Enable efficient batch-retrieval of Synapse annotations

Description

Feature request

What's the most efficient (or fastest) way to retrieve user-defined annotations for a large set of Synapse entities (e.g. over 1000)?

Context

We want to generate a table of annotations for all files within a Synapse project/folder. Currently, I'm iteratively calling the /entity/{entity}/annotations2 endpoint because to my knowledge, there's no equivalent POST endpoint where I can provide a list of entity IDs. In practice, each call takes 139 ms on average, which quickly adds up for large projects/folders (see profiling results). This functionality will be run as part of a web app, hence the pressure to keep run times below one minute.

Why not file views?

I realize that "a table of annotations for all files within a Synapse project/folder" is exactly what file views are. However, in this situation, we cannot assume that the users have edit-access to the parent project, and unfortunately, file views are bound to projects. Therefore, the only workaround I can think of is creating a new project with the purpose of creating a file view with the purpose of aggregating annotations across a project/folder, which is hacky at best.

This approach can be improved slightly by deleting the project once the tool has downloaded the annotations. That said, I prefer not to burden Synapse with short-lived projects. There's also the issue of potentially cluttering the user's account with projects if an error happens before the project clean-up (although steps can be taken to mitigate this risk, like a try-finally statement in Python).

Potential solutions

In no particular order, here are some unsolicited ideas for a long-term solution (assuming there isn't a better way to tackle this with existing endpoints/tools):

  • Allow file views to be bound to folders (I realize this has UI challenges)

  • Enable user-bound file views (like how users/orgs can have projects or packages in GitHub)

  • Create endpoint that creates an temporary unbound (or bound to an invisible project) file view behind the scenes for the purpose of generating a table of annotations

  • Create a POST endpoint for retrieving annotations for batches of Synapse entities (as long as the batch limit is at least 100)

Ultimately, I defer to the Platform deities to decide what's the best course of action.

Environment

None

Activity

Show:
Jay Hodgson
March 1, 2021, 10:15 PM

Thank you for the clarification about location in the hierarchy vs scope, as I was thinking the same thing.

I also do not have a strong opinion. We could start showing Tables/Views in the entity browser in the web UI, but the “Files” tab text would need to change.

Relying on the design team ( ) for guidance around UX and expectations.

John Hill
March 1, 2021, 10:03 PM

I do not have a strong opinion about adding views to folders.

However, I would like to clarify that a view's location has no bearing on what can be shown in the view. For example, I can create a view in project A and set its scope to be project B. Even though this view would be in project A, it would only show files from project B.

Milen Nikolov
March 1, 2021, 9:04 PM

As discussed on slack - I initially thought this was a bug when accidentally bound a fileview to a folder and it didn’t show on the web UI. But it sounds like it is a useful feature - perhaps having the ability to bind fileviews to folders is ok; and maybe there can be ‘admin’ toggle feature for showing/hiding fileviews in the web UI (though not sure how that would square w/ the current ‘Table’ that lists tables under ‘Project’).

Bruce Hoff
March 1, 2021, 8:56 PM

, , Please see 's comment above: Apparently you can add a view to a folder. It does not show up in a project. The question is whether we want to support or to block this.

Bruno Grande
March 1, 2021, 8:42 PM

Apparently, I can bind a file view to a folder. Because I can assume that the user has edit-access for a local folder (but not necessarily at the project level), this solves my problem quite elegantly. These folder-bound file views cannot be found using the web client, which I see as a perk because I don’t want these to appear on the website.

For more information, check out this conversation on Slack.

Won't Do

Assignee

Bruce Hoff

Reporter

Bruno Grande

Validator

Bruno Grande

Priority

Major

Labels

None

Development Area

None

Sprint

None

Fix versions

None

Release Version History

None

Story Points

None

Epic Link

None

Slack Channel

None