Downloading table file attachments can be slow/inefficient

Description

Related to https://sagebionetworks.jira.com/browse/SYNR-1024

I'm downloading 1800 files which average about 30MB each from a table using downloadTableColumns.

There is a 1GB limit in this REST call http://docs.synapse.org/rest/POST/file/bulk/async/start.html
so the client fetches a gigabyte worth of the files each time.

So the process looks like this:

The "Processing FileHandleId" lines take significantly longer than the "table_Downloading" lines. And it looks like it reprocesses dataFileHandleIds that haven't been downloaded by my local client yet, leading to a lot of redundant processing (or, at least, it seems this way). I'd like to be able to circumvent this issue by allowing bulk file downloads larger than 1GB, hopefully resulting in just a single "Processing FileHandleId" line and a single "table_Downloading" line.

Environment

None

Assignee

Unassigned

Reporter

Phil Snyder

Labels

None

Validator

Phil Snyder

Development Area

None

Release Version History

None

Epic Link

Components

Priority

Major
Configure