improve download speeds to be comparable to AWS

Description

Our data sizes are increasing, specifically for whole genome sequencing (500GB-1TB per file). Our download speeds are significantly slower than using AWS clients - my tests show at least 4x slower. In cases where we have data in external S3 buckets, our users are circumventing Synapse to get improved download speeds, and are (rightfully) complaining that data that does reside in Synapse S3 storage is slow to download when they know what kinds of speeds S3 can offer.

We have made significant improvements to data upload, and has indicated that is likely is possible to make similar improvements to download as well, but not trivially.

Environment

None

Assignee

Robert Minneker

Reporter

Kenneth Daily

Labels

Validator

Kenneth Daily

Development Area

None

Release Version History

None

Sprint

None

Fix versions

Priority

Critical
Configure