User getting 503 error when downloading

Description

We have a user in the AMP-AD project who is using the python client (v1.3) to download a lot of BAM files that are several GB each. They are getting about 2MB/sec transfer rate and the job failed with a 503. They are running the download from their internal hardware (not cloud).

Comments from user:

It managed to pull 48/60 BAM files before timing out:

ConnectionError: HTTPSConnectionPool(host='repo-prod.prod.sagebase.org', port=443): Max retries exceeded with url: /repo/v1/entity/syn4487045/version/1/file (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))

I don’t see a way to resume. I re-ran it earlier and it seemed to just overwrite existing files.

Phil is correct that the speed was consistently about 2 MB/s. It took 2 whole days to pull the 48 BAM files (400 GB).

Environment

None

Activity

Show:
Chris Bare
October 19, 2015, 6:48 PM

Chris Bare
October 19, 2015, 6:47 PM

This looks like throttling on the request to Synapse for a signed S3 URL. Retry was bypassed in this case; fixed as SYNPY-290.

Duplicate

Assignee

Chris Bare

Reporter

Kristen Dang

Labels

None

Validator

Kristen Dang

Development Area

None

Release Version History

None

Slack Channel

None

Components

Fix versions

Priority

Major