frequent upload/download failures for large BAM datasets

Description

For large BAM file datasets, user reports having to re-loop through uploads/downloads many times to recover from multiple failures each loop. Representative error below for uploads to syn5016925.

Uploading [########------------]42.15% 3.8GB/9.0GB /lrlhps/scratch/world/ngsdump/staging/merge_bam/rosmap/580_120521.bam retrying exception: ConnectionError HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /proddata.sagebase.org/3330804/30895ffc-8b77-4728-8d15-1aee43aef02b/580_120521.bam/779?AWSAccessKeyId=……&Expires=1446506146&Signature=lL4gKynzU81DOBRibaXpDT%2FyQsE%3D (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))
Uploading [##########----------]49.71% 5.1GB/10.3GB /lrlhps/scratch/world/ngsdump/staging/merge_bam/rosmap/171_120423.bam retrying exception: ConnectionError HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /proddata.sagebase.org/3330804/8ea33ada-72af-4bf6-91a3-74efd8469a3d/171_120423.bam/1053?AWSAccessKeyId=……&Expires=1446508723&Signature=Q38kO3Ti39y%2BgbhfKiaPEwgF8oo%3D (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))

Environment

None

Activity

Show:
Mike Kellen
January 22, 2016, 9:21 PM

New file upload mechanism should fix this. Please stress test 1.5 file upload.

Kristen Dang
January 12, 2016, 7:40 PM

Some errors that I encountered when uploading a series of small BAMs to Synapse:

Uploading [###############-----]76.64% 3.6GB/4.8GB /mnt/1796012098_B.FCD1LUUACXX_L1_IATCACG.snap.bam File "/usr/local/lib/python2.7/dist-packages/synapseclient/client.py", line 1959, in _chunkedUploadFile
p.map(upload_one_chunk_with_retry,chunkNumbers)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
raise self._value
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

Uploading [###############-----]74.84% 2.9GB/3.8GB /mnt/1796012074_B.FCD1K78ACXX_L4_IGGCTAC.snap.bam Traceback (most recent call last):
File "./run_transfer.py", line 22, in <module>
newEnt = syn.store(newEnt)
File "/usr/local/lib/python2.7/dist-packages/synapseclient/client.py", line 944, in store
fileSize=local_state.get('fileSize', None))
File "/usr/local/lib/python2.7/dist-packages/synapseclient/client.py", line 1779, in _uploadToFileHandleService
return self._chunkedUploadFile(filename, mimetype=mimetype)
File "/usr/local/lib/python2.7/dist-packages/synapseclient/client.py", line 1959, in _chunkedUploadFile
p.map(upload_one_chunk_with_retry,chunkNumbers)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
raise self._value
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

Mike Kellen
November 5, 2015, 11:41 PM

Moving to python client as I'm guessing this is an error handling issue. Please reassign if that's not the case.

I think next release of python client should include look at any open file transfer reliability work - we switched the retry logic in the R client and I'm not sure it's having the same issues anymore.

Fixed

Assignee

Chris Bare

Reporter

Kristen Dang

Labels

None

Validator

Kristen Dang

Development Area

None

Release Version History

None

Slack Channel

None

Components

Fix versions

Priority

Critical