upload failure from python client (threading)
Description
from collaborator:
">>> syn = synapseclient.Synapse()
>>> syn.login('jishuxu@broadinstitute.org','XXXXXXXX')
UPGRADE AVAILABLE
A more recent version of the Synapse Client (1.5.1) is available. Your version (1.2) can be upgraded by typing:
pip install --upgrade synapseclient
Python Synapse Client version 1.5.1 release notes
Adds Python 3 compatibility and enhanced file upload reliability
Welcome, jishuxu@broadinstitute.org!
>>> results = syn.query('SELECT id, name FROM entity WHERE parentId=="syn5848791"')
>>> results
{u'totalNumberOfResults': 0, u'results': []}
>>> results = syn.query('SELECT id, name FROM entity WHERE parentId=="syn2701448"')
>>> results
{u'totalNumberOfResults': 46, u'results': [{u'entity.id': u'syn5848791', u'entity.name': u'RawData'}, {u'entity.id': u'syn2701453', u'entity.name': u'ill450kAnno_finalQC_chr1.txt'}, {u'entity.id': u'syn2701469', u'entity.name': u'ill450kAnno_finalQC_chr10.txt'}, {u'entity.id': u'syn2701470', u'entity.name': u'ill450kAnno_finalQC_chr11.txt'}, {u'entity.id': u'syn2701471', u'entity.name': u'ill450kAnno_finalQC_chr12.txt'}, {u'entity.id': u'syn2701472', u'entity.name': u'ill450kAnno_finalQC_chr13.txt'}, {u'entity.id': u'syn2701473', u'entity.name': u'ill450kAnno_finalQC_chr14.txt'}, {u'entity.id': u'syn2701474', u'entity.name': u'ill450kAnno_finalQC_chr15.txt'}, {u'entity.id': u'syn2701475', u'entity.name': u'ill450kAnno_finalQC_chr16.txt'}, {u'entity.id': u'syn2701476', u'entity.name': u'ill450kAnno_finalQC_chr17.txt'}, {u'entity.id': u'syn2701477', u'entity.name': u'ill450kAnno_finalQC_chr18.txt'}, {u'entity.id': u'syn2701478', u'entity.name': u'ill450kAnno_finalQC_chr19.txt'}, {u'entity.id': u'syn2701461', u'entity.name': u'ill450kAnno_finalQC_chr2.txt'}, {u'entity.id': u'syn2701479', u'entity.name': u'ill450kAnno_finalQC_chr20.txt'}, {u'entity.id': u'syn2701480', u'entity.name': u'ill450kAnno_finalQC_chr21.txt'}, {u'entity.id': u'syn2701481', u'entity.name': u'ill450kAnno_finalQC_chr22.txt'}, {u'entity.id': u'syn2701462', u'entity.name': u'ill450kAnno_finalQC_chr3.txt'}, {u'entity.id': u'syn2701463', u'entity.name': u'ill450kAnno_finalQC_chr4.txt'}, {u'entity.id': u'syn2701464', u'entity.name': u'ill450kAnno_finalQC_chr5.txt'}, {u'entity.id': u'syn2701465', u'entity.name': u'ill450kAnno_finalQC_chr6.txt'}, {u'entity.id': u'syn2701466', u'entity.name': u'ill450kAnno_finalQC_chr7.txt'}, {u'entity.id': u'syn2701467', u'entity.name': u'ill450kAnno_finalQC_chr8.txt'}, {u'entity.id': u'syn2701468', u'entity.name': u'ill450kAnno_finalQC_chr9.txt'}, {u'entity.id': u'syn2701561', u'entity.name': u'ill450kMeth_chr10_740_imputed.txt'}, {u'entity.id': u'syn2701562', u'entity.name': u'ill450kMeth_chr11_740_imputed.txt'}, {u'entity.id': u'syn2701565', u'entity.name': u'ill450kMeth_chr12_740_imputed.txt'}, {u'entity.id': u'syn2701566', u'entity.name': u'ill450kMeth_chr13_740_imputed.txt'}, {u'entity.id': u'syn2701567', u'entity.name': u'ill450kMeth_chr14_740_imputed.txt'}, {u'entity.id': u'syn2703252', u'entity.name': u'ill450kMeth_chr15_740_imputed.txt'}, {u'entity.id': u'syn2703255', u'entity.name': u'ill450kMeth_chr16_740_imputed.txt'}, {u'entity.id': u'syn2703256', u'entity.name': u'ill450kMeth_chr17_740_imputed.txt'}, {u'entity.id': u'syn2703306', u'entity.name': u'ill450kMeth_chr18_740_imputed.txt'}, {u'entity.id': u'syn2703311', u'entity.name': u'ill450kMeth_chr19_740_imputed.txt'}, {u'entity.id': u'syn2701508', u'entity.name': u'ill450kMeth_chr1_740_imputed.txt'}, {u'entity.id': u'syn2703337', u'entity.name': u'ill450kMeth_chr20_740_imputed.txt'}, {u'entity.id': u'syn2703343', u'entity.name': u'ill450kMeth_chr21_740_imputed.txt'}, {u'entity.id': u'syn2703345', u'entity.name': u'ill450kMeth_chr22_740_imputed.txt'}, {u'entity.id': u'syn2701509', u'entity.name': u'ill450kMeth_chr2_740_imputed.txt'}, {u'entity.id': u'syn2701510', u'entity.name': u'ill450kMeth_chr3_740_imputed.txt'}, {u'entity.id': u'syn2701511', u'entity.name': u'ill450kMeth_chr4_740_imputed.txt'}, {u'entity.id': u'syn2701512', u'entity.name': u'ill450kMeth_chr5_740_imputed.txt'}, {u'entity.id': u'syn2701524', u'entity.name': u'ill450kMeth_chr6_740_imputed.txt'}, {u'entity.id': u'syn2701525', u'entity.name': u'ill450kMeth_chr7_740_imputed.txt'}, {u'entity.id': u'syn2701559', u'entity.name': u'ill450kMeth_chr8_740_imputed.txt'}, {u'entity.id': u'syn2701560', u'entity.name': u'ill450kMeth_chr9_740_imputed.txt'}, {u'entity.id': u'syn5843544', u'entity.name': u'tech_var_748qc_batch.txt'}]}
>>> syn.store(File(path=filetoupload,parent='syn5848791',synapseStore=True))
##################################################
Uploading file to Synapse storage
##################################################
..,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,..,Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 907, in store
mimetype=local_state.get('contentType', None))
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 1803, in _uploadToFileHandleService
return self._chunkedUploadFile(filename, mimetype=mimetype)
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 2003, in _chunkedUploadFile
exceptions._raise_for_status(response, verbose=self.debug)
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/exceptions.py", line 124, in _raise_for_status
raise SynapseHTTPError(message, response=response)
synapseclient.exceptions.SynapseHTTPError: 400 Client Error: Bad Request"
Environment
Activity
Based on the statement, from SYNPY-359, that the upload worked for the first 8 files before failing I think it is worth to try Kenny's suggestion and explicitly terminate the Pool at the end of multipart_upload.
If there is a limit of 64 threads, 8 files at 8 threads would deplete the pool.
I used an EC2, installed Python 3.4.0, wrote an explicit for loop in IPython interpreter over 10 500MB files and stored to Synapse - no issues.
Ran the following command in a python virtual environment on Belltown & as a job submitted to Sodo, and it worked fine with Python 1.5.1
synapse add --parentId syn3273961 foo.txt
duplicate
Also, Kenny pointed out that we don't call Pool.close() or Pool.terminate(). These should not be necessary as Pool.terminate() is called upon garbage collection but if there is a memory leak (as might be the case in the older version of python) this wouldn't happen.