upload failure from python client (threading)

Description

from collaborator:

">>> syn = synapseclient.Synapse()
>>> syn.login('jishuxu@broadinstitute.org','XXXXXXXX')

UPGRADE AVAILABLE

A more recent version of the Synapse Client (1.5.1) is available. Your version (1.2) can be upgraded by typing:
pip install --upgrade synapseclient

Python Synapse Client version 1.5.1 release notes

Adds Python 3 compatibility and enhanced file upload reliability

Welcome, jishuxu@broadinstitute.org!
>>> results = syn.query('SELECT id, name FROM entity WHERE parentId=="syn5848791"')
>>> results
{u'totalNumberOfResults': 0, u'results': []}
>>> results = syn.query('SELECT id, name FROM entity WHERE parentId=="syn2701448"')
>>> results
{u'totalNumberOfResults': 46, u'results': [{u'entity.id': u'syn5848791', u'entity.name': u'RawData'}, {u'entity.id': u'syn2701453', u'entity.name': u'ill450kAnno_finalQC_chr1.txt'}, {u'entity.id': u'syn2701469', u'entity.name': u'ill450kAnno_finalQC_chr10.txt'}, {u'entity.id': u'syn2701470', u'entity.name': u'ill450kAnno_finalQC_chr11.txt'}, {u'entity.id': u'syn2701471', u'entity.name': u'ill450kAnno_finalQC_chr12.txt'}, {u'entity.id': u'syn2701472', u'entity.name': u'ill450kAnno_finalQC_chr13.txt'}, {u'entity.id': u'syn2701473', u'entity.name': u'ill450kAnno_finalQC_chr14.txt'}, {u'entity.id': u'syn2701474', u'entity.name': u'ill450kAnno_finalQC_chr15.txt'}, {u'entity.id': u'syn2701475', u'entity.name': u'ill450kAnno_finalQC_chr16.txt'}, {u'entity.id': u'syn2701476', u'entity.name': u'ill450kAnno_finalQC_chr17.txt'}, {u'entity.id': u'syn2701477', u'entity.name': u'ill450kAnno_finalQC_chr18.txt'}, {u'entity.id': u'syn2701478', u'entity.name': u'ill450kAnno_finalQC_chr19.txt'}, {u'entity.id': u'syn2701461', u'entity.name': u'ill450kAnno_finalQC_chr2.txt'}, {u'entity.id': u'syn2701479', u'entity.name': u'ill450kAnno_finalQC_chr20.txt'}, {u'entity.id': u'syn2701480', u'entity.name': u'ill450kAnno_finalQC_chr21.txt'}, {u'entity.id': u'syn2701481', u'entity.name': u'ill450kAnno_finalQC_chr22.txt'}, {u'entity.id': u'syn2701462', u'entity.name': u'ill450kAnno_finalQC_chr3.txt'}, {u'entity.id': u'syn2701463', u'entity.name': u'ill450kAnno_finalQC_chr4.txt'}, {u'entity.id': u'syn2701464', u'entity.name': u'ill450kAnno_finalQC_chr5.txt'}, {u'entity.id': u'syn2701465', u'entity.name': u'ill450kAnno_finalQC_chr6.txt'}, {u'entity.id': u'syn2701466', u'entity.name': u'ill450kAnno_finalQC_chr7.txt'}, {u'entity.id': u'syn2701467', u'entity.name': u'ill450kAnno_finalQC_chr8.txt'}, {u'entity.id': u'syn2701468', u'entity.name': u'ill450kAnno_finalQC_chr9.txt'}, {u'entity.id': u'syn2701561', u'entity.name': u'ill450kMeth_chr10_740_imputed.txt'}, {u'entity.id': u'syn2701562', u'entity.name': u'ill450kMeth_chr11_740_imputed.txt'}, {u'entity.id': u'syn2701565', u'entity.name': u'ill450kMeth_chr12_740_imputed.txt'}, {u'entity.id': u'syn2701566', u'entity.name': u'ill450kMeth_chr13_740_imputed.txt'}, {u'entity.id': u'syn2701567', u'entity.name': u'ill450kMeth_chr14_740_imputed.txt'}, {u'entity.id': u'syn2703252', u'entity.name': u'ill450kMeth_chr15_740_imputed.txt'}, {u'entity.id': u'syn2703255', u'entity.name': u'ill450kMeth_chr16_740_imputed.txt'}, {u'entity.id': u'syn2703256', u'entity.name': u'ill450kMeth_chr17_740_imputed.txt'}, {u'entity.id': u'syn2703306', u'entity.name': u'ill450kMeth_chr18_740_imputed.txt'}, {u'entity.id': u'syn2703311', u'entity.name': u'ill450kMeth_chr19_740_imputed.txt'}, {u'entity.id': u'syn2701508', u'entity.name': u'ill450kMeth_chr1_740_imputed.txt'}, {u'entity.id': u'syn2703337', u'entity.name': u'ill450kMeth_chr20_740_imputed.txt'}, {u'entity.id': u'syn2703343', u'entity.name': u'ill450kMeth_chr21_740_imputed.txt'}, {u'entity.id': u'syn2703345', u'entity.name': u'ill450kMeth_chr22_740_imputed.txt'}, {u'entity.id': u'syn2701509', u'entity.name': u'ill450kMeth_chr2_740_imputed.txt'}, {u'entity.id': u'syn2701510', u'entity.name': u'ill450kMeth_chr3_740_imputed.txt'}, {u'entity.id': u'syn2701511', u'entity.name': u'ill450kMeth_chr4_740_imputed.txt'}, {u'entity.id': u'syn2701512', u'entity.name': u'ill450kMeth_chr5_740_imputed.txt'}, {u'entity.id': u'syn2701524', u'entity.name': u'ill450kMeth_chr6_740_imputed.txt'}, {u'entity.id': u'syn2701525', u'entity.name': u'ill450kMeth_chr7_740_imputed.txt'}, {u'entity.id': u'syn2701559', u'entity.name': u'ill450kMeth_chr8_740_imputed.txt'}, {u'entity.id': u'syn2701560', u'entity.name': u'ill450kMeth_chr9_740_imputed.txt'}, {u'entity.id': u'syn5843544', u'entity.name': u'tech_var_748qc_batch.txt'}]}
>>> syn.store(File(path=filetoupload,parent='syn5848791',synapseStore=True))

##################################################
Uploading file to Synapse storage
##################################################
..,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,..,Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 907, in store
mimetype=local_state.get('contentType', None))
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 1803, in _uploadToFileHandleService
return self._chunkedUploadFile(filename, mimetype=mimetype)
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/client.py", line 2003, in _chunkedUploadFile
exceptions._raise_for_status(response, verbose=self.debug)
File "/broad/dejagerlab/cogdec/jishu/softwares/Python/lib/python2.7/site-packages/synapseclient-1.2-py2.7.egg/synapseclient/exceptions.py", line 124, in _raise_for_status
raise SynapseHTTPError(message, response=response)
synapseclient.exceptions.SynapseHTTPError: 400 Client Error: Bad Request"

Environment

None

Activity

Show:
Larsson Omberg
April 1, 2016, 11:54 PM

Based on the statement, from SYNPY-359, that the upload worked for the first 8 files before failing I think it is worth to try Kenny's suggestion and explicitly terminate the Pool at the end of multipart_upload.

If there is a limit of 64 threads, 8 files at 8 threads would deplete the pool.

Kenneth Daily
April 1, 2016, 6:02 PM

I used an EC2, installed Python 3.4.0, wrote an explicit for loop in IPython interpreter over 10 500MB files and stored to Synapse - no issues.

Ben Logsdon
April 1, 2016, 5:58 PM

Ran the following command in a python virtual environment on Belltown & as a job submitted to Sodo, and it worked fine with Python 1.5.1

synapse add --parentId syn3273961 foo.txt

Larsson Omberg
April 1, 2016, 5:54 PM

duplicate

Larsson Omberg
March 31, 2016, 5:04 PM

Also, Kenny pointed out that we don't call Pool.close() or Pool.terminate(). These should not be necessary as Pool.terminate() is called upon garbage collection but if there is a memory leak (as might be the case in the older version of python) this wouldn't happen.

Fixed

Assignee

Larsson Omberg

Reporter

Ben Logsdon

Labels

None

Validator

Ben Logsdon

Development Area

None

Release Version History

None

Slack Channel

None

Fix versions

Priority

Critical