download failure with syncFromSynapse

Description

Report from user, see attachments:

I'm trying to downloading the round 2 data using Python synapseclient. I've tried several times and all failed after a few bam files downloaded.

The script I used was the following:

import synapseclient
import synapseutils

syn = synapseclient.Synapse(debug=True)
syn.login('my_username','my_password')
files = synapseutils.syncFromSynapse(syn, 'syn10141118')

This script stoped downloading after running for about 25 minutes. The STDOUT outputs are the attached. Would you please to have a look on them and give me some guide? Thank you so much .

Environment

None

Activity

Show:
Kristen Dang
July 26, 2017, 10:49 PM

Hi Ziming, it is sensitive data, but file sizes are going to be 10-20 GB each.

Ziming Dong
July 25, 2017, 7:05 PM

If possible, I'd like to know the synapse Id of the file downloaded so I can try to replicate the issue. If it has sensitive data, just let me know the file size and I can test on a dummy file of that size instead.

From reading the logs, It looked like the client had a inconsistent connection to amazon's S3. By the time it was able to connect, the provided presigned url had already expired. At the very end of the log (work.liwenhui.sh.20170721-0805.nohup), the client managed to download 140MB of data before the connection abruptly ended.

If you haven't already done so, please try the download again making sure that you are on the latest version of the python client (1.7.2 improved the download retry logic), and are using a wired Ethernet connection (WiFi can be flaky at times for larger file downloads).

Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Jordan Kiang

Reporter

Kristen Dang

Labels