downloaded files are reset when disk space exhausted

Description

From the Synapse forum:
https://www.synapse.org/#!Synapse:syn5637528/discussion/threadId=7464

I am trying to download via CLI, and using a recently updated client, the downloaded files keep erasing themselves and starting over once they get to 100MB. It will do this with six files simultaneously, and the "Downloaded" status bar will reverse, and the downloaded size will get smaller, not bigger(?!). Eventually, after about an hour, it will error out with:

ValueError: File download failed during sync
synapseclient 2.2.0 from bioconda

Environment

bioconda

Activity

Show:
Bruce Hoff
October 14, 2020, 9:48 PM

Tried another ubuntu instance:

... Looks like I have to update setuptools.

Bruce Hoff
October 14, 2020, 9:50 PM

I'm not going to be able to finish this today. I can try again Friday. If you need it sooner, please ask someone else to validate.

Jordan Kiang
October 14, 2020, 9:55 PM

An equivalent Mac command for a ramdisk would be

The volume would then be /Volumes/ramdisk

However the setuptools issue itself is concerning. I just installed it on a fresh ubuntu ec2 venv without aparent issue.

Jordan Kiang
October 14, 2020, 11:02 PM
Edited

Oh I see the version of Python you were using was 3.4 which is not supported by synapseclient (and hasn’t been since synapseclient 1.8.2). A python version with a reasonably modern setuptools will not allow you to even attempt the install of an unsupported version from pip, but this from your output indicates that this old python 3.4 installation in Ubuntu 14.04 LTS (Ubuntu 14.04 LTS is itself no longer supported) is using the older distutils which doesn’t honor the python_requires option and attempts the install anyway.

Sorry for the trouble but the validation steps should work on a supported Python version.

Alternately here is the Mac equivalent of the above steps (Python 3.7 is the default on Mac Catalina).

Bruce Hoff
October 16, 2020, 6:15 PM
Edited

Sorry for the delay. I validated using a docker container:

I verified that the download 'hangs'. In fact it made the Docker engine hang then crashed my computer.

Next step:

Nice: Exited cleanly with a clear message. Did NOT crash the Docker engine or my computer!

Assignee

Jordan Kiang

Reporter

Bruce Hoff

Labels

None

Validator

Bruce Hoff

Development Area

Synapse Core Infrastructure

Release Version History

None

Components

Sprint

None

Fix versions

Affects versions

Priority

Critical
Configure