Add download recovery by using the 'Range': 'bytes=xxx-xxx' header

Description

None

Environment

None

Activity

Show:
Larsson Omberg
August 27, 2016, 7:52 AM

Retries are never attempted for file entity downloads as the filehandle id is never passed to the _download function.

Larsson Omberg
August 27, 2016, 4:56 PM

Just ran a TB download test and it failed with:

Downloading [#########-----------]47.21% 3.2GB/6.8GB (7.5MB/s) MTa451-7.bam Traceback (most recent call last):
File "/gluster/home/lomberg/.virtualenvs/openblas/bin/synapse", line 9, in <module>
load_entry_point('synapseclient==1.5.1', 'console_scripts', 'synapse')()
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/_main_.py", line 775, in main
perform_main(args, syn)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/_main_.py", line 750, in perform_main
args.func(args, syn)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/_main_.py", line 142, in get
_recursiveGet(args.id, args.downloadLocation, syn) #Todo should be updated with destination folder instead of '.'
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/_main_.py", line 133, in _recursiveGet
_recursiveGet(result['entity.id'], new_path, syn)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/_main_.py", line 135, in _recursiveGet
syn.get(result['entity.id'], downloadLocation=path)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 696, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 872, in _getWithEntityBundle
file_handle_id=entityBundle['entity']['dataFileHandleId']))
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 1763, in _downloadFileEntity
return self._downloadFile(url, destination, file_handle_id, entity.get("md5", None))
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 1778, in _downloadFile
return returnDict(self._download_with_retries(url, destination, file_handle_id, expected_md5))
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 1786, in _download_with_retries
return self._download(url, destination, file_handle_id, expected_md5)
File "/gluster/home/lomberg/Dropbox/Sage/synapsePythonClient/synapseclient/client.py", line 1897, in _download
for nChunks, chunk in enumerate(response.iter_content(FILE_BUFFER_SIZE)):
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/requests/models.py", line 657, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 326, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 282, in read
data = self._fp.read(amt)
File "/gluster/toolbox/python/2.7.5/lib/python2.7/httplib.py", line 567, in read
s = self.fp.read(amt)
File "/gluster/toolbox/python/2.7.5/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 179, in recv
data = self.connection.recv(*args, **kwargs)
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1320, in recv
self._raise_ssl_error(self._ssl, result)
File "/gluster/home/lomberg/.virtualenvs/openblas/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1178, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

Larsson Omberg
August 29, 2016, 3:17 AM

To be conservative we will retry ALL exceptions on downloads at least 5 times. And more times if there was progress between attempts.

This logic is added by changing:

  • except (requests.exceptions.BaseHTTPError, requests.exceptions.RequestException) as ex:
    + except Exception as ex: #We will pass A downloadError always and proceed with retry
    in _download.

Larsson Omberg
August 29, 2016, 3:48 AM

Pushed changes to expanded retries in Pull request #319

Larsson Omberg
August 29, 2016, 4:43 PM

sion/2/file after progressing 150994944 bytes

Downloading [####################]100.00% 254.3MB/254.3MB (53.5MB/s) MN560.all.bw Done...
Downloading [####################]100.00% 153.8MB/153.8MB (13.9MB/s) MN563.all.bw Done...
Downloading [####################]100.00% 95.1MB/95.1MB (12.5MB/s) MN567.all.bw Done...
Downloading [####################]100.00% 215.3MB/215.3MB (18.9MB/s) MN571.all.bw Done...
Downloading [####################]100.00% 248.9MB/248.9MB (18.3MB/s) MN572.all.bw Done...
retrying on status code: 500
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>A7917E2D71380D7E</RequestId><HostId>TAeLaVJuYFyaPJ4vRNpQT9n4XmOZvzHDY+khaVpCA2LaIzj5nxEGqhqD4nMbIKiDx35VFR5NbFI=</HostId></Error>
total wait time 0 seconds
... Retrying in 1.0 seconds...
Downloading [####################]100.00% 224.3MB/224.3MB (26.4MB/s) AC7_A3.all.bw Done...

Assignee

Chris Bare

Reporter

Larsson Omberg

Labels

None

Validator

Larsson Omberg

Development Area

None

Release Version History

None

Components

Fix versions

Priority

Critical
Configure