Parallelize download syncs

Description

We can parallelize multiple downloads from a syncToSynapse by using a ThreadPoolExecutor shared by the downloads (currently we support paraellized part download of file chunks, but download the synced files themselves serially).

Environment

None

Activity

Show:
Jordan Kiang
July 22, 2020, 5:33 PM
Edited

Implementation details here: https://github.com/Sage-Bionetworks/synapsePythonClient/pulls

Some performance figures for this change. These were all run on an t2.xlarge (4 cores) in aws-us-east-1.

The client tested configurations are:

  1. SYNPY-1074 (this change)

  2. The current 2.1.1 Synapse client configured for single-threaded downloads (currently the default unless overridden)

  3. The current 2.1.1 Synapse client configured for multi-threaded downloads using the existing multi threaded download functionality (which supports multi threaded part downloads of any single file, but not multi threaded downloads across a sync of files, those are still downloaded serially).

Test 1

This change should be most beneficial to syncing folders with many small files, since they can be concurrently downloaded. syn22252168 is a Synapse project with 400 text files smaller than 50 bytes distributed in a file hierarchy 3 folders deep with 10 files in every subfolder. This file hierarchy can be recreated with the attached script. A recursive synapse sync was run on this project:

elapsed seconds:

 

2.1.1 single-threaded

2.1.1 multi-threaded

SYNPY-1074

1

124.82

200.23

59.22

2

121.88

205.93

55.23

3

124.84

193.30

59.77

Average

123.82

199.82

58.08

 

In this test the existing client when configured with the optional multi threaded mode actually performs worse than the client in single threaded mode because of the overhead of launching and running the download in threads isn’t efficient for the small files, and in both cases the files themselves are serially downloaded.

The changes with SYNPY-1074 allow files to be downloaded in parallel, however the individual files are downloaded with a single thread because the download code checks the reported size before deciding which implementation to use.

Test 2

This test is a real word folder syn21903917 with a mix of large and small files that was reported to have problems as part of SYNPY-1078. Some files are large video files of multiple gigabytes, some are small text files of a few bytes, and some images are a few hundred kB. The project itself is many terabytes, these tests downloaded 50GB of data before filling the available disk at which point the elapsed time was measured.

elapsed seconds

 

2.1.1 single-threaded

2.1.1 multi-threaded

SYNPY-1074

1

1875.34

1554.4

728.21

2

1818.2

1621.82

734.10

Average

1846.77

1588.11

731.16

 

In this case because of the larger files 2.1.1 multi threaded performs better than 2.1.1 single threaded, but the changes for SYNPY-1074 outperform both.

Test 3

The last test is a download of a single large file 7GB, syn21919857.

.This performance shouldn’t materially change but since the multi threaded download code was refactored to support a shared thread pool while syncing we want to ensure that performance hasn’t degraded.

elapsed seconds

 

2.1.1 single-threaded

2.1.1 multi-threaded

SYNPY-1074

1

81.35

43.56

44.21

2

77.8

44.87

44.18

3

79.78

42.46

43.47

Average

79.64

43.63

43.95

As expected/hoped, the changes do not materially affect performance over the existing multi threaded download in this case.

Jordan Kiang
July 25, 2020, 12:08 AM

This change has been merged to develop and will be in included in the next release candidate for validation.

Bruce Hoff
August 21, 2020, 8:41 PM

Can you think of a good internal user to assign as 'validator', i.e. someone who has a need to download batches of files in their work?

Jordan Kiang
August 26, 2020, 5:39 PM

I’ve asked Tobias Ross of the German Cancer Research Center to verify improved performance since he had previously had issues with sync performance (now partially resolved with a previous bug fix).

Jordan Kiang
August 28, 2020, 3:12 PM

Received the following back:

I followed up to ask if they collected any concrete numbers and will add them to the issue if received, but marking this as closed for validation/release purposes.

Assignee

Jordan Kiang

Reporter

Jordan Kiang

Labels

None

Validator

None

Development Area

Synapse Core Infrastructure

Release Version History

None

Time tracking

0m

Time remaining

0m

Components

Fix versions

Priority

Major
Configure