Concurrent migration of entities sharing the same file handle can result in an error

Description

The migrate_indexed_files function of synapseutils.migrate_functions can migrate entities concurrently depending on the threading configuration of the client. If two files/versions share the same file handle and a multipart copy occurs concurrently it can produce a DuplicateKeyException in the backend:

However this exposes that the utility should detect multiple files/versions sharing the same file handle and only copy that file handle once, updating the meta data of the additional files/versions to use the existing file handle id.

encountered this with a test project that had a file entity with multiple identical versions, but it's also possible to reproduce with separate files sharing the same file handle. It is reproducible with the following:

Environment

all

Activity

Show:
Jordan Kiang
February 12, 2021, 7:40 PM

Closing this per Nicole's validation of https://sagebionetworks.jira.com/browse/SYNPY-1103

Fixed

Assignee

Jordan Kiang

Reporter

Jordan Kiang

Labels

None

Validator

Nicole Kauer

Development Area

Synapse Core Infrastructure

Release Version History

None

Slack Channel

None

Components

Fix versions

Affects versions

Priority

Major