I want to migrate my existing non-empty folders to use STS storage locations

Description

We have 70TB of data across various projects and folders in Synapse Storage. We want to enable STS on this data without having to download and re-upload all of this data (which would be very expensive!) We would also like to keep existing Synapse IDs and metadata on our File Handles.

Proposed technical solution:

  1. Remove the restriction that STS storage locations can only be created in empty folders.

  2. API to move a FileEntity/FileHandle from one storage location to another.

    1. This may involve moving a file from one S3 key to another.

    2. Add the restriction that both storage locations must be in the same S3 bucket, to avoid possible file transfer costs.

    3. Check that moving a file within the same S3 bucket is cheap or free.

  3. Client-side helper class which calls the API from step 2 in a loop, to allow us to migrate all files in a folder to using STS.

    1. Would need docs to set the expectation that, if there are a lot of files in the folder, this script may need to be run repeatedly until the folder is in a consistent state.

We should also ensure that we don’t lift too many restrictions, so as to cause folders to drift towards being less consistent rather than more consistent. For example, we should still block adding files to an STS-enabled folder that aren’t part of the STS storage location.

Environment

None

Activity

Show:
Marco Marasca
March 3, 2021, 1:17 AM

I left it open because we are still investigating on to decide how to proceed. We can close it if we rename it and disconnect it from the STS related issues.

Ryan Luce
March 3, 2021, 1:13 AM

Is this epic still open? I’m wondering if the features connected to it that aren’t yet done can be disconnected from it. We can leave it in place, just wanted to check.

Marco Marasca
October 27, 2020, 7:14 PM

, the backend has all the needed bits in place, the final part is the integration with the python client: . Feel free to schedule a meeting!

Kelsey Montgomery
October 27, 2020, 7:06 PM

what is the status of this feature? It would be helpful to meet and review it when it is nearing completion.

Assignee

Marco Marasca

Reporter

Dwayne Jeng

Validator

Kelsey Montgomery

Priority

Major

Labels

Development Area

Synapse Core Infrastructure

Sprint

Fix versions

None

Release Version History

None

Story Points

None

Slack Channel

None

Epic Name

S3 Data Migration Support