Add synapseutil for migrating an existing external S3 bucket into Synapse

Description

Use case as discussed here:
https://sagebionetworks.slack.com/archives/C0AFCTGEQ/p1594922771322600

1. There is an external S3 bucket that has existing objects that have not yet been added to Synapse
2. The existing bucket itself is the storage location that will be used with Synapse (i.e. we are not copying these objects to a new bucket, but creating file handles for the objects where they already exist).
3. We want to add these objects as Files to a new Synapse Folder whose storage location is configured as the storage location above in #2
4. Files should be created along with meta data (annotations etc) as described in an input manifest if one is provided, or alternately all files in a listing of a bucket should be created (without any additional meta data).

Environment

None

Activity

Show:
Bruce Hoff
July 16, 2020, 7:29 PM

This sounds great.

Is there a use case in which the bucket owner (Alice) wants to share the indexed files with Bob, and Bob wants to access the files using the S3 CLI / boto?

Assignee

Jordan Kiang

Reporter

Jordan Kiang

Labels

None

Validator

Bruce Hoff

Development Area

Synapse Core Infrastructure

Release Version History

None

Components

Priority

Major
Configure