Support create External File Handle in Python client

Description

Previous title: Allow files in custom storage locations to be uploaded using sync function

===

I was working on linking and annotating files that were stored in an external S3 bucket, and I made a manifest file. I found the manifest structure is similar to the ones for uploading files in a local directory. It made me wonder if we can allow files in external custom storage locations to be uploaded using the `syncToSynapse` utility function?

Below is a demo s3 manifest file.

path

key

parent

name

assay

dataType

s3://testkeyname/rnaSeq/foo.bam

rnaSeq/foo.bam

syn123466

foo.bam

rnaSeq

geneExpression

s3://testkeyname/rnaSeq/info.csv

rnaSeq/info.csv

syn123466

info.csv

Not Applicable

metadata

Environment

None

Activity

Show:
Sara Gosline
August 12, 2019, 6:21 PM

Also tagging as this is a big use case for HTAN.

Larsson Omberg
June 14, 2019, 1:08 AM

We should probably not use the deprecated S3 syntax though and would also suggest adding an optional URL parameter for exteranalObjectStore file handles.

Larsson Omberg
June 14, 2019, 1:07 AM

Tagging , and as we had discussed the need for this functionality in a few places today. It looks like this functionality has been replicated by Phil, William, Kenny and Milen already.

I can imagine a syncFromS3.

 

 

Kimyen Truong
June 11, 2019, 6:59 PM

Code from

 

Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Jordan Kiang

Reporter

Xindi Guo

Validator

Xindi Guo