External File Management

Synapse supports upload/retrieval to file storage owned by users. An example is that of a private SFTP server.

Project Settings

Upload Destination on File Containers (Projects, Folders)

Synapse will generate a suggested file hierarchy into which the client uploads content. The client is responsible for generating missing folders.

Conventions for external URLs

Conventions for storing crendentials

(Cross reference to C-4 page)

Proposed Services Revision

(from Marcel's email of 11/17)

The current sequence for uploading a file goes like this:

Ask server for upload destinations for a synapse container

Get 1 or more upload destinations, of either S3 or external (SFTP)

Either let user choose, or upload to first default option

Upload to external file handle or ask for signed S3 url (using the upload destination)

This has a couple of problems, the most problematic one is that it doesn't handle the case where the R or Python client downloads a file, modifies it and then reuploads it again. We want the upload to got to the same destination, but if there is more than one option, that doesn't quite work.

Proposed solution (backwards compatible with the about-to-be released clients):

API

1) Add an API that gives a list of possible upload locations by ID and label for a given synapse container:

UploadDestinationLocation {

String uploadId;

String label;

}

UploadDestinationLocation[] getUploadDestinationLocations(String parentId);

2) Add an API that gives a single specific upload destination by uploadId (parentId is redundant, but used for extra check against the valid uploadIds):

UploadDestination getUploadDestination(String parentId, String uploadId)

Object Model

Add uploadId as a new optional field to an ExternalFileHandle (perhaps S3FileHandle has this field too).

Note: Need ExternallyManagedFileHandle to distinguish from non-managed ExternalFileHandle

Sequence

For uploading a new file:

Get upload destination locations for a synapse container

Have the user choose if more than one

For each new file:

Get upload destination for container and chosen upload Id

Upload to external file handle or ask for signed S3 url (using the upload destination)

Create new file handle

For modifying a file

Download file using file handle id

modify

Get upload destination for container and previous file handle id

Upload to external file handle or ask for signed S3 url (using the upload destination)

Create new file handle