External File Management
Synapse supports upload/retrieval to file storage owned by users. An example is that of a private SFTP server.
Project Settings
Upload Destination on File Containers (Projects, Folders)
Synapse will generate a suggested file hierarchy into which the client uploads content. The client is responsible for generating missing folders.
Conventions for external URLs
Conventions for storing crendentials
(Cross reference to C-4 page)
Proposed Services Revision
(from Marcel's email of 11/17)
The current sequence for uploading a file goes like this:
Ask server for upload destinations for a synapse container
Get 1 or more upload destinations, of either S3 or external (SFTP)
Either let user choose, or upload to first default option
Upload to external file handle or ask for signed S3 url (using the upload destination)
This has a couple of problems, the most problematic one is that it doesn't handle the case where the R or Python client downloads a file, modifies it and then reuploads it again. We want the upload to got to the same destination, but if there is more than one option, that doesn't quite work.
Proposed solution (backwards compatible with the about-to-be released clients):
API
1) Add an API that gives a list of possible upload locations by ID and label for a given synapse container:
UploadDestinationLocation {
String uploadId;
String label;
}
UploadDestinationLocation[] getUploadDestinationLocations(String parentId);
2) Add an API that gives a single specific upload destination by uploadId (parentId is redundant, but used for extra check against the valid uploadIds):
UploadDestination getUploadDestination(String parentId, String uploadId)
Object Model
Add uploadId as a new optional field to an ExternalFileHandle (perhaps S3FileHandle has this field too).
Note: Need ExternallyManagedFileHandle to distinguish from non-managed ExternalFileHandle
Sequence
For uploading a new file:
Get upload destination locations for a synapse container
Have the user choose if more than one
For each new file:
Get upload destination for container and chosen upload Id
Upload to external file handle or ask for signed S3 url (using the upload destination)
Create new file handle
For modifying a file
Download file using file handle id
modify
Get upload destination for container and previous file handle id
Upload to external file handle or ask for signed S3 url (using the upload destination)
Create new file handle