Synapse provides physical storage for files using Amazon S3, however, you can configure your own custom storage locations as well. For example, data files can physically reside in your own S3 bucket, SFTP serversGoogle Cloud Storage Bucket, or a local file server using a proxy servers. Creating a custom storage location allows you greater ownership and control of your files, especially when you have a large amount of data or when additional restrictions need to be set on the data.
...
To register the bucket with Synapse, you also need to create an object that proves to the Synapse service that you own this bucket. This can be done by creating a file named owner.txt that contains a line or comma separated list of user identifiers that are allowed to register and upload to the bucket. Valid user identifiers are a numeric Synapse user ID or the numeric ID of a team that you are a member of.
...
The ID of the user or the team can be obtained by navigating to the user profile or to the team page. The ID is the numeric value shown in the browser URL bar after the Profile: or Team: prefixes.
...
For convienance, AWS Cloudformation can be used to provision a custom AWS S3 bucket for use with Synapse. Using this approach will result in the exact same bucket as described in Setup with AWS Console.
Instructions:
Download the CF template.
Use the AWS Command Line or AWS Console to execute the template which will automatically provision the bucket.
Example using the the awscli:
...
Please see the REST docs for more information on setting external storage location settings using our REST API.
Using SFTP
To setup an SFTP as a storage location, the settings on the project need to be changed, specifically the storageLocation needs to be set. This is best done using either R or Python but has alpha support in the web browser. Customize the code below to set the storage location as your SFTP server:
Python
| Code Block |
|---|
import synapseclient
import json
syn = synapseclient.login()
destination = { "uploadType":"SFTP",
"concreteType":"org.sagebionetworks.repo.model.project.ExternalStorageLocationSetting",
"description":"My SFTP upload location",
"supportsSubfolders":True,
"url":"sftp://your-sftp-server.com",
"banner":"A descriptive banner, tada!"}
destination = syn.restPOST('/storageLocation', body=json.dumps(destination))
project_destination = {"concreteType":"org.sagebionetworks.repo.model.project.UploadDestinationListSetting",
"settingsType":"upload"}
project_destination['projectId'] = PROJECT
project_destination['locations'] = [destination['storageLocationId']]
project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination))
|
R
| Code Block |
|---|
library(synapseClient)
synapseLogin()
projectId <- 'syn12345'
destination <- list(uploadType='SFTP',
concreteType='org.sagebionetworks.repo.model.project.ExternalStorageLocationSetting',
description='My SFTP upload location',
supportsSubfolders=TRUE,
url='https://your-sftp-server.com',
banner='A descriptive banner, tada!')
destination <- synRestPOST('/storageLocation', body=destination)
projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
settingsType='upload')
projectDestination$locations <- list(destination$storageLocationId)
projectDestination$projectId <- projectId
projectDestination <- synRestPOST('/projectSettings', body = projectDestination)
|
Using a Proxy to Access a Local File Server or SFTP Server
For files stored outside of Amazon, an additional proxy is needed to validate the pre-signed URL and then proxy the requested file contents. View more information here about the process as well as about creating a local proxy or a SFTP proxy.
Set Project Settings for a Local Proxy
You must have a key (“your_secret_key”) to allow Synapse to interact with the file system.
Python
| Code Block | ||
|---|---|---|
| ||
import synapseclient
import json
syn = synapseclient.login()
PROJECT = 'syn12345'
destination = {"uploadType":"PROXYLOCAL",
"secretKey":"your_secret_key",
"proxyUrl":"https://your-proxy.prod.sagebase.org",
"concreteType":"org.sagebionetworks.repo.model.project.ProxyStorageLocationSettings"}
destination = syn.restPOST('/storageLocation', body=json.dumps(destination))
project_destination ={"concreteType": "org.sagebionetworks.repo.model.project.UploadDestinationListSetting",
"settingsType": "upload"}
project_destination['locations'] = [destination['storageLocationId']]
project_destination['projectId'] = PROJECT
project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination))
|
R
| Code Block | ||
|---|---|---|
| ||
library(synapser)
synLogin()
projectId <- 'syn12345'
destination <- list(uploadType='PROXYLOCAL',
secretKey='your_secret_key',
proxyUrl='https://your-proxy.prod.sagebase.org',
concreteType='org.sagebionetworks.repo.model.project.ProxyStorageLocationSettings')
destination <- synRestPOST('/storageLocation', body=toJSON(destination))
projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
settingsType='upload')
projectDestination$locations <- list(destination$storageLocationId)
projectDestination$projectId <- projectId
projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination))
|
Set Project Settings for a SFTP Proxy
You must have a key (“your_secret_key”) to allow Synapse to interact with the file system.
Python
| Code Block |
|---|
import synapseclient
import json
syn = synapseclient.login()
PROJECT = 'syn12345'
destination = {"uploadType":"SFTP",
"secretKey":"your_secret_key",
"proxyUrl":"https://your-proxy.prod.sagebase.org",
"concreteType":"org.sagebionetworks.repo.model.project.ProxyStorageLocationSettings"}
destination = syn.restPOST('/storageLocation', body=json.dumps(destination))
project_destination ={"concreteType": "org.sagebionetworks.repo.model.project.UploadDestinationListSetting",
"settingsType": "upload"}
project_destination['locations'] = [destination['storageLocationId']]
project_destination['projectId'] = PROJECT
project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination))
|
R
| Code Block |
|---|
library(synapser)
synLogin()
projectId <- 'syn12345'
destination <- list(uploadType='SFTP',
secretKey='your_secret_key',
proxyUrl='https://your-proxy.prod.sagebase.org',
concreteType='org.sagebionetworks.repo.model.project.ProxyStorageLocationSettings')
destination <- synRestPOST('/storageLocation', body=toJSON(destination))
projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
settingsType='upload')
projectDestination$locations <- list(destination$storageLocationId)
projectDestination$projectId <- projectId
projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination))
|
See also:
/wiki/spaces/DOCS/pages/2048426057See also: