...
| Info |
|---|
Note: You can only create STS storage locations on an empty folder. This is to ensure consistency between the STS storage location and the Synapse folder. To use STS on existing data, see the Migrating Your Data section below. |
Synapse-
...
Managed STS
...
Storage Locations
You can create an STS storage location using Synapse storage. Temporary S3 credentials will grant you access to the files and folders scoped to your STS storage location.
...
To set up the STS storage location using Synapse storage, first make sure you have an empty Synapse folder. Note that you will need write access to that folder. Then run the following code:
Python
| Code Block | ||
|---|---|---|
| ||
# Set storage location
import synapseclient
import json
syn = synapseclient.login()
FOLDER = 'syn12345'
destination = {'uploadType':'S3',
'stsEnabled':True,
'concreteType':'org.sagebionetworks.repo.model.project.S3StorageLocationSetting'}
destination = syn.restPOST('/storageLocation', body=json.dumps(destination))
project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
'settingsType': 'upload'}
project_destination['locations'] = [destination['storageLocationId']]
project_destination['projectId'] = FOLDER
project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination))
|
R
| Code Block | ||
|---|---|---|
| ||
#set storage location
library(synapser)
library(rjson)
synLogin()
folderId <- 'syn12345'
destination <- list(uploadType='S3',
stsEnabled=TRUE,
concreteType='org.sagebionetworks.repo.model.project.S3StorageLocationSetting')
destination <- synRestPOST('/storageLocation', body=toJSON(destination))
projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
settingsType='upload')
projectDestination$locations <- list(destination$storageLocationId)
projectDestination$projectId <- folderId
projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination))
|
Once the Synapse managed STS storage location is set up, you can upload files through the Synapse or the Synapse client of your choice.
External STS
...
Storage Locations
You can also create an STS storage location in an external AWS S3 bucket.
There are benefits of creating connections to Synapse from an external bucket. If you already have data stored in S3, or if you have large amounts of data that you want to transfer with the AWS command line interface, you can avoid uploading data to Synapse-managed storage by creating connections directly to the S3 bucket. Enabling an STS storage location in the external bucket allows access to the S3 directly for future computing.
Follow the steps in the Custom storage locations/wiki/spaces/DOCS/pages/2048327803 article to set read-write or read-only permissions on your external S3 bucket and enable cross-origin resource sharing (CORS). You may use AWS Cloudformation for set up.
...
| Note |
|---|
Important: If a baseKey is not specified, the temporary AWS credentials vended by STS will give users access to the whole bucket. To prevent access to the whole bucket, enter a folder path in your bucket that all files in the storage location should go into as the baseKey. |
Python
| Code Block |
|---|
# Set storage location
import synapseclient
import json
syn = synapseclient.login()
FOLDER = 'syn12345'
destination = {'uploadType':'S3',
'stsEnabled':True,
'bucket':'nameofyourbucket',
'baseKey':'nameofyourbasekey',
'concreteType':'org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting'}
destination = syn.restPOST('/storageLocation', body=json.dumps(destination))
project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
'settingsType': 'upload'}
project_destination['locations'] = [destination['storageLocationId']]
project_destination['projectId'] = FOLDER
project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination))
|
R
| Code Block |
|---|
#set storage location
library(synapser)
library(rjson)
synLogin()
folderId <- 'syn12345'
destination <- list(uploadType='S3',
stsEnabled=TRUE,
bucket='nameofyourbucket',
baseKey='nameofyourbasekey',
concreteType='org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting')
destination <- synRestPOST('/storageLocation', body=toJSON(destination))
projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting',
settingsType='upload')
projectDestination$locations <- list(destination$storageLocationId)
projectDestination$projectId <- folderId
projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination))
|
...
| Info |
|---|
Note: Synapse automatically generates folders for files uploaded through the Synapse website or clients. Files added directly to S3 may not match this folder structure. We recommend against mixing these two methods of adding files to prevent confusion in the folder structure. |
Obtaining
...
Temporary S3
...
Credentials
Once your STS storage location is set up, you can use Synapse to request temporary AWS credentials to access your data in S3 directly. These temporary credentials are active for 12 hours.
To get temporary credentials, Python and Java code is provided below. The REST interface is also available to request temporary credentials.
Java
| Code Block |
|---|
StsCredentials stsCredentials = synapseClient.getTemporaryCredentialsForEntity(folderEntityId, StsPermission.read_only); AWSCredentials awsCredentials = new BasicSessionCredentials(stsCredentials.getAccessKeyId(), stsCredentials.getSecretAccessKey(), stsCredentials.getSessionToken()); AWSCredentialsProvider awsCredentialsProvider = new AWSStaticCredentialsProvider(awsCredentials); AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(awsCredentialsProvider).build(); |
Python
| Code Block |
|---|
import synapseclient
import boto3
syn = synapseclient.login()
sts_credentials = syn.restGET(f"/entity/{FOLDER}/sts?permission=read_only")
client = boto3.client(
's3',
aws_access_key_id=sts_credentials['accessKeyId'],
aws_secret_access_key=sts_credentials['secretAccessKey'],
aws_session_token=sts_credentials['sessionToken'],
)
ent = syn.get("syn12345", downloadFile=False)
client.download_file(ent._file_handle['bucketName'],
ent._file_handle['key'],
ent.name)
# ent._file_handle['bucketName'] -- The name of the Synapse bucket to download from.
# ent._file_handle['key'] -- The path of the file in the Synapse bucket
# ent.name -- The path to the file to download to.
|
Migrating
...
Existing Data
From an
...
Existing S3
...
Bucket
If you have existing data in an S3 bucket, either stand-alone data or data from a previous Synapse project, you can use this sample code to migrate your S3 data to a new Synapse folder with a STS storage location.
| Info |
|---|
Note: You'll need to create a new folder with an STS-enabled storage location, as per the instructions above. Additionally, you must be the owner of the storage location in Synapse. |
From a
...
Project with
...
Multiple Buckets
If your Synapse project uses data from multiple S3 buckets, or if the data is in an S3 bucket you don’t own, then you may need to download the data and re-upload it a new Synapse folder with a STS storage location. Use this sample code to migrate the data in yourproject.
...
| Info |
|---|
Note: This method may incur data transfer costs from S3. |
Additional
...
Restrictions
STS storage locations can only be added to folders, not projects.
An STS storage location can only be added to, or removed from, empty folders.
An STS storage location cannot be added alongside other storage locations.
If a parent folder has an STS storage location, all sub-folders will have the same storage location and ACL as the parent.
A folder with an STS storage location can only contain files and other folders.
A file or folder in an STS storage location cannot be moved to a folder with a different storage location.
A file in an STS storage location must have a bucket and base key matching that storage location.
If an STS storage location is defined on a folder, that folder cannot be placed within another folder hierarchy that also defines an STS storage location (even if they are the same storage location).
See Also;Custom Storage Locations
/wiki/spaces/DOCS/pages/2048327803
| Include Page | ||||
|---|---|---|---|---|
|