The following provides instructions on how to log on to the Sage Scientific Compute workspace using your Synapse credentials, and how to use the products provided in the AWS Service Catalog to setup or modify EC2 instances and S3 buckets.
...
Note: You can add additional custom tags when provisioning resources however there are 3 reserved tags that you should avoid adding: Department, Project, and OwnerEmail. The owner email tag is automatically set to <Synapse Username>@synapse.org
Notifications
Please skip the Notifications pane. SNS notifications are not operational at this time.
...
The AWS SSM allows direct access to private instances from your own computer terminal. To setup access with the AWS SSM we need to create a special Synapse personal access token (PAT) that will work with the Sage Service Catalog. This is special PAT that can only be created using this workflow, creating a PAT from the Synapse personal token manager web page will NOT work.
Request a Synapse PAT by visiting https://sc.sageit.org/personalaccesstoken , for Sage employees, or https://ad.strides.sc.sageit.org/personalaccesstoken for AMP-AD members. (You may need to login to Synapse.) If you have already created a PAT through this mechanism and are repeating the process you must first visit the token management page in Synapse and delete the existing one with the same name.
After logging into Synapse a file containing the PAT, which is a long character string (i.e. eyJ0eXAiOiJ...Z8t9Eg), is returned to you. Save the file to your local machine and note the location where you saved it to then close the browser session.
Note: At this point you can verify that the PAT for the Service Catalog was successfully created by viewing the Synapse token management page. When the PAT expires you will need to repeat these steps to create a new PAT. The PAT should look something like this
...
Run an application on the EC2 (i.e. docker run -p 80:80 httpd)
Code Block [ec2-user@ip-10-49-26-50 ~]$ docker run -p 80:80 httpd Unable to find image 'httpd:latest' locally latest: Pulling from library/httpd 33847f680f63: Pull complete d74938eee980: Pull complete 963cfdce5a0c: Pull complete 8d5a3cca778c: Pull complete e06a573b193b: Pull complete Digest: sha256:71a3a8e0572f18a6ce71b9bac7298d07e151e4a1b562d399779b86fef7cf580c Status: Downloaded newer image for httpd:latest AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message [Thu Jul 22 23:54:12.106344 2021] [mpm_event:notice] [pid 1:tid 140706544895104] AH00489: Apache/2.4.48 (Unix) configured -- resuming normal operations [Thu Jul 22 23:54:12.107307 2021] [core:notice] [pid 1:tid 140706544895104] AH00094: Command line: 'httpd -D FOREGROUND'
To provide access to that app, an SC user can use the port forwarding feature to gain access to the app by running the AWS SSM CLI command:
Code Block aws ssm start-session --profile service-catalog \ --target i-0fd5c9ff0ef675ceb \ --document-name AWS-StartPortForwardingSession \ --parameters '{"portNumber":["80"],"localPortNumber":["9090"]}'To provide access to that app in the Windows Command Prompt use this syntax:
Code Block aws ssm start-session --profile service-catalog \ --target i-0fd5c9ff0ef675ceb \ --document-name AWS-StartPortForwardingSession \ --parameters "{\"portNumber\":[\"80\"],\"localPortNumber\":[\"9090\"]}"Now you should be able to access that app on your local machine at
http://localhost:9090.
...
Using the update action allows you to change parameters or update to a new version of the product. WARNING: changes to configuration parameters usually result in a recreation (“replacement”) of the instance, any data saved on the instance will be lost, and the nature of the update by Amazon is difficult to predict. We recommend that you save any important data to S3, provision a new instance and terminate the original.
Terminate
The terminate action deletes the instance permanently.
...
Please ask #sageit for help transferring ownership if that is deemed to be necessary rather than using this action. Otherwise, you may find that you cannot connect to your instance.
Cloud Storage (S3)
Note: S3 storage products currently are available only to Sage employees only.
...
Cloud Storage Products
To understand the cost of S3 buckets see https://aws.amazon.com/s3/pricing/ or use the AWS pricing calculator. Note that while data egress can be a substantial cost, our Service Catalog provisions buckets and EC2 instances in the same AWS region. Since AWS does not charge for egress to a location within a bucket’s region, accessing data from an instance in provisioned by our Service Catalog will avoid such costs.
S3 Private Encrypted Bucket
This product builds an encrypted AWS S3 bucket encrypted with private access accessible from any source.
If another user wishes to share your bucket they will provide you with their AWS IAM identifier, or ARN. You can enter it next to S3UserNames. If you wish to enter more than one, separate them by a comma (and no other white space).
Once the bucket is provisioned you can click on the “BucketUrl” in the provisioned product to go to the S3 Console where you can upload and download files. Alternately, if you add an AWS IAM user ARN, you can use the awscli or boto3 to perform bucket operations.
Which ARN to use?
Sage internal users can authenticate through Jumpcloud and aws-saml as described /wiki/spaces/IT/pages/405864455, then use an ARN constructed in this way: arn:aws:sts::<account_number>:assumed-role/<role_name>/<email_prefix>@sagebase.org. For example, if I authenticate as a “sandbox developer” and I am Jane Doe, the ARN looks like this: arn:aws:sts::563295687221:assumed-role/sandbox-developer/jane.doe@sagebase.org.
If an outside collaborator requires access, we recommend using a user ARN as described in the AWS documentation. In cases of PHI, please ensure that data sharing is appropriate.
...
. To access the bucket, see the section Using an S3 Bucket, below.
S3 Synapse Bucket
This product builds an AWS S3 bucket with private access for Synapse. Please see the instructions for “S3 Private Encrypted Bucket” above. The additional configuration parameter is the name of the Synapse user who will be allowed to link the bucket to a Synapse project.
...
When provisioning, you are prompted for two names. On the “Product Version” screen of the wizard, you must name your product. This is the name you will see listed in Service Catalog under “Provisioned Products” later. Please include your name in the product, e.g. if your name is Jane Doe and you are provisioning a bucket for your project Foo, you could name it jdoe-bucket-foo. On the “Parameters” screen, you have the option of naming the bucket itself (otherwise a name will be assigned). That is the name to use when accessing the bucket through the Amazon S3 client or via the Amazon S3 console.Another optional field is “S3UserARNs”. If another user wishes to share your bucket they will provide you with their AWS IAM identifier, or ARN, and you enter it in this field. If you wish to enter more than one, separate them by a comma (and no other white space). If you wish to have AWS CLI access, add the ARN for your sandbox user herebucket itself (otherwise a name will be assigned). That is the name to use when accessing the bucket through the Amazon S3 client or via the Amazon S3 console.
The S3 Synapse Bucket product requires the “SynapseUserName” field.
...
Using a bucket with the S3 client
If you added an IAM user or role ARN when provisioning the bucket, you can use the S3 client with credentials for said user or role to upload and download files. Note: You must include the ‘bucket-owner-full-control’ canned ACL when uploading files.To authenticate the S3 client for bucket access, follow the set-up steps under SSM access to an Instance, above. You can then access the bucket by including “--profile service-catalog” in the command, e.g. to download a file the syntax is:
| Code Block |
|---|
aws --profile service-catalog s3 cp s3://<your-bucket-name>/file.txt ./file.txt |
Using a bucket with the Synapse client
...
The terminate action removes the bucket product from the Service Catalog however the bucket is not removedimmediately deleted. The bucket and data in it will be placed in an archived state where no users will have access to the bucket. Upon expiration of the archive period (30 days) the bucket and data will get automatically purged from AWS. The owner of the bucket may request Sage IT to restore access to the bucket before it is purged
...
The benefits of the Service Catalog are that it is self-service, meant to fulfill the most common needs for compute and storage, and that it creates resources in a PHI-safe environment. We encourage you to use it preferentially. However, it will not fulfill all needs. For custom development in a PHI-safe environment, the “scicomp” account remains the preference for Sage employees. For custom development that does not concern PHI, the “sandbox” account can be used by Sage employees. For more information, see the Sage Bionetworks intranet article on computing. If you have any questions about which environment is most suitable, questions are welcome in the #sageit Slack channel!
Ephemeral Instances,
...
Persistent Data
We encourage you to treat your instances as ephemeral. It is very easy with the Service Catalog to create new instances, and since updating the parameters of an instance frequently results in their recreation, it is best not to get too attached to any one. This makes getting in the habit of storing your data in Synapse or S3 highly desirable. If you leave the organization you should again store any data your colleagues will need in Synapse or S3, then terminate your instances. Any instances left running will be terminated upon your departure.
...