Developer AWS Accounts

Use your individual AWS account under the Sage consolidated bill for AWS experiments. The rule of thumb is that if you cannot shut off what ever you are running while you are on vacation, it belongs in the Production AWS Account.

Production AWS Account

Use your IAM Account for:

Use the platform@sagebase.org Account for:

First time accessing the console

  1. Get your IAM credentials
  2. Then either

How To

Figure out if AWS is broken

AWS occasionally has issues. To figure out whether the problem you are currently experiencing is their fault or not:

  1. Check the AWS status console to see if they are reporting any problems http://status.aws.amazon.com/
  2. Check the most recent messages on the forums https://forums.aws.amazon.com/index.jsp Problems often get reported there first.
  3. If you still do not find evidence that the problem is AWS's fault, search the forums for your particular issue. Its likely that someone else has run into the same exact problem in the past.
  4. Still no luck? Ask your coworkers and/or post a question to the forums.

Run a Report to Know Who has Accessed What When

Use Elastic MapReduce to run a script on all our logs in the bucket logs.sagebase.org. There are some scripts in bucket emr.sagebase.org/scripts that will do the trick. If you want to change what they do, feel free to make new scripts.

Here is what a configured job looks like:

And here is some sample output from the job.  Note that:

Upload a dataset to S3

For the initial upload, a GUI tool called BucketExplorer (http://www.bucketexplorer.com/) is used. Uploads are done from the internal host fremont.fhcrc.org using the local access account 'platform', with the same password as the platform@sagebase.org account. The most efficient way to connect is to use an NX protocol client (http://www.nomachine.com/download.php) to get a virtual desktop as the user platform. Once connected the preconfigured BucketExplorer can be found in the application menu in the lower left corner of the screen.

Mac OSX Users I installed "NX Client for Mac OSX" but it complained that I was missing bin/nxssh and bin/nxservice. That stuff was not installed under Applications but instead under /Users/deflaux/usr/NX/

The initial datasets are stored in /work/platform/. This entire collection is mirrored exactly and can transfered by dragging and dropping into the data01.sagebase.org s3 bucket. This operation should be done as user platform, as all files should be readable by said user to facilitate the transfer.

BucketExplorer is very efficient, and will do hash comparisons and only transfer what files have changed. One can also get a visual comparison of what files have changed using the 'Comparer' button. During the transfer, the program will parallelize the transfer into 20 streams for very efficient use of outgoing bandwidth to the cloud.

Create a new IAM group

We are storing our access policies here: http://sagebionetworks.jira.com/source/browse/PLFM/trunk/configuration/awsIamPolicies

See the IAM documentation for more details about how to do this but its basically:

iam-groupcreate -g ReadOnlyUnrestrictedDataUsers
iam-groupuploadpolicy -g ReadOnlyUnrestrictedDataUsers -p ReadOnlyUnrestrictedDataPolicy -f ~/platform/trunk/configuration/awsIamPolicies/ReadOnlyUnrestrictedDataPolicy.txt
iam-groupadduser -u test -g ReadOnlyUnrestrictedDataUsers
iam-grouplistusers -g ReadOnlyUnrestrictedDataUsers

Create a new user and add them to IAM groups

Note that this is for adding Sage employees to groups by hand. The repository service will take care of adding Web Client and R Client users to the right IAM group(s) after they sign a EULA for a dataset.

See the IAM documentation for more details about how to do this but its basically:

iam-usercreate -u bruce.hoff -g Admins -k -v > bruce.hoff_creds.txt

Then give the user their credentials file.

Per Brian, he recommends that we store them in our server home directory on beltown/fremont so that they are backed up. If you have the Sage root password, you can get your credential file from the backup location:

~>ssh ndeflaux@fremont ls -la /home/ndeflaux/PlatformIAMCreds
total 40
drwxrwxr-x  2 ndeflaux FHCRC\domain^users 4096 2011-02-16 16:31 .
drwxr-xr-x 30 ndeflaux FHCRC\domain^users 4096 2011-02-16 17:16 ..
-r--------  1 ndeflaux FHCRC\domain^users  126 2011-02-16 16:31 brian.holt_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  126 2011-02-16 16:31 bruce.hoff_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  129 2011-02-16 16:31 david.burdick_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  125 2011-02-16 16:31 john.hill_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  127 2011-02-16 16:31 mike.kellen_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  130 2011-02-16 16:31 nicole.deflaux_creds.txt
-r--------  1 ndeflaux FHCRC\domain^users  332 2011-02-16 16:31 platform_cred.txt
-r--------  1 ndeflaux FHCRC\domain^users  120 2011-02-16 16:31 test_creds.txt

How to connect to RDS

The firewall currently only allows you to connect from a server inside the Fred Hutch network. If you are working from home, ssh to fremont and then do this.

This is the super user so be careful!

/usr/local/mysql/bin/mysql -u platform -h repo.c5sxx7pot9i8.us-east-1.rds.amazonaws.com -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 5.5.8-log Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| innodb             |
| mysql              |
| performance_schema |
| repositorydb       |
+--------------------+
5 rows in set (0.08 sec)

Notes for the sake of posterity

Gotchas Getting Started with Beanstalk

Here are some gotchas I ran into when using beanstalk for the first time:

  1. I created a key pair in US West and was confused when I couldn't get beanstalk to use that key pair.
  2. I could not ssh to my box even though I had the right key pair and the hostname.
  3. My serlvet didn't work right away and I wanted to look at stuff on disk.
  4. Error: java.lang.NoClassDefFoundError: javax/servlet/jsp/jstl/core/Config
  5. Here's what your deployment might look like when things are working well: