Page Comparison

...

Production AWS Account

Use your IAM Account the platform@sagebase.org account for:

S3
EC2
Elastic MapReduce (command line access only right now)
You will need to log into the AWS console with you IAM login and password: https://325565585839.signin.aws.amazon.com/console/ec2

Use the platform@sagebase.org Account for:

Elastic Beanstalk
console usage of Elastic MapReduce
Relational Database Service
Identity and Access Management Service
You will need to log into the AWS console with the platform@sagebase.org username and password: https://console.aws.amazon.com/

Credentials, passwords, ssh keys

...

You can also use your IAM account if you like but many AWS services do not support it yet such as Beanstalk. There is a different link to log into the AWS console with your IAM login and password: https://325565585839.signin.aws.amazon.com/console/ec2

Credentials, passwords, ssh keys

You can find them on our shared servers. When storing passwords locally on your laptop (which already has an encrypted drive, yay!) you might also consider using Password Safe.

Code Block

/work/platform>hostname
sodo
/work/platform>find PasswordsAndCredentials -type f
PasswordsAndCredentials/PlatformIAMCreds/brian.holt_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/david.burdick_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/john.hill_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/mike.kellen_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/nicole.deflaux_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/repository.service_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/test_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/integration.test@sagebase.org_creds.txt
PasswordsAndCredentials/SshCertificates/self-signed-crowd-tomcat.cer
PasswordsAndCredentials/PlatformAWSCredentials/cert-ACQDRLBJ7TXKIZ6KQNONJWH57GDPA2X4.pem
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.RDPCertificate
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.windowsPassword
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.windowsPassword~
PasswordsAndCredentials/PlatformAWSCredentials/pk-ACQDRLBJ7TXKIZ6KQNONJWH57GDPA2X4.pem
PasswordsAndCredentials/PlatformAWSCredentials/platformIAMCLI_CredentialFile.txt
PasswordsAndCredentials/passwords.txt
PasswordsAndCredentials/SshKeys/mysql-ssl-ca-cert.pem
PasswordsAndCredentials/SshKeys/PlatformKeyPairEast.pem
PasswordsAndCredentials/SshKeys/PlatformKeyPair.pem
PasswordsAndCredentials/SshKeys/tranSMARTDemo.pem
PasswordsAndCredentials/AtlassianAccountAWSCredentials/cert-MEF3B5GUK5T4LRUSBQ423ZYMXXRPRBJU.pem
PasswordsAndCredentials/AtlassianAccountAWSCredentials/elasticbamboo.pk
PasswordsAndCredentials/AtlassianAccountAWSCredentials/pk-MEF3B5GUK5T4LRUSBQ423ZYMXXRPRBJU.pem

First time accessing the console

Create a password for yourself using the IAM tools. You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on sodo.

Code Block
ssh you@sodo cd /work/platform bash source bin/setupIamClient.sh iam-useraddloginprofile -u YourFirstname.YourLastname -p aDecentPassword

Miscellaneous How To's

How to SSH to an EC2 Host

Connecting from Linux

ssh -i PlatformKeyPairEast.pem ec2-user@<the ec2 host>

For screen shots see EC2 docs

Connecting from Windows using Putty

For screen shots see EC2 docs

Window's users can also connect using PuTTY or WinSCP, however you will to first create a PuTTY private key file using puttygen.exe
Here is how to create the private key file:

Run the 'puttygen.exe' tool
Select the 'load' button from the UI.
From the file dialog select your the KeyPair file (i.e. PlatformKeyPairEast.pem)
A popup dialog should tell you the key file was imported sucessfully and to save it using "Save private Key"
Select 'Save Private Key' and give it a name such as PlatformKeyPairEast.ppk to create the PuTTY private key file.

Once you have a PuTTY private key file you can use it to connect to your host using PuTTY or WinSCP.
To connect with WinSCP:

Set the host name, and keep the default port (22). Note: Make sure port 22 is open on the box you are connecting to.
Set the user name to ec2-user
Select the '...' button under 'Private Key File' and select the .ppk file you created above.
Select 'Login'

Figure out if AWS is broken

...

/platform/PasswordsAndCredentials>ls
AtlassianAccountAWSCredentials     platformStagingEncryptionKey.txt
crowdServerCertificate             SshCertificates
passwords.txt                      SshKeys
PlatformAWSCredentials             StackCredentials
PlatformIAMCreds                   wildcard-sagebase.org-cert
platformPropertyEncryptionKey.txt

Miscellaneous How To's

How to SSH to an EC2 Host

Connecting from Linux

ssh -i PlatformKeyPairEast.pem ec2-user@<the ec2 host>

For screen shots see EC2 docs

Connecting from Windows using Putty

For screen shots see EC2 docs

Window's users can also connect using PuTTY or WinSCP, however you will to first create a PuTTY private key file using puttygen.exe
Here is how to create the private key file:

Run the 'puttygen.exe' tool
Select the 'load' button from the UI.
From the file dialog select your the KeyPair file (i.e. PlatformKeyPairEast.pem)
A popup dialog should tell you the key file was imported sucessfully and to save it using "Save private Key"
Select 'Save Private Key' and give it a name such as PlatformKeyPairEast.ppk to create the PuTTY private key file.

Once you have a PuTTY private key file you can use it to connect to your host using PuTTY or WinSCP.
To connect with WinSCP:

Set the host name, and keep the default port (22). Note: Make sure port 22 is open on the box you are connecting to.
Set the user name to ec2-user
Select the '...' button under 'Private Key File' and select the .ppk file you created above.
Select 'Login'

Figure out if AWS is broken

AWS occasionally has issues. To figure out whether the problem you are currently experiencing is their fault or not:

Check the AWS status console to see if they are reporting any problems http://status.aws.amazon.com/
Check the most recent messages on the forums https://forums.aws.amazon.com/index.jsp Problems often get reported there first.
If you still do not find evidence that the problem is AWS's fault, search the forums for your particular issue. Its likely that someone else has run into the same exact problem in the past.
Still no luck? Ask your coworkers and/or post a question to the forums.

...

Code Block

{
  "Id": "Policy1305325502034",
  "Statement": [
    {
      "Sid": "Stmt1305324625148",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::THE_BUCKET/*",
      "Principal": {
        "AWS": [
          "THE_PERSONS_AWS_ACCOUNT_NUMBER"
        ]
      }
    },
    {
      "Sid": "Stmt1305325498087",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::THE_BUCKET",
      "Principal": {
        "AWS": [
          "THE_PERSONS_AWS_ACCOUNT_NUMBER"
        ]
      }
    }
  ]
}

...

RDS and MySQL How To's

Create a new IAM group

How to connect to RDS

Use the MySQL client. You can install the IAM tools it locally on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on sodo.

We are storing our access policies in SVN: http://sagebionetworks.jira.com/source/browse/PLFM/trunk/configuration/awsIamPolicies

See the IAM documentation for more details about how to do this but here is an example of how one of our existing groups was created:

Code Block


ssh you@sodo
cd /work/platform
bash
source bin/setupIamClient.sh
iam-groupcreate -g ReadOnlyUnrestrictedDataUsers
iam-groupuploadpolicy -g ReadOnlyUnrestrictedDataUsers -p ReadOnlyUnrestrictedDataPolicy -f /work/platform/awsIamPolicies/ReadOnlyUnrestrictedDataPolicy.txt
iam-groupadduser -u test -g ReadOnlyUnrestrictedDataUsers
iam-grouplistusers -g ReadOnlyUnrestrictedDataUsers

Create a new user and add them to IAM groups

You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on sodo.

Note that this is for adding Sage employees to groups by hand. The repository service will take care of adding Web Client and R Client users to the right IAM group(s) after they sign a EULA for a dataset.

See the IAM documentation for more details about how to do this but here is an example of how Bruce's IAM user was created:

Code Block
ssh you@sodo cd /work/platform bash source bin/setupIamClient.sh iam-usercreate -u bruce.hoff -g Admins -k -v > PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt

Then give the user their credentials file.

RDS and MySQL How To's

How to connect to RDS

Use the MySQL client. You can install it locally on your machine (do this by installing a local MySQL database too.) Or you can use it on sodo.

The firewall currently only allows you to connect from a server inside the Fred Hutch network. If you are working from home, ssh to sodo and then do this. You can find the database password in sodo:/work/platform/PasswordsAndCredentials/passwords.txt

The produser account has full access to all databases, so be careful! The platform user is superuser and should only be used for creating new databases and users, and setting permissions.

...

(do this by installing a local MySQL database too.) Or you can use it on sodo.

The firewall currently only allows you to connect from a server inside the Fred Hutch network. If you are working from home, ssh to sodo and then do this. You can find the database password in sodo:/work/platform/PasswordsAndCredentials/passwords.txt

The produser account has full access to all databases, so be careful! The platform user is superuser and should only be used for creating new databases and users, and setting permissions.

Code Block


~>hostname
sodo
~>/usr/bin/mysql --ssl-ca=/work/platform/PasswordsAndCredentials/SshKeys/mysql-ssl-ca-cert.pem -u produser -h repo.c5sxx7pot9i8.us-east-1.rds.amazonaws.com -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6212
Server version: 5.5.8-log Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| innodb             |
| performance_schema |
| repositorydb       |
+--------------------+
4 |rows Databasein set (0.07 sec)

mysql> use repositorydb;
Reading table information |
+--------------------+
| information_schema |
| innodb             |
| performance_schema |
| repositorydb       |
+for completion of table and column names
You can turn off this feature to get a quicker startup with -A

showDatabase changed
mysql> show tables;
+------------------------+
4| rows Tables_in_repositorydb set (0.07 sec)

mysql> use repositorydb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

showDatabase changed
mysql> show tables;
+------------------------+
| Tables_in_repositorydb |
+------------------------+
| JDOANALYSISRESULT|
+------------------------+
| JDOANALYSISRESULT   |
| JDOANNOTATIONS      |
| JDODATASET          |
| JDODATASETANALYSIS  |
| JDODATEANNOTATION   |
| JDODOUBLEANNOTATION |
| JDOINPUTDATALAYER   |
| JDOLAYERLOCATION    |
| JDOANNOTATIONSJDOLAYERLOCATIONS   |
| JDOLONGANNOTATION   |
| JDODATASETJDOPROJECT          |
| JDORESOURCEACCESS JDODATASETANALYSIS  |
| JDOREVISION JDODATEANNOTATION     | | JDODOUBLEANNOTATION |
| JDOSCRIPT JDOINPUTDATALAYER    | | JDOLAYERLOCATION    |
| JDOLAYERLOCATIONSJDOSTRINGANNOTATION   |
| JDOLONGANNOTATION   |
| JDOPROJECT          |
| JDORESOURCEACCESS   |
| JDOREVISION         |
| JDOSCRIPT           |
| JDOSTRINGANNOTATION |
| JDOUSER             |
| JDOUSERGROUP        |
| NUCLEUS_TABLES         |
+------------------------+
18 rows in set (0.08 sec)

mysql> desc JDODATASET;
+---------------------+--------------+------+-----+---------+----------------+
| Field               | Type         | Null | Key | Default | Extra          |
+---------------------+--------------+------+-----+---------+----------------+
| ID                  | bigint(20)   | NO   | PRI | NULL    | auto_increment |
| ANNOTATIONS_ID_OID  | bigint(20)   | YES  | MUL | NULL    |                |
| CREATION_DATE       | datetime     | YES  |     | NULL    |                |
| CREATOR             | varchar(256) | YES  |     | NULL    |                |
| DESCRIPTION         | varchar(256) | YES  |     | NULL    |                |
| NAME                | varchar(256) | YES  |     | NULL    |                |
| NEXT_VERSION_ID_OID | bigint(20)   | YES  | MUL | NULL    |                |
| RELEASE_DATE        | datetime     | YES  |     | NULL    |                |
| REVISION_ID_OID     | bigint(20)   | YES  | MUL | NULL    |                |
| STATUS              | varchar(256) | YES  |     | NULL    |                |
+---------------------+--------------+------+-----+---------+----------------+
10 rows in set (0.07 sec)

mysql> select count(*) from JDODATASET ;
+----------+
| count(*) |
+----------+
|      114 |
+----------+
1 row in set (0.08 sec)

mysql> quit

...

Setup MySQL
- http://dev.mysql.com/doc/refman/5.0/en/macosx-installation.html
- http://dev.mysql.com/doc/refman/5.0/en/default-privileges.html

Create your empty database

Code Block

~/>/usr/local/mysql/bin/mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1910
Server version: 5.5.9 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.06 sec)

mysql> create database test2;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               | | test2              |
+--------------------+
5 rows in set (0.04 sec)

How to load a JDO schema into MySQL

Use the mysql client to connect to the database host and drop/create the database as needed
Set up the database configuration
- trunk/lib/jdomodels/src/main/resources/datanucleus.properties is already configured for a local MySQL instance
- For RDS, locally edit trunk/lib/jdomodels/src/main/resources/datanucleus.properties to include the RDS host, user, and password (do not check in any production database passwords into svn)

Run the Datanucleus schema creation tool

Code Block
cd trunk/lib/jdomodels mvn compile mvn datanucleus:enhance mvn datanucleus:schema-create

...

|
| test2              |
+--------------------+
5 rows in set (0.04 sec)

How to load a JDO schema into MySQL

Note that the repository service will create the schema upon startup if it does not already exist.

How to get the repository service to use local MySQL instead of HDSQL

...

Code Block

>ssh sodo
>cd /work/platform/DatasetMetadataLoader
/work/platform/DatasetMetadataLoader> curl http://dhcp149222.fhcrc.org:8080/services-repository-0.5-SNAPSHOT/repo/v1/dataset
{"results":[],"totalNumberOfResults":0,"paging":{}}
/work/platform/DatasetMetadataLoader>./datasetCsvLoader.py  -e http://dhcp149222.fhcrc.org:8080/services-repository-0.5-SNAPSHOT/repo/v1 -a http://dhcp149222.fhcrc.org:8080/services-authentication-0.5-SNAPSHOT/auth/v1/
lot of output here . . .
/work/platform/DatasetMetadataLoader> curl http://dhcp149222.fhcrc.org:8080/services-repository-0.5-SNAPSHOT/repo/v1/dataset?limit=1
{
   "results":[
      {
         "name":"Gastric Cancer ACRG",
         "annotations":"/repo/v1/dataset/0/annotations",
         "id":"0",
         "version":"0.0.1",
         "creator":"Asian Cancer Research Group, Inc., (ACRG)",
         "description":null,
         "creationDate":1299375144172,
         "status":"Future",
         "uri":"/repo/v1/dataset/0",
         "etag":"384011750",
         "releaseDate":null,
         "layer":"/repo/v1/dataset/0/layer",
         "hasExpressionData":false,
         "hasGeneticData":false,
         "hasClinicalData":false
      }
   ],
   "totalNumberOfResults":114,
   "paging":{
      "next":"/repo/v1/dataset?offset=2&limit=1"
   }
}

...

Versions Compared

Old Version 63

New Version 64

Key

Production AWS Account

Credentials, passwords, ssh keys

Credentials, passwords, ssh keys

First time accessing the console

Miscellaneous How To's

How to SSH to an EC2 Host

Connecting from Linux

Connecting from Windows using Putty

Figure out if AWS is broken

Miscellaneous How To's

How to SSH to an EC2 Host

Connecting from Linux

Connecting from Windows using Putty

Figure out if AWS is broken

RDS and MySQL How To's

Create a new IAM group

How to connect to RDS

Create a new user and add them to IAM groups

RDS and MySQL How To's

How to connect to RDS

How to load a JDO schema into MySQL

How to load a JDO schema into MySQL

How to get the repository service to use local MySQL instead of HDSQL