Page Comparison

...

Code Block

/work/platform>hostname
fremontbelltown
/work/platform>find PasswordsAndCredentials -type f
PasswordsAndCredentials/PlatformIAMCreds/brian.holt_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/david.burdick_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/john.hill_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/mike.kellen_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/nicole.deflaux_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/repository.service_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/test_creds.txt
PasswordsAndCredentials/PlatformIAMCreds/integration.test@sagebase.org_creds.txt
PasswordsAndCredentials/SshCertificates/self-signed-crowd-tomcat.cer
PasswordsAndCredentials/PlatformAWSCredentials/cert-ACQDRLBJ7TXKIZ6KQNONJWH57GDPA2X4.pem
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.RDPCertificate
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.windowsPassword
PasswordsAndCredentials/PlatformAWSCredentials/i-361d9b59.windowsPassword~
PasswordsAndCredentials/PlatformAWSCredentials/pk-ACQDRLBJ7TXKIZ6KQNONJWH57GDPA2X4.pem
PasswordsAndCredentials/PlatformAWSCredentials/platformIAMCLI_CredentialFile.txt
PasswordsAndCredentials/passwords.txt
PasswordsAndCredentials/SshKeys/mysql-ssl-ca-cert.pem
PasswordsAndCredentials/SshKeys/PlatformKeyPairEast.pem
PasswordsAndCredentials/SshKeys/PlatformKeyPair.pem
PasswordsAndCredentials/SshKeys/tranSMARTDemo.pem
PasswordsAndCredentials/AtlassianAccountAWSCredentials/cert-MEF3B5GUK5T4LRUSBQ423ZYMXXRPRBJU.pem
PasswordsAndCredentials/AtlassianAccountAWSCredentials/elasticbamboo.pk
PasswordsAndCredentials/AtlassianAccountAWSCredentials/pk-MEF3B5GUK5T4LRUSBQ423ZYMXXRPRBJU.pem

...

Create a password for yourself using the IAM tools. You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on fremontbelltown.

Code Block
ssh you@fremontyou@belltown cd /work/platform bash source bin/setupIamClient.sh iam-useraddloginprofile -u YourFirstname.YourLastname -p aDecentPassword

...

For the initial upload, a GUI tool called BucketExplorer (http://www.bucketexplorer.com/) is used. Uploads are done from the internal host fremontbelltown.fhcrc.org using the local access account 'platform', with the same password as the platform@sagebase.org account. The most efficient way to connect is to use an NX protocol client (http://www.nomachine.com/download.php) to get a virtual desktop as the user platform. Once connected the preconfigured BucketExplorer can be found in the application menu in the lower left corner of the screen.

...

You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on fremontbelltown.

We are storing our access policies in SVN: http://sagebionetworks.jira.com/source/browse/PLFM/trunk/configuration/awsIamPolicies

See the IAM documentation for more details about how to do this but here is an example of how one of our existing groups was created:

Code Block

ssh you@fremontyou@belltown
cd /work/platform
bash
source bin/setupIamClient.sh
iam-groupcreate -g ReadOnlyUnrestrictedDataUsers
iam-groupuploadpolicy -g ReadOnlyUnrestrictedDataUsers -p ReadOnlyUnrestrictedDataPolicy -f /work/platform/awsIamPolicies/ReadOnlyUnrestrictedDataPolicy.txt
iam-groupadduser -u test -g ReadOnlyUnrestrictedDataUsers
iam-grouplistusers -g ReadOnlyUnrestrictedDataUsers

...

You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on fremontbelltown.

Note that this is for adding Sage employees to groups by hand. The repository service will take care of adding Web Client and R Client users to the right IAM group(s) after they sign a EULA for a dataset.

See the IAM documentation for more details about how to do this but here is an example of how Bruce's IAM user was created:

Code Block
ssh you@fremontyou@belltown cd /work/platform bash source bin/setupIamClient.sh iam-usercreate -u bruce.hoff -g Admins -k -v > PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt

...

Use the MySQL client. You can install it locally on your machine (do this by installing a local MySQL database too.) Or you can use it on fremontbelltown.

The firewall currently only allows you to connect from a server inside the Fred Hutch network. If you are working from home, ssh to fremont belltown and then do this. You can find the database password in fremontbelltown:/work/platform/PasswordsAndCredentials/passwords.txt

This is the super user so be careful!

Code Block

~>hostname
fremontbelltown
~>/usr/bin/mysql -u platform -h repo.c5sxx7pot9i8.us-east-1.rds.amazonaws.com -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6212
Server version: 5.5.8-log Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| innodb             |
| mysql              |
| performance_schema |
| repositorydb       |
+--------------------+
5 rows in set (0.07 sec)

mysql> use repositorydb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

showDatabase changed
mysql> show tables;
+------------------------+
| Tables_in_repositorydb |
+------------------------+
| JDOANALYSISRESULT   |
| JDOANNOTATIONS      |
| JDODATASET          |
| JDODATASETANALYSIS  |
| JDODATEANNOTATION   |
| JDODOUBLEANNOTATION |
| JDOINPUTDATALAYER   |
| JDOLAYERLOCATION    |
| JDOLAYERLOCATIONS   |
| JDOLONGANNOTATION   |
| JDOPROJECT          |
| JDORESOURCEACCESS   |
| JDOREVISION         |
| JDOSCRIPT           |
| JDOSTRINGANNOTATION |
| JDOUSER             |
| JDOUSERGROUP        |
| NUCLEUS_TABLES         |
+------------------------+
18 rows in set (0.08 sec)

mysql> desc JDODATASET;
+---------------------+--------------+------+-----+---------+----------------+
| Field               | Type         | Null | Key | Default | Extra          |
+---------------------+--------------+------+-----+---------+----------------+
| ID                  | bigint(20)   | NO   | PRI | NULL    | auto_increment |
| ANNOTATIONS_ID_OID  | bigint(20)   | YES  | MUL | NULL    |                |
| CREATION_DATE       | datetime     | YES  |     | NULL    |                |
| CREATOR             | varchar(256) | YES  |     | NULL    |                |
| DESCRIPTION         | varchar(256) | YES  |     | NULL    |                |
| NAME                | varchar(256) | YES  |     | NULL    |                |
| NEXT_VERSION_ID_OID | bigint(20)   | YES  | MUL | NULL    |                |
| RELEASE_DATE        | datetime     | YES  |     | NULL    |                |
| REVISION_ID_OID     | bigint(20)   | YES  | MUL | NULL    |                |
| STATUS              | varchar(256) | YES  |     | NULL    |                |
+---------------------+--------------+------+-----+---------+----------------+
10 rows in set (0.07 sec)

mysql> select count(*) from JDODATASET ;
+----------+
| count(*) |
+----------+
|      114 |
+----------+
1 row in set (0.08 sec)

mysql> quit

...

You can use this to load data into a repository service whether it is running on beanstalk or locally on your laptop. You can run datasetCsvLoader.py -h and datasetNuker.py -h to see more documentation. The code can be found here PLFM/users/deflaux/scripts/DatasetLoader

Code Block

>ssh fremontbelltown
>cd /work/platform/DatasetMetadataLoader
/work/platform/DatasetMetadataLoader> curl http://dhcp149222.fhcrc.org:8080/repo/v1/dataset
{"results":[],"totalNumberOfResults":0,"paging":{}}
/work/platform/DatasetMetadataLoader>./datasetCsvLoader.py  -e dhcp149222.fhcrc.org:8080
lot of output here . . .
/work/platform/DatasetMetadataLoader> curl http://dhcp149222.fhcrc.org:8080/repo/v1/dataset?limit=1
{
   "results":[
      {
         "name":"Gastric Cancer ACRG",
         "annotations":"/repo/v1/dataset/0/annotations",
         "id":"0",
         "version":"0.0.1",
         "creator":"Asian Cancer Research Group, Inc., (ACRG)",
         "description":null,
         "creationDate":1299375144172,
         "status":"Future",
         "uri":"/repo/v1/dataset/0",
         "etag":"384011750",
         "releaseDate":null,
         "layer":"/repo/v1/dataset/0/layer",
         "hasExpressionData":false,
         "hasGeneticData":false,
         "hasClinicalData":false
      }
   ],
   "totalNumberOfResults":114,
   "paging":{
      "next":"/repo/v1/dataset?offset=2&limit=1"
   }
}

...

For links to more documentation and info about the ssl cert see PLFM-142 142

Gotchas Getting Started with Beanstalk

...

I created a key pair in US West and was confused when I couldn't get beanstalk to use that key pair.
- Beanstalk is only in US East so you have to make and use a key pair from US East
- Get the key pair PlatformKeyPairEast from fremontbelltown
I could not ssh to my box even though I had the right key pair and the hostname.
- I needed to edit the default firewall setttings to open up port 22
My serlvet didn't work right away and I wanted to look at stuff on disk.
Error: java.lang.NoClassDefFoundError: javax/servlet/jsp/jstl/core/Config
- In a tomcat container, such as Elastic Beanstalk, you have to include jstl.jar manually, hence this entry.
  Code Block
  <dependency> <groupId>javax.servlet</groupId> <artifactId>jstl</artifactId> <version>1.2</version> </dependency>
Here's what your deployment might look like when things are working well:

...

Versions Compared

Old Version 49

New Version 50

Key

Gotchas Getting Started with Beanstalk