Section | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Developer AWS Accounts
...
You can find them on our shared servers. When storing passwords locally on your laptop (which already has an encrypted drive, yay!) you might also consider using Password Safe.
Code Block |
---|
/work/platform>hostname
sodo
/work/platform/PasswordsAndCredentials>ls
AtlassianAccountAWSCredentials platformStagingEncryptionKey.txt
crowdServerCertificate SshCertificates
passwords.txt SshKeys
PlatformAWSCredentials StackCredentials
PlatformIAMCreds wildcard-sagebase.org-cert
platformPropertyEncryptionKey.txt
|
...
Window's users can also connect using PuTTY or WinSCP, however you will to first create a PuTTY private key file using puttygen.exe
Here is how to create the private key file:
...
Here is what a configured job looks like:
And here is some sample output from the job. Note that:
...
for the purpose of cutting-and-pasting:
Input Location: s3n://prodlogs.sagebase.org/
Output Location: s3n://emr.sagebase.org/output/report20110809
Mapper: s3n://emr.sagebase.org/scripts/downloadsByUserMapper.py
Reducer: s3n://emr.sagebase.org/scripts/downloadsByUserReducer.py
Amazon S3 Log Path: s3n://emr.sagebase.org/output/debugLogs
And here is some sample output from the job. Note that:
- All Sage employees will have their sagebase.org username as their IAM username
- Platform users register with an email address and we will use that email address as their IAM username.
User
d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440
is the platform@sagebase.org user.Code Block arn:aws:iam::325565585839:user/prod-nicole.deflaux@sagebase.org [09/Aug/2011:01:5507:5349 +0000] REST.GET.BUCKET - ...OBJECT 4621/0.0.0/mouse_model_of_sexually_dimorphic_atherosclerotic_traits.phenotype.zip d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:5648:3049 +0000] REST.GET.ACL 5031/0.0.0/rClient/5030/sangerIC50.zipBUCKET - d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:5655:3047 +0000] REST.HEADGET.OBJECT 5031/0.0.0/rClient/5030/sangerIC50.zipBUCKETPOLICY - d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:5655:3053 +0000] REST.GET.OBJECT 5031/BUCKET - ... d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:56:30 +0000] REST.GET.ACL 5031/0.0.0/rClient/5030/sangerIC50.zip Downloads per file: - 14 d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:56:30 +0000] REST.HEAD.OBJECT 5031/0.0.0/rClient/5030/sangerIC50.zip 3 4621/0.0.0/ d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [09/Aug/2011:01:56:30 +0000] REST.GET.OBJECT 5031/0.0.0/rClient/5030/sangerIC50.zip Downloads per file: - 14 5031/0.0.0/rClient/5030/sangerIC50.zip 3 4621/0.0.0/mouse_model_of_sexually_dimorphic_atherosclerotic_traits.phenotype.zip 1 Downloads per user: arn:aws:iam::325565585839:user/prod-nicole.deflaux@sagebase.org 1 d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 17
...
UPDATE: We no longer use bucket explorer to upload datasets. Instead we now use datasetCsvLoader.py. Run it with --help to see its options. the R client to perform uploads. You can still use bucket explorer to browse datasets.
...
The policy below gives someone full access (list, read, write, delete) to your bucket.
Code Block |
---|
{
"Id": "Policy1305325502034",
"Statement": [
{
"Sid": "Stmt1305324625148",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::THE_BUCKET/*",
"Principal": {
"AWS": [
"THE_PERSONS_AWS_ACCOUNT_NUMBER"
]
}
},
{
"Sid": "Stmt1305325498087",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::THE_BUCKET",
"Principal": {
"AWS": [
"THE_PERSONS_AWS_ACCOUNT_NUMBER"
]
}
}
]
}
|
...
The produser account has full access to all databases, so be careful! The platform user is superuser and should only be used for creating new databases and users, and setting permissions.
Code Block |
---|
~>hostname
sodo
~>/usr/bin/mysql --ssl-ca=/work/platform/PasswordsAndCredentials/SshKeys/mysql-ssl-ca-cert.pem -u produser -h repo.c5sxx7pot9i8.us-east-1.rds.amazonaws.com -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6212
Server version: 5.5.8-log Source distribution
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| innodb |
| performance_schema |
| repositorydb |
+--------------------+
4 rows in set (0.07 sec)
mysql> use repositorydb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
showDatabase changed
mysql> show tables;
+------------------------+
| Tables_in_repositorydb |
+------------------------+
| JDOANALYSISRESULT |
| JDOANNOTATIONS |
| JDODATASET |
| JDODATASETANALYSIS |
| JDODATEANNOTATION |
| JDODOUBLEANNOTATION |
| JDOINPUTDATALAYER |
| JDOLAYERLOCATION |
| JDOLAYERLOCATIONS |
| JDOLONGANNOTATION |
| JDOPROJECT |
| JDORESOURCEACCESS |
| JDOREVISION |
| JDOSCRIPT |
| JDOSTRINGANNOTATION |
| JDOUSER |
| JDOUSERGROUP |
| NUCLEUS_TABLES |
+------------------------+
18 rows in set (0.08 sec)
mysql> desc JDODATASET;
+---------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+--------------+------+-----+---------+----------------+
| ID | bigint(20) | NO | PRI | NULL | auto_increment |
| ANNOTATIONS_ID_OID | bigint(20) | YES | MUL | NULL | |
| CREATION_DATE | datetime | YES | | NULL | |
| CREATOR | varchar(256) | YES | | NULL | |
| DESCRIPTION | varchar(256) | YES | | NULL | |
| NAME | varchar(256) | YES | | NULL | |
| NEXT_VERSION_ID_OID | bigint(20) | YES | MUL | NULL | |
| RELEASE_DATE | datetime | YES | | NULL | |
| REVISION_ID_OID | bigint(20) | YES | MUL | NULL | |
| STATUS | varchar(256) | YES | | NULL | |
+---------------------+--------------+------+-----+---------+----------------+
10 rows in set (0.07 sec)
mysql> select count(*) from JDODATASET ;
+----------+
| count(*) |
+----------+
| 114 |
+----------+
1 row in set (0.08 sec)
mysql> quit
|
...
- Setup MySQL
Create your empty database
Code Block ~/>/usr/local/mysql/bin/mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1910 Server version: 5.5.9 MySQL Community Server (GPL) Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (0.06 sec) mysql> create database test2; Query OK, 1 row affected (0.00 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | | test2 | +--------------------+ 5 rows in set (0.04 sec)
...
Pass a property with your jdbc connection string:
Code Block |
---|
-DJDBC_CONNECTION_STRING=jdbc:mysql://localhost/test2
|
...
The default password is the empty string. If you want a different password, pass it via property -DPARAM2=myUsername
The relevant code is in trunk/lib/jdomodels/src/main/java/org/sagebionetworks/repo/model/jdo/PMF.java
.
How to load data into an empty repository service
You can use this to load data into a repository service whether it is running on beanstalk or locally on your laptop. You can run datasetCsvLoader.py -h
and datasetNuker.py -h
to see more documentation. The code can be found here PLFM/users/deflaux/scripts/DatasetLoader
...
DPARAM2=myUsername
The relevant code is in trunk/lib/jdomodels/src/main/java/org/sagebionetworks/repo/model/jdo/PMF.java
.
Beanstalk FAQ
How do I find the EC2 Instance being used to support a Beanstalk Environment?
...
The tomcat 7 log files are here:
Code Block |
---|
ls /opt/tomcat7/logs/
localhost_access_log.2011-06-13.txt
catalina.out
juli.2011-06-13.log
monitor_catalina.log
tail_catalina.log
monitor_catalina.log.lck
tail_catalina.log.lck
|
The tomcat 6 log files are here:
Code Block |
---|
/var/log
/var/log/tomcat6/monitor_catalina.log.lck
/var/log/tomcat6/tail_catalina.log
/var/log/tomcat6/tail_catalina.log.lck
/var/log/tomcat6/monitor_catalina.log
|
...
The log files are here:
Code Block |
---|
/var/log/httpd/error_log
/var/log/httpd/access_log
/var/log/httpd/elasticbeanstalk-access_log
/var/log/httpd/elasticbeanstalk-error_log
|
...
- I created a key pair in US West and was confused when I couldn't get beanstalk to use that key pair.
- Beanstalk is only in US East so you have to make and use a key pair from US East
- Get the key pair PlatformKeyPairEast from sodo
- I could not ssh to my box even though I had the right key pair and the hostname.
- I needed to edit the default firewall setttings to open up port 22
- My serlvet didn't work right away and I wanted to look at stuff on disk.
- Error: java.lang.NoClassDefFoundError: javax/servlet/jsp/jstl/core/Config
In a tomcat container, such as Elastic Beanstalk, you have to include jstl.jar manually, hence this entry.
Code Block <dependency> <groupId>javax.servlet</groupId> <artifactId>jstl</artifactId> <version>1.2</version> </dependency>
- Here's what your deployment might look like when things are working well:
...