Versions Compared

Old Version 48

changes.mady.by.user Nicole Deflaux

Saved on

compared with

New Version 49

changes.mady.by.user Nicole Deflaux

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Check the AWS status console to see if they are reporting any problems http://status.aws.amazon.com/
  2. Check the most recent messages on the forums https://forums.aws.amazon.com/index.jsp Problems often get reported there first.
  3. If you still do not find evidence that the problem is AWS's fault, search the forums for your particular issue. Its likely that someone else has run into the same exact problem in the past.
  4. Still no luck? Ask your coworkers and/or post a question to the forums.

Run a Report to Know Who has Accessed What When

Use Elastic MapReduce to run a script on all our logs in the bucket logs.sagebase.org. There are some scripts in bucket emr.sagebase.org/scripts that will do the trick. If you want to change what they do, feel free to make new scripts.

Here is what a configured job looks like:
Image Removed

And here is some sample output from the job.  Note that:

...

How to save money on the AWS bill

If you use EBS-backed AMIs you can "stop" (not "terminate") your instance when you are not using it. Your root partition and other EBS volumes stick around and you are only charged for EBS usage while the instance is "stopped". When you need to use it again you "start" the instance and then re-start your applications.

You can also start with a less expensive instance type easily upgrade to a larger size in this same manner. One thing to note is that you cannot switch between 32bit to 64bit OS - choose well for your initial choice.

Image Added

S3 How To's

Run a Report to Know Who has Accessed What When

Use Elastic MapReduce to run a script on all our logs in the bucket logs.sagebase.org. There are some scripts in bucket emr.sagebase.org/scripts that will do the trick. If you want to change what they do, feel free to make new scripts.

Here is what a configured job looks like:
Image Added

And here is some sample output from the job.  Note that:

  • All Sage employees will have their sagebase.org username as their IAM username
  • Platform users register with an email address and we will use that email address as their IAM username.
  • User d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 is the platform@sagebase.org user (also known as Brian Holt :-).  In general, you should only see activity from that user when we are using BucketExplorer to manage our files in S3.
    Code Block
    arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:44 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:56 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:58 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:45 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.LOGGING_STATUS	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19 +0000]	REST.HEAD.OBJECT	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:40 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:46 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.ACL	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19 +0000]	REST.GET.ACL	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.NOTIFICATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[17/Feb/2011:01:55:44 +0000]	REST.GET.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:13:42 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:22:02 +0000]	REST.GET.OBJECT	human_liver_cohort/expression/expression.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:23:06:17 +0000]	REST.HEAD.OBJECT	human_liver_cohort/readme.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:28:38 +0000]	REST.GET.ACL	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:39:57 +0000]	REST.GET.LOCATION	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:09 +0000]	REST.COPY.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:16 +0000]	REST.HEAD.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

. . .


Downloads per file:
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-network.txt	2
bxh_apoe/networks/muscle_male-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-log.txt	2
bxh_apoe/networks/SCN10_BxH-ApoE_Adipose_Male_batch_3_10_coexp.sif	2
bxh_apoe/networks/SCN19_muscle_female_bayesian.sif	2
human_liver_cohort/networks/deLiver_liver_all_adjusted_DE-Octave8_coexp-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-annotation.txt	2
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-nodes.txt	2
bxh_apoe/networks/muscle_female-nodes.txt	2
human_liver_cohort/networks/QuickChip_female_bayesian-annotation.txt	2
. . .
bxh_apoe/networks/liver_female_coexp-annotation.txt	2
bxh_apoe/networks/SCN15_BxH-ApoE_Muscle_Female_batch_1_11_coexp.sif	2
bxh_apoe/networks/SCN7_brain_female_bayesian.sif	2
human_liver_cohort/sage_bionetworks_user_agreement.pdf	5
bxh_apoe/phenotype/	1
bxh_apoe/networks/SCN14_BxH-ApoE_Liver_Male_batch_3_4_coexp.sif	2



Downloads per user:
arn:aws:iam::325565585839:user/nicole.deflaux	17
arn:aws:iam::325565585839:user/test	3
d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	931

...

BucketExplorer is very efficient, and will do hash comparisons and only transfer what files have changed. One can also get a visual comparison of what files have changed using the 'Comparer' button. During the transfer, the program will parallelize the transfer into 20 streams for very efficient use of outgoing bandwidth to the cloud.

IAM How To's

Create a new IAM group

You can install the IAM tools on your machine http://docs.amazonwebservices.com/IAM/latest/GettingStartedGuide/index.html?GetTools.html or use them on fremont.

...

Code Block
ssh you@fremont
cd /work/platform
bash
source bin/setupIamClient.sh
iam-usercreate -u bruce.hoff -g Admins -k -v > PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt

Then give the user their credentials file.

How to save money on the AWS bill

If you use EBS-backed AMIs you can "stop" (not "terminate") your instance when you are not using it. Your root partition and other EBS volumes stick around and you are only charged for EBS usage while the instance is "stopped". When you need to use it again you "start" the instance and then re-start your applications.

You can also start with a less expensive instance type easily upgrade to a larger size in this same manner. One thing to note is that you cannot switch between 32bit to 64bit OS - choose well for your initial choice.

...

 -u bruce.hoff -g Admins -k -v > PasswordsAndCredentials/PlatformIAMCreds/bruce.hoff_creds.txt

Then give the user their credentials file.

RDS and MySQL How To's

How to connect to RDS

...

For links to more documentation and info about the ssl cert see PLFM-142 142  

Gotchas Getting Started with Beanstalk

...