Versions Compared

Old Version 14

changes.mady.by.user Nicole Deflaux

Saved on

compared with

New Version 15

changes.mady.by.user Nicole Deflaux

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Get your IAM credentials
    • from Nicole
    • or from the backup copy on fremont:/home/ndeflaux/PlatformIAMCreds (if you have the Sage root password)
  2. Then either

How To

Run a Report to Know

...

Who has Accessed What When

Use Elastic MapReduce to run a script on all our logs in the bucket logs.sagebase.org. There are some scripts in bucket emr.sagebase.org/scripts that will do the trick. If you want to change what they do, feel free to make new scripts.

...

And here is some sample output from the job.  Note that:

  • All Sage employees will have their sagebase.org username as their IAM username
  • Platform users register with an email address and we will use that email address as their IAM username.
  • User d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 is the platform@sagebase.org user (also known as Brian Holt :-).  In general, you should only see activity from that user when we are using BucketExplorer to manage our files in S3.
    Code Block
    
arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:44 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:56 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:58 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:45 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.LOGGING_STATUS	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19 +0000]	REST.HEAD.OBJECT	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:40 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:46 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.ACL	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19 +0000]	REST.GET.ACL	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.NOTIFICATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[17/Feb/2011:01:55:44 +0000]	REST.GET.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:13:42 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:22:02 +0000]	REST.GET.OBJECT	human_liver_cohort/expression/expression.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:23:06:17 +0000]	REST.HEAD.OBJECT	human_liver_cohort/readme.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:28:38 +0000]	REST.GET.ACL	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:39:57 +0000]	REST.GET.LOCATION	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:09 +0000]	REST.COPY.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:16 +0000]	REST.HEAD.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

. . .


Downloads per file:
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-network.txt	2
bxh_apoe/networks/muscle_male-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-log.txt	2
bxh_apoe/networks/SCN10_BxH-ApoE_Adipose_Male_batch_3_10_coexp.sif	2
bxh_apoe/networks/SCN19_muscle_female_bayesian.sif	2
human_liver_cohort/networks/deLiver_liver_all_adjusted_DE-Octave8_coexp-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-annotation.txt	2
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-nodes.txt	2
bxh_apoe/networks/muscle_female-nodes.txt	2
human_liver_cohort/networks/QuickChip_female_bayesian-annotation.txt	2
. . .
bxh_apoe/networks/liver_female_coexp-annotation.txt	2
bxh_apoe/networks/SCN15_BxH-ApoE_Muscle_Female_batch_1_11_coexp.sif	2
bxh_apoe/networks/SCN7_brain_female_bayesian.sif	2
human_liver_cohort/sage_bionetworks_user_agreement.pdf	5
bxh_apoe/phenotype/	1
bxh_apoe/networks/SCN14_BxH-ApoE_Liver_Male_batch_3_4_coexp.sif	2



Downloads per user:
arn:aws:iam::325565585839:user/nicole.deflaux	17
arn:aws:iam::325565585839:user/test	3
d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	931

Upload a dataset to S3

For the initial upload, a GUI tool called BucketExplorer (http://www.bucketexplorer.com/) is used. Uploads are done from the internal host fremont.fhcrc.org using the local access account 'platform', with the same password as the platform@sagebase.org account. The most efficient way to connect is to use an NX protocol client (http://www.nomachine.com/download.php) to get a virtual desktop as the user platform. Once connected the preconfigured BucketExplorer can be found in the application menu in the lower left corner of the screen.

...