Versions Compared

Old Version 64

changes.mady.by.user Nicole Deflaux

Saved on

compared with

New Version 65

changes.mady.by.user Nicole Deflaux

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

You can also start with a less expensive instance type easily upgrade to a larger size in this same manner. One thing to note is that you cannot switch between 32bit to 64bit OS - choose well for your initial choice.

S3 How To's

How to enforce HTTPS-only access to S3

We enforce HTTPS-only access to S3 for all buckets. The bucket policies can be found here: http://sagebionetworks.jira.com/source/browse/PLFM/trunk/configuration/s3Policies

Run a Report to Know Who has Accessed What When

...

Here is what a configured job looks like: Image Removed

Image Added

And here is some sample output from the job.  Note that:

  • All Sage employees will have their sagebase.org username as their IAM username
  • Platform users register with an email address and we will use that email address as their IAM username.
  • User d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 is the platform@sagebase.org user (also known as Brian Holt :-).  In general, you should only see activity from that user when we are using BucketExplorer to manage our files in S3. Code Block arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:19:32:44.
    Code Block
    
arn:aws:iam::325565585839:user/prod-nicole.deflaux@sagebase.org	[09/Aug/2011:01:07:49 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:564621/0.0.0/mouse_model_of_sexually_dimorphic_atherosclerotic_traits.phenotype.zip

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:48:49 +0000]	REST.GET.BUCKET	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:55:47 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:58BUCKETPOLICY	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:55:53 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:45BUCKET	-

...

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:56:30 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17ACL	5031/0.0.0/rClient/5030/sangerIC50.zip

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:56:30 +0000]	REST.GETHEAD.LOGGING_STATUS	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19OBJECT	5031/0.0.0/rClient/5030/sangerIC50.zip

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[09/Aug/2011:01:56:30 +0000]	REST.HEADGET.OBJECT	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:19:32:40 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:17:47:46 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.BUCKET	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:42 +0000]	REST.GET.LOCATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[17/Feb/2011:01:48:51 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.ACL	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:19 +0000]	REST.GET.ACL	human_liver_cohort.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.HEAD.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:17 +0000]	REST.GET.NOTIFICATION	-

arn:aws:iam::325565585839:user/nicole.deflaux	[18/Feb/2011:23:42:57 +0000]	REST.GET.ACL	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[17/Feb/2011:01:55:44 +0000]	REST.GET.OBJECT	mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:13:42 +0000]	REST.GET.OBJECT	human_liver_cohort/readme.txt

arn:aws:iam::325565585839:user/test	[16/Feb/2011:23:22:02 +0000]	REST.GET.OBJECT	human_liver_cohort/expression/expression.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:23:06:17 +0000]	REST.HEAD.OBJECT	human_liver_cohort/readme.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:28:38 +0000]	REST.GET.ACL	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:39:57 +0000]	REST.GET.LOCATION	-

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:09 +0000]	REST.COPY.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	[16/Feb/2011:22:40:16 +0000]	REST.HEAD.OBJECT	bxh_apoe/causality_result/causality_result_adipose_male.txt

. . .


Downloads per file:
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-network.txt	2
bxh_apoe/networks/muscle_male-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-log.txt	2
bxh_apoe/networks/SCN10_BxH-ApoE_Adipose_Male_batch_3_10_coexp.sif	2
bxh_apoe/networks/SCN19_muscle_female_bayesian.sif	2
human_liver_cohort/networks/deLiver_liver_all_adjusted_DE-Octave8_coexp-nodes.txt	2
bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-annotation.txt	2
bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-nodes.txt	2
bxh_apoe/networks/muscle_female-nodes.txt	2
human_liver_cohort/networks/QuickChip_female_bayesian-annotation.txt	2
. . .
bxh_apoe/networks/liver_female_coexp-annotation.txt	2
bxh_apoe/networks/SCN15_BxH-ApoE_Muscle_Female_batch_1_11_coexp.sif	2
bxh_apoe/networks/SCN7_brain_female_bayesian.sif	2
human_liver_cohort/sage_bionetworks_user_agreement.pdf	5
bxh_apoe/phenotype/	1
bxh_apoe/networks/SCN14_BxH-ApoE_Liver_Male_batch_3_4_coexp.sif	2



5031/0.0.0/rClient/5030/sangerIC50.zip

Downloads per file:
-	14
5031/0.0.0/rClient/5030/sangerIC50.zip	3
4621/0.0.0/mouse_model_of_sexually_dimorphic_atherosclerotic_traits.phenotype.zip	1

Downloads per user:
arn:aws:iam::325565585839:user/prod-nicole.deflauxdeflaux@sagebase.org	17
arn:aws:iam::325565585839:user/test	3
1
d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440	93117

Upload a dataset to S3

UPDATE: We no longer use bucket explorer to upload datasets. Instead we now use datasetCsvLoader.py. Run it with --help to see its options. You can still use bucket explorer to browse datasets.

...

For links to more documentation and info about the ssl cert see PLFM-142         142          

Gotchas Getting Started with Beanstalk

...