...
Use Elastic MapReduce to run a script on all our logs in the bucket logs.sagebase.org. There are some scripts in bucket emr.sagebase.org/scripts that will do the trick. If you want to change what they do, feel free to make new scripts.
Here is what a configured job looks like:
And here is some sample output from the job:
| Code Block |
|---|
arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:19:32:44 +0000] REST.GET.OBJECT human_liver_cohort/readme.txt arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:19:32:56 +0000] REST.GET.OBJECT human_liver_cohort/readme.txt arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:19:32:58 +0000] REST.GET.OBJECT human_liver_cohort/readme.txt arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:17:47:45 +0000] REST.GET.LOCATION - arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:17 +0000] REST.GET.LOGGING_STATUS - arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:19 +0000] REST.HEAD.OBJECT human_liver_cohort.tar.gz arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:19:32:40 +0000] REST.GET.BUCKET - arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:17:47:46 +0000] REST.GET.BUCKET - arn:aws:iam::325565585839:user/nicole.deflaux [17/Feb/2011:01:48:42 +0000] REST.GET.BUCKET - arn:aws:iam::325565585839:user/nicole.deflaux [17/Feb/2011:01:48:42 +0000] REST.GET.LOCATION - arn:aws:iam::325565585839:user/nicole.deflaux [17/Feb/2011:01:48:51 +0000] REST.HEAD.OBJECT mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz arn:aws:iam::325565585839:user/nicole.deflaux [17/Feb/2011:01:48:51 +0000] REST.GET.ACL mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:17 +0000] REST.GET.ACL - arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:19 +0000] REST.GET.ACL human_liver_cohort.tar.gz arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:57 +0000] REST.HEAD.OBJECT mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:17 +0000] REST.GET.NOTIFICATION - arn:aws:iam::325565585839:user/nicole.deflaux [18/Feb/2011:23:42:57 +0000] REST.GET.ACL mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz arn:aws:iam::325565585839:user/test [17/Feb/2011:01:55:44 +0000] REST.GET.OBJECT mouse_model_of_sexually_dimorphic_atherosclerotic_traits.tar.gz arn:aws:iam::325565585839:user/test [16/Feb/2011:23:13:42 +0000] REST.GET.OBJECT human_liver_cohort/readme.txt arn:aws:iam::325565585839:user/test [16/Feb/2011:23:22:02 +0000] REST.GET.OBJECT human_liver_cohort/expression/expression.txt d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [16/Feb/2011:23:06:17 +0000] REST.HEAD.OBJECT human_liver_cohort/readme.txt d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [16/Feb/2011:22:28:38 +0000] REST.GET.ACL - d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [16/Feb/2011:22:39:57 +0000] REST.GET.LOCATION - d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [16/Feb/2011:22:40:09 +0000] REST.COPY.OBJECT bxh_apoe/causality_result/causality_result_adipose_male.txt d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 [16/Feb/2011:22:40:16 +0000] REST.HEAD.OBJECT bxh_apoe/causality_result/causality_result_adipose_male.txt . . . Downloads per file: bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-network.txt 2 bxh_apoe/networks/muscle_male-nodes.txt 2 bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-log.txt 2 bxh_apoe/networks/SCN10_BxH-ApoE_Adipose_Male_batch_3_10_coexp.sif 2 bxh_apoe/networks/SCN19_muscle_female_bayesian.sif 2 human_liver_cohort/networks/deLiver_liver_all_adjusted_DE-Octave8_coexp-nodes.txt 2 bxh_apoe/networks/BxH-ApoE_Liver_Male_batch_3_4_coexp-annotation.txt 2 bxh_apoe/networks/BxH-ApoE_Brain_Male_batch_3_14_coexp-nodes.txt 2 bxh_apoe/networks/muscle_female-nodes.txt 2 human_liver_cohort/networks/QuickChip_female_bayesian-annotation.txt 2 . . . bxh_apoe/networks/liver_female_coexp-annotation.txt 2 bxh_apoe/networks/SCN15_BxH-ApoE_Muscle_Female_batch_1_11_coexp.sif 2 bxh_apoe/networks/SCN7_brain_female_bayesian.sif 2 human_liver_cohort/sage_bionetworks_user_agreement.pdf 5 bxh_apoe/phenotype/ 1 bxh_apoe/networks/SCN14_BxH-ApoE_Liver_Male_batch_3_4_coexp.sif 2 Downloads per user: arn:aws:iam::325565585839:user/nicole.deflaux 17 arn:aws:iam::325565585839:user/test 3 d9df08ac799f2859d42a588b415111314cf66d0ffd072195f33b921db966b440 931 |
...