Page Comparison

...

Table of Contents

Assumptions

Where

Assume that the initial cloud we target is AWS but we plan to support additional clouds in the future.

For the near term we are using AWS as our external hosting partner. They have agreed to support our efforts for CTCAP. Over time we anticipate adding additional external hosting partners such as Google and Microsoft. Different scientists We imagine that different scientists and/or institutions will want to take advantage of different clouds.

...

For now we are assuming that we are dealing with files. Later on we can also envision providing access to data stored in a database and/or a data warehouse.

Who

Assume tens of thousands of users will eventually use the platform.

How

Assume that users who only want to download do not need to have an AWS account.

Assume that anyone wanting to interact with data on EC2 or Elastic MapReduce will have an AWS account and will be willing to give us their account number (which is a safe piece of info to give out, it is not an AWS credential).

Design Considerations

metadata
- how to ensure we have metadata for all stuff in the cloud
file formats
- tar archives or individual files on S3?
- EBS block devices per dataset?
file layout
- how to organize what we have
- how can we enforce a clean layout for files and EBS volumes?
- how to keep track of what we have
access patterns
- we want to make the right thing be the easy thing - make it easy to do computation in the cloud
- file download will be supported but will not be the recommended use case
- recommendations and examples from the R prompt for interacting with the data when working on EC2
security
- not all data is public
- encryption or clear text?
  - key management
- one time urls?
- intrusion detection
- how to manage ACLs and bucket policies
  - are there scalability upper bounds on ACLs? e.g., can't add more than X AWS accounts to an ACL
auditability
- how to have audit logs
- how to download them and make use of them
human data and regulations
- what recommendations do we make to people getting some data from Sage and some data from dbGaP and co-mingling that data in the cloud
monitoring - what should be monitored
- access patterns
- who
- when
- what
- how much
  - data foot print
  - upload bandwidth
  - download bandwidth
  - archive to cheaper storage unused stuff
cost
- read vs. write
- cost of allowing writes
- cost of keeping same data in multiple formats
- can we take advantage of the free hosting for http://aws.amazon.com/datasets even though we want to keep an audit log?
- how to meter and bill customers for usage
operations
- how to make it efficient to manage
- reduce the burden of administrative tasks
- how to enable multiple administrators
how long does it take to get files up/down?
- upload speeds - we are on the lambda rail
- shipping hard drives
durability
- data corruption
- data loss
scalability
- if possible, we want to only be the access grantors and then let the hosting provider take care of enforcing access controls and vending data

High Level Use Cases