...
Cloud Front Private Content
This could be very good for the download use cases.Cloud Front supports the notion of private content. Cloud front urls can be created with access policies such as an expires time and an IP mask.
Pros:
- This would work for the download of protected content use cases.
Cons: - Note that this is likely a bad solution for the EC2/EMR use cases because Cloud Front sits outside of AWS and users will incur the inbound bandwidth charges when they pull the data down to EC2/EMR.
Dev Pay
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?UsingDevPay.html
S3
Skipping a description of public data on S3 because the scenario is very straightforward - if get the URL you can download the resource. For example: http://s3.amazonaws.com/nicole.deflaux/ElasticMapReduceFun/mapper.R
Protected Data Scenario:
- All Sage data is stored on S3 and is not public.
- Users can only discover what data is available via the platform.
- Users can use the data for cloud computation by spinning up EC2 instances and downloading the files from S3 to the hard drive of their EC2 instance. See below for more details on this.
- Users can download the data from S3 to their local systemThere is an additional charge on top of the S3 hosting costs.
Open Questions:
- Since we do not need a CDN for the normal reason (to move often requested content closer to the user to reduce download times), does this buy us much over S3 Pre-Signed URLs? It seems like the only added benefit is an IP mask in addition to the expires time.
Resources:
Dev Pay
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?UsingDevPay.html
S3
Skipping a description of public data on S3 because the scenario is very straightforward - if get the URL you can download the resource. For example: http://s3.amazonaws.com/nicole.deflaux/ElasticMapReduceFun/mapper.R
Protected Data Scenario:
- All Sage data is stored on S3 and is not public.
- Users can only discover what data is available via the platform.
- Users can use the data for cloud computation by spinning up EC2 instances and downloading the files from S3 to the hard drive of their EC2 instance. See below for more details on this.
- Users can download the data from S3 to their local system. See below for more details on this.
- The platform directs users to sign a Sage-specified EULA prior to gaining access to these files in S3.
- Users must have a Sage platform account to access this data for download.
- The platform grants access to this data. See below for details.
- The platform will write to the audit log each time it grants access and to whom it granted access. S3 can also be configured to log all access to resources and this could serve as a means of intrusion detection.
- client IP address is available in the log
- "anonymous" or the users AWS canonical user id will appear in the log
- See proposals below regarding how users might pay for usage.
- The cost of hosting not free.
- Storage fees will apply.
- Bandwidth fees apply when data is uploaded.
- Data can also be shipped via hard drives and AWS Import fees would apply.
- Bandwidth fees apply when data is downloaded out of AWS. There is no charge when it is downloaded inside AWS (e.g., to an EC2 instance).
...
- Best Effort Server Log Delivery:
- The server access logging feature is designed for best effort. You can expect that most requests against a bucket that is properly configured for logging will result in a delivered log record, and that most log records will be delivered within a few hours of the time that they were recorded.
- However, the server logging feature is offered on a best-effort basis. The completeness and timeliness of server logging is not guaranteed. The log record for a particular request might be delivered long after the request was actually processed, or it might not be delivered at all. The purpose of server logs is to give the bucket owner an idea of the nature of traffic against his or her bucket. It is not meant to be a complete accounting of all requests.
- Usage Report Consistency
- It follows from the best-effort nature of the server logging feature that the usage reports available at the AWS portal might include usage that does not correspond to any request in a delivered server log.
- [Log format details | http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?LogFormat.html}
S3 Pre-Signed URLs for Private Content
Pros:
...
- of the nature of traffic against his or her bucket. It is not meant to be a complete accounting of all requests.
- Usage Report Consistency
- It follows from the best-effort nature of the server logging feature that the usage reports available at the AWS portal might include usage that does not correspond to any request in a delivered server log.
- [Log format details | http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?LogFormat.html}
S3 Pre-Signed URLs for Private Content
Pros:
- Simple!
- Scalable
- We control duration of the expiry window.
Cons:
- When the url has not yet expired, it is possible for others to use that same URL to download files.
- If a user gets her download url, and does not use it right away, she'll need to reload the WebUI page to get another or re-do the R prompt command.
Open Questions:
- How does this work with the new support for partial downloads?
Resources:
- "Query String Request Authentication Alternative: You can authenticate certain types of requests by passing the required information as query-string parameters instead of using the Authorization HTTP header. This is useful for enabling direct third-party browser access to your private Amazon S3 data, without proxying the request. The idea is to construct a "pre-signed" request and encode it as a URL that an end-user's browser can retrieve. Additionally, you can limit a pre-signed request by specifying an expiration time."
- http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?S3_QSAuth.html
- http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html
...
- Is there a cap on the number of users for IAM?
- Confirm that IAM only intended for managing groups and users where the base assumption is all activity is rolling up to a single AWS bill.
Resources:http://docs.amazonwebservices.com/IAM/latest/UserGuide/index.html?UsingWithS3.html
...
EBS snapshot ACL
Open questions:
- What is the max grant number for this?
Resources
What is the max grant number for this?
Custom Proxy
All files are kept completely private on S3 and we write a custom proxy that allows users with permission to download files whether to locations outside AWS or to EC2 hosts.
Pros:
- full flexibility
Cons:
- we are now the scalability bottleneck
If all else fails we can do this, but it will be more work operationally to manage a fleet of custom proxy servers.
Additional Details
Network Bandwidth
The Pacific Northwest Gigapop is the point of presence for the Internet2/Abilene network in the Pacific Northwest. The PNWGP is connected to the Abilene backbone via a 10 GbE link. In turn, the Abilene Seattle node is connected via OC-192 192 links to both Sunnyvale, California and Denver, Colorado.
PNWPG offers two types of Internet2/Abilene interconnects: Internet2/Abilene transit services and Internet2/Abilene peering at Pacific Wave International Peering Exchange. See Participant Services for more information.
...