...
Computing Resources
AWS
Google Big Compute
It should be easy to move the solution among different computing resources based on changing business needs. The solution should be tolerant to moderately high failures in worker nodes; it's highly likely that we'd want to use AWS spot instances to reduce cost as many of our use cases do not require immediate results.
Process Initialization
Sun Grid Engine
...
AWS Simple Workflow
AWS Simple Queue Service
Approach
Phase 1 approach:
...
- move queue to AWS SWF or SQS
- capture provenance in Synapse and expand to multistep analysis workflows.
Requirements (or at least desired initial functionality)
...