...
Automatically configuring the nodes for arbitrary requirements;
Resources
Sources of Cloud Computing Resources
AWS
Google Big Compute
It should be easy to move the solution among different computing resources based on changing business needs. The solution should be tolerant to moderately high failures in worker nodes; it's highly likely that we'd want to use AWS spot instances to reduce cost as many of our use cases do not require immediate results.
Process Initialization (allocate nodes, run generic 'worker' process on each node)
Sun Grid Engine
MapReduce/Hadoop
Job Assignment / Monitoring (create queue of jobs, assign jobs to nodes)
AWS Simple Workflow
AWS Simple Queue Service
...