...
All table operations are executed by a cluster of workers. This includes, query, updates, uploads, and downloads. This worker cluster is maintained by Amazon's Elastic Beanstalk and is currently configured to scale up/down on CPU usage. We currently run with a minimum of four machines and have scaled up to eight machines under load. In addition to the number of machines, we also control the number of concurrent instance instances of each worker across the cluster using a counting semaphore system. We also control how many threads each worker type can acquire on each machine. Each worker is triggered by a timer. All worker communication is facilitated using Amazon's SQS. When a worker timer fires, it first attempts to grab one the semaphores for its worker types. If successful, it will pull poll for messages from its queue. SQS ensures the same message is only issued to one worker at a time and automatically re-issues messages if the assigned worker fails to delete the message in a timely manner.
...
While the raw "truth" data of a table is stored in S3, and tracked in a single MySQL database, a cluster of MySQL databases is used to support table queries. A secondary worker (see Worker Cluster above) listens for table change events and attempts to build an index for a table in one of the database instances in the query cluster. This worker will update a table's index while holding an exclusive lock on the table. This workers primary job is to determine the current version of each row and ensure it is reflected in the index. In the future we plan to "rebalancedrebalance" the query database cluster by redistributing table incidences table instances based on the load of the of each database machine. Currently every table is indexed in the cluster but we plan only to only index "active" tables in the future.
...