...
Updates are process to a table in blocks of 25K rows. Each block will be issued a version number which also becomes the new version number of every row in that block. If all blocks are processed without failure then the metadata of each block is committed to the table's history in a single transaction making them a permanent part of the table's history. Any failure in any block results in a rollback of all blocks from that update. This makes each update to a table atomic regardless of the size of the update. Raw block data is stored in S3, while the metadata about the blocks is store in the main Synapse MySQL database.
Clustering
Worker Cluster
All table operations are executed by a cluster of workers. This includes, query, updates, uploads, and downloads. This worker cluster is maintained by Amazon's Elastic Beanstalk and is currently configured to scale up/down on CPU usage. We currently run with a minimum of four machines and have scaled up to eight machines under load. In addition to the number of machines, we also control the number of concurrent instance of each worker across the cluster using a counting semaphore system. We also control how many threads each worker type can acquire on each machine. Each worker is triggered by a timer. All worker communication is facilitated using Amazon's SQS. When a worker timer fires, it first attempts to grab one the semaphores for its worker types. If successful, it will pull for messages from its queue. SQS ensures the same message is only issued to one worker at a time and automatically re-issues messages if the worker fails to delete the message in a timely manner.
Database Cluster
While the row "truth" data of a table is stored in S3, and tracked in a single MySQL database, a cluster of database MySQL databases is used to support table queries. A secondary worker (see Worker Cluster above) listens for table change events and attempts to build an index for a table in one of the database instances in the query cluster. This worker will update a table's index while holding an exclusive lock on the table. This workers primary job is to determine the current version of each row and ensure it is reflected in the index. In the future we plan to "rebalanced" the query database cluster by redistributing table incidences based on the load of the of each database machine. Currently every table is indexed in the cluster but we plan to only index "active" tables in the future.
...