...
Anytime data is written to Synapse through the repository services, a message is generated and sent to an Amazon SNS Topic. The topic acts as a message syndication system, that pushes a copy of the message to all Amazon SQS queues that are registered with the topic. Each queue has a dedicated worker process watches is "own" messages pushed to the queue. The following sequence diagram shows how the repository service generates a message in response to a write:
...
- A Change Message is recorded with the original transaction, if the write is committed then, so is the record of the change in the change table.
- Messages are not published until after the transaction commits, so race conditions on message processing is not possible.
- Under normal condition messages are sent immediately.
- A system is in place to detect and re-send any failed messageslost message.
Message processing
While the workers Each queue has its own class of dedicated worker that pops messages from the queue and writes data to a secondary data source. These workers are all bundled into a special application called "workers". The works application is deployed to Elastic Beanstalk in the worksworkers.war , it file. Unlike the repository services, the workers application does not actually handle any web requests (other than administration support). Instead we are utilizing the "elastic" properties of Beanstalk, to manage a cluster of workers. This includes automatically scaling up and down, multi-zone deployment and failure recovery.
The workers application consists of a suite of Spring Beans that are each run on their own timer as part of a larger Quartz scheduler. Each worker timer is gated by a RDS backed semaphore that limit the number of current workers of each type that can be run across the cluster. This allows the work to be more evenly distributed across the cluster.
...