...
Each queue has its own class of dedicated worker that pops messages from the queue and writes data to a secondary data source. These workers are all bundled into a special application called "workers". The works application is deployed to Elastic Beanstalk in the workers.war file. Unlike the repository services, the workers application does not actually handle any web requests (other than administration support). Instead we are utilizing the "elastic" properties of Beanstalk, to manage a cluster of workers. This includes automatically scaling up and down, multi-zone deployment and failure recovery.
The workers application consists of a suite of Spring Beans that are each run on their own timer Each worker is run off its own Quartz Trigger as part of a larger Quartz scheduler. Each worker timer is gated by Worker concurrency across the cluster is controlled using a RDS backed semaphore that limit the number of current workers of each type that can be run across the cluster. This allows the work to be more evenly distributed across the cluster.Each worker in the suite has its own Amazon SQS queue that is "Semaphore". Each worker is assigned its own semaphore key, the maximum number of concurrent process across the entire cluster, and a maximum run-time (timeout). Some classes of workers must be run one-at-a-time while other are capable of running multiple instances in parallel.
The resilience of these works is provided by a combination of the features of Amazon's SQS, Elastic Beanstalk, and Quartz.