Modernizing and Migrating Synapse Workers to AWS ECS Fargate

Problem Statement & Objectives

Context

The Synapse Workers application is a core, high-throughput platform component currently deployed as a monolithic Java .war file on Apache Tomcat instances via AWS Elastic Beanstalk. To align with organizational infrastructure standards, reduce operational overhead, and achieve granular scalability, there is an immediate business push to migrate this workload to AWS ECS Fargate.

Observed Failure Mode

Initial migration attempts to ECS Fargate MicroVMs resulted in severe runtime instability:

• Lease Forfeiture: Supervisor threads regularly failed to renew SQS message visibility timeouts and database-backed global semaphore locks.

• Cascading Failures: Forfeited locks led to "split-brain" duplicate job execution, causing massive transaction rollbacks and pushing the primary database CPU close to 100% utilization.

• Message Backlogs: SQS messages were delayed, redelivered unnecessarily, and stuck in loops.

This document analyzes the root causes of this behavior and outlines a modern, Java 21-powered path forward.

Current Worker Framework Architecture

The framework acts as an application-level distributed scheduler. It orchestrates 120+ unique worker types across a cluster of nodes using two primary tiers of concurrency control: Global Cluster Limits (Database Semaphores) and Node-Level Limits (maxThreadsPerMachine).

Core Architectural Lifecycle

For every worker type, a dedicated Quartz scheduler trigger acts as the Supervisor Thread. The lifecycle proceeds as follows:

I/O Profile

The worker application is intensely I/O-intensive. Workers spend the vast majority of their lifecycles communicating off-box:

• Performing heavy relational database queries and massive batch updates via JDBC.

• Making blocking AWS API requests (SQS, S3, API Gateway Websockets).

• Reading and writing large, transient datasets to temporary disk files.

Infrastructure Execution Models: EC2 vs. ECS Fargate

The failure on Fargate is entirely an infrastructure mechanics problem. The underlying operating system handles multi-threading fundamentally differently in these two environments.

The EC2 Environment: Native Hardware Slicing

On our current Elastic Beanstalk cluster, we utilize dedicated EC2 instances running bare Linux operating systems with 2 dedicated vCPUs.

• The Linux CFS Scheduler: The operating system utilizes the Completely Fair Scheduler (CFS). The OS manages our 240+ OS-level threads (120 supervisors + 120 active worker slots) holistically.

• Supervisor Protection: When supervisor threads put themselves to sleep via Thread.sleep(1000) during their monitoring phase, they surrender their CPU time. When they wake up, the Linux kernel identifies that they have not consumed CPU resources recently. The kernel grants them high scheduling priority, momentarily pre-empting the I/O-intensive worker threads to give the supervisor its necessary microsecond of execution time. As a result, leases are always renewed on time.

The ECS Fargate Environment: The cgroup Capping Tax

On AWS ECS Fargate, applications do not run on raw virtual hardware; they run inside isolated Firecracker MicroVMs tightly constrained by Linux Control Groups (cgroups).

[ Fargate Task Container ]
├── 120 Supervisor Threads ──┐
└── 120 I/O-Intensive Workers ──┼─> Assigned to single cgroup ──> [ cpu.cfs_quota_us Exceeded ] ──> KVM FREEZES ENTIRE VM
                                                                                                    (Supervisors Cannot Run)

• The Greedy Worker Trap: Because our workers are highly active with I/O (waiting on network sockets, writing to files), they keep hundreds of OS-level threads registered as "active" or "waiting to execute" in the cgroup.

• The Context-Switching Tax: Managing 240+ heavy OS-level threads on a tiny Fargate allocation (e.g., 0.25 or 0.5 vCPU) forces the kernel to waste massive amounts of its assigned quota just performing CPU context-switches.

• CFS Bandwidth Throttling: Fargate enforces hard container limits via cpu.cfs_quota_us. If our task is allocated 0.25 vCPUs, it is allowed exactly 25ms of CPU execution time every 100ms. Because of the sheer thread volume and rapid HTTP short-polling, the application burns through those 25ms in the first few milliseconds of the window.

• The Freeze: Once the quota is blown, the kernel freezes the entire container for the remaining 75ms of the period. Because the container is completely frozen, our supervisor threads are completely starved of execution time. By the time the container unfreezes, the database semaphore or SQS lease window has lapsed, causing the cluster to drop the lock and trigger duplicate processing.

The Solution: Java 21 Virtual Threads

By taking advantage of our recent upgrade to Java 21 and Spring 6, we can fix this scheduling problem entirely inside the Java Virtual Machine using Project Loom Virtual Threads (spring.threads.virtual.enabled=true).

How Virtual Threads Bypass Throttling

When Virtual Threads are enabled, the 240+ supervisor and worker threads are no longer mapped 1:1 to heavy Linux OS threads. Instead, they become lightweight Virtual Threads (vthreads) managed in memory by the JVM. The JVM mounts these vthreads onto a tiny pool of Platform Carrier Threads whose size exactly matches the allocated vCPU count (e.g., exactly 1 carrier thread for a 0.25 or 0.5 vCPU Fargate task).

Exploiting I/O-Intensive Workloads

Virtual threads are designed specifically for I/O-bound architectures.

• Every time a worker thread blocks to perform a JDBC database call, read an SQS message, or write a temporary file, the JVM automatically unmounts that virtual thread from the single OS carrier thread.

• The OS carrier thread never blocks. It immediately picks up another virtual thread—such as a sleeping supervisor loop that just woke up.

Because scheduling is handled cooperatively inside the JVM rather than aggressively by the Linux kernel, OS-level context switching drops to near zero. The cgroup quota is spent purely on executing code rather than thread management, preventing Fargate from throttling the container.

Architectural Alternative: Spring Cloud AWS SqsListener

An alternative to our bespoke Quartz framework is migrating to the industry-standard Spring Cloud AWS 3.x messaging system using the @SqsListener annotation.

Benefits of the Standard

• Connection Efficiency: It natively utilizes the AWS Java SDK v2 Async Client powered by a non-blocking Netty event loop, eliminating the network connection pool pressures that forced us into short-polling.

• Declarative Retries: Built-in support for backoffs, dead-letter queues (DLQs), and framework-managed message deletions.

The Risk: Long-Running Batch Operations

Our current supervisor loop is highly resilient because it is decoupled from worker execution; it monitors and renews leases on a 1-second interval regardless of what the worker is doing.

Under standard @SqsListener behavior, if a worker executes a large, long-running database batch update, it blocks the thread. While you can inject a Visibility object to manually extend timeouts:

@SqsListener("queue-name")
public void process(String payload, Visibility visibility) {
    // If this batch update takes 2 minutes, we must remember to call visibility.extend()
    // inside our batch loops.
}

As observed in our past iterations, relying on individual developers to manually place lease renewal triggers inside complex business logic is fragile and prone to oversight. Therefore, a complete rewrite to @SqsListener is not recommended as an immediate fix.

Architectural Alternative: Functional Worker Tiering

Instead of running all 120 worker types on a single monolithic Fargate container, we can divide the application using Spring Profiles into functional "Tiers" (e.g., profile-tier-high, profile-tier-bulk, profile-tier-scheduled).

We would deploy 4 to 6 separate Fargate Services, each hosting only a subset (~20-30) of the worker types.

Do We Still Need Global Database Semaphores?

Yes, we absolutely still need the global database semaphores. Dividing the workers into tiers optimizes the internal thread density of an individual container, but it does nothing to coordinate concurrency across the cluster. For example, if the SearchIndexWorker is restricted to running on only 4 nodes globally (semaphoreMaxLockCount=4), and we scale our "Bulk Tier" Fargate task out to 12 containers to handle a massive backlog, the database semaphore remains our only mechanism to guarantee that only 4 of those 12 containers are actively indexing data at any one moment.

Strategic Recommendations & Phased Rollout Plan

To minimize engineering friction and de-risk the infrastructure migration, we recommend a strict, two-phase rollout strategy:

Phase 1: Runtime Optimization (Immediate)

• Action: Retain the current monolithic architecture and bespoke Quartz supervisor loops, but enable Java 21 Virtual Threads via application configuration:

  • Once on Spring Boot:

    spring.threads.virtual.enabled=true

  • If we are not on Spring Boot (our current implementation) we will need to manually configure all threads we created to be virtual and remove all thread pools. Note: Virtual threads do not require thread pools since they are so light.

• Justification: This addresses the root cause of the Fargate failure (cgroup throttling via OS thread thrashing) with minimal application logic changes and no architectural redesign. It preserves our highly reliable lease-renewal loop while dropping the platform thread footprint down to 1 or 2 carrier threads. Phase 1 now explicitly includes the required Virtual Thread guardrails, including JDBC driver verification, DBCP2-to-HikariCP migration, synchronous AWS SDK auditing, and remediation of any internal synchronized blocks around I/O.

• Success Criteria: Successful execution of the monolith on a 0.5 vCPU Fargate task under load without triggering lease forfeitures or DB spikes.

• Failure Criteria & Rollback: Phase 1 is considered unsuccessful if load testing or production canary deployment shows recurring jdk.VirtualThreadPinned events above the approved threshold, continued SQS visibility-timeout or DB semaphore lease forfeitures, sustained HikariCP connection wait times above 500ms, or renewed database CPU spikes attributable to duplicate processing. If any of these occur, roll back the Fargate deployment to the current EC2-backed Elastic Beanstalk runtime while the blocking issue is remediated, or temporarily increase Fargate vCPU allocation as a short-term containment measure.

Phase 2: Functional Tiering & Isolation (Mid-Term)

• Action: Segment the 120 workers into 4 distinct Spring Profiles based on priority and resource profiles (e.g., Fast/UI-driven, Bulk/Heavy, Scheduled Timers). Deploy these as independent ECS Fargate services.

• Justification: This provides absolute protection against a single rogue, CPU-bound worker crashing the entire worker ecosystem, and enables independent auto-scaling per tier to optimize AWS hosting costs.

Risk Mitigation & Virtual Thread Guardrails

While Virtual Threads solve the Linux cgroup context-switching tax, they introduce two runtime risks specific to the JVM layer: Carrier Thread Pinning and Database Connection Pool Starvation. This section outlines our mitigation and validation strategies.

Carrier Thread Pinning Mitigation

Virtual Threads yield the carrier thread cooperatively at JVM-managed blocking points (e.g., LockSupport.park()). However, if a thread blocks inside a synchronized block or method, the JVM cannot unmount it — the virtual thread becomes pinned to its carrier thread for the duration of that block, negating the benefit of Virtual Threads for that call. On a 0.5 vCPU Fargate task with only 1 carrier thread, a single pinning event blocks all other virtual threads from making progress.

Detection First

Before refactoring, enable pinning diagnostics to identify actual violations under load:

# Add to JVM startup flags — logs a full stack trace for every pinning event
-Djdk.tracePinnedThreads=full

Alternatively, monitor the JFR event jdk.VirtualThreadPinned during load testing. Any pinning event lasting longer than a few microseconds should be treated as a blocking bug.

Compliance Checklist

     <!-- REMOVE -->
     <dependency>
         <groupId>org.apache.commons</groupId>
         <artifactId>commons-dbcp2</artifactId>
         <version>2.9.0</version>
     </dependency>

     <!-- REPLACE WITH -->
     <dependency>
         <groupId>com.zaxxer</groupId>
         <artifactId>HikariCP</artifactId>
         <version>5.1.0</version>
     </dependency>

// BEFORE: Pins carrier thread during I/Onpublic synchronized byte[] fetchFromS3(String key) {n    return s3Client.getObjectAsBytes(req -> req.bucket(bucket).key(key))n                   .asByteArray();n}nn// AFTER: Safely yields carrier thread during I/Onprivate final ReentrantLock lock = new ReentrantLock();npublic byte[] fetchFromS3(String key) {n    lock.lock();n    try {n        return s3Client.getObjectAsBytes(req -> req.bucket(bucket).key(key))n                       .asByteArray();n    } finally {n        lock.unlock();n    }n}

Database Connection Pool Starvation

Virtual Threads make it trivially easy to have hundreds of concurrent tasks unblock simultaneously, but the database connection pool is still finite. With 240+ virtual threads potentially all requesting a connection at the same moment, we must ensure the pool acts as a bounded throttle.

• HikariCP maximumPoolSize should be set conservatively (e.g., 20–30 connections) relative to what the MySQL 8.4 instance can sustain.

• The existing DB Global Semaphore already bounds active workers per type across the cluster. Combined with a properly sized HikariCP pool, this provides two layers of back-pressure.

• Validation: Under load test, confirm that connection wait times (HikariPool.Wait metric) remain below 500ms and that no SQLTransientConnectionException (pool timeout) is thrown.