Problem Statement & Objectives

Context

The Synapse Workers application is a core, high-throughput platform component currently deployed as a monolithic Java .war file on Apache Tomcat instances via AWS Elastic Beanstalk. To align with organizational infrastructure standards, reduce operational overhead, and achieve granular scalability, there is an immediate business push to migrate this workload to AWS ECS Fargate.

Observed Failure Mode

Migration attempts to ECS Fargate resulted in severe runtime instability that was resolved in stages:

Stage 1: Memory Pressure & Task Churn (Resolved)

Initial deployment (prod-587, 2 vCPU × 8 containers, Java 11/Spring 5) configured the JVM to use 85% of task memory. Workers hit the 8GB memory ceiling, causing OOM kills, ECS task replacements, HTTP 499 health check failures, and cascading task churn. Connection spikes correlated with replacement events (new containers opening fresh pools while old containers drain).

Resolution: Reduced JVM heap to 50% of task memory. Task churn resolved completely.

Stage 2: Supervisor Thread Starvation (Unsolved — This Document's Focus)

After resolving memory pressure, change messages were replayed to trigger a full secondary index rebuild (tables, OpenSearch, object snapshots). With stable containers (no churn), the following was observed:

Stack traces confirm: supervisor threads failed to call changeMessageVisibility before receipt expiry:

AmazonSQSException: Value [receipt] for parameter ReceiptHandle is invalid.
Reason: Message does not exist or is not available for visibility timeout change.
(Service: AmazonSQS; Status Code: 400; Error Code: InvalidParameterValue)

A second test with 6 vCPU × 6 containers reduced renewal failures from thousands to hundreds — more CPU helped supervisors run more often — but did not eliminate the problem. It remained orders of magnitude worse than EC2.

The unsolved problem: Same application, same total vCPU (16), same workload. Near-zero supervisor failures under EC2, thousands under ECS Fargate. The only variable is how the runtime schedules threads.

This document analyzes the root cause of this behavior and outlines a Java 21 Virtual Threads solution.

Current Worker Framework Architecture

The framework acts as an application-level distributed scheduler. It orchestrates 120+ unique worker types across a cluster of nodes using two primary tiers of concurrency control: Global Cluster Limits (Database Semaphores) and Node-Level Limits (maxThreadsPerMachine).

Core Architectural Lifecycle

For every worker type, a dedicated Quartz scheduler trigger acts as the Supervisor Thread. The lifecycle proceeds as follows:

sequenceDiagram
    autonumber
    participant QZ as Quartz Trigger (Supervisor)
    participant DB as DB Global Semaphore
    participant SQS as AWS SQS Queue
    participant TP as Dedicated Thread Pool

    rect rgb(230, 245, 255)
    note right of QZ: 1. Global Concurrency Check
    QZ->+DB: attemptToGetLock(type, semaphoreMaxLockCount)
    alt Lock Denied
        DB-->QZ: Return False
        note right of QZ: Exit loop, wait for next Quartz trigger
    else Lock Granted
        DB-->>-QZ: Return True
    end
    end

    rect rgb(240, 255, 240)
    note right of QZ: 2. Main Supervisor Loop (while true)
    loop Active Execution
        alt Worker Type is SQS-Driven
            QZ->+SQS: Standard HTTP GET (Batch Size = maxThreadsPerMachine)
            SQS-->>-QZ: Return Messages
        end
        loop For Each Message / Timer Event
            QZ->>TP: Submit Worker Callable (Bounded by maxThreadsPerMachine)
        end

        rect rgb(255, 240, 240)
        note right of QZ: 3. Health & Lease Monitoring
        loop Every lockTimeout/3 seconds
            note right of QZ: If running time > 2/3 of Timeout:
            QZ->>SQS: Renew Message Visibility
            QZ->>DB: Renew Semaphore Lock Leasetime
            note right of QZ: If Worker finishes:
            QZ->>SQS: Delete Message
        end
        end
    end
    end

I/O Profile

The worker application is intensely I/O-intensive. Workers spend the vast majority of their lifecycles communicating off-box:

• Performing heavy relational database queries and massive batch updates via JDBC.

• Making blocking AWS API requests (SQS, S3, API Gateway WebSockets).

• Reading and writing large, transient datasets to temporary disk files.

Infrastructure Execution Models: EC2 vs. ECS Fargate

The failure on Fargate is an infrastructure mechanics problem. The underlying operating system handles multi-threading fundamentally differently in these two environments — even at identical vCPU allocations.

The EC2 Environment: Native Hardware Slicing

On our Elastic Beanstalk cluster, we utilize dedicated EC2 instances with 2 vCPUs each (8 instances, 16 total vCPU).

• The Linux CFS Scheduler: The OS utilizes the Completely Fair Scheduler (CFS) with access to dedicated physical CPU cores. The kernel manages our ~240 OS-level threads (120 supervisors + 120 active worker slots) holistically with full preemption authority.

• Supervisor Protection: When supervisor threads sleep during their monitoring phase (via Thread.sleep() in the exponential backoff loop), they surrender CPU time and accumulate "vruntime credit." When they wake, the kernel grants them high scheduling priority, immediately pre-empting worker threads. The supervisor gets its necessary microsecond of execution to renew leases. As a result, renewals are always timely.

The ECS Fargate Environment: The cgroup Bandwidth Tax

On ECS Fargate, applications run inside Firecracker MicroVMs constrained by Linux Control Groups (cgroups). Even at 2 vCPU (our test configuration), the scheduling model is fundamentally different:

[ Fargate Task Container — 2 vCPU ]
├── 120 Supervisor Threads ──┐
└── 120 I/O-Intensive Workers ──┼─> Single cgroup ──> [ cpu.cfs_quota_us = 200ms/100ms ]
                                                       │
                                                       ├── Context-switch overhead for 240+ threads
                                                       ├── I/O wait → threads stay "runnable"
                                                       └── Quota consumed by scheduling overhead
                                                           → Supervisors delayed past lease timeout

• The Greedy Worker Trap: Workers performing I/O (waiting on network sockets, writing files) keep hundreds of OS-level threads registered as "runnable" or "waiting to execute" in the cgroup. Unlike EC2 where the kernel has dedicated cores, Fargate's cgroup must time-slice ALL threads through the quota window.

• The Context-Switching Tax: Managing 240+ heavy OS-level threads on a Fargate allocation forces the kernel to waste significant portions of its quota performing CPU context-switches rather than executing useful code.

• CFS Bandwidth Throttling: Fargate enforces hard limits via cpu.cfs_quota_us. At 2 vCPU, the container gets 200ms of CPU time every 100ms period. With 240+ active threads and rapid I/O polling, the quota is consumed quickly.

• The Starvation: Unlike EC2 where the kernel's CFS "vruntime credit" mechanism guarantees sleeping threads get priority when they wake, Fargate's cgroup enforcement does not distinguish between "supervisor needs 1μs to renew a lease" and "worker thread returning from a JDBC call." Supervisors are delayed until the next quota window, and by then the lease has expired.

Why 6 vCPU helped but didn't fix it: More vCPU increases the quota (600ms per 100ms), reducing the probability of supervisors being delayed past the renewal deadline — but does not eliminate it. The fundamental problem is that platform thread scheduling under cgroup enforcement treats all threads equally, regardless of their criticality or CPU-time needs.

The Solution: Java 21 Virtual Threads

By leveraging our recent upgrade to Java 21 and Spring 6.2, we can fix this scheduling problem entirely inside the Java Virtual Machine using Project Loom Virtual Threads.

How Virtual Threads Bypass Throttling

With Virtual Threads, the 240+ supervisor and worker threads are no longer mapped 1:1 to heavy Linux OS threads. Instead, they become lightweight Virtual Threads (vthreads) managed in memory by the JVM. The JVM mounts these vthreads onto a small pool of Platform Carrier Threads whose size matches the vCPU count (e.g., 2 carrier threads for a 2 vCPU Fargate task).

Exploiting I/O-Intensive Workloads

Virtual threads are designed specifically for I/O-bound architectures:

• Every time a worker thread blocks on a JDBC call, SQS request, or file write, the JVM automatically unmounts that virtual thread from the carrier.

• The carrier thread never blocks. It immediately picks up another virtual thread — such as a supervisor loop that just woke from sleep.

• When a supervisor calls Thread.sleep(waitTimeMs), the VT unmounts from the carrier — no platform thread consumed. When it wakes, the JVM immediately mounts it on a carrier without needing OS scheduler cooperation.

Because scheduling is handled cooperatively inside the JVM rather than by the Linux kernel, OS-level context switching drops to near zero. The cgroup quota is spent purely on executing code, preventing Fargate from throttling the container. Supervisors always get timely execution for lease renewals.

Why Spring Boot's spring.threads.virtual.enabled=true Does NOT Apply

Spring Boot's virtual thread configuration only affects Spring-managed TaskExecutor beans used by @Async and @Scheduled annotations. Our workers application:

• Has zero@Async annotations

• Uses Quartz's internal thread pool (not Spring's task scheduling)

• Uses a custom Executors.newCachedThreadPool() in ConcurrentManagerImpl

Therefore, we must manually configure Virtual Threads for each thread-creation point. A Spring Boot migration is a worthwhile modernization initiative but is orthogonal to the Fargate VT fix and is not a prerequisite.

Concrete Implementation Changes

Quartz VT integration (Spring 6.1+ native approach via SchedulerFactoryBean.setTaskExecutor()):

<bean id="virtualThreadTaskExecutor"
      class="org.springframework.core.task.SimpleAsyncTaskExecutor">
    <property name="virtualThreads" value="true"/>
</bean>

<bean id="mainScheduler"
      class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="taskExecutor" ref="virtualThreadTaskExecutor"/>
    <property name="triggers" ref="workerTriggersList"/>
</bean>

This overrides Quartz's internal SimpleThreadPool with Spring's Virtual Thread-enabled executor. All 120+ supervisor threads become Virtual Threads mounted on only 2 carrier threads (matching the 2 vCPU allocation).

Architectural Alternative: Spring Cloud AWS SqsListener

An alternative to our bespoke Quartz framework is migrating to Spring Cloud AWS 3.x messaging using the @SqsListener annotation.

Benefits of the Standard

• Connection Efficiency: Natively utilizes the AWS Java SDK v2 Async Client powered by a non-blocking Netty event loop.

• Declarative Retries: Built-in support for backoffs, dead-letter queues (DLQs), and framework-managed message deletions.

The Risk: Long-Running Batch Operations

Our current supervisor loop is highly resilient because it is decoupled from worker execution; it monitors and renews leases independently of what the worker is doing.

Under @SqsListener, if a worker executes a long-running database batch update, it blocks the thread. While a Visibility object can be injected for manual timeout extension, relying on developers to place renewal triggers inside complex business logic is fragile.

Additionally, our ConcurrentWorkerStack supports FIFO queue ordering (receiving one message at a time per group to preserve sequence), which would be lost with @SqsListener.

Therefore, a complete rewrite to @SqsListener is not recommended as an immediate fix.

Architectural Alternative: Functional Worker Tiering

Instead of running all 120 worker types on a single monolithic Fargate container, we can divide the application using Spring Profiles into functional "Tiers" (e.g., profile-tier-high, profile-tier-bulk, profile-tier-scheduled).

We would deploy 4 to 6 separate Fargate Services, each hosting only a subset (~20-30) of the worker types.

Do We Still Need Global Database Semaphores?

Yes, we absolutely still need the global database semaphores. Dividing workers into tiers optimizes the internal thread density of an individual container, but does nothing to coordinate concurrency across the cluster. For example, if the SearchIndexWorker is restricted to running on only 4 nodes globally (semaphoreMaxLockCount=4), and we scale our "Bulk Tier" Fargate task out to 12 containers, the database semaphore remains our only mechanism to guarantee that only 4 of those 12 containers are actively indexing at any moment.

Strategic Recommendations & Phased Rollout Plan

Phase 1: Virtual Thread Adoption & Connection Pool Migration

• Action: Retain the current monolithic architecture and Quartz supervisor loops. Manually enable Virtual Threads at each thread-creation point (see implementation table above). Migrate all connection pools from DBCP2 to HikariCP. Convert synchronized blocks to ReentrantLock where they guard I/O.

• Justification: This addresses the root cause of the Fargate failure (supervisor thread starvation due to cgroup scheduling) with minimal architectural redesign. It preserves our reliable lease-renewal loop while dropping the platform thread footprint to 2 carrier threads on a 2 vCPU task.

• Scope: Both the workers WAR and repository WAR, since they share the same connection pools and synchronized code paths.

Success Criteria

• Renewal failure rate ≤ EC2 baseline (~64 per full index rebuild cycle)

• No jdk.VirtualThreadPinned events lasting > 1ms

• HikariCP connection wait times (HikariPool.Wait) < 500ms

• No database CPU spikes from duplicate processing

• Stable task count (no OOM-driven or health-check-driven replacements)

Validation Plan

Deploy VT-enabled workers to a Fargate test stack (2 vCPU × 8 containers). Replay change messages to trigger full secondary index rebuild. Compare renewal failures, connection metrics, and processing throughput to EC2 baseline.

Failure Criteria & Rollback

Phase 1 is considered unsuccessful if load testing shows recurring jdk.VirtualThreadPinned events above threshold, continued SQS visibility-timeout or DB semaphore lease forfeitures, sustained HikariCP connection wait times above 500ms, or renewed database CPU spikes. If any of these occur, the Fargate deployment stays blocked while the specific issue is investigated. The application can continue running on EC2 since VT changes are safe on multi-vCPU EC2 instances.

Phase 2: Functional Tiering & Isolation (Mid-Term)

• Action: Segment the 120 workers into 4 distinct Spring Profiles based on priority and resource profiles (e.g., Fast/UI-driven, Bulk/Heavy, Scheduled Timers). Deploy these as independent ECS Fargate services.

• Justification: Provides absolute protection against a single rogue CPU-bound worker crashing the entire ecosystem, and enables independent auto-scaling per tier to optimize hosting costs.

Risk Mitigation & Virtual Thread Guardrails

While Virtual Threads solve the cgroup scheduling problem, they introduce two runtime risks specific to the JVM layer: Carrier Thread Pinning and Database Connection Pool Starvation.

Carrier Thread Pinning Mitigation

Virtual Threads yield the carrier thread cooperatively at JVM-managed blocking points (e.g., LockSupport.park()). However, if a thread blocks inside a synchronized block or method, the JVM cannot unmount it — the virtual thread becomes pinned to its carrier thread. On a 2 vCPU Fargate task with only 2 carrier threads, pinning events block other virtual threads from making progress.

Detection

# Add to JVM startup flags — logs a full stack trace for every pinning event
-Djdk.tracePinnedThreads=full

Alternatively, monitor the JFR event jdk.VirtualThreadPinned during load testing. Any pinning event lasting longer than a few microseconds should be treated as a blocking bug.

Compliance Checklist — Dependencies

DBCP2 → HikariCP Migration (All 5 Pools)

The codebase has 5 independent BasicDataSource instances that ALL must be migrated:

<!-- REMOVE (all 5 pools) -->
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-dbcp2</artifactId>
    <version>2.9.0</version>
</dependency>

<!-- REPLACE WITH -->
<dependency>
    <groupId>com.zaxxer</groupId>
    <artifactId>HikariCP</artifactId>
    <version>5.1.0</version>
</dependency>

Synchronized Block Audit — Codebase Findings

The principle: any synchronized block guarding code that could perform I/O MUST be converted to ReentrantLock, because ReentrantLock.lock() is a Virtual Thread yield point (VT unmounts from carrier), while synchronized pins the carrier.

Critical (I/O under lock):

Moderate (high contention, no direct I/O):

Low risk (in-memory microsecond operations — acceptable pinning):

• JobTrackerImpl (lines 46, 58, 78): HashMap operations only

• MemoryCountingSemaphoreImpl (lines 39, 71, 96, 120): test/dev in-memory semaphore

Already correct (no action needed):

• CountingSemaphoreImpl: Uses Spring @Transactional, no synchronized blocks

• WebhookMetricsCollector, ThrottleRulesCache, ProjectStorageLimitsManager: Already use ConcurrentHashMap

Example refactoring pattern:

// BEFORE: Pins carrier thread during I/O
public synchronized byte[] fetchFromS3(String key) {
    return s3Client.getObjectAsBytes(req -> req.bucket(bucket).key(key))
                   .asByteArray();
}

// AFTER: Safely yields carrier thread during I/O
private final ReentrantLock lock = new ReentrantLock();
public byte[] fetchFromS3(String key) {
    lock.lock();
    try {
        return s3Client.getObjectAsBytes(req -> req.bucket(bucket).key(key))
                       .asByteArray();
    } finally {
        lock.unlock();
    }
}

Going-forward rule: New code must use ReentrantLock or java.util.concurrent primitives for any block that might contain I/O.

Database Connection Pool Starvation

Virtual Threads make it trivially easy to have hundreds of concurrent tasks unblock simultaneously, but the database connection pool is still finite. With 240+ virtual threads potentially all requesting a connection at the same moment, the pool must act as a bounded throttle.

• Critical change: The current main pool uses MaxTotal = -1 (unbounded, per PLFM-8344). With Virtual Threads, this becomes dangerous — hundreds of VTs can unblock simultaneously and exhaust MySQL's connection limit. HikariCP's bounded maximumPoolSize becomes the required throttle.

• HikariCP maximumPoolSize should be set conservatively (e.g., 20–30 connections) relative to what the MySQL instance can sustain.

• The existing DB Global Semaphore bounds active workers per type across the cluster. Combined with a properly sized HikariCP pool, this provides two layers of back-pressure.

• Validation: Under load test, confirm that connection wait times (HikariPool.Wait metric) remain below 500ms and that no SQLTransientConnectionException (pool timeout) is thrown.

Key takeaway: The DBCP2 → HikariCP migration is the most critical prerequisite. Without it, every database call pins the carrier thread, which on a 2 vCPU Fargate task (2 carrier threads) effectively serializes all work and recreates the exact starvation condition we're trying to fix — just at the JVM level instead of the cgroup level.