See

Flush the Bridge Worker Queues

1. Disable the Worker by setting Auto Scaling from Single Instance to Load Balancing, and then setting min and max instances to 0.

2. Use the script on https://github.com/DwayneJengSage/BridgeDJengScripts/blob/master/src/main/java/org/sagebionetworks/bridge/scripts/DownloadSqsMessages.java to extract the SQS messages from the 4 following SQS queues. (Note: The retention period on these queues is 14 days, which is the max for SQS. This step needs to be completed within 14 days.)

   • Bridge-UploadComplete-Notification-DLQ-prod

   • Bridge-UploadComplete-Notification-prod

   • Bridge-WorkerPlatform-DLQ-prod

   • Bridge-WorkerPlatform-Request-prod

3. Wait the participant version table on Synapse to finish spinning.

4. Bring the Worker back up by setting Auto Scaling to Single Instance.

Analyze Worker Messages

Use the following script to sort the worker messages https://github.com/DwayneJengSage/BridgeDJengScripts/blob/master/src/main/java/org/sagebionetworks/bridge/scripts/DownloadSqsMessages.java

The following command-line can be used. You’ll need to run it for each queue.

mvn compile exec:java -Dexec.mainClass=org.sagebionetworks.bridge.scripts.DownloadSqsMessages -Dexec.args="/Users/dwaynejeng/scripts-config-local.json /Users/dwaynejeng/tmp/DownloadSqsMessages-upload-dlq.results" &> ~/tmp/DownloadSqsMessages-upload-dlq.log

Worker Messages

1. cat DownloadSqsMessages-worker.results DownloadSqsMessages-worker-dlq.results > DownloadSqsMessages-worker-combined.results

2. Use https://github.com/DwayneJengSage/BridgeDJengScripts/blob/master/src/main/java/org/sagebionetworks/bridge/scripts/SortWorkerMessages.java

   • mvn compile exec:java -Dexec.mainClass=org.sagebionetworks.bridge.scripts.SortWorkerMessages -Dexec.args="/Users/dwaynejeng/tmp/DownloadSqsMessages-worker-combined.results /Users/dwaynejeng/tmp/redrive-20230626-" &> ~/tmp/SortWorkerMessages.log

3. cat redrive-20230626-uploads | sort | uniq > redrive-20230626-uploads-sorted

4. for i in arc biaffect-3 inv-arc mobile-toolbox; cat redrive-20230626-participants-$i | sort | uniq > redrive-20230626-participants-$i-sorted

Upload Complete Messages

1. Run the following query in Sumo Logic: _source=BridgeServer2-Prod MetricsFilter upload complete "\"status\":200," | parse "\"upload_id\":\"*\"" as uploadId

2. Download the results as a list of upload IDs.

3. tail -n +2 completed-uploads-20230620-to-20230625.csv | sed 's/"//g' | sort | uniq > completed-uploads-20230620-to-20230625 - This removes the header row.

4. cat DownloadSqsMessages-upload.results DownloadSqsMessages-upload-dlq.results > DownloadSqsMessages-upload-combined.results

5. grep org-sagebridge-upload-prod DownloadSqsMessages-upload-combined.results | sed 's/.*"key":"//' | sed 's/".*//' | sort | uniq > DownloadSqsMessages-upload-combined-sorted

6. comm -23 DownloadSqsMessages-upload-combined-sorted completed-uploads-20230620-to-20230625 > redrive-20230626-upload-complete

7. cat redrive-20230626-uploads-sorted redrive-20230626-upload-complete | sort | uniq > redrive-20230626-uploads-combined

Redrive Messages

Participant Versions

Upload redrive-20230626-participants-*-sorted to S3 bucket org-sagebridge-backfill-prod.

Write the following message to the Worker SQS queue:

The redrive participant worker redrives 10 participants per second.

Uploads

Upload redrive-20230626-uploads-combined to S3 bucket org-sagebridge-backfill-prod.

Write the following message to the Worker SQS queue:

The redrive upload worker redrives 0.5 uploads per second.

Miscellaneous Worker Messages

Preventative Action Items