Timeouts and deadlocks uploading CSVs to tables

Description

The last two days (Mar 24 and Mar 25), the exporter has been failing to upload some tables to Synapse. In some cases, we're getting an error from Synapse with stack trace

Other times, the upload CSV process fails to complete after 5 minutes (the current timeout configured for Bridge Exporter).

We're using the following code to upload CSVs https://github.com/Sage-Bionetworks/Bridge-Exporter/blob/develop/src/main/java/org/sagebionetworks/bridge/exporter/synapse/SynapseHelper.java#L604

We poll uploadCsvToTableAsyncGet() every second for 5 min. We're currently configured with at most 4 threads, and we throttle 10 requests per second against all Synapse APIs (see https://github.com/Sage-Bionetworks/Bridge-Exporter/blob/develop/src/main/resources/BridgeExporter.conf).

Bridge Exporter has automatic redrives every 15 minutes, maximum of 5 redrives, and it appears that these tables fail to upload deterministically.

Tables affected:
syn8128931
syn10284641
syn15664831
syn12977322
syn15673384
syn18403707
syn18071440

Additionally, in Synapse Website, some of these tables are stuck in "Waiting for the table index to become available..."
syn8128931
syn10284641
syn15664831
syn18403707
syn18071440

Environment

None

Assignee

John Hill

Reporter

Dwayne Jeng

Labels

None

Validator

Dwayne Jeng

Development Area

None

Release Version History

None

Components

Sprint

None

Fix versions

Priority

Critical
Configure