...

Note that this also includes the ExporterDate. This is exported to Synapse as UploadDate, but it is different from the UploadDate because the UploadDate represents that comes from the health data record in DDB. The UploadDate in DDB tracks when the health data was uploaded to Bridge (generally yesterday) while the ExporterDate represents when the Exporter was run and when the data hit Synapse. The design decision was made this way since researcher's would want to operate on "today's data drop" (ExporterDate) while the Exporter needs to set a cutoff for exporting (all data up through the end of yesterday, that is, UploadDate). The ExporterDate (UploadDate in Synapse) tracks when the data was upload to Synapse. Since the Bridge Exporter runs every day at ~2am and pulls yesterday's data, these dates are (in normal circumstances) always different.

This was done because the ExporterDate and UploadDate were done for very similar but different use cases. The UploadDate in DDB is for the Bridge Exporter to keep track of which data set it needs to export to Synapse. The ExporterDate (UploadDate in Synapse) is for researchers to keep track of which data they have processed, and to determine "today's data drop".

(worker) ExportWorker - An asynchronous execution. The Worker Manager will create one for each record, and may make more than if the record needs, for example, both the Health Data handler and the AppVersion handler. This is executed in the Worker Manager's asynchronous thread pool and tracked in the ExportTask's asynchronous execution queue.

...

• schedulerName (hash key) - Matches the Lambda function name. Used to distinguish between devo, staging, and prod.
• sqsQueueUrl - SQS queue to write requests to
• timeZone - Currently configured to America/Los_Angeles (equivalent to Pacific Time) for all envs. In the future, if we need to launch Bridge-EX stacks in other regions, this may have other values.
• requestOverrideJson - Optional. Request template that the Bridge-EX-Scheduler uses and fills in "date" with yesterday's date. Generally used for specialized stacks with special parameters or for testing. Example:

{

...

  "studyWhitelist":["api", "breastcancer", "parkinson"],

...

  "sharingMode":"PUBLIC_ONLY"

...

}

(ddbPrefix)SynapseMetaTables - Bridge-EX automatically writes to this table to keep track of meta tables (specifically appVersion tables and status tables). The key is the table name, generally of the form "parkinson-appVersion" or "parkinson-status", and it maps to the Synapse table ID. Bridge-EX uses this table to remember if it's already created a table, and if so, where to find that table.

(ddbPrefix)SynapseTables - Similar to SynapseMetaTables, Bridge-EX automatically writes to this table to keep track of tables, in this case, health data tables. The key is the schema name, flattened into the form "parkinson-TappingActivity-v6", which also maps to Synapse table IDs.

...

1. Make the Bridge-EX-Scheduler changes in your local repository and commit to the root in GibHub (generally via pull request): https://github.com/Sage-Bionetworks/Bridge-EX-Scheduler
2. In your local repository, run "mvn verify", then upload target/Bridge-EX-Scheduler-2.0.jar to AWS Lambda using the AWS Lambda console.
   1. Unfortunately, Travis doesn't support automated deployments of Java to AWS Lambda, so we have to do it manually.
3. To test, click the "Test" button in AWS Lambda.

Troubleshooting

...

1. in GibHub (generally via pull request): https://github.com/Sage-Bionetworks/Bridge-EX-Scheduler
2. In your local repository, run "mvn verify", then upload target/Bridge-EX-Scheduler-2.0.jar to AWS Lambda using the AWS Lambda console.
   1. Unfortunately, Travis doesn't support automated deployments of Java to AWS Lambda, so we have to do it manually.
3. To test, click the "Test" button in AWS Lambda.

Troubleshooting

Logs

NOTE: We're having data loss issues with Logentries. See Logentries support ticket (https://support.logentries.com/hc/en-us/requests/12415) and

Logs can be found at https://logentries.com/. Credentials to the root Logentries account can be found at belltown:/work/platform/PasswordsAndCredentials/passwords.txt. Alternatively, get someone with account admin access to add your user account to Logentries.

...

Redrives

Limitations

Legacy Hacks

* upload freeform text as attachments

* converting old surveys to health data

More Info

Bridge Data Pipeline

...