Versions Compared

Old Version 1

changes.mady.by.user Sonia Carlson

Saved on

compared with

New Version Current

changes.mady.by.user Sonia Carlson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This is the template for the agenda for the weekly Dev-Ops meeting. These are the things we should be looking at every week to make sure our services and data pipeline are healthy. (NOTE: The weekly agenda is still being developed, and information on this page is subject to change.)

Table of Contents

Bridge Server

NOTE: Jenkins may be failing to email us. Manually monitor the integ tests, and if it’s a frequent failure to email us, we might need to fix it or replace it.

Alarms

We use https://www.hubspot.com/ as our Dev-Ops ticketing system. Alarms notify us by sending an email to bridgeops@sagebase.org, which automatically creates a ticket in HubSpot. Click on Service → Tickets, then All Open Tickets and look at all open unassigned tickets. If tickets can be resolved right away, add a Note and then close the ticket. If tickets require further action, either assign the ticket to yourself, or close the ticket and link a Jira.

...

  • bridgeserver2-prod-AWSCWRequestErrorAlarm - Errors in the Bridge logs. Common errors include:

  • bridgeworker-prod-AWSCWRequestErrorAlarm - Similar to above, but for the Worker.

  • bridgeserver2-prod-AWSCW4XXAlarm-B675MRYR0N2M - HTTP 4XX’s from Bridge Server (excludes 401s, which happen all the time). This is

  • bridgeserver2-antivirus-prod-VirusScannerLambdaErrorAlarm - Errors in our antivirus scanner. For more documentation on (known) antivirus errors, see Virus Scanning of Files

  • Jenkins build became unstable / is still unstable - Our integration tests failed. See http://build-system.sagebase.org:8081/ This could either mean a bug was found in our code or our dev or staging environments are unhealthy. It is important to keep our integ test pipeline reliable, so that we can reliably catch bugs before they go to production.

Sumo Logic Dashboards

See https://www.sumologic.com/

...

  • Exporter 3.0 Errors (Server) - Yes, this graph is confirmed to work. You shouldn’t ever see any errors here, but if you do, it means that something in the server failed and needs to be investigated.

  • Worker Errors - May indicate that an export failed, so we should look at this closely. There are many errors related to timeouts to table syn26890553, which Dwayne is currently investigating. FitBit errors can currently be ignored.

  • Worker Retryable Errors - This is usually Synapse isn’t writable (Synapse in read-only state during weekly release). Shouldn’t be a problem unless it lasts a long time or happens at unexpected times.

  • BridgeEX numRecords per study - Despite the name, this is actually Exporter 2.0 exports per app. This is only a concern if the graph changes dramatically and unexpected (eg, a line spikes up or disappears completely).

  • Bridge Upload Validation Errors per study - Exporter 2.0 errors in Server. We track these so we know if/when things need to be redriven, but we don’t currently have the tools to dive deep into these.

  • Strict Upload Validation Errors - Exporter 2.0 messages related to incomplete or missing fields in a schema. Currently, none of these apps are configured to block exports on Strict Upload Validation, so this graph can mostly be ignored.

  • BridgeEX Daily Export avg runtime - Exporter 2.0 runtime in seconds. Currently averages around 2400 seconds (~40 minutes). Can be an issue if this spikes dramatically. Becomes an issue if it goes above 4 hours (14400 seconds).

  • BridgeEX Hourly Export avg runtime - A few apps export on an hourly basis. Averages 10-15 seconds. Not really a problem unless it spikes up to several minutes.

  • BridgeEX Errors - Exporter 2.0 errors. Most of these are Synapse not writable. Other errors probably require further action.

  • Upload Auto-Complete numUploads - This is the feature that receives notifications from our Upload bucket and automatically calls Upload Complete for redundancy. This is only a problem if the graph spikes up or plummets unexpectedly.

  • BridgeUDD avg runtime - Bridge User Data Download. Average runtime is about 240 seconds. This worker doesn’t get a lot of traffic, so the graph might be volatile. Probably not an issue unless the graph spikes up to 20-30 minutes or so.

  • BridgeServer 4XX Errors - This graph is equivalent to the 4XX alarm above. This exists mostly so that when the alarm goes off, we can look at the results in Sumo Logic quickly. Might also be an issue if the graph disappears completely.

  • Server Errors - Same as the Server Errors alarm above and exists for the same reason.

  • EX3.0 Participant Versions per App - Count of participant versions being written to app-wide Synapse projects. Mainly for informational purposes, but we should look for unexpected trends. Note that if there is no app-wide Synapse project but there are study-specific Synapse projects, the data point will not show up on this graph.

  • EX3.0 Participant Versions per Study - Same as above, but for studies. TODO Dwayne to make a code change to log this metric using study name instead of study ID.

  • EX3.0 Exports per App - Count of uploads exported to the app-wide Synapse projects.

  • EX3.0 Exports per Study - Count of uploads exported to the study-specific Synapse projects.

  • Append Participant Version Latency - Latency of the append to table Synapse APIs in milliseconds, from the async start until the async get returns a result. We have a client-side timeout of 5 minutes (300,000 seconds).

  • Append Participant Demographics Latency - Latency of the append to table call for participant demographics. Note that this feature has not yet been launched in production, so the graph is currently blank.

  • Antivirus Scanner Count - Number of files scanned by our antivirus scanner. Not a concern unless the line unexpectedly spikes up or plummets or dispapears.

  • Antivirus Scanner Latency - Antivirus scanner latency in milliseconds. TODO Determine what is normal for the antivirus scanner.

  • Antivirus Scanner Errors - See Virus Scanning of Files

  • Antivirus Scanner Results - Result of antivirus scanner. Currently, all results are clean. TODO What does it look like when the antivirus scans a virus? Talk to IT to figure out how we can test this.

  • Promotional SMS (All), Promotional SMS (Failed), Transactional SMS (All), Transactional SMS (Failed) - Promotional SMS (Failed) is only for study burst reminders and the failures are generally Opt Outs. This isn’t worth looking into. However, Transactional SMS (Failed) is used for things like SMS sign-in, so we need to look at all of these.

Example queries: Sumo copySumo Logic Examples

AWS Monitoring

  • Simple Email Service → Reputation Dashboard - Check the bounce rate and the complaint rate. If bounces or complaints get too high, AWS can shut down our ability to send email. This shuts down any study that uses email for authentication (which is a majority of studies).
    Note that the metrics are based on the last 10000 emails, and we send something like 10 per day, so this metric changes very slowly.
    To see individual bounces and complaints, go to https://groups.google.com/a/sagebase.org/g/bridge-bounce-notifications. Bounces are mostly fine. Complaints need to be looked at closely.

  • Simple Notification Service → Text Messaging (SMS) - Scroll down to Delivery Status Logs and look at SMS delivery since the last dev-ops meeting. Failures due to opted out are generally fine. Other failures should be looked at closely, especially if it’s widespread. This is no longer needed, because the logs now go to Sumo Logic.

Redrive Exports

Whenever an export in either 2.0 or 3.0 fails due to a bug or server error, we need to redrive those exports. Our process is as follows:

...

TODO

Jira Legacy
serverSystem JIRA
serverIdba6fb084-9827-3160-8067-8ac7470f78b2
keyDHP-315

Mobile Apps

under construction