Skip to end of banner
Go to start of banner

Weekly Dev-Ops Agenda

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

This is the template for the agenda for the weekly Dev-Ops meeting. These are the things we should be looking at every week to make sure our services and data pipeline are healthy. (NOTE: The weekly agenda is still being developed, and information on this page is subject to change.)

Bridge Server

Alarms

We use https://www.hubspot.com/ as our Dev-Ops ticketing system. Alarms notify us by sending an email to bridgeops@sagebase.org, which automatically creates a ticket in HubSpot. Click on Service → Tickets, then All Open Tickets and look at all open unassigned tickets. If tickets can be resolved right away, add a Note and then close the ticket. If tickets require further action, either assign the ticket to yourself, or close the ticket and link a Jira.

Note: To search for notes for past tickets, click the magnifying glass icon in the top nav bar. (The Search field in the Ticket view only searches ticket IDs, names, and descriptions, which rarely contains what we want.)

Note: If Hubspot isn’t working, navigate to the mailing list directly at https://groups.google.com/a/sagebase.org/g/bridgeops and look at new messages that have arrived since the last Dev-Ops meeting.

Common types of messages include (this list is not exhaustive):

Sumo Logic Dashboards

See https://www.sumologic.com/

It’s important to look at our dashboards. They inform us of unusual trends that might not individually be enough to set off an alarm, but in aggregate or over long time periods may require intervention. Some alarms indicate that some uploads and exports have failed and may need to be redriven.

A rundown of our dashboards graphs:

  • Exporter 3.0 Errors (Server) - Yes, this graph is confirmed to work. You shouldn’t ever see any errors here, but if you do, it means that something in the server failed and needs to be investigated.

  • Worker Errors - May indicate that an export failed, so we should look at this closely. There are many errors related to timeouts to table syn26890553, which Dwayne is currently investigating. FitBit errors can currently be ignored.

  • Worker Retryable Errors - This is usually Synapse isn’t writable (Synapse in read-only state during weekly release). Shouldn’t be a problem unless it lasts a long time or happens at unexpected times.

  • BridgeEX numRecords per study - Despite the name, this is actually Exporter 2.0 exports per app. This is only a concern if the graph changes dramatically and unexpected (eg, a line spikes up or disappears completely).

  • Bridge Upload Validation Errors per study - Exporter 2.0 errors in Server. We track these so we know if/when things need to be redriven, but we don’t currently have the tools to dive deep into these.

  • Strict Upload Validation Errors - Exporter 2.0 messages related to incomplete or missing fields in a schema. Currently, none of these apps are configured to block exports on Strict Upload Validation, so this graph can mostly be ignored.

  • BridgeEX Daily Export avg runtime - Exporter 2.0 runtime in seconds. Currently averages around 2400 seconds (~40 minutes). Can be an issue if this spikes dramatically. Becomes an issue if it goes above 4 hours (14400 seconds).

  • BridgeEX Hourly Export avg runtime - A few apps export on an hourly basis. Averages 10-15 seconds. Not really a problem unless it spikes up to several minutes.

  • BridgeEX Errors - Exporter 2.0 errors. Most of these are Synapse not writable. Other errors probably require further action.

  • Upload Auto-Complete numUploads - This is the feature that receives notifications from our Upload bucket and automatically calls Upload Complete for redundancy. This is only a problem if the graph spikes up or plummets unexpectedly.

  • BridgeUDD avg runtime - Bridge User Data Download. Average runtime is about 240 seconds. This worker doesn’t get a lot of traffic, so the graph might be volatile. Probably not an issue unless the graph spikes up to 20-30 minutes or so.

  • BridgeServer 4XX Errors - This graph is equivalent to the 4XX alarm above. This exists mostly so that when the alarm goes off, we can look at the results in Sumo Logic quickly. Might also be an issue if the graph disappears completely.

  • Server Errors - Same as the Server Errors alarm above and exists for the same reason.

  • EX3.0 Participant Versions per App - Count of participant versions being written to app-wide Synapse projects. Mainly for informational purposes, but we should look for unexpected trends. Note that if there is no app-wide Synapse project but there are study-specific Synapse projects, the data point will not show up on this graph.

  • EX3.0 Participant Versions per Study - Same as above, but for studies. TODO Dwayne to make a code change to log this metric using study name instead of study ID.

  • EX3.0 Exports per App - Count of uploads exported to the app-wide Synapse projects.

  • EX3.0 Exports per Study - Count of uploads exported to the study-specific Synapse projects.

  • Append Participant Version Latency - Latency of the append to table Synapse APIs in milliseconds, from the async start until the async get returns a result. We have a client-side timeout of 5 minutes (300,000 seconds).

  • Append Participant Demographics Latency - Latency of the append to table call for participant demographics. Note that this feature has not yet been launched in production, so the graph is currently blank.

  • Antivirus Scanner Count - Number of files scanned by our antivirus scanner. Not a concern unless the line unexpectedly spikes up or plummets or dispapears.

  • Antivirus Scanner Latency - Antivirus scanner latency in milliseconds. TODO Determine what is normal for the antivirus scanner.

  • Antivirus Scanner Errors - Yes, this graph is confirmed to work. We don’t currently have a good grasp on which errors are important to look at. See BRIDGE-3392 - Getting issue details... STATUS

  • Antivirus Scanner Results - Result of antivirus scanner. Currently, all results are clean. TODO What does it look like when the antivirus scans a virus? Talk to IT to figure out how we can test this.

AWS Monitoring

  • Simple Email Service → Reputation Dashboard - Check the bounce rate and the complaint rate. If bounces or complaints get too high, AWS can shut down our ability to send email. This shuts down any study that uses email for authentication (which is a majority of studies).
    Note that the metrics are based on the last 10000 emails, and we send something like 10 per day, so this metric changes very slowly.
    To see individual bounces and complaints, go to https://groups.google.com/a/sagebase.org/g/bridge-bounce-notifications. Bounces are mostly fine. Complaints need to be looked at closely.

  • Simple Notification Service → Text Messaging (SMS) - Scroll down to Delivery Status Logs and look at SMS delivery since the last dev-ops meeting. Failures due to opted out are generally fine. Other failures should be looked at closely, especially if it’s widespread.

Mobile Apps

under construction

  • No labels