How to tell if it's down
email received via sageInfo@sagebase.org or synapseeng@sagebase.org or platform@sagebase.org.
Jira ticket
Informal communication (hallway conversation, Slack)
Recovery
Judge the severity
Case 1: Portal is up but Synapse is non-functional (e.g. back end is down):
Step 1: Put "Synapse Down" banner on the portal
Go to app.pendo.io
Log in (verify can do this)
(to fix Pendo install, allowing Synapse website to load in an iframe): https://chrome.google.com/webstore/detail/ignore-x-frame-headers/gleekbfjekiniecknbkamfmkohkpodhe
Go to Guides
Go to System Issues Notice
Click manage guides in app
enter https://www.synapse.org
Set status (Disabled→Public), start date and end date on the notice
Step 2: Assess stack elements
- check which stack number we are on (
curl https://repo-prod.prod.sagebase.org/repo/v1/version or
traceroute repo-prod.prod.sagebase.org on mac or tracert on win)
Log in to aws.amazon.com under the synapse prod account, 325565585839
- Go to RDS console and look at 3 RDS, first ID gen, then others
- Go to Elastic beanstalk, scroll to synapse-prod-<stack> and see if green; Click on 'health" and "monitoring" at LHS.
Step 3: Recovery
Synapse Deployment Notes - 2018