Restoring from a DB Snapshot

An incident of Synapse outage might lead to restoring the database from a snapshot, which is an archive of the database from a prior state and should be considered a last resort in case the running database instance cannot be recovered itself. This procedure covers how to restore a snapshot and connect it to a running application.

1. Determine how far back to go

   1. Which snapshot do we need to revert to?

2. Restore the snapshot to a running db instance in the same VPC and security groupings as the current stack.

3. Verify the running state

   1. Confirm that transactions from after the snapshot time are not in the running db. Entities created after the snapshot time should not exist

   2. Verify that the db is in a consistent state

4. Put the stack in readonly mode

   1. Use the console to set the JDOSTACKSTATUS to RO

5. Recreate the stack using the saved restore job with the verified snapshot as a parameter with the stackbuilder

   1. Make sure the input size parameter is larger than the actual size of the snapshot

   2. Delete the log groups associated with the stack

6. After stacks finish building, toggle its state to read-write

   1. Use the console to set JDOSTACKSTATUS to RW

7. Verify running state by creating some new entities

Source issue: https://sagebionetworks.jira.com/browse/PLFM-6527