Document toolboxDocument toolbox

Synapse Deployment Notes - 2018

Owned by Xavier Schildwachter
Last updated: Nov 30, 2023
3 min read
versioncomment
2021/11/18Added tracking table
2018/07/5Created

A quick guide to deploying Synapse in 2018, this should be merged with existing articles
which contain a lot of relevant info but have obsolete procedure.

TODO: Add Synapse stack description here

To connect to the deployment system

  • on LastPass, get the credentials to connect to the deployment system (Jenkins):
    EMERGENCY: Search for 'emergency' in 'Shared-Platform' to get credentials to the emergency user
  • connect to vpn
  • connect to http://build-system-ops.sagebase.org:8080
  • login using the credentials from LastPass
  • you should see 3 build jobs (to deploy stacks: prod, staging, test) and 2 migration jobs (prod→staging, staging→test)
    NOTE: always double-check the actual parameters in case we deviate for whatever reason!

Prod stack changes

  • we try to avoid changing the live stack (no choice if shared resources are involved) so usually we deploy a parallel stack
  • implies access to Route53 since at some point DNS change needs to happen (TODO: investigate if this too could be wrapped and executed from build system)
    • EMERGENCY: To gain access to the AWS account should Khai or Xa be unexpectedly unavailable, search LastPass for 'emergency'.
      One site will popup with credentials to access the account. NOTE: Accessing the account requires an MFA device to be obtained from David/Diane.

To deploy a new stack (typically staging)

  • this assumes the simplest case: staging has been promoted to prod, the old prod stack has been deleted, staging artifacts have been built, and we just want to deploy the staging stack
  • information needed: artifacts version for portal and backend (e.g. '234.0-123456') (this is obtained by looking at the build outputs on the build server.
  • move the existing staging job properties to the prod job (so we can patch prod if needed)
    • TODO: Add details here (copy content of exec shell step from staging job into prod job)
  • edit the staging job properties for the current deployment
    • pick a deployment subnet (color) that is different from prod
    • TODO: Add details here (in shell exec step, edit stack number, versions, subnet color, beanstalk number as needed)
  • execute the job
  • verify that deployment went fine
    • TODO: Add details (cloudformation to see if templates exec'ed fine, EC2 target groups to verify all instances are actually up)

To patch an existing stack

Patching Prod requires deploying a new environment alongside the existing ones, testing and then switching to the new environment in Route53.
Patching Staging just updates an existing environment in place.

Patching Staging

  • information needed: artifact versions to deploy
  • edit the build-synapse-staging job, update the 'org.sagebionetworks.beanstalk.version.xxx' to the version desired
  • execute the job

Patching Prod

NOTE: The prod build is usually disabled, check that it's enabled (and pointing to the right bracnch) if you need to build artifacts for prod.

Typically, we patch portal by itself (i.e. create a new environment by changing 'org.sagebionetworks.beanstalk.number.portal') but if we have to patch the backend then we change all the environments (i.e. by changing 'org.sagebionetworks.beanstalk.number.xxx' for portal, repo and workers).

  • information needed: artifact versions to deploy
  • edit the build-synapse-staging job
    • update the 'org.sagebionetworks.beanstalk.version.xxx' property to the version desired
    • update the 'org.sagebionetworks.beanstalk.number.xxx' property to the number desired (typically increment current by one)
  • execute the job
  • in Route53, update the 'tst.synapse.org' CNAME to point to the newly created portal (since portal is updated regardless of front/backend patch)
  • test
  • when OK, update the CNAMEs for prod in Route53
    • in 'synapse.org' zone, update 'www.synapse.org'
    • in 'prod.sagebase.org' zone, update 'repo-prod.prod.sagebase.org'
  • delete the old prod stack

To make configuration changes to a stack

Typically, making a configuration change should just require modifying or adding a property in the corresponding job. If a code change
is required, it would go through the normal process and end up on the develop branch of the stack builder.

TODO: Document properties and their effect (i.e. for Prod stack, should we create a parallel stack for some property changes?)

  • information needed: property change, stack to apply to
  • in the corresponding job, modify/add property
  • execute the job

TODO: Map typical problems to fix in properties

NOTE: id_generator is treated differently