Deployments

Creating branches (usually Fri night or Sat morning)

Each week, we make sure release branches are merged back into develop at the stack release meeting. Before building the next staging stack (N+1), we tag the existing release branch and create branches for the next stack.

You can create a shell script to simplify the process:

#!/bin/bash
pushd ~/tmp/deploy
git clone git@github.com:Sage-Bionetworks/$1.git
pushd $1
git checkout release-$2
git tag -a $2.1 -m stack-$2-prod
git checkout develop
git merge release-$2
git push origin develop --tags
git checkout -b release-$3
git tag -a $3.0 -m stack-$3-staging
git push origin release-$3 --tags
popd
popd

Then use it (in this case, create-branch.sh) to close/tag the current staging branches and create new ones:

~/bin/create-branch.sh Synapse-Stack-Builder <N> <N+1>
~/bin/create-branch.sh SynapseWebClient <N> <N+1>
~/bin/create-branch.sh Synapse-Repository-Services <N> <N+1>

Now run the build jobs, pointing to the new branches:

Release builds point to release-<N+1>

Production builds point to release-<N>

Deploying dev stack (usually Fri night or Sat)

The dev stack runs the same version of the code as the staging stack in its production stack (i.e. there is no visible staging dev stack). Once we have artifacts from the builds above, we can deploy a staging version.

Deploying upcoming staging stack (usually Sat or Sun in between migrations below)

Final migrations

Migrations are scheduled Tue-Fri night and take a 2-3 hours. The goal is to be around 30 mins when going into the final migration (Sat night or Sun night depending on the progress of the table/search building and/or availability of migrator). To achieve that, we take advantage of a decrease of activity during the week-end and start several migrations during the day (at least one in the morning, one mid-afternoon to get the time around 1hr, then several back to back until the migration time dips towards 30 mins). In between migrations, perform final verifications such as checking that the deployed artifacts is the latest build, any validation left over after the stack meeting etc. Once we get to back-to-back migration, the job is usually setup to keep the destination in read-only mode.

The final migration, where both prod and staging are in read-only mode is usually timed early evening to minimize impact on users. Back-to-back migrations start around 2hrs prior to that, at some point the migration time is not decreasing anymore. That’s the clue that it’s ready to go…

Final migration - read-only mode

At the end of the job, you should only see a couple of lines with (null, null) values. these are migrateable tables that are empty.

04:34:38,031 INFO  - Currently running: 0 restore jobs.  Waiting to start 0 restore jobs.
04:34:38,158 INFO  - 	MULTIPART_UPLOAD_COMPOSER_PART_STATE:	mins:(null, null)	maxes:(null, null)
04:34:38,159 INFO  - 	COMMENT:	mins:(null, null)	maxes:(null, null)
04:34:38,159 INFO  - migration successful
04:34:38,159 INFO  - Destination remains in READ-ONLY mode.

If you see anything else do not go live without a clear understanding of what happened.

Open an issue with the type that’s the problem, and we’ll discuss on Monday.

Promoting staging stack to production

Removing old production stack

Initial migration

At this point, you have a live production stack and an empty staging stack. You can start migration for next week.

Note: typically after starting the job, I change the ‘org.sagebionetworks.remain.read.only.mode’ back to ‘false’ and queue up another job

Patching

Patching is deploying changes (parameters or artifacts) to a running stack. It happens routinely on the staging stack as we’re finding and fixing issues, less often on the production stack.

Patching staging

Patching staging is essentially deploying staging, just update the artifacts versions in the ‘Update the configuration of the build-synapse-staging-stack job’ above and run the job.

Patching production

Patching production is essentially deploying a ‘-1’ environment, mapping it as a ‘tst’ endpoint, validating the fixes and then promoting it to production.

Deploy ‘-1’ environment

Map environment to ‘tst’ website

Promote to production