Document toolboxDocument toolbox

Credentials rotation for Synapse admin services (PLFM-5017)

Owned by Xavier Schildwachter
Apr 17, 2023

This document focuses on Synapse clients connecting to the repository services to execute administrative calls ('/repo/v1/admin', ‘/repo/v1/migration’). These clients typically execute as jobs on the Jenkins ops system (http://build-system-ops.sagebase.org:8080).

How do the client connect?

job

method

job

method

file-archival, dataset-backfill

org.sagebionetworks.template.jobs.AsyncAdminJobExecutorMain (stackBuilder)

SynapseAdminClient gets secret from SecretsMgr (org.sagebionetworks.admin.auth.key and org.sagebionetworks.admin.auth.secret)

disable-certification, update-synapse-feature

curl, session token from an admin account (in the job)

remove-user, verify-oauth-client

curl, PAT from an admin account (bearer token), in the job

migration

SynapseAdminClient is passed basicAuth info: key (from config in job) and secret (from secrets manager, one for source and one for destination)

Same as AsyncAdminJobExecutor but already split

Rotating credentials for AsyncAdminJobExecutor

Solution 1: change the existing secret 'in-place'.

  1. change secret in SecMgr
    ==> at this point any app that uses the secret is broken

  2. redeploy the stack as '-1' (that will use the new secret)

  3. go live with '-1'
    ==> apps work again

Solution 2: rotate the secrets

To rotate the secrets, we would have to add a suffix ('.0', '.1'), and surface the suffix as a parameter for the stack builder to pull the correct values
for a given stack.

  1. New secrets with suffixes are created in Secrets Manager with the same value as the existing ones.

  2. Changes are made to Stack Builder with extra config to pick up using a suffix (e.g. '.0')

  3. To rotate the key in staging:

    • change the secret for '.1'

    • setup the suffix used by the Stack Builder to use '.1'

    • deploy staging
      ==> prod still uses '.0', staging uses '.1'

  4. Next deploy, both prod and staging will use '.1'

  5. For next rotation, just do the same but use '.0' for staging.

Solution 3: move away from using secret and use a regular admin account + PAT

  1. create a new PAT for the user

  2. update the clients to use the new PAT

    • this could be done with secrets manager

  3. delete the old PAT

Rotating credentials for the migration client

The migration app relies on 2 secrets to connect (one for source/prod, one for destination/staging).

  1. do not setup the next staging ahead of final migration (i.e. (staging==none))

  2. after final migration, change the destination secret in SecMgr

  3. deploy staging (uses the new secret), migration works fine

  4. after the final migration (b4 starting next stack migration), staging becomes prod so update the
    source secret to match staging secret
    ==> prod and staging secrets are same again
    ==> next staging can be deployed ahead of final migration (since its secret does not change)

Note1: Should work now.
Note2: There’s no way to rotate the key.