This document attempts to cover development and release practices for the synapser and PythonEmbedInR projects.

Table of Contents

outline	true

Github repositories

The code base for Synapse R client (wrap Python client) can be found at https://github.com/Sage-Bionetworks/synapser, which depends on https://github.com/Sage-Bionetworks/PythonEmbedInR. Modifying PythonEmbedInR is only required if some change is needed in how synapser interacts with its underlying Python library. Not all updates will require any associated changes to PythonEmbedInR.

Branches

master: tracking latest release
develop: tracking latest development work
release candidate branches: for each release, a new release candidate branch will be created with name set to the version to release (v1.0-rc).

Development and Release

Releasing a new version of synapser is a process that involves client engineers, product manager, validators, and end users.

Step 1: Identifying the next release

This step starts with client engineers and product manager going over R JIRA tickets and determining their priority, tagging them with the new release version, and assigning them to client engineers.

Step 2: Development

During development, an engineer works on their forks of synapser/PythonEmbedInR. Develop work will be checked into develop branch.

Fork https://github.com/Sage-Bionetworks/PythonEmbedInR and https://github.com/Sage-Bionetworks/synapser
Checkout a new branch to work on a feature/ Jira
GitHub Actions are configured in the repositories. Every push will build and test your changes in your forked repo (see the GitHub Actions tab of the repo). On your fork of the synapser repo (not needed for PythonEmbedInR), add each of the following secrets, the values can be obtained from the "Python Client dev stack openssl keys" entry of LastPass. These allow a test configuration to be installed during builds which enables vignette integration tests to run against test services.
1. encrypted_d17283647768_iv
2. encrypted_d17283647768_key
Once your change is ready for review, create a Pull Request with
1. base fork: Sage-Bionetworks/PythonEmbedInR or Sage-Bionetworks/synapser
2. base: develop
3. head fork: <your Github account>/PythonEmbedInR or <your Github account>/synapser
4. compare: <your feature branch>
Code view and reviewer merges your change

References:

...

Run

Code Block
styler::style_file()

Configuration

Add this file to your home directory, uncommenting headers and fields as necessary:

Code Block

language	bash
title	.synapseConfig
linenumbers	true

###########################
# Login Credentials       #
###########################
 
## Used for logging in to Synapse
## you may also specify an apikey instead of password. If both password and apikey are specified, the apikey is ignored
## Alternatively you can use rememberMe=True in synapseclient.login or login subcommand of the commandline client to
## cache your API key elsewhere.
#[authentication]
#username = <username>
#password = <password>
#apikey = <apikey>
 
## If you have projects with file stored on SFTP servers, you can specify your credentials here
## You can specify multiple sftp credentials
#[sftp://some.sftp.url.com]
#username= <sftpuser>
#password= <sftppwd>
#[sftp://a.different.sftp.url.com]
#username= <sftpuser>
#password= <sftppwd>
 
 
## If you have projects that need to be stored in an S3-like (e.g. AWS S3, Openstack) storage but cannot allow Synapse
## to manage access your storage you may put your credentials here.
## To avoid duplicating credentials with that used by the AWS Command Line Client,
## simply put the profile name form your ~/.aws/credentials file
## more information about aws credentials can be found here http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
#[https://s3.amazonaws.com/bucket_name] # this is the bucket's endpoint
#profile_name=local_credential_profile_name
 
 
###########################
# Caching                 #
###########################
## your downloaded files are cached to avoid repeat downloads of the same file. change 'location' to use a different folder on your computer as the cache location
# [cache]
# location = <your cache location>
 
 
###########################
# Advanced Configurations #
###########################
 
## If this section is specified, then the synapseclient will print out debug information
# [debug]
 
## Some integration tests require a second Synapse user. This should only be necessary for developers
#[test-authentication]
#username = <username>
#password = <password>
#principalid = <userId>
 
## Configuring these will cause the Python client to use these as Synapse service endpoints instead of the default prod endpoints.
# [endpoints]
# repoEndpoint=https://repo-staging.prod.sagebase.org/repo/v1
# authEndpoint=https://auth-staging.prod.sagebase.org/auth/v1
# fileHandleEndpoint=https://file-staging.prod.sagebase.org/file/v1
# portalEndpoint=https://staging.synapse.org/

Change Python Client Version in Local Environment

To make your local development environment depends on a pre-release version of the Python client, please run the following commands:

Code Block

language	bash
title	python client version
linenumbers	true

export PYTHON_CLIENT_GITHUB_USERNAME=Sage-Bionetworks
export PYTHON_CLIENT_GITHUB_BRANCH=develop

Changing The Version of the Underlying Python Client

The synapser package 'wraps' a specific version of the Synapse Python Client, specified in inst/python/installPythonClient.py, for example:

Code Block

language	bash
title	python client version
linenumbers	true

SYNAPSE_CLIENT_PACKAGE_VERSION = '2.4.0'

Upgrading the Python client dependency is a matter of changing this line. synapser will look for the specified version in PyPI (unless you've directed your local build to use a Github branch as described above).

Like all R packages, synapser has embedded reference doc's. These doc's are in a LaTeX-like format, have the file suffix ".Rd" and go in a directory called man/. This is a "source" folder which the package build process uses as input.

Ideally in the synapser package we would generate a perfect .Rd document for each Python command that we wrap in an R command. We can certainly generate a valid LaTeX file and copy over the content of the Python docstrings. However we cannot guarantee that the doc's will be perfect, since the docstrings often have code examples and other verbiage in Python. Our strategy is to generate draft .Rd files and perform a manual step of finalizing them. The procedure is as follows:

a. Build the R package. The draft .Rd files are written into a folder called auto-man/

b. Perform a git diff to see what auto-generated files have been added or changed. The changes should be a small fraction of the overall documents. Manually transfer the new documents and changes to the man/ folder, editing as needed. For example make sure that any python examples embedded within Python docstrings are properly translated into R equivalent code in the generated documentation.

c. git commit both the auto-generated and manually curated files.

Step 3: Deploy Staging

Once all JIRA tickets for the new release version are RESOLVED, we are ready to deploy staging version for validation process.

...

from the repo directory run the following command:

Code Block
R -e "pkgdown::build_site()"

...

Review the changes by inspecting docs/index.html file.

...

Commit the changes to the docs directory. When this release branch is validated and merged to master it will automatically publish to https://r-docs.synapse.org via GitHub Pages.

...

This document attempts to cover development and release practices for the synapser and synapserutils projects. As of synapser 1.0.0+, PythonEmbedInR is no longer used.

...

Table of Contents

minLevel	1
maxLevel	6
outline	true
type	list
printable	false

Github repositories

The code base for Synapse R client (wrap Python client) can be found at https://github.com/Sage-Bionetworks/synapser and the code base for Synapserutils package can be found at https://github.com/Sage-Bionetworks/synapserutils .

Branches

master: tracking latest release
develop: tracking latest development work
release candidate branches: for each release, a new release candidate branch will be created with name set to the version to release (v1.0-rc).

...

Development and Release

Releasing a new version of Synapser is a process that involves client engineers, product manager, validators, and end users.

Step 1: Identifying the next release

This step starts with client engineers and product manager going over R JIRA tickets and determining their priority, tagging them with the new release version, and assigning them to client engineers.

Step 2: Development

Develop work will be checked into develop branch.

Fork https://github.com/Sage-Bionetworks/synapser
Checkout a new branch from develop branch to work on a feature / Jira
GitHub Actions are configured in the repositories. Every push will build and test your changes in your forked repo (see the GitHub Actions tab of the repo). On your fork of the synapser repo, add each of the following secrets, the values can be obtained from the "Python Client dev stack openssl keys" entry of LastPass. These allow a test configuration to be installed during builds which enables vignette integration tests to run against test services.
1. encrypted_d17283647768_iv
2. encrypted_d17283647768_key
Once your change is ready for review, create a Pull Request with
1. base fork: https://github.com/Sage-Bionetworks/synapser
2. base: develop
3. head fork: <your Github account>/synapser
4. compare: <your feature branch>
Code view and reviewer merges your change to the develop branch.

References:

Commit Best Practices: http://r-pkgs.had.co.nz/git.html#commit-best-practices
Git Branch: http://r-pkgs.had.co.nz/git.html#git-branch
Style: tidyverse style guidelines
- Run
  Code Block
  styler::style_file()

Configuration

Add this file to your home directory, uncommenting headers and fields as necessary. Heres an example:https://github.com/Sage-Bionetworks/

...

Step 4: Validation

Validators can download and/or install the new pre-release version via install.packages() (recommended) or using devtools (not-recommended). After installing the pre-release version, a validator can validate the JIRA tickets that they were assigned and change their status to

Status

colour	Green
title	CLOSE

or

Status

colour	Blue
title	REOPEN

.

Option 1: Installing from Staging RAN

Staging RAN Repo: http://staging-ran.synapse.org

Code Block

language	bash
title	staging-ran

install.packages("synapser", repos=c("http://staging-ran.synapse.org"))

Option 2: Using devtools

Code Block

language	bash
title	devtools-staging

# replace version with the new release version 
devtools::install_github("Sage-Bionetworks/synapser@v1.0-rc")

Step 5: Patch Staging

When a critical bug is found during validation, and it needed to be addressed before releasing, a patch need to go into "staging" without changing the <major>.<minor> release version.

A client engineer fix a bug, run private build, and create Pull Request to the release branch (v1.0-rc)
After the Pull Request is merged, start the staging build to update the artifacts. Please see Step 3 to recreate a staging build.
Notify the validator to re-validate the ticket with the new artifact.

Step 6: Release

After all JIRA issue are CLOSED, the new version is ready to be released. Creating a production deployment is similar to the staging deployment and is accomplished by publishing a GitHub release:synapsePythonClient/blob/develop/synapseclient/.synapseConfig

Changing The Version of the Underlying Python Client in Synapser

The synapser package 'wraps' a specific version of the Synapse Python Client, specified in R/zzz.R, and tools/installPythonClient.R for example:

Code Block

language	bash

PYTHON_CLIENT_VERSION <- '2.7.0'

synapser will look for the specified version in PyPI. Like all R packages, synapser has embedded reference doc's. These doc's are in a LaTeX-like format, have the file suffix ".Rd" and go in a directory called man/. This is a "source" folder which the package build process uses as input.

Ideally in the synapser package we would generate a perfect .Rd document for each Python command that we wrap in an R command. We can certainly generate a valid LaTeX file and copy over the content of the Python docstrings. However we cannot guarantee that the doc's will be perfect, since the docstrings often have code examples and other verbiage in Python. Our strategy is to generate draft .Rd files and perform a manual step of finalizing them. The procedure is as follows:

Build the R package. The draft .Rd files are written into a folder called auto-man/
Code Block
R CMD INSTALL .
Perform a git diff to see what auto-generated files have been added or changed. The changes should be a small fraction of the overall documents. Manually transfer the new documents and changes to the man/ folder, editing as needed. For example make sure that any python examples embedded within Python docstrings are properly translated into R equivalent code in the generated documentation.
git commit both the auto-generated and manually curated files.

Info
Note: Make sure to add in new Rd files.

Changing The Version of the Underlying R Client in synapserutils

The synapserutils built on top of the synapser package and the synapser version is specified in the DESCRIPTION file in the Remotes section and would be attached to the package when installing it, for example:

Code Block
Sage-Bionetworks/synapser@2.1.1

Moreover, functions in synapserutils should mirror what are in the synapseutils. Changes need to be made to synapseUtilsWrapper.py and reference documents.

Step 3: Deploy Staging

Once all JIRA tickets for the new release version are RESOLVED, we are ready to deploy staging version for validation process.

Create a new release candidate branch from develop with branch name as the version to release (for example v1.0-rc)
Update the docs in the release candidate branch:
1. ensure you have the R pkgdown package installed (on a Mac you may need to brew install harfbuzz, fribidi, and pandoc if you haven't already)
2. Update the changelog contained in NEWS.md
3. Update the version in the DESCRIPTION file, as the version is reflected in the generated documentation.
4. from the repo directory run the following command:
  Code Block
  R -e "pkgdown::build_site()"
5. Review the changes by inspecting docs/index.html file.
6. Commit the changes to the docs directory and push to the release candicate branch
Create a new staging release
1. Go to the releases of the appropriate repo

...

1. https://github.com/Sage-Bionetworks/synapser/releases
2. Click the "Draft a new release" button and fill the following values:
  
  Tag version: X.Y-rc where X.Y where X.Y is the release version (e.g. 0.11-rc)
  Target: the previously created vX.Y-rc branch
  Release title: Same as tag version (X.Y

...

1. -rc)
  
  Important: Check the "Set as a pre-release" checkbox. This will cause the release to be deployed to a staging ran.
2. Use the GitHub release “Generate release notes” feature to create release notes. Copy related content from the NEWS.md to the top of the release notes.
3. Hit the

...

1. “Publish release“ button, this will trigger

...

Step 7: Users install the released version

Option 1: Installing from RAN

...

1. a GitHub Action that will test and deploy the staging release to http://staging-ran.synapse.org

...

Code Block

language	bash
title	ran

install.packages("synapser", repos=c("http://ran.synapse.org"))

Option 2: Using devtools

Code Block

language	bash
title	devtools-prod

# replace version with the new released version 
devtools::install_github("Sage-Bionetworks/synapser@v1.0")

Publishing Documentation

Users come to the synapser package through:

Synapse portal > docs site at (1) http://docs.synapse.org/r/
Github repo at (2) https://github.com/Sage-Bionetworks/synapser and its site at (3) https://Sage-Bionetworks.github.io/synapser/
Sage RAN including staging RAN at (4) http://staging-ran.synapse.org and production RAN at (5) http://ran.synapse.org

Each of the location above should include general information about synapser package and how to download it.

The package docs site (3) should be the central place to access all available documents. Each of the site above should link to it.

As a part of the release process, we generate and update the documents.

While building artifacts and pushing them to staging ran, we will generate docs and merge back to release candidate branch.
When we release the artifact to prod ran, we merge release candidate branch to master. This will update the docs site.

CI/CD configuration

As discussed above, the CI/CD for synapser/PythonEmbedInR is driven through GitHub Actions. The build is contained in the contained build.yml workflows of the respective repositories (synapser and PythonEmbedInR).

MacOS runner

Currently the builds utilize one external self hosted runner for Mac builds (Linux and Windows builds are run on ephemeral GitHub hosted runners). This is due to GitHub Actions currently only supporting Catalina based GitHub hosted runners, whereas the build uses a Mojave based runner in order to ensure better binary compatibility when installed on older versions of MacOS.

The current self hosted runner is a MacStadium machine which Sage subscribes to. The IP address and credential are available in LastPass. You can log into either via SSH or using VNC.

If the instance ever needs to be recreated, it can be set up on a new machine as follows:

...

Install each version needed (as listed in the build.yml workflows) using pkg installer from the R Mac install page.
After installing each version run the following command:
Code Block
language bash
pkgutil --pkgs | grep -i org.r-project | xargs -I {} sudo pkgutil --forget {}
Normally installing a version from the package installer will uninstall a previous version. "Forgetting" the previously installed version allows multiple versions to be installed at the same time.

If they haven't already been installed, install X-code command line tools

Code Block
xcode-select --install

...

Code Block

language	bash

# https://stackoverflow.com/a/58349403

for HEADER_PATH in /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/*
do
	INCLUDE_PATH="/usr/local/include/$(basename $HEADER_PATH)"
	if [ ! -f $INCLUDE_PATH ]; then
		sudo ln -s $HEADER_PATH $INCLUDE_PATH
	fi
done

Install brew https://brew.sh/ if it is not already installed. From brew, install pandoc:

Code Block

language	bash

brew install pandoc

For EACH the PythonEmbedInR and synapser GitHub repos, add a self-hosted runner by doing the following:

Got to the GitHub action runner settings for the repo:

1. /
Notify validators about the available version. The validation version will be in format <version-to-build>.<build-number> (For example: 1.0.87, for build number 87).

Step 4: Validation

Validators can download and/or install the new pre-release version via install.packages() (recommended) or using devtools (not-recommended). After installing the pre-release version, a validator can validate the JIRA tickets that they were assigned and change their status to

Status

colour	Green
title	CLOSE

or

Status

colour	Blue
title	REOPEN

.

Option 1: Installing from Staging RAN

Staging RAN Repo: http://staging-ran.synapse.org. For example, synapser

staging-ran

Code Block

language	bash

install.packages("synapser", repos=c("http://staging-ran.synapse.org"))

Option 2: Using devtools

devtools-staging

Code Block

language	bash

# replace version with the new release version 
devtools::install_github("Sage-Bionetworks/synapser@v1.0-rc")

Step 5: Patch Staging

When a critical bug is found during validation, and it needed to be addressed before releasing, a patch need to go into "staging" without changing the <major>.<minor> release version.

A client engineer fix a bug, run private build, and create Pull Request to the release branch (v1.0-rc)
After the Pull Request is merged, start the staging build to update the artifacts. Please see Step 3 to recreate a staging build.
Notify the validator to re-validate the ticket with the new artifact.

Step 6: Release

After all JIRA issue are CLOSED, the new version is ready to be released. Creating a production deployment is similar to the staging deployment and is accomplished by publishing a GitHub release:

Go to the releases of the appropriate repo https://github.com/Sage-Bionetworks/synapser/releases
Click the "Draft a new release" button and fill the following values:

Tag version: X.Y where X.Y is the release version (e.g. 0.11)
Target: the previously created vX.Y-rc branch
Release title: Same as tag version (X.Y)

Do NOT check the "This is a pre-release" checkbox.
Hit the publish button, this will trigger a GitHub Action that will test and deploy the production release to http://ran.synapse.org/
When this release branch is validated and merged to master it will automatically publish to https://r-docs.synapse.org via GitHub Pages.

Code Block
git checkout master git merge vX.Y.Z-rc git push upstream master

Merge the master branch to develop:

Code Block
git checkout develop git merge master git push upstream develop

Step 7: Users install the released version

Option 1: Installing from RAN

RAN Repo: http://ran.synapse.org

ran

Code Block

language	bash

install.packages("synapser", repos=c("http://ran.synapse.org"))

Option 2: Using devtools

devtools-prod

Code Block

language	bash

# replace version with the new released version 
remotes::install_github("Sage-Bionetworks/synapser@v1.0")

...

Publishing Documentation

Users come to the synapser package through:

Synapse portal > docs site at (1) http://docs.synapse.org/r/
Github repo at (2) https://github.com/Sage-Bionetworks

...

https://github.com/Sage-Bionetworks/synapser/settings/actions/runners

...

If a previous runner being replaced is listed remove it.

...

Follow the instructions to add a new runner, giving it a unique label that matches the label used in the workflow build.yml, e.g. "macos-mojave".

...

/synapser and its site at (3) https://Sage-Bionetworks.github.io/synapser/
Sage RAN including staging RAN at (4) http://staging-ran.synapse.org and production RAN at (5) http://ran.synapse.org

Each of the location above should include general information about synapser package and how to download it.

The package docs site (3) should be the central place to access all available documents. Each of the site above should link to it.

As a part of the release process, we generate and update the documents.

While building artifacts and pushing them to staging ran, we will generate docs and merge back to release candidate branch.
When we release the artifact to prod ran, we merge release candidate branch to master. This will update the docs site.

CI/CD configuration

As discussed above, the CI/CD for synapser is driven through GitHub Actions. The build is contained in the contained build.yml workflows of the respective repositories (synapser).

References:

R package docs site tools:
http
https://
hadley
pkgdown.
github
r-lib.
io
org/
pkgdown/
Github Pages instructions: https://
help
docs.github.com
/articles
/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages
/
-site

Page Comparison

Versions Compared

Old Version 3

New Version Current

Key

Github repositories

Branches

Development and Release

Step 1: Identifying the next release

Step 2: Development

References:

Configuration

Change Python Client Version in Local Environment

Changing The Version of the Underlying Python Client

Step 3: Deploy Staging

Github repositories

Branches

Development and Release

Step 1: Identifying the next release

Step 2: Development

Configuration

Step 4: Validation

Option 1: Installing from Staging RAN

Option 2: Using devtools

Step 5: Patch Staging

Step 6: Release

Changing The Version of the Underlying Python Client in Synapser

Changing The Version of the Underlying R Client in synapserutils

Step 3: Deploy Staging

Step 7: Users install the released version

Option 1: Installing from RAN

Option 2: Using devtools

Publishing Documentation

CI/CD configuration

MacOS runner

Step 4: Validation

Option 1: Installing from Staging RAN

staging-ran

Option 2: Using devtools

devtools-staging

Step 5: Patch Staging

Step 6: Release

Step 7: Users install the released version

Option 1: Installing from RAN

ran

Option 2: Using devtools

devtools-prod

Publishing Documentation

CI/CD configuration