Create and Manage a Challenge

One of the features of the Synapse.org platform is the ability to host a crowd-sourced Challenge. Challenges on Synapse are meant to be community competitions, designed to crowd-source new computational methods for fundamental questions in systems biology and translational medicine.

Running a Challenge on Synapse will require creating a Challenge space for Participants to learn about the Challenge, join the Challenge community, submit entries, track progress, and view results. This article is aimed at Challenge Organizers, and will focus on:

• Setting up the infrastructure

• Launching and updating the Challenge

• Monitoring the submissions

Before You Begin

Required Compute Power

At Sage Bionetworks, we generally provision an AWS EC2 Linux instance to run infrastructure of a Challenge, leveraging the SynapseWorkflowOrchestrator to run CWL workflows.  These workflows will be responsible for evaluating and scoring submissions.

• If Sage is responsible for providing the cloud compute services: we ask that you give us a general estimate of the computing power needed (e.g. memory, volume, GPU required?, etc).

Using Sensitive Data

Synapse has the ability to apply access restrictions to sensitive data (e.g. human data), so that legal requirements are met before Participants can access it. If human data are being used in the Challenge, or if you have any questions about the sensitivity of the Challenge data, contact the Synapse Access and Compliance Team (act@sagebase.org) for support with the necessary data access procedures.

If data is sensitive and cannot be hosted on Synapse (e.g. it cannot leave the external site or data provider), provide a remote server with the following:

• Support for Docker and, if possible, docker-compose

  • If Docker is not allowed, then support for Singularity and Java 8 is a must

• SynapseWorkflowOrchestrator repository

Note that it will be the data contributor’s responsibility to set up the infrastructure if Sage employees are not allowed access to the remote server. However, we are happy to consult if needed.

Challenge Infrastructure Setup

Requirements

• Synapse account

• Python 3.7+

• synapseclient

• challengeutils

• (for local testing) CWL runner of choice, e.g. cwltool

• Access to cloud compute services, e.g. AWS, GCP, etc.

Outcome

Once the infrastructure is set up, the selected Evaluation queue(s) will continuously be monitored for new submissions. Once a submission is received, it will undergo evaluation including validation and scoring. All submissions will be accessible to the Challenge Organizers, including the Docker image (if model-to-data Challenge) and/or prediction files. Participants may periodically receive email notifications about their submissions (e.g. status, scores), depending on the infrastructure configurations.

Steps

1. Create a workflow infrastructure GitHub repository for the Challenge. For the orchestrator to work, the repository must be public.

We have created two templates in Sage-Bionetworks-Challenges that you may use as a starting point. Their READMEs outline what will need to be updated within the scripts (under Configurations), but we will return to this later in Step 10.

2. Create the Challenge site on Synapse with challengeutils' create-challenge:

challengeutils create-challenge "challenge_name"

This command will create two Synapse projects:

• Staging - Organizers will use this project during the Challenge development to share files and draft the Challenge wiki. The create-challenge command initializes the wiki with the DREAM Challenge Wiki Template.

• Live - Organizers will use this project as the pre-registration page during Challenge development. When the Challenge is ready for launch, the project will then be replaced with the contents from staging.

You may think of these two projects as development (staging project) and production (live project), in that all edits must be done in the staging site, NOT the live site. Maintenance of both projects enables wiki content to be edited and previewed in the staging project before the content is published to the live project. Changes to the live site are synced over with challengeutils' mirror-wiki (see Update the Challenge below for more info).

create-challenge will also create four /wiki/spaces/DOCS/pages/1985446029 for the Challenge:

• Participants - This Synapse team includes the individual participants and teams who register to the Challenge.

• Organizers - The Challenge organizers must be added to this list to provide the permissions to share files and edit the wiki on the staging project.

• Admin - The Challenge administrators have administrator access to both the live and staging project. Organizers do not need to be Administrators. Ideally, all admin must have a good understanding of Synapse.

• Pre-registrants - This team is recommended for when the Challenge is under development. It allows participants to join a mailing list to receive notification of Challenge launch news.

Add Synapse users to the Organizers and Admin teams as needed.

3. On the live site, go to the Challenge tab and create as many /wiki/spaces/DOCS/pages/1985151345 as needed (for example, one per question/task) by clicking on Challenge Tools > Create Evaluation Queue.  By default, create-challenge will create an evaluation queue for writeups. More information on writeups and how to collect them here.

4. While still on the live site, go to the Files tab and create a new folder called "Logs" by clicking on the add-folder icon:

More information about the log folder and its folder structure provided below.

5. On the staging site, go to the Files tab and click on the upload icon to Upload or Link to a File:

6. In the pop-up window, switch tabs to Link to URL. For "URL", enter the web address to the zipped download of the workflow infrastructure repository.  You may get this address by going to the repository and clicking on Code > right-clicking Download Zip > Copy Link Address:

Name the file (e.g. "workflow"), then click Save.

7. Add an annotation to the file called ROOT_TEMPLATE by clicking on Files Tools > Annotations > Edit.  The "Value" will be the path to the workflow script, written as:

{infrastructure workflow repo}-{branch}/path/to/workflow.cwl

For example, this is the path to workflow.cwl of the model-to-data template repo:

model-to-data-challenge-workflow-main/workflow.cwl

8. Create a cloud compute environment with the required memory and volume specifications, then SSH into the instance.

• If you are a Sage employee: you can follow our internal instructions on /wiki/spaces/CHAL/pages/2806087732 here.

• If you are not a Sage employee: follow the instructions listed under "Setting up linux environment" to install and run Docker as well as docker-compose onto the compute environment of choice.
  

9. On the instance, clone the SynapseWorkflowOrchestrator repo if needed. Change directories to SynapseWorkflowOrchestrator/ and create a copy of the .envTemplate file as .env (or simply rename it to .env):

cd SynapseWorkflowOrchestrator/
cp .envTemplate .env

10. Open .env and enter values for the following property variables:

Other properties may also be updated if desired, e.g. SUBMITTER_NOTIFICATION_MASK, SHARE_RESULTS_IMMEDIATELY, MAX_CONCURRENT_WORKFLOWS, etc.  Refer to the "Running the Orchestrator with Docker containers" notes in the README for more details.

11. Return to the workflow infrastructure repo and clone it onto your local machine.  Using a text editor or IDE, make the following updates to the following scripts:

Data-to-model template:

Model-to-data template:

Push the changes up to GitHub when done.

12. On the instance, change directories to SynapseWorkflowOrchestrator/ and kick-start the orchestrator with:

docker-compose up -d

where -d will make it so the orchestrator is run in the background. This will enable you to log out of the instance without terminating the orchestrator.

If successful, the orchestrator will continuously monitor the Evaluation queues specified by EVALUATION_TEMPLATES for submissions with the status, RECEIVED.  Once it encounters a RECEIVED submission, it will run the workflow as specified by ROOT_TEMPLATE and update the submission status from RECEIVED to EVALUATION_IN_PROGRESS. If an error is encountered during any of the workflow steps, the orchestrator will update the submission status to INVALID and the workflow will stop.  If, instead, the workflow finishes to completion, the orchestrator will update the submission status to ACCEPTED.  Depending on how the workflow is set up (configured by Step 11), Participants may periodically be notified by email of their submission's progress.

13. To make changes to the .env file (e.g. change the number of concurrent submissions), stop the orchestrator with:

docker-compose down

Once you are done making updates, save the file and restart the orchestrator to apply the changes.

Log Files

As it’s running, the orchestrator will upload logs and prediction files to the folder as specified by WORKFLOW_OUTPUT_ROOT_ENTITY_ID. For each submitter, two folders will be created:

• <submitterid>

• <submitterid>_LOCKED

where Docker and TOIL logs are uploaded to <submitterid> and prediction files are uploaded to <submitterid>_LOCKED. Note that the LOCKED folders will NOT be accessible to the Participants – this is to help prevent data leakage.

The directory structure of “Logs” will look something like this:

Logs
 ├── submitteridA
 │  ├── submission01
 │  │  ├── submission01_log.txt
 │  │  └── submission01_logs.zip
 │  ├── submission02
 │  │  ├── submission02_log.txt
 │  │  └── submission02_logs.zip
 │ ...
 │
 ├── submitteridA_LOCKED
 │  ├── submission01
 │  │  └── predictions.csv
 │  ├── submission02
 │  │  └── predictions.csv
 │ ...
 │
...

Launch the Challenge

Requirements

• Synapse account

• Python 3.7+

• synapseclient

• synapseutils

Important TODOs:

1. Before proceeding with the launch, contact Sage Governance to ensure that a clickwrap is in-place for Challenge registration. You will need to provide Governance with the Synapse ID to the live Challenge site, as well as the team ID of the Participants team.

2. Share all needed Evaluation queues with the Participants team with Can submit permissions. Once the Challenge is over, we recommend updating their permissions to Can view – this will help enhance their user experience with Evaluation queues.

3. We also recommend sharing the Evaluation queues with the general public – that way, the leaderboards will be publicly accessible. If public leaderboards are not wanted, then at least grant all registered Synapse users with Can view permissions so that Synapse users can view the leaderboard(s).

4. After the Challenge is launched, create a Folder named “Data” and update its Sharing Settings. Share the “Data” Folder with the Participants team only! DO NOT make the Folder public or accessible to all registered Synapse users. The Local Sharing Settings for the “Data” Folder should look something like this:

   

5. Upload any Challenge Data that is to be provided to the Participants to the “Data” Folder. Remember to only do this once you have ensured that the Folder is only accessible to the Participants team.

To launch the Challenge, that is, to copy the Wiki pages of the staging site over to the live site, use synapseutils' copyWiki in a Python script, e.g.

import synapseclient
import synapseutils
syn = synapseclient.login()

synapseutils.copyWiki(
   syn, "syn1234",  # Synapse ID of staging site
   destinationId="syn2345",  # Synapse ID of live site
   destinationSubPageId=999999  # ID following ../wiki/ of live site URL
)

When using copyWiki, it is important to specify the destinationSubPageId parameter.  This ID can be found in the URL of the live site, where it is the integer following .../wiki/<some number>.

Once copyWiki has been used once, DO NOT USE IT AGAIN!!

Following this action, all changes to the live site should now be synced over with challengeutils' mirrow-wiki. More on updating the Wikis under the Update the Challenge section below.

Monitor the Submissions

As Challenge Organizers, we recommend creating a Submission View to easily track and monitor submissions as they come in. This table will especially be useful when Participants need help with their submissions.

Learn more about revealing scores and adding leaderboards in Evaluation Queues.

Steps

1. Go to the staging site and click on the Tables tab. Create a new Submission View by clicking on Add New… > Add Submission View. 

2. Under "Scope", add the Evaluation queue(s) you are interested in monitoring (you may add more than one), then click Next. On the next screen, select which information to display - this is known as its schema.

We recommend the following schema for monitoring Challenge submissions:

3. Click Save. A table of the submissions and their metadata will now be available for viewing and querying. Changes to the information displayed can be edited by clicking on the schema icon, followed by Edit Schema:

Update the Challenge

Challenge Site and Wikis

Any changes to the Challenge site and its wiki/sub-wiki contents must be done in the staging site, not live.  To update:

1. Make whatever changes needed to the staging project.

2. Use challengeutils' mirror-wiki to push the changes to the live project.

Evaluation Queue Quotas

To update the submission quotas, go to the Synapse live site, then head to the Challenge tab.  Edit the Evaluation queues as needed:

Workflow Steps

For any changes to the infrastructure workflow steps and/or scripts involved with the workflow (e.g. run_docker.py), simply make the edits to the scripts, then push the changes.