Page Comparison

One of the features to the Synapse.org platform is the ability to host a crowd-sourced challenge. Hosting challenges are a great way to crowd-source new computational methods for fundamental questions in systems biology and translational medicine.

...

Setting up the infrastructure for submission evaluation
Launching the challenge
Updating the challenge
Closing the challenge
Monitoring the submissions

...

Before You Begin

🖥️ Required Compute Power

At Sage Bionetworks, we generally provision an AWS EC2 Linux instance to run infrastructure of a challenge, leveraging the SynapseWorkflowOrchestrator to run CWL workflows. These workflows will be responsible for evaluating and scoring submissions.

If Sage is responsible for providing To budget the cloud compute services: we ask that you give us a general estimate of consider the computing power needed to validate and score the submissions (for example: memory, volumestorage, GPU required?, …network speed). If contracting with Sage Bionetworks, we can help estimate these costs as part of the Challenge budget.

📋 Using Sensitive Data as Challenge Data

Challenge data can be hosted on Synapse. If the data is sensitive (for example, human data), Synapse can apply access restrictions so that legal requirements are met before participants can access them. Contact the Synapse Access and Compliance Team (act@sagebase.org) for support with the necessary data access procedures for sensitive data.

🛑 Restricted Data Access

If data cannot leave the external site or data provider, it will be the data contributor’s responsibility to set up the challenge infrastructure. Contact the Challenges and Benchmarking team (cnb@sagebase.org) for consultations if needed.

...

Note that the steps outlined in this article will assume the orchestrator will be used.

...

Challenge Infrastructure Setup

...

Requirements

Synapse One Sage account
Python 3.78+
synapseclient
challengeutils
(for local testing) CWL runner of choice, e.g. cwltool
Access to cloud compute services, e.g. AWS, GCP, etc.

Outcome

This infrastructure setup will continuously monitor the challenge’s evaluation queue(s) for new submissions. Once a submission is received, it will undergo evaluation including validation and scoring. All submissions will be downloadable to the challenge organizers, including the Docker image (if model-to-data challenge) and/or prediction files. Participants may periodically receive email notifications about their submissions (such as status, scores), depending on the infrastructure configurations.

Steps

1. Create a GitHub repository for the challenge workflow infrastructure. For the orchestrator to work, this repo must be public.

...

Expand

title	If using data-to-model template:

Script	TODO	Required?
`workflow.cwl`	Update `synapseid` to the synID of the Challenge's goldstandardgold standard/groundtruth ground truth file	yes
	Set `errors_only` to `false` if an email notification about a valid submission should also be sent	no
	Add metrics and scores to `private_annotations` if they are to be withheld from the participants	no
`validate.cwl`	Update the base image if the validation code is not Python	no
`validate.cwl`	Remove the sample validation code and replace with validation code for the challenge	yes
`score.cwl`	Update the base image if the validation code is not Python	no
`score.cwl`	Remove the sample scoring code and replace with scoring code for the challenge	yes

Expand

title	If using model-to-data template:

Script	TODO	Required?
`workflow.cwl`	Provide the admin user ID or admin team ID for `principalid` (2 steps: `set_submitter_folder_permissions`, `set_admin_folder_permissions`)	yes
	Update `synapseid` to the synID of the Challenge's goldstandardgold standard / ground truth data	yes
	Set `errors_only` to `false` if an email notification about a valid submission should also be sent (2 steps: `email_docker_validation`, `email_validation`)	no
	Provide the absolute path to the data directory, denoted as `input_dir`, to be mounted during the container runs.	yes
	Set `store` to `false` if log files should be withheld from the Participants	no
	Add metrics and scores to `private_annotations` if they are to be withheld from the Participants	no
`validate.cwl`	Update the base image if the validation code is not Python	no
`validate.cwl`	Remove the sample validation code and replace with validation code for the Challenge	yes
`score.cwl`	Update the base image if the validation code is not Python	no
`score.cwl`	Remove the sample scoring code and replace with scoring code for the Challenge	yes

...

Once you are done making updates, save the file and restart the orchestrator to apply the changes.

Log Files

As it’s running, the orchestrator will upload logs and prediction files to the “Logs” folder. For each user or team that submits to the challenge, two folders will be created:

...

Code Block

Logs
 ├── submitteridA
 │  ├── submission01
 │  │  ├── submission01_log.txt
 │  │  └── submission01_logs.zip
 │  ├── submission02
 │  │  ├── submission02_log.txt
 │  │  └── submission02_logs.zip
 │ ...
 │
 ├── submitteridA_LOCKED
 │  ├── submission01
 │  │  └── predictions.csv
 │  ├── submission02
 │  │  └── predictions.csv
 │ ...
 │
...

...

Launch the Challenge

Requirements

Synapse One Sage account
Python 3.7+
synapseclient
synapseutils

Important TODOs:

Info
Following steps 1-5 should be done in the live project.

...

When using copyWiki, it is important to specify the destinationSubPageId parameter. This ID can be found in the URL of the live project, where it is the number following .../wiki/.

Once `copyWiki` has been used once, DO NOT RUN IT AGAIN!

Once the wiki has been copied over, all changes to the live project should now be synced with challengeutils' mirrow-wiki.

Panel

panelIconId	27a1
panelIcon	:arrow_right:
panelIconText	➡️
bgColor	#F4F5F7

Learn more about how to Update the Challenge below.

...

Monitor the Submissions

As challenge organizers, we recommend creating a Submission View to track and monitor submissions as they come in. This table will especially be useful when participants need help with their submissions.

Panel

panelIconId	27a1
panelIcon	:arrow_right:
panelIconText	➡️
bgColor	#F4F5F7

Learn more about revealing scores and adding leaderboards in Evaluation Queues.

Steps

1. Go to the staging project and click on the Tables tab. Create a new Submission View by clicking on Add New… > Add Submission View.

...

Column Name	Description	Facet values?
`evaluationid`	Evaluation ID (evaluation ID, but rendered as evaluation name) – recommended for SubmissionViews with multiple queues in scope	Recommended
`id`	Submission ID
`createdOn`	Date and time of the submission (in Epoch, but rendered as `MM/dd/yyyy, hh:mm:ss`)
`submitterid`	User or team who submitted (user or team ID, but rendered as username or team name)	Recommended
`dockerrepositoryname`	Docker image name – recommended for model-to-data challenges	Not recommended
`dockerdigest`	Docker SHA digest – recommended for model-to-data challenges	Not recommended
`status`	Workflow status of the submission (one of [`RECEIVED`, `EVALUATION_IN_PROGRESS`, `ACCEPTED`, `INVALID`])	Recommended
`submission_status`	Evaluation status of the submission (one of [None, `VALIDATED`, `SCORED`, `INVALID`])	Recommended
`submission_errors`	(if any) Validation errors for the predictions file	Not recommended
`orgSagebionetworksSynapseWorkflowOrchestratorSubmissionFolder`	synID to the submission’s logs folder	Not recommended
`prediction_fileid`	synID to the predictions file (if any)	Not recommended
(any annotations related to scores)	Submission annotations - names used depends on what annotations were used in the scoring step of the workflow

...

3. Click Save. A table of the submissions and their metadata will now be available for viewing and querying. Changes to the information displayed can be edited by clicking on the schema icon, followed by Edit Schema:

...

Update the Challenge

Updating Existing Wikis

1. Go to the staging project and navigate to the page(s) you wish to edit. Click on the pencil icon to Edit Project Wiki:

...

Use --dryrun to optionally preview which pages will be updated prior to doing an official sync.

Adding a New Wiki Page

1. Go to the staging project, click on Wiki Tools > Add Wiki Subpage. Enter a page title, then click OK. A new page should now be available.

...

Use --dryrun to optionally preview which pages will be updated prior to doing an official sync.

Extending the Deadline

It is not unheard of for there to be a change in the submission deadline. To extend the submission deadline date/time, you can either:

...

Panel

panelIconId	27a1
panelIcon	:arrow_right:
panelIconText	➡️
bgColor	#F4F5F7

Learn more posting announcements in Utilizing the Discussion Board.

Updating the Workflow

For any changes to the CWL scripts or run_docker.py, make edits as needed to the scripts, then push the changes. We highly recommend conducting dryruns immediately after, so that errors are addressed in a timely manner.

Evaluation Docker Image

If your workflow is using a Docker image in validate.cwl and/or score.cwl, and updates were made, pull the latest changes on the instance with:

Code Block
docker pull <image name>:<version>

...

Close the Challenge

Important TODOs:

Registration: remove/hide all "Register" buttons from the challenge site
- Pages to search through: main page, Participation Overview, Submission Tutorial(s)
- Past challenge as reference:
- You can replace the registration button with an alert well like this:

...

Versions Compared

Old Version 46

New Version Current

Key

Before You Begin

🖥️ Required Compute Power

📋 Using Sensitive Data as Challenge Data

🛑 Restricted Data Access

Challenge Infrastructure Setup

Requirements

Outcome

Steps

Log Files

Launch the Challenge

Requirements

Important TODOs:

Once `copyWiki` has been used once, DO NOT RUN IT AGAIN!

Monitor the Submissions

Steps

Update the Challenge

Updating Existing Wikis

Adding a New Wiki Page

Extending the Deadline

Updating the Workflow

Evaluation Docker Image

Close the Challenge

Important TODOs:

Page Comparison

Versions Compared

Old Version 46

New Version Current

Key

Before You Begin

🖥️ Required Compute Power

📋 Using Sensitive Data as Challenge Data

🛑 Restricted Data Access

Challenge Infrastructure Setup

Requirements

Outcome

Steps

Log Files

Launch the Challenge

Requirements

Important TODOs:

Once copyWiki has been used once, DO NOT RUN IT AGAIN!

Monitor the Submissions

Steps

Update the Challenge

Updating Existing Wikis

Adding a New Wiki Page

Extending the Deadline

Updating the Workflow

Evaluation Docker Image

Close the Challenge

Important TODOs:

Once `copyWiki` has been used once, DO NOT RUN IT AGAIN!