This guide helps organizers create a space within Synapse to host a crowd-sourced Challenge. Challenges are community competitions designed to crowd-source new computational methods for fundamental questions in systems biology and translational medicine. Learn more about Challenges and see examples of past and current projects by visiting Challenges and Benchmarking.
...
Synapse account
Python 3.7+
(for local testing) CWL runner of choice, e.g. cwltool
Access to cloud compute services, e.g. AWS, GCP, etc.
...
If you are not a Sage employee: follow the instructions listed under "Setting up linux environment" to install and run Docker as well as docker-compose onto the compute environment of choice.
...
Script | TODO | Required? |
| Update | yes |
Set | no | |
Add metrics and scores to | no | |
| Update the base image if the validation code is not Python | no |
Remove the sample validation code and replace with validation code for the Challenge | yes | |
| Update the base image if the validation code is not Python | no |
Remove the sample scoring code and replace with scoring code for the Challenge | yes |
...
Script | TODO | Required? |
| Provide the admin user ID or admin team ID for (2 steps: | yes |
Update | yes | |
Set (2 steps: | no | |
Provide the absolute path to the data directory, denoted as | yes | |
Set | no | |
Add metrics and scores to | no | |
| Update the base image if the validation code is not Python | no |
Remove the sample validation code and replace with validation code for the Challenge | yes | |
| Update the base image if the validation code is not Python | no |
Remove the sample scoring code and replace with scoring code for the Challenge | yes |
...
| Code Block |
|---|
docker-compose up -d |
where -d will make it so the orchestrator is run in the background. This will enable you to log out of the instance at any timewithout terminating the orchestrator.
| Info |
|---|
If |
...
If successful, the orchestrator will continuously monitor the Evaluation queues specified by EVALUATION_TEMPLATES for submissions with the status, RECEIVED. Once it encounters a RECEIVED submission, it will run the workflow as specified by ROOT_TEMPLATE and update the submission status from RECEIVED to EVALUATION_IN_PROGRESS. If an error is encountered during any of the workflow steps, the orchestrator will update the submission status to INVALID and the workflow will stop. If, instead, the workflow finishes to completion, the orchestrator will update the submission status to ACCEPTED. Depending on how the workflow is set up (configured by Step 11), participants Participants may periodically be notified by email of their submission's progress.
...
| Code Block |
|---|
docker-compose down |
Once you are done making updates, save the file and restart the orchestrator to apply the changes.
...
where Docker and TOIL logs are uploaded to <submitterid> and prediction files are uploaded to <submitterid>_LOCKED. Note that the LOCKED folders will NOT be accessible to the participants Participants – this is to help prevent data leakage.
...
| Note |
|---|
Before proceeding with the launch, contact Sage Governance to ensure that a clickwrap is in-place for Challenge registration. You will need to provide Governance with the Synapse ID to the live Challenge site, as well as the team ID of the Participants team. |
Requirements
Synapse account
Python 3.7+
synapseclient
To launch the Challenge, that is, to copy the Wiki pages of the staging site over to the live site, use synapseutils' copyWiki in a Python script, e.g.
...
When using copyWiki, it is important to specify the destinationSubPageId parameter. This ID can be found in the URL of the live site, where it is the integer following .../wiki/<some number>.
Once copyWiki has been used once, DO NOT USE IT AGAIN!!
Following this action, all changes to the live site should now be synced over with challengeutils' mirrow-wiki. More on updating the Wikis under the Update the Challenge section below.
...
As Challenge Organizers, we recommend creating a Submission View to easily track and monitor submissions as they come in. This table will especially be useful when participants Participants need help with their submissions.
...
3. Click Save. A table of the submissions and their metadata will now be available for viewing and querying. Changes to the information displayed can be edited by clicking on the schema icon, followed by Edit Schema:
...