This guide This guide helps organizers create a space within Synapse to host a crowd-sourced Challenge. Challenges are community competitions designed to crowd-source new computational methods for fundamental questions in systems biology and translational medicine. Learn more about Challenges and see examples of past and current projects by visiting Challenges and Benchmarking.
...
| Code Block |
|---|
Logs ├── submitteridA │ ├── submission01 │ │ ├── submission01_log.txt │ │ └── submission01_logs.zip │ ├── submission02 │ │ ├── submission02_log.txt │ │ └── submission02_logs.zip │ ... │ ├── submitteridA_LOCKED │ ├── submission01 │ │ └── predictions.csv │ ├── submission02 │ │ └── predictions.csv │ ... │ ... |
...
Launch the Challenge
...
Requirements
Synapse account
Python 3.7+
synapseclient
Important TODOs:
Before proceeding with the launch, contact Sage Governance to ensure that a clickwrap is in-place for Challenge registration. You will need to provide Governance with the Synapse ID to the live Challenge site, as well as the team ID of the
...
Participants team.
Share all needed Evaluation queues with the Participants team with
Can submitpermissions. Once the Challenge is over, we recommend updating their permissions toCan view– this will help enhance their user experience with Evaluation queues.After the Challenge is launched, create a Folder named “Data” and update its Sharing Settings.
Share the “Data” Folder with the Participants team only! DO NOT make the Folder public or accessible to all registered Synapse users. The Local Sharing Settings for the “Data” Folder should look something like this:
Upload any Challenge Data that is to be provided to the Participants to the “Data” Folder. Remember to only do this once you have ensured that the Folder is only accessible to the Participants team.
Requirements
Synapse account
Python 3.7+
synapseclient
To launch the Challenge, that is, to copy the Wiki pages of the staging site over to the live site, use synapseutils' copyWiki in a Python script, e.g.
...
When using copyWiki, it is important to specify the destinationSubPageId parameter. This ID can be found in the URL of the live site, where it is the integer following .../wiki/<some number>.
Once copyWiki has been used once, DO NOT USE IT AGAIN!!
Following this action, all changes to the live site should now be synced over with challengeutils' mirrow-wiki. More on updating the Wikis under the Update the Challenge section below.In addition to copying over the Wiki pages, share all needed Evaluation queues with the Participants team with Can submit permissions. Once the Challenge is over, we recommend updating their permissions to Can view (this help keep their interface clean).
...
Monitor the Submissions
As Challenge Organizers, we recommend creating a Submission View to easily track and monitor submissions as they come in. This table will especially be useful when Participants need help with their submissions.
...
Column Name | Description | Facet values? |
|---|---|---|
| Evaluation ID (evaluation ID, but rendered as evaluation name) – recommended for SubmissionViews with multiple queues in scope |
|
| Submission ID | |
| Date and time of the submission (in Epoch, but rendered as | |
| User or team who submitted (user or team ID, but rendered as username or team name) |
|
| Docker image name – recommended for model-to-data challenges |
|
| Docker SHA digest – recommended for model-to-data challenges |
|
| Workflow status of the submission (one of [ |
|
| Evaluation status of the submission (one of [None, |
|
| (if any) Validation errors for the predictions file |
|
| Synapse ID to the submission’s logs folder |
|
| Synapse ID to the predictions file (if any) |
|
(any annotations related to scores) | Submission annotations - names used depends on what annotations were used in the scoring step of the workflow |
Recommended
Not recommended...