This guide helps organizers create a space within Synapse to host a crowd-sourced Challenge. Challenges are community competitions designed to crowd-source new computational methods for fundamental questions in systems biology and translational medicine. Learn more about Challenges and see examples of past and current projects by visiting Challenges and Benchmarking.
...
Before proceeding with the launch, contact Sage Governance to ensure that a clickwrap is in-place for Challenge registration. You will need to provide Governance with the Synapse ID to the live Challenge site, as well as the team ID of the Participants team.Share all needed Evaluation queues with the Participants team with
Can submitpermissions. Once the Challenge is over, we recommend updating their permissions toCan view– this will help enhance their user experience with Evaluation queues.We also recommend sharing the Evaluation queues with the general public – that way, the leaderboards will be publicly accessible. If public leaderboards are not wanted, then at least grant all registered Synapse users with
Can viewpermissions so that Synapse users can view the leaderboard(s).After the Challenge is launched, create a Folder named “Data” and update its Sharing Settings.
Share the “Data” Folder with the Participants team only! DO NOT make the Folder public or accessible to all registered Synapse users. The Local Sharing Settings for the “Data” Folder should look something like this:
Upload any Challenge Data that is to be provided to the Participants to the “Data” Folder. Remember to only do this once you have ensured that the Folder is only accessible to the Participants team.
...
When using copyWiki, it is important to specify the destinationSubPageId parameter. This ID can be found in the URL of the live site, where it is the integer following .../wiki/<some number>.
Once copyWiki has been used once, DO NOT USE IT AGAIN!!
Following this action, all changes to the live site should now be synced over with challengeutils' mirrow-wiki. More on updating the Wikis under the Update the Challenge section below.
...
Column Name | Description | Facet values? |
|---|---|---|
| Evaluation ID (evaluation ID, but rendered as evaluation name) – recommended for SubmissionViews with multiple queues in scope |
|
| Submission ID | |
| Date and time of the submission (in Epoch, but rendered as | |
| User or team who submitted (user or team ID, but rendered as username or team name) |
|
| Docker image name – recommended for model-to-data challenges |
|
| Docker SHA digest – recommended for model-to-data challenges |
|
| Workflow status of the submission (one of [ |
|
| Evaluation status of the submission (one of [None, |
|
| (if any) Validation errors for the predictions file |
|
| Synapse ID to the submission’s logs folder |
|
| Synapse ID to the predictions file (if any) |
|
(any annotations related to scores) | Submission annotations - names used depends on what annotations were used in the scoring step of the workflow |
Recommended
Not recommended...