One of the features to the Synapse.org platform is the ability to host a crowd-sourced challenge. Hosting challenges are a great way to crowd-source new computational methods for fundamental questions in systems biology and translational medicine.
...
Before proceeding with the launch, we recommend contacting Sage Governance to add a clickwrap for challenge registration. With a clickwrap in-place, interested participants can only be registered if they agree to the terms and conditions of the challenge data usage.
If you are a Sage employee: submit a Jira ticket to the Governance board with the synID of the live project, as well as the team ID of the participants team.
Share all needed evaluation queues with the participants team with
Can submitpermissions. Once the challenge is over, we recommend updating the permissions toCan viewto prevent late submissions.We also recommend sharing the evaluation queues with the general public so that the leaderboards are openly accessible.
After the challenge is launched, create a folder called “Data” and update its Sharing Settings. Share the “Data” folder with the participants team only. Do not make the folder public or accessible to all registered Synapse users. The sharing settings of the “Data” folder should look something like this:
Upload any challenge data that is to be provided to the participants to the “Data” Folder. DO NOT UPLOAD DATA until you have updated its sharing settings.
...
When using copyWiki, it is important to specify the destinationSubPageId parameter. This ID can be found in the URL of the live project, where it is the number following .../wiki/.
Once copyWiki has been used once, DO NOT RUN IT AGAIN!
Once the wiki has been copied over, all changes to the live project should now be synced with challengeutils' mirrow-wiki.
...
Column Name | Description | Facet values? |
|---|---|---|
| Evaluation ID (evaluation ID, but rendered as evaluation name) – recommended for SubmissionViews with multiple queues in scope |
|
| Submission ID | |
| Date and time of the submission (in Epoch, but rendered as | |
| User or team who submitted (user or team ID, but rendered as username or team name) |
|
| Docker image name – recommended for model-to-data challenges |
|
| Docker SHA digest – recommended for model-to-data challenges |
|
| Workflow status of the submission (one of [ |
|
| Evaluation status of the submission (one of [None, |
|
| (if any) Validation errors for the predictions file |
|
| synID to the submission’s logs folder |
|
| synID to the predictions file (if any) |
|
(any annotations related to scores) | Submission annotations - names used depends on what annotations were used in the scoring step of the workflow |
Recommended
Not recommended...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Learn more posting announcements in Utilizing the Discussion Board. |
Updating the Workflow
For any changes to the CWL scripts or run_docker.py, make edits as needed to the scripts, then push the changes. We highly recommend conducting dryruns immediately after, so that errors are addressed in a timely manner.
...
Registration: remove/hide all "Register" buttons from the challenge site
Pages to search through: main page, Participation Overview, Submission Tutorial(s)
Past challenge as reference:
You can replace the registration button with an alert well like this:
...
| Code Block | ||
|---|---|---|
| ||
<div class="alert alert-info"> <h4><strong>Registration is closed.</strong></h4> <p>Thank you to everyone who joined the challenge!</p> </div> |
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
See Collecting Writeups for more information. |
Instances: stop and terminate all cloud compute instances for the challenge.
...