...
Setting up the Challenge infrastructure
Launching the Challenge space
Setting up the Challenge wiki
Updating the Challenge
Interacting with the submitted entries
...
Before You Begin
Computing Power
At Sage Bionetworks, we generally provision an EC2 Linux instance for a Challenge that leverages SynapseWorkflowOrchestrator to run CWL workflows. These workflows will be responsible for evaluating and scoring submissions (see model-to-data-challenge-workflow GitHub for an example workflow). If Sage Bionetworks is responsible for the cloud compute services, please give a general estimate of the computing power (memory, volume) needed. We can also help with the estimates if you are unsure.
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Example Let’s say a submission file that is very large and/or complex will require up to 10GB of memory for evaluation. If a max of four submissions should be run at the same time, then an instance of at least 40GB memory will be required (give or take some extra memory for system processes as well), whereas ten concurrent submissions would require at least 100GB. The volume of the instance will be dependent on variables such as the size of the input files and the generated output files. If running a model-to-data Challenge, Docker images should also be taken into account. On average, participants will create Docker images that are around 2-4 GB in size, though some have reached up to >10 GB. Always encourage participants to follow Dockerfile Best Practices to ensure they are submitting as optimal of images as possible. |
Using Sensitive Data
Synapse has the ability to apply access restrictions to sensitive data (e.g. human data), so that legal requirements are met before participants can access it. If human data are being used in the Challenge, or if you have any questions about the sensitivity of the Challenge data, contact the Synapse Access and Compliance Team (act@sagebase.org) for support with the necessary data access procedures.
...
If Sage is not allowed access to the server, then it is the external site’s responsibility to get the orchestrator running in whatever environment chosen. If Docker is not supported by the system, please let us know as we do have solutions for workarounds (e.g. using Java to execute, etc.).
...
Challenge Infrastructure Setup
Requirements
Synapse account
Python 3.7+
(for local testing) CWL runner of choice, e.g. cwltool
Access to cloud compute services, e.g. AWS, GCP, etc.
Steps
1. Create a workflow infrastructure GitHub repository for the Challenge. For the orchestrator to work, the repository must be public.
...
This command will create two Synapse projects /wiki/spaces/DOCS/pages/2048327716:
Staging - Organizers will use this project during the Challenge development to share files and draft the Challenge wiki. The
create-challengecommand initializes the wiki with the DREAM Challenge Wiki Template.Live - Organizers will use this project as the pre-registration page during Challenge development. When the Challenge is ready for launch, the project will then be replaced with the contents from staging.
You may think of these two projects as development (staging project) and production (live project), in that all edits must be done in the staging site, NOT the live site. Maintenance of both projects enables wiki content to be edited and previewed in the staging project before the content is published to the live project. Changes to the live site are synced over with challengeutils' mirror-wiki (see Update the Challenge for more).
| Info |
|---|
At first, the live site will be just one page where a general overview about the Challenge is provided. There will also be a pre-register button that Synapse users can click on if they are interested in the upcoming Challenge. |
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Important! For the initial deployment of the staging site to live, use |
create-challenge will also create four /wiki/spaces/DOCS/pages/1985446029 for the Challenge:
...
For a visual reference, a diagram of the orchestrator and its interactions with Synapse is provided below:
...
Display a Submissions Dashboard (Optional)
12. Go to the staging site and click on the TABLES tab. Create a new SubmissionView by clicking on Table Tools > Add Submission View. Under "Scope", add the evaluation queue(s) you are interested in monitoring (you may add more than one), then click Next. On the next screen, select which information to display - this is known as its schema. We recommend the following schema for a typical SubmissionView:
...
Once you click Save, a table of the submissions and their metadata will become available for viewing and querying. Changes to the information displayed can be edited by going to the SubmissionView, then clicking on Submission View Tools > Show Submission View Schema > Edit Schema.
Launch the Challenge (One-Time Only)
13. On the live site, go to the CHALLENGE tab and share the appropriate evaluation queues with the Participants team, giving them "Can submit" permissions.
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Important! After the initial copying, all changes to the live site should now be synced over with |
Stop the Orchestrator
15. On the instance, enter:
...
Otherwise, if you are not running the orchestrator in the background, read the logs on the terminal screen to determine whether there is current activity.
...
Wiki Setup
Use the following questions to help plan and set up the Challenge site and evaluation queues.
...
| Expand | ||
|---|---|---|
| ||
Should participants submit their writeups during submission evaluations or after the Challenge has closed? A writeup is a summary of all contributors, methods, scripts, code, and prediction files/Docker images that a team used for their submission. Writeups are required so that top-performing entries can be reproduced. |
...
Update the Challenge
Challenge Site and Wikis
Any changes to the Challenge site and its wiki/sub-wiki contents must be done in the staging site, not live. To update:
...
| Info |
|---|
Using the |
Evaluation Queue Quotas
To update the submission quotas, go to the Synapse live site, then head to the CHALLENGE tab. Edit the evaluation queues as needed:
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Example 2 A Challenge lasts for 4 months with a daily submission limit of 3. To implement this quota, click Edit on the desired queue, click Add Round and input the Duration and click Advanced Limits to pick daily/weekly/monthly limits. |
...
Workflow Steps
For any changes to the infrastructure workflow steps and/or scripts involved with the workflow (e.g. run_docker.py), simply make the edits to the scripts, then push the changes.
| Info |
|---|
Note: dry-runs should always follow a change to the workflow; this will ensure things are still working as expected. |
...
Interacting with Submissions
Throughout the Challenge, participants will continuously submit to the evaluation queues. To manage continuous submissions, organizers can automate validation and scoring with the Synapse python client evaluation commands.
Revealing Submissions and Scores
Organizers can create leaderboards when scores are ready to be revealed to participants using /wiki/spaces/DOCS/pages/2011070739.
...
Learn more about adding leaderboards in the /wiki/spaces/DOCS/pages/1985151345.
Related Articles
/wiki/spaces/DOCS/pages/1985151441, /wiki/spaces/DOCS/pages/1985151345, /wiki/spaces/DOCS/pages/2011070739
...