...
At Sage Bionetworks, we generally provision an EC2 Linux instance for a Challenge that leverages SynapseWorkflowOrchestrator to run CWL workflows. These workflows will be responsible for evaluating and scoring submissions (see model-to-data-challenge-workflow GitHub for an example workflow). If Sage Bionetworks is responsible for the cloud compute services, please give a general estimate of the computing power (memory, volume) needed. We can also help with the estimates if you are unsure.
By default, up to ten submissions can be evaluated concurrently, though this number can be increased or decreased accordingly within the orchestrator's .env file. Generally, the more submissions you want to run concurrently, the more power will be required of the instance.
...
Support for Docker and, if possible,
docker-composeIf Docker is not allowed, then support for Singularity and Java 8 is required
SynapseWorkflowOrchestrator repository
If Sage is not allowed access to the server, then it is the external site’s responsibility to get the orchestrator running in whatever environment chosen. If Docker is not supported by the system, please let us know as we do have solutions for workarounds (e.g. using Java to execute, etc.).
...
We have created two templates in Sage-Bionetworks-Challenges that you may use as a starting point. Their READMEs outline what will need to be updated within the scripts (under Configurations), but we will return to this later in Step 10.
Workflow Template | Submission Type |
|---|---|
Flat files (e.g. CSV) | |
Docker images |
2. Create the Challenge site on Synapse with challengeutils' create-challenge:
...
This command will create two Synapse /wiki/spaces/DOCS/pages/2048327716projects:
Staging - Organizers will use this project during the Challenge development to share files and draft the Challenge wiki. The
create-challengecommand initializes the wiki with the DREAM Challenge Wiki Template.Live - Organizers will use this project as the pre-registration page during Challenge development. When the Challenge is ready for launch, the project will then be replaced with the contents from staging.
You may think of these two projects as development (staging project) and production (live project), in that all edits must be done in the staging site, NOT the live site. Maintenance of both projects enables wiki content to be edited and previewed in the staging project before the content is published to the live project. Changes to the live site are synced over with challengeutils' mirror-wiki (see Update the Challenge for more).
| Info |
|---|
At first, the live site will be just one page where a general overview about the Challenge is provided. There will also be a pre-register button that Synapse users can click on if they are interested in the upcoming Challenge. |
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Important! For the initial deployment of the staging site to live, use |
create-challenge will also create four /wiki/spaces/DOCS/pages/1985446029 for the Challenge:
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
This file will be what links the evaluation queue to the orchestrator. Make note of this File ID for use later in Step 9. |
6. Add an annotation /wiki/spaces/DOCS/pages/2667708522 to the file called ROOT_TEMPLATE by clicking on Files Tools > Annotations > Edit. The "Value" will be the path to the workflow script, written as:
...
7. Create a cloud compute environment with the required memory and volume specifications. Once it spins up, log into the instance and clone the orchestrator. Follow the "Setting up linux environment" instructions in the README to install and run Docker as well as docker-compose onto the instance.
...
Other properties may also be updated if desired, e.g. SUBMITTER_NOTIFICATION_MASK, SHARE_RESULTS_IMMEDIATELY, MAX_CONCURRENT_WORKFLOWS, etc. Refer to the "Running the Orchestrator with Docker containers" notes in the README for more details.
...
12. Go to the staging site and click on the TABLES tab. Create a new SubmissionView Submission View by clicking on Table Tools > Add Submission View. Under Under "Scope", add the evaluation queue(s) you are interested in monitoring (you may add more than one), then click Next. On the next screen, select which information to display - this is known as its schema. We recommend the following schema for a typical SubmissionView:
...
Once you click Save, a table of the submissions and their metadata will become available for viewing and querying /wiki/spaces/DOCS/pages/2667642897. Changes to the information displayed can be edited by going to the SubmissionView, then clicking on Submission View Tools > Show Submission View Schema > Edit Schema.
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Important! After the initial copying, all changes to the live site should now be synced over with |
Stop the Orchestrator
...
Learn more about adding leaderboards in the /wiki/spaces/DOCS/pages/1985151345.
Related Articles
/wiki/spaces/DOCS/pages/1985151441, /wiki/spaces/DOCS/pages/1985151345, /wiki/spaces/DOCS/pages/2011070739
...