At the end of every Challenge, we aim to collect write-ups writeups from all participating users/teams, in which they will provide information about their submission(s). Currently, write-ups writeups are collected as a Synapse project (sample template here), and we ask the participants to include information such as their methodologies, external data sources (if any), and source code and/or Docker images. Typically, write-ups writeups are *required* in every Challenge, in order to be considered for “top performer” eligibility and other incentives, including byline authorship.
This guide will outline the steps to collect and validate write-ups writeups on Synapse.
...
Setup
Requirements
Synapse account
(for local testing) CWL runner of choice, e.g. cwltool
Access to cloud compute services, e.g. AWS, GCP, etc.
Steps
1. On the live version of the Challenge site, go to the Challenge tab and check whether there is already an Evaluation queue for collecting write-ups writeups (skip to Step 4 if so).
| Note |
|---|
Note that by default, Evaluation entities are only accessible to the Evaluation creator; if you are not currently an admin for the Challenge, double-check with other Organizers to ensure that you have the correct permissions to all available queues. |
...
2. Create a new Evaluation queue if one is not currently available. Click on Challenge Tools > Add Evaluation Queue, and name it something like <Challenge Name> Write-upWriteup
By default, the newly-created queue will only be accessible to the creator, in this case you. For now, update its Sharing Settings so that:
...
3. Update the quota of the Evaluation queue, such as the Duration (Round Start, Round End) and Submission Limits. Generally, there are no submission limits for write-upswriteups, so that field can be left blank.
...
Line Number | TODO | Motivation |
|---|---|---|
31 | Update | This will identify the Synapse team that are the organizers for thePTB Challenge. |
45 | Update | This will check that the participant/team did not submit the PTB Challenge site as their write-upwriteup. |
47-50 | Uncomment. | Lines 49-50 checks that the write-up writeup Project is AT LEAST accessible to the Organizers team (as defined by Line 31). Lines 47-48 checks the write-up writeup is accessible to anyone on the web. |
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
This file will be what links the evaluation queue to the orchestrator. Make note of this File ID for use later in Step 9. |
8. Add an annotation to the file called ROOT_TEMPLATE by clicking on the annotations icon:
...
b) create a new instance (a t3.small would be sufficient enough) and setup Orchestrator on that machine. See Steps 7-9, 11 of Creating and Managing a Challenge for more details on how to set this up.
Submit Writeup Dry-runs
Use and/or create a new Synapse project and submit it to the writeup queue. (We recommend saving this Project for re-use of future writeup queues setup).
You can directly submit to the queue within the Challenge tab of the live site:
...
Some edge cases we recommend testing for are:
Submitting the Challenge site (should be
INVALID)Submitting a Private Synapse project (should be
INVALID, given these configurations)Submitting a Private Synapse project that is shared with the Organizers team (should be
INVALID, given these configurations)Submitting a Public Synapse project (should be
VALID)
Once you are satisfied that the writeup workflow is to your expectations, remember to open the queue to the Challenge participants!
You can do so by updating its Sharing Settings so that the Participants team has Can submit permissions.