At the end of every Challenge, we aim to collect writeups from all participating users/teams, in which they will provide information about their submission(s). Currently, writeups are collected as Synapse Projects (sample template here), and we ask the participants to include information such as their methodologies, external data sources (if any), and source code and/or Docker images. Typically, writeups are *required* in every Challenge, in order to be considered for “top performer” eligibility and other incentives, including byline authorship.
...
b) create a new instance (a t3.small would be sufficient enough) and setup orchestrator on that machine. See Steps 7-9, 11 of Creating and Managing a Challenge for more details on how to set this up.
...
.
10. (optional but recommended, especially for Displaying Writeups in a Leaderboard) Create a SubmissionView so that you can easily track and monitor the writeup submissions. We recommend the following schema for monitoring writeup submissions:
Column Name | Description | Facet values? |
|---|---|---|
| Submission ID | |
| Date and time of the submission (in Epoch, but rendered as | |
| User or team who submitted (user or team ID, but rendered as username or team name) |
|
| Link to the writeup |
|
| Link to the writeup archive |
|
| Workflow status of the submission (one of [ |
|
| Evaluation status of the submission (one of [None, |
|
| (if any) Validation errors for the predictions file |
|
| Info |
|---|
The highlighted columns would need to be added manually by clicking the + Add Column button at the bottom of the Edit Columns window. |
Submit Dry-runs
Use and/or create a new Synapse project and submit it to the writeup queue. (We recommend saving this Project for re-use of future writeup queues setup).
...
Test Case | Workflow Configurations | Expected Outcome |
|---|---|---|
Submitting the Challenge site |
| |
Submitting a private Synapse project | Lines 49-50 is used (writeup should be accessible to the Organizers team) |
|
Submitting a private Synapse project | Lines 47-48 is used (writeup should be publicly accessible) |
|
Submitting a Private Synapse project that is shared with the Organizers team | Lines 49-50 is used (writeup should be accessible to the Organizers team) |
|
Submitting a Private Synapse project that is shared with the Organizers team | Lines 47-48 is used (writeup should be publicly accessible) |
|
Submitting a public Synapse project | Lines 47-48 and/or lines 49-50 are used |
|
Once you are satisfied that the writeup workflow is to your expectations, remember to open the queue to the Challenge participants!
You can do so by updating its Sharing Settings so that the Participants team has Can submit permissions.
Display Writeups in a Leaderboard
Once the Challenge has concluded and top-performers are ready to be announced, the “Final Results” leaderboard will typically be configured to include writeups alongside the final submissions and their scores. This will enable the sharing of solutions and algorithms submitted to the Challenge.
Creating this leaderboard can easily be achieved with a Materialized View table.
Steps
Go to the staging site and click on the Tables tab. Create a new Materialized View by clicking on Add New… > Add Materialized View.