Collect Writeups

At the end of every Challenge, we aim to collect writeups from all participating users/teams, in which they will provide information about their submission(s). Currently, writeups are collected as Synapse Projects (sample template here), and we ask the participants to include information such as their methodologies, external data sources (if any), and source code and/or Docker images. Typically, writeups are *required* in every Challenge, in order to be considered for “top performer” eligibility and other incentives, including byline authorship.

This article will outline the steps to collect and validate writeups on Synapse, as well as how to display them in a leaderboard combined with the evaluation results.

Workflow Setup

Requirements

• Synapse account

• (for local testing) CWL runner of choice, e.g. cwltool

• Access to cloud compute services, e.g. AWS, GCP, etc.

Steps

1. On the live version of the Challenge site, go to the Challenge tab and check whether there is already an Evaluation queue for collecting writeups (skip to Step 4 if so).

2. Create a new Evaluation queue if one is not currently available. Click on Challenge Tools > Add Evaluation Queue, and name it something like <Challenge Name> Writeup

By default, the newly-created queue will only be accessible to the creator, in this case you. For now, update its Sharing Settings so that:

• the Organizers team has Can score permissions

• the Admin team has Adminstrator permissions

• Anyone on the web has Can view permissions

We recommend not sharing the Evaluation queue with the Participants team until after the workflow has been tested. See Submit Writeup Dry-runs below for more details.

3. Update the quota of the Evaluation queue, such as the Duration (Round Start, Round End) and Submission Limits. Generally, there are no submission limits for writeups, so that field can be left blank.

4. Go to the Sage Bionetworks Challenges GH org, and create a New repo.

For Repository template, select “Sage-Bionetworks-Challenges/writeup-workflow”. The Owner can be left as default (“Sage-Bionetworks-Challenges”), and give any name you’d like for Repository name.

Past examples are:

• Anti-PD1-DREAM-Writeups

• CTD2-Chemosensitivity-Writeup

• RA2-Writeup-Infrastructure

Before creating the repo, remember to switch the view from Private to Public.

5. Clone the repo onto your machine. Using a text editor or IDE, make the following updates to workflow.cwl:

Push the changes up to GitHub when done.

6. On the staging site, go to the Files tab and click on the upload icon to Upload or Link to a File:

7. In the pop-up window, switch tabs to Link to URL. For "URL", enter the web address to the zipped download of the workflow infrastructure repository.  You may get this address by going to the repository and clicking on Code > right-clicking Download Zip > Copy Link Address:

Name the file (e.g. "writeup-workflow"), then click Save.

8. Add an annotation to the file called ROOT_TEMPLATE by clicking on the annotations icon:

The “Value” will be the path to the workflow script, written as:

{infrastructure workflow repo}-{branch}/{path/to/workflow.cwl}

For example:

my-writeup-repo-main/workflow.cwl

9. There are two approaches for running the writeup workflow. You can either:

a) tack on the evaluation + workflow onto an existing instance with orchestrator, by adding another key-value pair to EVALUATION_TEMPLATES in the .env file

b) create a new instance (a t3.small would be sufficient enough) and setup orchestrator on that machine. See Steps 7-9, 11 of Creating and Managing a Challenge for more details on how to set this up.

Submit Writeup Dry-runs

Use and/or create a new Synapse project and submit it to the writeup queue. (We recommend saving this Project for re-use of future writeup queues setup).

You can directly submit to the queue within the Challenge tab of the live site:

Some cases we recommend testing for are:

Once you are satisfied that the writeup workflow is to your expectations, remember to open the queue to the Challenge participants!

You can do so by updating its Sharing Settings so that the Participants team has Can submit permissions.

Display Writeups in a Leaderboard