Page Comparison

At the end of every Challenge, we aim to collect writeups from a typical challenge on Synapse, all participating users /and teams , in which they will provide information about will be asked to provide a writeup on their submission(s). Currently, writeups are collected as Synapse Projects (sample template here), and we ask the participants to include information such as their The writeup should include details such as methodologies, external data sources (if any), and source code and/or Docker images. Typically, writeups are *required* in every Challenge, in order to be considered for “top performer” eligibility and other incentives, including byline authorship.This article will outline the steps to collect and validate writeups on Synapse, as well as how to display them in a leaderboard combined with the evaluation results.. Participants and teams will submit their writeups as Synapse projects.

Although not true for all challenges, writeups are typically a criterion for any challenge incentives, such as “top performer” eligibility, byline authorship, and more.

If you are a challenge organizer and your challenge will require writeups, this article will help you:

• Set up the infrastructure for collecting and validating writeup submissions

• Display the writeups on a leaderboard

...

Workflow Setup

Requirements

• Synapse One Sage account

• (for local testing) CWL runner of choice, e.g. cwltool

• Access to cloud compute services, e.g. AWS, GCP, etc.

Outcome

Once set up, the workflow This infrastructure will continuously monitor the writeup queue for a new submissionssubmission, perform a quick validation, and, if valid, create an archive of the submission. An archive is created to ensure a copy of the writeup is always available to the Organizers organizers team, in case the Participant/Team delete project owner deletes the original version copy or remove accessibility.

Steps

1. On the live version of the Challenge siteproject, go to the Challenge tab and check whether there is already an Evaluation evaluation queue for collecting writeups (skip to Step 3 if so).

2. Create a new Evaluation evaluation queue if one is not currently available. Click on Challenge Tools > Add Evaluation Queue, and name it something like <Challenge Name> Writeup

By default, the newly-created queue will only be accessible to the creator, in this case, you. For now, update its Sharing Settings sharing settings so that:

• the Organizers The organizers team has Can score permissions

• the Admin The admin team has Adminstrator permissions

• Anyone on the web has Can view permissions

We recommend not sharing the Evaluation evaluation queue with the Participants participants team until after the workflow has been tested.

3. Update the quota of the writeup Evaluation evaluation queue, such as the Duration (Round Start, Round End) and Submission Limits. Generally, there are no submission limits for writeups, so this field can be left blank.

...

4. Go to Sage-Bionetworks-Challenges organization, and create a New repo.

For Repository “Repository template”, select “SageSage-Bionetworks-Challenges/writeup-workflow”workflow. The Owner “Owner” can be left as default (“Sage-Bionetworks-Challenges”), and give any name you’d like for Repository name“Repository name”.

Past naming examples are:

• Anti-PD1-DREAM-Writeups

• CTD2-Chemosensitivity-Writeup

• RA2-Writeup-Infrastructure

Before creating the repo, remember to switch the view from Private to Public.

...

...

6. On the staging siteproject, go to the Files tab and click on the upload icon to Upload or Link to a File:

...

7. In the pop-up window, switch tabs to Link to URL. For "URL", enter the web address to the zipped download of the workflow infrastructure repository.  You may get this address by going to the repository and clicking on Code > right-clicking Download Zip > Copy Link Address:

...

Name the file (e.g. "writeup-workflow"), then click Click Save.

8. Add an annotation to the file called ROOT_TEMPLATE by clicking on the annotations icon, followed by Edit:

...

The For “Value” will be , enter the path to the workflow script, written as:

{

...

name of repo}-{branch}/

...

workflow.cwl

...

For example: my-writeup-repo-main/workflow.cwl

9. There are two approaches for running the writeup workflow. You can either:

...

b) create a new instance (a t3.small would EC2 should be sufficient enough) and setup orchestrator on that machine. See Steps 7-9, 11 of Creating and Managing a Challenge for more details on how to set this up.

10. (optional but recommended for Displaying Writeups in a Leaderboard) Create a SubmissionView Submission View so that you can easily track and monitor the writeup submissions.

We recommend the following schema for monitoring writeup submissions:

...

Submit Dry-runs

Use and/or create a new Synapse project and submit it to the writeup queue. (We recommend saving this Project for re-use of future writeup queues setup). You can directly submit to the queue within the Challenge tab of the live siteproject:

...

Some cases we recommend testing for are:

Once you are satisfied that the writeup workflow is to your expectations, remember to open the queue to the

...

challenge participants!

You can do so by updating its Sharing Settings sharing settings so that the Participants participants team has Can submit permissions.

Display Writeups in a Leaderboard

Once the Challenge challenge has concluded and top-performers are ready to be announced, the a “Final Results” leaderboard will typically be configured created to include writeups alongside the final submissions and their scores. This will enable the sharing of solutions and algorithms submitted to the Challengechallenge.

Creating this leaderboard can easily be achieved with a Materialized View table.

Requirements

• Synapse ID synID of SubmissionView Submission View for final submissionsSynapse ID

• synID of SubmissionView Submission View for writeups

• challengeutils (for syncing changes)

Steps

1. Go to the staging siteproject and click on the Tables tab. Create a new Materialized View by clicking on Add New… > Add Materialized View.

...

This query will join the SubmissionView Submission View for submissions (s) with the SubmissionView Submission View for writeups (w), using the submitterids from both Views as the index. Rows are filtered to only include valid submissions from both Views (*.status = 'ACCEPTED') – these clauses are optional and can be removed if desired.

...

Columns selected from the writeups View (w) include:

• Synapse ID synID of the writeup (w.entityid)

• Synapse ID synID of the writeup archive (w.archive)

This The query is also assigning aliases, e.g. w.entityid AS writeup, for readability and easier querying.

...

3. Click Save. A table with the selected columns from each SubmissionView Submission View will now be available for viewing and querying. It should look something like this:

...

4. Continuing on the staging siteproject, switch to the Wiki tab and go to the “Results” page (create a new Wiki page if it’s not available). Click on the pencil icon to Edit Project Wiki:

...

SELECT * FROM syn123

where syn123 is the Synapse ID synID to the MaterializedView created earlier in Step 3 (not SubmissionViews!). Optionally include aliases and/or clauses to the query if desired, e.g. exclude certain columns such as archived_writeup. Click on the Preview button in the top-left corner to preview the prospective leaderboard, e.g.

...

6. Click Save once you are satisfied with the final leaderboard.

...

7. Sync changes to the live site with challengeutils' mirror-wiki:

challengeutils mirror-wiki staging-synid live-synid

...

🌟 For additional assistance or guidance, contact the Challenges and Benchmarking team at cnb@sagebase.org.