The purpose here is to start a discussion and engage some design thinking about how to automate more of the process of creating and administrating a Challenge on Synapse.

Right now, running a challenge requires a lot of manual steps that involve interacting with the REST API which means an engineer has to be in the loop. Chris (aka me) has written some nice convenience layers in Python, but that still requires a user comfortable on the command line. The goal is to get more challenge related functionality in the hands of the challenge organizers.

This page is more about requirements gathering than design or implementation. Not all of this is necessarily practical, but it's here to get ideas from the whole team.

Create Challenge object:

A button/wizard to turn a project into a challenge. Some functionality this might entail:

create team of access approved users
and the challenge object that links the project to the team
create admin team?
create splash screen wiki
add preregistration or registration button

Wikis

Challenge organizers need staging areas to collaborate on wikis prior before pushing changes to a live wiki.

Challenge site begins with a splash screen live
Staging area begins with template wiki (see syn2769515)
Editing of wiki pages in staging area visible to challenge organizers but not participants / public
Publish from staging area to live wiki

Registration

Challenges often have a preregistration phase where people can sign up for more information. Then participants register, apply for access to data in some arbitrary way and once they're approved they are added to a group that grants data access privileges.

needs to be easier integrate with external systems for data access approval
build on the experience with the Prostate Cancer Challenge and Project Data Sphere
and the experience with the PRO-ACT database on the ALS challenge

Create Evaluation queues

Define structure of questions / subchallenges and create required queues
Define rounds and quotas
Create associated leaderboard in wiki

Manage evaluation queues

Show lists of scored and invalid submissions
Enable diagnosis of submission problems
Allow exceptions to the date and quota restrictions on rounds
Download all scored submissions along with a table of metadata
Use tables for scoring rather than submissions status annotations

Scoring

Work towards a more plug-n-play way to run scoring scripts
Automate launch and provisioning of scoring machine AWS instance
Schedule scoring script
Define email templates (validation failure or reporting successful scoring) for feedback to participants
Collect errors, invalid submissions, submission statistics, perhaps as an extension of the Challenge Statistics page.

Ideas that are really far-out and wacky

Automate Elias's Bayesian robustness analysis to show separation between submissions

Browser not supported