Support for Challenges

The purpose here is to start a discussion and engage some design thinking about how to automate more of the process of creating and administrating a Challenge on Synapse.

Right now, running a challenge requires a lot of manual steps that involve interacting with the REST API which means an engineer has to be in the loop. Chris (aka me) has written some nice convenience layers in Python, but that still requires a user comfortable on the command line. The goal is to get more challenge related functionality in the hands of the challenge organizers.

This page is more about requirements gathering than design or implementation. Not all of this is necessarily practical, but it's here to get ideas from the whole team.

Create Challenge object:

A button/wizard to turn a project into a challenge. Some functionality this might entail:

• create team of access approved users
• and the challenge object that links the project to the team
• create admin team?
• create splash screen wiki
• add preregistration or registration button

Wikis

Challenge organizers need staging areas to collaborate on wikis prior before pushing changes to a live wiki.

• Challenge site begins with a splash screen live
• Staging area begins with template wiki (see syn2769515)
• Editing of wiki pages in staging area visible to challenge organizers but not participants / public
• Publish from staging area to live wiki

Registration

Challenges often have a preregistration phase where people can sign up for more information. Then participants register, apply for access to data in some arbitrary way and once they're approved they are added to a group that grants data access privileges.

• needs to be easier integrate with external systems for data access approval
• build on the experience with the Prostate Cancer Challenge and Project Data Sphere
• and the experience with the PRO-ACT database on the ALS challenge

Create Evaluation queues

• Define structure of questions / subchallenges and create required queues
• Define rounds and quotas
• Create associated leaderboard in wiki

Manage evaluation queues

• Show lists of scored and invalid submissions
• Enable diagnosis of submission problems
• Allow exceptions to the date and quota restrictions on rounds
• Download all scored submissions along with a table of metadata
• Use tables for scoring rather than submissions status annotations

Scoring

• Work towards a more plug-n-play way to run scoring scripts
• Automate launch and provisioning of scoring machine AWS instance
• Schedule scoring script
• Define email templates (validation failure or reporting successful scoring) for feedback to participants
• Collect errors, invalid submissions, submission statistics, perhaps as an extension of the Challenge Statistics page.

Ideas that are really far-out and wacky

• Automate Elias's Bayesian robustness analysis to show separation between submissions