The purpose here is to start a discussion and engage some design thinking about how to automate more of the process of creating and administrating a Challenge on Synapse.
...
This page is more about requirements gathering than design or implementation. Not all of this is necessarily practical, but it's here to get ideas from the whole team.
Create Challenge object:
A button/wizard to turn a project into a challenge. Some functionality this might entail:
...
- Work towards a more plug-n-play way to run scoring scripts
- Automate launch and provisioning of scoring machine AWS instance
- Schedule scoring script
- Define email templates (validation failure or reporting successful scoring) for feedback to participants
- Collect errors, invalid submissions, submission statistics, perhaps as an extension of the Challenge Statistics page.
Ideas that are really far-out and wacky
- Automate Elias's Bayesian robustness analysis to show separation between submissions
...