This was recently requested by the IDG-DREAM Challenge collaborators.
It would be helpful to allow us to require participants to fill out a survey to su bmit to a challenge queue. For example, in this challenge, we needed to gather methodologic information in a structured format to permit post challenge analysis. We asked questions like this:
What approach did you apply?
If you used any "named" algorithms or methods, please list them here:
What public training data did you use? Did you use any private data?
If you used private data, please describe
Did you ensemble multiple models?
But because this was a Google survey and we couldn't easily tie these results to a submission, we were also forced to ask questions like:
What is your Synapse username?
If submitting as part of a team, what is your team name?
What is your current submission ‘objectId’? (this can be found on the leaderboard)
to tie this to individual submissions. This requires a lot of legwork, and tracking people down and many people do not understand what an objectId is (so they submit the synapse ID of a synapse file that we don't have access to, for example). In addition, most people don't fill the survey out, so requiring it alongside submission might help this.
cc - is this something that would benefit challenges you have run?