Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

2. Create the Challenge site on Synapse.  This can easily be done with challengeutils (Installation instructions here:

Code Block
languagebash
challengeutils create-challenge "challenge_name"

...

Important!! After the initial copying, all changes to the live site should now be synced over with mirror-wiki; DO NOT use copyWiki again.  More on updating the Wikis under Update the Challenge.

Stop the Orchestrator

15. On the instance, enter: 
Ctrl + C  or cmd + C

...

Note that if the Challenge is currently active but you need to stop the orchestrator (for any reason, e.g. to make updates to the .env file, etc.), it may be helpful to first check whether any submissions are currently being evaluated.  If you are running the orchestrator in the background, you can monitor its activity by entering:

...

Otherwise, if you are not running the orchestrator in the background, read the logs on the terminal screen to determine whether there is currently current activity.

Wiki Setup

Use the following questions to help plan and set up the Challenge site and Evaluation Queues.

How many Challenge questions (“sub-challenges”) will there be? 

Will the participants submit a single file/model to answer all sub-challenges or will they submit a different file/model per sub-challenge?

What is the general timeline for the Challenge? 

Will there be rounds?  If so, how many? 

Using rounds may help increase participation levels throughout the Challenge, as submission activity is usually high near the end of  rounds/phases. It is best to have end dates during the mid-week if possible; this will ensure that there will be someone on-hand to help monitor and resolve issues should one arise.

Can users submit multiple submissions to a sub-challenge? 

If so, should there be a limit in frequency?  Examples: one submission per day, 3 submissions per week, 5 total, etc.

Setting a limit may help with potential overfitting as well as limit a user/team from monopolizing the compute resources.

What sort of submissions will the participants submit?
Common formats supported by Sage: prediction file (i.e. csv file), Docker image

When can the truth files (goldstandard) and training data (if any) be expected?
Will the data be released upon the challenge end? After the embargo? Never?

Is the data sensitive?
If so, will a clickwrap be needed (an agreement between the participant and data provider that requires the former to click a button that they will agree to the policies put in place regarding data usage)? Should log files be returned? Will there be a need to generate synthetic data?

Who will be responsible for providing/writing the validation and/or scoring scripts?
If Sage, please provide as many details regarding the format of a valid predictions file (e.g. number of columns, names of column headers, valid values, etc.) and all exceptional cases. For scoring, please provide the primary and secondary metrics, as well as any special circumstances for evaluations, i.e. CTD2 BeatAML primary metric is an average Spearman correlation, calculated from each drug’s Spearman correlation.

If not Sage, please provide the scripts in either Python or R.   If needed, we do provide sample scoring models that you may use as a template, available in both Python and R.

Are scores returned to the participants immediately or should they be withheld until the Challenge end?
A typical Challenge will immediately return the scores in an email upon evaluation completion, however, there have been past Challenges that did not return scores until after the end date.

There is also an “hybrid” approach, in which scores are immediately returned during the Leaderboard/Testing Phase but withheld during the Final/Validation Phase (in which participants do not know their performance until after the Challenge end).

When should the evaluation results/leaderboard be accessible to the participants?
Some past Challenges had active leaderboards (i.e. participants could readily view their ranking throughout the evaluation round) whereas other Challenges did not release the leaderboards until the round/Challenge was over.

Regarding writeups: when will these be accepted?
Should participants submit their writeups during submission evaluations or after the Challenge has closed?

A writeup is something we require of all participants in order to be considered for final evaluation and ranking. Within a writeup should be all contributing persons, a thorough description of their methods and usage of data outside of the Challenge data, as well as all of their scripts, code, and predictions file(s)/Docker image(s). We require all of these so that, should they be a top-performer, we can ensure their code and final output is reproducible.

Update the Challenge

Challenge Site

...