HPN Submission Scoring Workflow

HPN Submission Scoring Workflow

Below is the workflow and decision tree, both the 'standard' workflow and the 'administrative override'.

Quota Exceeded Email:  Your submission to the HPN xxxx Challenge (Synapse ID syn123456) has been received, but you have exceeded the submission quota of one submission per team per leader board round.  Please wait until the next round begins and submit again.  If you feel you've received this message in error, please contact Challenge support at http://support.sagebase.org/sagebase.

Invalid Submission Email:  Your submission to the HPN xxxx Challenge (Synapse ID syn123456) has been received, but was not accepted because it is not in a valid format.  The error message is: xxxxx.  You are still allowed one model submission for scoring in the current leader board round.   Please review your submission against the error code you received, make the necessary changes and try re-submitting.  Alternatively, you may submit a different model .  If you have questions, please contact Challenge support at http://support.sagebase.org/sagebase.

Submission Accepted Email:  Your valid submission to the HPN xxxx Challenge (Synapse ID syn123456) has been received and scored, as shown below.  Upon the next leader board update, your rank compared to other submissions will appear here:  https://www.synapse.org/#!Synapse:syn98765432.  If you have further questions, please contact Challenge support at http://support.sagebase.org/sagebase.

Score: Each value below is an AUC score, ranging from 0 to 1 where 0 is worst, 1 is best, and 0.5 is as good as a random guess.  There is a value for each cell line and stimulant, grouped by cell line:

Cell Line: BT20:

    IGF1: 0.5,

    PBS: 0.50673076923076932

    Serum: 0.41575091575091572

    NRG1: 0.55753968253968256

    Insulin: 0.36805555555555552

    HGF: 0.53606237816764124

    EGF: 0.58437500000000009

    FGF1: 0.27387914230019494

Cell Line: BT549:

    IGF1: 0.53645833333333337

    PBS: 0.51488833746898266

    Serum: 0.48958333333333326

    ...

Notes:

This is the workflow for scoring, but not for leader board updating.  This process will run frequently, to give users prompt feedback, while the leader board update will run infrequently, e.g. once per scoring round.

The weekly quota is tracked by counting the number of scored submissions having time stamps during the week in question, sent for a team.

We will provide an administrative override, allowing the challenge administrator to accept a submission even if it exceeds the weekly limit.  The process calculates the submission score, changes the submission status to SCORED, annotates the submission status with the reason for the override, and sends the Submission acceptance email.