Source material:
UI Design:
https://www.figma.com/file/E5zIAPsI2fagDc8btWGU06Ol/Challenge-Evaluation-Queue
Use case/desired behavior:
https://docs.google.com/document/d/19M9og2JW7CaG7YLNi8cCRhFuFwF-y6-nx4CJDAwr6qw/edit#heading=h.y0949lyt5na0
Motivations:
Existing Evaluations have a very inflexible way to define rounds:
{
"name":"SubmissionQuota",
"description":"Maximum submissions per team/participant per submission round. If round information is omitted, then this indicates the overall submission limit per team/participant.",
"properties":{
"firstRoundStart": {
"type": "string",
"format": "date-time",
"description": "The date/time at which the first round begins."
},
"roundDurationMillis":{
"type": "integer",
"description":"The duration of each round."
},
"numberOfRounds":{
"type": "integer",
"description":"The number of rounds, or null if there is no end."
},
"submissionLimit": {
"type": "integer",
"description":"the maximum number of submissions per team/participant per round."
}
}
}
The existing way to define Round limits assume that all rounds will last for the same duration and that later rounds will begin immediately as soon as the previous round ends. Users are also forced to perform unix timestamp calculations in order to set the round end datetime. There was also no fine grained submission limit control over Submission limits per user or participating team.
Proposed Changes
The goal is to provide users with more fine grained control over each Round in an Evaluation. This means:
Clearly defined start and end dates for each round.
No more startDate + roundNumber * roundDruation math for the end date
Per-round submission limits. These limits should be changeable even during an ongoing round
cumulative limit for the entire round
per day - defined as every 24 hours from the start date
per week - defined as every 7 days from the start date
per month
monthly reset on every n-th day of the month. Use n==31, for end of month.
By default could be filled in with same day of month as the start of round
Requires adding a time zone field for the entire Evaluation queue since the day could be off by 1 depending on the time zone
OR monthly reset every 30 days.
Add/remove/modify rounds without affecting other rounds
Change start date if round not yet started
Change end date if round not yet ended
Add/delete/change new rounds that start after the current time.
User’s Submissions will be automatically tagged with the index number of the current round
Additional metadata column in Submission views
Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions
Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance
API Schema
Evaluation
Add timeZone, rounds. Deprecate quota
timeZone will use long names (e.g. “US/Pacific”) instead of 3-letter names (e.g. PDT, PST) to account for Daylight savings
{
"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
"name": "Evaluation",
"properties": {
"....currently existing fields...":{}
"quota": {
"type":"object",
"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
},
"timeZone": {
"type": "string",
"description": "The time zone to be used for all round configurations."
},
"rounds": {
"type": "array",
"items": {
"$ref": "org.sagebionetworks.evaluation.model.SubmissionRound"
}
}
}
}
SubmissionRound (EvaluationRound?)
Defines roundStart and roundEnd dates.
Question: should roundStart and roundEnd be purely integers? or allow ISO timestamps? Other fields in Evaluation(e.g. createdOn ) use “date-time” as type so we would maintain consistency, but it makes a rather odd paring to have timeZone='US/Pacific' and roundStart='2020-08-11T16:45:10−08:00'
id is a UUID generated to identify rounds for updating
submissionLimit is used to set submission limits
{
"name":"SubmissionRound",
"description":"Defines the duration of a round and sets limits for maximum submissions per round",
"properties":{
"id": {
"type": "string",
"description": "The id of the SubmissionRound"
}, 2020-08-11T16:45:10+0000
"roundStart": {
"type": "string",
"format": "date-time",
"description": "The date/time at which the first round begins.",
"required": true
},
"roundEnd":{
"type": "string",
"format": "date-time",
"description":"The date/time at which the round ends.",
"required": true
},
"submissionLimit": {
"type": {
"$ref": "org.sagebionetworks.evaluation.model.SubmissionRoundLimit"
},
"description": "Optional. Sets limits for maximum submissions in this round."
}
}
}
SubmissionRoundLimit
Limits for day, week, month, and total
monthlySubmissionLimit is defined in a separate object
{
"name":"SubmissionRoundLimit",
"description": "Sets limits for maximum submissions in a SubmissionRound",
"properties":{
"totalSubmissionLimit": {
"type": "integer",
"description": "The maximum total number of submissions per team/participant for the entirety of the round."
},
"dailySubmissionLimit": {
"type": "integer",
"description": "The maximum total number of submissions per team/participant for each day of the round."
},
"weeklySubmissionLimit": {
"type": "integer",
"description": "The maximum total number of submissions per team/participant for every 7 days of the round."
},
"monthlySubmissionLimit": {
"type": {
"$ref": "org.sagebionetworks.evaluation.model.SubmissionRoundMonthlyLimit"
},
"description": "The maximum total number of submissions per team/participant per month. Unlike other limits, this can be configured to reset on the n-th day of every month.",
}
}
}
SubmissionRoundMonthlyLimit
{
"properties": {
"dayOfMonth": {
"type": "integer",
"description": "Day of the month on which this date this limit is reset."
},
"limit": {
"type": "integer",
"description": "The maximum total number of submissions per team/participant per month."
}
}
}
Example Evaluation
id will be back-end generated after it has been created
{
"....currently existing fields...":{},
"timeZone":"US/Pacific",
"rounds":[
{
"id":"bcbd1205-80e9-44c5-85e2-dbcb194e5e76",
"roundStart":1231412213213123123,
"roundEnd":"2020-08-11T16:45:10−08:00",
"submissionLimit":{
"totalSubmissionLimit":20,
"dailySubmissionLimit":2,
"weeklySubmissionLimit":8,
"monthlySubmissionLimit":{
"dayOfMonth":28,
"limit":10
}
}
},
{
"id":"254110e2-aead-4bf8-9ce4-3786be59a497",
"roundStart":1231412213213123123,
"roundEnd":"2020-08-11T16:45:10−08:00",
"submissionLimit":{
"totalSubmissionLimit":40,
"dailySubmissionLimit":8,
"weeklySubmissionLimit":12,
"monthlySubmissionLimit":{
"dayOfMonth":31,
"limit":20
}
}
}
]
}
EvaluationMaintenance
disallows all submissions
only 1 allowed per queue
{
"properties": {
"start": {
"type": "string",
"format": "date-time",
"description": "The date/time at which the maintenance begins.",
"required": true
},
"end":{
"type": "string",
"format": "date-time",
"description":"The date/time at which the maintenance ends.",
"required": true
},
}
}
API Endpoints
Endpoint | Request body | Return Body | Description |
|---|---|---|---|
None | Evaluation | EXISTING endpoint to get evaluation | |
Evaluation | Evaluation | EXISTING endpoint to create an evaluation | |
Evaluation | Evaluation | EXISTING endpoint to update evaluation | |
POST /evaluation/{evalId}/maintainance | EvaluationMaintenance | None or EvaluationMaintenance | Schedule a maintainance. Additional calls will override the existing maintenance window |
DELETE /evaluation/{evalId}/maintainance | None | None | Delete the existing maintenance window, if any |
API/Database Transition
A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.
For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and rounds when configuring a Evaluation. This allows time for clients to implement support for the new rounds field.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.