Page Comparison

Source material:

...

Motivations:

Existing Evaluations have a very inflexible way to define rounds:

...

...

Source material:

UI Design:
https://www.figma.com/file/E5zIAPsI2fagDc8btWGU06Ol/Challenge-Evaluation-Queue

Use case/desired behavior:
https://docs.google.com/document/d/19M9og2JW7CaG7YLNi8cCRhFuFwF-y6-nx4CJDAwr6qw/edit#heading=h.y0949lyt5na0

Motivations:

Existing Evaluations have a very inflexible way to define rounds:

{
	"name":"SubmissionQuota",
	"description":"Maximum submissions per team/participant per submission round.  If round information is omitted, then this indicates the overall submission limit per team/participant.",
	"properties":{
		"firstRoundStart": {
			"type": "integerstring",
			"descriptionformat": "date-time"the maximum number of submissions per team/participant per round,
			"description": "The date/time at which the first round begins."
		},
		"roundDurationMillis":{
	}
}

The existing way to define Round limits assume that all rounds will last for the same duration and that later rounds will begin immediately as soon as the previous round ends. Users are also forced to perform unix timestamp calculations in order to set the round end datetime. There was also no fine grained submission limit control over Submission limits per user or participating team.

Proposed Changes

The goal is to provide users with more fine grained control over each Round in an Evaluation. This means:

...

Clearly defined start and end dates for each round.

• No more startDate + roundNumber * roundDruation math for the end date

Per-round submission limits. These limits should be changeable even during an ongoing round

...

cumulative limit for the entire round

...

per day - defined as every 24 hours from the start date

...

per week - defined as every 7 days from the start date

...

			"type": "integer",
			"description":"The duration of each round."
		},
		"numberOfRounds":{
			"type": "integer",
			"description":"The number of rounds, or null if there is no end."
		},
		"submissionLimit": {
			"type": "integer",
			"description":"the maximum number of submissions per team/participant per round."
		}
	}
}

The existing way to define Round limits assume that all rounds will last for the same duration and that later rounds will begin immediately as soon as the previous round ends. Users are also forced to perform unix timestamp calculations in order to set the round end datetime. There was also no fine grained submission limit control over Submission limits per user or participating team.

Proposed Changes

The goal is to provide users with more fine grained control over each Round in an Evaluation. This means:

• Clearly defined start and end dates for each round.

  • No more startDate + roundNumber * roundDruation math for the end date

• Per-round submission limits. These limits should be changeable even during an ongoing round

  • cumulative limit for the entire round

  • per day - defined as every 24 hours from the start date

  • per week - defined as every 7 days from the start date

  • per month

    • monthly reset on every n-th day of the month. Use n==31, for end of month.

      • By default could be filled in with same day of month as the start of round

      • Requires adding a time zone field for the entire Evaluation queue since the day could be off by 1 depending on the time zone

    • OR monthly reset every 30 days.

• Add/remove/modify rounds without affecting other rounds

  • Change start date if round not yet started

  • Change end date if round not yet ended

  • Add/delete/change new rounds that start after the current time.

• User’s Submissions will be automatically tagged with the index number of the current round

  • Additional metadata column in Submission views

• Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions

  • Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance

...

Question: should roundStart and roundEnd be purely integers? or allow ISO timestamps? Other fields in Evaluation(e.g. createdOn ) use “date-time” as type so we would maintain consistency, but it makes a rather odd paring to have timeZone='US/Pacific' and roundStart='2020-08-11T16:45:10−08:00'

id is a UUID generated to identify rounds for updating can used for updates if we store rounds in a separate table

submissionLimit is used to set submission limits

{
   "name":"SubmissionRound",
   "description":"Defines the duration of a round and sets limits for maximum submissions per round",
   "properties":{
      "id": {
         "type": "string",
         "description": "The id of the SubmissionRound"
      },	2020-08-11T16:45:10+0000
      "roundStart": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the first round begins.",
         "required": true
      },
      "roundEnd":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the round ends.",
         "required": true
      },
      "submissionLimit": {
         "type": {
            "$ref": "org.sagebionetworks.evaluation.model.SubmissionRoundLimit"
         },
         "description": "Optional. Sets limits for maximum submissions in this round."
      }
   }
}

...

{
   "....currently existing fields...":{},
   
   
   "timeZone":"US/Pacific",
   
   "rounds":[
      {
         "id":"bcbd1205-80e9-44c5-85e2-dbcb194e5e761",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":20,
            "dailySubmissionLimit":2,
            "weeklySubmissionLimit":8,
            "monthlySubmissionLimit":{
               "dayOfMonth":28,
               "limit":10
            }
         }
      },
      
      
      {
         "id":"254110e2-aead-4bf8-9ce4-3786be59a4972",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }
      
      
   ]
}

...

API/Database Transition

A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.

For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and rounds when configuring a Evaluation. This allows time for clients to implement support for the new rounds field.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.

API/Database Transition

A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.

For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and rounds when configuring a Evaluation. This allows time for clients to implement support for the new rounds field.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.

Database Tables

2 Options:

• Store rounds List as JSON in a single database column.

  • This would require searching for the correct round based on start/end dates in memory

  • Evaluations object can be cleanly retrieved using query on single table

• Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd

  • Allows us to pull out one specific limit

  • Separate queries

    • query to retrieve list of all rounds for that evaluation,

      • On REST API GET, we care about all rounds

    • query to retrieve list single evaluation id,

      • On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time

    • query to pull from the Evaluations table

  • Limits are enforced in Java code so we can store the them as JSON

  • Question: should ID be created by the id-generator? or just index in rounds list?

    • generated unique ID makes it easy to perform updates on evaluations

      • with current API setup a list of round are submitted, making this pointless

    • index in round list as ID

      • this allows easier tagging of submission

      • updating rounds involves row deletion and addition if added

        • hash metdata to avoid pointless delete/add?

    • If we use roundStart, roundEnd to identify the round config.

      • any change to start,roundend would mean an insertion/deletion

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

Evaluation table

Add EVALUATION_ROUND, which is the round number during which it was submitted. This will be replicated into Submission Views.

Existing table:

ID BIGINT NOT NULL,
NAME varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
EVALUATION_ID BIGINT NOT NULL,
USER_ID BIGINT NOT NULL,
SUBMITTER_ALIAS varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
ENTITY_ID BIGINT NOT NULL,
ENTITY_BUNDLE mediumblob,
ENTITY_VERSION BIGINT NOT NULL,
CREATED_ON BIGINT NOT NULL,
TEAM_ID BIGINT,
DOCKER_REPO_NAME varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
DOCKER_DIGEST varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,