Page Comparison

Source material:

...

submissionLimit is used to set submission limits

Code Block

language	json

{
   "name":"EvaluationRound",
   "description":"Defines the duration of a round and sets limits for maximum submissions per round",
   "properties":{
      "id": {
         "type": "string",
         "description": "The id of the EvaluationRound"
      },
      "roundStart": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the first round begins.",
         "required": true
      },
      "roundEnd":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the round ends.",
         "required": true
      },
      "submissionLimit": {
         "type": {
            "$ref": "org.sagebionetworks.evaluation.model.EvaluationRoundLimit"
         },
         "description": "Optional. Sets limits for maximum submissions in this round."
      }
   }
}

...

disallows all submissions
only 1 allowed per queue

Code Block

language	json

{
   "properties": {
      "start": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the maintenance begins.",
         "required": true
      },
      "end":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the maintenance ends.",
         "required": true
      },
   }
}

...

Store rounds List as JSON in a single database column.
- This would require searching for the correct round based on start/end dates in memory
- Evaluations object can be cleanly retrieved using query on single table
Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd
- Allows us to pull out one specific limit
- Separate queries
  - query to retrieve list of all rounds for that evaluation,
    - On REST API GET, we care about all rounds
  - query to retrieve list single evaluation id,
    - On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time
  - query to pull from the Evaluations table
- Limits are enforced in Java code so we can store the them as JSON
- Question: should ID be created by the id-generator? or just index in rounds list?
  - generated unique ID makes it easy to perform updates on evaluations
    - with current API setup a list of round are submitted, making this pointless
  - index in round list as ID
    - this allows easier tagging of submission
    - updating rounds involves row deletion and addition if added
      - hash metdata to avoid pointless delete/add?
  - If we use roundStart, roundEnd to identify the round config.
    - any change to start,roundend would mean an insertion/deletion

evaluationId(BIGINT)(Foreign Key to Evaluations Table)

ID(BIGINT)

roundStart(TIMESTAMP)

roundEnd(TIMESTAMP)

Limits(JSON)

123

1

2020-01-01 04:40:10

Code Block

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

123

2

2020-01-01 04:40:10

…

456

1

2020-01-01 04:40:10

…

789

1

2020-01-01 04:40:10

…

...

Records maintenance windows for each evaluation.

evaluationId(BIGINT)(Foreign Key to Evaluations Table)	startDate(TIMESTAMP)	endDate(TIMESTAMP)
123	`2020-01-01 04:40:10`	`2020-01-01 04:40:10`
456	`2020-01-01 04:40:10`	`2020-01-01 04:40:10`

...

Existing table:

Code Block

language	sql

ID BIGINT NOT NULL,
NAME varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
EVALUATION_ID BIGINT NOT NULL,
USER_ID BIGINT NOT NULL,
SUBMITTER_ALIAS varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
ENTITY_ID BIGINT NOT NULL,
ENTITY_BUNDLE mediumblob,
ENTITY_VERSION BIGINT NOT NULL,
CREATED_ON BIGINT NOT NULL,
TEAM_ID BIGINT,
DOCKER_REPO_NAME varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,
DOCKER_DIGEST varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci,

...

Versions Compared

Old Version 7

New Version 8

Key

Source material: