Page Comparison

Source material:

...

Code Block

language	json

{
	"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
	"name": "Evaluation",
	"properties": {
	
		"....currently existing fields...":{}


		"quota": {
			"type":"object",
			"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
			"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
		},

		"timeZone": {
			"type": "string",
			"description": "The time zone to be used for all round configurations."
		},

		"rounds": {
			"type": "array",
			"items": {
				"$ref": "org.sagebionetworks.evaluation.model.SubmissionRound"
			}
		}
	}
}

SubmissionRound (EvaluationRound?)

Defines roundStart and roundEnd dates.

...

submissionLimit is used to set submission limits

Code Block

{
   "name":"SubmissionRoundEvaluationRound",
   "description":"Defines the duration of a round and sets limits for maximum submissions per round",
   "properties":{
      "id": {
         "type": "string",
         "description": "The id of the SubmissionRoundEvaluationRound"
      },
      "roundStart": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the first round begins.",
         "required": true
      },
      "roundEnd":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the round ends.",
         "required": true
      },
      "submissionLimit": {
         "type": {
            "$ref": "org.sagebionetworks.evaluation.model.SubmissionRoundLimitEvaluationRoundLimit"
         },
         "description": "Optional. Sets limits for maximum submissions in this round."
      }
   }
}

SubmissionRoundLimitEvaluationRoundLimit

Limits for day, week, month, and total

...

Code Block

language	json

{
   "name":"SubmissionRoundLimitEvaluationRoundLimit",
   "description": "Sets limits for maximum submissions in a SubmissionRoundEvaluationRound",
   "properties":{
      "totalSubmissionLimit": {
         "type": "integer",
         "description": "The maximum total number of submissions per team/participant for the entirety of the round."
      },
      "dailySubmissionLimit": {
         "type": "integer",
         "description": "The maximum total number of submissions per team/participant for each day of the round."
      },
      "weeklySubmissionLimit": {
         "type": "integer",
         "description": "The maximum total number of submissions per team/participant for every 7 days of the round."
      },
      "monthlySubmissionLimit": {
         "type": {
            "$ref": "org.sagebionetworks.evaluation.model.SubmissionRoundMonthlyLimitEvaluationRoundMonthlyLimit"
         },
         "description": "The maximum total number of submissions per team/participant per month. Unlike other limits, this can be configured to reset on the n-th day of every month.",

      }
   }
}

SubmissionRoundMonthlyLimitEvaluationRoundMonthlyLimit

Code Block

{
   "properties": {
      "dayOfMonth": {
         "type": "integer",
         "description": "Day of the month on which this date this limit is reset."
      },
      "limit": {
         "type": "integer",
         "description": "The maximum total number of submissions per team/participant per month."
      }
   }
}

...

Store rounds List as JSON in a single database column.
- This would require searching for the correct round based on start/end dates in memory
- Evaluations object can be cleanly retrieved using query on single table
Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd
- Allows us to pull out one specific limit
- Separate queries
  - query to retrieve list of all rounds for that evaluation,
    - On REST API GET, we care about all rounds
  - query to retrieve list single evaluation id,
    - On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time
  - query to pull from the Evaluations table
- Limits are enforced in Java code so we can store the them as JSON
- Question: should ID be created by the id-generator? or just index in rounds list?
  - generated unique ID makes it easy to perform updates on evaluations
    - with current API setup a list of round are submitted, making this pointless
  - index in round list as ID
    - this allows easier tagging of submission
    - updating rounds involves row deletion and addition if added
      - hash metdata to avoid pointless delete/add?
  - If we use roundStart, roundEnd to identify the round config.
    - any change to start,roundend would mean an insertion/deletion

evaluationId(BIGINT)(Foreign Key to Evaluations Table)

ID(BIGINT)

roundStart(TIMESTAMP)

roundEnd(TIMESTAMP)

Limits(JSON)

123

1

2020-01-01 04:40:10

Code Block

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

123

2

2020-01-01 04:40:10

…

456

1

2020-01-01 04:40:10

…

789

1

2020-01-01 04:40:10

…

...

Records maintenance windows for each evaluation.

evaluationId(BIGINT)(Foreign Key to Evaluations Table)	startDate(TIMESTAMP)	endDate(TIMESTAMP)
123	`2020-01-01 04:40:10`	`2020-01-01 04:40:10`
456	`2020-01-01 04:40:10`	`2020-01-01 04:40:10`

...

Versions Compared

Old Version 6

New Version 7

Key

Source material: