Page Comparison

Source material:

...

• Clearly defined start and end dates for each round.

  • No more startDate + roundNumber * roundDruation math for the end date

• Per-round submission limits. These limits should be changeable even during an ongoing round

  • cumulative limit for the entire round

  • per day - defined as every 24 hours from the start date

  • per week - defined as every 7 days from the start date

  • per month This will not be implemented until we have further clarification on its definition

    • monthly reset on every n-th day of the month. Use n==31, for end of month.

      • By default could be filled in with same day of month as the start of round

      • Requires adding a time zone field for the entire Evaluation queue since the day could be off by 1 depending on the time zone

    • OR monthly reset every 30 days.

• Add/remove/modify rounds without affecting other rounds

  • Change start date if round not yet started

  • Change end date if round not yet ended

  • Add/delete/change new rounds that start after the current time.

  • Validation

    • Disallow intersecting time intervals between rounds

    • Optional: total limit > month limit > week limit > day limit

• User’s Submissions will be automatically tagged with the index number of the current round

  • Additional metadata column in Submission views

• Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions

  • Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance

...

{
	"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
	"name": "Evaluation",
	"properties": {
	
		"....currently existing fields...":{}


		"quota": {
			"type":"object",
			"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
			"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
			}
	}
	}
}

EvaluationRoundRequest

Used to retrieve EvaluationRounds info

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
      "nextPageToken": {
         "type": "string",
         "description": "Token that can be used to get the next page. If null, first page is returned."
      }
}

EvaluationRoundResponse

Used to retrieve EvaluationRounds info

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
      "page": {
         "type": "array",
         "description": "The rounds in this evaluation.",
         "items": {
            "$ref": "org.sagebionetworks.evaluation.model.EvaluationRound"
         }
      },
      "nextPageToken": {
         "type": "string",
         "description": "Token that can be used to get the next page.  Null if there are no more results."
      }
}

EvaluationRound

Defines roundStart and roundEnd dates.

...

API/Database Transition

A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.

...

• Store rounds List as JSON in a single database column.

  • This would require searching for the correct round based on start/end dates in memory

  • Evaluations object can be cleanly retrieved using query on single table

• Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd

  • Allows us to pull out one specific limit

  • Separate queries

    • query to retrieve list of all rounds for that evaluation,

      • On REST API GET, we care about all rounds

    • query to retrieve list single evaluation id,

      • On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time

    • query to pull from the Evaluations table

  • Limits are enforced in Java code so we can store the them as JSON

  • Question: should ID be created by the id-generator? or just index in rounds list?

    • generated unique ID makes it easy to perform updates on evaluations

      • with current API setup a list of round are submitted, making this pointless

    • index in round list as ID

      • this allows easier tagging of submission

      • updating rounds involves row deletion and addition if added

        • hash metdata to avoid pointless delete/add?

    • If we use roundStart, roundEnd to identify the round config.

      • any change to start,roundend would mean an insertion/deletion

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

...

Records maintenance windows for each evaluation.

...