Page Comparison

Motivations:

Existing Evaluations have a very inflexible way to define rounds:

{
	"name":"SubmissionQuota",
	"description":"Maximum submissions per team/participant per submission round.  If round information is omitted, then this indicates the overall submission limit per team/participant.",
	"properties":{
		"firstRoundStart": {
			"type": "string",
			"format": "date-time",
			"description": "The date/time at which the first round begins."
		},
		"roundDurationMillis":{
			"type": "integer",
			"description":"The duration of each round."
		},
		"numberOfRounds":{
			"type": "integer",
			"description":"The number of rounds, or null if there is no end."
		},
		"submissionLimit": {
			"type": "integer",
			"description":"the maximum number of submissions per team/participant per round."
		}
	}
}

The existing way to define Round limits assume that all rounds will last for the same duration and that later rounds will begin immediately as soon as the previous round ends. Users are also forced to perform unix timestamp calculations in order to set the round end datetime. There was also no fine grained submission limit control over Submission limits per user or participating team.

Proposed Changes

The goal is to provide users with more fine grained control over each Round in an Evaluation. This means:

• Clearly defined start and end dates for each round.

  • No more startDate + roundNumber * roundDruation math for the end date

• Per-round submission limits. These limits should be changeable even during an ongoing round

  • cumulative limit for the entire round

  • per day - defined as every 24 hours from the start date

  • per week - defined as every 7 days from the start date

  • per month This will not be implemented until we have further clarification on its definition

    • monthly reset on every n-th day of the month. Use n==31, for end of month.

      • By default could be filled in with same day of month as the start of round

      • Requires adding a time zone field for the entire Evaluation queue since the day could be off by 1 depending on the time zone

    • OR monthly reset every 30 days.

• Add/remove/modify rounds without affecting other rounds

  • Change start date if round not yet started

  • Change end date if round not yet ended

  • Add/delete/change new rounds that start after the current time.

  • Validation

    • Disallow intersecting time intervals between rounds

    • Optional: total limit > month limit > week limit > day limit

• User’s Submissions will be automatically tagged with the index number of the current round

  • Additional metadata column in Submission views

• Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions

  • Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance

API Schema

Evaluation
Deprecate quota.

New evaluation round will be a list of objects which will be accessible via a new API

{
	"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
	"name": "Evaluation",
	"properties": {
	
		"....currently existing fields...":{}


		"quota": {
			"type":"object",
			"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
			"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
		}
	}
}

EvaluationRoundRequest

Used to retrieve EvaluationRounds info

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
      "nextPageToken": {
         "type": "string",
         "description": "Token that can be used to get the next page. If null, first page is returned."
      }
}

EvaluationRoundResponse

Used to retrieve EvaluationRounds info

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
      "page": {
         "type": "array",
         "description": "The rounds in this evaluation.",
         "items": {
            "$ref": "org.sagebionetworks.evaluation.model.EvaluationRound"
         }
      },
      "nextPageToken": {
         "type": "string",
         "description": "Token that can be used to get the next page.  Null if there are no more results."
      }
}

EvaluationRound

Defines roundStart and roundEnd dates.

id can used for updates if we store rounds in a separate table

evaluationLimit is used to set limits on submissions

...

...

Source material:

...

Source material:

UI Design:
https://www.figma.com/file/E5zIAPsI2fagDc8btWGU06Ol/Challenge-Evaluation-Queue

Use case/desired behavior:
https://docs.google.com/document/d/19M9og2JW7CaG7YLNi8cCRhFuFwF-y6-nx4CJDAwr6qw/edit#heading=h.y0949lyt5na0

Motivations:

Existing Evaluations have a very inflexible way to define rounds:

{
	"name":"SubmissionQuota",
	"description":"Maximum submissions per team/participant per submission round.  If round information is omitted, then this indicates the overall submission limit per team/participant.",
	"properties":{
		"firstRoundStart": {
			"type": "string",
			"format": "date-time",
			"description": "The date/time at which the first round begins."
		},
		"roundDurationMillis":{
			"type": "integer",
			"description":"The duration of each round."
		},
		"numberOfRounds":{
			"type": "integer",
			"description":"The number of rounds, or null if there is no end."
		},
		"submissionLimit": {
			"type": "integer",
			"description":"the maximum number of submissions per team/participant per round."
		}
	}
}

The existing way to define Round limits assume that all rounds will last for the same duration and that later rounds will begin immediately as soon as the previous round ends. Users are also forced to perform unix timestamp calculations in order to set the round end datetime. There was also no fine grained submission limit control over Submission limits per user or participating team.

High level changes

The goal is to provide users with more fine grained control over each Round in an Evaluation. This means:

• Clearly defined start and end dates for each round.

  • Initial design will not allow unbounded start or end dates.

    • The old SubmissionQuota system does allow unbounded start dates, end dates , or both.

  • No more startDate + roundNumber * roundDruation math for the end date

• Per-round submission limits. These limits should be changeable even during an ongoing round

  • cumulative limit for the entire round

  • per day - resets every day at 00:00 UTC

  • per week - resets every Monday at 00:00 UTC

  • per month - resets at 00:00 UTC on the 1st day of every month

• Add/remove/modify rounds without affecting other rounds

  • Change start datetime if round not yet started (current time < start time) AND 0 submissions were made

    • If 0 submissions made, the round may still be updated/deleted even if the time is past the start date

  • Change end date to any time in the future (current time < end time)

    • Setting the end datetime to a later time is always allowed

    • Setting end datetime to a time in the past is not allowed.

  • Add/delete/change new rounds that have not yet started.

  • Validation

    • Disallow intersecting time intervals between rounds

      • i.e. no overlap between time ranges of any 2 rounds belonging to the same Evaluation

    • SubmissionQuota and EvaluationRounds may not both exist. must pick one or the other

• User’s Submissions will be automatically tagged with the id of the current EvaluationRound

  • Additional metadata column in Submission views

• Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions

  • Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance Not implemented

API Schema

Evaluation
Deprecate quota.

New evaluation round will be a list of objects which will be accessible via a new API

{
	"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
	"name": "Evaluation",
	"properties": {
	
		"....currently existing fields...":{}


		"quota": {
			"type":"object",
			"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
			"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
		}
	}
}

EvaluationRoundRequest

Used to retrieve all EvaluationRounds associated with an Evaluation. Results are paginated. Use nextPageToken to retrieve the next page of results.

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
      "nextPageToken": {
         "type": "string",
         "formatdescription": "date-time",
         "description": "The date/time at which the first round begins.",
         "required": true
      },
      "roundEnd":Token that can be used to get the next page. If null, first page is returned."
      }
}

EvaluationRoundResponse

One page of EvaluationRounds associated with an Evaluation. Use the givennextPageToken in the next EvaluationRoundRequest to retrieve the next page of results.

{
   "description": "Response of EntityHeaders for the children of a given parent Entity",
   "properties": {
         "typepage": "string",{
         "formattype": "date-timearray",
         "description": "The date/time at which the round endsrounds in this evaluation.",
         "requireditems": true{
      },       "submissionLimit$ref": {"org.sagebionetworks.evaluation.model.EvaluationRound"
         "type":}
{      },
      "$refnextPageToken": "org.sagebionetworks.evaluation.model.EvaluationRoundLimit" {
          }"type": "string",
         "description": "OptionalToken that can be used to get the next page. Sets limitsNull forif maximumthere submissionsare inno thismore roundresults."
      }

  }
}

EvaluationRoundLimit

...

EvaluationRound

Defines roundStart and roundEnd dates.

id can used for updates if we store rounds in a separate table

limits is used to set limits on submissions

{
   	"namedescription":"EvaluationRoundLimit",Defines the duration  "description": "Setsof a round and sets limits for maximum submissions inper a EvaluationRoundround",
   	"properties":{
		"id": {
			"type":    "totalSubmissionLimit"string",
			"description": {"The id of the EvaluationRound"
     		},
		"etag": {
			"type": "integerstring",
			"description": "Synapse employs an Optimistic Concurrency Control (OCC) scheme to handle  "description": "The maximum total number of submissions per team/participant for the entirety of the round."
      },
      "periodicLimits": {
         concurrent updates. The eTag changes every time an EvaluationRound is updated; it is used to detect when a client's copy of an Evaluation is out-of-date."
		},
		"evaluationId": {
			"type": "string",
			"description": "The Evaluation to which this EvaluationRound belongs",
			"required": true
		},
		"roundStart": {
			"type": "arraystring",
			"format":         "date-time",
			"description": "limitsThe thatdate/time willat resetwhich periodically"the first round begins.",
			"required": true
		},
		"roundEnd":{
			"type": "string",
			"itemsformat": "org.sagebionetworks.evaluation.model.EvaluationRoundPeriodicLimit"
      }
   }
}

EvaluationRoundPeriodicLimit

Limits for total submission limit for the round, and a list of periodically resetting limits

{
   "name":"EvaluationRoundPeriodicLimit",
   "description": "Limits that will reset after every specified number of days",
   "properties":{
      "period": {
         "type": "integer",
         "description": "Maximum number of days required for the limit to reset"
      },
      "limit": {
         "type": "integer",
         "description": "Maximum number of submissions in this period"
      }
   }
}

Example Evaluation

id will be back-end generated after it has been created

{
   "....currently existing fields...":{},
   
   
   "timeZone":"US/Pacific",
   
   "rounds":[
      {
         "id":"1",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":20,
            "periodicLimits":[
              {
     date-time",
			"description":"The date/time at which the round ends.",
			"required": true
		},
		"limits": {
			"type": "array",
			"items": {
				"$ref": "org.sagebionetworks.evaluation.model.EvaluationRoundLimit"
			},
			"description": "Optional. Sets limits for maximum submissions in this round."
		}
	}
}

EvaluationRoundLimit

Limits for total submission limit for the round, and a list of periodically resetting limits.

{
	"description": "Sets limits for maximum submissions in a SubmissionRound. Each limitType may only appear once in an EvaluationRound's list of limits",
	"properties":{
		"limitType":{
			"type": "string",
			"name": "EvaluationRoundLimitType",
			"required": true,
			"enum": [
				{
					"name": "TOTAL",
					"description": "Limits maximum submissions for the entire duration of the round."
				},
				{
					"name": "DAILY",
					"description": "Limits maximum submissions per day. Resets daily at 00:00:00 UTC"
				},
				{
					"name": "WEEKLY",
					"description": "Limits maximum submissions per calendar week. Resets every Monday at 00:00:00 UTC"
				},
				{
					"name": "MONTHLY",
					"description": "Limits maximum submissions per calendar month. Resets on the 1st day of every calendar month at 00:00:00 UTC"
				}
			]
		},

		"maximumSubmissions": {
			"type": "integer",
			"required": true,
			"description": "maximum number of submissions allowed within the time period defined by the the limitType"
		}
	}
}

Example Evaluation

id will be back-end generated after it has been created. For roundStart and roundEnd, both UNIX timestamp in milliseconds and ISO time format are accepted

{
   "....currently existing fields...":{},
   
   
   "periodtimeZone": 1
  "US/Pacific",
   
          "limit"rounds":[
6      {
         },
              {
       "id":"1",
         "periodroundStart":1231412213213123123,
7         "roundEnd":"2020-08-11T16:45:10−08:00",
         "limitsubmissionLimit":[
 14        {
      }      "limitType": "TOTAL"
     ]       "maximumSubmissions":5
  }       },
         {
          {          "id"limitType": "2",
    WEEKLY"
    "roundStart":1231412213213123123,          "roundEndmaximumSubmissions":"2020-08-11T16:45:10−08:00",2
         "submissionLimit":{}
         ]
  "totalSubmissionLimit":40,     },
      
"periodicLimits":[      
      {
 {        "id":"2",
         "periodroundStart":1231412213213123123,
1         "roundEnd":"2020-08-11T16:45:10−08:00",
         "limitsubmissionLimit":[
4         {
     },       "limitType": "DAILY"
      {      "maximumSubmissions":2
         },
 "period": 7        {
            "limitlimitType": 18"MONTHLY"
            "maximumSubmissions":8
 }         }
   ]      ]
   }       }
      
      
   ]
}

EvaluationMaintenance Not implemented

disallows all submissions
only 1 allowed per queue

{
   "properties": {
      "start": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the maintenance begins.",
         "required": true
      },
      "end":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the maintenance ends.",
         "required": true
      },
   }
}

API Endpoints

...

Endpoint

...

Request body

...

Return Body

...

Description

...

GET /evaluation/{evalId}

...

None

...

Evaluation

...

EXISTING endpoint to get evaluation

...

POST /evaluation

...

Evaluation

...

Evaluation

...

EXISTING endpoint to create an evaluation

...

PUT /evaluation/{evalId}

...

Evaluation

...

Evaluation

...

EXISTING endpoint to update evaluation

...

POST /evaluation/{evalId}/maintainance

...

EvaluationMaintenance

...

None or EvaluationMaintenance

...

Schedule a maintainance. Additional calls will override the existing maintenance window

...

  }
}

API Endpoints

API/Database Transition

A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.

For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and the new EvaluationRounds API when configuring a Evaluation. This allows time for clients to implement support for the new API.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.

Database Tables

2 Options:

...

Store rounds List as JSON in a single database column.

• This would require searching for the correct round based on start/end dates in memory

• Evaluations object can be cleanly retrieved using query on single table

Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd

...

Allows us to pull out one specific limit

...

Separate queries

• query to retrieve list of all rounds for that evaluation,

  • On REST API GET, we care about all rounds

• query to retrieve list single evaluation id,

  • On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time

• query to pull from the Evaluations table

...

Limits are enforced in Java code so we can store the them as JSON

Question: should ID be created by the id-generator? or just index in rounds list?

...

generated unique ID makes it easy to perform updates on evaluations

• with current API setup a list of round are submitted, making this pointless

...

index in round list as ID

• this allows easier tagging of submission

• updating rounds involves row deletion and addition if added

  • hash metdata to avoid pointless delete/add?

If we use roundStart, roundEnd to identify the round config.

...

API/Database Transition

A SubmissionQuota can easily be mapped into into a list of SubmissionRounds using math, but the reverse is not true.

For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and the new EvaluationRounds API when configuring a Evaluation. This allows time for clients to implement support for the new API.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.
SubmissionQuota currently allows unbounded round start/end dates by setting null. This would not be desired behavior for EvaluationRounds because it would complicate queries that involve date ranges. We can mimic the unbounded time range by migrating the start/end to some time far in the past or future ( best of luck to the devs dealing with this in 2324 AD ! ).

Database Tables

2 Options:

• Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd

  • Allows us to pull out one specific R limit

  • Limits are enforced in Java code so we can store the them as JSON

  • generated unique ID makes it easy to perform updates on evaluations

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

Evaluation_maintenance table

Records maintenance windows for each evaluation.

JDOSUBMISSION table

Add EVALUATION_ROUND, which is the round number during which it was submitted. This will be replicated into Submission Views.

...