Page Comparison

Source material:

...

• Clearly defined start and end dates for each round.

  • No more startDate + roundNumber * roundDruation math for the end date

• Per-round submission limits. These limits should be changeable even during an ongoing round

  • cumulative limit for the entire round

  • per day - defined as every 24 hours from the start date

  • per week - defined as every 7 days from the start date

  • per month This will not be implemented until we have further clarification on its definition

    • monthly reset on every n-th day of the month. Use n==31, for end of month.

      • By default could be filled in with same day of month as the start of round

      • Requires adding a time zone field for the entire Evaluation queue since the day could be off by 1 depending on the time zone

    • OR monthly reset every 30 days.

• Add/remove/modify rounds without affecting other rounds

  • Change start date if round not yet started

  • Change end date if round not yet ended

  • Add/delete/change new rounds that start after the current time.

  • Validation

    • Disallow intersecting time intervals between rounds

    • Optional: total limit > month limit > week limit > day limit

• User’s Submissions will be automatically tagged with the index number of the current round

  • Additional metadata column in Submission views

• Allow Evaluation Queue Admins to schedule a Maintenance date range that disallows Submissions

  • Independent of defined rounds - neither the ongoing rounds nor later rounds will have their start/end date modified as a result of maintenance

API Schema

Evaluation
Add timeZone, rounds. Deprecate quota.timeZone will use long names (e.g. “US/Pacific”) instead of 3-letter names (e.g. PDT, PST) to account for Daylight savings

New evaluation round will be a list of objects which will be accessible via a new API

{
	"description": "An Evaluation is the core object of the Evaluation API, used to support collaborative data analysis challenges in Synapse.",
	"name": "Evaluation",
	"properties": {
	
		"....currently existing fields...":{}


		"quota": {
			"type":"object",
			"description":"DEPRECATED. Maximum submissions per team/participant per submission round",
			"$ref":"org.sagebionetworks.evaluation.model.SubmissionQuota"
		},

		"timeZone": {
			"type": "string",
			"description": "The time zone to be used for all round configurations."
		},

		"rounds": {
			"type": "array",
			"items": {
				"$ref": "org.sagebionetworks.evaluation.model.SubmissionRound"
			}
		}
	}
}

EvaluationRound

Defines roundStart and roundEnd dates. Question: should roundStart and roundEnd be purely integers? or allow ISO timestamps? Other fields in Evaluation(e.g. createdOn ) use “date-time” as type so we would maintain consistency, but it makes a rather odd paring to have timeZone='US/Pacific' and roundStart='2020-08-11T16:45:10−08:00' dates.

id can used for updates if we store rounds in a separate table

submissionLimit evaluationLimit is used to set submission limits on submissions

{
   "name":"EvaluationRound",
   "description":"Defines the duration of a round and sets limits for maximum submissions per round",
   "properties":{
      "id": {
         "type": "string",
         "description": "The id of the EvaluationRound"
      },
      "roundStart": {
         "type": "string",
         "format": "date-time",
         "description": "The date/time at which the first round begins.",
         "required": true
      },
      "roundEnd":{
         "type": "string",
         "format": "date-time",
         "description":"The date/time at which the round ends.",
         "required": true
      },
      "submissionLimit": {
         "type": {
            "$ref": "org.sagebionetworks.evaluation.model.EvaluationRoundLimit"
         },
         "description": "Optional. Sets limits for maximum submissions in this round."
      }
   }
}

EvaluationRoundLimit

Limits for day, week, month, and totalmonthlySubmissionLimit is defined in a separate objecttotal submission limit for the round, and a list of periodically resetting limits. Currently the maximum number of periodicLimits is 5

{
   "name":"EvaluationRoundLimit",
   "description": "Sets limits for maximum submissions in a EvaluationRound",
   "properties":{
      "totalSubmissionLimit": {
         "type": "integer",
         "description": "The maximum total number of submissions per team/participant for the entirety of the round."
      },
      "dailySubmissionLimitperiodicLimits": {
         "type": "integerarray",
         "description": "The maximum total number of submissions per team/participant for each day of the round."
      },: "limits that will reset periodically"
         "weeklySubmissionLimititems": { "org.sagebionetworks.evaluation.model.EvaluationRoundPeriodicLimit"
      }
   "type": "integer",}
}

EvaluationRoundPeriodicLimit

Limits for total submission limit for the round, and a list of periodically resetting limits

{
   "name":"EvaluationRoundPeriodicLimit",
     "description": "TheLimits maximumthat totalwill number of submissions per team/participant for every 7 days of the round."reset after every specified number of days",
   "properties":{
  },
      "monthlySubmissionLimitperiod": {
         "type": {"integer",
            "$refdescription": "org.sagebionetworks.evaluation.model.EvaluationRoundMonthlyLimit"
         },
  Maximum number of days required for the limit to reset"
      },
      "descriptionlimit": "The maximum total number of submissions per team/participant per month. Unlike other limits, this can be configured to reset on the n-th day of every month.",
{
         "type": "integer",
         "description": "Maximum number of submissions in this period"
      }
   }
}

...

Example Evaluation

id will be back-end generated after it has been created

{
   "properties....currently existing fields...": {{},
   
   
   "dayOfMonthtimeZone":"US/Pacific",
 {  
   "rounds":[
   "type": "integer",   {
         "descriptionid": "Day of the month on which this date this limit is reset.""1",
         "roundStart":1231412213213123123,
      },       "limitroundEnd": {"2020-08-11T16:45:10−08:00",
         "typesubmissionLimit": "integer",{
            "descriptiontotalSubmissionLimit":20,
"The maximum total number of submissions per team/participant per month."   "periodicLimits":[
   }    } }

Example Evaluation

id will be back-end generated after it has been created

{    "....currently existing fields...":{},  {
         "timeZone":"US/Pacific",        "roundsperiod":[ 1
       {          "id":"1",limit": 6
              },
          "roundStart":1231412213213123123,    {
      "roundEnd":"2020-08-11T16:45:10−08:00",           "submissionLimitperiod":{ 7
                 "totalSubmissionLimitlimit":20, 14
            "dailySubmissionLimit":2,  }
          "weeklySubmissionLimit":8,  ]
         }
"monthlySubmissionLimit":{      },
      
  "dayOfMonth":28,    
      {
    "limit":10     "id":"2",
       }  "roundStart":1231412213213123123,
       }  "roundEnd":"2020-08-11T16:45:10−08:00",
    },     "submissionLimit":{
            "totalSubmissionLimit":40,
  {          "idperiodicLimits":"2",[
            "roundStart":1231412213213123123,  {
       "roundEnd":"2020-08-11T16:45:10−08:00",          "submissionLimitperiod":{ 1
                 "totalSubmissionLimitlimit":40, 4
              "dailySubmissionLimit":8},
            "weeklySubmissionLimit":12,  {
                 "monthlySubmissionLimitperiod":{ 7
                 "dayOfMonthlimit":31, 18
              "limit":20}
            }]
         }
      }
      
      
   ]
}

EvaluationMaintenance

...

For the time being, I’m considering making the old deprecated quota field remain stored in the database, but enforce a rule such that user must choose between quota and rounds the new EvaluationRounds API when configuring a Evaluation. This allows time for clients to implement support for the new rounds fieldAPI.
Once we fully decide to remove SubmissionQuota, we can perform the conversion and store quotas asSubmissionRound.

...

• Store rounds List as JSON in a single database column.

  • This would require searching for the correct round based on start/end dates in memory

  • Evaluations object can be cleanly retrieved using query on single table

• Separate Table For Rounds (primary key is evaluationId , ID) index on roundStart and roundEnd

  • Allows us to pull out one specific limit

  • Separate queries

    • query to retrieve list of all rounds for that evaluation,

      • On REST API GET, we care about all rounds

    • query to retrieve list single evaluation id,

      • On challenge submissions during the submission limit enforcement, we only care about the specific round whose start/end interval encapsulate the current time

    • query to pull from the Evaluations table

  • Limits are enforced in Java code so we can store the them as JSON

  • Question: should ID be created by the id-generator? or just index in rounds list?

    • generated unique ID makes it easy to perform updates on evaluations

      • with current API setup a list of round are submitted, making this pointless

    • index in round list as ID

      • this allows easier tagging of submission

      • updating rounds involves row deletion and addition if added

        • hash metdata to avoid pointless delete/add?

    • If we use roundStart, roundEnd to identify the round config.

      • any change to start,roundend would mean an insertion/deletion

 {
         "id":"2",
         "roundStart":1231412213213123123,
         "roundEnd":"2020-08-11T16:45:10−08:00",
         "submissionLimit":{
            "totalSubmissionLimit":40,
            "dailySubmissionLimit":8,
            "weeklySubmissionLimit":12,
            "monthlySubmissionLimit":{
               "dayOfMonth":31,
               "limit":20
            }
         }
      }

...

Records maintenance windows for each evaluation.

...