Evaluation API

Overview

The Evaluation API is designed to support open-access Synapse data analysis and modeling challenges. This framework provides tools for administrators to create new evaluations in Synapse and manage the users and data objects associated with an evaluation. An evaluation owner can control access to evaluation resources and registration, as well as track user participation and evaluate submitted models. This API will support user-facing interfaces to monitor evaluations including activity notifications and on-demand scoring and leaderboarding.

Data Models

Synapse Evaluations are constructed around three primary objects: Evaluation and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services. The UMLet file used to generate the following image is available here.

Evaluation
Object representing a Synapse Evaluation

• ID - unique ID for the Evaluation
• eTag - optimistic concurrency flag; changes with each update
• ownerID - principal ID of the owner of the Evaluation
• name
• description
• contentSource - ID of the data source (e.g. entity) for the Evaluation (string)

• status - enum describing the state of the Evaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Submission
Immutable object to represent an entity submission to a Evaluation. A participant can create many Submissions for a given Evaluation.

• ID - unique ID for the Submission
• name
• ownerID - principal ID of the Submission owner
• evaluationID
• entityID - ID of the submitted entity
• version - specific entity version
• createdOn - submission date
• submitterAlias - the alias for the user or team creating the submission

SubmissionStatus
Object to track the status and score of a specific Submission. This object is generated at the time of submission, and can be modified by Evaluation administrators.

• submissionID
• eTag - optimistic concurrency flag; changes with each update
• status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID
• score - numerical score for this Submission (double between 0 and 1)
• report - String describing this Submission and/or its scoring. May contain additional scoring information, error logs, etc.
• annotations - Annotations object containing user-defined key-value annotations.  Note:  The contents of the annotations are mirrored to a second set of tables optimized for querying, as discussed here.

SubmissionBundle
Bundle object containing a Submission and its accompanying SubmissionStatus.

SubmissionStatusBatch

Used to upload a collection of Submission updates at once. 

REST API

Evaluation

Submission

Access Restrictions

Access Restrictions and approvals may be applied to Evaluations.  For details on the API see:  Data Access Control

cURL Examples

Evaluation

• Create an Evaluation

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
      "status":"PLANNED",
      "description":"description",
      "name":"my first evaluation",
      "contentSource":"contentSource"
  }' https://repo-staging.prod.sagebase.org/repo/v1/evaluation

• Get an Evaluation

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json 
  'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'

• Update an Evaluation

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
      "id":"1588317",
      "createdOn":"2013-01-16T16:30:56.727Z",
      "etag":"eed22bca-d88e-4b4a-8b1a-35dca7b7b8db",
      "status":"OPEN",
      "description":"description",
      "ownerId":"1588313",
      "name":"my first evaluation",
      "contentSource":"contentSource"
  }' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317

• Delete an Evaluation

  curl -i -k -H sessionToken:okxkSRyl3v0iIf0eutQfXA00 -H Accept:application/json -X DELETE 
  'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'

Participant

• Create a Participant

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
      "ownerId":"123456"
  }' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/987654/participant

• Delete a Participant

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -X DELETE 
  'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317/participant/1588313'

Submission

• Create a Submission

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
      "evaluationId":"1588317",
      "entityId":"1588315",
      "versionNumber":"1",
      "name":"some-name"
  }' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission

• Update a SubmissionStatus

  curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
      "id":"1589914",
      "modifiedOn":"2013-01-18T21:43:18.897Z",
      "etag":"13c88b1a-662c-4dd6-bb56-3976566516f7",
      "status":"SCORED",
      "score":0.42
  }' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589914/status

  
  
  

• Delete a Submission

  curl -i -k -H sessionToken:dPF2TqGUq121PR36AwJhXw00 -H Accept:application/json -X DELETE 
  'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589912'

Example Workflow

The following is a proposed workflow for interacting with the Evaluation Services. Please note that some components are still under development.

User Perspective

1. Find a Synapse Evaluation
2. Sign up as a Participant
3. Download relevant Synapse Entities referenced in Evaluation
4. Submit a Synapse entity to the Evaluation
5. Track status of personal Submissions and view global Evaluation status/leaderboard
6. Earn achievements for participation / success

Owner/Admin Perspective

1. Create a Synapse Evaluation
   
   1. Fill in details, attach relevant Synapse entities
   2. Specify access requirements
   3. Specify open/close dates
2. Listen for Submissions (using messaging service)
   
   1. Download unscored Submissions
   2. Perform scoring
   3. Push scores back to each SubmissionStatus
3. Track user participation and publish Evaluation status/leaderboard
4. Award achievements for user participation / success

Planned Improvements

The following features are planned but not yet implemented.

• Banned user lists
  Banned users should be prohibited from participating in one or more Evaluations, and any Submissions by a banned user should be isolated from the general populations of Submissions.
  PLFM-1646 - Getting issue details... STATUS
  
  
• Customizable Amazon Simple Notification Services messaging tools
  Evaluation administrators should have tools to configure a selective Amazon instance to listen for messages pertaining to Evaluations which they own. These messages should be sent to a queue dedicated to Evaluations.
  PLFM-1647 - Getting issue details... STATUS
  
  
• Immutable copies of submitted Entities
  Once an Entity is submitted to a Evaluation, a static copy of that Entity should become the property of the Evaluation administration.
  PLFM-1648 - Getting issue details... STATUS
  
  
• Administrator lists for Evaluations
  Evaluations should support administration by multiple users (rather than just the Evaluation owner). These admin lists should function as a lightweight ACL of sorts.
  PLFM-1649 - Getting issue details... STATUS