/
Evaluation API

Evaluation API

Apr 21, 2014

Overview

The Evaluation API is designed to support open-access Synapse data analysis and modeling challenges. This framework provides tools for administrators to create new evaluations in Synapse and manage the users and data objects associated with an evaluation. An evaluation owner can control access to evaluation resources and registration, as well as track user participation and evaluate submitted models. This API will support user-facing interfaces to monitor evaluations including activity notifications and on-demand scoring and leaderboarding.

Data Models

Synapse Evaluations are constructed around three primary objects: Evaluation and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services. The UMLet file used to generate the following image is available here.

Evaluation
Object representing a Synapse Evaluation

  • ID - unique ID for the Evaluation

  • eTag - optimistic concurrency flag; changes with each update

  • ownerID - principal ID of the owner of the Evaluation

  • name

  • description

  • contentSource - ID of the data source (e.g. entity) for the Evaluation (string)

  • status - enum describing the state of the Evaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Submission
Immutable object to represent an entity submission to a Evaluation. A participant can create many Submissions for a given Evaluation.

  • ID - unique ID for the Submission

  • name

  • ownerID - principal ID of the Submission owner

  • evaluationID

  • entityID - ID of the submitted entity

  • version - specific entity version

  • createdOn - submission date

  • submitterAlias - the alias for the user or team creating the submission

SubmissionStatus
Object to track the status and score of a specific Submission. This object is generated at the time of submission, and can be modified by Evaluation administrators.

  • submissionID

  • eTag - optimistic concurrency flag; changes with each update

  • status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID

  • score - numerical score for this Submission (double between 0 and 1)

  • report - String describing this Submission and/or its scoring. May contain additional scoring information, error logs, etc.

  • annotations - Annotations object containing user-defined key-value annotations.  Note:  The contents of the annotations are mirrored to a second set of tables optimized for querying, as discussed here.

SubmissionBundle
Bundle object containing a Submission and its accompanying SubmissionStatus.

SubmissionStatusBatch

Used to upload a collection of Submission updates at once. 

REST API

Evaluation

URL

HTTP Type

Description

URL

HTTP Type

Description

/evaluation

POST

Create a new Evaluation

/evaluation/{evaluationId}

GET

Get a Evaluation

/evaluation/{evaluationId}

PUT

Update a Evaluation

/evaluation/{evaluationId}

DELETE

Delete a Evaluation

/evaluation

GET

Batch Get Evaluations (Paginated)

Optional request parameters:
limit (long), offset (long)

/evaluation/count

GET

Get the number of Evaluations

/evaluation/name/{name}

GET

Find a Evaluation by name

Note that the name must be URL-encoded

/evaluation/available

GET

Get the paginated list of Evaluations which the user has joined as a participant. 
Optional request parameters:
limit (long), offset (long), status (OPEN,CLOSED,...)

 

Submission

URL

HTTP Type

Description

URL

HTTP Type

Description

/evaluation/submission?etag=<etag>

POST

Create a new Submission (and corresponding SubmissionStatus object).
Requires the current etag of the Entity in the Submission.

Required request parameter:
etag (string)

/evaluation/submission/{submissionId}

GET

Get a Submission
Requires ownership of the Submission, or admin rights on the Evaluation.

/evaluation/submission/{submissionId}/status

GET

Get the status of a Submission

/evaluation/submission/{submissionId}/status

PUT

Update the status of a Submission.
Requires admin rights on the Evaluation

/evaluation/submission/{submissionId}

DELETE

Delete a Submission.
Requires admin rights on the Evaluation.

/evaluation/{evaluationId}/submission/

/evaluation/{evaluationId}/submission/bundle/

GET

Batch get my Submissions/SubmissionBundles for a given Evaluation.
Results are paginated.
Returns only Submissions owned by the requesting user.

Optional request parameters:
limit (long), offset (long)

/evaluation/{evaluationId}/submission/all

/evaluation/{evaluationId}/submission/status/all

/evaluation/{evaluationId}/submission/bundle/all

GET

Batch get Submissions/SubmissionStatuses/SubmissionBundles for a given Evaluation.
Results are paginated.
Requires admin rights on the Evaluation.
Can be filtered by SubmissionStatusEnum.

Optional request parameters:
status (String), limit (long), offset (long)

/evaluation/submission/{submissionId}/file/{fileHandleId}

GET

Get a temporary pre-signed URL to a File contained within a specified Submission.
Requires admin rights on the Evaluation.

Optional request parameter:
redirect (Boolean)

/evaluation/{evaluationId}/submission/count

GET

Get the number of Submissions for a given Evaluation.