Versions Compared

Old Version 53

changes.mady.by.user Bennett Ng

Saved on

compared with

New Version Current

changes.mady.by.user Bruce Hoff

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

NOTE: The Evaluation services were initially introduced as the Competition services

Table of Contents

Overview

...

Synapse Evaluations are constructed around three primary objects: Evaluation , Participant, and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services. The UMLet file used to generate the following image is available here.

...

  • ID - unique ID for the Evaluation
  • eTag - optimistic concurrency flag; changes with each update
  • ownerID - principal ID of the owner of the Evaluation
  • name
  • description
  • contentSource - ID of the data source (e.g. entity) for the Evaluation (string)

  • status - enum describing the state of the Evaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Participant
Immutable object to represent a single user participating in an Evaluation. Evaluations can have many users, and users can participate in many Evaluations.

  • ownerID - principal ID of the Participant
  • evaluationID
  • createdOn - join date

Submission
Immutable object to represent an entity submission to a Evaluation. A participant can create many Submissions for a given Evaluation.

...

  • submissionID
  • eTag - optimistic concurrency flag; changes with each update
  • status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID
  • score - numerical score for this Submission (double between 0 and 1)
  • report - String describing this Submission and/or its scoring. May contain additional scoring information, error logs, etc.
  • annotations - Annotations object containing user-defined key-value annotations.  Note:  The contents of the annotations are mirrored to a second set of tables optimized for querying, as discussed here.

SubmissionBundle
Bundle object containing a Submission and its accompanying SubmissionStatus.

SubmissionStatusBatch

Used to upload a collection of Submission updates at once. 

REST API

Evaluation

URLHTTP TypeDescription
/evaluationPOST

Create a new Evaluation

/evaluation/{evaluationId}GETGet a Evaluation
/evaluation/{evaluationId}PUT

Update a Evaluation

/evaluation/{evaluationId}DELETEDelete a Evaluation
/evaluationGET

Batch Get Evaluations (Paginated)

Optional request parameters:
limit (long), offset (long)

/evaluation/countGETGet the number of Evaluations
/evaluation/name/{name}GETFind a Evaluation by name

Note that the name must be URL-encoded
/evaluation/availableGET

Get the paginated list of Evaluations which the user has joined as a participant. 
Optional request parameters:
limit (long), offset (long), status (OPEN,CLOSED,...)

Participant

URLHTTP TypeDescription
/evaluation/{evaluationId}/participantPOST

Join as a Participant in a Evaluation *

/evaluation/{evaluationId}/participant/{principalId}POSTAdd another user as a Participant in a Evaluation

.

*
Requires admin rights on the Evaluation

.

/evaluation/{evaluationId}/participant/{principalId}GETGet a Participant
/evaluation/{evaluationId}/participant/{principalId}DELETEDelete a Participant
/evaluation/{evaluationId}/participant/GET

Batch get Participants for a given Evaluation(Paginated)

Optional request parameters:
limit (long), offset (long)

/evaluation/{evaluationId}/participant/countGETGet the number of Participants in a given Evaluation

...


Submission

URLHTTP TypeDescription
/evaluation/submission?etag=<etag>POST

Create a new Submission (and corresponding SubmissionStatus object).
Requires the current etag of the Entity in the Submission.

Required request parameter:
etag (string)

/evaluation/submission/{submissionId}GET

Get a Submission
Requires ownership of the Submission, or admin rights on the Evaluation.

/evaluation/submission/{submissionId}/statusGET

Get the status of a Submission

/evaluation/submission/{submissionId}/statusPUT

Update the status of a Submission.
Requires admin rights on the Evaluation

/evaluation/submission/{submissionId}DELETEDelete a Submission.
Requires admin rights on the Evaluation.

/evaluation/{evaluationId}/submission/

/evaluation/{evaluationId}/submission/bundle/

GET

Batch get my Submissions/SubmissionBundles for a given Evaluation.
Results are paginated.
Returns only Submissions owned by the requesting user.

Optional request parameters:
limit (long), offset (long)

/evaluation/{evaluationId}/submission/all

/evaluation/{evaluationId}/submission/status/all

/evaluation/{evaluationId}/submission/bundle/all

GET

Batch get Submissions/SubmissionStatuses/SubmissionBundles for a given Evaluation.
Results are paginated.
Requires admin rights on the Evaluation.
Can be filtered by SubmissionStatusEnum.

Optional request parameters:
status (String), limit (long), offset (long)

/evaluation/submission/{submissionId}/file/{fileHandleId}GET

Get a temporary pre-signed URL to a File contained within a specified Submission.
Requires admin rights on the Evaluation.

Optional request parameter:
redirect (Boolean)

/evaluation/{evaluationId}/submission/countGETGet the number of Submissions for a given Evaluation.

...