Page Comparison

...

Overview

The Competition Evaluation API is designed to support open-access Synapse data analysis and modeling challenges. This framework provides tools for administrators to create new competitions evaluations in Synapse and manage the users and data objects associated with a competition. A competition an evaluation. An evaluation owner can control access to competition evaluation resources and registration, as well as track user participation and evaluate submitted models. This API will support user-facing interfaces to monitor competitions evaluations including activity notifications and on-demand scoring and leaderboarding.

Data Models

Synapse Competitions Evaluations are constructed around three primary objects: Competition, Participant, Evaluation and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services. The UMLet file used to generate the following image is available here.Competition

Image Added

Evaluation
Object representing a Synapse CompetitionEvaluation

id ID - unique ID for the CompetitionEvaluation
eTag - optimistic concurrency flag; changes with each update
ownerID - user principal ID of the owner of the CompetitionEvaluation
name
description
contentSource - descriptor ID of the data source (e.g. entity) for the Competition Evaluation (string)
status - enum describing the state of the CompetitionEvaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Participant
Immutable object to represent a single user participating in a Competition. Competitions can have many users, and users can participate in many Competitions.

ownerId - user ID of the Participant
competitionId
createdOn - join date

Submission
Immutable object to represent an entity submission to a CompetitionEvaluation. A participant can create many Submissions for a given Competition.Evaluation.

ID - unique ID for the Submission
name
ownerID - user principal ID of the Submission owner
competitionIDevaluationID
entityID - ID of the submitted entity
version - specific entity version
createdOn - submission date
submitterAlias - the alias for the user or team creating the submission

SubmissionStatus
Object to track the status and score of a specific Submission. This object is generated at the time of submission, and can be modified by Competition Evaluation administrators.

submissionID
eTag - optimistic concurrency flag; changes with each update
status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID
score - numerical score for this Submission (long)double between 0 and 1)
report - String describing this Submission and/or its scoring. May contain additional scoring information, error logs, etc.
annotations - Annotations object containing user-defined key-value annotations. Note: The contents of the annotations are mirrored to a second set of tables optimized for querying, as discussed here.

SubmissionBundle
Bundle object containing a Submission and its accompanying SubmissionStatus.

SubmissionStatusBatch

Used to upload a collection of Submission updates at once.

REST API

CompetitionEvaluation

URL

HTTP Type

Description

/competitionevaluation

POST

Create a new

Synapse Competition

Evaluation

/competitionevaluation/{competitionIdevaluationId}

GET

Fetch Get a Synapse CompetitionEvaluation

/competitionevaluation/{competitionIdevaluationId}

PUT

Update a Synapse Competition

Participant

URL	HTTP Type	Description
/competition/{competitionId}/participant	POST	Join as a new Participant in the Competition specified by competitionId. a Evaluation
/evaluation/{evaluationId}	DELETE	Delete a Evaluation
/evaluation	GET	Batch Get Evaluations (Paginated) Optional request parameters: limit (long), offset (long)
/evaluation/count	GET	Get the number of Evaluations
/evaluation/name/{name}	GET	Find a Evaluation by name Note that the name must be URL-encoded
/evaluation/available	GET	Get the paginated list of Evaluations which the user has joined as a participant. Optional request parameters: limit (long), offset (long), status (OPEN,CLOSED,...)

Submission

...

competition a Synapse Submission

URL	HTTP Type	Description
/competitionevaluation/submission?etag=<etag>	POST	Create a new Synapse Submission/competition Submission (and corresponding SubmissionStatus object). Requires the current etag of the Entity in the Submission. Required request parameter: etag (string)
/evaluation/submission/{submissionId}	GET	Get a Submission Requires ownership of the Submission, or admin rights on the Evaluation.
/evaluation/submission/{submissionId}/status	GET	Fetch Get the status of a Synapse Submission
/	evaluation/submission/{submissionId}/status	PUT	Update

Example Workflow

the status of a Submission. Requires admin rights on the Evaluation
/evaluation/submission/{submissionId}	DELETE	Delete a Submission. Requires admin rights on the Evaluation.
/evaluation/{evaluationId}/submission/ /evaluation/{evaluationId}/submission/bundle/	GET	Batch get my Submissions/SubmissionBundles for a given Evaluation. Results are paginated. Returns only Submissions owned by the requesting user. Optional request parameters: limit (long), offset (long)
/evaluation/{evaluationId}/submission/all /evaluation/{evaluationId}/submission/status/all /evaluation/{evaluationId}/submission/bundle/all	GET	Batch get Submissions/SubmissionStatuses/SubmissionBundles for a given Evaluation. Results are paginated. Requires admin rights on the Evaluation. Can be filtered by SubmissionStatusEnum. Optional request parameters: status (String), limit (long), offset (long)
/evaluation/submission/{submissionId}/file/{fileHandleId}	GET	Get a temporary pre-signed URL to a File contained within a specified Submission. Requires admin rights on the Evaluation. Optional request parameter: redirect (Boolean)
/evaluation/{evaluationId}/submission/count	GET	Get the number of Submissions for a given Evaluation.

Access Restrictions

Access Restrictions and approvals may be applied to Evaluations. For details on the API see: Data Access Control

cURL Examples

Evaluation

Create an Evaluation

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "status":"PLANNED",
    "description":"description",
    "name":"my first evaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation

Get an Evaluation

Code Block
curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json 'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'

Update an Evaluation

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1588317",
    "createdOn":"2013-01-16T16:30:56.727Z",
    "etag":"eed22bca-d88e-4b4a-8b1a-35dca7b7b8db",
    "status":"OPEN",
    "description":"description",
    "ownerId":"1588313",
    "name":"my first evaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317

Delete an Evaluation

Code Block
curl -i -k -H sessionToken:okxkSRyl3v0iIf0eutQfXA00 -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'

Participant

Create a Participant

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "ownerId":"123456"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/987654/participant

Delete a Participant

Code Block
curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317/participant/1588313'

Submission

Create a Submission

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "evaluationId":"1588317",
    "entityId":"1588315",
    "versionNumber":"1",
    "name":"some-name"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission

Update a SubmissionStatus

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1589914",
    "modifiedOn":"2013-01-18T21:43:18.897Z",
    "etag":"13c88b1a-662c-4dd6-bb56-3976566516f7",
    "status":"SCORED",
    "score":0.42
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589914/status

Delete a Submission

Code Block
curl -i -k -H sessionToken:dPF2TqGUq121PR36AwJhXw00 -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589912'

Example Workflow

The following is a proposed workflow for interacting with the Evaluation Services. Please note that some components are still under development.

User Perspective

Find a Synapse Evaluation
Sign up as a Participant
Download relevant Synapse Entities referenced in Evaluation
Submit a Synapse entity to the Evaluation
Track status of personal Submissions and view global Evaluation status/leaderboard
Earn achievements for participation / success

Owner/Admin Perspective

Create a Synapse Evaluation
1. Fill in details, attach relevant Synapse entities
2. Specify access requirements
3. Specify open/close dates
Listen for Submissions (using messaging service)
1. Download unscored Submissions
2. Perform scoring
3. Push scores back to each SubmissionStatus
Track user participation and publish Evaluation status/leaderboard
Award achievements for user participation / success

Planned Improvements

The following features are planned but not yet implemented.

Banned user lists
Banned users should be prohibited from participating in one or more Evaluations, and any Submissions by a banned user should be isolated from the general populations of Submissions.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
serverId ba6fb084-9827-3160-8067-8ac7470f78b2
key PLFM-1646
Customizable Amazon Simple Notification Services messaging tools
Evaluation administrators should have tools to configure a selective Amazon instance to listen for messages pertaining to Evaluations which they own. These messages should be sent to a queue dedicated to Evaluations.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
serverId ba6fb084-9827-3160-8067-8ac7470f78b2
key PLFM-1647
Immutable copies of submitted Entities
Once an Entity is submitted to a Evaluation, a static copy of that Entity should become the property of the Evaluation administration.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
serverId ba6fb084-9827-3160-8067-8ac7470f78b2
key PLFM-1648
Administrator lists for Evaluations
Evaluations should support administration by multiple users (rather than just the Evaluation owner). These admin lists should function as a lightweight ACL of sorts.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
serverId ba6fb084-9827-3160-8067-8ac7470f78b2
key PLFM-1649

Versions Compared

Old Version 5

New Version Current

Key

Table of Contents

Overview

Data Models

REST API

Example Workflow

Access Restrictions

cURL Examples

Evaluation

Participant

Submission

Example Workflow

Planned Improvements