Page Comparison

NOTE: The Evaluation services were initially introduced as the Competition services are still under development and are not yet available on Synapse.

Overview

The Competition Evaluation API is designed to support open-access Synapse data analysis and modeling challenges. This framework provides tools for administrators to create new competitions evaluations in Synapse and manage the users and data objects associated with a competition. A competition an evaluation. An evaluation owner can control access to competition evaluation resources and registration, as well as track user participation and evaluate submitted models. This API will support user-facing interfaces to monitor competitions evaluations including activity notifications and on-demand scoring and leaderboarding.

Data Models

Synapse Competitions Evaluations are constructed around three primary objects: CompetitionEvaluation, Participant, and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services.

CompetitionEvaluation
Object representing a Synapse CompetitionEvaluation

ID - unique ID for the CompetitionEvaluation
eTag - optimistic concurrency flag; changes with each update
ownerID - principal ID of the owner of the CompetitionEvaluation
name
description
contentSource - ID of the data source (e.g. entity) for the Competition Evaluation (string)
status - enum describing the state of the CompetitionEvaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Participant
Immutable object to represent a single user participating in a Competitionan Evaluation. Competitions Evaluations can have many users, and users can participate in many CompetitionsEvaluations.

ownerID - principal ID of the Participant
competitionIDevaluationID
createdOn - join date

Submission
Immutable object to represent an entity submission to a CompetitionEvaluation. A participant can create many Submissions for a given CompetitionEvaluation.

ID - unique ID for the Submission
name
ownerID - principal ID of the Submission owner
competitionIDevaluationID
entityID - ID of the submitted entity
version - specific entity version
createdOn - submission date

SubmissionStatus
Object to track the status and score of a specific Submission. This object is generated at the time of submission, and can be modified by Competition Evaluation administrators.

submissionID
eTag - optimistic concurrency flag; changes with each update
status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID
score - numerical score for this Submission (double between 0 and 1)

REST API

CompetitionEvaluation

URL	HTTP Type	Description
/competitionevaluation	POST	Create a new CompetitionEvaluation
/competitionevaluation/{competitionIdevaluationId}	GET	Get a CompetitionEvaluation
/competitionevaluation/{competitionIdevaluationId}	PUT	Update a CompetitionEvaluation
/competitionevaluation/{competitionIdevaluationId}	DELETE	Delete a CompetitionEvaluation
/competitionevaluation	GET	Batch Get Competitions Evaluations (Paginated) Optional request parameters: limit (long), offset (long)
/competitionevaluation/count	GET	Get the number of CompetitionsEvaluations
/competitionevaluation/name/{name}	GET	Find a Competition Evaluation by name Note that the name must be URL-encoded

Participant

URL	HTTP Type	Description
/competitionevaluation/{competitionIdevaluationId}/participant	POST	Join as a Participant in a Competition Evaluation *
/competitionevaluation/{competitionIdevaluationId}/participant/{principalId}	POST	Add another user as a Participant in a CompetitionEvaluation. * Requires admin rights on the CompetitionEvaluation.
/competitionevaluation/{competitionIdevaluationId}/participant/{principalId}	GET	Get a Participant
/competitionevaluation/{competitionIdevaluationId}/participant/{principalId}	DELETE	Delete a Participant
/competitionevaluation/{competitionIdevaluationId}/participant/	GET	Batch get Participants for a given Competition Evaluation(Paginated) Optional request parameters: limit (long), offset (long)
/competitionevaluation/{competitionIdevaluationId}/participant/count	GET	Get the number of Participants in a given CompetitionEvaluation

*A Synapse user can only join as a Participant in a Competition an Evaluation if the Competition Evaluation is OPEN. However, the Competition Evaluation owner can add other users as Participants at any time, regardless of the Competition Evaluation state.

Submission

URL	HTTP Type	Description
/competitionevaluation/submission	POST	Create a new Submission. (and corresponding SubmissionStatus object)
/competitionevaluation/submission/{submissionId}	GET	Get a Submission
/competitionevaluation/submission/{submissionId}/status	GET	Get the status of a Submission
/competitionevaluation/submission/{submissionId}/status	PUT	Update the status of a Submission. Requires admin rights on the CompetitionEvaluation
/competitionevaluation/submission/{submissionId}	DELETE	Delete a Submission. Requires admin rights on the CompetitionEvaluation.
/competitionevaluation/{compIdevaluationId}/submission/	GET	Batch get my Submissions for a given Competition Evaluation (Paginated). Returns only Submissions owned by the requesting user. Optional request parameters: limit (long), offset (long)
/competitionevaluation/{compIdevaluationId}/submission/all	GET	Batch get Submissions for a given Competition Evaluation (Paginated). Requires admin rights on the CompetitionEvaluation. Can be filtered by SubmissionStatusEnum. Optional request parameters: status (String), limit (long), offset (long)
/competitionevaluation/{compIdevaluationId}/submission/count	GET	Get the number of Submissions for a given CompetitionEvaluation

cURL Examples

Competition

Evaluation

Create a Competitionan Evaluation

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "status":"PLANNED",
    "description":"description",
    "name":"my first competitionevaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation

Get a Competitionan Evaluation

Code Block
curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json 'https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/1588317'

Update a Competitionan Evaluation

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1588317",
    "createdOn":"2013-01-16T16:30:56.727Z",
    "etag":"eed22bca-d88e-4b4a-8b1a-35dca7b7b8db",
    "status":"OPEN",
    "description":"description",
    "ownerId":"1588313",
    "name":"my first competitionevaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/1588317

Delete a Competitionan Evaluation

Code Block
curl -i -k -H sessionToken:okxkSRyl3v0iIf0eutQfXA00 -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/1588317'

Participant

Create a Participant

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "ownerId":"123456"
}' https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/987654/participant

Delete a Participant

Code Block
curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/1588317/participant/1588313'

Submission

Create a Submission

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "competitionIdevaluationId":"1588317",
    "entityId":"1588315",
    "versionNumber":"1",
    "name":"some-name"
}' https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/submission

Update a SubmissionStatus

Code Block

curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1589914",
    "modifiedOn":"2013-01-18T21:43:18.897Z",
    "etag":"13c88b1a-662c-4dd6-bb56-3976566516f7",
    "status":"SCORED",
    "score":0.42
}' https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/submission/1589914/status

Delete a Submission

Code Block
curl -i -k -H sessionToken:dPF2TqGUq121PR36AwJhXw00 -H Accept:application/json -X DELETE 'https://repo-staging.prod.sagebase.org/repo/v1/competitionevaluation/submission/1589912'

Example Workflow

The following is a proposed workflow for interacting with the Competition Evaluation Services. Please note that some components are still under development.

User Perspective

Find a Synapse CompetitionEvaluation
Sign up as a Participant
Download relevant Synapse Entities referenced in CompetitionEvaluation
Submit a Synapse entity to the CompetitionEvaluation
Track status of personal Submissions and view global competition Evaluation status/leaderboard
Earn achievements for participation / success

Owner/Admin Perspective

Create a Synapse CompetitionEvaluation
1. Fill in details, attach relevant Synapse entities
2. Specify access requirements
3. Specify open/close dates
Listen for Submissions (using messaging service)
1. Download unscored Submissions
2. Perform scoring
3. Push scores back to each SubmissionStatus
Track user participation and publish competition Evaluation status/leaderboard
Award achievements for user participation / success

Planned Improvements

The following features are planned but not yet implemented.

Banned user lists
Banned users should be prohibited from participating in one or more CompetitionsEvaluations, and any Submissions by a banned user should be isolated from the general populations of Submissions.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1646
Customizable Amazon Simple Notification Services messaging tools
Competition Evaluation administrators should have tools to configure a selective Amazon instance to listen for messages pertaining to Competitions Evaluations which they own. These messages should be sent to a queue dedicated to CompetitionsEvaluations.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1647
Immutable copies of submitted Entities
Once an Entity is submitted to a CompetitionEvaluation, a static copy of that Entity should become the property of the Competition Evaluation administration.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1648

Administrator pAdministrator lists for CompetitionsEvaluations
Competitions Evaluations should support administration by multiple users (rather than just the Competition Evaluation owner). These admin lists should function as a lightweight ACL of sorts.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1649
Multi-author Submissions
Competitions should support team Submissions, where more than one Synapse user claims authorship of a given Submission.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1650
Pagination support for fetching Participants and Submissions
Currently the API supports fetching all Participants or all Submissions for a given Competition. To support high-volume Competitions, pagination shoudl be supported on these requests.
Jira Legacy
server JIRA (sagebionetworks.jira.com)
key PLFM-1651

Versions Compared

Old Version 35

New Version 36

Key

Table of Contents

Overview

Data Models

REST API

cURL Examples

Competition

Evaluation

Participant

Submission

Example Workflow

Planned Improvements