Versions Compared

Old Version 11

changes.mady.by.user Bennett Ng

Saved on

compared with

New Version Current

changes.mady.by.user Bruce Hoff

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

NOTE: The Competition services are still under development and are not yet available on Synapse.

Table of Contents

Overview

The Competition Evaluation API is designed to support open-access Synapse data analysis and modeling challenges. This framework provides tools for administrators to create new competitions evaluations in Synapse and manage the users and data objects associated with a competition. A competition an evaluation. An evaluation owner can control access to competition evaluation resources and registration, as well as track user participation and evaluate submitted models. This API will support user-facing interfaces to monitor competitions evaluations including activity notifications and on-demand scoring and leaderboarding.

Data Models

Synapse Competitions Evaluations are constructed around three primary objects: Competition, Participant, Evaluation and Submission. These objects are defined by eponymous JSON schema files in the Synapse Repository Services. CompetitionThe UMLet file used to generate the following image is available here.

Image Added

Evaluation
Object representing a Synapse CompetitionEvaluation

  • ID - unique ID for the CompetitionEvaluation
  • eTag - optimistic concurrency flag; changes with each update
  • ownerID - user principal ID of the owner of the CompetitionEvaluation
  • name
  • description
  • contentSource - ID of the data source (e.g. entity) for the Competition Evaluation (string)

  • status - enum describing the state of the CompetitionEvaluation, one of: PLANNED, OPEN, CLOSED, COMPLETED

Participant
Immutable object to represent a single user participating in a Competition. Competitions can have many users, and users can participate in many Competitions.

  • ownerID - user ID of the Participant
  • competitionID
  • createdOn - join date

Submission
Immutable object to represent an entity submission to a CompetitionEvaluation. A participant can create many Submissions for a given CompetitionEvaluation.

  • ID - unique ID for the Submission
  • name
  • ownerID - user principal ID of the Submission owner
  • competitionIDevaluationID
  • entityID - ID of the submitted entity
  • version - specific entity version
  • createdOn - submission date
  • submitterAlias - the alias for the user or team creating the submission

SubmissionStatus
Object to track the status and score of a specific Submission. This object is generated at the time of submission, and can be modified by Competition Evaluation administrators.

  • submissionID
  • eTag - optimistic concurrency flag; changes with each update
  • status - enum describing the state of the Submission, one of: OPEN, CLOSED, SCORED, INVALID
  • score - numerical score for this Submission (long)

REST API

...

  • double between 0 and 1)
  • report - String describing this Submission and/or its scoring. May contain additional scoring information, error logs, etc.
  • annotations - Annotations object containing user-defined key-value annotations.  Note:  The contents of the annotations are mirrored to a second set of tables optimized for querying, as discussed here.

SubmissionBundle
Bundle object containing a Submission and its accompanying SubmissionStatus.

SubmissionStatusBatch

Used to upload a collection of Submission updates at once. 

REST API

Evaluation

URLHTTP TypeDescription
/competitionevaluationPOST

Create a new

Competition

Evaluation

/competitionevaluation/{competitionIdevaluationId}GETFetch Get a CompetitionEvaluation
/competitionevaluation/{competitionIdevaluationId}PUT

Update a CompetitionEvaluation

/competitionevaluation/{competitionIdevaluationId}DELETEDelete a Competition

Participant

Join as a new Participant in the Competition specified by competitionId
URLHTTP TypeDescription
/competition/{competitionId}/participantPOSTEvaluation
/evaluationGET

Batch Get Evaluations (Paginated)

Optional request parameters:
limit (long), offset (long)

/evaluation/countGETGet the number of Evaluations
/evaluation/name/{name}GETFind a Evaluation by name

Note that the name must be URL-encoded
/evaluation/availableGET

Get the paginated list of Evaluations which the user has joined as a participant. 
Optional request parameters:
limit (long), offset (long), status (OPEN,CLOSED,...)


Submission

URLHTTP TypeDescription
/competitionevaluation/submission?etag=<etag>POST

Create a new Submission (and

a

corresponding SubmissionStatus object).

/competition

Requires the current etag of the Entity in the Submission.

Required request parameter:
etag (string)

/evaluation/submission/{submissionId}GETFetch

Get a Submission

/competition

Requires ownership of the Submission, or admin rights on the Evaluation.

/evaluation/submission/{submissionId}/statusGET

Get the status of a Submission

/competitionevaluation/submission/{submissionId}/statusPUT

Update the status of a Submission.
Requires admin rights on the Evaluation

/evaluation/submission/{submissionId}DELETEDelete a Submission.
Requires admin rights on the Evaluation.

/evaluation/{evaluationId}/submission/

/evaluation/{evaluationId}/submission/bundle/

GET

Batch get my Submissions/SubmissionBundles for a given Evaluation.
Results are paginated.
Returns only Submissions owned by the requesting user.

Optional request parameters:
limit (long), offset (long)

/evaluation/{evaluationId}/submission/all

/evaluation/{evaluationId}/submission/status/all

/evaluation/{evaluationId}/submission/bundle/all

GET

Batch get Submissions/SubmissionStatuses/SubmissionBundles for a given Evaluation.
Results are paginated.
Requires admin rights on the Evaluation.
Can be filtered by SubmissionStatusEnum.

Optional request parameters:
status (String), limit (long), offset (long)

/evaluation/submission/{submissionId}/file/{fileHandleId}GET

Get a temporary pre-signed URL to a File contained within a specified Submission.
Requires admin rights on the Evaluation.

Optional request parameter:
redirect (Boolean)

/evaluation/{evaluationId}/submission/countGETGet the number of Submissions for a given Evaluation.

Access Restrictions

Access Restrictions and approvals may be applied to Evaluations.  For details on the API see:  Data Access Control

cURL Examples

Evaluation

  • Create an Evaluation

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "status":"PLANNED",
    "description":"description",
    "name":"my first evaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation

  • Get an Evaluation

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json 
'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'

  • Update an Evaluation

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1588317",
    "createdOn":"2013-01-16T16:30:56.727Z",
    "etag":"eed22bca-d88e-4b4a-8b1a-35dca7b7b8db",
    "status":"OPEN",
    "description":"description",
    "ownerId":"1588313",
    "name":"my first evaluation",
    "contentSource":"contentSource"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317

  • Delete an Evaluation

    Code Block
    curl -i -k -H sessionToken:okxkSRyl3v0iIf0eutQfXA00 -H Accept:application/json -X DELETE 
'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317'
Participant

  • Create a Participant

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "ownerId":"123456"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/987654/participant

 

  • Delete a Participant

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -X DELETE 
'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/1588317/participant/1588313'
Submission

  • Create a Submission

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -d '{
    "evaluationId":"1588317",
    "entityId":"1588315",
    "versionNumber":"1",
    "name":"some-name"
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission

  • Update a SubmissionStatus

    Code Block
    curl -i -k -H sessionToken:xxxxxxxxxxxxxxxx -H Accept:application/json -H Content-Type:application/json -X PUT -d '{
    "id":"1589914",
    "modifiedOn":"2013-01-18T21:43:18.897Z",
    "etag":"13c88b1a-662c-4dd6-bb56-3976566516f7",
    "status":"SCORED",
    "score":0.42
}' https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589914/status



  • Delete a Submission

    Code Block
    curl -i -k -H sessionToken:dPF2TqGUq121PR36AwJhXw00 -H Accept:application/json -X DELETE 
'https://repo-staging.prod.sagebase.org/repo/v1/evaluation/submission/1589912'

Example Workflow

The following is a proposed workflow for interacting with the Competition Evaluation Services. Please note that some components are still under development.

User Perspective

  1. Find a Synapse CompetitionEvaluation
  2. Sign up as a Participant
  3. Download relevant Synapse Entities referenced in CompetitionEvaluation
  4. Submit a Synapse entity to the CompetitionEvaluation
  5. Track status of personal Submissions and view global competition Evaluation status/leaderboard
  6. Earn achievements for participation / success

Owner/Admin Perspective

  1. Create a Synapse CompetitionEvaluation
    1. Fill in details, attach relevant Synapse entities
    2. Specify access requirements
    3. Specify open/close dates
  2. Listen for Submissions (using messaging service)
    1. Download unscored Submissions
    2. Perform scoring
    3. Push scores back to each SubmissionStatus
  3. Track user participation and publish competition Evaluation status/leaderboard
  4. Award achievements for user participation / success

Planned

...

Improvements

The following features are planned but not yet implemented.

  • Banned user lists
    Banned users should be prohibited from participating in one or more CompetitionsEvaluations, and any Submissions by a banned user should be isolated from the general populations of Submissions.
    Jira Legacy
    serverJIRA (sagebionetworks.jira.com)
    serverIdba6fb084-9827-3160-8067-8ac7470f78b2
    keyPLFM-1646


  • Customizable Amazon Simple Notification Services messaging tools
    Competition Evaluation administrators should have tools to configure a selective Amazon instance to listen for messages pertaining to Competitions Evaluations which they own. These messages should be sent to a queue dedicated to Competitions.Evaluations.
    Jira Legacy
    serverJIRA (sagebionetworks.jira.com)
    serverIdba6fb084-9827-3160-8067-8ac7470f78b2
    keyPLFM-1647


  • Immutable copies of the submitted Entities
    Once an Entity is submitted to a CompetitionEvaluation, a static copy of that Entity should become the property of the Competition Evaluation administration.
    Jira Legacy
    serverJIRA (sagebionetworks.jira.com)
    serverIdba6fb084-9827-3160-8067-8ac7470f78b2
    keyPLFM-1648


  • Administrator lists for CompetitionsEvaluations
    Competitions Evaluations should support administration by multiple users (rather than just the Competition Evaluation owner). These admin lists should function as a lightweight ACL of sorts.
    Multi-author Submissions
    Competitions should support team Submissions, where more than one Synapse user claims authorship of a given Submission.
    Jira Legacy
    serverJIRA (sagebionetworks.jira.com)
    serverIdba6fb084-9827-3160-8067-8ac7470f78b2
    keyPLFM-1649