Submission with Teams
- Bruce Hoff
- Jay Hodgson
The basic requirements are: (1) let challenge participants declare whom they're working with, this group being visible, and (2) impose submission quotas that apply to a participant even when someone else on their team is the one creating the submission in Synapse. This presentation elaborates on the basic requirements:
Specific Requirements
- To register a Team for a Challenge one must (1) be a registered Challenge participant, AND (2) be an administrator (captain) of the Team.
- Provide a service which accepts a user ID and returns a list of teams on whose behalf the user may submit, taking into account any restrictions.
- Send submission receipt notification to all contributors.
- Add an administrative service to add a contributor to a Submission. This allows someone who neglected to sign-up for the challenge but who contributed to a Submission to be given credit by Challenge Administration.
- Defer 'optOut' feature (allowing a user to 'drop off' a submission created by a teammate)
- When retrieving a TeamState, a hash of the object should be returned. The client submits the hash with the Submission and the server throws an exception if the hash is out of date. This ensures that the Team State at the time of submission is the same as that which the user saw.
- Need a service to register a Team based on an Evaluation ID (in contrast to registering based on a Challenge ID).
Submission restriction rule:
Each Submission has a submitter, an optional Team ID and a list of contributors (other user IDs). A submission is accepted if either:
A. It is a valid individual Submission, i.e. (1) it has no Team ID, (2) the submitter hasn't exceeded their per-round submission limit and (2) the submitter is not the submitter of or contributor to a Team Submission (see below) in the current round; OR
B. It is a valid Team Submission, i.e. (1) it has a Team ID of a registered Team, (2) the Team hasn't exceeded its per-round submission limit, (3) all contributors are registered for the challenge and are on the specified Team at the time of submission, and (4) neither the submitter nor any contributor has made an individual Submission or was the submitter of or contributor to a Team Submission having a different Team ID in the current round.
Design Approach
1) A Challenge is a Project with some extra metadata. Implement this as a new object which a Project can 'own', and on which CRUD operations can be done independently from the Project.
2) Introduce new Team-Challenge association to let participants associate their Team with a Challenge.
3) Explicitly limit the number of Submissions per 'round' ('phase'?) of a Challenge on a per-user basis. A Submission has a list of associated _users_ which is 'initialized' in the UI (prior to being submitted) to be the membership of a Team MINUS unregistered Team members MINUS Team members who exceeded their submission limit. The client submits a _finalized_ participant list as part of the Submission.
4) Users are notified of submissions with which they are associated and may opt out. (This prevents another user from consuming one's submission quota). Opting out must occur prior to scoring.
User Experience
Challenge administration:
- make a new Project
- create Participant Team
- create Challenge, specifying the participant Team
- make Evaluation queues, setting start/end of rounds, max submissions per participant per round, optOutDisallowed states.
Currently these operations are done from the command line. They could be scripted for convenience.
User registration:
User submission:
All check boxes will not be editable. For an unchecked-disabled Team member, one example reason may be: "This member has exceeded their submission quota. You may ask them to 'opt out' of pending submission(s) by other Team(s) they are on, then retry this submission."
Notes on page 2 (team selection):
- All teams that the current user has admin rights for are shown. On submit, the portal will also attempt to register the selected team to the associated challenge (if it is associated to a challenge).
- There will also be a link to "Create a new Team" which sends the user to the teams section of their profile page.
Opt out:
- user receives submission notification email:
Dear <username>:
We have received a submission to the <challenge name> challenge, created by <other user name>, which includes you as a contributor. This counts towards your quota of <n> submissions for this challenge round. If you feel this is incorrect you may "opt out" of the submission by following this link: http://www.synapse.org:#!Submissions:12345. Otherwise no action is required.
Sincerely,
Synapse Administration
- user following link in email to Portal.
- Portal list submissions for which user may opt out.
Objects
Entity Relationship Diagram
Note: For simplicity we leave out existing fields of legacy objects, Project, Team and Evaluation, and Submission.
Services
| URI | Method | Request Parameters | Request Body | Response Body | Authorization | Notes |
|---|---|---|---|---|---|---|
| /challenge | POST | -- | Challenge | Challenge | CREATE permission required in referenced project. | |
| /entity/{projectId}/challenge | GET | -- | -- | Challenge | READ permission required in project. | |
| /challenge/{challengeId} | GET | -- | -- | Challenge | READ permission required in project. | |
| /challenge | GET | participantId, limit, offset | -- | ChallengePagedResults | PUBLIC, but to be in the returned list the caller must have READ permission on the project 'owning' the Challenge. | returns the projects/challenges for which 'participantId' is registered. |
| /challenge/{challengeId} | PUT | -- | Challenge | -- | UPDATE permission required in referenced project. | |
| /challenge/{challengeId} | DELETE | -- | -- | -- | DELETE permission required in referenced project. | |
| /challenge/{challengeId}/participant | GET | limit, offset, affiliated | PaginatedIds | READ permission required in project. | Return challenge participants. If affiliated=true, return just participants affiliated with some registered Team. If false, return those affiliated with no Team. If missing return all participants. | |
| /challenge/{challengeId}/challengeTeam | POST | -- | ChallengeTeam | ChallengeTeam | Must be a member of the Participant Team and be an admin on the referenced Challenge Team. | |
| /challenge/{challengeId}/challengeTeam/{challTeamId} | GET | -- | -- | ChallengeTeam | Must have READ permission in the Challenge Project. | We defer implementation of this since there is no immediate use case. |
| /challenge/{challengeId}/challengeTeam | GET | limit, offset | -- | ChallengeTeamPagedResults | Must have READ permission in the Challenge Project. | |
| /challenge/{challengeId}/registratableTeam | GET | limit, offset | -- | PaginatedIds | Must have READ permission in the Challenge Project. | Returns the Teams which are NOT registered for the challenge and on which is current user is an ADMIN. |
| /challenge/{challengeId}/challengeTeam/{challTeamId} | PUT | -- | ChallengeTeam | ChallengeTeam | Must be a member of the Participant Team and be an admin on the referenced Challenge Team. | |
| /challengeTeam/{challTeamId} | DELETE | -- | -- | -- | Must be an admin on the referenced Challenge Team. | |
| /teamList | POST | IdSet | ListWrapper<Team> | PUBLIC | Returns a list of Teams for a given list of Team IDs. | |
/team/{id}/memberList | POST | IdSet | ListWrapper<TeamMember> | PUBLIC | Returns the Team Membership info for a given list of memberIds. | |
| /user/{id}/memberList | POST | IdSet | ListWrapper<TeamMember> | PUBLIC | Returns the user's Membership info for a given list of Teams. | |
| /challenge/{challengeId}/submissionTeams | GET | principalId, limit, offset | -- | PaginatedIds | Must be the user specified by 'principalId' or be a Synapse Admin | Returns a list of Teams either on whose behalf the user is eligible to submit, i.e. the Teams registered for the challenge and in which the user is a member. |
| /evaluation/submission | POST | etag, submissionEligibilityHash | Submission | Submission | Must have SUBMIT permission in the Evaluation referenced in the Submission. | If 'contributer' list is not empty then teamId is required and Submission is rejected if submitter or any specified contributers is not on the given team. If submissionEligibilityHash is included then teamId must also be included and submission will be rejected if computed TeamSubmissionEligibility does not match hash. Notifications sent to all contributers except for submitter. |
| /evaluation/submission/{submissionId}/contributor/{principalId} | POST | -- | -- | Administrative Service | Adds the given user as a contributor to the Submission. | |
| /evaluation/submission/{submissionId}/optOut | PUT | userId | -- | -- | userId must be the one making the request (or be a Synapse Administrator) | Sets 'optOut' field for given userId in the Submission. userId must be a contributer in the Submission (not the submitter himself) and Submission must not be in one of the Evaluation's 'optOutDisallowed' states. |
| /evaluation/{evaluationId}/team/{id}/submissionEligibility | GET | -- | -- | TeamSubmissionEligibility | user making the request must have SUBMIT permission in the Evaluation and must be a member of the specified Team. |