...
For above proposal to work we need to introduce new APIs for the evaluations so that the new type of annotations supported by the replication index can be added to submissions using the new annotation format and at the same time provide a way to create views for the submissions so that current the table/view query system can be replacedused as a replacement of the current submission query.
Evaluation API additions
...
Endpoint
...
Response Type
...
Description
...
PUT /evaluation/submission/{id}/annotations
...
...
Similar to the PUT /entity/{id}/annotations2, allows to add/update annotations on the submissions with the given id. The access scheme will reflect the current submission status annotations, so the caller must be granted ACCESS_TYPE.READ and ACCESS_TYPE.UPDATE_SUBMISSION on the evaluation the submission belongs to.
...
GET /evaluation/submission/{id}/annotations
...
...
Similar to the GET /entity/{id}/annotations2, allows to retrieve the annotations for the given submission. The access scheme will reflect the current submission status annotations, so the caller must be granted ACCESS_TYPE.READ and ACCESS_TYPE.READ_PRIVATE_SUBMISSION on the evaluation the submission belongs to.
Additionally the SubmissionBundle will also include a new property “annotations” of type AnnotationsThe Evaluations Service currently provide a way to annotate the submissions (which are immutable) through a mutable object, SubmissionStatus. This object contains a property named annotations that uses the old format supported by the submission query engine.
We can reuse the same exact APIs and add a new attribute to the SubmissionStatus named submissionAnnotations that can transport the annotations in the new format instead. The previous annotations attribute will be deprecated (but still supported) and not used to index the annotations in the replication index.
This has the advantage of not having to reimplement the logic around the ACL for submissions.
Entity and Table APIs additions
In order to support querying the annotated submissions we need to introduce a new type of view, similar to an EntityView. We propose to be explicit and create a EvaluationSubmissionView (That could extend the Table object or a more generic ObjectView in which case the OBJECT_TYPE should be included) SubmissionView that simply specifies the scope:
scopeIds: The list of evaluations ids that define the scope of the submissions to query
The EntityView currently allows to define a typeMask to filter the object sub-types. Views on submissions are not hierarchical and do not have different subtypes so we do not have the need for the type mask for now, we introduce a new super interface named View that both EntityView and SubmissionView implement. Since all the view types need to define a scope the View interface will embed the scopeIds, while the EntityView will have the current attribute for the type mask. This allows for the API to be backward compatible but more extensible.
For a SubmissionView the creator must be granted ACCESS_TYPE.READ_PRIVATE_SUBMISSION on each of the evaluations in the scope. Note that this is actually more strict that the current submission querying system, in fact today in order to query the submissions of an evaluation only ACCESS_TYPE.READ access is needed to query the submissions.
Note that the The system attributes of a submission that are currently indexed in the submission query system should also be available in the view (e.g. status, userId, teamId, submitterId, name, createdOn, submitterAlias etc). The way how those are indexed will work similarly to how the entity system attributes are indexed and exposed.
Note: The canCancel, cancelRequested and cancelControl will NOT be included in the first release. This is because they are currently serialized into the database as a blob values and they need to be normalized into their own columns in order to be efficiently retrieved when indexed.
Once a view is exposed it can be shared using the standard ACL controls on the view (which is an entity) so that it can be queried (e.g. in a leaderboard) by other users. This allows more flexibility than the current model which includes in the annotations an isPrivate boolean to “hide” non-public annotations. The user creating the view could create multiple views exposing different columns for different purposes.
The rest of the table APIs should largely remain the same with minor additions to accommodate the new view type, in particular:
Two new ColumnTypes: SUBMISSIONID and EVALUATIONID, so that they can be rendered in the view to point to the relative objects
A new EntityType: submissionview for the new type
...
of view
A new viewEntityType parameter in the https://rest-docs.synapse.org/rest/GET/column/tableview/defaults.html: This is needed since at the moment the only information that is passed is the viewTypeMask which is not enough to discriminate between different view types. By default it will be set to entityview to maintain backward compatibility. If the submissionview is used then the viewTypeMask will not be taken into consideration (as submission views do not have subtypes).
Similarly a new viewEntityType attribute will be added in the ViewScope object used in the https://rest-docs.synapse.org/rest/POST/column/view/scope.html API to discriminate between different view types.
The viewEntityType parameter/attribute will be an enumeration which is a subset of the EntityType enumeration and that will contain only the entity types that define a view: entityview and submissionview.
Submissions Schemas
In the future with the introduction of schemas for objects we could introduce binding of a schema to the annotations on submissions. In particular we could bind a schema to an evaluation so that the submissions annotations would abide by the rules set on the schema. At the same time creating a view on submissions could be guided by the same schema.