GET /evaluation/submission/query doesn't return results with user defined annotation


I mentioned this in the post scrum on Friday 4/9

A query using user defined annotations against the deprecated GET /evaluation/submission/query API doesn't seem to return the expected submission (the calling has READ_PRIVATE_SUBMISSION on the evaluation).

i.e. if a submission exists on evaluation 123 with entity id syn456 and annotation foo=1, the following query does eventually return the expected result if polled,

however the following query does not ever return any results:

The python client integration tests have a test related to evaluation submissions that has become recently
unstable against the dev stack seemingly due to some timing issues and some badly written polling code in the test. The part of the test that is failing quite old (7 years) and involves making a submission to an evaluation,
applying some annotations to the submission, and polling the above API. The above API does not return the expected result although this isn't the proximate cause of the instability (rather it seems just a timing change)

In fixing this test I'm removing the call to the above deprecated API but per our post scrum conversation
am filing this issue in case we want to review the above functionality.

The following is a more complete bit of Python code to reproduce this:




Bruce Hoff
6 days ago

Ahh, now I see: We have changed the SubmissionStatus object. It previously had an ‘annotations’ member and now has a ‘submissionAnnotations’ member. The former (noted as deprecated) is the member that gets indexed into the deprecated query indexed. The latter is the member that gets indexed into the Submission View. Clearly the Python client is pushing submission annotations into the new field, which I think is correct.


I think it’s reasonable for the Python client to stop testing the deprecated query. We will leave the query in the back end since it is used by legacy leaderboard wikis. But new versions of the Python client need not use the old submission annotations or query.

Bruce Hoff
6 days ago

I think the problem is that the annotation is a list, not a number. (I’m not even sure why a list was allowed.):


Bruce Hoff
6 days ago

After doing the previous step I created a wiki with a widget to display the evaluation queue with its single submission:



The wiki shows the submission but not the annotation:

Further, if I add a filter ‘where foo=1’ or ‘where foo=”1”’, the submission is removed from the list. I think this is the phenomenon reported by .

Bruce Hoff
6 days ago

I set up a test queue and submission:


Bruce Hoff
April 12, 2021, 3:03 PM

My inclination is not to prioritize fixing this since it's a deprecated API. It's worth checking whether the deprecated API is still in use. Can we simply remove it altogether?

Won't Fix
Your pinned fields
Click on the next to a field label to start pinning.


Bruce Hoff


Jordan Kiang


Bruce Hoff




Development Area