This is the design doc for integration between the Upload APIs and the Survey APIs. The goal of this design is to (a) export Survey API survey responses to Synapse and (b) facilitate migration from client-side surveys to server-side surveys.

Overview

When a researcher creates a survey:

1. The researcher creates, updates, and versions a survey as often as needed.
2. When the researcher publishes the survey, the Survey API creates the corresponding upload schema.
3. During the Bridge Exporter's next run, it will create the Synapse table corresponding to the survey's upload schema, and possibly populate it with data as described below.

When a user responds to a survey:

1. The user creates and appends answers to a survey response as many times as needed.
2. Once the user has answered all the questions or has called the survey response complete API, the Survey API creates the corresponding health data record.
3. During the Bridge Exporter's next run, it will export the survey response's corresponding health data record to the Synapse table described above.

*iOS Research Kit specific* Responding to a server-side survey using the upload API:

1. The user submits a complete survey response through the upload API, the same way they would submit a client-side survey response, except the info.json is tagged with the survey guid and created-on.
2. The Upload API detects that this is a server-side survey using the survey guid and created-on tags and converts the upload into a survey response.
3. The Upload API then calls the survey response and survey response complete API and things proceed as usual.

Design Details

Survey Schemas

We'll need a way to describe the Synapse tables, for creating the tables, exporting data to the tables, and identifying and validating the incoming tables. I feel like it's cleaner to auto-generate upload schemas from published surveys than to change the code in multiple places to understand surveys in place of upload schemas. Since upload schemas are auto-generated from published surveys and since published surveys are immutable, this minimizes the chance of a mismatch between the survey and the schema.

The alternative would be to change the health data records and the client-side surveys to look like server-side surveys, but from our earlier discussions, it sounds like we didn't want to do this. (Plus, it would almost certainly be messier.)

We should only create schemas when the researcher publishes a survey. This way, we don't create a bunch of intermediate schemas (and intermediate Synapse tables) for surveys that don't get exposed to end users.

We will add a new field schemaRevision to the Survey table. The Survey API uses this as the schemaRevision, the survey identifier as the schema ID, the survey name as the schema name, and maps survey questions into schema fields. A schema is needed because that's what's used to create and export health data records to Synapse.

For details of what a schema looks like, see Bridge Upload Data Format#Schemas

Survey Responses

*NEW API* We will need a new api (probably /api/v2/surveyresponses/{guid}/complete) for the user to signal the early completion of a survey. This is needed so the server knows when to start creating the health data record vs waiting for more survey answers. As a convenience, if the number of answers submitted in a survey response is equal to the number of questions, we can auto-detect that the survey response is complete and automatically start creating the health data record.

If for some reason, the user submits a response to survey without an upload schema (for example, the survey was never published, but was somehow leaked to users early), the survey response API will throw a 400 Bad Request.

Upload API

We want to make it easy for the current set of apps to migrate to server-side surveys. Otherwise, they might not migrate at all. So we'll allow server-side survey responses to be submitted through the Upload API. However, this won't allow partial responses or appending to existing responses, since the Upload API has no way to return that back to the user.

That said, given a chance to re-design the data format from scratch, we should have users call that Survey API directly. (Or at minimum, the Upload API data format for surveys is the same as the Survey API data format.)

For iOS Research Kit based apps, in order to use this feature, the info.json must contain fields "surveyGuid" and "surveyCreatedOn", corresponding to the surveyGuid and surveyCreatedOn from the Survey API.

Client-side surveys can't go through the Survey API, since client-side surveys don't exist on the server side. If the info.json doesn't contain the survey guid and created-on fields, it will go through the old code path for iOS surveys.

See Also

Bridge REST API

Bridge Upload Data Format