JSON Schema-based Access Requirements

Corresponding ticket: PLFM-9449

This document describes extending the Synapse API so Access Requirements (ARs) can describe additional, flexible information that must be collected from end users in data access requests. The solution provides services that facilitate managing the AR questionnaires.

The proposed solution involves adding new services for the Synapse Access and Compliance Team (ACT) to create and manage 'form template' objects, which describe how a registered JSON Schema is rendered to requesters as a multi-step form. A new Access Requirement type is added that references both a registered JSON Schema (the data contract) and a form template (the presentation). New services facilitate generating a JSON Schema and UI Schema for such an access requirement, which can be presented to a user as a form with validation rules when they initiate a data access request.

Background

Sensitive data in Synapse can be protected using one or more Access Requirements (ARs). When an AR is applied to a resource (e.g. FileEntity) in Synapse, end users must meet the terms of the AR to access the resource (e.g., download the file). Many ARs are "Managed" ARs, where information is collected from a requester in a "Data Access Request" (DAR). The DAR is then reviewed by a Data Access Committee (DAC). Using the information provided by the requester, the committee may approve or reject the DAR. Requester(s) meet the terms of the AR when their corresponding DAR is approved.

While many of our projects have diverse governance and data adjudication needs, Access Requirements currently only support prompting a requester with a subset of predefined questions, which makes it difficult to gather and review specific information from data requesters. To better support these scenarios, we propose a design that extends the existing Access Requirement/Data Access Request flow to use JSON Schema to describe additional information to gather in an Data Access Request.

For more information and use case information, see PSI-1 and TECH-184.

Goals

• Extend access requirement and data access submission services to collect custom data within the existing data access request flow

• For a given Access Requirement, ACT can describe additional form data that should be collected in a data access request.

• Custom information is gathered via a form presented to requester

  • ACT can provide an optional 'UI Schema' to customize form appearance

  • Support custom attachments (form fields with type file)

  • Form can be split into multiple steps to simplify and streamline interface for requesters

• Submission reviewers can review custom form data

• Requesters can save partial form progress and resume in a later session

• Add a path to using schemas for existing Access Requirements/approvals (support a one-time migration step from the existing ManagedACTAccessRequirement)

• Reviewer and submitter can export a PDF of the created submission (client side logic)

• Expose publicly-disclosable information from approved submissions (functionality extending or similar to POST /accessRequirement/{arId}/approvedSubmissionInfo)

• Support eDUC workflows

Non-Goals

See PSI-1.

• Supporting all types of form data (e.g. nested objects)

• Conditional logic (see Appendix)

• Enabling interoperability with other platforms (see Appendix)

• Coalescing requests to multiple ARs into one form (see Appendix)

Proposal Summary

Today, every Managed ACT Access Requirement collects the same fixed set of fields from a data requester. If a project requires asking for additional information not captured by the Managed ACT Access Requirement, there's no way to capture it inside the standard DAR flow. Workarounds exist, such as asking users to upload additional files described in the wiki, or requiring a specific format for the "Intended Data Use" statement, but data requesters often fail to properly follow these instructions.

This proposal extends the AR/DAR flow so ACT can attach a registered JSON Schema (the data contract for the request) and a Form Template (the presentation layer that maps schema properties to UI) to an Access Requirement. A requester's answers are validated against that schema, stored on the Submission, and presented to reviewers as a structured form. A new AR type (JsonSchemaAccessRequirement) utilizes this behavior; existing ManagedACTAccessRequirements continue to work unchanged unless ACT explicitly migrates them.

Summary of New Objects and Services

• JSON Schema describes the shape of the data collected from a requester, including per-property type, validation rules, and which properties are required. Schemas are registered and versioned in the existing Synapse JSON Schema registry. Multiple Access Requirements may reference the same schema.

• FormTemplate is a versioned, ACT-managed object that defines how a referenced JSON Schema is rendered as a multi-step form. It carries an ordered list of steps (display title, description), and within each step, an ordered list of field slots. Each field slot binds a JSON pointer into the schema to a UI definition (and, for file-upload fields, a template file handle). A template is pinned to one schema version; updating a template creates a new template version. Multiple Access Requirements may reference the same FormTemplate (N:1).

• JsonSchemaAccessRequirement is a new AR type that references a FormTemplate (by id and version). The template's pinned schema is the AR's data contract — there is no separate schema reference on the AR. Otherwise behaves like ManagedACTAccessRequirement (same accessor flow, expirations, approvals).

• Schema generation service resolves the AR's FormTemplate and the schema it references, and emits a per-step (jsonSchema, uiSchema) bundle. The UI uses this bundle to render the form for requesters or as a read-only display for reviewers. The same service can render a draft template body to support previewing during template authoring.

• Schema submission service accepts the requester's answers, validates them against the schema referenced by the AR's pinned template (filtered by the active requestType), and creates a Submission whose schemaData holds the validated payload.

• For JsonSchemaAccessRequirement, submissionData includes the schematized data submitted by the user. Fields not used by other services (e.g. research project, IDU statement, supplemental attachments, publications, summary of use) are no longer first-class properties. Equivalent functionality can now expressed as schema properties. Reviewer UI for these submissions reads from schemaData in addition to the first-class fields on the Submission object. The migration step (see below) does not modify past submissions, so the UI needs to support showing data from any field.

Example Workflow

ACT registers a JSON Schema in the existing schema registry that captures the data contract for an AR. They author a FormTemplate that pins to that schema and arranges its properties into one or more steps with UI hints. They preview the resulting form, save the template, and create a JsonSchemaAccessRequirement that references the template. Updating the form (renaming a step, regrouping fields, changing UI hints) is a single transactional update to the template; updating the data contract (changing types, adding/removing properties, changing validation) is a schema bump (which requires publishing a new template version pinned to the new schema). Updates to a template do not automatically cascade to ARs that reference it; ACT explicitly bumps an AR to a new template version.

Requester side: When a user wants access to a resource gated by a JsonSchemaAccessRequirement, the client calls the schema generation service for that AR version. The response is a multi-step JSON Schema + UI Schema bundle. The user fills out the form with their responses. Request data can be saved any time in the RequestInterface object using POST /dataAccessRequest (this is the existing flow for ManagedACTAccessRequirements). Once ready to submit, the submission service validates the payload against the schema pinned by the AR version and either creates the Submission or returns validation errors.

Reviewer side: A reviewer opens a submission as they do today. The submission carries schemaData plus a reference to the AR version it was submitted against; the UI re-runs schema generation for that AR version to render a read-only form filled with the requester's answers. The reviewer adjudicates the submission the same as they would adjudicate a submission against a ManagedACTAccessRequirement.

First-class vs Schema-driven Fields

In general, a field is kept first-class only if a server-side workflow or an external integration utilizes that field. This includes:

• The eDUC workflow, including traditional DUC fallback (institution, principal investigator, signing official, ducFileHandleId, eDucSignatureEnvelopeId)

• The accessor/renewal lifecycle (accessorChanges)

Other fields become schema-driven. Reviewer UIs always render any present first-class fields in addition to schema fields; they must never infer that a schematized AR lacks first-class data (even those fields that are ‘schematized’, because an AR could be migrated from ManagedACTAccessRequirement to JsonSchemaAccessRequirement).

API Design

We propose adding the following new services and new/changed objects.

Services

Objects

Open

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplate

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplateStep

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplateField

Custom JSON Schema definition for File Handles (example for a schema property targeted by a FormTemplateField):

To denote a JSON Schema property as a Synapse file upload:

• it must be of number type

• it must use the custom format synapse-filehandle-id

The file can be downloaded via the file handle ID by using the existing FileHandleAssociateType.DataAccessRequestAttachment.

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplateReference

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplateSearchRequest

org.sagebionetworks.repo.model.dataaccess.schema.FormTemplateSearchResponse

org.sagebionetworks.repo.model.dataaccess.schema.GenerateDataAccessSchemaRequestInterface

org.sagebionetworks.repo.model.dataaccess.schema.GenerateDataAccessSchemaFromAccessRequirement

org.sagebionetworks.repo.model.dataaccess.schema.GenerateDataAccessSchemaFromTemplateDraft

org.sagebionetworks.repo.model.dataaccess.schema.GenerateDataAccessSchemaResponse

org.sagebionetworks.repo.model.dataaccess.RequestInterface

org.sagebionetworks.repo.model.AccessRequirementReference

org.sagebionetworks.repo.model.HasExpiration

• interface factored out of ManagedACTAccessRequirement for reuse

org.sagebionetworks.repo.model.HasDataUseCertificate

• interface factored out of ManagedACTAccessRequirement for reuse

org.sagebionetworks.repo.model.JsonSchemaAccessRequirement

org.sagebionetworks.repo.model.dataaccess.Submission

The submission already includes the accessRequirementId and accessRequirementVersion, which can be used to get the schema this data was submitted against. For reviewers, the UI can display a 'read-only' version of the schema form filled in with the schemaData.

For submissions created against a JsonSchemaAccessRequirement:

• schemaData carries the validated submission payload.

• requestId is populated, and schemaData is snapshotted onto the Submission at submit-time.

• researchProjectSnapshot is null — its fields (institution, project lead, IDU, etc.) are now expressed as schema properties and live inside schemaData.

  • NOTE: The migration step will not backfill schemaData for historical ManagedACT submissions. The reviewer UI should support showing the first class properties and/or schemaData regardless of AR type.

org.sagebionetworks.repo.model.dataaccess.schema.PublicSubmissionInfoPageRequest

org.sagebionetworks.repo.model.dataaccess.schema.PublicSubmissionInfoPage

org.sagebionetworks.repo.model.dataaccess.schema.PublicSubmissionTemplateBundle

org.sagebionetworks.repo.model.dataaccess.schema.PublicSubmissionInfo

Sequence Diagram Examples

(Code for above)

(Code for above)

UI Mockups

The following images are for demonstration purposes only and are subject to change. They may be out-of-date as this design is updated.

Open

Video Demonstration:

Open

Open Questions

• Access Requirement updates can now substantially impact the submission requirements (new properties or validation rules may be added, the form layout may be reorganized, etc). How should we handle the following sequence?

  1. User begins to fill out the data access request form for AR 123, v4 (with a saved draft).

  2. AR 123 is updated to v5, which references a new schema or template version that adds required properties.

  3. User creates the submission with AR 123 v4 data.

     • Should the submission be accepted for review by the system? Should we only allow submission against the latest version of the AR (forcing the user to restart)?

     • What about the case where the submission is created, but has not been reviewed, before the AR update?

     • How should the draft service surface drift? (Likely the UI compares the draft's accessRequirement.accessRequirementVersionNumber against the current AR version and warns the user, but the contract should be explicit.)

Preventing duplicate inputs for fields required by eDUC

To support the eDUC flow, we must prompt for certain first-class fields, including PI name, PI email, institution, etc. Should these first-class form fields always be presented statically?

Or must we design a way for the JSON Schema to somehow customize these fields? If the fields must be dynamically controlled by the schema, options include

• eDUC-compatible ARs must include a $ref to a schema that describes the eDUC-required fields

• eDUC-compatible ARs must use a JSON Schema that includes properties matching eDUC-required fields.

Appendix

GA4GH Passport Visa

The research spike ticket includes the following "bonus" acceptance criteria":

Assess whether JSON submission can be stored/exchanged on Passport Visa

My assessment is that form responses could technically be encoded in a GA4GH Passport Custom Visa, which is a signed JWT. However, this may not be the best approach to disseminating access request responses. For example, the set of user responses could be very large. Using the Visa protocol to disseminate this kind of information seems counter to the intent of the specification.

Extension: Conditional Logic and Dynamic Requiredness

At a later date, we could extend the schema/template model to capture richer behaviors that are out of scope for v1:

• Conditional logic (show/hide a property based on other answers)

• Dynamically changing the 'requiredness' of a property based on other answers

• Cross-property validation (e.g. one of two properties must be filled in)

These would likely be expressed in the JSON Schema (using if/then/else, oneOf, etc.) with the FormTemplate's UI hints adapting to the schema's conditional structure.

Extension: Streamlined/Deduplicated Form

A future extension lets a requester applying to multiple ARs fill out one merged form. The v1 design enables this without breaking changes — the cases below describe what the extension would handle.

Case 1 — same FormTemplate. ARs sharing a template share a schema and a form. We display a single form, validate against the shared schema, and create one Submission per AR.

Case 2 — different templates, overlapping questions. AR1 asks A, B, C; AR2 asks A, B, D; merged form asks A–D once. Question identity is determined by using the same $refs to a registered sub-schema: properties $ref'd from the same sub-schema dedupe; properties defined inline do not. Fields without a shared $ref are asked once per AR. The client is responsible for facilitating subschema reuse.

We must still decide how to handle UI hint conflicts and step ordering mismatches, but it seems reasonable to just take the first value.