Requirements
These were collected from a meeting we held with Larsson, Brian, Thaneer, and the development team, and from follow-up interviews with Dan and Christine.
- We will have partners who contribute participants to existing studies, that is, users who download and use our app, and thus are contributing data to a larger population of participants (we propose to call these partners "sub-study partners");
- Sub-study partners may include their participants in (at least) a couple of ways:
- They may assign them an external ID, that in essence should identify the participant's sub-study membership as well;
- They may ask existing participants to join the study, such that signing a consent in Bridge should enroll that participant in the sub-study;
- Sub-study partners will also create user accounts to manage their own users and external identifiers (only)
- Sub-study participants may receive schedules (and thus, tasks or surveys) that are unique to their sub-study, but we believe in the main, these changes to the main study will be additive. However, Sage Bionetworks will be responsible for incorporating such changes so that they don't break the main study;
- Sub-study partners may access their list of participants and/or external identifiers, and can probably do anything an existing researcher can do with those entities... but only those that are in the sub-study. They cannot see participants or external IDs in other sub-studies (or no sub-study);
- Sub-study membership needs to be exported with all data exported to Synapse. The team that processes the data will use this membership information to create specific repositories for those sub-study partners;
- All users can be in multiple sub-studies (both participants, and administrative users). It's not clear what the requirements are for this (e.g. should a researcher see all participants in all the sub-studies they are a member of at once, or only one at a time with a mechanism to select their current sub-study?), but these requirements don't radically change the implementation design;
An in-progress list of tasks...
...
// Either or both of accountId and externalId need to be filled out. Would probably use ID column to avoid composite keys, but then would need to verify the results conform to the caller's study and sub-study.
CREATE TABLE `ExternalIds` {
`id` VARCHAR(60) NOT NULL,
`studyId` VARCHAR(60) NOT NULL,
`subStudyId` VARCHAR(60) NOT NULL,
`accountId` VARCHAR(255) NULL,
`externalId` VARCHAR(255) NULL,
PRIMARY KEY (`id`)
UNIQUE INDEX `StudyId-SubStudyId-Index` (`studyId` ASC, `subStudyId` ASC), // will return more than one
UNIQUE INDEX `StudyId-ExternalIdAccountId-Index` (`studyId` ASC, `externalId` `accountId` ASC), // could return more than one
UNIQUE INDEX `StudyId-SubStudyId-AccountIdExternalId-Index` (`studyId` ASC, `accountId` `externalId` ASC)
}
// An outstanding issue is dealing with batch adds... we should probably support that, but not batch deletes.
ExternalIdsServiceV2 {
listExternalIds(studyId, subStudyId, offsetBy, pageSize, includeDeleted)
createExternalId(externalIdObj)
getExternalId(studyId, subStudyId, externalId)
updateExternalId(externalIdObj)
deleteExternalId(studyId, subStudyId, externalId)
deleteExternalIdPermanently(studyId, subStudyId, externalId)
// These would update an add an associate record between Account and SubStudy with an account ID
assignExternalId(studyId, subStudyId, externalId, accountId)
unassignExternalId(studyId, subStudyId, externalId)
}
...
GET /v3/substudies/:subStudyId/externalids [list]
POST /v3/substudies/:subStudyId/externalids [create] <-- could take a list for batch creates
GET /v3/substudies/:subStudyId/externalids/:id [read]
POST /v3/substudies/:subStudyId/externalids/:id [update]
DELETE /v3/substudies/:subStudyId/externalids/:id [delete]
Note: in Hibernate, for this to be an entity, it'll need a primary composite key based on the foreign keys in the table. Or actually I like @IdClass for this purpose.Not sure how, in this case, we can leave accountId empty.
Further tasks in no particular order:
Migrate all the existing external IDs from DynamoDb to SQL tables and adjust all the existing APIs to use the new external Ids table. This means adjusting some APIs to look up users through this association rather than the DDB table.
Add a user's sub-study association to AccountSummary, StudyParticipant, and the UserSession.
Participant APIs should only include sub-studies that the caller belongs to. Admins might break this rule.
Users can be associated to a sub-study in one of several ways:
...
probably going to use a synthetic primary key rather than muck around with composite keys, particularly because accountId OR externalId can be null, but not both (maybe best enforced in the DAO).
AccountDao - mostly calls through to the ExternalIdService
- sub-study association needs to be restricted to the caller's sub-studies. For example, the set of sub-studies need to be passed in, or the StudyParticipant needs to be "tainted" with the caller's identity when it is constructed so permissions can be checked. Submitted record can participate in a set of sub-studies and then these are checked against membership of caller.
- constructAccount: set up eventual persistence of associative record
- Iterator/List calls should specify a sub-study (or perhaps a list, if we want BSM users to see all users they can see, across all studies)
Account, AccountSummary, StudyParticipant, UserSession
- The construction of all three of these arguments should return their sub-studies. It's conceivable we don't include in AccountSummary.
Migrate external IDs
- Create existing sub-studies
- read from both external ID systems, write only to new system
- Migrate external IDs to create associative records, removing external ID from the account table where appropriate.
- Remove old APIs from SDK
- Remove links to old tables
- Remove APIs from BridgePF
Update sub-populations so that signing the consent of a sub-population will assign a user to one ore more sub-studies, without an external ID (additional behaviors can be implemented as needed).
Uploads need to be tagged with the ID set of a user's sub-studies (all of them). Hence the need to have this available in the StudyParticipant record and the UserSession.Consider removing "unmanaged" external IDs as this adds some complexity to the logic of external IDs and I'm not sure this is used at all at this point, or will be.sub-study ids assigned to a user
Add the ability to filter by sub-studies using the Criteria object. Like tags, you should be able to add a set of sub-studies, at least one of which should match, or a set where none may match. The main use for this would be to schedule different sub-studies differently, in the context of an overarching multi-study design.I would also start by cleaning up accounts to remove GenericAccount and HibernateAccount and the copying between the two of them (subsequent changes will heavily involve the AccountsDao so it would help to simplify first).