Initially we referred to this as compliance, but because “compliance” also has meaning in a governance context, so we’re using “adherence” to describe the measurement of study member participation in the study.
...
The schedule itself, which is finite in length and described by a timeline;
The set of adherence records for the participant which describe what the participant has done in that timeline (we only work with the session records when calculating adherence);
A set of event timestamps that tell us where the participant is at in the timeline (ie what the user should have done and what they should currently be doing in the timelineschedule).
So far the Bridge server provides this information through separate APIs and does not keep track of the time of the participants. For the sake of efficiency, I think we want the Bridge server to combine this information and provide reports on the status of the account, which will be less resource intensive that serving out all the information to be processed by a client or worker process. The issues I see with this:
For the first time the server will need to have knowledge of the participant’s time and time zone(?);
Because this information depends on the time of the request, it is not very cacheable;
Nevertheless, the The reports probably update infrequently compared to the amount they will be read (the exact amount depends on many factors), while they may be read frequently in a couple of different formats. We will have to devise caching strategies to make the system performant when looking at many participants.
Persistent time windows will be excluded from all adherence reports. Completing assessments that are part of a persistent window do not count for or against adherence.
All of these views operate on the most recent timestamps for all events. Building schedules that rely on a mutable event changing, and triggering a new timeline of sessions to perform, will not work with these adherence APIs. That would be events like “session type X “do X when session type Y has been completed.” OTOH, Since it will show compliance with the most recent event time stream, and that might be all that mattersit may be sufficient anyway. Past time streams are no longer actionable.
...
Method | Path (Under /v5/studies/{studyId} | Description |
|---|---|---|
GET | /participants/{userId}/adherence/eventstream | List of SessionStream reports EventStreamAdherenceReport for one user. The only This view that shows includes scheduling for based on events the user does not have, this and is a detailed view of the entire schedule for one user of the whole schedule. |
GET | /participants/{userId}/adherence/weekly | A single WeeklyAdherenceReport for one usersuser. This API would generate and persist this report before returning it in the call, so that this call can be used by a worker process to build a record for every participant, but the call could also be called for up-to-date state information on a specific participantcalculates the “current week” of each event timestamp and aligns it to a seven day schedule. |
GET | /participants/adherence/weekly | A paged list where each record is a WeeklyAdherenceReport for one user. If no call has generated this report, it is not generated as part of this call, so a worker process will need to periodically generate these out-of-request. |
GET | /v1/apps/{appId}/studies/{studyId} | Worker endpoint to create and persist the weekly report for users, to enable the paged view across all accounts. |
...
| Code Block | ||
|---|---|---|
| ||
{
"activeOnly": false,
"timestamp": "2021-11-23T00:32:12.583Z",
"adherencePercent": 11,
"streams": [
{
"startEventId": "study_burst:main-sequence:02",
"eventTimestamp": "2021-12-26T20:00:00.000Z",
"byDayEntries": {
"0": [
{
"sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7",
"label": "Session #1",
"symbol": "1",
"startDay": 0,
"startDate": "2021-12-26",
"timeWindows": [
{
"sessionInstanceGuid": "vGQycgYTXNtk37Wcqbwv_w",
"timeWindowGuid": "GNp94CnfTTtR-s0OzrFeftrh",
"state": "not_yet_available",
"endDay": 0,
"endDate": "2021-12-26",
"type": "EventStreamWindow"
},
{
"sessionInstanceGuid": "JAy5-aSZnxUjAt2B9xmHeg",
"timeWindowGuid": "aRaHNKIY0yKgOl5CLuA3ZDHJ",
"state": "not_yet_available",
"endDay": 0,
"endDate": "2021-12-26",
"type": "EventStreamWindow"
}
],
"type": "EventStreamDay"
}
]
},
"type": "EventStream"
},
{
"startEventId": "study_burst:main-sequence:01",
"eventTimestamp": "2021-11-21T20:00:00.000Z",
"daysSinceEvent": 1,
"byDayEntries": {
"0": [
{
"sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7",
"label": "Session #1",
"symbol": "1",
"startDay": 0,
"startDate": "2021-11-21",
"timeWindows": [
{
"sessionInstanceGuid": "1aCbUaFYkixIsIJBf9WGpg",
"timeWindowGuid": "GNp94CnfTTtR-s0OzrFeftrh",
"state": "expired",
"endDay": 0,
"endDate": "2021-11-21",
"type": "EventStreamWindow"
},
{
"sessionInstanceGuid": "yQnubrShfYMY9ZzOE3zw3Q",
"timeWindowGuid": "aRaHNKIY0yKgOl5CLuA3ZDHJ",
"state": "expired",
"endDay": 0,
"endDate": "2021-11-21",
"type": "EventStreamWindow"
}
],
"type": "EventStreamDay"
}
]
},
"type": "EventStream"
},
],
{ "startEventId": "custom:event2",
"eventTimestamp": "2021-11-15T20:00:00.000Z",
"daysSinceEvent": 7,
"byDayEntries": {
"0": [
{
"sessionGuid": "z_jb4p2Lr9Q56z8AwiYNieqw",
"label": "Session #3",
"symbol": "3",
"startDay": 0,
"startDate": "2021-11-15",
"timeWindows": [
{
"sessionInstanceGuid": "KAxvwhsX6jSVl89a3-gdKw",
"timeWindowGuid": "gF6hy-UiipJLXqe7F_yK-wQc",
"state": "expired",
"endDay": 0,
"endDate": "2021-11-15",
"type": "EventStreamWindow"
}
],
"type": "EventStreamDay"
}
],
"3": [
{
"sessionGuid": "z_jb4p2Lr9Q56z8AwiYNieqw",
"label": "Session #3",
"symbol": "3",
"startDay": 3,
"startDate": "2021-11-18",
"timeWindows": [
{
"sessionInstanceGuid": "nQnWs_4ECvLtY4K1gYk67g",
"timeWindowGuid": "gF6hy-UiipJLXqe7F_yK-wQc",
"state": "expired",
"endDay": 3,
"endDate": "2021-11-18",
"type": "EventStreamWindow"
}
],
"type": "EventStreamDay"
}
]
},
"type": "EventStream"
},
{
"startEventId": "custom:event1",
"eventTimestamp": "2021-11-21T20:00:00.000Z",
"daysSinceEvent": 1,
"byDayEntries": {
"0": [
{
"sessionGuid": "eRLgI5gfe1kef_XRZDfdFU9I",
"label": "Session #2",
"symbol": "2",
"startDay": 0,
"startDate": "2021-11-21",
"timeWindows": [
{
"sessionInstanceGuid": "bDBVV02XrjrOG8NgBYSYFg",
"timeWindowGuid": "KZ1piANVdeD-r8PCHL2bviLh",
"state": "completed",
"endDay": 0,
"endDate": "2021-11-21",
"type": "EventStreamWindow"
}
],
"type": "EventStreamDay"
}
]
},
"type": "EventStream"
}
],
"type": "EventStreamAdherenceReport"
} |
The windows include the state of that session (which is the basis for adherence calculation). This adherence report could be displayed on a calendar since it is specific to a user and the server must know all the values to calculate it (and thus it’s the only report that can take a timestamp to indicate the time at which the report should be calculated).
The completion states are:
...
State
...
Description
...
Adherence
...
not_applicable
...
Participant does not have this event in their events, so these sessions will not currently ever be shown to the participant.
...
N/A
...
not_yet_available
...
Participant should not have seen or started this session. It’s in the future.
...
N/A
...
unstarted
...
Participant should see the session (they are being asked to do it now), but they have not started it.
...
unknown
...
started
...
Participant has started the session.
...
unknown
...
completed
...
Participant completed the session before it expired.
...
compliant
...
abandoned
...
Participant started or finished at least one assessment in the session, but there was more work to do and it expired before they finished it.
...
noncompliant
...
expired
...
Participant did not start the session and it is now expired.
...
noncompliant
We can calculate a compliance percentage from these values across a participant’s entire participation in the study. In the weekly report below, we can calculate adherence for that week alone.
Weekly Adherence Report
This report always returns seven days of adherence records for a given user. The report calculates this information by finding the “week since event N” for every event and every session, that is the week that has the day that falls on “today” (this timestamp won’t be adjustable since every timestamp would require a recalculation of all these reports—so it’ll have to be a date on the server). This means that the individual sessions listed in this report are not lined up by calendar date. The structure of this report would be as follows:
| Code Block | ||
|---|---|---|
| ||
{ "participant": { "firstName": "A-chan", "email": "alx.dark+achan@sagebase.org", "externalId": "asdfasdf", "identifier": "GqYpNUWolebxS2eQudF1hc-a", "type": "AccountRef" }, "timestamp": "2021-11-23T00:39:43.922Z", "weeklyAdherencePercent": 16, "dailyReports": [ { "day": 0, "sessions": [ { "sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7", "label": "Session #1 / Week 1 / Study Burst 1", "symbol": "1", "startDay": 0, "startDate": "2021-11-21", "timeWindows": [ { "sessionInstanceGuid": "1aCbUaFYkixIsIJBf9WGpg", "timeWindowGuid": "GNp94CnfTTtR-s0OzrFeftrh", "state": "expired", "endDay": 0, "endDate": "2021-11-21", "type": "EventStreamWindow" }, { "sessionInstanceGuid": "yQnubrShfYMY9ZzOE3zw3Q", "timeWindowGuid": "aRaHNKIY0yKgOl5CLuA3ZDHJ", "state": "expired", "endDay": 0, "endDate": "2021-11-21", "type": "EventStreamWindow" } ], "type": "EventStreamDay" }, { "sessionGuid": "eRLgI5gfe1kef_XRZDfdFU9I", "label": "Session #2 / Week 1", "symbol": "2", "startDay": 0, "startDate": "2021-11-21", "timeWindows": [ { "sessionInstanceGuid": "bDBVV02XrjrOG8NgBYSYFg", "timeWindowGuid": "KZ1piANVdeD-r8PCHL2bviLh", "state": "completed", "endDay": 0, "endDate": "2021-11-21", "type": "EventStreamWindow" } ], "type": "EventStreamDay" } ], "type": "DailyAdherenceReport" }, { "day": 1, "sessions": [ { "sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7", "label": "Session #1 / Week 1 / Study Burst 1", "symbol": "1", "startDay": 1, "startDate": "2021-11-22", "timeWindows": [ { "sessionInstanceGuid": "7HaIdOehYJrJk3VZcGeNxg", "timeWindowGuid": "GNp94CnfTTtR-s0OzrFeftrh", "state": "unstarted", "endDay": 1, "endDate": "2021-11-22", "type": "EventStreamAdherenceReport" } |
EventStreamWindow includes the state of that session (which is the basis for adherence calculation). This adherence report could be displayed on a calendar since it is specific to a user and the server must know all the calendrical values to calculate it (and it's the only report that can take a timestamp to indicate the time at which the report should be calculated).
The completion states are:
State | Description | Adherence |
|---|---|---|
not_applicable | Participant does not have this event in their events, so these sessions will not currently ever be shown to the participant. | N/A |
not_yet_available | Participant should not have seen or started this session. It’s in the future. | N/A |
unstarted | Participant should see the session (they are being asked to do it now), but they have not started it. | unknown |
started | Participant has started the session. | unknown |
completed | Participant completed the session before it expired. | compliant |
abandoned | Participant started or finished at least one assessment in the session, but there was more work to do and it expired before they finished it. | noncompliant |
expired | Participant did not start the session and it is now expired. | noncompliant |
We can calculate a compliance percentage from these values across a participant’s entire participation in the study. In the weekly report below, we can calculate adherence for that week alone.
Weekly Adherence Report
This report always returns seven days of adherence records for a given user. The report calculates this information by finding the “week since event N” for every event and every session (it will be the week that has the day that falls on “today.”) This timestamp won’t be adjustable since every timestamp would require a recalculation of all these reports—so it’ll have to be a date on the server, possibly adjustable per study to a different time zone. Note that the the individual sessions listed in this report are not lined up by calendar date (unless they were triggered by the same event). The structure of this report would be as follows:
| Code Block | ||
|---|---|---|
| ||
{ "participant": { "firstName": "A-chan", "email": "alx.dark+achan@sagebase.org", "externalId": "asdfasdf", "identifier": "GqYpNUWolebxS2eQudF1hc-a", "type": "EventStreamWindowAccountRef" }, "timestamp": "2021-11-23T00:39:43.922Z", "weeklyAdherencePercent": 16, { "dailyReports": [ { "sessionInstanceGuidday": "R0I7n1fEeIR88fYv6iVPTg"0, "sessions": [ "timeWindowGuid": "aRaHNKIY0yKgOl5CLuA3ZDHJ", { "statesessionGuid": "unstartedLcWpQFKaGY5FSQ0LT4tnvdO7", "label": "Session #1 / "endDay":Week 1, / Study Burst 1", "endDatesymbol": "2021-11-221", "typestartDay": "EventStreamWindow"0, } ]"startDate": "2021-11-21", "typetimeWindows": "EventStreamDay"[ }, { { "sessionGuidsessionInstanceGuid": "eRLgI5gfe1kef_XRZDfdFU9I1aCbUaFYkixIsIJBf9WGpg", "labeltimeWindowGuid": "SessionGNp94CnfTTtR-s0OzrFeftrh", #2 / Week 1", "symbolstate": "2expired", "startDayendDay": 10, "startDateendDate": "2021-11-2221", "timeWindowstype": ["EventStreamWindow" }, { "sessionInstanceGuid": "u-cc8Ou4xqUPVnht4oh-bw"yQnubrShfYMY9ZzOE3zw3Q", "timeWindowGuid": "KZ1piANVdeD-r8PCHL2bviLhaRaHNKIY0yKgOl5CLuA3ZDHJ", "state": "unstartedexpired", "endDay": 10, "endDate": "2021-11-2221", "type": "EventStreamWindow" } ], "type": "EventStreamDay" } ], "type": "DailyAdherenceReport" }, { "day": 5, "sessions": [ { "sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7eRLgI5gfe1kef_XRZDfdFU9I", "label": "Session #1#2 / Week 1 / Study Burst Week 1", "symbol": "12", "startDay": 50, "startDate": "2021-11-2621", "timeWindows": [ { "sessionInstanceGuid": "JfyiAbHZ-nlmTZui_N19TgbDBVV02XrjrOG8NgBYSYFg", "timeWindowGuid": "GNp94CnfTTtRKZ1piANVdeD-s0OzrFeftrhr8PCHL2bviLh", "state": "not_yet_availablecompleted", "endDay": 50, "endDate": "2021-11-2621", "type": "EventStreamWindow" }, ], { "type": "EventStreamDay" "sessionInstanceGuid": "g6kRO-JZgt0lGghDT8v37A", } ], "timeWindowGuidtype": "aRaHNKIY0yKgOl5CLuA3ZDHJDailyAdherenceReport", }, { "stateday": "not_yet_available",1, "sessions": [ { "endDaysessionGuid": 5"LcWpQFKaGY5FSQ0LT4tnvdO7", "label": "Session #1 / "endDate": "2021-11-26", Week 1 / Study Burst 1", "typesymbol": "EventStreamWindow1", "startDay": 1, } ]"startDate": "2021-11-22", "typetimeWindows": "EventStreamDay"[ }, { { "sessionGuidsessionInstanceGuid": "eRLgI5gfe1kef_XRZDfdFU9I7HaIdOehYJrJk3VZcGeNxg", "labeltimeWindowGuid": "SessionGNp94CnfTTtR-s0OzrFeftrh", #2 / Week 1", "symbolstate": "2unstarted", "startDayendDay": 51, "startDateendDate": "2021-11-2622", "timeWindowstype": [ "EventStreamWindow" }, { "sessionInstanceGuid": "-OoNi3vhifRUqN6X9WTMCgR0I7n1fEeIR88fYv6iVPTg", "timeWindowGuid": "KZ1piANVdeD-r8PCHL2bviLhaRaHNKIY0yKgOl5CLuA3ZDHJ", "state": "not_yet_availableunstarted", "endDay": 51, "endDate": "2021-11-2622", "type": "EventStreamWindow" } ], "type": "EventStreamDay" }, { "sessionGuid": "zeRLgI5gfe1kef_jb4p2Lr9Q56z8AwiYNieqwXRZDfdFU9I", "label": "Session #3#2 / Week 21", "symbol": "32", "startDay": 121, "startDate": "2021-11-2722", "timeWindows": [ { "sessionInstanceGuid": "2Ou-cc8Ou4xqUPVnht4oh-jPnpWOYZLx0VjBvvO8gbw", "timeWindowGuid": "gF6hy-UiipJLXqe7F_yK-wQcKZ1piANVdeD-r8PCHL2bviLh", "state": "not_yet_availableunstarted", "endDay": 121, "endDate": "2021-11-2722", "type": "EventStreamWindow" } ], "type": "EventStreamDay" } ], "type": "DailyAdherenceReport" }, ... etc. { "day": 6, "sessions": [ { "sessionGuid": "LcWpQFKaGY5FSQ0LT4tnvdO7", "label": "Session #1 / Week 1 / Study Burst 1", "symbol": "1", "startDay": 6, "startDate": "2021-11-27", "timeWindows": [ { "sessionInstanceGuid": "IUBXsO4ioFgJfi8GA9NSxw", "timeWindowGuid": "GNp94CnfTTtR-s0OzrFeftrh", "state": "not_yet_available", "endDay": 6, "endDate": "2021-11-27", "type": "EventStreamWindow" }, { "sessionInstanceGuid": "WdRjHkDWoqmQx4vxRuCVeg", "timeWindowGuid": "aRaHNKIY0yKgOl5CLuA3ZDHJ", "state": "not_yet_available", "endDay": 6, "endDate": "2021-11-27", "type": "EventStreamWindow" } ], "type": "EventStreamDay" }, { "sessionGuid": "eRLgI5gfe1kef_XRZDfdFU9I", "label": "Session #2 / Week 1", "symbol": "2", "startDay": 6, "startDate": "2021-11-27", "timeWindows": [ { "sessionInstanceGuid": "_-1AlFrrN9IfA5tK-OYvag", "timeWindowGuid": "KZ1piANVdeD-r8PCHL2bviLh", "state": "not_yet_available", "endDay": 6, "endDate": "2021-11-27", "type": "EventStreamWindow" } ], "type": "EventStreamDay" } ], "type": "DailyAdherenceReport" } ], "type": "WeeklyAdherenceReport" } |
The user-specific API will persist this data when it is calculated before returning the report so it can be retrieved as part of a wider study-scoped adherence report. This API would calculate and persist this report before returning it in the call, so that this call can be used by a worker process to build a record for every participant. The call could also be used to retrieve up-to-date state information for a specific participant. If this call is not called for a user, it will not show up in the paginated view of these records. We may want to hide a record that hasn’t been updated in the last 24 hours although a worker process will hopefully force a refresh on all accounts.
The SQL table for this report will not attempt to store the report normalized:
| Code Block | ||
|---|---|---|
| ||
CREATE TABLE `WeeklyAdherenceReports` ( `appId` varchar(255) NOT NULL, `studyId` varchar(255) NOT NULL, `userId` varchar(255) NOT NULL, `createdOn` bigint(20) unsigned NOT NULL, `labels` text NOT NULL, `weeklyAdherencePercent` int(3) unsigned NOT NULL, `json``reportData` mediumtext NOT NULL, PRIMARY KEY (`appId`, `studyId`, `userId`), CONSTRAINT `WeeklyAdherenceReports-Study-Constraint` FOREIGN KEY (`studyId`, `appId`) REFERENCES `Substudies` (`id`, `studyId`) ON DELETE CASCADE, CONSTRAINT `WeeklyAdherenceReports-Account-Constraint` FOREIGN KEY (`userId`) REFERENCES `Accounts` (`id`) ON DELETE CASCADE, ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; |
The “labels” column will pull out the labels from the JSON into a pipe-delimited string, to enable searching for records that match the label (this is a design requirement) in the study-scoped API (searching by label and percentage of adherence are UI design requirements).
Weekly Adherence Reports for Study
The weekly adherence API for a user will persist the report. Another final API will allow an admin to retrieve all these records using a paging API that is similar to the study participants search API. Those search criteria will be used to retrieve the page of user IDs, and then those will be used to retrieve and sort the appropriate set of adherence reports. If the adherence report hasn’t been requested for a specific user, it will come back empty. (A worker process will be created to run through users nightly and regenerate these reports, but the weekly adherence report API will accomplish the same refresh for one user).All the same arguments should exist as a paged set of weekly adherence records for all participants in the study. This retrieves the records from the database and does not calculate them. The paging search criteria should be the same as the study participants search API. In addition to the search arguments in the /v5/studies/{studyId}/participants/search API. In addition, it should be possible to filter and or sort on the following fields:
The adherence percentage;
The label of individual windows (through a like query on the labels column).
(I’m not sure how to do this in a way that’ll maintain paging against the study participant records…we’ll have to join this table and then maintain two sets of code for the core participant search and filter functionality)I would like to reuse the search code for participants, but it may be necessary to join tables in a way where that can’t be done, while still maintaining the desired paging logic. To start, it’s probably possible to just search and sort by adherence and labels.
Protocol adherence and notifications
Since all the reports give an adherence percentage (whole study or one week), we can use this to sort records by adherence in the weekly view, or to generate a message to administrators as part of the worker process that will be run nightly to update thesethe weekly reports. Any necessary configuration for this can be added to the Study model (in particular, at what percentage of adherence should we notify administrators that a participant is out of adherence).
A worker process would go through all apps nightly, looking for apps with studies that have schedules, and then for each of these studies, it would iterate through all participants in the study, calling the weekly adherence API endpoint for that participant to generate and cache their adherence report. It can also generate messages, if need be.
System Messaging APIs
One UI design I have seen shows UI designs show a list of notifications in an administrative UI, indicating specific participants that are out-of-adherence with the study protocol. This hints at a very large amount of functionality.a very large amount of functionality. Because “notifications” is heavily overloaded for Bridge, I will call these “system messages.”
We have had requirements to message both participants and study administrators over the years. For example, very early on we embedded a blog in an app for this purpose. Then we implemented support for remote push notifications and topics (rarely used). We often message participants through Instead in recent studies we have tried using local notifications based on scheduling information. Now we are talking about showing notifications to administrators about participants being out-of-adherence in an administrative UI.
...
API would be push-pull (write a message, request a list of active messages for a recipient; push-push via something like SMS or email is a lot more complicated);
recipients could include individuals or organizations (and other groups once we have them);
each recipient could mark the message as “read” to remove it from their UIs, separate from others, or “resolved” to remove it from everyone’s UIsfor everyone. They would not be deleted so they are auditable;messages could be set to expire;Messages could indicate if they can be read, resolved, or if they expire. For example, an adherence message might only allow you to mark it “resolved” since you don’t want multiple study coordinators all following up with a participant.
Messages could be assigned (change recipients), and indicate a status like “busy” along with the user currently working on the message. Now you have a very simple way to hand off work between team members.
...