🗞 Headlines
Our goal for the Mobile Toolbox Platform is to provide a complete end-to-end solution for the use of remotely-administered cognitive and other tests by researcher teams, without the need for any study-specific technology development. A complete solution must cover scenarios across the complete lifecycle of a study, and support research teams in:
Providing well documented validated measures.
Learning about available mobile cognitive measures and designing appropriate digital research protocols.
Managing collection of data from study participants, including support for enrollment, aggregate/high-level adherence tracking, and participant communications.
Providing secure transfer of de-identified data to analysts.
Disseminating data, results, and other findings from completed studies for forms that encourage reuse by other research teams, subject to FAIR principles.
To date, our efforts have been spread across these areas, with some areas still under construction. We have developed a new assessment browser which makes detailed information about each Mobile Toolbox assessment available, and a new tool for designing and configuring and implementing protocols for remote administration in the Mobile Toolbox app. We have also invested in developing the app internal APIs that allow assessments from any developer to easily plug into the same framework. Pre-existing Sage Platform technology has been adapted to support access to study data by analysts, as well as data distribution to 3rd parties customized to MTB’s needs. These tools were piloted during Mobile Toolbox validation and norming studies starting in summer 2021, beta tested with NU clinical partners starting in early 2022, and can be made available to selected early adopters in 2023. We are continuously assessing and improving all areas of the platform based on user feedback, following principles of agile software development.
Highlights
Key Platform features already delivered or in development:
Researchers can:
Construct study schedules including remote study bursts based on clinical visits or other triggers that have unpredictable dates.
Setup schedules that have recurring repetitions and custom time windows for participants to complete assessments.
Customize communications with participants through branded app messaging and pre-set notifications.
Optionally collect passive background phone data.
Preview the participant app experience before launching a study.
Asynchronously enroll participants and manage their enrollment status.
Track high-level individual participant adherence and “day in study” through the lifetime of the study.
Access processed score data (full automation complete by Q1 2023).
Provide study metadata
Engage in a BETA for custom survey authoring (coming Q2 2023)
Participants can:
Freely download the public Mobile Toolbox app from either the Google Play Store or the Apple App Store.
Securely login with a de-identified code.
Optionally receive notifications and reminders.
Automatically receive scheduled assessments per researcher instruction.
Can choose to skip, pause, and continue assessments at any time.
Receive practice instructions for each assessment.
Actively opt-in or out of background data collection, review privacy information at any time, have access to general study and IRB contact information to request withdrawal.
Analysts can:
Access all item-level data in Parquet format as well as post-processed scores as spreadsheet downloads. Analysts can access these resources using R, Python, or web clients.
Access continuously updated data (completion in Q1 2023)
View aggregate statistics and user activity metrics per study.
View documentation on these tools.
Lowlights
Incomplete data delivery as data analysis Beta phase ramps-up
Bugs and glitches throughout the study life cycle
Usability gaps and unsupported scenarios
Editing a schedule
Login/logout
Participant device switching
Detailed adherence reporting
Limited communication and visibility into status and impact
Risks
High priority work items competing for scarce resources.
Infrastructural investments needed for QA, development and staging environments, and continuous maintenance.
No maintenance resources for product documentation updates.
Other updates
As MTB completes Beta and addresses remaining work items for major feature delivery and bug fixes, the focus turns to hardening processes and scaling up:
QA & bug management
Customer Support
Fixing usability gaps
Providing self-serve access to Mobile Toolbox
Producing public facing documentation and materials
Adoption ramp-up
📈 Issue review
Issue ID | Issue | Description | Impact | Severity | Reported date | Resolved date | Team lead(s) | Resolution |
|---|---|---|---|---|---|---|---|---|
1 | Missing DCCS (Shape-Color Sorting) data | Response time is not always captured, causing the score computation to break down | Confirmed impact in MGH and UCSD data. Likely all sites impacted. | 1 | 1/13/2023 | New release in QA 1/26/2023 | Keith A. |
|
2 | Missing Android data uploads that have background recorders turned on | This problem occurs in participants whose background recorders are turned on. Upload fails. | All sites impacted. iOS data not impacted. | 1 | 1/23/2023 | 1/25/2023 | Nate B., Sonia | Android data is not retrievable
|
3 | Missing Flanker (Arrow Matching) data | Error in score computation fails to produce a score. Source data exists, but needs to be recomputed | Confirmed impact in UCSF data. Likely all sites impacted. | 2 | 1/20/2023 | In Progress | Keith A. |
|
4 | Missing Android data uploads | Upload files are sometimes broken, data is stuck on participant’s phone. *Some normal upload attrition is expected in remote mobile studies due to network errors, device types, and other unpredictable phone states. | All sites impacted at low prevalence | 3 | Investigation in progress | Sonia, Nate B. |
| |
5 | App freezes between Vocab and Spelling assessments | Participant completes Vocab assessment, Spelling assessment freezes and participant can’t progress | One report from UCSD, but expected to impact all sites | 1 | 1/19/2023 | New release in QA 1/26/2023 | Keith A. |
|
6 | Missing Android background data | Validation rules are too restrictive, causing some data to fail processing. | Impact is to background recorder microphone data | 3 | 1/10/2023 | In progress | Sonia, Phil, Vijay |
|
7 | Schedule delivery stops working | New activities stop being delivered to participant’s phone after first two of five timepoints | UCSD | 2 | 1/17/2023 | Open | Sonia, Nate B. | Painful workaround exists to assign participants a new ID, merge final data. |
8 | PSM - iOS | Swipe gestures near the notification bar during the PSM assessment would unintentionally restart the assessment. Users of 1st generation iPhone SE also experienced ‘squished' images or other layout and keyboard issues. | All sites impacted, but not all participants, estimated 5% prevalence | 2 | February 2022 | CLOSED May 2022 | Erin M., Keith A. |
|
9 | Double exposure | Some participants received an extra exposure to some assessments | All sites impacted, but not all participants. Estimated 10% prevalence | 2 | 4/21/2022 | CLOSED 5/16/2022 | Sonia, Nate B. | Fix released so apps would fetch adherence record on sign-in |
10 | PSM Construct Validation data missing | PSM data is missing from CV study | cannot proceed with analysis. All PSM CV data is impacted. | 2 | 1/27/2023 | Open | Larsson | Adjacent pairs and exact match scores missing from Parquet. Need parquet code and scoring fix |
📈 User facing feature requests
Feature | Description | Complexity | Priority | Delivery forecast |
|---|---|---|---|---|
Login/logout capability | Allow participant and preview accounts to logout and login so that data state is retained and cache i s properly cleared without deleting and reinstalling the app | high | 2 | TBD |
Editable schedules | Allow changes to a schedule after a study has been launched | high | 1 | TBD. New studies can currently be manually constructed to have more flexibility. Existing studies will not be editable. New work will ensure that ALL new studies will automatically have more flexibility. |
Spanish language | Provide app and assessments to run studies ALL in english or ALL in spanish | medium | 2 | summer 2023 |
Self-service account setup | Allow public users to onboard to Mobile Toolbox to explore, and to launch studies with a verified identity | high | 1 | summer 2023 |
Self-service access controls | Allow Study Admins to invite users and have granular control over what team members can see and do with PII and study data | high | 1 | summer 2023 |
Custom Survey Authoring | Researchers can collaborate and construct custom survey questions that can be delivered to participants as part of their schedule of activities | high | 2 | BETA in Q2 2023 |
Detailed Adherence completion reporting | Study Coordinators can see the details of completion status and date for every assessment administered | medium | 1 | Q1 2023 |
Adherence Alerts page | Study Coordinators can view alerts when participants fall below adherence thresholds or other events | medium | 2 | Q1 2023 |
New assessments | Availability of additional assessments (various) | medium | 2 | Varies. Several new assessments already integrated and in queue for May 2023 |
Schema refresh | Update the schemas of assessments to optimize data processing | medium | 2 | Q2 - TBD |
Automated scoring | Scores are automatically processed on-device or on a daily basis to keep up with ongoing data collection | medium | 1 | Q1 2023 |
Public website updates | Include suggested language for grant submissions, sample data, FAQs, About Us bios, more detailed assessment information | medium | 1 | Q1 and Q2 2023 |
Customer support helpdesk | Ticketing and tracking system and knowledge base to scale up responses to issues reported by coordinators | high | 2 | Q2 2023 |
Product documentation | Update and publish product documentation | low | 3 | May 2023 |
Crashlytics tuning | Improve crash and log collection from the mobile App to more quickly characterize and respond to bugs. | medium | 2 | Q2 2023 |
Data samples | Provide data samples so potential users can demo the app and explore data output without having to launch a study | low | 3 | Q1 2023 |
Demographics upload UI | Study Coordinators can upload demographics metadata so it can be automatically incorporated into data processing and analysis | medium | 3 | TBD |
🏁 Features by quarter 2023
🎯 OKRs and supporting initiatives - Work in progress….
<Objective 1>
ON TRACK / AT RISK / OFF TRACK
Key results |
|---|
|
<Objective 2>
ON TRACK / AT RISK / OFF TRACK
Key results |
|---|
|
<Objective 3>
ON TRACK / AT RISK / OFF TRACK
Key results |
|
(.) Other updates
💚 Engineering/Operational Health |
|---|
|
|---|
Customer insights📣 Marketing Highlights |
|---|
🎭 Competitor insights |
|---|
0 Comments