🗞 Headlines
Our goal for the Mobile Toolbox Platform is to provide a complete end-to-end solution for the use of remotely-administered cognitive and other tests by researcher teams, without the need for any study-specific technology development. A complete solution must cover scenarios across the complete lifecycle of a study, and support research teams in:
Delivering timely, reliable, high-fidelity data to researchers
Providing well documented validated measures.
Learning about available mobile cognitive measures and designing appropriate digital research protocols.
Managing the collection of data from study participants, including support for enrollment, aggregate/high-level adherence tracking, and participant communications.
Providing secure transfer of de-identified data to analysts.
Disseminating data, results, and other findings from completed studies for forms that encourage reuse by other research teams, subject to FAIR principles.
To date, our efforts have been spread across these areas, with some areas still under construction. We have developed a new assessment browser which makes detailed information about each Mobile Toolbox assessment available, and a new tool for designing and configuring and implementing protocols for remote administration in the Mobile Toolbox app. We have also invested in developing the app internal APIs that allow assessments from any developer to easily plug into the same framework. Pre-existing Sage Platform technology has been adapted to support access to study data by analysts, as well as data distribution to 3rd parties customized to MTB’s needs. These tools were piloted during Mobile Toolbox validation and norming studies starting in summer 2021, beta tested with NU clinical partners starting in early 2022, and can be made available to selected early adopters in 2023. We are continuously assessing and improving all areas of the platform based on user feedback, following principles of agile software development.
| Tip |
|---|
Highlights |
Key Platform features already delivered or in development:
Researchers can:
Construct study schedules including remote study bursts based on clinical visits or other triggers that have unpredictable dates.
Setup schedules that have recurring repetitions and custom time windows for participants to complete assessments.
Customize communications with participants through branded app messaging and pre-set notifications.
Optionally collect passive background phone data.
Preview the participant app experience before launching a study.
Asynchronously enroll participants and manage their enrollment status.
Track high-level individual participant adherence and “day in study” through the lifetime of the study.
Access processed score data (full automation complete by Q1 2023).
Provide study metadata
Engage in a BETA for custom survey authoring (coming Q2 2023)
Participants can:
Freely download the public Mobile Toolbox app from either the Google Play Store or the Apple App Store.
Securely login with a de-identified code.
Optionally receive notifications and reminders.
Automatically receive scheduled assessments per researcher instruction.
Can choose to skip, pause, and continue assessments at any time.
Receive practice instructions for each assessment.
Actively opt-in or out of background data collection, review privacy information at any time, have access to general study and IRB contact information to request withdrawal.
Analysts can:
Access all item-level data in Parquet format as well as post-processed scores as spreadsheet downloads. Analysts can access these resources using R, Python, or web clients.
Access continuously updated data (completion in Q1 2023)
View aggregate statistics and user activity metrics per study.
View documentation on these tools.
| Warning |
|---|
Lowlights |
Incomplete data delivery as data analysis Beta phase ramps-up
Bugs and glitches throughout the study life cycle
Usability gaps and unsupported scenarios
Editing a schedule
Login/logout
Participant device switching
Detailed adherence reporting
Limited communication and visibility into status and impact
| Note |
|---|
Risks |
High priority work items competing for scarce resources.
Infrastructural investments needed for QA, development and staging environments, and continuous maintenance.
No maintenance resources for product documentation updates.
| Info |
|---|
Other updates |
As MTB completes Beta and addresses remaining work items for major feature delivery and bug fixes, the focus turns to hardening processes and scaling up:
QA & bug management
Customer Support
Fixing usability gaps
Providing self-serve access to Mobile Toolbox
Producing public facing documentation and materials
Adoption ramp-up
📈 Issue review
Issue ID
Issue
Description
Impact
Severity
Reported date
Resolved date
Team lead(s)
Resolution
1
Missing DCCS (Shape-Color Sorting) data
Response time is not always captured, causing the score computation to break down
Confirmed impact in MGH and UCSD data. Likely all sites impacted.
1
1/13/2023
New release in QA 1/26/2023
Keith A.
fixed logic in shared json schema to capture/validate response time going forward. Potential changes needed to Parquet processing and scoring.
evaluate cases to possibly reconstruct existing data scores from user interaction data. May need to adjust scoring code for one-off re-computation of existing data
2
Missing Android data uploads that have background recorders turned on
This problem occurs in participants whose background recorders are turned on. Upload fails.
All sites impacted. iOS data not impacted.
1
1/23/2023
1/25/2023
Nate B., Sonia
Android data is not retrievable
hotfix released
3
Missing Flanker (Arrow Matching) data
Error in score computation fails to produce a score. Source data exists, but needs to be recomputed
Confirmed impact in UCSF data. Likely all sites impacted.
2
1/20/2023
In Progress
Keith A.
fix scoring code, update to parquet code required
recompute scores for all sites (1-2 days) after updating parquet and scoring code
4
Missing Android data uploads
Upload files are sometimes broken, data is stuck on participant’s phone.
*Some normal upload attrition is expected in remote mobile studies due to network errors, device types, and other unpredictable phone states.
All sites impacted at low prevalence
3
Investigation in progress
Sonia, Nate B.
data prior to Android build 26 (mid-January 2023 release) has old MD5 hash on the upload files, causing upload failure even though the queue retries the upload.
potential to fix and retrieve data from phone when participant opens app
5
App freezes between Vocab and Spelling assessments
Participant completes Vocab assessment, Spelling assessment freezes and participant can’t progress
One report from UCSD, but expected to impact all sites
1
1/19/2023
New release in QA 1/26/2023
Keith A.
fix adds missing resources to repo
6
Missing Android background data
Validation rules are too restrictive, causing some data to fail processing.
Impact is to background recorder microphone data
3
1/10/2023
In progress
Sonia, Phil, Vijay
Loosen validation rules
re-run Parquet
7
Schedule delivery stops working
New activities stop being delivered to participant’s phone after first two of five timepoints
UCSD
2
1/17/2023
Open
Sonia, Nate B.
Painful workaround exists to assign participants a new ID, merge final data.
8
PSM - iOS
Swipe gestures near the notification bar during the PSM assessment would unintentionally restart the assessment. Users of 1st generation iPhone SE also experienced ‘squished' images or other layout and keyboard issues.
All sites impacted, but not all participants, estimated 5% prevalence
2
February 2022
CLOSED May 2022
Erin M., Keith A.
released fixes for gesture management over the notification bar
released fixes for zoom settings and layout
1st gen iPhone SE is an unsupported device and users will continue to have layout and keyboard issues with PSM and other assessments
9
Double exposure
Some participants received an extra exposure to some assessments
All sites impacted, but not all participants. Estimated 10% prevalence
2
4/21/2022
CLOSED
5/16/2022
Sonia, Nate B.
Fix released so apps would fetch adherence record on sign-in
10
PSM Construct Validation data missing
PSM data is missing from CV study
cannot proceed with analysis. All PSM CV data is impacted.
2
1/27/2023
Open
Larsson
Adjacent pairs and exact match scores missing from Parquet. Need parquet code and scoring fix
11
Missing spelling and vocab in UCSF score file
some iOS spell_theta, spell_se, vocab_theta, vocab_se scores missing in the UCSF score file
TBD
TBD
1/27/2023
Open
Sonia
| Tip |
|---|
Highlights - March 2023 |
Met criteria for Go-ahead to restart MGH study. Data collection has started and is at 100% fidelity
Fixed in production edge scenarios with schedule delivery and sign-in for mobile clients
Crashlytics monitoring deployed on iOS
Progress on defining weekly release cadence and criteria. Will finalize process over the next two sprint cycles
New devops viz tool identified an issue with Parquet exporter which has reduced ‘stuck’ data.
Baseline QA metrics defined. Next step to socialize, signoff and build dashboard
Centralized issue tracker up and in-use for active triage. Still needs refinement.
| Warning |
|---|
Lowlights - March 2023 |
Reports of PSM issues
Need further triage of remaining ‘stuck’ data
| Note |
|---|
Risks |
Need to close on background recorder decisions
| Info |
|---|
Other updates |
manual onboarding process is documented
have received new binaries from Monash group
met to provide feedback, Monash has some additional work to do
server work is complete to support growth of assessment library
upcoming bug bash of researcher UI portal
📈 Monthly summary
Where can I find information about…
*reach out to Sonia or Ann if you can’t access links
Issue Triage and Prioritization Scale
Data completion downloadable .csv
Jira Service Desk bug template - in progress
Jira Metrics - in progress
Jira Incident Manager - not started
Jira projects, backlogs, boards
ETL / Data Processing
Synapse Mobile Toolbox parent project
What are we working on? April 2023 our focus is on:
Description | Type | Expected output by end of month | Future work |
|---|---|---|---|
compute scores for DCCS (shape-color sorting) based on inferred Response Time for Android users where direct collection of RT is missing. This is to recover from a bug where RT was not collected from Android users prior to build 26. | User facing |
|
|
Detailed adherence report visible in the Researcher Portal UI that shows each assessment completion status, skipped status, and completion timestamp | User facing | In Researcher Portal UI, a coordinator can click into an individual participant and see the detailed completion status at the assessment level. Accessible via staging.studies.mobiletoolbox.org |
|
Finalize reqs and implementation plan for schema refresh to support upcoming ICAR measure validation studies and update of existing measures. | Planning |
|
|
Publication of internal viz tools (Jupyter notebook) for data monitoring | Infrastructure |
|
|
Updates to batch processing for 24hr scoring | Infrastructure | move to AWS batch jobs to address 10min timeout limitation | automate addition of new study IDs |
Add data pipeline changes to release notes | Process | Monthly release notes include downstream changes from ETL and scoring | respond to feedback |
Jira Service Desk | Process | Mechanism for internal and external collaborators to file bugs in a template that propagates through to staging for triage |
|
Data re-processing of data ‘stuck’ in parquet queue | Issue resolution |
|
|
Resolve PSM restart issues | Issue resolution |
| monitor in production |
Jira Service Desk Metrics | Process | Define and agree on metrics for bug response time | implement through Jira Service Desk |
Communications
UCSF response and instructions for PSM and for some duplicate data collections
📈 User facing feature requests
Feature | Description | Complexity | Priority | Delivery forecast | |||||
|---|---|---|---|---|---|---|---|---|---|
Login/logout capability | Allow participant and preview accounts to logout and login so that data state is retained and cache i s properly cleared without deleting and reinstalling the app | high | 1 | TBD | |||||
Turn off background recorders | part 1: turn off capability for researchers to optionally turn on recorders and default all study configs to OFF part 2: excise the implementation of the recorders on both Android and iOS | 2 | TBD | ||||||
Editable schedules | Allow changes to a schedule after a study has been launched | high | 12 | TBD. New studies can currently be manually constructed to have more flexibility. Existing studies will not be editable. New work will ensure that ALL new studies will automatically have more flexibility. | |||||
Spanish language | Provide app and assessments to run studies ALL in english or ALL in spanish | medium | 2 | summer earliest Fall 2023 | |||||
Self-service account setup | Allow public users to onboard to Mobile Toolbox to explore, and to launch studies with a verified identity | high | 1 | target late summer 2023 | |||||
Self-service access controls | Allow Study Admins to invite users and have granular control over what team members can see and do with PII and study data | high | 1 | target late summer 2023 | |||||
Custom Survey Authoring | Researchers can collaborate and construct custom survey questions that can be delivered to participants as part of their schedule of activities | high | 2BETA | in Q2 2023Deprioritized. Target beta in Q3/Q4 | |||||
Detailed Adherence completion reporting | Study Coordinators can see the details of completion status and date for every assessment administered | medium | 1 | Q1 April 2023 | |||||
Adherence Alerts page | Study Coordinators can view alerts when participants fall below adherence thresholds or other events | medium | 2 | Q1 2023 | New assessments | Availability of additional assessments (various) | medium | 2 | Varies. Several new assessments already integrated and in queue for May Q2 2023 |
Schema refresh | Update the schemas of assessments to optimize data processing | medium | 2Q2 - TBD | pending implementation plan for schema refresh. Ask is to complete for May/June start of clinical study | |||||
Automated scoring | Scores are automatically processed on-device or on a daily basis to keep up with ongoing data collection | medium | 1Q1 | 2023V1 complete | |||||
Public website updates | Include suggested language for grant submissions, sample data, FAQs, About Us bios, more detailed assessment information | medium | 12 | Q1 and Q2 Q3 2023 | |||||
Customer support helpdesk | Ticketing and tracking system and knowledge base to scale up responses to issues reported by coordinators | high | 2 | Q2 Q3 2023 | |||||
Product documentation | Update and publish product documentation | lowmedium | 3 | May 2023 | |||||
Crashlytics tuning | Improve crash and log collection from the mobile App to more quickly characterize and respond to bugs. | medium | 2 | Q2 2023 | |||||
Data samples | Provide data samples so potential users can demo the app and explore data output without having to launch a study | low | 3 | Q1 Q2 2023 | |||||
Demographics upload UI | Study Coordinators can upload demographics metadata so it can be automatically incorporated into data processing and analysis | medium | 3 | TBD |
🏁 Features by quarter 2023
🎯 OKRs and supporting initiatives - Work in progress….
<Objective 1>
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
Key results |
|---|
|
| Expand | ||
|---|---|---|
| ||
<Objective 2>
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
Key results |
|---|
|
| Expand | ||
|---|---|---|
| ||
<Objective 3>
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
Key results |
|
| Expand | ||
|---|---|---|
| ||
(.) Other updates
💚 Engineering/Operational Health |
|---|
|
|---|
📣 Marketing Highlights |
|---|
🎭 Competitor insights |
|---|