The purpose of the study is to use a community-based participatory research strategy to refine and test the usability of an AI-powered smartphone app, AICaries, to be used by children’s parents/caregivers for dental caries detection in their children.
We used an integrate, design, assess, and share framework [
Components of AICaries smartphone app. The AICaries app prototype houses the AICaries tool for caries detection and other developed app components including maternal and child’s caries risk assessment, and education and clinic resources. AI: artificial intelligence.
AI-powered caries detection algorithm: our team has archived a data set with more than 100,000 high-quality intraoral photos including front and posterior teeth. Using a semiauto dental caries annotation software developed in house, trained and calibrated dentists have annotated approximately 40,000 individual tooth photos for dental caries using the International Caries Diagnosis System [
Maternal risk assessment and child risk assessment: an interactive caries risk assessment tool for mothers and young children. We modified the elements in the American Dental Association (ADA) Caries Risk Assessment system for less health-literate individuals. Through these assessments, mothers can visualize how daily oral hygiene behavior and diet could impact their and their children’s caries risk.
Validated perinatal oral health literacy index: to measure mother’s oral health literacy [
Maternal and children’s oral health education resources: we assembled a series of informative educational materials that provide appropriately timed information specific to pregnant women’s oral health importance, children’s tooth development, children’s oral hygiene, and diet recommendations.
List of available dental clinics that accept dental insurance for low-income groups (eg, Medicaid): built upon the list developed by the University of Rochester Eastman Institute for Oral Health (EIOH) and our community partner Healthy Baby Network via a New York State Department of Health grant “MICHC Oral Health Manual and Toolkit”
The AICaries app prototype houses the AICaries tool for caries detection and other developed app components including maternal and child’s caries risk assessment, and education and clinic resources.
This AICaries study will use the qualitative study design to assess the feedback and usability of the app component, app flow, and whether parents can take photo of children’s teeth on their own. The proposed human study will not assess the AI performance, which will be assessed in a separate study.
Specifically, we will conduct iterative moderated usability testing and refinement for the AICaries app that was developed and will be refined by our study team. Briefly, we will conduct the study in 2 steps, detailed in
AICaries usability test study flow.
Step 1 AICaries usability test: we will conduct individual usability tests among 10 pairs of end users (parents with young children: 10 parents and 10 children) to facilitate app module modification and fine-tuning using think-aloud and instant data analysis strategies. Only one study session was designed.
Step 2 AICaries field testing: we will conduct unmoderated field testing for app feasibility and acceptability. Field testing will be conducted in the real world among 32 pairs of parents with their young children (a total of 64 participants: 32 parents and 32 children younger than 5 years) to assess the usability and acceptability of AICaries, including assessing the number/quality of teeth images taken by the parents for their children and parents’ satisfaction. Two study sessions are designed (detailed in the section Study Procedures).
A total of up to 84 participants, 42 parents and 42 children younger than 5 years of age will be enrolled in the study. The study population will consist primarily of economically and socially disadvantaged parents 18 years and older (mainly mothers) and their young children (<5 years of age). We expect the participants to be 40% White, 45% Black or African American, 5% Asian, and 10% other race; we expect the composition of ethnical groups among the study sample is 80% non-Hispanic and 20% Hispanic. Since mothers are the primary caregiver for children, the ratio between female and male of the parent participants is expected to be 4:1, with 80% mothers and 20% fathers. Both male and female children will be recruited. Detailed inclusion and exclusion criteria are shown in
Inclusion criteria
Provide signed and dated informed consent
Have an Android smartphone that could be installed with the AICaries app
Willing to comply with all study procedures and be available for the duration of the study
Female or male, 18 years or older
Have at least 1 child aged between 1-5 years
Eligible for state-supported Medicaid type of insurance (eg, Medicaid, Blue Choice, MVP Option, Fidelis Medicaid; note, we are using the insurance eligibility to select low-income participants)
English speaking
Exclusion criteria
Participants who have decisional impairment and are incapable of making an informed decision about their participation in the study
Inclusion criteria
Female or male, aged between 1-5 years
Eligible for state-supported Medicaid type of insurance (eg, Medicaid, Blue Choice, MVP Option, Fidelis Medicaid)
Exclusion criteria
Orofacial deformity (cleft lip, cleft palate, oral-pharyngeal mass)
Down syndrome or other developmental disabilities
We will conduct face-to-face recruitment at the following two sites. The characteristics of these clinics make them suitable for the recruitment in the proposed study.
University of Rochester Medical Center (URMC)–EIOH Perinatal Dental Clinic: the URMC-EIOH Perinatal dental clinic was founded in March 2018 and is dedicated to serving socioeconomically disadvantaged pregnant women, women post partum, and young children. Since its founding, URMC-EIOH Perinatal Dental Clinic has provided dental care to more than 300 mothers and their children. A majority of the patients at the EIOH Perinatal Dental Clinic are eligible for state-supported Medicaid type of insurance, which is one of the eligibility requirements of the study.
URMC, Highland Family Medicine (HFM): URMC-HFM is one of the largest family medicine residency teaching practices in Monroe County, New York. HFM provides comprehensive prenatal care to the mothers and care to the baby from birth through childhood. Medical records showed that more than 600 children younger than 5 years (44% African American, 39% Caucasian, 7% Asian, and 10% other) are seen at HFM each year.
The AICaries step 2 field testing involves 2 in-person study sessions. The research assistant (RA) will phone or text to remind participants about the upcoming study visits approximately 3 days prior to the session. We will try to accommodate their social, economic, or physical barriers to coming for a study session, such as providing transportation and parking assistance.
We will conduct 10 in-person, one-on-one, 60-minute usability tests of the smartphone to facilitate app module modification and fine-tuning using think-aloud [
The parent will be asked to bring one child with them to the study site (URMC-EIOH), involving a quiet room equipped with Wi-Fi and videotaping, staffed by two trained RAs.
Following informed consent, the RA who is moderating will ask the parent to complete a brief survey (part 1 of
Prior to the testing, one RA will briefly demonstrate how to use the app and think-aloud strategy to express their feedback while testing the app.
The parent will then use the app while thinking aloud throughout the entire process.
The key usability tasks include navigation of the app interface; accessing and completing the ADA caries risk assessment; and, notably, taking usable (diagnostic) photographs of their child’s front teeth using the photo-taking interface.
The RA will provide encouragement and affirmation throughout, and only offer assistance if the parent hits a clear road block. The RA will provide facilitating statements (eg, “I noticed you did X. Can you explain why?”) or echoing comments. At the end of the session, the parent will be asked about suggestions for improvement.
Each session will be videotaped. The recording will start with participants’ verbal permission. We will inform the participants when the recording stops. The videotape will be reviewed by the study team immediately after the session using instant data analysis [
The unmoderated field testing allows the end users to interact with the AICaries app in their natural environment. We will enroll 32 pairs of parents and their young children, a total of 64 eligible participants, for step 2 of the AICaries study. There are two in-person study sessions.
At the first in-person session, approximately 30 to 45 minutes, following informed consent, the participants will complete a baseline questionnaire (part 1 of
The second in-person study session will take place after the 2-week app testing; the participants will come to the study site, complete a brief in-person survey (part 2 of
We expect to reach data saturation [
A sample size of 28 produces a two-sided 95% CI with a width equal to 0.3 when 80% of parents produce usable tooth images for their children using AICaries. Considering a dropout rate of 10%, we plan to recruit 32 parents and their children.
The transcribed data for the step 1 AICaries usability test and step 2 AICaries field testing will be coded with predetermined open codes. Thematic content will be further analyzed using categorizing and contextualizing strategies to understand the needed improvement for the AICaries app.
For the step 2 AICaries field testing, the System Usability Scale (SUS) will be used to assess the acceptance of the AICaries (part 2 of