To gain a deeper understanding of phenomena in their context of use, a case study methodology is recommended [50]. The case study will be guided by Yin’s [51,52] approach, using 5 components: research questions, propositions or purpose, units of analysis, determination of how the data are linked to the propositions, and criteria to interpret the findings. The first step to developing ORALS is to gain a deep understanding of existing approaches to recording visits (see Table 1). We will conduct site visits to the Ryan Family Practice, Ludington, MI; University of Texas Medical Branch Cancer Center Victory Lakes, League City, TX; and Barrow Neurosurgical Institute, Phoenix, AZ. Each of these sites has a unique approach to recording visits and thus important insight to share that will help guide the development of ORALS.

Our purpose is to gain insight from pioneer clinics to guide the development of ORALS. In stages 2, 3, and 4, we will use case study findings to develop ORALS and test the software in a human-computer interaction laboratory and in clinic settings. The ic3d team and patient representatives will be involved in stage 1 by guiding the study design and interpreting findings. Figure 2 illustrates the critical steps, from the challenges of accurate physical recording to creating a secure data store for safe multiple use access.

We will adopt a multiple case design including 3 sites where recordings of medical visits are shared with patients [51]. Each recording system represents a case (ie, the unit of analysis). Embedded in each case will be clinicians who support recording and those who do not, patients and their families, and clinic management and administrative staff.

This research has received Institutional Review Board (IRB) approval from the Committee for the Protection of Human Subjects at Dartmouth College (Study #29380).

Participants will include patients, family members, clinical staff, management, and administrators. We will use key contacts to arrange interviews and include clinicians who have chosen not to offer patients recordings. Interviewees will be 18 years or older and able to communicate in English. We plan to conduct a minimum of 6 interviews per stakeholder group at each site until data saturation is reached [53].

Audio recordings will be transcribed verbatim. Data analysis will take place simultaneously with data collection, which, in turn, will assist in the iterative development of the interview guides. We will use a framework approach to analyze the data. This approach consists of 6 steps: familiarizing ourselves with the data, identifying a framework, and indexing, charting, mapping, and interpreting the data [54,55]. The research questions will provide the analytic framework with thematic comparisons across cases. Identifying recurring themes across sites is a strength of the multiple case study design. Two researchers will apply an initial codebook, and they will meet and include new codes into a revised codebook before conducting a secondary assessment of interview data. Emergent codes will be added to existing codes. Codes, memos, and short narrative summations of data will be entered into ATLAS.ti.

Case study information will determine the scope and functionality of Open Recording Automated Logging System (ORALS) and the user interface designed using usability engineering [56].

Usability engineering will provide iterative, formative feedback from ORALS target users. Vinter et al [57] reported that usability errors dominate 60% of software problem reports. Myers and Rosson [58] reported that about half of code development is devoted to user interface design. To mitigate these challenges, early stakeholder engagement in ORALS design before initial software development and iteratively thereafter is crucial.

A member of the interaction design team will be embedded in stage 1 of the case study. We will analyze which moments during the clinic visit matter most to patients and their families (RQ4, Table 1) to arrive at a standardized set of tags for the user interface (eg, test result, treatment plan). This set will be categorized into terms such as specific medical conditions or treatments and topics such as treatment decisions and plans. Case study data will also support the development of a set of activity scenarios and associated tasks for participants to complete using prototypes. This will include tasks such as authenticating into ORALS to allow the secure linkage of recordings to patients, controlling recordings (eg, starting and stopping), finding a recent appointment, finding tagged information in the recordings, and other derived tasks.

Paper prototyping activities will guide early software development cycles by engaging potential ORALS users. This work creates an interactive interface by using paper, pens, markers, Post-It notes, and other media to mock up multiple screens and overlays for users [59]. As users complete tasks, a member of the design team responds by changing the paper interface screens and overlays. This prototyping process offers the benefits of getting users involved in the design early, when there is evidence that users are much more likely to offer feedback because the interface appears more malleable and capable of change.

The design team will develop a usability specification for the set of tasks and define the expected interaction paths and the amount of time to complete each task. The usability specification will be used as a baseline for determining changes. Survey questions will be used to collect demographics, preassessment of user expectations of the system, posttask design feedback, and postassessment of user impressions of the system.

We anticipate 3 rounds of ORALS usability evaluation with stakeholder users: (1) paper prototyping as described, (2) formative evaluation with an early software prototype midway in the software development cycle, and (3) summative evaluation at the end of the development cycle to understand how the ORALS will work in the field and guide final changes.

Facilitator-led paper prototyping activities will take place in person with prospective users in the human-computer interaction lab, ic3d lab, Geisel School of Medicine; observers will take notes. For formative and summative usability evaluation, activities can be conducted in person or facilitated remotely using Web conferencing tools. We will use a software usability evaluation product called Morae (TechSmith Corp), which supports screen recording of software use and a log of mouse and keyboard interactions. Each session will be designed to last about an hour.

We will work with a minimum of 6 potential users, patients, and clinicians in each design round. Participants will be identified both from local clinics and the in-clinic test site at UTMB. Participants will receive $25 for their participation.

Paper prototyping sessions will provide data through observational notes, think-aloud statements, and resulting design artifacts from changes in each session. Software evaluation sessions will collect performance measures, think-aloud statements, survey responses, and recordings of the user interactions with the system.

The primary evaluation goal is to refine the user interface design, providing a layout, interface controls, and workflow for creating and reviewing recordings in ORALS. A member of the research team with expertise in user design (CG) will analyze the results of the paper prototyping sessions for themes. CG will also review recordings of the software session for usability issues, including deviations from the expected interaction paths. Finally, CG will analyze performance measures, coded sessions, and survey responses. Modifications and enhancements to the interface will be based on commonality of themes.

After the first round of usability evaluation through paper prototyping, we will create a set of software requirements for the initial ORALS prototype, and we will modify and improve the user interfaces for the software prototype based on feedback from the second and third rounds of usability evaluation.

ORALS software will be developed and implemented as a secure Web-based system consisting of user interfaces (stage 2) supported by recording, transcribing, and automated tagging software (stage 3). User-facing portions (stage 2) of the software will be implemented in the Ruby on Rails Web development framework; the automated-tagging portion of the system will be written in the Python language (see Figure 3). The automated tagging software will have 2 components: speech recognition and tag identification. Speech recognition software has improved considerably in accuracy over the past 2 decades and is used in clinical settings for medical transcription [61]. We will use voice-to-text and text analytics approaches from IBM Research for speech-to-text transcription. For tag identification, we will use Weka [62] and SciKit [63], both open-source and widely used machine learning libraries.

In this stage, our goal is to take the text output from the speech recognition component and develop machine learning methods that will support accurate tag identification.

As noted in stage 2, we will formalize a list of standard tags that are either terms or topics. To capture the context of terms and phrases in text in our tagging system, first we apply a named-entity recognition system on input text to identify medical concepts and their corresponding classes. For this named-entity recognition task we will use Apache cTAKES information extraction framework [64] and Unified Medical Language System (UMLS) [65]. cTAKES is open-source software that identifies terms and phrases in text that correspond to UMLS concepts in high-level classes such as anatomy, symptom, procedure, disorder, and drug. To identify topics of interest within the transcriptions, we will use a supervised machine learning classifier based on a support vector machine (SVM) framework [66]. The text, extracted concepts, and their associated classes will be provided as input features to the SVM classifier. This classifier will identify the texts that are most related to a topic based on these features. If the accuracy of the SVM approach is less than 90%, we will try the alternative approach of Hidden Markov Model (HMM) [67]. We will implement the automated tagging software using the Weka, Natural Language Toolkit, and SciKit libraries in Python.

We have access to a set of 120 existing patient audio recordings from a primary care setting and will apply the speech-to-text software to translate the recordings into written text using voice-to-text and text analytics approaches from IBM Research. The accuracy of the automated speech-to-text transcriptions will be measured as the percentage of words correctly transcribed. This will be measured by first formatting all transcripts into a one-word-per-row format, then applying the OpenDiff tool to count the number of differences between the manual and software-generated transcripts. Percent correct will be expressed as the number of differences divided by the total number of words per transcript. We expect that an accuracy rate of 90% will be needed in order to have acceptable transcription performance for automated tagging.

Four medical students will act as reviewers and will manually annotate the tags of interest in the text. Tags of interest will be derived through case study interviews with patients, caregivers, and clinicians. Students will work in pairs and will review 60 recordings each. An annotation guide will be created with a primary care clinician, who will also assist in training medical students in annotation. We will measure the Cohen's kappa coefficient between the 2 sets of reviewers and assess the interrater reliability, with a target kappa of >.8 [68]. Discrepancies will be resolved by consensus with other members of the research team.

To automatically identify the tags related to terms, we will apply an SVM classifier to the transcriptions and their corresponding annotations using Apache cTAKES named-entity recognition method. We use 10-fold cross-validation to evaluate the classifier against the manual annotations (our reference standard). In this evaluation, we measure accuracy, precision, recall, and F1 score of our method. If the accuracy is below 90%, we will consider and evaluate the performance of other machine learning approaches for topic identification in text, such as an HMM.

The usability testing and machine learning efforts in stages 2 and 3 will overlap so that the ORALS system can be developed and tested in an iterative agile approach. After paper prototyping is completed in stage 2, we will implement the initial Web-based prototype, which will include the manually tagged recordings, for the planned formative evaluation. For the planned summative evaluation, the ORALS system will include patient recordings that have been automatically tagged with terms and topics.

We will undertake 2 steps to ensure the privacy of the patient and physician participants who provided the initial recordings in our testing with other patient participants in usability evaluation. First, the project team will rerecord the transcription for the subset of recordings used in usability testing so that recording will be in the voices of project team members and not the original patient or clinician. Second, we will ensure that all protected health information defined by the Health Insurance Portability and Accountability Act (HIPAA) is removed from these recordings.

We will develop ORALS as a secure stand-alone Web-based application that can be accessed via the Internet from a home computer. The system will be hosted on HIPAA-compliant servers within Dartmouth College. For deployment and testing in stage 4, patient participants will access their individual recordings through authenticated log-in over an https secure online connection, ensuring the recording is linked to the correct patient. In addition, we will design audiorecording control interfaces that can be used from a native Android app on any compatible device. The interface will allow for the collection of the recording in the exam room via the Android device and the uploading of the recording in an encrypted manner directly to the ORALS database server.

The final stage will involve rapid cycle testing of ORALS using the Model of Improvement approach, which consists of 3 improvement questions and Plan-Do-Study-Act (PDSA) cycles [69] (described below). The Centers for Medicaid and Medicare Innovations recommends this approach for testing innovative interventions. PDSA cycles are iterative small-scale tests of change consisting of a hypothesis for improvement (Plan), study protocol to implement and test the proposed improvement (Do), analysis and interpretation of the data (Study), and iteration of what to do next based on the study (Act) [69-72].

Our aim is to produce a version of ORALS that is widely accepted by clinicians and used by patients and their families. Each cycle will lead to refinements of ORALS, improving performance, usability, and acceptability. The introduction of ORALS will be considered a success if all of the following occur:

We will test ORALS in the 3 clinics of UTMB Family Medicine—Dickinson, Island East, and Island West—that are served by approximately 40 clinicians, with 15 to 25 daily patient visits per clinician. We will focus testing in one of these clinics, to be decided in conjunction with UTMB in year 2. UTMB has a dedicated quality improvement team with experience in rapid cycle testing that will assist in this stage.

Patients aged 18 years and older with access to the Internet who can communicate in English will be eligible for inclusion. Clinicians from the selected UTMB Family Practice clinic will be eligible for inclusion.

We will transcribe interviews and identify commonly reported issues and solutions in each cycle. A descriptive analysis of ORALS usage will be conducted with continuous data presented as means and standard deviations and categorical data as proportions and ranges. Paired t tests will compare PAM-SF and PCS scores before and after ORALS use. PAM-SF will be our primary outcome of interest. We will analyze PAM-SF on a continuous scale; scores range from 0 (low activation) to 100 (high activation). On average, interventions to promote engagement have required a 3- to 6-point change in PAM-SF to be a minimally important clinical difference [76,77]. With 200 patients, we will have 80% power to detect a 3-point change in PAM-SF with an alpha level of 0.05. We will aim for a minimum of 200 patients (25 patients per clinician per week) in the final cycle of PDSA testing. All data analysis will be conducted using Stata 14 (StataCorp LLC).