Aim 1 focuses on modeling speech, language, emotional expression, and interaction patterns during routine patient-nurse encounters (verbal communication) as markers of early cognitive decline in HHC. The primary analytic cohort includes non-Hispanic Black and non-Hispanic White patients receiving HHC services from VNS Health. These groups were selected because they are highly represented in the study setting, enable adequately powered comparisons within a single HHC system, and facilitate evaluation of model performance across racial groups—particularly important given well-documented disparities in dementia diagnosis and care for Black patients.

Potential participants are identified through EHR-based screening and clinician referral workflows within VNS Health. Prespecified EHR indicators are used to identify likely cases (eg, documented symptoms of cognitive decline) and likely controls (no evidence of impairment). Eligible patients are approached during an active episode of HHC. Recruitment is monitored to achieve representation of both racial groups and, when feasible, balance across key characteristics, age, sex as a biological variable, and education.

Eligible participants are aged ≥60 years, plan to receive VNS Health services during the study period, have sufficient English proficiency to communicate independently with HHC nurses, have adequate vision/hearing to complete cognitive testing, and can provide written informed consent. Patients are excluded if they (1) are unable to communicate independently with the HHC nurse in English and (2) have speech or language disorders due to neurological conditions other than MCI-ED (eg, Parkinson disease or seizure disorders). Full eligibility criteria are provided in Multimedia Appendix 1.

We use a 2-stage approach to identify patients for case and control groups. First, we identify potential cases using available ICD-10 (International Statistical Classification of Diseases, Tenth Revision) diagnoses in the EHR (ICD-10 G31.84 for MCI) and identify potential controls as patients without documented cognitive impairment. Second, all consented participants without an existing MCI diagnosis complete cognitive assessments—the Montreal Cognitive Assessment (MoCA) [30,31] and Clinical Dementia Rating (CDR) [31]—in their homes, administered by a trained research assistant who audio-records responses to support final group assignment.

A study clinician with expertise in cognitive impairment detection reviews the recorded cognitive assessments together with relevant clinical context (medical history and nurse assessment information from OASIS) to confirm group assignment. Based on prespecified criteria, participants are classified as MCI-ED cases when findings are consistent with early cognitive impairment (anticipated CDR 0.5-1 and MoCA ~16-25, with consideration of EHR evidence when available) and as cognitively normal controls when findings are within normal limits (CDR 0 and MoCA ≥26), and there is no EHR evidence of cognitive impairment. Participants meeting criteria for moderate to severe impairment (eg, CDR 2-3 or MoCA <16) are excluded because the protocol focuses on MCI-ED.

After group allocation, patients in both the case and control groups are invited to provide additional consent for the next phase of Aim 1, which includes audio-recording routine patient-nurse encounters.

For each enrolled participant, 3 routine patient-nurse encounters are audio-recorded during the HHC episode of care. Recording multiple encounters provides repeated observations to capture within-person variability in speech, language, and interactional patterns across visits, while minimizing participant and clinician burden. When both patient and nurse provide consent, a trained research assistant attends the visit and operates a Saramonic Blink audio-recording device [32], minimizing burden on clinical staff. This portable device, with dual wireless microphones that attach to clothing, provides clear speech transmission to devices like an iPod and offers dual-channel storage.

Audio-recorded encounters are linked to EHR data extracted from the VNS Health system, including the OASIS [33,34]—a federally mandated HHC assessment capturing patient health status, functional status, and living arrangements—as well as supplemental structured data (eg, medications) and free-text clinical notes. Free-text notes include visit notes, documenting each nurse encounter, and care coordination notes, capturing communications with other clinicians, physicians, and family members.

Early cognitive decline affects multiple aspects of spoken communication, including speech motor control, language organization, emotional expression, and social interaction. In HHC settings, these changes are expressed during spontaneous patient-nurse verbal communications rather than structured speech production tasks (eg, reading task). Building on our preliminary feasibility and pilot work, Aim 1 focuses on systematically modeling these communication patterns using an automated speech analysis system. The objective is to extract complementary acoustic, linguistic, emotional, and interactional features from naturally occurring patient-nurse communications that may signal MCI-ED. The analytic framework for Aim 1 consists of 5 components, summarized in Table 1, which together capture core dimensions of speech production and interaction relevant to cognitive decline.

For reproducibility, all speech- and interaction-based parameters extracted in Aim 1 are explicitly specified in Table 1, organized by analytic component (phonetic motor planning, emotional expression, syntactic and semantic language organization, and patient-nurse interaction). Table 1 provides the operational definition and measurement domain for each parameter. Acoustic features are computed using established toolkits (OpenSMILE [78] and PRAAT [38]); linguistic features are derived from automatically transcribed speech using NLTK, LIWC [79], and distilled RoBERTa; and interactional features are computed from speaker-labeled timestamps generated by Amazon Web Services (AWS) Transcribe.

Impairment in phonetic motor planning is a well-documented consequence of neurodegenerative disorders, including MCI-ED, and manifests as reduced articulation precision, altered speech rhythm [44,80,81], and increased disfluency. To characterize these changes, we analyze acoustic parameters across six domains (Table 1, component 1): (1) speech fluency [45,46,82], (2) rhythmic structure [48,49,83], (3) frequency and spectral characteristics [45,84,85], (4) voice instability [45,69,86], (5) voice quality [87-89], and (6) voice intensity [45,90]. These measures quantify temporal and spectral aspects of speech that reflect the patient’s ability to plan and execute vocal motor actions.

Alterations in emotional expression often develop alongside cognitive decline and can negatively affect communication quality and interpersonal interaction [91,92]. Emotion is conveyed both through nonverbal vocalization and semantic content [93-96]. To model vocal expression of emotion, we use the GeMAPS [39], which captures affect-related changes in autonomic arousal and vocal musculature via frequency-, energy-, and spectral-domain parameters (Table 1, component 2). To capture the semantic expression of emotion, we extract linguistically encoded emotional indicators using the LIWC dictionary. Emotion-related linguistic markers (eg, sadness and anxiety) have been associated with cognitive dysfunction [97] and adverse health outcomes [98].

Language impairment in MCI-ED is characterized by reduced lexical diversity, simplified syntax [44,97,99], word-finding difficulties [44,100], and impaired memory-related discourse [44,101], which together contribute to reduced coherence. We model language organization using features in four domains (Table 1, component 3): (1) lexical richness [64,65,102], (2) syntactic complexity and grammaticality [66,103], (3) semantic fluency [104], and (4) patient recall ability [73,74]. Linguistic features are derived from automatically transcribed speech and processed using standard NLP tools. In addition, we will use distilled RoBERTa [42] to generate contextual language representations that capture semantic relationships beyond surface-level lexical features.

Cognitive impairment also affects social communication, including turn-taking, timing, and responsiveness [105,106]. Patients with MCI-ED may show recurrent interactional patterns, such as longer turns, delayed responses, or reduced interactivity [107]. To capture these phenomena, we will model patient-nurse interaction using easily measurable dialogue features [76,77,108] (Table 1, component 4), including patient turn counts, dialog interactivity, turn density, turn duration, and relative timing of turns. These interactional measures reflect how patients engage with clinicians in real-world care encounters and provide information beyond speech content alone.

Acoustic parameters for components 1 and 2 will be computed at the utterance level (continuous blocks of uninterrupted patient speech) using OpenSMILE [37] and PRAAT toolkits [38]. Verbal communications are automatically transcribed using AWS Transcribe, after which linguistic features for component 3 are computed at the encounter level using the NLTK toolkit and distilled RoBERTa. Interactional features for component 4 are derived from AWS Transcribe metadata, including speaker labels and time stamps. All extracted features will be aggregated at the patient level for downstream integration with clinical data in Aim 3.

Many clinical indicators of MCI-ED—including symptoms, lifestyle risk factors, and communication difficulties—are documented in free-text clinical notes or expressed during patient-nurse conversations [109], but are not captured in structured EHR fields [6,110]. Aim 2 uses LLMs to systematically extract this information from HHC clinical notes and transcripts of patient-nurse verbal communication. The goal is to convert unstructured text into standardized, patient-level variables that can be integrated with speech features (Aim 1) and structured assessment data (OASIS) for multimodal screening (Aim 3).

For reproducibility, all MCI-ED–related information identified in Aim 2 is defined using an information schema summarized in component 1 (Information targets and schema). The schema specifies the target information families (clinical symptoms, lifestyle risk factors, and communication deficits) and associated attributes, and is applied consistently across human annotation and LLM-based identification. MCI-ED–related information is normalized to standard clinical terminologies—Unified Medical Language System (UMLS) [111] concepts, when available—and represented in a structured patient-level format, supporting reproducible integration with Aim 1 features and OASIS variables.

We define a schema for the 3 MCI-ED–related risk factor categories, including clinical symptoms, lifestyle risk factors, and communication deficits. For each identified item, the system records (1) the related terms; (2) a normalized clinical concept identifier when available (UMLS [111] concepts); (3) clinically relevant attributes: assertion (present/absent/possible), temporality (current/historical), and experiencer (patient/caregiver); (4) severity and frequency; and (5) duration. The schema is used consistently across clinical notes and transcripts of patient-nurse communication.

Using the information schema defined in component 1, we create a human-annotated gold-standard dataset to support LLM adaptation and evaluation. The schema specifies the target information categories (clinical symptoms, lifestyle risk factors, and communication deficits) and associated attributes, including assertion, temporality, and experiencer.

Two trained nurse annotators independently annotate a stratified sample of HHC clinical notes and encounter transcripts according to this predefined schema. The annotation sample is stratified by race, sex, and visit type to ensure representation of diverse documentation patterns. Interannotator agreement is assessed using Cohen κ [112] for each information category and attribute, calculated on double-annotated samples prior to adjudication. Discrepancies are resolved through adjudication meetings to produce a finalized gold-standard dataset. The annotated corpus is subsequently partitioned into training, development, and test sets to enable LLM fine-tuning and unbiased performance evaluation.

We use a hybrid LLM-based strategy that combines prompted extraction and instruction tuning [113], both aligned with the information schema (component 1) and supervised by the human-annotated gold-standard dataset (component 2). First, prompted extraction (baseline): as a baseline approach, we apply structured prompts that explicitly define the 3 information families (clinical symptoms, lifestyle risk factors, and communication deficits) and required attributes (assertion, temporality, and experiencer). Prompts instruct the model to produce schema-compliant JSON outputs and to provide a supporting text span for each identified item to ensure evidence-grounded extraction. This prompted approach provides an interpretable, rapidly adjustable method for early experiments and error analysis. Second, instruction tuning (schema-guided supervised adaptation): to improve reliability on HHC-specific language and documentation patterns, we perform instruction tuning using the training split of the human-annotated gold-standard dataset. Training examples pair the input text (note or transcript segment) with the target output formatted as schema-compliant JSON, including the identified item type, attributes (assertion, temporality, and experiencer), and supporting span. This teaches the model to follow the extraction instructions consistently and to produce outputs that match the schema across diverse note styles and conversational phrasing. Instruction tuning is implemented using parameter-efficient methods (eg, Low-Rank Adaptation [114]/ Quantized Low-Rank Adaptation [114,115]) to reduce computational burden in the secure environment.

For each identified item in the text, we normalize the item (mention) to a standardized clinical concept identifier (UMLS, when available) using a controlled vocabulary lookup supplemented by string similarity matching for common variants. When multiple items map to the same concept within a document, we merge them into a single record while preserving the schema attributes (assertion, temporality, experiencer, and—when present—severity and frequency/duration) and retaining the supporting text spans for traceability. We then aggregate document-level outputs to the patient level to produce predictors for Aim 3. Patient-level variables summarize the presence of each normalized concept and its attributes across the patient’s available HHC notes and encounter transcripts. The final deliverable is a structured patient-level table of normalized MCI-ED–related symptoms, lifestyle risk factors, and communication deficits for integration with OASIS data and Aim 1 speech and interaction features.

We evaluate performance against the human-annotated gold-standard dataset using (1) span-level precision/recall/F₁ under exact and overlap matching, (2) attribute performance (assertion, temporality, experiencer, and severity/frequency when applicable), and (3) concept normalization accuracy. We report results overall and stratified by race, sex, and age group to monitor for systematic performance differences. We conduct routine error analysis (eg, common false positives from templated note language, negation errors, or transcript artifacts) and use findings to refine prompts, update normalization resources, and adjust fine-tuning settings. Low-confidence outputs are flagged for targeted review during development to guide iteration and reduce systematic errors.

The final Aim 2 deliverable is a structured, patient-level dataset summarizing clinical symptoms, lifestyle risk factors, and communication deficits identified from HHC notes and transcripts, including normalized concept identifiers and clinically meaningful attributes. These variables are used as candidate predictors and complementary signals in the multimodal screening algorithm developed in Aim 3.

The objective of Aim 3 is to develop and evaluate a multimodal screening algorithm for identifying HHC patients with MCI-ED. The algorithm integrates complementary information from three routinely generated data sources: (1) speech and interaction features extracted from patient-nurse communication (Aim 1), (2) MCI-ED–related information identified from clinical notes and transcripts (Aim 2), and (3) structured assessment data from the OASIS.

Input variables include (1) acoustic, linguistic, emotional, and interactional features derived from patient-nurse verbal communication (Aim 1); (2) normalized clinical symptoms, lifestyle risk factors, and communication deficits identified from clinical notes and encounter transcripts (Aim 2); and (3) structured OASIS variables capturing sociodemographic characteristics, diagnoses, medications, functional status, and related clinical information.

Prior to model development, we assess data quality and address missingness, inconsistency, and integrity issues using a predefined data quality framework [116]. Continuous variables are transformed and scaled as appropriate to ensure comparability across modalities. To reduce dimensionality and mitigate overfitting in the presence of a large number of candidate predictors, we apply Joint Mutual Information Maximization [117] as a feature selection method. Joint Mutual Information Maximization is selected for its suitability in small to moderate sample settings with high-dimensional data, where it balances relevance to the outcome with redundancy among features.

We develop screening models using supervised discriminative machine learning algorithms that are appropriate for tabular and multimodal clinical data, including logistic regression, support vector machines (SVMs) [118], and ensemble tree-based methods [119-122]. These models are chosen for their interpretability, robustness, and reduced risk of overfitting in clinical datasets. Multimodal integration is performed by combining features from speech, clinical text, and OASIS data within a unified modeling framework. Models are trained to estimate the probability of MCI-ED at the patient level. Temporal aspects of speech-derived features are summarized at the patient level prior to modeling, rather than modeled using complex sequence architectures, to maintain feasibility and stability given sample size considerations.

Model training and hyperparameter tuning are conducted using stratified nested cross-validation, with inner loops for parameter selection and outer loops for performance estimation. Model performance is evaluated using the area under the receiver operating characteristic curve (AUC-ROC) and area under the precision-recall curve, along with calibration measures to assess agreement between predicted risk and observed outcomes. To evaluate equitable performance, we assess model metrics stratified by race, sex, and age group. Fairness-related measures, including group-wise differences in sensitivity and specificity and calibration across subgroups, are examined. When systematic performance differences are observed, we explore mitigation strategies such as reweighting or threshold adjustment and reassess model performance.

In the final step, the selected model is evaluated on an independent validation dataset to provide an unbiased estimate of performance. The screening algorithm produces a patient-level risk score indicating the likelihood of MCI-ED, which is intended to support clinical awareness and referral for further cognitive evaluation rather than serve as a diagnostic tool. The resulting algorithm and associated feature sets are prepared for downstream evaluation of clinical use and integration into HHC workflows.

Acoustic and temporal speech features were encoded using SpeechDETECT, including parameters related to phonetic motor planning (Table 1, component 1). Vocal emotion-related cues were encoded using GeMAPS (Table 1, component 2). Linguistic features included handcrafted measures capturing lexical richness, syntactic complexity, and semantic/fluency markers (eg, repetition and filler words; Table 1, component 3), and psycholinguistic indicators were extracted using LIWC 2015. In addition, we evaluated pretrained transformer language models for transcript-based representations.

When modeling patient speech alone, DistilBERT achieved the strongest performance among evaluated BERT-based models (F₁=69.39; AUC-ROC=69.36). For clinical notes, BioClinicalBERT yielded the best performance among evaluated language models (F₁=64.29; AUC-ROC=69.17). Among traditional classifiers, a linear SVM performed well using patient speech features (F₁=75.0; AUC-ROC=75.94). Models using structured EHR/OASIS variables achieved their best performance with logistic regression (F₁=75.56; AUC-ROC=79.70).

Incorporating nurse speech and interactional measures (Table 1, component 4) resulted in improved discrimination (SVM F₁=85.0; AUC-ROC=86.47), suggesting that patient-nurse interaction captures complementary information beyond patient speech alone.

In multimodal analyses integrating speech features, interactional measures, and structured EHR/OASIS variables, the SVM achieved the highest overall performance (F₁=88.89; AUC-ROC=90.23). Examination of model contributions suggested that reduced lexical diversity, longer patient pauses, increased nurse dominance in conversation, selected psycholinguistic markers, and specific EHR variables (eg, non–insulin-dependent diabetes, pressure ulcers, and living alone) contributed to discrimination.

Overall, these results support the feasibility of extracting and integrating multimodal signals for MCI-ED screening in HHC. Final model evaluation, subgroup performance assessment, and fairness analyses will be conducted using the prespecified validation procedures after completion of recruitment and the finalized analytic dataset.