Chronic pain is a pervasive health issue that leads to poor patient outcomes and high health care costs. It affects one-fourth of adults in the United States, ranks as a leading cause of disability, and costs the nation over US $700 billion annually [1-3]. The prevalence and severity of chronic pain among Veterans of the US military are disproportionately high compared with the general population [4]. As such, chronic pain management is a high priority for health care providers and systems, including the Veterans Health Administration (VA) [5].

As pain is a subjective, multidimensional experience, its assessment is complex but central to evaluating treatment efficacy and effectiveness. The IMMPACT (Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials) consensus recommends assessment of multiple core outcome domains for chronic pain treatment trials, including pain (with its intensity, quality, and temporal aspects), physical functioning, and emotional functioning [6,7]. Assessing these domains, whether capturing the subjective experience of pain severity or outcomes concerning pain’s interference with functioning (eg, general activity) and well-being (eg, enjoyment of life), relies heavily on patients’ self-reports.

While electronic health records (EHRs) are increasingly integrating self-reported pain data, with VA leading the shift from basic “fifth vital sign” scores [8] to comprehensive measures [9,10], systematic implementation of multidimensional patient-reported pain measures is rare [11]. The commonly used Numerical Rating Scale (NRS) for pain intensity, a single-item numeric measurement, was intended for clinical screening and has limited utility for longitudinally tracking outcomes or capturing pain’s broad interference in patient functioning and well-being [6,9,12-14]. Although multidimensional instruments like the Brief Pain Inventory, PEG (Pain, Enjoyment, and General Activity) scale, and Patient-Reported Outcomes Measurement Information System (PROMIS) measures offer richer insights, they are typically administered in specific research settings or specialty clinics rather than in routine care [13]. As pain is often discussed with providers as part of routine care, descriptions of pain interference, treatment plans, and associated progress may offer an alternative way to monitor the multidimensional aspects of pain management, supplementing targeted efforts to capture patient-reported pain outcomes.

As part of a prior quality improvement initiative, VA conducted a nationwide survey across 18 VA medical centers participating in the Whole Health Flagship initiative, collecting patient-reported outcomes (PROs) from Veterans initiating complementary and integrative health (CIH) therapies [15,16]. The survey used measures from the PEG scale, a validated and widely used instrument to assess pain severity and its interference with functioning and quality of life [13]. Over the 2-year period spanning from March 2021 to March 2023, patient surveys were used to track longitudinal PROs from 3725 patients. Although providing valuable patient-reported data, surveys are resource-intensive, suffer from low response rates, and are difficult to scale for continuous population-level monitoring.

An alternative approach to obtaining outcomes equivalent to PROs at scale is through clinical notes, which contain narratives of patient reports and patient-provider discussions. Recent advances in machine learning and natural language processing (NLP) have generated techniques to extract pain and self-reported outcomes from narrative text data [17,18]. Prior works have leveraged NLP methods to achieve objectives such as constructing an ontology of pain over diverse attributes [19], identifying attributes of pain interference [20], and using fine-tuned large language models (LLMs) to parse musculoskeletal pain features from unstructured clinical notes [21]. Notably, recent studies using advanced Transformer-based deep learning models [22] found that model-extracted features of pain experience were in agreement with domain experts [20,23].

Despite the growing body of literature demonstrating the feasibility of using NLP to extract outcomes similar to PROs of chronic pain, several questions remain. First, documentation patterns for pain vary significantly across clinics and patient sociodemographic characteristics [24]. How such variation manifests within the heterogeneous VA population and whether it introduces systematic bias into algorithmic models remains largely unknown. Second, while the broad notion of pain is frequently documented, documentation of concepts related to subjective aspects such as patient-reported functioning is reportedly more sparse compared with aspects like severity or site [25,26]. A quantitative understanding of how completely clinical text data captures pain severity and functional interference domains remains underexamined. With these challenges in mind, we aim to develop a domain-informed NLP framework to operationalize these pain outcomes, leveraging linked longitudinal survey data as the benchmark to systematically quantify documentation completeness and validate the model’s clinical utility.

The primary objective of this study is to develop a domain-informed NLP framework for operationalizing pain severity and functional interference outcomes directly from clinical text data that can be scaled at the population level. Leveraging our survey dataset of 3725 Veterans with longitudinal PEG measures and an encounter with the VA within a year of the survey as the benchmark, we pursue 3 specific aims:

To our knowledge, this is the first large-scale study to validate clinical text-based pain outcome extraction against patient-reported measures in a longitudinal nationwide survey. Ultimately, we aim to establish the feasibility of using this NLP framework as a scalable, automated alternative to resource-intensive surveys, enabling health care systems to longitudinally monitor severe pain intensity and interference at the population level.

This protocol outlines an ongoing study to develop and validate an NLP framework for identifying severe pain intensity and interference from clinical narratives across VA medical centers, using longitudinal PROs from a nationwide survey as the benchmark (Figure 1). At the time of writing, the project has finalized its methodological design and is currently in the model development phase. In the initial phase, we have completed the construct operationalization, translating PEG scale domains into a concrete seed lexicon derived from psychometrically validated instruments for use in developing annotation guidelines for identifying PEG-related concepts in clinical notes and rating each of the 3 PEG domains as either not severe (<7/10) or severe (≥7/10).

Currently, we are in the process of adapting LLMs within VA’s secure computing environment through prompt engineering, incorporating annotation guidelines to guide evidence extraction from clinical notes. Early LLM output has identified 600 candidate encounters (200 for each of the 3 pain domains) that are being verified by SMEs. This ground truth validation dataset will be used to assess the performance of 2 candidate LLMs. Although these resource-intensive LLMs are not currently scalable for reviewing EHR notes at the population level, this phase provides initial proof-of-concept validation. To enable scalable processing at the population level, we will adopt a knowledge distillation approach, using the best-performing LLM to generate a large library of span-level annotations that will be used to train a lightweight language model. This step will also allow transparency in reviewing which key language elements the LLM parses to determine severity levels within each of the 3 PEG domains. The lightweight model will then be assessed for performance relative to the best-performing LLM. Finally, we will compare the model-generated PEG domain ratings from EHR encounters to temporally adjacent PEG scores reported by patients to assess how closely EHR reports agree with PROs. Throughout these steps, we will conduct a 3-stage validation process to characterize the completeness of pain intensity and interference documentation in clinical narratives, evaluate the inference accuracy of the LLM-as-annotator and the lightweight model, and assess concordance between the lightweight model’s output and PROs captured independently from the EHR. The completed TRIPOD+AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis and Artificial Intelligence) reporting checklist [27] is provided in Checklist 1.

This study constitutes a secondary analysis of PROs collected from the VA CIH Therapy Patient Experience Survey quality improvement evaluation [15,16]. This evaluation was deemed a nonresearch activity under the Federal Policy for the Protection of Human Subjects and was initiated and executed as an internal quality improvement project for VHA operations in accordance with VA Program Guide 1200.21 [28]. The present secondary analysis is conducted under an existing Memorandum of Understanding between the study team and the VA Office of Patient-Centered Care and Cultural Transformation, which governs data access for this work. No additional institutional review board review or waiver was required for this secondary analysis, as both the survey data from the initial evaluation project and the development of NLP evaluation tools to monitor pain interference in clinical narratives were deemed nonresearch.

All analyses are conducted within VA’s Health Insurance Portability and Accountability Act (HIPAA)–compliant computing environment via the VA Informatics and Computing Infrastructure. All study data, including clinical notes, structured EHR fields, and survey responses, remain within VA firewalls at all times. Open-weight LLMs and the lightweight model are hosted on local VA graphics processing unit infrastructure, with no protected health information (PHI) transmitted to external API end points or third-party services; proprietary commercial LLMs accessed through external APIs are excluded from this study’s pipeline. Access to the underlying Corporate Data Warehouse is role-based and limited to personnel credentialed by the VA to conduct internal quality improvement data analytics, with all access events logged through VA Informatics and Computing Infrastructure’s standard auditing infrastructure.

The Veterans Health Administration is the integrated health care system within the US Department of Veterans Affairs, providing care to enrolled Veterans through a national network of medical centers and outpatient sites. Enrollment is open to individuals with qualifying military service histories, although not all eligible Veterans enroll. Veterans receiving VA care carry a particularly heavy burden of chronic pain, with prevalence substantially higher than that of the general US adult population. This makes the cohort clinically important and represents a high-impact setting in which to develop scalable pain measurement tools. The Whole Health Flagship sites are 18 VA medical centers selected to lead implementation of VA’s Whole Health System of Care, a patient-centered health care model emphasizing what matters to the Veteran rather than what is the matter with the Veteran. The 6 priority CIH therapies expanded under this initiative are acupuncture, chiropractic care, massage, Tai Chi, yoga, and meditation.

Data for this study were derived from a secondary analysis of longitudinal PROs collected by Office of Patient-Centered Care and Cultural Transformation’s CIH Therapy Patient Experience Survey and corresponding EHRs from the VA Corporate Data Warehouse. The survey study protocol has been published [29,30]. VA’s Whole Health Flagship initiative, initially funded by the Comprehensive Addiction and Recovery Act of 2016, led to the expansion of CIH therapies across VA. The CIH Therapy Patient Experience Survey was conducted to assess the impact of this expansion on outcomes including pain severity and functional interference. Veterans invited to participate in the survey were aged 18 to 89 years, had been diagnosed with chronic musculoskeletal pain, and had newly initiated 1 of the 6 priority CIH therapies at any of the 18 VA medical centers participating in the Whole Health Flagship initiative.

Baseline surveys were sent on a weekly basis following identification of CIH therapy initiation between March 2021 and September 2022, with 6-month follow-up survey data collected through March 2023. Veterans included in this quality improvement evaluation were participants who had EHR-documented or self-reported use of a priority CIH therapy during the 6-month study period and who had completed self-reported pain assessment surveys at baseline and 6-month time points. This ensured that patients included in our study population had chronic pain and provided complete self-reported pain outcome data for analysis.

The analysis focuses on replicating the individual domains of the PEG scale, a 3-item psychometric instrument consisting of 1 item for pain severity (“P”) and 2 items for pain interference with Enjoyment of life (“E”) and General activity (“G”) [13] (Figure 2). For each domain, the goal is to determine whether there is information in the clinical narrative deemed to represent a patient’s score of severe (≥7/10) or not severe (<7/10). The threshold of 7 for severe was determined from prior literature, which observed that this threshold explained the highest proportion of variance in patient-reported interference, as introduced by Serlin et al [31] and replicated in many different patient populations [32-34]. Future efforts may address methods for combining the 3 domains into a summary score and identifying finer-level gradients of pain interference; because the 3 PEG domains are certain to be documented at different frequencies in clinical narratives relative to survey collection, where all domains are structurally asked at the same time, combining the individual domains extracted from clinical narratives is a substantially different analytic task that will depend on the intermediate findings of this analysis.

The lightweight model is designed for scalable, population-level deployment rather than individual-level diagnostic classification. Accordingly, the evaluation of concordance is framed around whether the model’s output can serve as a reliable surrogate for PROs at population scale, complementing rather than replacing direct patient self-report. As such, we will assess the concordance between severe pain and interference identified by the lightweight model in clinical narratives and patient-reported PEG scores in the full 3725-patient cohort, with patient-reported scores serving as the benchmark (Multimedia Appendix 3). This analysis will be performed on the baseline data for each individual.

The primary discrimination metric is area under the receiver operating characteristic curve, computed individually for each PEG domain using the proportion of severe pain and interference notes within a survey-anchored window as the continuous ranking score against the patient’s binary per-domain survey label (severe ≥7/10 vs not severe <7/10).

Patients without any pain-related clinical narrative within a given temporal window are excluded from that window’s analysis, as no narrative observation exists. This exclusion is reflected in the reduced sample size at narrower windows (Multimedia Appendix 3). Patients with pain-related narratives but without documentation of a specific PEG domain are retained as observations of low evidence density for that domain.

In addition to discrimination, we will assess calibration to evaluate how closely model-predicted rates of severe pain interference align with patient-reported rates at the aggregate level. Calibration will be reported through 3 complementary measures. Calibration plots will bin patients by decile of predicted probability and compare predicted rates against observed patient-reported rates within each bin. Brier scores will summarize overall probabilistic performance across the full cohort, with lower values indicating closer agreement between predicted and observed outcomes. Calibration-in-the-large will be computed as the ratio of predicted to observed severe pain interference prevalence, providing a single-number summary of systematic overprediction or underprediction.

Each analysis will be repeated for a range of temporal windows around the survey date (±7, ±30, ±90, and ±180 d) to assess how the breadth of the aggregation window affects concordance. The 7-day window reflects the anchoring period of the PEG instrument, which asks patients to consider the past week. All point estimates will be reported with 95% CIs derived from a nonparametric bootstrap resampling patients as the independent unit over 1000 replicates.

Given the model’s intended role as a population-scale complement to represent a similar concept as patient self-report of pain severity and interference, the following minimum performance criterion is calibrated to surrogate utility rather than individual-level diagnostic accuracy. The prespecified minimum performance criterion goal for Stage 3 is area under the receiver operating characteristic curve ≥0.75 for each of the 3 domains, requiring discrimination meaningfully above chance while accommodating the attenuation expected when validating an EHR-derived surrogate against patients’ self-reports. We anticipate performance will deteriorate as encounters/clinical narratives become less temporally related to the survey date.

Performance will additionally be evaluated within prespecified subgroups of interest (age, sex, primary pain diagnosis, and clinic type) to determine whether the model systematically overestimates or underestimates severe pain interference for these subgroups. Subgroup differences will be assessed by comparing bootstrap-derived confidence intervals across subgroups.

Together, the 3 validation stages assess the full pipeline from documentation completeness to model inference accuracy and population-level utility. These results will determine the feasibility of monitoring severe pain and interference at population scale and whether our NLP framework can serve as a scalable surrogate for resource-intensive patient-reported surveys.

Chronic pain is a complex health issue. Current screening tools embedded in structured EHR data cannot fully capture this complexity. Our framework extracts multidimensional pain information, including pain severity and interference with physical and emotional functioning, directly from clinical narratives at scale. If validated, this approach could offer a scalable alternative to resource-intensive surveys for monitoring severe pain and interference. To the best of our knowledge, this is the first study to investigate the feasibility of using EHR data, validated against large-scale PROs, for routine monitoring of severe pain and interference.

This study adopts a domain-informed approach, grounding the LLM’s inference in a seed lexicon derived from established psychometric instruments. This enables rigorous mapping of abstract pain-related PEG domains into concrete terms found in clinical documentation. To enable scalable processing, our approach employs knowledge distillation: an LLM first generates high-quality annotations on a subset of notes, which then serve as training data for a lightweight language model capable of processing the entire cohort. This design leverages LLM capabilities for nuanced evidence extraction while enabling scalable inference through a lightweight model. Our validation strategy extends beyond standard benchmarking against expert annotations. By first examining documentation completeness, we will assess whether disparities in clinical documentation practices, which have been identified in prior work [24-26] and may introduce systemic bias, exist in our patient population. By validating against PROs, we will evaluate whether the framework’s inferences align with how patients themselves report their pain and functional interference.

Subject to successful validation, this framework could improve how pain is monitored in health care systems. Surveys collecting PROs often suffer from low response rates and nonresponse bias, as prior research found that nonresponders had higher pain-catastrophizing scores and more pain at enrollment [12]. A validated NLP tool could serve as a passive approach to identify severe pain and functional interference, complementing active survey collection and reaching patients who are engaged in care but unreached by surveys. A validated severe pain phenotype would also enable longitudinal monitoring of treatment outcomes. While randomized controlled trials establish efficacy, they are limited in tracking long-term functional improvements at scale. By extracting pain and functional interference directly from EHR data, researchers could evaluate the efficacy of high-cost or invasive interventions such as spinal cord stimulation in reducing severe pain and interference in broader, real-world populations.

There are several limitations to this protocol. First, the LLMs’ inference relies on EHR documentation, which may be incomplete in recording the full picture of patients’ experience of pain. We address this limitation by quantifying documentation completeness in our validation. Second, pain fluctuates over time, and temporal gaps between clinical visits and survey dates may reduce concordance between EHR-derived and survey-reported outcomes. Third, significant heterogeneity in clinical documentation practices and reports of interference from pain will likely lead to underidentification of interference. Fourth, documentation biases in the EHR may propagate into model inference. If pain is systematically underdocumented for certain patient populations, the language model’s inference for those groups may inherit this bias. We will conduct subgroup analyses by demographic and socioeconomic characteristics, comparing both documentation patterns (eg, the frequency and coverage of the PEG domains in annotations) and model inference accuracy metrics (eg, sensitivity, concordance with PROs) across groups to identify such disparities.

Fifth, patient representatives were not directly consulted during the development of the seed lexicon, annotation guidelines, or interpretation of model outputs in this secondary analysis, as the parent CIH Therapy Patient Experience Survey was conducted between 2021 and 2023 and has since concluded. This gap is partially mitigated by the fact that patient perspectives are embedded indirectly through the psychometric instruments used to construct the seed lexicon (PROMIS-PI, WHYMPI, Oswestry), each of which was developed with patient-engaged validation. Nonetheless, future deployment work would benefit from direct patient input on interpretability and face validity.

Finally, the direct comparison between aggregated NLP-derived domain evidence and PEG survey scores is an imperfect proxy for true concordance. Clinical documentation of pain is episodic and contextually driven, whereas the PEG reflects a patient’s deliberate summary of their experience over time; simple aggregation of sparse span-level extractions may not fully bridge this representational gap. A natural extension of this work, contingent on demonstrating adequate NLP accuracy against SME annotations, would be to train a supervised model that learns to map heterogeneous, potentially sparse clinical documentation patterns onto PEG-equivalent scores, rather than relying on direct aggregation.

This protocol outlines a framework for identifying severe pain intensity and interference from clinical narratives, addressing a gap in health care system surveillance. By grounding inference in established psychometric instruments and validating against PROs, we aim to develop an NLP-empowered tool that can reliably extract pain severity and functional interference from EHR data. If successful, this approach could provide a scalable complement to resource-intensive surveys, enabling longitudinal monitoring of severe pain interference at the population level.