As injury and death by firearm continues to increase in the United States, so too does secondary exposure to firearm violence (witnessing a shooting, being threatened by a firearm, or the injury or death of a loved one or acquaintance by gun violence). It is estimated that 3 million US children are exposed to firearm violence each year and that 54% of all US adults have experienced firearm violence [1,2]. Primary and secondary exposure can lead to adverse social, emotional, physical, and behavioral health impacts [1,3-6]. Despite the high prevalence of exposure to firearm violence, structured data (eg, diagnostic codes) on firearm violence is largely limited to acute primary exposure (ie, direct injury or death) and there are no International Classification of Diseases (ICD) codes that specifically capture follow-up care from injury or exposure without injury. As a result, most firearm violence research is conducted using acute injury data from hospitals, emergency departments, or death data and is therefore limited to individuals who sustained acute physical injury [7].

Comprehensive data from ambulatory care settings (primary and behavioral health care) are needed to better understand the short- and long-term impacts associated with firearm violence exposure [7-9]. ICD-10 diagnosis codes now exist to document adverse socioeconomic and community circumstances, but these codes are used infrequently across health systems and there is no specific code documenting secondary exposure to gun violence [10,11]. Another opportunity to identify this exposure in clinical settings is through the development and use of custom data collection instruments, such as a trauma history “SmartForm” described in prior work [12]. While these documentation practices show promise, they remain inconsistently available and underused.

Another promising emerging source of data is electronic health record (EHR) clinical notes [13]. Clinical notes include relevant medical care notes that are needed for comprehensive, longitudinal patient health care. Accessing unstructured clinical data using advanced natural language processing (NLP) methods has demonstrated utility in similar types of firearm and trauma research, yet the application of these methods to outpatient settings remains largely unexplored [13-17]. Early work using a keyword search of firearm violence terms identified that exposure to firearm violence is available in unstructured data in ambulatory care settings [18]. This finding provided rationale for developing an NLP text classifier to identify cases of firearm violence exposure documented in clinical notes.

This protocol describes the development of an NLP text classifier to identify firearm violence exposure in primary care and behavioral health care clinical notes. Ambulatory clinics providing primary and behavioral health care offer a novel source of data on firearm violence, particularly exposures that may not have resulted in acute emergency care, hospital encounters, or death. The rich contextual information (eg, patient social history and comorbidities) contained in primary care and behavioral health clinical notes presents an opportunity to expand our understanding of the short- and long-term adverse effects of firearm violence exposure [19-21]. The addition of data from this novel source, combined with learnings from more established data sources on firearm injury and death, could greatly expand our understanding of risks and outcomes associated with exposure [7,22].

This research protocol describes the development of a stakeholder-informed NLP model to identify patients with exposure to firearm violence using clinical notes from a large, multistate network of ambulatory health care centers by classifying text as indicative as exposure to firearm violence or not.

This protocol describes the development of a stakeholder-informed NLP text classifier to identify patients with firearm violence exposure documented in unstructured clinical notes during a primary care or behavioral health care encounter. The current study builds on pilot work that confirmed the presence of unstructured data in clinical notes that included firearm violence keywords that had contextual meaning of exposure [18]. Study findings, when available, will be reported following guidance for reporting machine learning model development in biomedical research [23].

The data used in this study come from OCHIN, a multistate network of community health organizations using a single instance of Epic EHR [24]. OCHIN is a nonprofit health care innovation center that offers a fully hosted, highly customized instance of Epic practice management and EHR solutions at over 2000 health care delivery sites across 40 states. OCHIN member health organizations provide health care services to patients in low-resource settings, regardless of their ability to pay. We will access clinical notes from unstructured records in natural language serving as a record of treatment, diagnoses, and clinical progress from ambulatory primary care and behavioral health care encounters to patients aged ≥5 years at encounter from January 1, 2012, to December 31, 2022. These chart notes will be used to develop an NLP model that will classify the note text as representative of firearm violence exposure or not.

OCHIN community health organizations, as part of their contract with OCHIN, authorize OCHIN to create limited datasets of member information for certain research activities consistent with applicable law and aligning with best practices of research, including institutional review board oversight of study activities.

This study was reviewed and deemed exempt by the Norwich University Institutional Review Board. Only research staff who are OCHIN employees have access to clinical notes, and efforts were made to remove direct patient identifiers prior to review.

Data for NLP model development will be identified by searching for keyword phrases in clinical notes for study-eligible patient encounters. We define study-eligible patient encounters as those occurring between January 1, 2012, and December 31, 2022, with patients who were aged ≥5 years on their date of encounter (4,024,926 patients). This study period aligns with the date range of EHR data availability at the time of study start. The age restriction was imposed to reduce the number of “false positive” firearm exposure measures resulting from screening for household gun ownership commonly assessed in early childhood encounters [25]. We further restricted eligible ambulatory encounters to those categorized as either primary or behavioral health, using a previously defined algorithm (31,801,530 encounters) [26].

The NLP model development will be guided by an 8- to 10-person stakeholder advisory committee (SAC), a multidisciplinary group of individuals who will review methods, provide guidance on testing and evaluation, participate in interpreting and disseminating study findings, and evaluate the social and ethical merits for ongoing study and model deployment. SAC members include people with lived experience of firearm violence exposure, clinicians, researchers, clinical informaticists, and data scientists.

Development of the SAC will be conducted during the first 3 months of the study. SAC members will meet monthly, alternating between synchronous and asynchronous meeting formats, to guide study activities and will be compensated for their time (the mean compensation will be $185 per hour). Due to the dynamic nature of the research activities and the novel application of a SAC for NLP development around firearm violence, the ongoing contribution of SAC members after lexicon development and review (see below) will continue to evolve as study activities progress. We anticipate that SAC members will participate in interpreting research findings and provide ongoing input into the applicability of the model in clinical practice.

During pilot work, a list of keywords informed by MacPhaul et al [13] was applied to a set of notes for patients with known exposure to firearm violence. A random sample of 30 notes from each of 3 text string search categories (broad, gun-only, and shooting) were reviewed to contextualize how keywords are used in clinical notes and to identify lexicon refinements (eg, remove preset template EHR phrases and add keywords[18]). Among the 90 notes reviewed, 13 (14%) were determined to be true “cases,” while all other notes did not indicate exposure to or injury from firearm violence.

In this study, the lexicon (Multimedia Appendix 1) was expanded to include additional terms used by Chew et al [27] and will be used to extract a random sample of 5000 clinical notes that contain keywords for review. Applying a case rate of 14%, we anticipate that 700 of these clinical notes will indicate exposure to firearm violence. The study team worked together to develop a case note selection decision guide to clarify inclusion and exclusion criteria and reduce variation among study team annotators (Multimedia Appendix 2). The guide defines a case as “primary or secondary exposure to firearm violence as a direct witness of firearm violence or the acute aftermath.” The decision guide clarifies that cases will include any mention in the clinical note of gun threats (primary or secondary) experienced by a patient. In addition, cases include mention of retained bullets in the body, sequalae from firearm violence, patients brandishing or perpetrating crime with a gun, or self-inflicted gunshot wounds. Following pilot methods, the sample of clinical notes will be posted as a shared Microsoft Excel document for review. Study team members independently review notes for keywords and contextualize the meaning of the note as case or noncase, copying over relevant sentences that informed their decision into the document. When case status is not apparent from the clinical note even though it contains lexicon words, the reviewer will indicate in an electronic comment that additional review and adjudication is needed. An additional 2 study team reviewers will review the clinical note, document their assessment, and the 3 reviewers will subsequently meet to make a determination. If additional input is needed for categorization the study team will involve the clinician informaticists on the study team, one of whom is a primary care physician and the other a behavioral health clinician. If needed, the decision guide will be updated iteratively by the study team. Counts of total notes available, the prevalence of keywords, and the prevalence of firearm violence–indicative notes will be recorded and detailed in future publications.

The study team will also engage the SAC to review keywords and suggest additional terms. These terms will be evaluated for inclusion into the lexicon by performing a keyword search to identify potentially related notes, followed by a review of a sample (up to 30 per term, if available in notes) of these notes. Terms that identify at least one note indicative of firearm violence exposure will be added to the lexicon. Engaging stakeholders is an important approach for incorporating broad community perspectives in developing and applying NLP and other artificial intelligence in health care, reducing bias and improving applicability [28-30].

The adjudicated notes will become the labeled dataset for training and testing our NLP text classifier. The study team will use Python (Python Software Foundation) to build the NLP text classifier. The output of our classifier will result in a new structured data variable that identifies exposure to firearm violence from behavioral health or primary care encounters in ambulatory settings.

The NLP text classifier will be trained as follows:

The study team is approaching development of the NLP text classifier with a lens toward minimizing biases that may emerge [39-41]. Our primary strategy to address bias in model development is through the engagement of a SAC throughout the study. SAC members will participate in discussions about the use of an NLP-based classifier in a clinical context. Discussions with SAC members on the implications of false positives and false negatives at both the patient level and research findings level will help guide how potential NLP models will be evaluated (ie, which metrics are of higher importance). The SAC will be asked to consider bias based on patient characteristics and provider practices as well as algorithmic bias. Inclusion of SAC-suggested terms indicative of firearm violence and considerations of model performance across subpopulations are key evaluative characteristics for considering, identifying, and then mitigating potential biases in a universal firearm violence NLP algorithm. For additional diversity in input, our research team will meet with OCHIN’s primary care and behavioral health clinical workgroups, attended by clinical leaders from member health organizations during the study start up (Year 1, quarter 1).

Potential biases in our training data include missing patients where firearm violence exposure is not present in a clinical note, possibly due to care received outside the health center, variation in provider care, or documentation practices, and patients who do not self-report exposure. These biases are inherent to all EHR-based research, and generalizability should be interpreted in light of the convenience-based, albeit large, nature of the dataset used [42,43] Our data come from primarily low-resource populations but it is not necessarily representative of all subgroups or individuals most likely to experience exposure to firearm violence. Likewise, we do not have reliable data on many variables in which model bias may exist (eg, gender).

There is the possibility of annotation bias during adjudication of text excerpts from clinical notes, even with our experienced team members. To minimize this bias, the study team developed a case note selection decision guide to support review by different study team members. We did not assess inter-rater reliability but would recommend this be performed in future projects.

We will attempt to identify potential model bias by calculating model performance across subpopulations of sex, age, race and ethnicity, and the intersections of these subpopulations (ie, group fairness) as these are protected attributes that are reliably documented in our data. We will provide this information to the SAC for further consideration along with discussions of stereotypical bias that can occur by evaluating models over such groups, under-representation bias due to low numbers in some subgroups, and possible mitigation strategies.

There may be other types of biases present in an NLP model that require more in-depth analysis to identify and mitigate. Such analyses include counterfactual evaluation (eg, replacing pronouns), adversarial testing, checking for bias in word embeddings, and evaluating model bias on an external dataset. However, due to the short duration of this study that spans lexicon development to the evaluation of NLP models, and without the intent to put this model in production at the end of this study, we leave these advanced and necessary bias identification methods to future work. If we do identify any biases in this study, we will document and provide transparency to these biases. We do not foresee time to effectively mitigate all potential biases within this study period. If there are identified biases and time left in our study period, we will consider and use appropriate mitigation strategies, which will most likely involve postprocessing mitigation techniques such as calibration and threshold adjustment given time constraints.

Data acquisition started in February 2024 and was completed in April 2024, with >85 million primary care and behavioral health notes from >7 million patients. The study team extracted text from a random sample of 5000 notes with at least 1 firearm violence term for review and adjudication between March 2024 and May 2024. The first NLP template for baseline models was run in July 2024. Model development will continue throughout the study period to identify the best performing model for potential deployment. We expect that the final model will be selected by August 2025, and we will publish results of NLP model development and the final model performance in 2026.

The SAC was established during the first 3 months of the study in 2023 and includes 12 members: 5 community advocates or patients with lived experience; 1 physician; 2 clinician researchers; 2 clinical informaticists; and 2 data scientists with firearm violence data and research experience. The SAC will continue to meet monthly for the duration of the study. SAC contribution includes providing guidance on lexicon development, review of NLP performance, and application of NLP model in clinical practice [30].