This study will use a retrospective chart review design to evaluate the feasibility and performance of LLM-based clinical data extraction from workplace injury medical records. The study design will follow a proof-of-concept framework, beginning with a pilot sample to establish baseline model performance before proceeding to the full dataset. The study is registered with the Data Sciences Institute at the University of Toronto (grant number DSI-CIDSY4R2P01).

Data will be obtained from electronic medical charts within the WSIB specialty programs at THP, a large hospital system in the Greater Toronto Area, Ontario, Canada. THP offers seven specialty recovery programs for workplace injuries, funded by the WSIB: Back and Neck, Shoulder and Elbow, Hand and Wrist, Neurology, Foot and Ankle, Hip and Knee, and COVID-19 Assessment Program (CAP). Each program focuses on injury-specific rehabilitation to facilitate rapid recovery and RTW. Previous analyses have demonstrated that participation in four of the seven programs (Back and Neck, Shoulder and Elbow, Hand and Wrist, and Neurology) is effective in reducing depression and anxiety, although these analyses did not address outcomes in the context of RTW [4]. Table 1 provides an overview of the WSIB specialty programs at THP.

For this pilot study, we will begin with 50 retrospective medical charts from the WSIB Back and Neck specialty program, selected because records in this program are typically more concise, thereby reducing the time required for initial chart review and model evaluation. Records will span from January 2018 to December 2024. The broader dataset comprises approximately 1738 records across all WSIB specialty programs and will be used in subsequent phases pending pilot results.

A standardized extraction schema will be developed to capture clinical, rehabilitative, and sociodemographic variables from each medical chart. Table 2 presents the categories of variables targeted for extraction.

Charts in the Back and Neck specialty program combine narrative clinical documentation with a structured discharge outcome record. The narrative component includes WSIB referral documentation, the initial assessment, progress notes recorded across the rehabilitation episode, and the discharge report, and it is the principal source for variables such as occupation and industry, mechanism of injury, functional limitations described in clinical language, and treatment content, where a single variable may be described in more than one section of a chart and in different terms by different clinicians. The structured component is a standardized discharge data template completed for each patient, which records administrative and clinical information in coded fields: program and treatment stream; treatment start and end dates; total number of visits; dates of injury and of surgery, where applicable; primary area of injury; physician diagnoses selected from region-specific categories; and confounding factors affecting length of stay, such as depression or anxiety, posttraumatic stress disorder, or financial and transportation barriers, selected from a defined list.

RTW status is likewise captured in coded form at baseline and at discharge, recorded as working full-time or part-time with regular or modified duties or as one of several defined not-working categories, together with a staged rating of RTW readiness. Standardized outcome instruments are administered at program entry and at discharge and recorded in the template as numeric scores. The “depression and anxiety screening scores” listed in Table 2 are the anxiety and depression subscale scores of the Hospital Anxiety and Depression Scale, captured at baseline and discharge. Pain severity is recorded on the Numeric Pain Rating Scale, condition-specific function on the Oswestry Disability Index and the Neck Disability Index for this program, fear of movement on the Tampa Scale of Kinesiophobia, and health-related quality of life on the EQ-5D-3L, alongside measured functional tests, including timed walk, lifting, carrying, and stair tasks. The variables in Table 2 therefore arrive in mixed representation: some are read directly from coded template fields or instrument scores, others must be extracted from free narrative text, and some, such as RTW barriers, may appear in both and in different terms.

Human annotators and language models will work from the same source documents and the same extraction schema, so any difference in performance reflects a difference in extraction rather than a difference in the information available. For each variable in Table 2, the schema will specify the permitted value set, the document sections in which the variable is expected to appear, and the rule to apply when a chart contains repeated or conflicting entries, for example, by taking the most recent value recorded before discharge. A variable that is absent from a chart will be recorded explicitly as not documented rather than left blank, so failure to extract a value that was recorded can be distinguished from correct recognition that no value exists. The completed schema, including field definitions and coding rules, will be released as supplementary material.

Eligible records are those of adults injured at work and referred to WSIB specialty programs with board-approved claims. No maximum age will be imposed, although the sample will primarily consist of individuals younger than 65 years of age, given the nature of employment-based injuries. The study will use preexisting clinical records created in the course of care rather than data collected for research purposes. No participant will be contacted, and no clinical intervention will be introduced.

We will evaluate four models in their off-the-shelf (no fine-tuning) form: two general-purpose models (Qwen3-VL-8B and InternVL3.5-8B) and two domain-specific biomedical models (MedGemma-27B and LLaMA-3-Meditron-8B). The general-purpose models are included because their broader pretraining may enable stronger performance on nonclinical or semistructured variables, such as sociodemographic information, where domain-specific knowledge is less critical. In addition, the two biomedical models (MedGemma-27B and LLaMA-3-Meditron-8B) will be further evaluated under a fine-tuning setting using the annotated pilot dataset to assess potential gains in extracting clinically nuanced variables. All models will be prompted with structured queries (eg, “What is the patient’s primary diagnosis?”) to extract predefined variables across sociodemographic, clinical, treatment, and mental health domains. Table 3 summarizes the models, their characteristics, and intended roles in this study.

To establish a reference standard for evaluating model performance, human annotation will be performed on the full pilot sample of 50 medical charts. Three independent annotators with clinical or research expertise in occupational health will review each chart using the standardized extraction schema. Annotators will extract the same set of variables targeted by the AI models. Interannotator agreement will be assessed using Cohen κ to evaluate consistency across annotators and to identify variables that require additional clarification or standardization of extraction criteria [29]. Discrepancies will be resolved through consensus discussion. The consensus annotations will serve as the ground truth against which all model outputs will be compared.

Prompt development, fine-tuning, and final evaluation will be carried out on separate, nonoverlapping sets of charts. Using the same records for all three purposes would allow information from the evaluation data to influence model configuration and would yield performance estimates that are optimistically biased and unlikely to reproduce on unseen records. This failure mode is well documented across applied ML and is a recognized source of irreproducible findings [30], and its avoidance is an explicit requirement of current consensus reporting standards for ML-based science [31].

The 50 charts will be partitioned once, before any model is run, into a development set of 10 (20%) charts, a training set of 25 (50%) charts, and a held-out test set of 15 (30%) charts. Partitioning will be performed at the chart level so that no record contributes to more than one set and will be stratified by sex and by broad age band to reduce the chance that a demographic subgroup is absent from the test set. The allocation will be generated with a fixed random seed, which will be reported so that the partition can be reproduced.

The development set will be the only data used to design and revise extraction prompts and to select few-shot exemplars. The training set will be used exclusively for supervised fine-tuning of the two biomedical models. The held-out test set will not be inspected during prompt development or fine-tuning and will be used once, at the end of the study, to produce the reported performance and fairness estimates for all four models under all three extraction approaches. Where fine-tuning requires a validation signal for early stopping or hyperparameter selection, that signal will be obtained by cross-validation within the training set and never from the test set. Should any test chart be examined for an unanticipated reason, the event and the charts involved will be reported alongside the results.

Human annotation will cover all 50 charts, so consensus ground truth will be available for every partition, and annotators will not be informed of partition membership. Reporting will follow the TRIPOD-LLM statement for studies developing or evaluating LLMs in health care [32], together with the TRIPOD+AI (Transparent Reporting of a Multivariable Prediction Model for individual Prognosis or Diagnosis+AI) statement for prediction model reporting [33].

Model performance will be evaluated by comparing AI-extracted data against the human-annotated ground truth. Performance metrics will be selected based on the type of variable being extracted. For binary variables, we will report sensitivity (true-positive rate), specificity (true-negative rate), the positive predictive value (precision), and the F₁-score. For multiclass classification tasks, macro- and microaveraged F₁-scores and confusion matrices will be used. For continuous variables, including numerical clinical measures, we will report the mean absolute error (MAE) and the root mean squared error (RMSE). For text span and entity extraction tasks, such as diagnosis codes and free-text clinical concepts, we will compute token- and span-level precision, recall, and the F₁-score. Following the approach described by Bannett et al [19], we will assess model explainability by qualitatively examining extraction errors to identify systematic patterns of misclassification. Performance will be compared across the two tiers (general-purpose vs domain-specific models) and the three extraction approaches (zero-shot, few-shot, and fine-tuned). All figures reported as final performance will be computed on the held-out test set defined earlier. Performance on the development or training partitions, where reported, will be labeled as such and will not be presented as an estimate of generalization performance.

The primary model performance outcome is the macroaveraged F₁-score across the categorical variables listed in Table 2, computed on the held-out test set and reported separately for each model and each extraction approach. Macroaveraging is used in preference to microaveraging so that variables with few documented instances contribute equally to the primary outcome rather than being dominated by the most frequently documented categories, which also preserves sensitivity to the subgroup effects examined in the fairness assessment. Secondary outcomes are the per variable F₁-score, the MAE and RMSE for continuous variables, the token- and span-level F₁-score for narrative extraction, and the fairness metrics reported in Table 4.

As a descriptive benchmark for interpreting model performance, we will also compute a human reference ceiling, defined as the macroaveraged F₁-score of each individual annotator against the consensus ground truth, averaged across the three annotators and computed on the same test set and the same variables as the model estimates. This places model and human performance on a common scale and indicates how much of the shortfall from perfect agreement reflects the difficulty of the charts themselves rather than a limitation of the models. It is reported as a context for interpretation and is not a component of the progression criterion.

Progression to the larger study will be considered justified if the best-performing model achieves a macroaveraged F₁-score of at least 0.80 on the held-out test set, without evidence of systematic failure on clinically important variables. This threshold is stated as a pragmatic feasibility criterion for deciding whether to proceed to the larger dataset, not as a validated threshold for acceptable clinical performance or for deployment in compensation decision-making; no such threshold has been established for this task. Evidence of systematic failure will be assessed by inspecting per variable performance for the injury characteristics and RTW outcome variables in Table 2, which carry the most direct consequences for compensation and care, and any variable on which extraction fails consistently will be reported and discussed rather than absorbed into the aggregate.

Continuous-variable performance, narrative extraction performance, and subgroup fairness results will be reported as supporting outcomes rather than as components of the progression criterion. This reflects the precision available from a held-out test set of 15 charts, in which subgroup cells in particular will be small, so that committing in advance to numerical disparity thresholds would imply a precision the pilot cannot support. All primary and supporting estimates will be reported with 95% CIs, and where a CI spans the progression threshold, the result will be reported as indeterminate rather than as meeting or failing it. The fairness results will inform the design and analysis plan of the larger study, in which subgroup sample sizes permit meaningful estimation.

A critical component of this study will be the assessment of algorithmic fairness in AI-driven data extraction. We will evaluate whether model performance differs systematically across demographic subgroups defined by age, sex, language, occupation type, and industry. Two primary fairness metrics will be applied, as described in Table 4. Fairness metrics will be computed on the held-out test set against the same consensus ground truth used for accuracy. Because subgroup sample sizes within a 50-chart pilot will be small, subgroup estimates will be reported with exact CIs and read as preliminary signals for the larger study rather than as evidence for or against the presence of bias. Systematic reviews of medical language models report demographic disparities across a wide range of tasks and find mitigation methods to be comparatively immature [34], and closed models have been shown to reproduce racial and gender patterns present in clinical text [35], which supports reporting observed disparities transparently rather than only after a mitigation step has been applied.

In addition, qualitative analysis of extraction errors will be conducted to identify potential sources of harm, including misclassification patterns that may disproportionately affect specific worker populations. This assessment will be informed by clinical expertise and contextualized judgment from team members with experience in occupational health and rehabilitation practice.

Beyond quantitative model evaluation, this study will examine the ethical and social implications of AI-assisted chart review in occupational health. This component will analyze three data sources generated by the study itself: the catalog of extraction errors produced on the held-out test set, the subgroup performance differences produced by the fairness assessment, and the recorded disagreements between annotators, which identify the variables whose meaning is contested even among trained human reviewers.

Analysis will proceed in two stages. First, each error type observed in the test set will be mapped to the care or compensation decision it could plausibly affect. An omitted RTW restriction bears on workplace accommodation, whereas a misclassified injury mechanism bears on claim adjudication. This mapping will produce a structured risk register recording, for each variable in Table 2, the observed error rate, the direction of error, the decision affected, and the party who would bear the consequence. Second, the research team will hold structured analysis sessions in which occupational therapy, clinical, and data science members will review the register together and assess where automated extraction would shift interpretive authority away from clinicians and injured workers and toward the operators of the extraction system. These sessions will be documented, and points of disagreement within the team will be recorded rather than resolved into a single position.

The outputs of this component will be the completed risk register, a list of variables identified as unsuitable for automated extraction without human verification, and a written account of the conditions under which the research team judges LLM-assisted chart review to be defensible in a compensation context. These outputs are intended to inform governance requirements for the larger study rather than to serve as a general endorsement of the method.

Given that this study involves identifiable personal health information (PHI), a secure hybrid architecture will be implemented to enable graphics processing unit (GPU)–accelerated LLM processing, while ensuring that PHI never leaves hospital custody. Two categories of data are distinguished throughout this protocol: (1) identifiable PHI, comprising the original clinical records, which remains within THP at all times, and (2) deidentified health data, that is, the study dataset produced after deidentification, which is the only data transmitted to cloud compute resources. The architecture will separate data custody from compute execution: all identifiable PHI will be retained within THP’s on-premises secure sandbox environment (MCVHDSMAN90), while Amazon Web Services (AWS) cloud resources will be used exclusively as transient compute layers. Figure 1 presents an overview of this architecture. Key architectural features include the following:

This architecture will ensure compliance with THP enterprise governance requirements and Canadian privacy legislation, while enabling the computational resources necessary for LLM fine-tuning and inference workloads.

This manuscript describes a planned study. No chart has been accessed, transferred, or processed, and no data collection will begin before research ethics approval is in place. Approval will be obtained from the research ethics boards of both THP, which holds custody of the records, and the University of Toronto, prior to any data access at either site. The protocol will not proceed to data collection under any other authorization.

Consent provided by a worker to participate in a treatment program is not consent to the research use of that worker’s record, and this protocol does not treat it as such. Both applications will therefore request a waiver of the requirement to seek individual consent for secondary use of identifiable information under Article 5.5A of the Tri-Council Policy Statement [36]. The grounds for the request are that the records were created for clinical and compensation purposes between 2018 and 2024, that many workers completed their programs several years ago and are no longer in contact with the hospital, that recontacting individuals across a 6-year retrospective window is impracticable, that the research could not reasonably be carried out without the waiver, and that the safeguards described next limit residual risk to participants. Whether these grounds are met is a determination for the research ethics boards, and the study will not proceed if the waiver is refused.

Records will be identifiable at the point of access, because sociodemographic and clinical detail must be read from the original documents. Deidentification will therefore be performed inside the THP secure environment before any content enters the processing pipeline described earlier. Direct identifiers will be removed and replaced with study codes, the linking file will be held only within hospital infrastructure and will be accessible only to hospital-based members of the team, and no identifiable content will cross the trust boundary at any stage. References elsewhere in this protocol to deidentified data describe the state of the data after this step rather than at the point of retrieval.

Two distinct retention periods apply, and this distinction matters because the study draws on records maintained for clinical reasons, independent of the research. The deidentified research dataset created for this study will be retained for 5 years, in accordance with THP research data retention policy, after which it will be securely destroyed. The underlying clinical records will remain in the custody of THP and will be retained indefinitely for clinical purposes, as they would be irrespective of this study; the research team will hold no copy of them beyond the research retention period. The file linking study codes to patient identifiers will be destroyed once data collection is complete and annotation has been finalized. Retention periods, storage locations, and destruction procedures will be set out in the ethics applications and followed, as approved.

Data will be accessed, processed, and stored in accordance with the secure hybrid architecture described earlier, so PHI will always remain under hospital custody. Locally deployed open-weight models were selected, in part, for this reason, since they allow inference without transmitting clinical text to an external model provider [24,25]. The research team includes collaborators from THP clinical practice who will contribute expertise in medical chart annotation and interpretation.

This protocol describes the first systematic investigation of LLM-based clinical data extraction applied to workplace injury medical charts within a workers’ compensation rehabilitation context. The study will address a critical gap at the intersection of AI in health care, occupational rehabilitation, and algorithmic fairness. By using both general-purpose and domain-specific language models, this study will generate foundational evidence regarding the feasibility, accuracy, and equity implications of automated chart review in occupational health settings.

The model evaluation strategy compares locally deployed open-weight models under both prompting-based and fine-tuning settings. We will assess the off-the-shelf performance of general-purpose models (Qwen3-VL-8B and InternVL3.5-8B) and biomedical models (MedGemma-27B and LLaMA-3-Meditron-8B), as well as the impact of fine-tuning on the biomedical models. This comparison is particularly relevant for clinical settings where the cost and complexity of model fine-tuning must be weighed against the accessibility and flexibility of prompt-based approaches.

Although prior studies have demonstrated the utility of LLMs in clinical data extraction across domains such as pediatric medication monitoring [19], classification of free-text pathology reports [37], and prediction of clinical and operational outcomes from clinical notes [38], no previous work has applied these methods to workplace injury rehabilitation records. The WSIB chart review context introduces domain-specific challenges, including highly variable documentation structures across specialty programs, the integration of clinical and occupational information within single records, and the need to extract variables (eg, RTW barriers and functional capacity assessments) that are not well represented in general biomedical training corpora. This study will provide the first empirical data on LLM performance in this context.

The inclusion of a formal fairness assessment using the demographic parity and equalized odds metrics will also distinguish this study from most existing clinical AI evaluations, which typically focus on aggregate accuracy rather than stratified analyses across demographic groups [20]. Given that workplace injury outcomes are influenced by sociodemographic factors, including age, sex, occupation, and industry, equitable model performance is essential for responsible deployment.

Several limitations should be acknowledged, together with the reasoning behind the design choices that produce them. First, the pilot draws 50 charts from a single specialty program at a single facility, which limits the generalizability of initial findings. Restricting the pilot to the Back and Neck specialty program follows from the annotation burden: consensus ground truth requires three independent annotators to review every chart, and within a 12-month grant of CAD $10,000 (US $7305), the inclusion of a second program would either halve the sample available per program or exhaust the annotation capacity that makes the ground truth credible. Three features of the design limit the cost of this restriction. The extraction schema in Table 2 contains no variable specific to spinal injury and is intended to transfer unchanged to the remaining six programs. Performance will be reported per variable rather than only in aggregate, so variables that depend on program-specific documentation can be distinguished from those that do not. The subsequent phase will draw on the full dataset of approximately 1738 records across all seven programs, and the pilot is structured to produce the schema, the partitioning procedure, and the annotation protocol that this phase requires.

Second, the retrospective design means that documented variables are subject to confounding not captured by the extraction schema. Because the objective is to measure extraction accuracy against what clinicians recorded rather than to estimate causal effects on RTW, this limitation constrains the interpretation of any secondary analysis rather than the primary outcome. Third, a consensus ground truth established by three annotators may not capture the full range of interpretive variability present in clinical chart review. Reporting Cohen κ per variable and retaining the record of preconsensus disagreement will make visible those variables on which human agreement is unstable, and model performance on such variables will be interpreted against a ground truth that is itself uncertain rather than treated as error.

Fourth, the models were not trained on occupational health data, which may limit domain-specific performance. This is the reason the design pairs off-the-shelf evaluation with fine-tuning on the training partition, so the gap attributable to domain mismatch is measured rather than assumed. Fifth, the secure hybrid architecture introduces computational overhead that may affect throughput. This cost is accepted because transmitting clinical text to an external model provider is incompatible with hospital data governance, and processing time will be recorded so that the operational cost of the architecture can be reported alongside accuracy. Sixth, a held-out test set of 15 charts yields small subgroup sample sizes for the fairness analysis. The pilot is designed to detect gross disparities and to establish the measurement procedure rather than to produce precise subgroup estimates, which is a principal reason for proceeding to the larger dataset.

This protocol outlines a rigorous and ethically informed approach to evaluating LLM-based data extraction in workplace injury rehabilitation. By combining quantitative performance assessment with fairness analysis and critical ethical inquiry, the study aims to establish a foundational methodology for the responsible integration of AI in occupational health. Findings from this pilot will inform the design of a larger-scale study across all WSIB specialty programs at THP, contribute to the development of ethical guidelines for AI-assisted chart review, and support evidence-based policy recommendations for AI deployment in workers’ compensation clinical workflows. The study directly aligns with the Data Sciences Institute’s mission to ensure that data science methodologies are applied in a reproducible, fair, and ethical manner.