Evaluation of the Care Pathway in the Context of the Dispensing of Emicizumab (Hemlibra) in Community Pharmacies in France: Protocol for a Cross-sectional Study Based on the Kirkpatrick Model

Background: Commercialized since 2019, emicizumab (Hemlibra) was available only in French hospital pharmacies for prophylaxis of hemophilia A with or without inhibitors. Since June 15, 2021, patients can choose between a hospital and community pharmacy. These changes in the care pathway have important organizational consequences for patients, their relatives


Introduction Background
Severe hemophilia A is an inherited bleeding disorder due to the deficiency of clotting factor VIII (FVIII), which causes bleeds occurring internally into joints and muscles, or external bleeding from minor cuts, surgical procedures, or injuries.The frequency and intensity of bleeding depends on the level of FVIII deficiency.In high-income countries, the life expectancy of patients with severe hemophilia A is decreased by 37%, the prevalence is estimated to be approximately 6 per 100,000 males, and prevalence at birth is to be approximately 9.5 cases per 100,000 males [1].In France, the prevalence of severe hemophilia A has been estimated to be approximately 2095 patients in September 2022 [2].Conventional management consists of FVIII replacement therapy.To maintain a sufficient level of clotting factor in the bloodstream to prevent bleeds, patients with severe hemophilia A are typically prescribed prophylactic treatment that consists of regular administration of FVIII, for example, every day, week, or month, according to how long the factor lasts in the body.Although effective, this treatment can only be administered intravenously, and in France, as well as in other countries [3][4][5][6], dispensation FVIII is only available from hospital pharmacies; they both contribute to organizational and geographical constraints [7] as well as a significant mental burden for patients and their caregivers, particularly for the parents of children with hemophilia [8].

Dispensing Circuit of Emicizumab in France
Emicizumab (Hemlibra) is a laboratory-engineered protein that works by performing a key function in the clotting cascade that is normally carried out by the FVIII protein [9][10][11][12].It can be prescribed for routine prophylaxis to prevent or reduce the frequency of bleeding episodes in adults and children of all ages, newborn and older, with hemophilia A, with or without treatment with an FVIII inhibitor [13][14][15].
In 2019, emicizumab received marketing authorization for severe hemophilia A with or without an inhibitor in Europe [16].This drug was initially available only from French hospital pharmacies, but from June 15, 2021, it has also been available in community pharmacies.The availability of emicizumab in community pharmacies was accompanied by the implementation of a rigorous new organization of the care pathway (Figure 1), ensuring that all professionals involved in the management of patients were identified by community pharmacists for an optimized exchange of information with the hospital pharmacy and the national reference center for hemophilia.Community pharmacists have the possibility to follow the training program by making a request to the reference center for hemophilia.The e-learning course called "HEMOPHAR" is based on the disease and its issues, and was designed by a multidisciplinary team of health care professionals from the reference center for hemophilia, as well as the platform for exchange and research on blood-derived medicines and their recombinant analogues (Plateforme d'Echange et de Recherche sur les Médicaments DErivés du Sang et leurs analogues recombinants) of the French society of clinical pharmacy (Société Française de Pharmacie Clinique).Also, a short training session organized by Roche, which that markets the drug, validated by the French medicines agency (Agence nationale de sécurité du médicament et des produits de santé) is proposed to pharmacists who order the drug; this is dispensed by telephone and aims to review the key points on the proper use of this medicinal product (including storage methods and ordering).The gray arrows represent the setting up of the organization for the first time initiated by the patient, whose steps are numbered in the order of implementation.The patient indicates his/her chosen community pharmacy to one of the 33 care centers.The care center requests the connexion codes for the e-learning course "HEMOPHAR" from the reference center for hemophilia, and the community pharmacist is invited to follow the training.To facilitate the relationships between hospital and community pharmacists, practical information sheets are shared, including a contact sheet, checklist for dispensing emicizumab, and liaison sheet between the community pharmacist and the care center.The hospital pharmacy is informed by the care center about the transition of dispensing emicizumab in community pharmacy.

Objectives of the PASODOBLEDEMI Study
These changes in the dispensing circuit have important organizational consequences for community pharmacists, becoming both new stakeholders and contacts in the patient care pathway [17].The effectiveness of this new organization, and in particular the training, therefore needs to be evaluated.To address this, we designed a national study that aims to evaluate the direct impact of the training programs provided to community pharmacists in the context of the dispensing of emicizumab, and to evaluate patient satisfaction with their treatment whether they choose dispensation from a community pharmacy or retained dispensation from the hospital pharmacy.

Overview
The Kirkpatrick model (KM) represents a reference for the evaluation of training programs.Studies of various designs and purposes were addressed using the KM; for example, to evaluate a nutritional education program provided by health care professionals [18], to evaluate a train-the-trainer program for health care providers [19], to assess the training of cardiopulmonary resuscitation for nurses [20], to evaluate educational programs for undergraduate dental students regarding appropriate prescribing behavior during their early educational years [21], or to determine the effectiveness of a course for community health workers [22].
The popularity of the KM can be traced to several factors [23].First, it addressed the need for training professionals to understand training evaluation in a systematic way by providing a straightforward language for communicating about training outcomes.Second, the KM provides trainers with a way to measure the results of what they do in terms of economic impacts and outcomes, which also measures the success of the organization.In addition, to simplify the complex process of training evaluation, a guide to the type of questions and the criteria that may be appropriate to focus on at different temporalities and information levels is provided.

Level 1: Reaction
The first level of evaluation consists of collecting immediate feedback from participants after completing the training.This information can be used to determine which elements of the training are working well or appreciated, and which ones need to be improved.Indicators such as participation rate, completion rate, or time spent on training are typically collected to supplement participant feedback.

Level 2: Learning
The second level is to accurately measure what participants did or did not learn during the training.Usually, the KM suggests a pre-and posttraining evaluation to quantify progress, for example, through knowledge tests or competency interviews.In contrast to immediate reactions to training, this level allows XSL • FO RenderX for the measurement of specific outcomes by identifying Learning objectives or targeting expected effects of training.When pretraining data are missing, the knowledge acquired post training is evaluated in accordance with the expected knowledge.

Level 3: Behavior
The third level is to determine if the training had a direct impact on the participants' behaviors.This level of evaluation is the most time-consuming because it takes several weeks or months for participants to build confidence and apply their knowledge in their professional practice.To measure Behavior, data collection and feedback experience are needed to give participants the opportunity to apply the skills they learned during training; for example, by exposing them to a simulated situation.It is important to mention that a lack of behavioral change does not necessarily mean that the training was ineffective.

Level 4: Results
The fourth level consists of assessing the impact of behavioral changes of the participants on the systemic organization, and in determining if the training reached the expected impact or improved the organization.

Study Design
We designed a cross-sectional study based on the KM.The first 3 levels are based on the data directly collected from the pharmacists.Level 1 refers to the evaluation of the immediate reaction of community pharmacists who followed the HEMOPHAR e-learning program and level 2 to their knowledge acquisition.Level 3 consists of evaluating the professional practice of both hospital and community pharmacists in the dispensation of emicizumab.The fourth level of the KM refers to the patients' satisfaction related to treatment, whether dispensed from a hospital or a community pharmacy (Figure 2).

Patients With Severe Hemophilia A With or Without an Inhibitor
All patients with severe hemophilia A, with or without an inhibitor, regardless of age, treatment with emicizumab, and whether they chose dispensation from a community pharmacy or retained dispensation from a hospital pharmacy, were eligible for the evaluation of the results.For those younger than 18 years, a legal representative fills out the questionnaire, and for those aged 18 years or older who need assistance in completing the questionnaire, this may be provided by caregivers or relatives.

Time Frames
A clinical research associate (CRA) is specifically dedicated to contact community pharmacists, informing them of the study modalities and outcomes, and inviting them to complete the e-questionnaires at the appropriate time (Table 1).Immediately after training is completed, the community pharmacists are invited to complete a questionnaire; this will be used to evaluate Reaction.The CRA is also able to incite community pharmacists-who have received their connexion codes but who have not yet started the HEMOPHAR learning program-to finalize the HEMOPHAR training program.During the HEMOPHAR training program, community pharmacists also complete 4 quizzes related to the following sections: presentation of the disease, therapeutic management, organization of care, and practice in the community pharmacy.The responses to the quizzes will be used to evaluate Learning.All community pharmacists are invited to complete the e-questionnaire related to Behavior at least 3 months after the implementation of the organization.Patients treated with emicizumab are invited to complete the appropriate questionnaire related to Results, an e-questionnaire specific to dispensation from hospital pharmacies, and a second e-questionnaire specific to dispensation from community pharmacies.

Development of the Questionnaires
The e-questionnaire related to Reaction was intentionally short and composed of 8 questions, including 5 questions based on the 4-item Likert-scale (not at all satisfied, not very satisfied, somewhat satisfied, and very satisfied) and 3 closed-ended questions (yes/no; Multimedia Appendix 1).The estimated time to complete this e-questionnaire is estimated to be approximately 1 minute.
The e-questionnaire related to Behavior was composed of 13 multiple-choice questions (Multimedia Appendix 2).The estimated time to complete this questionnaire is estimated to be approximately 8 minutes.
The e-questionnaires related to Results were adapted to the dispensing choice of the patient (Multimedia Appendices 3 and 4).They were composed of 33 questions.The estimated time to complete this questionnaire is estimated to be approximately 6 minutes.
The content of the questionnaires was initially defined by a multidisciplinary team composed of hospital and community pharmacists and research scientists specialized in methodology and care accessibility.A scientific committee was specifically constituted for the PASODOBLEDEMI study in order to challenge and validate the purposes, methodology, questionnaires, and expected outcomes.This committee was coordinated by the reference center for hemophilia and composed of hospital and community pharmacists, physicians, and the French association of patients with hemophilia (Association Française des Hémophiles).Following the recommendations of the scientific committee, the content of the e-questionnaires has evolved by integrating operational and organizational aspects and optimizing literacy.The e-questionnaires were finally tested in real-life conditions on a sample of eligible populations.The content of the questionnaires was finally validated by the scientific committee in July 2022.

Ethical Considerations
The study complies with the reference methodology for studies and evaluations in the health field [24]; therefore, it did not require written consent to participate.The ethics committee of the Hospices Civils de Lyon approved the study (n°2022-06-01 obtained on June 14, 2022).The PASODOBLEDEMI study was registered in ClinicalTrials.gov(NCT05449197 related to the evaluation of Reaction, Learning, and Behavior, and NCT05450640 related to the evaluation of Results).

Administration
All e-questionnaires may be filled out using a PC or mobile device.The hospital and community pharmacists are invited by email to complete the e-questionnaires accessible using a login and password sent by the CRA for the evaluation of Reaction and Behavior; this also allowed the linking of data between evaluation levels.The e-questionnaire related to Results is accessible to eligible patients via a QR code or a specific URL presented on a paper information note, hand-delivered by the community pharmacist or hospital pharmacist when the patient comes for treatment or consultation.

Data Collection
Data collection for the first 3 levels started on July 1, 2022, and for the fourth level, the questionnaires for patients were available since September 12, 2022.The study ended on December 31, 2022.

Anticipated Findings
The availability of emicizumab in community pharmacies represents a significant advancement of the care pathway for patients with hemophilia A. The direct expected benefit concerns the improvement of the patients' satisfaction by reducing the mental burden due to the geographical and organizational constraints to hospital pharmacies [8].As observed in the management of other diseases, distance to the pharmacy may play a role in pharmaceutical management, leading, for example, to inappropriate and inadequate medication and management of asthma for patients living further from their preferred pharmacy [25].Also, the dispensing by community pharmacies may provide better access and satisfaction to the patients [26].For example, it is reported that the dispensing in community pharmacies of medicines previously only available from hospital pharmacies during the first wave of the COVID-19 pandemic in Portugal led to a significant increase in satisfaction levels in all evaluated domains: pharmacist's availability, opening hours, waiting time, privacy conditions, and overall experience [27].In the United States, satisfaction of patients with schizophrenia was improved by dispensing a long-acting injectable medication in community pharmacies during the COVID-19 pandemic [28].In addition, a collaboration between opioid treatment programs and community pharmacists supported the feasibility and acceptability of physician prescribing and community pharmacy for access to methadone [29].
The new organization for dispensing emicizumab to patients with hemophilia A in French community pharmacies must be accompanied by optimal safety and quality conditions due to the risk of serious and urgent bleeding situations in the management of rare bleeding diseases.Based on the KM, the PASODOBLEDEMI study focuses on the evaluation of the training program and patient satisfaction 1 year after the implementation of the community-based dispensing of emicizumab.Widely used in the literature [30], the KM presents advantages, including ease of implementation and provision of evaluation of the systemic impacts on the organization.Considering that single outcome measures cannot adequately reflect the complexity of the organization, the KM provides 4 distinct outcomes.
The KM presents limitations [23] because the 4-level implementation can be time-consuming and requires an expensive investment to collect data and analyze learning outcomes.It is also difficult to establish a direct link between outcomes and training.
Other methodologies have been developed and used for evaluation of training programs, but which were not adapted to the PASODOBLEDEMI study.For instance, the Context-Inputs-Process-Product model (or the Stufflebeam model) [31,32] is based on the comparison of pre-and posttraining program evaluation to assess the gain in knowledge and confidence of participants; the Levels of Learning model (or the Kaufman model) is based on the use of specific evaluation scales (Kaufman Test of Educational Achievement) as an individually administered battery that provides in-depth assessment and evaluation of skills [33], but the design of this model is too rigid for the objectives of this study; and the Success Case Method (or the Brinkerhoff model) that is closer to an economically oriented performance scale with a study design based on a survey and semistructured interviews [34].

Conclusions
The elaboration of the PASODOBLEDEMI protocol has already had a positive impact with the commitment of all health professionals, physicians, hospital and community pharmacists, as well as the patient community.The results will be disseminated to the French authorities and will enable, if necessary, proposing this access model to other rare diseases.

Multimedia Appendix 1
Questionnaire for the evaluation of Reaction.

Multimedia Appendix 4
Questionnaire for the evaluation of Results (To patients or legal representatives or caregivers who have retained hospital pharmacy dispensing).

Figure 1 .
Figure 1.New organization of the dispensation circuit of emicizumab since June 15, 2021, in France.The black arrows indicate the routine dispensing organization, including emergency management of bleeding episodes.The gray arrows represent the setting up of the organization for the first time initiated by the patient, whose steps are numbered in the order of implementation.The patient indicates his/her chosen community pharmacy to one of the 33 care centers.The care center requests the connexion codes for the e-learning course "HEMOPHAR" from the reference center for hemophilia, and the community pharmacist is invited to follow the training.To facilitate the relationships between hospital and community pharmacists, practical information sheets are shared, including a contact sheet, checklist for dispensing emicizumab, and liaison sheet between the community pharmacist and the care center.The hospital pharmacy is informed by the care center about the transition of dispensing emicizumab in community pharmacy.

Figure 2 .
Figure 2. The Kirkpatrick model adapted to the PASODOBLEDEMI study.

Table 1 .
Objectives, the eligible population, time frames, and measurement tool per levels of evaluation.