Published on in Vol 10, No 2 (2021): February

Preprints (earlier versions) of this paper are available at, first published .
A Value-Based Comparison of the Management of Ambulatory Respiratory Diseases in Walk-in Clinics, Primary Care Practices, and Emergency Departments: Protocol for a Multicenter Prospective Cohort Study

A Value-Based Comparison of the Management of Ambulatory Respiratory Diseases in Walk-in Clinics, Primary Care Practices, and Emergency Departments: Protocol for a Multicenter Prospective Cohort Study

A Value-Based Comparison of the Management of Ambulatory Respiratory Diseases in Walk-in Clinics, Primary Care Practices, and Emergency Departments: Protocol for a Multicenter Prospective Cohort Study


1Axe Santé des populations et Pratiques optimales en santé, Centre de recherche du CHU de Québec-Université Laval, Québec, QC, Canada

2Department of Family and Emergency Medicine, Université Laval, Québec, QC, Canada

3Department of Community Health sciences, Université de Sherbrooke, Campus de Longueuil, Longueuil, QC, Canada

4Centre de recherche Charles-Le Moyne - Saguenay-Lac-Saint-Jean sur les innovations en santé, Longueuil, QC, Canada

5Department of Social and Preventive Medicine, Université Laval, Québec, QC, Canada

6VITAM - Centre de recherche en santé durable, Québec, QC, Canada

7Centre de recherche du Centre intégré de santé et de services sociaux de Chaudière-Appalaches, Lévis, QC, Canada

8Ministère de la santé et des services sociaux, Gouvernement du Québec, Québec, QC, Canada

9Department of Emergency Medicine, Queen's University, Kingston, ON, Canada

10Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada

11Schwartz/Reisman Emergency Medicine Institute, Sinai Health System, Toronto, ON, Canada

12Faculty of Nursing, Université de Montréal, Montréal, QC, Canada

13Operations and Decision Systems Department, Faculty of Administrative Sciences, Université Laval, Québec, QC, Canada

14Department of Medicine, University of Toronto, Toronto, ON, Canada

15Department of Family and Emergency Medicine, Université de Montréal, Montréal, QC, Canada

16Ministry of Health and Long Term Care, Government of Ontario, Toronto, ON, Canada

17Departments of Family Medicine and Community Health Sciences, University of Calgary, Calgary, AB, Canada

18Division of Emergency Medicine, Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada

19Hôpital du Sacré-Coeur-de-Montréal, Centre intégré universitaire de santé et de services sociaux du Nord-de-l'Île-de Montréal, Montréal, QC, Canada

20Canadian Institute for Health Information, Ottawa, ON, Canada

21Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada

22Laboratoire ARIMED, GMF-U de Saint-Charles-Borromée, Québec, QC, Canada

23Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada

24Department of Emergency Medicine, Sunnybrook Research Institute, University of Toronto, Toronto, ON, Canada

25Institut universitaire de cardiologie et de pneumologie de Québec, Québec, QC, Canada

26Department of Critical Care Medicine, Medicine and Community Health Sciences, O'Brien Institute for Public Health, University of Calgary, Calgary, AB, Canada

27Department of Emergency Medicine, St Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada

Corresponding Author:

Simon Berthelot, MSc, MD, FRCP, CCFP(EM)

Axe Santé des populations et Pratiques optimales en santé

Centre de recherche du CHU de Québec-Université Laval

2705 Boulevard Laurier

Québec, QC, G1V 4G2


Phone: 1 418 525 4444 ext 46095


Background: In Canada, 30%-60% of patients presenting to emergency departments are ambulatory. This category has been labeled as a source of emergency department overuse. Acting on the presumption that primary care practices and walk-in clinics offer equivalent care at a lower cost, governments have invested massively in improving access to these alternative settings in the hope that patients would present there instead when possible, thereby reducing the load on emergency departments. Data in support of this approach remain scarce and equivocal.

Objective: The aim of this study is to compare the value of care received in emergency departments, walk-in clinics, and primary care practices by ambulatory patients with upper respiratory tract infection, sinusitis, otitis media, tonsillitis, pharyngitis, bronchitis, influenza-like illness, pneumonia, acute asthma, or acute exacerbation of chronic obstructive pulmonary disease.

Methods: A multicenter prospective cohort study will be performed in Ontario and Québec. In phase 1, a time-driven activity-based costing method will be applied at each of the 15 study sites. This method uses time as a cost driver to allocate direct costs (eg, medication), consumable expenditures (eg, needles), overhead costs (eg, building maintenance), and physician charges to patient care. Thus, the cost of a care episode will be proportional to the time spent receiving the care. At the end of this phase, a list of care process costs will be generated and used to calculate the cost of each consultation during phase 2, in which a prospective cohort of patients will be monitored to compare the care received in each setting. Patients aged 18 years and older, ambulatory throughout the care episode, and discharged to home with one of the aforementioned targeted diagnoses will be considered. The estimated sample size is 1485 patients. The 3 types of care settings will be compared on the basis of primary outcomes in terms of the proportion of return visits to any site 3 and 7 days after the initial visit and the mean cost of care. The secondary outcomes measured will include scores on patient-reported outcome and experience measures and mean costs borne wholly by patients. We will use multilevel generalized linear models to compare the care settings and an overlap weights approach to adjust for confounding factors related to age, sex, gender, ethnicity, comorbidities, registration with a family physician, socioeconomic status, and severity of illness.

Results: Phase 1 will begin in 2021 and phase 2, in 2023. The results will be available in 2025.

Conclusions: The end point of our program will be for deciders, patients, and care providers to be able to determine the most appropriate care setting for the management of ambulatory emergency respiratory conditions, based on the quality and cost of care associated with each alternative.

International Registered Report Identifier (IRRID): PRR1-10.2196/25619

JMIR Res Protoc 2021;10(2):e25619



The Problem: Emergency Department Overuse and Misuse

Emergency departments (EDs) are specialized and costly resources designed to provide care for patients with urgent or life-threatening conditions [1]. In Canada, low-acuity ambulatory patients, who do not require a gurney or constant observation, represent 30%-60% of all ED visits [2-7]. This situation is increasingly considered as overuse and misuse of ED resources and a threat to the quality of care received by patients whose needs are more urgent [8]. Delays experienced in an overcrowded ED can lead to mortality, morbidity, and reduced quality of life [9-14]. ED overcrowding is widely regarded as a serious but largely avoidable public health risk exacerbated by ambulatory patients [15,16].

An Important Policy Issue

Many Canadian regional health authorities have developed policies so that low-acuity ambulatory emergency patients preferably present to walk-in clinics or primary care practices [17-19]. Over the past decade, numerous innovations have been implemented to improve timely access to primary care, such as extended walk-in clinic hours [17-19] and the advanced access model (timely access to a care provider) for registered patients [20-26]. In Ontario and Québec, governments have invested massively in new models of primary care to improve access to emergency care and thereby decrease ED visits by patients who are treatable in non-ED settings [18,27]. These health policy priorities rely on the assumption that walk-in clinics and primary care facilities offer less costly, more accessible, and more efficient alternatives to the local population [17,28,29] than overcrowded EDs [3,4,7,8,30-34]. As reasonable as this assumption may appear, data supporting it are scarce and equivocal [35,36].

Determining the Best Care Setting for Ambulatory Emergency Patients: A Knowledge Gap

Few studies have tested the hypothesis that walk-in or primary care clinics offer better care than EDs to ambulatory patients with acute health concerns.

The Costs

A prospective study in Ontario in 2005 [28] concluded that for similar cases, ED costs were 3 to 4 times higher than the costs incurred in a family physician’s office or a walk-in clinic. However, compared costs were not adjusted for comorbidities or severity of disease and did not include out-of-pocket expenses (eg, parking) and indirect costs to patients (eg, loss of income). Other studies, mainly from the United States, have reached similar conclusions [17,37-39] but using charges as proxies of health care costs, which has been shown to be an inaccurate costing method [40,41]. Some reports even suggest that walk-in clinics may in fact increase overall health care costs by duplicating care with frequent return visits after an initial visit [42-45].

The Quality of Care

Very few studies have considered quality of care and patient health outcomes in determining the best alternative setting for treating ambulatory emergency patients [8]. A 2017 review (Cochrane) of prospective studies comparing mortality, morbidity, and adherence to practice guidelines in walk-in clinics, primary care practices, and EDs found that none met this criterion [46]. However, three retrospective studies [47-49] and one study evaluating costs and return visits [28] suggested that (1) inappropriate use of antibiotics for self-resolving acute respiratory conditions occurs more frequently following visits to urgent care centers and family medicine offices than to EDs [47-50]; (2) the choice of antibiotics is more concordant with practice guidelines in walk-in clinics than in EDs and family medicine practices [48]; and (3) return visit likelihood within 72 hours is higher after an ED care episode than after any other outpatient clinic visit [28]. However, these fragmented and incomplete data come mostly from the United States. A comprehensive research program comparing acute care received in EDs, walk-in clinics, and primary care practices in Canada is long overdue.

The Patient Perspective

Deciders often prioritize certain care settings based on potential cost savings, auctioning off care paths to the lowest bidder from the government’s perspective [51]. However, studies have shown that from a patient’s perspective, the choice to seek care in either a primary care practice, a walk-in clinic, or an ED is determined not only by ease of primary care access but also by factors such as convenience and perceived severity of illness and previous health care experiences [35,52-56]. What patients value the most differs considerably from what other stakeholders tend to value [57]. The patient’s perspective must be considered to determine the best ambulatory emergency care option. To our knowledge, no studies have compared these alternative settings from a patient’s perspective.

Conceptual Framework: A Value-Based Approach

To compare the different care setting possibilities for ambulatory emergency patients, we propose value-based assessment, an approach first described by Michael Porter in 2006 [58,59] and widely adopted since by researchers and health quality organizations around the world [60-66]. Value is defined in terms of health outcomes achieved per dollar spent [58,67,68]. It promotes the best care at the lowest cost, without isolating clinical issues from economic issues. Two essential components are needed: (1) a feasible and reliable costing method and (2) valid, reliable, and readily available outcome indicators, consistent with the priorities of patients, deciders, and care providers. This comprehensive paradigm aligns patients, deciders, and clinicians behind shared goals, based on patient preferences and scientific evidence.

Previous Preliminary Work

Our team has conducted a pilot study in which an ED and a primary care clinic offering walk-in services for frequent ambulatory acute conditions were compared in terms of costs of care and compliance with practice guidelines [69,70]. We reviewed the medical records of 918 adults with one of 13 targeted ambulatory acute conditions during the 2015 and 2016 fiscal year and applied a time-driven activity-based costing method. Time-driven activity-based costing has been found to provide more precise accounting than methods based on diagnosis-related groups and is simpler than conventional activity-based costing [41,71-73]. It assumes that the cost of a care episode is proportional to the time that the patient spends receiving the care. Costs of care are determined by allocating all direct costs (eg, staff salaries) and overhead (eg, building maintenance) to activities related to patient care, including physician charges [74,75]. This costing method has been used successfully in many care settings [65,71,76,77], and we adapted it for use in EDs and primary care practices [41,69,78,79]. The adjusted mean costs in each clinical setting for upper respiratory tract infection (URTI), a condition for which antibiotics and x-rays are generally not recommended [80,81] were determined and the clinical settings were compared on the basis of the process of care applied (Table 1).

Table 1. Mean cost of care and percentage of use of nonrecommended care applied to upper respiratory tract infection in a primary care practice and an emergency department.
VariablePrimary care practice (n=102)Emergency department (n=52)P value
Cost of carea (US $), (mean 95% CI)45.4 (38.4-53.4)59.8 (49.4-72.3)<.001
Process of care, % (95% CI)

Chest x-ray13.7 (7.7-22.0)26.9 (15.6-41.0).05

Antibiotics44.1 (34.3-54.3)5.8 (1.2-16.0)<.001

aMean value adjusted for age, sex, vital signs, comorbidities, and number of regular medications for upper respiratory tract infection.

On the basis of this preliminary study, we conclude that (1) time-driven activity-based costing is feasible in ED and primary care settings without requiring advanced information technologies or rigorously coded electronic medical records, 2 major barriers to conducting research in outpatient clinics, and (2) significant variations in costs and quality of care may exist between EDs and clinics, suggesting that a multicenter cohort study is warranted. However, this retrospective study highlighted major issues that only a prospective design can resolve: comorbidities (crucial to risk adjustment), disposition plans (crucial to assessing quality of care), and discharge diagnosis are not readily extractable from databases in the outpatient setting and are often missing or incomplete in medical notes. By manually reviewing thousands of visits logged in electronic records, our research assistants identified eligible cases one chart at a time. These major hurdles apply to outpatient clinics in all Canadian provinces. A retrospective design for a multicenter cross-jurisdictional study would have major methodological flaws because of the unlikeliness of obtaining comparable information across settings. More importantly, a retrospective study on administrative databases would not allow us to assess patients’ perspectives. Finally, a randomized controlled trial is not feasible for the population and settings under study because randomization would have to occur before any contact with the health system to assign patients to their treatment group. For these reasons, we believe that a prospective cohort study is the most appropriate design for identifying the best care setting for ambulatory emergency patients.


Our goal is to compare the health outcomes and costs of care received in EDs, walk-in clinics, and primary care practices by ambulatory patients presenting with acute respiratory conditions, namely, URTI, sinusitis, otitis media, pharyngitis, tonsillitis, bronchitis, influenza-like illness, pneumonia, acute asthma, or acute exacerbation of chronic obstructive pulmonary disease (COPD). We selected these conditions because many performance metrics have been validated previously for assessing the quality of care provided [82,83]. Highly prevalent in ambulatory emergency care before the COVID-19 pandemic [16,50], acute respiratory conditions are now putting even greater strain on already overstretched health care systems. In addition, the pandemic has shifted primary care services significantly toward telemedicine (ie, remote consultation by phone or videoconferencing) [84]. Determining where these patients can get the most effective care is a crucial issue. Our 3 specific objectives are to (1) estimate the costs of care processes administered by care providers in EDs, walk-in clinics, and primary care practices for acute respiratory conditions from the public payer’s perspective; (2) estimate and compare the cost of care episodes in EDs, walk-in clinics, and primary care practices for acute respiratory conditions from the public payer’s and patient’s perspectives; and (3) compare patient health outcomes and quality of care in these care settings when treating acute ambulatory respiratory conditions from the public payer’s and patient’s perspectives. To achieve these objectives, we propose a 4-year (from April 1, 2021, to March 31, 2025) research plan in 2 phases: a time-driven activity-based costing method study (objective 1) and a prospective cohort study (objectives 2 and 3).

Phase 1: A Time-Driven Activity-Based Costing Method Study


A time-driven activity-based costing study will first be performed for fiscal year April 1, 2021, to March 31, 2022. We shall estimate the cost of care processes administered by care providers (Objective 1) in 3 different models of ambulatory emergency care in Québec and Ontario: (1) discontinuous care in the ED (by physicians unfamiliar with the patients); (2) discontinuous care in a walk-in clinic (by physicians unfamiliar with the patients); and (3) continuous care in primary care clinic (patients attached to a primary care practice, seen by their family physician or a colleague on a same-day appointment for urgent needs).

We have confirmed the participation of 14 of the 15 planned patient recruitment sites (ED 5/5, walk-in 5/5, primary care 4/5; Multimedia Appendix 1). They have been selected in different types of urban areas, including small (Joliette), medium (Kingston), large (Québec City, Ottawa), and metropolitan (Montreal) cities. In each participating region, an ED will be paired with a nearby walk-in clinic and primary care practice. We are currently securing our final additional clinic in Ottawa with the help of BeACCoN (Better Access and Care for Complex Needs), a provincial primary care research network.


The time-driven activity-based costing method [72,85] will enable us to derive for each setting the cost of care processes (eg, triage) and traceable supplies (eg, medication) potentially provided to patients with acute respiratory conditions, which includes telemedicine. This costing method requires only 2 parameters, namely, the unit cost of supplying capacity and the duration of processes, and comprises the following steps:

  1. Process (eg, salbutamol in acute asthma) and resource (eg, respiratory therapist) mapping through discussion with local teams for each respiratory condition (Figure 1)
  2. Validation of process maps and durations by on-field research assistants prospectively observing patients and measuring process duration using time-motion software (UMT Plus [Laubrass])
  3. Calculation, with local administrative teams, of total annual overhead costs (eg, building maintenance) related to the care of ambulatory patients with acute conditions (Multimedia Appendix 2 for allocation rules)
  4. Estimation of cost per time unit ($/minute) for the following cost elements obtained by dividing yearly expenses for a cost element by the total yearly number of minutes worked by professionals to care for patients in this facility (Multimedia Appendix 3): (1) human resources (eg, nurses, physicians) or equipment (eg, x-ray machine), (2) consumable supplies (eg, gloves, needles, paper), and (3) overhead costs
  5. Estimation of the cost of traceable supplies (eg, laboratory testing)
  6. Calculation of the average cost of each health care process (Multimedia Appendicies 3 and 4)
Figure 1. Process mapping for upper respiratory tract infections in the emergency department (truncated). Each box represents a process with its duration. Colors identify human resources (red=nurses; yellow=clerks; green=physicians). ED: emergency department.
View this figure

The cost of a care process is proportional to the mean duration measured on field. For example, the cost of triage is estimated by adding up the expenses associated with the triage nurse, consumables, and overhead. These elements will be estimated by multiplying the mean triage duration by their unit costs as follows:

Cost of triage = mean triage duration × (unit cost of nurse + unit cost of consumables+ unit cost of overhead) = 7.1 min × (US $0.78/min + US $0.07/min + US $0.17/min) = US $7.24 (Can $1 [US $0.76])

The cost of telemedicine will be estimated following the same steps, from resource mapping and time measurement through allocation of overhead and consumables, all the way to average cost calculation.

Where applicable, the following adjustments will be made so that the estimated costs reflect the public payer’s perspective: (1) expenses paid by physicians or owners of a participating clinic will be subtracted from the yearly expenses related to the appropriate cost element (eg, salaries, overhead); (2) similarly, government funding received by a clinic apart from physician remuneration will be added.

Financial Data Sources

The accounting department at each participating site will provide all financial data, except for physician charges. To calculate the unit cost for physicians, the total amount charged by all physicians per site per year will be obtained from local private billing agencies. This sum will be divided by the number of minutes spent delivering patient care, which will be obtained from physician schedules.

Intermediate Outputs of Phase 1

In addition to institution-specific costs, upon completing phase 1, we will create a list of standardized costs of care for each process and associated traceable supplies based on the average costs estimated in the 15 institutions (ED, walk-in, primary care, both provinces). Use of standardized costs will eliminate price effects because of differential costings between sites and provinces, thereby facilitating comparisons between the 3 clinical settings. In phase 2, the cost of a care episode will be calculated per individual by summing the standardized costs of care processes, supplies, and drugs received by each patient during their visit. Fixed and variable costs will be broken down to estimate and compare the care settings in terms of the marginal cost of each new patient assessed [86].

Phase 2: A Prospective Cohort for Comparing the 3 Health Care Settings

Design and Setting

A multicenter prospective cohort study will be conducted in the institutions included in phase 1 to compare the value of care in EDs, walk-in clinics, and primary care practices (Objectives 2 and 3).

Selection of Participants

We shall include patients (1) aged 18 years and older; (2) seen in person or via telemedicine in an ED, a walk-in clinic, or the primary care practice where they are registered; (3) ambulatory during the entire visit or consultation; and (4) discharged home with a diagnosis of URTI, sinusitis, otitis media, pharyngitis, tonsillitis, bronchitis, influenza-like illness, pneumonia, acute asthma, or acute exacerbation of COPD. We shall exclude patients (1) transported by ambulance, (2) not covered by the provincial health insurance plan, (3) having consulted for a similar problem in the previous 30 days as patients with refractory diseases representing a population with different care needs, (4) living in a long-term health care facility or incarcerated, or (5) receiving palliative care.

Recruitment Procedures on the Initial Visit

A research nurse in collaboration with local clerks at each site will screen eligible patients after on-site registration or web-based scheduling, but before assessment by a physician, based on presenting complaints suggestive of acute respiratory conditions. After assessment and once a targeted diagnosis is confirmed, the same research nurse will prospectively (1) obtain consent from patients; (2) ensure that the discharge diagnosis, comorbidities, and disposition plans are fully documented; and (3) administer a questionnaire to assess patient experience of care and motivation for choosing one care setting over the other. Motivation will be classified into the 6 domains of the Conceptual Model of Emergency Department Use [35]. Participants will be asked to specify whether their choice of care setting was based on accessibility, convenience, their perception of the severity of illness, their beliefs and knowledge regarding these care settings, referral and advice from a care professional or an acquaintance, or costs. They will also be requested to rate their perception of illness severity. For on-site participants only, the research nurse will also (1) assess the severity using the Pandemic Medical Early Warning Score (PMEWS), a validated severity score allowing points for age, vital signs, comorbidity, social situation, and functional status [87], and (2) perform spirometry (measured parameter: forced expiratory volume in the first second [FEV1]) on patients with acute asthma. A random sampling recruitment schedule will be planned to ensure a proportional representation of the hours of operation for each recruiting site. Participating EDs will recruit on a schedule similar to their paired participating clinics to include participants who could have consulted in an alternative setting and exclude night patients who differ significantly from patients seeking care during the day [88]. Recruitment will occur over a full year to encompass seasonal variability in the incidence of respiratory diseases.

Data Collection and Follow-Up Phone Calls

Research assistants at each site will complete data collection from local medical records. For on-site participants and, where appropriate, for those assessed by telemedicine, they will compile the following information: age, sex, gender, ethnicity, postal code, distance from facility to home, referral by the provincial telephone consultation service (811, Telehealth), enrollment with a family physician, presenting complaints, comorbidities, regular medications, date and time of arrival and discharge, vital signs upon arrival, investigations and interventions during care episode, discharge diagnosis, and prescriptions upon discharge. A follow-up phone call will be made to all participants 10 days after the initial visit to collect data initially unavailable in medical records and to evaluate primary and secondary outcome metrics. Patient-reported outcome and cost measures will be completed by the participants at this moment, either on the phone with the research assistant or independently using a secured online survey link, depending on the participant’s preference. Text messaging and email reminders will be sent to improve participant retention [89]. We shall obtain information on health outcomes (eg, mortality) and physician charges via provincial databases. The charges billed by any physician 7 days after the initial visit will be used to estimate the costs of care for subsequent return visits and hospital admissions.

Outcome Measures

A value-based comparative assessment requires the simultaneous evaluation of health outcomes and costs. Our outcome measures were chosen from a guideline on the assessment of ED performance [90] and recent literature on patient experience assessment (Table 2) [91-95]. The initial visit, from arrival at a participating site to discharge, represents the unit of analysis for all outcome measures; however, the health system or patient costs incurred during the following week will be estimated and added to the cost of the initial visit. For participants assessed in person, the outcome will be scored per care setting and further stratified per discharge diagnosis and by province, using institution-specific costs for interprovincial cost comparisons. We will analyze the same outcome measures separately in the case of patients evaluated by telemedicine, as missing data (eg, vital signs) will prevent us from adjusting for the severity of their illness.

Table 2. Main study outcomes.

Incidence of return visit (Oa)Proportion of patients returning to any EDb or outpatient clinic at 72 hours and 7 days after the initial visit [83,96-99]. An adjudication committee will review records of return visits to classify them as planned or unplanned and avoidable or unavoidable.Follow-up call at 10 days

Mean cost of care−the Ministry of Health perspective (Cc)The cost per care episode calculated by summing the costs of all care processes delivered to a patient during the initial visit plus the costs of return visits and admissions at 72 hours and 7 days.Electronic medical records and provincial billing databases

Median PROM-EDd patients scores (O)Developed and validated by team member SV, the PROM-ED questionnaire provides a measurement of patient-reported outcome expressed as scores for symptom relief, understanding of health concern, reassurance, and having a plan for care [91,94].Follow-up call at 10 days

Median scores on a PREMe (O)We adapted and are validating a tool from patient experience surveys used in EDs and primary care clinics in Ontario [100-102]. This tool evaluates the patient’s view of care delivery and measures various dimensions of patient experience relevant to all care alternatives, such as attitude of providers.At the end of the initial visit

Mean CoPaQf (C)A questionnaire measuring patients’ and caregivers’ out-of-pocket expenses (eg, travel) and indirect costs (eg, loss of income) will be proposed to participants. This questionnaire was developed and validated by members of our team (ML, JG, SB) and further adapted for use in this study.Follow-up call at 10 days

Incidence of admission, intensive care unit, or mortality (O)Proportion of patients who were admitted to the hospital or the intensive care unit or died because of one of the targeted respiratory conditions within 30 days [83,103] after the initial visit.Provincial databases: Med-Echo, ICES, death registries

Wait timesMedian and mean length of stay and time spent waiting to see a physician.Electronic medical records

aO: health outcome.

bED: emergency department.

cC: health cost.

dPROM-ED: patient-reported outcome measure for ED.

ePREM: patient-reported experience measure.

fCoPaQ: cost-for-patient questionnaire.

To evaluate the quality of care in each group under study, compliance with practice guidelines (eg, corticosteroid prescription for asthma) for the treatment of respiratory diseases [104-109] will be compared (the full list of outcome measures is given in Multimedia Appendix 5 [83,91,94,96-103]). Return visits will be reported by the participants during the 10-day follow-up phone call. The Canadian Institutes of Health Research bridge grant obtained in April 2020 allowed our team to adapt questionnaires assessing patient perspective (Patient-Reported Experience Measure [PREM], patient-reported outcome measure for ED patients [PROM-ED], and cost-for-patient questionnaire [CoPaQ]) for use in any setting under evaluation. Their use for patients seen in person or by telemedicine in ambulatory patients with acute respiratory conditions will be validated further in the fall of 2020.

Sample Size

As our main analyses focus on patients assessed in person, our sample size calculation is based solely on their numbers. We estimate that the rate of return visit for ambulatory emergency conditions varies from 1% to 13% depending on the care setting [17,110,111]. To account for the potential similarity in outcomes among individuals in each of the 15 clusters, we assumed a realistic intracluster correlation of =0.02 based on previous studies and applied a correction to inflate our sample size calculation [112,113]. Using data from Campbell et al [28], at least 1485 patients (approximately 99 per cluster) will be needed to reveal a 5% difference in the proportion of return visits within 72 hours (eg, 5% vs 10%), assuming a 20% loss to follow-up and at least 30 participants per condition, based on multivariate logistic regression power analysis, type I error (α) at .05, and power at 80% (1−β). Assuming 240 recruitment days over a year at each site and the recruitment of at least 1 to 2 patients per day, our final cohort should include over 4000 patients and reach the minimal sample size in both participating provinces, which will allow for more robust comparisons and analyses.

Statistical Analysis

All main analyses will be conducted primarily on participants assessed in person in any of the care settings. Participants assessed by telemedicine will be analyzed and compared separately between sites where it is implemented. The value delivered at each participating site and on average in each care setting type will be illustrated with an operational effectiveness graphic [114]. Adjusted costs of care for acute ambulatory respiratory conditions will be plotted on the x-axis and adjusted return visits within 72 hours on the y-axis. Points closest to 0 on both axes represent the highest value of care (Multimedia Appendix 6). Indeed, the lower the return visit proportion and cost of care, the higher the value of care. Similar graphics will be used for patient-centered outcome measures. To compare EDs, walk-in clinics, and primary care physician practices, multilevel generalized linear models will be used with probability distributions adapted to the outcome under evaluation. To adjust for confounding (differences in case mix between care settings), an overlap weights approach [115,116] will be used, wherein each individual receives a weight factor that is proportional to the inverse of the probability of choosing a particular setting. Subjects that differ fundamentally between settings are attributed a weight of 0 and are thus excluded. This approach is of the greatest interest when groups are initially very different [116]. Intuitively, overlap weights create a pseudopopulation for which treatment is independent of measured confounders, thus mimicking a randomized trial of those confounders. Overlap weights have been shown to be more robust than conventional inverse probability weighting and matching based on propensity scores [115]. The weights will be estimated using a multinomial logistic regression model in which the dependent variable is the chosen care setting and independent variables are potential confounders or risk factors [117] for the outcomes identified in the literature: age [118,119], sex and gender [118,120], ethnicity [118,121-123], registration with a family physician [20,124], comorbidities (the Charlson index; number of regular medications) [118,125,126], asthma, FEV1 among patients with asthma, the Canadian deprivation index [127-129], patient perception of illness severity [130], and vital signs [131-135] as proxies for severity. The same independent variables will be used to adjust for differences in case mix between settings in the telemedicine cohort, excluding vital signs and FEV1. Multiple imputation will be considered as a possible means to adjust for these variables in this cohort. Overlap weights will be calculated using the values predicted by this model. We shall verify that the care setting groups are comparable according to the measured confounders after weighting by computing standardized mean differences. Differences below 10% will be considered to indicate good balance [117]. If residual imbalances are present, the weighting model will be revised. Once an appropriate balance is achieved, separate models for each outcome will be fitted to the weighted data, for which the care setting will be the only independent variable. The robustness of results with respect to unmeasured confounding will be assessed using the E-value [136,137]. Clustering by setting (eg, province, practice unit) will be taken into account using multilevel modeling (random intercept on province and practice unit). Reported cost estimates will be calculated with item-specific standardized costs (eg, Québec and Ontario average nurse unit cost). Patients referred to the ED from a participating outpatient facility but discharged home after ED assessment will be analyzed in the care setting group where they first presented, and the ED referral will be considered as a return visit. The costs of any return visits and admissions up to 7 days after the initial visit will be estimated separately and attributed to the care setting where the initial visit took place. Results of the 3 questionnaires from the patient perspective will be reported as proportions (PREM), mean costs (CoPaQ), and median scores (PROM-ED) and adjusted using the overlap weights approach. As patients seek care for symptoms, subanalysis based on presenting complaints instead of discharge diagnosis will be conducted to provide meaningful patient-oriented results. Other subanalyses will evaluate which patient profile (eg, gender [120], motivation for choosing a facility), and institutional characteristics (eg, access to x-ray) predict high quality and low costs, keeping in mind that our value assessment might not yield similar results for all subgroups or even within a group of patients with the same diagnosis. Statistical differences will be assessed with a significance threshold set at .05.

Sensitivity Analyses

To assess potential uncontrolled confounding of the results, sensitivity analyses will be conducted by excluding separately and concurrently the participants most likely to influence the effects of the 3 types of care settings: (1) ≥65 years; (2) with ≥1 comorbidity; (3) with either asthma or COPD; (4) with ≥1 regular medication; (5) with any abnormal vital signs; (6) in the lowest and highest quartile of the deprivation index; and (7) smokers. The analyses will be repeated using PMEWS instead of vital signs as a marker of illness severity. To control for a potential Hawthorne effect, the analyses will be repeated, with the first 3 months of recruitment excluded to focus on the data collected after the providers have become used to being observed.

Study Preparation

From our pilot studies reported earlier until now, our team has made significant progress to reach its goal of identifying the care pathways providing the highest value to ambulatory emergency patients. We have assembled a very strong research team composed of patients, clinicians, administrators, and researchers. Together, we have created this paper. Two patient partners met with us regularly and provided helpful comments to make our research plan more patient centered. We have secured 14 of 15 planned participating sites. We have adapted the 3 patient-centered tools (PREM, PROM-ED, and CoPaQ) and are currently validating their use on ambulatory emergency patients whether they receive care in an ED, a walk-in clinic, or a primary care practice.

Protocol Endorsement

Our protocol has been endorsed by the Network of Canadian Emergency Researchers (NCER). The broad support for our research initiative from leading Canadian organizations in emergency (NCER, Canadian Association of Emergency Physicians) and primary care (Réseau-1, Réseau de recherche axée sur les pratiques de première ligne, BeACCoN Ontario, Réseau sur les Innovations en soins de santé de première ligne et intégrés, Strategy for Patient-Oriented Research Unit), from the Ministries of Health of Ontario and Québec, and from organizations dedicated to improving health care throughout Canada (PULSAR, Canadian Institute for Health Information, Institut national d’excellence en santé et services sociaux, ICES) demonstrates the importance of the issue being addressed.

Study Timeline

Phase 1 will begin in 2021 and will allow us to compare the cost of care from the public payer perspective in 3 different settings and 2 Canadian provinces. We expect that the results from this phase will be available in 2023. Phase 2 will begin in 2023 and will evaluate the value of the care in each setting under study. The final results will be published in 2025 and 2026. Our 4-year program covering the period of April 1, 2021, to March 31, 2025, is presented in a Gantt diagram available in Multimedia Appendix 7.


Our unique multidimensional approach to examining the quality and cost of care using both patient and system perspectives will provide knowledge that will be helpful in determining whether EDs, walk-in clinics, or primary care practices offer the best value to patients with acute ambulatory respiratory conditions. We expect our study to yield tangible benefits for all stakeholders.

  1. For guiding policy and decisions: Despite weak evidence, Canadian provinces have invested massively in alternative care pathways to get ambulatory patients with urgent care needs to rely less on hospital EDs. Data generated by the proposed study will have an immediate impact by providing hard evidence in support of health care planning decisions intended to improve the service quality/cost ratio and hence outcomes in the largest patient category.
  2. For patients: Current policies are designed for statistically average ambulatory emergency patients without considering patient perspectives and the widely variable severity of each diagnosed illness. As the needs and preferences of patients with pharyngitis likely differ from those with exacerbated COPD, our stratified results per condition will enable policy makers to structure urgent care systems to provide better-adapted higher value services to each specific category of patients. Our comprehensive research initiative will bring patient preferences and perspectives into policy making.
  3. For clinicians: Our study will be a powerful driver for quality improvement in all care settings involved. Care quality can vary considerably, and we hope to generate unique opportunities for valid and meaningful comparisons and for quality improvement initiatives throughout the country.

Challenges and Mitigation Strategies

First, as patients choose their facility, those presenting at the 3 types of setting will likely represent different populations. However, we believe that the potential confounding bias due to self-selection of the care setting can be overcome using the overlap weights approach. Extensive testing of the robustness of our findings by sensitivity analyses should allow us to avoid reaching false conclusions under the influence of uncontrolled confounding. Second, the Québec and Ontario health systems might differ enough to yield results that will not be easy to generalize. When applicable, the sources of heterogeneity will be investigated. However, Canadian provincial health care systems have fundamental similarities that reduce the risk of poor generalizability. All are based on universal coverage; all suffer from a lack of integration between primary and urgent care resources [138,139]; institutions follow the recommendations of the same accreditation organizations; and care providers are trained according to the same standards and guidelines. Third, because of the pandemic, many outpatient clinics have ceased their activities or shifted to telemedicine. Our research plan already includes participants evaluated by telemedicine and will adapt easily to any increase in this practice. If the pandemic is still ongoing in November 2021 when phase 1 is launched, we will be able to collect financial data from participating institutions, which can be done remotely. Time measurement of care processes can be postponed until phase 2 in 2023, during which the recruitment of participants is planned. Finally, if the pandemic is still a factor in 2023, we will select clinics that continue to assess patients with acute respiratory disorders.


Ambulatory emergency patients account for 30% to 60% of all ED visits in Canada. This burden on emergency care is now exacerbated by the COVID-19 pandemic. This category of patients is thought to be amenable to using walk-in clinics or primary care practices and is the focus of redirection strategies meant to decrease ED overuse. However, current knowledge is inadequate for reaching any firm conclusions about which care settings are best suited for this purpose. The aim of this study is to compare the value of the care that these patients receive in EDs, walk-in clinics, and primary care practices, thereby providing arm administrators and care providers with new and robust knowledge that will enable them to determine the best care setting for the management of respiratory ambulatory emergency conditions. We all agree that the system can only benefit from patients receiving timely care in the proper setting from the most suitable provider.


This work was funded by a Canadian Institutes of Health Research Project under Grant RN #169210. The authors would like to thank Yves Bolduc, Jean-Sébastien Audette, Michel Lafrenière, Benoît Maheu, Susan Phillips, Bruno J L’Heureux, Danièle Roberge, and Myriam Nadeau for their contributions.

Authors' Contributions

SB is the grant holder and nominated principal investigator of the project. MB and JG are the coprincipal investigators. All authors contributed ideas and read and approved the final manuscript.

Conflicts of Interest

Dr. Patrick M. Archambault has completed research contracts with Thales Digital Solutions to develop medical decision support systems. Dr. Alexandre Messier is the inventor of a redirection solution via a web application and works as a medical consultant for Logibec, the company responsible for its marketing and distribution.

Multimedia Appendix 1

Participating sites per province and care setting.

PDF File (Adobe PDF File), 289 KB

Multimedia Appendix 2

Summary of overhead expenses.

PDF File (Adobe PDF File), 306 KB

Multimedia Appendix 3

Cost per time unit (Can $/min) of cost elements and estimated cost (Can $) of important care processes.

PDF File (Adobe PDF File), 194 KB

Multimedia Appendix 4

Steps of the time-driven activity-based costing method.

PDF File (Adobe PDF File), 342 KB

Multimedia Appendix 5

Complete list of study outcomes.

PDF File (Adobe PDF File), 143 KB

Multimedia Appendix 6

Example of an operational effectiveness graphic with hypothetical numbers.

PDF File (Adobe PDF File), 282 KB

Multimedia Appendix 7

Gantt diagram—value project.

PDF File (Adobe PDF File), 211 KB

  1. Closer than you think: Linking primary care to emergency department use in Quebec. St. Mary’s Research Centre. 2013.   URL: http:/​/www.​​ignitionweb/​data/​media_centre_files/​737/​Closer%20than%20you%20think%20_%20Feb%2025.​pdf [accessed 2016-09-27]
  2. Les urgences au Québec: Évolution de 2003-2004 à 2012-2013. Le Commissaire à la santé et au bien-être du Québec. 2014.   URL: [accessed 2016-09-27]
  3. Vertesi L. Does the Canadian Emergency Department Triage and Acuity Scale identify non-urgent patients who can be triaged away from the emergency department? CJEM 2004 Oct;6(5):337-342. [CrossRef] [Medline]
  4. Ismail SA, Gibbons DC, Gnani S. Reducing inappropriate accident and emergency department attendances: a systematic review of primary care service interventions. Br J Gen Pract 2013 Dec;63(617):e813-e820 [FREE Full text] [CrossRef] [Medline]
  5. Apprendre des meilleurs : Étude comparative des urgences du Québec. Le Commissaire à la santé et au bien-être du Québec. 2016.   URL: [accessed 2016-09-27]
  6. A Matter of Urgency: Reducing Emergency Department Overuse. New England Health Institute. 2010.   URL: [accessed 2016-10-03]
  7. Schull MJ, Kiss A, Szalai J. The effect of low-complexity patients on emergency department waiting times. Ann Emerg Med 2007 Mar;49(3):257-64, 264.e1. [CrossRef] [Medline]
  8. Morgan SR, Chang AM, Alqatari M, Pines JM. Non-emergency department interventions to reduce ED utilization: a systematic review. Acad Emerg Med 2013 Oct;20(10):969-985. [CrossRef] [Medline]
  9. Singer AJ, Thode HC, Viccellio P, Pines JM. The association between length of emergency department boarding and mortality. Acad Emerg Med 2011 Dec;18(12):1324-1329 [FREE Full text] [CrossRef] [Medline]
  10. Bernstein SL, Aronsky D, Duseja R, Epstein S, Handel D, Hwang U, et al. The effect of emergency department crowding on clinically oriented outcomes. Acad Emerg Med 2009 Jan;16(1):1-10 [FREE Full text] [CrossRef] [Medline]
  11. Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ 2011 Jun 01;342:d2983 [FREE Full text] [CrossRef] [Medline]
  12. Kennebeck SS, Timm NL, Kurowski EM, Byczkowski TL, Reeves SD. The association of emergency department crowding and time to antibiotics in febrile neonates. Acad Emerg Med 2011 Dec;18(12):1380-1385 [FREE Full text] [CrossRef] [Medline]
  13. Pines JM, Localio AR, Hollander JE, Baxt WG, Lee H, Phillips C, et al. The impact of emergency department crowding measures on time to antibiotics for patients with community-acquired pneumonia. Ann Emerg Med 2007 Dec;50(5):510-516. [CrossRef] [Medline]
  14. Sills M, Fairclough D, Ranade D, Mitchell M, Kahn M. Emergency department crowding is associated with decreased quality of analgesia delivery for children with pain related to acute, isolated, long-bone fractures. Acad Emerg Med Dec 2011;18(12):1330-1338. [CrossRef] [Medline]
  15. Howard M, Goertzen J, Kaczorowski J, Hutchison B, Morris K, Thabane L, et al. Emergency Department and Walk-in Clinic Use in Models of Primary Care Practice with Different After-Hours Accessibility in Ontario. Healthc Policy 2008 Aug;4(1):73-88 [FREE Full text] [Medline]
  16. Sources of Potentially Avoidable Emergency Department Visits. Canadian Institute for Health Information. 2014.   URL: [accessed 2016-10-05]
  17. Weinkauf DJ, Kralj B. Medical Service Provision and Costs: Do Walk-in Clinics Differ from Other Primary Care Delivery Settings? Canadian Public Policy / Analyse de Politiques 1998 Dec;24(4):471. [CrossRef]
  18. Meilleur accès aux soins de première ligne ? Le ministre Barrette annonce la mise en oeuvre des super-cliniques. Ministère de la santé et des services sociaux du Québec. 2016.   URL: [accessed 2019-09-13]
  19. Broekhuis SM, van Dijk WD, Giesen P, Pavilanis A. Walk-in clinics in Quebec, Canada: patients and doctors do not agree on appropriateness of visits. Fam Pract 2014 Mar;31(1):92-101. [CrossRef] [Medline]
  20. McCusker J, Tousignant P, Borgès Da Silva R, Ciampi A, Lévesque J, Vadeboncoeur A, et al. Factors predicting patient use of the emergency department: a retrospective cohort study. CMAJ 2012 May 03;184(6):E307-E316 [FREE Full text] [CrossRef] [Medline]
  21. Glazier RH, Kopp A, Schultz SE, Kiran T, Henry DA. All the right intentions but few of the desired results: lessons on access to primary care from Ontario's patient enrolment models. Healthc Q 2012;15(3):17-21. [CrossRef] [Medline]
  22. La performance du système de santé et de services sociaux québécois 2015. Commissaire à la santé et au bien-être.   URL: http:/​/www.​​fileadmin/​www/​2015/​PerformanceGlobale/​CSBE_Rapport_Global_2015_Accessible.​pdf [accessed 2016-02-19]
  23. Comparison of Primary Care Models in Ontario by Demographics, Case Mix and Emergency Department Use, 2008/09 to 2009/10. Institute for Clinical Evaluative Sciences. 2012.   URL: https:/​/www.​​flip-publication/​comparison-of-primary-care-models-in-ontario-by-demographics/​files/​assets/​basic-html/​index.​html#1 [accessed 2016-09-30]
  24. The Impact of Extended Hours Primary Care on Emergency Department Use Among Medicaid/SCHIP Enrollees in Houston, TX. The University of Texas School of Public Health, 1Texas Children’s Health Plan.   URL: [accessed 2016-10-01]
  25. Van Der Biezen M, Adang E, Van Der Burgt R, Wensing M, Laurant M. The impact of substituting general practitioners with nurse practitioners on resource use, production and health-care costs during out-of-hours: a quasi-experimental study. BMC Fam Pract 2016 Sep 13;17(1):132 [FREE Full text] [CrossRef] [Medline]
  26. Breton M, Maillet L, Paré I, Abou Malham S, Touati N. Perceptions of the first family physicians to adopt advanced access in the province of Quebec, Canada. Int J Health Plann Manage 2017 Oct;32(4):e316-e332. [CrossRef] [Medline]
  27. Hutchison B, Levesque J, Strumpf E, Coyle N. Primary health care in Canada: systems in motion. Milbank Q 2011 Jul;89(2):256-288. [CrossRef] [Medline]
  28. Campbell M, Silver R, Hoch J, Ostbye T, Stewart M, Barnsley J, et al. Re-utilization outcomes and costs of minor acute illness treated at family physician offices, walk-in clinics, and emergency departments. Can Fam Physician 2005 Jan;51:82-83 [FREE Full text] [Medline]
  29. Nath JB, Costigan S, Lin F, Vittinghoff E, Hsia RY. Access to Federally Qualified Health Centers and Emergency Department Use Among Uninsured and Medicaid-insured Adults: California, 2005 to 2013. Acad Emerg Med 2019 Feb;26(2):129-139 [FREE Full text] [CrossRef] [Medline]
  30. Gentile S, Vignally P, Durand A, Gainotti S, Sambuc R, Gerbeaux P. Nonurgent patients in the emergency department? A French formula to prevent misuse. BMC Health Serv Res 2010 Mar 15;10:66 [FREE Full text] [CrossRef] [Medline]
  31. Smulowitz PB, Honigman L, Landon BE. A novel approach to identifying targets for cost reduction in the emergency department. Ann Emerg Med 2013 Mar;61(3):293-300. [CrossRef] [Medline]
  32. Cadre de référence pour la prévention et la gestion des maladies chroniques physiques en première ligne. Ministère de la santé et des services sociaux du Québec. 2012.   URL: [accessed 2016-02-26]
  33. Signature d'une entente de principe entre le gouvernement du Québec et la Fédération des médecins omnipraticiens du Québec. Ministère de la santé et des services sociaux du Québec. 2015.   URL: [accessed 2017-09-09]
  34. Plan stratégique du Ministère de la Santé et des Services sociaux du Québec 2019-2023. Ministère de la santé et des services sociaux du Québec. 2019.   URL: https:/​/cdn-contenu.​​cdn-contenu/​adm/​min/​sante-services-sociaux/​publications-adm/​plan-strategique/​PL_19-717-02W_MSSS.​pdf [accessed 2019-11-16]
  35. Uscher-Pines L, Pines J, Kellermann A, Gillen E, Mehrotra A. Emergency department visits for nonurgent conditions: systematic literature review. Am J Manag Care 2013 Jan;19(1):47-59 [FREE Full text] [Medline]
  36. Van den Heede K, Van de Voorde C. Interventions to reduce emergency department utilisation: A review of reviews. Health Policy 2016 Dec;120(12):1337-1349 [FREE Full text] [CrossRef] [Medline]
  37. Baker LC, Baker LS. Excess cost of emergency department visits for nonurgent care. Health Aff (Millwood) 1994;13(5):162-171. [CrossRef] [Medline]
  38. Martin BC. Emergency medicine versus primary care: a case study of three prevalent, costly, and non-emergent diagnoses at a community teaching hospital. J Health Care Finance 2000;27(2):51-65. [Medline]
  39. Warren BH, Isikoff SJ. Comparative costs of urgent care services in university-based clinical sites. Arch Fam Med 1993 May;2(5):523-528. [CrossRef] [Medline]
  40. Finkler SA. The distinction between cost and charges. Ann Intern Med 1982 Jan;96(1):102-109. [CrossRef] [Medline]
  41. Yun BJ, Prabhakar AM, Warsh J, Kaplan R, Brennan J, Dempsey KE, et al. Time-Driven Activity-Based Costing in Emergency Medicine. Ann Emerg Med 2016 Jun;67(6):765-772. [CrossRef] [Medline]
  42. Salisbury C, Munro J. Walk-in centres in primary care: a review of the international literature. Br J Gen Pract 2003 Jan;53(486):53-59 [FREE Full text] [Medline]
  43. Jones M. Walk-in primary medical care centres: lessons from Canada. BMJ 2000 Oct 14;321(7266):928-931 [FREE Full text] [CrossRef] [Medline]
  44. Bell N, Szafran O. Use of walk-in clinics by family practice patients. Can Fam Physician 1992;38:507-513.
  45. Ashwood JS, Gaynor M, Setodji CM, Reid RO, Weber E, Mehrotra A. Retail Clinic Visits For Low-Acuity Conditions Increase Utilization And Spending. Health Aff (Millwood) 2016 Mar;35(3):449-455. [CrossRef] [Medline]
  46. Chen CE, Chen CT, Hu J, Mehrotra A. Walk-in clinics versus physician offices and emergency rooms for urgent care and chronic disease management. Cochrane Database Syst Rev 2017 Feb 17;2:CD011774 [FREE Full text] [CrossRef] [Medline]
  47. Mehrotra A, Gidengil CA, Setodji CM, Burns RM, Linder JA. Antibiotic prescribing for respiratory infections at retail clinics, physician practices, and emergency departments. Am J Manag Care 2015 Apr;21(4):294-302 [FREE Full text] [Medline]
  48. Mehrotra A, Liu H, Adams JL, Wang MC, Lave JR, Thygeson NM, et al. Comparing costs and quality of care at retail clinics with that of other medical settings for 3 common illnesses. Ann Intern Med 2009 Sep 01;151(5):321-328 [FREE Full text] [CrossRef] [Medline]
  49. Palms DL, Hicks LA, Bartoces M, Hersh AL, Zetts R, Hyun DY, et al. Comparison of Antibiotic Prescribing in Retail Clinics, Urgent Care Centers, Emergency Departments, and Traditional Ambulatory Care Settings in the United States. JAMA Intern Med 2018 Sep 01;178(9):1267-1269 [FREE Full text] [CrossRef] [Medline]
  50. The evolving role of retail clinics. RAND Corporation. 2016.   URL: [accessed 2018-06-28]
  51. Desrosiers E. Québec révisera la règle du plus bas soumissionnaire. Le Devoir 2019:30 [FREE Full text]
  52. Masso M, Bezzina AJ, Siminski P, Middleton R, Eagar K. Why patients attend emergency departments for conditions potentially appropriate for primary care: reasons given by patients and clinicians differ. Emerg Med Australas 2007 Aug;19(4):333-340. [CrossRef] [Medline]
  53. Le Sage N, Gagnon M, Fratru R, Emond M. PACSUNU: problematic of ambulatory care visits that are semi-urgent and non-urgent. CJEM Mai 2010;12(3):265.
  54. Butler PA. Medicaid HMO enrollees in the emergency room: use of nonemergency care. Med Care Res Rev 1998 Mar;55(1):78-98. [CrossRef] [Medline]
  55. Schwartz MP. Office or emergency department: what's the difference? South Med J 1995 Oct;88(10):1020-1024. [CrossRef] [Medline]
  56. Tan S, Mays N. Impact of initiatives to improve access to, and choice of, primary and urgent care in the England: a systematic review. Health Policy 2014 Dec;118(3):304-315. [CrossRef] [Medline]
  57. Sharp AL, Cobb EM, Dresden SM, Richardson DK, Sabbatini AK, Sauser K, et al. Understanding the value of emergency care: a framework incorporating stakeholder perspectives. J Emerg Med 2014 Oct;47(3):333-342. [CrossRef] [Medline]
  58. Porter ME. A strategy for health care reform--toward a value-based system. N Engl J Med 2009 Jul 09;361(2):109-112. [CrossRef] [Medline]
  59. Porter M, Teisberg E. Redefining Health Care: creating value-based competition on results. In: Harvard Business School - Faculty & Research. Boston: Harvard Business School Press; 2006:1-59139.
  60. Better value in the NHS: The role of changes in clinical practice. The King's Fund. 2015 Jul.   URL: https:/​/www.​​sites/​default/​files/​field/​field_publication_file/​better-value-nhs-Kings-Fund-July%202015.​pdf [accessed 2018-08-06]
  61. Value-based healthcare in Europe: Laying the foundation. The Economist Intelligence Unit. 2016.   URL: [accessed 2018-08-06]
  62. Value-based healthcare summit: Transforming healthcare by redefining value. Canadian Foundation for Healthcare Improvement. 2018.   URL: https:/​/www.​​sf-docs/​default-source/​documents/​health-system-transformation/​vbhc-summary-report-e.​pdf?sfvrsn=6abcab44_2 [accessed 2018-08-06]
  63. Yount KW, Turrentine FE, Lau CL, Jones RS. Putting the value framework to work in surgery. J Am Coll Surg 2015 May;220(4):596-604. [CrossRef] [Medline]
  64. Wylie K, Crilly J, Toloo GS, FitzGerald G, Burke J, Williams G, et al. Review article: Emergency department models of care in the context of care quality and cost: a systematic review. Emerg Med Australas 2015 May;27(2):95-101. [CrossRef] [Medline]
  65. Kirkpatrick JR, Marks S, Slane M, Kim D, Cohen L, Cortelli M, et al. Using value-based analysis to influence outcomes in complex surgical systems. J Am Coll Surg 2015 May;220(4):461-468. [CrossRef] [Medline]
  66. Baron RJ, Davis K. Accelerating the adoption of high-value primary care--a new provider type under Medicare? N Engl J Med 2014 Jan 09;370(2):99-101. [CrossRef] [Medline]
  67. Porter ME. What is value in health care? N Engl J Med 2010 Dec 23;363(26):2477-2481. [CrossRef] [Medline]
  68. Kaplan RS, Porter ME. How to solve the cost crisis in health care. Harv Bus Rev 2011 Sep;89(9):46-52, 54, 56. [Medline]
  69. Berthelot S, Mallet M, Baril L, Dupont P, Bissonnette L, Stelfox H, et al. P017: A time-driven activity-based costing method to estimate health care costs in the emergency department. CJEM 2017 May 15;19(S1):S83. [CrossRef]
  70. Berthelot S, Mallet M, Simonyan D, Guertin J, Moore L, Boilard C, et al. PL04: Comparison of the cost and the quality of the care provided to low acuity patients in an emergency department and a walk-in clinic. CJEM 2019 May 2;21(S1):S6. [CrossRef]
  71. Öker F, Özyapıcı H. A new costing model in hospital management: time-driven activity-based costing system. Health Care Manag (Frederick) 2013;32(1):23-36. [CrossRef] [Medline]
  72. Kaplan RS, Anderson SR. Time-driven activity-based costing. Harv Bus Rev 2004 Nov;82(11):131-8, 150. [Medline]
  73. Erhun F, Mistry B, Platchek T, Milstein A, Narayanan VG, Kaplan RS. Time-driven activity-based costing of multivessel coronary artery bypass grafting across national boundaries to identify improvement opportunities: study protocol. BMJ Open 2015 Aug 25;5(8):e008765 [FREE Full text] [CrossRef] [Medline]
  74. Canadian Patient Cost Database Technical Document: MIS Patient Costing Methodology, January 2019. Canadian Institute for Health Information. 2019.   URL: [accessed 2021-01-08]
  75. Sutherland JM. Pricing hospital care: Global budgets and marginal pricing strategies. Health Policy 2015 Aug;119(8):1111-1118 [FREE Full text] [CrossRef] [Medline]
  76. Demeere N, Stouthuysen K, Roodhooft F. Time-driven activity-based costing in an outpatient clinic environment: development, relevance and managerial impact. Health Policy 2009 Oct;92(2-3):296-304. [CrossRef] [Medline]
  77. Akhavan S, Ward L, Bozic KJ. Time-driven Activity-based Costing More Accurately Reflects Costs in Arthroplasty Surgery. Clin Orthop Relat Res 2016 Jan;474(1):8-15 [FREE Full text] [CrossRef] [Medline]
  78. Berthelot S, Mallet M, Baril L, Dupont P, Bissonnette L, Stelfox H, et al. P017: A time-driven activity-based costing method to estimate health care costs in the emergency department. CJEM 2017 May 15;19(S1):S83. [CrossRef]
  79. Lee MH, Schuur JD, Zink BJ. Owning the cost of emergency medicine: beyond 2%. Ann Emerg Med 2013 Dec;62(5):498-505. [CrossRef] [Medline]
  80. Rhinosinusite aiguë chez l'adulte. Institut national d'excellence en santé et services sociaux du Québec. 2016.   URL: [accessed 2021-01-09]
  81. Halls A, Van't Hoff C, Little P, Verheij T, Leydon GM. Qualitative interview study of parents' perspectives, concerns and experiences of the management of lower respiratory tract infections in children in primary care. BMJ Open 2017 Oct 15;7(9):e015701 [FREE Full text] [CrossRef] [Medline]
  82. Cameron PA, Schull MJ, Cooke MW. A framework for measuring quality in the emergency department. Emerg Med J 2011 Oct;28(9):735-740. [CrossRef] [Medline]
  83. Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, et al. Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM 2011 Oct;13(5):300-9, E28. [CrossRef] [Medline]
  84. Bhaskar S, Bradley S, Chattu VK, Adisesh A, Nurtazina A, Kyrykbayeva S, et al. Telemedicine as the New Outpatient Clinic Gone Digital: Position Paper From the Pandemic Health System REsilience PROGRAM (REPROGRAM) International Consortium (Part 2). Front Public Health 2020;8:410 [FREE Full text] [CrossRef] [Medline]
  85. Kaplan RS. Improving value with TDABC. Healthc Financ Manage 2014 Jun;68(6):76-83. [Medline]
  86. Friedman AB. The Uncertain Economics of Insurance Enabling More Emergency Department Visits. Ann Emerg Med 2017 Aug;70(2):226-228. [CrossRef] [Medline]
  87. Challen K, Bright J, Bentley A, Walter D. Physiological-social score (PMEWS) vs. CURB-65 to triage pandemic influenza: a comparative validation study using community-acquired pneumonia as a proxy. BMC Health Serv Res 2007 Mar 01;7:33 [FREE Full text] [CrossRef] [Medline]
  88. O'Keeffe C, Mason S, Jacques R, Nicholl J. Characterising non-urgent users of the emergency department (ED): A retrospective analysis of routine ED data. PLoS One 2018;13(2):e0192855 [FREE Full text] [CrossRef] [Medline]
  89. Varner C, McLeod S, Nahiddi N, Borgundvaag B. Text messaging research participants as a follow-up strategy to decrease emergency department study attrition. CJEM 2018 Jan;20(1):148-153. [CrossRef] [Medline]
  90. Schull M, Hatcher C, Guttmann A, Leaver C, Vermeulen M, Rowe B. Development of a Consensus on Evidence-Based Quality of Care Indicators for Canadian Emergency Departments: ICES Investigative Report. In: ICES Investigative Report. Toronto: Institute for Clinical Evaluative Sciences; 2010:978.
  91. Dainty KN, Seaton B, Laupacis A, Schull M, Vaillancourt S. A qualitative study of emergency physicians' perspectives on PROMS in the emergency department. BMJ Qual Saf 2017 Sep;26(9):714-721 [FREE Full text] [CrossRef] [Medline]
  92. PROM-ED: Patient-reported outcome measure for emergency department patients. Research at St. Michael's Hospital. 2019.   URL: [accessed 2019-09-22]
  93. Bos N, Sturms LM, Stellato RK, Schrijvers AJP, van Stel HF. The Consumer Quality Index in an accident and emergency department: internal consistency, validity and discriminative capacity. Health Expect 2015 Oct;18(5):1426-1438. [CrossRef] [Medline]
  94. Vaillancourt S, Seaton MB, Schull MJ, Cheng AHY, Beaton DE, Laupacis A, et al. Patients' Perspectives on Outcomes of Care After Discharge From the Emergency Department: A Qualitative Study. Ann Emerg Med 2017 Dec;70(5):648-658. [CrossRef] [Medline]
  95. Bos N, Sturms LM, Schrijvers AJ, van Stel HF. The Consumer Quality index (CQ-index) in an accident and emergency department: development and first evaluation. BMC Health Serv Res 2012 Aug 28;12:284 [FREE Full text] [CrossRef] [Medline]
  96. The Emergency Department Return Visit Quality Program: Results from the first year. Health Quality Ontario. 2017.   URL: [accessed 2018-08-09]
  97. Jones P, Harper A, Wells S, Curtis E, Carswell P, Reid P, et al. Selection and validation of quality indicators for the Shorter Stays in Emergency Departments National Research Project. Emerg Med Australas 2012 Jul;24(3):303-312. [CrossRef] [Medline]
  98. Sørup CM, Jacobsen P, Forberg JL. Evaluation of emergency department performance - a systematic review on recommended performance and quality-in-care measures. Scand J Trauma Resusc Emerg Med 2013 Aug 09;21(1):62 [FREE Full text] [CrossRef] [Medline]
  99. Trivedy CR, Cooke MW. Unscheduled return visits (URV) in adults to the emergency department (ED): a rapid evidence assessment policy review. Emerg Med J 2015 Apr;32(4):324-329. [CrossRef] [Medline]
  100. Weinick RM, Becker K, Parast L, Stucky BD, Elliott MN, Mathews M, et al. Emergency Department Patient Experience of Care Survey: Development and Field Test. Rand Health Q 2014 Dec 30;4(3):5 [FREE Full text] [Medline]
  101. Ontario Emergency Department Patient Experience of Care Survey (Ontario EDPEC). Ontario Hospital Association.   URL: [accessed 2019-08-23]
  102. Primary Care Patient Experience Survey. Health Quality Ontario.   URL: https:/​/www.​​Portals/​0/​documents/​qi/​primary-care/​primary-care-patient-experience-survey-en.​pdf [accessed 2019-08-23]
  103. West R. Objective standards for the emergency services: emergency admission to hospital. J R Soc Med 2001;94 Suppl 39:4-8 [FREE Full text] [Medline]
  104. Clinical guides in antibiotic treatment-1st series. Institut national d'excellence en santé et services sociaux du Québec. 2017.   URL: [accessed 2017-09-14]
  105. Hodder R, Lougheed MD, Rowe BH, FitzGerald JM, Kaplan AG, McIvor RA. Management of acute asthma in adults in the emergency department: nonventilatory management. CMAJ 2010 Mar 09;182(2):E55-E67 [FREE Full text] [CrossRef] [Medline]
  106. Balter MS, La Forge J, Low DE, Mandell L, Grossman RF, Chronic Bronchitis Working Group, Canadian Thoracic Society, Canadian Infectious Disease Society. Canadian guidelines for the management of acute exacerbations of chronic bronchitis: executive summary. Can Respir J 2003;10(5):248-258. [CrossRef] [Medline]
  107. McAlister FA, Lin M, Bakal J, Dean S. Frequency of low-value care in Alberta, Canada: a retrospective cohort study. BMJ Qual Saf 2018 May;27(5):340-346. [CrossRef] [Medline]
  108. O'Donnell DE, Aaron S, Bourbeau J, Hernandez P, Marciniuk DD, Balter M, et al. Canadian Thoracic Society recommendations for management of chronic obstructive pulmonary disease - 2007 update. Can Respir J 2007 Oct;14 Suppl B:5B-32B [FREE Full text] [CrossRef] [Medline]
  109. FitzGerald JM, Lemiere C, Lougheed MD, Ducharme FM, Dell SD, Ramsey C, et al. Recognition and management of severe asthma: A Canadian Thoracic Society position statement. Canadian Journal of Respiratory, Critical Care, and Sleep Medicine 2017 Dec 11;1(4):199-221. [CrossRef]
  110. Rohrer JE, Angstman KB, Furst JW. Impact of retail walk-in care on early return visits by adult primary care patients: evaluation via triangulation. Qual Manag Health Care 2009;18(1):19-24. [CrossRef] [Medline]
  111. All-Cause Readmission to Acute Care and Return to the Emergency Department. Canadian Institute for Health Information. 2012.   URL: [accessed 2017-09-15]
  112. Campbell MK, Fayers PM, Grimshaw JM. Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research. Clin Trials 2005 Sep;2(2):99-107. [CrossRef] [Medline]
  113. Ben Charif A, Croteau J, Adekpedjou R, Zomahoun HTV, Adisso EL, Légaré F. Implementation Research on Shared Decision Making in Primary Care: Inventory of Intracluster Correlation Coefficients. Med Decis Making 2019 Aug;39(6):661-672. [CrossRef] [Medline]
  114. What is strategy? Institute of Business Management, Lahore. 2014 Apr 13.   URL: [accessed 2019-02-22]
  115. Li F, Li F. Propensity score weighting for causal inference with multiple treatments. Ann. Appl. Stat 2019 Dec;13(4):2389-2415. [CrossRef]
  116. Thomas LE, Li F, Pencina MJ. Overlap Weighting: A Propensity Score Method That Mimics Attributes of a Randomized Clinical Trial. JAMA 2020 Jun 16;323(23):2417-2418. [CrossRef] [Medline]
  117. Li L, Greene T. A weighting analogue to pair matching in propensity score analysis. Int J Biostat 2013 Jul 31;9(2):215-234. [CrossRef] [Medline]
  118. Ismail SA, Pope I, Bloom B, Catalao R, Green E, Longbottom RE, et al. Risk factors for admission at three urban emergency departments in England: a cross-sectional analysis of attendances over 1 month. BMJ Open 2017 Jun 22;7(6):e011547 [FREE Full text] [CrossRef] [Medline]
  119. Lowthian J, Straney LD, Brand CA, Barker AL, Smit PDV, Newnham H, et al. Unplanned early return to the emergency department by older patients: the Safe Elderly Emergency Department Discharge (SEED) project. Age Ageing 2016 Mar;45(2):255-261. [CrossRef] [Medline]
  120. Spitzer DL. Engendering health disparities. Can J Public Health 2005;96 Suppl 2:S78-S96 [FREE Full text] [Medline]
  121. Todd KH, Samaroo N, Hoffman JR. Ethnicity as a risk factor for inadequate emergency department analgesia. JAMA 1993;269(12):1537-1539. [Medline]
  122. Blanchard JC, Haywood YC, Scott C. Racial and ethnic disparities in health: an emergency medicine perspective. Acad Emerg Med 2003 Dec;10(11):1289-1293 [FREE Full text] [CrossRef] [Medline]
  123. Veenstra G. Racialized identity and health in Canada: results from a nationally representative survey. Soc Sci Med 2009 Aug;69(4):538-542. [CrossRef] [Medline]
  124. McAlister FA, Bakal JA, Green L, Bahler B, Lewanczuk R. The effect of provider affiliation with a primary care network on emergency department visits and hospital admissions. CMAJ 2018 Mar 12;190(10):E276-E284 [FREE Full text] [CrossRef] [Medline]
  125. Huntley AL, Johnson R, Purdy S, Valderas JM, Salisbury C. Measures of multimorbidity and morbidity burden for use in primary care and community settings: a systematic review and guide. Ann Fam Med 2012 Mar 12;10(2):134-141 [FREE Full text] [CrossRef] [Medline]
  126. Perkins AJ, Kroenke K, Unützer J, Katon W, Williams JW, Hope C, et al. Common comorbidity scales were similar in their ability to predict health care costs and mortality. J Clin Epidemiol 2004 Oct;57(10):1040-1048. [CrossRef] [Medline]
  127. Indice de défavorisation, Québec, 2016. Institut national de santé publique du Québec. 2016.   URL: [accessed 2017-09-21]
  128. L'indice de défavorisation matérielle et sociale en bref. Gamache P, Hamel D, Blaser C. 2019.   URL: https:/​/www.​​sites/​default/​files/​santescope/​indice-defavorisation/​guidemethodologiquefr.​pdf [accessed 2019-08-06]
  129. Pampalon R, Hamel D, Gamache P, Raymond G. A deprivation index for health planning in Canada. Chronic Dis Can 2009;29(4):178-191 [FREE Full text] [Medline]
  130. McCallum J, Shadbolt B, Wang D. Self-rated health and survival: a 7-year follow-up study of Australian elderly. Am J Public Health 1994 Jul;84(7):1100-1105. [CrossRef] [Medline]
  131. McCusker J, Ionescu-Ittu R, Ciampi A, Vadeboncoeur A, Roberge D, Larouche D, et al. Hospital characteristics and emergency department care of older patients are associated with return visits. Acad Emerg Med 2007 May;14(5):426-433 [FREE Full text] [CrossRef] [Medline]
  132. Moore L, Hanley JA, Turgeon AF, Lavoie A. Comparing regression-adjusted mortality to standardized mortality ratios for trauma center profiling. J Emerg Trauma Shock 2012 Oct;5(4):333-337 [FREE Full text] [CrossRef] [Medline]
  133. Gabayan GZ, Gould MK, Weiss RE, Derose SF, Chiu VY, Sarkisian CA. Emergency Department Vital Signs and Outcomes After Discharge. Acad Emerg Med 2017 Jul;24(7):846-854. [CrossRef] [Medline]
  134. Winter J, Waxman MJ, Waterman G, Ata A, Frisch A, Collins KP, et al. Pediatric Patients Discharged from the Emergency Department with Abnormal Vital Signs. West J Emerg Med 2017 Aug;18(5):878-883. [CrossRef] [Medline]
  135. Thompson M, Coad N, Harnden A, Mayon-White R, Perera R, Mant D. How well do vital signs identify children with serious infections in paediatric emergency care? Arch Dis Child 2009 Dec;94(11):888-893. [CrossRef] [Medline]
  136. Haneuse S, VanderWeele TJ, Arterburn D. Using the E-Value to Assess the Potential Effect of Unmeasured Confounding in Observational Studies. JAMA 2019 Mar 12;321(6):602-603. [CrossRef] [Medline]
  137. VanderWeele TJ, Ding P. Sensitivity Analysis in Observational Research: Introducing the E-Value. Ann Intern Med 2017 Aug 15;167(4):268-274. [CrossRef] [Medline]
  138. Affleck E. The Culture of Care. Healthc Pap 2016;15(3):31-36. [Medline]
  139. Bryan S, Donaldson C. Taking Triple Aim at the Triple Aim. Healthc Pap 2016;15(3):25-30. [Medline]

BeACCoN: Better Access and Care for Complex Needs
CoPaQ: cost-for-patient questionnaire
COPD: chronic obstructive pulmonary disease
ED: emergency department
FEV1: forced expiratory volume in the first second
NCER: Network of Canadian Emergency Researchers
PMEWS: Pandemic Medical Early Warning Score
PREM: patient-reported experience measures
PROM-ED: patient-reported outcome measure for emergency department patients
URTI: upper respiratory tract infection

Edited by T Derrick; This paper was peer reviewed by the Canadian Institutes of Health Research. submitted 19.11.20; accepted 18.12.20; published 22.02.21


©Simon Berthelot, Mylaine Breton, Jason Robert Guertin, Patrick Michel Archambault, Elyse Berger Pelletier, Danielle Blouin, Bjug Borgundvaag, Arnaud Duhoux, Laurie Harvey Labbé, Maude Laberge, Philippe Lachapelle, Lauren Lapointe-Shaw, Géraldine Layani, Gabrielle Lefebvre, Myriam Mallet, Deborah Matthews, Kerry McBrien, Shelley McLeod, Eric Mercier, Alexandre Messier, Lynne Moore, Judy Morris, Kathleen Morris, Howard Ovens, Paul Pageau, Jean-Sébastien Paquette, Jeffrey Perry, Michael Schull, Mathieu Simon, David Simonyan, Henry Thomas Stelfox, Denis Talbot, Samuel Vaillancourt. Originally published in JMIR Research Protocols (, 22.02.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.