This paper is in the following e-collection/theme issue:

Non-randomized Protocols and Methods (ehealth) (571) Telehealth and Telemonitoring (1710) mHealth for Symptom and Disease Monitoring, Chronic Disease Management (1421) Heart Failure Self-Management (154)

Published on in Vol 7, No 5 (2018): May

This is a member publication of Centre for Global eHealth Innovation, Toronto, ON, Canada

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/9911, first published .
Implementation and Evaluation of a Smartphone-Based Telemonitoring Program for Patients With Heart Failure: Mixed-Methods Study Protocol

Implementation and Evaluation of a Smartphone-Based Telemonitoring Program for Patients With Heart Failure: Mixed-Methods Study Protocol

Implementation and Evaluation of a Smartphone-Based Telemonitoring Program for Patients With Heart Failure: Mixed-Methods Study Protocol

Authors of this article:

Patrick Ware1, 2 Author Orcid Image ;   Heather J Ross3, 4, 5 Author Orcid Image ;   Joseph A Cafazzo1, 2, 6 Author Orcid Image ;   Audrey Laporte1, 7 Author Orcid Image ;   Emily Seto1, 2 Author Orcid Image
Article Authors Cited by (12) Tweetations (4) Metrics

    Protocol

    1Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

    2Centre for Global eHealth Innovation, Techna Institute, University Health Network, Toronto, ON, Canada

    3Ted Rogers Centre for Heart Research, University Health Network, Toronto, ON, Canada

    4Department of Medicine, University of Toronto, Toronto, ON, Canada

    5Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada

    6Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, ON, Canada

    7Canadian Centre for Health Economics, Toronto, ON, Canada

    Corresponding Author:

    Patrick Ware, MPH

    Institute of Health Policy, Management and Evaluation

    Dalla Lana School of Public Health

    University of Toronto

    155 College Street

    Toronto, ON, M5T 3M6

    Canada

    Phone: 1 647 227 6015

    Email: patrick.ware@mail.utoronto.ca


    Background: Meta-analyses of telemonitoring for patients with heart failure conclude that it can lower the utilization of health services and improve health outcomes compared with the standard of care. A smartphone-based telemonitoring program is being implemented as part of the standard of care at a specialty care clinic for patients with heart failure in Toronto, Canada.

    Objective: The objectives of this study are to (1) evaluate the impact of the telemonitoring program on health service utilization, patient health outcomes, and their ability to self-care; (2) identify the contextual barriers and facilitators of implementation at the physician, clinic, and institutional level; (3) describe patient usage patterns to determine adherence and other behaviors in the telemonitoring program; and (4) evaluate the costs associated with implementation of the telemonitoring program from the perspective of the health care system (ie, public payer), hospital, and patient.

    Methods: The evaluation will use a mixed-methods approach. The quantitative component will include a pragmatic pre- and posttest study design for the impact and cost analyses, which will make use of clinical data and questionnaires administered to at least 108 patients at baseline and 6 months. Furthermore, outcome data will be collected at 1, 12, and 24 months to explore the longitudinal impact of the program. In addition, quantitative data related to implementation outcomes and patient usage patterns of the telemonitoring system will be reported. The qualitative component involves an embedded single case study design to identify the contextual factors that influenced the implementation. The implementation evaluation will be completed using semistructured interviews with clinicians, and other program staff at baseline, 4 months, and 12 months after the program start date. Interviews conducted with patients will be triangulated with usage data to explain usage patterns and adherence to the system.

    Results: The telemonitoring program was launched in August 2016 and patient enrollment is ongoing.

    Conclusions: The methods described provide an example for conducting comprehensive evaluations of telemonitoring programs. The combination of impact, implementation, and cost evaluations will inform the quality improvement of the existing program and will yield insights into the sustainability of smartphone-based telemonitoring programs for patients with heart failure within a specialty care setting.

    International Registered Report Identifier (IRRID): RR1-10.2196/resprot.9911

    JMIR Res Protoc 2018;7(5):e121

    doi:10.2196/resprot.9911

    Keywords

    heart failure; telemedicine; self-management; health services research; costs and cost analysis 


    Background

    Currently, one of the greatest challenges for health care systems worldwide is the growing fiscal and social burden of preventing and managing chronic diseases [1]. Heart failure (HF) is one of the most expensive chronic diseases, partly because 50% of patients with HF get readmitted within 1 year [2]. Evidence suggests approximately half of all readmissions are preventable and result from inadequate discharge teaching, nonadherence to medication, or failure to have early follow-up with a clinician [3].

    There is mounting evidence that embedding self-care within existing health care services provides an effective model to meet these needs for a number of chronic diseases, including HF [4-7]. Although more traditional self-care interventions such as health coaching or patient education can be effective [8,9], their implementation often proves challenging [10,11]. Health information technology is one avenue that can support the delivery of self-care interventions. For example, telemonitoring (TM) is thought to be crucial to offer patients the right care at the right time [12] by allowing patients to collect clinical data at home, which is then transmitted via technology to be viewed and acted upon by a clinician at a distant location [13].

    In HF, meta-analyses conclude that TM lowers the utilization of health services, improves HF health outcomes, and improves health-related quality of life (QoL) [14-20]. However, results vary widely between individual trials [21], and 2 of the largest studies, the Tele-HF [22] and BEAT-HF trials [23], reported null results. One problem is that results come from trials of varying quality, and there is often little discussion about the intervention itself or the degree to which the patients have adhered to it over the course of the study [24]. Furthermore, studies looking at the implementation of TM have identified technical, cost, organizational, and behavioral barriers, which may explain why these technologies are not yielding consistent positive outcomes [12,25]. In addition, this may explain why, despite some evidence of cost-effectiveness [26], there is still inconclusive evidence regarding the impact of TM in terms of its cost to patients with HF [26].

    Although the ubiquity of mobile phones is believed to have the potential to make TM interventions more accessible and cost-effective [27,28], additional knowledge gaps exist in the emerging use of smartphones for TM of chronic diseases. Randomized controlled trials (RCTs) employing mobile phone–based TM interventions have been conducted and have shown similar positive results to those reported for more traditional TM [29,30]. However, this type of TM is novel, and therefore, there have been no comprehensive implementation studies on smartphone-based TM interventions completed to date.

    Implementation of a Smartphone-Based Telemonitoring Program

    The Intervention

    An algorithm-based smartphone-based TM program for patients with HF, called the Medly Program, is being implemented as part of the standard of care at the University Health Network’s Ted Rogers Center of Excellence for Heart Function in Toronto, Canada (hereafter referred to as the HF clinic). The objective of the Medly Program is similar to other HF TM programs: improve patient self-management and decrease health care utilization [31]. Patients enrolled in the Medly Program will receive a Medly kit that includes a smartphone (Samsung Galaxy Grand Prime) with a limited data plan and the Medly app already downloaded. In addition, patients receive a Bluetooth-enabled weight scale and blood pressure cuff. Patients are instructed to take daily weight and blood pressure readings using these devices and record their symptoms using the Medly app first thing in the morning. Automated self-care instructions are immediately displayed in the Medly app after these 3 parameters are processed by an algorithm that was developed in close consultation with HF clinicians. In addition, patients have access to graphs displaying historical trends for these parameters. To assist in compliance, patients receive an automated call on their primary phone line if they have not taken their readings before 10 AM. The Medly Program is initially providing patients with all the equipment to mitigate the potential operational and software development challenges of offering the service on different devices during the critical early stages of implementation. However, a transition to a bring your own device model is planned, which would enable patients with smartphones to use their own devices by downloading the Medly app.

    If there are signs of deteriorating health of a patient, the Medly algorithm generates an alert to a HF clinician who is part of the patient’s care team. The alerts are made available to clinicians in 2 formats. The first is through an automated email containing the latest weight, blood pressure, and symptoms along with the patient’s target ranges. In addition, the email contains the patients’ latest medication list and HF–related laboratory results, and contact information. Second, clinicians can choose to view the alerts by accessing a secure Web portal that presents a list of all the alerts triggered. Here, clinicians can review details of the latest alert and graphs showing historical weight, blood pressure, symptoms, and HF–specific laboratory results, which are visually contextualized according to the patient’s target ranges. The clinician will follow-up with the patient depending on the clinical need, documenting all actions and decisions taken in response to the alert in the hospital electronic medical record (EMR).

    The Medly Program is intended to be delivered as part of the standard of care; as such, the goal is for it to be seamlessly integrated within the existing workflows of the HF clinic. Before deployment, a service blueprint was created through ethnographic methods including observation and informal interviews with clinicians and support staff in the HF clinic. This helped identify areas where the processes of the Medly Program could be incorporated. In addition, an RCT of 100 patients with an embedded qualitative component was previously conducted in the HF clinic with an earlier version of the Medly system. This study concluded that patients receiving the intervention experienced improvements in QoL and self-care maintenance compared with a control group [30]. Lessons learned from this RCT helped justify the decision to implement the Medly Program as part of the standard of care and informed the program’s implementation strategy. The current program offers an opportunity to evaluate the implementation and effectiveness of Medly under real-world conditions.

    User Training and Support

    Training as to how to use the Medly app and associated devices is provided to patients at the time of enrollment into the Medly Program by a telehealth analyst. Clinicians monitoring patients through the Medly system participated in a formal training session approximately 1 month before program deployment. Both patients and clinicians are provided with a user manual to supplement the in-person training along with contact information of the telehealth analyst who offers technical support during normal business hours.

    Evaluation Objectives and Research Questions

    Objective 1

    The first objective is to evaluate the impact of the Medly Program on health service utilization, patient health outcomes, and their ability to self-care. Alerts sent to patients and clinicians will help identify periods of symptom exacerbation and volume overload. The impact of this is expected to permit earlier intervention for worsening conditions, thus avoiding trips to the emergency department (ED) and hospitalizations. For patients who are enrolled into the Medly Program on hospital discharge, the program is expected to reduce 30-day readmission rates. In addition, participation in the Medly Program is expected to improve patients’ ability to self-care, leading to improved clinical outcomes and QoL.

    Objective 2

    The second objective is to evaluate the degree to which the Medly Program was implemented as intended and to identify the contextual barriers and facilitators of implementation. This objective will answer the following questions:

    • To what extent did the HF clinic implement the Medly Program as intended?
    • What contextual factors influence the implementation of the Medly Program?
    • What adaptations were needed to implement and sustain the program within existing clinical workflows?
    Objective 3

    The third objective is to describe patient usage patterns to determine adherence and other behaviors in the Medly Program. This objective will answer the following questions:

    • To what degree do patients adhere to the Medly Program and how do adherence patterns change over time?
    • What factors influence patient adherence?
    Objective 4

    The fourth objective is to evaluate the costs associated with the implementation of the TM program from the perspective of the health care system (ie, public payer), the hospital, and patients.


    Overview of the Study Design and Evaluation Framework

    Data for the 4 objectives will be collected using mixed-methods. This approach will include a multiple pre-and posttest design for the evaluation of patient-level impacts, patient adherence, and cost. Quantitative data will include data collected as part of the standard of care (including health care utilization data and laboratory results obtained using a chart review) and usage data from the TM system. Additional patient-level data will be collected using questionnaires at baseline, 1 month, 6 months, 12 months, and 24 months. The qualitative component will take the form of an embedded single case study [32]. The 2 embedded subunits of analysis include clinicians and patients, as it relates to their adoption and use of the Medly system. The case is defined as the Medly Program at the HF clinic for the duration of 1 year starting from the program’s launch date (August 23, 2016). Qualitative methods, including semistructured interviews and document reviews, will be used to gain insights regarding patient self-care practices (objective 1), the barriers and facilitators to program implementation (objective 2), and explanations for patient adherence and usage of the system (objective 3).

    Study Participants

    Representatives from all stakeholder groups involved in the implementation of the Medly Program will be recruited for participation.

    Patients

    Patients can be enrolled into the Medly Program provided they (1) are 18 years or older, (2) have been diagnosed with HF and are followed by a cardiologist at the HF clinic, (3) can speak and read English (or have an informal caregiver who does) to adequately understand the text prompts in the Medly app, and (4) are able to comply with using Medly (eg, able to stand on the weight scale, able to answer symptom questions). As the Medly program is being implemented as part of the standard of care, there is no explicit exclusion criteria for participating in the program and its evaluation. The duration of program participation will be decided by patients and their treating cardiologist on an individual basis.

    Upon enrollment into the Medly Program, patients will be presented with the option of answering questionnaires and participating in interviews related to the program evaluation. Patients will be asked to sign a written consent form before participating in the evaluation.

    Program Staff

    All members of the Medly Program staff will be asked to participate in semistructured interviews. These include clinicians providing care for patients with HF in the HF clinic (n=7), telehealth analyst (n=1), project manager (n=1), and members of the implementation team (n=2). These individuals will be asked to sign a consent form before their first interview.

    Data Collection and Analysis

    Objective 1: Measuring the Impact of the Medly Program
    Impact Indicators

    The primary outcome for evaluating the impact of the Medly Program is the number of hospitalizations because of HF in the 6 months before versus the 6 months after enrollment. Secondary impact outcomes comparing 6-month to baseline values are described below.

    Health Service Utilization

    The number of hospitalizations (all-cause), 30-day readmission rate, days in hospital (HF and all-cause), number of ED visits (HF and all-cause), visits to family doctor (HF and all-cause), number of HF–related outpatient visits, and changes to medication will be recorded.

    Left Ventricular Fraction and Laboratory Tests

    The following HF–specific clinical parameters will be collected: left ventricular ejection fraction (LVEF), brain natriuretic peptide (BNP), creatinine, sodium, potassium, hemoglobin, and uric acid levels.

    Mortality and Prediction of Survival

    Patient mortality will be tracked. In addition, the Seattle Heart Failure Model (SHFM) will be calculated at program entry. Projected SHFM survival versus actual survival will be compared [33]. The calculation of this score requires data on age, gender, New York Heart Association classification, weight, LVEF, systolic blood pressure, list of medications (including diuretics), laboratory results (hemoglobin, lymphocytes, uric acid, total cholesterol, and sodium), and QRS interval.

    Dyspnea

    Patients will be asked to describe their level of breathlessness using a visual analogue scale for dyspnea on a scale ranging from 0 (no shortness of breath) to 10 (shortness of breath is the worst it can be).

    Quality of Life

    The EQ-5D-5L is a measure of generic health status and will be administered to all patients as a measure of QoL [34,35]. This 5-item instrument has undergone validity and reliability testing for several conditions including HF. Due to the generic nature of this measure, it is recommended that it be administered along with supplementary measurements to capture more disease-specific aspects related to QoL. Hence, the Minnesota Living with Heart Failure Questionnaire (MLHFQ) will also be administered. The questionnaire contains 21 items scored on a 5-point Likert scale, the responses of which are summed to produce a total score and subscores for the domains of physical and emotional QoL. The MLHFQ is widely used in studies involving HF–related QoL and has been shown to have a high level of reliability and validity [36].

    Self-Care

    The self-care of HF index asks respondents to respond to 22 items on a 4-point Likert scale to assess their ability to self-care across 3 subscales (maintenance, management, and confidence). The tool has undergone validity and reliability testing involving patients with HF [37].

    Demographic Variables

    Demographic information will also be collected using a questionnaire, which includes questions on the following: age, sex, income, native language, living arrangements (living alone, with a partner, or other), whether or not they have a caregiver (formal or informal), and type of living areas (eg, urban, suburban, or rural). In addition, patient comorbidities will be tracked. Finally, questions will be asked to assess the patient’s experience and comfort level with technology and smartphones, including frequency of use.

    Data Acquisition

    Data collection will occur at baseline, 1 month, 6 months, 12 months, and 24 months, or until the patient exits the program. Data collected in addition to the primary evaluation period (baseline to 6 months) will provide an opportunity for post-hoc analyses aimed at quality improvement, including recommended duration of use. For example, the 1-month time point was included to determine if the bulk of changes to self-care and QoL occur immediately after enrollment. Similarly, longitudinal data will help determine whether patient-level impacts are sustained or change when using the system long term.

    Health service utilization, laboratory results, mortality, prediction of survival, and select demographic information will be obtained from the hospital EMR. In addition, health service utilization information will be verified by patient participants through self-reports via a questionnaire.

    Baseline and follow-up questionnaires containing the validated survey tools listed in Table 1 will be distributed to patients during regularly scheduled visits. Upon study enrollment, patients will be asked whether they prefer to be mailed the questionnaire or completing it using an online survey tool (SurveyMonkey [38]) in the event that they do not have a clinic visit scheduled for the data collection time point. In these situations, the questionnaires will be sent to the patient according to the preferred format and they will be given 2 weeks to respond, after which a member of the evaluation team will call the patient to remind them to complete the questionnaire.

    Table 1. Timing of outcome assessments for the impact evaluation.
    Domain and measureBaseline1 month6 months12 months24 monthsExit
    Health service utilization






    		30-day readmission
    		Xa




    		Number of hospitalizationsX
    		XXXX

    		Number of days in hospitalX
    		XXXX

    		Number of emergency department visitsX
    		XXXX

    		Number of heart failure-related outpatient visitsX
    		XXXX

    		Number of visits to family doctorsX
    		XXXX

    		Changes to medicationX
    		XXXX
    Clinical outcomes






    		Left ventricular ejection fractionX

    		XX

    		Blood work: BNPb, creatinine, sodium, potassium, hemoglobin, and uric acidX
    		XXXX

    		Visual analogue scale for dyspneaXXXXXX

    		SHFMcX




    Self-care






    		SCHFId [37]XXXXXX

    		EQ-5D-5Le [34,35]XXXXXX

    		MLHFQf [36]XXXXXX

    aX: data is collected at this time point.

    bBNP: brain natriuretic peptide.

    cSHFM: Seattle Heart Failure Model.

    dSCHFI: Self-Care of Heart Failure Index.

    eEQ-5D-5L: EuroQol five-dimensional.

    fMLHFQ: Minnesota Living with Heart Failure Questionnaire.

    Planned Analyses

    The primary analyses will be paired Student t tests and Wilcoxon signed rank tests comparing baseline and 6-month values for all patient-level outcomes. For patients enrolled in the Medly Program on hospital discharge, the 30-day readmission rate will be compared with the readmission rate before the launch of the Medly Program, as determined using hospital administrative data. Secondary analyses aimed at determining the longitudinal impact of the Medly Program (ie, using outcome data from the additional time points), and the correlation of independent variables (eg, patient characteristics and adherence rates) with outcomes will be analyzed using general linear mixed model procedures. In addition, descriptive statistics will be produced for all variables collected, which may inform necessary subgroup analyses. All statistical analyses will be performed using the statistical software application SPSS (IBM Corporation, USA) [39].

    Power Consideration

    We will aim to recruit at least 108 patients into the Medly Program at the HF clinic before analyses of patient-level outcomes are undertaken. This number is based on being able to detect a small effect size (Cohen d=0.3) in the number of hospitalizations because of HF within the first 6 months of enrollment with 80% power and an alpha of .05 (two-sided) [40]. This number considers that approximately 20% of patients will be “lost to follow-up,” which includes patient mortality and those who withdraw from the program before the 6-month time point. We anticipate recruiting this number in the first 18 months of the program.

    Objective 2: Implementation Evaluation

    The mixed-methods implementation evaluation will be guided by the framework by Proctor et al, which describes outcomes that can serve as indicators of implementation success [41], and the Consolidated Framework for Implementation Research (CFIR). The CFIR describes factors influencing implementation success according to 5 domains: (1) intervention characteristics, (2) outer setting, (3) inner setting, (4) characteristics of individuals, and (5) process [42].

    Table 2. Implementation outcome indicators.
    Implementation outcomeDefinitionsaIndicator
    AdoptionThe intention, initial decision, or action to try or employ an innovation or evidence-based practice. Adoption may also be referred to as “uptake”Number of clinicians having decided to use Medly to monitor patients
    Implementation costThe cost impact of the implementation effortSee objective 4 for cost outcomes associated with implementation of the Medly Program
    FeasibilityThe extent to which a new treatment, or an innovation, can be successfully used or carried out within a given agency or settingNumber of patients enrolled; rate of patient enrollment; number of patient-initiated dropouts from the program; number of physician-initiated dropouts from the program—or no uptake
    FidelityThe degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the program developersNumber and nature of calls or emails made to the telehealth analyst; proportion of the number of—alerts acknowledged over the total number of alerts, phone calls to patients over the total number of alerts triggered, and false or inappropriate alerts over the total alerts triggered
    PenetrationThe integration of a practice within a service setting and its subsystemsPercentage of clinicians using Medly over the total number of potential clinician users in the HF clinic

    aDefinitions are based on the definitions provided by Proctor et al [41].

    Implementation Outcome Indicators

    Implementation outcomes are defined by Proctor et al as “the effects of deliberate and purposive actions to implement new treatments, practice, and services” [41]. A total of 5 implementation outcomes were selected as quantitative indicators and are presented in Table 2. These data will be obtained from the hospital’s EMR, the Medly system’s audit trails, and technical support logs.

    Semistructured Interviews With Program Staff

    Separate semistructured interview guides based on the constructs in the CFIR will be formulated for each category of program staff previously outlined. Interviews will be conducted at baseline, 4 months, and 12 months to align with the different phases of Stetler et al’s typology of formative evaluations [43]. The interviews are expected to last approximately 30 min and will be conducted at a location convenient to the participants. All interviews will be audiotaped for later transcription and analysis.

    Planned Analyses

    Descriptive statistics will be produced for the indicators of the implementation outcome to provide an objective measure of implementation success.

    Furthermore, 2 independent researchers will analyze interview transcripts and documents using the Framework Method of qualitative analysis [44]. An initial round of coding will use a deductive approach by looking for themes that match the constructs in the CFIR. A second round of coding will be inductive using an open coding approach, which will involve researchers looking for unexpected themes that are not represented in the guiding CFIR framework. Throughout the analysis, both reviewers will discuss the themes and codes from their independent analyses to come up with a single analytical framework. This finalized framework will be applied in a final coding of all transcriptions using NVivo (QSR International, Doncaster, Victoria, Australia) [45].

    Quantitative and qualitative data will be triangulated such that the interview data will help explain success or failure of the implementation.

    Objective 3: Describing Patient Adherence and Usage Patterns

    A sequential explanatory mixed-methods approach [46] will be used consisting of a quantitative measurement of patient adherence and semistructured interviews with patients at 4 time points.

    Patient Adherence

    Adherence will be assessed by analyzing patient usage rates of the Medly system—specifically, the proportion of days for which the patient took a complete reading (weight, blood pressure, and symptoms) over the previous 30 days. Usage data will be obtained and exported on a regular basis through Google Analytics [47].

    Semistructured Interviews

    A sample of patients enrolled in the Medly Program will participate in semistructured interviews aimed at understanding reasons for adherence or nonadherence and their general experiences with the Medly Program. Interview guides will be formulated based on the constructs of the Unified Theory of Acceptance and Use of Technology 2 [48]. For example, we will explore how patients’ expectations, ease of use of the intervention, facilitating conditions (eg, quality of technical support services), and the direct or indirect influence of clinicians and loved ones could explain levels of uptake, adherence, and use. Unlike adherence, which will be calculated for all patients enrolled in the Medly Program, interviews will be conducted until information saturation is reached using maximum variation sampling [49] based on age, sex, experience with technology, health status, time since enrollment, and level of adherence. As recommended by Francis et al for theory-based interview studies, an a priori target of 10 patients is being set as the initial analysis sample [50]; however, interviews will continue until no new themes emerge [50]. For patients who withdraw from the Medly Program before the end of the evaluation period, reasons for this withdrawal will be documented as part of standardized off-boarding procedures and a sample will be asked to participate in an interview. The interviews are expected to last 20 to 30 min and will be conducted in a quiet and private space within the clinic (eg, consultation room) during a regular clinic visit or over the telephone. All interviews will be audiotaped and transcribed for later analysis.

    Planned Analyses

    Monthly adherence rates will be examined using descriptive statistics to identify any patterns in the increase or decrease of patient adherence to Medly over time. In addition, general linear mixed model procedures will be performed to determine if any baseline patient characteristics (eg, age, sex, and HF severity) or time since program enrollment predicts patient adherence to the Medly system. Semistructured interviews will be analyzed using the Framework Method [44], as previously described.

    Objective 4: Cost Impact of Implementing the Medly Program

    The costs associated with implementing the Medly Program will be determined from the perspectives of the public payer, hospital, and patient. In reporting these results, costs will be interpreted in relation to the patient-level impacts determined in objective 1.

    Data Acquisition

    Costs will be calculated using a 6-month time frame. Specifically, we will compare costs before the implementation of the Medly Program (assessed at baseline) versus the costs after enrollment of patients into the program (measured at 6 months). This time frame was chosen because it represents the time horizon over which most of the health effects and costs of using the Medly Program are expected. Most of the cost variables will be self-reported by patients and triangulated using administrative data whenever possible (eg, EMR data). Therefore, questions related to the cost will be added to the patient outcome questionnaires (objective 1) and will be administered at baseline and 6 months.

    Public Payer Perspective

    Costs to the public payer will be determined by looking at health care utilization of patients enrolled in the Medly Program before versus after their enrollment. These will include hospitalizations, ED visits, HF clinic visits, family physician visits, and use of home care services. In addition, costs for inpatient medications will be considered.

    Hospital Perspective

    Costs from the hospital perspective will be valued based on time spent by human resources involved in the Medly Program. This time will be converted to costs based on those individuals’ respective salaries. This will include time the clinicians spent reviewing and responding to Medly alerts as well as time the clinicians spend in training sessions, learning to use the system, and seeking technical support. In addition, costs for the hospital perspective will include equipment costs (Medly kit, smartphone data plan, and server) and the salary for employing a telehealth analyst responsible for recruiting, training, onboarding, managing inventory support, and providing training.

    Patient Perspective

    Costs for patients will primarily be determined by the time they spend accessing care for HF. This will involve determining their employment status and annual income as well as time they spend traveling to and from appointments, time spent at appointments, and how much work time was missed because of their HF condition (vacation or unpaid). In addition, this will include the time patients spent learning to use the Medly system, time spent using the system, and time getting technical support. Additional costs considered include travel, parking, and all other out-of-pocket costs related to accessing care or using the intervention. Moreover, time of informal caregivers (eg, friend or family member who helps the patient to manage their HF) will be valued as part of the patient perspective. Costs for informal caregivers will be based on the average hourly rate of personal support workers.


    The Medly Program was launched in August 2016. As of April 4, 2018, 166 patients have been enrolled. The primary impact analysis is expected to be conducted by January 2019.


    This study aims to evaluate the implementation and impact of a smartphone-based TM program being implemented as part of the standard of care in a specialty care setting in a large Canadian city.

    Limitations

    Unlike TM systems evaluated in the context of academic research, the lack of strict patient inclusion and exclusion criteria for the Medly Program has the potential to lead to heterogeneity among evaluation participants, which will make it difficult to generalize the results to other health care settings. Another important limitation is that the nature of this evaluation and the availability of data do not allow for a distinct comparator group. Without discounting these limitations, we believe that one of the strengths of this evaluation is its pragmatic nature and that these threats to internal and external validity will be mitigated through a detailed description of the context and participants when results are reported. The interpretation of the Medly Program evaluation results will include comparisons with previous TM RCTs conducted within the HF clinic [30] and other comparable settings, as well as other RCTs evaluating interventions designed to promote self-care through education and health coaching conducted within the HF clinic [51]. This is possible because of the selection of outcome metrics common in other TM studies.

    Conclusions

    Unlike other TM studies that focus primarily on quantitative outcomes, this evaluation will also examine the context and mechanisms that lead to them. Therefore, this pragmatic mixed-methods study will allow for an interpretation of results using realist evaluation principles [52]. The information gathered during this evaluation will inform if, and how, a smartphone-based TM system improves the self-care capacities, clinical management, and health outcomes of patients with HF. This evaluation will lead to quality improvement of the current program and provide evidence that will inform the implementation and sustainability of other TM programs.

    Acknowledgments

    The authors wish to thank the clinical staff for providing feedback used in the design of the TM system and whose insights informed the implementation of the Medly Program. Funding for this program and its evaluation is provided by the Ted Rogers Centre for Heart Research. Additional funding covering a graduate student’s stipend comes from a Natural Sciences and Engineering Research Council of Canada Discovery Grant.

    Authors' Contributions

    PW and ES led the overall design of the evaluation; HJR, AL, and JAC also contributed to the design. PW drafted the manuscript, and ES, HJR, AL, and JAC edited and reviewed the manuscript. ES is the principal investigator of this evaluation.

    Conflicts of Interest

    None declared.

    1. Prince MJ, Wu F, Guo Y, Gutierrez Robledo LM, O'Donnell M, Sullivan R, et al. The burden of disease in older people and implications for health policy and practice. Lancet 2015;385(9967):549-562 [FREE Full text]
    2. Heidenreich PA, Albert NM, Allen LA, Bluemke DA, Butler J, Fonarow GC, American Heart Association Advocacy Coordinating Committee, Council on Arteriosclerosis, Thrombosis and Vascular Biology, Council on Cardiovascular Radiology and Intervention, Council on Clinical Cardiology, Council on Epidemiology and Prevention, Stroke Council. Forecasting the impact of heart failure in the United States: a policy statement from the American Heart Association. Circ Heart Fail 2013 May;6(3):606-619 [FREE Full text] [CrossRef] [Medline]
    3. White M, Garbez R, Carroll M, Brinker E, Howie-Esquivel J. Is “teach-back” associated with knowledge retention and hospital readmission in hospitalized heart failure patients? J Cardiovasc Nurs 2013;28(2):137-146. [CrossRef] [Medline]
    4. Effing T, Monninkhof EM, Van der Valk PD, Van der Palen J, van Herwaarden CL, Partidge MR, et al. Self-management education for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2007 Oct 17(4):CD002990. [CrossRef] [Medline]
    5. McAlister FA, Lawson FM, Teo KK, Armstrong PW. A systematic review of randomized trials of disease management programs in heart failure. Am J Med 2001 Apr;110(5):378-384. [CrossRef]
    6. Weintraub A, Gregory D, Patel AR, Levine D, Venesy D, Perry K, et al. A multicenter randomized controlled evaluation of automated home monitoring and telephonic disease management in patients recently hospitalized for congestive heart failure: the SPAN-CHF II trial. J Card Fail 2010 Apr;16(4):285-292. [CrossRef] [Medline]
    7. Kruis AL, Smidt N, Assendelft WJ, Gussekloo J, Boland MR, Rutten-van Mölken M, et al. Integrated disease management interventions for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2013 Oct 10(10):CD009437. [CrossRef] [Medline]
    8. Jovicic A, Holroyd-Leduc JM, Straus SE. Effects of self-management intervention on health outcomes of patients with heart failure: a systematic review of randomized controlled trials. BMC Cardiovasc Disord 2006 Nov 02;6:43 [FREE Full text] [CrossRef] [Medline]
    9. DeWalt DA, Malone RM, Bryant ME, Kosnar MC, Corr KE, Rothman RL, et al. A heart failure self-management program for patients of all literacy levels: a randomized, controlled trial [ISRCTN11535170]. BMC Health Serv Res 2006 Mar 13;6:30 [FREE Full text] [CrossRef] [Medline]
    10. Holden RJ, Schubert CC, Mickelson RS. The patient work system: an analysis of self-care performance barriers among elderly heart failure patients and their informal caregivers. Appl Ergon 2015 Mar;47:133-150 [FREE Full text] [CrossRef] [Medline]
    11. Inglis SC, Clark RA, McAlister FA, Stewart S, Cleland JG. Which components of heart failure programmes are effective? A systematic review and meta-analysis of the outcomes of structured telephone support or telemonitoring as the primary component of chronic heart failure management in 8323 patients: Abridged Cochrane Review. Eur J Heart Fail 2011 Sep;13(9):1028-1040. [CrossRef] [Medline]
    12. Boyne JJ, Vrijhoef HJ. Implementing telemonitoring in heart failure care: barriers from the perspectives of patients, healthcare professionals and healthcare organizations. Curr Heart Fail Rep 2013 Sep;10(3):254-261. [CrossRef] [Medline]
    13. Inglis S. Structured telephone support or telemonitoring programmes for patients with chronic heart failure. J Evid Based Med 2010;3(4):228. [CrossRef]
    14. Inglis SC, Clark RA, Dierckx R, Prieto-Merino D, Cleland JG. Structured telephone support or non-invasive telemonitoring for patients with heart failure. Cochrane Database Syst Rev 2015(10):CD007228. [CrossRef] [Medline]
    15. Lin MH, Yuan WL, Huang TC, Zhang HF, Mai JT, Wang JF. Clinical effectiveness of telemedicine for chronic heart failure: a systematic review and meta-analysis. J Investig Med 2017 Mar 22;65(5):899-911. [CrossRef] [Medline]
    16. Kotb A, Cameron C, Hsieh S, Wells G. Comparative effectiveness of different forms of telemedicine for individuals with heart failure (HF): a systematic review and network meta-analysis. PLoS ONE 2015 Feb 25;10(2):e0118681. [CrossRef]
    17. Nakamura N, Koga T, Iseki H. A meta-analysis of remote patient monitoring for chronic heart failure patients. J Telemed Telecare 2014 Jan;20(1):11-17. [CrossRef] [Medline]
    18. Or CK, Tao D, Wang H. The effectiveness of the use of consumer health information technology in patients with heart failure: a meta-analysis and narrative review of randomized controlled trials. J Telemed Telecare 2017 Jan;23(1):155-166. [CrossRef] [Medline]
    19. Polisena J, Tran K, Cimon K, Hutton B, McGill S, Palmer K, et al. Home telemonitoring for congestive heart failure: a systematic review and meta-analysis. J Telemed Telecare 2010;16(2):68-76. [CrossRef] [Medline]
    20. Xiang R, Li L, Liu SX. Meta-analysis and meta-regression of telehealth programmes for patients with chronic heart failure. J Telemed Telecare 2013 Jul;19(5):249-259. [CrossRef] [Medline]
    21. Paré G, Moqadem K, Pineau G, St-Hilaire C. Clinical effects of home telemonitoring in the context of diabetes, asthma, heart failure and hypertension: a systematic review. J Med Internet Res 2010;12(2):e21 [FREE Full text] [CrossRef] [Medline]
    22. Chaudhry SI, Mattera JA, Curtis JP, Spertus JA, Herrin J, Lin Z, et al. Telemonitoring in patients with heart failure. N Engl J Med 2010 Dec 9;363(24):2301-2309 [FREE Full text] [CrossRef] [Medline]
    23. Ong MK, Romano PS, Edgington S, Aronow HU, Auerbach AD, Black JT, Better Effectiveness After Transition–Heart Failure (BEAT-HF) Research Group. Effectiveness of remote patient monitoring after discharge of hospitalized patients with heart failure: the Better Effectiveness After Transition -- Heart Failure (BEAT-HF) randomized clinical trial. J Am Med Assoc Intern Med 2016 Mar;176(3):310-318 [FREE Full text] [CrossRef] [Medline]
    24. Mattera JA. Patients' Adoption and Adherence to a Heart Failure Telemonitoring Intervention. Boston: Boston University; 2011.
    25. Kato NP, Johansson P, Okada I, de Vries AE, Kinugawa K, Strömberg A, et al. Heart failure telemonitoring in Japan and Sweden: a cross-sectional survey. J Med Internet Res 2015 Nov 13;17(11):e258 [FREE Full text] [CrossRef] [Medline]
    26. Grustam AS, Severens JL, van Nijnatten J, Koymans R, Vrijhoef HJ. Cost-effectiveness of telehealth interventions for chronic heart failure patients: a literature review. Int J Technol Assess Health Care 2014 Jan;30(1):59-68. [CrossRef] [Medline]
    27. Boulos MN, Wheeler S, Tavares C, Jones R. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng Online 2011;10(1):24.
    28. Sarasohn-Kahn J. CHCF.: California HealthCare Foundation How smartphones are changing health care for consumers and providers   URL: https://www.chcf.org/wp-content/uploads/2017/12/PDF-HowSmartphonesChangingHealthCare.pdf [accessed 2018-04-24] [WebCite Cache]
    29. Scherr D, Kastner P, Kollmann A, Hallas A, Auer J, Krappinger H, MOBITEL Investigators. Effect of home-based telemonitoring using mobile phone technology on the outcome of heart failure patients after an episode of acute decompensation: randomized controlled trial. J Med Internet Res 2009 Aug 17;11(3):e34 [FREE Full text] [CrossRef] [Medline]
    30. Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Mobile phone-based telemonitoring for heart failure management: a randomized controlled trial. J Med Internet Res 2012;14(1):e31 [FREE Full text] [CrossRef] [Medline]
    31. Ware P, Seto E, Ross HJ. Accounting for complexity in home telemonitoring: a need for context-centred evidence. Can J Cardiol 2018 Jan 14;17(1):156 (forthcoming) [FREE Full text] [CrossRef] [Medline]
    32. Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: Sage Publications; 2013.
    33. Levy WC, Mozaffarian D, Linker DT, Sutradhar SC, Anker SD, Cropp AB, et al. The Seattle Heart Failure Model: prediction of survival in heart failure. Circulation 2006 Mar 21;113(11):1424-1433 [FREE Full text] [CrossRef] [Medline]
    34. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res 2011 Dec;20(10):1727-1736 [FREE Full text] [CrossRef] [Medline]
    35. Janssen MF, Pickard AS, Golicki D, Gudex C, Niewada M, Scalone L, et al. Measurement properties of the EQ-5D-5L compared to the EQ-5D-3L across eight patient groups: a multi-country study. Qual Life Res 2013;22(7):1717-1727. [Medline]
    36. Rector TS, Cohn JN. Assessment of patient outcome with the Minnesota Living with Heart Failure questionnaire: reliability and validity during a randomized, double-blind, placebo-controlled trial of pimobendan. Pimobendan Multicenter Research Group. Am Heart J 1992 Oct;124(4):1017-1025. [Medline]
    37. Riegel B, Lee CS, Dickson VV, Carlson B. An update on the self-care of heart failure index. J Cardiovasc Nurs 2009;24(6):485-497 [FREE Full text] [CrossRef] [Medline]
    38. SurveyMonkey. San Mateo, California, USA: SurveyMonkey Inc; 2017.   URL: https://www.surveymonkey.com/ [accessed 2018-04-24] [WebCite Cache]
    39. IBM Corporation. IBM. Armonk, NY: IBM Corp; 2013. SPSS Statistics for Mac, version 22   URL: https://www-01.ibm.com/support/docview.wss?uid=swg21646821 [accessed 2018-04-24] [WebCite Cache]
    40. Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007 May;39(2):175-191. [Medline]
    41. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2010 Oct 19;38(2):65-76. [CrossRef]
    42. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50. [Medline]
    43. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006 Feb;21(S2):S1-S8. [CrossRef] [Medline]
    44. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117 [FREE Full text] [CrossRef] [Medline]
    45. QSR International Pty Ltd. Qsrinternational. Doncaster, Victoria, Australia: QSR International Pty Ltd; 2012. NVivo qualitative data analysis software   URL: https://www.qsrinternational.com/nvivo/home [accessed 2018-04-24] [WebCite Cache]
    46. Clark VP, Creswell JW. Designing and Conducting Mixed Methods Research. 2nd ed. Thousand Oaks, CA: SAGE Publications Inc; 2011.
    47. Ledford JL, Teixeira J, Tyler ME. Google Analytics. Indianapolis: Wiley Publishing; 2010.
    48. Venkatesh V, Thong JY, Xu X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 2012;36(1):157-178.
    49. Sandelowski M. Sample size in qualitative research. Res Nurs Health 1995 Apr;18(2):179-183. [Medline]
    50. Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health 2010 Dec;25(10):1229-1245. [CrossRef] [Medline]
    51. Nolan RP, Payne AY, Ross H, White M, D'Antono B, Chan S, et al. An internet-based counseling intervention with email reminders that promotes self-care in adults with chronic heart failure: randomized controlled trial protocol. JMIR Res Protoc 2014 Jan 30;3(1):e5 [FREE Full text] [CrossRef] [Medline]
    52. Pawson R. The Science of Evaluation: A Realist Manifesto. Thousand Oaks, CA: Sage Publications; 2013.

    ‎
    BNP: brain natriuretic peptide
    CFIR: Consolidated Framework for Implementation Research
    ED: emergency department
    EMR: electronic medical record
    EQ-5D-5L: EuroQol five-dimensional
    HF: heart failure
    LVEF: left ventricular ejection fraction
    MLHFQ: Minnesota Living with Heart Failure Questionnaire
    QoL: quality of life
    RCTs: randomized controlled trials
    SCHFI: Self-Care of Heart Failure Index
    SHFM: Seattle Heart Failure Model
    TM: telemonitoring

    Edited by G Eysenbach; submitted 22.01.18; peer-reviewed by J Hensel, I Kedan; comments to author 21.02.18; revised version received 02.03.18; accepted 16.03.18; published 03.05.18

    Copyright

    ©Patrick Ware, Heather J Ross, Joseph A Cafazzo, Audrey Laporte, Emily Seto. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 03.05.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.