Published on in Vol 12 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Web-Based Patient Recommender Systems for Preventive Care: Protocol for Empirical Research Propositions

Web-Based Patient Recommender Systems for Preventive Care: Protocol for Empirical Research Propositions

Web-Based Patient Recommender Systems for Preventive Care: Protocol for Empirical Research Propositions

Authors of this article:

Pamella Howell 1 Author Orcid Image ;   Arun Aryal 1 Author Orcid Image ;   Crystal Wu 1 Author Orcid Image


Department of Information Systems, College of Business and Economics, California State University, Los Angeles, Los Angeles, CA, United States

Corresponding Author:

Pamella Howell, MBA, MIS, PhD

Department of Information Systems

College of Business and Economics

California State University, Los Angeles

5151 State University Drive,

Los Angeles, CA, 90032

United States

Phone: 1 323 343 2929


Background: Preventive care helps patients identify and address medical issues early when they are easy to treat. The internet offers vast information about preventive measures, but the sheer volume of data can be overwhelming for individuals to process. To help individuals navigate this information, recommender systems filter and recommend relevant information to specific users. Despite their popularity in other fields, such as e-commerce, recommender systems have yet to be extensively studied as tools to support the implementation of prevention strategies in health care. This underexplored area presents an opportunity for recommender systems to serve as a complementary tool for medical professionals to enhance patient-centered decision-making and for patients to access health information. Thus, these systems can potentially improve the delivery of preventive care.

Objective: This study proposes practical, evidence-based propositions. It aims to identify the key factors influencing patients’ use of recommender systems and outlines a study design, methods for creating a survey, and techniques for conducting an analysis.

Methods: This study proposes a 6-stage approach to examine user perceptions of the factors that may influence the use of recommender systems for preventive care. First, we formulate 6 research propositions that can be developed later into hypotheses for empirical testing. Second, we will create a survey instrument by collecting items from extant literature and then verify their relevance using expert analysis. This stage will continue with content and face validity testing to ensure the robustness of the selected items. Using Qualtrics (Qualtrics), the survey can be customized and prepared for deployment on Amazon Mechanical Turk. Third, we will obtain institutional review board approval because this is a human subject study. In the fourth stage, we propose using the survey to collect data from approximately 600 participants on Amazon Mechanical Turk and then using R to analyze the research model. This platform will serve as a recruitment tool and the method of obtaining informed consent. In our fifth stage, we will perform principal component analysis, Harman Single Factor test, exploratory factor analysis, and correlational analysis; examine the reliability and convergent validity of individual items; test if multicollinearity exists; and complete a confirmatory factor analysis.

Results: Data collection and analysis will begin after institutional review board approval is obtained.

Conclusions: In pursuit of better health outcomes, low costs, and improved patient and provider experiences, the integration of recommender systems with health care services can extend the reach and scale of preventive care. Examining recommender systems for preventive care can be vital in achieving the quadruple aims by advancing the steps toward precision medicine and applying best practices.

International Registered Report Identifier (IRRID): PRR1-10.2196/43316

JMIR Res Protoc 2023;12:e43316



Current State of Recommender Systems in Health Care

The integration of recommender systems into eHealth or e-medicine is an underexplored area in research [1]. With the abundance of medical information available on the web, expertise and context-specific knowledge are needed to analyze it effectively [2]. Implementing technology such as recommender systems in health care services can bring about several benefits. First, it can enable aging population groups such as baby boomers, Generation X, and millennials to effectively process and filter the information they find on the web or receive from medical professionals [3]. Second, it can assist insurers and health care providers to shift toward a merit-based payment system, in which providers are compensated based on their patients’ outcomes and the ability to lower costs while keeping patients healthy [4]. To facilitate this transition, recommender systems will act as part of the patient-centered team-based workflow, by supporting patient compliance with the treatment protocol. This shift can encourage providers to adopt a preventive approach, rather than just treating secondary and tertiary medical conditions.


This study examines the extant literature and formulates quantitative propositions to explore the factors influencing patient use of recommender systems for preventive health decision-making. A proposition is a statement formulated for empirical testing that may be true or false based on some observable phenomena. It also proposes a study design and methods for developing a survey and conducting an analysis. Preventive care helps patients to identify diseases that can cause medical problems before they become serious. Many Americans are unaware of simple tests that could save their lives or prevent years of costly treatment [5,6]. For example, the US Preventive Services Task Force recommends initiating low-dose aspirin use for the primary prevention of cardiovascular disease. In 2019, the Centers for Disease Control and Prevention reported that stroke (cerebrovascular disease) was the fifth leading cause of death in the United States. Patient use of recommender systems can predict and enhance the treatment protocol for diseases such as stroke [7] and could reduce the number of deaths.

The existing literature lacks information about the factors motivating patients to use recommender systems for preventive care. The propositions developed in the study are intended to assist patients in deciding whether to use the available recommender systems for preventive care. We also aim to provide a starting point for future researchers interested in using recommender systems for preventive care. We begin this inquiry into the eHealth recommender system by describing the types of recommender systems. We develop practice-based empirical propositions and further the applied research angle by identifying salient issues with recommender systems. Finally, we discuss general issues with recommender systems and provide strategies to address them to improve the use of recommender systems.

Recommender System Types and Applications


Traditional recommender systems can be classified as collaborative, content based, and hybrid [8]. Recommender systems also use the following algorithmic approaches to recommend items: group and knowledge-based recommendations [9] and demographic filtering [10]. Finally, there is a context-aware filtering classification.

Collaborative Filtering

Collaborative filtering is the most used type of recommender system built on a collaborative algorithm that uses 1 user’s choice to predict subsequent user preferences [11]. Correlation among users plays an essential role in this type of recommender system. Collaborative filtering applications are heavily used in e-tourism, where the preferences of previous visitors are used to predict activities that new travelers may engage in [10]. In addition to traditional collaborative filtering methods, matrix factorization improves the algorithm by using a baseline [12]. Other proposed enhancements to the collaborative filtering method include using trust as a replacement for the evaluation performed by a user. Results from studies indicate that using trust as a filler for rating preserves the accuracy and coverage of the predictions provided by the recommender systems [13]. To improve customer resource management, researchers modified the collaborative filtering method to include a multifactor for a bipartite network—that is, an affiliation network. The proposed model improved the recommendations’ accuracy and diversity [14].

Researchers have proposed a hybrid method to address the limitations of the traditional collaborative filtering approach. By incorporating the Big 5 personality dimensions, the collaborative filtering algorithm’s active learning capabilities are enhanced, allowing for the use of personality type to overcome the problem of sparsity, resulting in improved accuracy and a great number of recommendations [15]. Other modifications include developing new systems that can learn and adapt to user feedback to elicit changes in health behaviors. For example, researchers have proposed a collaborative filtering-based computer-tailored health communications system that delivers personalized messages to individual patients based on explicit feedback (ratings) about past messages. These studies suggest that most users have positive opinions about smoking cessation support messages recommended based on their past ratings [16]. Another example of adapting to user feedback is creating an algorithm to recommend treatment based on patients’ pretreatment preferences and posttreatment satisfaction. This study demonstrates the feasibility of using a collaborative filtering recommender system to provide accurate recommendations and guide shared decision-making [17].

Furthermore, collaborative filtering and clustering can be used to develop a drug recommendation system that suggests medications for patients with diabetes by comparing a specific patient profile with those of patients with similar characteristics [18]. Similarly, collaborative filtering and clustering can be used to develop a recommender system for patients with cardiovascular diseases [19]. Although the number of studies on collaborative filtering systems is increasing, their application in the health care domain remains limited. The use of this type of recommender system for computer-tailored interventions and health recommendations is still in its early stages. However, there is potential for these systems to determine appropriate drugs and treatments for patients and deliver personalized messages that motivate behavior change.

Content-Based Filtering

Content-based recommender systems (CBRSs) rely primarily on item descriptions [20]. They evaluate options and recommend a product, person, or service based on a user’s previous acquisitions or suggested preferences [21]. The process of a CBRS is divided into 3 steps. First, the content is represented in a processable format. Then, the algorithm learns about the user. Finally, a recommendation is made for that specific user. The advantage of a content-based system is that it offers personalized recommendations and does not require a rating or previous documentation of an item, thus making the recommendation process more efficient [22]. However, the complexity of the current e-service and e-product landscape requires CBRS users to integrate it with other recommender algorithms.

CBRSs have a range of applications in health care, such as promoting data set reusability, refining the research process, and recommending health education. They accomplish this by streamlining the process of identifying relevant data sets. For example, by deducing researchers’ academic interests from their list of publications, the system can make data set recommendations that users may find useful or interesting [23]. In addition, the process of conducting clinical research studies is refined by calculating the similarity between annotated publications and the research query and recommending the most relevant literature. This enhanced recommender system can be integrated into research platforms to support future medical studies by helping researchers refine their research questions [24]. Furthermore, CBRS can be used to develop a content-based system that recommends health educational websites to users based on the metadata of the videos they have watched. The system suggests relevant websites for a given health video and facilitates the search for additional related content [25]. Although existing studies propose novel approaches, they have yet to be implemented in the broader health care context, and their limitations warrant further exploration. However, as the volume of health-related information increases, solutions that assist users in selecting relevant information become increasingly important. This recommender system can support health information retrieval by reducing the burden of searching for relevant information.

Hybrid Filtering

The hybrid filtering recommender system aims to compensate for the shortcomings of a classification model or enhance others with the advantages of an algorithm [21]. Hybrid recommender systems are a special type that combines ≥2 recommendation strategies. For example, researchers combine collaborative and knowledge-based filtering systems to create hybrids in the restaurant industry. When using hybrid systems, rich data, such as how a particular food helped reduce weight [26], can act as a filtering mechanism for group members. Results show that semantic knowledge-based ratings improve the collaborative filtering model [27]. Subsequent studies build upon previous findings by including an additional contextual modification that improves the performance of the recommender system [27]. Traditional methods of accessing videos, audio, and television have changed; a technological development is IP television, which streams content over proprietary IP-based networks. The recommender systems proposed by researchers suggest videos to users, which they would not have selected otherwise. Results show that recommender systems provide valuable recommendations despite high demand [28]. Most models today are developed as hybrids because they are more robust.

An example of hybrid collaborative filtering is a system that recommends search terms for a given patient to clinicians. The proposed hybrid model outperforms top-N recommendations by improving the efficiency and effectiveness of retrieving relevant patient information [29]. Another example is a Drug Development Recommendation model that supports pharmaceutical companies in determining which drug groups to develop. The Drug Development Recommendation model supports decision-making before companies initiate new drug development and improves the success rate by making recommendations tailored to a specific company and its unique business conditions. Another hybrid recommender system was developed to generate peer and content recommendations for patients with diabetes based on their profile, health status, and ratings of personalized content. This development offers a glimpse into the potential of recommender systems to promote a positive, healthy lifestyle via the implementation of a personalized, mobile peer support system [30]. The applications of hybrid recommender systems in health care are emerging; however, they are not used extensively in the health domain. The number of published studies on the use of hybrid models in health care is minimal so far. However, the limitations of content-based and collaborative recommender systems suggest that new studies will likely explore applications of the hybrid approach further. 

Group Recommendation

Group recommendations are based on the collaboration of an entire group [9,31]. Alternatively, some refer to this recommendation type as community based [32]. An individual’s community may include colleagues, friends, and family; members typically have a common outlook. Expansions in this area include travel and location-based services, where the trust walker algorithm enhances users’ perception of the reliability of location recommendations given [32]. Another study concentrated on folk recommendations with a tripartite composition, suggesting that this type of filtering connects users with the same interests via social tagging; a well-known example of this recommender system is Flickr [33]. The group recommender system matrix is sparse, and users’ preferences are unknown. Sparsity is a problem in a group recommender system matrix, and researchers developed a memory-based technique to resolve the sparsity issue. The results indicated that the proposed method performs better than existing approaches in generating group recommendations [34].

Group recommender systems support decision-making by integrating the preferences or attributes of members. There is an increase in the number of studies that evaluate group recommender systems; however, their exploration remains low in the eHealth and e-medicine categories. Previous studies have assessed the composition of a group and ways to optimize the accuracy of recommendations. A study found that personality could improve the accuracy of predictions [35]. Another network-based recommender algorithm in an Italian health care organization computed similarities among patients. It generated a ranked list of physicians and hospitals suitable for a given patient profile using health data shared by the community [36]. It will be challenging to generate a real group consensus in the health care domain; however, studies show that there are ways for group recommender systems to achieve a high level of agreement among their members [37].

Context-Aware Recommender Systems

The context-aware recommendation system (CARS) category is relatively new; it modifies the content-based or collaborative filtering classes. This category is noteworthy owing to its potential for expanding research on recommender systems. Certain factors define CARSs, some of which include purchase purpose, location, and time. The e–health care, e-music, and e-travel domains successfully use the CARS algorithm. Change over time is an assessment of contextual factors that points to change over time to determine static or dynamic elements. Factors are assessed based on what is known about the facet and whether it is unobservable, partially observable, or fully observable. There are 3 forms of context-aware recommendations (prefiltering, postfiltering, and context modeling), which include the context as part of the rating process [38]. Adding context and trust enhances the collaborative filtering algorithm to combat the sparsity problem. The results from studies including these modifications indicate better recommendations [39]. A prefilling contextual technique referred to as item splitting is used to separate the characteristics of an item into a minimum of 2 distinct contextual factors; this is another method of improving the sparsity problem [40]. Hybreed is a tool developed to integrate context in the recommender system design [41]. Using accuracy and diversity as performance measures, a comparison of the 3 forms of contextual recommendation models indicates that the task affected both performance measures. Therefore, further studies in this area are needed, as no attempts have been made to reproduce the 3780 experiments conducted in this study [42].

A detailed review of context awareness and health awareness in eHealth revealed that both factors are essential for developing user-personalized recommender systems [43]. Considerable amounts of data have accumulated in clinical databases. Recommender systems can supply patients with basic descriptions of information in health records to facilitate the comprehension of their health status. A study hypothesized whether the retrieval precision of an advanced health recommender system was higher than that of a naive system. The results indicated that advanced health recommender systems enhanced patient understanding of health records more than a naive system [44]. Researchers from a separate study proposed a cloud-based hybrid predictive model for a personalized health information service. Experiment results discovered that participants have a positive response toward usability measurement dimensions [45-47] of the system, such as perceived usefulness, expectation confirmation, perceived value, satisfaction, and perceived trust [48].

Overview and Workflow

This study will use a multistage qualitative methodology to explore patient perceptions about using recommender systems for preventive care decisions. Stage 1 of the framework focuses on developing practice-based empirical propositions. Stage 2 includes the development of survey instrument. Then, we will obtain institutional review board (IRB) approval for the study in stage 3, before data collection and analysis can be conducted in stages 4 and 5, respectively (Figure 1).

Figure 1. Study design and workflow. IRB: institutional review board; HINT: Human Intelligence Tasks.

Stage 1—Practice-Based Empirical Propositions


Although recommender systems are seen as valuable in health care, no studies synthesize the factors leading patients to use this technology. The following sections present the propositions developed from constructs in previous studies that may influence patient recommender systems. Examining factors influencing the use of this technology is vital because it will support achieving the quadruple health aims—improving the patient population, reducing the cost of care, enhancing the patient experience, and improving provider satisfaction [49].


Privacy is an issue within the literature on recommender systems [50]. Privacy concerns stem from the need for users to disseminate personalized information to allow for the best recommendation [51]. Researchers have developed methods to combat privacy challenges; however, this factor may still affect the patient’s perception of usability. Strategies to protect privacy include the secure processing architecture and the anonymous contributions architecture [52]. Another attempt to reduce privacy concerns in recommender systems is justification messages to improve the experience with a recommender system [53]. A recommender system can also assist individuals in making decisions regarding their privacy by providing them recommendations and warnings based on their privacy preferences [54].

Privacy concerns present unique challenges when adopting technology in health care. For example, users of wearable medical devices are vulnerable to data security and privacy protection issues [55]. Wearable technology data security can be classified into technical security, data management, or laws and regulations. Privacy concerns challenge mental health care providers with each advancement in technology used to deliver psychotherapy services [56]. Although using email, SMS text messaging, telehealth, electronic medical records, and mobile apps aid mental health providers, such technologies simultaneously increase the risks of unintended breaches of confidentiality. As privacy and confidentiality are regarded as essential considerations in mental health care, telepsychology is limited by concerns around privacy. To address these concerns, several researchers develop a risk assessment tool to assess whether health ITs (HITs) comply with privacy policy standards [57]. They find that users of HITs emphasize the need for transparency and policies that define the type of data collected and used. Thus, as privacy risks emerge as a common concern for health care professionals and patients, privacy issues undermine users’ confidence and willingness to use HITs. Privacy concerns affect the adoption of technology in health care. Therefore, maintaining the security and privacy of protected health information will be crucial for the future of health care.

Proposition 1—Privacy concerns may negatively affect the use of health care recommender systems for preventive care.


A social network of interlinked nodes connected by their confidence relations is a trust network [58,59]. Trust-based models help resolve sparsity problems [60]. Modifying existing recommender system classifications with trust improves ratings’ accuracy [61]. An early study on peer production services revealed that trust-based recommender systems can substantially enhance services and overcome the issue of information overload [62]. Trust can be aggregated into ontology and used to update user profiles to improve the recommendation process [63]. In scenarios where patients or individuals face conflicting recommendations, trust can help the user to make a better choice regarding the accuracy of the results [64]. Trust represents a vital antecedent of the acceptance and use of technology in health care. A recent study identifies perceived usefulness, usability, training, and technical support as trust facilitators. In contrast, the lack of privacy, perceived risks, and cost and security issues represent the main barriers [49]. Similarly, another study analyzes the enablers and impediments of trust in digital health. These trust factors indicate whether to place trust in digital health technologies [65]. Asan et al [66] focus on clinicians as primary users of artificial intelligence (AI) systems and how trust shapes their use and adoption of AI. In this context, trust refers to applying AI-based systems to support clinical decision-making. The lack of trust in AI deters users’ adoption of this technology in health care. As trust is the most notable factor influencing the adoption of wearable health devices [67], understanding how users define trust in the health care technology context provides insight into the use of technologies.

Proposition 2—Trust may positively affect the use of health care recommender systems for preventive care.


An effective recommendation will help a customer make a better choice or choose a product that they would otherwise not have selected. Studies have shown that a combination of rating and content can increase effectiveness [68]. Systems can provide less effective measures by overestimating or underestimating a recommendation for a particular patient [69]. Variations in predictions should be carefully considered during the development of a health system. For preventive care measures, overestimation may cause the most ripple effects. For example, a test ordered very frequently could increase costs, or inaccurate preventive measures could have considerable medical consequences. The lack of effectiveness is an issue when adopting technology in health care. For example, technological tools, including telehealth, mobile app, and SMS text messaging, can address the mental health crisis during the pandemic. Each tool helps provide the needed psychological support; for instance, the effectiveness of telehealth is comparable with that of in-person counseling services [70]. A similar study assesses the effectiveness of mobile health (mHealth) technology in suicide prevention. The results suggest that these mHealth technology tools effectively reduce suicide-specific outcomes [71]. In addition, digital health technology effectively monitors and treats asthma [72]. Ineffectiveness is a considerable adoption barrier because it indicates the future utility of a given technology in health care.

Proposition 3—Effective recommendations positively affect the use of health care recommender systems for preventive care.


Accuracy is one of the most frequently used measures for evaluating a recommender system; however, these often include complex calculations that average customers are not privileged to access. The patient’s perception of whether the system is making correct predictions may influence use [69]. Service providers use accuracy measures to help customers make better product and service selections. Will the patient’s perception that the recommendations are correct affect their opinion and usability of preventive health recommendations? Accuracy is also an issue when adopting technology in health care because the low accuracy of these technologies raises concerns regarding the threat posed to patient safety [73]. The adoption of these technologies should be carefully considered, especially if their performance is inferior to that of current practices. A recent study examines the quality of information provided by web-based symptom checkers, where physicians outperform symptom checkers regarding the accuracy of diagnostic data [73]. This accuracy concern highlights the need for further studies on symptom checkers. Users cannot solely depend on the technology; for example, the Apple Watch overestimates the value for energy expenditure of patients with cardiovascular disease but accurately measures their heart rate [74]. These accuracy issues imply that it is premature to use the Apple Watch for cardiac rehabilitation. Therefore, caution should be exercised when adopting technology with accuracy issues in health care.

Proposition 4—Perceived accuracy positively affects the use of health care recommender systems for preventive care.

User Satisfaction

User satisfaction refers to the perception of the system services after personally experiencing the system [75]. The level of satisfaction that patients receive from using community-based health care recommender systems may vary across individuals. If customers are more satisfied, they tend to implement the recommendations; the opposite is true if they are dissatisfied [48]. HIT literature reports high user satisfaction leading to increased adoption of technology in health care in various health care settings. For example, patients’ acceptance of the health technology program increased with high patient satisfaction with the digital tools incorporated into the program [76]. Patient satisfaction with telemedicine determines its long-term viability beyond the pandemic [77]. Patients experienced higher satisfaction with web-based visits than with in-person visits, which is not surprising considering that health care professionals can still address patients’ questions and concerns without compromising quality. High user satisfaction leads to great acceptability for lifestyle apps in mHealth [78]. User satisfaction indicates that a specific technology sufficiently satisfies patients; however, low satisfaction rates suggest that the barrier to adopting that technology in health care is important.

Proposition 5—User satisfaction positively affects the use of health care recommender systems for preventive care.


A fundamental business system requirement is efficiency. For example, one of the largest recommender systems, Google, searches in seconds. Similarly, more specialized recommender systems are expected to perform tasks quickly or reduce the time it takes to make choices [79]. Efficiency is the user’s perception that the recommendation improved decision-making [47]. When adopting HIT such as recommender systems, efficiency is important. HIT refers to technology that assists in preventing, diagnosing, and treating diseases. HITs, such as electronic health records and health information exchanges (HIEs), facilitate care coordination by making information available and providing real-time decision support to physicians [80]. HIEs enable the sharing of patient health information electronically among providers, patients, and payers across different health care settings [81,82]. Researchers have assessed the impact of HIT’s efficiency. The findings demonstrated that HIT improves efficiency by decreasing use rates [83]. A similar study found that adopting HIT enhances hospital efficiency. However, adopting HIT and increasing the employment of physicians nullifies the positive impact on hospital efficiency [84]. Finally, another study aimed to determine whether participating in HIE networks improves hospital efficiency. The author concludes that participating in HIEs improves hospital efficiency but calls for more research [85]. Similar to the implementation of other large-scale HIT, recommender systems may affect patient efficiency.

Proposition 6—Perceived efficiency positively affects the use of health care recommender systems for preventive care.

Stages 2 and 3—Instrument Development and Operationalization of Variables

Researchers can develop an instrument after literature reviews and interviews with subject matter experts, including information systems professionals, physicians, and patients. An expert in survey design will be consulted to ensure that the questions are phrased appropriately to elicit proper responses for the measured factors. The review team can consist of 3 to 5 individuals to represent the abovementioned expertise, including the principal investigator. Before the final items are selected for the survey, we will assess content and face validity using a short questionnaire that includes the items and construct definitions [86,87]. IRB approval may not be required for this process; 25 to 30 professionals or students can be used to elicit the validation results. Items will be rated on a 7-point scale from 1 (not at all) to 7 (completely), aligned with the construct’s definition. Items that do not match precisely with the construct definitions—responses below the mean—should be refined or removed from the final instrument. Discarded questions can be replaced with new ones, and the validation process can be repeated. Figure 2 depicts the plausible measurement model for the likelihood of using recommender systems for preventive care with several reflective indicators. Table 1 presents the constructs and references.

Figure 2. Research model—propositions for evaluating a recommender system for preventive care.
Table 1. Recommender system constructs and references.
Constructs and itemsStudies
Privacy concerns Xu et al [88,89]

I am concerned that the recommender system will collect large amount of personal information from me. a

I am concerned that the recommender system will use my personal information for unauthorized purposes. 

I am concerned that unauthorized people will have access to my personal information. 
Trust Liu and Tao [90] and Kim et al [91]

The recommender system is trustworthy. 

The content of the recommender system is reliable. 

Overall, I trust the recommender system. 
Competency-related trust Esmaeilzadeh [92], Mpinganjira [93], and McKnight et al [94]

The recommender system is competent.

The recommender system performs its role very well. 
Effectiveness Liu and Tao [90]

The recommender system provides personalized recommendations that are based on my information. 

The recommender system personalizes my health care experience by acquiring my personal preferences. 

The recommender system delivers personalized health care services. 
Accuracy Yang et al [95]

The recommender system offers consistent results over time. 

I feel confident that the recommender system offers error-free results. 
User satisfaction Park et al [96]

Overall, I am satisfied with the recommender system. 

The recommender system meets my expectations. 

I recommend the recommender system to others in health care. 
Efficiency Garg et al [97]

The recommender system will help in providing health care services. 

The recommender system will increase the speed of services. 

The recommender system will increase efficiency. 

aNot applicable.

The dependent variable—the use of recommender systems—can be modeled based on the theory of planned behavior using a binary item [98-100]. This theory contends that behavioral, normative, and control beliefs affect a person’s intentions to perform a particular behavior; the strength of these intentions influences an individual’s actual behavior [101,102]. This proposed study design adopts the theory of planned behavior as a fundamental behavior theory to represent patient behavior, especially in terms of their use of recommender systems. Researchers can examine patients’ adoption of recommender systems by extending the theory of planned behavior to the health care context. Individual attitudes, subjective norms, and perceived control influence patients’ intention to use recommender systems. Therefore, individual perception and social influences are antecedents of their intention to use recommender systems; the greater this intention, the more likely that the adoption behavior will be realized. The use variable can be dichotomous (yes or no).

Privacy concerns can include 3 items [88,89]. When formulating trust items, researchers can combine two trust constructs: (1) general trust [90,91] and (2) competency-related trust [92-94]. The effectiveness construct can use 3 reflective items [90]. The accuracy construct can be measured using 2 items [95]. Short-form measures have limitations; however, they are as effective at capturing a construct as extended scales when adequately designed [103]. Possible robustness checks are discussed in the Stage 6—Data Analysis section. Overall, 3 reflective items can be used to measure user satisfaction [96] and efficiency [97]. Scale scores can be computed as the averages of individual items and anchored to strongly agree, agree, somewhat agree, strongly disagree, disagree, and somewhat disagree. 

Stage 4—Ethics Approval

This study has been submitted for ethics approval to the California State University Institutional Review Board (reference number: 2038900-1) through a secure web portal. Approval may take 4 to 6 weeks, depending on whether revisions are required and the frequency with which the board meets. It will be approved if the board finds that this study involves minimal risk to the participants. Participants are informed that they can skip questions they do not feel comfortable answering or choose not to participate. The survey is anonymous, and participants cannot be identified.

Stage 5—Initial Data Collection: Amazon Mechanical Turk


This study will be conducted via the web; this aligns with the study’s aims to examine the perceptions of adults in need of preventive care. Data will be collected using Qualtrics (Qualtrics) via an anonymous survey link. Qualtrics is a web-based data collection tool; its functionality allows users to create a customized survey.

Amazon Mechanical Turk is a hosted platform that allows researchers access to a wide range of participants [104]. Participants are called turkers or crowdworkers. They browse their profile for Human Intelligence Tasks (HINT), which includes a brief description of the responsibilities. Posting the survey on the HINT list will initiate participant recruitment for the study. Amazon Mechanical Turk allows requesters to select worker requirements to enforce the inclusion and exclusion criteria. To be included, participants should be aged ≥18 years without serious mental impairments and consent to participate in the study. Participants must be in the United States and be able to read and write in English. The procedure for recruiting individuals and how participants access and complete the survey is depicted in Figure 3.

Figure 3. Amazon Mechanical Turk (M-Turk) workflow. HINT: Human Intelligence Tasks.

The survey cover page will be used to distribute the recruitment letter and obtain informed consent. The letter will also provide an overview of the study, participation criteria, and any benefits or risks to the participant. Completed survey data are automatically stored if Qualtrics is used. It is reasonable to assume that the time needed to complete the survey will be approximately 10 to 15 minutes. Before beginning the survey, all participants will receive a vignette [105] describing what a recommender system is and an example of how it will be used in their care.

Sample Vignette

Recommender systems are the systems that are designed to suggest things to the user based on many different factors. They are used by companies such as Google, Instagram, Spotify, Amazon, Reddit, and Netflix, to entice users with relevant content. A health recommender system may propose preventive measures to patients based on their clinical history and demographic characteristics. For example, based on available data, it may recommend talking to a primary care provider about the decision to initiate low-dose aspirin use for the primary prevention of cardiovascular disease.

Participants will answer 19 to 20 reflective questions to measure each construct, as outlined in Table 1. They will answer demographic questions on ethnicity, race, sex, income, education, employment, and the number of chronic conditions. No compensation will be provided for participating in the study. A sample size of approximately 600 participants may result in a medium effect size where the r value varies from 0.3 to 0.4. Effect size is a quantitative measure of the magnitude of the experimental effect [106].

Stage 6—Data Analysis

Researchers can use R to analyze the research model; it is stage 5 of the proposed study design. The following robustness check can be used to evaluate the measurement instrument. Principal component analysis will be performed; only factors with eigenvalues >1 will be selected; and then, a visual assessment will be conducted using a scree plot. A parallel analysis will be completed to review the eigenvalues for the 6 factors in the conceptual model. A Harman Single Factor test can be performed to test the effect of common bias to ensure that a single factor does not emerge. To assess discriminant validity, exploratory factor analysis will be performed. Then, any factor loading <0.70 will be examined; a loading value of 0.55 is considered as good and 0.45 is considered as fair [107]. Reliability and validity will be examined using Cronbach α; a value >0.7 is satisfactory. Multicollinearity should be examined as part of the analysis; correlation should fall below the conventional threshold value of 0.6 [108,109].

To establish the reliability and convergent validity of individual items, the average variance extracted for each construct will be calculated. Composite reliability measures the overall reliability of a collection of heterogeneous but similar items. Composite reliability >0.70 threshold and extracted variance >0.50 threshold are recommended [110]. The analysis of variance inflation factors will determine whether multicollinearity exists among the tested variables. Variance inflation factors are a scaled version of the multiple correlation coefficients between 1 variable and the remaining independent variables. Variance inflation factors below the cutoff value of 10 indicate that multicollinearity among the variables is not supported [111]. Researchers should perform confirmatory factor analysis. The goodness-of-fit indices [112-114], adjusted goodness of fit, comparative fit index, Normed Fit Index by Bentler and Bonett [115], root mean square error approximation, and root mean square residual will be examined.

Data collection and analysis will begin after institutional review board approval is obtained.

This study proposes practice-based empirical propositions for developing recommender systems in health care, focusing on identifying the factors influencing patients’ use of these systems. We also provide a study design and methods for creating a survey and conducting an analysis. Our study highlights the potential of recommender systems to improve population health by educating patients about disease prevention and motivating them to use preventive services. This study aims to guide future research in the health care domain by exploring the application of recommender systems in preventive care.

Clinical quality measures such as preventive metrics are essential for monitoring health care quality. Often, patients are unaware of the many preventive quality metrics that can help improve their health care outcomes. Improved patient awareness of preventive measures helps not only them but also physicians and insurers to meet the metrics of clinical quality initiatives from advanced payment models such as primary care first, accountable care organization, merit-based incentive payment system, and Healthcare Effectiveness Data and Information Set. Hybrid recommender systems can help patients filter and use data about preventive quality metrics, making the information more accessible. Using the propositions mentioned in this paper, health care professionals and researchers can examine the factors that will lead patients to use recommender systems for preventive care.

There are advantages to implementing recommender systems and general issues that may influence patient use of the systems. As discussed in the propositions, patient privacy concerns may affect patients’ likelihood of using recommender systems. Issues stemming from privacy concerns may lead patients to not share their medical records, resulting in the cold start problem. The cold start problem affects recommender systems because new users do not have a profile, thereby limiting the ability of the recommendation systems to provide meaningful information. Another privacy concern is the intrusive nature of recommender systems [69]. Algorithms require direct feedback that is linkable to a particular user; this may engender privacy concerns. Patients may be less likely to seek recommendations if their searches or history can be linked to them by administrators.

To improve patient use of recommender systems, developers must be able to improve the trust in the systems they use or the administrators governing the data collection, processing, and dissemination processes. An issue with recommender systems is overspecification, which could result in duplicate preventive testing. Patients who receive duplicated recommendations may reduce their trust in the system’s capabilities. Duplicate testing may result in 2 issues: patient fatigue may reduce trust or overuse may increase costs. To reduce the influence of overspecification on preventive recommendations, developers may consider integrating current structured data from the patient’s medical records to train and test recommender systems.

Perceptions about the effectiveness and accuracy of recommender systems also reduce the likelihood of patients using them. A factor that could cause efficiency or accuracy issues with algorithms is the sparsity problem, in which some services have many ratings and others do not. Data accumulation is a method to rectify the sparsity problem to facilitate accurate predictions [8]. Efforts to address this data sparsity issue should include developers working closely with insurers, providers, and patients to limit the factors affecting their likelihood of sharing data. Data blocking on the physician’s side may also lead to sparsity; in response, the Cures Act should limit this practice by 2023. Furthermore, a patient may choose to share the data; however, it may not be exportable or interoperable with data from the internet or other sources. Unfortunately, data from multiple inoperable sources can lead to low efficiency. Health care organization professionals can tackle this challenge by implementing systematic data standards such as Systematized Nomenclature of Medicine, Logical Observation Identifiers Names and Codes, Current Procedural Terminology, and International Classification of Diseases and the fast health care interoperable resource standards.

This study does not address all the psychological aspects of recommender systems. Further studies are needed to better understand the physiological effects of these systems on users [69]. The trust construct developed in this study does not relate to the patient-physician trust. However, when addressing concerns about privacy and sensitive patient data, it is crucial to consider the patient-physician relationship in the design of web-based recommender systems. Developers should include features for effective communication and collaboration between patients and qualified health professionals.

Providing access to IT support services and mobile options for accessing preventive recommendations can improve patient satisfaction. Efficiency and scalability also play key roles in determining the success of recommender systems. As the amount of data available on the web continues to grow, algorithms must be able to handle the increased computational demands. Studies have shown that accuracy does not necessarily increase with large amounts of data, but computation time may [116]. Health care professionals may consider partnering with large IT companies that offer cloud computing resources to address scalability issues. Federal insurers such as Medicaid may also consider using a portion of cost savings from tracking preventive care to fund cloud computing hubs.

Additional studies are necessary on the physiological aspects of recommender systems [69]. In doing so, developers may improve user satisfaction and, therefore, the likelihood of use. Health care records can contain some of the most sensitive data about a person’s life. Addressing the intrusive nature of recommender systems and the sensitivity of patient data leveraging the patient-physician relationship is paramount. Developers of web-based recommender systems should include features for effective communication and collaboration between patients and qualified health professionals. It may not be feasible to grant patients access to their private physicians; however, allowing access to a professional may improve patient satisfaction. Recommender systems should provide, at minimum, some IT support services and offer a mobile way to access preventive recommendations [117]. Many large IT companies such as Amazon, Microsoft, and IBM have entered the health care arena; it may be opportunistic for health care professionals to partner with them to access cloud computing resources.


The authors received no financial support for the study, authorship, or publication of this paper.

Data Availability

The data sets generated during and analyzed in this study will be available from the corresponding author upon reasonable request after completion of the study.

Conflicts of Interest

None declared.

  1. Hors-Fraile S, Rivera-Romero O, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit-Balcells A, et al. Analyzing recommender systems for health promotion using a multidisciplinary taxonomy: a scoping review. Int J Med Inform 2018 Jun;114:143-155. [CrossRef] [Medline]
  2. Hall A, Walton G. Information overload within the health care system: a literature review. Health Info Libr J 2004 Jun;21(2):102-108 [FREE Full text] [CrossRef] [Medline]
  3. Zhao YC, Zhao M, Song S. Online health information seeking behaviors among older adults: systematic scoping review. J Med Internet Res 2022 Feb 16;24(2):e34790 [FREE Full text] [CrossRef] [Medline]
  4. Levine S, Malone E, Lekiachvili A, Briss P. Health care industry insights: why the use of preventive services is still low. Prev Chronic Dis 2019 Mar 14;16:E30 [FREE Full text] [CrossRef] [Medline]
  5. Elnicki D. Patient-perceived barriers to preventive health care among indigent, rural appalachian patients. Arch Intern Med 1995 Feb 27;155(4):421-424. [CrossRef]
  6. Tam DY, Lo YY, Tsui W. Knowledge, practices and expectations of preventive care: a qualitative study of patients attending government general outpatient clinics in Hong Kong. BMC Fam Pract 2018 May 09;19(1):58 [FREE Full text] [CrossRef] [Medline]
  7. Tran TN, Felfernig A, Trattner C, Holzinger A. Recommender systems in the healthcare domain: state-of-the-art and research issues. J Intell Inf Syst 2020 Dec 17;57(1):171-201. [CrossRef]
  8. Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 2005 Jun;17(6):734-749. [CrossRef]
  9. Felfernig A, Jeran M, Ninaus G, Reinfrank F, Reiterer S. Toward the next generation of recommender systems: applications and research challenges. In: Multimedia Services in Intelligent Environments. Smart Innovation, Systems and Technologies, vol 24. Heidelberg: Springer; 2013.
  10. Gavalas D, Konstantopoulos C, Mastakas K, Pantziou G. Mobile recommender systems in tourism. J Netw Comput Appl 2014 Mar;39:319-333. [CrossRef]
  11. Breese J, Heckerman D, Kadie C. Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence. 1998 Presented at: UAI'98: Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence; Jul 24 - 26, 1998; Madison Wisconsin.
  12. Koren Y. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008 Presented at: KDD08: The 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Aug 24 - 27, 2008; Las Vegas Nevada USA. [CrossRef]
  13. Massa P, Avesani P. Trust-aware recommender systems. In: Proceedings of the 2007 ACM conference on Recommender systems. 2007 Presented at: RecSys07: ACM Conference on Recommender Systems; Oct 19 - 20, 2007; Minneapolis MN USA. [CrossRef]
  14. Xu C. Personal recommendation using a novel collaborative filtering algorithm in customer relationship management. Discrete Dynamics Nature Soc 2013;2013:1-9. [CrossRef]
  15. Elahi M, Braunhofer M, Ricci F, Tkalcic M. Personality-based active learning for collaborative filtering recommender systems. In: Advances in Artificial Intelligence. Cham: Springer; 2013.
  16. Marlin B, Adams RJ, Sadasivam R, Houston TK. Towards collaborative filtering recommender systems for tailored health communications. AMIA Annu Symp Proc 2013;2013:1600-1607 [FREE Full text] [Medline]
  17. Shaoibi A, Neelon B, Lenert LA. Shared decision making: from decision science to data science. Med Decis Making 2020 Feb 06;40(3):254-265. [CrossRef]
  18. Granda Morales LF, Valdiviezo-Diaz P, Reátegui R, Barba-Guaman L. Drug recommendation system for diabetes using a collaborative filtering and clustering approach: development and performance evaluation. J Med Internet Res 2022 Jul 15;24(7):e37233 [FREE Full text] [CrossRef] [Medline]
  19. Mustaqeem A, Anwar SM, Majid M. A modular cluster based collaborative recommender system for cardiac patients. Artif Intell Med 2020 Jan;102:101761. [CrossRef] [Medline]
  20. Pazzani M. A framework for collaborative, content-based and demographic filtering. Artif Intell Rev 1999;13(5-6):393-408. [CrossRef]
  21. Ricci F, Rokach L, Shapira B. Introduction to recommender systems handbook. In: Recommender Systems Handbook. Boston, MA: Springer; 2011.
  22. Lops P, de Gemmis M, Semeraro G. Content-based recommender systems: state of the art and trends. In: Recommender Systems Handbook. Boston, MA: Springer; 2011.
  23. Patra B, Roberts K, Wu H. A content-based dataset recommendation system for researchers-a case study on Gene Expression Omnibus (GEO) repository. Database (Oxford) 2020 Jan 01;2020:1 [FREE Full text] [CrossRef] [Medline]
  24. Almeida J, Silva JF, Matos S, Pazos A, Oliveira JL. A recommender system to help refining clinical research studies. In: Public Health and Informatics. Amsterdam: IOS Press; 2021.
  25. Sanchez Bocanegra CL, Sevillano Ramos JL, Rizo C, Civit A, Fernandez-Luque L. HealthRecSys: a semantic content-based recommender system to complement health videos. BMC Med Inform Decis Mak 2017 May 15;17(1):63 [FREE Full text] [CrossRef] [Medline]
  26. van Pinxteren Y, Geleijnse G, Kamsteeg P. Deriving a recipe similarity measure for recommending healthful meals. In: Proceedings of the 16th international conference on Intelligent user interfaces. 2011 Presented at: IUI '11: 16th International Conference on Intelligent User Interfaces; Feb 13 - 16, 2011; Palo Alto CA USA. [CrossRef]
  27. Burke R. Hybrid recommender systems: survey and experiments. User Model User Adap Inter 2002;12(4):331-370. [CrossRef]
  28. Pripuzic K, Žarko IP, Podobnik V, Lovrek I, Čavka M, Petković I, et al. Building an IPTV VoD recommender system: an experience report. In: Proceedings of the 12th International Conference on Telecommunications. 2013 Presented at: 12th International Conference on Telecommunications; Jun 26-28, 2013; Zagreb, Croatia.
  29. Ren Z, Peng B, Schleyer TK, Ning X. Hybrid collaborative filtering methods for recommending search terms to clinicians. J Biomed Inform 2021 Jan;113:103635 [FREE Full text] [CrossRef] [Medline]
  30. Chomutare T, Arsand E, Hartvigsen G. Mobile peer support in diabetes. Stud Health Technol Inform 2011;169:48-52. [Medline]
  31. Bobadilla J, Ortega F, Hernando A, Gutiérrez A. Recommender systems survey. Knowl Based Syst 2013 Jul;46:109-132. [CrossRef]
  32. Ravi L, Vairavasundaram S. A collaborative location based travel recommendation system through enhanced rating prediction for the group of users. Comput Intell Neurosci 2016;2016:1291358-1291328 [FREE Full text] [CrossRef] [Medline]
  33. Godoy D, Corbellini A. Folksonomy-based recommender systems: a state-of-the-art review. Int J Intell Syst 2015 Aug 14;31(4):314-346. [CrossRef]
  34. Ghazarian S, Nematbakhsh MA. Enhancing memory-based collaborative filtering for group recommender systems. Expert Syst Appl 2015 May;42(7):3801-3812. [CrossRef]
  35. Quijano-Sanchez L, Recio-Garcia JA, Diaz-Agudo B, Jimenez-Diaz G. Social factors in group recommender systems. ACM Trans Intell Syst Technol 2013 Feb;4(1):1-30. [CrossRef]
  36. Narducci F, Musto C, Polignano M, Gemmis M, Lops P, Semeraro G. A recommender system for connecting patients to the right doctors in the HealthNet social network. In: Proceedings of the 24th International Conference on World Wide Web. 2015 Presented at: WWW '15: 24th International World Wide Web Conference; May 18 - 22, 2015; Florence, Italy. [CrossRef]
  37. Castro J, Quesada FJ, Palomares I, Martínez L. A consensus-driven group recommender system. Int J Intell Syst 2015 Apr 01;30(8):887-906. [CrossRef]
  38. Adomavicius G, Tuzhilin A. Context-aware recommender systems. In: Recommender Systems Handbook. Boston, MA: Springer; 2010.
  39. Linda S, Bharadwaj K. A fuzzy trust enhanced collaborative filtering for effective context-aware recommender systems. In: Proceedings of First International Conference on Information and Communication Technology for Intelligent Systems: Volume 2. Cham: Springer; 2017.
  40. Baltrunas L, Ricci F. Experimental evaluation of context-dependent collaborative filtering using item splitting. User Model User Adap Inter 2013 Feb 7;24(1-2):7-34. [CrossRef]
  41. Hussein T, Linder T, Gaulke W, Ziegler J. Hybreed: a software framework for developing context-aware hybrid recommender systems. User Model User Adap Inter 2012 Dec 28;24(1-2):121-174. [CrossRef]
  42. Panniello U, Tuzhilin A, Gorgoglione M. Comparing context-aware recommender systems in terms of accuracy and diversity. User Model User Adap Inter 2012 Dec 27;24(1-2):35-65. [CrossRef]
  43. User Modeling and Adaptation for Daily Routines Providing Assistance to People with Special Needs. London: Springer; 2013.
  44. Wiesner M, Pfeifer D. Health recommender systems: concepts, requirements, technical basics and challenges. Int J Environ Res Public Health 2014 Mar 03;11(3):2580-2607 [FREE Full text] [CrossRef] [Medline]
  45. Carenini G, Mittal VO, Moore JD. Generating patient-specific interactive natural language explanations. Proc Annu Symp Comput Appl Med Care 1994:5-9 [FREE Full text] [Medline]
  46. Sinha R, Swearingen K. The role of transparency in recommender systems. In: Proceedings of the CHI '02 Extended Abstracts on Human Factors in Computing Systems. 2002 Presented at: CHI02: Human Factors in Computing Systems; Apr 20 - 25, 2002; Minneapolis, Minnesota, USA. [CrossRef]
  47. Tintarev N, Masthoff J. A survey of explanations in recommender systems. In: Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop. 2007 Presented at: 2007 IEEE 23rd International Conference on Data Engineering Workshop; Apr 17-20 , 2007; Istanbul, Turkey. [CrossRef]
  48. Wang S, Chen YL, Kuo AM, Chen H, Shiu YS. Design and evaluation of a cloud-based mobile health information recommendation system on wireless sensor networks. Comput Electrical Eng 2016 Jan;49:221-235. [CrossRef]
  49. Abbas R, Carroll N, Richardson I, Beecham S. Trust factors in healthcare technology: a healthcare professional perspective. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies-HEALTHINF. 2018 Presented at: 11th International Joint Conference on Biomedical Engineering Systems and Technologies -HEALTHINF; Jan 19-21 , 2018; Funchal, Madeira, Portugal. [CrossRef]
  50. Resnick P, Varian HR. Recommender systems. Commun ACM 1997 Mar;40(3):56-58. [CrossRef]
  51. Jeckmans A, Beye M, Erkin Z, Hartel P, Lagendijk R, Tang Q. Privacy in recommender systems. In: Social Media Retrieval. London: Springer; 2012.
  52. Hoens TR, Blanton M, Steele A, Chawla NV. Reliable medical recommendation systems with patient privacy. ACM Trans Intell Syst Technol 2013 Oct 08;4(4):1-31. [CrossRef]
  53. Knijnenburg BP, Kobsa A. Making decisions about privacy: information disclosure in context-aware recommender systems. ACM Trans Interact Intell Syst 2013 Oct 1;3(3):1-23. [CrossRef]
  54. Rasmussen C, Dara R. Recommender systems for privacy management: a framework. In: Proceedings of the 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering. 2014 Presented at: 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering; Jan 09-11 , 2014; Miami Beach, FL, USA. [CrossRef]
  55. Jiang D, Shi G. Research on data security and privacy protection of wearable equipment in healthcare. J Healthc Eng 2021 Feb 5;2021:6656204-6656207 [FREE Full text] [CrossRef] [Medline]
  56. Lustgarten SD, Garrison YL, Sinnard MT, Flynn AW. Digital privacy in mental healthcare: current issues and recommendations for technology use. Curr Opin Psychol 2020 Dec;36:25-31 [FREE Full text] [CrossRef] [Medline]
  57. LaMonica HM, Roberts AE, Lee GY, Davenport TA, Hickie IB. Privacy practices of health information technologies: privacy policy risk assessment study and proposed guidelines. J Med Internet Res 2021 Sep 16;23(9):e26317 [FREE Full text] [CrossRef] [Medline]
  58. Yuan W, Guan D, Lee Y, Lee S, Hur SJ. Improved trust-aware recommender system using small-worldness of trust networks. Knowledge Based Syst 2010 Apr;23(3):232-238. [CrossRef]
  59. Yuan W, Guan D, Lee S, Wang J. Skeleton searching strategy for recommender searching mechanism of trust-aware recommender systems. Comput J 2014 Mar 19;58(9):1876-1883. [CrossRef]
  60. Zheng X, Chen C, Hung J, He W, Hong F, Lin Z. A hybrid trust-based recommender system for online communities of practice. IEEE Trans Learning Technol 2015 Oct 1;8(4):345-356. [CrossRef]
  61. Abderrahim N, Benslimane SM. Towards improving recommender system: a social trust-aware approach. Int J Modern Educ Comput Sci 2015 Feb 08;7(2):8-15. [CrossRef]
  62. Li Y, Kao C. TREPPS: a trust-based recommender system for peer production services. Expert Syst Applications 2009 Mar;36(2):3263-3277. [CrossRef]
  63. Martinez-Cruz C, Porcel C, Bernabé-Moreno J, Herrera-Viedma E. A model to represent users trust in recommender systems using ontologies and fuzzy linguistic modeling. Inform Sci 2015 Aug;311:102-118. [CrossRef]
  64. Bedi P, Vashisth P. Empowering recommender systems using trust and argumentation. Inform Sci 2014 Sep;279:569-586. [CrossRef]
  65. Adjekum A, Blasimme A, Vayena E. Elements of trust in digital health systems: scoping review. J Med Internet Res 2018 Dec 13;20(12):e11254 [FREE Full text] [CrossRef] [Medline]
  66. Asan O, Bayrak AE, Choudhury A. Artificial intelligence and human trust in healthcare: focus on clinicians. J Med Internet Res 2020 Jun 19;22(6):e15154 [FREE Full text] [CrossRef] [Medline]
  67. Asadi S, Abdullah R, Safaei M, Nazir S. An integrated SEM-neural network approach for predicting determinants of adoption of wearable healthcare devices. Mobile Inform Syst 2019 Feb 14;2019:1-9. [CrossRef]
  68. Symeonidis P, Nanopoulos A, Manolopoulos Y. Justified recommendations based on content and rating data. Citeseerx.   URL: https:/​/citeseerx.​​document?repid=rep1&type=pdf&doi=2d3ea40560d4e6b598f5347d451a30652ce3540b [accessed 2023-03-06]
  69. Tintarev N, Masthoff J. Designing and evaluating explanations for recommender systems. In: Recommender Systems Handbook. Boston, MA: Springer; 2010.
  70. Figueroa CA, Aguilera A. The need for a mental health technology revolution in the COVID-19 pandemic. Front Psychiatry 2020 Jun 3;11:523 [FREE Full text] [CrossRef] [Medline]
  71. Melia R, Francis K, Hickey E, Bogue J, Duggan J, O'Sullivan M, et al. Mobile health technology interventions for suicide prevention: systematic review. JMIR Mhealth Uhealth 2020 Jan 15;8(1):e12516 [FREE Full text] [CrossRef] [Medline]
  72. Mosnaim G, Safioti G, Brown R, DePietro M, Szefler SJ, Lang DM, et al. Digital health technology in asthma: a comprehensive scoping review. J Allergy Clin Immunol Pract 2021 Jun;9(6):2377-2398 [FREE Full text] [CrossRef] [Medline]
  73. Wallace W, Chan C, Chidambaram S, Hanna L, Iqbal FM, Acharya A, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digit Med 2022 Aug 17;5(1):118 [FREE Full text] [CrossRef] [Medline]
  74. Falter M, Budts W, Goetschalckx K, Cornelissen V, Buys R. Accuracy of Apple watch measurements for heart rate and energy expenditure in patients with cardiovascular disease: cross-sectional study. JMIR Mhealth Uhealth 2019 Mar 19;7(3):e11889 [FREE Full text] [CrossRef] [Medline]
  75. Wang S, Wu C. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Applications 2011 Sep;38(9):10831-10838. [CrossRef]
  76. Baumel A, Correll C, Hauser M, Brunette M, Rotondi A, Ben-Zeev D, et al. Health technology intervention after hospitalization for schizophrenia: service utilization and user satisfaction. Psychiatr Serv 2016 Sep 01;67(9):1035-1038. [CrossRef] [Medline]
  77. Nanda M, Sharma R. A review of patient satisfaction and experience with telemedicine: a virtual solution during and beyond COVID-19 pandemic. Telemed J E Health 2021 Dec;27(12):1325-1331. [CrossRef] [Medline]
  78. Overdijkink SB, Velu AV, Rosman AN, van Beukering MD, Kok M, Steegers-Theunissen RP. The usability and effectiveness of mobile health technology-based lifestyle and medical intervention apps supporting health care during pregnancy: systematic review. JMIR Mhealth Uhealth 2018 Apr 24;6(4):e109 [FREE Full text] [CrossRef] [Medline]
  79. SYMEONIDIS P, Nanopoulos A, Papadopoulos A, Manolopoulos Y. Collaborative recommender systems: combining effectiveness and efficiency. Expert Syst Applications 2008 May;34(4):2995-3013. [CrossRef]
  80. Jha AK, Doolan D, Grandt D, Scott T, Bates DW. The use of health information technology in seven nations. Int J Med Inform 2008 Dec;77(12):848-854. [CrossRef] [Medline]
  81. Kern LM, Kaushal R. Health information technology and health information exchange in New York State: new initiatives in implementation and evaluation. J Biomed Inform 2007 Dec;40(6 Suppl):S17-S20 [FREE Full text] [CrossRef] [Medline]
  82. Shen N, Bernier T, Sequeira L, Strauss J, Silver MP, Carter-Langford A, et al. Understanding the patient privacy perspective on health information exchange: a systematic review. Int J Med Inform 2019 May;125:1-12. [CrossRef] [Medline]
  83. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006 May 16;144(10):742-752 [FREE Full text] [CrossRef] [Medline]
  84. Cho N, Chang J, Atems B. The effects of health information technology adoption and hospital-physician integration on hospital efficiency. Am J Manag Care 2014 Nov;20(11 Spec No. 17):eSP9-eS15 [FREE Full text] [Medline]
  85. Walker DM. Does participation in health information exchange improve hospital efficiency? Health Care Manag Sci 2018 Sep 24;21(3):426-438. [CrossRef] [Medline]
  86. Maynes TD, Podsakoff PM. Speaking more broadly: an examination of the nature, antecedents, and consequences of an expanded set of employee voice behaviors. J Appl Psychol 2014 Jan;99(1):87-112. [CrossRef] [Medline]
  87. Meier S. Measuring Change in Counseling and Psychotherapy. New York: Guilford Publications; 2008.
  88. Xu H, Dinev T, Smith H, Hart P. Examining the formation of individual’s privacy concerns: toward an integrative view. In: Proceedings of the International Conference on Information Systems 2008. 2008 Presented at: International Conference on Information Systems 2008; Dec 14 - 17 , 2008; Paris, France.
  89. Xu H, Dinev T, Smith J, Hart P. Information privacy concerns: linking individual perceptions with institutional privacy assurances. J Assoc Inform Syst 2011;12(12). [CrossRef]
  90. Liu K, Tao D. The roles of trust, personalization, loss of privacy, and anthropomorphism in public acceptance of smart healthcare services. Comput Human Behav 2022 Feb;127:107026. [CrossRef]
  91. Kim DJ, Ferrin DL, Rao HR. A trust-based consumer decision-making model in electronic commerce: the role of trust, perceived risk, and their antecedents. Decis Support Syst 2008 Jan;44(2):544-564. [CrossRef]
  92. Esmaeilzadeh P. The impacts of the perceived transparency of privacy policies and trust in providers for building trust in health information exchange: empirical study. JMIR Med Inform 2019 Nov 26;7(4):e14050 [FREE Full text] [CrossRef] [Medline]
  93. Mpinganjira M. Precursors of trust in virtual health communities: a hierarchical investigation. Inform Manage 2018 Sep;55(6):686-694. [CrossRef]
  94. McKnight DH, Choudhury V, Kacmar C. Developing and validating trust measures for e-commerce: an integrative typology. Inform Syst Res 2002 Sep;13(3):334-359. [CrossRef]
  95. Yang Q, Al Mamun A, Hayat N, Jingzu G, Hoque ME, Salameh AA. Modeling the intention and adoption of wearable fitness devices: a study using SEM-PLS analysis. Front Public Health 2022 Jul 6;10:918989 [FREE Full text] [CrossRef] [Medline]
  96. Park E, Kim H, Ohm JY. Understanding driver adoption of car navigation systems using the extended technology acceptance model. Behav Inform Technol 2014 Oct 07;34(7):741-751. [CrossRef]
  97. Garg P, Gupta B, Chauhan AK, Sivarajah U, Gupta S, Modgil S. Measuring the perceived benefits of implementing blockchain technology in the banking sector. Technol Forecast Soc Change 2021 Feb;163:120407. [CrossRef]
  98. Ajzen I. From intentions to actions: a theory of planned behavior. In: Action Control. Berlin, Heidelberg: Springer; 1985.
  99. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process 1991 Dec;50(2):179-211. [CrossRef]
  100. Madden TJ, Ellen PS, Ajzen I. A comparison of the theory of planned behavior and the theory of reasoned action. Pers Soc Psychol Bull 2016 Jul 02;18(1):3-9. [CrossRef]
  101. Ajzen I. Constructing a theory of planned behavior questionnaire. TPB Questionnaire Construction. 2006 Jan.   URL: https:/​/www.​​publication/​235913732_Constructing_a_Theory_of_Planned_Behavior_Questionnaire [accessed 2023-03-06]
  102. Bosnjak M, Ajzen I, Schmidt P. The theory of planned behavior: selected recent advances and applications. Eur J Psychol 2020 Aug 31;16(3):352-356 [FREE Full text] [CrossRef] [Medline]
  103. Gogol K, Brunner M, Goetz T, Martin R, Ugen S, Keller U, et al. “my questionnaire is too long!” the assessments of motivational-affective constructs with three-item and single-item measures. Contemporary Educ Psychol 2014 Jul;39(3):188-205. [CrossRef]
  104. Ibarra JL, Agas JM, Lee M, Pan JL, Buttenheim AM. Comparison of online survey recruitment platforms for hard-to-reach pregnant smoking populations: feasibility study. JMIR Res Protoc 2018 Apr 16;7(4):e101 [FREE Full text] [CrossRef] [Medline]
  105. Lupiáñez-Villanueva F, Folkvord F, Vanden Abeele M. Influence of the business revenue, recommendation, and provider models on mobile health app adoption: three-country experimental vignette study. JMIR Mhealth Uhealth 2020 Jun 04;8(6):e17272 [FREE Full text] [CrossRef] [Medline]
  106. Cohen J. A power primer. Psychol Bull 1992;112(1):155-159. [CrossRef]
  107. Comrey AL, Lee HB. Interpretation and application of factor analytic results. In: A First Course in Factor Analysis. London, England, United Kingdom: Psychology Press; 1992.
  108. Peng J, So TH. Logistic regression analysis and reporting: a primer. Understand Stat Educ 2002 Feb;1:31-70. [CrossRef]
  109. Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Market Res 2018 Nov 28;18(1):39-50. [CrossRef]
  110. Foody J, Benner JS, Frishman W. Adherence. J Clin Hypertens (Greenwich) 2007 Apr;9(4):271-275 [FREE Full text] [CrossRef] [Medline]
  111. Toothaker LE, Aiken LS, West SG. Multiple regression: testing and interpreting interactions. J Operat Res Society 1994 Jan;45(1):119. [CrossRef]
  112. Tabachnick B, Fidell L. Using Multivariate Statistics. Manhattan, New York, United States: Harper & Row; 1989.
  113. Bentler PM. Theory and Implementation of EQS A Structural Equations Program. California, Los Angeles: BMDP Statistical Software, Incorporated; 1985.
  114. Hu L, Bentler PM. Fit indices in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol Method 1998 Dec;3(4):424-453. [CrossRef]
  115. Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull 1980 Nov;88(3):588-606. [CrossRef]
  116. Sharma L, Gera A. A survey of recommendation system: research challenges. Int J Eng Trends Technol 2013;4(5):1989-1992.
  117. Unni P, Staes C, Weeks H, Kramer H, Borbolla D, Slager S, et al. Why aren't they happy? An analysis of end-user satisfaction with Electronic health records. AMIA Annu Symp Proc 2016;2016:2026-2035 [FREE Full text] [Medline]

AI: artificial intelligence
CARS: context-aware recommendation system
CBRS: content-based recommender system
HIE: health information exchange
HINT: Human Intelligence Tasks
HIT: health IT
IRB: institutional review board
mHealth: mobile health

Edited by A Mavragani; submitted 07.10.22; peer-reviewed by K Taylor, F Medina; comments to author 23.11.22; revised version received 21.01.23; accepted 07.02.23; published 30.03.23


©Pamella Howell, Arun Aryal, Crystal Wu. Originally published in JMIR Research Protocols (, 30.03.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.