The availability of the Internet and expectations of the public to become more involved in health care decisions have resulted in unprecedented demand of consumer health information, such as that provided by the National Library of Medicine (MedlinePlus). Consumer health information is about health problems, self-care, disease prevention, and health care services, and is intended for the general public, including patients and their relatives [1,2]. Accessing Web-based health information will increase as Internet access is almost ubiquitous in many countries. For example, a national survey of a representative population sample reveals that accessing Web-based consumer health information in the United States has doubled since 2001, while access from other sources has decreased, and the Internet is the most frequently used platform for accessing consumer health information [3]. In 2012, 35% of US adults reported using health information found online “to try to figure out the medical condition they or someone else might have” [4].

Using evidence-based health information may contribute to improving the health of populations, when this information use leads consumers to improve their knowledge and their participation in health decision-making [5-8]. Another potential impact of better-informed consumers is on the use of health services. For example, in one study, better-informed consumers were less likely to use emergency room services [9]. Such outcomes are particularly important in primary health care where patients, their friends, relatives, and home care aids, play an active role in health decision-making and frequently search for information [10]. For example, among adult Internet users in Canada, 54% have looked for health information online [11]. The aging of society and the prevalence of chronic diseases will continue to increase [12], which will, in turn, drive an increase in the use of Web-based consumer health information.

In line with more consumer-centered health services, health information providers are producing more Web-based resources for the general public and for patients [13]. However, there are no comprehensive instruments to evaluate whether Web-based consumer health information is valuable from a consumers’ viewpoint. While existing tools typically focus on experts’ evaluation of the quality of information sources [14], new tools are needed to better understand the role of the Internet in health, and specifically to evaluate and improve the impact of information on patients and their families [15]. Consequently, our objective is to develop and validate a questionnaire that can be used by the general public, including patients, to assess outcomes associated with seeking and receiving Web-based consumer health information. To do this, we engaged organizational partners, such as consumer health information providers who expressed a need for a clear and user-friendly, comprehensive assessment method. This method will be used to stimulate reflection on information, enabling a two-way knowledge translation between information providers and consumers, also referred to as participatory production of knowledge with lay people. In this paper, we report on the development and validation of the Information Assessment Method for patients and consumers (IAM4all). We will present the theoretical model, the literature review, the three-step content validation process, the results, and the discussion, including examples of application and further validation study.

In line with usual content validation procedures for psychometric assessments [47,48], we followed three steps that involved researchers, laypersons, and a panel of experts, respectively. These three steps are described in the “Data Collection and Analysis” section below. In step 1, we created a first version of the IAM4all questionnaire using the theoretical model, and items from the critical literature review and previous work [17]. In step 2, the second version was based on interviews with 16 laypersons. For step 3, experts contributed to produce the third version by rating all items for relevance, clarity, representativeness, and specificity. Ethical approval was obtained from the McGill University institutional review board.

Study participants were 16 laypersons (health information consumers), and 20 experts (co-authors) from McGill University, and 3 organizational partners. First, laypersons were recruited by co-authors from their adult acquaintances. They were fluent in English or French, and had no training or experience in health sciences or information studies.

Second, our partners were the Canadian Pharmacists Association (CPhA), the College of Family Physicians of Canada (CFPC), and the Centre for Literacy Québec (CLQ). We have been conducting participatory research since 2005 with the CPhA and the CFPC for the development of an IAM checklist for clinicians (nurses, pharmacists, physicians). The CPhA and the CFPC are nonprofit professional associations. The CPhA provides recommendations for clinicians (eg, patient self-care), and is starting to provide Web-based consumer health information. The CFPC produces Web-based patient education material for family physicians and patients. The CPhA and the CFPC are interested in assessing their products. For their part, the CLQ was interested in the development and assessment of effective interventions providing Web-based consumer health information. The 3 partners helped to formulate the objectives for this work. Our partnership followed the principles of participatory research with organizations, also called collaborative action research [49,50]. This approach leads to improve knowledge and practice (organizational learning), to engage organization members (as reflective practitioners), and to involve organization partners in all research aspects (integrated knowledge translation).

For each level of outcomes, results are summarized in Tables 1-4. Each table reports all steps of the content validation procedure. All interviewees reported frequently searching for health information on the Internet (16 of 16). The majority of interviewees were women (11 of 16). In terms of health literacy, all 16 interviewees had an Adequate Functional Health Literacy level [51] (ie, can read and interpret most health texts).

The ACA-LO model provided the four interdependent levels of outcomes of information targeted by the IAM4all questionnaire. Findings of our critical review suggested 25 items to operationalize these levels. There were (1) seven types of consumer objectives for the situational relevance of information, (2) five types of cognitive/affective impact of information on consumers, (3) five types of consumer use of information, and (4) eight types of health benefits for consumers. The consultation with 8 researchers yielded the first version of the questionnaire that included 26 items.

The analysis of interviews with 16 laypersons resulted in 18 items being confirmed, six revised, three excluded, and five added. All six revisions involved rewording of the questionnaire items. Interviewees suggested excluding the reason for searching item “to share information about health with members of a support group” and the information use item “I used (will use) this information to persuade a health professional to make a change in the management of my health” as they were not representative of the consumer experience. Interviewees brought to our attention two new reasons for searching: “To make a decision about seeing a health professional”, and “to find options different from those given by a health professional”. The three other added items reflected the interviewees experience with finding information that motivated them to learn more, using the found information to be more certain about the management of their health (or health care), and information being harmful. Six of 16 interviewees (37.5%) mentioned that retrieved information often increased their anxiety about health issues, and 2 (12.5%) said that information could be harmful:

Another interviewee shared this feeling by saying:

To address this aspect using information, the item “this information made me more worried” was excluded as a cognition item and transformed as a new health benefit question: “Did something negative come out from using this information?” This analysis resulted in the second version of the questionnaire, which included 29 response items.

The analysis of experts’ ratings and suggestions resulted in nine items being confirmed, 19 revised (reworded), zero added, and one removed. Regarding the relevance and clarity of each item, expert agreement was satisfactory for 28 of 29 items (96.6%). Agreement was not satisfactory for the item “the information could be harmful”. Seven experts found this item problematic (eg, 2 suggested that the user would not be able to determine if the information is harmful). However, interviewees did not perceive this item to be problematic. In fact, 2 reported that this item is important. Thus, in line with ecological validity, the users’ viewpoint was prioritized, and the item was kept. Using experts’ comments, the wording was nevertheless improved to read “I think this information could be harmful”.

Representativeness and specificity of items per level of outcomes were discussed. For each level, expert comments were addressed as follows. Level 1: the item “to educate myself” was seen as redundant with the item “to satisfy my curiosity”, because 9 experts considered curiosity as the motivation that encompasses the will to educate oneself. As a result, the item “to educate myself” was excluded. Level 2: The item “I understood this information” was transformed into a response to a new screening question, “did you find the information you were looking for?”: “Yes but I did not understand it”. Level 3: The only changes consisted of rewording three items. Level 4: Five experts were concerned about the lack of negative types of outcomes. To address this issue, the question “did something negative come out from using this information?” was added with Yes/No response options and a comment box for those who answered yes.

Finally, the wording of 19 items and their readability for users with a low level of literacy have been improved, and inconsistencies in using a passive or active voice have been solved. For example, the item “to address a question about the health of a relative or a friend” was changed to “to answer a question about the health of someone else (eg, a family member)”. As another example, the concept of “health professional” has been explained using a parenthesis with examples (a nurse, a doctor, a pharmacist, or other clinician).

We developed and content-validated a questionnaire that can be used to assess four levels of outcomes associated with seeking and receiving Web-based consumer health information. For information seeking situations (pull: active acquisition of information), the current version of the IAM4all questionnaire is presented in Figure 1. The verb tense of questions and items must be adapted to the moment of the evaluation. For example, the future tense in the fourth question (will you use this information?) is appropriate for quasimomentary assessment (intention to use), while the past tense (did you use this information?) is appropriate for follow-up evaluation (self-reported use). In addition, we adapted this questionnaire for situations where information is received (push: passive acquisition of information) such as email alerts (push questionnaire available on request). To our knowledge, this validation study is the first to propose such practical method to evaluate how Web-based consumer health information objects are valuable from a consumer perspective, whatever the type of access to information (professionally-mediated, direct, peer-mediated), for pull and push situations, and for quasimomentary and follow-up evaluations.

The proposed questionnaire flow is as follows: (1) the title of the information object under scrutiny is displayed (eg, Web page title), (2) the last four questions are disabled when the answer to question #2 is “yes but I did not understand it” or “no” (did you find the information you were looking for?), and (3) the last two questions are disabled when the answer to question #4 is “no” (did you or will you use this information for yourself?). Negative answers to question #2 refer to information that is not relevant in the situation at hand. In accordance to our model, there is no level 1 outcome, which precludes levels 2, 3, and 4 outcomes (cognitive impact, information use, and health benefits). Negative answers to question #4 refer to information that is relevant, but not used. In accordance to our model, there is no level 3 outcome, which precludes level 4 outcome (health benefits).

Based on our literature review and the interviews with laypersons the following subitems can be displayed when there are negative answers to question #2 (information not relevant): the information was difficult to find; the search engine was difficult to use; there was too much information to look over; I did not have enough time; my Internet connection was slow; there was no information in a language I am comfortable in; it was hard to tell what information to trust. In addition, question #6 allows consumers to share their concerns with information use (can something negative come out from using this information?) since the question #5 may be seen as too positive (expected health benefits).

We claim that any organization wishing to evaluate and improve their Web-based consumer health information from a consumer’s perspective can use this method. By way of illustration, we are planning a project in partnership with Défibami (an association for patients living with cardioverter-defibrillators). The association emails a monthly newsletter to their members, and is interested to know how this information is valuable for their users. Members will be asked to read the main article of the newsletter, and rate it using the push IAM4all questionnaire (version named IAM4patient, available on request) as a contribution to the improvement of the information provided by their association.

Another example is a two-way knowledge translation project that we are planning with the Chagnon foundation that produces “Naître et grandir” (N&G). The objective of N&G are to inform parents and increase social awareness of the importance of early childhood development in order to create conditions and environments that are conducive to educational success. N&G comes in a variety of formats, including a website, a magazine, and a communication initiative. N&G also circulates content and maintains contact with parents through a free, personalized weekly newsletter and social networks like Facebook and Twitter. Specifically, about 100,000 families receive the weekly newsletter that gives access to a highlighted Web page from where the public can browse other pages, thus creating a push-pull system and opportunities for serendipitous information retrieval. For part of this collaboration, a button will be available on each page offering access to the IAM4all questionnaire (version named IAM4parents, available on request). The reader’s trajectory will be tracked. Users accessing Web pages from the newsletter will be offered a push-pull version of the IAM4all questionnaire, for which the first question was revised (“why did you look for this information?” was replaced by “why did you read this information?”). Users accessing Web pages from searching the Internet will be offered the pull version of the IAM4all questionnaire.

This project has already contributed to the development of the IAM4all questionnaire in three ways: the production of a push-pull version, a cross-cultural adaptation in French, and an adaptation for a broader topic (the well-being of children and their parents). The word “health” was replaced by “child” for all questions and items, and the sentence “improve my health” was replaced by “improve the well-being or health of my child” (fifth question). Based on the N&G experience with a Web-based survey, no difficulties are expected with regard to the number of responses due to the sense of community among information users (13,000 responses were obtained from an N&G survey in the past). In sum, the IAM4all questionnaire is expected to: stimulate N&G users’ reflection, thus increase information use (mere measurement effect) [19], and continuously collect constructive feedback from the N&G users, which can be used by N&G providers to improve their information services.

The IAM4all is theory-driven, and unique in terms of comprehensiveness and content validity of items. However, it constitutes only a proposal, which needs to be further validated using statistical analysis of data collected from a larger sample (construct validity). Our content validation study faces one main limitation. The participants constituted a convenience sample of Web-based health information consumers, and were not representative of the general public in terms of demographic, educational, and sociocultural characteristics (eg, they had an adequate functional health literacy level). The implementation of the IAM4all with Défibami and N&G will allow us to conduct focus groups with diverse health information consumers for further content ecological validation study [54], and collect hundreds of completed IAM4all questionnaires for further construct validation study using classical test theory and/or item response theory.

By active involvement throughout the development of this innovative information assessment method, our partners ensured that it took their needs into account, which may improve its use. This method can respond to the needs of information providers because it can document the consumer information interaction, and enable a two-way knowledge translation between information providers and information end-users. The former updates and delivers the best available information. The latter assesses this information, and can submit constructive feedback. In turn, providers may use this feedback to improve their services, which is beneficial for all parties. Such two-way knowledge translation process constitutes a participatory production of knowledge with lay people (eg, N&G with IAM4all).

To our knowledge, the IAM4all is a unique and original method to assess how Web-based consumer health information is valuable from the consumers’ viewpoint, specifically the use of information and its expected health benefits. The integration of the IAM4all within electronic knowledge resources can help information providers to evaluate and improve their Web-based consumer health information services. The IAM4all can stimulate reflection and feedback about health information; this feedback can then be used to improve information services.