This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Online consumer health information addresses health problems, self-care, disease prevention, and health care services and is intended for the general public. Using this information, people can improve their knowledge, participation in health decision-making, and health. However, there are no comprehensive instruments to evaluate the value of health information from a consumer perspective.
We collaborated with information providers to develop and validate the Information Assessment Method for all (IAM4all) that can be used to collect feedback from information consumers (including patients), and to enable a two-way knowledge translation between information providers and consumers.
Content validation steps were followed to develop the IAM4all questionnaire. The first version was based on a theoretical framework from information science, a critical literature review and prior work. Then, 16 laypersons were interviewed on their experience with online health information and specifically their impression of the IAM4all questionnaire. Based on the summaries and interpretations of interviews, questionnaire items were revised, added, and excluded, thus creating the second version of the questionnaire. Subsequently, a panel of 12 information specialists and 8 health researchers participated in an online survey to rate each questionnaire item for relevance, clarity, representativeness, and specificity. The result of this expert panel contributed to the third, current, version of the questionnaire.
The current version of the IAM4all questionnaire is structured by four levels of outcomes of information seeking/receiving: situational relevance, cognitive impact, information use, and health benefits. Following the interviews and the expert panel survey, 9 questionnaire items were confirmed as relevant, clear, representative, and specific. To improve readability and accessibility for users with a lower level of literacy, 19 items were reworded and all inconsistencies in using a passive or active voice have been solved. One item was removed due to redundancy. The current version of the IAM4all questionnaire contains 28 items.
We developed and content validated the IAM4all in partnership with information providers, information specialists, researchers and representatives of information consumers. This questionnaire can be integrated within electronic knowledge resources to stimulate users’ reflection (eg, their intention to use information). We claim that any organization (eg, publishers, community organizations, or patient associations), can evaluate and improve their online consumer health information from a consumers’ perspective using this method.
The availability of the Internet and expectations of the public to become more involved in health care decisions have resulted in unprecedented demand of consumer health information, such as that provided by the National Library of Medicine (MedlinePlus). Consumer health information is about health problems, self-care, disease prevention, and health care services, and is intended for the general public, including patients and their relatives [
Using evidence-based health information may contribute to improving the health of populations, when this information use leads consumers to improve their knowledge and their participation in health decision-making [
In line with more consumer-centered health services, health information providers are producing more Web-based resources for the general public and for patients [
The Information Assessment Method (IAM) is based on information studies [
For clinicians, the ACA-LO model is operationalized using an IAM checklist that stimulates reflective learning within continuing education programs. With regard to Web-based health information, completing an IAM questionnaire will stimulate consumers’ reflection on the relevance of retrieved information for the situation at hand, the cognitive/affective impact of information (eg, learning something new), the intention to use information (eg, for doing something differently), and the expected health benefits from using it. In line with studies of the mere measurement effect [
For the present work, we conducted a critical review of the literature (all disciplines) on the four levels of outcomes of information with regards to online consumer health information. We examined 27 articles reporting literature reviews, original studies, and doctoral theses. For each level of outcome, this review suggested a preliminary list of types of outcomes. There were (1) seven types of consumer objectives for assessing the situational relevance of information, (2) five types of cognitive/affective impact of information on consumers, (3) five types of consumer use of information, and (4) eight types of health benefits for consumers. Findings from this review were grouped in accordance with three types of information-seeking: professionally-mediated access (through clinicians and librarians), direct access, and peer-mediated access (through relatives and social media) to Web-based consumer health information. These types are complementary (eg, consumers can search by themselves and with peers, then check with a health professional). Consideration of all three types of information-seeking was important to propose a comprehensive list of types of outcomes.
The literature shows that consumers are provided with Web-based health information by professionals for an educational purpose. The acquisition of information is mediated by a nurse, a doctor, an allied health professional, or a librarian [
Web-based health information can be found by consumers to address the following objectives: answer their own clinical questions (about themselves or relatives); update their knowledge; find more information to complement what was provided by clinicians or librarians; check this information; and satisfy their curiosity [
Peers usually provide Web-based health information based on personal opinions or experiences, and encouragement or emotional support (subjective); they can also provide information based on guidelines (normative), or research (objective), or a combination of subjective, normative, and objective information [
A systematic literature review shows that cognitive/affective impacts of Web-based consumer health information on patients are mainly learning (patients learn something new) and reassurance for decision-making (patients feel reassured for making decisions about their health and health care) [
The literature suggests cognitive/affective impacts are commitment to change in health care (eg, a commitment to change in prevention or treatment of illness), learning (better understanding of health or health care), and reassurance for decision-making [
An overview of the field by experts suggests cognitive/affective impacts of information from members of support groups are learning (better understanding of specific issues about health or health care), reassurance (gaining confidence), and confirmation that consumers are doing the right thing [
Web-based consumer health information is used by professionals to communicate with patients, and can be prescribed to patients (eg, in preparation for counseling and decision-making) [
Web-based consumer health information is used by people for themselves, their relatives, and friends. Direct access may lead to a consultation with a clinician to get an explanation or another opinion, and to modify health care when needed. Although rarely, using Web-based consumer health information may lead to disagreements between patients and clinicians. Clinicians may be defensive, or analyze the information with patients and guide them to high quality resources [
In online support groups, the flow of information involves multiple members and moves in multiple directions; it may contain inaccuracies, but most inaccuracies are rapidly corrected by informal leaders and group members. Information from these groups may be often used to comply with health care management [
The use of Web-based consumer health information may be associated with an increase of patients’ participation in health care, a gain in patient satisfaction, the prevention of health disease, and health improvement such as a reduction of depression and of the level of anxiety. Interactive interventions that tailor information to users’ needs can be more effective compared with noninteractive booklet or email interventions [
The use of Web-based consumer health information may affect the relationships between health professionals and patients, as it may (1) augment information provided by clinicians, (2) help patients to make informed health care choices, and (3) enable shared decision-making when there are different options for health care. It may transform the traditional clinician-patient relationship (from the clinician with expertise-based power to the clinician sharing information with patients), which can be challenging for some clinicians and patients. For example, consumer health information may create frustration when cyberchondriac patients bring in lengthy printouts of Web-based information about illnesses they might have [
There is no strong evidence regarding the positive or negative effects of Web-based consumer health information such as that from online support groups, discussion forums, and mailing lists. While online cancer support groups may help group members to cope more effectively with their disease, it has been suggested that people may rely on groups for too long, which can delay needed health care [
In line with usual content validation procedures for psychometric assessments [
Study participants were 16 laypersons (health information consumers), and 20 experts (co-authors) from McGill University, and 3 organizational partners. First, laypersons were recruited by co-authors from their adult acquaintances. They were fluent in English or French, and had no training or experience in health sciences or information studies.
Second, our partners were the Canadian Pharmacists Association (CPhA), the College of Family Physicians of Canada (CFPC), and the Centre for Literacy Québec (CLQ). We have been conducting participatory research since 2005 with the CPhA and the CFPC for the development of an IAM checklist for clinicians (nurses, pharmacists, physicians). The CPhA and the CFPC are nonprofit professional associations. The CPhA provides recommendations for clinicians (eg, patient self-care), and is starting to provide Web-based consumer health information. The CFPC produces Web-based patient education material for family physicians and patients. The CPhA and the CFPC are interested in assessing their products. For their part, the CLQ was interested in the development and assessment of effective interventions providing Web-based consumer health information. The 3 partners helped to formulate the objectives for this work. Our partnership followed the principles of participatory research with organizations, also called collaborative action research [
We created the first version of the IAM4all questionnaire using the four levels of outcomes of information (ACA-LO model): (level 1) situational relevance of information, (level 2) its cognitive impact, (level 3) information use, and (level 4) subsequent health benefits. For each level, items were derived from our critical review of the literature on consumer health information. Then, we edited and revised these items using previous work on the IAM. Finally, items of the IAM4all questionnaire were then reviewed by 8 researchers who belonged to the Information Technology Primary Care Research Group (ITPCRG) at McGill University.
More than 34 papers and book chapters are published about our previous work on the IAM checklist for clinicians (
The first version of the IAM4all questionnaire was tested using individual interviews with 16 laypersons (VG). The level of health literacy of participants was tested using the Short Test of Functional Health Literacy in Adults (S-TOFHLA) [
Interviews were both inductive and deductive. Participants were asked to describe the overall experience of consumers looking for health information, which enabled the emergence of new items. Then, they were asked for their feedback on the existing items. For each level of outcomes, the interviewer asked open questions (eg, “when you find health information on the Internet, how do you use it?”) followed by semistructured questions for each item (eg, “let me know if the item is clear; if it is not clear, tell me why”), and ended with an open question, “would you suggest any other items?”. All interviews were recorded and summarized for analysis. For each questionnaire item, the interviewees’ responses were interpreted (VG) as a confirmation (item is clear), or a revision (change wording, move to a different question, merge with another item), an addition (new item), or an exclusion. Additional comments related to each item were also recorded and interpreted. All interpretations were reviewed with the first author (PP), thus creating the second version of the IAM4all questionnaire.
In line with the usual definition of content validity for psychometric assessments [
All experts rated each item for relevance and clarity. To rate item relevance, experts were asked whether the item was appropriate to document the corresponding level of outcomes (eg, they were asked if the item “I used (will use) this information to do it differently” was appropriate to document the use of information). Four response options were available ranging from “I strongly agree” to “I strongly disagree”. To rate clarity (readability), four response options were offered ranging from “I strongly agree (accept this item without revision)” to “I strongly disagree (reject this item because of a major clarity issue)”.
In addition, all experts rated the representativeness and specificity of items for each level of outcomes in the ACA-LO model. To rate representativeness, participants were asked whether the items were representative of all aspects of the corresponding level of outcomes (eg, participants were asked if the six proposed items covered all aspects [or dimensions] of the level health benefits). To rate specificity, participants were asked whether the items for each level of outcomes were specific (ie, no item is redundant with another).
For all questions, response options ranged from “I strongly agree” to “I strongly disagree”. Participants were asked to justify their “disagree” and “strongly disagree” responses in a comment box. All experts answered all questions, and we calculated the proportion of expert agreement by combining the number of responses “I strongly agree” and “I agree” (n=20). We considered 66% or more as an acceptable proportion of expert agreement (at least 14 experts agreed). In addition, experts provided suggestions for addition, revision, or exclusion of items. The analysis of experts’ ratings and suggestions led us to confirm, revise, add, and remove items. In line with ecological validity, which is defined as the usability and adaptation of a tool from the users’ viewpoint [
For each level of outcomes, results are summarized in
Content validation of IAM4all items: level 1 - situational relevance.
Post interview items | Prepanel item development | Expert panel (N=20) | |||||||
Theoretical model | Critical review | Previous work on the IAM | Consultation with 8 researchers | Interviews with 16 laypersons | Item relevance | Clarity | Representa-tiveness | Specificity | |
Situational relevance | x | x | x | x | x |
|
|
85% | 50% |
1. To answer a question about my health |
|
x | x | x | x | 100% | 100% |
|
|
2. To address a question about the health of a relative or a friend |
|
x | x | x | x | 95% | 90% |
|
|
3. To educate myself about health |
|
x | x | x | x | 90% | 95% |
|
|
4. To satisfy my curiosity about health |
|
x | x | x | x | 80% | 90% |
|
|
5. To follow-up on the information given by a health professional |
|
x | x | x | x | 100% | 95% |
|
|
6. To prepare myself before talking to a health professional |
|
|
x | x | x | 100% | 100% |
|
|
7. To make a decision about seeing a health professional |
|
|
|
|
x | 100% | 95% |
|
|
8. To find options different from those given by a health professional |
|
|
|
|
x | 95% | 95% |
|
|
Content validation of IAM4all items: level 2 - cognitive impact.
Post interview items | Prepanel item development | Expert panel (N=20) | |||||||
Theoretical model | Critical review | Previous work on the IAM | Consultation with |
Interviews with 16 laypersons | Item relevance | Clarity | Representa-tiveness | Specificity | |
Cognitive impact | x | x | x | x | x |
|
|
75% | 55% |
9. I learned something new |
|
x | x | x | x | 100% | 100% |
|
|
10. This information confirmed I did (am doing) the right thing |
|
x | x | x | x | 100% | 85% |
|
|
11. I was reassured |
|
x | x | x | x | 80% | 80% |
|
|
12. I was reminded of something I already knew |
|
|
x | x | x | 100% | 95% |
|
|
13. I am motivated to learn more |
|
|
|
|
x | 95% | 90% |
|
|
14. I understood this information |
|
x | x | x | x | 90% | 95% |
|
|
15. I was dissatisfied |
|
|
x | x | x | 85% | 75% |
|
|
16. There is a problem with this information |
|
|
x | x | x | 85% | 85% |
|
|
17. This information could be harmful |
|
|
x |
|
x | 65% | 65% |
|
|
Content validation of IAM4all items: level 3 - information use.
|
Prepanel item development | Expert panel (N=20) | |||||||
Theoretical model | Critical review | Previous work on the IAM | Consultation with |
Interviews with 16 laypersons | Item relevance | Clarity | Representa-tiveness | Specificity | |
Information use | x | x | x | x | x |
|
|
55% | 85% |
18. I was doing or going to do something concerning my health, and I used (will use) this information to do it differently |
|
x | x | x | x | 95% | 80% |
|
|
19. I did not know what to do, and this information (did) will help to justify a decision concerning my health |
|
|
x | x | x | 90% | 80% |
|
|
20. I thought I knew what to do, and I used (will use) this information to be more certain about the management of my health (or health care) |
|
|
x | x | x | 85% | 75% |
|
|
21. This information (did) will help to better understand a particular issue related to my health |
|
|
x | x | x | 90% | 75% |
|
|
22. I used (will use) this information in a discussion with a health professional |
|
x | x | x | x | 100% | 95% |
|
|
Content validation of IAM4all items: Level 4 - expected benefits.
|
Prepanel item development | Expert panel (N=20) | |||||||
Theoretical model | Critical review | Previous work on the IAM | Consultation with |
Interviews with |
Item relevance | Clarity | Representa-tiveness | Specificity | |
Expected benefits | x | x | x | x | x |
|
|
60% | 60% |
23. This information decreased my worries about a health problem |
|
x | x | x | x | 90% | 85% |
|
|
24. This information increased my satisfaction with the care I receive |
|
x | x | x | x | 90% | 90% |
|
|
25. This information allowed (will allow) me to receive additional information from a health professional |
|
x | x | x | x | 75% | 70% |
|
|
26. Because of this information, I am (will be) more involved in decisions around my health |
|
x | x | x | x | 100% | 90% |
|
|
27. This information helped (will help) me to better handle a problem with my health |
|
x | x | x | x | 100% | 90% |
|
|
28. This information helped (will help) me to prevent a disease or the worsening of a disease |
|
x | x | x | x | 85% | 80% |
|
|
29. This information helped (will help) to improve my health |
|
x | x | x | x | 100% | 85% |
|
|
The ACA-LO model provided the four interdependent levels of outcomes of information targeted by the IAM4all questionnaire. Findings of our critical review suggested 25 items to operationalize these levels. There were (1) seven types of consumer objectives for the situational relevance of information, (2) five types of cognitive/affective impact of information on consumers, (3) five types of consumer use of information, and (4) eight types of health benefits for consumers. The consultation with 8 researchers yielded the first version of the questionnaire that included 26 items.
The analysis of interviews with 16 laypersons resulted in 18 items being confirmed, six revised, three excluded, and five added. All six revisions involved rewording of the questionnaire items. Interviewees suggested excluding the reason for searching item “to share information about health with members of a support group” and the information use item “I used (will use) this information to persuade a health professional to make a change in the management of my health” as they were not representative of the consumer experience. Interviewees brought to our attention two new reasons for searching: “To make a decision about seeing a health professional”, and “to find options different from those given by a health professional”. The three other added items reflected the interviewees experience with finding information that motivated them to learn more, using the found information to be more certain about the management of their health (or health care), and information being harmful. Six of 16 interviewees (37.5%) mentioned that retrieved information often increased their anxiety about health issues, and 2 (12.5%) said that information could be harmful:
I tend to over-worry, I tend not to look up for too much information and prefer to talk to the doctor about it. The more information I have the more I will worry.
Another interviewee shared this feeling by saying:
Seldom, information on the Internet decreases your worries, it adds to the anxiety.
To address this aspect using information, the item “this information made me more worried” was excluded as a cognition item and transformed as a new health benefit question: “Did something negative come out from using this information?” This analysis resulted in the second version of the questionnaire, which included 29 response items.
The analysis of experts’ ratings and suggestions resulted in nine items being confirmed, 19 revised (reworded), zero added, and one removed. Regarding the relevance and clarity of each item, expert agreement was satisfactory for 28 of 29 items (96.6%). Agreement was not satisfactory for the item “the information could be harmful”. Seven experts found this item problematic (eg, 2 suggested that the user would not be able to determine if the information is harmful). However, interviewees did not perceive this item to be problematic. In fact, 2 reported that this item is important. Thus, in line with ecological validity, the users’ viewpoint was prioritized, and the item was kept. Using experts’ comments, the wording was nevertheless improved to read “I think this information could be harmful”.
Representativeness and specificity of items per level of outcomes were discussed. For each level, expert comments were addressed as follows. Level 1: the item “to educate myself” was seen as redundant with the item “to satisfy my curiosity”, because 9 experts considered curiosity as the motivation that encompasses the will to educate oneself. As a result, the item “to educate myself” was excluded. Level 2: The item “I understood this information” was transformed into a response to a new screening question, “did you find the information you were looking for?”: “Yes but I did not understand it”. Level 3: The only changes consisted of rewording three items. Level 4: Five experts were concerned about the lack of negative types of outcomes. To address this issue, the question “did something negative come out from using this information?” was added with Yes/No response options and a comment box for those who answered yes.
Finally, the wording of 19 items and their readability for users with a low level of literacy have been improved, and inconsistencies in using a passive or active voice have been solved. For example, the item “to address a question about the health of a relative or a friend” was changed to “to answer a question about the health of someone else (eg, a family member)”. As another example, the concept of “health professional” has been explained using a parenthesis with examples (a nurse, a doctor, a pharmacist, or other clinician).
We developed and content-validated a questionnaire that can be used to assess four levels of outcomes associated with seeking and receiving Web-based consumer health information. For information seeking situations (pull: active acquisition of information), the current version of the IAM4all questionnaire is presented in
The proposed questionnaire flow is as follows: (1) the title of the information object under scrutiny is displayed (eg, Web page title), (2) the last four questions are disabled when the answer to question #2 is “yes but I did not understand it” or “no” (did you find the information you were looking for?), and (3) the last two questions are disabled when the answer to question #4 is “no” (did you or will you use this information for yourself?). Negative answers to question #2 refer to information that is not relevant in the situation at hand. In accordance to our model, there is no level 1 outcome, which precludes levels 2, 3, and 4 outcomes (cognitive impact, information use, and health benefits). Negative answers to question #4 refer to information that is relevant, but not used. In accordance to our model, there is no level 3 outcome, which precludes level 4 outcome (health benefits).
Based on our literature review and the interviews with laypersons the following subitems can be displayed when there are negative answers to question #2 (information not relevant): the information was difficult to find; the search engine was difficult to use; there was too much information to look over; I did not have enough time; my Internet connection was slow; there was no information in a language I am comfortable in; it was hard to tell what information to trust. In addition, question #6 allows consumers to share their concerns with information use (can something negative come out from using this information?) since the question #5 may be seen as too positive (expected health benefits).
We claim that any organization wishing to evaluate and improve their Web-based consumer health information from a consumer’s perspective can use this method. By way of illustration, we are planning a project in partnership with Défibami (an association for patients living with cardioverter-defibrillators). The association emails a monthly newsletter to their members, and is interested to know how this information is valuable for their users. Members will be asked to read the main article of the newsletter, and rate it using the push IAM4all questionnaire (version named IAM4patient, available on request) as a contribution to the improvement of the information provided by their association.
Information Assessment Method for all (IAM4all).
Another example is a two-way knowledge translation project that we are planning with the Chagnon foundation that produces “Naître et grandir” (N&G). The objective of N&G are to inform parents and increase social awareness of the importance of early childhood development in order to create conditions and environments that are conducive to educational success. N&G comes in a variety of formats, including a website, a magazine, and a communication initiative. N&G also circulates content and maintains contact with parents through a free, personalized weekly newsletter and social networks like Facebook and Twitter. Specifically, about 100,000 families receive the weekly newsletter that gives access to a highlighted Web page from where the public can browse other pages, thus creating a push-pull system and opportunities for serendipitous information retrieval. For part of this collaboration, a button will be available on each page offering access to the IAM4all questionnaire (version named IAM4parents, available on request). The reader’s trajectory will be tracked. Users accessing Web pages from the newsletter will be offered a push-pull version of the IAM4all questionnaire, for which the first question was revised (“why did you look for this information?” was replaced by “why did you read this information?”). Users accessing Web pages from searching the Internet will be offered the pull version of the IAM4all questionnaire.
This project has already contributed to the development of the IAM4all questionnaire in three ways: the production of a push-pull version, a cross-cultural adaptation in French, and an adaptation for a broader topic (the well-being of children and their parents). The word “health” was replaced by “child” for all questions and items, and the sentence “improve my health” was replaced by “improve the well-being or health of my child” (fifth question). Based on the N&G experience with a Web-based survey, no difficulties are expected with regard to the number of responses due to the sense of community among information users (13,000 responses were obtained from an N&G survey in the past). In sum, the IAM4all questionnaire is expected to: stimulate N&G users’ reflection, thus increase information use (mere measurement effect) [
The IAM4all is theory-driven, and unique in terms of comprehensiveness and content validity of items. However, it constitutes only a proposal, which needs to be further validated using statistical analysis of data collected from a larger sample (construct validity). Our content validation study faces one main limitation. The participants constituted a convenience sample of Web-based health information consumers, and were not representative of the general public in terms of demographic, educational, and sociocultural characteristics (eg, they had an adequate functional health literacy level). The implementation of the IAM4all with Défibami and N&G will allow us to conduct focus groups with diverse health information consumers for further content ecological validation study [
By active involvement throughout the development of this innovative information assessment method, our partners ensured that it took their needs into account, which may improve its use. This method can respond to the needs of information providers because it can document the consumer information interaction, and enable a two-way knowledge translation between information providers and information end-users. The former updates and delivers the best available information. The latter assesses this information, and can submit constructive feedback. In turn, providers may use this feedback to improve their services, which is beneficial for all parties. Such two-way knowledge translation process constitutes a participatory production of knowledge with lay people (eg, N&G with IAM4all).
To our knowledge, the IAM4all is a unique and original method to assess how Web-based consumer health information is valuable from the consumers’ viewpoint, specifically the use of information and its expected health benefits. The integration of the IAM4all within electronic knowledge resources can help information providers to evaluate and improve their Web-based consumer health information services. The IAM4all can stimulate reflection and feedback about health information; this feedback can then be used to improve information services.
The Information Assessment Method - published papers.
Acquisition Cognition Application - Level of Outcomes model
College of Family Physicians of Canada
Centre for Literacy Québec
Canadian Pharmacists Association
Information Assessment Method
Information Assessment Method for all
Information Technology Primary Care Research Group
Naître et grandir
Short Test of Functional Health Literacy in Adults
Pierre Pluye holds a research salary award of the ‘Fonds de recherche du Québec – Santé’ (FRQ-S). Gillian Bartlett received funding from the Canadian Institutes of Health Research for a Primary and Community-Based Health Care Catalyst Grant titled “Empowering Patients: Making the personal health record accessible and relevant for primary care.”
None declared.