RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.06.16 in Vol 5, No 2 (2016): Apr-Jun

This paper is in the following e-collection/theme issue:

    Original Paper

    Consumers’ Patient Portal Preferences and Health Literacy: A Survey Using Crowdsourcing

    1Department of Bioengineering, University of California, Los Angeles, Los Angeles, CA, United States

    2Department of Biology, Florida Agricultural and Mechanical University, Tallahassee, FL, United States

    3Medical Imaging Informatics, Department of Radiological Sciences, University of California, Los Angeles, Los Angeles, CA, United States

    Corresponding Author:

    Mary Zide, MLIS, PhD

    Department of Bioengineering

    University of California, Los Angeles

    924 Westwood Blvd

    Suite 420

    Los Angeles, CA, 90024

    United States

    Phone: 1 5102075012

    Fax:1 3107943544

    Email:


    ABSTRACT

    Background: eHealth apps have the potential to meet the information needs of patient populations and improve health literacy rates. However, little work has been done to document perceived usability of portals and health literacy of specific topics.

    Objective: Our aim was to establish a baseline of lung cancer health literacy and perceived portal usability.

    Methods: A survey based on previously validated instruments was used to assess a baseline of patient portal usability and health literacy within the domain of lung cancer. The survey was distributed via Amazon’s Mechanical Turk to 500 participants.

    Results: Our results show differences in preferences and literacy by demographic cohorts, with a trend of chronically ill patients having a more positive reception of patient portals and a higher health literacy rate of lung cancer knowledge (P<.05).

    Conclusions: This article provides a baseline of usability needs and health literacy that suggests that chronically ill patients have a greater preference for patient portals and higher level of health literacy within the domain of lung cancer.

    JMIR Res Protoc 2016;5(2):e104

    doi:10.2196/resprot.5122

    KEYWORDS

    Crowdfunding campaign to support this specific research

    We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

    keyboard with crowdfunding key instead of enter key

    Suggested contribution levels: $20/$50/$100



    Introduction

    Life expectancy has nearly doubled in the last century [1], due in part to preventive care, screening protocols, and the conversion of previously lethal diseases into manageable, chronic conditions through medical advances. Recently, lung cancer screening with computed tomography (CT) has been shown to significantly reduce lung cancer mortality and has resulted in paradigm shifts such that screening is now a covered medical procedure by both third party payers and Medicare [2], providing the opportunity for more patients to be screened. However, lung cancer is still the single greatest cause of cancer-related death. Accurate and timely patient education is considered to be one of the critical elements underlying improvements in public health, to better inform patients of the underlying causes of their conditions, the measures they can take to mitigate risk, and the basis, and balance of benefits to risks, of certain interventions.

    However, low levels of health literacy are seen as a hurdle to accessing health information [3]. The US Department of Health and Human Services defines health literacy as patients’ ability to acquire, read, and understand health information in order to make health decisions appropriate to their situation [4]. The National Center for Education Statistics maintains that health literacy is crucial for all adults to understand and improve their health, as they encounter health information in a variety of digital and hardcopy formats (eg, websites, blogs, federated search engines, magazine articles, pamphlets, prescription directions) [5].

    Further study of patients’ health literacy in the subject of lung cancer is necessary to inform the design of patient education tools. Lung cancer screening protocols have recently been revised due to the findings in [2], demonstrating the efficacy of annual CT scans for high-risk individuals. Given the major policy change in lung cancer screening, it is expected that moderate patient education will be required to better understand the etiologies of lung cancer and basis for screening. eHealth apps are a potentially potent medium for the task of providing educational materials.

    The term eHealth can be broadly described as the use of information technology in health care, examples of which include patient portals, personal fitness apps that run on mobile phones or personal computers, and digital consumer health educational guidelines. eHealth tools are increasingly available; however, availability does not ensure greater health literacy or increased use of these tools. The term usability refers to the ease of use and learnability of a tool, the measure of which can serve as one gauge of the effectiveness of a tool in helping the user reach their objectives [6]. Usability is necessary to ensure use of and application of the information within eHealth technologies [7,8]. Contributing to perceived usability is perceived feasibility. Feasibility, which can be defined as the effectiveness of an intervention, can be further divided into eight areas of focus, including acceptability, for example, how users react to an intervention, and demand, for example, an estimated need of a tool [8].

    While no design will be able to anticipate all information needs, the content of a patient portal has the potential to influence perceived usability and satisfaction, as well as improve health literacy and outcomes [9,10]. Generally, a patient portal is a secure website that provides individuals with access to their personal medical information [11]. Although there has not been lengthy discussion on the content of portals in particular, the content of personal health records (PHRs) has received more attention. Basic consensus has been reached on certain data points that should be included: problem lists, procedures, lab tests, diagnoses, and notes [7,12-15]. These data points would likely prove beneficial in a portal as well. Portals also have a range of functions not available in PHRs, such as the ability to email a physician, renew a prescription, view clinical reports, and make appointments. However, patient information needs may differ by diagnosis, and in order to improve health literacy and outcomes, some diagnoses may require more specific additional information, educational content, and functions [7,16]. For instance, cancer patients have been shown to want access to targeted information about their particular diagnosis (benign vs cancerous), treatments (chemotherapy, radiation, surgery), and prognosis (quality of life, 5-year survival) [17]. However, the specific needs of cancer patients by diagnosis are not well known. Identifying these needs, as well as their literacy levels on specific subjects, would prove helpful in the design of tools meant to serve specific populations, such as lung cancer. Before this can be done, it is necessary to establish a baseline of a general population’s information needs.

    Additionally, as lung cancer is the leading cause of cancer death and the majority of lung cancer cases are due to tobacco smoke, it is important that the general population be informed regarding lung cancer and the means through which risk can be mitigated (smoking cessation) [18]. While campaigns to inform the general population of this health issue have been wide spread, it is important to document how effective these campaigns have been in providing health consumers with knowledge, especially when considering recent reforms to screening protocol. Assessment of lung cancer literacy can help to inform portal module design aimed at improving literacy rates.

    As an initial starting point, we were interested in two areas of study: (1) understanding the perceived usability and, in turn, feasibility of information content of patient portals of a convenience cohort taken from the general public, and (2) the demand for lung cancer‒specific health literacy of a convenience cohort taken from the general public. Our goal was to establish a baseline to understand how perspectives and literacy vary by basic patient (consumer) demographics.


    Methods

    To assess baseline health information needs and preferences, as well as lung cancer knowledge, a survey was developed based on previously validated surveys [19,20], which were designed for the purposes of documenting patients’ health portal preferences and lung cancer knowledge. While the questions in our survey are taken from these validated tools, not all questions from the original surveys were used. Instead, in an effort to keep our survey concise and focused, we chose those questions that most closely focused on our concerned topics, perceived usability and feasibility of patient portals, and lung cancer knowledge. We chose to rely on prior validated surveys, as developing and validating a survey would require considerable additional research beyond our current scope. Our instrument consisted of three modules (see Multimedia Appendix 1). The first module captured information needs and preferences regarding patient portals posed as statements that were rated by participants on a 7-point Likert scale ranging from 1 (completely disagree) to 7 (completely agree). The second module consisted of factual statements about lung cancer and computed tomography with multiple choice answers, of which only one answer was correct. The third module contained demographic questions, based primarily on the US census [21]. Two free-text questions allowed participants to describe concerns and the effects of using a portal. This study was certified exempt by an internal review board committee at University of California, Los Angeles (UCLA).

    Recruitment

    The survey was distributed via the website MTurk, a crowdsourcing Internet site devoted to human intelligence tasks [22] that has been used in prior medical informatics studies [23-26] and in the evaluation of consumer health tools [27,28]. The MTurk site was used as it has been demonstrated by those studies as a reliable method to survey a convenience sample and because it provided access to individuals across the country, as opposed to one geographical area. The survey was made available only to those MTurk participants within the United States. The survey was posted on the MTurk website for 3 weeks in May 2015. Participants were invited to complete the survey with the guarantee of compensation of US $1 per completed survey; each participant completed only one survey. The first 500 individuals to fully complete the survey were chosen as a convenience sample for purposes of analysis. We sought 250 participants age 41 and older, and 250 participants age 40 and younger. To do this, the survey was distributed twice, once as a survey available to those who were age 40 and younger, and once as a survey to those who were age 41 and older. Participants were asked to affirm their age cohort.

    Statistical Analysis

    Power and Statistical Tests

    A power analysis indicated that in order to demonstrate a 0.95 power with alpha=.025, we would require 324 participants. Survey results were analyzed using SPSS version 20 software. Statistically significant results are those with a P value ≤.05. Independent t-tests and one-way analyses of variance (ANOVAs) were used in univariate analyses to determine differences in information needs and preferences based on mean values of demographic variables. Chi-square was used in univariate analysis to identify differences in health literacy based on mean demographic variables. To determine if combinations of demographic variables can predict dependent variables, stepwise logistic regression was done using all significant variables from univariate analysis, with each model using the independent variable(s) that had significant univariate results for a particular question (P ≤.05). Logistic regression was used in order to make the data as parsimonious as possible, as the number of cases per group were limited. Alongside P values, log-odds, and prediction probabilities, Nagelkerke R2values were also reported. The Nagelkerke R2statistic is used to demonstrate how useful the independent variables are in predicting the dependent variable [29].

    Variables

    All variables can be seen in Multimedia Appendix 2. Due to sparse data, the demographic variables sex and race were collapsed into dichotomous variables (seen Multimedia Appendix 1). The variable sex included one response listed as “other.” This single response was eliminated from univariate analysis of portal preferences based on sex. For logistic regression, the 7-point Likert scales used to record patient portal responses were also dichotomized as follows: Agree (Scores 5-7) and Disagree (Score 1-4). For regression analysis, a score of 4 on Likert scale responses was included within the Disagree category, as the purpose of our analysis was to compare those who agreed to others. For lung cancer knowledge variables, binary values were coded as “0” for incorrect and “1” for correct.


    Results

    Only surveys with no missing data elements were analyzed. Participants who returned surveys with missing data were contacted via email and invited to supply the missing data points. There was no consistent pattern to the types of data elements not completed. The surveys of participants who supplied the missing data were included in analysis. In total, out of the 500 surveys issued, 473 complete surveys were collected.

    User Demographics

    The majority of participants were white (389/473, 82.2%) (see Table 1). Although accounting for fewer respondents, Asian (26/474, 5.4%) and black (25/473, 5.2%) participants were the second largest group and were roughly equivalent. The majority of participants had some college education, with very few (27/473, 5.7%) participants listing high school as their highest level of educational attainment. Roughly half of respondents reported an annual income of US $35,000 or less. Participants tended to spend over 11 hours a week online (349/473, 73.7%), but most had used a portal 10 times or less (216/473, 45.6%) or had never used a patient portal (174/473, 36.7%). This rate of portal use suggests that responses to the survey are both a measure of needs for those with experience using a portal, and expectations for those who have not used a portal before.

    Table 1. Demographic results.
    View this table

    Evaluation Outcomes

    Influence of Participant Characteristics on Patient Portal Questions

    Patient response frequencies to all survey questions can be seen in Multimedia Appendix 3 , and results to all univariate analyses can be seen in Multimedia Appendix 4. Demographic variables that significantly influenced participant perceptions of patient portal usability and feasibility are shown in Table 2. For all significant results, females rated statements about portal perceptions with higher positive responses. For example, the average Likert response to the question “Portals are not difficult to use” was 4.97 for males and 5.32 for females (P=.009). Females tended to have higher, more positive average Likert scores across all questions relating to portal use.

    Respondents reporting a chronic illness also tended to have more positive views of portals, with higher average ratings than those without chronic illness. The average response was significantly higher for chronically ill participants for all four questions relating to portals (see Table 2). This trend was also seen across all questions relating to portal preferences.

    Differences in responses significantly varied based on the number of times participants had used a patient portal. However, there was no consistent trend seen across answers. Additionally, while not statistically significant, 33% of participants reported concern about “unauthorized access” to their patient portal.

    Table 2. Statistically significant results of univariate analysis of patient portal preferences.
    View this table
    Influence of Patient Characteristics on Lung Cancer Screening Knowledge

    Questions pertaining to lung cancer and chest CT had multiple choice answers, of which one answer was correct. The chronic illness predictor was most frequently associated with correct responses (see Table 3). Those reporting a chronic illness had a higher rate of correct answers for three of the four significant results. However, those not reporting chronic illness performed better on the two questions with the outcome variable “Someone who has quit smoking…” and “Lung cancer is one of the most common cancers.”

    Table 3. Statistically significant results of univariate analysis of demographic variables on lung cancer knowledge.
    View this table

    Other predictors with significant test results included time spent online, income, education, smoking habit, sex, and portal use. Time spent online had the second highest number of significant tests with three results. However, there was no consistent pattern observed within the rates of answers, with those who spent more time sometimes outperforming and sometimes underperforming those who spent less time. For the income predictor, correct answers to the question “Coughing up blood is a frequent sign of lung cancer” increased as income increased, until the level of US $76,000 or more. Those who made US $76,000 or more a year had a lower rate of correct answers than those who made less. For the smoking habit predictor, those who smoked had a slight but significantly lower rate of correct answers (78.4% versus 78.7%) for the question “Lung cancer is one of the most common cancers.” Men had a significantly higher rate of correct answers for the question “Coughing up blood is a frequent sign of lung cancer” than women (75.4% vs 73.7%). As portal use increased, correct answers to the question “Lung cancer is infectious” decreased. There was no pattern observed in response rates for the question “In the past, before the CT scan was introduced…” when stratified by the education predictor, nor for the question “A change of cough pattern is a frequent sign of lung cancer” response rates when stratified by the income predictor.

    Logistic Regression

    Stepwise logistic regression was performed for demographic variables that had statistically significant relationships as seen in Tables 2 and 3. Statistically significant results for logistic regression are seen in Table 4. Increased portal use was positively associated with agreeing with the statement “It should be easy to become skillful at using a portal,” meaning participants who used a portal more than 10 times were more likely to agree with the statement. However, prediction success was weak with 32.6% of outcomes correctly predicted. The chronic illness predictor was positively associated with having the correct answer for “A change of cough pattern is a frequent sign of lung cancer,” indicating that those with a chronic illness were more likely to choose the correct answer for the question. However, prediction was not improved in comparison to the constant model, remaining at 52.4%.

    Table 4. Statistically significant results for logistic regression analysis.
    View this table

    Discussion

    Principal Results and Comparisons With Prior Work

    In this study, a convenience cohort of the general public was asked to rate their perceptions of usability and feasibility of patient portals and knowledge of lung cancer, in order to demonstrate the demand for lung cancer knowledge and status of patient portal usability. Overall, we observed that respondents reporting chronic illness tended to have more positive opinions of patient portals and to perform better on lung cancer knowledge questions than those without a chronic condition. Chronically ill patients having a more positive opinion of eHealth is a finding that has also been observed [30,31]. Similarly, a review of literature on portals found that participant interest in patient portals varied by health status [32].

    Interestingly, some studies have found that chronically ill patients may have lower health literacy [33,34]. In contrast, a study found that those with a chronic illness reported higher rates of literacy regarding test results [35]. In our work, it may be that the higher performance we observed relates to higher levels of educational attainment, as those respondents reporting chronic illness in our survey more frequently had Associate, Bachelor’s, and Master’s degrees than those reporting no chronic illness (14.6% vs 12.9%, 39.8% vs 36.8%, and 9.7% vs 8.8% respectively).

    Having used a portal was associated with significant difference in opinions of patient portals. However, there was no consistent trend observed, with some statements being rated in higher agreement by those who had used portals more and other statements being rated higher by those who had used them less. It is also worth noting that for those who had never used a portal, these statements are measuring expectations of portals, while with those who have used portals, these statements are measuring experience. These results suggest that opinions about the usability of portals do not demonstrate a clear trend whether being drawn based on expectations or experience.

    We also found that women reported more positive ratings of portals. While use does not equal preference, positive ratings may be influenced by the higher use of portals we observed in this survey (19.7% of women had used a portal over ten times, compared to 14.3% of men), which is consistent with higher eHealth resource use observed in women [36-38]. We also documented a statistically significant difference in correct responses for lung cancer knowledge stratified by the amount of time spent online. Others have found similar relationships between Internet use and health literacy [34], but our results showed no consistent trend and are inconclusive.

    Although two logistic regression models had significant P value results, neither model had strong prediction success. While the low P values (<.05) indicate that the findings are unlikely due to chance, the prediction values, as well as the Nagelkerke R2values, indicate that these models will not perform prediction well. Further research is required to determine predictors that would predict answer cohorts to these questions successfully.

    Finally, 33% of participants noted that they were “concerned about unauthorized access,” which was the most commonly chosen option (seem Multimedia Appendix 4 , Question 26). In other work, security was a concern for two thirds of health information consumers, although users of PHRs had less observed concern [39]. The overall percentage of participants concerned with security in this study was less than in [39]. However, a similar trend was observed with 32.5% of those who had used a portal more than 10 times were concerned with unauthorized access while 33.1% of those who had used a portal 10 times or less were concerned. Similarly, security has also been a concern [40]. This common theme suggests that eHealth users may associate eHealth tools with a lack of security. This concern has the potential to impact portal use, as it would likely limit satisfaction and perceived usability and, thus, feasibility.

    Limitations

    There are several limitations to this study that may have influenced our results. Most importantly, we used a convenience sample of the first 500 respondents to a survey posted on MTurk. As such, the results derive from Internet users; those with less Internet experience may well have different views regarding patient portals and lung cancer health literacy than those documented here. Our respondents also had higher levels of education than seen in the US population [41], which may have influenced health literacy. Moreover, although we specifically asked respondents not to look up answers to lung cancer questions, we have no guarantee that responses were not informed by additional online queries.

    An additional limitation is the potential bias that may be introduced by the use of a digital survey format. Others have found that patients who have used eHealth technology have more positive opinions of it than those who do not [42], suggesting that experience with technology can cause one to regard it more favorably. Here, completing a digital survey on eHealth may bias respondents to rate statements about patient portals more highly.

    Conclusion

    This work documents a baseline for consumer information needs and health literacy within the domain of lung cancer. Our results suggest that women and those with chronic illness had more positive views of patient portals and that chronically ill patients had higher health literacy in lung cancer. Although chronically ill patients are a likely user group for patient portals, non‒chronically ill patients can also benefit from opportunities to manage and learn about their health as well as mitigate risk via portals. Given this, the results suggest a need for further lung cancer education with opportunities to use patient portals to better educate individuals who do not have a chronic illness about lung cancer. Further study is also needed in order for portals to better address the information needs of patients who are not chronically ill in order to improve perceived usability and feasibility.

    This baseline can be used in future comparison studies as well as to inform portal design to improve usability and raise lung cancer health literacy with educational modules. Our future work includes designing a patient portal influenced by these results and then surveying a patient population at UCLA who will have access to the portal, to compare those survey results to this baseline, in order to identify any difference between them.

    Acknowledgments

    This work is supported by NIH/NCI Grant R01 LM011333, NIH/NCI Grant 5T32 EB01664003, and UCLA HBCU Grant 285247.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    PDF File (Adobe PDF File), 43KB

    Multimedia Appendix 2

    PDF File (Adobe PDF File), 64KB

    Multimedia Appendix 3

    PDF File (Adobe PDF File), 42KB

    Multimedia Appendix 4

    PDF File (Adobe PDF File), 68KB

    References

    1. Burton DM. National Institute on Aging.: US Department of Health and Human Services; 2007. Living Long and Well in the 21st Century: Strategic Directions for Research on Aging   URL: https:/​/www.​nia.nih.gov/​about/​living-long-well-21st-century-strategic-directions-research-aging/​introduction [accessed 2015-09-10] [WebCite Cache]
    2. National Lung Screening Trial Research Team, Aberle DR, Adams AM, Berg CD, Black WC, Clapp JD, et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 2011 Aug 4;365(5):395-409 [FREE Full text] [CrossRef] [Medline]
    3. Zeng QT, Tse T, Crowell J, Divita G, Roth L, Browne AC. Identifying consumer-friendly display (CFD) names for health concepts. In: AMIA Annu Symp Proc. 2005 Presented at: AMIA Annual Symposium; 10/22/2005-10/26/2005; Washington, DC p. 859-863   URL: http://europepmc.org/abstract/MED/16779162 [WebCite Cache]
    4. US Department of Health and Human Services. National Action Plan to Improve Health Literacy. Washington, DC; 2010.   URL: http://health.gov/communication/hlactionplan/pdf/Health_Literacy_Action_Plan.pdf [accessed 2016-05-11] [WebCite Cache]
    5. Kutner M, Greenberg E, Jin Y, Paulsen C. Results from the 2003 National Assessment of Adult Literacy. Washington, DC: US Department of Education; 2006. The Health Literacy of America's Adults   URL: https:/​/www.​healthypeople.gov/​2020/​tools-resources/​evidence-based-resource/​health-literacy-america%E2%80%99s-adults-results-2003-national [accessed 2016-05-11] [WebCite Cache]
    6. Rosenbaum S. Usability evaluations versus usability testing: when and why? IEEE Transactions on Professional Communication 1989;32(4):283-288.
    7. Rosenbaum S. Usability evaluations versus usability testing: when and why? IEEE Trans Profess Commun 2011;32(4):210-216. [CrossRef]
    8. Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med 2009 May;36(5):452-457 [FREE Full text] [CrossRef] [Medline]
    9. Tang PC, Overhage JM, Chan AS, Brown NL, Aghighi B, Entwistle MP, et al. Online disease management of diabetes: engaging and motivating patients online with enhanced resources-diabetes (EMPOWER-D), a randomized controlled trial. J Am Med Inform Assoc 2013 May 1;20(3):526-534 [FREE Full text] [CrossRef] [Medline]
    10. Krist AH, Woolf SH, Rothemich SF, Johnson RE, Peele JE, Cunningham TD, et al. Interactive preventive health record to enhance delivery of recommended care: a randomized trial. Ann Fam Med 2012;10(4):312-319 [FREE Full text] [CrossRef] [Medline]
    11. Health Information Technology. 2013. What is a Patient Portal?   URL: https://www.healthit.gov/providers-professionals/faqs/what-patient-portal [accessed 2016-05-11] [WebCite Cache]
    12. Tobacman JK, Kissinger P, Wells M, Prokuski J, Hoyer M, McPherson P, et al. Implementation of personal health records by case managers in a VAMC general medicine clinic. Patient Educ Couns 2004 Jul;54(1):27-33. [CrossRef] [Medline]
    13. Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13(2):121-126 [FREE Full text] [CrossRef] [Medline]
    14. Bridgford A, Davis TME. A comprehensive patient-held record for diabetes. Part one: initial development of the Diabetes Databank. Pract Diab Int 2001 Sep;18(7):241-245. [CrossRef]
    15. Raisinghani MS, Young E. Personal health records: key adoption issues and implications for management. Int J Electron Healthc 2008;4(1):67-77. [CrossRef] [Medline]
    16. Hess R, Bryce CL, McTigue K, Fitzgerald K, Zickmund S, Olshansky E, et al. The diabetes patient portal: patient perspectives on structure and delivery. Diabetes Spectrum 2006;19(2):106-109. [CrossRef]
    17. McNamara M, Arnold C, Sarma K, Aberle DR, Garon E, Bui AAT. Patient Portal Preferences: Perspectives on Imaging Information. J Assoc Inf Sci Technol 2015 Aug 1;66(8):1606-1615 [FREE Full text] [CrossRef] [Medline]
    18. Siegel R, Ma J, Zou Z, Jemal A. Cancer statistics, 2014. CA Cancer J Clin 2014;64(1):9-29 [FREE Full text] [CrossRef] [Medline]
    19. van den Bergh KAM, Essink-Bot ML, van Klaveren RJ, de Koning HJ. Informed participation in a randomised controlled trial of computed tomography screening for lung cancer. Eur Respir J 2009 Sep;34(3):711-720 [FREE Full text] [CrossRef] [Medline]
    20. Wu H. Chapter 3: Questionnaire Surveys. In: Exploring Healthcare Consumer Acceptance of Personal Health Information Technology. College Park, MD: Department of Information Systems University of Maryland; 2013:348.
    21. US Census Data.: US Census Bureau; 2013.   URL: http:/​/factfinder.​census.gov/​faces/​tableservices/​jsf/​pages/​productview.​xhtml?pid=ACS_13_5YR_DP05&src=pt [WebCite Cache]
    22. Amazon Mechanical Turk.   URL: https://requester.mturk.com/ [accessed 2015-08-11] [WebCite Cache]
    23. Gottlieb A, Hoehndorf R, Dumontier M, Altman RB. Ranking adverse drug reactions with crowdsourcing. J Med Internet Res 2015;17(3):e80 [FREE Full text] [CrossRef] [Medline]
    24. Kristan J, Suffoletto B. Using online crowdsourcing to understand young adult attitudes toward expert-authored messages aimed at reducing hazardous alcohol consumption and to collect peer-authored messages. Transl Behav Med 2015 Mar;5(1):45-52 [FREE Full text] [CrossRef] [Medline]
    25. Cui L, Carter R, Zhang G. Evaluation of a novel Conjunctive Exploratory Navigation Interface for consumer health information: a crowdsourced comparative study. J Med Internet Res 2014;16(2):e45 [FREE Full text] [CrossRef] [Medline]
    26. MacLean DL, Heer J. Identifying medical terms in patient-authored text: a crowdsourcing-based approach. J Am Med Inform Assoc 2013;20(6):1120-1127 [FREE Full text] [CrossRef] [Medline]
    27. Yu B, Willis M, Sun P, Wang J. Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk. J Med Internet Res 2013;15(6):e108 [FREE Full text] [CrossRef] [Medline]
    28. Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. J Med Internet Res 2012;14(3):e79 [FREE Full text] [CrossRef] [Medline]
    29. Bewick V, Cheek L, Ball J. Statistics review 14: Logistic regression. Crit Care 2005 Feb;9(1):112-128 [FREE Full text] [CrossRef] [Medline]
    30. Parker RM, Ratzan SC, Lurie N. Health literacy: a policy challenge for advancing high-quality health care. Health Affairs 2003;22(4):147-153 [FREE Full text] [Medline]
    31. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. Journal of Medical Internet Research 2012;14(1):19 [FREE Full text] [CrossRef] [Medline]
    32. Irizarry T, DeVito Dabbs A, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    33. Parker RM, Ratzan SC, Lurie N. Health literacy: a policy challenge for advancing high-quality health care. Health Aff (Millwood) 2003;22(4):147-153 [FREE Full text] [Medline]
    34. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    35. Mák G, Smith FH, Leaver C, Hagens S, Zelmer J. The Effects of Web-Based Patient Access to Laboratory Results in British Columbia: A Patient Survey on Comprehension and Anxiety. J Med Internet Res 2015;17(8):e191 [FREE Full text] [CrossRef] [Medline]
    36. Ralston JD, Rutter CM, Carrell D, Hecht J, Rubanowice D, Simon GE. Patient use of secure electronic messaging within a shared medical record: a cross-sectional study. J Gen Intern Med 2009 Mar;24(3):349-355 [FREE Full text] [CrossRef] [Medline]
    37. Beckjord EB, Finney Rutten LJ, Squiers L, Arora NK, Volckmann L, Moser RP, et al. Use of the internet to communicate with health care providers in the United States: estimates from the 2003 and 2005 Health Information National Trends Surveys (HINTS). J Med Internet Res 2007 Jul;9(3):e20 [FREE Full text] [CrossRef] [Medline]
    38. Fox S, Jones S. The social life of health information: Americans' pursuit of health takes place within a widening network of both online and offline sources. Washington, DC: Pew Internet and American Life Project; 2009.   URL: http://www.pewinternet.org/files/old-media//Files/Reports/2009/PIP_Health_2009.pdf [accessed 2014-10-01] [WebCite Cache]
    39. Udem T. Consumers and Health Information Technology. A National Survey. Oakland, CA: California Health Care Foundation; 2010.   URL: http:/​/www.​chcf.org/​publications/​2010/​04/​consumers-and-health-information-technology-a-national-survey [accessed 2015-09-10] [WebCite Cache]
    40. Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and Facilitators to Online Portal Use Among Patients and Caregivers in a Safety Net Health Care System: A Qualitative Study. J Med Internet Res 2015;17(12):e275 [FREE Full text] [CrossRef] [Medline]
    41. Kena G, Musu-Gillette L, Robinson J, Wang X, Rathbun A, Zhang J, et al. The Condition of Education. Washington, DC: National Center for Education Statistics; 2015.   URL: http://nces.ed.gov/pubs2015/2015144.pdf [accessed 2015-09-10] [WebCite Cache]
    42. Ancker JS, Silver M, Miller MC, Kaushal R. Consumer experience with and attitudes toward health information technology: a nationwide survey. J Am Med Inform Assoc 2013 Jan 1;20(1):152-156 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    ANOVA: one-way analysis of variance
    CT: computed tomography
    PHRs: personal health records
    UCLA: University of California, Los Angeles


    Edited by G Eysenbach; submitted 10.09.15; peer-reviewed by E Boogerd, B Crotty; comments to author 22.11.15; revised version received 05.01.16; accepted 23.03.16; published 08.06.16

    ©Mary Zide, Kaitlyn Caswell, Ellen Peterson, Denise R Aberle, Alex AT Bui, Corey W Arnold. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 08.06.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.