Background: eHealth apps have the potential to meet the information needs of patient populations and improve health literacy rates. However, little work has been done to document perceived usability of portals and health literacy of specific topics.
Objective: Our aim was to establish a baseline of lung cancer health literacy and perceived portal usability.
Methods: A survey based on previously validated instruments was used to assess a baseline of patient portal usability and health literacy within the domain of lung cancer. The survey was distributed via Amazon’s Mechanical Turk to 500 participants.
Results: Our results show differences in preferences and literacy by demographic cohorts, with a trend of chronically ill patients having a more positive reception of patient portals and a higher health literacy rate of lung cancer knowledge (P<.05).
Conclusions: This article provides a baseline of usability needs and health literacy that suggests that chronically ill patients have a greater preference for patient portals and higher level of health literacy within the domain of lung cancer.
Life expectancy has nearly doubled in the last century , due in part to preventive care, screening protocols, and the conversion of previously lethal diseases into manageable, chronic conditions through medical advances. Recently, lung cancer screening with computed tomography (CT) has been shown to significantly reduce lung cancer mortality and has resulted in paradigm shifts such that screening is now a covered medical procedure by both third party payers and Medicare [ ], providing the opportunity for more patients to be screened. However, lung cancer is still the single greatest cause of cancer-related death. Accurate and timely patient education is considered to be one of the critical elements underlying improvements in public health, to better inform patients of the underlying causes of their conditions, the measures they can take to mitigate risk, and the basis, and balance of benefits to risks, of certain interventions.
However, low levels of health literacy are seen as a hurdle to accessing health information . The US Department of Health and Human Services defines health literacy as patients’ ability to acquire, read, and understand health information in order to make health decisions appropriate to their situation [ ]. The National Center for Education Statistics maintains that health literacy is crucial for all adults to understand and improve their health, as they encounter health information in a variety of digital and hardcopy formats (eg, websites, blogs, federated search engines, magazine articles, pamphlets, prescription directions) [ ].
Further study of patients’ health literacy in the subject of lung cancer is necessary to inform the design of patient education tools. Lung cancer screening protocols have recently been revised due to the findings in , demonstrating the efficacy of annual CT scans for high-risk individuals. Given the major policy change in lung cancer screening, it is expected that moderate patient education will be required to better understand the etiologies of lung cancer and basis for screening. eHealth apps are a potentially potent medium for the task of providing educational materials.
The term eHealth can be broadly described as the use of information technology in health care, examples of which include patient portals, personal fitness apps that run on mobile phones or personal computers, and digital consumer health educational guidelines. eHealth tools are increasingly available; however, availability does not ensure greater health literacy or increased use of these tools. The term usability refers to the ease of use and learnability of a tool, the measure of which can serve as one gauge of the effectiveness of a tool in helping the user reach their objectives . Usability is necessary to ensure use of and application of the information within eHealth technologies [ , ]. Contributing to perceived usability is perceived feasibility. Feasibility, which can be defined as the effectiveness of an intervention, can be further divided into eight areas of focus, including acceptability, for example, how users react to an intervention, and demand, for example, an estimated need of a tool [ ].
While no design will be able to anticipate all information needs, the content of a patient portal has the potential to influence perceived usability and satisfaction, as well as improve health literacy and outcomes [, ]. Generally, a patient portal is a secure website that provides individuals with access to their personal medical information [ ]. Although there has not been lengthy discussion on the content of portals in particular, the content of personal health records (PHRs) has received more attention. Basic consensus has been reached on certain data points that should be included: problem lists, procedures, lab tests, diagnoses, and notes [ , - ]. These data points would likely prove beneficial in a portal as well. Portals also have a range of functions not available in PHRs, such as the ability to email a physician, renew a prescription, view clinical reports, and make appointments. However, patient information needs may differ by diagnosis, and in order to improve health literacy and outcomes, some diagnoses may require more specific additional information, educational content, and functions [ , ]. For instance, cancer patients have been shown to want access to targeted information about their particular diagnosis (benign vs cancerous), treatments (chemotherapy, radiation, surgery), and prognosis (quality of life, 5-year survival) [ ]. However, the specific needs of cancer patients by diagnosis are not well known. Identifying these needs, as well as their literacy levels on specific subjects, would prove helpful in the design of tools meant to serve specific populations, such as lung cancer. Before this can be done, it is necessary to establish a baseline of a general population’s information needs.
Additionally, as lung cancer is the leading cause of cancer death and the majority of lung cancer cases are due to tobacco smoke, it is important that the general population be informed regarding lung cancer and the means through which risk can be mitigated (smoking cessation) . While campaigns to inform the general population of this health issue have been wide spread, it is important to document how effective these campaigns have been in providing health consumers with knowledge, especially when considering recent reforms to screening protocol. Assessment of lung cancer literacy can help to inform portal module design aimed at improving literacy rates.
As an initial starting point, we were interested in two areas of study: (1) understanding the perceived usability and, in turn, feasibility of information content of patient portals of a convenience cohort taken from the general public, and (2) the demand for lung cancer‒specific health literacy of a convenience cohort taken from the general public. Our goal was to establish a baseline to understand how perspectives and literacy vary by basic patient (consumer) demographics.
To assess baseline health information needs and preferences, as well as lung cancer knowledge, a survey was developed based on previously validated surveys [, ], which were designed for the purposes of documenting patients’ health portal preferences and lung cancer knowledge. While the questions in our survey are taken from these validated tools, not all questions from the original surveys were used. Instead, in an effort to keep our survey concise and focused, we chose those questions that most closely focused on our concerned topics, perceived usability and feasibility of patient portals, and lung cancer knowledge. We chose to rely on prior validated surveys, as developing and validating a survey would require considerable additional research beyond our current scope. Our instrument consisted of three modules (see ). The first module captured information needs and preferences regarding patient portals posed as statements that were rated by participants on a 7-point Likert scale ranging from 1 (completely disagree) to 7 (completely agree). The second module consisted of factual statements about lung cancer and computed tomography with multiple choice answers, of which only one answer was correct. The third module contained demographic questions, based primarily on the US census [ ]. Two free-text questions allowed participants to describe concerns and the effects of using a portal. This study was certified exempt by an internal review board committee at University of California, Los Angeles (UCLA).
The survey was distributed via the website MTurk, a crowdsourcing Internet site devoted to human intelligence tasks  that has been used in prior medical informatics studies [ - ] and in the evaluation of consumer health tools [ , ]. The MTurk site was used as it has been demonstrated by those studies as a reliable method to survey a convenience sample and because it provided access to individuals across the country, as opposed to one geographical area. The survey was made available only to those MTurk participants within the United States. The survey was posted on the MTurk website for 3 weeks in May 2015. Participants were invited to complete the survey with the guarantee of compensation of US $1 per completed survey; each participant completed only one survey. The first 500 individuals to fully complete the survey were chosen as a convenience sample for purposes of analysis. We sought 250 participants age 41 and older, and 250 participants age 40 and younger. To do this, the survey was distributed twice, once as a survey available to those who were age 40 and younger, and once as a survey to those who were age 41 and older. Participants were asked to affirm their age cohort.
Power and Statistical Tests
A power analysis indicated that in order to demonstrate a 0.95 power with alpha=.025, we would require 324 participants. Survey results were analyzed using SPSS version 20 software. Statistically significant results are those with a P value ≤.05. Independent t-tests and one-way analyses of variance (ANOVAs) were used in univariate analyses to determine differences in information needs and preferences based on mean values of demographic variables. Chi-square was used in univariate analysis to identify differences in health literacy based on mean demographic variables. To determine if combinations of demographic variables can predict dependent variables, stepwise logistic regression was done using all significant variables from univariate analysis, with each model using the independent variable(s) that had significant univariate results for a particular question (P ≤.05). Logistic regression was used in order to make the data as parsimonious as possible, as the number of cases per group were limited. Alongside P values, log-odds, and prediction probabilities, Nagelkerke R2values were also reported. The Nagelkerke R2statistic is used to demonstrate how useful the independent variables are in predicting the dependent variable .
All variables can be seen in. Due to sparse data, the demographic variables sex and race were collapsed into dichotomous variables (seen ). The variable sex included one response listed as “other.” This single response was eliminated from univariate analysis of portal preferences based on sex. For logistic regression, the 7-point Likert scales used to record patient portal responses were also dichotomized as follows: Agree (Scores 5-7) and Disagree (Score 1-4). For regression analysis, a score of 4 on Likert scale responses was included within the Disagree category, as the purpose of our analysis was to compare those who agreed to others. For lung cancer knowledge variables, binary values were coded as “0” for incorrect and “1” for correct.
Only surveys with no missing data elements were analyzed. Participants who returned surveys with missing data were contacted via email and invited to supply the missing data points. There was no consistent pattern to the types of data elements not completed. The surveys of participants who supplied the missing data were included in analysis. In total, out of the 500 surveys issued, 473 complete surveys were collected.
The majority of participants were white (389/473, 82.2%) (see). Although accounting for fewer respondents, Asian (26/474, 5.4%) and black (25/473, 5.2%) participants were the second largest group and were roughly equivalent. The majority of participants had some college education, with very few (27/473, 5.7%) participants listing high school as their highest level of educational attainment. Roughly half of respondents reported an annual income of US $35,000 or less. Participants tended to spend over 11 hours a week online (349/473, 73.7%), but most had used a portal 10 times or less (216/473, 45.6%) or had never used a patient portal (174/473, 36.7%). This rate of portal use suggests that responses to the survey are both a measure of needs for those with experience using a portal, and expectations for those who have not used a portal before.
|Age in years|
|Prefer not to answer||1||0.2|
|Unknown or prefer not to answer||3||0.6|
|Two or more races||19||4|
|$76,000 or more||55||11.6|
|Prefer not to answer||6||1.2|
|Times using a portal|
|51 times or more||9||1.9|
|Prefer not to answer||3||0.6|
|Time online, hours|
|11 hours or more||349||73.7|
|Prefer not to answer||2||0.4|
Influence of Participant Characteristics on Patient Portal Questions
Patient response frequencies to all survey questions can be seen in, and results to all univariate analyses can be seen in . Demographic variables that significantly influenced participant perceptions of patient portal usability and feasibility are shown in . For all significant results, females rated statements about portal perceptions with higher positive responses. For example, the average Likert response to the question “Portals are not difficult to use” was 4.97 for males and 5.32 for females (P=.009). Females tended to have higher, more positive average Likert scores across all questions relating to portal use.
Respondents reporting a chronic illness also tended to have more positive views of portals, with higher average ratings than those without chronic illness. The average response was significantly higher for chronically ill participants for all four questions relating to portals (see). This trend was also seen across all questions relating to portal preferences.
Differences in responses significantly varied based on the number of times participants had used a patient portal. However, there was no consistent trend seen across answers. Additionally, while not statistically significant, 33% of participants reported concern about “unauthorized access” to their patient portal.
|Independent variable||Survey question||Eta2||P|
|Portals are not difficult to use.||0.014||.009|
|Using a portal can make me accomplish tasks (eg, review my diagnoses and tests) quickly in managing my personal health information.||0.009||.044|
|A portal can be useful to manage my personal health information.||0.025||.003|
|Using a portal can make me accomplish tasks (eg, review my diagnoses and tests) quickly in managing my personal health information.||0.29||.001|
|A personalized portal can suit my needs of managing my personal health information.||0.014||.034|
|It should be easy to become skillful at using a portal.||0.027||.001|
|A personalized portal can suit my needs of managing my personal health information.||0.036||.002|
|Using a portal can make me accomplish tasks (eg, review my diagnoses and tests) quickly in managing my personal health information.||0.035||.002|
|A portal can be useful to manage my personal health information.||0.023||.025|
|Using a portal with a health encyclopedia can provide me with health care knowledge and education.||0.023||.025|
|It should be easy to become skillful at using a portal.||0.030||.007|
aIndependent variable that utilized an independent t test; all others used ANOVAs.
Influence of Patient Characteristics on Lung Cancer Screening Knowledge
Questions pertaining to lung cancer and chest CT had multiple choice answers, of which one answer was correct. The chronic illness predictor was most frequently associated with correct responses (see). Those reporting a chronic illness had a higher rate of correct answers for three of the four significant results. However, those not reporting chronic illness performed better on the two questions with the outcome variable “Someone who has quit smoking…” and “Lung cancer is one of the most common cancers.”
|Independent variable||Survey question||P|
|In the past, before the CT scan was introduced, the chance of dying due to lung cancer after diagnosis was high.||.005|
|A change of cough pattern is a frequent sign of lung cancer.||.034|
|Coughing up blood is a frequent sign of lung cancer.||.030|
|Lung cancer is one of the most common cancers.||.036|
|CT images are made with X-rays.||.016|
|To complete a CT scan, subjects must undress their upper body.||.006|
|Lung cancer is infectious.||.030|
|Coughing up blood is a frequent sign of lung cancer.||<.001|
|CT images are made with X-rays.||.012|
|To complete a CT scan, subjects must undress their upper body.||.017|
|A change of cough pattern is a frequent sign of lung cancer.||.014|
|Lung cancer is infectious.||<.001|
|Lung cancer is infectious.||<.001|
Other predictors with significant test results included time spent online, income, education, smoking habit, sex, and portal use. Time spent online had the second highest number of significant tests with three results. However, there was no consistent pattern observed within the rates of answers, with those who spent more time sometimes outperforming and sometimes underperforming those who spent less time. For the income predictor, correct answers to the question “Coughing up blood is a frequent sign of lung cancer” increased as income increased, until the level of US $76,000 or more. Those who made US $76,000 or more a year had a lower rate of correct answers than those who made less. For the smoking habit predictor, those who smoked had a slight but significantly lower rate of correct answers (78.4% versus 78.7%) for the question “Lung cancer is one of the most common cancers.” Men had a significantly higher rate of correct answers for the question “Coughing up blood is a frequent sign of lung cancer” than women (75.4% vs 73.7%). As portal use increased, correct answers to the question “Lung cancer is infectious” decreased. There was no pattern observed in response rates for the question “In the past, before the CT scan was introduced…” when stratified by the education predictor, nor for the question “A change of cough pattern is a frequent sign of lung cancer” response rates when stratified by the income predictor.
Stepwise logistic regression was performed for demographic variables that had statistically significant relationships as seen inand . Statistically significant results for logistic regression are seen in . Increased portal use was positively associated with agreeing with the statement “It should be easy to become skillful at using a portal,” meaning participants who used a portal more than 10 times were more likely to agree with the statement. However, prediction success was weak with 32.6% of outcomes correctly predicted. The chronic illness predictor was positively associated with having the correct answer for “A change of cough pattern is a frequent sign of lung cancer,” indicating that those with a chronic illness were more likely to choose the correct answer for the question. However, prediction was not improved in comparison to the constant model, remaining at 52.4%.
|Independent variable||Survey question||Log-odds||P||Nagelkerke R2||Prediction percentage correct, %|
|It should be easy to become skillful at using a portal.||0.996||.016||0.024||32.6|
|A change of cough pattern is a frequent sign of lung cancer.||0.611||.008||0.020||52.4|
Principal Results and Comparisons With Prior Work
In this study, a convenience cohort of the general public was asked to rate their perceptions of usability and feasibility of patient portals and knowledge of lung cancer, in order to demonstrate the demand for lung cancer knowledge and status of patient portal usability. Overall, we observed that respondents reporting chronic illness tended to have more positive opinions of patient portals and to perform better on lung cancer knowledge questions than those without a chronic condition. Chronically ill patients having a more positive opinion of eHealth is a finding that has also been observed [, ]. Similarly, a review of literature on portals found that participant interest in patient portals varied by health status [ ].
Interestingly, some studies have found that chronically ill patients may have lower health literacy [, ]. In contrast, a study found that those with a chronic illness reported higher rates of literacy regarding test results [ ]. In our work, it may be that the higher performance we observed relates to higher levels of educational attainment, as those respondents reporting chronic illness in our survey more frequently had Associate, Bachelor’s, and Master’s degrees than those reporting no chronic illness (14.6% vs 12.9%, 39.8% vs 36.8%, and 9.7% vs 8.8% respectively).
Having used a portal was associated with significant difference in opinions of patient portals. However, there was no consistent trend observed, with some statements being rated in higher agreement by those who had used portals more and other statements being rated higher by those who had used them less. It is also worth noting that for those who had never used a portal, these statements are measuring expectations of portals, while with those who have used portals, these statements are measuring experience. These results suggest that opinions about the usability of portals do not demonstrate a clear trend whether being drawn based on expectations or experience.
We also found that women reported more positive ratings of portals. While use does not equal preference, positive ratings may be influenced by the higher use of portals we observed in this survey (19.7% of women had used a portal over ten times, compared to 14.3% of men), which is consistent with higher eHealth resource use observed in women [- ]. We also documented a statistically significant difference in correct responses for lung cancer knowledge stratified by the amount of time spent online. Others have found similar relationships between Internet use and health literacy [ ], but our results showed no consistent trend and are inconclusive.
Although two logistic regression models had significant P value results, neither model had strong prediction success. While the low P values (<.05) indicate that the findings are unlikely due to chance, the prediction values, as well as the Nagelkerke R2values, indicate that these models will not perform prediction well. Further research is required to determine predictors that would predict answer cohorts to these questions successfully.
Finally, 33% of participants noted that they were “concerned about unauthorized access,” which was the most commonly chosen option (seem, Question 26). In other work, security was a concern for two thirds of health information consumers, although users of PHRs had less observed concern [ ]. The overall percentage of participants concerned with security in this study was less than in [ ]. However, a similar trend was observed with 32.5% of those who had used a portal more than 10 times were concerned with unauthorized access while 33.1% of those who had used a portal 10 times or less were concerned. Similarly, security has also been a concern [ ]. This common theme suggests that eHealth users may associate eHealth tools with a lack of security. This concern has the potential to impact portal use, as it would likely limit satisfaction and perceived usability and, thus, feasibility.
There are several limitations to this study that may have influenced our results. Most importantly, we used a convenience sample of the first 500 respondents to a survey posted on MTurk. As such, the results derive from Internet users; those with less Internet experience may well have different views regarding patient portals and lung cancer health literacy than those documented here. Our respondents also had higher levels of education than seen in the US population , which may have influenced health literacy. Moreover, although we specifically asked respondents not to look up answers to lung cancer questions, we have no guarantee that responses were not informed by additional online queries.
An additional limitation is the potential bias that may be introduced by the use of a digital survey format. Others have found that patients who have used eHealth technology have more positive opinions of it than those who do not , suggesting that experience with technology can cause one to regard it more favorably. Here, completing a digital survey on eHealth may bias respondents to rate statements about patient portals more highly.
This work documents a baseline for consumer information needs and health literacy within the domain of lung cancer. Our results suggest that women and those with chronic illness had more positive views of patient portals and that chronically ill patients had higher health literacy in lung cancer. Although chronically ill patients are a likely user group for patient portals, non‒chronically ill patients can also benefit from opportunities to manage and learn about their health as well as mitigate risk via portals. Given this, the results suggest a need for further lung cancer education with opportunities to use patient portals to better educate individuals who do not have a chronic illness about lung cancer. Further study is also needed in order for portals to better address the information needs of patients who are not chronically ill in order to improve perceived usability and feasibility.
This baseline can be used in future comparison studies as well as to inform portal design to improve usability and raise lung cancer health literacy with educational modules. Our future work includes designing a patient portal influenced by these results and then surveying a patient population at UCLA who will have access to the portal, to compare those survey results to this baseline, in order to identify any difference between them.
This work is supported by NIH/NCI Grant R01 LM011333, NIH/NCI Grant 5T32 EB01664003, and UCLA HBCU Grant 285247.
Conflicts of Interest
Multimedia Appendix 1PDF File (Adobe PDF File), 43KB
Multimedia Appendix 2PDF File (Adobe PDF File), 64KB
Multimedia Appendix 3PDF File (Adobe PDF File), 42KB
Multimedia Appendix 4PDF File (Adobe PDF File), 68KB
- Burton DM. National Institute on Aging.: US Department of Health and Human Services; 2007. Living Long and Well in the 21st Century: Strategic Directions for Research on Aging URL: https://www.nia.nih.gov/about/living-long-well-21st-century-strategic-directions-research-aging/introduction [accessed 2015-09-10] [WebCite Cache]
- National Lung Screening Trial Research Team, Aberle DR, Adams AM, Berg CD, Black WC, Clapp JD, et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 2011 Aug 4;365(5):395-409 [FREE Full text] [CrossRef] [Medline]
- Zeng QT, Tse T, Crowell J, Divita G, Roth L, Browne AC. Identifying consumer-friendly display (CFD) names for health concepts. In: AMIA Annu Symp Proc. 2005 Presented at: AMIA Annual Symposium; 10/22/2005-10/26/2005; Washington, DC p. 859-863 URL: http://europepmc.org/abstract/MED/16779162 [WebCite Cache]
- US Department of Health and Human Services. National Action Plan to Improve Health Literacy. Washington, DC; 2010. URL: http://health.gov/communication/hlactionplan/pdf/Health_Literacy_Action_Plan.pdf [accessed 2016-05-11] [WebCite Cache]
- Kutner M, Greenberg E, Jin Y, Paulsen C. Results from the 2003 National Assessment of Adult Literacy. Washington, DC: US Department of Education; 2006. The Health Literacy of America's Adults URL: https://www.healthypeople.gov/2020/tools-resources/evidence-based-resource/health-literacy-america%E2%80%99s-adults-results-2003-national [accessed 2016-05-11] [WebCite Cache]
- Rosenbaum S. Usability evaluations versus usability testing: when and why? IEEE Transactions on Professional Communication 1989;32(4):283-288.
- Rosenbaum S. Usability evaluations versus usability testing: when and why? IEEE Trans Profess Commun 2011;32(4):210-216. [CrossRef]
- Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med 2009 May;36(5):452-457 [FREE Full text] [CrossRef] [Medline]
- Tang PC, Overhage JM, Chan AS, Brown NL, Aghighi B, Entwistle MP, et al. Online disease management of diabetes: engaging and motivating patients online with enhanced resources-diabetes (EMPOWER-D), a randomized controlled trial. J Am Med Inform Assoc 2013 May 1;20(3):526-534 [FREE Full text] [CrossRef] [Medline]
- Krist AH, Woolf SH, Rothemich SF, Johnson RE, Peele JE, Cunningham TD, et al. Interactive preventive health record to enhance delivery of recommended care: a randomized trial. Ann Fam Med 2012;10(4):312-319 [FREE Full text] [CrossRef] [Medline]
- Health Information Technology. 2013. What is a Patient Portal? URL: https://www.healthit.gov/providers-professionals/faqs/what-patient-portal [accessed 2016-05-11] [WebCite Cache]
- Tobacman JK, Kissinger P, Wells M, Prokuski J, Hoyer M, McPherson P, et al. Implementation of personal health records by case managers in a VAMC general medicine clinic. Patient Educ Couns 2004 Jul;54(1):27-33. [CrossRef] [Medline]
- Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13(2):121-126 [FREE Full text] [CrossRef] [Medline]
- Bridgford A, Davis TME. A comprehensive patient-held record for diabetes. Part one: initial development of the Diabetes Databank. Pract Diab Int 2001 Sep;18(7):241-245. [CrossRef]
- Raisinghani MS, Young E. Personal health records: key adoption issues and implications for management. Int J Electron Healthc 2008;4(1):67-77. [CrossRef] [Medline]
- Hess R, Bryce CL, McTigue K, Fitzgerald K, Zickmund S, Olshansky E, et al. The diabetes patient portal: patient perspectives on structure and delivery. Diabetes Spectrum 2006;19(2):106-109. [CrossRef]
- McNamara M, Arnold C, Sarma K, Aberle DR, Garon E, Bui AAT. Patient Portal Preferences: Perspectives on Imaging Information. J Assoc Inf Sci Technol 2015 Aug 1;66(8):1606-1615 [FREE Full text] [CrossRef] [Medline]
- Siegel R, Ma J, Zou Z, Jemal A. Cancer statistics, 2014. CA Cancer J Clin 2014;64(1):9-29 [FREE Full text] [CrossRef] [Medline]
- van den Bergh KAM, Essink-Bot ML, van Klaveren RJ, de Koning HJ. Informed participation in a randomised controlled trial of computed tomography screening for lung cancer. Eur Respir J 2009 Sep;34(3):711-720 [FREE Full text] [CrossRef] [Medline]
- Wu H. Chapter 3: Questionnaire Surveys. In: Exploring Healthcare Consumer Acceptance of Personal Health Information Technology. College Park, MD: Department of Information Systems University of Maryland; 2013:348.
- US Census Data.: US Census Bureau; 2013. URL: http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_13_5YR_DP05&src=pt [WebCite Cache]
- Amazon Mechanical Turk. URL: https://requester.mturk.com/ [accessed 2015-08-11] [WebCite Cache]
- Gottlieb A, Hoehndorf R, Dumontier M, Altman RB. Ranking adverse drug reactions with crowdsourcing. J Med Internet Res 2015;17(3):e80 [FREE Full text] [CrossRef] [Medline]
- Kristan J, Suffoletto B. Using online crowdsourcing to understand young adult attitudes toward expert-authored messages aimed at reducing hazardous alcohol consumption and to collect peer-authored messages. Transl Behav Med 2015 Mar;5(1):45-52 [FREE Full text] [CrossRef] [Medline]
- Cui L, Carter R, Zhang G. Evaluation of a novel Conjunctive Exploratory Navigation Interface for consumer health information: a crowdsourced comparative study. J Med Internet Res 2014;16(2):e45 [FREE Full text] [CrossRef] [Medline]
- MacLean DL, Heer J. Identifying medical terms in patient-authored text: a crowdsourcing-based approach. J Am Med Inform Assoc 2013;20(6):1120-1127 [FREE Full text] [CrossRef] [Medline]
- Yu B, Willis M, Sun P, Wang J. Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk. J Med Internet Res 2013;15(6):e108 [FREE Full text] [CrossRef] [Medline]
- Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. J Med Internet Res 2012;14(3):e79 [FREE Full text] [CrossRef] [Medline]
- Bewick V, Cheek L, Ball J. Statistics review 14: Logistic regression. Crit Care 2005 Feb;9(1):112-128 [FREE Full text] [CrossRef] [Medline]
- Parker RM, Ratzan SC, Lurie N. Health literacy: a policy challenge for advancing high-quality health care. Health Affairs 2003;22(4):147-153 [FREE Full text] [Medline]
- Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. Journal of Medical Internet Research 2012;14(1):19 [FREE Full text] [CrossRef] [Medline]
- Irizarry T, DeVito Dabbs A, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]
- Parker RM, Ratzan SC, Lurie N. Health literacy: a policy challenge for advancing high-quality health care. Health Aff (Millwood) 2003;22(4):147-153 [FREE Full text] [Medline]
- Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
- Mák G, Smith FH, Leaver C, Hagens S, Zelmer J. The Effects of Web-Based Patient Access to Laboratory Results in British Columbia: A Patient Survey on Comprehension and Anxiety. J Med Internet Res 2015;17(8):e191 [FREE Full text] [CrossRef] [Medline]
- Ralston JD, Rutter CM, Carrell D, Hecht J, Rubanowice D, Simon GE. Patient use of secure electronic messaging within a shared medical record: a cross-sectional study. J Gen Intern Med 2009 Mar;24(3):349-355 [FREE Full text] [CrossRef] [Medline]
- Beckjord EB, Finney Rutten LJ, Squiers L, Arora NK, Volckmann L, Moser RP, et al. Use of the internet to communicate with health care providers in the United States: estimates from the 2003 and 2005 Health Information National Trends Surveys (HINTS). J Med Internet Res 2007 Jul;9(3):e20 [FREE Full text] [CrossRef] [Medline]
- Fox S, Jones S. The social life of health information: Americans' pursuit of health takes place within a widening network of both online and offline sources. Washington, DC: Pew Internet and American Life Project; 2009. URL: http://www.pewinternet.org/files/old-media//Files/Reports/2009/PIP_Health_2009.pdf [accessed 2014-10-01] [WebCite Cache]
- Udem T. Consumers and Health Information Technology. A National Survey. Oakland, CA: California Health Care Foundation; 2010. URL: http://www.chcf.org/publications/2010/04/consumers-and-health-information-technology-a-national-survey [accessed 2015-09-10] [WebCite Cache]
- Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and Facilitators to Online Portal Use Among Patients and Caregivers in a Safety Net Health Care System: A Qualitative Study. J Med Internet Res 2015;17(12):e275 [FREE Full text] [CrossRef] [Medline]
- Kena G, Musu-Gillette L, Robinson J, Wang X, Rathbun A, Zhang J, et al. The Condition of Education. Washington, DC: National Center for Education Statistics; 2015. URL: http://nces.ed.gov/pubs2015/2015144.pdf [accessed 2015-09-10] [WebCite Cache]
- Ancker JS, Silver M, Miller MC, Kaushal R. Consumer experience with and attitudes toward health information technology: a nationwide survey. J Am Med Inform Assoc 2013 Jan 1;20(1):152-156 [FREE Full text] [CrossRef] [Medline]
|ANOVA: one-way analysis of variance|
|CT: computed tomography|
|PHRs: personal health records|
|UCLA: University of California, Los Angeles|
Edited by G Eysenbach; submitted 10.09.15; peer-reviewed by E Boogerd, B Crotty; comments to author 22.11.15; revised version received 05.01.16; accepted 23.03.16; published 08.06.16Copyright
©Mary Zide, Kaitlyn Caswell, Ellen Peterson, Denise R Aberle, Alex AT Bui, Corey W Arnold. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 08.06.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.