Published on 18.11.14 in Vol 3, No 4 (2014): Oct-Dec
Examining the Use of an Open Digital Health Library for Professionals
Background: The Norwegian Electronic Health Library (The Library) is a website for health personnel. Most of the content is also open to the public. Usage statistics have risen sharply in the years 2010-2013.
Objective: We wanted to find out whether the rise was caused by health personnel, the general public, or other factors.
Methods: Since we lacked direct information, we had to use proxy data to shed light on our questions. We applied mixed methods (database of registered users, user survey, usage statistics, and statistics from suppliers), and triangulated between them.
Results: Health personnel were our largest user group, but The Library was also accessed by students, patients, and other groups. Content in Norwegian was preferred to English language content. Concise, practical information was preferred to more comprehensive information. Patient leaflets were the most popular information type. Mobile phone visits differed from personal computer visits both in terms of time of day and what kind of information was viewed.
Conclusions: The Library was used mostly by health personnel, as intended, but our data are inconclusive regarding a possible change in user groups. There was a large degree of consistency in results when using different investigation methods. The survey points toward health personnel being the largest user group, and the usage statistics show that patient leaflets are the most popular content, being viewed by both health personnel and patients.
JMIR Res Protoc 2014;3(4):e66
- libraries, medical;
- access to information;
- information dissemination;
- search engine /statistics;
- Web log analysis
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
The Norwegian Electronic Health Library (The Library) is a publicly funded website (Helsebiblioteket.no) for professionals, established in 2006. It is marketed to health personnel, but not to the general public. The Library provides free access to important sources of knowledge intended for health personnel, including point-of-care tools (reference works), bibliographic databases, and a large number of scientific journals .
The Library is also a sharing platform for guidelines, patient leaflets, and clinical procedures. It contains links to all its purchased sources, as well as open sources like rating scales, reports, summarized research, patient leaflets, and guidelines published elsewhere.
Thus, The Library is partly a traditional library service with purchased content, and partly a sharing platform where information resources are published.
Most of the content is available to anyone with a Norwegian Internet protocol (IP) address, including free access for the entire Norwegian population  to the five largest general medical journals and point-of-care tools like “UpToDate” and “BMJ Best Practice”. To our knowledge, The Library is the only one with national licenses for major international journals and point-of-care tools.
Users do not have to be logged in when using open access sources or subscribed material with a national access license, and may go directly to the original content provider, such as a journal’s home page. Some of the content is, for economic reasons, only available to health personnel and students. Such access is given according to recognized institutional IP addresses (hospitals, health institutions, universities, university colleges) or personal username and password assigned from The Library. As long as users are at their workplace, the IP address is recognized, so they don’t have to log in. If users are at home or anywhere else outside of the workplace they must log in to get full access to all of The Library’s resources.
From 2010 to 2013, The Library saw a sharp rise in usage. We investigated the use of The Library to find out whether this rise was caused by professionals, the general public, or other factors.
Since information on the individual visitors to The Library is not recorded, we had to rely on proxy variables. A proxy variable is something that is probably not in itself of great interest, but from which a variable of interest can be obtained .
Proxy variables serving as indicators of who is using The Library could be: where users are coming from, at what time of day the website is visited, what kind of information is most frequently used, and what kind of device people use. We collected data from several sources in order to shed light on the issue from different angles.
The data were taken from The Library’s database of registered personal users, a survey of our users within a given period, usage statistics for our website, and usage statistics from our content suppliers.
Database of Registered Personal Users
The Library has a database of registered personal users, but we do not record their activities on the website. We have information on professional background for 48,950 of 85,270 registered users (data extracted March 31, 2014), but the quality of the data varies.
We did a user survey from October 1 - October 9, 2013, asking the website visitors if they were health personnel, and if so, which professional group they belonged to.
The users who came to the website were served a pop-up that they could accept or reject to take part in the survey. The survey pop-up was not displayed again after the users had accepted or rejected it.
We asked the participants whether they were health personnel, students, patients/dependents, non-health personnel employees in the health services, researchers, or “other”. If they were health personnel, they were asked which personnel group they belonged to. Health personnel were also asked which sector they worked in.
We analyzed usage statistics for The Library website from the years 2010-2013 by Google Analytics, and we looked particularly at “pageviews” of certain types of information like patient leaflets and guidelines. A “pageview” is recorded each time a user visits a webpage . A single visitor can conduct many pageviews on a website. Each time the visitor returns to the page, a new pageview is recorded. Google Analytics distinguishes between new visitors and returning visitors by using cookies [ ].
We analyzed bounce rates for different parts of our website. Bounce rate is defined by Google as the percentage of single-page sessions, that is, sessions in which the visitor left the site from the entrance page without interacting with the page .
Sessions are defined by Google as the number of individual sessions initiated by all the users of a site. If a user is inactive on the site for 30 minutes or more, any future activity is attributed to a new session. Users who leave the site and return within 30 minutes are counted as part of the original session .
In the analysis of pageviews for guidelines versus patient leaflets, we had data for only the first five months of 2014; therefore, we extrapolated the data as if the numbers were representative for the whole year.
Statistics From Content Suppliers
We used available statistics from our suppliers of bibliographic databases, journals, and point-of-care tools to get data on the usage of their resources. The statistics for journals are based on successful full text downloads according to the COUNTER standard (Counting Online Usage of Networked Electronic Resources) . There is a slight difference between this and a pageview, but for the practical purpose of our comparison we have chosen to ignore this. The statistics on bibliographic databases are based on number of executed searches (number of times a user has pressed the Search button) according to the COUNTER standard. The usage statistics cover the period 2010-2013, and the sources for these statistics are shown in .
The Norwegian point-of-care tool Handbook of Emergency Medicine (Legevakthåndboken) became available in September 2012. Our statistic for the Handbook of Emergency Medicine is based on Google Analytics.
In our comparison of the statistics from the suppliers with the usage statistics for our own website, we have regarded a full text download or a search as a pageview. We did this to be able to compare the data from different sources.
|OVID (AMED, Embase, Medline, Ovid Nursing, PsycINFO)||Ovid (LWW, Nursing Full Text)||UpToDate|
|Ebsco (Cinahl)||Informa Healthcare||BMJ (Best Practice)|
|APA (PsycARTICLES)||Pharmaceutical Press (BNF Children og Martindale)|
|BMJ (23 titles)||Gyldendal Akademiske (Handbook of Emergency Medicine, in Norwegian)|
|JAMA Network (10 titles)|
|The Lancet (4 titles)|
|Annals of Internal Medicine (including ACP Journal Club)|
|New England Journal of Medicine|
Database of Registered Users
In the database of registered personal users, we have information on professions for 56.98% (48,590/85,270) of the users (data extracted March 31, 2014). The largest group of personal users was nurses, followed by physicians, physiotherapists, psychologists, pharmacists, and assistant nurses (see).
|Profession||Number of registered users|
|Assistant nurses||804 (1.65)|
All in all, 2563 (4.27%) visitors took part in the 9-day survey out of an estimated number of approximately 60,000 unique visitors over the same period. Not all respondents answered all the questions of the questionnaire.
A total of 55.95% (1434/2563) of the respondents reported to be health personnel, 15.02% (385/2563) students, 11.86% (304/2563) patients/dependents, 5.93% (152/2563) employees of the health services (not health personnel), 4.37% (112/2563) researchers, and 6.87% (176/2563) other.
Among the health personnel (n=1438), physicians were the largest group (38.66%, 556/1438), followed by nurses (29.62%, 426/1438), psychologists (8.34%, 120/1438), physiotherapists (5.63%, 81/1438), and pharmacists (4.66%, 67/1438).
The respondents were asked which sector of the health services they worked in: 65.51% (940/1435) of health personnel came from hospitals and specialist health services, 28.08% (403/1435) from primary care, and the rest from educational institutions, industry, research, etc.
From our usage statistics, we can see that patient leaflets had the sharpest increase in pageviews from 2011 through 2013, and it was the most frequently viewed information type in 2013 (see). There was a sharp increase in pageviews of guidelines in the period 2010-2013.
The Handbook of Emergency Medicine immediately became very popular and this accounts for a large proportion of the growth in pageviews for point-of-care tools in 2012 and 2013 (see).
The share of users coming from Google and other search engines increased throughout the period 2010-2013 and constituted approximately two-thirds of the traffic in 2013. The share of new visitors increased steadily from 2010 through 2013.
Each visitor spent a shorter time per session on the website in 2013 than in 2010.
The share of visitors using mobile phones increased from 1.00% (10,074/1,007,395) in 2010 to 21.00% (552,859/2,632,660) in 2013. This is a general trend for websites and The Library does not differ considerably from other websites in this respect. Mobile phone users stayed for a shorter length of time on the site than personal computer (PC) users. They also viewed fewer pages than PC users. Mobile phone users spent much more time on patient leaflet pages than on guideline pages. Mobile phone users also viewed more patient leaflet pages than guideline pages. Pageviews of patient leaflets increased throughout the day, while pageviews of guidelines peaked during office hours (see).
While visits by PC users peaked in the office hours, visits by mobile and tablet users increased steadily throughout the day (see).
More users came from recognized workplace networks in 2013 than in 2010, but the relative share of these decreased (see).
From 2010 to 2013, we saw a 2% increase in nightly use (midnight to 8am) and a 4% increase in evening use (4pm to midnight), and a relative reduction in office hours (8am to 4pm) use.
People who viewed patient leaflets spent 4 minutes or more on each page and the bounce rate was very high.
Based on the usage statistics, the most frequently used type of information from the website was patient leaflets (see). Other frequently used types of information on the website were guidelines and procedures.
Both the number of documents (patient leaflets, guidelines, and procedures) as well as the number of topics covered by these documents increased sharply from 2010 to 2013.
We found a difference between the kind of information that was viewed by mobile users and PC users. Patient leaflets were viewed on mobile phones almost as frequently as on PC screens. Guidelines were much more frequently viewed on PCs (see) and very rarely on mobile devices.
|Number of pageviews - all days||3,361,563||3,416,930||4,309,538||5,737,733|
|Number of pageviews - weekdays||2,852,482||2,903,624||3,604,040||4,724,635|
|Number of visits||1,007,395||1,125,686||1,605,288||2,632,660|
|Search engine traffic visits, n (%)||312,292 (31.00%)||427,761 (38.00%)||931,067 (58.00%)||1,676,394 (63.68%)|
|Direct traffic visits, n (%)||392,884 (39.00%)||382,733 (34.00%)||353,163 (22.00%)||579,185 (22.00%)|
|Other traffic visits, n (%)||302,219 (30.00%)||315,192 (28.00%)||321,058 (20.00%)||368,572 (14.00%)|
|Pages per visit||3.34||3.04||2.69||2.18|
|Time spent per visit||4m15s||3m25s||2m38s||1m56s|
|New visitors||366,786 (36.41%)||453,252 (40.26%)||771,366 (48.05%)||1,308,465 (49.70%)|
|Mobile phones, % of visits||10,074 (1.00%)||33,771 (3.00%)||144,476 (9.00%)||552,859 (21.00%)|
|Tablets, % of visits||0%||0%||6%||13%|
|Time spent per visit, desktop||4m17s||3m36s||3m3s||2m25s|
|Time spent per visit, mobile phone||2m14s||2m21s||40s||40s|
|Pages per visit, desktop||3.33||3.05||2.90||2.61|
|Pages per visit, mobile phone||2.43||2.62||1.39||1.24|
|Visits from “workplace” networks||511,845||546,407||654,617||903,048|
|Office hoursbpageviews, % of total on weekdays||1,965,360 (68.90%)||1,991,886 (68.60%)||2,378,666 (66.00%)||2,962,346 (62.70%)|
|Evening hourscpageviews, % of total on weekdays||761,613 (26.70%)||772,364 (26.60%)||1,027,151 (28.50%)||1,450,463 (30.70%)|
% of total on weekdays
|125,509 (4.40%)||142,278 (4.90%)||198,222 (5.50%)||311,826 (6.60%)|
aovernight hours: midnight to 8am
bevening hours: 4pm to midnight
coffice hours: 8am to 4pm
Statistics From Content Suppliers
Based on statistics from our suppliers (), the usage of journals, point-of-care tools, and bibliographic databases showed a steady growth in the period we analyzed (2010-2013).
The statistics from our suppliers combined with the usage statistics (see) make it possible to compare pageviews of journals, point-of-care tools, database searches, and The Library website. Pageviews in the website increased more than the pageviews of purchased sources. In 2013, there were 5.7 million pageviews of the website pages, against 4.0 million pageviews for purchased resources like point-of-care tools and journals. Point-of-care tools rose sharply from September 2012 when the Norwegian language point-of-care tool Handbook of Emergency Medicine was introduced. The Handbook of Emergency Medicine is available online to anyone with a Norwegian IP. It can also be downloaded as an app. This source now has more pageviews than all other point-of-care tools combined.
The Library website and its purchased content are widely used by both health personnel and the general public. Over the last few years, there has been a sharper rise in the use of the website than in the use of most content located outside the website. This could be due to better visibility in Google and other search engines, but that is probably not the only reason. There has also been an increase in the number of documents published on the website.
Norwegian language seems to play a role. Norwegian guidelines, patient leaflets, and the Handbook of Emergency Medicine had a sharper rise in popularity than the rest of our content. In fact, the Handbook of Emergency Medicine had more pageviews than all the other point-of-care tools combined in 2013. A study among Norwegian physicians has shown that it is highly valued to get information in their mother tongue .
Cultural factors, ease-of-use, and how practical the information is, may also play a role. The Handbook of Emergency Medicine is very concise and practical, but it covers only a fraction of the topics that our English language point-of-care tools cover. Patient leaflets are also very concise and highly popular among our users. A study on preferred sources of information for primary care physicians from 1997 states that “Information resources for answering clinical questions should be readily available, familiar, and quick to use” .
There was large growth in the number of mobile phone users visiting the website. We found that visits from PC users peaked during office hours, while visits from mobile users increased throughout the day. Nicholas et al  and Cronk [ ] have shown that there is a difference in the time of day when PC users visit a website and when mobile users visit the same site, and the difference is the same as we found: PCs peak in office hours and mobile phones peak in the evening. Even so, we still have more PC users in the evening than we have mobile users.
Nicholas et al  found that mobile users typically viewed other kinds of material than PC users, typically shorter texts and fewer pages. We found the same—mobile phone users viewed relatively more patient leaflets and fewer guidelines than PC users on The Library website.
Strengths and Limitations
The use of different methods to analyze the usage is a strength of this study. It is also a strength that we have followed the usage over several years.
There are some weaknesses regarding the database, survey, and the Google Analytics data.
The data quality of the database of registered personal users is not optimal. The users report their professional position in free text, which makes classification difficult. Our survey covered only 9 days, and the response rate was low. According to the survey software supplier Surveygizmo , external surveys like customer satisfaction surveys generally get 10-15% response rates, while internal surveys get 30-40% response rates. Our survey had an estimated response rate of 4%. This estimate is probably too low, since we didn’t measure non-responders directly, but based the estimate on Google Analytics’ unique visitor statistic and cookies.
Google Analytics is based on samples of data and hence may be skewed. We have, however, followed the use of the website through Google Analytics over 4 years, and the tendencies are consistent over all these years.
Some of the Google Analytics data give a skewed picture even if they are technically correct. Guidelines pageviews, for example, are reported a bit too high, due to each guideline being split into several documents on our website.
The data from the usage statistics for the website indicate that patient leaflets are becoming more popular. At the same time, the survey data indicate that the majority of users are health personnel. How can this be explained?
The Library is primarily a link portal. Patient leaflets are, along with guidelines, recommendations, and procedures, the only content that is published on the website itself. The same development as we have seen for patient leaflets can be seen for guidelines. Their use (registered as pageviews) also increases sharply. In other words, what we are seeing is an increase in the usage of content that is located on the website itself.
The percentage of users coming from the Google search engine underpins this. The content of the website is more visible to Google than the content of our purchased material. The proportion of users coming from search engines has increased from 31% to 64%.
The high Google ranking may lead more people from outside the ranks of health personnel to our pages. This may be one of the reasons why time spent on the portal and pages per visit both go down. Some of these visitors probably don’t find what they expect on the website.
Is it patients then who read the patient leaflets? Our data give us an indication, but are not conclusive.
The hour of the day of viewing of patient leaflets shows a pattern that resembles both the visits by PC users and the visits by mobile phone users. There is a small peak in office hours, but the number of pageviews increases again from 4 pm and peaks at 10 pm. This indicates that a large proportion of the readers of patient leaflets are the general public or patients, or at least not people at work. But the peak in office hours indicates that the patient leaflets are also used by health personnel at work.
We have to remember that most health personnel are not physicians, and they might very well make good use of patient leaflets. It may also be that physicians visit the website and print the patient leaflets for their patients. Professionals and patients alike seem to embrace simple, short, and trustworthy information.
The inclusion of the patient in the decision process is becoming more common in health care. To make this inclusion meaningful, the patients need to be informed. According to Longo , patients have different information needs depending on where they are in the process. Since The Library is quite unique with its national licenses for highly respected point-of-care tools and journals, we didn’t find any other studies examining patients’ use of similar websites.
The importance of visibility in search engines for medical websites was shown by Giustini as early as 2005 . This visibility also makes it possible to retrieve professional information using devices like mobile phones far away from the office.
We found two studies on how a large public website is accessed with different devices. Cronk studied the use of the British government website gov.uk and found that mobile phones and tablets were used more in the evenings and weekends, while desktop/laptop PCs dominated the office hours . Nicholas et al compared mobile users with PC users of Europeana, the European fulltext cultural website. They found that mobile users were the fastest growing group and that their visits were different from PC user visits. Mobile visits were typically shorter, less interactive, and less content was viewed per visit. Mobile use peaked in the evenings and weekends. They found that tablet users behaved very much like PC users [ ].
We found the same pattern as Cronk  and Nicholas et al [ ]. PCs peaked during working hours, while mobile phones increased throughout the day and peaked around 10 pm. We also found that pageviews of patient leaflets followed a pattern in between the PCs and mobile phones. Patient leaflets showed a small peak during office hours and a larger peak in the evening. This indicates that patient leaflets are actually read by the general public, but they are probably also read or handed out by health personnel during office hours.
We are now redesigning The Library website, partly to make it more mobile friendly, and the findings in this study are being taken into consideration in this process. It was decided to keep the patient information leaflets, partly as a consequence of the findings in this study.
We assume our largest user group is health personnel and students, as intended. The Library is widely used among health personnel, and to some extent also by the public. Norwegian language content is more popular than English language content. We cannot conclude whether the high and increasing popularity of patient leaflets is caused by patients using The Library more than before.
Conflicts of Interest
All authors are employees of the Norwegian Knowledge Centre for the Health Services, the parent organization of the Norwegian Electronic Health Library.
- Nylenna M, Eiring Ø, Strand G, Røttingen JA. Wiring a nation: putting knowledge into action. Lancet 2010 Mar 20;375(9719):1048-1051. [CrossRef] [Medline]
- Tjensvoll K. National licensing for the Norwegian Electronic Health Library. Serials: The Journal for the Serials Community 2011 Mar 07;24(1):52 [FREE Full text] [CrossRef] [Medline]
- Wikipedia.: Wikimedia Proxy (statistics) URL: http://en.wikipedia.org/wiki/Proxy_(statistics) [accessed 2014-10-28] [WebCite Cache]
- TechTerms. 2007 Oct 26. Page View Definition URL: http://www.techterms.com/definition/pageview [accessed 2014-10-28] [WebCite Cache]
- Lewis A. Search Engine Watch. 2013 Jul 09. How to use the Google Analytics new vs returning report URL: http://searchenginewatch.com/article/2280159/How-to-Use-the-Google-Analytics-New-vs-Returning-Report [accessed 2014-10-28] [WebCite Cache]
- Google Support. Bounce rate: Learn what a bounce rate is, and how to improve it URL: https://support.google.com/analytics/answer/1009409?hl=en [accessed 2014-10-28] [WebCite Cache]
- Google Support. How sessions are calculated in Analytics URL: http://support.google.com/analytics/answer/2731565?hl=en [accessed 2014-10-28] [WebCite Cache]
- Project COUNTER. COUNTER: Counting Online Usage of NeTworked Electronic Resources URL: http://www.projectcounter.org/ [accessed 2014-10-28] [WebCite Cache]
- Gulbrandsen P, Schroeder TV, Milerad J, Nylenna M. Paper or screen, mother tongue or English: which is better? A randomized trial. JAMA 2002 Jun 5;287(21):2851-2853 [FREE Full text] [CrossRef] [Medline]
- Thompson ML. Characteristics of information resources preferred by primary care physicians. Bull Med Libr Assoc 1997 Apr;85(2):187-192 [FREE Full text] [Medline]
- Nicholas D, Clark D, Rowlands I, Jamali HR. Information on the go: a case study of Europeana mobile users. J Am Soc Inf Sci Technol 2013 May 10;64(7):1311-1322. [CrossRef]
- Cronk P. Inside GOV.UK.: Government Digital Service; 2013 Nov 14. Desktop, tablet and mobile use by time of day URL: http://insidegovuk.blog.gov.uk/2013/11/14/desktop-tablet-and-mobile-use-by-time-of-day/ [accessed 2014-10-28] [WebCite Cache]
- SurveyGizmo. 2010 Jan 28. Survey response rates URL: http://www.surveygizmo.com/survey-blog/survey-response-rates/ [accessed 2014-10-28] [WebCite Cache]
- Wikipedia.: Wikimedia Foundation PageRank URL: http://en.wikipedia.org/wiki/PageRank [accessed 2014-10-28] [WebCite Cache]
- Check PageRank. URL: http://checkpagerank.net/ [accessed 2014-10-28] [WebCite Cache]
- Longo DR, Woolf SH. Rethinking the information priorities of patients. JAMA 2014 May 14;311(18):1857-1858. [CrossRef] [Medline]
- Giustini D. How Google is changing medicine. BMJ 2005 Dec 22;331:1487 [FREE Full text] [CrossRef] [Medline]
|COUNTER: Counting Online Usage of Networked Electronic Resources|
|IP: Internet protocol|
|PC: personal computer|
Edited by G Eysenbach; submitted 29.08.14; peer-reviewed by M Rethlefsen; comments to author 19.09.14; revised version received 09.10.14; accepted 19.10.14; published 18.11.14
©Runar Eggen, Kjell Tjensvoll, Magne Nylenna. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 18.11.2014.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.