Published on 13.05.20 in Vol 9, No 5 (2020): May
Evaluation and Dissemination of a Checklist to Improve Implementation of Work Environment Initiatives in the Eldercare Sector: Protocol for a Prospective Observational Study
Background: To measure sustainable improvements in the work environment, a flexible and highly responsive tool is needed that will give important focus to the implementation process. A digital checklist was developed in collaboration with key stakeholders to document the implementation of changes in eldercare sector workplaces.
Objective: This paper describes the study protocol of a dissemination study that aims to examine when, why, and how the digital checklist is spread to the Danish eldercare sector following a national campaign particularly targeting nursing homes and home care.
Methods: This prospective observational study will use quantitative data from Google Analytics describing use of the checklist as documented website engagement, a survey among members in the largest union in the sector, information from a central business register, and monitoring of campaign activities. The evaluation will be guided by the five elements of the RE-AIM framework: reach, effectiveness, adoption, implementation, and maintenance.
Results: The study was approved in June 2016 and began in October 2018. The campaign that is the foundation for the evaluation began in 2017 and ended in 2018. However, the webpage where we collect data is still running. Results are expected in 2020.
Conclusions: This protocol provides a working example of how to evaluate dissemination of a checklist to improve implementation of work environment initiatives in the eldercare sector in Denmark. To our knowledge, implementation in a nationwide Danish work environment has not been previously undertaken. Given that the checklist is sector-specific for work environment initiatives and developed through systematic collaboration between research and practice, it is likely to have high utility and impact; however, the proposed evaluation will determine this. This study will advance dissemination research and, in particular, the evaluation of the impact of these types of studies. Finally, this study advances the field through digital tools that can be used for evaluation of dissemination efforts (eg, Google Analytics associated with website) in the context of a rigorous research design activity.
International Registered Report Identifier (IRRID): DERR1-10.2196/16039
JMIR Res Protoc 2020;9(5):e16039
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
Currently many countries are facing shortages of health care workers, and this is jeopardizing capacity to deliver residential care services . The shortage of health care workers is associated with high rates of sickness absence, turnover, and early retirement among this professional group [ ]. It is well known that high physical and psychosocial work demands are important risk factors for long-term sickness absence, turnover, and early retirement from the eldercare sector [ , ]. Thus, increasing and sustaining the eldercare workforce demands urgent attention. Several initiatives, including complex multilevel interventions, have been introduced to improve the work environment of eldercare workers during the past decades in Denmark and other countries [ - ]. However, despite availability of extensive research and policy efforts, employees in the eldercare sector have only experienced limited improvements in the work environment.
A reason for the lack of improvement in the work environment in this setting may be that the eldercare sector is continuously changing through care regimes, political reforms, and high turnover rates . Such changes affect the stability of the organizations and may also challenge implementation of new knowledge and work environment policies or initiatives [ ]. The work environment can be considered a moving target with continuously changing terms and starting points [ ]. Therefore, interventions for such a moving target must be flexible and highly responsive to the changing needs in the work environment and facilitate the implementation process [ ]. Therefore, we decided to collaborate with stakeholders from the target group to collect information about how to efficiently implement changes in the work environment in the eldercare sector [ ]. The knowledge base was condensed to the Hitting the Moving Target framework, which summarized 11 components to consider in order to succeed with implementation [ ]. The framework targets both managers and employees in the eldercare sector. From recommendations from the target group, the 11 components from the Hitting the Moving Target framework were used to develop an implementation tool in the form of a digital checklist containing the 11 concepts.
Another reason for the lack of improvement in the work environment in this sector may be the challenge of translation of policies and research knowledge into practice . Many factors can influence whether the translation of research knowledge into practice is successful and whether policies or evidence-based practices are accepted and used by the target users [ ]. Dissemination of research findings is an important step to bridge the gap between research and practice. Effective dissemination strategies include formative research to customize dissemination strategies to fit audience needs and preferences [ ]. Distribution strategies should focus on ensuring that messages and materials from research reach intended audiences by use of multicomponent dissemination strategies (eg, mailings, websites, publications, webinar or in-person presentations, interpersonal connections, and mass media, among others) [ , ]. To be most effective, distribution should engage the channels that intended audiences already trust and access for information [ ]. Previously, national campaigns have been used to reach a large proportion of the population for reducing musculoskeletal disorders [ , ]. But such campaigns are expensive and need to be well planned. In addition, the packaging and communication used to disseminate evidence-based knowledge determines the dissemination success [ ]. Different approaches have been attempted to overcome this challenge. For example, in Sweden, guidelines aimed at improving the psychosocial work environment have been coproduced with practitioners [ ]. Still, there is a shortage of knowledge on how to optimize dissemination, and the reach and effect of such initiatives is difficult to evaluate and rarely investigated.
This paper is the dissemination protocol that examines when, why, and how the checklist is spread to the Danish eldercare sector, in particular nursing homes and homecare. The protocol presents both the process of developing the digital checklist, planning the dissemination strategy (ie, the communication of the checklist through a national campaign), and the evaluation plan. Specifically, the protocol aims to investigate the adoption, reach, implementation, maintenance, and effectiveness of the checklist after a national campaign to improve implementation of work environment initiatives among eldercare workers in Denmark. The following four research questions will be investigated:
- How many Danish eldercare workplaces use the checklist and what characterizes those who do from those who don’t? (adoption)
- Across Danish eldercare workplaces, what proportion of eldercare workers know about the campaign and what characterizes those who do from those who don’t? (reach and representativeness)
- Among the users, how is the checklist used and for what purposes? (implementation and maintenance)
- Is the work environment practice improved among users of the checklist compared with nonusers? (effectiveness)
This is a prospective observational study using a range of quantitative data collection approaches to accomplish the study aims. The study is described according to the Standards for Reporting Implementation Studies (StaRI) statement.
Study Setting and Population
The study setting is the eldercare sector in Denmark, and more specifically, selected nursing homes and homecare settings. In Denmark, there are approximately 5000 workplaces within the eldercare sector that employ about 100,000 eldercare workers.
Ethics Approval and Consent to Participate
According to Danish law as defined in Committee Act §2 and §1, the study described should not be further reported to the local ethics committee. The data use is approved by the Danish Data Protection Agency. According to Danish law, questionnaire- and register-based studies do not need approval by ethical and scientific committees or informed consent by participants.
Development of the Digital Checklist (Dissemination Object)
The translation from Hitting the Moving Target framework to a digital checklist was conducted in close collaboration with relevant partners. We involved two groups of stakeholders: (1) a practice-based research team (PBR) team and (2) a municipality team. They functioned as codevelopers with a central role in operationalizing the 11 implementation components from the Hitting the Moving Target framework into the checklist. The PBR team group consisted of 10 stakeholders with representatives from the largest union in the nursing home sector, Local Government Denmark, which is the central organization of all Danish municipalities, the Danish Work Environment Authority, the Knowledge Center for Work Environment, and the Sector-Specific Work Environment Community Organization for Public and Welfare workplaces. The municipality-based stakeholder group consisted of 5 work environment consultants employed in 4 different Danish municipalities, and this group helped us in the development, especially in making the checklist adaptable to existing practices and structures. As an example, the municipality-based stakeholder group helped us decide that the checklist should be directed at the Occupational Health and Safety (OHS) groups at the workplaces, which are groups consisting of both managers and employees with specific tasks related to OHS.
The feasibility of the checklist was assessed through cognitive interviews with a range of intended beneficiaries, both employees and managers in the eldercare sector (primarily those involved in OHS groups). The prototype of the Web-based checklist was then included in 40 tests conducted with employees, mainly OHS representatives at different nursing homes, to fit the digital solution to the user needs and make sure they could use the checklist as intended. The final 11 checkpoints can be seen in.
Content and Use of the Web-Based Checklist
The checklist is an interactive digital platform that can be assessed through a website. Users (primarily the OHS groups) can use it in their work environment practice when implementing new routines, projects, or initiatives (termed actions in the checklist) to improve the work environment. The steps in the process of using the checklist are described in. The user has to log in with their affiliation and then define the action (free of choice) they want to implement. The next step is to go through the checklist and pick one concept in the checklist that they want to focus on. Working with defined actions to support implementation is a task that the OHS groups already have in their portfolio to maintain a good work environment. After having gone through all the points of the checklist they get a result that indicates how many facilitators for implementation are in place, and they get information about the concept to focus on. It is possible to print a diploma that shows this and place it visibly at the workplace and automatically send an email to communicate the actions to others (ie, coworkers and upper management). In addition, it is possible to get tips for implementation on the website [ ].
The aim with the campaign was to increase awareness of the existence of the checklist. The campaign was developed with our PBR and municipality teams. An initial workshop was held to identify central target users when disseminating the checklist to the eldercare sector. Through persona analysis, we characterized all potential users working with or within the eldercare sector and made a description of their role in the work environment, and thereby we were able to describe who would be the most relevant target group for the checklist. This workshop highlighted the importance of targeting the checklist directly to those who work within the work environment (eg, the OHS groups) at the workplaces. We planned the campaign to be nationwide and primarily driven through network on social media, websites, through newsletters, magazines, conferences, letters, and a campaign film by the researchers and stakeholders (especially the PBR team). The campaign was planned to run for 1 year (from September 2017 to September 2018).
Campaign Materials and Methods
The campaign materials included paper and digital elements. Paper elements included printed versions of the checklist such as postcards and letters, which were sent to the administrative departments of all municipalities and all identified home care units and nursing homes in Denmark informing about the new checklist. The digital elements included a short campaign movie, small instruction movies, and newsletters. We produced a campaign movie aimed at increasing awareness about the new checklist. The movie was distributed via social media (eg, Facebook, LinkedIn, and Twitter), and we aimed to create awareness of the newly developed checklist among the entire target population in the eldercare sector. Additionally, we produced small instruction movies showing how to use the checklist. Both were uploaded on the same webpage .
Presentations and Dissemination Partners
The researchers presented the checklist whenever possible at conferences and workshops where the target group was present. Furthermore, researchers presented the checklist for consultants working in this industry—for example, to consultants employed at a central position in the municipality or physiotherapists or occupational therapists working at one or more nursing homes. We focused on building a network with central work environment representatives in the municipalities of Denmark, emphasizing the close collaboration between researchers and practitioners. This enabled exchange of information from the municipalities about the work environment and for us to inform about the checklist and how to use it.
The stakeholders (especially our PBR team) functioned as dissemination partners. They referred to the checklist on their respective websites, Facebook pages, in magazines, in newsletters, etc. The PBR team also functioned as ambassadors for disseminating the checklist (eg, when visiting the nursing homes). Furthermore, the entire corps of inspectors from the Danish Work Environment Authority (N-90), who visit and inspect workplaces in Denmark regarding the work environment, were trained in the checklist and informed about the checklist before going out to the workplaces. Additionally, we established a partnership with work environment consultants connected to the eldercare sector. They were informed about the checklist when relevant—for example, when talking with the OHS groups about potential initiatives for improving the work environment and how to succeed with their initiatives.
This study aims to investigate the dissemination and effectiveness on work environment improvements of a checklist to improve implementation of work environment initiatives among eldercare workers in Denmark. A commonly used framework in the evaluation of implementation research is the RE-AIM (reach, efficacy/effectiveness, adoption, implementation, and maintenance) framework [- ]. RE-AIM guides areas to consider when seeking to evaluate the potential public health impact of a program and consists of the 5 dimensions, reach, efficacy/effectiveness, adoption, implementation, and maintenance.
In this dissemination study, the indicators of reach, implementation, and maintenance are somewhat overlapping and may be used as indicators of more than one evaluation component. This is because the intervention can be considered as both the campaign, the checklist, and the action plan at the workplace and thus reach and implementation can occur at several levels (ie, societal, organizational, and individual level), and we aim to describe all levels in this study (see).
We will use multiple data sources including data on campaign activities, data from the Central Business Register (CVR), data from Google Analytics, data regarding activity on the project website (use of the checklist), and data from a survey among members in the largest union in the industry. In, the campaign (activities) and the data collection are illustrated.
Central Business Register
The CVR contains information about all registered workplaces in Denmark. Factors available in the CVR are size (number of employees in intervals), type of workplace (nursing home, home care, hospital, etc), age (start-up date of company/workplace), and geographical position of the workplace (ie, municipality).
The website  was associated with a Google Analytics account. Information on day-to-day visits and month-by-month geographical data on city level will be downloaded.
Online Checklist Website
From the backend of the website, we will collect user-specific information from visitors, defined actions to improve the work environment, and answers to the checklist.
The largest trade union for eldercare workers in Denmark organizes approximately 180,000 members primarily in the public sector. The union has more than 90,000 employed members engaged in social and health sector. Members belonging to the social and health sector can voluntarily sign up as a survey member to receive a questionnaire 4 to 6 times a year about work environments and other work-related topics. Union members can register and drop out as they want. The union survey is sent to approximately 7500 survey members each time.
The data sources above will be used to answer the research questions according to the RE-AIM framework.shows each of the components of the RE-AIM framework and their definition. In addition, the appendix shows the specific research question(s) related to each component and their respective data source and operationalization.
Using descriptive statistics, we will compare adopting (workplace that use the checklist) and nonadopting workplaces in terms of the number of employees, type of workplaces, and age and geographical position of the workplaces. To report reach, we will use descriptive statistics to compare gender, age, manager, position of trust, employer, and workplace of respondents with knowledge of the campaign and those not reached by the campaign. In addition, we will test for differences in adopters and nonadopters and in reached and nonreached, respectively, by t test or analysis of variance.
Effectiveness will be evaluated by comparing the change in the prioritization of the work environment efforts experienced by respondents from before the campaign until after the campaign. Analyses regarding the effectiveness will be performed after 12 months of campaign by means of analysis of covariance comparing employees with knowledge of the campaign and those without knowledge of the campaign. We will test whether it is relevant to control for confounders such as age and gender.
The study started in September 2017 with the 1-year campaign. The main data collection was completed by September 2018, but data collection through the website is ongoing. Dissemination of results is expected in early 2020.
This paper presents the protocol for the evaluation and dissemination of a checklist to improve implementation of work environment initiatives in the eldercare sector in Denmark. To our knowledge, this has not previously been undertaken in a Danish work environment context, and it is the first nationwide sector-specific checklist for implementing work environment initiatives. Also, evaluation of which checklist points have been most frequently (or rarely) ticked may give valuable information regarding which implementation challenges workplaces easily handle and which challenges they postpone.
The project will expand the understanding of the determinants of implementation and dissemination success and failure. New knowledge will be generated on industry-specific dissemination, which potentially can be used in industries other than nursing homes. It is hoped that the planned fine-grained evaluation outcomes will provide practical information that will give managers and government agencies well-tested tools and comprehensive process insights that with enable interventions to be more quickly and more effectively implemented to generate improved work environments and alleviate some of the shortages in the eldercare workforce.
Strengths and Limitations
A limitation in the evaluation of the impact of the checklist is that use is measured only through the website and not the potential use outside this setting. For example, the postcards with a print of the checklist given to workplaces can be used instead of logging in to the website checklist. Therefore, the evaluation will likely not cover the full dissemination and may actually underrepresent uptake of the checklist. Further, we have observed that some employees in eldercare may have limited access to computers during work hours. On the other hand, tablets and laptops are often present at the workplaces and Web-based work environment and safety systems are becoming a more regular practice. A limitation to the study is that we cannot generalize beyond eldercare. It is also a limitation that the data are self-reported and contained nonvalidated information. However, a strength is the use of multiple data sources and new digital data sources.
To have a wide and lasting impact, tools often need to be adapted to or reconstructed within practice settings. Consequently, to maximize the utility of our model and the resulting implementation checklist, extensive and systematic involvement of stakeholders was undertaken. Importantly, we also used this codesign process to develop the dissemination strategy. We intended that the ownership generated in key stakeholders would assist to ensure the development of a fit-for-purpose checklist and dissemination strategy and therefore generate strong positive outcomes. There is an ever-increasing demand for research to make a practical difference in policy and in society in general, but many products and programs fail deliver such outcomes. Given the methods used to develop our interventions and the approach to evaluation, we expect that the data generated from this study will generate novel insights into how to generate real-world impacts at scale. In conclusion, this study will advance dissemination research and, in particular, the evaluation of the impact of these types of studies. Finally, this study advances the field through digital tools that can be used for evaluation of dissemination efforts (eg, Google Analytics associated with website) in the context of a rigorous research design activity.
The study is funded by the Danish Working Environment Research Fund (33-2016-03 20165101121). The funders had no role in study design, data collection, and analysis; decision to publish; or preparation of the manuscript.
CNR acquired funding for the project, designed the study, drafted the first version of the manuscript, and wrote the final version of the manuscript. HHJ designed the study, participated in discussions around the study, and critically revised the manuscript. AKL, PKM, IJ, RO, LL, and LK participated in discussions around the study and critically revised the manuscript. MBJ acquired funding for the project, designed the study, participated in discussions around the study, and critically revised the manuscript. All authors have read and approved the manuscript.
Conflicts of Interest
Overview of evaluation and operationalization of the components in the RE-AIM framework: adoption, reach, implementation, effectiveness, and maintenance.DOCX File , 15 KB
Funding documentation. Declaration of translation into English from Danish.DOCX File , 37 KB
- Campbell JD, Buchan J, Pozo-Martin F, Guerra AM, Leone C, Siyam A, et al. A Universal Truth: No Health Without a Workforce. Geneva: Global Health Workforce Alliance and World Health Organization; 2013. URL: https://www.who.int/workforcealliance/knowledge/resources/GHWA-a_universal_truth_report.pdf?ua=1 [accessed 2020-03-11]
- Jensen LD, Ryom PK, Christensen MV, Andersen JH. Differences in risk factors for voluntary early retirement and disability pension: a 15-year follow-up in a cohort of nurses’ aides. BMJ Open 2012 Nov 12;2(6):e000991. [CrossRef]
- Clausen T, Borg V. Do positive work-related states mediate the association between psychosocial work characteristics and turnover? A longitudinal analysis. Int J Stress Manag 2010;17(4):308-324. [CrossRef]
- Andersen LL, Clausen T, Mortensen OS, Burr H, Holtermann A. A prospective cohort study on musculoskeletal risk factors for long-term sickness absence among healthcare workers in eldercare. Int Arch Occup Environ Health 2012 Aug;85(6):615-622. [CrossRef] [Medline]
- Larsen AK, Holtermann A, Mortensen OS, Punnett L, Rod MH, Jørgensen MB. Organizing workplace health literacy to reduce musculoskeletal pain and consequences. BMC Nurs 2015;14:46 [FREE Full text] [CrossRef] [Medline]
- Aust B, Rugulies R, Finken A, Jensen C. When workplace interventions lead to negative effects: learning from failures. Scand J Public Health 2010 Feb;38(3 Suppl):106-119. [CrossRef] [Medline]
- Nielsen K. Review article: how can we make organizational interventions work? Employees and line managers as actively crafting interventions. Human Relations 2013 Apr 23;66(8):1029-1050. [CrossRef]
- Rasmussen CDN, Holtermann A, Mortensen OS, Søgaard K, Jørgensen MB. Prevention of low back pain and its consequences among nurses' aides in elderly care: a stepped-wedge multi-faceted cluster-randomized controlled trial. BMC Public Health 2013 Nov 21;13:1088 [FREE Full text] [CrossRef] [Medline]
- Olsen O, Albertsen K, Nielsen ML, Poulsen KB, Gron SMF, Brunnberg HL. Workplace restructurings in intervention studies: a challenge for design, analysis and interpretation. BMC Med Res Methodol 2008 Jun 13;8:39 [FREE Full text] [CrossRef] [Medline]
- Højberg H, Rasmussen C, Osborne R, Jørgensen M. Identifying a practice-based implementation framework for sustainable interventions for improving the evolving working environment: hitting the Moving Target Framework. Appl Ergon 2018 Feb;67:170-177 [FREE Full text] [CrossRef] [Medline]
- Dugan AG, Punnett L. Dissemination and implementation research for occupational safety and health. Occup Health Sci 2017 Dec;1(1-2):29-45 [FREE Full text] [CrossRef] [Medline]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
- Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond implementation strategies: classifying the full range of strategies used in implementation science and practice. Implement Sci 2017 Dec 03;12(1):125 [FREE Full text] [CrossRef] [Medline]
- McCormack L, Sheridan S, Lewis M, Boudewyns V, Melvin CL, Kistler C, et al. Communication and dissemination strategies to facilitate the use of health-related evidence. Evid Rep Technol Assess (Full Rep) 2013 Nov(213):1-520. [CrossRef] [Medline]
- Buchbinder R, Jolley D. Effects of a media campaign on back beliefs is sustained 3 years after its cessation. Spine (Phila Pa 1976) 2005 Jun 01;30(11):1323-1330. [CrossRef] [Medline]
- Andersen LL, Geisle N, Knudsen B. Can beliefs about musculoskeletal pain and work be changed at the national level? Prospective evaluation of the Danish national Job & Body campaign. Scand J Work Environ Health 2018 Jan 01;44(1):25-36. [CrossRef] [Medline]
- Chaudoir SR, Fisher JD, Simoni JM. Understanding HIV disclosure: a review and application of the Disclosure Processes Model. Soc Sci Med 2011 May;72(10):1618-1629 [FREE Full text] [CrossRef] [Medline]
- Kwak L, Wåhlin C, Stigmar K, Jensen I. Developing a practice guideline for the occupational health services by using a community of practice approach: a process evaluation of the development process. BMC Public Health 2017 Jan 18;17(1):89 [FREE Full text] [CrossRef] [Medline]
- MEDvirkNU. URL: https://medvirknu.dk/ [accessed 2020-03-11]
- Om MEDvirk. URL: https://medvirknu.dk/om-medvirk [accessed 2020-03-11]
- Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med 2007 May;32(5 Suppl):S119-S126. [CrossRef] [Medline]
- Van Acker R, De Bourdeaudhuij I, De Cocker K, Klesges LM, Cardon G. The impact of disseminating the whole-community project '10,000 Steps': a RE-AIM analysis. BMC Public Health 2011 Jan 4;11(1):3. [CrossRef]
- Cocker KD, Cardon G, Bennie JA, Kolbe-Alexander T, Meester FD, Vandelanotte C. From evidence-based research to practice-based evidence: disseminating a web-based computer-tailored workplace sitting intervention through a health promotion organisation. Int J Environ Res Public Health 2018 May 22;15(5) [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [Medline]
|CVR: Central Business Register|
|PBR: practice-based research|
|OHS: Occupational Health and Safety|
|RE-AIM: reach, efficacy/effectiveness, adoption, implementation, and maintenance|
|StaRI: Standards for Reporting Implementation Studies|
Edited by G Eysenbach; submitted 28.08.19; peer-reviewed by D Eerd, M Hong; comments to author 19.10.19; revised version received 13.01.20; accepted 04.02.20; published 13.05.20
©Charlotte Diana Nørregaard Rasmussen, Helene Højberg, Anne Konring Larsen, Pernille Kold Munch, Richard Osborne, Lydia Kwak, Irene Jensen, Laura Linnan, Marie Birk Jørgensen. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 13.05.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.