This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Advance care planning (ACP) promotes conversations about future health care needs, enacted if a person is incapable of making decisions at end-of-life that may be communicated through written documentation such as advance care directives. To meet the needs of multicultural and multifaith populations in Australia, an advance care planning website, ACP
The purpose of this paper was to utilize the context, input, process, and product (CIPP) framework to conduct a formative and summative evaluation of ACP
The CIPP framework was used, which revolves around 4 aspects of evaluation: context, input, process, and product. Context: health professionals’ solutions for the website were determined through thematic analysis of exploratory key stakeholder interviews. Included religions were determined through an environmental scan, Australian population statistics, and documentary analysis of project steering committee meeting minutes.
Input: Project implementation and challenges were examined through documentary analysis of project protocols and meeting minutes.
Process: To ensure religion-specific content was accurate and appropriate, a website prototype was built with content review and functionality testing by representatives from religious and cultural organizations and other interested health care organizations who completed a Web-based survey.
Product: Website analytics were used to report utilization, and stakeholder perceptions were captured through interviews and a website survey.
Context: A total of 16 key stakeholder health professional (7 general practitioners, 2 primary health nurses, and 7 palliative care nurses) interviews were analyzed. Website solutions included religious and cultural information, communication ideas, legal information, downloadable content, and Web-based accessibility. Christian and non-Christian faiths were to be included in the religion-specific content.
Input: Difficulties gaining consensus on religion-specific content were overcome by further state and national religious organizations providing feedback.
Process: A total of 37 content reviewers included representatives of religious and cultural organizations (n=29), health care (n=5), and community organizations (n=3). The majority strongly agree or agree that the content used appropriate language and tone (92%, 34/37), would support health professionals (89%, 33/37), and was accurate (83%, 24/29).
Product: Resource usage within the first 9 months was 12,957 page views in 4260 sessions; majority were (83.45%, 3555/4260) from Australia. A total of 107 Australian-based users completed the website survey; most felt information was accurate (77.6%, 83/107), easy to understand (82.2%, 88/107), useful (86.0%, 92/107), and appropriate (86.0%, 92/107). A total of 20 nurses (general practice n=10, palliative care n=8, and both disciplines n=2) participated in stakeholder interviews. Qualitative findings indicated overall positivity in relation to accessibility, functionality, usefulness, design, and increased knowledge of advance care planning. Recommended improvements included shortened content, a comparable website for patients and families, and multilingual translations.
The CIPP framework was effectively applied to evaluate the development and end product of an advance care planning website.Although overall findings were positive, further advance care planning website development should consider the recommendations derived from this study.
Integration of advance care planning (ACP) is associated with improved quality of life, better adherence to patients’ wishes, and reduced hospitalizations [
Although national ACP frameworks and guidelines promote inclusion of cultural and religious beliefs in end-of-life decision making [
To address this gap, an Australian-based website, ACP
Systematic evaluation is required to assess the effectiveness, efficiency, and appropriateness of activities [
The purpose of this paper was to present a formative and summative evaluation of an ACP website, ACP
The CIPP framework consists of four evaluation types that are summarized below [
Context evaluation is used to judge and assess project needs, problems, assets, opportunities, and contextual conditions
Input evaluation is concerned with program planning through assessing budgets, procedural plans, feasibility, challenges, and targets
Process evaluation examines implementation of planned and actual processes
Product is concerned with examining program outcomes, how the program effectively addressed needs, and achieved its goals
The CIPP model enables evaluators to design specific core questions for each evaluation type that are relevant to the intended project to be assessed. On the basis of the ACP
A project protocol was developed inclusive of three main phases: (1) an exploratory study, (2) religious leader interviews that would be used to derive religion-specific website content, and (3) an evaluation. The study received ethical approval by a human research ethics committee (CHREC 02-05-10-15).
Project governance was established with a project steering committee (PSC) and project working group (PWG) convened. The PWG developed project and research aims, protocol, data collection, and management procedures and reported outcomes to the PSC. The PSC consisted of expert representatives from religious and cultural organizations, universities, palliative care, and nursing researchers and general practitioners (GPs) who were responsible for providing overall project oversight, approving protocols, procedures, recommending linkages for recruitment, and website feedback. Documentation resulting from these committees included PSC meeting minutes, the project protocol, email correspondence, and reports.
Overview of context, input, process, and product (CIPP) model for ACP
Type and questions | Rationale | Evaluation methods | |||
What were the recommended solutions of key stakeholders (Health professionals—general practitioners and nurses)? | It was important to understand the recommendations of potential website users (ie, key stakeholders) to tailor content to the intended audience | Thematic analysis of interviews with health professionals (exploratory study) | |||
What religious or cultural backgrounds were included in website content? | Appropriate consideration should be undertaken to ensure that the website contains religion-specific information representative of Australian-based groups, with integration of cultural information where relevant | Review of Australian cultural diversity data, conduct of an environmental scan of Australian religious and cultural ACP resources, and documentary analysis of meeting minutes | |||
What challenges were encountered during project implementation? | Creating a website integrating information representative of multiple religious and cultural groups requires an appropriate governance structure, timelines, and procedures | Documentary analysis of project protocol and meeting minutes | |||
What process was utilized to ensure that the website content supported ACP conversations within appropriate religious and cultural contexts? | It was important to ensure accuracy of different religion-specific content as this was a key project requirement | Development of website prototype, pretesting of functionality and content review by religious and cultural leaders. Analysis of religion-specific Web-based content review surveya | |||
How was the website used within the initial 9 months following release? | Measurement of website utilization is important to assess impact and inform strategies for improvements | Reporting of analytic data integrated in the website | |||
What were stakeholders’ (website users and nurses) perceptions of the website? | Obtaining stakeholder perceptions of website benefits, weaknesses, and suggestions for improvements to inform further development | Analysis of website user feedback surveya and thematic analysis of stakeholder (nurse) interviews |
aReported as per the
ACP
An environmental scan [
Given that ACP conversations can occur throughout the illness trajectory [
Key stakeholders—GPs and palliative care and general practice nurses (minimum qualification clinical nurse specialists) were invited to participate in the exploratory study. Following consent, semistructured interviews were conducted using a questionnaire based on the cultural awareness framework by Jirwe [
Findings from the environmental scan [
A prototype of the website was initially built by website developers, with functional and design specifications determined by the PWG and PSC. This enabled content review and functionality testing of a religion-specific search function embedded in the website, which allowed users to select a religion and denomination (if applicable) from a predetermined drop-down list and view content pertinent to that religion. To ensure accurate and appropriate representation of religious information, content underwent an extensive review process with religious and cultural leader interviewees, representatives from state and national religious and cultural organizations, and health care organizations in specialized end-of-life roles. Reviewers were invited to view the website prototype, in particular information about their religion (if applicable), and were then invited to participate in a closed, Web-based content review survey.
Although the Web-based content review survey obtained feedback about religion-specific content, a website feedback survey was used to examine stakeholder perceptions of the end product, the ACP
Both the Web-based content review survey and website feedback survey were based on the core program evaluation standards, utility, feasibility, propriety, and accuracy [
For the Web-based content review survey, questions were about accuracy of content, appropriateness to the intended audience, detail and depth, ease of reading, completeness of information, perceptions of utility, and least and most useful section. Questions were nonrandomized, located on one page, and two out of five total questions were compulsory. Answers to the least and most useful sections and comments were not compulsory.
For the website feedback survey, to examine users’ perceptions of the website, an open, voluntary, Web-based feedback survey using SurveyMonkey [
Google analytics [
General practice and palliative care nurses were invited to participate in the evaluation of ACP
Qualitative interview data (context and product evaluation) was examined using thematic analysis [
What are your experiences of having advance care planning (ACP) conversations with people from different religious and cultural backgrounds?
What do you think are some of the benefits of the website? [Ask to provide an example of how the website has been useful]
What are some of the weaknesses? [Ask to provide an example of any issues or concerns or difficulties]
How do you think ACP
How do you feel the website has contributed to your understanding of ACP in a culturally diverse context?
Would you recommend the resource to colleagues? If no, why not?
Are there any additional comments or questions you have and feel would be important for the evaluation and improvement of this resource?
A total of 16 health professionals (GPs: n=7, primary health nurses: n=2, and palliative care nurses: n=7) were included in the exploratory study analysis (participation rate 26%, 17/65). One participant’s interview was inaudible and therefore not included in the analysis. Characteristics are summarized in
All interviewed GPs and nurses had ACP experiences with people across the lifespan from different religious and cultural backgrounds; in rural, urban, community, residential, and hospital settings.
In providing a resource to support religiously and culturally appropriate ACP, health professionals stated the following solutions (N indicates nurse; GP indicates general practitioner):
Although health professionals asserted the importance of nonpresumptive individualized care in facilitating ACP, they expressed solutions for enhanced religious and cultural information provision, which encompassed:
An understanding of varied religious beliefs and related requirements, N5 stated:
I think if you had an easy guide to—like 4 or 5 dot points under some of the more common cultural or religious beliefs that would be helpful...
Religious or cultural information outlining communication ideas to introduce ACP and decision making with different religious groups. This would include generically safe words to use and ideas for how to explain medical and technical ACP-related terms in sensitive ways. N6 exclaimed:
It would be good to have something to support you and guide you in those conversation.
Preparatory information for dealing with potential reactions. This could include education with role plays and discussions. GP3 stated:
...you could have an education session that involved role playing...just doing examples of various ways in which there may be problems with communication or difficulties getting over certain stumbling blocks.
Legal information applicable to one’s work jurisdiction
Additional support contacts, for example, interpreter details
ACP multilingual resources for patients or families
Online, textual, and audio-visual; N7 stated:
...a video on iPad or computer that has a person from that culture speaking about how they approached advance care planning.
Ease of access was mentioned, with GP2 requesting:
...online resources available and easily accessible, and if you're not sure, you can just go and have a look at those resources. That would be quite handy.
Link-in with well-known palliative care websites
Easy to follow downloadable resources guides
The environmental scan identified seven Australian-based ACP websites and seven ACP informational booklets with cultural and religious information representative of Aboriginal and Torres Strait Islander (n=5), Sikh (n=1), and Italian communities (n=1) [
The major project challenge related to difficulties in recruiting GPs to the exploratory study. Further recruitment strategies were employed, including liaison with primary health care networks to assist in recruitment and reimbursing providers operating solely under fee-for-service.
It was determined by the PSC that religion-specific content would need to incorporate heterogeneity within religions. The strategy to minimize conflicting information was attempting to interview at least two leaders from these groups so that dichotomist views could be investigated. Despite this, there were some issues with differences in beliefs within certain religions. State and national religious and cultural organizations were called upon to review content to ensure accuracy.
ACP
Characteristics of interviewed health professionals.
Characteristics | n (%) or median years (range) | |
General practitioner | 7 (44) | |
Primary health nurse | 2 (12) | |
Palliative care nurse | 7 (44) | |
Health professional experience, median years (range) | 9.5 (1.5-40) | |
Male | 8 (50) | |
Age, median years (range) | 51 (38-67) | |
Australia | 11 (69) | |
Overseas | 5 (31) | |
If born overseas, time living in Australia, median years (range) | 29 (9-62) | |
Christian | 7 (44) | |
None | 8 (50) | |
Hindu | 1 (6) | |
Strongly disagree or disagree | 6 (37) | |
Neutral | 3 (19) | |
Strongly agree or agree | 7 (44) |
Religion-specific search function.
Religion-specific information.
A total of 37 individuals (participation rate 67%, 37/55 [survey responders/invited participants]), representative of religious and cultural organizations (n=29) and members of other interested organizations (health care n=5, community organization n=3), reviewed the website content from February 17, 2016 to April 25, 2016 and completed the Web-based survey (
For example, interviewed religious and cultural leaders spoke about practices and rituals after death; this was not included in the initial content for inclusion in the website given that the website was focused on ACP discussions. Several reviewers of Christian and non-Christian faiths indicated it was necessary, as people may request postdeath rituals as part of the planning process; hence, this was written, and sections were reviewed. Content refinement was completed within the allocated time frame of 3 months, and no project delays occurred.
Quick Navigation.
Religion-specific content reviewer responses (n=37).
Data from Google analytics indicated that from September 1, 2016 to May 14, 2017, there were 12,957 page views and 4260 sessions by 2920 users. Of this, 68.33%, (2911/4260) were new sessions and bounce rate, which indicates people visiting a page, and leaving without further exploration was 40.40%, (1721/4260). Most sessions were from Australia (83.45%, 3555/4260), with a bounce rate of 40.84%, 1452/3555). Most Australian users resided in the states of Victoria (46.81%, 1664/3555) and New South Wales (27.99%, 995/3555). Other countries where the website was used included the United States (5.19%, 221/4260, bounce rate 65.16%, 144/221) and Russia (3.71%, 158/4260, bounce rate 6.96%, 11/158). Users spent an average of 3.21 min on ACP
From March 1, 2017 to May 14, 2017, 107 Australian-based website users completed the website evaluation survey (10% participation rate [participating users/unique site visitors determined by Internet protocol address]), of which the majority (88.8%, 95/107) indicated that they were health professionals. Of those that were health professionals, approximately 57% (49/86) were nurses, 14% (12/86) allied health, 14% (12/86) GPs, 5% (4/86) medical specialists, and 10.5% (9/86) indicated other (ie, practice managers, students, and program coordinators).
The majority of responders resided in Victoria (n=53), found out about the website via email (n=35), and explored the home page (n=52) and religious and cultural resources (n=49). Survey responders’ perceptions of the information on the website (
Interviews were conducted with 20 registered nurses (NS indicating nurse stakeholder; working in palliative care n=8, general practice n=10, and across both disciplines n=2). Characteristics are summarized in
All except NS6 were experienced in offering ACP, and five nurse stakeholders (NS11, NS12, NS9, NS14, and NS26) also described ACP education roles in health care teams.
Thematic findings follow.
Participants were complimentary about the ACP
...it took me a little bit of time and I’m talking minutes rather than hours to actually work out how the website worked.
Specific appealing design features mentioned included no login required, responsive design for tablet and mobile phone use enabling access on homecare visits, (religion-specific) search function that avoided the need for scrolling, and downloadable information. Comments included the following:
For a lot of us in primary care we’re doing home visits so you’re out there with a laptop and you can flag stuff and print it off and send it out to people and that kind of stuff
I think it’s set out...clear enough that you don’t get lost within it...
Web-based survey responders’ background data (ACP: advance care planning).
Survey questions | n | |
New South Wales | 13 | |
Queensland | 10 | |
South Australia | 18 | |
Tasmania | 5 | |
Victoria | 53 | |
Western Australia | 8 | |
Search engine (ie, Google) | 4 | |
Other website | 7 | |
35 | ||
General practitioner office or staff | 4 | |
Hospital | 10 | |
Place of worship | 1 | |
Electronic newsletter | 7 | |
Printed media | 22 | |
Professional association | 14 | |
Other | 18 | |
Home page | 52 | |
Religion-specific content | 20 | |
About this website | 26 | |
ACP discussion scripts | 29 | |
ACP law | 28 | |
General information | 40 | |
External ACP resources | 20 | |
ACP videos | 16 | |
Religious and cultural resources | 49 | |
Religious and cultural events | 12 |
aAble to indicate more than one answer for the question.
Web-based survey respondents' perceptions of website information (n=107).
Web-based survey respondents' experiences with using the website (n=107). ACP: advance care planning.
Characteristics of interviewed nurses.
Characteristics | n (%) or median years (range) | |
Palliative care | 8 (40) | |
General practice | 10 (50) | |
Both | 2 (10) | |
Nursing experience, median years (range) | 29 (5-44) | |
Female | 18 (90) | |
Age, median years (range) | 52 (32-63) | |
Australia | 12 (60) | |
Overseas | 8 (40) | |
If born overseas, time living in Australia, median years (range) | 19 (5-38) | |
Christian | 12 (60) | |
None | 6 (30) | |
Mixed religions | 2(10)a | |
Strongly disagree or disagree | 5 (25) | |
Neutral | 3 (15) | |
Strongly agree or agree | 12 (60) |
aMore than one religion stated by individuals.
Participants commented on the comprehendible, concise, sufficient, and manageable information; and links to relevant sites. NS25 stated the following:
If I had someone culturally diverse and I could just click on that (home page) and get into the link straight away without wasting hours and searching. I could get the accurate information for whatever I needed. I really liked that.
Individuals added that it addressed a knowledge gap or was a good addition to existing cultural care resources. Especially agreeable were consistent informational headers across religions; multimedia (textual and video) and multilingual (video) learning options.
Many commended specific information provided, including different EoLC procedures associated with varied traditions, acknowledgment of two main Buddhism streams, description of Ramadan, links to state-specific legal information, interpreter contact details, religious festival dates, scripts on guiding ACP conversations, suggestions for approaching difficult situations, and advice that ACP
Although some participants had not viewed the entire ACP
I think it’s assisting me in understanding the sensitivities that come with different groups and that we need to be respectful and sensitive in our conversations.
My intention is for us to use the information to frame our initial contact a little bit differently. At the moment, we tend to pretty much say the same script to everybody, which in light to this kind of information may not be the best way to go.
Many found ACP
I would use it at the drop of a hat for any sort of unfamiliar territory.
All had already used and/or recommended ACP
I have used it for educational purposes for myself and directed people towards it as a resource...in that way it has helped me.
Many also expected that the website would enable improved staff-client relationships through extending health professionals’ cultural ACP competency. Furthermore, occasional participants considered the website could potentially assist volunteers, patients, and families.
Four participants could not think of ways to improve the website, and some offered minor design critiques, for example, on menu visuals. Some also commented on omission of materials beyond the website’s remit, for example, advance directive forms, or content which were already included, for example, an educational video on ACP with indigenous Australians. A number also described technical issues related to Web-based links or Web-enabled personal devices.
Suggested technical website improvements included increased search and home page button visibility and reduced steps (clicks) to reach Christian pages. Two participants also believed that information presented on Aboriginal and Torres Strait Islander peoples could be more prominent.
Almost all the participants thought that the website had enough information; however, individuals suggested additional information was needed on minority religions such as scientology and how to approach multicultural families before commencing ACP conversations. One who lived in an area with “huge” Chinese and Vietnamese populations queried whether information on these groups was also needed. Additionally, individuals recommended an advanced directive example, interpretations of complex legal-based ACP information, comparable websites for patients and families, and a related online chat service. Also suggested was further clarity on ACP terms through a larger glossary and acknowledging that different terms in different Australian states can refer to the same thing, as illustrated in the following quote:
Quite often when somebody is talking to you about power of attorney they’ll also talk to you about power of enduring guardianship...The medical power of attorney and some of those languages cross over.
Further recommendations included increasing content to include a “frequently asked questions” page and further condensing each religion’s information into dot points. Extending comprehension of website content to health professionals without English as a first language was also recommended through presenting more information visually and multilingual translations. NS7 stated the following:
I certainly know Doctors who are practicing in this country who speak English well enough to practice the mechanics of medicine but don’t speak English well and confidently enough to have sensitive conversations with people about advance care planning...Medical practitioners might actually benefit from...having it written in their own language.
This study demonstrates that the CIPP evaluation model can be applied to conduct a formative and summative evaluation of an ACP website development and end product. In terms of website development, health professionals provided the following solutions: religious or cultural information guides, communication examples, education featuring role-plays, legal information, additional support contact details (ie, interpreters), and multilingual ACP resources. Recommended design and functionality included online accessibility, audio-visuals, linkages with existing websites, and downloadable content. Most of the requested content elements and design features were integrated into the ACP
Online resources identified through the environmental scan were linked to ACP
Utilization of both state and national religious and cultural organizations ensured accuracy of religion-specific content and resolved ambiguities. Importantly, process evaluation determined that an overwhelming majority of religious leaders felt content was easy to read, used appropriate language and tone, featured accurate content, and would support health professionals in ACP. Refinements were made to content where appropriate, that is, inclusion of end-of-life rituals and practices based on feedback that this was important.
Google analytics data revealed the majority of users were new, with a bounce rate of 40%. The bounce rate is reflective of people visiting the landing page and leaving without further exploration, which may be suggestive of people either finding what they are searching for and then leaving the website or leaving the website after unintentional visitation. Australia had the highest usage, which was to be expected; interestingly Russia accounted for approximately 4% of users with a bounce rate of about 7%, indicating that residents in Russia were exploring the website.
Overall, website feedback was favorable, and the majority of survey responders were health professionals, predominantly nurses, followed by GPs and allied health and medical specialists, indicative of the target audience. The remaining survey responders may have been members of the general community, which illustrates reach of website users broader than the targeted audience and the need for such a resource within the community. Most survey responders were positive about information accuracy, ease of understanding, usefulness, appropriateness, and website design. A lower proportion of survey responders (though still a majority) felt that the website increased their knowledge and awareness of ACP and assisted in preparation of ACP with people from different religious and cultural backgrounds. This may be explained by users having existing knowledge and expertise in ACP, with experience in discussions already among diverse religious and cultural communities.
Interviewed nurse stakeholders commented on the benefits of online accessibility; user friendliness; appealing design features and functionality, including the religion-specific search function; and absence of log-in requirements. Some nurses, however, suggested that menu visuals could be improved by increasing search function and home page button visibility. Many felt that the content was useful in addressing knowledge gaps, adding to existing religious and cultural resources, and would recommend the website to others. A comprehensive list of recommendations for consideration of further Web-based development included a glossary for varying state-based legal terms, condensing religion-specific content, comparable website for patients and families, an online chat service, and multilingual translations. Though multilingual translations were mentioned in the exploratory study and end-product evaluation, this was beyond the scope of the project resources and timelines.
Participation rate for the exploratory health professional study was low (26%), and health professionals were not from the most culturally diverse Australian-based communities as initially intended. Although difficulties in recruitment of GPs are consistent with the literature [
Multiple ACP eHealth apps are available, such as governmental and commercial websites, Web-based ACD registries, and educational material [
In contrast to these studies, ACP
This study demonstrates that all facets of the CIPP framework can be effectively applied for evaluation of eHealth technologies, specifically ACP websites, with assessment of ACP
advance care directive
advance care planning
context, input, process, and product
electronic health
end-of-life care
general practitioner
Making Your Wishes Known
project steering committee
project working group
The authors would like to thank the health care professionals, religious leaders, website content reviewers, project steering committee members, website designers and developers, and members of the community who participated in or supported the project.
This study was supported by funding from the Australian Government Department of Health. The funding body did not influence study design, data collection, data analysis, or interpretation and reporting of findings.
APS designed and led the research program. APS, PM, and LB carried out the research program. APS and PM developed the content of the website. COC led qualitative data analysis with interrater reliability from PM and APS. APS analyzed quantitative data. All authors contributed to the writing and/or critical revision of the paper.
None declared.