Published on 01.03.16 in Vol 5, No 1 (2016): Jan-Mar
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/5142, first published Sep 16, 2015.
Development and Initial Evaluation of the Web-Based Self-Management Program “Partner in Balance” for Family Caregivers of People With Early Stage Dementia: An Exploratory Mixed-Methods Study
Background: People with dementia increasingly depend on informal caregivers. Internet-based self-management interventions hold considerable promise for meeting the educational and support needs of early stage dementia caregivers (EDCs) at a reduced cost.
Objective: This study aimed to (1) develop an online self-management program for EDC to increase self-efficacy and goal attainment, and (2) evaluate the program’s feasibility and report preliminary data on effectiveness.
Methods: Based on the Medical Research Council (MRC) framework for the development and evaluation of complex interventions, a stepwise approach was adopted to explore potential user needs and develop and validate the content by means of (1) focus group discussions with dementia caregivers (N=28), (2) interviews with dementia care professionals (N=11), and (3) individual think-aloud usability tests with EDC (N=2) and experts (N=2). A pilot evaluation was conducted with EDC (N=17) to test the feasibility and establish preliminary effects. Self-report measures of feasibility were completed after the completion of intervention. Self-efficacy and goal attainment were evaluated before and after the intervention.
Results: The different steps provided useful information about the needs of potential users regarding the content and delivery of the program. This resulted in the newly developed “Partner in Balance” program. At the start, system failures resulted in a high noncompleter rate (7/17, 41%), but at the end, an acceptable feasibility score of 209 (range 54-234) was found. The convenience of completing the program at home, the tailored content, and the guidance (face-to-face and online) were appraised positively. Preliminary effects on caregiver self-efficacy (P<.05) and goal attainment (T>50) were promising.
Conclusions: Adaptations were made to the program to limit the amount of system failures and prevent high noncompleter rates. As recommended by the MRC framework, confirming the feasibility and preliminary effectiveness is a valuable step toward examining the effectiveness of this newly developed intervention.
Trial Registration: Dutch Trial Register (NTR): NTR4217; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=4217 (Archived by WebCite at http://www.webcitation.org/6f6B8lvRP).
JMIR Res Protoc 2016;5(1):e33
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
Chronic illness and decreased well-being are expected to become global public health challenges , with dementia being one of the most common disorders in elderly individuals [ ]. With less formal health care available and more people in need of care, the caring role has now shifted to the informal caregivers at home [ ]. However, caregivers of people with dementia are at an increased risk of burden, stress, and have a fourfold risk of becoming depressed compared with noncaregivers [ , ]. As such, this transition of friend/family member into the caring role increases the need for effective caregiver interventions to improve their mood and quality of life.
Although recent face-to-face caregiver interventions appeared to be promising [, ], the increasing gap between care supply and demand calls for alternative and cheaper methods for providing education and support to informal caregivers [ , ]. Internet interventions may help caregivers cope with the challenges of caring for a person with dementia [ ]. A recent literature review [ ] showed that the currently available Internet interventions for caregivers of people with dementia have promising effects on their confidence and burden, given that they included multiple components and were tailored to the individual participant (caregiver). In addition, Internet-delivered caregiver support may prevent accessibility problems for informal caregivers who are isolated or have difficulties accessing traditional health care services [ , ].
At present, remote support for dementia caregivers is increasing, and new Internet interventions are being developed [- ]. These programs are, however, mainly focused on dealing with dementia-related problems (eg, neuropsychiatric symptoms) that occur at an advanced stage of the caregiver career; by contrast, early stage interventions can prepare caregivers for their future tasks at a stage where stress and burden are relatively low [ ]. Early intervention and support for caregivers have proven to be effective in reducing strain, increasing caregiver confidence, and delaying institutionalization of the person with dementia [ - ]. Moreover, early therapeutic interventions may help caregivers identify their needs based on their individual situation and facilitate the adaptation process [ ]. The Stress and Coping paradigm by Lazarus and Folkman [ ] and the Social Learning theory by Bandura [ ] propose that taking charge of the changes in one’s life increases self-efficacy, and can therefore reduce caregiver stress and its impact on general well-being [ ]. Following these theories, an early stage support program for caregivers should focus at positively managing life with dementia rather than managing the dementia itself [ ]. Self-management programs suit the caring role transition and have previously been used to support informal caregivers of several chronic diseases with promising results [ , , ].
Iterative Development Process
This study describes the development of an online self-management program for early stage dementia caregivers (EDC) to improve self-efficacy and goal attainment. We closely followed the iterative process of the new Medical Research Council (MRC) Framework for the development of complex interventions . The first 2 steps in the intervention development are described elsewhere [ , ]. The current paper describes the next 4 steps ( ) spread over a 2-year period (2012-2014). These steps are described in the following sections.
Methods and Results
Step 1: Explore Potential User Views: Focus Groups
In-depth exploratory focus group interviews were conducted to explore EDC’s views on the content and format of an early stage intervention (see Ref  for detailed methodology). A context-mapping approach [ ] was used: a booklet examining personal Internet and computer use and a collage displaying chosen themes based on significance during the early stages. Available themes were preselected based on existing interventions, early stage dementia care literature [ , ], and expert knowledge. Blank cards were also provided. The most often selected and highest appraised themes were compiled. Focus group interviews were transcribed verbatim and analyzed independently based on deductive content analysis by 2 of the study authors (LMMB and MEdV). Topics that were mentioned frequently and explicitly served as the basis for categorization. Categories were merged into common themes in a consensus meeting (LMMB and MEdV).
Participant characteristics (N=28) are presented in. The booklet on computer use was completed by 18 participants. Reasons for noncompletion were (1) overlooking the booklet (N=7) and (2) not understanding its value (N=3). Participants used the computer for multiple purposes, for example, finding information (15/18, 83%), email (14/18, 78%), financial transactions (14/18, 78%), writing (14/18, 78%), viewing photos (9/18, 50%), playing games (8/18, 44%), (video) chatting with family members (3/18, 17%), and shopping (2/18, 11%). However, 3 participants did not use a computer and were not inclined to do so in the near future.
During the focus groups, the majority of the participants considered Internet interventions as efficient due to the high level of accessibility, especially when feeling pressed for time or being bound to one’s home. Receiving answers to urgent queries was also considered very positive.
The advantages are no travelling time and the possibility to search what I want to know when it is convenient for me.
I would like to be able to extract the information that is important for me at that particular moment, because we’re all so different. When you’re in need of an answer, a personal response would be great.
Blended care (face-to-face care combined with online modules) was preferred over online care only, due to the personal contact with a professional.
People experience emotions, while a computer is just an object. Seeing the person you are talking to is really important. Once you know each other, email or telephone is fine for information exchange.
|Characteristics||n (%) or mean|
|Relationship to the care recipient|
|Living together with care recipient|
|Care recipient diagnosis|
|Mild cognitive impairment||8 (32)|
|Alzheimer’s disease (AD)||11 (44)|
|Vascular dementia||3 (12)|
|Dementia not otherwise specified||2 (8)|
|Care recipient years of diagnosis|
All participants (N=28) completed the collage of themes during the interviews. Participants stressed the importance of an intervention tailored to the stage of the disease and the individual caregiver’s situation, with less focus on coping with dementia and negative stigmatizing information about the future. Learning how to stay healthy by positively managing one’s life and learning to accept the changes were considered important, and so is the significance of information provided by other caregivers. A more flexible choice of themes, based on personal needs and areas of interest, was considered desirable. The themes most often mentioned and highest appraised are listed in.
|Appraised most important|
|Practical tips||27 (96)||14 (50)|
|Role and relationship changes||22 (79)||5 (18)|
|Information about the disease||21 (75)||12 (43)|
|Balance in activities||19 (68)||2 (7)|
|Focus on the positive||19 (68)||5 (18)|
|Communication||18 (64)||5 (18)|
|Acceptance||16 (57)||9 (32)|
|Insecurities and worrying||12 (43)||3 (11)|
|Social relationships and support||12 (43)||1 (4)|
|Emotions and tension||10 (36)||1 (4)|
Step 2: Develop and Validate Program Content and Structure
Consulting Dementia Care Experts
Individual in-depth interviews were conducted to explore dementia care experts’ views on EDC Internet support. Experts from different institutions and regions within the Netherlands were recruited via email. Inclusion criteria were (1) professional caregiver in the Dutch dementia care field, (2) daily interaction with people with dementia (PwD) and their caregivers, and (3) ample experience in supporting EDC. The number of participants (N=11) was determined by data saturation. The redundancy of themes emerged from interviews . Professional backgrounds of the experts were psychiatrists (N=1), clinical neuropsychologists (N=3), registered health psychologists (N=4), occupational therapists (N=1), social psychiatric nurses (N=1), and nurse practitioners (N=1), with an average of 13.64 (SD 7.43) years of professional experience. A semistructured interview guide was developed by authors LMMB and MEdV, which was validated by author FRJV. Topics included EDC needs, relevance and feasibility of EDC support, and themes for an EDC intervention. Brief summaries of the key points were made throughout the interview to obtain participant verification [ ]. All interviews were audiotaped. The content of the verbatim transcriptions of the interviews was analyzed by summarizing common themes based on deductive content analysis.
The experts emphasized that caregiver support needs to be tailored to the dementia stage. Concerns were raised about providing early support in the absence of later-stage problems, when caregivers are not in need of help yet and could possibly reject early stage support. Experts considered education on the disease and its course as most important. Other important themes involved accepting the disease, coping with relationship changes, stress, role management, and rumination. The importance of interaction between PwD, caregivers, and environment was also stressed. Correcting or accepting care recipients’ mistakes and notifying social network can be primary stressors in daily interaction. Too much negative information not fitting the early stages, for example, behavioral problems, care homes, and end-of-life decisions, could lead to adverse reactions and should be avoided in EDC support.
Combining the themes chosen by the experts and the caregivers resulted in 9 separate themes (). The theme “practical tips” was incorporated in the module structure to provide tips thematically.
|Acceptance||Identify changeable and unchangeable situations|
|Adapt expectations and learn to let go|
|Balance in activities||Change in daily and pleasant activities|
|Identify personal carrying capacity and burden|
|Communication with family member and environment||Communication changes due to memory problems|
|Effective communication with adaptations|
|Coping with stress||Relationship between stress and health problems|
|Identify and cope with stress in daily life|
|Focusing on the positive||Identify activities and situations that are still possible|
|Find alternatives and accept adaptations|
|Insecurities and rumination||Recognize rumination signals and control thoughts|
|Prevent looking ahead; live in the moment|
|Self-understanding||Self-evaluation in caregiver encounters|
|Personal strengths and areas of improvement|
|The changing family member||The changing memory and behavior|
|Influence of memory decline on daily life together|
|Social relations and support||Value and maintenance of social relations|
|Types of support|
Content Proposal: Existing Evidence and Conceptual Frameworks on Self-Care
Intervention content was proposed by authors LMMB and MEdV based on a literature review , EDC needs [ ], identified themes in Step 1, and conceptual frameworks on self-management. The Stress and Coping paradigm [ ] served as the theoretical basis for the content of the modules. According to this model, stress is experienced when a person perceives that the demands (caring for a person with dementia) exceed their personal and social resources. Caregivers’ responses to their stress situation might be mediated by their understanding of the situation and their beliefs about their ability to cope. The latter fits Bandura’s [ ] concept of self-efficacy (belief in one’s capabilities). Consistent with this theory, models of dementia management emphasize the need to maintain self-worth and control [ ]. An intervention aimed at increasing self-efficacy should not only educate the caregiver, but should also foster self-management by combining education with problem-solving skills, and work toward a change in behavior [ ].
The proposed self-management intervention program “Partner in Balance” (PiB) encourages caregivers to actively manage their lives and identify solutions for their specific needs . Increasing knowledge, identifying and setting goals, and learning skills to achieve these previously set goals served as the basis for the intervention program. Module content was focused on role management (eg, balancing activities in daily life) and emotional management (eg, dealing with fear and insecurity about the future) [ ]. Formulating, planning, and executing personal goals can be learned using a proactive 5-step change plan ( ) often used in self-management [ ], which was integrated into each module. By formulating and planning a personal change plan, caregivers learn to anticipate on stressful situations and gain confidence in their ability to take care of the situation and themselves [ ]. Because caregivers greatly varied in their needs, personal goals, and interest, a flexible choice of modules was used. Successful elements that were identified in the literature review [ ], including tailored caregiving strategies and contact with a coach and/or other caregivers, were included in the program content likewise.
Validating Proposed Program Content
Dementia care experts (N=4) not involved in the initial interviews were asked to read the material and provide comments with respect to language, tone, amount of information, and significance and propriety of the content. They worked as clinical neuropsychologist (N=1), registered health psychologist (N=1), occupational therapist (N=1), and social psychiatric nurse (N=1), with an average of 9.75 (SD 2.22) years of professional experience.
The experts provided comments concerning the content of each module. Stigmatizing, complex, or unclear language was reported and alternatives were provided. The textual content of the program was adapted accordingly.
Step 3: Testing the Feasibility and Preliminary Effectiveness
Think-Aloud Usability Testing
A Web-based fully operational program was developed based on the senior-friendly website checklist . Initial usability flaws were tested with the “think-aloud” method [ ]. Potential users were asked to think aloud while using the system, allowing the researchers to understand the reasons behind their usage behavior. Participants (N=4) were randomly selected from the focus group interviews (N=2) and the expert interviews (N=2). All aspects of the senior-friendly website guidelines were checked (eg, font size, contrast, menus and navigation, button style and size, phrasing, illustrations and videos, and Web assistance) and additional comments on user-friendliness were explored in-depth. The interviews were audiotaped and the verbatim transcripts were combined with field notes made by author LMMB on experienced difficulties during the walk-through. A coding scheme based on the interview protocol was used [ ].
All participants (N=4) commented on the layout, font size, contrast, tone, and navigation. The website layout was considered professional and attractive, although a uniform composition of all pages was proposed to foster cohesion. Alterable font sizes and increased contrast with the background color were suggested. Vignettes of caregivers were considered useful, but addressing people with their given name instead of their family name was considered more appealing. The tips from other caregivers were perceived as crucial. It was suggested to conclude every module with these tips to increase layout uniformity.
An uncontrolled pre-post-intervention pilot study with EDC was conducted to establish feasibility, as this is recommended before moving on to a larger scale effect study [, ]. The Medical Ethics Committee of the Maastricht University Medical Centre approved this study (No NL44475.068.13, Dutch Trial registration number NTR4217). Caregivers were included in the study if they (1) were spousal caregivers of people with mild cognitive impairment [ ] or mild dementia of all subtypes [ ], and (2) had access to the Internet. Exclusion criteria were (1) insufficient cognitive abilities to engage in the online self-management program, (2) overburdening, (3) severe health problems (determined by the study staff), or (4) caring for PwD caused by human immunodeficiency virus, acquired brain impairment, Down syndrome, Huntington’s chorea, or alcohol abuse. Participants were recruited at memory clinics and ambulatory mental health clinics. Based on comparable feasibility studies, we aimed to include 10 participants [ , ]. Of those contacted, 17 of 43 caregivers (40%) were willing to participate and signed the informed consent form.
Feasibility was evaluated face-to-face at the caregiver’s home by a semistructured interview developed for this study—the Program Participation Questionnaire (PPQ). The PPQ was based on measurement scales for perceived usefulness and ease of use and overall acceptance of information technology [, ] and included 30 items on usability, clarity, comfort with, and acceptability of the format on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). For the individual PPQ items, see . Mean scores were calculated with descriptive statistics. Because there were no external criteria to properly define feasibility [ ], we followed the conventional strategy of using the median score of the questionnaire as a cutoff. This approach was previously adopted in a Delphi study as evidence of agreement of intervention feasibility [ ]. Based on the PPQ scale (range 54-234, median 144), scores of 145 or higher were considered “acceptable feasibility.” Mean item scores (range 1-7) will be used to make decisions on positively and negatively appraised aspects of the program. Mean item scores of 5 (slightly agree) or higher will be considered positive, mean item score below 4 (slightly disagree or lower) will need further revisions. Participants were asked to elaborate their scores. Comments were audio recorded and transcribed verbatim. Meaningful data units based on the PPQ items were identified and derived independently from the qualitative data by authors LMMB and HEJW with deductive content analysis. Furthermore, the actual accessed data use of the program (number of log-ins and features used) was compared with self-reported data.
The study population consisted of 17 participants, of whom 10 completed the postintervention assessment. Participants who did not complete the postintervention assessment were replaced to meet the sample size suggested by previous studies [, ]. The main reasons for not completing the program and the postintervention assessment were difficulties with the online aspect of the program (N=4), private circumstances (N=2), and disagreements with the care recipient (N=1). Completer and noncompleter characteristics are listed in .
|Age caregiver, mean (SD)||68.10 (6.54)||67.43 (5.65)|
|Age care recipient (people with dementia [PwD]), mean (SD)||69.90 (4.33)||71.57 (8.46)|
|Hours care per week, mean (SD)||44.20 (56.85)||76.43 (71.98)|
|Gender, N (%)|
|Male||7 (70)||2 (29)|
|Female||3 (30)||5 (71)|
|Education, N (%)|
|High school||1 (10)||3 (43)|
|College||7 (70)||4 (57)|
|Graduate school||2 (20)||0 (0)|
|PwD diagnosis, N (%)|
|Mild cognitive impairment||7 (70)||5 (72)|
|Alzheimer’s disease||3 (30)||1 (14)|
|Vascular dementia||0 (0)||1 (14)|
The PPQ showed a good internal consistency (alpha=.89) and had a mean sum score of 209 (SD 22.14). Given the threshold of 145 or higher, this score indicated an acceptable feasibility. Mean item scores above 5 (slightly agree or higher) were found for convenience of completing the program at home (5.9, SD 1.8), clarity of the website (6.0, SD 1.2), module structure (6.1, SD 0.6) and content (6.6, SD 0.5), privacy (6.6, SD 1.3), tailored assignments (6.1, SD 1.3), guidance by the coach (6.6, SD 0.5), and general contentment (6.4, SD 0.9). A mean item score below 4 (slightly disagree or lower) was found for usefulness of the discussion forum (2.8, SD 2.7).shows the positive, negative, and neutral themes derived from the additional comments. Self-report usage data were comparable to tracked usage data: 106.41 (SD 96.15) minutes spent per module, including scoping the website (4.5 minutes, SD 4.13), completing the assignments and change plan (79.14 minutes, SD 77.16), contacting the personal coach (15.31 minutes, SD 16.96), and visiting the discussion forum (7.46 minutes, SD 7.09), spread out over 2.38 (SD 1.38) weeks.
|Online||At home||Writing personal situations feels more confronting than verbalizing them||—||I thought it was useful to have the ability to do it at my own convenience and only when I was in the mood for it. [P10]|
|Work in own time|
|Website||Quick response during problems||Technical problems: login and communication||It was pleasant that the information was presented in parts. [P8]|
|Personal coach||Indispensable support||Goal setting during intake difficult||Help of coach during goal setting necessary, not able to do it alone.||It is important that you have someone you can rely on during the program. It stimulates you even more and you know you are not alone. [P8]|
|Feedback boosts self-confidence and motivation|
|Video clips||Relatable examples||Not applicable to every individual||Background of caregivers unclear||I could not identify with the video clips because it was not clear who was talking and I cannot relate to the addressed problems just yet. [P3]|
|Discussion forum||Too difficult to use||Coaches could feed the forum, ask questions or outline situations||I am not going to write something out of the blue on a discussion forum. You might say the wrong things, even if you have good intentions. [P2]|
|Difficult to start conversation|
|No nonverbal communication|
|Change plan||Useful||Felt like an obligation||Difficult to set personal goals||The self-management assignment teaches you a lesson. How are you going to solve these things? [P10]|
|Application in daily life||Communication tips||Already apply tips in daily life||The coach provided me with very good tips that I applied in daily life. [P12]|
|General contentment||Boosts confidence||No focus on practical decisions possibly faced in the future||On moments that you are feeling insecure, this program gives you some kind of confirmation. It makes you doubt yourself less. [P10]|
|Recommend to other caregivers|
aP (number): participants number in the feasibility study.
Piloting Preliminary Effects
Preliminary understanding of the effectiveness of the program was based on the baseline and postintervention assessment 8 weeks later, completed at the participant’s own convenience in an uncontrolled pilot study. At participant’s request, paper questionnaires were used. The Caregiver Self-Efficacy Scale (CSES) was used to measure domain-specific caregiver self-efficacy . The subscales include 4 items on service use and 6 items on care management, with scores ranging from 1 (not at all certain) to 10 (very certain). We found a good internal consistency for both service use (alpha=.73) and care management (alpha=.87). Paired samples t tests were conducted to evaluate pre-post-intervention changes. The goal attainment scaling (GAS) [ ] method was used to rate treatment-related change and to compare relative success of previously set personal goals. Baseline scores were set at −2. Postintervention scores can range from −2 (much lower than expected) to +2 (much better than expected), with a score of 0 meaning goal attained. Raw scores were transformed into an individual mean GAS score (T score) to determine goal attainment with a potential weight assigned to the goal(s) [ ]. T scores of 50 or more (SD 10) indicate effective goal achievement.
Postintervention, participants (N=10) had significantly higher scores on both the CSES care management subscale (mean 41.1, standard error [SE]=2.5; t9=−2.5, P=.03) and service use subscale (mean 32.6, SE=1.7; t9=−3.5, P=.01) compared with preintervention scores (mean 36.1 and SE=3.2, and mean 23.2, SE=3.4, respectively), although effect sizes were small (d=0.14 and 0.41, respectively).
In this study, 8 program completers set 13 goals in total; 2 program completers were not able to set goals due to personal difficulties verbalizing the desired change. Two goals (of 2 participants with multiple goals) could not be scored after the intervention, because the goals changed during the course of the study. In total, 8 goals were attained (2 attained, 5 higher than expected, and 1 much higher than expected), and 3 goals were unattained (1 much lower than expected and 1 lower than expected). The mean T score at baseline (set at the −2 level) was 27.8 (SD 3.04). The mean achieved T score after the intervention was 53.7 (SD 12.03).shows the number of goals for each domain, with most goals set on communication with the care recipient (N=7), followed by maintaining positive activities together (N=2), obtaining social support (N=2), and planning time alone (N=2).
Step 4: Adapting the Program—Final Intervention
Following the iterative development process, the program was adapted according to the results obtained in the feasibility study. The discussion forum was expanded with regular posts from personal coaches with practical tips, literature, and events related to EDC. The role and background of the person in the video clips was clarified. In addition, the content of often-mentioned early stage situations and problems  was expanded and later-stage problems were made less prominent in the video clips. Furthermore, technical issues with logging in and communicating with the personal coach were resolved with the team of Web experts.
PiB consists of three elements, namely, (1) face-to-face intake session with a personal coach, (2) online period guided by the personal coach (psychologist or psychiatric nurse with ample experience with dementia caregivers), and (3) face-to-face evaluation session with the personal coach.
In the intake session participants are introduced to the website and the self-management concept of the program. The coach and participant set personal goals using a motivational interviewing technique frequently used to identify change objectives and enhance intrinsic motivation . Based on the discussed areas for improvement, participants select 4 of the 9 modules that were previously identified by experts and caregivers. Participants are provided with personal login codes to access their selected modules and edit their personal information. After the online period, participants will discuss their personal goals and their ability to cope with future difficulties in the evaluation session.
During the online period of the intervention, participants follow the chosen modules during an 8-week period. The website consists of (1) a home page with a short description of the goal of the program, personal login option, contact information of the researcher, and the institutional affiliation (Maastricht University); (2) a personal page with a link to the chosen modules and a mailbox for exchanging emails with the personal coach; and (3) an online forum to interact with other caregivers, moderated by the researcher and personal coaches.
Every module has a fixed design of the following 4 components: (1) video clip of fellow family caregivers, (2) education, (3) self-reflection assignment, and (4) the 5-step change plan, guided by the personal coach who will provide individualized online feedback after completion of each module and offers assistance when needed. For every module, 2 weeks are reserved as a starting point. However, participants are allowed to complete the modules at their own pace as informed by the self-management approach . The first week of a module addresses Components 1-4. Participants can send their assignment and the 5-step plan to their coach. The second week of every module is reserved for feedback from the coach, after which participants can adjust their 5-step plan if necessary.
The personal coach is an experienced dementia care professional (psychologist or psychiatric nurse). Coaches will receive a 1-day training in self-management techniques and online help before the start of the intervention. They will receive experienced supervision from an experienced professional in psychology and self-management during the course of the intervention period to ensure quality and alignment of the feedback of the coaches according to self-management principle. Coaches are asked to support participants in choosing modules that fit their personal situation, help participants identify feasible goals, offer techniques to achieve goals, and provide participants with general constructive feedback on their assignments. Using a personal login code, coaches will be matched with the participants assigned to them. The CONSORT EHEALTH checklist is presented as.
In this paper, the iterative development process of the Web-based self-management program PiB for EDC was presented. Use of the MRC framework enabled us to develop an intervention based on existing research, theoretical frameworks, and user and professional input. Including potential users during the design process enabled us to gain unique insights into usage behavior and challenges to adapt the technology to the needs of the target audience. A similar design has been successfully used in previous studies [- ].
During the exploration phase, caregivers greatly varied in their need for information. Previous self-management studies confirm that personal caregiver needs should be used as a starting point . Blended care was preferred over online care only, due to the personal contact with a professional. Previous studies support the value of this format as participants highly appreciated the connection with their coach or therapist [ ] and felt more motivated to complete Web-based interventions [ ]. By contrast, adding face-to-face contacts increases the costs of the online program and reducing the number of face-to-face contacts might harm treatment outcomes when online components are not used [ , ]. However, blending online modules with regular face-to-face therapy can increase the adherence and effectiveness of the treatment. Scientific validation of blended care interventions is warranted for the development and adaptation of future treatments [ ] and can provide important support for the use of blended care interventions rather than online therapy only. The former can be more easily implemented by health services, therapists, and clients than online therapy, as they can be integrated into existing treatment and care settings [ ].
Our results showed acceptable rates of satisfaction with PiB. Caregivers greatly appreciated the use of online resources due to the convenience of completing the program from their homes, which is in line with previous studies on Web-based caregiver interventions [, - ]. Furthermore, PiB supported the participants in the process of caregiving and boosted their self-confidence, probably due to the combination of support, self-management, and a tailored approach [ , ]. However, the noncompleter rate was high due to initial technical difficulties, which were resolved later on in the study. Unfamiliarity with using the website also caused difficulties among the older age group, resulting in a relatively young sample. A recent study confirmed that younger dementia caregivers were more likely to use the Internet for health-related purposes [ ]. However, other research showed that homebound older adults with limited computer skills who receive computer training at the start of an intervention can participate without difficulties [ ]. In addition, including more potential users in the thinking-aloud procedure could be helpful [ ].
The discussion forum was negatively reviewed and hardly used, due to the unclear purpose, the anonymity of participants, and the high threshold for starting a conversation. However, other studies suggest that forums can serve as a valuable addition to share experiences and support [, , ]. Adding new tips or developments for caregivers could increase the use of a forum because the aspect of reading posts was already considered useful [ ]. Furthermore, participants reported struggles with goal setting. This could be due to the relatively lower objective burden of EDC, compared with caregivers of people in the later stages of the disease. EDC might experience more subjective burden, which is more difficult to translate into specific needs. Problems with accepting and adapting to their new role may also have hindered goal setting [ ]. Therefore, goal setting for EDC should focus on the enhancement of positive overall experiences and facilitation of the personal adaptation process, rather than exclusively aiming for change. This approach may help to reduce or prevent negative consequences of caregiving (eg, overburdening) at a later stage [ ].
Preliminary effects on caregiver self-efficacy and goal attainment were small, yet positive. This finding is in line with Bandura’s theory on self-efficacy, which states that caregivers’ objective understanding of the situation and belief in one’s capabilities (ie, the self-efficacy level) can increase if provided with the right tools . Furthermore, this finding is congruent with previous research on online support for dementia caregivers [ , , ].
The program in this study was evaluated in a homogenous group of primary caregivers (eg, spousal caregivers), with specific attention for the spousal relationship. However, the themes may apply to a broader target group, as demonstrated by previous studies [- ]. PiB could potentially be suitable for other primary carers, which should be further investigated in the upcoming effect study using a larger sample.
The small sample size, the lack of a control condition, and a possible sampling bias based on caregivers with access to the Internet make it difficult to generalize the results. However, previous studies adopting a development and feasibility approach have used similar methodology and sample sizes [, ], fitting the purpose of formative research [ ]. In addition, users and experts were closely involved in the development of the intervention, and the content and adaptations relied on in-depth participant and expert feedback. Future research should consider inclusion of caregivers in the proposed content validation, to ensure potential user feedback in every step of the development. Furthermore, drawing conclusions from the adopted median cutoff score, which is an arbitrary value, may not be justified. However, in this study, the overall feasibility score was not leading. Mean item scores were used to make program improvements.
This study used paper questionnaires. Although seniors’ use of the Internet is expected to increase over time , dementia caregivers seem to be less active in health-related Internet use compared with the population at large [ ]. The high noncompleter rate resulted in missing post-test data from the noncompleters as these were collected after the last module. However, reasons for noncompletion and characteristics of noncompleters were provided, giving insight into their possible motives. Future effectiveness studies should include noncompleter data after the treatment and at follow-up and consider using Web-based questionnaires, with the advantages of low costs, no missing data or data entry errors, and time flexibility [ ].
Our tailored intervention approach appeared to be feasible for informal EDC and provided them with important support when dealing with the difficulties of caregiving. Feasibility results were used to improve the intervention. Confirmation of the feasibility and preliminary effectiveness is a valuable step toward examining the effectiveness of this intervention, as recommended by the MRC framework . The PiB course is currently (November 2015) available for caregivers who are interested in participating in the effectiveness study. At the course website (Partner in Balans) they can express their interest by emailing the researcher, after which they will receive additional information about the course and the effectiveness study.
This study is part of a larger study funded by Alzheimer Nederland (Grant no. WE03-2010-08), and the Alzheimer Research Fund Limburg. We thank the caregivers and experts who participated in this study and Inge Klinkenberg for her contribution to the assessments.
LMMB, MEdV, GIJMK, and FRJV designed the study. LMMB collected the data, performed the analyses and wrote the manuscript. MEdV and HEJW performed the second independent qualitative analysis for different stages in the development. MEdV supervised the data collection and analysis. MEdV, HEJW, GIJMK, and FRJV reviewed and approved this manuscript.
Conflicts of Interest
Multimedia Appendix 1
Program Participation Questionnaire.PDF File (Adobe PDF File), 127KB
Multimedia Appendix 2
CONSORT-EHEALTH checklist V1.6.2 .PDF File (Adobe PDF File), 761KB
- World Health Organization. World Health Statistics. Geneva, Switzerland: World Health Organization; 2014. URL: http://apps.who.int/iris/bitstream/10665/112738/1/9789240692671_eng.pdf?ua=1 [accessed 2016-02-06] [WebCite Cache]
- Alzheimer’s Disease International. Policy Brief for Heads of Government: The Global Impact of Dementia 2013–2050. London, UK: Alzheimer’s Disease International; 2013. URL: https://www.alz.co.uk/research/GlobalImpactDementia2013.pdf [accessed 2015-09-15] [WebCite Cache]
- Alzheimer's A. 2014 Alzheimer's disease facts and figures. Alzheimers Dement 2014 Mar;10(2):e47-e92. [Medline]
- Brodaty H, Donkin M. Family caregivers of people with dementia. Dialogues Clin Neurosci 2009;11(2):217-228 [FREE Full text] [Medline]
- Joling KJ, van Hout Hein P J, Schellevis FG, van der Horst Henriette E, Scheltens P, Knol DL, van Marwijk Harm W J. Incidence of depression and anxiety in the spouses of patients with dementia: a naturalistic cohort study of recorded morbidity with a 6-year follow-up. Am J Geriatr Psychiatry 2010 Feb;18(2):146-153. [CrossRef] [Medline]
- Van Mierlo Lisa D, Meiland Franka J M, Van der Roest Henriëtte G, Dröes RM. Personalised caregiver support: effectiveness of psychosocial interventions in subgroups of caregivers of people with dementia. Int J Geriatr Psychiatry 2012 Jan;27(1):1-14. [CrossRef] [Medline]
- Selwood A, Johnston K, Katona C, Lyketsos C, Livingston G. Systematic review of the effect of psychological interventions on family caregivers of people with dementia. J Affect Disord 2007 Aug;101(1-3):75-89. [CrossRef] [Medline]
- Macdonald A, Cooper B. Long-term care and dementia services: an impending crisis. Age Ageing 2007 Jan;36(1):16-22 [FREE Full text] [CrossRef] [Medline]
- White MH, Dorman SM. Online support for caregivers. Analysis of an Internet Alzheimer mailgroup. Comput Nurs 2000;18(4):168-76; quiz 177. [Medline]
- Boots L M M, de Vugt M E, van Knippenberg R J M, Kempen G I J M, Verhey F R J. A systematic review of Internet-based supportive interventions for caregivers of patients with dementia. Int J Geriatr Psychiatry 2014 Apr;29(4):331-344. [CrossRef] [Medline]
- Serafini JD, Damianakis T, Marziali E. Clinical practice standards and ethical issues applied to a virtual group intervention for spousal caregivers of people with Alzheimer's. Soc Work Health Care 2007;44(3):225-243. [CrossRef] [Medline]
- Kirsch SE, Lewis FM. Using the World Wide Web in health-related intervention research. A review of controlled trials. Comput Inform Nurs 2004;22(1):8-18. [Medline]
- Hattink B, Meiland F, van der Roest Henriëtte, Kevern P, Abiuso F, Bengtsson J, et al. Web-Based STAR E-Learning Course Increases Empathy and Understanding in Dementia Caregivers: Results from a Randomized Controlled Trial in the Netherlands and the United Kingdom. J Med Internet Res 2015;17(10):e241 [FREE Full text] [CrossRef] [Medline]
- Blom MM, Zarit SH, Groot Zwaaftink Rob B M, Cuijpers P, Pot AM. Effectiveness of an Internet intervention for family caregivers of people with dementia: results of a randomized controlled trial. PLoS One 2015;10(2):e0116622 [FREE Full text] [CrossRef] [Medline]
- Cristancho-Lacroix V, Moulin F, Wrobel J, Batrancourt B, Plichart M, De RJ, et al. A web-based program for informal caregivers of persons with Alzheimer's disease: an iterative user-centered design. JMIR Res Protoc 2014;3(3):e46 [FREE Full text] [CrossRef] [Medline]
- Hayden LJ, Glynn SM, Hahn TJ, Randall F, Randolph E. The use of Internet technology for psychoeducation and support with dementia caregivers. Psychol Serv 2012 May;9(2):215-218. [CrossRef] [Medline]
- Kajiyama B, Thompson LW, Eto-Iwase T, Yamashita M, Di MJ, Marian TY, et al. Exploring the effectiveness of an internet-based program for reducing caregiver distress using the iCare Stress Management e-Training Program. Aging Ment Health 2013;17(5):544-554 [FREE Full text] [CrossRef] [Medline]
- Cristancho-Lacroix V, Wrobel J, Cantegreil-Kallen I, Dub T, Rouquette A, Rigaud AS. A web-based psychoeducational program for informal caregivers of patients with Alzheimer's disease: a pilot randomized controlled trial. J Med Internet Res 2015;17(5):e117 [FREE Full text] [CrossRef] [Medline]
- van der Roest Henriëtte G, Meiland Franka J M, Jonker C, Dröes RM. User evaluation of the DEMentia-specific Digital Interactive Social Chart (DEM-DISC). A pilot study among informal carers on its impact, user friendliness and, usefulness. Aging Ment Health 2010 May;14(4):461-470. [CrossRef] [Medline]
- Verhey F, de VM. Mild cognitive impairment is becoming more psychosocial. Aging Ment Health 2013;17(3):265-266. [CrossRef] [Medline]
- Ducharme F, Dube V, Levesque L, Saulnier D, Giroux F. An online stress management training program as a supportive nursing intervention for family caregivers of an elderly person. Can J Nurs Inform 2011 Jun;6(2):13-44.
- Gaugler JE, Kane RL, Kane RA, Newcomer R. Early community-based service utilization and its effects on institutionalization in dementia caregiving. Gerontologist 2005 Apr;45(2):177-185. [Medline]
- Kjallman AA, Hellzen O, Norbergh KG. Experiences of long term ongoing structured support in early stage of dementia - a case study. Int J Older People Nurs 2014 Dec;9(4):289-297 [FREE Full text] [CrossRef] [Medline]
- Boots Lizzy M M, Wolfs Claire A G, Verhey Frans R J, Kempen Gertrudis I J M, de Vugt Marjolein E. Qualitative study on needs and wishes of early-stage dementia caregivers: the paradox between needing and accepting help. Int Psychogeriatr 2015 Jun;27(6):927-936. [CrossRef] [Medline]
- Lazarus R, Folkman S. Stress, Appraisal and Coping. New York: Springer; 1984.
- Bandura A. Self-Efficacy: The Exercise of Control. New York: W. H. Freeman and Company; 1997.
- Acton GJ, Kang J. Interventions to reduce the burden of caregiving for an adult with dementia: a meta-analysis. Res Nurs Health 2001 Oct;24(5):349-360. [Medline]
- Toms GR, Quinn C, Anderson DE, Clare L. Help yourself: perspectives on self-management from people with dementia and their caregivers. Qual Health Res 2015 Jan;25(1):87-98. [CrossRef] [Medline]
- Lorig KR, Ritter PL, Dost A, Plant K, Laurent DD, McNeil I. The Expert Patients Programme online, a 1-year study of an Internet-based self-management programme for people with long-term conditions. Chronic Illn 2008 Dec;4(4):247-256. [CrossRef] [Medline]
- Forducey PG, Glueckauf RL, Bergquist TF, Maheu MM, Yutsis M. Telehealth for persons with severe functional disabilities and their caregivers: facilitating self-care management in the home setting. Psychol Serv 2012 May;9(2):144-162 [FREE Full text] [CrossRef] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655 [FREE Full text] [Medline]
- Sleeswijk VF, Stappers PJ, van der Lugt R. Contextmapping: experiences from practice. CoDesign: International Journal of CoCreation in Design and the Arts 2005 Mar ;1(2):119-149. [CrossRef]
- Joosten-Weyn Banningh B. Learning to Live with Mild Cognitive Impairment. Development and Evaluation of a Psychological intervention for patients with Mild Cognitive Impairment, and Their Significant Others. Nijmegen, The Netherlands: Radboud University Nijmegen, Medical Centre; 2012.
- Denzin N, Lincoln Y. Collecting and Interpreting Qualitative Materials. London, UK: Sage; 1998.
- Hennink MM. International Focus Group Research. A Handbook for the Health and Social Sciences. Cambridge, UK: Cambridge University Press; 2007.
- Trappenburg J, Jonkman N, Jaarsma T, van OMH, Kort H, de WN, et al. Self-management: one size does not fit all. Patient Educ Couns 2013 Jul;92(1):134-137. [CrossRef] [Medline]
- Mountain GA, Craig CL. What should be in a self-management programme for people with early dementia? Aging Ment Health 2012;16(5):576-583. [CrossRef] [Medline]
- Lorig KR, Holman H. Self-management education: history, definition, outcomes, and mechanisms. Ann Behav Med 2003 Aug;26(1):1-7. [Medline]
- Nahm ES, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004;22(6):326-34; quiz 335. [Medline]
- Jaspers Monique W M. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009 May;78(5):340-353. [CrossRef] [Medline]
- Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med 2009 May;36(5):452-457 [FREE Full text] [CrossRef] [Medline]
- Winblad B, Palmer K, Kivipelto M, Jelic V, Fratiglioni L, Wahlund LO, et al. Mild cognitive impairment--beyond controversies, towards a consensus: report of the International Working Group on Mild Cognitive Impairment. J Intern Med 2004 Sep;256(3):240-246 [FREE Full text] [CrossRef] [Medline]
- American Psychiatric Association (APA). Neurocognitive Disorders. In: Diagnostic and Statistical Manual of Mental Disorders. 5th edition. Washington DC: APA; 2013.
- Fick DM, Steis MR, Mion LC, Walls JL. Computerized decision support for delirium superimposed on dementia in older adults. J Gerontol Nurs 2011 Apr;37(4):39-47 [FREE Full text] [CrossRef] [Medline]
- Lai Claudia K Y, Wong LF, Liu KH, Lui W, Chan MF, Yap Louise S Y. Online and onsite training for family caregivers of people with dementia: results from a pilot study. Int J Geriatr Psychiatry 2013 Jan;28(1):107-108. [CrossRef] [Medline]
- Teel CS, Leenerts MH. Developing and testing a self-care intervention for older adults in caregiving roles. Nurs Res 2005;54(3):193-201. [Medline]
- Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989 Sep;13(3):319. [CrossRef]
- National Alliance for Caregiving and American Association of Retired Persions (NAC & AARP). Caregiving in the U.S. 2004 Appendix A: Detailed Methodology. Washington DC: NAC and AARP; 2004. URL: http://assets.aarp.org/rgcenter/il/us_caregiving.pdf [accessed 2015-09-15] [WebCite Cache]
- Campbell SE, Azuara-Blanco A, Campbell MK, Francis JJ, Greene AC, Ramsay CR, et al. Developing the specifications of an open angle glaucoma screening intervention in the United Kingdom: a Delphi approach. BMC Health Serv Res 2012;12:447 [FREE Full text] [CrossRef] [Medline]
- Lorig K, Chastain RL, Ung E, Shoor S, Holman HR. Development and evaluation of a scale to measure perceived self-efficacy in people with arthritis. Arthritis Rheum 1989 Jan;32(1):37-44. [Medline]
- Kiresuk TJ, Sherman RE. Goal attainment scaling: A general method for evaluating comprehensive community mental health programs. Community Ment Health J 1968 Dec;4(6):443-453. [CrossRef] [Medline]
- Kiresuk TJ, Smith A, Cardillo JE. Goal Attainment Scaling: Applications, Theory, Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994.
- Miller W, Rollnick S. Motivational Interviewing: Preparing People for Change. Second edition. New York: The Guilford Press; 2002.
- Robinson L, Brittain K, Lindsay S, Jackson D, Olivier P. Keeping In Touch Everyday (KITE) project: developing assistive technologies with people with dementia and their carers to promote independence. Int Psychogeriatr 2009 Jun;21(3):494-502. [CrossRef] [Medline]
- Savitch N, Zaphris P. British HCI Conference: HCI and the Older Population. Learning from people with dementia to develop research methods for older people URL: http://www.dcs.gla.ac.uk/utopia/workshop/savitch.pdf [accessed 2015-09-15] [WebCite Cache]
- Hanson E, Magnusson L, Arvidsson H, Claesson A, Keady J, Nolan M. Working together with persons with early stage dementia and their family members to design a user-friendly technology-based support service. Dementia 2007 Aug 01;6(3):411-434. [CrossRef]
- Barlow J, Wright C, Sheasby J, Turner A, Hainsworth J. Self-management approaches for people with chronic conditions: a review. Patient Educ Couns 2002;48(2):177-187. [Medline]
- van der Vaart Rosalie, Witting M, Riper H, Kooistra L, Bohlmeijer ET, van Gemert-Pijnen Lisette J E W C. Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry 2014;14:355 [FREE Full text] [CrossRef] [Medline]
- Wilhelmsen M, Lillevoll K, Risør MB, Høifødt R, Johansen ML, Waterloo K, et al. Motivation to persist with internet-based cognitive behavioural treatment using blended care: a qualitative study. BMC Psychiatry 2013;13:296 [FREE Full text] [CrossRef] [Medline]
- Ruwaard J, Kok R. Wild west eHealth: Time to hold our horses? Eur Health Psychol 2015 Feb;17(1):45-49.
- Rozental A, Andersson G, Boettcher J, Ebert DD, Cuijpers P, Knaevelsrud C, et al. Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interv 2014 Mar;1(1):12-19. [CrossRef]
- Carlbring P, Andersson G. Internet and psychological treatment. How well can they be combined? Comput Human Behav 2006 May;22(3):545-553. [CrossRef]
- Glueckauf RL, Loomis JS. Alzheimer's Caregiver Support Online: lessons learned, initial findings and future directions. NeuroRehabilitation 2003;18(2):135-146. [Medline]
- Lorig KR, Ritter PL, Dost A, Plant K, Laurent DD, McNeil I. The Expert Patients Programme online, a 1-year study of an Internet-based self-management programme for people with long-term conditions. Chronic Illn 2008 Dec;4(4):247-256. [CrossRef] [Medline]
- Glynn SM, Randolph ET, Garrick T, Lui A. A proof of concept trial of an online psychoeducational program for relatives of both veterans and civilians living with schizophrenia. Psychiatr Rehabil J 2010;33(4):278-287. [CrossRef] [Medline]
- Torp S, Hanson E, Hauge S, Ulstein I, Magnusson L. A pilot study of how information and communication technology may contribute to health promotion among elderly spousal carers in Norway. Health Soc Care Community 2008 Jan;16(1):75-85. [CrossRef] [Medline]
- Stjernswärd S, Östman M, Löwgren J. Online self-help tools for the relatives of persons with depression -- a feasibility study. Scand J Caring Sci 2012 Mar;26(1):70-80. [CrossRef] [Medline]
- Kim H. Understanding Internet Use Among Dementia Caregivers: Results of Secondary Data Analysis Using the US Caregiver Survey Data. Interact J Med Res 2015;4(1):e1 [FREE Full text] [CrossRef] [Medline]
- Choi I, Sharpe L, Li S, Hunt C. Acceptability of psychological treatment to Chinese- and Caucasian-Australians: Internet treatment reduces barriers but face-to-face care is preferred. Soc Psychiatry Psychiatr Epidemiol 2015 Jan;50(1):77-87. [CrossRef] [Medline]
- McKechnie V, Barker C, Stott J. The effectiveness of an Internet support forum for carers of people with dementia: a pre-post cohort study. J Med Internet Res 2014;16(2):e68 [FREE Full text] [CrossRef] [Medline]
- Glueckauf RL, Ketterson TU, Loomis JS, Dages P. Online support and education for dementia caregivers: overview, utilization, and initial program evaluation. Telemed J E Health 2004;10(2):223-232. [CrossRef] [Medline]
- Coulehan M. Impact of a Web-based educational and peer-support intervention for dementia caregivers. Washington, DC: American Society on Aging; 2011 Apr 27 Presented at: Aging in America Conference; 2011 Apr 27; San Francisco, CA.
- Fledderus M, Schreurs KM, Bohlmeijer ET, Vollenbroek-Hutten MM. Development and pilot evaluation of an online relapse-prevention program based on acceptance and commitment therapy for chronic pain patients. JMIR Human Factors 2015 Jan 05;2(1):e1. [CrossRef]
- Eastman JK, Iyer R. The elderly's uses and attitudes towards the Internet. J Consum Mark 2004 May;21(3):208-220. [CrossRef]
- Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res 2007;9(3):e25 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]
|CSES: Caregiver Self-Efficacy Scale|
|EDC: early stage dementia caregivers|
|GAS: goal attainment scaling|
|MRC: Medical Research Council|
|PiB: Partner in Balance|
|PPQ: Program Participation Questionnaire|
|PwD: people with dementia|
|SE: standard error|
Edited by A Moorhead; submitted 16.09.15; peer-reviewed by F Meiland, C Smits, C Barker; comments to author 09.10.15; revised version received 06.11.15; accepted 23.12.15; published 01.03.16
©Lizzy MM Boots, Marjolein E de Vugt, Hanneke EJ Withagen, Gertrudis IJM Kempen, Frans RJ Verhey. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 01.03.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.