Design and Methods for a Comparative Effectiveness Pilot Study: Virtual World vs. Face-to-Face Diabetes Self-Management
Background: Type 2 diabetes (diabetes) is a serious threat to public health in the United States and disproportionally affects many racial/ethnic minority groups, including African Americans. Limited access to treatment and high attrition rates further contribute to health disparities in diabetes-related morbidity and mortality among minorities. Greater opportunities for increasing access and decreasing barriers to treatment are needed. Technology-based interventions have potential for accomplishing this goal but evidence of feasibility and potential effectiveness is lacking, especially for populations that traditionally have limited educational attainment and low computer literacy.
Objective: This paper describes the design and methods of a pilot randomized clinical trial that will compare the feasibility and potential efficacy of delivering a diabetes self-management intervention via a virtual world vs. a face-to-face format.
Methods: Study participants (n=100) will be African American women with uncontrolled type 2 diabetes recruited from primary care practices and affiliated health centers at a large safety net hospital in Massachusetts. Participants will be randomized into a virtual world-based (VW) intervention condition or a face-to-face control condition. Both conditions provide the same theory-based curriculum and equivalent exposure to the self-management program (eight group sessions), and both will be delivered by a single intervention team (a dietitian and a diabetes educator). Assessments will be conducted at baseline and 4 months. Feasibility will be determined by evaluating the degree to which participants engage in the VW-based intervention compared to face to face (number of sessions completed). Potential efficacy will be determined by comparing change in physiological (glycemic control) and behavioral (self-reported dietary intake, physical activity, blood glucose self-monitoring, and medication adherence) outcomes between the experimental and control groups.
Results: The primary outcomes of interest are feasibility of the VW intervention and its potential efficacy on glucose control and diabetes self-management behaviors, compared to the face-to-face condition. Analysis will use a two-sample Kolmogorov-Smirnov test for changes in variable distribution. P values will be calculated using binomial tests for proportions and t tests for continuous variables.
Conclusions: If the intervention is found to be feasible and promising, it will be tested in a larger RCT.
JMIR Res Protoc 2012;1(2):e24)
- Virtual systems;
- Patient education;
- Minority health;
- Health disparities;
- African Americans;
- Type 2 diabetes;
- Health behavior;
- Clinical trials
Type 2 diabetes (diabetes) is a serious threat to public health in the United States and the world and is a costly disease at the individual and societal levels. In the United States, its prevalence has risen among all ethnic groups, especially among certain racial/ethnic minority groups such as African-Americans who have had a near doubling of prevalence (now 13%) since 1988 [, ]. Diabetes has numerous and serious health risks associated with poor control [ ]. Expert guidelines promote counseling for patient health behavior change targeting increased physical activity, reduction in caloric intake and weight, and promotion of adherence to medication [ ]. Among individuals with diabetes, women and African Americans report the lowest levels of physical activity and more than half (66%) of African Americans with diabetes report high-fat diets [ , ]. Although self-management programs have proven efficacy, attrition rates are high [ ]. Factors reported as contributing to attrition include competing family responsibilities, conflict with program hours, and distance to services [ ]. Web-based methods to improve diabetes outcomes could improve access to behavior counseling and have shown promise in improving health behaviors and glycemic control [ ]. However, the feasibility of intervening through web-based methods has been questioned due to a real or perceived digital divide or socioeconomic inequalities among individuals in access to information technologies. For example, only 49% of African Americans compared to 68% of Caucasians have a broadband Internet connection at home [ ]. In addition to this divide in access to connectivity and hardware, researchers have identified a skill and knowledge divide on the basis of technological competency and digital literacy [ ].
Virtual world (VW) environments are potentially suitable environments for supporting patient education delivery, including diabetes self-management programming. VWs are 3D, immersive, online places where people enter and participate as an avatar. Once in the VW, participant avatars can walk, run, fly, talk, travel, and interact with the other people or structures present in the virtual space. The rich visual landscape, in combination with the mediated presence via avatar, gives the participant a strong sense of “being there”. VWs offer vast opportunities for interaction, intense engagement, and opportunities for scripted immersive experiences, simulations, role-playing, and constructivist learning. Sense of presence, easy access, and anonymity afforded by the avatar may lead to greater interaction and group cohesiveness, as well as engagement and attention, compared to other web-based approaches [, ]. Currently there are over 300 such online VWs, such as Jibe, OpenSim, Spoton3D, Open Wonderland, and Second Life (SL). Created and maintained by companies, universities, or individuals, some of these online environments are free of charge or some assess monthly or per-use fees. Although the use of VWs has been shown to influence behavior in the “real” world [ ], studies are limited and have not targeted the spectrum of diabetes self-management behaviors.
The objective of this paper is to describe the design and methods of a pilot study that will compare the feasibility and potential efficacy of delivering a diabetes self-management intervention via a VW (delivered in Second Life, or SL) versus a face-to-face format.
Women in Control is a pilot randomized clinical trial funded by the National Library of Medicine/National Institutes of Health that will evaluate the feasibility and potential efficacy of a diabetes self-management intervention delivered through interactive sessions within a Virtual World (VW), compared to a face-to-face format, to improve glucose control and enhance adherence to diabetes self-management behaviors in a population of African American women.
We hypothesize that the VW-based experimental intervention condition will produce outcomes related to feasibility and potential efficacy that are comparable or greater than those observed in the face-to-face control condition.
The intervention content will be adapted from the CDC/NIH program “Power to Prevent” (P2P) , a culturally appropriate, evidence-based behavior-change curriculum designed for face-to-face delivery to African American patients with diabetes or pre-diabetes. The intervention approach is theory-based [ ] and thus seeks to enhance diabetes knowledge, optimize attitudes toward diabetes self-management (ie, self-efficacy, outcome expectations), and develop behavioral self-management skills (eg, goal setting, tracking self-management behaviors and glucose levels, problem-solving) to facilitate changes in dietary intake, physical activity, blood glucose self-monitoring, and medication adherence. Curriculum objectives and topics are summarized in .
We will deliver this curriculum through a series of weekly sessions that begin with an individual intake session, followed by 8 weekly sessions. Sessions will be delivered in either a VW or face-to-face environment in groups of approximately 12-15 participants by a single intervention team (a dietitian and a diabetes educator) for the same duration (90-minute sessions) using the same protocol and materials for both conditions. Participants in both conditions will receive 2 sessions of computer training and will be provided with an Internet-enabled laptop computer upon training completion.summarizes the specific activities conducted during each computer training.
The face-to-face group sessions will take place in a large conference room at Boston Medical Center. All face-to-face participants will receive transportation vouchers to facilitate attendance. VW-based sessions will take place in an open-air VW forum built especially for the intervention with adequate structures and required visuals/displays (seeto ). Although the VW sessions will last 90 minutes, these participants will be asked to log in 30 minutes prior to each session in order to test their sound and troubleshoot any connection problems. To facilitate session flow, a triage system will assign participants to “producers” (technical support staff) as technology needs arise. As in the face-to-face condition, both interventionists will attend every group session (alternating facilitation). In the VW condition, the non-leading interventionist will take notes in the local chat and provide general back-up, while in the face-to-face condition, the non-leading interventionist will provide logistical assistance. compares similarities and differences of intervention implementation among these conditions in greater detail.
Two members of the research team will oversee fidelity of intervention content and style for both conditions via direct observation and review of recorded sessions. Interventionists will be provided with feedback on errors of omission and commission, and additional support will be given on an ongoing basis as needed.
Study Setting and Population
The study will recruit 100 African-American (Afro-Caribbean) women recruited from primary care practices and affiliated health centers at a large safety net hospital in Massachusetts. Eligible subjects will be age >18 years, English-speaking, with a diagnosis of type 2 diabetes, and an HbA1c > 8 at their last outpatient visit, which has to occur within the year preceding recruitment. Subjects with the following conditions will be excluded: ulcerative colitis, renal failure, complications following abortion and ectopic and molar pregnancies, angina pectoris, and other forms of unstable ischemic heart disease and conditions precluding brisk walking.
The research protocol was approved by the Institutional Review Boards at Boston Medical Center and the University of Massachusetts Medical School. Potential participants will be identified from the medical record data warehouse at Boston Medical Center and affiliated community health centers based on the above criteria. Identified patients will receive a letter informing them about the study. The letter will provide a description of the study, announce a phone call from a study staff to determine final eligibility, and give patients the option to call in or opt out. During the telephone call, the staff will assess final eligibility (ie, self-reported ability to view a computer screen without difficulty, ability to read, no use of glucocorticoid therapy, no current participation in a weight loss program, and availability for weekly meetings), answer any questions they may have about the study, and invite fully eligible women to participate. A maximum of five calls will be made to each participant on different days and varying times. Interested women will be scheduled for an enrollment visit at the Boston Medical Center General Clinical Research Unit in which participants provide written informed consent. A reminder call will be made the evening prior to the appointment.
Upon completion of all study assessments, we will randomize participants to either the VW-based experimental intervention condition or the face-to-face control condition. A block randomization scheme with a block size of 4, developed by StudyTRAX software (v3.0.0103), will be used. Randomization will be stratified on age and hemoglobin A1c measured at baseline. Participants will be informed about their randomization assignment via phone.
Outcomes and Study Measures
The primary outcomes of interest are feasibility of the VW intervention and its potential efficacy compared to the face-to-face condition. Feasibility will be determined by evaluating the degree to which participants engage in the VW-based intervention compared to face-to-face (number of sessions completed). Potential efficacy will be determined by comparing change in glucose control and self-management behaviors between the experimental and control groups, from baseline to the 4-month follow-up.
Assessments will be performed at baseline and at 4-month follow-up. HbA1c tests will be used to assess average blood glucose level over the previous 2-3 months. Non-fasting venous blood samples from an antecubital vein will be collected after the subject remains seated for 10 minutes, in accordance with standard research protocols . Boston Medical Center laboratories will analyze the specimens. At each time point, dietary intake will be assessed by two unannounced 24-hr dietary recalls (24HRs) [ ] administered on randomly selected days (60% probability of a weekday and 40% probability of a weekend day). 24HRs are well suited for the characteristics of the target population (ie, language, illiteracy/low educational level, ethnic foods) that limit use of other/additional assessment methods. Also at each time point, 24-hour physical activity recalls will be administered immediately after the 24-hr dietary recalls [ ], followed by recalls of blood glucose self-monitoring and medication adherence. Other measures will include blood pressure [ ], cholesterol, body mass index, waist and hip circumference, depressive symptoms [ , ], self-efficacy for diabetes management [ ], health literacy [ ], social support [ , ], perceived stress [ ], and quality of life [ ]. Demographic factors, smoking history, alcohol intake, current use of prescription and non-prescription medications, as well as use of home remedies, and experience with computers and the Internet will be assessed via survey. We will assess patient satisfaction at the 4-month follow-up. Intervention implementation costs (costs that would be incurred if the intervention were to be implemented outside the context of the research project) will be tracked.
StudyTrax (v3.0.0103), a web-based electronic data capture software, will be developed to house case report forms for data collected throughout the study. Case report forms will be developed to include field-specific validation code, procedures to check data ranges, and reminders to enter required data to ensure data integrity. Research staff utilizing the database will be provided role-specific privileges to the database. Two members of the study staff will perform double data entry on all clinical measures. Dietary data will be collected using the Nutrition Data System for Research (NDSR) database. Physical activity data will be collected on case report forms designed in Microsoft Access.
Quantitative Evaluation: Analytic Plan
Sample Size and Power Calculations
As a comparative effectiveness study, patients in both arms will receive interventions potentially resulting in improvements in the primary outcomes. Hence, a very large sample size would be required to demonstrate either statistically significant comparative effectiveness or superiority of one group over the other. As a result, and in recognition of the project’s status as a feasibility study, the study was powered to be able to demonstrate within group improvements in one of the key outcomes: physical activity. A sample size of 47 was determined to be sufficient to show a within group improvement, from pre- to post-, of 20% of the sample from inactive to moderate intensity physical activity, with a power of 80% at 0.05% level of significance.
Analysis and Statistical Tests
Our null hypothesis is that there will be no significant differences in outcome measures between the VW and face-to-face conditions. We will test this null hypothesis against the alternative that the VW condition is not as effective as face-to-face intervention using a two-sample Kolmogorov-Smirnov test for changes distributions of A1c. In other words, the VW intervention will be as good as or better than face-to-face intervention in decreasing A1c. The P values will be calculated using binomial tests for proportions and t tests for continuous variables. We will use RStudio (version 0.96.330)  for statistical analysis.
Following completion of the intervention delivery phase, focus groups will be conducted to qualitatively compare the experiences of participants assigned to the VW vs. the face-to-face condition. A total of 40 randomly selected participants will be invited to participate in four focus groups (expecting that 8-10 participants per group will attend): two groups will include VW participants, and the other two will include face-to-face participants. Participants will be consented for participation in, and recording of, the group discussions. We will offer a monetary incentive of US $40 and travel expenses to encourage participation. Two analysts will independently review transcripts of discussions, identifying key words and themes (in vivo coding) related to their experiences of living with diabetes, decision-making about treatments and lifestyle changes, and involvement in the VW or face-to-face intervention. A grounded theory approach will be used . Constant comparative analysis will be used to relate the transcribed focus group interview data through ideas to core concerns. The first approach will involve open coding using a meaningful phrase by meaningful phrase strategy. During the open coding process, data will be coded for any and all categories. Focused codes will be created for sorting and synthesizing the initial codes to organize the data, using the most significant and/or frequent earlier codes to sift through the data. From focused codes, memos will be generated theorizing about how codes relate to ideas and characterizing the emerging theory. Memos will be sorted into concepts demonstrating relationships between concepts that derive from memos. The formulation of themes and codes drawn directly from different sections of the data, and then used to code the transcripts themselves, is a pivotal step toward building an in-depth analysis.
Glycemic control is critical to preventing health disparities in diabetes morbidity and mortality among racial/ethnic minorities such as African Americans. Strategies to maximize access to treatment and self-management support are needed for this and other vulnerable populations. Technology-based interventions may be able to increase access but further evidence is needed to support the feasibility of such interventions with populations that have limited educational and computer literacy. By comparing a VW-based intervention format to a traditional face-to-face format for delivery of a diabetes self-management intervention to a sample of low-income African American women, this study will provide evidence of feasibility for a population with considerable health disparities in diabetes prevalence and outcomes and who also have limited experience with computers.
The strengths of the study are many. They include: (1) assessment of feasibility of utilizing a VW platform for patient health education, (2) randomization of study participants to a face-to-face condition (ie, traditional approach) vs. a VW environment condition, holding constant the intervention content, (3) recruitment of inner city minority participants thus addressing so called digital-divide issues, (4) standardization of hardware used to access the VW, (5) use of a control group that has similar hardware and training, (6) robust measurement of behavioral outcomes, and (7) inclusion of a qualitative post-intervention evaluation.
Limitations of this study include: (1) the pilot nature of the study. Adequately powering a comparative effectiveness study, given the characteristics of our primary outcomes, will require a larger sample size, beyond the scope of this project, and (2) the study focus is on a subset of the population, namely low-income African American women with uncontrolled diabetes. Thus, the generalizability of our findings to men and to non-underserved populations might be limited.
Evidence of feasibility and potential efficacy is needed so that new interventions can be designed for VW implementation and high reach to diverse populations. Our previous work has demonstrated the ability of VW-based training to change the professional behavior of clinicians [, ]. We anticipate that the current study will produce valuable data and insights to help guide application of these findings to VW-based interventions targeting not only diabetes self-management but also other chronic conditions and health behaviors, including such critical topics as medication adherence, substance abuse cessation and treatment, individual and group-based mental health treatments, many possible rehabilitative services, and other health services.
This study is funded by a grant from the National Institutes of Health/National Library of Medicine (grant # RC1 LM010412).
Conflicts of Interest
Multimedia Appendix 1
Comparison of virtual world (VW) versus face-to-face implementation of the Women in Control curriculum.PDF File (Adobe PDF File), 173KB
Multimedia Appendix 2
NIH Reviewers Statements.PDF File (Adobe PDF File), 47KB
Multimedia Appendix 3
Response to Reviewers Critiques.PDF File (Adobe PDF File), 103KB
- World Health Organization (WHO). Diabetes - fact sheet. 2012 Sep. URL: http://www.who.int/mediacentre/factsheets/fs312/en/index.html [accessed 2012-10-25] [WebCite Cache]
- National Diabetes Information Clearinghouse (NDIC). National diabetes statistics, 2011 URL: http://diabetes.niddk.nih.gov/dm/pubs/statistics/ [accessed 2012-10-25] [WebCite Cache]
- American Diabetes Association. Standards of medical care in diabetes--2012. Diabetes Care 2012 Jan;35 Suppl 1:S11-S63. [CrossRef] [Medline]
- Huffman FG, Vaccaro JA, Exebio JC, Zarini GG, Katz T, Dixon Z. Television watching, diet quality, and physical activity and diabetes among three ethnicities in the United States. J Environ Public Health 2012;2012:191465 [FREE Full text] [CrossRef] [Medline]
- Nelson KM, Reiber G, Boyko EJ. Diet and exercise among adults with type 2 diabetes: findings from the third national health and nutrition examination survey (NHANES III). Diabetes Care 2002 Oct;25(10):1722-1728. [Medline]
- Norris SL, Engelgau MM, Narayan KM. Effectiveness of self-management training in type 2 diabetes: a systematic review of randomized controlled trials. Diabetes Care 2001 Mar;24(3):561-587. [Medline]
- Gucciardi E, Demelo M, Offenheim A, Stewart DE. Factors contributing to attrition behavior in diabetes self-management programs: a mixed method approach. BMC Health Serv Res 2008;8:33 [FREE Full text] [CrossRef] [Medline]
- Ramadas A, Quek KF, Chan CK, Oldenburg B. Web-based interventions for the management of type 2 diabetes mellitus: a systematic review of recent evidence. Int J Med Inform 2011 Jun;80(6):389-405. [CrossRef] [Medline]
- US Department of Commerce. Exploring the digital nation: Home broadband internet adoption in the United States. Washington, DC: National Telecommunication and Information Administration; 2010. URL: http://www.esa.doc.gov/sites/default/files/reports/documents/report.pdf [accessed 2012-10-25] [WebCite Cache]
- US Department of Commerce. Falling through the net: A survey of the "have nots" in rural and urban America. Washington, DC: National Telecommunications and Information Administration; 1995 Jul. URL: http://www.ntia.doc.gov/ntiahome/fallingthru.html [accessed 2012-10-25] [WebCite Cache]
- Taylor TL. Living digitally: Embodiment in virtual worlds. In: Schroeder R, editor. The Social Life of Avatars. London: Springer; 2002:40-62.
- Hansen MM. Versatile, immersive, creative and dynamic virtual 3-D healthcare learning environments: A review of the literature. J Med Internet Res 2008;10(3):e26 [FREE Full text] [CrossRef] [Medline]
- Blascovich J, Bailenson JN. Infinite Reality - Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution. New York: William Morrow; 2011.
- National Institutes of Health, Centers for Disease Control and Prevention. Power to prevent: A family lifestyle approach to diabetes prevention URL: http://ndep.nih.gov/media/power-to-prevent-508.pdf?redirect=true [accessed 2012-10-25] [WebCite Cache]
- Bandura A. Self-efficacy: The Exercise of Control. New York: W.H. Freeman; 1997.
- Merriam PA, Ockene IS, Hebert JR, Rosal MC, Matthews CE. Seasonal variation of blood cholesterol levels: Study methodology. J Biol Rhythms 1999 Aug;14(4):330-339. [Medline]
- Krantzler NJ, Mullen BJ, Schutz HG, Grivetti LE, Holden CA, Meiselman HL. Validity of telephoned diet recalls and records for assessment of individual food intake. Am J Clin Nutr 1982 Dec;36(6):1234-1242 [FREE Full text] [Medline]
- Matthews CE, Freedson PS, Hebert JR, Stanek EJ, Merriam PA, Ockene IS. Comparing physical activity assessment methods in the Seasonal Variation of Blood Cholesterol Study. Med Sci Sports Exerc 2000 May;32(5):976-984. [Medline]
- Chobanian AV, Bakris GL, Black HR, Cushman WC, Green LA, Izzo JL, National Heart‚ Lung‚ and Blood Institute Joint National Committee on Prevention‚ Detection‚ Evaluation‚ and Treatment of High Blood Pressure, National High Blood Pressure Education Program Coordinating Committee. The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure: the JNC 7 report. JAMA 2003 May 21;289(19):2560-2572. [CrossRef] [Medline]
- Sawyer-Radloff L. The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement 1977;1(3):385-401.
- Reuland DS, Cherrington A, Watkins GS, Bradford DW, Blanco RA, Gaynes BN. Diagnostic accuracy of Spanish language depression-screening instruments. Ann Fam Med 2009;7(5):455-462 [FREE Full text] [CrossRef] [Medline]
- Rosal MC, Carbone ET, Goins KV. Use of cognitive interviewing to adapt measurement instruments for low-literate Hispanics. Diabetes Educ 2003;29(6):1006-1017. [Medline]
- Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD. Brief report: Screening items to identify patients with limited health literacy skills. J Gen Intern Med 2006 Aug;21(8):874-877 [FREE Full text] [CrossRef] [Medline]
- Sherbourne CD, Stewart AL. The MOS social support survey. Soc Sci Med 1991;32(6):705-714. [Medline]
- Sherbourne CD, Meredith LS. Quality of self-report data: A comparison of older and younger chronically ill patients. J Gerontol 1992 Jul;47(4):S204-S211. [Medline]
- Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983 Dec;24(4):385-396. [Medline]
- Jenkinson C, Layte R, Jenkinson D, Lawrence K, Petersen S, Paice C, et al. A shorter form health survey: Can the SF-12 replicate results from the SF-36 in longitudinal studies? J Public Health Med 1997 Jun;19(2):179-186 [FREE Full text] [Medline]
- RStudio: Integrated development environment for R (version 0.96.330) computer software. 2012. URL: http://www.rstudio.com/ide/ [accessed 2012-10-30] [WebCite Cache]
- Charmaz K. Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis. London, UK: Sage Publications; 2006.
- Mitchell S, Heyden R, Heyden N, Schroy P, Andrew S, Sadikova E, et al. A pilot study of motivational interviewing training in a virtual world. J Med Internet Res 2011;13(3):e77 [FREE Full text] [CrossRef] [Medline]
- Wiecha J, Heyden R, Sternthal E, Merialdi M. Learning in a virtual world: Experience with using second life for medical education. J Med Internet Res 2010;12(1):e1 [FREE Full text] [CrossRef] [Medline]
Edited by G Eysenbach; submitted 31.10.12; peer-reviewed by M Hansen; accepted 16.11.12; published 17.12.12
©Milagros C. Rosal, Robin Heyden, Roanne Mejilla, Maria Rizzo DePaoli, Chetty Veerappa, John M. Wiecha. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 17.12.2012.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.