This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
The Department of Veterans Affairs (VA) has developed health information technologies (HIT) and resources to improve veteran access to health care programs and services, and to support a patient-centered approach to health care delivery. To improve VA HIT access and meaningful use by veterans, it is necessary to understand their preferences for interacting with various HIT resources to accomplish health management related tasks and to exchange information.
The objective of this paper was to describe a novel protocol for: (1) developing a HIT Digital Health Matrix Model; (2) conducting an Analytic Hierarchy Process called pairwise comparison to understand how and why veterans want to use electronic health resources to complete tasks related to health management; and (3) developing visual modeling simulations that depict veterans’ preferences for using VA HIT to manage their health conditions and exchange health information.
The study uses participatory research methods to understand how veterans prefer to use VA HIT to accomplish health management tasks within a given context, and how they would like to interact with HIT interfaces (eg, look, feel, and function) in the future. This study includes two rounds of veteran focus groups with self-administered surveys and visual modeling simulation techniques. This study will also convene an expert panel to assist in the development of a VA HIT Digital Health Matrix Model, so that both expert panel members and veteran participants can complete an Analytic Hierarchy Process, pairwise comparisons to evaluate and rank the applicability of electronic health resources for a series of health management tasks.
This protocol describes the iterative, participatory, and patient-centered process for: (1) developing a VA HIT Digital Health Matrix Model that outlines current VA patient-facing platforms available to veterans, describing their features and relevant contexts for use; and (2) developing visual model simulations based on direct veteran feedback that depict patient preferences for enhancing the synchronization, integration, and standardization of VA patient-facing platforms. Focus group topics include current uses, preferences, facilitators, and barriers to using electronic health resources; recommendations for synchronizing, integrating, and standardizing VA HIT; and preferences on data sharing and delegation within the VA system.
This work highlights the practical, technological, and personal factors that facilitate and inhibit use of current VA HIT, and informs an integrated system redesign. The Digital Health Matrix Model and visual modeling simulations use knowledge of veteran preferences and experiences to directly inform enhancements to VA HIT and provide a more holistic and integrated user experience. These efforts are designed to support the adoption and sustained use of VA HIT to support patient self-management and clinical care coordination in ways that are directly aligned with veteran preferences.
Historically, health information technology (HIT) applications and systems have been developed under the auspices of independent organizational program offices, and as a result, they may not optimally support or enable an integrated patient experience across technology platforms [
To enable a more patient-centered and integrated approach to HIT tools and services, the Department of Veterans Affairs (VA) chartered a Connected Health task force in 2012 to develop strategic recommendations that would enable a seamless, unified veteran experience across all VA sponsored patient-facing technologies (ie, any technologies that a patient uses directly). Task force recommendations included the development of a centralized governance structure that would integrate and standardize the development and deployment of VA digital health tools and services. The Office of Connected Health of the Veterans Health Administration (VHA) was established in 2013, and represents a centralized governance model for multiple VHA program offices responsible for patient-facing technology systems, including My Health
As emphasized in the VA Strategic Plan [
The development and proliferation of virtual access to care supports an organizational approach that is personalized, proactive, and patient-driven...Advances in virtual care expand where health care services can be accessed, reduce the need for travel to medical facilities, and transform VA’s delivery of health care and its effect on patients’ health outcomes. (pg. 18). [
To date, the VA has invested in a multitude of electronic health resources, such as Telehealth, VetLink Kiosks, an electronic health record, a tethered patient portal known as My Health
To accomplish this effectively, it is crucial to understand the nuances of veteran preferences for using various types of HIT resources to accomplish common user tasks and to exchange information. Concurrently, a detailed assessment of current and future VA patient-facing technology platforms is needed in order to identify integrated approaches that will best support a more patient-centered experience. In alignment with the Office of Connected Health, this VA funded research is designed to: (1) conduct a comprehensive assessment of current and future patient-facing technology resources based on the input of an expert panel that includes organizational subject matter experts and key stakeholders; (2) learn which resources veterans prefer to use to accomplish their health related tasks within a given context; (3) identify veteran preferences for using VA resources to exchange information; and (4) explore how veterans want to interact with digital health resource interfaces (eg, look, feel, and function). The study aims are designed to support veterans’ self-management and task accomplishment (eg, accessing lab test results, refilling a prescription, scheduling an appointment, communicating electronically, etc), and to improve continuity of care through the integrated use of the VA’s electronic health resources. Specifically they are to: (1) explore the preferences of veterans with chronic comorbid conditions for using various VA electronic health resources, using participatory research methods (develop and refine visual modeling simulations of VA electronic resources based on direct veteran input); (2) develop a comprehensive Digital Health Matrix Model of current and future VA patient-facing electronic health resources and conduct an Analytical Hierarchy Process, a pairwise comparison process with expert panel members and veteran participants; and (3) collaborate with VHA Program Offices as operational partners to directly inform current and future patient-facing HIT redesign efforts.
In this paper, we describe our development processes and study protocol, which leverage stakeholder groups (subject matter experts, clinicians, and veterans) and a participatory research approach that purposively recruits participants as expert informants to express their preferred vision for the future of VA’s system of electronic health resources. Products of this research will be used in tandem with VA operational efforts to increase the usability and usefulness of the VA’s electronic health resources, and to support a more integrated and effective veteran experience in use of VA HIT electronic health resources.
This mixed-methods participatory descriptive study [
To complete data collection, two rounds of veteran participant focus groups will be conducted, along with self-administered surveys to elicit veteran perspectives. Based on the first round of veteran focus groups, the research team will collaborate with members of the VHA Human Factors team to develop relevant process models, and then create a set of visual modeling simulations. A second round of focus groups with the same veterans from the first round will then be conducted to elicit feedback about the visual modeling simulations, and to complete pairwise comparisons adapted to the focus group format. These focus groups will contribute to a final set of visual modeling simulations and a refined Digital Health Matrix Model. This study is approved and regulated by the VA Central Institutional Review Board.
In qualitative research, sample size relies on the quality and richness of information obtained [
Organizational subject matter experts and key stakeholders will comprise the expert panel. This panel will include representation from all relevant VHA Program Offices and key clinical disciplines. The majority of these individuals are well known by the research team as key representatives for each of VA’s electronic health resource platforms, clinicians, subject matter experts who work in this area of research, and Office of Connected Health representatives.
Snowball sampling will be used as needed to recruit members that represent all key stakeholders (eg, My Health
The research team is purposively recruiting up to N=48 veterans who have expressed interest in using HIT electronic resources as “expert informants” to inform the outcomes of this project. The sample will include English speaking veterans age 35 years and older, with at least two chronic comorbid conditions (eg, diabetes and high blood pressure), and who report using two or more VA electronic health resources or non-VA electronic resources more than once a month. Therefore, study exclusion criteria include veterans younger than 35 years of age with less than two comorbid conditions, who use fewer than two VA electronic health resources less than once a month. Because of the nature of this study, we exclude those who do not speak English; and/or who report a visual, hearing, or cognitive impairment. This part of the study is being conducted at two VA Medical Centers: (1) the James A Haley Veterans Hospital (Tampa, Florida); and (2) the VA New England Health Care System, (Bedford, Massachusetts).
Purposive sampling will yield a sample pool for veteran recruitment efforts. We used administrative data to identify veterans registered for My Health
A structured screening interview tool was developed that includes items to address study age criteria and the occurrence of at least two chronic comorbid conditions (eg, diabetes, high blood pressure, COPD, etc). Based on Agency for Healthcare Research and Quality (AHRQ) recommendations, the screening interview tool also includes items to ensure veteran use of at least two VA electronic health resources (to make transactions; and access, store, manage, organize, and track information) [
Study team members will contact potential study participants for recruitment via telephone utilizing the screening tool until domain and theme saturation is reached in data collection. Veterans will receive up to US $50 for their participation (US $25 for participating in each round of focus groups).
Veteran participant recruitment flow chart.
Data will be collected from expert panel members using a series of teleconference calls to inform the development of the Digital Health Matrix Model, and communication via email to request and obtain individual responses for a structured pairwise comparison activity that is based on Analytical Hierarchy Process techniques. Expert panel members will also be invited to participate in an education session via teleconference that provides additional instructions on completing the requested pairwise comparisons. Data will be collected from veteran participants using a self-administered survey and focus group scripts.
Study flow chart.
Pairwise comparison is an analytic hierarchy process (ie, method for understanding complex decision making) that directs participants to compare a series of items and decide which item is preferred. Participants then quantify their preferences using a numerical scale [
The second step entails structuring the decision hierarchy and emphasizes expanding content developed in the first step through an information gathering process with subject matter experts, stakeholders, and veteran input. To complete this second step, we will develop a Digital Health Matrix Model that represents a detailed inventory of available electronic resources, their features, characteristics, and contexts for use.
To develop the Digital Health Matrix Model, an expert panel will provide the appropriate clinical, administrative, and operational expertise and perspective. Expert panel stakeholder groups include clinicians (physicians, nurses, and patient educators) and representatives aligned with each of the VA electronic health resources (My My Health
The multi-axis Digital Health Matrix Model will include both currently available and future VA patient-facing platforms, their features, their availability, and conditions for appropriate use. Due to the complexity of VA electronic health resources and elements of interest, the Digital Health Matrix Model will be developed using an electronic Excel document with several sheets representing topics discussed during focus group interviews such as access, function, preferences, barriers to use, relevant user tasks, etc. Within each sheet, there will be a two-axis inventory of: (1) each electronic health resource (represented by row); and (2) domains (ie, broad categories describing related items) that emerge from focus group data and are relevant to the topic of each sheet. This matrix model will provide a descriptive blueprint for veteran decision making, and support the continued development of a more integrated system of VA patient-facing platforms and electronic health resources that meet the needs and preferences of veteran users. The organization of the matrix document will also allow content to be evaluated and prioritized based on the perceived usefulness of each electronic health resource within specific contexts. Due to the length and breadth of detail contained within the model, we will develop search and categorization options to allow expert panel members to easily select and compare two (or more) resources while completing the pairwise comparison activity.
Both expert panel members and veterans will conduct a single activity that will complete the third and fourth steps of the process. There are two separate processes that will be used to complete these activities with the panel members and veterans. Using email, expert panel members will be provided a series of “worksheets” in a single document, with each worksheet representing a single health management task. Panel members will be asked to consider veterans preferences, cost, convenience, and workflow consequences, when completing the pairwise comparisons to determine which electronic health resources most effectively support health care management within the specific, predefined contexts. An adapted process will be completed by veterans during the second round of focus groups to ensure that their input is represented in the final model. Veterans will complete the pairwise comparisons and rank their preferences as a group to promote discussion.
In these final steps, electronic resources are compared (step 3) and ranked (step 4). Comparisons in step 3, allow the individual(s) to consider the value of using electronic health resources for a given health management task. In step 4, the selected electronic health resources are ranked to determine their level of priority over other options; numerical priorities are assigned on a scale of 1 to 3. This is done to calculate numeric weights for each alternative. The final scores provide a decision-making model that compares alternative electronic health resources for accomplishing specific tasks for managing a health condition.
Pairwise comparison worksheet example.
As previously noted, veteran participants will participate in two rounds of focus groups. At the first round of focus groups, veteran participants (up to N=48) will be consented and complete a baseline participant survey and assessment packet with a research team member in a private room. To collect demographic data, they will complete a 9-item demographic survey to ascertain age/date of birth, gender, race/ethnicity, education level, income level, marital status, and current medical conditions. There are thirty additional items that will be included to assess their electronic resource use (such as use of computer, Internet, smart phone, mobile technology, etc) and use of VA specific HIT resources such as My Heath
Veteran participants will be recruited as “expert informants”. The first round will enable veterans to describe their preferences and needs when using HIT to manage their health and exchange information. Using data collected during this phase, the research team and VA Human Factors partners will create interactive visual modeling simulations of patient-facing electronic health resources. During the second round of focus groups, participants will review the visual modeling simulations and discuss how well they represent veteran preferences and needs. Veterans will have an opportunity to provide additional input and suggest specific revisions to ensure the visual modeling simulations effectively represent their needs and preferences. Participants will also complete an adapted version of the pairwise comparison process.
Focus groups will last up to 2 hours and will be audio recorded. A moderator and facilitator will lead the focus groups, while VA operational partners and VA Human Factors representatives attend via teleconference technology. Participants will be introduced to the moderator, facilitator, and the teleconference attendees, and informed of their role and purpose for being present.
Participants will be asked questions about their preferences for using VA and non-VA electronic health resources, and specific reasons for use. Focus group questions will focus on, “The now”, resources they use (My Health
Preferences for exchanging information with VA:
Sharing information with your health care team is important, how would you like to be able to provide your health care team with information (for example, information that you keep track of at home)?
Use of electronic resources:
Please list all of the technology such as online resources, services, and tools (both VA and Non-VA) you currently use to manage your (condition).
Let’s talk about the tasks you do to manage your (condition), using these technologies and applications.
Probes,
-How do you use (electronic resource) to manage (condition)?
-What is the primary reason you use (electronic resource)?
-Which features of (electronic resource) are most useful to you?
Interacting with VA technology in the future:
If you were going to design the way you interact with VA using technology to send and receive information, what would it look like? Let’s draw what this system might look like together (use Post-it Pads and markers).
Can you give me an example of how VA electronic technologies could be used as an ideal system of VA electronic services that work together?
If you use the (electronic resource) to (task), would you expect the (electronic resource) to look exactly the same? If yes, how so?
Probes,
-How would you expect to see your information across tools (format)?
-How would you prefer labeling, colors, and backgrounds to be?
On smartphones and computers, there is a main page, or “dashboard” from which a user can navigate to all of their tools. How do you feel about having a dashboard of VA services and tools?
During the focus group, participants will be provided with pens, pads, markers, and large pieces of paper to allow them to write down their thoughts and draw out imagery that represents their preferences and needs. They will also be invited to bring any mobile devices that they use, and to share their preferred electronic resources throughout the focus group discussion. The study moderator will also transcribe notes onto large sheets of paper displayed on the wall in order to allow participants to review and refer to notes and topics throughout the focus group discussion. This method will assist the study team in guiding the discussion.
After each focus group, all notes developed during the group discussion will be immediately transcribed, and all imagery drawn by participants during the session will be photographed and saved in an electronic format. All of these assets will be transferred to the assigned Human Factors team to inform the immediate development of visual modeling simulations based on all of the data provided by veterans in the first round of focus groups.
Based on the data collected in round one of the focus groups, visual model simulations will be created by VA Human Factors experts using iRise version 8 (iRise, Enterprise Visualization Platform). Visual model simulations provide an effective method to rapidly create a graphic display of an electronic interface, but with limited functionality. This technique allows the user to see the graphic display (also known as wireframes), and experience it in its limited function. Focus group summaries and visual asset data (eg, drawings and sketches) gathered during the focus groups will be used to ensure that the participants’ perspectives are represented in the simulations. The study team will use focus group data to develop detailed test case scenarios and process models that will be simulated to the specified look and feel using the iRise software by the VA Human Factors team. These simulated visual models will consist of mock application screens and Web pages for various resources of interest (ie, My Health
The second round of veteran focus groups will be conducted with the same veteran participants from round one. In the second round of focus groups, participants will complete two activities: (1) provide feedback about the visual model simulations; and (2) conduct the pairwise comparison process group activity. We will provide exposure to the visual modeling simulations in group settings so that veteran focus group participants can react and provide specific feedback about these simulations and the degree to which they represent their needs and preferences. These interactive sessions will consist of a group setting, in which participants engage in a semiscripted simulation to review each of the visual modeling simulations of patient-facing platforms. As recommended by Kushniruk [
Initial responses to visual model simulations:
What are your initial thoughts about the (electronic resource simulation).
Probe,
-What are 3 things you most like/dislike about the (electronic resource simulation)?
-Does this (electronic resource simulation) reflect the feedback you gave us when we met previously?
Feedback about format and layout:
What do you like/dislike about the (colors, size, layout)?
Probe,
-How would you change the (colors, size, layout) to make (electronic resource simulation) (more useful, easier to use)?
Usability and ease of use:
Describe a scenario in which you would use this (electronic resource simulation).
How would you (navigation/task) if you wanted to (health management task) for your (health condition)?
Morae version 3.3 (Morae from TechSmith) usability software and audio recorders will be used to record and analyze revisions recommended by veteran focus group participants. This software has been utilized successfully in testing Web-based software [
The veteran focus group participants will also complete the pairwise comparison activity as a group to enable discussion about commonalities and differences in preferences. To accomplish this, adapted versions of the worksheets will be posted on large presentation style poster paper so that all participants can see the layout. A research team member will facilitate the pairwise comparison process with the group, allowing each participant an opportunity to select and rank electronic resources for a series of health condition management tasks. Pairwise comparisons completed by veterans will then be compared to those completed by expert panel members.
Pairwise comparisons completed by participating expert panel members and veteran focus group participants will be reviewed by the study team and then collated to identify the preferred platform for each task and function based on expressed preferences, needs, benefits, and barriers. The collation process will allow all respondent’s comparisons to be tallied to rank the usefulness of various electronic health resource platforms for completing distinct self-management and health care related tasks. The study team will also assess similarities and differences in the input provided by expert panel members versus veteran focus group participants.
Descriptive analysis of survey and assessment data completed by veteran focus group participants will be conducted to identify sample characteristics. Qualitative data from the first round of veteran focus groups will be analyzed at two levels by three study team members. First, participant input captured on Post-it Notes will be transcribed to an electronic document. Study team members with qualitative research expertise will code the data from the Post-it Notes topographically into major domains and subdomains in order to organize and summarize the data. Using this preliminary data analysis, the team will then use this input from veterans to expand the Digital Health Matrix Model in preparation for the pairwise comparisons. Analysis of this data will also inform the creation of the visual modeling simulations.
Due to the rapid iterative nature of this project, the second level of analysis will be completed when both rounds of veteran focus groups have been completed. The audio files will be transcribed and uploaded into the qualitative data analysis software program ATLAS.ti version 7.1 (ATLAS.ti Scientific Software Development GmbH) for comprehensive analysis. Additionally, Morae recordings will be reviewed to allow the team to document notes and relevant content in alignment with the audio recorded data. This data will be compared and compiled with qualitative transcripts to collate data assets.
Once all data are collated, content analysis will be completed in two primary steps, first, we will identify domains and taxonomic structures; and second, we will evaluate coding schemas for reliability and credibility. In the first step, we will identify codes, extract meaningful statements, and identify group domains and taxonomies. Participant comments will be organized to develop codes, and codes will be merged to develop categories. Categories will be grouped into a taxonomic structure that describes the dataset. Themes identified in the preliminary analysis will be compared to those identified in subsequent transcript analyses, and results will be integrated into a final taxonomic thematic structure [
This paper describes a novel process and protocol for developing and implementing a mixed-methods participatory approach to evaluating and understanding veterans’ preferences and vision for an integrated system of HIT electronic health resources to support health care and self-management. Leveraging expert panel stakeholders and subject matter experts, along with veteran patients as expert consultants, we describe a participatory approach that can be used in future research to dynamically evaluate user preferences for HIT systems and tools. This approach informs the development of a more integrated and connected system of electronic health resources that will support a more holistic patient experience across multiple platforms and tools, based on a patient-centered approach to virtual care. This study is finished with recruitment, is in final stages of data collection, and the preliminary stages of data analysis.
The optimal use of health information technology to improve health care delivery and help patients become active participants in their care and self-management is essential to address patients’ ongoing health care needs [
Overall the goal of this study is to inform the VA’s vision of an integrated system of HIT electronic health resources from the shared perspective of veterans, clinicians, subject matter experts, and other key stakeholders (eg, VA operational partners). This study illustrates an innovative approach to using participatory research methods with diverse stakeholders and technology resources to create a vision for an integrated user-friendly system of HIT patient-facing resources. To our knowledge, this is one of the few published protocols that inform the development of an integrated system of HIT resources within a large health care system that serves more than 2.5 million users (Veterans and Consumers Health Informatics Office, U.S. Department of Veterans Affairs, unpublished data 2014).
Although this protocol is useful in developing valuable knowledge to inform system improvements, our study has limitations. First, although our sample size will be comparable to other qualitative mixed-methods studies [
Our use of mixed-methods to collect, analyze, and converge data from distinctly different sources supports the development of a product that is informed by users, clinicians, and operational Program Office representatives to identify an integrated set of electronic health resources that focus on usability and usefulness. These efforts are guided by best practices and will support a user-based design to promote integration, synchronization, and standardization across an integrated system of patient-facing platforms and tools. In alignment with VA goals and the mission of VA’s Office of Connected Health, these data will support the development and proliferation of user-friendly electronic resources that support virtual access to care that is personalized, proactive, and patient-driven to increase access, and transform the VA’s delivery of health care [
Agency for Healthcare Research and Quality
Computer-Email-Web
eHealth Literacy Scale
health information technology
Rapid Estimate of Adult Literacy in Medicine
Department of Veterans Affairs
Veterans Health Administration
The Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development Service, and National eHealth Quality Enhancement Research Initiative (QUERI) Coordinating Center (RRP 12-495) supported the development of this manuscript. This manuscript was also supported in part by the Center of Innovation for Disability and Rehabilitation Research at the James A. Haley Veterans Hospital.
The contents of this manuscript do not represent the views of the Department of Veterans Affairs or the U.S. Government.
The authors would like to acknowledge the efforts of the VA Human Factors team, including Nancy Wilck, Abigail Noonan, and Ashley Cook for their significant contribution to the success of this project. We would also like to thank Susan Woods and Jeffrey Sartori for their generosity of time and consultation throughout the conceptualization and implementation of this project.
None declared.