Background: Young adults who experience homelessness are exposed to environments that contribute to risk behavior. However, few studies have examined how access to housing may affect the health risk behaviors of young adults experiencing homelessness.
Objective: This paper describes the Log My Life study that uses an innovative, mixed-methods approach based on geographically explicit ecological momentary assessment (EMA) through cell phone technology to understand the risk environment of young adults who have either enrolled in housing programs or are currently homeless.
Methods: For the quantitative arm, study participants age 18-27 respond to momentary surveys via a smartphone app that collects geospatial information repeatedly during a 1-week period. Both EMAs (up to 8 per day) and daily diaries are prompted to explore within-day and daily variations in emotional affect, context, and health risk behavior, while also capturing infrequent risk behaviors such as sex in exchange for goods or services. For the qualitative arm, a purposive subsample of participants who indicated engaging in risky behaviors are asked to complete an in-depth qualitative interview using an interactive, personalized geospatial map rendering of EMA responses.
Results: Recruitment began in June of 2017. To date, 170 participants enrolled in the study. Compliance with EMA and daily diary surveys was generally high. In-depth qualitative follow-ups have been conducted with 15 participants. We expect to recruit 50 additional participants and complete analyses by September of 2019.
Conclusions: Mixing the quantitative and qualitative arms in this study will provide a more complete understanding of differences in risk environments between homeless and housed young adults. Furthermore, this approach can improve recall bias and enhance ecological validity.
International Registered Report Identifier (IRRID): DERR1-10.2196/12112
Risk environment has been defined as the space—whether social or physical—in which factors external to a person interact to increase the chances of certain health risk behaviors [- ]. Young adults between the ages of 18 and 25 years old, sometimes referred to as transition-aged youth, who experience homelessness live in an unstable and sometimes chaotic risk environment that has resulted in high rates of substance use and sexually transmitted infections, including HIV [ - ]. For example, Noell et al found the incidence of sexually transmitted infections in a homeless adolescent population to be as high as 17% and the prevalence and incidence of certain infections to be 10 to 12 times higher than those found in the same age group among the general population [ ]. Homelessness services programs that provide housing to young adults have the capacity to change their risk environment, and housing might serve as a protective factor by providing safe, independent living arrangements that can alleviate stress related to being homeless [ , ]. Housing programs may also expose young adults to positive social influences, because street-based peers have been associated with risk behaviors in this population [ , ]. However, these programs may also change the contextual factors that influence health risk behaviors in unforeseen or unexpected ways; for example, being placed in a housing program may increase social pressures from network members who are still experiencing homelessness and need a place to stay or the privacy afforded by housing could permit misuse and sale of drugs or easier engagement in risky sex [ , ]. To date, there has been limited investigation of young adults with a history of homelessness who have enrolled in housing programs [ ] or differences in the risk environment between those experiencing homelessness and those who have moved into housing programs. Furthermore, past studies typically relied on self-report methods that are burdened by recall bias [ , , ].
The Log My Life (LML) study seeks to fill this gap in the literature by developing an innovative, mixed-methods approach using geographically explicit ecological momentary assessments (GEMA) to understand the risk environment of young adults who have either enrolled in housing programs or are currently experiencing homelessness. GEMA is considered the gold standard for capturing valid intensive longitudinal self-report information that is embedded in important contextual factors and can be used to understand and predict health risk behaviors [- ]. The conceptual model for this study (see ) is based on prior research, showing that within-day variation in various psychosocial characteristics (eg, mood and substance use craving) affects both drug use and sexually risky behavior [ - ]. We hypothesize that access to housing for individuals experiencing homelessness will affect where, when, and with whom they spend time daily [ - ]. Furthermore, we hypothesize that these contextual factors can influence HIV risk and within-day psychosocial characteristics [ , , - ], which in turn could impact one’s ability to access or maintain stable housing.
This paper describes the protocols of the LML study and highlights innovative aspects of its design, including the use of geospatial data, ecological momentary assessments (EMA), dynamic social contexts, and in-depth interviews to assess the built and social context and psychosocial factors influencing risky health behavior. We also present preliminary recruitment progress to date and describe additional avenues of potential inquiry.
Leveraging the widespread use of smartphone technology, including among homeless populations [, ], this study uses a mixed-methods, prospective longitudinal design to recruit young adults who are in either of the following sampling frames: (1) enrolled in a housing program (eg, transitional housing and permanent supportive housing) or (2) currently homeless. For the quantitative arm, participants in both sampling frames complete a baseline questionnaire and are observed for 1 week by responding to repeated momentary surveys prompted and administered through a smartphone app developed for this study. A 7-day period was chosen to have adequate power (ie, 8 prompts per day for a total of 56 within-subject observations) to detect within-subject changes over time and to capture variation in behavior based on the day of the week. Both EMAs and daily diaries are prompted, and responses are used to explore within-day and daily variations in emotional affect, context, and risk behavior, while also capturing infrequent risk behaviors such as sex in exchange for goods or services. The geospatial location of the participant is also continuously recorded for the monitoring period, to the degree permitted by the phone. For the qualitative arm, a purposive subsample of participants who indicated engaging in risky behaviors are asked to complete an in-depth qualitative interview using an interactive, personalized geospatial map rendering of EMA responses as an elicitation device. Similar methods have been used to better understand contextual factors that influence tobacco use [ ]. Mixing the quantitative and qualitative arms in this study provides a more complete understanding of differences in risk environments between homeless and formerly homeless young adults. Study protocols were approved by the institutional review board at the University of Southern California.
Participants include young adults residing in Los Angeles County who either have or are experiencing homelessness. Approximately 200 participants are currently being recruited from 11 agencies that run permanent supportive or transitional living housing programs as well as from shelter sites and drop-in facilities serving youth experiencing homelessness. To investigate the various contextual mechanisms that could explain risk behaviors among young adults, a power analysis was conducted to determine that we need approximately 100 young adults in housing programs and 100 young adults who remain homeless. In both sampling frames, individuals are eligible to participate if they can be interviewed in English, can read and understand smartphone items in English without assistance, and are willing to provide written informed consent. To be included in the housed sample, young adults must be aged between 18 and 25 years and residing in a housing program that serves homeless young adults. Individuals as old as 27 years are allowed to participate if they entered the housing program before the age of 25 years. Young adults are considered to be part of the unhoused sample if they are aged between 18 and 25 years and meet the McKinney-Vento Homeless Assistance Act  definition of homelessness that specifies lack of a fixed, regular, and adequate nighttime residence.
Young adults are being recruited through flyers and informational sessions held at housing programs or drop-in facilities. Study staff members have or will recruit youth at 6 permanent housing programs, 9 transitional living programs, and 6 drop-in facilities across Los Angeles County. Young adults recruited at permanent or transitional living programs are considered to be eligible if they are enrolled in the housing program and fit inclusion criteria. During informational sessions at drop-in sites, youth complete a self-administered screener on an electronic tablet that indicates whether they meet the eligibility criteria for homelessness or are enrolled in a housing program.
Upon enrollment, participants receive an iPad (Apple, USA) to complete a self-administered questionnaire via a secure Web-based platform. Due to the potentially sensitive nature of the questions, the questionnaire is administered using computer-assisted self-interviewing techniques. These baseline meetings last approximately 45 to 75 min and include a questionnaire with 2 components: one that assesses demographics and historical experiences and another that explores participants’ social network (subsequently described). Participants then have the option to use a study-provided phone, usually a third-generation MotoG (Motorola, USA) smartphone that has an unlimited data plan, or their personal smartphone if they own an Android-based phone that is compatible with the study smartphone app. Youth who agree to use their personal phone receive an additional US $10 to offset the cost of cellular data. Throughout the study period, momentary and daily surveys are prompted using a custom software app for smartphones running the Android operating system (Google, USA). Participants can earn up to US $130 for completing the main study components, but some incentives are task-based (ie, each participant’s total incentive amount is driven by compliance with prompted EMAs and daily diaries). Research staff members assist the participant in setting up the smartphone app during baseline meetings, and each participant completes 1 practice EMA and daily diary demonstration. During setup, participants specify normal sleep and wake times, so they do not receive prompts outside their typical waking hours.
For the next 7 days, participants complete EMAs on the smartphone app. EMAs allow repeated collection of real-time data, eliminate the need for retrospective recall, and are particularly well suited for examining episodic behavior that may be affected by context such as substance use [, - ]. Participants receive 7 to 8 prompts per day, depending on the number of waking hours. Participants are prompted randomly during 30-min windows separated by 2-hour intervals. For example, if a participant is normally awake from 9 am to 11 pm, 1 EMA survey will be triggered randomly between 9:30 am and 10 am, 11:30 am and 12 pm, 1:30 pm and 2 pm, and so on. The app delivers a push notification with a chime or vibration, if the phone is not set to silent. Participants are instructed to stop their current activity and complete a short EMA survey on their phone. This process requires about 1 to 3 min. If no entry is made, the app emits up to 3 push notifications at 3-min intervals (3, 6, and 9 min after the first prompt). Each EMA survey becomes inaccessible 10 min after the initial prompt, unless the participant is answering the survey. Participants are instructed to ignore signals that occur during an incompatible activity (eg, driving, sleeping, or bathing). EMA surveys contain skip logic to minimize burden on the participant while optimizing the quality of data received in each brief survey. A minimum of 15 items appear at each EMA prompt. A maximum of 31 items appear in the event that a participant reports all permutations of risk behavior and social context. Soliciting multiple EMA entries per day has been shown to be acceptable in previous studies with youth and adults [ - ] and had been pilot tested for this study.
Starting on the second day of the observation period and continuing for 7 consecutive days, participants are asked to complete a daily diary in which they reflect on their behavior during the previous day. Participants can self-initiate and complete the daily diary at any point during the day, but they are also automatically prompted to do so if the daily diary has not already been completed. Participants select 3 times throughout the day (eg, 9 am, 12 pm, and 2 pm) to receive reminder prompts to complete the daily diary. Reminders do not deploy within 15 min of EMA prompting windows to avoid conflict between surveys. Although momentary data capture can be considered an improvement over self-report methods, there is still a need to include daily assessments as a complement to EMA as we continue to build these methods . Capturing assessments in both momentary and daily methods will provide an opportunity to evaluate how well daily diaries replicate momentary data. Use of daily methods permits more flexibility in how questions are asked. In addition, daily diaries are useful for querying infrequent risk behaviors that would be unnecessary to ask multiple times per day (eg, frequency of sex), as is often done with studies deploying EMA methods. To date, studies comparing collection methods have found close approximate aggregated ratings of behavior or affect between daily and momentary collection [ ], but daily collection provides a much better representation of real-time experiences than longer recall methods [ , ]. Responses on both EMA and daily diaries are encrypted, wirelessly uploaded after each entry, and stored on a server accessible to the research team for compliance monitoring.
During the monitoring week, study personnel contact participants by phone twice to check on progress, encourage compliance, and address any technical issues. Participants can also email, call, or text a study helpline number any time they have issues or questions. Google Voice is used to maintain a record of calls, texts, and emails as well as allow multiple staff members to address concerns and mask their own personal phone numbers. After the monitoring period is completed, participants meet with the study staff to complete a 30-min exit appointment, during which they respond to additional questionnaires, receive compensation (calculated based on their compliance), and return the study phone and charger if borrowed.
Study personnel invite a purposive subsample of study participants (n=30) who (1) indicate high-risk behavior (eg, hard drug use and frequent alcohol or marijuana use) during their observation week or during their lifetime and (2) display adequate compliance (ie, 70% or greater) on EMA and daily diaries to participate in an additional in-depth, 45- to 60-min qualitative interview. This interview is used to explore dynamic socioenvironmental factors that affect health risk behaviors and how youth navigate risky environments. The structured open-ended interview also uses an interactive, personalized geospatial map rendering of EMA responses that are generated through the smartphone’s built-in location-finding system as an elicitation device, similar to a method proposed by McQuoid et al . These interviews are conducted in private rooms at drop-in sites or housing programs. Audio of the session are recorded and transcribed using a professional service, and participants receive US $30.
Baseline and Exit Questionnaires
The baseline questionnaire addresses factors and characteristics shown to be related to housing stability among youth [, - ], including demographics (eg, age, sex, gender, race and ethnicity, education, income, and current and past employment), physical and mental health conditions, lifetime and recent drug and alcohol use, trauma history, homelessness history, life skill development, and emotional regulation. Participant mental health is assessed with the Patient Health Questionnaire [ , ], General Anxiety Disorder Scale [ ], Primary Care PTSD Screener [ ], Difficulties in Emotion Regulation Scale Short Form [ ], Perceived Stress Scale [ ], and the COPE scale [ ], all of which have been validated with young adult populations. Mental health diagnoses, suicidality, and care engagement are also assessed. Participants rate their current physical health, list their chronic illnesses, assess sleep impairment, and complete a checklist about difficulties in health care access. Items to address youth sexual history and sex-related HIV risk behaviors are based on the Centers for Disease Control and Prevention Youth Risk Behavior Survey’s sexual behaviors subsection [ - ], a validated tool that details past 90-day sexual behaviors of adolescents. Items in this portion of the baseline questionnaire also assess HIV and sexually transmitted infection testing, status, and treatment and are based on previous studies by this research team [ , ]. Youth provide a detailed account of lifetime and past 30-day substance use, including type of drug, typical quantity and frequency of consumption, and route of administration (eg, injected, swallowed, or smoked). Probable alcohol or other substance use disorder is determined with the CAGE substance abuse screening tool [ , ], a short, validated assessment that examines substance use issues among individuals aged 16 years or above. In addition to risky behaviors, positive youth development, such as ability to communicate, look forward, manage money, or practice self-care, is measured with the Ansell-Casey Life Skills assessment [ , ].
The baseline questionnaire additionally focuses on participants’ historical life experiences such as duration of homelessness, foster care involvement, and justice system involvement, also based on measures used in other studies with homeless youth [, ]. Childhood trauma is assessed with the University of California, Los Angeles Post-Traumatic Stress Disorder Reaction Index for the Diagnostic and Statistical Manual of Mental Disorders-IV [ ], and participants also reflect on past experiences with trauma in their community, including incidents of robbery, violent assault, intimate partner violence, or other experiences perceived as discrimination. Participants are asked to recall their past experiences with police and gangs and discuss their access to guns. In addition to past experiences, youth complete the Stress on the Streets Checklist [ ] to describe their current level of stress with their living environment and the Food Insecurity Scale [ ] to assess their typical access to food. Finally, youth comment on their knowledge of housing options, engagement with meaningful activities, and access to supportive services (eg, drop-in center, shelter, and mental health counseling).
To assess social networks, participants also complete a short egocentric social network inventory (based on REALYST ; also refer to the study by Burt [ ]). Participants initially identify 5 people (commonly referred to as alters in social network analysis [ ]) that they interact with most frequently (eg, friend from the street, family member, romantic partner, or caseworker). Then, participants respond to questions about each named individual’s characteristics, including the alter’s gender, age, sexual orientation, race and ethnicity, nature of relationship, frequency of contact, substance use belief and behaviors, sexual beliefs and behaviors, and whether the alter is a source of advice or support.
During the exit meeting, participants complete an additional computer-assisted questionnaire that includes items about life skill development  and if housed, their current housing experience based on items from the Housing Experience Survey [ ]. During the exit meeting, youth have an opportunity to describe any events that may have made the observation week atypical and reflect on their experience in the study, such as whether EMA prompts or time spent responding to surveys interfered with their daily life, caused any stress or anxiety, or altered their typical behavior. Participants also indicate the extent to which they felt comfortable answering items honestly and accurately and whether they generally had a negative or positive experience using the smartphone app.
Ecological Momentary Assessment and Daily Diaries
Ecological Momentary Assessments
Surveys prompted by the phone during each EMA query momentary positive and negative affect, hunger, significant events (eg, involved in a physical fight), and further details and contextual factors concerning alcohol use, other drug use, and temptation to drink or use drugs. These items have been successfully applied in other EMA studies of affect and substance use [, , , - ]. EMA also prompts request information about the participant’s social context (subsequently described), and each response is time stamped. A complete list of EMA items and their response options along with sample screenshots can be found in and .
Daily diaries capture risk behaviors of the previous day and infrequent behavior that may be missed by EMAs. In daily diaries, participants respond to items that aim to provide more in-depth details about any drug use events (eg, quantity and mode of use) or sexual encounters that occurred during the previous day (eg, partner’s gender, nature of relationship with partner, and use of a condom). Participants also reflect on their sleep behavior during the past evening, including the duration, location, and quality. Items in the daily diary, available in, were adapted from the Youth Risk Behavior Surveillance System survey [ ] and tools developed by our team in previous studies [ ].
Dynamic Social Context
The names of the 5 individuals with whom the participant interacts most (ie, alters) elicited from the baseline questionnaire are entered into the smartphone app during setup. The app stores each entry, subsequently adding the variables as responses into the social context items of each EMA and daily diary. At the start of each EMA (see), participants select which alters (if any) they interacted with during the past 2 hours; if any alters were present, participants indicate whether alcohol, tobacco, or other drugs were consumed. A participant may also specify the presence of other individuals, as desired. If a participant reports interacting with someone other than 1 of the 5 alters, an additional follow-up question gathers data on the relationship between that person and the participant. Although EMA items assess the presence of alters and other individuals at a given moment, daily diaries assess with which specific alters the participant may have consumed alcohol or drugs, if at all (see ). Relevant alter-level characteristics from the baseline questionnaire (eg, alter uses illicit substances) can be used to generate risk profiles corresponding to the social context of each assessed interaction. For example, prompt-level data may contain a sum of the total number of alters who use illicit substances with whom the respondent reported interacting during that prompt (range: 0-5), and day-level data may contain the overall sum of illicit substance–using alters across all prompts that day or the proportion of prompts to which the respondent reported interacting with any substance-using alter.
Location and Geographic Information Systems Data
Location data are collected once every minute using a background system process on the participant’s device. Android’s location system uses a multiple-mode sensing method to estimate location relying on a combination of WiFi, cellular triangulation, and global positioning system (GPS) satellites. The software reports accuracy as a 68% CI (1 SD) in meters; epochs with a CI greater than 100 meters were excluded. Activity spaces for participants are defined at the day level using minimum convex hulls and standard deviational ellipses (at 1, 2, and 3 SD). A minimum convex hull is a rudimentary algorithm that creates the smallest possible simple convex polygon encompassing all the points in a dataset, whereas standard deviational ellipses are mean-centered ellipses that cover 68%, 95%, and 99% of GPS data, depending on the specified SD . Location activity spaces are calculated using Zone 5 of the California State Plane Coordinate System (Datum: WSG84) based on the expected geospatial distribution of data. Coordinates for activity spaces can be used to map additional data layers using geographic information systems, such as crime rate and proximity to services.
In-Depth Follow-Up Qualitative Interviews
As part of the in-depth qualitative follow-up interviews, interactive geospatial maps personalized with GEMA data are shown to participants as a visual elicitation tool to explore participant risk behavior and living environment. Maps are generated in Google Maps (see example in), and responses can be displayed on satellite imagery, a road map, or Google Street View. The maps are generated using location sensor data and momentary self-report survey data. They display where the participant traveled during the monitoring week and the locations of any drug use, stressful events, and particular areas participants may have felt positive or negative emotions (ie, happier or sadder than usual).
Interviewers first highlight responses associated with higher levels of risk behaviors (ie, drug use or risky sexual behavior) and ask participants to discuss these instances and any patterns they perceive. For example, interview questions that aim to solicit a conversation on substance use and the social contexts that affect use include: “What are your thoughts as to whether these locations influenced your using?” and “You also indicated you were/weren’t with [list any alter identified]? What role do you think this person(s) played in your using?” During the interview, participants can interact with the map, and different responses can be displayed based on any set of EMA items. Interviewers are also trained to request geospatial identifiers for daily activities, interaction with network members, and HIV risk and prevention behaviors that are not already part of the EMA response.
Quantitative Data Integration
EMA and daily diary data from smartphones are encrypted and then subsequently wirelessly uploaded to a secure server for further data processing. The differences in temporality between location (minute level), EMA (multiple times per day), daily diary (day level), and questionnaire (week level) data are reconciled after questionnaire data are deidentified and location and EMA data are unencrypted. Days are offset by 3 hours, ending at 2:59 am and beginning at 3:00 am to account for delayed sleep schedules in the study population. Coordinates for mean latitude and longitude during the 30-min period surrounding an EMA prompt are generated based on minute-level location data. An aggregate measure is used to limit missing data in the event that a location estimate is unavailable at the exact moment a participant answered a survey. Similarly, daily diary location data is generated using the previous day’s coordinates for mean latitude and longitude, in addition to the area of the minimum convex hull and standard deviational ellipses for that day. Daily diary data are then merged with EMA data, repeating daily diary entries across all prompts for each person-day, and questionnaire data are merged, repeating questionnaire responses across all prompts for each participant.
All data are screened for violations of statistical assumptions, such as non-normality or outliers, and transformed to satisfy assumptions for subsequent data analyses. Pairwise correlations are used to screen for multicollinearity and exclude variables that represent similar constructs. Generalized linear models are used to examine relationships between baseline items (ie, demographics and history) with exit questionnaire outcomes. Furthermore, these models are used to predict the likelihood that an individual is assigned to supportive housing for each individual factor, as depicted in.
Generalized linear mixed models (GLMMs) are used to address the primary aims of the study for day-level and intraday analyses and missing data analyses. Given the expected differences in contextual and psychosocial factors contingent on housing status (see), GLMMs are conducted separately for participants in supportive housing and participants who are not housed. Model fit parameters, such as random effects and specification of variance-covariance matrices, are specified for each GLMM. All level 1 (ie, within-subject) predictors (eg, affect and hunger) are disaggregated into 2 variables through grand-mean centering and person-mean centering to examine both interindividual and intraindividual effects on the outcomes, respectively. Level 2 (ie, between-subject) factors act as covariates in all GLMMs. Depending on the outcome, contextual or psychosocial factors may act as intraday predictors, mediators, or moderators.
Qualitative Data Analysis and Integration
In-depth qualitative interviews are analyzed using a comparative case study analysis . Case studies emphasize uniqueness in context and are used to consider complex phenomenon with interrelated influences that can exist on multiple levels (eg, individual, interpersonal, and structural). Following standard procedures for case study analysis [ , ], a case record or summary is developed for each participant using information extracted from transcripts. Transcripts are thematically analyzed using codes that help identify whether risk or protective factors are influenced by participants’ social networks, housing environment, activities participants tend to engage in, or some combination. The use of a case summary matrix that displays salient information from individual case summaries in a table format facilitates cross-case comparisons that can be used to identify broader themes [ ].
Integrating qualitative and quantitative findings is done by adding significant quantitative findings to the case summary matrix to facilitate discussion of comparisons between the results. This triangulating process is used to determine the extent to which qualitative findings converge with, are complementary to, or expand upon the quantitative findings [, ]. Discrepant findings are also noted and further considered. This information is then added to the mixed-methods matrix [ ].
Recruitment began in June of 2017. To date, 185 people have attended information sessions and were screened to participate in the study. Furthermore, 170 individuals enrolled in the study and 165 started EMA (). Out of the 159 participants with usable data, 6 did not initially complete the protocol but then restarted the EMA and daily log component to satisfy study parameters, either by borrowing a new study phone (n=4), restarting on their personal phone after losing a study phone (n=1), or restarting on a new study phone after attempting to finish on their personal phone (n=1). Overall, 6 study phones were not returned at the end of the study, although 1 participant restarted the EMA protocol on a personal phone and the other participants opted to complete the exit questionnaire. In-depth qualitative follow-ups have been conducted with 9.5% (15/158) participants with usable data. Initial analyses of quantitative and qualitative data have begun with mixed-methods analysis planned for the near future. We expect to recruit 50 additional participants and complete analyses by September of 2019.
This paper presents the protocols of a mixed-methods prospective longitudinal study designed to explore risk behavior of recently or currently homeless youth. This study is one of the few studies that have used EMA with homeless populations [, ] and is the first, to our knowledge, that incorporates social networks with specific, identified alters in EMA, which can help improve our understanding of the role of social context. This study is also one of the first to leverage smartphone location-sensing capabilities and phone-based EMA to permit participants to report on a variety of contextual (location and physical or social surroundings) and psychosocial factors that vary throughout the day (current affect, substance use cravings, and hunger) [ ]. The use of GEMA has clear methodological advantages over other approaches used in studies on homeless youth behavior, in that it can improve recall bias and ecological validity and is particularly well suited to studying substance use given its episodic nature and relation to context and current affect [ ]. The use of geographic information systems also provides the ability to overlay neighborhood- or community-level data such as crime rates or density of alcohol outlets or marijuana dispensaries.
Strengths and Limitations
We note the importance of using a mixed-methods GEMA that can provide insights into the strengths and weaknesses of traditional EMA studies . EMA surveys are generally short and do not necessarily capture the richness of experiences, which is something that this study addresses through in-depth qualitative interviews that incorporate geographical considerations. Further consideration will be needed when integrating quantitative and qualitative data, especially if those data sources appear to be in conflict. Although this is the first study to use this type of mixed-methods approach to examine substance use and sexual risk behavior among homeless youth, mixed-methods studies that have included geographical data have focused on how environment affects access to public transit [ ], arrhythmia in old age [ ], or tobacco use [ ].
Although the goal of this paper is to describe study protocols and highlight innovative aspects of the study design, we also note that the study has been successful in recruiting homeless youth who have been considered hard to engage and formerly homeless youth living in housing programs who have been understudied . To date, we have had high EMA compliance and few lost phones. Further efforts are needed to understand if an incentive structure based on compliance, which has resulted in high compliance, may adversely affect data quality. Other potential challenges may include discrepancies between qualitative and quantitative reporting, missing or inaccurate geospatial data, and youth failing to admit to drug use or other risky behavior due to stigma.
Despite limitations, this mixed-methods design provides rich data difficult to collect with traditional survey methodology. We know context influences health behavior , but we know little about the daily environments of recently homeless young adults, particularly how their movements in space and time can result in dangerous substance use or sexual activity. At a minimum, information gained from this project can inform providers of the typical risks experienced by their clients and potentially inform structural interventions in housing programs. Geospatial data could point to hotspots (see Veldhuizen et al [ ]) or specific risky environments in which young adults often interact. Furthermore, ecologically valid methods allow researchers to approximate what might be occurring in real time and bring us one step closer to identifying leverage points where intervention might be particularly fruitful. The combination of EMA and geospatial data can greatly enhance the development of ecological momentary interventions or just-in-time adaptive interventions [ , ]. Theoretically, these interventions harness the power of mobile phone technology, particularly geospatial sensors and app-based momentary prompts, to intervene at just the right time to shift an individual’s behavior. Mobile interventions are potentially affordable solutions to intervening among populations that are typically hard to reach. We now have the technology to deliver preventive interventions in situ, but we do not yet fully understand the dynamic nature of health risk behaviors [ ]. We hope that studies such as this can begin to untangle this complexity.
This project was supported by funding from the National Institutes of Health/National Institute of Mental Health (1R01MH110206). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or National Institute of Mental Health.
Conflicts of Interest
Multimedia Appendix 1
Ecological momentary assessment (EMA) items.PDF File (Adobe PDF File), 40KB
Multimedia Appendix 2
Sample smartphone app screen images.PDF File (Adobe PDF File), 147KB
Multimedia Appendix 3
Daily log items.PDF File (Adobe PDF File), 47KB
- Latkin C, German D, Vlahov D, Galea S. Neighborhoods and HIV: a social ecological approach to prevention and care. Am Psychol 2013;68(4):210-224 [FREE Full text] [CrossRef] [Medline]
- Rhodes T, Simic M. Transition and the HIV risk environment. Br Med J 2005 Jul 23;331(7510):220-223 [FREE Full text] [CrossRef] [Medline]
- Strathdee S, Hallett T, Bobrova N, Rhodes T, Booth R, Abdool R, et al. HIV and risk environment for injecting drug users: the past, present, and future. Lancet 2010 Jul;376(9737):268-284 [FREE Full text] [CrossRef]
- Beech B, Myers L, Beech D. Hepatitis B and C infections among homeless adolescents. Fam Community Health 2002;25(2):28-36 [FREE Full text] [CrossRef]
- Rice E, Milburn N, Monro W. Social networking technology, social network composition, and reductions in substance use among homeless adolescents. Prev Sci 2011 Mar;12(1):80-88 [FREE Full text] [CrossRef] [Medline]
- Rice E. The positive role of social networks and social networking technology in the condom-using behaviors of homeless young people. Public Health Rep 2010;125(4):588-595 [FREE Full text] [CrossRef] [Medline]
- Noell J, Rohde P, Ochs L, Yovanoff P, Alter MJ, Schmid S, et al. Incidence and prevalence of chlamydia, herpes, and viral hepatitis in a homeless adolescent population. Sex Transm Dis 2001;28(1):4-10 [FREE Full text] [CrossRef]
- Coates J, McKenzie-Mohr S. Out of the frying pan, into the fire: trauma in the lives of homeless youth prior to and during homelessness. J Sociol Soc Welfare 2010 Dec;37(4):65-96 [FREE Full text]
- Edidin J, Ganim Z, Hunter S, Karnik N. The mental and physical health of homeless youth: a literature review. Child Psychiatry Hum Dev 2012 Jun;43(3):354-375 [FREE Full text] [CrossRef] [Medline]
- Rice E, Monro W, Barman-Adhikari A, Young S. Internet use, social networking, and HIV/AIDS risk for homeless adolescents. J Adolesc Health 2010 Dec;47(6):610-613 [FREE Full text] [CrossRef] [Medline]
- Rice E, Barman-Adhikari A, Milburn N, Monro W. Position-specific HIV risk in a large network of homeless youths. Am J Public Health 2012 Jan;102(1):141-147 [FREE Full text] [CrossRef] [Medline]
- Henwood B, Rhoades H, Hsu H, Couture J, Rice E, Wenzel S. Changes in social networks and HIV risk behaviors among homeless adults transitioning into permanent supportive housing: a mixed-methods pilot study. J Mix Methods Res 2016 Jul 08;11(1):124-137 [FREE Full text] [CrossRef]
- Wenzel S, Rice E, Henwood B, Rhoades H. Grantome. Bethesda, MD: National Institute of Drug Abuse; 2014. HIV Risk, Drug Use, Social Networks: Homeless Persons Transitioned to Housing URL: http://grantome.com/grant/NIH/R01-DA036345-04 [accessed 2018-11-22] [WebCite Cache]
- Pierce S, Grady B, Holtzen H. Daybreak in Dayton: assessing characteristics and outcomes of previously homeless youth living in transitional housing. Child Youth Serv Rev 2018 May;88:249-256 [FREE Full text] [CrossRef]
- McQuoid J, Thrul J, Ling P. A geographically explicit ecological momentary assessment (GEMA) mixed method for understanding substance use. Soc Sci Med 2018 Dec;202:89-98 [FREE Full text] [CrossRef] [Medline]
- Kirchner T, Shiffman S. Spatio-temporal determinants of mental health and well-being: advances in geographically-explicit ecological momentary assessment (GEMA). Soc Psychiatry Psychiatr Epidemiol 2016 Dec;51(9):1211-1223 [FREE Full text] [CrossRef] [Medline]
- Stone A, Shiffman S. Ecological validity for patient reported outcomes. In: Steptoe A, editor. Handbook of Behavioral Medicine: Methods and Applications. New York, NY: Springer; 2010:99-112.
- Hufford M, Shiffman S, Paty J, Stone A. Ecological momentary assessment: Real-world, real-time measurement of patient experience. In: Fahrenberg J, Myrtek M, editors. Progress in Ambulatory Assessment: Computer-Assisted Psychological and Psychophysiological Methods in Monitoring and Field Studies. Ashland, NC: Hogrefe & Huber; 2001:69-92.
- Shiffman S, Stone A, Hufford M. Ecological momentary assessment. Annu Rev Clin Psychol 2008 Apr;4(1):1-32 [FREE Full text] [CrossRef]
- Santa Maria D, Padhye N, Yang Y, Gallardo K, Santos G, Jung J, et al. Drug use patterns and predictors among homeless youth: results of an ecological momentary assessment. Am J Drug Alcohol Abuse 2018;44(5):551-560 [FREE Full text] [CrossRef] [Medline]
- Linas B, Latkin C, Westergaard R, Chang L, Bollinger R, Genz A, et al. Capturing illicit drug use where and when it happens: an ecological momentary assessment of the social, physical and activity environment of using versus craving illicit drugs. Addiction 2015 Feb;110(2):315-325 [FREE Full text] [CrossRef] [Medline]
- Epstein D, Marrone G, Heishman S, Schmittner J, Preston K. Tobacco, cocaine, and heroin: craving and use during daily life. Addict Behav 2010 Apr;35(4):318-324 [FREE Full text] [CrossRef] [Medline]
- Epstein D, Willner-Reid J, Vahabzadeh M, Mezghanni M, Lin J, Preston K. Real-time electronic diary reports of cue exposure and mood in the hours before cocaine and heroin craving and use. Arch Gen Psychiatry 2009 Jan;66(1):88-94 [FREE Full text] [CrossRef] [Medline]
- Kennedy A, Epstein D, Phillips K, Preston K. Sex differences in cocaine/heroin users: drug-use triggers and craving in daily life. Drug Alcohol Depend 2013 Sep 01;132(1-2):29-37 [FREE Full text] [CrossRef] [Medline]
- Lin JL, Vahabzadeh M, Mezghanni M, Epstein DH, Preston KL. A high-level specification for adaptive ecological momentary assessment: real-time assessment of drug craving, use and abstinence. AMIA Annu Symp Proc 2005:455-459 [FREE Full text] [Medline]
- Milburn N, Rice E, Rotheram-Borus M, Mallett S, Rosenthal D, Batterham P, et al. Adolescents exiting homelessness over two years: the risk amplification and abatement model. J Res Adolesc 2009 Dec 01;19(4):762-785 [FREE Full text] [CrossRef] [Medline]
- Padgett D. There's no place like (a) home: ontological security among persons with serious mental illness in the United States. Soc Sci Med 2007 May;64(9):1925-1936 [FREE Full text] [CrossRef] [Medline]
- Padgett D, Henwood B, Abrams C, Drake R. Social relationships among persons who have experienced serious mental illness, substance abuse, and homelessness: implications for recovery. Am J Orthopsychiatry 2008 Jul;78(3):333-339 [FREE Full text] [CrossRef] [Medline]
- Jennings J, Woods S, Curriero F. The spatial and temporal association of neighborhood drug markets and rates of sexually transmitted infections in an urban setting. Health Place 2013 Sep;23:128-137 [FREE Full text] [CrossRef] [Medline]
- Skelly A, Arcury T, Gesler W, Cravey A, Dougherty M, Washburn S, et al. Sociospatial knowledge networks: appraising community as place. Res Nurs Health 2002 Mar 14;25(2):159-170 [FREE Full text] [CrossRef]
- Petering R, Rice E, Rhoades H, Winetrobe H. The social networks of homeless youth experiencing intimate partner violence. J Interpers Violence 2014 Aug;29(12):2172-2191 [FREE Full text] [CrossRef] [Medline]
- El-Bassel N, Gilbert L, Wu E, Chang M. A social network profile and HIV risk among men on methadone: do social networks matter? J Urban Health 2006 Jul;83(4):602-613 [FREE Full text] [CrossRef] [Medline]
- Friedman S, Mateu-Gelabert P, Curtis R, Maslow C, Bolyard M, Sandoval M, et al. Social capital or networks, negotiations, and norms? A neighborhood case study. Am J Prev Med 2007 Jun;32(6 Suppl):S160-S170 [FREE Full text] [CrossRef] [Medline]
- Latkin C, Forman V, Knowlton A, Sherman S. Norms, social networks, and HIV-related risk behaviors among urban disadvantaged drug users. Soc Sci Med 2003 Feb;56(3):465-476 [FREE Full text] [CrossRef]
- Rice E, Lee A, Taitt S. Cell phone use among homeless youth: potential for new health interventions and research. J Urban Health 2011 Dec;88(6):1175-1182 [FREE Full text] [CrossRef] [Medline]
- Rhoades H, Wenzel S, Rice E, Winetrobe H, Henwood B. No digital divide? Technology use among homeless adults. J Soc Distr Homeless 2017 Mar 22;26(1):73-77 [FREE Full text] [CrossRef]
- United States Congress. 2009. The McKinney-Vento Homeless Assistance Act As amended by S. 896 The Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act of 2009 URL: https://www.hudexchange.info/resources/documents/HomelessAssistanceActAmendedbyHEARTH.pdf [accessed 2018-11-23] [WebCite Cache]
- Tyler K, Olson K. Examining the feasibility of ecological momentary assessment using short message service surveying with homeless youth: lessons learned. Field Methods 2018 Apr 16;30(2):91-104 [FREE Full text] [CrossRef]
- Field M. Ecological momentary assessment - new insights and opportunities. Addiction 2015 Jun;110(6):1043-1044. [CrossRef] [Medline]
- Smyth J, Smyth JM. Ecological momentary assessment research in behavioral medicine. J Happiness Stud 2003;4(1):35-52 [FREE Full text] [CrossRef]
- Dunton G, Atienza A, Castro C, King A. Using ecological momentary assessment to examine antecedents and correlates of physical activity bouts in adults age 50+ years: a pilot study. Ann Behav Med 2009 Dec;38(3):249-255 [FREE Full text] [CrossRef] [Medline]
- Dunton G, Kawabata K, Intille S, Wolch J, Pentz M. Assessing the social and physical contexts of children's leisure-time physical activity: an ecological momentary assessment study. Am J Health Promot 2012;26(3):135-142 [FREE Full text] [CrossRef] [Medline]
- Dunton G, Liao Y, Intille S, Wolch J, Pentz M. Physical and social contextual influences on children?s leisure-time physical activity: an ecological momentary assessment study. J Phys Act Health 2011 Jan;8(s1):S103-S108 [FREE Full text] [CrossRef]
- Dunton G, Whalen C, Jamner L, Floro J. Mapping the social and physical contexts of physical activity across adolescence using ecological momentary assessment. Ann Behav Med 2007 Jun;34(2):144-153 [FREE Full text] [CrossRef]
- Trull T, Ebner-Priemer U. Using experience sampling methods/ecological momentary assessment (ESM/EMA) in clinical assessment and clinical research: introduction to the special section. Psychol Assess 2009 Dec;21(4):457-462 [FREE Full text] [CrossRef] [Medline]
- Broderick JE, Schwartz JE, Schneider S, Stone AA. Can End-of-day reports replace momentary assessment of pain and fatigue? J Pain 2009 Mar;10(3):274-281 [FREE Full text] [CrossRef] [Medline]
- Stockwell T, Donath S, Cooper-Stanbury M, Chikritzhs T, Catalano P, Mateo C. Under-reporting of alcohol consumption in household surveys: a comparison of quantity-frequency, graduated-frequency and recent recall. Addiction 2004 Aug;99(8):1024-1033 [FREE Full text] [CrossRef] [Medline]
- Broderick J, Schwartz J, Vikingstad G, Pribbernow M, Grossman S, Stone A. The accuracy of pain and fatigue items across different reporting periods. Pain 2008 Sep 30;139(1):146-157 [FREE Full text] [CrossRef] [Medline]
- Perron J, Cleverley K, Kidd S. Resilience, loneliness, and psychological distress among homeless youth. Arch Psychiatr Nurs 2014 Aug;28(4):226-229 [FREE Full text] [CrossRef] [Medline]
- Tyler K, Melander L. Child abuse, street victimization, and substance use among homeless young adults. Youth Soc 2013 Jan 06;47(4):502-519 [FREE Full text] [CrossRef]
- Heerde J, Scholes-Balog K, Hemphill S. Associations between youth homelessness, sexual offenses, sexual victimization, and sexual risk behaviors: a systematic literature review. Arch Sex Behav 2015 Jan;44(1):181-212 [FREE Full text] [CrossRef] [Medline]
- Kroenke K, Spitzer R, Williams J. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef]
- Spitzer R, Williams J, Kroenke K, Linzer M, deGruy FV, Hahn S, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. J Am Med Assoc 1994 Dec 14;272(22):1749-1756. [Medline]
- Spitzer R, Kroenke K, Williams J, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
- Prins A, Ouimette P, Kimerling R, Camerond RP, Hugelshofer D, Shaw-Hegwer J, et al. The primary care PTSD screen (PC–PTSD): development and operating characteristics. Prim Care Psych 2004 Jan 01;9(1):9-14 [FREE Full text] [CrossRef]
- Kaufman E, Xia M, Fosco G, Yaptangco M, Skidmore C, Crowell S. The Difficulties in Emotion Regulation Scale Short Form (DERS-SF): validation and replication in adolescent and adult samples. J Psychopathol Behav Assess 2015 Nov 23;38(3):443-455 [FREE Full text] [CrossRef]
- Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983 Dec;24(4):385 [FREE Full text] [CrossRef]
- Carver C, Scheier M, Weintraub J. Assessing coping strategies: a theoretically based approach. J Pers Soc Psychol 1989;56(2):267-283 [FREE Full text] [CrossRef]
- Brener N, Collins J, Kann L, Warren C, Williams B. Reliability of the Youth Risk Behavior Survey Questionnaire. Am J Epidemiol 1995 Mar 15;141(6):575-580. [Medline]
- Brener N, Kann L, Mcmanus T, Kinchen S, Sundberg E, Ross J. Reliability of the 1999 Youth Risk Behavior Survey Questionnaire. J Adolesc Health 2002 Oct;31(4):336-342 [FREE Full text] [CrossRef]
- Centers for Disease Control and Prevention. 2015 Youth Risk Behavior Survey Questionnaire URL: https://www.cdc.gov/healthyyouth/data/yrbs/index.htm [accessed 2018-11-21] [WebCite Cache]
- Rice E. Grantome. Bethesda, MD: National Institute of Mental Health; 2011. HIV Risk in Large Networks of Homeless Youth URL: http://grantome.com/grant/NIH/R01-MH093336-01 [accessed 2018-11-23] [WebCite Cache]
- Ewing J. Detecting alcoholism: the CAGE questionnaire. J Am Med Assoc 1984 Oct 12;252(14):1905 [FREE Full text] [CrossRef]
- Brown RL, Rounds LA. Conjoint screening questionnaires for alcohol and other drug abuse: criterion validity in a primary care practice. Wis Med J 1995;94(3):135-140. [Medline]
- Casey K, Reid R, Trout A, Hurley K, Chmelka M, Thompson R. The transition status of youth departing residential care. Child Youth Care Forum 2010 Apr 29;39(5):323-340 [FREE Full text] [CrossRef]
- Ansell D, Morse J, Nollan K, Hoskins R. Life Skills Guidebook. Seattle, WA: Casey Family Programs; 2004.
- Steinberg A, Brymer M, Decker K, Pynoos R. The University of California at Los Angeles post-traumatic stress disorder reaction index. Curr Psychiatry Rep 2004 Mar;6(2):96-100 [FREE Full text] [CrossRef]
- Rew L, Powell T, Thompson S. Stress on the streets: validation of a measurement. J Soc Dist Homeless 2016 Jan 08;25(1):11-18 [FREE Full text] [CrossRef]
- Coates J, Swindale A, Bilinsky P. Food and Nutrition Technical Assistance. Household Food Insecurity Access Scale (HFIAS) for Measurement of Household Food Access: Indicator Guide (v. 3) URL: http://www.fao.org/fileadmin/user_upload/eufao-fsi4dm/doc-training/hfias.pdf [accessed 2018-11-23] [WebCite Cache]
- Research, Education and Advocacy Co-Lab for Youth Stability and Thriving. 2018. Homeless youth risk and resilience survey URL: https://www.realyst.org/hyrrs [accessed 2018-11-21] [WebCite Cache]
- Burt R. Network items and the general social survey. Soc Networks 1984 Dec;6(4):293-339 [FREE Full text] [CrossRef]
- Wasserman S, Faust K. Social Network Analysis: Methods and Applications. New York: Cambridge University Press; 1994.
- Kloos B, Shah S. A social ecological approach to investigating relationships between housing and adaptive functioning for persons with serious mental illness. Am J Community Psychol 2009 Dec;44(3-4):316-326 [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, McHugo G, Xie H, Dobbins K, Young M. Comparing retrospective reports to real-time/real-place mobile assessments in individuals with schizophrenia and a nonclinical comparison group. Schizophr Bull 2012 May;38(3):396-404 [FREE Full text] [CrossRef] [Medline]
- Epstein D, Preston K. Daily life hour by hour, with and without cocaine: an ecological momentary assessment study. Psychopharmacology (Berl) 2010 Aug;211(2):223-232 [FREE Full text] [CrossRef] [Medline]
- Kowalczyk W, Phillips K, Jobes M, Kennedy A, Ghitza U, Agage D, et al. Clonidine maintenance prolongs opioid abstinence and decouples stress from craving in daily life: a randomized controlled trial with ecological momentary assessment. Am J Psychiatry 2015 Aug 01;172(8):760-767 [FREE Full text] [CrossRef] [Medline]
- Phillips K, Epstein D, Vahabzadeh M, Mezghanni M, Lin J, Preston K. Substance use and hepatitis C: an ecological momentary assessment study. Health Psychol 2014 Jul;33(7):710-719 [FREE Full text] [CrossRef] [Medline]
- Vahabzadeh M, Lin J, Mezghanni M, Epstein D, Preston K. Automation in an addiction treatment research clinic: computerised contingency management, ecological momentary assessment and a protocol workflow system. Drug Alcohol Rev 2009 Jan;28(1):3-11 [FREE Full text] [CrossRef] [Medline]
- Jones M, Pebley A. Redefining neighborhoods using common destinations: social characteristics of activity spaces and home census tracts compared. Demography 2014 Jun;51(3):727-752 [FREE Full text] [CrossRef] [Medline]
- Stake R. Multiple Case Study Analysis. New York: Guilford Press; 2005.
- Yin R. Case Study Research: Design and Method. Thousand Oaks, CA: Sage Publications; 2003.
- Miles MB, Huberman AM, Saldana J. Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications; 2013.
- Plano Clark VL, Creswell JW. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications; 2010.
- Palinkas L, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed-methods designs in mental health services research: a review. Psychiatr Serv 2011 Mar;62(3):255-263. [CrossRef] [Medline]
- O'Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. Br Med J 2010 Sep 17;341:c4587. [CrossRef] [Medline]
- Hernandez D, Rossel C. Inequality and access to social services in Latin Americapace-time constraints of child health checkups and prenatal care in Montevideo. J Transp Geogr 2015;44:24-32. [CrossRef]
- Lager D, Van Hoven B, Huigen P. Rhythms, ageing and neighborhoods. Environ Plan 2018;48(8):1565-1580. [CrossRef]
- Henwood BF, Redline B, Semborski S, Rhoades H, Rice E, Wenzel S. A grounded theory of the relationship between ontological security, mental health, social relationships, and identity formation for young adults in supportive housing. Cityscape 2019 (forthcoming).
- Veldhuizen EM, Osté J, Kunst A. Environmental characteristics of hotspots of alcohol-related health incidents outside of the entertainment districts of Amsterdam. GeoJournal 2017 Oct 6;83(5):1053-1066 [FREE Full text] [CrossRef]
- Nahum-Shani I, Smith S, Spring B, Collins L, Witkiewitz K, Tewari A, et al. Just-in-time adaptive interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann Behav Med 2018 May 18;52(6):446-462 [FREE Full text] [CrossRef] [Medline]
- Intille SS. Technological innovations enabling automatic, context-sensitive ecological momentary assessment. In: Stone A, Shiffman S, Atienza A, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Report in Health Research. Oxford: Oxford University Press; 2007:308-337.
- Riley W, Rivera D, Atienza A, Nilsen W, Allison S, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med 2011 Mar;1(1):53-71 [FREE Full text] [CrossRef] [Medline]
|EMA: ecological momentary assessment|
|GEMA: geographically explicit ecological momentary assessments|
|GLMM: generalized linear mixed model|
|GPS: global positioning system|
|LML: Log My Life|
Edited by G Eysenbach; submitted 05.09.18; peer-reviewed by D Santa Maria, J Thrul, N Nguyen; comments to author 14.10.18; revised version received 24.10.18; accepted 24.10.18; published 10.01.19Copyright
©Benjamin F Henwood, Brian Redline, Eldin Dzubur, Danielle R Madden, Harmony Rhoades, Genevieve F Dunton, Eric Rice, Sara Semborski, Qu Tang, Stephen S Intille. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 10.01.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.