The e-workbook was developed as a Web-based, rather than native (ie, device-specific) application. Although this approach requires the user to have Internet access, an advantage is that Web-based tools are accessible to providers on any of the wide range of web-connected devices. A native application would ultimately require reprogramming and updating for each operating system (Android, iOS, Windows). Although no technology can be completely “future proof,” it was determined that Web-based tools would have the highest potential to remain useful in meeting the aims of this project as technology progresses. Native versions of the app will likely represent the final step toward disseminating this resource once it is fully evaluated and refined.

The application is responsive; that is, it detects the type of device that is in use and adjusts to it for optimal look and feel. For example, the application can detect an iPad and use JavaScript to convert desktop mouse actions to accelerometer actions. This functionality enables the use of features such as shaking the device to trigger a response, inertial finger drawing, and the built-in microphone. Conversely, those actions would naturally degrade to normal mouse actions if a user were to access the resource via desktop computer. As the application detects different devices, the responsive layout naturally adapts to the devices’ native display parameters. This achieves a fully integrated and device-agnostic application and increases potential for adoption across practice settings.

Qualitative data, collected regularly as part of consultation calls conducted in TF-CBT training programs, provided valuable direction in the technical and content development phases. Specifically, these data were used to determine which components of TF-CBT are most vulnerable to drift and what activities could be developed to overcome challenges to child engagement and provider fidelity in each TF-CBT component. For example, providers who were trained in TF-CBT reported that they used the psychoeducation, anxiety management, and coping components of TF-CBT at least 50% of the time with their child trauma cases; but that the exposure components were reported least frequently (26-50%) [40]. Therefore, greater attention was paid to emphasizing exposure-specific elements of TF-CBT rather than general psychotherapy skills. The research team also carefully reviewed data collected from expert clinicians [39] to identify areas in which technology-based resources would have high potential to enhance (1) provider fidelity; (2) child engagement in treatment activities; (3) child or caregiver understanding of key treatment concepts; (4) likelihood of skill acquisition; or (5) patient adherence to homework exercises. Taken together, these data and observations were used to create the content and format of the e-workbook activities and resources listed in Table 1.

We developed numerous resources (Table 1) or “chapters” for use by providers during individual sessions with each child. Introduction videos were created for children and are available on the first screen of each chapter. Each video depicts a teenager who explains the rationale for the chapter and presents brief examples that illustrate completion of the activity. Our decision to feature older youth (ages 15-16) in most of the videos was based on focus group feedback from children aged 8-15 years. Specifically, youth at the younger end of the age range stated that they would be equally pleased with younger or older actors in the videos, whereas youth at the older end of the age range agreed unanimously that actors should be older adolescents, and that they would likely experience very little connection to a younger actor. Some videos are brief (ie, 30-60 second) clips designed for a provider to use after introducing a concept or teaching a skill with the goal of enhancing engagement and reinforcing what the provider taught during the session (eg, videos demonstrating the CBT triangle in the cognitive coping component of TF-CBT). Some chapters feature interactive touch screen games, such as drag-and-drop activities, drawing tools, trivia-style card games, and animated relaxation activities. Each activity was developed to address an element of the TF-CBT protocol that was identified by experts and providers as challenging to implement with high fidelity and engagement.

Some resources included activities for providers to complete with caregivers. These consisted of an extensive collection of video clips with narration that demonstrate a wide range of effective behavior management skills, including common mistakes and how to correct them. Videos were not intended to replace provider instruction and demonstration, but to support providers’ attempts to teach caregivers how to apply skills across a broad range of situations and settings. Additional resources were developed for the provider. These consisted primarily of content that assists with session preparation tasks, such as setting session goals and potential agenda items, tracking, updating assigned homework activities, and guiding their clinical decision making as families progress in TF-CBT. We also created provider tabs for most resources that provide discussion points, tips, and ideas for supplemental exercises.

The primary purpose of alpha testing in software and intervention resource development is to assess reactions and obtain direct input from end users regarding design, content, and functionality. These data are needed to guide improvements to the resource. Participants were provided with tablets (ie, iPads) in either individual interviews or focus group settings to interact with a select set of resources within the TF-CBT e-workbook. Providers were given access to all activities listed in Table 1, and children were given access to a representative sample of activities. Participants were asked a series of semistructured questions administered by trained interviewers. The questions asked participants to provide reactions to each tool, ranging from reactions to the overall “look and feel” of the interface, ease of navigation, and technical problems they experience. An average of 5 minutes was dedicated to review and discuss each resource. Qualitative data collected during these sessions were audiotaped for transcription, coding, and data analysis.

Alpha testing was completed via focus groups with providers (n=22) and via focus groups or individual interviews with children (n=24). Focus group sizes were generally between 5-8 participants.

Children aged 8-11 years and 12-16 years participated in separate groups; average age was 13.0 years. Children were recruited from clinical sites (local child advocacy and mental health treatment centers) and local schools with on-site mental health services with high rates of trauma exposure. Inclusion/exclusion criteria for clinic-referred children were consistent with those used in prior RCTs with TF-CBT: children were 8-16 years old, had a history of at least one potentially traumatic event, had current clinically elevated symptoms of PTSD, and did not have active suicidal or homicidal ideations or symptoms of psychosis. Participants recruited from the schools were also 8-16 years old but not required to have a reported trauma history or a particular symptom profile. This combined clinical/nonclinical sample was used to maximize input on the look-and-feel of the TF-CBT e-workbook in addition to applicability and clarity of content to youth in the targeted age range. Sample of children was 65% female, 86% Black/African American, 14% white.

Providers were recruited from several child advocacy and mental health centers. Providers were eligible if they were fully trained in TF-CBT and carried active child trauma cases at the time of the study. There was diversity in the provider sample with respect to discipline (64% counseling, 18% social work, 18% psychology), credentials (18% doctorate, 23% MSW/LCSW/LISW/LMSW, 36% LPC, 23% other Master’s degree), and years of experience delivering child trauma treatment (46% 5<years, 27% 3-5 years, 27% 1-3 years). Providers also varied in the settings in which they worked (45% mental health clinic, 55% child advocacy center), with several providers delivering TF-CBT in schools or patients’ homes through outreach clinics. The provider sample was 86% female, 73% Caucasian, 18% Black/African American, 4.5% Native American, and 4.5% other (unspecified).

Focus group and interview responses were audiorecorded and transcribed. First, a content analysis of the interview responses was conducted through multiple close readings of the transcriptions by two independent coders (authors on this paper). Each coder generated an independent list of thematic categories and subcategories based on their review of the data (eg, usefulness of a specific tool; age appropriateness). These themes were then further developed and ordered by the primary coder and reviewed and edited by the second coder. The coders then met in a consensus conference to discuss the categories, resolve questions, and refine the thematic categories. Once this was accomplished, the themes were again reviewed. After additional discussion to review and refine categories and resolve questions, the final thematic categories were completed and higher-order categories were developed. We have previously used similar analytic approaches to qualitative research with a range of public sector health care patient and provider populations. [39,41-43] Results of these analyses, reported in Table 2 (providers) and Table 3 (children), were used to refine and fine tune the TF-CBT e-workbook. The vast majority of recommendations given by children and providers were addressed in the context of the current grant prior to feasibility testing. Some recommendations were cost prohibitive, but can be addressed with future funding in preparation for a large-scale RCT.

Beta testing followed the alpha testing phase and is a form of usability testing in which refined or revised resources are evaluated by a new set of providers for acceptability and functionality. This was to ensure that any errors existing within the TF-CBT e-workbook were identified and corrected in preparation for the feasibility trial. Members of the development and investigative team reviewed all the revised components of the resource, and additional edits and refinements to content, appearance, and functionality were made in response. Figures 1-7 include screenshots of several e-workbook activities.

The goal of the ongoing feasibility trial is to examine the feasibility of implementing the TF-CBT e-workbook in community mental health service agencies, and to demonstrate the randomized controlled trial (RCT) methodology we propose to use in a future large-scale trial to test the effectiveness of the TF-CBT e-workbook in clinics across the state. Resources and data resulting from this project are designed to position us well for a large-scale RCT to examine the incremental benefit of e-workbooks to best-practice interventions compared to standard care without technology. The strengths and limitations associated with conducting a small-scale feasibility RCT versus an open trial were carefully considered. A major benefit of the latter approach included increased quantity of feasibility data related to the use of the e-workbook. On balance, this option was outweighed by the importance of demonstrating the feasibility of the full RCT methodology. This would allow us to identify and address barriers in recruitment, retention, fidelity assessment, and other procedures essential to successful conduct of a RCT. This approach is consistent with expert recommendations to pilot test the feasibility of methods to be used in large RCTs, use data yielded by such studies to “debug” the methodology, and assess optimal strategies to executing the RCT [44].

The feasibility trial will involve 10 providers recruited from four participating community-based mental health and child welfare agencies where TF-CBT is delivered. Providers will have been fully trained in TF-CBT and carry active child trauma cases at the onset of the trial. Providers will be block randomized by site to either the e-workbook facilitated TF-CBT (n=5) or standard TF-CBT (TAU) (n=5) conditions. Each provider will treat at least two children for a total sample of 20 children. We will also pursue an exploratory aim regarding the feasibility of using the e-workbook across multiple treatment delivery settings by including a small subsample of mental health providers who are primarily based in school settings (at least two per study condition).

An independent, trained evaluator who is blind to the study condition will conduct baseline, mid-treatment, and post-treatment clinical assessments. Families will be reimbursed $30 for each administration of the assessment battery. All sessions will be audiorecorded and coded for fidelity and child engagement by independent coders.

Alternatives to this design decision were considered carefully. For example, the design could have included each provider treating one e-workbook case and one standard TF-CBT case, with randomized ordering of conditions by provider. An advantage of this approach is that provider factors are less likely to have strong impact on clinical outcomes in this small-scale trial. A weakness of the approach is that use of the e-workbook could affect performance on subsequent standard TF-CBT cases for providers assigned to deliver TF-CBT using the e-workbook first. Additionally, it was determined that if providers within an agency were assigned to different starting conditions, this could increase risk of contamination across conditions with standard TF-CBT providers hearing about, viewing, and potentially even using elements of the e-workbook with standard TF-CBT cases. Overall, it was concluded that block randomization by clinic to the two study conditions was a preferable design strategy.

A trained evaluator both blinded to treatment condition and fully trained in the administration of all measures will administer the assessment battery. All measures are well-validated and widely used instruments in the mental health field and in the treatment outcome research. Assessment will occur on three occasions: at baseline (ie, during the screening interview), at mid-treatment (ie, after the sixth treatment session, approximately two months post-baseline), and at post-treatment (ie, after the twelfth treatment session, approximately four months post-baseline). Treatment fidelity and child engagement will be measured via observational coding of audiotapes.

Fidelity to the TF-CBT and e-workbook-facilitated TF-CBT protocols will be measured via coding of audiotaped treatment sessions by independent, trained raters. Treatment sessions will be audiotaped for both study conditions (n=20 x ~10 sessions average=~200 audiotapes). Ratings will be completed using a behaviorally specific coding system of TF-CBT provider behavior that was modified for the current study to ensure relevance to the e-workbook condition [45]. The coding system will be used to calculate providers' fidelity to each TF-CBT component. An additional eight items focus on general therapy skills, not specific to TF-CBT, including establishing an agenda, providing a treatment rationale, and assigning homework. Additional items were created to identify use of the e-workbook activities to differentiate the two study conditions. Two independent raters will listen to audiorecorded treatment session tapes and complete the modified fidelity measure to code the use and extensiveness of specific treatment techniques depicted on the recordings. Raters will be trained in the coding system and meet biweekly throughout the remainder of the feasibility trial to ensure maintenance of acceptable levels of accuracy and interrater reliability. Discrepant ratings will be reviewed until consensus is achieved. If the two raters cannot reach consensus, the PI and Co-Is will make final decisions.

Child engagement will be measured via coding of audiotaped sessions by independent, trained raters. The Child Involvement Ratings Scale (CIRS) [46-47], a 6-item scale that measures child engagement for each session, will be used for this purpose. Four “positive” involvement items and two “negative” involvement items are rated for each session on a 6-point scale (ie, “not at all” to “a great deal present”). The positive-involvement items emphasize the extent to which children initiate discussions, demonstrate enthusiasm, self-disclose, and demonstrate understanding. Negative-involvement items address withdrawal or avoidance in treatment. Coders will provide ratings based on two 10-min segments of session audiotapes (ie, beginning at min 10 and min 40). Child engagement ratings on the CIRS have been associated with clinical outcomes [47] and provider flexibility in delivery of EBTs [48]. Excellent internal consistency and strong interrater reliability have been reported for this measure [47-48].

To obtain direct input on the e-workbook’s design, content, and functionality, post-treatment assessments will include semistructured interviews with children, caregivers, and providers after completion of e-workbook-facilitated TF-CBT. Patients and providers will complete semistructured interviews about their experiences with the e-workbook and how it affected treatment. Interviews will be audiotaped, transcribed, and interpreted using the same approach used during alpha testing.

For both conditions, several patient- and provider-level variables, as well as data collection procedures will be assessed and described. Patient-level variables will include the percentage of eligible patients recruited, treatment attrition, study retention, and session attendance. Recruitment will be assessed by the proportion of patients who agree to participate as compared to the total number solicited to enroll. Attrition will be assessed by examining the proportion of patients who prematurely terminate treatment. Qualitative analyses will be conducted to identify themes in termination. Study retention refers to the proportion of patients who complete all assessment points associated with the treatment protocol, including those who terminate treatment prematurely.

Provider-level variables will include provider recruitment to participate, fidelity to TF-CBT procedures, use of the resources within the e-workbook, and adherence to the session audiotaping protocol. Recruitment will be assessed by the proportion of providers who agreed to participate as compared to the total number approached. Provider fidelity will consist of the proportion of completed treatment components for the intervention according to the fidelity measure described previously. Providers will be interviewed at the end of treatment with e-workbook cases to provide feedback on the usability of the e-workbook activities. Other data to be summarized include kappa coefficients between independent fidelity raters, scheduling and logistical barriers to completing assessment, and communication successes and failures between study staff and clinic site staff around recruitment efforts to demonstrate feasibility of study procedures for our planned RCT.