The ePST program is directly based on a manualized, evidence-based treatment for depression known as problem-solving treatment for primary care (PST-PC). PST-PC has been demonstrated to be effective for the treatment of depression [3-5]. (For a detailed description of PST-PC, see Hegel and Arean [6].) The basis of PST-PC is that enhancing problem-solving skills and attitudes and working to solve problems in one’s life can reduce depression. PST-PC involves 6 steps: (1) clarifying the problem, (2) establishing an achievable goal, (3) brainstorming alternative solutions, (4) evaluating the pros and cons of each solution and selecting one or more solutions, (5) developing an action plan to implement the solution(s), and (6) evaluating the success of the implementation and troubleshooting as needed. Each session concludes with scheduling pleasant activities.

The mechanism by which problem-solving treatment works has not yet been clearly identified [7]. PST-PC activates individuals to take steps toward changing their situation. Evidence is emerging to suggest that, along with typical predictors of improvement such as treatment adherence [8], the most important element of PST-PC for overcoming depression may be helping the patient to overcome an avoidant coping style [7,9]. Because PST-PC has strong face validity and is easy for most patients to understand, acceptance and satisfaction from patients is high, with very low dropout rates, as reported in clinical trials of PST-PC [5,10].

An interactive media program was produced to deliver ePST. The goal of ePST was to automate an empirically supported treatment and to provide it in the context of a simulated helping relationship. The ePST program was built for the US National Aeronautics and Space Administration (NASA) as part of a suite of self-guided programs to help astronauts manage their own psychosocial problems on long missions [11]. The system is used autonomously, confidentially, and in a self-directed manner. Although the development of ePST was funded by NASA, it was designed to also be evaluated among the public with a wide variety of patients. Therefore, a NASA and a general public version were produced, which differ only by a few video clips. As such, ePST is intended for use by both astronauts and nonastronauts.

ePST is based on the Virtual Practicum Model [12,13], an approach to designing professional education programs, which is based on Boisot’s “epistemological space” [14], Kolb’s “learning cycles” [15], and Schon’s “reflective practicums” [16]. The Virtual Practicum Model has been used to teach clinicians how to conduct counseling in prevention of human immunodeficiency virus infection [17], manage patients with human immunodeficiency virus/acquired immunodeficiency syndrome [18], and conduct genetic counseling and testing [19]. The basis of the Virtual Practicum Model is the simulation of the interactive experience of being trained one-on-one by a master clinician—a practitioner who is both a master of his or her specialty and a master teacher. The goal of the program is to provide trainees both the concrete information they need to manage clinical cases and to impart the soft skills needed to interact effectively with patients, and the case conceptualization skills needed to understand each unique case. ePST builds on the Virtual Practicum Model by providing persons seeking treatment for depression an opportunity to receive treatment (in a simulated fashion) from a master clinician (author MTH) who is an expert in problem-solving treatment. However, not only does the on-camera clinician walk the user through the concrete steps of problem-solving treatment, he conveys warmth and empathy—the soft skills of a skilled psychotherapist. The goal of both the VPM and ePST is for users to receive training or treatment from an expert in the field, and thereby to increase the number of persons who can benefit from that professional.

ePST is a multimedia-based, video-intensive program that simulates therapy based on the PST-PC treatment manual [6]. It includes six sessions, intended to be completed once per week. In each session, users are welcomed by a therapist (author MTH) presented via audio and video, who appears in a warmly lit professional office. The user then completes the Patient Health Questionnaire-9 (PHQ-9) [20] measure of depression. Tailored feedback about his or her progress is provided by the virtual therapist, discussing the user’s current level of depression and change from the previous session (see Figure 1).

The first session provides psychoeducation about depression and the process of problem-solving treatment; subsequent sessions offer decreasing guidance on the process of problem-solving treatment. During each session, the virtual therapist guides users through all steps of problem-solving treatment, plus making a list of enjoyable activities (entering their own or choosing from an extensive list of common activities) and scheduling them for the week.

Sessions 2 through 6 begin with a check-in on all active problems. The therapist provides feedback about the user’s success or failure in solving problems compared with the previous session and helps him or her identify ways to improve problem solving, both on individual problems and in general.

Although the process of problem solving remains consistent throughout the program, a variety of audio and video clips are used to maintain a sense of novelty in each session and to respond to user inputs. At the end of the treatment session, the program produces a printout that summarizes the work the user has done on each problem, including his or her action plans, plus a day-by-day schedule of enjoyable activities for the coming week.

The program’s interfaces are designed for ease of use: where free text entry is requested, no other options are available to the user; where menus are used, five or fewer choices are generally made available. Nonetheless, in the background a highly complex program is running, with substantial branching of video and audio, failure analysis algorithms, and data handling methods that adapt to the user’s clinical status and problem-solving history within ePST.

Fogg and colleagues have identified factors that engender users’ trust in software or websites and found that the credibility depends largely on three factors: (1) the usability of the program (especially that it does what the user expects it to do at any given moment), (2) the brand or source behind it, and (3) the professionalism of the interface and media [21,22]. If a program behaves in an unpredictable manner, is presented by an unknown entity, or looks amateurish, users are unlikely to trust it. An effort was made to enhance the credibility of ePST by addressing these three domains: (1) the program was designed to be simple to use and went through initial usability testing by persons who had been treated for depression, (2) familiar brands are referenced in the program (Harvard, NASA, Dartmouth), and (3) the video and audio for ePST were produced in a broadcast television facility, meeting industry standards for production values, and ePST’s interfaces were developed by professional graphic artists.

The initial version of the program was delivered via universal serial bus flash drive, which is a convenient format for astronauts, since they often train in settings without Internet access, and bandwidth on space missions is limited. However, ePST can readily be adapted for delivery via the Internet.

ePST advances the field of computer-automated interventions for depression by (1) approximating actual treatment via the use of rich media, (2) minimizing the amount of text that a user needs to read, and (3) performing failure analyses to assist users to improve their problem-solving effectiveness.

Information richness refers to “the potential information carrying capacity of data” [23]. Media can be ranked on richness by the number of channels of communication they employ: verbal (the words said), paraverbal (how they are said), and nonverbal (body language) communication [24]. Although some information is best conveyed by text (eg, procedures or numerical data), emotionally laden information is more effectively communicated with richer media [25]. Psychotherapy falls into the latter category. Through rich media (branching video and audio clips tailored to users’ inputs), a conversation is approximated between the user and a competent, caring therapist. The intent is to make the ePST program feel more like interacting with a person than with a computer. ePST contains 148 video and 225 audio clips, although each user receives only a fraction of them, based on his or her clinical status and what he or she does in the program.

Although ePST was created for one of the most highly educated groups of individuals in the world—astronauts—it requires less reading skill than most self-help programs, due to the extensive use of audio and video. Text on the screen is written at a low literacy level, mainly limited to labels and occasional short sentences. The user does need to be able to input free text to write his or her own problem statement, goals, and action plan; however, proper grammar and spelling are not required. The writing skill necessary is comparable to that required for sending text messages or emails.

In in-person problem-solving treatment, a therapist guides the client and provides feedback on how he or she could improve on problem solving. However, without a therapist, the challenge of ePST is to guide users through the process of problem-solving treatment while simultaneously teaching them how to evaluate and improve their own work. Failure analysis is a process of examining failures, identifying reasons for them, and planning how to prevent them in the future. It is used in engineering [26], business [27], and software design [28,29]. Through failure analysis algorithms in ePST, the computer assists users to identify ways to improve their problem solving in general and where to revise work on specific problems, much as a live therapist would do. In this way, ePST tailors each session to what the user has done in the program, thus far.

The primary purpose of the pilot study was to test research methods and collect preliminary data on which to base a larger clinical trial of ePST. As such, we posed the following research questions, noting that the number of participants would likely not be sufficient to test hypotheses: (1) To what extent do participants find ePST to be usable? (2) To what extent do participants find ePST to be acceptable and credible? and (3) To what extent do participants who use ePST improve in depression, compared with those who do not use it?

Because there are fewer than 100 astronauts [30] (and most do not have depression at any given point), it is necessary to study persons who bear some demographic similarities to astronauts. To support generalization from the test population to astronauts, each participant was between the ages of 30 and 60 years, had completed at least 4 years of college, used a computer at least twice per week, and read and spoke English well enough to use the program. We enrolled only persons meeting the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, text revision (DSM-IV-TR) [31] criteria for minor depression and scoring higher than 10 (moderate or worse depressive symptom severity) on the self-report 17-item Hamilton Depression Inventory (HDI) [32].

Exclusion criteria were a history of bipolar disorder or schizophrenia, current substance abuse or dependence, dysthymic disorder, or neurological disorders. Study participants with active suicidal ideation, self-injurious behavior, or previous suicide attempts were also excluded.

We sent 500 email messages to persons listed on a registry of those interested in receiving information about clinical trials for depression. Of these, 104 persons responded. A total of 54 phone screening interviews (to establish age, education, computer experience, and exclusionary diagnoses obtained by history) were successfully completed. Others did not respond to messages. Of the 54 persons screened, we excluded 40 (26 lacked the required education level, 5 reported suicidal ideation or attempts, 4 were excluded for other psychiatric or medical conditions, and 5 were unavailable during the study timeframe). All persons excluded from the study were provided a list of referrals for treatment. This left 14 individuals, who were invited for in-person evaluation. All 14 met eligibility criteria and were enrolled in the study. We randomly assigned 7 participants to each condition.

We conducted the Mini-International Neuropsychiatric Interview (MINI) [33] to establish enrollment eligibility. After enrollment, a baseline assessment was administered and participants were assigned to the waitlist control or ePST experimental groups using block randomization to ensure equal numbers of participants in each group. Because the number of available participants was small, we made no attempt to balance groups on any variable.

Participants in the ePST condition were requested to use the program weekly on site at the clinic, although they were permitted to schedule sessions less frequently. However, we decided a priori that a longer than 3-week interval between sessions would be considered a protocol violation. We contacted persons in the waitlist (control) group 3 weeks after enrollment to assess depression and safety. A follow-up assessment of depression was conducted at week 6 for control group participants and at posttreatment 1 week after the final ePST session (nominally at 7 weeks) for ePST group participants. We conducted an additional follow-up with those in the ePST group 4 weeks after they completed ePST. Participants received payment based on the number of times they were required to visit the clinic: nine times for the ePST participants for a total of US $470 and twice for the waitlist participants for a total of US $170. Following the week 6 assessment, waitlist participants were offered off-study use of the ePST program.

We recruited a sample of 14 participants, 7 per group. Because the primary aims of this evaluation were to gauge the usability and acceptability of ePST and refine the evaluation procedures, the number of participants was based on resources available, not on a power analysis. For the pilot study, due to the small sample size, nonparametric statistical analyses were conducted on all measures. These included the Wilcoxon signed rank test and the Mann-Whitney U test. Cronbach alpha was calculated for Cred-Q and AST. The numerical values are presented with mean (SD) or median (range), or both. We used the Statistical Analysis Software (SAS) program, version 9.3 (SAS Institute, Cary, NC, USA) for the analysis.

As Figure 2 shows, 54 persons responded to an advertisement for the study and were screened by telephone. A total of 35 of them did not meet all inclusion criteria and 5 were excluded for other reasons; 14 participants were enrolled.

Mean HDI scores for the ePST group at pretreatment were 15.61 (SD 9.64), and the median score was 15.4 (range 3.2–31.1). For the waitlist control group, the mean HDI score was 18.71 (SD 6.65) and the median was also 15.4 (range 11.4–26). A Mann-Whitney U test indicated no significant difference in HDI scores between the groups (P = .62) at baseline. We did note demographic differences between the groups (see Table 1).

One participant in the control group was lost to follow-up. All ePST group participants completed all ePST sessions and posttreatment evaluations; however, 2 persons had a gap of greater than 3 weeks between two sessions, constituting a protocol violation. They were permitted to continue using ePST, and we conducted separate analyses on the intent-to-treat sample (n = 13) and the subset completing treatment according to the protocol (n = 11). The mean time to complete each session was 53 (SD 31) minutes, with earlier sessions running longer than later ones.

The ePST group completed the SUS after session 1 and session 6. On the 100-point SUS, the mean score was 80.36 (SD 19.28) and median was 85 (range 42.5–97.5) after session 1. After session 6, the mean SUS was 85.36 (SD 15.91) and median was 95 (range 57.5–100). No significant difference was found for the SUS between the two time points (P < .09; Wilcoxon signed rank test).

Cronbach alpha was .922 for the Cred-Q and .830 for the AST. Nonetheless, since the Cred-Q was created for this study and since data have not been previously published on the AST, an item-level analysis of those questionnaires is more appropriate than summary data. Mean and median scores for each Cred-Q item are presented in Table 2; AST scores are presented in Table 3.

At posttreatment, in the intent-to-treat analysis (n = 7), the mean HDI score for persons who received ePST was 8.81 (SD 5.91) and median was 6.3 (range 1.1–16.7) at posttreatment. For the ePST group, the mean percentage change in HDI score from baseline to posttreatment was –35.86% (SD 40.16%) (median –51.02%, range –33.87% to 79.74%). The mean HDI score for the waitlist control group at 6-week follow-up was 17.78 (SD 7.52) (median 18.55, range 8.2–28.4). The mean percentage change in depression for the control group was –11.12% (SD 18.70%) (median –17.78%, range 14.29%–28.07%; see Table 4). The difference in percentage change in depression between the two groups, using the Mann-Whitney U test, was not significant in the intent-to-treat analysis (P = .25). A Wilcoxon signed rank test indicated that depression scores remained stable at 4-week follow-up for the ePST group (P = .90), as depicted in Figure 3.

Persons who completed treatment with no more than a 3-week gap between any sessions showed a trend toward greater improvement in depression. For this subset of the ePST treatment group, depression decreased by an average of 52.65% (SD 28.28%) (median –61.35%, range –79.74% to –5.56%; P =.05). There was no significant difference in depression level between 1-week and 4-week follow-ups in the completer analysis (mean 6.24, SD 6.51; median 4.1, range 0.3–12.4). A Wilcoxon signed rank test indicated that depression scores remained stable at 4-week follow-up for the ePST group (P = .90), as depicted in Figure 3.

The clinical evaluation component of this study was secondary to the technology development, in the amount of time and funding devoted to it, and was primarily intended to establish a methodology to evaluate the program in future studies. As such, the feasibility study has many limitations, including the limited sample size and unique characteristics of the sample (ie, high education and frequent computer use), inequivalent compensation and assessment time points, and lack of an active control condition. Therefore, given these limitations all conclusions must be considered to be highly preliminary, and the ePST program should be evaluated with a larger sample that includes an active control condition.

Although we used validated measures wherever possible, none existed for some domains of interest, and we therefore created them. The Cred-Q and AST, which were used for the first time in this study, may have been subject to response set bias. It is notable that most Cred-Q and AST item means are similar to each other, with little variability. Of note are apparently conflicting results to AST questions about preference for using a computer versus being treated by a live clinician: “I would rather do problem-solving treatment with a therapist than with the computer” (mean 5.0, SD 1.4) and “I would rather use a computer to help myself privately than go to a therapist” (mean 5.0, SD 0.8). Therefore, the Cred-Q and AST should be revised if used in subsequent studies to better detect response set. Approaches to designing Likert scales to detect and prevent response set have been advanced by Shulruf et al [41] and Barnette [42].

Future studies could directly compare the interactive media approach to delivering problem-solving treatment (which is both bandwidth intensive and costly) versus a text-based version that functions using the same logic. This could determine the merit of using interactive media, compared to the cost to produce and deliver it.

Further evaluation of the effectiveness of ePST is warranted; however, the question is what the best design for such as study would be. Historically, computer-automated behavioral interventions have generally been compared with a waitlist (as in this study), usual care, or live therapy. However, there are drawbacks to each of these designs. Some treatment is likely to be better than no treatment, making a waitlist control a straw man (although in many clinics patients linger on waitlists for weeks and months, making this a de facto usual care comparison). Usual care comparisons may have more real-world applicability, in which participants are randomly assigned to either use ePST or receive treatment as usual. But usual care varies between settings, and even within a single study group may include many interventions, such as receiving medication, intensive therapy, or nothing at all. Therefore, the usual care control is actually an amalgam of multiple types of control groups.

Randomized “John Henry” [43] (man versus machine) trials that compare ePST versus live problem-solving treatment clinicians following the same protocol would establish noninferiority. This type of data would be most useful for clinics deciding whether to deliver problem-solving treatment via computer or a live clinician (and for which patients); however, many settings do not have the resources to hire or train sufficient numbers of problem-solving treatment clinicians to meet the demand for treatment. A more useful comparison might be randomization to either ePST or another automated treatment of depression, now that several exist in the marketplace. Clinics that are unable to hire specialty staff may well be able to provide automated depression treatment, and few head-to-head comparisons of computer-automated depression treatments have been published.

A limitation across all of the above comparisons is that they evaluate the automated treatment as a static entity and preclude the opportunity for its improvement during the trial. The criterion-based development model proposed by Carter [44] is a cyclical model of testing and evaluating to criterion. In this model, originally for the evaluation of self-instructional interactive media programs, an a priori target level of learner performance is established, which signifies an acceptable level of mastery over a skill. Essentially, “X% of learners should perform skill Y at level Z.” The interactive media program is then tested and revised in a spiral development model [45] until the training criteria are met. The criterion-based development model can be applied to the evaluation of self-treatment software, as well, by establishing a meaningful reduction of depression symptoms and functional impairment for users, and testing and revising the program to reach that criterion.

The advantage of the criterion-based development model over randomized trials is that evaluation data are immediately applied to the improvement of the program, maximizing the potential of the program to effect change. Rather than comparing treatment conditions, which may yield statistically, but not clinically, significant differences, evaluation group outcomes are compared with predetermined, objective standards of health and functioning. It is possible to test whether this meaningful goal is met or exceeded. Clearly there is a need for both randomized trial and criterion-based evaluations, and ePST should be tested using both models.

Further development of ePST could tailor the program to special populations, such as cancer patients, older persons, or adolescents. To increase accessibility, a range of hosts could be provided, to better match the user’s demographics, including his or her language.

Although other computer-automated treatments for depression have previously been developed and shown to be effective, that does not mean work in the field should stop, any more than drug discovery should cease once an effective medication has been found for a disease. ePST was designed to overcome some limitations of current computer-automated therapies, such as reliance on text, limited interactivity, and little if any feeling of human connection. In doing so, it presents a model for next-generation interactive, media-based interventions.