A 2-day multidisciplinary conference in Toronto, Canada was held to develop consensus regarding the important pain inputs, pain management advice, and feature requirements of the mobile phone-based pain management system for adolescents with cancer. The content and structure of the conference was decided upon using modifications to the methods of a previous and successful health care consensus conference [49]. A facilitator for the meeting with research and clinical expertise in pediatric pain and pediatric oncology, as well as expertise in consensus conference methods and facilitation, was selected to lead the proceedings. Research Ethics Board approval was obtained at the host institution.

A systematic review of the published pediatric oncology pain management literature was conducted prior to the meeting to understand the current body of knowledge pertaining to pediatric cancer pain management. Results of this review, along with the meeting agenda, were disseminated to conference participants in advance of the meeting for their review and consideration.

Fifteen participants were selected based on their professional and research expertise in pediatric pain, pediatric oncology, or mHealth technology development and testing. Health care experts provided input on the needed clinical aspects of the system, and technology experts provided input on the technical practicality and feasibility of the system suggestions made by clinicians. A consumer (ie, a university educated adult living with chronic pain) was also included. The pain consumer did not have cancer but was asked to participate because of their extensive experience with pain and participation in scientific research and clinical projects. Table 1 outlines the demographic characteristics of conference participants.

This conference used nominal group technique to develop consensus on the necessary features of the smartphone-based pain management system. Nominal group technique is a method used to pool judgment from a group of subject matter experts through quantitative and qualitative data collection methods [50-52]. Briefly, nominal group technique involves the use of a highly structured face-to-face meeting consisting of 2 facilitated rounds of voting on solutions to a problem posed to the group [51,53]. In the consensus conference conducted for the present study, an introductory presentation to conference participants on Day 1 highlighted (1) results of the systematic literature review, (2) overall goals of the 2-day meeting, and (3) reviewed the nominal group method. When the proceedings began, a question was posed to the group (eg, “What are the ‘patient-driven’ pain treatment techniques adolescents may use to manage their pain?”). A list of all questions posed to the group is shown in Multimedia Appendix 1. Each participant then had 2 uninterrupted minutes to address the group with 1 particular answer to the question and their rationale for selecting this answer. Answers were recorded and participants voted on the validity of each answer. Results of the vote were presented to the conference participants by the facilitator and each participant had an additional opportunity to address the group regarding the selected answers. A second round of voting followed. Consensus was considered achieved when 75% of the participants endorsed a given answer. All answers that did not reach 75% endorsement were discarded or reformulated through discussion. Conference proceedings were audio-recorded and field notes were made during both days.

Following the consensus conference, participant answers to the posed questions were used to develop a catalog of system requirements and a standardized pain treatment algorithm. When clarification on the data generated during the conference was required, consensus conference participants were queried by email and provided feedback. Audio-recordings and field notes were also referred to as necessary.

Prior to commencement of Phase 2 of this study, Research Ethics Board approval was obtained from the participating institution. A catalogue of system requirements and a prototype algorithm were developed based on Phase 1 results and iterative cycles of individual qualitative interviews were conducted with pediatric oncology and pain health care providers and adolescents with cancer to vet and refine the algorithm and requirements. A computer-based functional mock-up of the app was also developed to test with participants to improve their understanding of the intervention under development and focus their recommendations.

Participants were recruited from 1 large pediatric academic health care institution (Toronto, Canada) over a 3-month period in 2013. Health care professionals were included in the study if they specialized in either pediatric pain or oncology, were English speaking, were from any clinical profession, and had worked for at least 1 year. Health care professionals were excluded if they were in training and had not achieved full licensure. Adolescents who were English speaking, 12-18 years of age, diagnosed with cancer at least 3 months before testing, actively undergoing cancer treatment, and having experienced pain of any intensity in the week preceding testing were eligible to participate. The adolescent exclusion criterion was severe cognitive impairment as identified by a member of their health care team. A convenience sampling strategy, augmented with purposive sampling, was used to attempt to achieve maximum sample variation in age, sex, and diagnosis for adolescents. Maximum variation in profession was sought from health care professionals. This sampling strategy provided information-rich cases and ensured insight from a heterogeneous sample. Characteristics of both groups are shown in Table 2.

Audio-recorded individual interviews were conducted with adolescents with cancer and health care professionals. Interviews were conducted with each participant group until data redundancy or the point when no new data were gathered that had not previously been categorized [54]. Each adolescent participant completed brief demographic and disease-related questionnaires, as well as a survey related to smartphone use. Additional information was obtained from medical charts as needed. Each interview lasted between 20 and 45 minutes. Semistructured interview guides were used to lead the discussion. Participants were given a brief description of the app and its intended function. Adolescent participants were additionally shown a semifunctional mock-up of the app using laptop-based smartphone emulator software. Very early in the interview process, it became clear that adolescents were not comfortable commenting on and critiquing the developed algorithm. As such, the process of algorithm vetting was conducted only with health care professionals because they are accustomed to algorithm use in the clinical setting and could provide valuable clinical insight.

Interview questions moved from general to specific and focused on eliciting adolescent and health care professional perceptions and suggestions on the system and algorithm [55,56]. The interview guides used with both adolescent and health care professional participants are shown in Multimedia Appendix 2. The interviewer recorded field notes on participant comfort with the interview process and other nonverbal communication. All audio-recorded interviews were transcribed and entered into NVivo 10.0 software [57] for coding. Field notes were also transcribed and coded alongside transcripts using NVivo 10.0.

The qualitative data analysis process began as soon as the first interview was conducted. Two people independently conducted data analyses and discussed themes that emerged from the interviews with reference to field note data. Discrepancies in opinion were resolved using group discussion with a third party. A conventional qualitative content analysis approach was used to code data [58]. Specifically, each analyst independently made notes on impressions and thoughts (codes) related to the interview, according to the study objectives [55]. Codes were then grouped into meaningful categories or themes that reflected perceptions and suggestions for the smartphone-based pain management system and its clinical algorithm. Changes to the catalogue of system requirements and the algorithm were made following individual interviews. Further interviews were conducted until neither the adolescents nor health care professionals had further suggestions for changes. All descriptive quantitative data (for both study phases) were analyzed using SAS 9.1 software [59].

The 2-day consensus conference established the key elements of the smartphone pain management system, which could be used to build the catalogue of system requirements and design the preliminary pain care algorithm.

Clinically important pain inputs defined the clinical characteristics of an adolescent’s pain experience that should result in pain management support from the app. Specifically, a clinically important pain input represented a threshold of adolescent-reported pain that facilitates the generation of automated help. Using the nominal group technique, 11 pain inputs were originally suggested by clinical experts. Following voting and a second round of discussion, 4 inputs were retained as thresholds for system-delivered pain management advice. These adolescent inputs were (1) pain of any severity now, (2) pain of any severity in the previous 12 hours, (3) pain interference with activities of daily living, and (4) a sense of not being able to control pain. Below, a clinical psychologist discussed the rationale for including multidimensional pain assessment responses as important inputs:

“We’ve talked a lot about assessing the intensity of pain…but from the psychology side one of the missing ingredients is the interpretation of pain in some way…So, it becomes important to understand how someone is coping with pain and the emotional climate when pain is occurring…because pain interpretation is an important part of self-management techniques.”

The appropriate pain management advice, which the smartphone system could provide to adolescents, was also established. Pain management advice was grouped as being pharmacological, physical, or psychological in nature. Endorsement for this grouping is illustrated in a quote from a pediatric pain management expert:

“I agree that you need the broad categories up front. And then, you know, you may…click on a selection and you have everything within it and you can delve in a bit deeper for some suggestions about things you might use.”

Clinical experts concentrated their suggestions on pain management advice that was considered appropriate for an adolescent to undertake on their own. Therefore, complex pain management strategies, or strategies requiring advanced clinical training and licensure (eg, medication adjustments), were not suggested as potential advice for adolescents.

Consensus conference participants suggested 2 pharmacological options, which were retained throughout voting. Pharmacological options were reminders to take prescribed medications. Six physical strategies were suggested and 4 were retained after both rounds of voting, including suggestions such as the use of hot/cold therapies. Eighteen psychological pain management therapies were originally suggested, with 9 being retained after voting. Psychological recommendations included using distraction and seeking social support, among others.

Finally, nominal group technique established the needed system functionalities to maximize the effectiveness of the app. Following 2 rounds of voting, 5 requirements were retained and included in the smartphone system design. Table 3 details the system requirements and the rationale for including each in the pain management system.

All participants endorsed the smartphone system and thought that it would be beneficial to adolescents with cancer pain. Specific beneficial aspects of the system from the point of health care professionals included the ability to manage pain the moment it occurs and to intervene when adolescents experience cancer pain in their home environments. The benefits of the system as cited by adolescents included having a record of pain and pain care to discuss with their health care team at subsequent clinic visits, the ability to connect through the system with other adolescents with cancer, and the ability to manage pain the moment it occurs, including through the use of several psychological and physical management strategies. Adolescent endorsement of the ability to receive several diverse pain management strategies is highlighted in the quote below.

“[I like] that there’s different options available. It’s not like, ‘Ok, take this medication or something’. Yeah, I think I like that about the app…Like, you can learn more about pain and…you know, you could like write a journal too.”

Overwhelmingly, both oncology health care professionals and adolescents endorsed the inclusion of a clinical expert in the smartphone system design. The app was originally designed as a stand-alone system that would provide pain management advice to adolescents without the active input of a health care professional. However, following the interview process it was decided that the active input of a clinical expert (eg, an oncology-trained registered nurse) would improve both the effectiveness and safety of the system. The system design was then changed so that adolescent pain reports considered clinically important (eg, 3 consecutive reports of moderate-to-severe pain [pain ≥ 4/10]) will trigger an email alert to a registered nurse. The nurse will then contact the adolescent to assist in clinical decision making. A representative quote from an adolescent about the addition of a health care professional is below:

“Um, if [a health care professional] contacts you, I think that also makes sense. But, um, like I’m not sure if you need them to contact for every single pain. But, I think if it falls under certain categories then that’ll be good.”

Both health care professionals and adolescents recognized that individualization of pain management advice is important. Both groups highlighted that individuals may respond to pain management advice uniquely based on their personal characteristics and preferences. One adolescent stated:

“I don’t know, I think it depends on the person, I think. Like, I wouldn’t mind it talking to my parents, but that’s just me I guess.”

As such, a mechanism that allows adolescents to rate the likability and effectiveness of the pain management advice the system offers was added to the design. Through this mechanism, when future pain management advice is given to adolescents, the advice adolescents liked and used most often will be offered first. A capacity for the system to individualize pain management advice to a specific adolescent was endorsed by health care professionals as highlighted in the below quote:

“You can kind of think about this as an N of 1 trial each time somebody is doing it. So if there is a way to build in that learning, then ultimately [the app] becomes evidence-based for that particular person.”

Vetting of the algorithm occurred with health care professionals who recommended changes to the algorithm based on their clinical and research expertise. Required major changes included revision of the threshold for a change in pain management, suggestion that the wording used in app be changed to improve an adolescent’s ability to understand questions and management advice, and the recommendation that adolescents be informed regarding time to contact from the registered nurse monitoring the system. For instance, a health care professional said this regarding the wording used in this app:

“This could be clearer in terms of the language here. It’s almost like in needs to be black and white, ‘take your narcotic’.”

In response to these required changes: (1) the threshold for the system to offer additional advice was lowered so that if pain was not improved 1 hour after initial management advice was made, new advice may be delivered; (2) adolescent-appropriate wording of system questions and management instructions was used; and (3) an alert was added to the system to tell adolescents that a health care provider was being informed about their pain. The alert further stated that if the adolescent did not hear from the health care professional within a given amount of time, they should talk to their caregivers and/or seek medical help. Following these revisions, clinicians suggested no further changes and the algorithm was considered finalized.