This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
The majority of all surgeries are performed on an outpatient basis (day surgery). The Recovery Assessment by Phone Points (RAPP) app is an app for the Swedish Web-version of Quality of Recovery (SwQoR), developed to assess and follow-up on postoperative recovery after day surgery.
The objectives of this study are (1) to estimate the extent to which the paper and app versions of the SwQoR provide equivalent values; (2) to contribute evidence as to the feasibility and acceptability of a mobile phone Web-based app for measuring postoperative recovery after day surgery and enabling contact with a nurse; and (3) to contribute evidence as to the content validity of the SwQoR.
Equivalence between the paper and app versions of the SwQoR was measured using a randomized crossover design, in which participants used both the paper and app version. Feasibility and acceptability was evaluated by a questionnaire containing 16 questions regarding the value of the app for follow-up care after day surgery. Content validity evaluation was based on responses by day surgery patients and the staff of the day surgery department.
A total of 69 participants completed the evaluation of equivalence between the paper and app versions of the SwQoR. The intraclass correlation coefficient (ICC) for the SwQoR was .89 (95% CI 0.83-0.93) and .13 to .90 for the items. Of the participants, 63 continued testing the app after discharge and completed the follow-up questionnaire. The median score was 69 (inter-quartile range, IQR 66-73), indicating a positive attitude toward using an app for follow-up after day surgery. A total of 18 patients and 12 staff members participated in the content validity evaluation. The item-level content validity index (I-CVI) for the staff group was in the 0.64 to 1.0 range, with a scale-level content validity index (S-CVI) of 0.88. For the patient group, I-CVI was in the range 0.30 to 0.92 and S-CVI was 0.67. The content validity evaluation of the SwQoR, together with three new items, led to a reduction from 34 to 24 items.
Day surgery patients had positive attitudes toward using the app for follow-up after surgery, and stated a preference for using the app again if they were admitted for a future day surgery procedure. Equivalence between the app and paper version of the SwQoR was found, but at the item level, the ICC was less than .7 for 9 items. In the content validity evaluation of the SwQoR, staff found more items relevant than the patients, and no items found relevant by either staff or patients were excluded when revising the SwQoR.
Day surgery (outpatient surgery) is an expanding and well-established practice in the United States and in many European countries [
The access to reliable and validated instruments to measure and evaluate the quality of postoperative recovery is important in both research and clinical practice. Furthermore, an assessment of recovery can lead to reduced readmissions to the hospital after day surgery [
The first phase of this study was cross-sectional to estimate equivalency of the two versions of the SwQoR; for this estimate the order of the versions were randomized. The second phase used a prospective design. The study was approved by the regional ethical review board in Uppsala, Sweden (2014/456).
The study was carried out from January to May, 2015. A total of 70 participants were recruited consecutively in two day surgery settings in Sweden. The inclusion criteria were that patients must be adults over 17 years of age, be admitted for day surgery, be able to understand the Swedish language both in speech and in writing, and undergo general anesthesia. The exclusion criterion was that the patient does not having access to a mobile phone with Internet access and a Web browser (smartphone).
For evaluating the content validity of the SwQoR, 18 patients participating in the study and 12 staff members were recruited. The staff group worked in the day surgery departments participating in this study and included 4 anesthesiologists, 4 surgeons, and 4 nurses.
RAPP is a Web-based app that is suitable for all mobile phone models. The participants’ own mobile phones were used for this study, following the principle of bring your own device (BYOD) [
The final question in the app ”Do you want to be contacted by a nurse?”. To be answered with ”Yes” or ”No”.
When making the appointment for their operation, preoperative patients were provided with written information about the study and were requested to bring their mobile phone to the day surgery department on the day of surgery. At admission, the participants were verbally informed about the aim of the study, and those who agreed to participate provided informed consent. Both preoperatively and prior to discharge from the hospital, the participants were thoroughly trained and informed about the app’s functionalities and navigation, and they learned how to document their postoperative recovery. Inclusion, information, and follow-up were conducted by one member of the research team (KD). If the question “Do you want to be contacted by a nurse?” was ever answered with “Yes” by a patient, an immediate email was sent with the participant’s study code to one member of the research group who had access to the code set. This research team member then contacted a nurse at the day surgery department where the surgery was performed, and the nurse contacted the participant. In one of the day surgery settings, all the requested contact calls were conducted by one specific nurse, and in the other setting, the nurse in charge of all incoming phone calls that day made contact with the patient.
To measure the equivalence between the app and paper versions of the SwQoR, participants were randomized into one of the following two conditions for answering its 31 items: (1) a paper questionnaire followed by the app questionnaire, or (2) the app questionnaire followed by the paper questionnaire. In both cases, 30 minutes elapsed between the app and paper measurements for both groups. This interval was guided by an earlier study conducted by Gower et al [
After completing the first version of SwQoR, the participants were not able to see their previous answers when they responded in the second round.
All included participants were asked to use the RAPP (answer the 31 items in the SwQoR, as well as the final question “Do you want to be contacted by a nurse?”) each day for 7 days after discharge. One member of the research team (KD) was always available during the study period (both by phone and by email) if the participants had any problems using the app. On the 7th day, a follow-up phone call was made by a member of the research team (KD), who used a questionnaire to ask the participants for feedback on using a mobile phone app to assess their postoperative recovery. The follow-up questionnaire was designed for this study and guided by a similar questionnaire used by Ainsworth et al [
Questions that were suggested as missing yet relevant for postoperative recovery were included in the content validity review described below.
To assess the content validity of the SwQoR used after a day surgery, staff and patients evaluated the SwQoR together with the items suggested by the participants as missing in the follow-up. The staff members and patients rated the items regarding intelligibility and relevance on a 4-point scale with 1 representing
Each participant was assigned a study code and no personal data, such as social security number, name, age, gender or telephone number were stored in the app. Only one member of the research team had access to the code set and could identify who was answering the app. The paper questionnaires were also coded. The codes were stored separately from the questionnaires. Data transmission between the mobile phone and the server used for the test occurred via the mobile network General Packet Radio Service (GPRS), and the data were stored in a secure server that required a login and password to access the answers from the app.
The equivalence testing between the paper and app versions was analyzed using the intraclass correlation coefficient (ICC) (one-way, single measures). An ICC value of .7 or above was considered acceptable [
To our knowledge, no previous studies have compared paper and digital/electronic postoperative questionnaires. Thus, the number of participants for this study was guided by two earlier studies in other contexts that compared paper and electronic questionnaires. Salaffi et al [
Surgery was canceled for one of the 70 included participants. The characteristics of the remaining 69 patients are presented in
Patient characteristics (N=69).
Characteristics | n (%) | |
Sex | ||
Men | 41 (59) | |
Women | 28 (41) | |
Age, mean (SD) | 50 (15) | |
Surgery type | ||
General | 33 (48) | |
Orthopedic | 26 (38) | |
Gynecology | 4 (6) | |
Hand | 3 (4) | |
Ear, nose, throat | 3 (4) |
Flowchart describing the recruitment of participants and data collection.
The agreement between the app and paper versions is presented in
Agreement between the app and paper versions of the SwQoR (N=66).
Category | Paper mediana (IQR) | App median (IQR) | ICC (95% CI) | |
Able to breathe easy | 10 (9.75-10) | 10 (9.75-10) | .25 | .81 (0.71-0.88) |
Sleeping well | 10 (8-10) | 10 (8-10) | .28 | .89 (0.82-0.93) |
Being able to enjoy food | 10 (8-10) | 10 (8-10) | .12 | .80 (0.69-0.87) |
Feeling rested | 9 (7-10) | 8 (5-10) | .008 | .80 (0.67-0.87) |
Having a general feeling of well-being | 9 (8-10) | 9 (6-10) | .03 | .73 (0.59-0.83) |
Feeling in control | 9 (8-10) | 9 (7-10) | .75 | .82 (0.72-0.88) |
Feeling relaxed | 9 (8-10) | 9 (7-10) | .14 | .77 (0.65-0.85) |
Speaking normally | 10 (9-10) | 10 (8-10) | .013 | .71 (0.57-0.81) |
Able to look after personal hygiene | 10 (9-10) | 10 (8-10) | .63 | .68 (0.53-0.80) |
Able to write as usual | 10 (10-10) | 10 (9-10) | .04 | .86 (0.78-0.91) |
Able to return to work or usual duties about the home | 5 (1-10) | 5 (2-9) | .82 | .90 (0.84-0.94) |
Nausea and/or vomiting | 0 (0-3) | 0 (0-2.75) | .06 | .89 (0.82-0.93) |
Feeling restless | 0 (0-1) | 0.5 (0-4) | .001 | .49 (0.28-0.66) |
Shivering or twitching | 0 (0-0) | 0 (0-1) | .003 | .36 (0.12-0.55) |
Feeling too cold | 0 (0-3) | 1 (0-3) | .20 | .76 (0.64-0.85) |
Dizziness | 1 (0-4.75) | 1.5 (0-5) | .23 | .61 (0.43-0.75) |
Pain in the surgical wound | 2.5 (0.25-6.75) | 3 (0-7) | .27 | .72 (0.57-0.82) |
Anxiety | 0 (0-2) | 0 (0-2) | .87 | .66 (0.50-0.78) |
Depressed | 0 (0-1) | 0 (0-1) | .16 | .80 (0.69-0.87) |
Feeling lonely | 0 (0-0) | 0 (0-1) | .03 | .68 (0.53-0.79) |
Difficulties getting to sleep | 0 (0-1) | 0 (0-1.25) | .82 | .64 (0.48-0.77) |
Nightmares | 0 (0-0) | 0 (0-0) | .02 | .87 (0.80-0.92) |
Headache | 0 (0-2) | 0 (0-2) | .39 | .81 (0.71-0.88) |
Muscle pain | 0 (0-2) | 0 (0-2) | .67 | .74 (0.61-0.83) |
Back pain | 0 (0-0) | 0 (0-1) | .07 | .83 (0.74-0.89) |
Sore throat | 0 (0-1) | 0 (0-2) | .02 | .79 (0.68-0.87) |
Sore mouth | 0 (0-0) | 0 (0-0.25) | .05 | .78 (0.66-0.86) |
Difficulties concentrating | 0 (0-2) | 0 (0-2) | .22 | .78 (0.66-0.86) |
Trouble urinating | 0 (0-0) | 0 (0-0) | .37 | .73 (0.59-0.83) |
Diarrhea | 0 (0-0) | 0 (0-0) | .07 | .30 (0.06-0.51) |
Feeling constipated | 0 (0-0) | 0 (0-0) | .03 | 0.13 (-0.12-0.36) |
a0= none of the time, 10=all of the time.
bWilcoxon signed ranks test.
The RAPP was answered over a mean of 5 days (min 1, max 7). When asked about reasons for not answering the RAPP all 7 days, 8 participants reported not remembering to answer, 5 reported technical issues such as the app logging out or problems with the network, and 2 were re-admitted to the hospital. Those participants who forgot to answer declared that they wanted a daily reminder.
Results from the follow-up questionnaire showed that the participants had a positive attitude toward using the app, felt comfortable using the technology, and took a reasonable amount of time to answer the items in the app (
Results from the follow-up questionnaire, questions 1 to 11 (N=63).
Question | Mediana | IQR | Min, max |
I felt familiar with using this type of technology | 1 | 1-1 | 1, 3 |
I would like to use this type of postoperative follow-up again if undergoing surgery | 1 | 1-1 | 1, 4 |
I think other people would find the software tool easy to use | 2 | 1-3 | 1, 5 |
This type of systematic follow-up helped me and would help other patients in the same situation | 1 | 1-2 | 1, 4 |
Answering the questions made me feel better | 5 | 3-7 | 1, 7 |
It was difficult to answer the questions | 7 | 7-7 | 2, 7 |
I would like to avoid answering the questions | 7 | 7-7 | 2, 7 |
Answering the questions took a lot of time | 7 | 7-7 | 4, 7 |
It was difficult to keep track of what the questions were asking | 7 | 6-7 | 4, 7 |
It was inconvenient to answer the questions using my smartphone | 7 | 7-7 | 2, 7 |
Answering the questions made me feel worse | 7 | 7-7 | 1, 7 |
Total scoreb (positive items reversed) | 69 | 66-73 | 45, 77 |
a1= strongly agree, 7=strongly disagree.
bMinimum possible score 11, maximum possible score 77.
The request to be contacted by a nurse via the app was used 15 times (3.4%, 15/441) in relation to the total number of chances to request contact (441 instances with 63 participants using the app for 7 days each). The reasons for the contact were the following: (1) questions concerning the surgical wound regarding the dressing, stitches, swelling, etc (44%, 7/16), (2) pain and/or pain management (19%, 3/16), (3) general information (13%, 2/16), (4) constipation (13%, 2/16), (5) request for a medical certificate (6%, 1/16), and (6) nausea (6%, 1/16). The opportunity to get in contact with a nurse via the app provided a sense of security and was appreciated by all except one of the participants, who wanted to use the telephone for initiating contact instead of the app. Participants (25%, 16/63) also expressed that it is typically difficult to contact a caregiver and that this opportunity provided a simple solution for that problem. Three additional items were suggested by the participants: fever, reddened surgical wound, and swollen surgical wound.
In total, 34 items were included when evaluating content validity (ie, the original 31 items and 3 additional items). Five surveys from the patients were incomplete, 13 (72%, 13/18) were included in this analysis. Results of the content validity are presented in
An I-CVI rating less than 0.78 by both patient and staff led to the removal of the following 7 items in the SwQoR: (1) able to enjoy food, (2) able to write, (3) feeling restless, (4) shaking or twitching, (5) feeling too cold, (6) feeling alone, and (7) backache. When calculating the S-CVI after removing these 7 items, the S-CVI was 0.94 for staff and 0.72 for patients. Four related items that were considered by patients to be very similar (had a good sleep, feel rested, had difficulty falling asleep, had bad dreams
Revision of items in SwQoR.
Revision | SwQoR 31 | SwQoR 24 |
Merged into one item | Sleeping well | Had sleeping difficulties |
Nightmares | ||
Difficulties getting to sleep | ||
Feeling rested | ||
Not changed | Able to breathe easy | Able to breathe easy |
Having a general feeling of well-being | Having a general feeling of well-being | |
Feeling in control | Feeling in control | |
Feeling relaxed | Feeling relaxed | |
Speaking normally | Speaking normally | |
Able to look after personal hygiene | Able to look after personal hygiene | |
Able to return to work or usual duties about the home | Able to return to work or usual duties about the home | |
Nausea and/or vomiting | Nausea and/or vomiting | |
Dizziness | Dizziness | |
Pain in the surgical wound | Pain in the surgical wound | |
Anxiety | Anxiety | |
Depressed | Depressed | |
Headache | Headache | |
Muscle pain | Muscle pain | |
Sore throat | Sore throat | |
Sore mouth | Sore mouth | |
Difficulties concentrating | Difficulties concentrating | |
Trouble urinating | Trouble urinating | |
Feeling constipated | Feeling constipated | |
Diarrhea | Diarrhea | |
Included after content validity assessment | N/A | Reddened surgical wound |
Fever | ||
Swollen surgical wound | ||
Excluded after content validity assessment | Feeling restless | N/A |
Shivering or twitching | ||
Feeling too cold | ||
Being able to enjoy food | ||
Able to write as usual | ||
Back pain | ||
Feeling lonely |
The present study shows agreement between the paper and app versions of the SwQoR, but on an item level in the SwQoR, the ICC was less than 0.7 for 9 items. The participants were very positive toward using the app for a follow-up survey after undergoing day surgery, did not find it to take too long to fill in, and were willing to use this follow-up method if admitted for a future day surgery. The content validity showed that more items were found to be relevant by the staff group compared to the patient group.
When measuring equivalence between the paper and app version of the SwQoR, we used parametric statistics even though the SwQoR collects ordinal level data. This allows results from the study to be compared with results from previous studies on the QoR instrument [
In our study, both the app and paper version of the SwQoR showed excellent internal consistency (Cronbach’s alpha .91 and .91, respectively), which is a similar result to previous studies regarding the QoR with Cronbach’s alpha values of .93 [
To our knowledge, there has been only one previous study that tested an app for follow-up after day surgery. Semple at al [
Similar results were also reported in a study by Stomberg et al [
The principle of BYOD was used in this study, which resulted in excluding 18 otherwise eligible patients. This could of course have affected the results regarding feasibility and acceptability, since it would be natural that a person with access to a mobile phone would have a more positive attitude toward using mobile apps. However, the number of mobile phone users is increasing and will most likely continue to increase in coming years. Further, a BYOD-approach eliminates the cost of providing tablets or mobile phones (smartphones) in the health care system and in the study [
In the content validity evaluation in this study, the analyses for staff and patients were conducted separately as these two groups utilize the SwQoR assessment from different points of view. Staff members in the day surgery department assess postoperative recovery to follow-up on, evaluate, and improve anesthetic and postoperative care. Patients personally experience the postoperative recovery and thus use the SwQoR to report on that recovery. In the evaluation of content validity, staff rated the items higher than the patients. Only for 9 items was I-CVI greater than 0.78 in the patient group, whereas I-CVI was greater than 0.78 for 27 items in the staff group. The rank order of the items was similar in both the patient and staff groups, as the patients just tended to consider items to be less relevant than the staff in this study. In contrast to our results, Myles at al [
When revising the SwQoR, the decision was made to retain any item with I-CVI greater than 0.78 in either the patient or the staff group, since that item was considered relevant for the perspectives and contexts of each group. Revision of the QoR was previously described by Stark et al [
The items in the QoR have been previously summarized and reported across the five dimensions
This study was conducted in two day surgery departments in Sweden including participants familiar with using mobile phones and participants who spoke and could read the Swedish language. Only day surgery patients who underwent general anesthesia were included. Further studies including all types of anesthesia and surgeries should be conducted, as well as studies including non-Swedish speaking participants answering in their own language. There was also a technical limitation including no opportunity for the participants to change their prior answers in the app and this might have affected the reported answers.
On the basis of our results, we recommend some changes to be implemented in the next version and further development of the RAPP app (
A value on the numeric VAS should be chosen before the respondent can continue to the next question.
Incorporate the ability to go back and check or change prior answers.
Reformulate all positive (n=8) items into negative items.
Include a daily reminder to fill in the app, which is only possible in a native app.
Develop the web-based app as a native app.
Testing of the new items that was included after the content validity.
Day surgery patients had positive attitudes toward using this app for follow-up after surgery and wanted to use the app again if admitted for future day surgeries. The ability to get in contact with a nurse via the app was very much appreciated and made the participants feel secure. Equivalence between the app and paper versions of the SwQoR showed agreement (ICC .89), but at the item level, the ICC was less than .7 for 9 items. This study shows the importance of evaluating an instrument converted from paper to electronic assessment formats and the need to evaluate the specific app for this assessment. In the content validity evaluation of the SwQoR, staff found more items relevant than the patients. The content validity evaluation of the SwQoR together with 3 new items led to a reduction from 34 to 24 items in the SwQoR.
Content validity SwQoR.
bring your own device
intraclass correlation coefficient
item-level content validity index
patient reported outcome
Quality of Recovery instrument
Recovery Assessment by Phone Points
scale-level content validity index
Swedish Web-version of Quality of Recovery
visual analogue scale
KD, ME, MJ, and UN made substantial contributions to the conception and design. KD, ME, MJ, and UN drafted the article or revised it critically for important intellectual content. Final approval of the version to be published was made by KD, ME, MJ, and UN.
This study is supported by FORTE (the Swedish Research Council for Health, Working Life and Healthcare), grant number 2013-4765; and Vetenskapsrådet (The Swedish Research Council), grant number 2015-02273.
Author Ulrica Nilsson and Örebro University Enterprise AB hold shares in RAPP-AB.