Published on in Vol 13 (2024)

Preprints (earlier versions) of this paper are available at, first published .
A Digital Intervention to Improve Skin Self-Examination Among Survivors of Melanoma: Protocol for a Type-1 Hybrid Effectiveness-Implementation Randomized Trial

A Digital Intervention to Improve Skin Self-Examination Among Survivors of Melanoma: Protocol for a Type-1 Hybrid Effectiveness-Implementation Randomized Trial

A Digital Intervention to Improve Skin Self-Examination Among Survivors of Melanoma: Protocol for a Type-1 Hybrid Effectiveness-Implementation Randomized Trial


1Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, United States

2University of Colorado Anschutz Medical Campus, Aurora, CO, United States

3RTI International, Research Triangle Park, NC, United States

Corresponding Author:

Sharon Manne, PhD

Rutgers Cancer Institute of New Jersey

195 Little Albany St,

New Brunswick, NJ, 08901

United States

Phone: 1 (732) 357 7978


Background: Although melanoma survival rates have improved in recent years, survivors remain at risk of recurrence, second primary cancers, and keratinocyte carcinomas. The National Comprehensive Cancer Network recommends skin examinations by a physician every 3 to 12 months. Regular thorough skin self-examinations (SSEs) are recommended for survivors of melanoma to promote the detection of earlier-stage, thinner melanomas, which are associated with improved survival and lower treatment costs. Despite their importance, less than a quarter of survivors of melanoma engage in SSEs.

Objective: Previously, our team developed and evaluated a web-based, fully automated intervention called mySmartSkin (MSS) that successfully improved SSE among survivors of melanoma. Enhancements were proposed to improve engagement with and outcomes of MSS. The purpose of this paper is to describe the rationale and methodology for a type-1 hybrid effectiveness-implementation randomized trial evaluating the enhanced MSS versus control and exploring implementation outcomes and contextual factors.

Methods: This study will recruit from state cancer registries and social media 300 individuals diagnosed with cutaneous malignant melanoma between 3 months and 5 years after surgery who are currently cancer free. Participants will be randomly assigned to either enhanced MSS or a noninteractive educational web page. Surveys will be collected from both arms at baseline and at 3, 6, 12, and 18 months to assess measures of intervention engagement, barriers, self-efficacy, habit, and SSE. The primary outcome is thorough SSE. The secondary outcomes are the diagnosis of new or recurrent melanomas and sun protection practices.

Results: Multilevel modeling will be used to examine whether there are significant differences in survivor outcomes between MSS and the noninteractive web page over time. Mixed methods will evaluate reach, adoption, implementation (including costs), and potential for maintenance of MSS, as well as contextual factors relevant to those outcomes and future scale-up.

Conclusions: This trial has the potential to improve outcomes in survivors of melanoma. If MSS is effective, the results could guide its implementation in oncology care and nonprofit organizations focused on skin cancers.

International Registered Report Identifier (IRRID): RR1-10.2196/52689

JMIR Res Protoc 2024;13:e52689




More than 97,610 cases of cutaneous malignant melanomas will be diagnosed in the United States in 2023, making it the fifth most commonly diagnosed cancer. With incidence rates more than tripling from 1975 to 2020, the population of survivors of melanoma is estimated at >1 million. Among persons diagnosed with localized, regional, or distant-stage melanoma, the 5-year survival rates have improved in the last decade, with 99%, 65%, and 25% survival rates, respectively [1]. Improved survival and increased incidence have translated to an estimated annual treatment cost for melanoma of approximately US $2.5 billion [2].

Survivors of melanoma remain at risk of recurrence, second primary cancers, and keratinocyte carcinomas. Recurrence rates depend on tumor thickness and nodal involvement and range from 3% to 24% among persons with thinner lesions to 51% among persons with thicker lesions or lymph node involvement [3-5]. Recurrent or new primary melanomas occur most commonly during the first 5 years after diagnosis but can arise many years later [6,7]. Recurrent melanoma typically occurs at local or regional sites, and approximately half of distant recurrences present within the skin or lymph nodes [8,9]. Psychosocial morbidity is an issue for patients and survivors, with studies reporting higher anxiety and depressive symptoms than in the general population [10] and elevated anxiety about recurrence.

For survivors of melanoma, National Comprehensive Cancer Network guidelines recommend several actions, including skin examinations by a physician every 3 to 12 months and regular thorough skin self-examination (SSE) [11-13], which entails a deliberate, systematic inspection of all areas of the body using a mirror or the assistance of another person to examine hard-to-view areas [14]. Patient education about SSE by a health care professional is also recommended. Engagement in regular sun protection behavior is also recommended to reduce the risk of subsequent skin cancers as UV radiation from the sun is a contributing factor for melanoma and other skin cancers [2].

Limited data suggest that performance of SSE is associated with differential survival rates among patients with melanoma, and professional recommendations for regular SSE are supported by 3 research findings. First, more than half of recurrences and new primary melanomas are detected by survivors themselves [11,15,16]. Second, individuals who perform SSE are diagnosed with significantly earlier-stage melanomas than those who do not. Detection and treatment of recurrent disease and new primary cancers at earlier stages leads to improved survival, which is not accounted for by lead-time bias [17]. Third, melanomas identified through SSE are thinner than those found incidentally [16,18,19]. Retrospective studies suggest that individuals who perform SSE have lower tumor thickness. Thinner melanomas are associated with better survival [19-23]. Thus, promoting regular SSE will likely enhance the early detection of easier-to-treat recurrences and new primary cancers among survivors. For example, Robinson et al [24] found that 13.4% of patients in their SSE randomized controlled trial (RCT) developed a new melanoma over the 2-year follow-up. Patients in the SSE intervention condition detected 81% of these melanomas, approximately 34% of which were invasive, with only a 1% increase in physician visits. The costs of treating earlier- versus later-stage melanomas are significantly lower [25,26]. In summary, regular thorough SSE is recommended for survivors of melanoma and results in the detection of earlier-stage, thinner melanomas, which is associated with improved survival and lower treatment costs.

Despite its importance, engagement in regular, thorough SSE is low. Compliance rates with thorough SSE range between 7% and 17% among survivors of melanoma [14,20,22,27,28]. Studies document wide variation in SSE performance, with figures dependent upon the way that SSE is measured and the time frame for assessment. When patients are asked whether they have performed any form of SSE in the past 2 months, high rates of performance are observed (71.5%) [29]. SSE rates are higher when specifying any performance in the past year (84.3%) [30]. However, significantly lower rates are found if SSE is defined by its thoroughness. We reported that 13.7% of survivors checked 4 key areas and had someone assist them or used a mirror for hard-to-see areas [30]. Loescher et al [31] found that 16% of women and 7% of men examined each of the 7 designated body parts in the previous 2 months. Mujumdar et al [32] reported that 17% of survivors examined a minimum number of areas of the body (8 out of 9 areas) in the previous 2 months. In our recent work, 65% reported having conducted an SSE in the previous 2 months, but only 7.5% of the sample checked all 15 body parts. Hard-to-see areas were missed, including the scalp (37.9%), buttocks (40.4%), soles of the feet (41.6%), and genitals (44.4%). Less than half (46.2%) [33] reported using a mirror to view hard-to-see places, and only 39% reported asking for assistance. In total, 2% reported using a mole map to guide their most recent SSE.

A limited number of interventions have been evaluated to improve SSE among survivors of melanoma [24,34-37]. Existing interventions have used in-person, print, or web-based delivery modes. Several have demonstrated improvements in SSE [34,35]. Limitations of previous studies include lack of a comparison group [35]; being an in-person or partner-assisted intervention, which compromises the ability to disseminate it [24]; lack of inclusion of long-term outcomes [24]; lack of inclusion of SSE performance [36] as an outcome; and low intervention use and high dropout [35]. No existing intervention is fully automated and, thus, potentially cost-effective and highly scalable. To this end, our team developed and evaluated mySmartSkin (MSS), a web-based, fully automated intervention to improve SSE among survivors of melanoma. MSS is a behaviorally based program that is delivered via the internet, tailored to the user, and fully automated with no human clinical support. MSS was compared with usual care (UC) in an RCT of 430 survivors of melanoma from New Jersey [38,39]. The results indicated a beneficial impact of MSS versus UC on the performance of thorough SSE at all follow-ups up to 1 year [24,34-37]. Effect sizes were in the small to medium magnitude range for SSE, but these effects were not sufficiently strong to scale up for implementation in the existing form.

In addition to SSE, professional agencies recommend engagement in regular sun protection behaviors, such as staying in the shade, applying sunscreen with a sun protection factor of at least 30, and wearing protective clothing (eg, hats and long sleeves). Survivors of melanoma report engaging in higher levels of sun protection behaviors than the general population [40], but their sun protection behaviors do not meet the recommended guidelines [30,32,41]. To date, only 2 intervention studies have targeted improved sun protection behaviors among survivors of melanoma [34,38]. Bowen et al [34] reported significant improvements in some sun protection behaviors (eg, wearing sunglasses and staying in the shade). In our prior work [38], we evaluated the effects of MSS on sun protection behaviors, which was included as a component of the original intervention. The effects of MSS compared with UC on sun protection behaviors were only statistically significant at the 24-week follow-up in analyses that did not control for baseline sun protection behaviors or other potential covariates. Stronger and more consistent improvements in sun protection would be an important goal for future work.

Theoretical Frameworks

Previous empirical findings regarding factors associated with SSE [29,30,32,42,43] and sun protection, as well as the preventive health model (PHM) [44,45], inform the content of MSS. The PHM posits that the performance of preventive behaviors is influenced by background, affective, cognitive, and social factors. Background factors include sociodemographic characteristics, risk factors for skin cancer, medical history, and knowledge about melanoma. Affective factors include concerns about melanoma recurrence and distress about the diagnosis. Cognitive factors include perceived controllability of melanoma, self-efficacy, and benefits and barriers. Social normative factors include family and friend support and physician recommendations. In addition to our previous research on skin cancer risk reduction practices among family members of patients with melanoma [46,47], the PHM has been used successfully to understand and promote screening for colorectal and prostate cancer [44,48]. The original MSS focused on improving the perceived controllability of melanoma through the performance of comprehensive SSE; increasing the perceived benefits of SSE and sun protection; reducing barriers to SSE and sun protection; enhancing self-efficacy for SSE and sun protection; and enhancing family, friend, and physician support for sun protection.

A major challenge for intervention research is that most interventions that demonstrate a beneficial impact are not tested in effectiveness or dissemination trials and are not conducive to “real world” delivery or use. As a result, they do not penetrate the general population and, unfortunately, are not received by those who most need them [49]. Our work is guided by the Practical, Robust Implementation and Sustainability Model (PRISM), which extends the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) [50] framework to consider not only implementation outcomes but also multilevel contextual factors influencing implementation. Reach refers to the percentage and representativeness of persons exposed to a program. Effectiveness refers to the impact on key survivor-level outcomes. As MSS is a free-standing, recipient-facing intervention, adoption is defined as the proportion and representativeness of recipients (in this case, survivors) who adopt (ie, at least log into) the program. Implementation refers to the degree to which a program is delivered and received as intended. This is measured through costs, engagement, acceptability, feasibility, and appropriateness. Maintenance is the extent to which a program and its survivor-level effects are sustained over time. The integration of these models with the study’s aims is shown in Figure 1 [51-55]. In the proposed study, in aim 1, the goal of the iterative process of enhancing the MSS intervention using stakeholder feedback and usability testing is to improve RE-AIM outcomes, which will be assessed in aims 2 and 3. In aim 2, we focus on the effectiveness of the enhanced MSS, testing its effects on survivor-level outcomes, including clinical outcomes (eg, melanomas found). In aim 3, we address the remaining RE-AIM dimensions. In addition, to proactively identify barriers to and facilitators of future scale-up and widespread dissemination and implementation of MSS, we explore multilevel contextual factors identified by key stakeholders drawn from the PRISM domains of recipients, external environment, intervention design, and implementation and sustainability infrastructure. We anticipate that incorporating PRISM and RE-AIM throughout the study aims will ensure that the enhanced intervention is responsive to key stakeholder preferences and that we “design for dissemination” [56], recognizing potential barriers to and facilitators of future scale-up and informing our next stage of developing dissemination and implementation strategies to maximize the public health impact of MSS.

Figure 1. Integration of preventive health model and implementation frameworks with the study aims. RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance.

Study Objectives


The purpose of this type-1 hybrid effectiveness-implementation study is to evaluate the effectiveness of a web-based intervention, MSS, and explore its implementation and relevant contextual factors. A type-1 hybrid effectiveness-implementation approach requires the engagement of multilevel stakeholders throughout the research process, focus on evaluating the effectiveness of the enhanced MSS, and identification of critical factors that could promote or prevent wide-scale implementation. The study’s 3 aims are guided by implementation frameworks and behavior change theories.

Aim 1

The first aim is to enhance MSS by collaborating with multilevel stakeholders. We will collaborate with stakeholders to enhance MSS through qualitative interviews and usability testing of potential intervention enhancements. Enhancements are based on empirically validated behavior change techniques (BCTs) and lessons learned from survivors’ feedback in our previous study. We will use an iterative process that includes (1) key informant interviews with survivors, providers, and professional organization representatives regarding proposed enhancements; (2) conversion to an enhanced mobile-based delivery platform; (3) usability testing; and (4) iterative program refinements.

Aim 2

The second aim is to evaluate the effects of the enhanced MSS on thorough SSE (primary outcome) and examine its impact on the diagnosis of new or recurrent melanomas (secondary outcome) and sun protection practices (secondary outcome). We will conduct an RCT comparing MSS and a noninteractive educational web page with 300 survivors to test the intervention effects on these outcomes.

Aim 2 Hypothesis

We propose that MSS participants will be more likely to perform thorough SSE over the 18-month follow-up period. We will explore the impact of MSS on new or recurrent melanomas and sun protection behaviors. We anticipate diagnosing a greater number of earlier-stage melanomas and observing higher sun protection behaviors among those in MSS compared with those in UC.

Aim 3

The third aim is to assess selected implementation outcomes and identify factors relevant to future scale-up for widespread dissemination, implementation, and maintenance of MSS. We will use mixed methods to assess implementation outcomes and explore perspectives from survivors, care providers, and professional organizations to inform the selection of strategies to best disseminate and implement MSS on a broad scale. The subaims are as follows:

  1. Aim 3a is to estimate program costs and assess the cost-effectiveness of MSS relative to control. We hypothesize that MSS costs will be higher than UC costs. We expect that MSS will be a more cost-effective strategy given its previous positive effects on SSE and the identification of new or recurrent melanoma. If our findings support this as expected, exploratory cost-effectiveness analyses from the health care and societal perspectives will be conducted using simulation models of melanoma-related costs, disease progression, and survival over 5- and 10-year analysis horizons.
  2. Aim 3b is to examine the reach, adoption, implementation (ie, engagement, acceptability, appropriateness, and feasibility), and maintenance of MSS. For reach, we predict that the demographic characteristics of those exposed to MSS will not differ from those of the general population of survivors of melanoma. For adoption, we propose that the proportion of contacted and eligible survivors randomized to MSS who consent, complete the baseline questionnaire, and log into MSS will be equal to or greater than the proportion in our previous efficacy trial. For engagement, we propose that 80% of MSS participants will log into the intervention at least once. For acceptability, we predict that MSS will be rated as highly acceptable. We do not have hypotheses regarding appropriateness, feasibility, and maintenance as these data will be evaluated in terms of future implementation of MSS should it have an impact on SSE and sun protection outcomes. For each of these implementation outcomes, we will assess the demographic characteristics of MSS participants to determine whether outcomes differ among survivors in specific subgroups (eg, by age, sex, race, ethnicity, and socioeconomic status).
  3. Aim 3c is to identify and describe contextual factors experienced by multilevel stakeholders as key to scale-up and widespread implementation of MSS, including consideration of potential delivery settings, timing of delivery, and resources needed to promote its implementation.

Ethics Approval

Study procedures were approved by the Rutgers University institutional review board (IRB; protocol Pro2022000948).

Aim 1: Enhancements to the Existing Version of MSS Using Multilevel Stakeholders


This phase engages multilevel stakeholders in optimizing MSS by targeting multiple RE-AIM outcomes: effectiveness, adoption, implementation, and maintenance. Iterative enhancements will be made to MSS intended to increase the effects on SSE by increasing adoption, acceptability, appropriateness, feasibility, and maintenance. This process will include key informant interviews with survivors, health care delivery stakeholders, and professional organizations to inform the selection of enhancements from the potential theory-based strategies we identified based on the findings and survivor feedback from our previous RCT. This user-centered process will involve assessment of preferences regarding content (eg, “What are your thoughts on the option of uploading pictures of your moles?”), use (eg, “How useful would it be to log in to MSS three months after you complete it and what would you use it for?”), and preferred ways to motivate targeted behaviors (eg, “What are your thoughts about incentives such as free products?”). This phase of our design for dissemination approach will use an iterative process that includes (1) key informant interviews with survivors of melanoma and care providers to inform the selection of proposed theory-based enhancements, (2) conversion to a mobile-based intervention delivery platform incorporating selected enhancements, (3) usability testing with survivors who use the program for 1 month, and (4) program refinement.

Proposed Enhancements

The existing MSS content was guided by PHM and evidence-based BCTs. The original MSS included two categories of BCTs: (1) prompts (participants could set an email reminder to perform SSE on specific dates for the duration of study participation) and (2) planning (participants could set a goal related to SSE and select up to 2 action steps to address barriers). The other material primarily targeted SSE knowledge (eg, how to recognize a suspicious growth) and self-efficacy (building confidence in identifying suspicious growths), which were the mediators of MSS effects on SSE.

Our selection of proposed enhancements was guided by two goals: (1) to optimize engagement with MSS and (2) to increase sustained performance of SSE. A summary of enhancements is shown in Textbox 1. We focused on optimizing engagement as participants who used MSS more were more likely to perform SSE across the follow-up period, which we are now extending. Thus, increasing engagement with MSS should enhance its impact over time.

Textbox 1. Proposed enhancements to mySmartSkin (MSS).

Self-monitoring and performance feedback

  • Upload pictures of moles each month on mole map
  • Illustrate body parts checked each month
  • Bar graph of completed skin self-examinations (SSEs) over time from baseline to 1 year on the MSS home page
  • Monthly feedback on progress with SSE

Incentives for success

  • Incentives for self-reported SSE, core completion, and SSE and sun protection goal setting: rulers, reminder stickers, sun hats, sunscreen, sun protection factor ChapStick, sun safety checklist magnet, calendars, hanging door signs with the ABCDEFs (asymmetry, border irregularity, color variation, diameter of >6 mm, evolving, and funny looking) of melanoma, sun umbrella, among others (up to 6 incentives)


  • Personalized text reminder delivered the day before SSE is due each month to perform it, with a link to the self-check program
  • Tailored reminder of body parts to remember to check

Goals and planning

  • Identification of personal barriers to SSE and sun protection behaviors
  • Continued troubleshooting of ways to address SSE barriers and sun protection behaviors
  • Regular personalized feedback and ability to track progress on SSE and sun protection goals

Social support

  • Discuss the importance of support for change
  • Identify someone to support SSE (eg, spouse) and how the patient would like to talk to and involve them
  • Provide SSE information via email to authorized support person

To optimize initial engagement, we are implementing four enhancements: (1) enhancing participant-facing recruitment materials (eg, advertisements, emails, and study home page) by including narratives from survivors and oncologists about MSS and the importance of thorough SSE, (2) streamlining enrollment materials and processes (eg, screening and baseline surveys and instructions), (3) reaching out to participants who do not log into MSS by day 3 to provide a brief orientation session fostering ease of access (ie, review modules and topics and the importance of completing modules, show the participant how to log in, and discuss SSE benefits for finding recurrence early), and (4) informing participants about incentives for completion of each core and other tasks within the program.

To improve sustained SSE, we identified 6 categories of BCTs for the proposed enhancements [57]; we included (1) self-monitoring and performance feedback to improve self-efficacy (eg, uploading photos of moles and saving them to compare with the next SSE); (2) behavioral incentives (eg, provided for completing SSE); (3) additional prompts for SSE (eg, programmed reminders about missed areas of the body); (4) adding a more comprehensive approach to setting goals and planning, including continued troubleshooting of barriers; (5) including more content on receiving assistance and support from others in performing SSE (including the option to email an authorized support person about assisting with the participant’s SSE); and (6) adding a goal-setting component that fosters habit formation and maintenance of behavior change. We created an in-depth goal-setting section that allows users to track their progress and make updates to their goals and progress over time. The goal-setting component is tailored based on the user’s current SSE and sun protection behaviors, and the purpose of this component is to assist in the creation of a plan to improve both behaviors.

In terms of sun protection behaviors, we included a sun safety core with content categorized into chapters addressing (1) learning more about sun safety with topics such as risks of UV rays, risks of unprotected sun exposure, sunscreen education, how to avoid sunburns, and other sun protection behaviors; (2) assessing current sun protection behaviors; (3) increasing confidence in and motivation for sun protection; and (4) setting sun protection goals. A more detailed summary of the content of each core is provided in Table 1.

Table 1. Enhanced mySmartSkin intervention content.
SectionKey contentInteractive features
  • Overview of navigation, sections, and features
  • N/Aa
Core 1: learn about spots
  • Personal melanoma history
  • Skin cancer facts and figures
  • Melanoma risk factors
  • Risk of recurrence
  • Purpose of SSEb
  • How to conduct an SSE
  • The ABCDEFsc of melanoma
  • Your experience doing SSE
  • What to do when you find a suspicious spot
  • Importance of SSE and physician skin examinations
  • Confidence in checking spots
  • Mole facts and fiction
  • ABCDEF-identifying challenge quiz
  • Suspicious or not mole challenge
  • Vignette and physician video and audio clips
  • SSE goal setting
Core 2: sun-safe behaviors
  • Sun-safe behaviors
  • Risks of sun exposure
  • UV index
  • Types of sunscreen
  • Avoiding sunburns during outdoor activities
  • Risk assessment for outdoor activities
  • Sun-safe clothing
  • Limiting sun exposure
  • Increasing confidence and motivation for sun safety
  • Prioritizing sun-safe behaviors
  • SSE goal setting
  • Sunscreen FAQsd
  • Sunscreen facts and fiction
  • Tanning facts and fiction
  • Assessing current tanning beliefs
  • Vignette and physician video or audio clips
  • Sun-safe behavior importance rating
  • Sun safety goal setting
  • Goal summary
  • SSE progress tracker
  • Badges and prizes
  • Goal progress tracker

aN/A: not applicable.

bSSE: skin self-examination.

cABCDEF: asymmetry, border irregularity, color variation, diameter of >6 mm, evolving, and funny looking.

dFAQ: frequently asked question.

Key Informant Interviews to Gather Input on Enhancements

This phase of aim 1 has been completed. The procedures are described in the following sections, and the results are described later in the paper (see the Results section).


We planned to recruit 10 survivors of melanoma who had completed treatment within up to 3 years since diagnosis from the Rutgers Cancer Institute of New Jersey (CINJ) through new case ascertainment by research staff using electronic medical records. On the basis of our past experience in qualitative assessments, this sample was expected to provide thematic saturation of the data, although additional interviews could be conducted if needed.


Following informed consent procedures, participants completed a brief demographic questionnaire before their interview began. Participants self-reported their age, sex or gender, race, ethnicity, education, current insurance coverage, school enrollment, employment, and marital status. Patients also reported current SSE practices.


The project coordinator (AS) conducted the interviews, which were audio recorded with participant permission and approximately 45 minutes in duration. During the interviews, participants were shown prototypes and wireframe illustrations of the proposed enhancements. To ensure that feedback was gathered on all aspects of the MSS platform, interviewers highlighted different sections, including the goal-setting activity sequence and skin self-check body map. The semistructured interviews elicited feedback on the proposed enhancements. We included probes about enhancements based on stakeholders’ positive, neutral, or negative reactions (eg, “What parts of the site did you think were most interesting and helpful?” and “What parts were less helpful or seemed less relevant to you?”). Phase 1 participants received US $50 for the interviews. These interviews were conducted in person at the CINJ or using videoconferencing software. The interviewer took field notes, and the interviews were transcribed verbatim.

Refining Enhancements

The study team reviewed the field notes and verbatim transcriptions of key informant interviews. The responses to each proposed enhancement were coded as positive, neutral, negative, or mixed. Although potential enhancements were based on findings from the previous RCT, suggestions from participants, BCTs, and proven ways to optimize engagement, if particular enhancements were perceived negatively by multiple stakeholders, we used these interviews to explore alternatives that would be more acceptable. For those enhancements with multiple negative or mixed comments, the team decided whether the issue could be addressed feasibly and crafted changes to the content and approach that were congruent with stakeholder feedback. The stakeholders in later interviews reviewed the changes made in response to feedback given in earlier interviews to provide confirmation on whether the issue was adequately addressed. The study team provided feedback to the web developer, Radiant (Radiant Creative Group LLC), throughout the interview process for changes to be incorporated in a timely manner.

Conversion to a Mobile-Based MSS Platform, Usability Testing, and Refinements With Multilevel Stakeholders

Stakeholder-informed enhancements to content and program features will be incorporated with the existing content and migrated to the mobile-based delivery platform in collaboration with Radiant. Testing will be conducted throughout the preproduction and production stages to ensure the suitability of the content for the intended audience. Weekly meetings with Radiant will be conducted to review uploaded content.

Usability Testing, Feedback, and Refinements

Testing will be accomplished by conducting focused interviews with up to 10 survivors of melanoma, 5 health care providers, and 5 organization representatives who will each receive US $100. Testing is modeled after a National Cancer Institute website design project [58]. The process will involve checking the content with participants for attractiveness, comprehension, acceptability, and persuasion. In addition, participants will be able to use MSS for approximately 1 week before the interview so that we can gather input on features that are planned over time, such as self-monitoring and performance feedback, incentives, and prompts. A series of questions and structured guides will be prepared in advance related to the elements to be evaluated.

In the set of interviews, all enhancements and engagement tools, including reminder prompts, will be activated, providing an opportunity to evaluate all components of the enhanced MSS. After that, we will set up a virtual meeting in which the participant will provide feedback about acceptability, ease of use or intuitiveness, and satisfaction. During this session, we will discuss the topic areas and review MSS using a shared screen to allow participants to react to each screen. Participants will be asked to provide feedback on features. These sessions will be digitally video recorded and transcribed, and AS will take notes. The data will be summarized and reviewed to determine further modifications to be made. These efforts will yield a set of understandable, acceptable, and appealing enhancements to MSS that are customized to the needs, capabilities, and preferences of patients with melanoma, providers, and organizational stakeholders. We will conduct weekly meetings that include Radiant and the study team to review and produce content. Once the final beta website has been agreed upon by the full research team, it will undergo extensive testing by Radiant to ensure that the programming is functioning correctly.

Aim 2: Evaluate Effects of MSS Versus a Noninteractive Educational Web Page on SSE


Phase 2 is a randomized effectiveness trial in which MSS will be tested in an RCT comprising 2 groups (MSS vs a noninteractive educational web page) and 5 assessments (baseline, 3 months, 6 months, 1 year, and 18 months) with 300 survivors of melanoma. The main goal is to evaluate the effects of MSS on thorough SSE and examine the impact on clinical outcomes (eg, diagnosed early-stage skin cancers).


We will recruit participants from 2 sources: state cancer registries and Facebook. This approach was adopted to prepare for dissemination by assisting the team in comparing the reach of the 2 methods of recruitment. This will foster our ability to determine future settings. The registries include all patients with melanoma in each state, which facilitates eventual generalizability. We chose NJ and CA registries because the 2 states are geographically disparate and sociodemographically diverse, and we have worked successfully with both registries on current projects (CA2219854 and CA221854). We chose Facebook as a second approach as the vast majority of Americans (84% in 2021) [59] use social media, with use steadily increasing each year. This strategy will reach a broad audience of survivors. In addition, many melanoma and skin cancer advocacy groups are active on social media, which will inform the potential for working with them on future dissemination.


We will recruit from the New Jersey State Cancer Registry (NJSCR) and the Cancer Registry of Greater California (CRGC). Owing to state laws, allowable recruitment procedures differ. NJSCR procedures entail case ascertainment, a letter to the physician, an address check, a letter mailed to the patient, up to 12 calls to verify that they received the letter, and the provision of verbal consent to send contact information to the CINJ. The CINJ reaches out to the patient to describe the study and send a link to the MSS website. We have enhanced the CRGC procedures to be as similar as possible to those of the NJSCR. The CRGC queries their database, confirms that the patient is alive, and checks this information with the provider. Case information is sent to the CINJ, a letter is sent explaining the study followed by up to 12 calls to the patient to obtain their email address, and a link to the study web page is emailed. The process takes approximately 2 months.


We will use paid Facebook advertisements. We will work with a social media marketing company to develop keywords and advertisements targeting individuals with characteristics similar to those of survivors of melanoma (eg, older age). As noted previously, this approach has been successfully used in 2 previous and one current skin cancer–related intervention project [60-62]. This approach will also recruit a broader population of survivors of melanoma than the previous efficacy trial.


The eligibility for this trial is as follows:

  1. Diagnosis of primary pathological stage-0 to stage-III cutaneous malignant melanoma
  2. Being 3 months to 5 years after surgery
  3. No current evidence of cancer
  4. No adherence to thorough SSE [14] (ie, did not check the entire body at least once during the past 3 months)
  5. Age of ≥18 years
  6. Internet access
  7. Ability to speak and read English
  8. Ability to provide informed consent

The incidence is most common among non-Hispanic White individuals, who have an annual rate of 28 cases per 100,000 compared with 7 in American Indian or Alaska Native individuals; 5 in Hispanic individuals; and 1 in non-Hispanic Black, Asian, and Pacific Islander individuals (92.4% among non-Hispanic White individuals). Consistent with population rates, we will oversample minority groups from cancer registries to ensure a minimum of 7.6% racial and ethnic minority group survivor enrollment.


The projected sample size is 300. Participants will be recruited using 2 methods. Half of the sample will be recruited from 2 state cancer registries. We expect >10,000 eligible patients with melanoma from the 2 registries during our recruitment period. We will select a random sample from each diagnosis year to balance the year of diagnosis. The acceptance rate for the previous MSS study was 40.9%, but we anticipate a lower recruitment from the registry participants in this study based on our recent work with both CRGC and NJSCR patients (CA221854) [63]. Both cancer registries have a large number of survivors annually. Thus, we will randomly select a subset of cases from each of the retrospective 5-year time frames. The other half of the sample will be recruited via Facebook. We will use paid Facebook advertisements targeted at persons who follow skin cancer–related causes and organizations. We will end recruitment from each source when the required sample size is reached.

Enrollment, Randomization, Intervention Delivery, and Follow-Up Surveys

Potential participants recruited via registries will be asked for their email when they verbally consent to participate in the study. Staff will then email these potential participants the link to the study’s web page. Participants recruited via social media advertisements will be automatically linked to the study web page when they click on the “learn more” part of the advertisement. Once on the web page, the following automated, completely web-based procedures will be used for enrollment. On the study’s web page, there is a screening survey to determine eligibility. Potential participants will complete the eligibility screener. If eligible, the program automatically moves to the electronic consent. Once consent is obtained, the program automatically moves to the baseline survey. After survey completion, the program randomizes the participant to MSS or the comparison condition and automatically moves to either the MSS landing page or the comparison condition landing page. Randomization will be stratified according to disease stage and time since diagnosis (3 months-3 years and 4-5 years) and, then, within strata, block randomized using blocks of 20 participants overseen by the study statistician following standard operating procedures of the CINJ Biostatistics Shared Resource.

Participants who do not log into the MSS program by day 3 will be contacted by a study coordinator via SMS text message, email, or phone 3 times between days 3 and 10 to remind them about logging into MSS. Participants who log in but do not review core 1 by day 5 will be contacted 3 times via email, SMS text message, or calls to invite them to review core 1. These interactions will also serve as an orientation for the participants allowing them to ask any questions about the study or the intervention.

Survey Procedures

Participants will be prompted (via automated email, as well as via SMS text message, telephone, and mail, as necessary) to complete the following web-based surveys: follow-up 1 at 3 months after the baseline, follow-up 2 at 6 months after the baseline, follow-up 3 at 1 year after the baseline, and follow-up 4 at 18 months after the baseline. Participants who do not complete the surveys will be reminded as follows: (1) reminder email and SMS text message with a link to the survey if it is not returned after 1 week, (2) call with a voicemail if no contact is made and email with a link to the survey if it is not returned after 2 weeks, and (3) the same procedure as at 2 weeks if the survey is not returned after 3 weeks. Participants who do not return the survey after a month will be considered lost to follow-up at that time point. Our primary effectiveness analyses will focus on individuals’ performance of thorough SSE at 18 months, which will ensure that all participants’ SSE practices are assessed over a long time frame and increase the likelihood of detecting new or recurrent melanomas. Participants will receive incremental gift card amounts for each survey (maximum total of US $120).

Intervention Procedures

Participants will have a unique identifying user code and create a password to access the interventions. If users forget their password, they will provide a unique answer to a personal question and the database will return the forgotten password. If users forget both their user ID and password, they can contact the technical assistance hotline. In the previous study, navigation issues were rare. We anticipate that it will take participants approximately 1 hour to complete the entire MSS intervention once and approximately 15 minutes to complete the comparison condition. Participants will have access to the interventions for 18 months.

Comparison Group

We chose a noninteractive educational web page comparison group for the following reasons: (1) the information incorporated is widely available to the public and used in dermatology and primary care, (2) the digital delivery method is similar to that of MSS, (3) it allows for a comparison of MSS effects with a minimal knowledge-based educational intervention, (4) it permits a comparison of MSS effects with the results of previous studies that used minimal comparison interventions, (5) it presents a standardized intervention rather than a variable treatment-as-usual intervention, (6) a no-intervention control would be unethical for this high-risk population, and (7) we considered using an intervention for another health issue (eg, nutrition) but anticipated that survivors of melanoma would be less likely to participate. We will create a static educational web page whose content will be based on freely available, accurate information relevant to melanoma, melanoma risk, and SSE. Possible topics include the warning signs of melanoma [64], the risk of recurrence among survivors of melanoma, and how to perform an SSE [65].

Aim 2 Measures

The demographic information collected will include age, sex, race, ethnicity, education, marital status, state of residence, employment, income, and insurance.

Clinical Characteristics

The clinical characteristics collected will be date of diagnosis, stage, and treatment received.

Melanoma Risk

The melanoma risk information collected will be eye color, natural hair and skin color, skin reactivity to the sun, freckling, moles, sunburn history, indoor UV tanning, and family history.

Environmental Factors

The environmental factors collected will be month of assessment, residential location, and average UV index at solar noon over the 3 months before each time point.

Primary Outcome Measure: Skin Self-Check Comprehensiveness

A full list of outcome measures and covariates is shown in Table 2. The primary outcome consists of the following items: (1) “Have you, or anyone else, not including a doctor or other healthcare professional, ever checked any part of your body for signs of skin cancer?” (yes or no), (2) “If yes, in the last three months, did you check your body for early signs of skin cancer?” (yes or no), and (3) “Doing a thorough skin check or skin self-examination means spending time looking at the skin systematically and deliberately. Please indicate whether you thoroughly checked each of the following areas of your body the last time you checked your body for early signs of skin cancer: Scalp, face, neck, shoulders, front of arms, back of arms, chest, stomach, upper back, lower back, front of legs, back of legs, bottom of feet, buttocks, and genitals” (yes or no for each). For all time points, comprehensiveness will be based on the total number of body parts checked among participants who conducted an SSE in the last 3 months. Participants who did not conduct an SSE in the last 3 months will be coded as 0 for this outcome.

Table 2. Aim 2 measures.
MeasureBaselineTime 2Time 3Time 4Time 5

Date of birth


Marital status

Race and ethnicity


Insurance status

Employment status

Time since diagnosis

Month of assessment

Residential location

Average UV index at solar noon

Skin self-examination

Sun protection behaviors

Melanoma diagnosis, stage, thickness, and date

Clinician confirmation of cancer
Secondary Outcomes
Sun Protection Behaviors

Participants report how often they engage in 4 behaviors when outside on a sunny day: wearing sunscreen with a sun protection factor of ≥30, wearing a long-sleeved shirt, wearing a wide-brimmed hat, and staying in the shade [66]. Items are rated on a 5-point Likert scale (1=never; 5=always).

Clinical Outcomes

Participants report whether a new melanoma or other skin cancer was diagnosed and the stage, thickness, and date. Physician confirmation of this information will be collected.

Analyses of the Impact of MSS on SSE, Sun Protection, and Clinical Outcomes
Analyses of Thorough SSE and Sun Protection

Demographics and baseline variables will be compared between the 2 conditions using chi-square tests and ANOVAs for categorical and continuous variables, respectively. These variables, including outcomes, will also be compared by recruitment source. Missing data will be handled using multiple imputation with 50 imputed samples [67]. Tests of intervention effects on thorough SSE and sun protection will be conducted separately for each wave of postintervention data collection using logistic regression models that treat baseline SSE or sun protection, respectively, as a covariate. Given the large number of potential additional demographic and medical covariates, additional analyses will include the covariates that are significant predictors of the outcome.

Sample Size and Power

On the basis of our previous study, we expect follow-up survey completion rates between 80% and 93% at the 1-year time point and 80% to 85% at the 18-month time point, which will result in an expected final sample size of N=255. In the previous trial, thorough SSE occurred at a rate of approximately 30% in MSS at each time point, and effect sizes for conducting a thorough SSE ranged from w=0.29 (8 weeks) to w=0.21 (6 months). Given that the MSS intervention will be enhanced, we expect that the SSE rate will be higher (45%), which will result in approximately w=0.36. A sensitivity power analysis using G*Power indicated that a sample of N=255 will be able to detect an even smaller effect of w=0.22 with a 2-tailed test, Cronbach α of .05, and power of 0.95. Thus, the expected effect of w=0.36 with a sample size of N=255 will be detectable with a power of 0.99. Given that sun protection behavior is a secondary outcome, we are not powering the trial for this variable.

Analyses of the Impact of MSS on the Diagnosis of New or Recurrent Melanoma

The impact of MSS would be strengthened with data suggesting that survivors who use MSS detect suspicious growths and bring them to physicians for appropriate workup and diagnosis. Given the likely low recurrence rate over the 18 months that participants are followed for in this trial, the study is not powered for testing hypotheses on this end point. Rather, we will conduct an exploratory analysis to examine whether diagnoses differ between MSS and UC. Specifically, the number of diagnoses of earlier-stage new and recurrent melanoma at 18 months will be compared for the 2 conditions using either negative binomial regression models or binary logistic models depending on the distribution of new diagnoses (ie, if there are patients with more than one new diagnosis during the period). If our findings show that more melanomas are detected in MSS, we will conduct exploratory cost-effectiveness analyses from the health care and societal perspectives by creating simulation models of melanoma-related costs, disease progression, and survival over 5- and 10-year analysis horizons from the start of the trial (aim 3a).

Aim 3: Implementation Outcomes and Contextual Factors Relevant to Future Scale-Up

The type-1 hybrid effectiveness-implementation approach allows us to obtain vital data to inform future scale-up, dissemination, and implementation. This aim has 3 parts. In aim 3a, we will assess the costs of delivery of MSS and the control condition website; if indicated by the results of aim 2, we will conduct cost-effectiveness analyses. In aim 3b, guided by RE-AIM, we will evaluate the implementation outcomes of MSS from the aim 2 effectiveness trial. In aim 3c, we will elicit actionable feedback on contextual factors relevant to future scale-up, dissemination, implementation, and maintenance from multilevel stakeholders, including survivors, providers, and representatives of organizations invested in melanoma survivorship care. Quantitative and qualitative data from aims 3a, 3b, and 3c will be analyzed and interpreted together to provide valuable insights and directions regarding implementation strategies needed to inform the scale-up and widespread implementation, focusing particularly on reach, adoption, implementation, and maintenance. The results of aim 3 as a whole will answer the following key questions: how can we increase the number of MSS users to maximize adoption? What dissemination channels are preferred to increase our reach to survivors? How can MSS be maintained to provide a sustainable intervention for survivors? What resources or adaptations are needed for organizations and practices to integrate MSS into their survivorship care offerings?

Aim 3a: Costs and Cost-Effectiveness
Assessment of Program Costs

The research team will track the costs of delivering both MSS and the noninteractive educational web page. Costs will first be estimated from a program perspective, capturing the explicit resources required to deliver and maintain the program after all start-up costs (eg, development and programming costs) have been incurred. Costs that are needed to deliver the intervention will be collected to inform the cost of scale-up, but any research-related costs that would not be required for standard program implementation will also be tracked. Sunk costs, which are one-time upfront costs, will be reported separately from ongoing program implementation costs. Program cost data will be collected using a modified version of previous cost surveys developed by the research team and adapted for several interventions [68-73]. The cost surveys capture all relevant labor- and non–labor-related inputs necessary to quantify costs. Labor costs for program staff time (via questionnaires) will be valued using actual or estimated wages. Nonlabor costs will be obtained from program billing records; these include materials and supplies used to support program and intervention activities and costs of facilities and contracted services. The feasibility of including the value of participant time spent on the intervention will also be explored, where participants’ time will be valued using age- and sex-specific wage rates. In addition, if the results of the analysis in aim 2 indicate that more melanomas are found among MSS participants than among the educational web page arm participants, costs to the health care sector will be estimated. These include the costs of diagnosing and treating recurrent melanoma. Published estimates of stage-specific medical costs of melanoma will be used to calculate the treatment costs of melanoma by stage at detection [74-79]. To estimate costs from a societal perspective, the costs to each stakeholder group will be summed, including program implementation costs, costs to the health care sector, and the value of participants’ time.

Cost Analysis

Total and incremental program costs will be estimated for each arm. Program development costs, a one-time capital investment, will be calculated separately from ongoing implementation costs, or the costs to maintain program delivery for the duration of the intervention. Analyses will distinguish between fixed costs, those that do not vary with enrollment rates (eg, server maintenance), and variable costs, which increase for each participant added. As most costs for MSS are expected to be fixed, the mean cost per participant will be driven largely by the number of enrolled participants. To explore the costs of scale-up, sensitivity analyses will be conducted regarding additional program dissemination efforts and variations in uptake rates. The primary cost-effectiveness analysis will evaluate the additional costs per thorough SSE completed for MSS versus educational web page participants over the 18-month trial. This analysis will be conducted from the program, health care sector (including program costs and costs of health care use for lesions detected), and societal (ie, health care sector plus participant costs) perspectives. The incremental cost-effectiveness of MSS relative to the educational web page will be determined as the difference in mean costs between the 2 arms divided by the difference in mean outcome changes [80,81].

Simulation Models of Melanoma-Related Costs, Disease Progression, and Survival

If the findings show that more melanomas are detected in early stages for the MSS arm relative to the educational web page arm as expected, secondary or exploratory cost-effectiveness analyses will be conducted from the health care sector and societal perspectives by creating statistical simulation models of melanoma-related costs and disease progression over 5- and 10-year analysis horizons from the start of the trial. Assuming the equivalent likelihood of a melanoma recurrence for MSS and educational web page participants, trial data on stage at diagnosis for melanomas detected during the trial (from aim 2) and published estimates of average stage at diagnosis for survivor recurrences in the general population will be used. Melanoma treatment costs beyond the trial period will be simulated for both arms, where the hypothesis is that MSS participants will have lower treatment costs than those in the educational web page arm as earlier detection requires less aggressive treatment. Mortality will also be simulated for both arms to create an effectiveness measure of simulated life years gained (LYGs) for MSS versus the educational web page comparison group. To estimate LYGs, reductions in melanoma mortality for MSS versus the comparison group will be combined with life expectancy data [82]. As health-related quality of life among survivors of melanoma is similar to health-related quality of life in the general population, there will be no quality adjustment of LYGs as any differences across arms in quality-adjusted life years would be driven by differences in mortality [83-88]. The cost-effectiveness analysis will examine the total incremental costs of MSS compared with the educational web page arm. Incremental costs include program implementation costs (excluding sunk and research costs), health care use costs incurred during the trial and any melanoma treatment costs simulated to occur beyond the trial, and the value of patients’ time to participate in the intervention. Incremental effectiveness will be calculated as LYGs for MSS compared with the educational web page arm. Base analyses will discount future costs and life years and use bootstrapping methods to generate 95% CIs for the incremental cost-effectiveness ratio. Sensitivity analyses will explore factors such as rates of detection of thin lesions through SSE, which may make MSS more cost-effective than the educational web page comparator.

Aims 3b and 3c: Exploration of Implementation Outcomes and Contextual Factors Relevant for Future Scale-Up
MSS Participants

All participants in the intervention trial assigned to MSS will complete surveys assessing specific implementation outcomes. A subset of MSS participants (n=30) will be purposively selected to participate in key informant interviews to reflect a diversity of MSS outcomes, MSS use, ratings of acceptability, and sociodemographic and clinical characteristics.

Additional Stakeholders

To ensure that we obtain a breadth of perspectives on the potential for scale-up, dissemination, widespread implementation, and maintenance of MSS, we will also identify and invite 20 individuals from multiple groups involved in melanoma treatment and survivorship to participate in key informant interviews. These stakeholders will include additional health care providers and representatives from the same organizations participating in aim 1. Participants in aim 3 key informant interviews will receive US $50 for completing the interviews, which will be conducted in person or via videoconferencing software.

Procedures, Measures, and Analyses

Aim 2 addresses the effectiveness of MSS. Aim 3a addresses the costs associated with delivering MSS, an important factor in the future potential scale-up of the intervention. In aims 3b and 3c, we use mixed methods to focus on the remaining RE-AIM outcomes and PRISM domains using data from the aim 2 RCT combined with surveys and stakeholder interviews.

Quantitative Implementation Outcomes and Analyses

As this is a type-1 hybrid effectiveness-implementation study, our analyses of implementation outcomes are primarily descriptive and meant to inform future planning for scale-up, dissemination, and implementation. Analyses of Reach will describe the proportion of contacted individuals who express interest by completing an eligibility survey. The clinical and sociodemographic characteristics of potential participants will be described and compared with those of the population of survivors of melanoma. Analysis of Adoption will describe the proportion of eligible participants who enter the intervention website (Table 3). Analyses of Implementation will describe the indicators listed in Table 3 and examine their variability by participant clinical characteristics, sociodemographic characteristics, and melanoma risk factors assessed in aim 2. We will also examine the variability in indicators of implementation by recruitment source (registry vs social media advertisements). The Acceptability of Intervention Measure (AIM) is a brief, quantitative measure of intervention acceptability that has demonstrated content and structural validity and test-retest reliability [89] as well as face validity [90]. Participants respond to 4 statements using 5-point Likert-type scales, with higher scores indicating higher acceptability. The AIM will be completed by MSS arm participants at the first follow-up. Acceptability ratings will be summarized using descriptive statistics and compared across clinical, sociodemographic, and risk factor groups.

Table 3. Aim 3 implementation measures.
OutcomeDefinitionMeasurementData source
  • Proportion and representativeness of individuals who express interest in MSSa
  • Proportion of contacted participants who complete an eligibility survey
  • Sociodemographic and medical characteristics of individuals who screen vs the population of survivors
  • Recruitment and enrollment data
  • National survivor data set (eg, NHISb and HINTSc)
  • Proportion of individuals who begin an intervention
  • Proportion of contacted participants who consent, complete baseline survey, and enter the intervention website
  • MSS automated data
  • How much individuals use MSS
  • Number of core modules completed
  • SSEsd performed with self-check program
  • Number of views of MSS
  • Use of each enhancement (incentives)
  • MSS automated data
Program costs and cost-effectiveness
  • Total and incremental costs of MSS vs comparison arm; cost-effectiveness of MSS
  • Surveys and billing and invoicing documents
  • Staff, service provider, and participant surveys; billing and invoicing, including payments; and published literature
  • Perception that MSS is feasible for use, dissemination, and delivery
  • Qualitative key informant interviews
  • Health care providers and professional organizations
  • Perception that MSS is satisfactory
  • Acceptability of Intervention Measure
  • Qualitative key informant interviews
  • Participant surveys
  • Survivors, providers, and organizations
  • Perception that MSS is an appropriate fit for intended use
  • Qualitative key informant interviews
  • Survivors, health care providers, and organizations
  • Perceptions of the likelihood, needs, and resources for maintaining MSS delivery
  • Qualitative key informant interviews
  • Health care providers and organizations
PRISMe contextual factors
  • Perceived contextual factors related to scale-up and D&If
  • Qualitative key informant interviews
  • Survivors, health care providers, and organizations

aMSS: mySmartSkin.

bNHIS: National Health Interview Survey.

cHINTS: Health Information National Trends Survey.

dSSE: skin self-examination.

ePRISM: Practical, Robust Implementation and Sustainability Model.

fD&I: dissemination and implementation.

Qualitative Key Informant Interviews

Interviews will be conducted with stakeholders from each group (ie, participants, providers, and organizations) to assess the perceptions of potential barriers and facilitators to consider for future scale-up, widespread implementation, and maintenance. MSS participant interviews will be conducted within the month following their final follow-up during the aim 2 trial. Interviews with other stakeholders will be conducted following completion of the effectiveness trial during the final study year. Semistructured interview guides will include open-ended questions and probes regarding [91] implementation outcomes of feasibility, acceptability, and appropriateness of widespread implementation of MSS for survivors of melanoma. The potential for MSS maintenance will also be explored using open-ended questions and probes, including resources available and needed for the sustainability of the intervention from multiple stakeholder perspectives. In addition, we plan to focus on three PRISM domains: (1) organizational, provider, and patient perspectives on the intervention; (2) characteristics of the recipients of the intervention (survivors); and (3) implementation and sustainability infrastructure (if MSS were to be disseminated and supported by specific organizations outside the research context). To elicit perspectives on the 3 PRISM domains of interest to scale-up, dissemination, implementation, and maintenance, interview questions (as appropriate for each stakeholder group) will focus on (1) current approaches and practices relevant to the promotion of SSE; (2) the benefits and shortcomings of the MSS intervention; (3) barriers to MSS use (for survivors and those who may suggest it as a resource); (4) possible strategies to disseminate MSS to survivors; (5) pros and cons of suggested dissemination channels and strategies; (6) the “fit” of MSS with the missions and objectives of providers, practices, and organizations; and (7) resources, infrastructure, and other factors needed to sustain MSS.

Qualitative Analyses of Implementation Outcomes

We will follow the guidelines established by the COREQ (Consolidated Criteria for Reporting Qualitative Research) for data collection, analysis, and reporting. The audio recordings of the interviews and focus groups will be transcribed verbatim, deidentified, and imported into the ATLAS.ti software (ATLAS.ti Scientific Software Development GmbH) for analysis. Directed content analysis will be used. This structured method allows researchers to specify constructs of interest a priori (eg, acceptability, feasibility, appropriateness, and PRISM domains) and obtain detailed descriptions or elaborations of them, classifying and coding themes within and across constructs [92]. The study investigators will develop a preliminary codebook for each stakeholder group. A primary and secondary coder will independently code and discuss an initial subset of interviews to explore thematic content and then merge and explore them for concordant and discordant coding. After refining the codebooks, they will recode the first set of transcripts and determine the interrater reliability. After achieving consistency, the coding process will continue. In total, 4 randomly selected transcripts will be identified for double coding and evaluation of discordant coding, followed by any needed modifications and recoding to achieve consensus. Once the initial coding has been completed, 10% of the sample (ie, 1-2 participants from each stakeholder group) will be randomly selected and invited to participate in a member-checking process to determine whether additional data collection is necessary and ensure that valid inferences are made through coding procedures [93]. Following any further corrections, the team will develop a summative grid of emergent themes across and within stakeholder groups.

Synthesis of Quantitative and Qualitative Data to Inform Scale-Up

Mixed methods data will be analyzed and integrated using a concurrent parallel design [94]. As described previously, each type of data (quantitative and qualitative) will initially be collected and analyzed separately. A mapping and matrix approach will facilitate the integration and interpretation of the results, and joint displays will present the integrated results, connecting qualitative themes with quantitative outcomes. The results will be summarized within and across stakeholder groups to allow for consideration of multilevel dissemination and implementation strategies needed for future scale-up, dissemination, implementation, and maintenance.

Data Safety and Monitoring

Before initiating this study, the Rutgers Scientific Review Board will review the study procedures to ensure their scientific merit, safety, legality, and technical feasibility per the established policy. The proposed procedures will then be reviewed for protection against risks by the Rutgers health sciences IRB. Adverse events will be reported through the Rutgers CINJ for processing as per the established policy. This policy includes specific timelines for reporting events that are stipulated by the IRB. In addition to the IRB, the Rutgers CINJ Protocol Monitoring Committee reviews all adverse events for investigator-initiated trials as they occur. The project coordinator, under the direct supervision of the principal investigator, will be responsible for reporting any adverse events that are documented on the safety or adverse events form or reported by the study interventionist. The Rutgers CINJ Protocol Monitoring Committee will oversee the validity and integrity of the data by conducting periodic audits of the study records. The committee is empowered to suspend or close studies with major deficiencies and provides direction to investigators in the development of corrective action plans to rectify and meet identified deficiencies. As part of the Protocol Monitoring Committee function, accrual is monitored for clinical trials. All clinical trials undergo a semiannual review by the Protocol Monitoring Committee, at which time accrual figures are reviewed. Specific accrual rates for each trial are required under protocol monitoring policy. The Rutgers CINJ uses an internal audit program to address retention of participants, adherence to protocol, and data completeness. This audit program is reviewed and governed by the Protocol Monitoring Committee. In addition, there is a Data Safety and Monitoring Advisory Board. This team will consist of a dermatologist, a psychologist, and a survivor of melanoma from the community. The board, along with the study principal investigators, will convene for an in-person or virtual meeting annually. The agenda of the annual meeting will be to review risk procedures, adverse event reporting, and quality assurance. The Data Safety and Monitoring Advisory Board will review any serious adverse events reported as well as investigator adherence to eligibility rules.

To date, we have completed the first phase of aim 1; the remaining phases of this aim are currently in progress. The prototype and content for the MSS intervention were developed in collaboration with Radiant Inc. Following the procedures described in the Methods section, one set of stakeholder interviews was conducted with 5 survivors of melanoma who were shown the SSE body map component. A second set of stakeholder interviews was conducted with another sample of 5 survivors of melanoma who were shown the goal-setting module. Participants were shown the prototypes and wireframe. The responses were summarized in field notes, and we reviewed the notes and verbatim transcriptions of key informant interviews. Each proposed enhancement was coded as supportive, neutral, negative, or mixed. For those enhancements that had multiple negative or mixed comments, the team decided whether the issue could be addressed and crafted changes to the content and approach that were congruent with stakeholder feedback.

For the 2 sets of interviews, 19 patients were approached. Of these 19 patients, 10 (53%) were enrolled and 9 (47%) refused. The sample consisted of 50% (5/10) women and 50% (5/10) men, ranging in age from 32 to 75 (mean 57, SD 14.55) years. They were primarily non-Hispanic White (9/10, 90%) individuals, with 10% (1/10) of the participants identifying as non-Hispanic Black. Most (9/10, 90%) were employed, and 50% (5/10) had a college degree or higher education. In total, 40% (4/10) had conducted a partial SSE in the last year, another 30% (3/10) had conducted a comprehensive SSE in the last year, and 30% (3/10) had not conducted an SSE in the last year. A summary of the comments regarding design and navigation suggestions is provided in Multimedia Appendix 1. A summary of comments about content is shown in Table 4.

The team provided the recommended changes based on the feedback to the web developer. The interviewees in later interviews reviewed the changes made in response to feedback given in earlier interviews to provide confirmation on whether the issue was adequately addressed.

Table 4. Feedback regarding mySmartSkin (MSS) content and coding for phase 1.
FeedbackCodingTeam decision
Mixed reviews on the incentive optionsMixedChanged incentive options based on feedback
MSS would be very helpful for survivorship carePositivea
Wish MSS was available for public usePositive
Content is thorough but quick to get through, which will motivate patients to use the appPositive
Helps user feel in control of their healthPositive
Content seems overwhelmingMixedAs some users felt that the content was manageable whereas others felt that it was overwhelming, we added a time estimate for each chapter so users will know how long it will take to complete so they can break the content into manageable sessions based on their own schedule.
Make the app more funNegativeIncluded gamification aspects such as badges and prizes that users can win
The importance of SSEb and MSS comes through in the contentPositive
Content is straightforward, direct, and concisePositive
The app is visually pleasingPositive
User likes the ability to make the text biggerPositive
User likes the use of statistics to reinforce the importance of SSEPositive
Want to share app with family and friendsPositive

aNo decision made as feedback as positive.

bSSE: skin self-examination.

Principal Findings

Once the analyses are completed, we anticipate that MSS participants will be more likely to perform thorough SSE and sun safety behaviors over the 18-month follow-up period and propose that there will be more earlier-stage melanomas diagnosed in MSS than in UC. In terms of the cost analysis, we expect that MSS will be a more cost-effective strategy given its greater effectiveness in increasing SSE and identifying new or recurrent melanoma. For the reach outcome, we predict that the demographic variables of survivors exposed to MSS will not differ from those of the general population of survivors of melanoma. For adoption, we propose that the proportion of contacted or eligible survivors randomized to MSS who consent, complete the baseline survey, and log into MSS will be equal to or greater than the proportion who adopted the intervention in our previous efficacy trial. For engagement, we propose that 80% of MSS participants will log into the intervention site at least once. For acceptability, we predict that MSS will be rated as highly acceptable, with mean acceptability ratings of ≥4 (out of 5) on the AIM.

Strengths, Limitations, and Unanticipated Problems

The study’s strengths are the focus on both effectiveness and implementation through the type-1 hybrid approach; multilevel stakeholder engagement throughout the trial; a novel integration of RE-AIM, PRISM, and health behavior frameworks; the enhancement of a promising fully automated intervention; the scalability of this mobile or web-based intervention; the cost analysis; the longitudinal study design; and the inclusion of clinical outcomes. There are few interventions that have been evaluated to improve SSE among survivors of melanoma. No published intervention is fully automated, which represents a potentially cost-effective and scalable intervention delivery method. This study will advance the science of cancer survivorship by optimizing a promising intervention for an underserved group of survivors and preparing for widespread dissemination and implementation. Few studies have focused on implementation science aspects of consumer-facing digital interventions. Our use of a type-1 hybrid effectiveness-implementation trial will inform not only our understanding of the effects and implementation of MSS but also our understanding of future large-scale dissemination of self-administered web-based interventions. By evaluating and modeling the clinical outcomes, cost, and cost-effectiveness of MSS, our work will provide information about the costs associated with possible implementation and value.

There are at least 2 limitations. First, some individuals will not use MSS despite its high dissemination. Second, SSE outcomes are measured through self-report. However, this weakness is mitigated by the fact that self-report measures have excellent reliability and validity [14,66,95-100] and self-report has been recommended as the most appropriate assessment approach for wide-scale skin cancer risk reduction research [101].

Future Directions

If effective, MSS could be disseminated and delivered via dermatologist practices, public health organizations such as the American Cancer Society, and nonprofit organizations focused on melanoma or in partnership with existing social media channels. Future research should evaluate potential dissemination and implementation strategies to reach survivors building upon what we will learn in this study about the contextual factors that may impede or promote the future reach, adoption, implementation, and maintenance of MSS.


This project was supported by the National Cancer Institute under R01CA265548 awarded to SM and CJH. Services, results, or products in support of the research project were generated by the Rutgers Cancer Institute of New Jersey Cancer Prevention and Outcomes Data Support Shared Resource. The authors acknowledge the following individuals for their valuable contributions to this project: Carolina Lozada, Lisa Paddock, and Radiant Inc.

Data Availability

The data sets generated and analyzed during this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Peer review report from the Science of Implementation in Health and Healthcare Study Section Healthcare Delivery and Methodologies Integrated Review Group Center For Scientific Review (SIHH, USA).

DOCX File , 17 KB

  1. Survival rates for melanoma skin cancer. American Cancer Society. URL: https:/​/www.​​cancer/​melanoma-skin-cancer/​detection-diagnosis-staging/​survival-rates-for-melanoma-skin-cancer-by-stage.​html#references [accessed 2023-08-30]
  2. Kao SY, Ekwueme DU, Holman DM, Rim SH, Thomas CC, Saraiya M. Economic burden of skin cancer treatment in the USA: an analysis of the Medical Expenditure Panel Survey Data, 2012-2018. Cancer Causes Control. Mar 30, 2023;34(3):205-212. [CrossRef] [Medline]
  3. Francken AB, Accortt NA, Shaw HM, Colman MH, Wiener M, Soong SJ, et al. Follow-up schedules after treatment for malignant melanoma. Br J Surg. Nov 2008;95(11):1401-1407. [CrossRef] [Medline]
  4. Francken AB, Bastiaannet E, Hoekstra HJ. Follow-up in patients with localised primary cutaneous melanoma. Lancet Oncol. Aug 2005;6(8):608-621. [CrossRef] [Medline]
  5. Leiter U, Buettner PG, Eigentler TK, Bröcker EB, Voit C, Gollnick H, et al. Hazard rates for recurrent and secondary cutaneous melanoma: an analysis of 33,384 patients in the German Central Malignant Melanoma Registry. J Am Acad Dermatol. Jan 2012;66(1):37-45. [CrossRef] [Medline]
  6. Christianson DF, Anderson CM. Close monitoring and lifetime follow-up is optimal for patients with a history of melanoma. Semin Oncol. Jun 2003;30(3):369-374. [CrossRef] [Medline]
  7. NCCN clinical practice guidelines in oncology: melanoma; version 4.2011. National Comprehensive Cancer Network. URL: [accessed 2024-01-25]
  8. Huang CL, Provost N, Marghoob AA, Kopf AW, Levin L, Bart RS. Laboratory tests and imaging studies in patients with cutaneous malignant melanoma. J Am Acad Dermatol. Sep 1998;39(3):451-463. [CrossRef] [Medline]
  9. Lifchez SD, Kelamis JA. Melanoma: workup and surveillance. Clin Plast Surg. Jan 2010;37(1):55-63. [CrossRef] [Medline]
  10. Lisy K, Lai-Kwon J, Ward A, Sandhu S, Kasparian NA, Winstanley J, et al. Patient-reported outcomes in melanoma survivors at 1, 3 and 5 years post-diagnosis: a population-based cross-sectional study. Qual Life Res. Aug 05, 2020;29(8):2021-2027. [CrossRef] [Medline]
  11. Francken AB, Shaw HM, Accortt NA, Soong SJ, Hoekstra HJ, Thompson JF. Detection of first relapse in cutaneous melanoma patients: implications for the formulation of evidence-based follow-up guidelines. Ann Surg Oncol. Jun 15, 2007;14(6):1924-1933. [CrossRef] [Medline]
  12. Wolff T, Tai E, Miller T. Screening for skin cancer: an update of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. Feb 03, 2009;150(3):194-198. [FREE Full text] [CrossRef] [Medline]
  13. NCCN Guidelines® insights - melanoma: cutaneous, version 2.2021. National Comprehensive Cancer Network. 2021. URL: [accessed 2024-01-25]
  14. Weinstock MA, Risica PM, Martin RA, Rakowski W, Smith KJ, Berwick M, et al. Reliability of assessment and circumstances of performance of thorough skin self-examination for the early detection of melanoma in the Check-It-Out Project. Prev Med. Jun 2004;38(6):761-765. [CrossRef] [Medline]
  15. Moore Dalal K, Zhou Q, Panageas KS, Brady MS, Jaques DP, Coit DG. Methods of detection of first recurrence in patients with stage I/II primary cutaneous melanoma after sentinel lymph node biopsy. Ann Surg Oncol. Aug 30, 2008;15(8):2206-2214. [CrossRef] [Medline]
  16. McPherson M, Elwood M, English DR, Baade PD, Youl PH, Aitken JF. Presentation and detection of invasive melanoma in a high-risk population. J Am Acad Dermatol. May 2006;54(5):783-792. [CrossRef] [Medline]
  17. Leiter U, Garbe C. Epidemiology of melanoma and nonmelanoma skin cancer--the role of sunlight. Adv Exp Med Biol. 2008;624:89-103. [CrossRef] [Medline]
  18. De Giorgi V, Grazzini M, Savarese I, Gori A, Papi F, D'Errico A, et al. The impact of body area in melanoma self-detection: a retrospective study. Eur J Cancer Prev. Jul 2015;24(4):343-346. [CrossRef] [Medline]
  19. Pollitt RA, Geller AC, Brooks DR, Johnson TM, Park ER, Swetter SM. Efficacy of skin self-examination practices for early melanoma detection. Cancer Epidemiol Biomarkers Prev. Nov 2009;18(11):3018-3023. [CrossRef] [Medline]
  20. Murali R, Haydu LE, Long GV, Quinn MJ, Saw RP, Shannon K, et al. Clinical and pathologic factors associated with distant metastasis and survival in patients with thin primary cutaneous melanoma. Ann Surg Oncol. Jun 2012;19(6):1782-1789. [CrossRef] [Medline]
  21. Berwick M, Armstrong BK, Ben-Porat L, Fine J, Kricker A, Eberle C, et al. Sun exposure and mortality from melanoma. J Natl Cancer Inst. Feb 02, 2005;97(3):195-199. [CrossRef] [Medline]
  22. Schneider JS, Moore DH2, Mendelsohn ML. Screening program reduced melanoma mortality at the Lawrence Livermore National Laboratory, 1984 to 1996. J Am Acad Dermatol. May 2008;58(5):741-749. [CrossRef] [Medline]
  23. Lo SN, Scolyer RA, Thompson JF. Long-term survival of patients with thin (T1) cutaneous melanomas: a Breslow thickness cut point of 0.8 mm separates higher-risk and lower-risk tumors. Ann Surg Oncol. Apr 12, 2018;25(4):894-902. [CrossRef] [Medline]
  24. Robinson JK, Wayne JD, Martini MC, Hultgren BA, Mallett KA, Turrisi R. Early detection of new melanomas by patients with melanoma and their partners using a structured skin self-examination skills training intervention: a randomized clinical trial. JAMA Dermatol. Sep 01, 2016;152(9):979-985. [FREE Full text] [CrossRef] [Medline]
  25. Gogebakan KC, Mukherjee K, Berry EG, Sonmez K, Leachman SA, Etzioni R. Impact of novel systemic therapies on the first-year costs of care for melanoma among Medicare beneficiaries. Cancer. Aug 15, 2021;127(16):2926-2933. [FREE Full text] [CrossRef] [Medline]
  26. Serra-Arbeloa P, Rabines-Juárez ÁO, Álvarez-Ruiz MS, Guillén-Grima F. Cost of cutaneous melanoma by tumor stage: a descriptive analysis. Actas Dermosifiliogr. Apr 2017;108(3):229-236. [FREE Full text] [CrossRef] [Medline]
  27. Duffecy J, Sanford S, Wagner L, Begale M, Nawacki E, Mohr DC. Project onward: an innovative e-health intervention for cancer survivors. Psychooncology. Apr 21, 2013;22(4):947-951. [FREE Full text] [CrossRef] [Medline]
  28. Regan MW, Reid CD, Griffiths RW, Briggs JC. Malignant melanoma, evaluation of clinical follow up by questionnaire survey. Br J Plast Surg. Jan 1985;38(1):11-14. [CrossRef] [Medline]
  29. Coups EJ, Manne SL, Stapleton JL, Tatum KL, Goydos JS. Skin self-examination behaviors among individuals diagnosed with melanoma. Melanoma Res. Feb 2016;26(1):71-76. [CrossRef] [Medline]
  30. Manne S, Lessin S. Prevalence and correlates of sun protection and skin self-examination practices among cutaneous malignant melanoma survivors. J Behav Med. Oct 20, 2006;29(5):419-434. [CrossRef] [Medline]
  31. Loescher LJ, Harris RB, Lim KH, Su Y. Thorough skin self-examination in patients with melanoma. Oncol Nurs Forum. Jan 2, 2007;33(3):633-637. [CrossRef]
  32. Mujumdar UJ, Hay JL, Monroe-Hinds YC, Hummer AJ, Begg CB, Wilcox HB, et al. Sun protection and skin self-examination in melanoma survivors. Psychooncology. Oct 2009;18(10):1106-1115. [FREE Full text] [CrossRef] [Medline]
  33. Manne SL, Heckman CJ, Kashy D, Lozada C, Gallo J, Ritterband L, et al. Prevalence and correlates of skin self-examination practices among cutaneous malignant melanoma survivors. Prev Med Rep. Sep 2020;19:101110. [FREE Full text] [CrossRef] [Medline]
  34. Bowen DJ, Burke W, Hay JL, Meischke H, Harris JN. Effects of web-based intervention on risk reduction behaviors in melanoma survivors. J Cancer Surviv. Jun 26, 2015;9(2):279-286. [FREE Full text] [CrossRef] [Medline]
  35. Loescher LJ, Hibler E, Hiscox H, Quale L, Harris R. An internet-delivered video intervention for skin self-examination by patients with melanoma. Arch Dermatol. Aug 16, 2010;146(8):922-923. [FREE Full text] [CrossRef] [Medline]
  36. Robinson JK, Gaber R, Hultgren B, Eilers S, Blatt H, Stapleton J, et al. Skin self-examination education for early detection of melanoma: a randomized controlled trial of internet, workbook, and in-person interventions. J Med Internet Res. Jan 13, 2014;16(1):e7. [FREE Full text] [CrossRef] [Medline]
  37. Robinson JK, Turrisi R, Stapleton J. Efficacy of a partner assistance intervention designed to increase skin self-examination performance. Arch Dermatol. Jan 01, 2007;143(1):37-41. [CrossRef] [Medline]
  38. Coups EJ, Manne SL, Ohman Strickland P, Hilgart M, Goydos JS, Heckman CJ, et al. Randomized controlled trial of the mySmartSkin web-based intervention to promote skin self-examination and sun protection behaviors among individuals diagnosed with melanoma: study design and baseline characteristics. Contemp Clin Trials. Aug 2019;83:117-127. [FREE Full text] [CrossRef] [Medline]
  39. Manne SL, Heckman CJ, Kashy DA, Ritterband LM, Thorndike FP, Lozada C, et al. Randomized controlled trial of the mySmartSkin web-based intervention to promote skin self-examination and sun protection among individuals diagnosed with melanoma. Transl Behav Med. Jul 29, 2021;11(7):1461-1472. [FREE Full text] [CrossRef] [Medline]
  40. Mayer D, Layman A, Carlson J. Sun-protection behaviors of melanoma survivors. J Am Acad Dermatol. Jan 2012;66(1):e9-10. [FREE Full text] [CrossRef] [Medline]
  41. Freiman A, Yu J, Loutfi A, Wang B. Impact of melanoma diagnosis on sun-awareness and protection: efficacy of education campaigns in a high-risk population. J Cutan Med Surg. May 5, 2004;8(5):303-309. [CrossRef] [Medline]
  42. Bowen D, Jabson J, Haddock N, Hay J, Edwards K. Skin care behaviors among melanoma survivors. Psychooncology. Dec 21, 2012;21(12):1285-1291. [CrossRef] [Medline]
  43. Oliveria SA, Christos PJ, Halpern AC, Fine JA, Barnhill RL, Berwick M. Evaluation of factors associated with skin self-examination. Cancer Epidemiol Biomarkers Prev. Nov 1999;8(11):971-978. [Medline]
  44. Myers RE, Ross E, Jepson C, Wolf T, Balshem A, Millner L, et al. Modeling adherence to colorectal cancer screening. Prev Med. Mar 1994;23(2):142-151. [CrossRef] [Medline]
  45. Bruce AF, Theeke L, Mallow J. A state of the science on influential factors related to sun protective behaviors to prevent skin cancer in adults. Int J Nurs Sci. Jul 10, 2017;4(3):225-235. [FREE Full text] [CrossRef] [Medline]
  46. Manne S, Jacobsen PB, Ming ME, Winkel G, Dessureault S, Lessin SR. Tailored versus generic interventions for skin cancer risk reduction for family members of melanoma patients. Health Psychol. Nov 2010;29(6):583-593. [FREE Full text] [CrossRef] [Medline]
  47. Manne SL, Coups EJ, Jacobsen PB, Ming M, Heckman CJ, Lessin S. Sun protection and sunbathing practices among at-risk family members of patients with melanoma. BMC Public Health. Feb 21, 2011;11:122. [FREE Full text] [CrossRef] [Medline]
  48. Myers RE, Hyslop T, Wolf TA, Burgh D, Kunkel EJ, Oyesanmi OA, et al. African-American men and intention to adhere to recommended follow-up for an abnormal prostate cancer early detection examination result. Urology. May 2000;55(5):716-720. [CrossRef] [Medline]
  49. Proctor EK, Colditz GA, Brownson RC. Dissemination and Implementation Research in Health: Translating Science to Practice. New York, NY. Oxford University Press; 2012.
  50. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. Sep 1999;89(9):1322-1327. [CrossRef] [Medline]
  51. Blackman KC, Zoellner J, Berrey LM, Alexander R, Fanning J, Hill JL, et al. Assessing the internal and external validity of mobile health physical activity promotion interventions: a systematic literature review using the RE-AIM framework. J Med Internet Res. Oct 04, 2013;15(10):e224. [FREE Full text] [CrossRef] [Medline]
  52. Carnevale TD. Universal adolescent depression prevention programs: a review. J Sch Nurs. Jun 26, 2013;29(3):181-195. [CrossRef] [Medline]
  53. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. Oct 15, 2006;21(5):688-694. [CrossRef] [Medline]
  54. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to "employ" the RE-AIM model? Eval Health Prof. Mar 21, 2013;36(1):44-66. [CrossRef] [Medline]
  55. Klesges LM, Dzewaltowski DA, Glasgow RE. Review of external validity reporting in childhood obesity prevention research. Am J Prev Med. Mar 2008;34(3):216-223. [CrossRef] [Medline]
  56. Kwan BM, Brownson RC, Glasgow RE, Morrato EH, Luke DA. Designing for dissemination and sustainability to promote equitable impacts on health. Annu Rev Public Health. Apr 05, 2022;43(1):331-353. [FREE Full text] [CrossRef] [Medline]
  57. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. Aug 2013;46(1):81-95. [FREE Full text] [CrossRef] [Medline]
  58. Atkinson NL, Massett HA, Mylks C, Hanna B, Deering MJ, Hesse BW. User-centered research on breast cancer patient needs and preferences of an internet-based clinical trial matching system. J Med Internet Res. May 15, 2007;9(2):e13. [FREE Full text] [CrossRef] [Medline]
  59. Auxier B, Anderson M. Social media use in 2021. Pew Research Center. Apr 7, 2021. URL: [accessed 2021-06-24]
  60. Manne S, Buller D, Devine K, Heckman C, Pagoto S, Frederick S, et al. Sun safe partners online: pilot randomized controlled clinical trial. J Med Internet Res. Sep 17, 2020;22(9):e18037. [FREE Full text] [CrossRef] [Medline]
  61. Heckman CJ, Riley M, Khavjou O, Ohman-Strickland P, Manne SL, Yaroch AL, et al. Cost, reach, and representativeness of recruitment efforts for an online skin cancer risk reduction intervention trial for young adults. Transl Behav Med. Oct 23, 2021;11(10):1875-1884. [FREE Full text] [CrossRef] [Medline]
  62. Manne S, Day A, Coups EJ, Kashy D. Sun safe partners: a pilot and feasibility trial of a couple-focused intervention to improve sun protection practices. Prev Med Rep. Dec 2018;12:220-226. [FREE Full text] [CrossRef] [Medline]
  63. Manne S, Hudson S, Frederick S, Mitarotondo A, Baredes S, Kalyoussef E, et al. e-Health self-management intervention for oral and oropharyngeal cancer survivors: design and single-arm pilot study of empowered survivor. Head Neck. Nov 2020;42(11):3375-3388. [FREE Full text] [CrossRef] [Medline]
  64. Melanoma warning signs. The Skin Cancer Foundation. URL: [accessed 2023-09-01]
  65. Self-exams save lives: how to do a skin check. The Skin Cancer Foundation. URL: [accessed 2023-09-01]
  66. Glanz K, Yaroch AL, Dancel M, Saraiya M, Crane LA, Buller DB, et al. Measures of sun exposure and sun protection practices for behavioral and epidemiologic research. Arch Dermatol. Feb 2008;144(2):217-222. [CrossRef] [Medline]
  67. Lang KM, Little TD. Principled missing data treatments. Prev Sci. Apr 4, 2018;19(3):284-294. [CrossRef] [Medline]
  68. Cho BH, Hicks KA, Honeycutt AA, Hupert N, Khavjou O, Messonnier M, et al. A tool for the economic analysis of mass prophylaxis operations with an application to H1N1 influenza vaccination clinics. J Public Health Manag Pract. 2011;17(1):E22-E28. [CrossRef] [Medline]
  69. Honeycutt AA, Khavjou OA, Bradley C, Neuwahl S, Hoerger TJ, Bellard D, et al. Intervention costs from communities putting prevention to work. Prev Chronic Dis. Jul 28, 2016;13:E98. [FREE Full text] [CrossRef] [Medline]
  70. Honeycutt AA, Khavjou OA, Jones DJ, Cuellar J, Forehand RL. Helping the noncompliant child: an assessment of program costs and cost-effectiveness. J Child Fam Stud. Feb 7, 2015;24(2):499-504. [FREE Full text] [CrossRef] [Medline]
  71. Khavjou OA, Honeycutt AA, Hoerger TJ, Trogdon JG, Cash AJ. Collecting costs of community prevention programs: communities putting prevention to work initiative. Am J Prev Med. Aug 2014;47(2):160-165. [CrossRef] [Medline]
  72. Khavjou OA, Honeycutt AA, Yarnoff B, Bradley C, Soler R, Orenstein D. Costs of community-based interventions from the Community Transformation Grants. Prev Med. Jul 2018;112:138-144. [CrossRef] [Medline]
  73. Yarnoff B, Khavjou O, Bradley C, Leis J, Filene J, Honeycutt A, et al. Standardized cost estimates for home visiting: pilot study of the home visiting budget assistance tool (HV-BAT). Matern Child Health J. Apr 13, 2019;23(4):470-478. [CrossRef] [Medline]
  74. Kakushadze Z, Raghubanshi R, Yu W. Estimating cost savings from early cancer diagnosis. Data. Sep 04, 2017;2(3):30. [CrossRef]
  75. Guy JGJ, Zhang Y, Ekwueme DU, Rim SH, Watson M. The potential impact of reducing indoor tanning on melanoma prevention and treatment costs in the United States: an economic analysis. J Am Acad Dermatol. Feb 2017;76(2):226-233. [FREE Full text] [CrossRef] [Medline]
  76. Guy GPJ, Machlin SR, Ekwueme DU, Yabroff KR. Prevalence and costs of skin cancer treatment in the U.S., 2002-2006 and 2007-2011. Am J Prev Med. Feb 2015;48(2):183-187. [FREE Full text] [CrossRef] [Medline]
  77. Styperek A, Kimball AB. Malignant melanoma: the implications of cost for stakeholder innovation. Am J Pharm Benefits. 2012;4(2):66-76.
  78. Alexandrescu DT. Melanoma costs: a dynamic model comparing estimated overall costs of various clinical stages. Dermatol Online J. 2009;15(11) [FREE Full text] [CrossRef]
  79. SEER cancer statistics review (CSR). National Cancer Institute. URL: [accessed 2024-01-25]
  80. Haddix AC, Teutsch SM, Corso PS. Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation. Oxford, UK. Oxford University Press; 1996.
  81. Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG. Cost-Effectiveness in Health and Medicine. Oxford, UK. Oxford University Press; 2016.
  82. Arias E, Bastian B, Xu J, Tejada-Vera B. U.S. State life tables, 2018. National Vital Statistics Reports. Mar 11, 2021. URL: [accessed 2024-01-25]
  83. Sampogna F, Paradisi A, Iemboli ML, Ricci F, Sonego G, Abeni D. Comparison of quality of life between melanoma and non-melanoma skin cancer patients. Eur J Dermatol. Apr 01, 2019;29(2):185-191. [CrossRef] [Medline]
  84. Blackford S, Roberts D, Salek MS, Finlay A. Basal cell carcinomas cause little handicap. Qual Life Res. Apr 1996;5(2):191-194. [CrossRef]
  85. Schlesinger-Raab A, Schubert-Fritschle G, Hein R, Stolz W, Volkenandt M, Hölzel D, et al. Quality of life in localised malignant melanoma. Ann Oncol. Dec 2010;21(12):2428-2435. [FREE Full text] [CrossRef] [Medline]
  86. Rhee JS, Matthews BA, Neuburg M, Smith TL, Burzynski M, Nattinger AB. Skin cancer and quality of life. Dermatol Surg. 2004;30(4):525-529. [CrossRef]
  87. Sampogna F, Tabolli S, Abeni D. Impact of different skin conditions on quality of life. G Ital Dermatol Venereol. Jun 2013;148(3):255-261. [Medline]
  88. Schubert-Fritschle G, Schlesinger-Raab A, Hein R, Stolz W, Volkenandt M, Hölzel D, et al. Quality of life and comorbidity in localized malignant melanoma: results of a German population-based cohort study. Int J Dermatol. Jun 2013;52(6):693-704. [CrossRef] [Medline]
  89. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. Aug 29, 2017;12(1):108. [FREE Full text] [CrossRef] [Medline]
  90. De Marchis EH, Hessler D, Fichtenberg C, Adler N, Byhoff E, Cohen AJ, et al. Part I: a quantitative study of social risk screening acceptability in patients and caregivers. Am J Prev Med. Dec 2019;57(6 Suppl 1):S25-S37. [FREE Full text] [CrossRef] [Medline]
  91. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 19, 2011;38(2):65-76. [FREE Full text] [CrossRef] [Medline]
  92. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. Nov 2005;15(9):1277-1288. [CrossRef] [Medline]
  93. Fielding NG, Fielding JL. Linking Data. Thousand Oaks, CA. SAGE Publications, Inc; Mar 13, 1986.
  94. Creswell JW, Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA. SAGE Publications, Inc; 2007.
  95. Weinstock MA, Martin RA, Risica PM, Berwick M, Lasater T, Rakowski W, et al. Thorough skin examination for the early detection of melanoma. Am J Prev Med. Oct 1999;17(3):169-175. [CrossRef] [Medline]
  96. Glanz K, Schoenfeld ER, Steffen A. A randomized trial of tailored skin cancer prevention messages for adults: project SCAPE. Am J Public Health. Apr 2010;100(4):735-741. [FREE Full text] [CrossRef] [Medline]
  97. O'Riordan DL, Glanz K, Gies P, Elliott T. A pilot study of the validity of self-reported ultraviolet radiation exposure and sun protection practices among lifeguards, parents and children. Photochem Photobiol. May 2008;84(3):774-778. [FREE Full text] [CrossRef] [Medline]
  98. O'Riordan DL, Nehl E, Gies P, Bundy L, Burgess K, Davis E, et al. Validity of covering-up sun-protection habits: association of observations and self-report. J Am Acad Dermatol. May 2009;60(5):739-744. [FREE Full text] [CrossRef] [Medline]
  99. Hillhouse J, Turrisi R, Jaccard J, Robinson J. Accuracy of self-reported sun exposure and sun protection behavior. Prev Sci. Oct 27, 2012;13(5):519-531. [FREE Full text] [CrossRef] [Medline]
  100. Køster B, Søndergaard J, Nielsen JB, Olsen A, Bentzen J. Reliability and consistency of a validated sun exposure questionnaire in a population-based Danish sample. Prev Med Rep. Jun 2018;10:43-48. [FREE Full text] [CrossRef] [Medline]
  101. Glanz K, Mayer JA. Reducing ultraviolet radiation exposure to prevent skin cancer methodology and measurement. Am J Prev Med. Aug 2005;29(2):131-142. [CrossRef] [Medline]

AIM: Acceptability of Intervention Measure
BCT: behavior change technique
CINJ: Cancer Institute of New Jersey
COREQ: Consolidated Criteria for Reporting Qualitative Research
CRGC: Cancer Registry of Greater California
IRB: institutional review board
LYG: life year gained
MSS: mySmartSkin
NJSCR: New Jersey State Cancer Registry
PHM: preventive health model
PRISM: Practical, Robust Implementation and Sustainability Model
RCT: randomized controlled trial
RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance
SSE: skin self-examination
UC: usual care

Edited by A Mavragani; The proposal for this study was peer-reviewed by Science of Implementation in Health and Healthcare Study Section Healthcare Delivery and Methodologies Integrated Review Group Center For Scientific Review (SIHH, USA). See the Multimedia Appendix for the peer-review report;submitted 12.09.23; accepted 26.10.23; published 12.02.24.


©Sharon Manne, Carolyn J Heckman, Sara Frederick, Alexis A Schaefer, Christina R Studts, Olga Khavjou, Amanda Honeycutt, Adam Berger, Hao Liu. Originally published in JMIR Research Protocols (, 12.02.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.