Working with our Advisory Board, we develop and refine a survey that includes common FP intervention components and associated mechanisms, as well as a qualitative interview to distribute to FPs and caregivers who provided or received one-to-one FP support. We will survey 50 FPs and 50 caregivers to report on components delivered (FPs) or received (caregivers) and perceptions of each component’s utility (ie, mechanism identification) in the first year of providing or receiving FP support. A subset of purposively sampled of 48 respondents will be invited to complete qualitative interviews to share in-depth qualitative data on intervention components, target mechanisms, and perceived effect on outcomes to inform aim 2. Purposive sampling criteria are described in the Participants and Procedures section.

Participants for the aim 1 Survey will be recruited via pathways identified in the Study Setting section. Aim 1 participants will include trained FPs (including FP supervisors or trainers) and caregivers who have received FP support. FP participants must meet the following criteria: (1) have received formal training in providing FP support, (2) have provided at least one session of one-to-one FP support in the past 3 months to a caregiver of a child aged 5-17 years with mental health needs, (3) currently work in a program that represents outpatient care, (4) speak English, and (5) be at least 18 years of age. FP supervisors or trainers must meet the same FP inclusion criteria but only need to have provided one-to-one support in the past 6 months and engage in training or supervising others to provide FP support.

Caregiver participants must meet the following criteria: (1) received at least one instance of one-to-one FP support in an outpatient setting within the past 3 months, (2) identify as a primary caregiver of a child with mental health needs aged 5-17 years at receipt of FP support, (3) focus of FP support was related to child mental health need, (4) speaks English, and (5) is at least 18 years of age.

For qualitative interviews, we will purposively sample 48 respondents, including 32 FPs (aiming for approximately half to be trainers or supervisors). They will be selected to reflect a range of intervention components delivered, mechanisms targeted, work settings, backgrounds, and years of experience reported. We will also invite 16 caregivers, selected to reflect a range of intervention components received, as well as settings in which seeking FP care and types of support received. We will sample primarily for diversity in FP work setting while aiming to achieve representation across backgrounds, including training model and years of experience.

All survey development procedures will be conducted in collaboration with our Advisory Board to ensure that surveys apply to the diversity of existing FP models and will support the logic model of the problem and change development. Final survey content and design will be determined in collaboration with the study Advisory Board.

The aim 1 survey will use tailored design principles [42], including clear and thoughtful survey development, efforts to minimize respondent burden through brevity, and structured follow-up contacts to maximize accessibility and completion rates. Prior to distributing the survey, we will gather Advisory Board feedback on individual items (including intervention components and mechanisms) as well as feedback on survey accessibility and length. They will report on any components or mechanisms that they think are missing, any that seem redundant, and any sources of confusion. They will also help refine wording to ensure that the survey is easily understood by both FPs and caregivers. These suggestions will be discussed with the research team and integrated into a revised survey that will be shared back with the Advisory Board for additional feedback and refinement prior to survey roll-out. The aim 1 survey will expand an existing validated measure of FP adherence, the family contact log [24,26] to capture content across FP programs and models. We will consult other existing FP quality measures like the family journey assessment [27] and tools used internally by FREDLA to be applicable to a wider range of FP programs. We will use this expanded measure to create both an FP-reported and caregiver-reported version.

We anticipate the final measure will focus on support provided or received in the first year of FP support. For the FP-reported version, FPs will first be asked to identify a specific session in the past 3 months conducted with a caregiver of a child aged 5-17 years they have been working with for a year or less. They will then be asked to report on the extent to which they discussed a range of topics and used a range of common FP intervention strategies with caregivers in that identified session. They will be asked how typical was of their sessions with caregivers, if they discussed any topics or used any intervention strategies in this session that were not included in the list and if they generally use any topics or intervention strategies not included in the list. They will then be presented with the list of strategies that they said they used in the session and asked to select the three most important strategies. Finally, they will be asked to rank the top three target mechanisms from a list provided that they targeted in this identified session. Throughout the survey, respondents will be able to free-respond to share on any intervention components or mechanisms they do not see in the survey. Respondents will also be asked to report on their demographic characteristics and workplace, as well as brief questions on the identified session. For the caregiver version, questions will be revised to instead focus on FP report received in the same format described above but with a focus on general receipt versus identification of a single session. As with the FP version, caregivers will have the option to share any intervention components or mechanisms not listed through a free response option. Caregivers will also complete brief questions about their and their child’s demographics, their child’s mental health needs, and the setting in which they received FP support. Each version of the survey will take ~30 minutes; all participants will be compensated US $35. Refer to Table 3 for full list of measures to be presented across study aims.

To help ensure comprehensibility of survey, we will collect feedback on the draft tool up to 4 FPs individually using cognitive interviewing [43] procedures. We will use their responses to refine the survey prior to final administration. We will prioritize getting feedback from those who have not already provided feedback through Advisory Board. Respondents will be compensated US $50 for their time.

In addition to the aim 1 survey, we will use qualitative interviews to gather in-depth information about common intervention components and perceptions of how each links to target mechanisms to deepen findings gathered from survey data (QUAN→qual approach [44]). To participate in the aim 1 qualitative interview, a participant must have participated in the aim 1 survey and been invited to participate based on their survey response, based on sampling criteria described in the Participants and Procedures section.

Interview guides for FPs and caregivers will be refined in collaboration with our Advisory Board. Interviews will first ask respondents general questions about the perceived impact of FP support, more specific questions about identified intervention components respondents reported they often used or received on the aim 1 survey, and ask FPs or caregivers to elaborate on how they operationalize these techniques in practice. Then, we will query respondents about intended mechanisms targeted by components used or received often and outcomes of interest. We will also include questions about how FPs disclose and address cultural factors in their work, and about how relevant contextual factors influence FPs’ intervention decisions and families’ response to FP recommendations.

For FP respondents, we will assess perceived impact of FP support using open-ended questions (eg, “How do you think receiving Family Peer support impacts families?”). To help operationalize intervention components, we will ask questions like: “When you provide informational support to a family, what does that look like?” To gather information about intended target mechanisms, we may ask questions like: “When you deliver emotional support, what are you hoping to achieve or what changes are you hoping to see in the caregiver?” To support respondents in engaging causal thinking, we may also use fill-in-the-blank statements such as “because” and “if-then” to follow up on specific intended mechanisms. Questions to gather information on outcomes of interest associated with mechanisms will rely on open-ended questions that specifically query areas of improvement seen in the literature (service engagement, changes in caregiver well-being, changes in parenting approach or caregiver-child relationship). We may also reference open-ended responses provided in their survey about mechanism targets. For caregivers, we will ask a directly corresponding set of questions. Interviews will take approximately 45 minutes; participants will be compensated US $50. Interviews will be recorded and transcribed for analysis.

Consistent with a QUAN→qual approach [44], quantitative data will first be collected and inspected. Qualitative data will then be used to explore and hypothesis test what is learned in the quantitative data. In the quantitative survey data, we will identify intervention components commonly used by FPs and their perceived links to hypothesized mechanisms from FP and caregiver perspectives. We will conduct basic descriptive statistics on the survey data, inspect missingness, and characterize the demographics of the study sample. We will aggregate all open-ended survey responses reporting on intervention components and mechanisms, determine whether content is unique from items presented in the survey, and develop a set of codes to categorize them. We will map them onto the hypothesized mechanisms in Table 2 and generate more categories of potential mechanisms as needed. We will use qualitative analysis to contextualize quantitative results, analyze the open-ended questions, and elaborate on mechanistic pathways that emerge from the quantitative data.

To analyze qualitative results, we will rely on Rapid Qualitative Analysis (RQA) procedures. This approach provides an alternative to traditional qualitative analysis and has been shown to reduce time needed to complete analysis without sacrificing rigor [45-47]. Given the abbreviated window between collection of all aim 1 qualitative data and development and distribution of the aim 3 survey, the focused research questions, and the mixed methods nature of the data collected, we believe this project is well-aligned with RQA procedures [48]. We will first develop our summary template with domains corresponding to study questions informed by existing theory, identified intervention components from the survey, and mechanisms described in the survey. Once the template is fully developed, the first author and all other research team members involved in RQA will assess the template using three randomly selected transcripts. For training and calibration, we will also compare summarizing styles across team members and seek to understand and reconcile any areas of difference. We will continue this process until summaries look equivalent across research team members. The template will include space for other relevant content not reflected in current captures domains. Once the template has been verified and consistency across team members is established, we will divide up transcripts to summarize. Consistent with RQA best practices, the study PI will also audit 20% of the individually prepared remaining summaries using the original transcripts [48].

These summaries will then be placed into a matrix where they can be examined by respondent group. When possible, we will make efforts to complete summary templates immediately after study interviews [48]. Informed by the aim 1 survey, we will use qualitative results and synthesize results into a draft set of core interventions, intended target mechanisms, and links to proximal outcomes for aim 2. We will examine our matrix across and between respondent groups to see if components or mechanisms vary. We will also see if both qualitative and quantitative responses vary by FP background or work setting.

Procedures for aim 2 will occur in conjunction with the study Advisory Board. We will present aim 1 data to the Advisory Board. Over a series of meetings, we will collaboratively refine a logic model of change delineating how each intervention component impacts change mechanisms and outcomes. The finalized logic model provides a comprehensive list of FP intervention components with intended mechanisms and informs the tool to be tested in aim 3.

Guided by the IM approach of integrating data, theory, and community perspectives to build logic models, we will synthesize Aim 1 data into a summary of common intervention components, mechanisms, and qualitative descriptions to present to the Advisory Board for feedback. We will use visual displays of potential causal pathways using a program like Lucidchart [49] to improve comprehensibility. The focus of one Advisory Board meeting will be on refining the list of intervention components. We will present a comprehensive list of all intervention components identified across survey and qualitative interview results. We will provide accompanying example quotes to show how caregivers, FPs, and trainers describe them. We will ask for global impressions of the list, whether they think anything is missing and whether there are redundancies. We will refine the list based on feedback. At the next Advisory Board meeting, we will present the comprehensive list of target mechanisms endorsed across Aim 1 data with a focus on qualitative and open-ended responses grouped by theme. We will include example quotes of how FPs, trainers, and families perceive these mechanisms to be engaged. We will gather feedback on the links between the intervention component and mechanisms. Finally, at the next meeting, we will present a comprehensive logic model of change that presents each intervention component tied to the mechanism or mechanisms of change that are most strongly linked, and the outcomes of interest. We will briefly present the qualitative data on outcomes of interest for feedback. The Advisory Board will use this drafted logic model to create a comprehensive logic model of change. If there are disagreements among Advisory Board members, we will note these conflicting opinions and try to come to consensus in the meeting. If this is not possible, the PI will reexamine descriptive statistics from aim 1 data to see whether they suggest a direction. If direction remains ambiguous, we will keep both opinions represented in the logic model.

Informed by the final list of discrete interventions included in the logic model, we will create a comprehensive list of common FP intervention components that appear across protocols, how they are delivered, and theorized target mechanisms of change. The components will include clearly operationalized definitions informed both by qualitative interviews and Advisory Board meeting discussions. The list of common intervention components will comprise the first comprehensive list of FP intervention components across FP protocols, which may be used in future effectiveness studies of FP support. These components and their targets also will serve as the basis for: (1) the self-report measure of use of FP intervention components that will be developed for and evaluated in aim 3, and (2) the coding manual to code intervention component use within standardized walk-throughs (refer to aim 3) to validate the self-report tool.

Using a systematic community-partnered approach [50], we will then adapt the self-report of intervention components tool from Aim 1 to align with the comprehensive logic model of change to be validated in Aim 3. This tool will ask FPs to report on the extensiveness of the intervention components they delivered in a specific session with a caregiver on a 4-point Likert scale. We will solicit feedback from the Advisory Board on the comprehensiveness of the tool and to ensure intervention components are described in easily comprehensible lay language for those working in outpatient settings, which may include community, home-based, and clinic models. After revisions, we will present the tool to 4-to-6 FPs individually and elicit feedback using cognitive interviewing [43]. These FPs will be external to the Advisory Board and will be compensated US $50 for their time. Interviewers will ask interviewees to complete the tool to identify any sources of confusion or response error, using “think aloud” procedures and “verbal probing” cognitive interviewing techniques [43]. Interviewees will also provide feedback on the operationalized intervention descriptions included in the tool and list of common intervention components. Results will inform measure refinements to address common sources of confusion or error until saturation is reached (ie, no additional areas of confusion are identified). We will present the revised tool for final feedback alongside the comprehensive list of common intervention components to the Advisory Board.

We will validate the self-report tool developed across aims 1 and 2. In total, 200 FPs will complete the tool following a recent session with a caregiver, along with brief measures of acceptability, feasibility, perceived target mechanisms, and demographics. We will evaluate psychometric properties including internal consistency and factor structure as well as acceptability and pairs of intervention components and mechanisms.

For additional perspectives on components and associated mechanisms, a subset of 40 FP caregiver dyads will complete a parallel report on a selected session. For validation, an additional 40 FPs will complete up to two standardized walk-through encounters with associated self-report tools, which will be compared to independent observational coding of their sessions. Aim 3 activities will result in refined materials necessary to use in future large-scale evaluations of FP programs.

Aim 3 FP participants may include those who participated in Aim 1. We will also recruit additional FPs to meet our sample size goal. We will rely on the same recruitment methods from Aim 1. Aim 3 FP participants must meet the following criteria: (1) Have received formal training in providing FP support, (2) work in an ambulatory setting, (3) speak English, (4) be at least 18 years of age, and (5) have a caregiver on their caseload who meets the following criteria: caregiver of a child ages 5-17 with mental health needs, receiving individual support from the FP, received at least one session of FP support in past 3 months, and has been seeing the FP for less than a year.

Aim 3 surveys will be administered electronically and include: the developed self-report of common FP intervention components tool, acceptability of the tool, feasibility of the tool, a target mechanism questionnaire, and a demographics questionnaire. In line with validation procedures for other measures to capture fidelity to other intervention components [51], 200 FPs will complete the self-report tool finalized in Aim 2 for a selected session with a representative caregiver. We will then survey them about the acceptability and feasibility of the self-report tool to capture use of common FP intervention components in standard practice, using the validated 4-item Acceptability of Intervention Measure (AIM) [52] to report on acceptability and the 4-item Feasibility of Intervention Measure (FIM) to report on feasibility. Respondents will also answer the target mechanism questionnaire and select what they were trying to change with their intervention delivery from a list of target mechanisms (finalized in Aim 2 activities), based on items they endorsed in the tool. Respondents will also report on the perceived strength of the relationship between intervention component delivery and target mechanisms on a 4-point Likert scale from “not at all” to “a lot.” We will collect demographics using the same items as in Aim 1. They will also provide brief deidentified information about the representative caregiver and family, the session, and any factors that affected intervention delivery. The research team will support prompt survey completion to foster session recall using reminders.

For the Aim 3 Caregiver and FP surveys, we will purposively sample FPs by work setting and training background as reported in the Aim 3 survey. If selected based on the Aim 3 survey, FPs will be asked to identify a caregiver with whom they are working, who we will ask to complete the self-report tool a second time for a selected session. We will support FPs to identify caregivers of youth aged 5-17 years with whom they are currently working. Because services may differ for caregivers early vs later in their FP support, we will randomly ask FPs within this subset to invite either: (1) a caregiver within their first 3 months of FP service who has had at least 2 prior sessions, or (2) a caregiver with whom they have been working for 3-12 months to participate. While no formal consensus exists on the ideal length of time involved in FP support, studies suggest one-to-one FP support can range from 3 months to 9 months or longer in length [12,20,53,54]; thus, these windows were selected to differentiate early vs later phase content that might be delivered.

To recruit caregivers, selected FPs will be given a handout and link to provide to an identified caregiver with whom they are working. FPs will be instructed to select a session that is not a first or last session, as the content of these is likely to be distinct from the typical content of FP support. Once provided with the handout and link, the caregiver will have the option to speak with the study team or complete a consent form independently. Once a caregiver consents, they will be texted a link to their survey immediately after their indicated session. This survey will include the self-report tool to capture receipt of common FP interventions alongside the target mechanism questionnaire, session information, and brief demographic questions. FPs will also be texted a parallel survey link immediately after the selected session. If the caregiver does not consent prior to the session but does want to participate, while we will aim for survey completion within 48 hours of the session, they will be eligible to consent within a week of the session and have up to three weeks after the session to complete the parallel self-report tool. However, we will make all efforts to have consent obtained prior to the session to allow for reporting on content as close to the session as possible. There will be no recording of caregivers.

To further validate the self-report tool, we will select another subset of 40 FPs based on their Aim 3 survey results to complete a standardized role play (walk-through paradigm. In selecting the subset of FPs for the walk-through, we will aim for a diverse sample in terms of work setting, training background, and whether the FP reports high, medium, or low use of intervention components on the Aim 3 FP survey. The walk-through is meant to simulate a prototypical session with a caregiver and relies on a trained FP to act as a standardized patient and role-play a caregiver who is receiving FP support [28]. We have selected the walk-through approach over video or audio recording sessions due to acceptability concerns related to recording. This method has been used with FPs to evaluate service delivery and has been found to be feasible and acceptable [24,28]. However, it is time-intensive and not suitable for scale-up, underscoring the importance of validating a pragmatic, self-report tool to capture use of common FP intervention components.

Selected FPs will be assigned to complete up to two randomly selected walk-through scripted vignettes that may represent different phases of treatment (eg, beginning, middle, and end of treatment), excluding first or last sessions. Walk-through scripts will include three possible scenarios adapted to ensure that they allow FPs to use the full range of common FP intervention components from the self-report tool. After the walk-through, each FP will complete the self-report tool to report on their own use of the common FP intervention components delivered in the walk-through. Walk-through recordings will also be coded by trained raters using a comprehensive coding scheme for all possible FP interventions delineated on a scale that parallels that of the self-report tool. To establish coding reliability, research team members including the PI, research assistant, and student worker will code alongside a trained FP and supervisor who is part of our Advisory Board. To be consistent with benchmarks for other observation coding scales [55], raters will be considered reliable and able to code independently after achieving intraclass correlation coefficients of >0.70 with the consensus code.

We will examine the psychometric properties of the FP completed self-report tool to capture use of common intervention components, including internal consistency for the measure using Cronbach α. To examine response patterns, we will look at a histogram of all item responses. We will also conduct Exploratory Factor Analysis (EFA) to see which constructs appear and which intervention components are endorsed together. After data screening, we will create a correlation matrix of pairwise and partial correlations. To determine data factorability, we will look at the correlation magnitudes and their statistical significance. We anticipate many items will be highly correlated, indicating that they are endorsed as frequently delivered together in a session. We will look at Kaiser measure of sampling adequacy to ensure that it is greater than 0.6, indicating factor analysis suitability. If many correlations are significant and the Kaiser measure of sampling is adequate, we will extract factors via maximum likelihood factoring. For interpretability, we will rotate the factors using an oblique rotation (promax). To determine the number of factors, we will use a combination of parallel analysis with Glorfeld correction [56], minimum average partial [57], interpretability, and evaluation of the scree plot for where the eigenvalues begin to form a downward linear trend (the “elbow”). We anticipate at least 5 factors based on what is in the literature (Table 2); however, as noted above, it is possible additional key content areas may emerge based on aim 1 activities, so the final measure may include more factors. If a smaller number of factors is adequate, we will retain the most parsimonious factor model. In the interest of keeping the measure as brief as possible, we will inspect factor loadings, item thresholds/intercepts, and face validity to determine inclusion or removal. We will examine eigenvalues to determine the amount of variance explained by each factor and examine the variance accounted for by the factors. We expect the factor structure to account for at least 60% of the total variance [58]. We will examine if psychometric properties vary based on FP background or work setting. We will also examine if these variables impact components reported and associated mechanisms from the mechanism survey. High variability may suggest additional measure revisions necessary to ensure items translate across FP models. Finally, we will also examine feasibility metrics such as questions missed and time taken to complete measure, as well as results from the Acceptability of Intervention Measure (AIM) and Feasibility of Intervention Measure (FIM) [52].

Post hoc sample size adequacy will be determined by (1) evaluation of the loading standard errors, and (2) bootstrapped resampling and refitting of the factor structure to check its stability. The latter will involve Procrustes-rotating each solution to the full-sample target (to handle rotational indeterminacy, sign flips, and column reordering) and examining the distribution of loadings and frequency with which items change factor assignment with each resample.

We will examine convergent validity in the subsample of 40 FPs who complete the standardized walk-throughs by comparing results of coded recordings to the FP-completed tool from the walk-through encounters. We will use paired t tests to identify any differences between results from the walk-through and the self-report tool for each item. We will also examine responses collected from FPs about acceptability and feasibility of the measure and how long it took them to complete the measure.

We will also examine convergence of FP responses and caregiver responses for the subset of 40 dyads. We expect that FPs and caregivers may vary in their report of what occurred given that there is known variability in provider and client agreement on therapy session content [59,60]. We will analyze any area of discrepancy to inform understanding of common components as well as possible mechanisms.

We will preliminarily validate the mechanistic pathways identified in aims 1 and 2 by creating a correlation matrix of items from the self-report tool of common FP intervention components and survey items from the target mechanism questionnaire. We will identify mechanisms most highly correlated with each intervention component. The most correlated variables will be selected for formal validation in a subsequent R01 proposal using prospective caregiver data to measure mechanisms. We will also specifically examine pathways identified in the logic model with these correlations. For example, if the logic model says advocacy support increases self-efficacy, we will examine the correlation between these items. We will consider correlations significant above a threshold of 0.30 to be evidence of preliminary validation of the association. Analyses are not intended to be fully powered; correlation magnitude will inform model refinement for fully powered testing in a subsequent trial.

Consistent with IM, we will present all relevant findings to the Advisory Board. We will discuss psychometric data, including discrepancies between observational coding, caregiver, and FP-report to determine if modifications to the FP self-reported tool are warranted before use in subsequent studies. We will also present preliminarily validated mechanistic pathways for feedback and revision.