Published on in Vol 10, No 12 (2021): December

Preprints (earlier versions) of this paper are available at, first published .
Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial


1Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom

2Risk and Information Systems Research Group, School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom

3Department of Physiotherapy, Barts Health NHS Trust, London, United Kingdom

Corresponding Author:

Cabella Lowe, MSc

Centre for Sports & Exercise Medicine

William Harvey Research Institute

Queen Mary University of London

Mile End Hospital

Bancroft Road

London, E1 4DG

United Kingdom

Phone: 44 07976315105


Background: Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing demand for health care use. Digital technologies to improve health care outcomes and efficiency are considered a priority; however, innovations are rarely tested with sufficient rigor in clinical trials, which is the gold standard for clinical proof of safety and efficacy. We have developed a new musculoskeletal digital assessment routing tool (DART) that allows users to self-assess and be directed to the right care. DART requires validation in a real-world setting before implementation.

Objective: This pilot study aims to assess the feasibility of a future trial by exploring the key aspects of trial methodology, assessing the procedures, and collecting exploratory data to inform the design of a definitive randomized crossover noninferiority trial to assess DART safety and effectiveness.

Methods: We will collect data from 76 adults with a musculoskeletal condition presenting to general practitioners within a National Health Service (NHS) in England. Participants will complete both a DART assessment and a physiotherapist-led triage, with the order determined by randomization. The primary analysis will involve an absolute agreement intraclass correlation (A,1) estimate with 95% CI between DART and the clinician for assessment outcomes signposting to condition management pathways. Data will be collected to allow the analysis of participant recruitment and retention, randomization, allocation concealment, blinding, data collection process, and bias. In addition, the impact of trial burden and potential barriers to intervention delivery will be considered. The DART user satisfaction will be measured using the system usability scale.

Results: A UK NHS ethics submission was done during June 2021 and is pending approval; recruitment will commence in early 2022, with data collection anticipated to last for 3 months. The results will be reported in a follow-up paper in 2022.

Conclusions: This study will inform the design of a randomized controlled crossover noninferiority study that will provide evidence concerning mobile health DART system clinical signposting in an NHS setting before real-world implementation. Success should produce evidence of a safe, effective system with good usability, potentially facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This rigorous approach to mobile health system testing could be used as a guide for other developers of similar applications.

Trial Registration: NCT04904029;

International Registered Report Identifier (IRRID): PRR1-10.2196/31541

JMIR Res Protoc 2021;10(12):e31541




Musculoskeletal disorders (MSDs) are the leading contributors to years lived with disability worldwide and have shown an increase in disease burden over the past decade [1-3]. Musculoskeletal conditions can affect as many as 1 in 4 adults and are set to continue rising, being associated with decreased life expectancy and reduced activity [4,5]. MSDs are prevalent throughout the life span and associated with early work retirement and reduced ability to participate socially [5]. In developed countries, they present the most significant proportion of lost productivity in the workplace, leading to a significant impact on the gross domestic product and health care costs [6,7].

In the United Kingdom, this poses a financial and societal challenge, costing >£4.76 billion (US $6.35 billion) of the UK National Health Service (NHS) resources and using up to 30% of primary care physician visits annually [8,9]. A freedom of information request has revealed that the average waiting time for NHS musculoskeletal outpatient physiotherapy services exceeded 6 weeks in the year to April 2019, with some patients waiting 4 months for routine appointments [10]. Longer waiting times can result in delays to physiotherapy services, with potentially detrimental effects on pain, disability, and quality of life [11,12]. The increasing proportion of burdens on public health services because of MSDs has highlighted the need for a targeted policy response [3,13].

Access to the right person, right place, first time is considered a key factor in improving musculoskeletal condition outcomes and reducing unwarranted variation in clinical pathways, such as unnecessary secondary care consultations and investigations [14]. Musculoskeletal triage as a single point of access is effective across various outcome measures, including user satisfaction, diagnostic agreement, appropriateness of referral, and reduction in patient waiting times [15]. Importantly, triage has also shown a reduction in costs across the musculoskeletal pathway, which is particularly crucial in overburdened health care systems, where triage can be performed effectively via several methods and by a range of clinicians [16-18].

Remote triage services such as telephone and video consultations or web-based or digital applications have the potential to reduce waiting times and musculoskeletal caseload [15,19]. Direct access to these services with initial assessments by physiotherapists may be a viable, cost-effective solution for managing the growing burden of MSD demand and workloads [19-22], with recent advances being made in digital primary care triage applications [23,24]. Some research has suggested that physiotherapy-led telephone triage is shown to be clinically as effective as usual care [21,25] and broadly acceptable to patients with MSDs seeking early physiotherapy advice [26]. However, barriers include the time required to reach a triage outcome, limited patient and professional trust, and interoperability problems. It should also be noted that comparisons between studies are hampered by variations in outcome measures and lack of randomization and statistical power, making any generalizations across health care settings problematic.

Mobile health (mHealth) technology has been proposed as a cost-effective solution for improving health care delivery [27,28]; however, this requires robustly tested and validated web-based triage platforms to signpost patients with MSDs to an appropriate level of care [19,29]. Standards and guidance for safe and effective implementation of mHealth apps have been published by several national and international organizations, all specifying a requirement for evidence of clinical safety and effectiveness [30-38]. A UK evidence standards framework specifically requires as the best practice standard a high-quality randomized controlled study or studies done in settings relevant to the UK health and social care system, which compare the digital health technology with a relevant comparator and demonstrate consistent benefits, including clinical outcomes in the target population [30].

To date, there is limited evidence regarding the use of web-based or digital triage platforms for MSDs specifically, and most investigations have focused on the performance of generic symptom checkers, covering a wide range of clinical presentations. However, the evidence from these studies concerning clinical- and cost-effectiveness, signposting to appropriate services, patient compliance, and safety was found to be weak or inconsistent [29]. A review of 36 primary care generic diagnostic and triage symptom checkers on web-based or mobile platforms found that appropriate triage advice was given for only 49% of the 688 vignette cases, with appropriate triage advice being given most frequently for emergency (63%) or urgent care (56%) and nonurgent care being accurate in only 30% of cases [39]. This finding is consistent with previous literature that evaluated web-based generic symptom checkers for self-diagnosis and triage [40]. A systematic review of generic health digital and web-based symptom checkers found algorithm-based triage to be inferior and more risk averse in providing appropriate triage advice compared with doctors and health practitioners [29]. Although most system developers consider this to be a safe approach, incorrect or unnecessary clinical escalations have been shown to adversely affect user trust and system adoption [41]. It is proposed that the digital assessment routing tool (DART) may overcome the limitations of existing generic symptom checkers and triage platforms by narrowing the scope of the tool and refining clinical algorithms to specifically address MSD presentations. The potential to improve triage efficiency and timely signposting of patients with MSDs to an appropriate level of care is potentially significant.

DART Overview

DART (Optima Health) is the first contact mHealth system designed specifically for the management of MSDs. The clinical algorithms are configured to provide the patient with a recommendation for the correct intervention level. The patient can self-assess using a computer, tablet, or smartphone. Alternatively, the content can be delivered by a remotely situated clinician or nonclinical administrator by telephone or video call. Once the affected body region has been selected, the patient is presented with a varying number of questions, depending on the nature of their symptoms and previous responses. Serious pathology is screened for, and appropriate signposting is given at the start of the assessment, with less urgent medical referrals being identified as the patient passes through the questioning. Algorithms are configured to match the provider’s clinical services based on evidence-based practice and sector-specific referral criteria. DART can be applied across any number of health care systems, including public and private services, and typically signposts to emergency or routine medical assessment, specific condition specialists, physiotherapy, self-management programs, or psychological support services. DART has an integral reporting function that allows the analysis of individual and amalgamated patient data to assess the system and clinical pathway performance (Figure 1).

Figure 1. Digital assessment routing tool mobile health system.
View this figure

Previous Work

This pilot study is part of a larger project, bringing DART from concept to implementation through a series of clinical and academic research work packages.

To assess the algorithm’s clinical validity, 2 reports were commissioned by Optima Health and undertaken by a panel of 5 consultant clinicians experienced in the musculoskeletal field, which comprised a consultant rheumatologist, a consultant orthopedic surgeon, a consultant sports and exercise physician and senior clinical lecturer, an honorary general practitioner (GP) in emergency care, and a consultant physiotherapist and academic lead. The first round of desktop evaluation comprised the panel inputting symptoms from 100 clinical scenarios (including red flags and complex presentations) into DART. The DART recommendation was then assessed by the panel as being correct, arguably correct, or disagree. Feedback from the panel was incorporated into a new iteration, leading to improved DART accuracy during the second review. On the basis of their opinion, the panel recommended that clinical validity was sufficient to allow DART to proceed to further research studies.

Real-world usability testing has been completed using an iterative convergent mixed methods design incorporating patient and public involvement [42], the results of which will be reported later in 2021. This study optimized usability in the final DART iteration that will be used in this pilot study and subsequent main trial.

Research Aims and Objectives

The aim of this study is to facilitate the delivery of a future trial by exploring key aspects of trial methodology, assessing the procedures, and collecting exploratory data to inform the design of a definitive, randomized, crossover, noninferiority trial to assess DART safety and effectiveness in an NHS primary care setting.

Primary Objective

The primary outcome measure is to collect exploratory data about the agreement of triage decisions made by the DART system and physiotherapy-led triage, which will provide a variance (SD) estimate required for the sample size calculation in the main trial.

Secondary Objectives

The secondary outcome measures are as follows:

  1. Evaluate the number of people who sign up and are retained, with a dropout rate and identification of when dropouts happen
  2. Evaluate the systems for randomization and data collection (effectiveness, process of implementation, allocation concealment, and bias)
  3. Identify the burden on the patient and therapist (treatment delay, DART procedure complexity, and additional questions)
  4. Identify barriers in the proposed intervention delivery processes


A pilot randomized, single-blinded, crossover, noninferiority trial will be conducted to compare the safety and efficacy of DART signposting with physiotherapy-led triage outcomes in an NHS primary care setting. This preliminary study was designed in accordance with the CONSORT guidelines for pilot and feasibility trials [43], CONSORT guidelines for equivalence and noninferiority randomized trials [44], and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist [45]. Moreover, the trial design will not influence the care or triage decisions made by the usual care clinicians. Participants will complete both a DART assessment and a physiotherapy-led triage assessment on the same day, with randomization determining the order in which this is done (Figure 2).

The outcomes available to the physiotherapist will be matched by those available within the DART, allowing direct comparison between the 2 assessment outcomes (Table 1). Following their DART assessment, the participant will use a web-based questionnaire to complete the system usability scale (SUS) [46] to measure user satisfaction.

Figure 2. Study design and participant flowchart. DART: digital assessment routing tool; MSD: musculoskeletal disorder; NHS: National Health Service.
View this figure
Table 1. All possible outcomes and suboutcomes from the digital assessment routing tool and physiotherapist-led triage assessment.
Medical careFCPa physiotherapistPhysiotherapy careRemote self-management
Emergency care (accident and emergency referral/NHS111)N/AbPost fracture or surgery physiotherapySupported self-management
Urgent primary care physician (GPc)Urgent FCPUrgent physiotherapy referralContinue self-management advice
Routine primary care physician (GP)Routine FCPRoutine physiotherapy referralWeb-based support material
Consultant reviewN/APhysiotherapy referral plus psychosocial supportDigital self-management

aFCP: first contact practitioner.

bN/A: not applicable.

cGP: general practitioner.


A total of 76 participants will be recruited for this pilot study using purposive sampling from a single NHS general practice. Monthly MSD referral volumes far exceed the number required for the study, which should support the completion of data collection within the anticipated study timescale. Patients with an MSD wishing to access support from the practice, either from a GP or physiotherapist, will be recruited into the study using an advert on the practice website, posters and leaflets in the GP waiting room, and by invitation from the practice administrators if the patient telephones in to make an appointment. Interested patients will be able to access the participant information sheet on the web or through hard copies available in practice, which will provide the researcher’s contact details and a request to contact them if they wish to participate (Multimedia Appendix 1). The researcher will make contact, answer any further questions, complete eligibility screening, and gain initial verbal consent. An appointment will then be made for the next available study clinic.

Inclusion Criteria

The study participant inclusion criteria were as follows: (1) aged >18 years, (2) able to speak and read English, (3) registered as a patient at the primary care practice, (4) having a current musculoskeletal condition for which they are seeking treatment, and (5) able to access the internet either by themselves or with the help of a family or friend.

Exclusion Criteria

The study participant exclusion criteria were as follows: (1) cognitive impairments or learning disabilities that limit a participant’s ability to follow study-related procedures, (2) unwillingness or inability to follow protocol-related procedures, and (3) Optima Health or Queen Mary University of London employees.

Informed Consent

Each participant will receive the participant information sheet (Multimedia Appendix 1), which outlines the purpose of the study and the nature of their participation. This includes information about the format of the interaction, potential risks, confidentiality and protection of their personal data, the anonymity of study findings, and their right to withdraw at any time without prejudice. The participant will be given the opportunity to raise any questions with the researcher during the formal consenting process when they attend the practice. Formal written consent will be completed by the researcher with the patient in the practice waiting room using a web-based form. Failure to provide consent will result in the participant receiving a usual care physiotherapy-led triage without a DART assessment.

Randomization and Blinding

Participants will be randomized to either (1) a DART assessment followed by a physiotherapist-led triage assessment or (2) a physiotherapist-led triage assessment followed by a DART assessment. This is to account for the order effects in the crossover design (Figure 2). This will be achieved by block randomization with permuted blocks of random size and without stratification factors to avoid selection bias and unequal arms [47-49]. After gaining consent, the researcher will open a sealed envelope that contains the randomization to be used for the participant. Allocation will be performed using a computer-generated list of random numbers, which will lead to a randomization sequence in Microsoft Excel 2020. The allocation ratio between the arms will be 1:1.

Blinding in this study will be ensured at 3 different points: (1) the physiotherapist leading the triage will be blinded to group allocation and DART assessment outcomes; (2) participants will be blinded to the DART assessment outcome and the physiotherapist triage outcome until they have completed both assessments and the SUS; and (3) the analysis and interpretation of the study results will be conducted by researchers blinded to the intervention group allocation.

To minimize potential bias created during the physiotherapist assessment, the physiotherapist will be required to minimize any information or advice they give to the participant until after they have completed their subjective assessment and arrived at a provisional management recommendation. This will include discussing any possible diagnoses or giving any condition management advice. Once the participant has completed both assessments and data collection is complete, they will return to the physiotherapist, who will then complete the objective assessment and continue with normal care.

Data Collection

Physiotherapy-Led Triage

Participants will receive a usual care physiotherapy-led triage assessment from a first contact practitioner (FCP). An FCP physiotherapist is a qualified autonomous clinician who can assess, diagnose, treat, and discharge a patient without a medical referral, where appropriate. They have completed additional postgraduate training to provide expert assessment, check for red flags, and provide advice on self-management. If needed, they prescribe medication, order investigations, refer for physiotherapy, or provide onward referral to secondary care services such as rheumatology or orthopedics [50]. As such, the FCP physiotherapist will provide a rigorous comparator against which to measure DART signposting recommendations.

The physiotherapist-led triage assessment will be completed within a 20-minute appointment. The management pathways available to the FCPs are matched by those that can be generated by DART to allow an objective comparison (Table 1). Participants may seek help elsewhere or opt out of the study at any point, which will not affect their usual physiotherapy-led care.

DART Assessment

Participants will access DART using a tablet device in a treatment room or a quiet area in the practice waiting room. The researcher will log onto DART and enter the participant study number but will have no further contact with the participant until they have completed their DART assessment. The DART assessment will be completed either before or after their appointment with the physiotherapist, depending on their randomization allocation. A unique reference ID will be generated in the DART system and linked to the participant’s study number. The participant will complete the DART assessment, which will result in a signposting outcome. This will not be visible to the participant but will be stored in the DART system for later retrieval and analysis. Thereafter, the participant will complete a web-based version of the SUS, capturing their experience of using DART. All participants will be given the physiotherapist triage assessment outcome by the physiotherapist, who will complete any associated management actions or referrals. Both assessments will be completed on the same day in close succession to reduce variations in clinical presentation.

Panel Assessment

An independent panel comprising 3 experts in musculoskeletal physiotherapy and general practice qualified to the consultant level will provide consensus on all disagreements between DART and physiotherapy-led triage that would yield a safety concern (Textbox 1). In addition, a random sample of cases will be assessed by the panel to decide which they consider to be the correct outcome based on the patient’s presentation from the physiotherapist’s clinical record of their assessment. The panel decision will provide the definitive gold standard outcome against which the physiotherapist outcome will be compared. The triage outcomes that are amended by the panel will be deemed the most appropriate outcome in preference to those from the physiotherapy assessment and will provide the outcome against which DART is compared.

Adverse triage outcomes in medical care, physiotherapy care, and self-management that would yield a safety concern, including a delay in intervention likely to result in a poor outcome.

Adverse triage outcomes that would yield a safety concern

  • Physiotherapy or self-management when it should have been urgent medical care (accident and emergency/NHS111 referral or urgent general practitioner)
  • Self-management when it should have been physiotherapy, first contact practitioner, or medical care
  • Routine care when it should have been urgent care
Textbox 1. Adverse triage outcomes in medical care, physiotherapy care, and self-management that would yield a safety concern, including a delay in intervention likely to result in a poor outcome.
Clinical Signposting Outcomes

The primary outcome measure will be assessment outcomes from physiotherapy-led triage and DART assessments (Table 1). These decisions are classified into 4 categories (medical care, FCP, physiotherapy care, and remote self-management), with further suboutcomes in each category. This allows for precise comparisons within each category and subcategory and is based on usual care signposting approaches in musculoskeletal clinical practice. Adverse triage outcomes that could yield safety concerns will also be identified (Textbox 1). Our aim for the pilot data is to explore the agreement rate between DART and physiotherapy-led triage across assessment outcomes, with additional analysis of suboutcomes. This will inform the suitability of the outcome measure and analysis for the future main trial. Additional data will be collected from DART and the physiotherapist. Demographic variables include age and gender. Clinical characteristics include the musculoskeletal pathway related to the body site.

Process Outcomes

The process outcomes will help to determine whether the implementation of the main trial design is feasible. Anonymized data will be collected, including the proportion of participants who showed interest in participating against those who were recruited into the study. Participant dropout rates at each stage of the trial (and, where possible, reasons for dropping out) will be collected. System process outcomes include errors reported in randomization, allocation concealment, blinding, or data collection. Any evidence for selection bias or other sources of bias will be explored. Other outcomes include the overall time burden: average times from initial participant contact to the first assessment, along with any treatment delay because of the additional time required to perform the research procedures. Technical problems with DART or other comments from the physiotherapist or participants that pose a barrier to intervention delivery or trial procedures will be explored.

Study Duration

Following ethics approval, 3 months will be allocated for the collection of data from the required 76 participants. The exact duration will be dependent on the volume of physiotherapy referrals entering the physiotherapy service and recruitment uptake.

Data Analysis

Calculation of Main Study Sample Size

As the nature of this pilot trial is to explore trial design and feasibility, a formal power-based sample size calculation will not be conducted. Our sample size is based on the estimated stepped rules of thumb from Whitehead et al [51] to demonstrate an extra small standardized effect size (σ<0.1) at a 90% powered main trial. The obtained variance estimate of the outcome measures from the pilot data will allow sample size calculation for the main trial using the noncentral T-distribution approach from Julious and Owen [52]. Our plan is to recruit a total of 76 participants over a 3-month period in this trial.

Primary Outcome

The primary analysis will involve an absolute agreement intraclass correlation (A,1) estimate with 95% CIs between DART and the clinician on triage outcomes with recommended management pathways, which will be calculated using SPSS, version 23 (IBM Corporation) and based on a single rating, 2-way mixed-effects model [53,54]. The analysis includes intention-to-treat and per-protocol analyses, with a subanalysis of categories (medical care, physiotherapy care, and self-management) and adverse triage outcomes. A predefined margin of correlation with an intraclass correlation ≥0.90 will be set and based on both clinical recommendations and the literature in which a correlation was demonstrated in clinical management decisions taken between telehealth and face-to-face physiotherapy [55]. In addition, diagnostic properties (sensitivity, specificity, and predictive values) will be calculated for DART and reported with 95% CI [56]. A descriptive summary of the variables includes the mean, SD or CI, median, and IQR as appropriate. The amalgamated SUS score will be reported as a mean and used to calculate a percentile score to allow benchmarking of the DART system usability against other systems [57].

Evaluation of Participant Recruitment and Retention

The number of participants referring to physiotherapy will be reported, including the proportion that meets the eligibility criteria, shows initial interest, and consents to participate in the trial. Participants who show initial interest in participating but do not consent will also be reported (and, where possible, reasons for not participating). Participants who opt out of the trial at any stage, such as between interventions, will also be reported. Differences in dropout rates between the intervention groups will be compared. As recruitment rates vary in randomized controlled trials [58], a conservative margin will be set. Dropouts seem unlikely to occur as there is only a single visit that will coincide with the physiotherapy appointment. Thus, a predefined criterion of 50% and 95% will be considered satisfactory for the proportion of identified participants recruited and retained, respectively.

Evaluation of Randomization, Allocation Concealment, Blinding, Data Collection, and Bias

Any underpinning errors in systems responsible for procedural randomization, allocation, blinding, or data collection will be reported. Baseline characteristics will be compared between the intervention groups (DART-physiotherapy-triage and physiotherapy-triage-DART) using analysis of variance for continuous variables or chi-square tests for categorical variables. Homogeneity between groups will indicate successful randomization and minimized risk of selection bias. Discrepancies in allocation concealment or unblinding of participants or therapists will be further compared between intervention groups and reported. Unsuccessful blinding is considered when physiotherapists become unblinded to group allocation or DART assessment advice.

Identification of Trial Burden and Barriers to Intervention Delivery

Administrative and physiotherapist burden in terms of additional time required to administer the study process will be assessed and extrapolated to understand the implications for the main study. Feedback from the service administrators and participating physiotherapists will also be recorded and reported.


This study is funded by Optima Health, the developers of DART, and therefore, is at risk of bias. The lead researcher (CL) is an employee of Optima Health and enrolled in a PhD program at the Queen Mary University of London. To mitigate bias, participants will be excluded if they are employees of Optima Health or the Queen Mary University of London. Participants will not have previously seen or used DART. Recruitment is through promotion directly to the medical center patients and not by the researcher contacting them using a database. There is no financial reward offered to people to participate in the study. After gaining of formal consent and ensuring that the participant has logged on to DART, the researcher has no further contact with participants, with data being collected through DART, by the physiotherapist, or the SUS web-based questionnaire. The expert panel will comprise senior musculoskeletal clinicians who are not employed in any form by Optima Health.

Risks and Benefits

There will be no form of physical intervention during this study, and participants will have no extra travel in addition to that required for their physiotherapy appointment. Normal care, as determined by the triage physiotherapist, will be followed in all cases, and participants will have full access to all existing clinical pathways available to them. Participants will not be given the DART signposting recommendation, so there will be no conflict with the recommendation given by the physiotherapist.

Data Management

Participants will have the right to withdraw from the study at any time. If they do, data collected up to the point they withdraw will be retained but not added to. Electronic and paper data will be managed and stored securely in accordance with the general data protection regulations. Study data will be collected outside the NHS firewall. Personal data collected in DART will be confined to age and sex at birth and linked to other research data using a DART system unique reference number. The data inputted by the participant during the DART assessment will be generated entirely by them, and no data will be extracted from their NHS records. DART system information security and data protection will be covered by Optima Health’s certification and compliance with Cyber Essentials Plus and ISO 27001. Data collected by the physiotherapist will comprise physiotherapist assessment recommendations and study numbers only. The web-based SUS questionnaire will only contain the participant’s study number. Research data will be stored separately to personal data and linked by a unique reference number that is only accessible to the researchers. Access to the participant’s physiotherapy assessment record forms a part of the study consenting process; however, only the assessment completed as a part of the study will be reviewed by the panel and no other part of the NHS record. The data collected during this pilot may be reused for a later definite randomized noninferiority trial in a deidentified format.

Ethics approval was submitted in June 2021. This study has been registered at NCT04904029. Recruitment will commence early 2022, and data collection is anticipated to last for up to 3 months. The results will be reported in a follow-up paper in 2022.


The demand for an mHealth system to correctly assess and signpost patients with MSDs has been demonstrated. Research into the patterned use of generic symptom checkers indicates that MSDs are among the most common reasons for accessing web-based or digital triage applications in primary care [59,60]. Studies regarding the effectiveness of generic symptom checkers have shown variable levels of system validity [23,24,29,40], and it cannot be assumed that these research findings translate to an mHealth system with a well-defined scope, such as DART with MSDs. To date, there are no published studies providing a proven methodology for evaluating the real-world validity of similar musculoskeletal mHealth systems. This pilot trial will explore the feasibility and study design for a future large-scale, noninferiority trial determining whether DART’s efficacy and safety are noninferior or not unacceptably worse to FCP physiotherapist-led triage, a usual practice comparator. During earlier DART development, the desktop validity of DART clinical algorithms was appraised as being good by an expert panel using vignettes; however, this is not representative of the real-world validation required for safe and effective implementation into routine clinical practice [61]. DART has demonstrated good usability through an iterative convergent mixed methods study, leading to the version to be used in this pilot trial. This pilot study protocol will give a greater understanding of how to assess the validity of an mHealth system such as DART within an NHS setting and provide a template for other researchers and developers to use across triage and referral mHealth systems.

Methodological Limitations

The purpose of this pilot is to validate or provide information for improvement in the study design to underpin the successful completion of the subsequent main trial. The crossover design can create a delay in the participant receiving normal care, albeit only a few minutes, and there is an increase in the amount of service administration time required. The most common errors in crossover trials are failure to adapt stratification in the order of treatments and analysis of the group rather than separately between sequence groups [62]. This is accounted for by the permuted block randomization and the independent analysis of results per group. A washout period between interventions is normally recommended to prevent carryover effects; however, there is no therapeutic effect from either DART or the physiotherapy-led assessment. The physiotherapist being asked to refrain from giving management advice as they assess the patient may be a change in normal practice for some clinicians but should not affect their signposting recommendation. Carryover effects may involve the participant becoming primed to answer questions about their health differently after gaining more insight into their health problem from having completed the previous assessment. Data analysis comprises assessing clinical agreement between the physiotherapist and DART system management recommendations. This study is not designed to examine the impact of differing DART recommendations on individual patient management and how this may positively or negatively affect care complexity, case duration, or cost. This would form the basis of a future implementation study to assess these factors across the entire MSD pathway.

Methodological Strengths

The findings from this pilot trial will constitute the design of a future noninferiority trial, as well as provide preliminary data on DART safety and effectiveness. The key benefit of a crossover design using the same participant for both arms is that differences in clinical presentation as a confounder are minimized. The large variety of MSD symptomology, patients’ general health, and psychosocial status means that using a study design where different participant results are compared, such as in a parallel study, would require an unacceptably large sample size. Assessing the participant twice within minutes reduces the risk of a change in symptoms that could lead to a different signposting recommendation between the physiotherapist and DART assessments. The use of real-world NHS participants rather than clinician testing using vignettes supports patient and public involvement, a crucial component of testing for any system designed to be used with patients. Piloting within the NHS ensures that DART is tested across a wide range of clinical presentations and patient demographics, including varying socioeconomic status, eHealth literacy, fluency of English speaking, age, and employment status.


This study has been funded by Optima Health. Support for this study has been provided by Haydock Medical Center and Northwest Boroughs Hospitals National Health Service Trust Musculoskeletal Service.

Authors' Contributions

CL conceived the study topic and wrote the protocol, with input from HHS and DM. CL and HHS will perform data analysis. CL drafted the manuscript with input from HHS, which was reviewed by DM, WM, and HHS before submission.

Conflicts of Interest

Optima Health has developed the DART system and is the owner of the associated intellectual property. The principal investigator (CL) is an employee of Optima Health and a PhD Research Student at Queen Mary University of London. HHS is employed by Optima Health to work as a research assistant on this study.

Multimedia Appendix 1

Patient information sheet.

DOCX File , 65 KB

  1. GBD 2016 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 2017 Sep 16;390(10100):1211-1259 [FREE Full text] [CrossRef] [Medline]
  2. GBD 2017 DALYs and HALE Collaborators. Global, regional, and national disability-adjusted life-years (DALYs) for 359 diseases and injuries and healthy life expectancy (HALE) for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018 Nov 10;392(10159):1859-1922 [FREE Full text] [CrossRef] [Medline]
  3. Sebbag E, Felten R, Sagez F, Sibilia J, Devilliers H, Arnaud L. The world-wide burden of musculoskeletal diseases: a systematic analysis of the World Health Organization Burden of Diseases Database. Ann Rheum Dis 2019 Jun;78(6):844-848. [CrossRef] [Medline]
  4. Musculoskeletal conditions. World Health Organization.   URL: [accessed 2021-11-26]
  5. GBD 2017 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018 Nov 10;392(10159):1789-1858 [FREE Full text] [CrossRef] [Medline]
  6. The impact of musculoskeletal disorders on Americans — Opportunities for action. BoneJoint Initiative USA.   URL: [accessed 2021-11-26]
  7. Board assurance prompt: improving the effectiveness of musculoskeletal health for NHS providers. Good Governance Institute.   URL: [accessed 2021-11-26]
  8. Ahmed N, Ahmed F, Rajeswaran G, Briggs TR, Gray M. The NHS must achieve better value from musculoskeletal services. Br J Hosp Med (Lond) 2017 Oct 02;78(10):544-545. [CrossRef] [Medline]
  9. Musculoskeletal conditions. NHS England.   URL: https:/​/www.​​ourwork/​clinical-policy/​ltc/​our-work-on-long-term-conditions/​musculoskeletal/​ [accessed 2021-11-26]
  10. Patients waiting up to four months to see a physiotherapist in England. Equipsme.   URL: [accessed 2021-11-26]
  11. Deslauriers S, Déry J, Proulx K, Laliberté M, Desmeules F, Feldman DE, et al. Effects of waiting for outpatient physiotherapy services in persons with musculoskeletal disorders: a systematic review. Disabil Rehabil 2021 Mar;43(5):611-620. [CrossRef] [Medline]
  12. Lewis AK, Harding KE, Snowdon DA, Taylor NF. Reducing wait time from referral to first visit for community outpatient services may contribute to better health outcomes: a systematic review. BMC Health Serv Res 2018 Nov 20;18(1):869 [FREE Full text] [CrossRef] [Medline]
  13. Steel N, Ford J, Newton J, Davis A, Vos T, Naghavi M, et al. Changes in health in the countries of the UK and 150 English Local Authority areas 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 2018 Nov 03;392(10158):1647-1661 [FREE Full text] [CrossRef] [Medline]
  14. Transforming musculoskeletal and orthopaedic elective care services. NHS England.   URL: [accessed 2021-11-26]
  15. Joseph C, Morrissey D, Abdur-Rahman M, Hussenbux A, Barton C. Musculoskeletal triage: a mixed methods study, integrating systematic review with expert and patient perspectives. Physiotherapy 2014 Dec;100(4):277-289. [CrossRef] [Medline]
  16. Oakley C, Shacklady C. The clinical effectiveness of the extended-scope physiotherapist role in musculoskeletal triage: a systematic review. Musculoskeletal Care 2015 Dec;13(4):204-221. [CrossRef] [Medline]
  17. Bornhöft L, Larsson ME, Thorn J. Physiotherapy in Primary Care Triage - the effects on utilization of medical services at primary health care clinics by patients and sub-groups of patients with musculoskeletal disorders: a case-control study. Physiother Theory Pract 2015 Jan;31(1):45-52. [CrossRef] [Medline]
  18. Bornhöft L, Thorn J, Svensson M, Nordeman L, Eggertsen R, Larsson ME. More cost-effective management of patients with musculoskeletal disorders in primary care after direct triaging to physiotherapists for initial assessment compared to initial general practitioner assessment. BMC Musculoskelet Disord 2019 May 01;20(1):186 [FREE Full text] [CrossRef] [Medline]
  19. Effectiveness of remote triage: a systematic review. Washington, DC: Evidence Synthesis Program, Health Services Research and Development Service, Office of Research and Development, Department of Veterans Affairs.   URL: [accessed 2021-11-26]
  20. Babatunde OO, Bishop A, Cottrell E, Jordan JL, Corp N, Humphries K, et al. A systematic review and evidence synthesis of non-medical triage, self-referral and direct access services for patients with musculoskeletal pain. PLoS One 2020 Jul 6;15(7):e0235364 [FREE Full text] [CrossRef] [Medline]
  21. Mallett R, Bakker E, Burton M. Is physiotherapy self-referral with telephone triage viable, cost-effective and beneficial to musculoskeletal outpatients in a primary care setting? Musculoskeletal Care 2014 Dec;12(4):251-260. [CrossRef] [Medline]
  22. Harland N, Blacklidge B. Physiotherapists and General Practitioners attitudes towards 'Physio Direct' phone based musculoskeletal Physiotherapy services: a national survey. Physiotherapy 2017 Jun;103(2):174-179. [CrossRef] [Medline]
  23. Verzantvoort N, Teunis T, Verheij T, van der Velden AW. Self-triage for acute primary care via a smartphone application: practical, safe and efficient? PLoS One 2018 Jun 26;13(6):e0199284 [FREE Full text] [CrossRef] [Medline]
  24. Cowie J, Calveley E, Bowers G, Bowers J. Evaluation of a digital consultation and self-care advice tool in primary care: a multi-methods study. Int J Environ Res Public Health 2018 May 02;15(5):896 [FREE Full text] [CrossRef] [Medline]
  25. Salisbury C, Montgomery AA, Hollinghurst S, Hopper C, Bishop A, Franchini A, et al. Effectiveness of PhysioDirect telephone assessment and advice services for patients with musculoskeletal problems: pragmatic randomised controlled trial. BMJ 2013 Jan 29;346:f43. [CrossRef] [Medline]
  26. Pearson J, Richardson J, Calnan M, Salisbury C, Foster NE. The acceptability to patients of PhysioDirect telephone assessment and advice services; a qualitative interview study. BMC Health Serv Res 2016 Mar 28;16:104 [FREE Full text] [CrossRef] [Medline]
  27. Personalised health and care 2020 using data and technology to transform outcomes for patients and citizens. HM Government. 2014.   URL: https:/​/assets.​​government/​uploads/​system/​uploads/​attachment_data/​file/​384650/​NIB_Report.​pdf [accessed 2021-11-26]
  28. Buechi R, Faes L, Bachmann LM, Thiel MA, Bodmer NS, Schmid MK, et al. Evidence assessing the diagnostic performance of medical smartphone apps: a systematic review and exploratory meta-analysis. BMJ Open 2017 Dec 14;7(12):e018280 [FREE Full text] [CrossRef] [Medline]
  29. Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Med J Aust 2020 Jun;212(11):514-519. [CrossRef] [Medline]
  30. Unsworth H, Dillon B, Collinson L, Powell H, Salmon M, Oladapo T, et al. The NICE Evidence Standards Framework for digital health and care technologies - Developing and maintaining an innovative evidence framework with global impact. Digit Health 2021 Jun 24;7:20552076211018617 [FREE Full text] [CrossRef] [Medline]
  31. Guidance A guide to good practice for digital and data-driven health technologies. GOV.UK. 2021.   URL: [accessed 2021-11-26]
  32. Ergonomics of human-system interaction — Part 210: human-centred design for interactive systems. ISO. 2019.   URL: [accessed 2021-11-26]
  33. Green Paper on mobile health (mHealth). World Health Organization.   URL: [accessed 2021-11-26]
  34. Van Velthoven MH, Smith J, Wells G, Brindley D. Digital health app development standards: a systematic review protocol. BMJ Open 2018 Aug 17;8(8):e022969 [FREE Full text] [CrossRef] [Medline]
  35. Medical devices: software applications (apps). GOV.UK.   URL: [accessed 2021-11-26]
  36. NHS digital, data and technology standards. NHS Digital.   URL: [accessed 2021-11-26]
  37. DCB0129: Clinical Risk Management: its application in the manufacture of health IT systems. NHS Digital.   URL: [accessed 2021-11-26]
  38. Digital Technology Assessment Criteria (DTAC). NHS (2021).   URL: [accessed 2021-11-26]
  39. Semigran H, Linder J, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 2015 Jul 08;351:h3480 [FREE Full text] [CrossRef] [Medline]
  40. Chambers D, Cantrell AJ, Johnson M, Preston L, Baxter SK, Booth A, et al. Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open 2019 Aug 01;9(8):e027743 [FREE Full text] [CrossRef] [Medline]
  41. Using technology to ease the burden on primary care. Healthwatch.   URL: https:/​/healthwatchenfield.​​wp-content/​uploads/​2019/​01/​Report_UsingTechnologyToEaseTheBurdenOnPrimaryCare.​pdf [accessed 2021-11-26]
  42. Lowe C, Hanuman Sing H, Browne M, Alwashmi MF, Marsh W, Morrissey D. Usability testing of a digital assessment routing tool: protocol for an iterative convergent mixed methods study. JMIR Res Protoc 2021 May 18;10(5):e27205 [FREE Full text] [CrossRef] [Medline]
  43. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, PAFS consensus group. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud 2016 Oct 21;2:64 [FREE Full text] [CrossRef] [Medline]
  44. Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG, CONSORT Group. Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA 2012 Dec 26;308(24):2594-2604. [CrossRef] [Medline]
  45. Eysenbach G. CONSORT-EHEALTH: implementation of a checklist for authors and editors to improve reporting of web-based and mobile randomized controlled trials. Stud Health Technol Inform 2013;192:657-661. [Medline]
  46. Brooke J. SUS: A 'Quick and Dirty' Usability Scale. In: Usability Evaluation In Industry. Boca Raton, Florida: CRC Press; 1996.
  47. Suresh K. An overview of randomization techniques: an unbiased assessment of outcome in clinical research. J Hum Reprod Sci 2011 Jan;4(1):8-11 [FREE Full text] [CrossRef] [Medline]
  48. Efird J. Blocked randomization with randomly selected block sizes. Int J Environ Res Public Health 2011 Jan;8(1):15-20 [FREE Full text] [CrossRef] [Medline]
  49. Broglio K. Randomization in clinical trials: permuted blocks and stratification. JAMA 2018 Jun 05;319(21):2223-2224. [CrossRef] [Medline]
  50. Elective care high impact interventions: first contact practitioner for MSK services. NHS.   URL: https:/​/www.​​publication/​elective-care-high-impact-interventions-first-contact-practitioner-for-msk-services/​ [accessed 2021-11-26]
  51. Whitehead AL, Julious SA, Cooper CL, Campbell MJ. Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Stat Methods Med Res 2016 Jun;25(3):1057-1073 [FREE Full text] [CrossRef] [Medline]
  52. Julious S, Owen R. Sample size calculations for clinical studies allowing for uncertainty about the variance. Pharm Stat 2006;5(1):29-37. [CrossRef] [Medline]
  53. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016 Jun;15(2):155-163 [FREE Full text] [CrossRef] [Medline]
  54. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods 1996;1(1):30-46. [CrossRef]
  55. Cottrell M, O'Leary SP, Swete-Kelly P, Elwell B, Hess S, Litchfield M, et al. Agreement between telehealth and in-person assessment of patients with chronic musculoskeletal conditions presenting to an advanced-practice physiotherapy screening clinic. Musculoskelet Sci Pract 2018 Dec;38:99-105. [CrossRef] [Medline]
  56. Parikh R, Mathai A, Parikh S, Chandra Sekhar G, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 2008;56(1):45-50 [FREE Full text] [CrossRef] [Medline]
  57. Sauro J. A Practical Guide to the System usability Scale: Background, Benchmarks & Best Practices IBSN. California: Createspace; 2012.
  58. Walters SJ, Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques RM, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open 2017 Mar 20;7(3):e015276 [FREE Full text] [CrossRef] [Medline]
  59. Eccles A, Hopper M, Turk A, Atherton H. Patient use of an online triage platform: a mixed-methods retrospective exploration in UK primary care. Br J Gen Pract 2019 May;69(682):e336-e344 [FREE Full text] [CrossRef] [Medline]
  60. Edwards H, Marques E, Hollingworth W, Horwood J, Farr M, Bernard E, et al. Use of a primary care online consultation system, by whom, when and why: evaluation of a pilot observational study in 36 general practices in South West England. BMJ Open 2017 Nov 22;7(11):e016901 [FREE Full text] [CrossRef] [Medline]
  61. Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med 2020;3:14 [FREE Full text] [CrossRef] [Medline]
  62. Wellek S, Blettner M. On the proper use of the crossover design in clinical trials: part 18 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2012 Apr;109(15):276-281 [FREE Full text] [CrossRef] [Medline]

CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth
DART: digital assessment routing tool
FCP: first contact practitioner
GP: general practitioner
mHealth: mobile health
MSD: musculoskeletal disorder
NHS: National Health Service
SUS: system usability scale

Edited by G Eysenbach; submitted 24.06.21; peer-reviewed by A Korchi, U Bork; comments to author 16.10.21; revised version received 29.10.21; accepted 04.11.21; published 13.12.21


©Cabella Lowe, Harry Hanuman Sing, William Marsh, Dylan Morrissey. Originally published in JMIR Research Protocols (, 13.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.