Published on in Vol 10, No 5 (2021): May

Preprints (earlier versions) of this paper are available at, first published .
Predicting Risk of Hospital Admission in Patients With Suspected COVID-19 in a Community Setting: Protocol for Development and Validation of a Multivariate Risk Prediction Tool

Predicting Risk of Hospital Admission in Patients With Suspected COVID-19 in a Community Setting: Protocol for Development and Validation of a Multivariate Risk Prediction Tool

Predicting Risk of Hospital Admission in Patients With Suspected COVID-19 in a Community Setting: Protocol for Development and Validation of a Multivariate Risk Prediction Tool


1Department of Surgery and Cancer, Imperial College London, London, United Kingdom

2Patient Safety Translational Research Centre, Institute of Global Health Innovation, Imperial College London, London, United Kingdom

3Center for Health Technology and Services Research / Department of Community Medicine, Health Information and Decision (CINTESIS/MEDCIDS), Faculty of Medicine, University of Porto, Porto, Portugal

4Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom

5Whole Systems Integrated Care, North West London Clinical Commissioning Group, London, United Kingdom

Corresponding Author:

Ana Belen Espinosa-Gonzalez, PhD

Department of Surgery and Cancer

Imperial College London

St Mary's Hospital

Praed St

London, W2 1NY

United Kingdom

Phone: 44 747 203 5868


Background: During the pandemic, remote consultations have become the norm for assessing patients with signs and symptoms of COVID-19 to decrease the risk of transmission. This has intensified the clinical uncertainty already experienced by primary care clinicians when assessing patients with suspected COVID-19 and has prompted the use of risk prediction scores, such as the National Early Warning Score (NEWS2), to assess severity and guide treatment. However, the risk prediction tools available have not been validated in a community setting and are not designed to capture the idiosyncrasies of COVID-19 infection.

Objective: The objective of this study is to produce a multivariate risk prediction tool, RECAP-V1 (Remote COVID-19 Assessment in Primary Care), to support primary care clinicians in the identification of those patients with COVID-19 that are at higher risk of deterioration and facilitate the early escalation of their treatment with the aim of improving patient outcomes.

Methods: The study follows a prospective cohort observational design, whereby patients presenting in primary care with signs and symptoms suggestive of COVID-19 will be followed and their data linked to hospital outcomes (hospital admission and death). Data collection will be carried out by primary care clinicians in four arms: North West London Clinical Commissioning Groups (NWL CCGs), Oxford-Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC), Covid Clinical Assessment Service (CCAS), and South East London CCGs (Doctaly platform). The study involves the use of an electronic template that incorporates a list of items (known as RECAP-V0) thought to be associated with disease outcome according to previous qualitative work. Data collected will be linked to patient outcomes in highly secure environments. We will then use multivariate logistic regression analyses for model development and validation.

Results: Recruitment of participants started in October 2020. Initially, only the NWL CCGs and RCGP RSC arms were active. As of March 24, 2021, we have recruited a combined sample of 3827 participants in these two arms. CCAS and Doctaly joined the study in February 2021, with CCAS starting the recruitment process on March 15, 2021. The first part of the analysis (RECAP-V1 model development) is planned to start in April 2021 using the first half of the NWL CCGs and RCGP RSC combined data set. Posteriorly, the model will be validated with the rest of the NWL CCGs and RCGP RSC data as well as the CCAS and Doctaly data sets. The study was approved by the Research Ethics Committee on May 27, 2020 (Integrated Research Application System number: 283024, Research Ethics Committee reference number: 20/NW/0266) and badged as National Institute of Health Research Urgent Public Health Study on October 14, 2020.

Conclusions: We believe the validated RECAP-V1 early warning score will be a valuable tool for the assessment of severity in patients with suspected COVID-19 in the community, either in face-to-face or remote consultations, and will facilitate the timely escalation of treatment with the potential to improve patient outcomes.

Trial Registration: ISRCTN registry ISRCTN13953727;

International Registered Report Identifier (IRRID): DERR1-10.2196/29072

JMIR Res Protoc 2021;10(5):e29072




During 2020, it became clear that assessment of the severity of COVID-19 infection required clinical tools specific to the condition and that repurposing tools such as the National Early Warning Score (NEWS2), designed for the early diagnosis of sepsis, would not be safe clinical practice [1]. The management of COVID-19 by clinicians is challenged by uncertainty about the disease progression [2]. There is evidence that a small percentage of patients present a dramatic deterioration of clinical status around the 8th to 10th day of disease, often associated with unperceived low oxygen saturations (known as “silent hypoxia”) that may require hospital and intensive care unit (ICU) admissions [3-5]. The inability to predict which patients will experience clinical deterioration adds an additional level of complexity to the clinical challenge and diagnostic uncertainty that general practitioners (GPs) have faced during the pandemic, particularly as most of the consultations are carried out remotely (usually by telephone and occasionally by video) to minimize the risk of transmission [6].

It was initially suggested that NEWS2 could be used to assess severity of patients with COVID-19 [7]. NEWS2 is calculated from patient’s temperature, pulse rate, respiratory rate, systolic blood pressure, pulse oximetry reading, and presence of new onset of acute confusion [8]. It is commonly used in hospital settings and ambulance service prior to transfer to hospital to assess the risk of deterioration of a patient [9]. However, NEWS2 seems to be a late indicator of decompensation, typically triggering within the last 12 hours before a transfer to ICU is considered necessary and, therefore, this limits its application and validity in a primary care or community care setting where an earlier warning would be preferred [9,10].

The Roth score (originally developed as a measure of breathlessness in cardiopulmonary disease [11]) was briefly considered by the Royal College of General Practitioners (RCGP) as possibly useful in the assessment of breathlessness when assessing patients with signs and symptoms of COVID-19 [12]. However, a rapid literature review concluded that the Roth score might have a low sensitivity (ie, a normal score in patients with “silent hypoxia”), and therefore should not be used by GPs when assessing patients over the phone or in video consultations [13].

Justification and Study Objective

This new condition and the forced shift toward remote consultations during the pandemic have increased the challenges and uncertainty commonly faced in general practice [6]. Primary care clinicians need a tool to guide the management of patients with suspected COVID-19 to be able to identify those whom they can reassure, those that need monitoring, and those that require urgent further assessment or referral to hospital. Even though the validity of NEWS2 for this purpose was a subject of intense debate during the height of the first COVID-19 wave, the score is still being used by primary care clinicians to assess patients prior to transfer to hospital [9]. The use of NEWS2 outside the hospital setting has not been validated, and it was not designed to capture the idiosyncrasies of COVID-19 infection. Therefore, there is need to develop an early warning score that incorporates key features of acute COVID-19 and that can be safely used by GPs when assessing patients remotely [14].

We reviewed the literature on COVID-19 early warning scores, then conducted a series of focus groups with 72 primary care clinicians (mostly GPs and including advanced nurse practitioners and paramedics) to derive elements that might form part of a suitable score, value sets, and appropriate SNOMED terms [15]. This paper describes the process of quantitative development and validation of the Remote COVID-19 Assessment in Primary Care (RECAP) score. The objective was to produce a multivariate risk prediction tool to facilitate the early identification, by primary care physicians and other clinicians working in the community, of those patients with COVID-19 that are at higher risk of becoming severely ill and inform the early escalation of their treatment, while also reducing unnecessary referrals in low-risk patients, with the aim of improving patient outcomes.

Study Design

This primary care data linkage study follows a prospective cohort observational design, whereby patients presenting in primary or community care with signs and symptoms suggestive of COVID-19 will be followed and their data linked with hospital outcomes, particularly focusing on hospital admission, ICU admission, and death. For data collection purposes, the initial set of items identified in earlier qualitative work [15], known as RECAP-V0, will be integrated into an electronic template to be used by primary care physicians (see Figure 1 for a summary of items included in RECAP-V0). This will enable the standardized recording of patients’ signs and symptoms and subsequent linkage with hospital and mortality data. Data collected will be used to develop and validate a multivariate regression model to predict hospital admission, ICU admission, and death.

Figure 1. Summary of RECAP-V0 items. Source: [15]. RECAP: Remote COVID-19 Assessment in Primary Care.
View this figure

Data Collection


The development of the RECAP score will require the use of primary and secondary data. The collection of patients’ signs and symptoms as they present in primary care requires the involvement of primary care clinicians, who will be asked to assess those patients with a clinical diagnosis of suspected COVID-19 using the RECAP electronic template.

The recruitment of clinicians (study sites) and patients (study participants) will be carried out by four different arms depending on clinician and participant location and service used to seek medical care:

  1. North West London (NWL) Clinical Commissioning Groups (CCGs) arm: this arm has its own integrated linked database (Whole Systems Integrated Care [WSIC]) and a secure environment (Imperial’s Clinical Analytics, Research and Evaluation [iCARE] secure environment) to hold the data. Recruitment of practices will be facilitated by the NWL clinical research network (CRN). General practitioners will use EMIS [16] or TPP SystmOne [17] electronic health record systems to capture patients’ data.
  2. RCGP Research and Surveillance Centre (RSC) arm: this is a national network of practices within the RCGP developed to contribute with data for disease surveillance and research [18], which is held in the Oxford RCGP Clinical Informatics Digital Hub (ORCHID) secure environment [19]. Subject to the patient’s consent, data from RSC network practices (collected from computerized medical record systems EMIS or TPP SystmOne, the United Kingdom’s most used systems, using Ardens RECAP electronic templates [20]) will be pseudonymized and extracted via a Wellbeing Software extraction system and linked to outcomes.
  3. Covid Clinical Assessment Service (CCAS) arm: this service is organized within the National Health Service (NHS) 111 Online service (managed by South Central Ambulance Service) for the clinical assessment and management of patients with a clinical diagnosis of suspected COVID-19. It is staffed by general practitioners and uses the Adastra electronic health record system [21]. Upon patients’ consent, the data collected will be transferred to ORCHID and linked to hospital outcomes.
  4. Doctaly arm: this private health care platform has been commissioned by South East London CCGs to provide a home monitoring service for patients with a diagnosis of COVID-19 (positive result in laboratory test) in South East London. Patients’ medical history and assessment data are collected using a chatbot via the WhatsApp mobile app. The questions asked via the Doctaly chatbot were designed to reflect the same concepts as the RECAP-V0 set. Data collected will be also transferred to the Oxford secure environment and linked to outcome data.

Figure 2 below depicts study data sources and data flow. Primary care data collected by practices in NWL and held in iCARE are already linked to hospital outcomes (ie, hospital admission, ICU admission, and death). Data held in the University of Oxford secure environment (RCGP RSC, CCAS, and Doctaly data) will be linked to outcome data contained in the Hospital Episode Statistics (HES) and Office of National Statistics (ONS) databases using an encrypted NHS number. Hospital admission and mortality data are available in HES and ONS; however, ICU admission information is not available.

Figure 2. Data flowchart. CCG: Clinical Commissioning Group; NHS: National Health Service; NWL: North West London; ORCHID: Oxford RCGP Clinical Informatics Digital Hub; RCGP: Royal College of General Practitioners; RSC: Research and Surveillance Centre; SE: South East.
View this figure
Selection Criteria

Our main cohort includes patients clinically diagnosed with COVID-19 that are being assessed and managed in primary care. Additional cohorts include patients with signs and symptoms suggestive of COVID-19 assessed by the NHS 111 CCAS and patients with established COVID-19 that are assessed as part of a primary care–led home monitoring service (Doctaly).

In the NWL, RCGP RSC, and CCAS arms, participants will be identified by primary care clinicians and enrolled in the study if they satisfy the following inclusion criteria:

  1. The patient is willing and able to provide informed consent for data linkage (exceptions are described in detail in the Overview section of the Results)
  2. The patient has signs and symptoms that are judged by the clinician to be suggestive of acute COVID-19 and time since onset of symptoms is ≤14 days.
  3. The participant is 18 years of age or older.
  4. The clinician is able to use the electronic template that contains the RECAP codes.
  5. Data collected by the clinician can be linked to the following hospital outcomes: hospital admission, ICU admission (only for NWL CCGs arm data), and hospital outcome (either discharge or cause of death).

For data collected in South East London CCGs (Doctaly) arm, the selection criteria consist of participant age (ie, 18 years old or older) and having a data sharing or consent procedure in place, since the other criteria are already satisfied (ie, patients are offered home monitoring after receiving a positive result from a COVID-19 test and the monitoring tool was specifically designed to include RECAP codes).

Template Development

In order to collect primary data from primary care or community care settings, the RECAP-V0 items that captured patients’ signs and symptoms along with other characteristics (sociodemographic information and comorbidities) are transferred into an electronic template using SNOMED and Read codes. These codes have been identified by the study team and collaborators and have been reviewed by NHSX, NHS England, and the UK Faculty of Clinical Informatics. The templates have been deployed for COVID-19 management via electronic health record systems—such as Ardens EMIS and SystmOne, TPP SystmOne, or Adastra—used by clinicians in GP practices, COVID-19 hubs, and CCAS, or via the patient-facing platform Doctaly. This will enable the collection of patients’ signs and symptoms in large data sets that will be stored in two secure environments (ORCHID and iCARE secure environments).

Sample Size

A total of 2880 participants will be necessary to develop a model with a minimum 85% specificity, assuming 10% prevalence of hospital admission and 6% missing data. We will split the sample into two consecutive groups, taking the first 50% of participants’ data for model development and the last 50% for model validation. CCAS will also collect 2880 participants as we wish to explore the hypothesis that, on account of case mix and spectrum bias, patients already triaged to the national service may require a separate model. We will then separately develop and validate a model for CCAS. Doctaly will provide an additional validation data set for RECAP-V1 score.

Data Analysis


A detailed statistical analysis plan (SAP) written before inspecting the data will be followed for analysis. The SAP provides a detailed description of data handling, RECAP-V1 model development and validation, and any planned secondary outcomes analysis. Given the complexity of issues to be addressed, including missing data not at random, potential correlations between clinical measurements; regression models and machine learning; and the relationships between the four different data sets, the SAP will be the subject of a separate article.

RECAP-V1 Early Warning Score Development and Validation

We will use multivariate logistic regressions to develop and validate the score. Table 1 contains a list of the items we included in the RECAP-V0 electronic template along with their SNOMED codes that will be used as inputs in the model.

The template has been designed to support the assessment of patients via both face-to-face and remote consultations; however, we anticipate that there are certain observations, such as respiratory rate or oxygen saturation, whose recording in remote consultations may be challenging. Therefore, we included information on patients’ symptoms that could be used as a proxy of quantitative items if they were unavailable. The factors for the model (predictor variables) can then be summarized as follows: heart rate, respiratory rate or shortness of breath, trajectory of breathlessness, oxygen saturation or level of tiredness, temperature or feeling feverish, days from onset of symptoms, muscle aches, and cognitive decline. Moreover, we will extract other patient characteristics such as age, gender, body mass index, ethnicity, presence of comorbidities (eg, diabetes, hypertension, coronary heart disease, and chronic kidney disease), and whether the patient is or has been on a COVID-19 shielding list. During the conduct of the study, the QCOVID score [22] has been adopted as a measure of baseline risk and used to populate the COVID-19 shielding list in health record systems [23]. We expect that patient characteristics ought to be able to be represented by the shielding term and will test this hypothesis. Missing data will be handled with standard methodologies for the multiple imputation of missing data [24].

Regarding the outputs of the model, we are interested in hospital admission (defined as an overnight hospital stay within 28 days of onset of symptoms), ICU admission (only available in NWL’s WSIC/iCARE database), and death (either at the hospital or at home within 28 days of onset of symptoms).

We will also conduct exploratory analyses, using machine learning algorithms for outcome prediction (nonlinear classifiers) including random forest, gradient boosting, and neural networks, alongside machine learning approaches for imputation of missing data.

Table 1. RECAP-V0 template items.
Variable nameMeasurement levelDescription/parameterConcept ID
Heart rateContinuous
  • Heart rate measured at systemic artery (observable entity)
  • 78564009
Respiratory RateContinuous
  • Respiratory rate (observable entity)
  • 86290005
Shortness of breathNominal
  • No breathlessness (situation)
  • Breathless, moderate exertion (finding)
  • Breathless, mild exertion (finding)
  • Unable to complete a sentence in one breath (finding)
  • 161938003
  • 161939006
  • 161940008
  • 407588003
Trajectory of breathlessnessNominal
  • Patient condition improved (finding)
  • Patient condition unchanged (finding)
  • Patient condition deteriorating (finding)
  • Symptom very severe (finding)
  • 268910001
  • 359748005
  • 275723000
  • 162471005
Oxygen saturation (rest)Continuous
  • Peripheral blood oxygen saturation on room air at rest (observable entity)
  • 866661000000106
Oxygen saturation (exertion)Continuous
  • Peripheral blood oxygen saturation on room air on exertion (observable entity)
  • 866681000000102
Level of tirednessNominal
  • Not tired (situation)
  • Fatigue (finding)
  • Unable to get on and off a bed (finding)
  • 161869003
  • 84229001
  • 301663005
  • Tympanic temperature (observable entity)
  • 703421000
Feeling feverishNominal
  • No temperature (situation)
  • Feeling hot (finding)
  • Rigor symptom (finding)
  • 161851007
  • 373904004
  • 248457000
Date of onset of symptomsContinuous
  • Date of onset of symptoms (observable entity)
  • 520191000000103
Muscle achesNominal
  • Myalgia (finding)
  • 68962001
Cognitive declineNominal
  • Mentally alert (finding)
  • Clouded consciousness (finding)
  • Acute confusion (finding)
  • On examination, decreased level of consciousness (finding)
  • 248234008
  • 40917007
  • 130987000
  • 417473004
COVID-19 shielding list as defined using the QCOVID score [23]Nominal
  • High-risk category for developing complication from COVID-19 infection (finding)
  • 1300561000000107
  • Current chronological age (observable entity)
  • 424144002
Body mass indexContinuous
  • Body mass index (observable entity)
  • 60621009
Patient sexNominal
  • Patient sex (observable entity)
  • 184100006
  • Diabetes (disorder)
  • 73211009
  • Hypertension (disorder)
  • 38341003
Coronary heart diseaseNominal
  • Coronary heart disease (disorder)
  • 53741008
Chronic kidney diseaseNominal
  • Chronic kidney disease (disorder)
  • 709044004
Ethnicity/related nationality dataNominal
  • Ethnicity/related nationality data (observable entity)
  • 186034007
Participant’s consentNominal
  • Consent given to participate in research study (finding)
  • 873771000000107


Recruitment started in October 2020. Initially, only the NWL CCGs and RCGP RSC arms were actively recruiting. In order to engage clinicians with the study and facilitate participation, we have run three webinars or training workshops where the study team described the study objectives and deadlines, and provided a detailed description of the RECAP template and how it would be used at the clinical front line. These webinars took place in October 2020 and January 2021. The study team has been in close contact with participating practices through the Imperial College London arm office, which has overall responsibility for the project and is directly in charge of data collection in North West London, and the University of Oxford arm office, which has direct responsibility for data collection from RCGP RSC practices. The CCAS and Doctaly platform arms joined the study in February 2021.

The initial data set to be used for model development will consist of RCGP RSC and NWL arms data and will be complete by the end of March 2021; this includes the primary care data on recruited patients’ signs and symptoms collected by these two arms linked to outcomes 28 days later. Two stages of data extraction and analysis of this integrated data set have been identified: first, RECAP-V1 development using the first half of the data set will start in April 2021, and second, model validation using the second half of the data set will follow. As of March 24, 2021, we have recruited a combined sample of 3827 participants (173 active primary care practices enrolled) in these two arms. The CCAS arm started the recruitment process on March 15, 2021, and we expect to reach the desired sample size in this arm (2880 participants) by the end of May 2021. Data sharing agreements are being developed to access data that have already been collected from around 1400 participants by clinicians using the Doctaly platform. The CCAS and Doctaly platform data sets will be used to validate the RECAP-V1 model and will be analyzed independently. Once we have produced the model, and subject to findings, the RECAP-V1 score will be ready to be deployed and used by clinicians to guide the management of patients with suspected COVID-19 according to their predicted risk.

The study is sponsored by Imperial College London and ethical approval was granted by the North West-Greater Manchester East Research Ethics Committee and Health Research Authority on May 27, 2020 (Integrated Research Application System number: 283024, Research Ethics Committee reference number: 20/NW/0266). An amendment to include the CCAS and Doctaly arms was approved on February 1, 2021. Due to the low risk associated with participation in this study and the remote nature (telephone/video consultation) of most patient encounters, the review committee agreed that obtaining verbal consent for data linkage was acceptable.

To access and link retrospective data collected by the NHS 111 CCAS and Doctaly platforms in South East London (ie, data that have already been collected by the services prior to study participation) we requested the last ethics amendment submitted to be assessed under the Control of Patient Information (COPI) notice, data sharing provisions that allow public authorities and research bodies the use of COVID-19–relevant patient-level data without the need for patients’ explicit consent [25]. For NHS 111 CCAS prospective data—that is, data from patients seeking medical care after the RECAP template has been installed in Adastra—we will apply the same mechanism to seek consent that has been followed in the NWL and RCGP RSC arms, and patients in the clinical queue will receive an SMS text message with information on the study and how to participate.

Data and all appropriate documentation will be stored in accordance with General Data Protection Regulation (Data Protection Act 2018) for a minimum of 10 years after the completion of the study, including the follow-up period. Participants can withdraw from the study at any point by informing their GP or a member of the study team. They will be asked whether the data obtained before withdrawal can be retained for analysis or they would like their data to be destroyed instead.

The study was included in the National Institute of Health Research (NIHR) Clinical Research Network Portfolio (CPMS number: 45890) on September 25, 2020, and badged as NIHR Urgent Public Health Study on October 14, 2020. These measures facilitate the rapid mobilization of resources from NIHR and clinical research networks toward study dissemination and participant recruitment and help ensure that high-quality data can be collected on a timely basis (trial registration number: ISRCTN13953727).

Dissemination Plan and Patient and Public Involvement

The RECAP-V0 template has already been disseminated nationally through CRNs facilitating the standardization of clinical records of patients with COVID-19. Its use has been encouraged through webinars and invited talks arranged by CRNs. Once the risk prediction tool has been developed and validated, we will seek endorsement for it to be incorporated into the electronic template to support clinical decision making when assessing patients with COVID-19. We expect to reach wide national and international dissemination of the RECAP-V1 risk prediction tool through submission to academic journals and international conferences.

Patient and public participation has been incorporated at different stages of the project. Patients were involved, along with primary care clinicians, in the qualitative study carried out to identify the set of elements to be included in the RECAP-V0 [15]. Once the RECAP-V1 tool has been developed and validated, public participation will be sought to coproduce project lay summaries, which will be valuable to disseminate the study findings to a wider audience.

The RECAP-V1 early warning score will, we anticipate, facilitate the stratification of the severity of patients with COVID-19 and their appropriate management and escalation of treatment. This study also promotes the standardization of assessment of patients with COVID-19, of collection of medical records, and record keeping thanks to the electronic templates developed, which can all have a positive impact in patients’ care, continuity, and safety [26]. Moreover, since November 2020, NHS England and NHS Improvement have led the establishment of the COVID Oximetry @home pathway, offered to patients with symptomatic COVID-19 who are aged 65 years or older or who are clinically extremely vulnerable to COVID-19 [27]. This service is being delivered by general practice, with referrals from NHS 111, CCAS, and hospital emergency departments, and involves an initial face-to-face or remote clinical assessment followed by monitoring of home oximetry readings for 14 days, to aid early recognition of deterioration. Items in the RECAP-V1 risk prediction tool in development are consistent with suggested clinical markers for triage on this pathway, and we anticipate that the tool will provide a unified quantitative risk score that will fit the monitoring needs of the service. Finally, we would like to emphasize the value of the study as an example of a digital clinical study, whose practice has been upheld by national research institutions on the basis of its cost-effectiveness and patient-centeredness due to the potential to recruit participants and collect large amounts of data with minimum inconvenience for the patient [28]. This is an example of leveraging the power of the NHS as a learning health system [29].


This study is funded by the Community Jameel and the Imperial College President’s Excellence Fund, the Economic and Social Research Council, the UK Research and Innovation, Health Data Research UK, the NIHR Imperial Biomedical Research Centre (the study was enabled by the Imperial Clinical Analytics Research and Evaluation (iCARE) environment and Whole Systems Integrated Care and used the iCARE and WSIC team and data resources,, the NIHR Oxford Biomedical Research Centre and the NIHR Imperial Patient Safety Translational Research Centre. We also thank our collaborators at EMIS, NWL CCGs, TPP and Wellbeing medical systems. Finally, we would like to acknowledge patients and practices of the Oxford-RCGP RSC, NWL CCGs, CCAS and SE London CCGs, who allowed data sharing for research, and the NIHR Clinical Research Networks for their help in enrolling and liaising with practices.

Conflicts of Interest

SdL is the Director of the RCGP RSC. All other authors declare no conflicts.

Multimedia Appendix 1

Peer review document.

PDF File (Adobe PDF File), 125 KB

Multimedia Appendix 2

Peer review comments.

PDF File (Adobe PDF File), 217 KB


  1. Carr E, Bendayan R, Bean D, Stammers M, Wang W, Zhang H, et al. Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study. BMC Med 2021 Jan 21;19(1):23 [FREE Full text] [CrossRef] [Medline]
  2. Koffman J, Gross J, Etkind SN, Selman L. Uncertainty and COVID-19: how are we to respond? J R Soc Med 2020 Jun;113(6):211-216. [CrossRef] [Medline]
  3. Chen S, Feng H, Xu H, Huang S, Sun J, Zhou L, et al. Patterns of Deterioration in Moderate Patients With COVID-19 From Jan 2020 to Mar 2020: A Multi-Center, Retrospective Cohort Study in China. Front Med (Lausanne) 2020;7:567296 [FREE Full text] [CrossRef] [Medline]
  4. Bickler PE, Feiner JR, Lipnick MS, McKleroy W. 'Silent' Presentation of Hypoxemia and Cardiorespiratory Compensation in COVID-19. Anesthesiology 2021 Feb 01;134(2):262-269 [FREE Full text] [CrossRef] [Medline]
  5. Wilkerson RG, Adler JD, Shah NG, Brown R. Silent hypoxia: A harbinger of clinical deterioration in patients with COVID-19. Am J Emerg Med 2020 Oct;38(10):2243.e5-2243.e6 [FREE Full text] [CrossRef] [Medline]
  6. Greenhalgh T, Koh GCH, Car J. Covid-19: a remote assessment in primary care. BMJ 2020 Mar 25;368:m1182. [CrossRef] [Medline]
  7. COVID-19 rapid guideline: critical care in adults. National Institute for Health and Care Excellence (NICE). 2020 Mar 20.   URL: [accessed 2021-05-11]
  8. National Early Warning Score (NEWS) 2. Royal College of Physicians (UK).   URL: [accessed 2021-03-07]
  9. Patel R, Nugawela MD, Edwards HB, Richards A, Le Roux H, Pullyblank A, et al. Can early warning scores identify deteriorating patients in pre-hospital settings? A systematic review. Resuscitation 2018 Nov;132:101-111. [CrossRef] [Medline]
  10. Finnikin S, Wilke V. What's behind the NEWS? National Early Warning Scores in primary care. Br J Gen Pract 2020 Jun;70(695):272-273 [FREE Full text] [CrossRef] [Medline]
  11. Chorin E, Padegimas A, Havakuk O, Birati EY, Shacham Y, Milman A, et al. Assessment of Respiratory Distress by the Roth Score. Clin Cardiol 2016 Nov;39(11):636-639 [FREE Full text] [CrossRef] [Medline]
  12. RCGP clarification on the use of Roth scores in the assessment of patients with potential COVID-19. Royal College of General Practitioners. 2020 Apr 2.   URL: https:/​/www.​​about-us/​news/​2020/​april/​rcgp-clarification-on-the-use-of-roth-scores-in-the-assessment-of-patients-with-potential-covid-19 [accessed 2021-03-07]
  13. Greenhalgh T. Should the Roth score be used in the remote assessment of patients with possible COVID-19? Centre for Evidence-Based Medicine, Oxford. The Centre for Evidence-Based Medicine, Oxford.   URL: [accessed 2021-05-11]
  14. Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ 2020 Apr 07;369:m1328 [FREE Full text] [CrossRef] [Medline]
  15. Greenhalgh T, Thompson P, Weiringa S, Neves AL, Husain L, Dunlop M, et al. What items should be included in an early warning score for remote assessment of suspected COVID-19? qualitative and Delphi study. BMJ Open 2020 Nov 12;10(11):e042626. [CrossRef] [Medline]
  16. EMIS Health.   URL: [accessed 2021-03-31]
  17. Clinical software to revolutionise healthcare. The Phoenix Partnership (TTP).   URL: [accessed 2021-03-31]
  18. RCGP Research and Surveillance Centre (RSC). Royal College of General Practitioners.   URL: [accessed 2021-03-07]
  19. de Lusignan S, Lopez Bernal J, Byford R, Amirthalingam G, Ferreira F, Akinyemi O, et al. Influenza and Respiratory Virus Surveillance, Vaccine Uptake, and Effectiveness at a Time of Cocirculating COVID-19: Protocol for the English Primary Care Sentinel System for 2020-2021. JMIR Public Health Surveill 2021 Feb 19;7(2):e24341 [FREE Full text] [CrossRef] [Medline]
  20. Ardens Health Informatics Ltd.   URL: [accessed 2021-03-31]
  21. Adastra Clinical Patient Management.   URL: [accessed 2021-03-31]
  22. Clift AK, Coupland CAC, Keogh RH, Diaz-Ordaz K, Williamson E, Harrison EM, et al. Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ 2020 Oct 20;371:m3731 [FREE Full text] [CrossRef] [Medline]
  23. COVID-19 Population Risk Assessment. NHS Digital.   URL: [accessed 2021-03-21]
  24. Lang KM, Little TD. Principled Missing Data Treatments. Prev Sci 2018 Apr;19(3):284-294. [CrossRef] [Medline]
  25. Control of patient information (COPI) notice. NHS Digital.   URL: https:/​/digital.​​coronavirus/​coronavirus-covid-19-response-information-governance-hub/​control-of-patient-information-copi-notice [accessed 2021-03-21]
  26. Tanner C, Gans D, White J, Nath R, Pohl J. Electronic health records and patient safety: co-occurrence of early EHR implementation with patient safety practices in primary care settings. Appl Clin Inform 2015;6(1):136-147 [FREE Full text] [CrossRef] [Medline]
  27. COVID Oximetry @home - digital and data services. NHS Digital.   URL: [accessed 2021-03-09]
  28. Inan OT, Tenaerts P, Prindiville SA, Reynolds HR, Dizon DS, Cooper-Arnold K, et al. Digitizing clinical trials. NPJ Digit Med 2020;3:101 [FREE Full text] [CrossRef] [Medline]
  29. Scobie S, Castle-Clarke S. Implementing learning health systems in the UK NHS: Policy actions to improve collaboration and transparency and support innovation and better use of analytics. Learn Health Syst 2020;4(1):e10209 [FREE Full text] [CrossRef] [Medline]

CCG: Clinical Commissioning Group
COPI: Control of Patient Information
CRN: clinical research network
GP: general practitioner
HES: Hospital Episode Statistics
iCARE: Imperial’s Clinical Analytics, Research and Evaluation database
ICU: intensive care unit
NEWS2: National Early Warning Score
NHS: National Health Service
NIHR: National Institute of Health Research
ONS: Office of National Statistics
ORCHID: Oxford RCGP Clinical Informatics Digital Hub
RCGP: Royal College of General Practitioners
RECAP: Remote COVID-19 Assessment in Primary Care
RSC: Research and Surveillance Centre
SAP: statistical analysis plan
WSIC: Whole Systems Integrated Care

Edited by T Derrick; this paper was peer reviewed by the Peer Review Office of Imperial College London and Imperial College Healthcare NHS Trust, as well as the UKRI Economic and Social Research Council. See the Multimedia Appendix for the peer-review reports; submitted 24.03.21; accepted 01.04.21; published 25.05.21


©Ana Belen Espinosa-Gonzalez, Ana Luisa Neves, Francesca Fiorentino, Denys Prociuk, Laiba Husain, Sonny Christian Ramtale, Emma Mi, Ella Mi, Jack Macartney, Sneha N Anand, Julian Sherlock, Kavitha Saravanakumar, Erik Mayer, Simon de Lusignan, Trisha Greenhalgh, Brendan C Delaney. Originally published in JMIR Research Protocols (, 25.05.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.