Implementation and Effects of Risk-Dependent Obstetric Care in the Netherlands (Expect Study II): Protocol for an Impact Study

Background Recently, validated risk models predicting adverse obstetric outcomes combined with risk-dependent care paths have been made available for early antenatal care in the southeastern part of the Netherlands. This study will evaluate implementation progress and impact of the new approach in obstetric care. Objective The objective of this paper is to describe the design of a study evaluating the impact of implementing risk-dependent care. Validated first-trimester prediction models are embedded in daily clinical practice and combined with risk-dependent obstetric care paths. Methods A multicenter prospective cohort study consisting of women who receive risk-dependent care is being performed from April 2017 to April 2018 (Expect Study II). Obstetric risk profiles will be calculated using a Web-based tool, the Expect prediction tool. The primary outcomes are the adherence of health care professionals and compliance of women. Secondary outcomes are patient satisfaction and cost-effectiveness. Outcome measures will be established using Web-based questionnaires. The secondary outcomes of the risk-dependent care cohort (Expect II) will be compared with the outcomes of a similar prospective cohort (Expect I). Women of this similar cohort received former care-as-usual and were prospectively included between July 1, 2013 and December 31, 2015 (Expect I). Results Currently, women are being recruited for the Expect Study II, and a total of 300 women are enrolled. Conclusions This study will provide information about the implementation and impact of a new approach in obstetric care using prediction models and risk-dependent obstetric care paths. Trial Registration Netherlands Trial Register NTR4143; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=4143 (Archived by WebCite at http://www.webcitation.org/6t8ijtpd9)


Rebuttal Spaanderman et al: External validity and impact of first-trimester obstetric prediction rules in the Netherlands. 50-50200-98-063
Reviewer id R 2012 524

Objective problem definition and assignment
The reviewer is overall content but expresses two concerns.
1. One concern is that the "test characteristics of most first trimester rules are generally believed to be not good enough (ie both sensitive and specific enough) to use clinically, especially given the low prevalence for many outcomes".
Unfortunately, the reviewer does not produce any literature showing that this is a general belief, or what arguments it is based upon.
Furthermore, despite what the reviewer implies, sensitivity and specificity are not fixed features of a prediction rule; instead they depend on which cut-off point is chosen. Even if, at a certain cut-off, sensitivity and specificity are not both optimal, a prediction test can be very useful in clinical practice. For instance, for outcomes with a low prevalence of 5% such as those in our study, a predictive test with a specificity of 98% and a sensitivity of only 70% will still detect more than 2/3s of the women with the prospective outcome although only a limited fraction of the women (5%) are test positive; of the test positives about 65% will have the outcome (if not prevented).

2.
A second concern of the reviewer is that he or she does not "know of any intervention or care path that will decrease the risk of preterm birth, preeclampsia, SGA or GDM".
We would like to refer to a paragraph under "Relevance" (next to last) where we mention a number of interventions that have been proven, mostly in RCTs, to either reduce either the risk of these adverse outcomes or the severity of morbidity associated with these outcomes. Although the net effect of each of these individual interventions may be restricted, we believe that their combined effect can lead to significant reductions in maternal and child morbidity and mortality. It is important to mention also that interventions can also be initiated earlier in comparison with the present situation in which the VIL (Verloskundige Indicatie Lijst) is used. Therefore better preventive effects can be obtained.
We do not agree that care paths should be developed before the start of the study, because the prediction rules we identified so far are promising and the ingredients for effective care paths are available. They can be developed during the 1 st part of the study (validation study).

Strategy
The reviewer refers to point 2 above (available effective interventions). We have explained why we think that beneficial effects can be achieved. In addition, prediction based care paths can reduce health care costs as (usually more expensive) 2 nd line care actions will be focused on specific risks while women can remain under supervision of a 1 st line midwife for general follow-up.

Project group
Our project group indeed comprises all relevant disciplines.

Feasibility
Feasibility is good according to the reviewer.

Overall quality assessment
We have argued that beneficial effects can be achieved by use of combinations of effective interventions (see earlier). Therefore we think that judging overall quality as "M" is unjustified. That rating is also surprising in view of the fact that the four previous component ratings range from S to VG, and the average would be G.

Budget
Is reasonable according to the reviewer.

Reviewer id R 2012 525
We have serious problems with the work of this reviewer. His or her comments give us the strong impression that the application was either not carefully read, misapprehended, or both. In our opinion, we think it would be very unfair to give this reviewer's assessment much weight in the overall judgment of the application. We feel that his or her handling of the application is out of balance with the dedication, time and thought given to it by our team. We hope that the below considerations will make this further clear.

Objective, problem definition and assignment
The reviewer asks: "Will the predicting value of each article be assessed selarately or will rules be selected from the articles". Articles usually have no predicting value; and we think it is obvious from the proposal text (e.g. Strategy, par. 1.1, Model selection phase) that models will be selected from articles.
Another question is: "Will SGA, PTB, GDM, and PET studied independently?" As shown under 1.1, Model selection phase, published prediction rules predict one outcome each and it would be very illogical to validate prediction rules for other purposes than for which they were developed. Under 1.3, Data analyses, first sentence, it can be read that each prediction rule will be evaluated separately.
The reviewer further states that "It is unclear which models for predicting rules will be used". We are not planning to predict rules with models. Furthermore, we have made clear reference (again under Strategy, par. 1.1, Model selection phase) to published models that are promising and which will be evaluated in the validation study. More prediction rules may be published in the meantime (NB, 5 were published in 2011 alone) and it would be unwise not to leave room for any additional prediction rules to be included in the validation study.

Strategy
The reviewer seems to be basing his or her judgment on only 1 argument, namely that "it is uncertain how many participants will have access to the internet". Internet penetration rate is high in the Netherlands: 90% for all ages (December 2011), and virtually 100% among women of childbearing age.
Although the question is valid, the reviewer's judgment (M) is however completely out of balance, since all other features of the strategy seem to be ignored. (Clarity, adequacy in terms of problem definition/assignment, adequacy of chosen methods and analyses, if there is a target group: the way in which the strategy reflects the factors gender, age, ethnicity and/or other characteristics relevant to the objective; degree of collaboration with intermediate and/or ultimate target group (the patient/client perspective).)

Feasibility
The reviewer rates this aspect as "unsatisfactory" because he or she thinks this part is "difficult to assess as it is uncertain how many studies will be used to select prediction rules". We do not understand why feasibility would be difficult to assess when the eventual number of prediction rules to be selected is still uncertain. The variables contained in the prediction models are required to be easily collectable either via routine care or questionnaire, so the number of studies/prediction rules is not relevant for feasibility.
Here again the reviewer ignores other aspects that should be evaluated (possibility to achieve the objectives using this strategy, availability of facilities/staff, realistic phasing and timetable).
Therefore, the judgment does not indicate that the application was read carefully, and it is again out of balance.

Objective, problem definition and assignment
In the appendix we will give more details about the prediction rules that we found to be promising at the time of the writing of the application. We have chosen the outcomes PTB, SGA, LGA, GDM, PE(T)/HELLP syndrome, and asphyxia as eligible outcomes for prediction rules to be evaluated in the validation study (1.1 model selection phase), because they are relatively prevalent and important contributors to maternal and neonatal morbidity and mortality. We do not agree with the reviewer that the eligible outcomes in the validation study are mainly maternal. All have a possible impact on child health, and 4 of the 6 outcomes are predominantly child outcomes (PTB, SGA, LGA, asphyxia).
The reviewer is right in stating that the composite outcome in the impact study is neonatal. We chose not to combine specifically maternal and neonatal components in the composite outcome as interpretation of values for such an outcome is problematic. Therefore we separated maternal from neonatal outcomes, and although the composite outcome was the one we based our sample size upon, we defined a number of important secondary outcomes focused on maternal health.
The component outcomes in the composite outcome were chosen in such a way that they all had a clear association with neonatal morbidity as well as mortality. The components are: perinatal death, asphyxia, NICU admission, SGA p2.3, very preterm birth. Each of these outcomes alone is not prevalent enough to be used as an individual primary outcome. We chose not to use SGA 10.0 or preterm birth (<37 wks) as their association with morbidity / mortality is weaker than SGA 2.3 and very preterm birth (<32 wks) and at the same time they would put to much weight into the composite outcome (prevalences of 9-10% and 7.7% instead of 2.2% and 1.5%, respectively).

Strategy
The reviewer is generally content with the strategy. He or she states that "As a rule, a prediction model will need to have not more than 10 variables if there are 100 events." This applies to model development, not model validation. For model validation, a rule of 100 events and 100 non-events can be maintained, irrespective of the number of variables in the prediction rule (Vergouwe et al 2005).
We think that the "treatment paradox" is a non-issue here since, during the validation study, we will not disclose predicted probabilities, nor will we apply care paths that may influence outcome.
The reviewer wants more information on how the literature search will be done, and if there will be any restrictions with respect to language or setting. Literature search will be done in PubMed, first by use of broad (sensitive) search terms, and then, on the basis of a review of 100 titles, abstracts and associated MeSH terms the search will be made more specific so as to be able to exclude nonrelevant articles in an automated matter. Language will be restricted to English, Dutch, German, French, and Spanish, and in principle no restrictions will be applied to setting, but all papers will be evaluated with respect to applicability on a general population.
We are not sure what the reviewer means with the "4 components" involved in the study.
1.3 Project group: good

Feasibility
Again it is unclear why the reviewer thinks that the time frame will be insufficient. In any case, we will confer with other groups (UMCU and AMC) if data collection can be combined so that this part of the study can be shortened.

Overall quality assessment
The reviewer rates the application as good quality. Multiple outcomes is indeed an issue in both the validation and the impact study. Data analysis and ranking of prediction rules (validation study) is described under 1.3. The main outcome variable of the impact study is the composite neonatal outcome as for this outcome we calculated sample size. All other outcomes (mentioned under 2.2.2) will however be analyzed (2.3) and addressed in separate papers where necessary. As the reviewer suggests, we suggest that funding of the first study is conditional on the outcome of the first.