Validation of a model to predict adverse outcomes in patients with pulmonary embolism

Abstract

Aims To validate a model for quantifying the prognosis of patients with pulmonary embolism (PE). The model was previously derived from 10 534 US patients.

Methods and results We validated the model in 367 patients prospectively diagnosed with PE at 117 European emergency departments. We used baseline data for the model's 11 prognostic variables to stratify patients into five risk classes (I–V). We compared 90-day mortality within each risk class and the area under the receiver operating characteristic curve between the validation and the original derivation samples. We also assessed the rate of recurrent venous thrombo-embolism and major bleeding within each risk class. Mortality was 0% in Risk Class I, 1.0% in Class II, 3.1% in Class III, 10.4% in Class IV, and 24.4% in Class V and did not differ between the validation and the original derivation samples. The area under the curve was larger in the validation sample (0.87 vs. 0.78, P=0.01). No patients in Classes I and II developed recurrent thrombo-embolism or major bleeding.

Conclusion The model accurately stratifies patients with PE into categories of increasing risk of mortality and other relevant complications. Patients in Risk Classes I and II are at low risk of adverse outcomes and are potential candidates for outpatient treatment.

Introduction

The mortality of acute pulmonary embolism (PE) varies widely, ranging from >95% in patients who experience cardiorespiratory arrest to <2% in patients with a non-massive embolus.^1–3 A risk stratification tool that accurately quantifies the prognosis of patients with PE may be useful in guiding the intensity of initial treatment. For example, patients estimated to be at low risk could be discharged early or managed entirely as outpatients using low-molecular-weight heparin,⁴ whereas patients estimated at high risk may benefit from a more intensive surveillance.

We previously developed a clinical prognostic model that accurately stratifies patients into five severity classes of increasing risk of 30-day mortality and other adverse medical outcomes such as non-fatal recurrent venous thrombo-embolism (VTE) and major bleeding.⁵ The model comprised 11 routinely available clinical parameters and provides physicians a beside risk assessment tool for patients with PE, without any need for imaging studies such as echocardiography or laboratory tests.⁵

Geographic and follow-up period transportabilities are important components of the generalizability of a prognostic model.⁶ These components require that the model's predictions remain reliable and accurately discriminate key outcomes when re-tested in patients from other locations, using different periods of follow-up.⁶ To assess these dimensions of generalizability, we validated our prognostic model in a multicentre study of PE conducted in France and Belgium using a follow-up period of 90 days.⁷

Methods

Initial derivation of the prognostic model

Using administrative and clinical data from the Pennsylvania Health Care Cost Containment Council and the MediQual Atlas databases, we derived the original prognostic model from 10 534 adult patient discharges from 186 Pennsylvania hospitals with an ICD-9-CM diagnosis of PE.⁵ To derive our rule, we used patient demographics (age and sex), comorbid conditions (cancer, heart failure, ischaemic heart disease, chronic lung disease, chronic renal disease, cerebrovascular disease, severe neurological disease, and smoking status), and physical examination findings (body temperature, pulse, systolic blood pressure, respiratory rate, mental status, and arterial oxygen saturation). All of these predictor variables were recorded at the time of patient presentation and were previously shown to be associated with short-term mortality in patients with PE or other acute diseases.^8–15

The primary study outcome was death from all causes within 30 days of hospitalization, and this outcome was ascertained using data from the US National Death Index.¹⁶ The original prediction rule consisted of 11 predictors of mortality (Table 1): two demographic variables (age and male sex), three comorbid conditions (cancer, heart failure, and chronic lung disease), and six clinical factors (temperature <36°C, pulse ≥110 per minute, systolic blood pressure <100 mmHg, respiratory rate ≥30 per minute, altered mental status, and arterial oxygen saturation <90%).⁵ Each predictor variable was assigned an integer score directly proportional to the magnitude of its beta-coefficient derived from a logistic regression model of mortality. We computed a prognostic score for each patient by adding their age in years and the assigned points for each of the predictor variables documented for that patient. We partitioned the full range of risk scores into quintiles, with each quintile representing a prognostic risk class for 30-day mortality (Class I, very low risk; Class II, low risk; Class III, intermediate risk; Class IV, high risk; and Class V, very high risk) (Table 1).⁵ In the original derivation sample, the 30-day overall mortality was 9.2% and the risk-class-specific mortality varied from 1.1% in Risk Class I to 24.5% in Risk Class V.⁵ Detailed information concerning the derivation process and results were published elsewhere.⁵

Validation sample

We externally validated our model in a cohort of patients prospectively identified to assess the appropriateness of diagnostic testing for patients with suspected PE.⁷ These patients were enrolled in emergency departments at 117 teaching (n=23) and non-teaching (n=94) hospitals in France and Belgium between 13 January 2003 and 16 February 2003. Patients were excluded from this study if the diagnosis of PE was documented before the time of presentation or was made after a hospital stay of >2 days, or if diagnostic testing for PE could not be performed because the patient rapidly died or left the hospital against medical advice.⁷ In the validation population, we considered only patients with objectively confirmed PE, defined by a positive spiral computerized tomography or pulmonary angiography, a high-probability ventilation/perfusion lung scan, or proximal deep vein thrombosis documented by compression ultrasonography.

Baseline demographic (age and sex) and clinical information (cancer, heart failure, chronic lung disease, pulse, blood pressure, respiratory rate, temperature, and arterial oxygen saturation measured by pulse oximetry) and whether patients received thrombolysis were prospectively collected by emergency department physicians and recorded on a standardized data collection instrument.⁷ Using the baseline data for the prognostic variables in the prognostic model, we calculated a risk score and risk class for each patient in the validation population. Because mental status was not recorded, we assumed mental status to be normal in all patients in the validation cohort. Missing values for all prognostic variables were assumed to be normal, a strategy used in the original derivation of the model and in the development of the Pneumonia Severity Index, a widely used risk stratification tool for community-acquired pneumonia.^5,17

Our primary outcome measure was all-cause mortality 90 days after presentation for PE. We assessed mortality using patient or proxy interviews, interview of the patient's primary care physician, and/or hospital chart review. Interviews were performed through telephone and administered by local study coordinators. Two investigators adjudicated all deaths as definite, fatal PE and possible, fatal PE (e.g. sudden death without any obvious cause) or death from other causes. Our secondary outcomes were objectively confirmed non-fatal VTE (PE or deep vein thrombosis) and non-fatal major bleeding, defined as any bleeding resulting in disability, hospitalization, or blood transfusion.⁷ For this project, we excluded all patients who refused follow-up or who were lost during follow-up.

Methods of analysis

We compared baseline patient's characteristics between the validation and the original derivation samples⁵ using χ² statistics for categorical variables. To assess the performance of our prognostic model, we compared the proportions of patients per risk class and mortality rates within each risk class between the validation and the original derivation samples.⁵ To assess the model's discriminatory power to predict mortality, we compared the area under the receiver operating characteristic (ROC) curve between the validation and the original derivation samples.^5,18 For all analyses, a two-tailed P-value of less than 0.05 was used to define statistical significance. To assess the accuracy of our model to predict mortality, we also compared sensitivity, specificity, and positive and negative predictive values and likelihood ratios for low (Risk Classes I and II) vs. higher risk patients (Risk Classes III–V) between derivation and validation samples.

Results

Of 1529 patients included in the study, 393 had objectively confirmed PE, defined by a positive spiral computerized tomography or pulmonary angiography, a high-probability ventilation/perfusion lung scan, or proximal deep vein thrombosis documented by compression ultrasonography. Of these, we excluded 26 (6.6%) who refused to give informed consent or who were lost to follow-up, establishing a validation sample of 367 patients. Excluded patients had the same demographic and clinical characteristics as enrolled patients. Overall, 23.1% of excluded patients were classified in Risk Classes I and II, 11.5% in Class III, 30.8% in Class IV, and 11.5% in Class V.

Baseline characteristics and outcomes in the derivation and validation samples

Although patients in the validation sample were older, they were less likely to have cancer, chronic lung disease, hypothermia, systolic hypotension, and an altered mental status when compared with patients in the derivation sample (Table 2). The higher frequency of hypoxaemia in the validation sample probably reflects the measurement of arterial oxygenation without supplemental oxygen in the validation sample. Thrombolysis rates varied from 2.0 to 8.9% and did not differ across risk classes (P=0.32).

Despite a longer duration of follow-up in the validation sample (30 vs. 90 days), mortality was lower in the validation sample (6.3%) than in the derivation sample (9.2%). Of the 23 patients in the validation sample who died, 11 (47.8%) died from definite or possible PE, two (8.7%) from intracranial haemorrhage, and 10 (43.5%) from other causes. Overall, 15 (65.2%) patients in this sample died during the first 30 days following diagnosis. Only one patient (0.3%) had non-fatal recurrent VTE and nine (2.5%) developed non-fatal major bleeding in the validation sample.

Comparison of the prognostic model in the derivation and validation samples

The prognostic model classified a larger proportion of patients into Risk Classes II and III and a smaller proportion into Risk Class V in the validation sample than that in the derivation sample (Table 3). Because age is the most powerful predictor of adverse outcomes in the model, almost no patients aged ≥65 were classified into Risk Class I (<0.5%) in both samples, whereas only few patients under 65 years were classified into Risk Class V in the derivation (7.7%) and validation samples (2.7%). All five risk-class-specific mortality rates were similar in the derivation and validation samples and ranged from 0–1.1% in Class I to 24.4–24.5% in Class V (Table 3). The discriminatory power of the model, expressed as the area under the ROC curve, was larger in the validation sample (0.87 vs. 0.78, P=0.01) (Figure 1).

In the validation sample, no patients in Risk Classes I, II, III, and V had non-fatal recurrent VTE during follow-up; only one patient (1.6%) in Risk Class IV developed deep vein thrombosis. No patients in Classes I and II, six (6.3%) in Class III, two (3.2%) in Class IV, and one (2.2%) in Class V had non-fatal major bleeding during follow-up.

When dichotomized as low risk (Risk Classes I and II) vs. higher risk (Risk Classes III–V), the model had a negative predictive value of 98–99% and a negative likelihood ratio of 0.09–0.2 for predicting mortality (Table 4). Because this cutpoint was specifically chosen to identify low-risk patients with PE, the positive predictive values (11–14%) and the positive likelihood ratios (1.6–1.8) for predicting mortality were low.

Discussion

Our findings validate a previously developed prognostic model for patients with PE and confirm its geographic transportability to an independent patient population over a longer follow-up period.⁵ Our model accurately identifies patients with PE who are at low risk of fatal and non-fatal medical outcomes: none of the patients in Risk Class I in the validation sample died, had non-fatal recurrent VTE, or major bleeding within 90 days of the initial PE. Among patients in Risk Class II, only 1% of patients died and no patients had recurrent VTE or major bleeding. The negative predictive value for predicting mortality among low-risk patients (Risk Classes I and II) varied from 98 to 99%.

There is growing evidence that many patients with non-massive PE can be safely discharged early or treated entirely as outpatients using low-molecular-weight heparin.^19–21 The British Thoracic Society recommends outpatient treatment of clinically stable patients with non-massive PE.²² However, outpatient treatment of such patients is not widely accepted, because eligibility criteria of prior studies examining outpatient treatment of PE were relatively unspecific^19–21 and explicit criteria to identify low-risk patients with PE were not available. Our model, which is entirely based on simple and objective parameters, could help physicians identify low-risk patients with PE (Risk Classes I and II) who could be safely treated in the outpatient setting with low-molecular-weight heparin or if hospitalized, they are the candidates for early hospital discharge. A recent cost-effectiveness analysis demonstrated that outpatient treatment or early discharge of only a small proportion of patients with PE is likely to result in substantial cost-savings.²³ However, before low-risk patients with PE based on our model can be treated as outpatients, the safety and clinical usefulness of this approach must be tested in a clinical trial. In addition, aspects of model reliability, for example interobserver agreement for classifying patients as low vs. higher risk, should be assessed in a future study. It is also important to note that our rule is intended to supplement, not to replace clinical judgement. The initial site of treatment decision for patients with PE must also consider psychosocial contraindications to outpatient care (e.g. lack of treatment adherence). Likewise, physicians would be unlikely to discharge a previously healthy 40-year-old woman who has severe hypoxaemia and no additional pertinent prognostic factors, even if she was classified as very low risk (Class I) by the rule.

Whether patients in Risk Class V who have a short-term mortality of >20% may potentially benefit from more intensive forms of care and surveillance (e.g. in an intensive care unit setting) remains to be shown.

The original model was derived from a retrospective cohort of 10 354 inpatients with PE from 186 Pennsylvania hospitals and internally validated in another 5177 inpatients with PE from the same sites.⁵ The model also performed well in a small external validation sample of 221 patients prospectively identified with PE from three Swiss and French emergency departments.⁵ Thirty-day mortality ranged 0–1.6% for patients in Risk Class I, 1.7–3.5% in Class II, 3.2–7.1% in Class III, 4.0–11.4% in Class IV, and 10.0–24.5% in Class V in these three samples.⁵ Patients in Risk Classes I and II had also a low rate of non-fatal cardiogenic shock, cardiorespiratory arrest, recurrent VTE, and major bleeding.⁵ Overall, the accuracy and the generalizability of the model are now supported by its derivation and validation in four patient cohorts, comprising 16 119 patients from 305 teaching and non-teaching hospitals in four countries (USA, France, Switzerland, and Belgium) in North America and Western Europe. Patients in these cohorts had a broad spectrum of disease severity, ranging from non-massive PE to PE with cardiorespiratory arrest, and were followed-up to 90 days after the initial diagnosis.

Our work has potential limitations. First, our validation sample may not reflect the full prognostic spectrum of patients with PE, because patients had fewer comorbid diseases and fewer signs of physiological instability than patients in the derivation sample. However, no patient died in Risk Class I and only 1% of patients died in Risk Class II, on the basis of our model. Secondly, information about mental status was not explicitly recorded in the study used to validate our model.⁷ Although it is very unlikely that more than a few patients had an altered mental status, we cannot exclude the possibility that disease severity may have been underestimated in these patients. Thirdly, 6.6% of patients with PE had to be excluded from our analyses, because follow-up information was not available. However, baseline characteristics of excluded and enrolled patients were not different, making a selection bias unlikely. Finally, we could not estimate the potential impact of treatments (e.g. quality of oral anticoagulation during follow-up) on patient outcomes, because this information was not documented in our databases.

In conclusion, we successfully validated our clinical prognostic model in an independent patient sample with PE over a more extended duration of follow-up. Although this model reliably identifies patients at low risk of mortality and non-fatal adverse outcomes, a clinical trial is required to assess whether outpatient treatment or abbreviated inpatient treatment of patients in Risk Classes I and II is as safe and effective as traditional inpatient treatment.

Acknowledgements

The study was partly funded by a grant from the US National Heart, Lung, and Blood Institute (1 R21 HL075521-01A1). D.A. has received financial support from the Clinical Epidemiology Center, University of Lausanne, Switzerland. M.J.F. was supported by a K-24 career development award from the National Institute of Allergy and Infectious Diseases. This work was performed at the University of Lausanne, Switzerland and the University of Pittsburgh, USA.

Conflict of interest: none.

References