Measures of diagnostic accuracy and clinical usefulness. Epidemiological methodologies applied to the use of lung ultasound among heart failure patients

Bovaro F.*1, Manasievska M.*2,3, Merletti F.3, Ciliberto E.3, Maule M.M.3, Lupia E.4,5, Pivetta E.2,3,4
  1. Residency Program in Emergency Medicine, University of Turin
  2. PhD Program in Experimental Medicine and Therapy, Department of Medical Sciences, University of Turin
  3. Cancer Epidemiology Unit, Department of Medical Sciences, University of Turin
  4. Emergency Medicine Division, AOU Città della Salute e della Scienza di Torino
  5. Deparment of Medical Sciences, University of Turin


It is still not clear which is the best methods for evaluating accuracy and clinical usefulness of new diagnostic tools.
To evaluate performances of an integrated diagnostic approach with the lung ultrasound (LUS) in diagnosing acute heart failure using seversal methodologies.
Materials And Methods
We calculated the area under the ROC curve (AUC), Brier score, Youden index, net reclassification index (NRI) and net benefit (NB) for the clinical and the LUS integrated approaches in a subcohort of patients enrolled at Molinette Hospital in a previous multicenter study.
NRI and NB seemed to be more informative for understanding the usefulness of a diagnostic tool.


Heart failure (HF) is one of the most relevant problems in developed countries and its incidence is increasing progressively with age (1). HF is defined as clinical syndrome with symptoms and signs that can result from any structural or functional cardiac disorder that impairs the ability of the ventricle to eject blood (2).
Acute heart failure is a complex and heterogeneous clinical syndrome defined as the rapid onset or change in symptoms and signs of heart failure requiring immediate medical attention and urgent therapy. It is a leading indication for hospitalization, associated with high short-term (intra-hospital) and long-term (6 to 12 month) mortality (2).
Typical HF symptom is shortness of breath (i.e. dyspnea), which is one of the most common complaints in the Emergency Department (ED), causing over 3 million evaluations/year in the United States (3)(4). It is defined as a subjective experience of breathing difficulty. Dyspnea can have two main etiologies, cardiogenic and non-cardiogenic. The diagnosis of HF based on combination of patient’s history, physical examination and traditional diagnostic approach (i.e. chest radiography, electrochadiogram, and dosage of natriuretic peptides) is often difficult, and a large number of the initial etiological diagnoses made by emergency physicians are modified after further examinations leading to dangerous diagnostic delays.
Lung ultrasound (LUS) is a basic application of point-of-care ultrasound(5). It can be quickly performed bedside and it leads to rapid therapeutic decisions (6).
Multiple vertical artifacts (i.e. B lines) at LUS evaluation have been proposed as a sonographic sign of pulmonary congestion (7). they are a good indicator of alveolar interstitial syndrome, but are not specific for acute HF AHF (8). Combination of sonographic and clinical finidings might improve diagnostic accuracy of an acute dyspnea etiology assessment (6).
The recent guidelines from the European Society of Cardiology (ESC), published in June 2016 (2), do not modify the general approach to patients with suspected AHF. The guidelines propose an integrated approach for the diagnosis of HF that should be based on detailed symptoms history, physical examination and further diagnosis confirmation using additional investigations such as electrocardiogram, chest radiograph, echocardiography and biomarkers such as natriuretic peptides (2). Therefore, the only relevant difference compared to the 2012 ESC guidelines is the recommendation to use natriuretic peptides.
The guidelines mention the LUS without indicating its level of efficacy, but suggesting the use of bedside LUS for evaluation of signs of interstitial edema and pleural effusion if expertise was available (2).
Several epidemiological methods have been suggested to evaluate accuracy and clinical usefulness of different diagnostic tools, but none of them was demostrated to perform better than the traditional receiver operating characteristic (ROC) curve, mainly in terms of frequency of use.


With this study we aimed to evaluate different performances of an integrated diagnostic approach, by implementing clinical assessment in combination with the bedside LUS in differentiating AHF from noncardiogenic causes of acute dyspnea in the ED.

Materials and methods

Two different forms of TTP have been classified: Congenital and Acquired forms. Both cases, genetic mutations and immune-mediated mechanisms, are responsable of the ADAMTS13 protein deficiency. The ultralarge multimeter of the von-Willebrand factor (ULvWF) is not appropriately cleaved, causing spontaneous and massive platelet aggregation for intense endothelial stress, especially in the cerebral, cardiac and renal microcirculation. Some predisposing factors are: occult neoplasia or metastatic condition, the intake of drugs, among which the quinidine and ticlopidine [10], and HIV infection [11]. Acquired TTP usually occurs in a previously healthy individual, more commonly in young females as evident in our case report.
The presence of microangiopathic haemolytic anaemia and thrombocytopenia is ample for the diagnosis in the absence of other evident causes [12]. Our patient also presented two of the other three elements of the classical pentade (acute renal failure, neurological changes and fever). These signs are no longer necessary and are associated to advanced organ damage [13]. As in our case, symptomatology can be quite varied, including hemorrhagic events and the most fearful thrombotic manifestations (cardiovascular, renal and neurological). Non specific gastroenterological manifestations, such as fever, abdominal pain, diarrhoea and vomiting are very frequent. Recently McDonald et. [14] described the association with new acute pancreatitis.
Sometimes, the identification of the syndrome can be difficult, being often in overlap conditions with hemolytic-uraemic syndrome (HUS) and autoimmune disorders (antiphospholipid antibody syndrome, Evans syndrome).
Rapid diagnosis must be supported by laboratory tests; the presence of≥1schistocytes on a peripheral blood smearschistocytes is considered suggestive for diagnostic purposes [15, 16]. 
TTP treatment involves the use of plasmapheresis [17] or, in alternative, plasma infusion by evaluating the individual fluid load tolerance [18]. These therapeutic approaches reduce mortality from 80-90% to 10% [9]. The effectiveness of plasma exchange therapy is due to the removal from the circulation of anti-ADAMTS13 antibodies in the immune-mediated forms and the correction of protease deficiency in congenital forms. A delay (over 24 hours) of treatment can compromise its effectiveness. 
The use of glucocorticoids for autoimmune etiology seems to be rational in the event of poor response to initial treatment with plasmapheresis [16, 19]. It is also useful when the platelet count does not rise after several days of plasmapheresis or if it returns to fall when treatment sessions are reduced or suspended [12]. In recent years there has been an increase in the use of anti-CD20 monoclonal antibodies (Rituximab) to suppress the production of anti-ADAMTS13 antibodies by depletion of B lymphocytes [20, 21].
Several studies suggest a complete clinical and laboratory response within 1-3 weeks of starting Rituximab treatment, in about 95% of patients with idiopathic PTT, refractory to plasma exchange and corticosteroid treatments, or relapsing [22]. Intravenous immunoglobulin administration may be effective. Splenectomy may actually be considered in chronic recurrent PTT forms that are refractory to other therapeutic approaches. Its effectiveness is based on the removal of an important production site of anti-ADAMTS13 autoantibodies. Platelet aggregation inhibitors such as ticlopidine, clopidogrel, acetylsalicylic acid, and dipyridamole have no indication [9]; they are unable to inhibit platelets aggregation induced by ULvWF multimers [23]. Blood transfusions, folic acid supplements, and possible antithrombotic prophylaxis with low molecular weight heparin may be useful support therapy. For such a fearful pathology, it is right to perform a follow-up with blood count and LDH dose. In fact, several cases of recurrence are described within 30 days of the disease remission. At the moment, validated maintenance therapies are not described.


The sub-cohort analyzed in this study consists of 310 patients presented to the ED of the Città della Salute e della Scienza di Torino for acute dyspnea, of whom 152 (49%) patients received a final diagnosis of heart failure. The area under the ROC (AUC) of the clinical evaluation, the integrated approach and CXR was 0.874, 0.974 and 0.774, respectively.
The NRI of the approach integrated with LUS for cardiogenic and non-cardiogenic dyspneas were 12.5 (95% CI: 6.9-18.1) and 7.6 (95% CI: 3.9-12.1), respectively. The NB of the clinical and the integrated valuations varied from 13.1 to 10, respectively with a prevalence of heart failure ranging from 40 to 50%.
The Brier score for the clinical and integrated evaluations were 0.11 and 0.03, respectively.
The results for the Youden index for the clinical diagnosis and the integrated approach was 0.747 and 0.948, respectively.


The diagnostic accuracy and clinical usefulness of a diagnostic tool could be expressed in several different ways. Although several methods have been proposed, AUC is the most reported measure of accuracy. Despite a widespread use of AUC, NRI and NBs might be more informative, in particular for understanding the usefulness of a diagnostic tool.


  1. Redfield MM. Heart Failure — An Epidemic of Uncertain Proportions. N Engl J Med. 2002 Oct 31;347(18):1442–4.
  2. Ponikowski P, Voors AA, Anker SD, Bueno H, Cleland JGF, Coats AJS, et al. 2016 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur Heart J. 2016;37(27):2129–2200m.
  3. Parshall MB, Schwartzstein RM, Adams L, Banzett RB, Manning HL, Bourbeau J, et al. An official American thoracic society statement: Update on the mechanisms, assessment, and management of dyspnea. Am J Respir Crit Care Med. 2012;185(4):435–52.
  4. Ambrosino N, Serradori M. Determining the cause of dyspnoea: linguistic and biological descriptors. Chron Respir Dis. 2006;3(3):117–22.
  5. Moore CL CJ. Point-of-care ultrasonography. N Engl J Med. 2011;364(8):749–57.
  6. Pivetta E, Goffi A, Lupia E, Tizzani M, Porrino G, Ferreri E, et al. Lung ultrasound-implemented diagnosis of acute decompensated heart failure in the ED: A SIMEU multicenter study. Chest. 2015;148(1):202–10.
  7. Martindale JL. Resolution of sonographic B-lines as a measure of pulmonary decongestion in acute heart failure. Am J Emerg Med. 2016;34(6):1129–32.
  8. Volpicelli G, Elbarbary M, Blaivas M, Lichtenstein DA, Mathis G, Kirkpatrick AW, et al. International evidence-based recommendations for point-of-care lung ultrasound. Intensive Care Med. 2012;38(4):577–91.
  9. MS P. The Statistical Evaluation of Medical Tests for Classification and Prediction. New York: Oxford University Press; 2004. 302 p.
  10. Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biometrical J. 2008;50(3):419–30.
  11. Brier GW. Verification of forecasts expersses in terms of probaility. Mon Weather Rev. 1950;78(1):1–3.
  12. Vickers AJ. Decision analysis for the evaluation of diagnostic tests, prediction models and molecular markers. Am Stat. 2008;62(4):314–20.
  13. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Vol. 19, Book. 2009. 500 p.
  14. Steyerberg EW, Pencina MJ, Lingsma HF, Kattan MW, Vickers AJ, van Calster B. Assessing the incremental value of diagnostic and prognostic markers: A review and illustration. Eur J Clin Invest. 2012;42(2):216–28.