Elsevier

Medical Image Analysis

Volume 30, May 2016, Pages 108-119
Medical Image Analysis

A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI

https://doi.org/10.1016/j.media.2016.01.005Get rights and content

Highlights

  • Deep learning for segmentation.

  • Excellent agreement.

  • High correlation for indices.

Abstract

Segmentation of the left ventricle (LV) from cardiac magnetic resonance imaging (MRI) datasets is an essential step for calculation of clinical indices such as ventricular volume and ejection fraction. In this work, we employ deep learning algorithms combined with deformable models to develop and evaluate a fully automatic LV segmentation tool from short-axis cardiac MRI datasets. The method employs deep learning algorithms to learn the segmentation task from the ground true data. Convolutional networks are employed to automatically detect the LV chamber in MRI dataset. Stacked autoencoders are used to infer the LV shape. The inferred shape is incorporated into deformable models to improve the accuracy and robustness of the segmentation. We validated our method using 45 cardiac MR datasets from the MICCAI 2009 LV segmentation challenge and showed that it outperforms the state-of-the art methods. Excellent agreement with the ground truth was achieved. Validation metrics, percentage of good contours, Dice metric, average perpendicular distance and conformity, were computed as 96.69%, 0.94, 1.81 mm and 0.86, versus those of 79.295.62%, 0.87–0.9, 1.76–2.97 mm and 0.67–0.78, obtained by other methods, respectively.

Introduction

Cardiac magnetic resonance imaging (MRI) is now routinely being used for the evaluation of the function and structure of the cardiovascular system (Yuan, Kerwin, Ferguson, Polissar, Zhang, Cai, Hatsukami, 2002, Lima, Desai, 2004, Frangi, Niessen, Viergever, 2001, Petitjean, Dacher, 2011, Tavakoli, Amini, 2013, Heimann, Meinzer, 2009, Suinesiaputra, Cowan, Al-Agamy, Elattar, Ayache, Fahmy, Khalifa, Gracia, Jolly, Kadish, Lee, Margeta, Warfield, Young, 2014). Segmentation of the left ventricle (LV) from cardiac MRI datasets is an essential step for calculation of clinical indices such as ventricular volume, ejection fraction, left ventricular mass and wall thickness as well as analyses of the wall motion abnormalities.

Manual delineation by experts is currently the standard clinical practice for performing the LV segmentation. However, manual segmentation is tedious, time consuming and prone to intra- and inter-observer variability (Frangi, Niessen, Viergever, 2001, Petitjean, Dacher, 2011, Tavakoli, Amini, 2013, Heimann, Meinzer, 2009, Suinesiaputra, Cowan, Al-Agamy, Elattar, Ayache, Fahmy, Khalifa, Gracia, Jolly, Kadish, Lee, Margeta, Warfield, Young, 2014). To address this, it is necessary to reproducibly automate this task to accelerate and facilitate the process of diagnosis and follow-up. To date, several methods have been proposed for the automatic segmentation of the LV. A review of these methods can be found in Frangi et al. (2001); Petitjean and Dacher (2011); Tavakoli and Amini (2013); Heimann and Meinzer (2009); Suinesiaputra et al. (2014).

To summarize, there are several challenges in the automated LV segmentation from cardiac MRI datasets: heterogeneities in the brightness of LV cavity due to blood flow; presence of papillary muscles with signal intensities similar to that of the myocardium; complexity in segmenting the apical and basal slice images; partial volume effects in apical slices due to the limited resolution of cardiac MRI; inherent noise associated with cine cardiac MRI; dynamic motion of the heart and inhomogeneity of intensity; considerable variability in shape and intensity of the heart chambers across patients, notably in pathological cases, etc (Tavakoli, Amini, 2013, Petitjean, Dacher, 2011, Queiros, Barbosa, Heyde, Morais, Vilaca, Friboulet, Bernard, D’hooge, 2014). Due to these technical barriers, the task of automatic segmentation of the heart chambers from cardiac MRI is still a challenging problem (Petitjean, Dacher, 2011, Tavakoli, Amini, 2013, Suinesiaputra, Cowan, Al-Agamy, Elattar, Ayache, Fahmy, Khalifa, Gracia, Jolly, Kadish, Lee, Margeta, Warfield, Young, 2014).

Current approaches for automatic segmentation of the heart chambers can be generally classified as: pixel classification (Kedenburg, Cocosco, Köthe, Niessen, Vonken, Viergever, 2006, Cocosco, Niessen, Netsch, Vonken, Lund, Stork, Viergever, 2008), image-based methods (Jolly, 2009, Liu, Hu, Xu, Song, 2012), deformable methods (Billet, Sermesant, Delingette, Ayache, 2009, Ben Ayed, Li, Ross, 2009, Chang, Valentino, Chu, 2010, Pluempitiwiriyawej, Moura, Wu, Ho, 2005), active appearance and shape models (AAM/ASM) (Zhang, Wahle, Johnson, Scholz, Sonka, 2010, Assen, Danilouchkine, Frangi, Ords, Westenberg, Reiber, Lelieveldt, 2006) and atlas models (Zhuang, Hawkes, Crum, Boubertakh, Uribe, Atkinson, Batchelor, Schaeffter, Razavi, Hill, 2008, Lorenzo-Valdés, Sanchez-Ortiz, Elkington, Mohiaddin, Rueckert, 2004). Pixel classification, image-based and deformable methods suffer from a low robustness and accuracy and require extensive user interaction (Petitjean and Dacher, 2011). Alternatively, model-based methods such as AAM/ASM and atlas models can overcome the problems related to the previous methods and reduce user interaction at the expense of a large training set to build a general model. However, it is very difficult to build a model that is general enough to cover all possible shapes and dynamics of the heart chambers (Petitjean, Dacher, 2011, Jolly, Xue, Grady, Guehring, 2009). Small datasets lead to a large bias in the segmentation, which makes these methods inefficient when the heart shape is outside the learning set (e.g., congenital heart defects, post-surgical remodeling, etc).

Furthermore, current learning-based approaches for LV segmentation have certain limitations. For instance, methods using random forests (Margeta, Geremia, Criminisi, Ayache, 2012, Lempitsky, Verhoek, Noble, Blake, 2009, Geremia, Clatz, Menze, Konukoglu, Criminisi, Ayache, 2011) rely on image intensity and define the segmentation problem as a classification task. These methods employ multiple stages of intensity standardization, estimation and normalization, which are computationally expensive and affect the success of further steps. As such, their performance is rather mediocre at basal and apical slices and overall inferior to the state-of-the-art. Also, methods that use Markov random fields (MRFs) (Cordero-Grande, Vegas-Sánchez-Ferrero, Casaseca-de-la Higuera, San-Román-Calvar, Revilla-Orodea, Martín-Fernández, Alberola-López, 2011, Huang, Pavlovic, Metaxas, 2004), conditional random fields (CRFs) (Cobzas, Schmidt, 2009, Dreijer, Herbst, du Preez, 2013) and restricted Boltzman machines (RBMs) (Ngo and Carneiro, 2014) have been considered. MRF and RBM are generative models that try to learn the probability of input data. Computing the image probability and parameter estimation in generative models is usually difficult and can lead to reduced performance if oversimplified. Besides, they use the Gibbs sampling algorithm for training, which can be slow, become stuck for correlated inputs, and produce different results each time it is run due to its randomized nature. Alternatively, CRF methods try to model the conditional probability of latent variables, instead of the input data. However, they are still computationally difficult, due to complexity of parameter estimation, and their convergence is not guaranteed (Dreijer et al., 2013).

Motivated by these limitations, and given the fact that manual segmentation by experts is the ground truth in cardiac MRI, we tackle the complex problem of LV segmentation utilizing a combined deep-learning (LeCun, Bengio, Hinton, 2015, Hinton, Salakhutdinov, 2006, Bengio, 2009, Bengio, Courville, Vincent, 2013, Ng, Deng, Yu, 2014, Baldi, 2012) and deformable-models approach. We develop and validate a fully automated, accurate and robust LV segmentation method from cardiac MRI. In terms of novelty and contributions, our work is one of the early attempts of employing deep learning algorithms for cardiac MRI segmentation. It is generally believed that since current practices of deep learning have been trained on huge amount of data, deep learning cannot be effectively utilized for medical image segmentation due to the lack of training data. However, we show that even with limited amount of training data, using artificial data enlargement, pre-training and careful design, deep learning algorithms can be successfully trained and employed for cardiac MRI segmentation. Nevertheless, we solve some of the shortcomings of classical deformable models, i.e., shrinkage and leakage and sensitivity to initialization, using our integrated approach. Furthermore, we introduce a new curvature estimation method using quadrature polynomials to correct occasional misalignment between image slices. The proposed framework is tested and validated on the MICCAI database (Radau et al., 2009).

The remainder of the manuscript is as follows. In Section 2, our method is described in detail. In Section 3, the implementation elements are provided. Section 4 presents the validation experiments. The results are presented in Section 5. In Section 6, we discuss the results, performance and comparison with the state-of-the-arts methods, and finally, Section 7 concludes the paper.

Section snippets

Datasets

We used the MICCAI 2009 challenge database (Radau et al., 2009) to train and assess the performance of our methodology. The MICCAI database obtained from the Sunnybrook Health Sciences Center, Toronto, Canada is publicly available online (Radau et al., 2009) and contains 45 MRI datasets, grouped into three datasets. Each dataset contains 15 cases, divided into four ischemic heart failure (SC-HF-I), four non-ischemic heart failure (SC-HF-NI), four LV hypertrophy (SC-HYP) and three normal (SC-N)

Implementation details

Images and contours of all the cases in the training dataset of the MICCAI challenge (Radau et al., 2009) were collected and divided into the large-contour and small-contour groups. Typically, the large contours belong to image slices near the base/middle and the small contours belong to the apex of the heart since the contours near the apex are much smaller than the contours at the base. As such, there are around 135 and 125 images in each group, respectively. We artificially enlarged the

Validation process

We assess the performance of our automated method by comparing its accuracy with that of the gold standard (manual annotations by experts). Accordingly, average perpendicular distance (APD), Dice metric, Hausdorff distance, percentage of good contours and the conformity coefficient (Chang et al., 2009) were computed. As recommended in Radau et al. (2009), a segmentation is classified as good if the APD is less than 5 mm. The average perpendicular distance measures the distance from the

Illustrative results

To better understand the role of each step, the outcome of the deformable model with no shape constraint (α3=0), deep learning (shape inference, Step 2) and the integrated deformable model and deep learning method (final step) for two typical images are shown in Fig. 8.

Fig. 9 illustrates the automatic and manual LV segmentation results for a typical cardiac MRI dataset from the base to apex as well as three views of the reconstructed LV chamber (front, base and apex views). The segmentation

Discussion

In this study, we developed and validated an automated LV segmentation method based on deep learning algorithms. We broke down the problem into localization, shape inferring and segmentation tasks. Convolutional networks were chosen for localization and extracting a ROI because they are invariant to spacial translation and changes in scale and pixels’ intensity (LeCun, Kavukcuoglu, Farabet, Sermanet, Eigen, Zhang, Mathieu, Fergus, LeCun, 2014). We also chose a stacked AE for shape inferring

Conclusion

In summary, a novel method for fully automatic LV segmentation from cardiac MRI datasets is presented. The method employs deep learning algorithms for automatic detection and inferring the LV shape. The shape was incorporated into deformable models and brought more robustness and accuracy, particularly for challenging basal and apical slices. The described approach is shown to be accurate and robust compared to the other state-of-the-art methods. Excellent agreement and a high correlation with

Acknowledgments

This work is partially supported by a grant from American Heart Association (14GRNT18800013) to Prof. Kheradvar and Conexant-Broadcom Endowed Chair of Prof. Jafarkhani.

References (63)

  • Lorenzo-ValdésM. et al.

    Segmentation of 4D cardiac MR images using a probabilistic atlas and the EM algorithm

    Med. Image Anal.

    (2004)
  • PetitjeanC. et al.

    A review of segmentation methods in short axis cardiac MR images

    Med. Image Anal.

    (2011)
  • PetitjeanC. et al.

    Right ventricle segmentation from cardiac mri: A collation study

    Med. Image Anal.

    (2015)
  • QueirosS. et al.

    Fast automatic myocardial segmentation in 4d cine CMR datasets

    Med. Image Anal.

    (2014)
  • SchaererJ. et al.

    A dynamic elastic model for segmentation and tracking of the heart in MR image sequences

    Med. Image Anal.

    (2010)
  • SuinesiaputraA. et al.

    A collaborative resource to build consensus for automated left ventricular segmentation of cardiac MR images

    Med. Image Anal.

    (2014)
  • TavakoliV. et al.

    A survey of shaped-based registration and segmentation techniques for cardiac images

    Comput. Vis. Image Underst.

    (2013)
  • AssenH.C. et al.

    Spasm: a 3d-asm for segmentation of sparse and arbitrarily oriented cardiac MRI data

    Med. Image Anal.

    (2006)
  • BabalolaK.O. et al.

    Comparison and Evaluation of Segmentation Techniques for Subcortical Structures in Brain MRI

    (2008)
  • BaldiP.

    Autoencoders, unsupervised learning, and deep architectures.

    ICML Unsuperv. Transf. Learn.

    (2012)
  • BarajasJ. et al.

    Correction of misalignment artifacts among 2-D cardiac mr images in 3-D space

    First International Workshop on Computer Vision for Intravascular and Intracardiac Imaging, MICCAI 2006

    (2006)
  • Ben AyedI. et al.

    Embedding overlap priors in variational left ventricle tracking

    IEEE Trans. Med. Imaging

    (2009)
  • BengioY.

    Learning deep architectures for AI

    Found. Trends Mach. Learn.

    (2009)
  • BengioY. et al.

    Representation learning: a review and new perspectives

    IEEE Trans. Patt. Anal. Mach. Intel.

    (2013)
  • BilletF. et al.

    Cardiac motion recovery and boundary conditions estimation by coupling an electromechanical model and cine-mri data

    Funct. Imaging Model. Heart

    (2009)
  • BlandJ.M. et al.

    Statistical methods for assessing agreement between two methods of clinical measurement

    Lancet

    (1986)
  • BoureauY.-L. et al.

    A theoretical analysis of feature pooling in visual recognition

    Proceedings of the 27th International Conference on Machine Learning (ICML-10)

    (2010)
  • ChanT. et al.

    Active contours without edges

    IEEE Trans. Img. Proc.

    (2001)
  • ChandlerA.G. et al.

    Correction of misaligned slices in multi-slice cardiovascular magnetic resonance using slice-to-volume registration

    J. Cardiovasc. Magn. Reson.

    (2008)
  • ChangH.-H. et al.

    Active shape modeling with electric flows

    IEEE Trans. Vis. Comp. Gr.

    (2010)
  • CobzasD. et al.

    Increased discrimination in level set methods with embedded conditional random fields

    Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009

    (2009)
  • Cited by (526)

    View all citing articles on Scopus

    This paper was recommended for publication by “HANDLING Dr. James Duncan”.

    View full text