Elsevier

Medical Image Analysis

Volume 14, Issue 6, December 2010, Pages 770-783
Medical Image Analysis

Detection of neuron membranes in electron microscopy images using a serial neural network architecture

https://doi.org/10.1016/j.media.2010.06.002Get rights and content

Abstract

Study of nervous systems via the connectome, the map of connectivities of all neurons in that system, is a challenging problem in neuroscience. Towards this goal, neurobiologists are acquiring large electron microscopy datasets. However, the shear volume of these datasets renders manual analysis infeasible. Hence, automated image analysis methods are required for reconstructing the connectome from these very large image collections. Segmentation of neurons in these images, an essential step of the reconstruction pipeline, is challenging because of noise, anisotropic shapes and brightness, and the presence of confounding structures. The method described in this paper uses a series of artificial neural networks (ANNs) in a framework combined with a feature vector that is composed of image intensities sampled over a stencil neighborhood. Several ANNs are applied in series allowing each ANN to use the classification context provided by the previous network to improve detection accuracy. We develop the method of serial ANNs and show that the learned context does improve detection over traditional ANNs. We also demonstrate advantages over previous membrane detection methods. The results are a significant step towards an automated system for the reconstruction of the connectome.

Introduction

Neural circuit reconstruction, i.e. the connectome (Sporns et al., 2005), is currently one of the grand challenges facing neuroscientists. Similarly, the National Academy of Engineering has listed reverse-engineering the brain as one its grand challenges.1 While neural circuits are central to the study of the nervous system, relatively little is known about differences in existing neuronal classes, patterns, and connections. Electron microscopy (EM) is an unique modality for scientists attempting to map the anatomy of individual neurons and their connectivity because it has a resolution that is high enough to identify synaptic contacts and gap junctions. These are important indicators for types of neuron topology and are required for neural circuit reconstruction. Several researchers have undertaken extensive EM imaging projects in order to create detailed maps of neuronal structure and connectivity (Fiala and Harris, 2001, Briggman and Denk, 2006a). Early work in this area, by White et al. (1986), includes the complete mapping of the nematode Caenorhabditis elegans nervous system. This is a simple organism, containing just over 300 neurons and 6000 synapses, yet it took nearly a decade to identify all the relevant structures and reconstruct the connectivity.2 In comparison, newer imaging techniques are producing much larger volumes of very complex organisms, with thousands of neurons and millions of synapses (Briggman and Denk, 2006b, Anderson et al., 2009). Thus, automating the reconstruction process is of paramount importance.

The ability to reconstruct neural circuitry at ultrastructural resolution is of substantial clinical importance. Retinal degenerative diseases, including pigmentosa and macular degeneration, result from a loss of photoreceptors. Photoreceptor cell stress and death induces subsequent changes in the neural circuitry of the retina resulting in corruption of the surviving retinal cell class circuitry. Ultrastructural examination of the cell identity and circuitry reveal substantial changes to retinal circuitry with implications for vision rescue strategies (Marc et al., 2008, Marc et al., 2007, Marc et al., 2003, Jones and Marc, 2005, Jones et al., 2003, Jones et al., 2005, Peng et al., 2000). These findings in retinal degenerative disease mirror findings in epilepsy where neural circuits also undergo remodeling in presumed response to abnormal electrical activity clinically manifested as seizures. Scientists are interested in examining normal and pathological synaptic connectivities and how neuronal remodeling contributes to neuronal pathophysiology (Sutula et al., 2002, Pollard et al., 1994, Koyama et al., 2004). Examination of synaptic and dendritic spine formation during development provide insight into the adaptivity of neural circuits (Sorra et al., 2000, DeBello et al., 2001). Ultrastructural evaluation of multiple canonical volumes of neural tissue are critical to evaluate differences in connectivity between wild type and mutants. The complexity and size of the these datasets, often approaching 17 terabytes, makes human segmentation of the complex textural information of electron microscopic imagery a difficult task. Moreover, population or screening studies become unfeasible since fully manual segmentation and analysis would require multiple years of manual effort per specimen. As a result, better image processing techniques are needed to help with automated segmentation of EM data including identification of neurons and the connections.

The modality we have chosen for reconstructing the connectome at the individual cell level is serial-section transmission electron microscopy (TEM). It provides scientists with images that capture the relevant structures; however, it poses some interesting challenges for image processing. Most importantly, serial-section TEM offers a relatively wide field of view to identify large sets of cells that may wander significantly as they progress through the sections. It also has an in-plane resolution that is high enough for identifying synapses. In collecting images through TEM, sections are cut from a specimen and suspended so that an electron beam can pass through it creating a projection. The projection can be captured on a piece of film and scanned or captured directly as a digital image. An important trade-off occurs with respect to the section thickness. Thinner sections are preferable from an image analysis point of view because structures are more easily identifiable due to less averaging. However, from an acquisition point of view, thinner sections are harder to handle and impose a limit on the area of the section that can be cut. For instance, in the rabbit retina, scientists need to study sections with areas as large as 250 μm in diameter to gain a sufficient understanding of neural connectivity patterns. Sections of this size can be reliably cut at 50–90 nm thickness with the current serial-section TEM technology. This leads to an extremely anisotropic resolution, 2–5 nm in-plane compared to 50–90 nm out-of-plane, and poses two image processing challenges. First, the cell membranes can range from solid dark curves for neurons that run approximately perpendicular to the cutting-plane, to grazed grey swaths for others which run more obliquely and suffer more from the averaging effect. Consequently, segmentations of neurons in these 2-D images, are difficult given the change in membrane contrast and thickness. Second, due to the large physical separation between sections, shapes and positions of neurons can change significantly between adjacent sections.

There are alternative specimen preparation and EM imaging techniques that can be used for neural circuit reconstruction such as Serial-Block Face Scanning Electron Microscopy. Briggman and Denk proposed a specimen preparation which only highlights extracellular spaces removing almost all contrast from intracellular structures (Briggman and Denk, 2006b). However, it is not possible to identify synapses with that approach. Identification of synapses is an important part of neural circuit reconstruction because it determines which cells are communicating, and where in the circuitry they connect. To highlight synapses in TEM, scientists must use a stain that also highlights intracellular structures, such as vesicles and mitochondria, as well as neuron membranes. Therefore, image segmentation techniques must account for these data characteristics in order to identify and successfully track neurons across hundreds of sections.

There are two general approaches for neuron segmentation. One approach focuses first on the detection of neuron membranes in each 2-D section (Jurrus et al., 2008, Macke et al., 2008, Venkataraju et al., 2009). These boundaries can be used to identify individual neurons, which are then linked across sections to form a complete neuron. Unfortunately, accurate detection of neuron membranes in EM is a difficult problem given the presence of intracellular structures. This makes simple thresholding, edge detection (i.e. Canny), and region growing methods ineffective for the detection of neuron membranes. Some example images and results with traditional image processing methods are shown in Fig. 1. The other approach to neuron segmentation is to directly use the 3-D characteristics of the data (Andres et al., 2008, Jain et al., 2007). However, full 3-D approaches are difficult due to the anisotropic nature of the data. As mentioned earlier, in serial-section EM, there is a trade-off between section thickness and section loss rate. The datasets used in this paper to demonstrate membrane detection are from the C. elegans ventral nerve cord and from the rabbit retina. For these datasets, the nerve cord has a resolution of 6 nm × 6 nm × 33 nm and the retina has a resolution of 2 nm × 2 nm × 80 nm. This large section thickness often causes features to shift significantly between sequential images, decreasing the potential advantages of a direct 3-D approach. For these reasons, we follow the first approach which is to first perform a 2-D segmentation followed by a linking of the segmented regions in 3-D. This approach is particularly suitable for datasets in which a majority of the neurons run in a general direction which is roughly orthogonal to the sectioning plane such as the datasets considered in this paper. The main focus of this paper is to improve the 2-D neuron segmentation in each section. This information can then be used to link the segmentation in each section to obtain the full 3-D reconstruction.

Recent related work indicates that machine learning methods are an effective approach for detection of neuron membranes. These methods all use different representations for learning membrane pixels, most of which include training a single instance of a classifier on image derived features, such as Hessian eigenspaces (Venkataraju et al., 2009, Mishchenko, 2009) and local statistical features (Andres et al., 2008). Inspired by Tu’s auto-context shape classification approach (Tu, 2008), the method described in this paper uses a series of classifiers to more accurately detect membranes in EM images, which is a necessary step for improved 3-D neuron segmentation as discussed above. However, unlike Tu’s auto-context (Tu, 2008) which uses boosting to select features from a large pool of candidates such as Haar wavelet responses, we use a series of artificial neural networks (ANNs) that operate on a fixed set of features. The first ANN uses as input the intensity values sampled directly from the image. The input to the subsequent ANNs in the series is comprised of the same set of image values, in addition to the output of the previous ANN sampled on a stencil of nearby pixels (as depicted in Fig. 4). The ANNs in the series, therefore, have different inputs even though they have a common desired output. The advantages of this method are twofold. First, the classifier uses raw data, that is, the image intensities, rather than a constrained version of the image as given by responses to a large filter bank or statistical features that will not scale well for large datasets. Second, the use of the serial ANNs provides context, which is information from nearby pixels that contributes to the learning, providing increasing amounts of relative information at each stage of the network. As a result, the series of ANNs learns to remove vesicles and mitochondria from the membrane detection and close gaps in places where the membrane is weak. In this paper, we demonstrate the improvement from the combined use of stencils and the series of ANNs for two datasets with distinctly different characteristics.

Section snippets

Related work

There are several methods that attempt to segment EM images of neural tissue. Active contours, in both parametric and level set forms (Jurrus et al., 2009, Macke et al., 2008, Vazquez et al., 1998), can provide smooth, accurate segmentations of cells. However, they are very sensitive to initialization, which must be close to the neuron membrane, and often confuse internal structures for neuron membranes. If given an edge term that suppresses internal structures, such as one that is derived from

Method

The method developed here for neuron membrane detection combines ANN classifiers and image stencil neighborhood feature vectors. The following sections provide details on each of these components.

Results

Two TEM datasets are used as test cases for the proposed method. The first dataset is a stack of 50 sections from the ventral nerve cord of the C. elegans. The second dataset is a single section from the 16TB rabbit retina dataset. These datasets contain very different types of neural cells. The C. elegans data has a resolution of 6 nm × 6 nm × 33 nm and each 2-D section is 662 × 697 pixels. Neuron membranes in the C. elegans data appear as intensity valleys; however, not all valleys in the data are

Conclusion and future work

In this paper a new approach for neuron membrane detection is proposed. Inspired by Tu’s auto-context framework (Tu, 2008), our approach introduces two major contributions. The first contribution is the introduction of a serial ANN classifier and its application to neuron membrane detection. The use of context allows the classifier to close gaps in weak membranes and suppress intracellular structures by using increasingly non-local information with each ANN in the series. The second

Acknowledgments

This work was supported by NIH R01 EB005832 (T.T.), NIH EY0015128 (R.M.), EY002576 (R.M.), NEI Vision Core EY014800 (R.M.), HHMI (E.M.J.), and NIH NINDS 5R37NS34307-15 (E.M.J.). We also thank the anonymous reviewers whose comments helped improve this paper.

References (59)

  • J. Anderson et al.

    A computational framework for ultrastructural mapping of neural circuitry

    PLoS Biol.

    (2009)
  • B. Andres et al.

    Segmentation of SBFSEM volume data of neural tissue by hierarchical classification

  • Awate, S.P., Whitaker, R.T., 2006. Unsupervised, information-theoretic, adaptive image filtering for image restoration....
  • Awate, S.P., Tasdizen, T., Whitaker, R.T., 2006. Unsupervised texture segmentation with nonparametric neighborhood...
  • Borenstein, E., Sharon, E., Ullman, S., 2004. Combining top-down and bottom-up segmentation. In: Proceedings of the...
  • Buades, A., Coll, B., Morel, J.-M., 2005. A non-local algorithm for image denoising. In: Proceedings of IEEE Conference...
  • G.W. Cottrell

    Extracting features from faces using compression networks

  • G. Cybenko

    Approximation by superpositions of a sigmoidal function

    Math. Control, Signals, Syst.

    (1989)
  • W.M. DeBello et al.

    Adaptive axonal remodeling in the midbrain auditory space map

    J. Neurosci.

    (2001)
  • Dollar, P., Tu, Z., Belongie, S., 2006. Supervised learning of edges and object boundaries. In: IEEE Computer Society...
  • J.C. Fiala et al.

    Extending unbiased stereology of brain ultrastructure to three-dimensional volumes

    J. Am. Med. Inform. Assoc.

    (2001)
  • Fowlkes, C., Belongie, S., Chung, F., Malik, J., 2004. Spectral grouping using the Nyström method. In: IEEE...
  • Freund, Y., Schapire, R.E., 1995. A decision-theoretic generalization of on-line learning and an application to...
  • S. Haykin

    Neural Networks – A Comprehensive Foundation

    (1999)
  • Jain, V., Murray, J., Roth, F., Turaga, S., Zhigulin, V., Briggman, K., Helmstaedter, M., Denk, W., Seung, H., 2007....
  • Y. Jin et al.

    Control of type-D GABAergic neuron differentiation by C. elegans UNC-30 homeodomain protein

    Nature

    (1994)
  • B.W. Jones et al.

    Retinal remodeling triggered by photoreceptor degenerations

    J. Comp. Neurol.

    (2003)
  • B.W. Jones et al.

    Retinal remodelling

    Clin. Exp. Optom.

    (2005)
  • Jurrus, E., Whitaker, R., Jones, B., Marc, R., Tasdizen, T., 2008. An optimal-path approach for neural circuit...
  • Cited by (77)

    • Patient-specific modeling of blood flow in the coronary arteries

      2023, Computer Methods in Applied Mechanics and Engineering
    • ScanEV – A neural network-based tool for the automated detection of extracellular vesicles in TEM images

      2021, Micron
      Citation Excerpt :

      The term «neural networks» (NNs) denotes a class of algorithms, which mimic some aspects of animal brain functioning; these algorithms are highly efficient in image analysis and object detection (Al-Saffar et al., 2017; Anwar et al., 2018). In the field of electron microscopy, the NNs can be used for the detection of nanoparticles (Oktay and Gurses, 2019) and single molecules (Wagner et al., 2019), classification of images obtained by cellular electron cryo-tomography (Che et al., 2018), detection of neuron membranes (Jurrus et al., 2013, 2010), and in other applications. Development of a NN for object detection usually consists of several steps: preparation of the datasets (train, validation, and test), development of the network itself, including the preprocessing and post-processing steps, optimization, and testing.

    • A neuron image segmentation method based Deep Boltzmann Machine and CV model

      2021, Computerized Medical Imaging and Graphics
      Citation Excerpt :

      For example, the application in image recognition effectively improves the performance of image classification and target detection tasks (Krizhevsky et al., 2017; Zeiler and Fergus, 2014; Song et al., 2017; Aakur et al., 2017; Sugianto et al., 2018; Aravindh Mahendran and Andrea Vedaldi, 2016). Recently, these successful applications have been extended to semantic segmentation of dense predictive output (Tuzel et al., 2016; You et al., 2020; Shelhamer et al., 2017; Xing and Yang, 2016; Ronneberger et al., 2015), such as neuron membrane segment-ation of electron microscope images in the field of neuroscience research (Jurrus et al., 2010; Ahmed et al., 2021). Although the performance of various deep learning models in object recognition tasks has approached the human level, their performance in some dense prediction output problems still lags far behind human experts, especially in the brain connector’s construction involving high-dimensional and high-resolution electron microscope image analysis (Urakubo et al., 2019; Zhang et al., 2019; Xiao et al., 2018).

    • An adaptive genetic algorithm as a supporting mechanism for microscopy image analysis in a cascade of convolution neural networks

      2020, Applied Soft Computing Journal
      Citation Excerpt :

      Regardless of the use of the microscope, any magnification allows scientists to analyze very small biological structures. The obtained images are quite often of high resolutions with a very large number of structures and details [6]. One of the most popular analyzes is the counting of cells in a given sample or the analysis of the stage of their life [7].

    • Detection and segmentation in microscopy images

      2020, Computer Vision for Microscopy Image Analysis
    • Deep multilevel contextual networks for biomedical image segmentation

      2019, Handbook of Medical Image Computing and Computer Assisted Intervention
    View all citing articles on Scopus
    View full text