Detection of neuron membranes in electron microscopy images using a serial neural network architecture
Introduction
Neural circuit reconstruction, i.e. the connectome (Sporns et al., 2005), is currently one of the grand challenges facing neuroscientists. Similarly, the National Academy of Engineering has listed reverse-engineering the brain as one its grand challenges.1 While neural circuits are central to the study of the nervous system, relatively little is known about differences in existing neuronal classes, patterns, and connections. Electron microscopy (EM) is an unique modality for scientists attempting to map the anatomy of individual neurons and their connectivity because it has a resolution that is high enough to identify synaptic contacts and gap junctions. These are important indicators for types of neuron topology and are required for neural circuit reconstruction. Several researchers have undertaken extensive EM imaging projects in order to create detailed maps of neuronal structure and connectivity (Fiala and Harris, 2001, Briggman and Denk, 2006a). Early work in this area, by White et al. (1986), includes the complete mapping of the nematode Caenorhabditis elegans nervous system. This is a simple organism, containing just over 300 neurons and 6000 synapses, yet it took nearly a decade to identify all the relevant structures and reconstruct the connectivity.2 In comparison, newer imaging techniques are producing much larger volumes of very complex organisms, with thousands of neurons and millions of synapses (Briggman and Denk, 2006b, Anderson et al., 2009). Thus, automating the reconstruction process is of paramount importance.
The ability to reconstruct neural circuitry at ultrastructural resolution is of substantial clinical importance. Retinal degenerative diseases, including pigmentosa and macular degeneration, result from a loss of photoreceptors. Photoreceptor cell stress and death induces subsequent changes in the neural circuitry of the retina resulting in corruption of the surviving retinal cell class circuitry. Ultrastructural examination of the cell identity and circuitry reveal substantial changes to retinal circuitry with implications for vision rescue strategies (Marc et al., 2008, Marc et al., 2007, Marc et al., 2003, Jones and Marc, 2005, Jones et al., 2003, Jones et al., 2005, Peng et al., 2000). These findings in retinal degenerative disease mirror findings in epilepsy where neural circuits also undergo remodeling in presumed response to abnormal electrical activity clinically manifested as seizures. Scientists are interested in examining normal and pathological synaptic connectivities and how neuronal remodeling contributes to neuronal pathophysiology (Sutula et al., 2002, Pollard et al., 1994, Koyama et al., 2004). Examination of synaptic and dendritic spine formation during development provide insight into the adaptivity of neural circuits (Sorra et al., 2000, DeBello et al., 2001). Ultrastructural evaluation of multiple canonical volumes of neural tissue are critical to evaluate differences in connectivity between wild type and mutants. The complexity and size of the these datasets, often approaching 17 terabytes, makes human segmentation of the complex textural information of electron microscopic imagery a difficult task. Moreover, population or screening studies become unfeasible since fully manual segmentation and analysis would require multiple years of manual effort per specimen. As a result, better image processing techniques are needed to help with automated segmentation of EM data including identification of neurons and the connections.
The modality we have chosen for reconstructing the connectome at the individual cell level is serial-section transmission electron microscopy (TEM). It provides scientists with images that capture the relevant structures; however, it poses some interesting challenges for image processing. Most importantly, serial-section TEM offers a relatively wide field of view to identify large sets of cells that may wander significantly as they progress through the sections. It also has an in-plane resolution that is high enough for identifying synapses. In collecting images through TEM, sections are cut from a specimen and suspended so that an electron beam can pass through it creating a projection. The projection can be captured on a piece of film and scanned or captured directly as a digital image. An important trade-off occurs with respect to the section thickness. Thinner sections are preferable from an image analysis point of view because structures are more easily identifiable due to less averaging. However, from an acquisition point of view, thinner sections are harder to handle and impose a limit on the area of the section that can be cut. For instance, in the rabbit retina, scientists need to study sections with areas as large as 250 μm in diameter to gain a sufficient understanding of neural connectivity patterns. Sections of this size can be reliably cut at 50–90 nm thickness with the current serial-section TEM technology. This leads to an extremely anisotropic resolution, 2–5 nm in-plane compared to 50–90 nm out-of-plane, and poses two image processing challenges. First, the cell membranes can range from solid dark curves for neurons that run approximately perpendicular to the cutting-plane, to grazed grey swaths for others which run more obliquely and suffer more from the averaging effect. Consequently, segmentations of neurons in these 2-D images, are difficult given the change in membrane contrast and thickness. Second, due to the large physical separation between sections, shapes and positions of neurons can change significantly between adjacent sections.
There are alternative specimen preparation and EM imaging techniques that can be used for neural circuit reconstruction such as Serial-Block Face Scanning Electron Microscopy. Briggman and Denk proposed a specimen preparation which only highlights extracellular spaces removing almost all contrast from intracellular structures (Briggman and Denk, 2006b). However, it is not possible to identify synapses with that approach. Identification of synapses is an important part of neural circuit reconstruction because it determines which cells are communicating, and where in the circuitry they connect. To highlight synapses in TEM, scientists must use a stain that also highlights intracellular structures, such as vesicles and mitochondria, as well as neuron membranes. Therefore, image segmentation techniques must account for these data characteristics in order to identify and successfully track neurons across hundreds of sections.
There are two general approaches for neuron segmentation. One approach focuses first on the detection of neuron membranes in each 2-D section (Jurrus et al., 2008, Macke et al., 2008, Venkataraju et al., 2009). These boundaries can be used to identify individual neurons, which are then linked across sections to form a complete neuron. Unfortunately, accurate detection of neuron membranes in EM is a difficult problem given the presence of intracellular structures. This makes simple thresholding, edge detection (i.e. Canny), and region growing methods ineffective for the detection of neuron membranes. Some example images and results with traditional image processing methods are shown in Fig. 1. The other approach to neuron segmentation is to directly use the 3-D characteristics of the data (Andres et al., 2008, Jain et al., 2007). However, full 3-D approaches are difficult due to the anisotropic nature of the data. As mentioned earlier, in serial-section EM, there is a trade-off between section thickness and section loss rate. The datasets used in this paper to demonstrate membrane detection are from the C. elegans ventral nerve cord and from the rabbit retina. For these datasets, the nerve cord has a resolution of 6 nm × 6 nm × 33 nm and the retina has a resolution of 2 nm × 2 nm × 80 nm. This large section thickness often causes features to shift significantly between sequential images, decreasing the potential advantages of a direct 3-D approach. For these reasons, we follow the first approach which is to first perform a 2-D segmentation followed by a linking of the segmented regions in 3-D. This approach is particularly suitable for datasets in which a majority of the neurons run in a general direction which is roughly orthogonal to the sectioning plane such as the datasets considered in this paper. The main focus of this paper is to improve the 2-D neuron segmentation in each section. This information can then be used to link the segmentation in each section to obtain the full 3-D reconstruction.
Recent related work indicates that machine learning methods are an effective approach for detection of neuron membranes. These methods all use different representations for learning membrane pixels, most of which include training a single instance of a classifier on image derived features, such as Hessian eigenspaces (Venkataraju et al., 2009, Mishchenko, 2009) and local statistical features (Andres et al., 2008). Inspired by Tu’s auto-context shape classification approach (Tu, 2008), the method described in this paper uses a series of classifiers to more accurately detect membranes in EM images, which is a necessary step for improved 3-D neuron segmentation as discussed above. However, unlike Tu’s auto-context (Tu, 2008) which uses boosting to select features from a large pool of candidates such as Haar wavelet responses, we use a series of artificial neural networks (ANNs) that operate on a fixed set of features. The first ANN uses as input the intensity values sampled directly from the image. The input to the subsequent ANNs in the series is comprised of the same set of image values, in addition to the output of the previous ANN sampled on a stencil of nearby pixels (as depicted in Fig. 4). The ANNs in the series, therefore, have different inputs even though they have a common desired output. The advantages of this method are twofold. First, the classifier uses raw data, that is, the image intensities, rather than a constrained version of the image as given by responses to a large filter bank or statistical features that will not scale well for large datasets. Second, the use of the serial ANNs provides context, which is information from nearby pixels that contributes to the learning, providing increasing amounts of relative information at each stage of the network. As a result, the series of ANNs learns to remove vesicles and mitochondria from the membrane detection and close gaps in places where the membrane is weak. In this paper, we demonstrate the improvement from the combined use of stencils and the series of ANNs for two datasets with distinctly different characteristics.
Section snippets
Related work
There are several methods that attempt to segment EM images of neural tissue. Active contours, in both parametric and level set forms (Jurrus et al., 2009, Macke et al., 2008, Vazquez et al., 1998), can provide smooth, accurate segmentations of cells. However, they are very sensitive to initialization, which must be close to the neuron membrane, and often confuse internal structures for neuron membranes. If given an edge term that suppresses internal structures, such as one that is derived from
Method
The method developed here for neuron membrane detection combines ANN classifiers and image stencil neighborhood feature vectors. The following sections provide details on each of these components.
Results
Two TEM datasets are used as test cases for the proposed method. The first dataset is a stack of 50 sections from the ventral nerve cord of the C. elegans. The second dataset is a single section from the 16TB rabbit retina dataset. These datasets contain very different types of neural cells. The C. elegans data has a resolution of 6 nm × 6 nm × 33 nm and each 2-D section is 662 × 697 pixels. Neuron membranes in the C. elegans data appear as intensity valleys; however, not all valleys in the data are
Conclusion and future work
In this paper a new approach for neuron membrane detection is proposed. Inspired by Tu’s auto-context framework (Tu, 2008), our approach introduces two major contributions. The first contribution is the introduction of a serial ANN classifier and its application to neuron membrane detection. The use of context allows the classifier to close gaps in weak membranes and suppress intracellular structures by using increasingly non-local information with each ANN in the series. The second
Acknowledgments
This work was supported by NIH R01 EB005832 (T.T.), NIH EY0015128 (R.M.), EY002576 (R.M.), NEI Vision Core EY014800 (R.M.), HHMI (E.M.J.), and NIH NINDS 5R37NS34307-15 (E.M.J.). We also thank the anonymous reviewers whose comments helped improve this paper.
References (59)
- et al.
Towards neural circuit reconstruction with volume electron microscopy techniques
Curr. Opin. Neurobiol.
(2006) - et al.
Towards neural circuit reconstruction with volume electron microscopy techniques
Curr. Opin. Neurobiol.
(2006) Approximation capabilities of multilayer feedforward networks
Neural Networks
(1991)- et al.
Retinal remodeling during retinal degeneration
Exp. Eye Res.
(2005) - et al.
Axon tracking in serial block-face scanning electron microscopy
Med. Image Anal.
(2009) - et al.
Contour-propagation algorithms for semi-automated reconstruction of neural processes
J. Neurosci. Methods
(2008) - et al.
Neural remodeling in retinal degeneration
Prog. Retinal Eye Res.
(2003) - et al.
Correlation between reactive sprouting and microtubule protein expression in epileptic hippocampus
Neuroscience
(1994) - et al.
Promising research: vision-based robot positioning using neural networks
Image Vis. Comput.
(1996) - et al.
The sensory circuitry for sexual attraction in C. elegans males
Curr. Biol.
(2007)
A computational framework for ultrastructural mapping of neural circuitry
PLoS Biol.
Segmentation of SBFSEM volume data of neural tissue by hierarchical classification
Extracting features from faces using compression networks
Approximation by superpositions of a sigmoidal function
Math. Control, Signals, Syst.
Adaptive axonal remodeling in the midbrain auditory space map
J. Neurosci.
Extending unbiased stereology of brain ultrastructure to three-dimensional volumes
J. Am. Med. Inform. Assoc.
Neural Networks – A Comprehensive Foundation
Control of type-D GABAergic neuron differentiation by C. elegans UNC-30 homeodomain protein
Nature
Retinal remodeling triggered by photoreceptor degenerations
J. Comp. Neurol.
Retinal remodelling
Clin. Exp. Optom.
Cited by (77)
Patient-specific modeling of blood flow in the coronary arteries
2023, Computer Methods in Applied Mechanics and EngineeringScanEV – A neural network-based tool for the automated detection of extracellular vesicles in TEM images
2021, MicronCitation Excerpt :The term «neural networks» (NNs) denotes a class of algorithms, which mimic some aspects of animal brain functioning; these algorithms are highly efficient in image analysis and object detection (Al-Saffar et al., 2017; Anwar et al., 2018). In the field of electron microscopy, the NNs can be used for the detection of nanoparticles (Oktay and Gurses, 2019) and single molecules (Wagner et al., 2019), classification of images obtained by cellular electron cryo-tomography (Che et al., 2018), detection of neuron membranes (Jurrus et al., 2013, 2010), and in other applications. Development of a NN for object detection usually consists of several steps: preparation of the datasets (train, validation, and test), development of the network itself, including the preprocessing and post-processing steps, optimization, and testing.
A neuron image segmentation method based Deep Boltzmann Machine and CV model
2021, Computerized Medical Imaging and GraphicsCitation Excerpt :For example, the application in image recognition effectively improves the performance of image classification and target detection tasks (Krizhevsky et al., 2017; Zeiler and Fergus, 2014; Song et al., 2017; Aakur et al., 2017; Sugianto et al., 2018; Aravindh Mahendran and Andrea Vedaldi, 2016). Recently, these successful applications have been extended to semantic segmentation of dense predictive output (Tuzel et al., 2016; You et al., 2020; Shelhamer et al., 2017; Xing and Yang, 2016; Ronneberger et al., 2015), such as neuron membrane segment-ation of electron microscope images in the field of neuroscience research (Jurrus et al., 2010; Ahmed et al., 2021). Although the performance of various deep learning models in object recognition tasks has approached the human level, their performance in some dense prediction output problems still lags far behind human experts, especially in the brain connector’s construction involving high-dimensional and high-resolution electron microscope image analysis (Urakubo et al., 2019; Zhang et al., 2019; Xiao et al., 2018).
An adaptive genetic algorithm as a supporting mechanism for microscopy image analysis in a cascade of convolution neural networks
2020, Applied Soft Computing JournalCitation Excerpt :Regardless of the use of the microscope, any magnification allows scientists to analyze very small biological structures. The obtained images are quite often of high resolutions with a very large number of structures and details [6]. One of the most popular analyzes is the counting of cells in a given sample or the analysis of the stage of their life [7].
Detection and segmentation in microscopy images
2020, Computer Vision for Microscopy Image AnalysisDeep multilevel contextual networks for biomedical image segmentation
2019, Handbook of Medical Image Computing and Computer Assisted Intervention