Multi-scale deep neural network for mitosis detection in breast cancer histological images

Accurate assessment of the breast cancer deterioration degree plays a crucial role in making medical plan, and the important basis for degree assessment is the number of mitoses in a given area of the pathological image. We utilized deep multi-scale fused fully convolutional neural network (MFF-CNN) combing with conditional random felid (CRF) to detect mitoses in hematoxylin and eosin stained histology image. Analyze the characteristics of mitotic detection ----scale invariance and sparsity, as well as the difficulties ---small amount of data , inconsistent image staining and sample class unbalanced. Based on this, mitotic detection model is designed. In this paper, a tissue-based staining equalization method is used, and to establish an effective training sample set, we select training samples by using CNN. A mitotic detection model fusing multi-level and multi-scale features and context information was designed, and the corresponding training strategy was made to reduce over-fitting. As preliminarily validated on the public 2014 ICPR MITOSIS data, our method achieves a better performance in term of detection accuracy than ever recorded for this dataset.


Introduction
Mitosis counting refers to the number of dividing cell identified in a fixed number of high power fields (40x magnification in this experiment), which is laborious, subjective [1].In hematoxylin and eosin (H&E) stained breast cancer sections, mitoses are discernible as hyperchromatic objects with dark color that lack clear nuclear membranes and have irregularity shape properties.In fact, mitosis is a complex process during which a cell nucleus undergoes four phase and exhibits highly variable appearance, moreover in most stages a mitotic nucleus looks like a non-mitotic nucleus shown in Fig.
1 [2].Therefore, the identification of mitosis may often suffer from disagreement between inter-observers.The computer-aid mitosis detector for breast cancer becomes a promising solution for these issues.Because it is a hard task to extract high-efficiency features manually, the performance of early studies based on handcraft features were not so impressive [3][4][5].In recent years, deep convolution neural networks (CNN) exhibits outstanding performance in classification [6][7] [8], which can learn high-level feature representation from the raw dataset.Nowadays, more and more researchers begin to apply CNN model to mitosis detection.
Fig. 1.The example of mitoses and non-mitoses (each column in red rectangle represents prophase, metaphase, anaphase and telophase of mitosis respectively, and non-mitoses with similarity appearance as mitosis were shown in the green rectangle) The latest studies show that deep CNN with hierachical feature representation has made breakthroughs in mitosis detection [9], Table .1 lists the papers using deep learning in mitosis detection.
Haibo Wang et al. proposed an cascaded strategy combining the handcraft features and the features learned from deep CNN model together to generate more comprehensive features for mitosis detection [10].This method was evaluated on the public ICPR12 mitosis dataset and yielded an F-measure of 0.7345 being the second best performance ever recorded in this dataset.
Ciresan et al. applied a DNN model to classify each pixel of the H&E stained mitosis images, consequently won the ICPR 2012 mitosis detection competition [11].They proposed an approach to select relatively rare challenging non-mitosis samples based on the output of DNN model, which allowing the model to learn the significant differences between mitotic and non-mitotic nuclei efficiently.However, this training dataset includes 95.4% non-mitosis class samples and only 6.6% mitosis samples, so the detector tend to classify the mitosis as non-mitosis.In addition, the pixel-wise classifier is time-consuming, costing roughly 8 minutes per image.
Hao Chen et al. designed a faster mitosis detector by leveraging the FCN model, moreover, they proposed an innovative cascaded CNN model to detect mitosis [12].The detection accuracy outperformed other methods by a large margin in 2014 ICPR MITOS-ATYPIA challenge.
However, above work didn't discuss the impact of the different level features from CNN on final accuracy.The complete breast cancer metastasis probability assessment process is shown in Fig. 2, including the interest region extraction steps, the staining normalization procedure, and the mitotic detection procedure.The area of interest has been given by medical professionals, and the goal of this paper is to design computer-aided system models for mitotic detection of breast cancer.Among above jobs, both the MFF-CNN+CRF model with corresponding training strategy are first time applied in the field of mitosis detection.The rest of this paper organized as follow.The section 2 will explain the new methodology in detailed.The section 3 will describe the experiment and exhibit the results.The section 4 will present the conclusion for this work.

Stain-Normalization and training dataset building
The inconsistency of stain condition makes the appearances of H&E stained histology drastically different, so the classify performance was degraded [13][14].For instance, many false mitosis may arise when the histopathology slide is over-stained.This experiment performed staining unmixing (separation of the hematoxylin and eosin stains) and normalized each hematoxylin and eosin image separately [16].This method was based on what was described in [17].First of all, we use the stacked autoencoder to transfer the feature space of image, and then use K-clustering method to represent all kinds of tissues and separate the different kinds of tissues.Finally, we stain normalized each type of tissue separately.The whole process is shown in Fig.To build an effective dataset , we used g the CNN model as samples selector by leveraging the cascaded selection strategy., we extracted the non-mitosis patches by leveraging the output probability images of the previous stage.The values of probability images represent the likelihood of mitosis and were used to the select the challenging samples.And then, we augmented the mitosis training samples by rotations of {0;45;90;135;180;215;270} degrees and translation.

Multi-scale Fused Fully Convolutional Network
The features from lower layers of CNN model responds to the general attributes such as edge, color, texture and so on, while the higher layer features are more classspecific and abstract [18][19] [20].Inspiring by the previous work which combined the handcrafted features and the features learned from deep CNN [15], this study proposed a sampler and efficient approach in which linked the output of The multi-scale FF-CNN model has two branches with different scale features and the final class prediction for each pixel is the joint prediction from all two branches.More contextual information could be captured by the larger scale branch, which helps detect more real mitotic samples.Combining predictions achieve better mitotic detection.As we know, it will be more difficult to train a multi-scale FCN model, so we    are the compress-ratio of the width and height respectively.Actually, the "compress-ratio" is roughly equal to the stride S of whole model .According to our compress-ratio, the value of displacement should be chosen as 32.In this up-sample approach, there will be 1024 32 32   operations to be implemented per image.The computation is redundant and time-consuming, finally, we set the sliding stride as 16 ,that is sliding number reduced to 4, ensuring a comparative accuracy at the same time.The method is explained in paper [21].

Layer Feature maps
Filter size Stride padding Learning rate input ---

FF-CNN+CRF model
Conditional random field model (CRF) can predict pixel-level labels according to the context of the image and globally observe the image.In this paper, the CRF model connected to the FF-CNN model.The FF-CNN+CRF-RNN model is shown as Figure 5.The FF-CNN predicts the class probability to each pixel, and the CRF-RNN uses the predicted information as its input to globally optimize the segmentation results and they based on the intensity and location information of each pixel.Firstly, the whole slice image is taken as the input of FFCNN, and two probability images  The stain-normalization method was leveraged to preprocess the images for the slice of A03, A07, A11, A14 , A15, A17 and A18, which made the edges of mitosis clearer and color consistent shown as Fig. 6, and to augment the training data for the slice of A04 and A05.

Evaluation and Performance
Evaluation was performed according to the ICPR 2014 content criteria, if the detected mitoses whose coordinates are closer than 30 pixels (8 m  ) to true mitoses are defined as true positives (TP).Those detection locating without 32 pixels of true mitosis are counted as false positive (FP).The true mitoses which were not detected by the model are defined as false negative (FN).We compute the performance measure including precision:

Fig. 2 .
Fig.2.The process of breast cancer metastasis probability assessment process

Fig. 3 .
Fig.3.The process of stain normalization based on tissue

EnlargeFig. 4 .
Fig. 4.An overview of proposed Multi-scale FF-CNN model combining different level and scale features together

1 )
Where M represents the number of the scale , here M=2; i  represents the weight of the ith scale., ,j jxy represents the value of the output and the real label respectivelyIn order to focus on the discrimination characteristics of each scale, we will initialize larger i  .Then we gradually lower i  and focus on the joint training.

Fig. 5 .
Fig. 5.The combination of MFF-CNN and CRF_RNN In this work, the performance of model was evaluated on 2014 ICPR MITOSIS dataset including 1,200 training images and 496 testing images.Due to different condition during the tissue acquisition process, the appearance of tissues are various, which makes the detection task more challenging.The spatial resolution of images acquired by the widely-used Apero-XT scanner is 0.25 pixel m /  and the magnification is 40.

FNN
refers to the number of true positive, false positive and false negative, respectively.We evaluated the performance of FF-CNN model and the oneway plain FCN model, FCN model, MFF-CNN model and MFF-CNN+CRF model and the effectiveness of frequency weighted loss function[20].

Table 1 .
The overview of papers using deep learning in mitosis detection

Table 2 .
The architecture parameter of final mitosis detector based on AlexNet FCN model

Table 3 .
Result of 2014 ICPR MITOSIS Datast The results of mitosis detection are showed in the Table.2("/" denotes that the results are not released).The MFF-CNN+CRF model yields the highest F-measure 0.437.In the experiment, we compared with MFF-CNN model and MFF-C+CRF model, we can clearly find that the number of misjudged mitosis greatly decreased from 135 to 105 shown in the Table.