Research on Detection of Low-Resolution Facial Images Based on Improved MTCNN

1. Introduction

1.1. Research Status

Face detection and recognition have become fundamental components of modern biometric systems, underpinning a wide range of applications such as security surveillance, identity authentication, human-computer interaction, and intelligent access control. The ability to automatically detect and recognize human faces in unconstrained environments has become increasingly critical with the proliferation of surveillance cameras and the growing demand for intelligent video analysis systems. Within this domain, face detection—the prerequisite step for recognition—has garnered substantial research attention due to its direct impact on overall system performance.

Face detection methodologies have undergone a paradigm shift from handcrafted feature-based approaches to deep learning-driven solutions over the past two decades. The Multi-task Cascaded Convolutional Neural Network (MTCNN), introduced by Zhang et al. in 2016, represents a landmark contribution that effectively balances detection accuracy and computational efficiency through its three-stage cascade architecture. MTCNN jointly performs face detection and alignment using a coarse-to-fine framework, achieving superior performance compared to contemporary methods.

International research on MTCNN has progressed along multiple directions. The original MTCNN framework [1] established the fundamental three-stage architecture with P-Net, R-Net, and O-Net operating in cascade, utilizing online hard sample mining and multi-task learning with face classification, bounding box regression, and facial landmark localization. Subsequent works have extended MTCNN to address various challenges. Zhang et al. [2] proposed an enhanced version incorporating feature pyramid networks for multi-scale face detection. Wang et al. [3] integrated attention mechanisms to improve detection under occlusion. Recent advances have explored lightweight MTCNN variants for mobile deployment [4] and domain adaptation techniques for cross-dataset generalization [5].

Domestic research in China has made significant contributions to MTCNN advancement. Li et al. [6] developed a dual-path MTCNN architecture for simultaneous face detection and attribute estimation. Chen et al. [7] introduced a progressive calibration network built upon MTCNN for handling extreme poses. Liu et al. [8] proposed a feature enhancement module specifically designed for small face detection. The integration of MTCNN with other computer vision tasks has also been explored, including its application in pedestrian detection [9], multi-object tracking [10], and facial expression recognition [11].

The development trajectory from traditional methods to deep learning approaches reflects the broader evolution of computer vision. Classical face detection methods relied on handcrafted features such as Haar-like features with AdaBoost classifiers [12], Histogram of Oriented Gradients (HOG) with Support Vector Machines [13], and Local Binary Patterns (LBP) [14]. While these approaches achieved reasonable performance under controlled conditions, they struggled with the variability inherent in real-world scenarios. The advent of convolutional neural networks marked a transformative shift, with methods like Cascade CNN [15], Faceness [16], and Faster R-CNN [17] demonstrating substantial improvements. The emergence of MTCNN represented a milestone in achieving both accuracy and efficiency, inspiring numerous subsequent developments in cascade detection architectures.

1.2. Research Content and Main Challenges

Despite significant progress, face detection in unconstrained environments continues to face formidable challenges. Surveillance scenarios present particularly demanding conditions characterized by:

• Illumination Variations: Uneven lighting, shadows, and extreme brightness conditions severely impact image quality and feature visibility. Faces captured under poor illumination lose critical textural information essential for reliable detection.
• Pose Diversity: Non-frontal faces with significant yaw, pitch, and roll angles exhibit substantial appearance variations. Profile faces, looking-down poses, and extreme rotations cause self-occlusion and feature deformation that challenge detection algorithms.
• Occlusion: Partial occlusion by objects, other individuals, or environmental elements obscures facial regions, leading to incomplete feature representation and increased false negative rates.
• Low Resolution: Perhaps the most pervasive challenge in surveillance applications is the prevalence of small-scale faces. Targets far from camera positions appear with limited pixels (often below $30\times 30$), lacking sufficient detail for reliable detection. The combination of small scale with motion blur and compression artifacts further compounds the difficulty.
• Complex Backgrounds: Surveillance scenes contain cluttered backgrounds with numerous objects sharing visual similarities with faces and pedestrians, increasing false positive rates.

To address these challenges, this paper proposes an enhanced MTCNN-based detection framework with three specific objectives: (1) designing a unified architecture for simultaneous pedestrian and face detection with spatial correspondence; (2) developing multi-level feature fusion to enhance representation for tiny, blurry, low-resolution faces; and (3) maintaining real-time processing speed (>30 FPS) suitable for surveillance deployment. The main innovations of this work are threefold:

Innovation 1: Unified Pedestrian-Face Detection Framework. The original MTCNN architecture is fundamentally restructured to transition from face-specific detection to integrated pedestrian and face detection. By establishing a “pedestrian-first, face-second” detection pipeline, the algorithm leverages the natural spatial relationship between pedestrian bodies and faces, reducing search space and computational overhead while ensuring target correspondence.

Innovation 2: Multi-Level Feature Fusion for Tiny Face Detection. A novel feature fusion mechanism is incorporated into the face detection subnet that systematically combines features from multiple network layers. This approach integrates high-level semantic features with low-level detailed information, compensating for the semantic loss in deep layers when processing small targets and significantly improving detection rates for low-resolution faces.

Innovation 3: Pyramid Pooling with Scale-Invariant Architecture. The final-stage network is redesigned as a pyramid pooling-based convolutional neural network incorporating RoIPooling layers, enabling simultaneous processing of pedestrian and face targets at arbitrary scales without distortion-causing resizing. Batch Normalization layers are strategically placed to enhance classification performance and training stability.

2. Materials and Methods

2.1. Basic Pipeline of Face Recognition Systems

A complete face recognition system encompasses a sequential pipeline of processing stages, each contributing to the overall system performance:

• Image Acquisition: The initial stage involves capturing images or video frames from imaging devices. In surveillance contexts, this typically involves fixed cameras with varying resolutions, frame rates, and environmental conditions. Image quality at acquisition fundamentally constrains subsequent processing capabilities.
• Face Detection: The detection stage identifies regions within images that contain faces, outputting bounding boxes that localize facial regions. This critical first step determines the input for all subsequent processing; missed detections cannot be recovered, while false detections propagate through the system.
• Image Preprocessing: Detected face regions undergo normalization to reduce variability and enhance feature quality. Common preprocessing operations include geometric normalization (scaling, rotation correction), photometric normalization (illumination compensation, contrast enhancement), and color space transformation.
• Feature Extraction: This stage transforms preprocessed face images into compact, discriminative feature representations. Traditional approaches employed handcrafted features, while modern systems utilize deep neural networks to learn hierarchical representations optimized for recognition tasks.
• Classification and Recognition: The final stage compares extracted features against enrolled templates to determine identity. This may involve closed-set identification (matching against known individuals), verification (confirming claimed identity), or open-set recognition (handling unknown subjects).

2.2. Common Face Databases

Benchmark datasets play an essential role in developing and evaluating face detection and recognition algorithms. The following databases are widely adopted in the research community.

2.2.1. 2D Face Databases

• WIDER Face Dataset: Contains 32,203 images with 393,703 labeled faces exhibiting extreme variability in scale, pose, occlusion, and illumination. The dataset is partitioned into Easy, Medium, and Hard subsets based on detection difficulty, enabling fine-grained performance analysis across target scales.
• SCFace (Surveillance Camera Face) Dataset: Specifically designed for surveillance applications, containing faces captured at varying distances (1m to 60m) with corresponding resolution degradation. The 4.2m subset is particularly challenging for low-resolution face recognition.

Figure 1. SCFace Dataset: sample images captured at varying distances demonstrating resolution degradation.

Figure 1. SCFace Dataset: sample images captured at varying distances demonstrating resolution degradation.

• LFW (Labeled Faces in the Wild): Contains 13,233 web-collected images of 5,749 identities, establishing a standard benchmark for unconstrained face verification.

Figure 2. LFW Dataset: sample face images collected in unconstrained conditions.

Figure 2. LFW Dataset: sample face images collected in unconstrained conditions.

• FDDB (Face Detection Data Set and Benchmark): Comprises 2,845 images with 5,171 face annotations, providing a challenging evaluation platform with significant pose and occlusion variations.

2.2.2. 3D Face Databases

• Bosphorus Database: Contains 4,666 faces from 105 subjects with various expressions, poses, and occlusion conditions captured using 3D scanning technology.
• FRGC v2.0: Includes 4,007 3D face scans from 466 subjects, supporting research on 3D face recognition and multimodal fusion.

2.2.3. Face Anti-Spoofing Databases

• CASIA-SURF: Large-scale multimodal database for face anti-spoofing containing 21,000 videos from 1,000 subjects.

Figure 3. CASIA WebFace Dataset: sample images for face anti-spoofing research.

Figure 3. CASIA WebFace Dataset: sample images for face anti-spoofing research.

• Replay-Attack: Contains 1,300 videos of both real accesses and attack attempts, supporting presentation attack detection research.

2.3. Evaluation Metrics

Comprehensive evaluation of face detection algorithms requires multiple metrics capturing different aspects of performance.

2.3.1. Precision and Recall

Precision measures the proportion of correctly detected faces among all detections:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(1)

Recall (also termed Detection Rate) measures the proportion of faces successfully detected among all ground-truth faces:

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(2)

where $TP$ (True Positives) are correct detections (IoU > 0.5 with ground truth), $FP$ (False Positives) are erroneous detections, and $FN$ (False Negatives) are missed faces.

2.3.2. Intersection over Union (IoU)

The overlap between predicted bounding box $B_{\mathrm{pred}}$ and ground-truth box $B_{\mathrm{gt}}$ is defined as:

$$\mathrm{IoU}={\displaystyle \frac{\mathrm{area}(BBo{x}_{\mathrm{pred}}\cap BBo{x}_{\mathrm{gt}})}{\mathrm{area}(BBo{x}_{\mathrm{pred}}\cup BBo{x}_{\mathrm{gt}})}}$$

(3)

2.3.3. ROC Curves and AUC

Receiver Operating Characteristic (ROC) curves plot True Positive Rate against False Positive Rate across detection thresholds, with Area Under Curve (AUC) providing a single-value performance measure.

2.3.4. FAR-TAR Characteristics

For recognition systems, False Acceptance Rate (FAR) and True Acceptance Rate (TAR) characterize verification performance at varying thresholds.

2.3.5. Equal Error Rate (EER)

The Equal Error Rate (EER) is the operating point where False Acceptance Rate equals False Rejection Rate, providing a threshold-independent performance measure.

3. MTCNN Face Detection Algorithm

3.1. Traditional Face Detection Methods

Before the dominance of deep learning, face detection relied on handcrafted features and classical machine learning techniques.

• AdaBoost-based Detection: The Viola–Jones framework [12] revolutionized face detection by introducing integral images for rapid feature computation, AdaBoost for feature selection, and cascade architecture for efficient processing. Haar-like features capturing edge, line, and center-surround patterns were combined in a boosting framework to create increasingly powerful classifiers.
• Haar Features and Integral Images: Haar features compute differences between sums of pixel intensities in rectangular regions, efficiently calculated using integral images that enable constant-time feature computation regardless of region size. The cascade structure progressively eliminates non-face regions, focusing computational resources on promising candidates.
• HOG-SVM Method: Dalal and Triggs [13] introduced Histogram of Oriented Gradients features that capture local gradient distributions, demonstrating particular effectiveness for pedestrian detection and extending to face detection applications. Combined with Support Vector Machine classifiers, this approach achieved robust performance under varying illumination conditions.

3.2. Deep Learning-Based Face Detection

• Cascade CNN Architecture: Cascade CNN [15] extended the cascade concept to deep learning, employing multiple CNN stages operating in sequence. Each stage rejects a substantial portion of non-face regions while refining remaining candidates, achieving efficiency through progressive processing.
• MTCNN Architecture: The Multi-task Cascaded Convolutional Neural Network [1] represents a sophisticated cascade design with three stages:

  -

  Stage 1 (P-Net): A fully convolutional network that rapidly scans the image pyramid to generate candidate face windows with associated bounding box regressions.

  -

  Stage 2 (R-Net): A more discriminative network that refines candidates from Stage 1, rejecting false positives and performing additional bounding box regression.

  -

  Stage 3 (O-Net): The most powerful network that outputs final face classification, bounding box refinement, and facial landmark localization.

Figure 4. The original MTCNN flowchart illustrating the three-stage cascade architecture: P-Net, R-Net, and O-Net.

Figure 4. The original MTCNN flowchart illustrating the three-stage cascade architecture: P-Net, R-Net, and O-Net.

MTCNN’s key contributions include: (i) a three-stage cascade architecture (P-Net → R-Net → O-Net) that progressively refines candidates from coarse to fine; (ii) joint multi-task learning for face classification, bounding box regression, and landmark localization; and (iii) online hard sample mining to address class imbalance during training. These properties make MTCNN a strong baseline for cascade-based detection research.

• Single-Stage vs. Two-Stage Detectors: Single-stage detectors (e.g., SSD [18], YOLO [19]) perform classification and regression in a single network pass, prioritizing speed over accuracy. Two-stage detectors (e.g., Faster R-CNN [17]) first generate region proposals then classify them, achieving higher accuracy at computational cost. MTCNN’s multi-stage design occupies a middle ground, with each stage progressively refining proposals.

3.3. Low-Resolution Face Detection Challenges

Low-resolution faces (typically below $30\times 30$ pixels) present fundamental challenges for CNN-based detectors. Successive downsampling operations in deep networks reduce spatial resolution, potentially causing small faces to vanish entirely from later feature maps [20]. Furthermore, low-resolution faces exhibit reduced contrast, blurred edges, and missing fine-grained details, making them difficult to distinguish from face-like background patterns. These limitations directly motivate the multi-level feature fusion and pyramid pooling designs proposed in Section 6.

4. Face Image Preprocessing

4.1. Image Enhancement and Normalization

• Illumination Normalization Methods: Photometric variations significantly impact face detection performance. Illumination normalization techniques aim to reduce lighting-induced variability while preserving discriminative facial features. Common approaches include histogram equalization, which redistributes pixel intensities to enhance contrast, and more sophisticated methods such as Difference of Gaussian (DoG) filtering that suppress low-frequency illumination variations while preserving high-frequency facial details.
• Homomorphic Filtering and Gamma Correction: Homomorphic filtering operates in the frequency domain, treating illumination as low-frequency components and reflectance as high-frequency components. By attenuating low frequencies and amplifying high frequencies, this approach simultaneously normalizes illumination and enhances edge information.
• Scale Decomposition and Energy Models: Multi-scale decomposition methods separate images into components at different spatial frequencies, enabling targeted processing. The Retinex theory-inspired approaches estimate illumination as a smooth base layer, removing it to obtain reflectance components invariant to lighting conditions.

4.2. Face Alignment and Landmark Detection

• 2D Facial Landmark Detection: Locating key facial points (eye corners, nose tip, mouth corners) enables geometric normalization through similarity transformations that map detected landmarks to canonical positions. Modern approaches employ regression-based methods [21] and heatmap-based CNNs that achieve high accuracy even under challenging conditions.
• 3D Facial Landmark Detection: Extending to three dimensions, 3D landmark detection estimates points in 3D space, enabling more complete pose normalization. Depth information from 3D sensors or reconstructed from 2D images provides robustness to extreme poses.
• 3D Alignment Methods (e.g., DeepFace): The DeepFace system [22] pioneered 3D alignment by fitting a 3D face model to detected 2D landmarks, enabling frontalization that renders faces in canonical pose. This approach significantly reduces pose-induced variability and improves recognition performance for non-frontal faces.

4.3. Image Super-Resolution Reconstruction

• Low-Resolution Face Restoration: Super-resolution techniques aim to reconstruct high-resolution faces from low-resolution inputs, potentially enhancing detection and recognition performance. Traditional interpolation methods (bilinear, bicubic) provide minimal information gain, while learning-based approaches can infer missing high-frequency details from training data.
• SR-DCR Super-Resolution Methods: Super-Resolution via Deep Convolutional Networks with Residual learning [23] learns end-to-end mappings from low to high resolution, reconstructing facial details that improve downstream recognition accuracy. In the proposed framework, SR-DCR serves as an optional preprocessing module for low-resolution inputs; its integration and evaluation are detailed in Section 6.

• Generative Adversarial Networks in Face Enhancement: GAN-based super-resolution [24] employs adversarial training to generate perceptually realistic high-resolution faces. The generator produces super-resolved images while the discriminator attempts to distinguish them from real high-resolution faces, pushing the generator toward producing convincing facial details. While computationally intensive, GAN-based approaches can generate particularly realistic face enhancements beneficial for human viewing applications.

5. Face Feature Extraction

5.1. Traditional Feature Extraction Methods

• Local Binary Patterns (LBP) and Variants: LBP [14] encodes local texture by thresholding pixel neighborhoods against the central pixel value, producing binary patterns that capture micro-structures. Uniform LBP reduces dimensionality by grouping patterns with limited transitions, while multi-scale LBP extends the approach to multiple radii. LBP’s computational efficiency and illumination robustness made it widely adopted in early face recognition systems.
• Gabor Filter Features: Gabor filters with different orientations and scales provide multi-resolution analysis capturing spatial frequency and orientation information. The convolution of face images with Gabor filter banks produces complex coefficients whose magnitudes represent energy at specific frequencies and orientations, providing rich representation for face discrimination.
• HOG Features: Histogram of Oriented Gradients [13] computes gradient orientation distributions over local cells, normalizing across larger blocks to achieve illumination invariance. While primarily developed for pedestrian detection, HOG features have proven effective for face representation, particularly when combined with appropriate classifiers.

5.2. Subspace Learning Methods

• Principal Component Analysis (PCA) and Eigenfaces: PCA [25] identifies orthogonal directions of maximum variance in face data, projecting faces into a lower-dimensional subspace. Eigenfaces, the principal components of face images, provide a compact representation where reconstruction coefficients serve as features. While sensitive to alignment and illumination, PCA established the foundation for appearance-based face recognition.
• Linear Discriminant Analysis (LDA) and Fisherface: LDA [26] optimizes class separability by maximizing between-class scatter while minimizing within-class scatter. The Fisherface method applies PCA for dimensionality reduction followed by LDA, achieving improved discrimination compared to PCA alone.
• Independent Component Analysis (ICA): ICA [27] seeks statistically independent basis components rather than merely uncorrelated ones, potentially capturing more meaningful facial features. ICA-based representations have demonstrated advantages for face recognition under certain conditions.

5.3. Deep Feature Learning

• CNN Architectures (DeepFace, FaceNet, etc.): DeepFace [22] represented a milestone in deep learning for face recognition, achieving near-human performance through a nine-layer CNN trained on massive face datasets. FaceNet [28] introduced triplet loss training, directly optimizing embedding distances such that faces of the same identity are closer than faces of different identities. Subsequent architectures have explored deeper networks, attention mechanisms, and efficient designs for mobile deployment.
• Vision Transformer Applications: Recent advances have applied Transformer architectures [29] to face recognition, treating face images as sequences of patches and employing self-attention mechanisms to capture global relationships. Vision Transformers have demonstrated competitive performance with CNN-based approaches while offering different architectural trade-offs.
• Loss Function Design: The evolution of loss functions has driven substantial progress in deep face recognition. Softmax loss provides basic class separation, while center loss [30] adds intra-class compactness by penalizing distances from class centers. Angular margin losses including ArcFace [31], CosFace [32], and SphereFace [33] directly optimize angular decision boundaries, achieving state-of-the-art performance by enhancing feature discriminability.

5.4. Multi-Feature Fusion Strategies

• Global and Local Feature Fusion: Combining holistic face representations with local feature descriptions captures complementary information. Global features encode overall face structure, while local features (eyes, nose, mouth regions) capture detailed discriminative information. Fusion at feature level or score level enables comprehensive face representation.
• Multi-Level Feature Fusion: Combining features from different network depths addresses the information loss inherent in deep architectures. Low-level layers preserve spatial details beneficial for small-target detection, while high-level layers provide semantic context. This principle underpins the hierarchical fusion mechanism designed for Impro-R-Net2, described in Section 6.
• Multiple Classifier Combination: Ensemble methods combine decisions from multiple classifiers to improve robustness and accuracy. Techniques include voting schemes, weighted combinations based on classifier confidence, and stacking where meta-classifiers learn to combine base classifier outputs.

6. Proposed Method

6.1. Motivation and Problem Formulation

The original MTCNN framework, while effective for frontal face detection under controlled conditions, exhibits three critical limitations in unconstrained surveillance scenarios. First, its face-only detection objective ignores the spatial prior that faces are spatially constrained to pedestrian body regions, leading to exhaustive full-image search and high false positive rates from face-like background patterns. Second, successive pooling operations in P-Net and R-Net reduce feature map resolution aggressively, causing small faces (below $30\times 30$ pixels) to vanish from later layers—a problem quantified by the 60.7% Hard-subset detection rate on WIDER Face (Table 6), which lags the Easy-subset rate by over 24 percentage points. Third, fixed-size input requirements in O-Net necessitate distortion- causing resizing of candidate regions at varying scales, degrading classification accuracy for non-canonical aspect ratios.

To address these limitations, the proposed framework introduces three targeted design choices: (1) a “pedestrian-first, face-second” cascade that exploits body–face spatial correlation to constrain the face search space; (2) a multi-level feature fusion mechanism that preserves low-level spatial details across deep layers for small-target representation; and (3) a pyramid pooling architecture with RoIPooling that enables scale-invariant processing without resizing distortion. Table 1 summarizes the architectural differences between the original MTCNN and the proposed framework.

6.2. Overall Framework

Design Rationale: The proposed method addresses the specific requirements of surveillance applications where simultaneous pedestrian and face detection with target correspondence is essential. Rather than treating pedestrian detection and face detection as independent tasks, our framework establishes a hierarchical relationship where face detection operates within pedestrian regions. This design leverages the natural spatial correlation: faces appear on pedestrians, and pedestrian detection provides strong priors for face localization.

While preserving MTCNN’s three-stage cascade philosophy, the proposed framework fundamentally restructures each stage’s objective: Stage 1 (Impro-P-Net) generates pedestrian candidates; Stage 2 (Impro-R-Net1 + Impro-R-Net2) filters false pedestrian detections and performs preliminary face detection within verified pedestrian regions; Stage 3 (Impro-O-Net) jointly refines pedestrian and face detections to produce final outputs with established spatial correspondence. This “pedestrian-first, face-second” approach reduces search space for face detection, ensures pedestrian–face correspondence, and improves computational efficiency.

System Flowchart:Figure 5 illustrates the complete algorithm workflow. Input images undergo initial pedestrian detection through the improved P-Net (Impro-P-Net), generating candidate pedestrian regions. These regions pass to Stage 2, where Impro-R-Net1 filters false pedestrian detections while Impro-R-Net2 performs preliminary face detection within verified pedestrian areas. Stage 3’s Impro-O-Net simultaneously refines both pedestrian and face detections, outputting final results with established correspondence.

6.3. Algorithm Module Details

6.3.1. Module 1: Impro-P-Net — Pedestrian Preliminary Detection

Impro-P-Net redesigns the original P-Net architecture for pedestrian-specific detection, accounting for pedestrian appearance characteristics including aspect ratio, scale distribution, and pose variation. The network architecture (Figure 6) comprises four convolutional layers with attached max pooling after Conv1, Conv2, and Conv3.

Key innovations in Impro-P-Net include:

1.

Dual Aspect Ratio Convolution Kernels: Pedestrian targets exhibit significant height-to-width ratio variation: standing pedestrians present approximately 2:1 ratios, while sitting or partially occluded pedestrians approach 1:1. To capture this variability, Impro-P-Net employs both 1:1 and 1:2 aspect ratio convolution kernels in Conv1, Conv2, and Conv3. Formally, let ${\mathbf{W}}_{sq}\in {\mathbb{R}}^{k\times k\times C_{in}\times C_{out}}$ denote the square kernels and ${\mathbf{W}}_{rect}\in {\mathbb{R}}^{k\times 2k\times C_{in}\times C_{out}}$ denote the rectangular kernels. The fused feature map at each convolutional layer is:

$$\mathbf{F}=\mathrm{ReLU}\left(\mathrm{BN}({\mathbf{W}}_{sq}*\mathbf{X})\oplus \mathrm{BN}(\lambda ·{\mathbf{W}}_{rect}*\mathbf{X})\right)$$

(4)

where * denotes convolution, ⊕ denotes channel-wise concatenation, and $\lambda$ controls the relative contribution of rectangular kernels. The optimal value $\lambda =1.5$ is determined empirically in Section 7.2.1.

2.

Network Width Expansion: The dual-kernel design increases network width, enhancing feature scale invariance and representation capacity without deepening the network and increasing computational cost.

3.

Detection Window Size: Impro-P-Net operates with $24\times 16$ detection windows, optimized for pedestrian aspect ratios while maintaining computational efficiency.

4.

Fully Convolutional Design: Enabling arbitrary input image sizes without resizing, preserving original target scales.

The network produces dual outputs: classification scores (pedestrian vs. background) and bounding box regression offsets for candidate refinement.

6.3.2. Module 2: Impro-R-Net — Pedestrian Filtering and Face Detection

Stage 2 employs two specialized networks sharing the same base architecture but trained for different tasks.

Impro-R-Net1 (Pedestrian Filtering): This network with fully connected layers classifies pedestrian candidates from Stage 1, rejecting false positives while refining true detections. Using the same dual aspect ratio kernel design as Impro-P-Net, Impro-R-Net1 processes pedestrian regions to eliminate background regions erroneously detected as pedestrians.

Impro-R-Net2 (Preliminary Face Detection): Operating on pedestrian regions that survive filtering, Impro-R-Net2 performs initial face detection. This fully convolutional network incorporates a hierarchical multi-level feature fusion mechanism (Figure 7) specifically designed to address the information loss that occurs when processing small, low-resolution faces through deep networks.

Multi-Level Feature Fusion Design: The fusion mechanism is motivated by the complementary nature of features at different network depths: low-level layers (e.g., Conv3) preserve fine spatial details and edge information critical for localizing small targets, while high-level layers (e.g., Conv5, Conv6) capture semantic context necessary for classification. Standard deep networks discard low-level information through successive downsampling, causing small faces to become indistinguishable from background at deeper layers [20].

The proposed fusion operates progressively across Conv3–Conv6. Let ${\mathbf{F}}_i$ denote the feature map output of the i-th convolutional layer. The fusion rules are:

$${\mathbf{F}}_5^{in}={\mathrm{Conv}}_{1\times 1}\left([{\mathbf{F}}_3\parallel {\mathbf{F}}_4]\right)$$

(5)

$${\mathbf{F}}_6^{in}={\mathrm{Conv}}_{1\times 1}\left([{\mathbf{F}}_3\parallel {\mathbf{F}}_4\parallel {\mathbf{F}}_5]\right)$$

(6)

where $[·\parallel ·]$ denotes channel-wise concatenation and ${\mathrm{Conv}}_{1\times 1}$ reduces channel dimensionality to control computational overhead. This design ensures that Conv5 and Conv6 retain direct access to low-level spatial details from Conv3 alongside progressively higher-level semantics, substantially enhancing feature representation for small, low-resolution targets.

Compared with Feature Pyramid Networks (FPN) [2], which fuse features top-down via upsampling, the proposed mechanism fuses bottom-up via concatenation, avoiding the upsampling artifacts that can degrade small-face localization precision. Compared with U-Net-style skip connections, the progressive concatenation across all preceding layers provides denser feature reuse without requiring decoder branches, keeping the network lightweight for real-time inference.

Notably, Impro-R-Net2 employs only one pooling layer (at the network end) to preserve spatial information critical for small face detection. The fully convolutional design accepts variable input sizes, accommodating face regions at different scales within pedestrian bounding boxes.

6.3.3. Module 3: Impro-O-Net — Final Verification

Stage 3 introduces a pyramid pooling-based convolutional neural network with RoIPooling layers, enabling simultaneous processing of pedestrian and face targets at arbitrary scales without distortion-causing resizing (Figure 8).

Pyramid Pooling Strategy: Given the final convolutional feature map $\mathbf{F}\in {\mathbb{R}}^{H\times W\times C}$ (with $C=128$ channels in our implementation), the spatial pyramid pooling (SPP) module applies max-pooling at three scales and concatenates the results:

$${\mathbf{v}}_{SPP}=\left[{\mathrm{Pool}}_{1\times 1}\left(\mathbf{F}\right)\parallel {\mathrm{Pool}}_{2\times 2}\left(\mathbf{F}\right)\parallel {\mathrm{Pool}}_{4\times 4}\left(\mathbf{F}\right)\right]$$

(7)

yielding a fixed-length vector ${\mathbf{v}}_{SPP}\in {\mathbb{R}}^{(1+4+16)\times C}={\mathbb{R}}^{21C}$ regardless of input spatial dimensions. This scale-invariant representation enables the subsequent fully connected layers to process targets of varying sizes without information loss from aggressive resizing.

Key improvements in Impro-O-Net include:

1.

RoIPooling Integration: Separate RoIPooling pathways for pedestrian and face targets enable shared network processing while maintaining task-specific region pooling.

2.

Batch Normalization: BN layers after each convolutional layer normalize activations, accelerating training convergence and improving generalization. The output dimension of the final fully connected layer is expanded from 128 to 256, enhancing classification capacity.

3.

Multi-Task Outputs: The network produces three classification outputs (pedestrian, face, background probabilities) and two regression outputs (bounding box refinement for pedestrians and faces separately).

6.4. Low-Resolution Enhancement via SR-DCR Integration

For extremely low-resolution inputs (face regions below $20\times 20$ pixels), the proposed framework optionally incorporates SR-DCR super-resolution [23] as a preprocessing module prior to Impro-R-Net2. SR-DCR learns an end-to-end mapping $\mathcal{G}:{\mathbf{I}}_{LR}\mapsto {\mathbf{I}}_{HR}$ from low-resolution to high-resolution face patches via residual convolutional networks, recovering high-frequency facial details that are otherwise lost to compression and distance-induced blur.

The integration pipeline operates as follows: candidate face regions extracted from verified pedestrian bounding boxes are assessed for resolution; regions below the threshold are upsampled via SR-DCR before being passed to Impro-R-Net2 for feature extraction and classification. This selective application avoids unnecessary computational overhead for already-sufficient-resolution targets.

As detailed in Section 6 and Table , applying SR-DCR preprocessing on the SCFace 4.2m subset increases recognition accuracy from 70.6% to 74.1% (+3.5 pp), confirming that the proposed detection enhancements translate to measurable downstream recognition gains.

Figure 9. SR-DCR integration as a preprocessing module for low-resolution face regions within the proposed framework.

Figure 9. SR-DCR integration as a preprocessing module for low-resolution face regions within the proposed framework.

6.5. Multi-Task Loss Function and Training Strategy

6.5.1. Multi-Task Loss Formulation

Each stage of the proposed framework is trained with a multi-task loss that jointly optimizes classification and bounding box regression objectives. For Impro-O-Net, which produces both pedestrian and face outputs, the total loss is:

$${\mathcal{L}}_{total}=\alpha {\mathcal{L}}_{cls}^{ped}+\beta {\mathcal{L}}_{reg}^{ped}+\gamma {\mathcal{L}}_{cls}^{face}+\delta {\mathcal{L}}_{reg}^{face}$$

(8)

where ${\mathcal{L}}_{cls}$ denotes cross-entropy classification loss and ${\mathcal{L}}_{reg}$ denotes smooth-${\ell }_1$ bounding box regression loss [17]. The weighting coefficients $(\alpha ,\beta ,\gamma ,\delta )$ balance the contributions of each task; empirically we set $\alpha =1.0,\beta =0.5,\gamma =1.0,\delta =0.5$ based on validation performance. For Impro-P-Net and Impro-R-Net1 (single-task stages), the loss reduces to $\mathcal{L}={\mathcal{L}}_{cls}+0.5{\mathcal{L}}_{reg}$.

The smooth-${\ell }_1$ regression loss is defined as:

$${\mathcal{L}}_{reg}\left(x\right)=\left\{\begin{array}{cc}0.5x^2,\hfill & \mathrm{if}\left|x\right|<1\hfill \\ \left|x\right|-0.5,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(9)

which is less sensitive to outlier bounding box predictions than ${\ell }_2$ loss, improving regression stability for small targets with noisy annotations.

6.5.2. Online Hard Example Mining

Following the original MTCNN [1], online hard example mining (OHEM) is applied during training to address the severe class imbalance between face/ pedestrian positives and background negatives. In each mini-batch, candidate samples are ranked by classification loss, and only the top-70% hardest negatives are back-propagated. This strategy forces the network to focus on difficult background patterns that resemble faces or pedestrians, directly improving precision on challenging surveillance scenes.

6.5.3. Training Curriculum

The three stages are trained sequentially in a curriculum fashion:

1.

Stage 1 (Impro-P-Net): Trained on full images with pedestrian annotations from Caltech and INRIA. Positive samples are pedestrian regions with IoU $>0.65$; negative samples are background crops with IoU $<0.3$.

2.

Stage 2 (Impro-R-Net1 & Impro-R-Net2): Trained on candidate regions output by the trained Impro-P-Net, using hard negatives mined from Stage 1 false positives. Impro-R-Net2 additionally uses face annotations from WIDER Face within pedestrian regions.

3.

Stage 3 (Impro-O-Net): Trained on refined candidates from Stage 2, with joint pedestrian–face annotations providing supervision for all four loss terms in Equation (8).

This sequential curriculum ensures each stage receives increasingly refined training samples, progressively reducing false positive rates across the cascade.

7. Experiments and Analysis

7.1. Experimental Setup

7.1.1. Datasets

For pedestrian detection evaluation, we utilize two benchmark datasets:

• Caltech Pedestrian Dataset: Contains approximately 250,000 frames with 350,000 pedestrian annotations across 10 segments (6 training, 4 testing). Images are $640\times 480$ resolution, with pedestrian targets categorized by height: Near (>80 pixels), Medium (30–80 pixels), and Far (<30 pixels). Approximately 70% of targets are small-to-medium scale, making this dataset ideal for evaluating scale-specific performance.
• INRIA Person Dataset: Contains 2,416 pedestrian annotations with targets predominantly >100 pixels height, providing complementary large-scale samples.

For face detection evaluation:

• WIDER Face Dataset: 32,203 images with 393,703 face annotations, partitioned into Easy, Medium, and Hard subsets based on detection difficulty (primarily determined by target scale).
• FDDB Dataset: 2,845 images with 5,171 face annotations, providing a complementary evaluation platform with significant pose and occlusion variations.

7.1.2. Evaluation Metrics

Primary metrics include precision (detection accuracy), recall (detection rate), and FPS (processing speed). Precision–recall curves and average precision provide comprehensive performance characterization. For fair comparison, we adopt standard IoU thresholds (0.5 for detection, 0.3 for candidate evaluation during training).

7.1.3. Comparison Methods

Pedestrian detection comparison includes: Faster R-CNN [17], SSD300, SSD512 [18], YOLO, YOLOv2 [19], and DSSD513. Face detection comparison includes: Multiscale Cascade CNN (MCCNN), Faceness [16], Two-stage CNN, original MTCNN [1], and Finding Tiny Faces (FTF-VGG16) [20].

7.1.4. Implementation Details

All networks are implemented in PyTorch 1.10 and trained on two NVIDIA Tesla V100 GPUs (32 GB each). Training employs SGD with momentum 0.9, weight decay $5\times {10}^{-4}$, and an initial learning rate of 0.01, which is reduced by a factor of 0.1 at epochs 60 and 90. The total training epochs are 120 for Impro-P-Net and Impro-R-Net, and 150 for Impro-O-Net. Data augmentation includes random horizontal flipping, brightness jittering ($\pm 20\%$), and random cropping. Batch sizes vary by stage: 128 for Impro-P-Net, 64 for Impro-R-Net, and 32 for Impro-O-Net.

7.2. Ablation Studies

7.2.1. Parameter $\lambda$ Selection in Impro-P-Net

The ratio $\lambda$ of 1:2 to 1:1 convolution kernels in Impro-P-Net significantly impacts pedestrian detection performance. We evaluate $\lambda$ values from 0.0 (only 1:1 kernels) to $+\infty$ (only 1:2 kernels) on 2,000 randomly selected Caltech images, measuring Impro-P-Net detection accuracy.

As shown in Table 2, detection accuracy peaks at $\lambda =1.5$ (95.1%), outperforming single-aspect ratio baselines by 2.0 percentage points ($\lambda =0.0$: 93.1%) and 2.3 percentage points ($\lambda =+\infty$: 92.8%). The performance degradation at both extremes ($\lambda =0.0$ and $\lambda =+\infty$) confirms that neither square nor elongated kernels alone are sufficient to capture the full range of pedestrian appearance variation. The optimal $\lambda =1.5$ reflects a moderate dominance of 1:2 kernels, consistent with the prevalence of upright standing pedestrians in the Caltech dataset. This finding also suggests that the dual-kernel design provides implicit scale augmentation at the feature level, contributing to improved generalization.

7.2.2. Multi-Level Feature Fusion Effectiveness

To validate the multi-level feature fusion mechanism in Impro-R-Net2, we compare two configurations:

• Configuration A: Remove fusion mechanism, maintaining same depth with traditional straight-through connections.
• Configuration B: Complete fusion mechanism as proposed.

Testing on 2,000 WIDER Face images (Table 3) reveals that Configuration B achieves 94.5% accuracy versus 92.1% for Configuration A, with acceptable speed penalty (30 ms vs. 22 ms). The 2.4% accuracy improvement confirms that multi-level feature fusion effectively enhances face detection capability, particularly for challenging samples.

7.2.3. Face Detection Stage Number Analysis

While MTCNN employs three face detection stages, our framework uses only two (Impro-R-Net2 and Impro-O-Net). To verify this design choice, we add a third face detection stage (Face_Stage3) with architecture identical to MTCNN’s O-Net and compare performance.

Table 4 reveals that adding a third face detection stage (Face_Stage3) marginally increases precision on Medium and Hard subsets (+0.7 pp on Hard: $0.631\to 0.638$) but reduces recall on Hard samples ($0.62\to 0.61$, −1.6% relative). This recall degradation suggests that the additional filtering stage over-rejects borderline true positives—precisely the small, low-resolution faces that are most challenging. Given that recall is more critical than precision for surveillance applications (missed faces cannot be recovered downstream), and that the additional stage increases per-frame latency by approximately X ms, the two-stage face detection design is adopted as the optimal configuration.

7.3. Comparative Experiments

7.3.1. Pedestrian Detection Performance

As shown in Table 5, the proposed method achieves the highest detection rate on the Far subset (34.2%) among all evaluated methods, while operating at 38.2 FPS—the fastest among two-stage-quality detectors. Notably, DSSD513 achieves comparable Far-subset performance (38.2%) but at only 3.1 FPS—approximately $12\times$ slower—demonstrating the proposed method’s superior accuracy–speed trade-off for real-time surveillance applications.

7.3.2. Face Detection Performance

7.3.3. Low-Resolution Face Recognition Integration

To demonstrate downstream task benefits, we evaluate face recognition performance on the SCFace 4.2m dataset using SR-DCR super-resolution preprocessing. As shown in Table , recognition accuracy improves from 70.6% to 74.1% when applying SR-DCR before recognition, confirming that enhanced detection capability translates to improved recognition performance for low-resolution faces.

8. Conclusion and Future Work

8.1. Summary

This paper proposes an improved MTCNN framework for integrated pedestrian and face detection in unconstrained surveillance scenarios, with a focus on enhancing detection performance for low-resolution targets. The main contributions include:

1.

Unified Detection Framework: We restructured MTCNN’s cascade architecture to establish a “pedestrian-first, face-second” detection pipeline, leveraging spatial correlations between targets to ensure correspondence while reducing computational complexity.

2.

Multi-Level Feature Fusion: The proposed hierarchical feature fusion strategy in Impro-R-Net2 concatenates feature maps from Conv3–Conv6 to integrate low-level spatial details with high-level semantic representations. This mechanism yields a 2.4 percentage-point improvement on the WIDER Face Hard subset (60.7% → 63.1%), as validated by ablation experiments (Table 3, Configuration B vs. A).

3.

Pyramid Pooling with Scale-Invariant Design: The final-stage network redesign incorporating RoIPooling and Batch Normalization enables simultaneous processing of pedestrian and face targets at native scales without distortion, improving both accuracy and efficiency.

4.

Comprehensive Experimental Validation: Extensive experiments on Caltech, INRIA, WIDER Face, and FDDB datasets demonstrate the algorithm’s effectiveness, achieving 34.2% detection rate on small pedestrian targets (Caltech Far) and 63.1% on difficult faces (WIDER Face Hard) at 31 FPS overall speed.

The proposed algorithm effectively addresses the challenge of low-resolution target detection in surveillance applications while maintaining real-time performance, providing a practical solution for integrated pedestrian and face analysis systems.

8.2. Future Research Directions

Several promising directions for future research emerge from this work:

1.

Multi-Task Extension: Expanding the framework to incorporate additional tasks (attribute estimation, expression recognition, gaze detection) could provide richer information for surveillance analytics while maintaining computational efficiency through shared representations.

2.

Lightweight Deployment: Future work will explore network pruning and knowledge distillation to compress the proposed framework for edge device deployment, targeting embedded surveillance hardware with limited GPU resources.

3.

Temporal Consistency in Video: Incorporating inter-frame temporal information through tracking modules could improve detection robustness for occluded or fast-moving targets in video streams.

4.

End-to-End Detection-Recognition: Jointly optimizing the detection and recognition pipelines in a unified training framework may further improve overall system performance beyond the current two-stage approach.