High-Precision Depth Estimation Networks Using Low-Resolution Depth and RGB Image Sensors for Low Cost MR Glasses

1. Introduction

Depth estimation is important in computer vision for robot control [1], multi-view generation for 3D exhibition [2,3,4], autonomous driving [5] and hand-gesture recognition in mixed reality (MR) [6] and so forth. In general, the depth estimations can be roughly catalogued into image-based and sensor-based approaches. Image-based approaches including stereo matching [7] and monocular depth estimation [9,10,11,12,13,14,15,16] which purely exploit the RGB image resource to achieve the depth estimation task.

The monocular depth estimation networks can be cataloged into supervised [8], unsupervised [13] and semi-supervised [15,16] approaches. The training resource of supervised monocular depth estimation networks could primarily come from the infrared, LiDAR, radar, and ultrasonic sensors [17]. Thus, the primary depths are manually corrected according to the accurate depth distances to obtain the ground truth (GT) depth maps. However, in most cases, labeling and annotations of GT depths cannot be easily available. Thus, the unsupervised networks adopt paired stereo color images captured by dual cameras to indirectly infer the GT depth map for each other. The convenience of developing the unsupervised network is that the GT depth maps of TOF camera is not needed at all. The depth map predicted from a color view is applied to render the virtual RGB image of another view using depth-image-based rendering (DIBR) [3]. Thus, the prediction of depth loss can be transferred to the RGB domain that the training loss is only associated with the varied RGB image differences. The semi-supervised networks are conceptually similar to the unsupervised ones that they can be divided into two classes. The first class utilizes dual-view RGB images combined with semantic segmentation information and 3D-reconstructed surface normal information to achieve multi-functional networks [15]. The embeddings of normal, segmentation and depth can be informatively harmonized to enhance the accuracy of depth estimation. The second class integrates the dual-view RGB images and low-resolution depth maps [15] that their loss functions are similar to that of unsupervised approaches where the calibrations are partially required in supervised processing.

The users would not like to adopt the parallelized RGB cameras for obtaining the parallax in depth estimation, when the persistence of on-line calibration is troublesome for the use of moving stereo cameras with inevitable vibrations. Thus, the monocular depth estimation network appears common in use for cost-effective depth estimation. Further, the monocular depth estimation with the aid of LiDAR sensor [17] or time-of-flight (ToF) sensor [18] is preferred for predicting the real-world distances. The ToF sensors of LED light source are cheap but supply the low accuracies and the shorter range of sensed depths relative to the LiDAR sensors. The MR glasses in mobile features are often with computation-constrained resource, making the complex state-of-the-art depth methods be hardly designed in it. Hence, the low power-consumption ToF depth sensor is the first choice for the installation in smart MR glasses. Although, the low-cost ToF depth sensor can only offer spare depths, in principle, the high effective compensation of spare depth map and RGB image shall capably figure out the high-precision depth map with partial practical distances. To obtain precise high dimension depths for the MR glasses, in this paper, we presented lightweight depth estimation networks that the main contributions are threefold.

• The proposed structure of our network targeted at the utilization of low-cost MR glasses can attain the optimal fusion of the sparse depth map and RGB image to achieve the accurate high-precision dense depth map.
• A novel adaptive dynamic range bins estimator is proposed for promptly estimating the depth distribution of the captured scene, making the finally resulted depth maps be considerably appropriate for specified employments.
• The proposed networks with two decoding variants have been successfully implemented on the Jorjin MR glasses [19] for the hand-gesture MR and the augmented reality (AR) applications.

2. Related Work

Related to this study, the technologies of monocular depth estimation, adaptive bins estimation and depth completion should be briefly reviewed in the following three subsections.

2.1. Depth Estimation

Monocular depth estimation tackles the problem of estimating scene depth via a single RGB image. Estimating depth/distance from a single RGB image is an ill-pose problem physically in the viewpoint of computer vision. However, numerous deep learning approaches have been developed to address this challenge in terms of the maximum probability. Most of these methods are based on the autoencoder architecture. The encoder extracts the embedding features from RGB images and then the decoder with learnable upsampling layers to progressively recover the depth predictions. Depth estimation tasks can be divided into two major categories: unsupervised and supervised approaches. Supervised learning approaches [8,9,10,11] use ground truth depth to perform per-pixel regression of dense depth values and unsupervised methods [13,14] do not need the GT depth data for the training process. In general, the pure supervised learning networks need the precise high-dimensional GT depth maps for deepening the learning and the pure unsupervised learning demands the sufficient RGB stereo image datasets. Thus, they appear unsuitable for estimating the high-precision real-world depths under the constraints of only using the low-power equipment and anticipating the dense practical perspective distances. Hence, we herein design the lightweight depth predictive networks with variant decoders under the conditions that a color camera and a ToF depth-sparse sensor are only available.

2.2. Depth Completion

With sparse samples obtained from LiDAR or ToF sensors, depth completion [17,18] can reconstruct the dense depth by utilizing high resolution RGB images and sparse depth measures. Ma et al. develop a CNN-based depth network using RGB-D raw data to achieve the accurate and reliable LiDAR super-resolution [20]. This work can identify that, by the aid of only few LiDAR depths, the high-dimensional depth map estimation can be improved. The autoencoder-based network can likely recover the dense depth map from the sparse depth samples by direct regression routine [21]. However, the generated depth maps of these approaches often have blurry depth boundaries. To solve this problem, Cheng et al. [22] proposed an architecture termed convolutional spatial propagation network (CSPN) to effectively mitigate this blurring phenomenon in depth completion task. This architectural concept is inspired by the spatial propagation network (SPN) [23] that SPN can learn the affinities for local neighborhood of pixels and propagate the information of affinities iteratively over spatial dimensions. As a result, the blurry depth maps and the coarse-profile segmentation masks can be sharpened and refined.

2.3. Adaptive Bins Estimation

Early CNN-based depth estimation approaches can be viewed as a per-pixel regression task. Some approaches employ the autoencoder to extract features for making the extensive per-pixel regression to construct the depth maps of rational naturalness. Compared to CNN-based depth methods based on one-step regression without the alignment of depth anchors, AdaBins can provide the individual alignments of estimated depth centers for different real-word perspective ranges. The common sense of AdaBins is to split depth range into several subintervals that the division of bins is estimated per-image adaptively. Thus, the probabilities of per-pixel depth falling into these dynamic bins can be measured. Finally, the depth of each pixel can be predicted by linear combination of each bin center and their corresponding probabilities. Bhat et al. [12] show that monocular depth estimation can be treated as a pixel-based classification task by utilizing an AdaBins module. The AdaBins modules can be useful appropriated tools to offer the high-precision depth estimation for remote sensing and aerial images [24,25]. There are several AdaBins structures suggested to further promote the depth estimation performances with semantic, mask transformer, multi-scale and knowledge distillation techniques [25,26,27,28] to alleviate the computation. Hence, the AdaBins module can support the high practicality of monocular depth estimation network in real-distance detection tasks.

3. Proposed Method

In the proposed depth estimation network, the inputs captured by MR glasses are a high-resolution (HR) RGB image and a low-resolution (LR) depth map. The HR images and their corresponding LR depth maps are fed into the feature extractors to extract their high and low level features separately. The extracted features then pass through a share multilevel share decoder to fuse all the level features of image and depth together to reconstruct the dense depth features gradually, as Figure 1 shown. The features decoded from low level to high level are fed into an adaptive bins estimator which performs global analysis of depth features. The adaptive bins estimator predicts the bins divisions, which can be treated as the predictive depth distributions of current scenes. Finally, the output dense depth map is calculated by the linear combination of bins divisions and the corresponding probabilities.

3.1. Lightweight Encoder

The proposed encoder is designed as a lightweight dual-path feature extractor where one path is called the RGB extraction branch and another is called the depth extraction branch, as shown in Figure 2. The inputs of proposed encoder are the RGB image with size 224×224×3 and the LR depth map with the size of p×p×1 that p = 8 needs to be set for the Jorjin J7EF Plus AR glasses [19]. The HR RGB image and LR depth map are fed into color extraction branch and depth extraction branch, respectively, that the LR depth maps are bi-linearly interpolated to the same spatial dimension as the RGB images before feed into depth extraction branch.

To let the proposed model be suited for low-cost MR glasses, the primary consideration of designing modules should be high efficiency in computation. Inspired by FastDepth [10], we adopt MobileNet [30] with depthwise separable convolution layers as our feature extractors since they provide extreme computational efficiency. Then, we reduce the convolution layers on the original MobileNet layers to further achieve higher model throughput. The reduced encoder contains 6 depthwise separable convolution layers which can be regarded as a simplified MobileNet module termed MobileNet-s. The multilevel outputs of MobileNet-s are fed forward to the multilevel decoder through the multilevel skip connections to generate the fused multilevel decoding features.

3.2. Multilevel Shared-Decoder

The proposed multilevel shared-decoder can successfully fuse the RGB and depth features together, shown as Figure 3, where the multilevel shared-decoder, which consists of five cascading upsample blocks, i.e., UP₀, UP₁, UP₂, UP₃, and UP₄. For the nth upsample block (UP_n), two nth level features, r_n and d_n, coming from RGB and depth branches, respectively, are applied layer by layer to progressively reconstruct multilevel decoding features, i.e., D₀, D₁, D₂, D₃, and D₄, which will be further processed by the adaptive bins estimator for depth estimation.

In the study, two types of upsampling blocks are designed to cover different application scenarios, which are SimpleUp for high-efficiency and UpCSPN-k for high-accuracy applications.

3.2.1. Upsampling by SimpleUp

For the applications of mobiles, the design of upsampling block must be very lightweight to fit the restrictions of target hardware. Therefore, we suggest a simple high-efficient module for the upsampling block called SimpleUp. As shown in Figure 4, the encoding results r_n and d_n are blended with the last-level decoding result D_n-1 together in the nth upsampling block (UP_n) by element-wise addition. We adopt element-wise addition rather than concatenation, because the concatenation will increase the channels of features and consume the additional resource for the subsequent convolution operation. The SimpleUp consists of a 5×5 convolution kernel to reduce the channel number by half followed by a bilinear interpolation to double the spatial resolution at each layer. In the convolution layers, adopting the depthwise separable convolution over the standard convolutions can further reduce the parameters of the decoding layers, making SimpleUp more efficient without performance degradation.

3.2.2. Upsampling by UpCSPN-k

With relative adequate resource, the complexity-moderate upsampling module called UpCSPN-k can be selected for the high-accuracy applications. The UpCSPN-k utilizes CSPNs over the layers of decoder to iteratively produce the refined depths. As shown in Figure 5, the input RGB feature r_n and depth feature d_n with the size of (H, W, C) in the nth layer are passed through a 5×5 convolution then divided into two branches, convolution branch and CSPN branch.

In Figure 5, the convolution branch performs a 3×3 standard convolution and outputs the features with the size of (H, W, (C/2)-1). In the meanwhile, the CSPN branch is composed of two prediction heads, which produce a coarse depth map $d_n^c$with the size of (H, W, 1) and a predicted affinities map $\widehat{\kappa }(x)$with the size of (H, W, k²-1) where k is the kernel size of CSPN. For each pixel x in $\widehat{\kappa }(x)$, the convolution weights of channels can help to convey the information of pixels through the spatial average. Thus, the refined depth $d_n^r$ through the CSPN processing in each decoding layer can be obtained as

$$d_x^r={\kappa }_x\left(x\right)\odot d_x^c+{\displaystyle \sum _{x_n\in {\mathcal{N}}_x}{\kappa }_x\left(x_n\right)}\odot d_{x_n}^c,$$

(1)

where x denotes the pixel indices, x_n denotes the local neighborhood N_x around x and “$\odot$” is the element-wise production. The refined affinity of x is given as

$${\kappa }_x\left(x\right)=1-{\displaystyle \sum _{x_n\in {\mathcal{N}}_k}{\kappa }_x\left(x_n\right),}$$

(2)

where the normalization of predicted affinities map $\widehat{\kappa }(x)$ is performed by

$${\kappa }_x\left(x_n\right)={\widehat{\kappa }}_x\left(x_n\right)/{\displaystyle \sum _{x_n\in {\mathcal{N}}_x}\left|{\widehat{\kappa }}_x\left(x_n\right)\right|}.$$

(3)

After the CSPN processing, the refined depth is concatenated with the features of convolution branch, and then the spatial resolution of concatenated results is doubled by a bilinear interpolator for each decoding layer. Although the UpCSPN-k consumes more computation resource, the upsampling by UpCSPN-k can achieve the more accurate interpolation with the lower error rate than that by SimpleUp.

3.3. Adaptive Bins Estimator

After the multilevel shared decoder, the multilevel decoding features, D₀, D₁, D₂, D₃, and D₄ are sent into the proposed adaptive dynamic range bins (AdaDRBins) estimator. The AdaDRBins estimator employ these multilevel decoding features and the original LR depth map, d_pxp to produce a set of predicted bins width {b₀, b₁, b₂, b₃} and the predicted depth map , as shown in Figure 6.

The AdaDRBins estimator consists of four types of multiple layer perceptron (MLP): features embedding MLP (E_n-symbolized block), bins initialization MLP (INIT-symbolized block), bins splitter MLP (S_n-symbolized block) and bias prediction MLP (B_n-symbolized block). Those MLPs exploit the individual fully-connected (FC) structures to project the data into the different specified dimensions. The combination of those MLP can make the gaps of bins be more and more fine-grained in the bins estimator

3.3.1. Feature Embedding MLPs (E_n)

The decoded features are first passed through Feature embedding MLPs layers. The embedding MLPs project the nth decoding features D_n with the size of (H_n, W_n, C_n) to become the nth embedding features, e_n, with the size of (H_n, W_n, C_e). The channel dimension C_e of embedding features now become the same size, which is referred as “embedding space”. Tab. 1 shows the detailed architecture of feature embedding MLPs that the proper channel dimension is assigned by C_e =32 in our experiments. The subsequent layers within the AdaDRBins module utilize these embedding features to determine the appropriate bin division for each video frame.

Table 1. Details of feature embedding MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	Cn	64	GeLU
FC	64	Ce=32	-

Table 1. Details of feature embedding MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	Cn	64	GeLU
FC	64	Ce=32	-

3.3.2. Bins Initialization MLPs (INIT):

The input of INIT is the first embedding feature denoted by e₀ which is obtained from bottleneck of the decoder. The INIT generates an initial bin division, which is treated as coarse bin division, to serve as the starting point of bin division refinement. In the subsequent bins estimating layers, each bin will be consecutively split into two new bins layer by layer. To fit the range of training datasets, the predicted bins division is re-scaled by

$$b_0=d_{\min }^0+\left(d_{\max }^0-d_{\min }^0\right)\cdot \mathrm{norm}\left(b_0\right),$$

(4)

where $d_{\min }^0$and $d_{\max }^0$denote the predicted minimal depth and maximal depth of bias prediction MLP B₀, respectively. Tab. 2 shows the detailed architecture of INIT MLPs.

Table 2. Details of INIT MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	64	GeLU
FC	64	b	RLU
Global AvgPool	-	--	-

Table 2. Details of INIT MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	64	GeLU
FC	64	b	RLU
Global AvgPool	-	--	-

3.3.3. Bins Splitter MLPs (S_n)

After bins initialization, the initial bin division is sent to the bins splitter MLPs to perform the coarse-to-fine bins generation strategy. The bins splitter MLPs play an important role in determining the distribution of bins, based on their respective bin embeddings. In essence, they acts as decision-makers to determine how the previous bin division should be divided for improving the precision and the resolution of bins. As shown in Figure 6, the interpolated embedding features of previous layer are element-wisely added with the embedding features of current layer and then fed into the next bin splitter MLP. After the FC layer in each bins splitter MLP, the features are passed through a Sigmoid activation to obtain a split score vector denoted by$\alpha =\left({\alpha }^0,{\alpha }^1,\dots ,{\alpha }^{2^{k-1}b}\right)$. The scores in this vector determine how each previous bin is divided into two bins. Let $b_{n-1}$ denote the previous bins division, $b_n$ denoting the bins division after the bin splitter MLP becomes

$$b_n=\left\{{\alpha }^0{b}_{n-1}^0,\left(1-{\alpha }^0\right)b_{n-1}^0,{\alpha }^1{b}_{n-1}^1,\left(1-{\alpha }^1\right)b_{n-1}^1,\cdots ,{\alpha }^{2^{n-1}b}{b}_{n-1}^{2^{n-1}b},\left(1-{\alpha }^{2^{n-1}b}\right)b_{n-1}^{2^{n-1}b}\right\}$$

(5)

This binning processing increases the granularity and resolution as the layers progress, ultimately improving the quality of the output. Similar to INIT, the output bin divisions are re-scaled to fit the depth range of the training dataset, given by

$$b_n=d_{\min }^n+\left(d_{\max }^n-d_{\min }^n\right)\cdot \mathrm{norm}\left(b_n\right),$$

(6)

where $d_{\min }^n$ and $d_{\max }^n$ are the predicted minimal depth and maximal depth of bias prediction MLP B_n. Table 3 details the architecture of the nth splitting MLPs.

Table 3. Details of the nth splitting MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	64	GeLU
FC	64	$2^{n-1}\cdot b$	-
Global AvgPool	-	--	-

Table 3. Details of the nth splitting MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	64	GeLU
FC	64	$2^{n-1}\cdot b$	-
Global AvgPool	-	--	-

3.3.4. Bias Prediction MLPs (B_n)

For low-resolution depth inputs, we only have a rough understanding of the depth distribution in the scene. Our general idea is to constrain the predicted depth range, making the bins prediction closer to the ground truth depth range. Since the minimum and maximum depths in low-resolution depth map may not be identical to the true minimum and maximum depths in ground truth, a bias term is needed for compensating the predictive offset. The bias MLP consists of two inputs: the low-resolution depth map d_pxp and the embedding feature e_n. The outputs are the estimated minimum depth and maximum depth of the scene. The embedding feature is passed through a bias prediction MLP to estimate the minimum and maximum depths. The estimated biases are respectively added to the minimum value and maximum value of the resolution depth map as

$$d_{\min }^n=\min (d_{p\times p})+bia{s}_{\min }^n\mathrm{and} d_{\max }^n=\min (d_{p\times p})+bia{s}_{\max }^n,$$

(7)

where $bia{s}_{\min }^k$and $bia{s}_{\max }^k$are the predicted biases generated by the bias prediction of B^k. The predicted minimum depth and maximum depth are then fed into bin initialization MLP and bin splitter MLP with the more accurate dynamic depth range of current scene. Tab.4 details the architecture of the nth bias MLPs.

Table 4. Details of the nth bias MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	32	GeLU
FC	64	2	-
Global AvgPool	-	--	-

Table 4. Details of the nth bias MLPs while keeping the same H_n and W_n.

Layer type	Input channels	Output channels	Activation
FC	32	32	GeLU
FC	64	2	-
Global AvgPool	-	--	-

After the prediction of these MLPs, the final bin division b₃ is obtained. In the final stage of AdaDRBins, a hybrid regression layer (REG) is used on the final decoded feature D₀ to estimate the probabilities of each pixel falling into these bins. The hybrid regression layer consists of a 3×3 convolution followed by a softmax activation unit which results in the predicted probability p_i for the i^h bin. Then, the predicted probabilities are applied to perform the linear combination with the predicted final bin-centers $c\left(b_3^i\right)$, which is given by

$$c\left(b_3^i\right)=d_{\min }^3+\left(d_{\max }^3-d_{\min }^3\right)\left(b_3^i/2+{\displaystyle \sum _{j=1}^{i-1}{b}_3^j}\right),$$

(8)

for achieving the final predicted depth map $\widehat{d}$. The predicted depth at (x, y) in $\widehat{d}$ can be obtained by

$$\widehat{d}(x,y)={\displaystyle \sum _{i=1}^{N}c}\left(b_3^i\right)p_i(x,y),$$

(9)

where p_i(x,y) is the probability of predicted depth at (x,y) falling in the i^h bin.

3.4. Loss Functions

Similar to AdaBins, we employ two loss terms to assemble the loss function which is designed as

$${\mathcal{L}}_{total}=\alpha {\mathcal{L}}_{reg}+\beta {\mathcal{L}}_{bins},$$

(10)

where ${\mathcal{L}}_{reg}$ and ${\mathcal{L}}_{bins}$ denote the per-pixel regression loss and bin-centers distribution loss, respectively. With extensive experiments, the scalars $\alpha =10$ and $\beta =0.5$ are selected to attain the stability of training our network.

3.4.1. Scale-Invariant Log Loss

We use masked scale-invariant log (SILog) loss proposed by Eigen et al. [5] as the per-pixel regression loss. The loss term is defined by:

$${\mathcal{L}}_{reg}=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _i{g}_i^2}-{\displaystyle \frac{\lambda }{N^2}}{\left({\displaystyle \sum _i{g}_i}\right)}^2},$$

(11)

with

$$g_i=\log d_i-\log {\widehat{d}}_i,$$

(12)

where $d_i$ and ${\widehat{d}}_i$ denote ground truth depth and predicted depth at pixel i, respectively, and N denotes number of valid pixels in ground truth depth (i.e., $d_i>0$). Similar to AdaBins, we set λ = 0.85 in our experiments.

3.4.2. Bin-Centers Distribution Loss

Similar to primary AdaBins, the chamfer distance [31] is adopted as the bin center distribution loss in order to supervise the distribution of predicted bin centers. Let ${\mathcal{C}}_{pred}=\left\{c_1,c_2,c_3,c_4\right\}$ denotes the set of predicted bin centers at different scale and $d_{gt}\in {ℝ}^{HW}$ denotes the set of valid ground truth values. The loss term is given by

$${\mathcal{L}}_{bins}={\displaystyle \sum _{b\in \mathcal{B}}CD}\left(c,d_{gt}\right),$$

(13)

where the chamfer distance is defined as

$$CD(X,Y)={\displaystyle \frac{1}{\left|X\right|}}{\displaystyle \sum _{x\in X}\underset{y\in Y}{\min }}‖x-y‖+{\displaystyle \frac{1}{\left|Y\right|}}{\displaystyle \sum _{y\in Y}\underset{x\in X}{\min }}‖x-y‖.$$

(14)

This loss term promotes the alignment between the distribution of bin centers and the distribution of ground truth depth values by pushing the bin centers to match the actual depth values in average.

3.5. LR Depth Generation for Training

In order to simulate the LR depth maps obtained from MR glasses for training, we introduce a depth sampling method that generates low-resolution depths randomly from the ground truth. Figure 7 shows the results of LR depth generation with p = 8 by our proposed simulation strategy, where the proposed two-step LR depth generation is stated as follows:

Step 1.

Subblocking: To obtain a p×p low-resolution depth map $d_{p\times p}$ from the ground truth depth map d, we first divide the ground truth depth map into p×p subblocks.

Step 2a.

Median pooling: We simply take the median value of each sub-block as the LP depth value to obtain the simulated LR depth map.

Step 2b.

Max-depth filtering (optional): In practical, the depth range on most MR glasses depth sensors is limited to a short distance range. To simulate the short distance range depth during training, we apply max-depth filtering on the sampled low-resolution depth map. As the depth value is greater than a max-depth threshold, max-depth filtering will reset the depth value to zero (i.e., invalid pixel)

4. Experimental Results

In this section, we will first describe the datasets and implementation settings. Then, we will exhibit the depth estimation performance achieved by the proposed network in various evaluation criteria.

4.1. Datasets and Implementation Settings

We use NYU Depth V2 Dataset [32] and EgoGesture Dataset [33] as our training datasets. The NYU Depth V2 provides the pairs of RGB images and the corresponding depth map for indoor scenes. The image and depth map resolution of the dataset are captured at 640×480 from Microsoft Kinect which renders the maximum sensible distance of 10 meters. We employ the official train/test split in our experiments, which comprises 12K training samples and 654 testing samples. The EgoGesture Dataset provides RGB images and the depth map of diverse hand gestures. The dataset is gathered by Intel RealSense SR300 with resolution 640×480. The dataset encompasses four indoor scenes and two outdoor scenes. In total, the dataset contains 2,953,224 hand gesture frames collected for the RGB and depth modality, respectively.

Our experiments are implemented with Python 3.9, PyTorch 1.10.2 [34], CUDA 11.3 and cuDNN 8.2 on Ubuntu 22.04 LTS operating system. We trained our network using single NVIDIA GeForce RTX 4090 GPU with 24 GB memory and Intel i7-13700K CPU. During training, we utilize the batch size of 16 and Adam [34] optimizer with (0.95, 0.99) betas and 0.1 weight-decay. The proposed depth estimation network is trained over 50 epochs with the learning rate starting from 0.000357 and decaying 20% every 5 epochs.

4.2. Proposed Network Implementation on Low-cost MR Glasses

The profile and components of Jorjin J7EF Plus AR glasses are described by Figure 8 which contains two major parts: an MR goggle with several sensors and a computation device connected through USB type C connector. There are several useful sensors built on this MR goggle, including time-of-flight (ToF), RGB camera, gyroscope, accelerator and magnetometer. The resolution of the ToF sensor is 8×8 and its sensing distance range is roughly within 3 meters, which is comparably low. Our goal is to utilize the LR depth map and RGB images from Jorjin’s MR glasses to get precise and high-resolution depth map by using EPSON BO-IC400 platform [35], which adopts Qualcomm SXR1130 processor with 4GB memory and 64GB storage.

For Jorjin MR-glasses sensor, the resolutions of captured raw RGB image and depth map are 640×480 and 8×8, respectively. Since the spatial positions of RGB and depth images must be fully aligned, we first resize the raw RGB image into 360×270 and crop the region RGB_crop = {(x, y): 68 ≤ x ≤ 191, 23 ≤ y ≤ 246} to make the cropped color images be 224×224 RGB images for further aligning with depth maps. We tried the spatial transformer network to align the RGB and depth images. However, this deep learning network not only consumes additional cost, but also causes some unpredictable inaccuracies in the alignment. Hence, we adopt a semi-automatic alignment by the aid of easy manual refinement that an alignment result for Jorjin MR-glasses data is displayed by Figure 9. Because the calculation of depth-estimation loss directly exploits the high-resolution GT depth maps provided by NYU Depth V2 dataset and EgoGesture dataset. Those high-resolution GT depth maps are downsampled as the sensor-labeled 8×8 depth maps for training our networks.

For verifying the feasibility of our model for resource-limited MR glasses, we implement the simplest model with the sole-SimpleUp decoder w/o AdaDRBins. The Unity with version: 2020.3.29f1 is used as our engine to build the virtual scene on MR glasses. Jorjin provided the software development kit (SDK) for MR glasses on Unity, we can access the MR sensor data with Unity scripting API in C# language. We use Barracuda Unity 3.0.0 as our model inference library. Barracuda Unity provides a lightweight and cross-platform neural network inference API for Unity. It can inference neural network in ONNX (Open Neural Network Exchange) format directly on both CPU and GPU of supported platforms.

For indoor scenes, the model is trained on the NYU Depth V2 dataset, we apply max-depth filtering of 3 meters on the depth training samples. For hand gesture scenes, the model is trained on the EgoGesture dataset, we apply max-depth filtering of 1.5 meters on the depth training samples. Figure 10 displays the estimated high-precision depths for indoor and dynamic hand scenes on Jorjin MR glasses by the simplest model with only SimpleUp decoder w/o AdaDRBins. There are three windows, 8×8 depths captured by the ToF sensor, the depth estimation result predicted by our proposed network, and RGB image captured by Jorjin MR glasses. For the result of indoor scenes, we can see the network roughly understand the depth information of the scene. However, the depth sensing range of the MR glasses is very low. Consequently, the quality of the predicted images is not good as what we demonstrated in the experiments. For the hand gesture scenes, since the distance range of hand gestures can fit the short depth sensing range of ToF sensor, the position and distance of hand making a gesture can be clearly identified. We believe that with the high-resolution depth information on MR glasses, many applications can be more accurate. To see the hand gesture demos in details, one can download Video S1: EgoGesture demo.mp4 at: www.mdpi.com/xxx/s1.

4.3. Performance Evaluation Results

For the experiments, we adopt MobileNet-s as the encoder. We use the official testing split, which contains 654 testing samples, to test the performances by using average evaluation metrics. The resolution of RGB images is set to 224×224 and the resolution of simulated LR depth is set to 8×8.

We compare the proposed models to the FastDepth [10] with ResNet50+NNConv5, FastDepth [10] with MobileNet+NNConv5, and the MonoDepth [13] with ResNet50+ DispNet for 224×224 RGB input images. As Table 5 shown, the computational burdens of two compared latters are about 4.8 G MACs and 0.74 G MACs, respectively. In our proposed network, the Dual MobileNet encoder including 6 depthwise separable convolution layers and one pointwise convolution totally requires about 0.185G MACs. In Table 5 , three proposed models using the solo-SimpleUp decoder, the SimpleUp+AdaDRBins decoder and the UpCSPN-7+AdaDRBins decoder are chosen to display their computational complexities in order. These three models demand 0.675 (i.e., 0.185+0.49), 1.335 (i.e.,0.185+1.15) and 9.445 (i.e.,0.185+9.26) G MACs, respectively. Not only the proposed simplest model, which adopts the solo-SimpleUp decoder, is lighter than the fastest version of FastDepth in MACs, but also achieves the lower 0.233 RMSE than its 0.599 RMSE obtained for testing the NYU Depth V2 dataset. It is noted that the MonoDepth tested on the KITTI dataset attains 4.392 RMSE in monotonous outdoor scenes. Thus, for the more complex indoor dataset, e.g., the NYU Depth V2 dataset, the RMSE of MonoDepth shall be larger than 4.392 such that we enlisted the RMSE of MonoDepth as “>4.392” in Table 5.

Table 5. Comparisons of FastDepth, MonoDepth and the various proposed variants by using the NYU Depth V2 dataset.

Network	Encoder	Decoder	Weights (M)	MAC(G)	RMSE
FastDepth [10]	ResNet50	NNConv5	25.60	4.190	0.568
FastDepth [10]	MobileNet	NNConv5	3.19	0.740	0.599
MonoDepth [13]	ResNet50	DispNet	30.00	4.800	>4.392.
Proposed	DMobileNet_s	SimpleUp	2.18	0.675	0.223
Proposed	DMobileNet_s	SimpleUp+ AdaDRBins	2.28	1.150	0.199
Proposed	DMobileNet_s	UpCSPN-7+ AdaDRBins	43.57	9.445	0.185

Table 5. Comparisons of FastDepth, MonoDepth and the various proposed variants by using the NYU Depth V2 dataset.

Network	Encoder	Decoder	Weights (M)	MAC(G)	RMSE
FastDepth [10]	ResNet50	NNConv5	25.60	4.190	0.568
FastDepth [10]	MobileNet	NNConv5	3.19	0.740	0.599
MonoDepth [13]	ResNet50	DispNet	30.00	4.800	>4.392.
Proposed	DMobileNet_s	SimpleUp	2.18	0.675	0.223
Proposed	DMobileNet_s	SimpleUp+ AdaDRBins	2.28	1.150	0.199
Proposed	DMobileNet_s	UpCSPN-7+ AdaDRBins	43.57	9.445	0.185

Figure 11 shows the qualitative results of our proposed network on NYU Depth V2 Dataset. We observe that the depth maps achieved by the combination of UpCSPN-k mand AdaDRBins modules preserve more details and clearer boundaries than those obtained by SimpleUp. However, the depth maps resulted from the model with the solo-SimpleUp decoder are still clear and recognizable. For considering the balance of complexity and precision, the combination of SimpleUP and AdaDRBins could be a good solution for the resource-limited platforms. Particularly, observing the 2nd and 4th rows of Figure 11, the rectangle-highlighted depth areas resulted by the proposed network using UpCSPN-7+AdaDRBins decoder can be even better than the corresponding depth areas in GT maps in terms of visual rationality.

4.4. Ablation Study

For the ablation study, we exhibit the quantitative metrics of proposed networks with different amounts and complexities of ingredients in Table 6 and Table 7. The simulation results show that our simplest architecture of the decoder, i.e., Solo-SimpleUp, achieves the comparable performance on NYU Depth V2 with incredibly fast inference speed, makes it more suitable to be implemented on mobile devices.

Table 6. Quantitative results of the proposed networks achieved by different structures that REL expresses the relative error.

Decoder	Bins Estimator	RMSE (m)	REL	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$		${\mathit{\delta }}_3$
SimpleUp	--	0.223	0.046	97.36	99.43	99.85
UpCPSN-3	--	0.206	0.039	97.87	99.56	99.89
UpCPSN-5	--	0.197	0.036	97.87	99.55	99.89
UpCPSN-7	--	0.202	0.036	97.87	99.54	99.89
SimpleUP	AdaBins	0.197	0.037	98.04	99.64	99.91
SimpleUp	AdaDRBins	0.199	0.038	98.06	99.64	99.91
UpCSPN-3	AdaDRBins	0.190	0.036	98.21	99.67	99.92
UpCSPN-5	AdaDRBins	0.191	0.036	98.21	99.67	99.92
UpCSPN-7	AdaDRBins	0.185	0.034	98.24	99.67	99.92

Table 6. Quantitative results of the proposed networks achieved by different structures that REL expresses the relative error.

Decoder	Bins Estimator	RMSE (m)	REL	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$		${\mathit{\delta }}_3$
SimpleUp	--	0.223	0.046	97.36	99.43	99.85
UpCPSN-3	--	0.206	0.039	97.87	99.56	99.89
UpCPSN-5	--	0.197	0.036	97.87	99.55	99.89
UpCPSN-7	--	0.202	0.036	97.87	99.54	99.89
SimpleUP	AdaBins	0.197	0.037	98.04	99.64	99.91
SimpleUp	AdaDRBins	0.199	0.038	98.06	99.64	99.91
UpCSPN-3	AdaDRBins	0.190	0.036	98.21	99.67	99.92
UpCSPN-5	AdaDRBins	0.191	0.036	98.21	99.67	99.92
UpCSPN-7	AdaDRBins	0.185	0.034	98.24	99.67	99.92

Table 7. Computational analyses of the proposed networks with different structures where MAC expresses the number of multiply-accumulate operations.

Decoder	Bins Estimator	#Bins	Weights (M)	MAC(G)		FPS
SimpleUp	--	-	2.18	0.49	1170
UpCPSN-3	--	--	41.85	8.04	367
UpCPSN-5	--	--	42.13	8.36	356
UpCPSN-7	--	--	42.57	8.60	337
SimpleUp	AdaBins	256	4.52	4.43	425
SimpleUp	AdaDRBins	32	2.28	1.15	560
UpCSPN-3	AdaDRBins	32	42.85	8.70	340
UpCSPN-5	AdaDRBins	32	43.14	8.92	330
UpCSPN-7	AdaDRBins	32	43.57	9.26	310

Table 7. Computational analyses of the proposed networks with different structures where MAC expresses the number of multiply-accumulate operations.

Decoder	Bins Estimator	#Bins	Weights (M)	MAC(G)		FPS
SimpleUp	--	-	2.18	0.49	1170
UpCPSN-3	--	--	41.85	8.04	367
UpCPSN-5	--	--	42.13	8.36	356
UpCPSN-7	--	--	42.57	8.60	337
SimpleUp	AdaBins	256	4.52	4.43	425
SimpleUp	AdaDRBins	32	2.28	1.15	560
UpCSPN-3	AdaDRBins	32	42.85	8.70	340
UpCSPN-5	AdaDRBins	32	43.14	8.92	330
UpCSPN-7	AdaDRBins	32	43.57	9.26	310

Although the combinations of UpCSPN-k and AdaDRBins have lower inference speeds, their performances are much higher than the model with the solo-SimpleUp decoder. As Table 6 shown, the decoder with UpCSPN-7 and AdaDRBins can make the generated depth maps acquire a very high precision with RMSE 0.185, which is lower the RMSEs achieved by the existing AdaBin-based networks [12,21,22,23,24,25], i.e., with RMSEs higher than 0.19 for various task-specific testing images

5. Discussion

In this study, for low cost MR applications, the proposed depth estimation algorithms with lightweight encoders and simpleUp multilevel share decoder. The simplest network with only 2.18M parameters is successfully implemented in Jorjin J7EF Plus AR glasses. It is noted that the proposed network with lightweight encoders, UpCPSN-7 multilevel share decoder and AdaDRbins can achieve the highest precision performance in depth estimation, better than AdaBines-based networks.

We can divide the models of proposed network into 4 groups that their decoders are “Solo-SimpleUp”, “UpCPSN-k only”, “SimpleUP+AdaxBins” (AdaxBins can be AdaBins and AdaDRBins), “UpCSPN-k+AdaDRBins”. Observing Table 7 , we find that the RMSEs of models between different groups have the salient differences, and the RMSEs of models in the identical group have close RMSEs. Even that, in the second group, i.e., UpCPSN-k only, the “UpCPSN-5 only” decoder is better than the “UpCPSN-7 only” decoder. This fact can manifest an evidence for the deep learning investigation that the more stacked layers could not guarantee better performance. We can claim that the better structure design is more important than the use of larger amounts of kernels.

In this study, we follow the rule of gradually increasing the stacked ingredients and layers to develop our networks. The content of Tab.6 can imply that the model with the “SimpleUP+AdaBins” decoder could be a teacher network to distillate the model with the solo-SimpleUp decoder. Similarly, the model with the UpCSPN-l+AdaDRBins decoder can play a teacher role for the model the UpCSPN-m+AdaDRBins decoder for l > m. Hence, based on this design strategy, the teacher-and-student structure can be easily attained by arbitrarily reducing the layers of a complex model for distillation.

Because the depths captured by the ToF depth camera in MR-glasses are very sparse, the original depths near fingers are mostly lost during hand-moving actions. The high-resolution depth predicted by our proposed networks can greatly support the accurate estimation of hand gestures in complex backgrounds. The predicted depths of finger pixels could provide better foundation for recognition of subtle 3D hand gestures.

It is noted that the proposed network should further investigate the vulnerabilities posed by adversarial examples when the MR glasses involve security sensitive applications, such as medical surgery and military security checks, etc. In the future, the proposed networks could consider a robust technique involving camouflage through the use of adversarial examples [36].

6. Conclusions

In this paper, we introduced the lightweight dual-path autoencoder and the adaptive dynamic range bins estimator architecture for estimation of highly accurate depth maps for resource-limited MR applications, which are based on an RGB image and a low-resolution depth map. The proposed approaches with different effective decoding modules yield impressive depth estimation results, particularly for indoor scenes and hand gestures. According to simulation results, we firmly believe that the proposed models with various combinations are well-suited for the deployment on light-weight mobile devices, MR glasses and AR apparatuses. By providing precise and dense depth information, the proposed networks can help greatly enhancing the accuracy of MR applications.