DSER: Spectral Epipolar Representation for Efficient Light Field Depth Estimation

1. Introduction

Dense depth estimation is a fundamental problem in 3D vision. Light field imaging is especially attractive because it captures both spatial and angular radiance, enabling geometry-aware inference beyond monocular and stereo cues [1,2,3,4]. In practice, however, light field depth estimation remains challenging due to sparse angular sampling, photometric inconsistency, weak texture, aliasing, and depth discontinuities. Existing methods face a clear trade-off: deep models improve prediction quality but often require large annotated datasets and underexploit epipolar structure [3,5], while classical methods such as gradient-based estimation, plane sweeping, and EPI analysis are geometrically grounded but are, respectively, unstable in low-texture regions, computationally expensive, or prone to oversmoothing fine structures [6,7,8].

We propose Deep Spectral Epipolar Representation (DSER), a hybrid framework for dense light field depth estimation. The key idea is a spectral epipolar prior that models frequency-consistent structure in horizontal and vertical EPIs to regularize multi-view correspondence. DSER combines this prior with three complementary estimators: Least Squares Gradient (LSG) for fast local initialization, plane sweeping for global cost aggregation, and fine-to-coarse EPI refinement for structure-preserving recovery. A final occlusion-aware Directed Random Walk (DRW) propagates reliable disparity along edge-consistent paths and suppresses ambiguity near occlusion boundaries [9]. Experiments on the Heidelberg Light Field Benchmark and Stanford Lytro Archive show that DSER improves structural consistency and boundary fidelity while maintaining a favorable accuracy-runtime trade-off [10,11].

Contributions

• We introduce DSER, a light field depth estimation framework that injects spectral regularization into the epipolar domain for dense disparity reconstruction.
• We develop a hybrid inference pipeline that unifies LSG initialization, plane-sweeping aggregation, multiscale EPI refinement, and occlusion-aware directed random walk propagation.
• We show on benchmark and real-world light field datasets that DSER improves structural fidelity and achieves a strong balance between reconstruction accuracy and computational efficiency.

2. Related Work

Classical light field depth estimation methods recover disparity through multi-view correspondence, plane sweeping, and epipolar-plane analysis [12,13]. While geometrically interpretable, they are often computationally expensive and brittle under weak texture, noise, and occlusion [14,15]. In particular, single-cue formulations degrade when photometric consistency is violated, motivating hybrid methods that combine complementary geometric cues [6,7].

Learning-based methods have significantly advanced depth estimation using convolutional, recurrent, and attention-based architectures [3,5,16]. However, many are designed for monocular or stereo inputs rather than plenoptic data, and therefore underutilize angular redundancy and explicit epipolar geometry [17,18]. Their performance also typically depends on large labeled datasets and often degrades under real-world domain shift [19,20].

Recent works incorporate angular consistency, geometric priors, and EPI-based reasoning into light field depth pipelines [21,22,23,24]. These studies show that epipolar structure is a strong supervisory signal because it directly encodes scene geometry across views [8,25]. However, existing approaches still face a trade-off between accuracy, robustness, and computational efficiency [9,15]. In contrast, DSER introduces a spectral epipolar prior that regularizes disparity in the frequency domain and couples it with hybrid multistage refinement, yielding a stronger balance between structural fidelity and efficiency [26,27].

3. Method

We introduce DSER, a hybrid framework for dense light field depth estimation that combines spectral epipolar regularization, multi-view geometric matching, and confidence-guided multiscale refinement. Given a 4D light field $L(x,y,u,v)$, DSER estimates disparity by integrating complementary cues from spatial-angular gradients, cost-volume aggregation, and epipolar consistency. The framework consists of four components: local disparity initialization, global cost-volume estimation, spectral EPI refinement, and confidence-guided coarse-to-fine propagation.

3.1. Data and Preprocessing

We evaluate on the 4D Light Field Benchmark, where each sample contains a $9\times 9$ angular grid of $512\times 512$ views [10], and additionally test generalization on higher-resolution light field data with $17\times 17$ angular sampling and $960\times 1280$ spatial resolution [11]. Preprocessing includes intensity normalization, resizing, invalid-region inpainting, and mask-guided foreground-background separation.

3.2. Least Squares Gradient Initialization

We first obtain a fast local disparity estimate using spatial-angular gradients. Under disparity-induced parallax,

$$L(x,y,u,v)=L(x-d{\Delta }_x,y-d{\Delta }_y,u+{\Delta }_x,v+{\Delta }_y).$$

(1)

We minimize the local reconstruction error

$$\begin{array}{c}E={\int }_{\alpha }\sum _p[L(x,y,u,v)\hfill \\ -L(x-d{\Delta }_x,y-d{\Delta }_y,u+{\Delta }_x,v+{\Delta }_y){]}^2.\hfill \end{array}$$

(2)

which yields the closed-form estimate

$$d^{*}={\displaystyle \frac{{\sum }_p(L_x{L}_u+L_y{L}_v)}{{\sum }_p(L_x^2+L_y^2)}}.$$

(3)

This stage is efficient and provides subpixel initialization but is unstable in weak-texture and occluded regions [14,15].

3.3. Plane-Sweeping Cost Volume

To improve global consistency, we construct a variance-based cost-volume. For each disparity hypothesis d, the sheared light field is

$$L_d(x,y,u,v)=L(x+ud,y+vd,u,v),$$

(4)

and the matching cost is

$$C(x,y,d)={\displaystyle \frac{1}{\left|U\right|\left|V\right|}}\sum _{u,v}{\left[L_d(x,y,u,v)-\overline{L_d}(x,y)\right]}^2.$$

(5)

The disparity is selected by minimizing $C(x,y,d)$. Plane sweeping improves robustness in textured regions but is substantially more expensive than local estimation [9,10].

3.4. Spectral EPI Refinement

The key novelty of DSER is a spectral epipolar prior that regularizes correspondence estimation in the frequency domain. Horizontal and vertical EPIs encode disparity as oriented epipolar structures; DSER exploits their frequency-consistent patterns to suppress noisy matches, sharpen object boundaries, and recover missing structure in occluded regions [7,8,17,21].

3.5. Confidence-Guided Depth Propagation

We estimate an edge-aware confidence map from the central view:

$$C_e(x,y)=\sum _{(x^{\prime },y^{\prime })\in \mathcal{N}(x,y)}\parallel I(x,y)-I(x^{\prime },y^{\prime })\parallel .$$

(6)

For each disparity hypothesis, a color-density score is computed, as

$$\begin{array}{cc}\hfill S(x,y,d)& ={\displaystyle \frac{1}{\left|R\right|}}\sum _{r\in R(x,y,u,v,d)}K(r-\overline{r}),\hfill \\ \hfill d^{*}(x,y)& =arg\underset{d}{max}S(x,y,d).\hfill \end{array}$$

(7)

Disparities are retained only when the confidence

$$C_d(x,y)=C_e(x,y)·\parallel S_{max}-\overline{S}\parallel$$

(8)

exceeds a threshold, allowing reliable disparity to propagate while suppressing ambiguous estimates [9,18].

3.6. Multiscale Spectral Refinement

To refine low-confidence regions, DSER performs multiscale EPI optimization. We minimize

$$\begin{array}{cc}\hfill E\left(D\right)& =\sum _{x,y}{\rho }_d\left(I(x,y)-I_D(x,y;D)\right)\hfill \\ \hfill & +\lambda \sum _{(x,y),(x^{\prime },y^{\prime })\in \mathcal{N}}{\rho }_s\left(D(x,y)-D(x^{\prime },y^{\prime })\right).\hfill \end{array}$$

(9)

where the data term enforces photometric consistency and the smoothness term imposes anisotropic spatial regularization [23,25]. Spatial-angular evidence is aggregated into an adaptive cost volume

$$\begin{array}{cc}\hfill C(x,y,d)& =\sum _{i,j}{w}_{ij}(x,y,d)|I_i(x,y)\hfill \\ \hfill & -I_j(x+\Delta x\left(d\right),y+\Delta y\left(d\right))|.\hfill \end{array}$$

(10)

where $w_{ij}$ unreliable correspondences are down-weighted [19,24]. Disparity is estimated independently along horizontal and vertical angular axes, fused, and refined by bilateral and median filtering. A coarse-to-fine optimization then solves Eq. 9 at progressively lower-to-higher resolutions, improving global consistency while preserving local depth discontinuities [26,28].

4. Experiments

Datasets and metrics

We evaluate on Boxes, Dino, and Cotton from the Heidelberg Light Field Benchmark [10] and additionally test on the Stanford Lytro Archive [11]. Depth is recovered from disparity via $Z=fb/d$ and evaluated using PSNR and MSE:

$$\mathrm{PSNR}=10{log}_{10}\left({\displaystyle \frac{MA{X}_I^2}{MSE}}\right),\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(Z_i-{\widehat{Z}}_i)}^2.$$

(11)

Higher PSNR and lower MSE indicate better reconstruction quality [27,29].

5. Results and Analysis

5.1. Classical and Learning-Based Baselines

Gradient-based estimators such as LSG are the fastest ($\approx 12$–20 s) but yield the lowest PSNR (21–24 dB), reflecting their failure in textureless and occluded regions. Plane sweeping achieves the highest single-scene PSNR ($36.53$ dB on Boxes) through exhaustive hypothesis testing but incurs prohibitive runtime ($\approx 350$ s), making it impractical for efficient deployment. Learning-based CNN and attention baselines improve average PSNR to 31–32 dB, but remain slower than DSER and require large annotated training sets [24,25].

5.2. EPI-Based and Proposed Methods

Pure EPI methods provide a stronger balance between quality and efficiency but degrade on low-texture scenes such as Cotton. In contrast, DSER achieves the best average PSNR ($28.71$ dB) and the highest per-scene PSNR on Cotton ($26.86$ dB), exceeding Plane Sweeping by $1.52$ dB at only $\approx 20$ s runtime. This shows that spectral epipolar regularization, coupled with hybrid multiscale refinement, offers a more favorable accuracy-efficiency trade-off than any single-paradigm baseline. Table 1 summarizes results across fifteen methods and five paradigms.

5.3. Real-World Generalisation

Table 2, Table 3 and Table 4 report per-scene PSNR and runtime for all methods. Plane Sweeping achieves the highest PSNR on Boxes and Dino but is outperformed by EPI-FCR on Cotton ($26.86$ vs. $25.34$ dB), demonstrating the benefit of epipolar-domain refinement in occluded low-texture scenes [7,8]. LSG is the fastest ($\approx 19$ s) but least accurate; EPI2 attains near-peak PSNR at a fraction of Plane Sweeping’s cost, making it practical for real-time deployment [20,30].

Figure 1 compares depth reconstructions across all methods. Plane sweeping provides a strong baseline but shows quantization artifacts and boundary leakage in occluded or specular regions [6,7]. LSG captures coarse structure efficiently but degrades in textureless areas. EPI-based methods improve boundary precision and depth continuity, with EPI2 best recovering thin structures such as the Dino appendages and Boxes mesh through stronger angular and multiscale constraints [21,22].

Figure 2 shows that EPI2 generalizes well to real-world light field data, producing sharper structural boundaries and lower per-pixel error than all baselines. The remaining failure cases are concentrated in heavily occluded or texture-ambiguous regions, indicating a promising direction for future adaptive refinement [30,31].

5.4. Depth Sampling Analysis

Figure 3 directly compares EPI2 against ground truth and LSG residuals. LSG exhibits large errors in flat or texture-poor regions, especially in the cotton background and boxes’ side walls. In contrast, EPI2 errors are largely confined to sharp discontinuities, corroborating the PSNR gains in Table 2 [24].

Figure 4 shows clear scene-dependent behavior. EPI2 most strongly improves over Plane Sweeping on Cotton, a low-texture, heavily occluded scene where exhaustive matching over-smooths structure. The gap is smaller on the more textured Boxes and Dino scenes, indicating that epipolar regularization is most beneficial in low-evidence regions [7,8].

Figure 5 confirms diminishing returns with increasing depth-plane count: MSE drops quickly at small $N_d$ but plateaus beyond $N_d=11$, where additional planes provide only marginal gains at higher computational cost [26,27]. We therefore fix $N_d=11$ in all primary experiments.

Figure 6 highlights the large runtime gap between methods. Plane sweeping requires $\approx 350$ s on average due to exhaustive search, whereas EPI2 runs in $\approx 20$ s by restricting expensive matching to uncertain regions and replacing global search with epipolar filtering.

6. Ablation Study

To isolate the contribution of each pipeline component we conduct a systematic ablation on the three Heidelberg benchmark scenes (Boxes, Dino, Cotton). Starting from the bare LSG initializer (A1), we incrementally add: plane-sweeping cost-volume aggregation (A2), spectral EPI refinement (A3), the occlusion-aware Directed Random Walk (A4), and multiscale coarse-to-fine optimization (A5), which together constitute the full DSER / EPI2 model. We additionally study the effect of the number of depth planes $N_d$ and the spectral regularization strength ${\lambda }_s$ separately in Table 6 and Table 7.

Component contributions

Table 5 shows that every added component strictly improves average PSNR. The largest single gain comes from plane sweeping (A1→A2, $+2.94$ dB), which resolves the ill-conditioned low-texture failures of LSG (Proposition D.3). Spectral EPI refinement (A2→A3) contributes a further $+1.15$ dB at a cost of only $\approx 8$ s, consistent with Theorem C.2: aligning correspondences to the spectral support locus suppresses noisy matches without exhaustive re-search. The DRW propagation step (A3→A4) provides the second-largest improvement on Cotton ($+1.22$ dB), the scene with the heaviest occlusion, validating the edge-aligned propagation guarantee of Corollary G.5. Multiscale refinement (A4→A5) yields modest but consistent gains ($+0.17$ dB average), predominantly on fine boundary structures in Dino, in line with the error-contraction bound of Theorem H.1.

Depth-plane count $N_d$

Table 6 reports PSNR and runtime as $N_d$ increasing from 3 to 64. MSE drops quickly for $N_d\le 11$ and plateaus thereafter, consistent with the cubic-decay result of Proposition I.3. We therefore fix $N_d=11$ in all primary experiments as the knee of the cost-accuracy curve.

Spectral regularization weight ${\lambda }_s$

Table 7 sweeps ${\lambda }_s$ over two orders of magnitude. Very small values (${\lambda }_s={10}^{-4}$) leave the spectral prior inactive, recovering performance close to A2. Very large values (${\lambda }_s=1.0$) over-regularize, washing out fine structures in Dino ($-1.8$ dB relative to the optimum). The best trade-off is obtained at ${\lambda }_s=0.1$, which we adopt as the default.

6.1. Limitations

DSER improves depth fidelity and reduces dependence on exhaustive search, but several limitations remain. First, gains are scene-dependent: the method shows the largest improvement on Cotton but only modest gains over Plane Sweeping on Dino, indicating sensitivity to texture and scene geometry [7,8]. Second, spectral refinement introduces extra computation that may limit deployment in strict real-time or very large-scale settings. Third, evaluation with PSNR alone is incomplete; metrics such as SSIM, depth accuracy, and more detailed runtime profiling would provide a more holistic assessment [27,29].

7. Discussion

DSER addresses the accuracy-efficiency trade-off in light field depth estimation by combining complementary estimators with epipolar-domain refinement. LSG provides efficient local initialization, plane sweeping offers accurate but expensive global matching, and EPI-FCR improves structural consistency through angular regularization and multiscale refinement [18,21,22]. Experiments show that EPI-based refinement approaches Plane Sweeping accuracy at substantially lower cost, especially in scenes with occlusion and texture variation [23,24]. More broadly, DSER suggests that epipolar-domain priors are an effective mechanism for scalable, geometry-aware depth estimation and may transfer to related tasks such as multi-view stereo and volumetric reconstruction [16,32]. Future work includes adaptive model selection, larger and more diverse plenoptic training data, and integration with RGB-D fusion [20,28].

8. Conclusions

We presented DSER, a hybrid light field depth estimation framework that unifies LSG initialization, plane-sweeping cost aggregation, and EPI-based multiscale refinement within a single scalable pipeline. Across Boxes, Dino, and Cotton, the proposed EPI2 variant achieved the best accuracy-efficiency trade-off, approaching Plane Sweeping accuracy at substantially lower runtime while consistently outperforming LSG in reconstruction quality [10,14]. These results demonstrate that spectral epipolar priors and multiscale refinement constitute an effective and practical strategy for robust dense light field reconstruction, with clear potential for extension to broader 3D vision tasks [20,32].

9. Broader Impact Statement

This work presents DSER, a hybrid light field depth estimation framework that improves the accuracy–efficiency trade-off for dense disparity reconstruction from plenoptic imagery. We discuss the foreseeable positive and negative societal consequences of this research.

Intended applications and positive impact

Dense, geometrically consistent depth maps are a core primitive for a wide range of socially beneficial technologies. DSER’s favorable runtime ($\approx 20$ s on a single consumer-grade GPU, a $\sim 17\times$ speedup over exhaustive plane sweeping at comparable accuracy) lowers the computational barrier for practitioners without access to large-scale hardware, democratizing high-quality 3D reconstruction. Foreseeable beneficial applications include:

• Medical imaging and surgical robotics. Light field endoscopes and depth-from-focus microscopes require fast, structure-preserving depth estimates in real or near-real time. DSER’s occlusion-aware propagation and boundary sharpness are particularly relevant for tissue segmentation and instrument localization, where depth discontinuities carry diagnostic significance.
• Assistive technology. Robust light field depth estimation can improve obstacle detection and scene understanding in mobility aids and wearable navigation systems for visually impaired users, especially in texture-poor indoor environments where gradient-only methods degrade.
• Cultural heritage and scientific digitization. High-fidelity 3D reconstruction of artifacts, archaeological sites, and natural specimens benefits from the structural consistency and low boundary error that DSER achieves on low-texture and partially occluded scenes.
• Autonomous systems and robotics. Accurate, efficient depth estimation is a critical perception primitive for path planning, 3D mapping, and manipulation in service and field robots. DSER’s efficiency profile makes it viable for onboard processing under strict power and latency budgets.

Limitations and risks

We acknowledge several potential negative or unintended consequences:

• Surveillance and privacy. Like all dense 3D reconstruction methods, DSER could, in principle, be integrated into surveillance pipelines that reconstruct the geometry of individuals or spaces without consent. We do not develop any surveillance application, and the present work is limited to controlled benchmarks and publicly available real-world light field datasets. We encourage practitioners who deploy this or related work in public-facing systems to comply with applicable privacy regulations and to implement appropriate safeguards.
• Dual-use in autonomous weaponry. Improved depth perception could be applied to autonomous targeting or navigation in military platforms. The authors neither design nor intend DSER for such use and note that existing, highly mature depth-sensing modalities (LiDAR, structured light) already serve this domain. The marginal capability uplift from this work in a military context is therefore minimal.
• Dataset and benchmark bias. Our primary evaluation uses the Heidelberg Light Field Benchmark and the Stanford Lytro Archive, both of which contain controlled laboratory or indoor scenes captured with specific plenoptic hardware. Performance may degrade in scenes with diverse illumination, outdoor conditions, or non-standard sensor configurations, and conclusions about accuracy or efficiency may not generalize uniformly to all deployment contexts. We report this limitation explicitly in Section 6.1 and encourage evaluation on broader, more demographically and geographically diverse scene sets as the field matures.
• Environmental cost. Although DSER is significantly faster than exhaustive plane sweeping and does not require large-scale model training (unlike deep learning baselines), iterative spectral refinement and cost-volume construction remain non-trivial computationally. For large-scale or continuous-deployment scenarios, the aggregate energy consumption of inference should be weighed against the application benefit.

Data and model transparency

All experiments use publicly released benchmark datasets with documented licenses. No personally identifiable information, biometric data, or sensitive human-subject content is involved. We will release source code, pre-computed results, and configuration files upon acceptance to facilitate reproducibility and independent auditing of our claims.

Summary

On balance, we believe the benefits of DSER, more accessible and geometrically accurate 3D reconstruction from light fields outweighthe foreseeable risks, which are largely shared with the broad category of 3D computer vision research and are not unique to this contribution. We remain committed to responsible disclosure, open evaluation, and constructive engagement with the broader research community on the ethical dimensions of 3D perception technology.

Appendix B.1. The Two-Plane Parameterisation

A 4D light field $\mathcal{L}:{\mathbb{R}}^2\times {\mathbb{R}}^2\to \mathbb{R}$ is parameterized by the two-plane model [12], where $(x,y)\in \mathcal{S}$ denotes a point on the spatial (image) plane and $(u,v)\in \mathcal{A}$ a direction on the angular (aperture) plane. The radiance of a ray through the aperture point $(u,v)$ hitting the spatial plane at $(x,y)$ is $\mathcal{L}(x,y,u,v)$.

Definition A1 

(Sub-aperture view). The sub-aperture view at angular position $(u,v)$ is the 2D slice

$$I_{u,v}(x,y)\mathcal{L}(x,y,u,v).$$

(A1)

Definition A2 

(Epipolar Plane Image (EPI)). Fixing y and v, the horizontal EPI is the 2D slice

$$E_h(x,u)\mathcal{L}(x,y_0,u,v_0).$$

(A2)

The vertical EPI $E_v(y,v)\mathcal{L}(x_0,y,u_0,v)$ is defined analogously.

Appendix B.2. The Epipolar Disparity Constraint

Under the Lambertian assumption and a fronto-parallel surface at depth Z, the projective shift induced by a lateral aperture displacement $({\Delta }_u,{\Delta }_v)$ is a disparity $d=fb/Z$, where f is the focal length and b is the baseline.

Proposition A1 

(Epipolar shift). For a Lambertian point at disparity d,

$$\mathcal{L}(x,y,u,v)=\mathcal{L}(x-d{\Delta }_u,y-d{\Delta }_v,u+{\Delta }_u,v+{\Delta }_v).$$

(A3)

Proof. 

Let $\mathbf{p}={(X,Y,Z)}^{\top }$ be the 3D scene point. The perspective projection onto sub-aperture view $(u,v)$ gives image coordinates

$$x_{u,v}={\displaystyle \frac{fX}{Z}}+u,y_{u,v}={\displaystyle \frac{fY}{Z}}+v.$$

A displacement $({\Delta }_u,{\Delta }_v)$ on the aperture plane shifts the image coordinates by $(-d{\Delta }_u,-d{\Delta }_v)$ with $d=f/Z·b$, which is exactly Eq. (A3).    □

Corollary A1 

(EPI line slope). In the horizontal EPI $E_h(x,u)$, a point at disparity d traces a line with slope $\partial x/\partial u=-d$. Disparity is therefore directly readable as the negated slope of iso-intensity lines in the EPI.

Corollary A1 is the geometric foundation of all EPI-based depth methods in DSER: reliable disparity estimation reduces to robust slope estimation in the 2D epipolar domain.

Appendix C.1. Frequency-Domain Formulation

Definition A3 

(2D Fourier transform of the EPI). Let’s ${\widehat{E}}_h(\xi ,\mu )$ denote the 2D Fourier transform of the horizontal EPI:

$${\widehat{E}}_h(\xi ,\mu )=\mathcal{F}\left\{E_h\right\}(\xi ,\mu )=\int \int E_h(x,u)e^{-2\pi i(\xi x+\mu u)}\mathrm{d}x\mathrm{d}u.$$

(A4)

Theorem A1 

(Spectral epipolar constraint). For a Lambertian surface of constant disparity d, the Fourier spectrum of the EPI is supported on the line

$$\mu =-d\xi .$$

(A5)

Equivalently, the energy of ${\widehat{E}}_h$ is concentrated in a wedge centred on this line, whose angular width is determined by the spatial bandwidth of the radiance function.

Proof. 

From Proposition A1, $E_h(x,u)=g(x+du)$ for some 1D radiance profile g. Taking the Fourier transform:

$$\begin{array}{cc}\hfill {\widehat{E}}_h(\xi ,\mu )& =\int \int g(x+du)e^{-2\pi i(\xi x+\mu u)}\mathrm{d}x\mathrm{d}u.\hfill \end{array}$$

(A6)

Substituting $s=x+du$:

$$\begin{array}{cc}\hfill {\widehat{E}}_h(\xi ,\mu )& =\widehat{g}\left(\xi \right)\int e^{-2\pi i(\mu +d\xi )u}\mathrm{d}u=\widehat{g}\left(\xi \right)\delta (\mu +d\xi ),\hfill \end{array}$$

(A7)

which is nonzero only when $\mu =-d\xi$, completing the proof.    □

Remark A1. 

Theorem A1 implies that multi-layer scenes produce a mixture of spectral lines. DSER separates these contributions by treating disparity estimation as spectral line decomposition, enabling frequency-consistent regularisation of the correspondence field.

Appendix C.2. Spectral Regularisation as a Frequency-Consistent Prior

Let ${\widehat{E}}_h^{\left(n\right)}$ denote the EPI spectrum estimated from noisy observations. The spectral prior in DSER is:

$$p(D\mid {\widehat{E}}_h)\propto exp\left(-{\lambda }_s\sum _{x,u}{\left|{\widehat{E}}_h^{\left(n\right)}(\xi ,\mu )-{\widehat{E}}_h(\xi ,-d\xi )\right|}^2\right),$$

(A8)

which penalises deviations from the theoretical spectral support locus. Maximising this prior is equivalent to finding the disparity field whose implied spectral lines best explain the observed EPI spectra.

Proposition A2 

(Equivalence to angular consistency). Minimising the spectral penalty in Eq. (A8) is equivalent to maximising angular consistency:

$$\begin{array}{c}\hfill \sum _{(u,v)\ne (u^{\prime },v^{\prime })}\parallel I_{u,v}(x,y)-I_{u^{\prime },v^{\prime }}(x+(d_{u,v}-d_{u^{\prime },v^{\prime }}){\Delta }_u,\\ \hfill y+(d_{u,v}-d_{u^{\prime },v^{\prime }}){\Delta }_v){\parallel }^2.\end{array}$$

(A9)

Proof. 

Parseval’s theorem gives ${∥{\widehat{E}}_h^{\left(n\right)}-{\widehat{E}}_h∥}_2^2={∥E_h^{\left(n\right)}-E_h∥}_2^2$. Expanding the spatial-domain residual over all angular pairs yields the stated angular consistency objective.    □

Appendix D.1. Derivation of the Closed-Form Estimator

The LSG estimator minimises the linearised reconstruction residual. Expanding Eq. (A3) to first order in $({\Delta }_u,{\Delta }_v)$:

$$\begin{array}{cc}\hfill \mathcal{L}(x,y,u,v)& -\mathcal{L}(x-d{\Delta }_u,y-d{\Delta }_v,u+{\Delta }_u,v+{\Delta }_v)\hfill \\ \hfill & \approx d(L_x{\Delta }_u+L_y{\Delta }_v)+(L_u{\Delta }_u+L_v{\Delta }_v)=0.\hfill \end{array}$$

(A10)

where $L_x,L_y,L_u,L_v$ are partial derivatives. This gives the per-ray linear constraint

$$d=-{\displaystyle \frac{L_u{\Delta }_u+L_v{\Delta }_v}{L_x{\Delta }_u+L_y{\Delta }_v}}.$$

(A11)

Aggregating Eq. (A11) over a local neighbourhood $\mathcal{N}(x,y)$ and all angular samples $({\Delta }_u,{\Delta }_v)$ in an overconstrained least-squares system yields:

Theorem A2 

(LSG closed-form solution). The least-squares disparity estimate that minimises

$$E_{\mathrm{LSG}}=\sum _{p\in \mathcal{N}}{(L_x{L}_u+L_y{L}_v)}^2/{(L_x^2+L_y^2)}^2$$

(A12)

is given by the closed form

$$d^{*}={\displaystyle \frac{{\displaystyle \sum _p(L_x{L}_u+L_y{L}_v)}}{{\displaystyle \sum _p(L_x^2+L_y^2)}}}.$$

(A13)

Proof. 

Setting the derivative of the sum-of-squared residuals with respect to d to zero:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial }{\partial d}}\sum _p{\left[d(L_x^2+L_y^2)+(L_x{L}_u+L_y{L}_v)\right]}^2=0,\end{array}$$

(A14)

rearranging gives Eq. (A13) directly.    □

Appendix D.2. Bias-Variance Analysis

Proposition A3 

(LSG estimator bias). Under additive zero-mean Gaussian noise $\eta \sim \mathcal{N}(0,{\sigma }^2)$ on the light field intensities, the LSG estimator is asymptotically unbiased:

$$\mathbb{E}\left[d^{*}\right]\to d_{\mathrm{true}}\mathrm{as}\left|\mathcal{N}\right|\to \infty .$$

(A15)

Proof. 

Write the noisy gradient ${\tilde{L}}_x=L_x+{\eta }_x$. The numerator and denominator of Eq. (A13) become sums of products of independent noise terms. By the law of large numbers, cross terms ${\sum }_p{\eta }_x{\eta }_u\to 0$ while the signal terms dominate as $\left|\mathcal{N}\right|\to \infty$, proving asymptotic unbiasedness.    □

Proposition A4 

(Breakdown under low texture). The LSG estimator is undefined whenever ${\sum }_p(L_x^2+L_y^2)\approx 0$, i.e., in regions where the spatial gradient is near zero. The condition number of the associated $2\times 2$ normal equations grows as $\kappa \propto 1/{\lambda }_{min}$, where ${\lambda }_{min}$ is the smallest eigenvalue of the local structure tensor $\mathbf{J}={\sum }_p{\nabla }_{s}L{\nabla }_s{L}^{\top }$.

Proposition A4 formally characterizes why LSG fails in textureless regions and motivates supplementing it with the plane-sweeping cost volume.

Appendix D.3. Overall Pipeline

Algorithm A1 summarizes the proposed framework.

Algorithm A1 Light Field Depth Estimation

Require:  Light field $L(x,y,u,v)$, mask $M(x,y)$ Ensure:  Refined disparity map $D(x,y)$ 1: Normalise, resize, inpaint, and mask the input light field 2: Compute initial disparity $d_{\mathrm{LSG}}$ from spatial-angular gradients 3: for all disparities d do 4:     Warp sub-aperture views; compute cost $C(x,y,d)$ 5: end for 6: $d_{\mathrm{sweep}}(x,y)\leftarrow arg{min}_{d}C(x,y,d)$ 7: for all$(x,y)$do 8:     Refine via EPI consistency score $S(x,y,d)$ 9: end for 10: while resolution > threshold do 11:     Downsample, refine, upsample to next scale 12: end while

Appendix E.1. Variance-Based Matching Cost

For a sheared light field $L_d(x,y,u,v)=L(x+ud,y+vd,u,v)$, the matching cost is the variance over angular views:

$$C(x,y,d)={\displaystyle \frac{1}{\left|\mathcal{A}\right|}}\sum _{(u,v)\in \mathcal{A}}{\left[L_d(x,y,u,v)-{\overline{L}}_d(x,y)\right]}^2,$$

(A16)

where ${\overline{L}}_d(x,y)={\displaystyle \frac{1}{\left|\mathcal{A}\right|}}{\sum }_{u,v}{L}_d(x,y,u,v)$.

Theorem A3 

(Minimum-variance consistency). Under the Lambertian model, $C(x,y,d)$ attains its global minimum at $d=d_{\mathrm{true}}(x,y)$, with minimum value $C_{min}=0$ in the noise-free case.

Proof. 

At the true disparity, $L_d(x,y,u,v)=g(x,y)$ for all $(u,v)$ by Proposition A1, so all angular views agree and the variance is zero. For any $d\ne d_{\mathrm{true}}$, the shear introduces parallax residuals, which by Jensen’s inequality yield $C(x,y,d)>0$. □

Appendix E.2. Statistical Efficiency of the Variance Cost

Proposition A5 

(CRLB for variance-based disparity). Under additive i.i.d. noise $\eta \sim \mathcal{N}(0,{\sigma }^2)$, the Cramér-Rao lower bound on the variance of any unbiased disparity estimator is

$$\mathrm{Var}\left[\widehat{d}\right]\ge {\displaystyle \frac{{\sigma }^2}{{\displaystyle \sum _{u,v}{\left[{\nabla }_{x}L·{(u,v)}^{\top }\right]}^2}}}.$$

(A17)

The variance cost achieves this bound asymptotically as $\left|\mathcal{A}\right|\to \infty$.

Proof. 

The log-likelihood under the Gaussian noise model is $\ell \left(d\right)=-{\displaystyle \frac{1}{2{\sigma }^2}}{\sum }_{u,v}{[L(x+ud,y+vd,u,v)-g(x,y)]}^2$. Computing the Fisher information $\mathcal{I}\left(d\right)=-\mathbb{E}[{\partial }^2\ell /\partial d^2]$ gives the denominator of Eq. (A17), and the Cramér-Rao inequality $\mathrm{Var}\left[\widehat{d}\right]\ge 1/\mathcal{I}\left(d\right)$ completes the proof. □

Appendix E.3. Complexity vs. Accuracy Trade-Off

The cost volume requires $O\left(\right|\mathcal{S}\left|·\right|\mathcal{A}\left|·N_d\right)$ operations, where $N_d$ is the number of discrete disparity hypotheses. In contrast, LSG requires $O\left(\right|\mathcal{S}\left|·\right|\mathcal{A}\left|\right)$ operations. The ratio $N_d$ (typically 64–256 in practice) quantifies the runtime penalty of exhaustive search, motivating the hybrid DSER pipeline that uses plane sweeping only to resolve LSG failures and EPI-refinement to sharpen the result.

Appendix F.1. Data and Smoothness Terms

The full disparity field D is estimated by minimising

$$\begin{array}{cc}\hfill E\left(D\right)& =\underset{\mathrm{data}\mathrm{term}}{\underbrace{{\sum }_{x,y}{\rho }_d\left(I(x,y)-I_D(x,y;D)\right)}}\hfill \\ \hfill & +\lambda \underset{\mathrm{smoothness}\mathrm{term}}{\underbrace{{\sum }_{(x,y)\sim (x^{\prime },y^{\prime })}{\rho }_s\left(D(x,y)-D(x^{\prime },y^{\prime })\right)}}.\hfill \end{array}$$

(A18)

where $(x,y)\sim (x^{\prime },y^{\prime })$ denotes spatial neighbourhood pairs.

Definition A4 

($\rho$-function). Both ${\rho }_d$ and ${\rho }_s$ are convex, non-decreasing, and sub-quadratic loss functions (e.g. Huber, truncated quadratic, or Charbonnier):

$$\rho (r;ϵ)=\sqrt{r^2+{ϵ}^2}-ϵ,ϵ>0.$$

(A19)

Appendix F.2. Existence and Uniqueness

Theorem A4 

(Well-posedness). Let ${\rho }_d$ and ${\rho }_s$ be strictly convex. Then $E\left(D\right)$ attains a unique global minimiser $D^{*}$ on any compact, convex admissible set $\mathcal{D}\subset {\mathbb{R}}^{\left|\mathcal{S}\right|}$.

Proof. 

E is continuous and strictly convex as a sum of strictly convex functions composed with linear maps. By the extreme value theorem, E attains its minimum on the compact set $\mathcal{D}$. Strict convexity guarantees uniqueness. □

Remark A2. 

In practice, ${\rho }_s$ is often chosen to be non-strictly convex (e.g. truncated quadratic) to allow sharp discontinuities. In this case, uniqueness is not guaranteed globally, but the functional remains lower semicontinuous and its minimisers correspond to piecewise-smooth disparity fields that respect depth boundaries.

Appendix F.3. Anisotropic Smoothness and Edge Preservation

The smoothness weight is chosen anisotropically:

$$\lambda (x,y,x^{\prime },y^{\prime })={\lambda }_{0}exp\left(-\beta {∥I(x,y)-I(x^{\prime },y^{\prime })∥}_2^2\right),$$

(A20)

where $\beta >0$ controls edge sensitivity.

Proposition A6 

(Edge-preserving behaviour). As $\beta \to \infty$, $\lambda (x,y,x^{\prime },y^{\prime })\to 0$ across image edges (where ${∥I(x,y)-I(x^{\prime },y^{\prime })∥}_2^2\gg 0$) and $\lambda \to {\lambda }_0$ in homogeneous regions. Consequently, the minimiser $D^{*}$ of E with anisotropic weights exhibits unconstrained variation across edges and penalised variation within homogeneous regions, formally justifying depth discontinuity preservation.

Appendix G.1. Edge Confidence

The edge confidence $C_e(x,y)$ aggregates local photometric contrast:

$$C_e(x,y)=\sum _{(x^{\prime },y^{\prime })\in \mathcal{N}(x,y)}{∥I(x,y)-I(x^{\prime },y^{\prime })∥}_2.$$

(A21)

Proposition A7 

(Relationship to structure tensor). $C_e(x,y)$ is proportional to the trace of the local structure tensor $\mathbf{J}(x,y)$, i.e. $C_e(x,y)\propto tr\left(\mathbf{J}\right)={\lambda }_1+{\lambda }_2$, where ${\lambda }_1,{\lambda }_2$ are the eigenvalues of $\mathbf{J}$. Hence $C_e$ is high at edges and corners and low in textureless regions.

Appendix G.2. Colour-Density Score via Mean Shift

The depth confidence score is defined via kernel density estimation across angular views:

$$S(x,y,d)={\displaystyle \frac{1}{\left|R\right|}}\sum _{r\in R(x,y,u,v,d)}K\left({\displaystyle \frac{r-\overline{r}}{h}}\right),$$

(A22)

where K is a Gaussian kernel with bandwidth h, and $\overline{r}$ is the mean-shift mode iterate.

Theorem A5 

(Mean-shift convergence). The mean-shift update $\overline{r}\leftarrow {\sum }_{r}K(r-\overline{r})r/{\sum }_{r}K(r-\overline{r})$ converges to a local mode of the kernel density estimate $\widehat{p}\left(r\right)={\displaystyle \frac{1}{\left|R\right|h}}{\sum }_{r}K((r-\overline{r})/h)$. The disparity $d^{*}(x,y)={arg\; max}_{d}S(x,y,d)$ corresponds to the colour mode most consistent with a fronto-parallel surface at depth d.

Proof. 

Mean-shift convergence follows from Cheng (1995): the update is a fixed-point iteration that ascends the gradient of $\widehat{p}$ in the RKHS induced by K, and is therefore guaranteed to converge to a stationary point from any initialisation. □

Appendix G.3. Directed Random Walk as Graph Regularisation

Definition A5 

(Depth graph). Define an undirected graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{W})$ where $\mathcal{V}=\mathcal{S}$ (image pixels), $\mathcal{E}$ connects 4-adjacent pixels, and the edge weight is

$$W_{(x,y),(x^{\prime },y^{\prime })}=exp\left(-\gamma {∥\nabla I(x,y)∥}_2^2\right)·C_d(x,y)·C_d(x^{\prime },y^{\prime }),$$

(A23)

with $C_d(x,y)=C_e(x,y)·∥S_{max}-\overline{S}∥$ the joint confidence.

Theorem A6 

(DRW as MAP estimation). Propagating disparity values along the directed random walk is equivalent to MAP estimation in a Gaussian Markov Random Field (GMRF) on $\mathcal{G}$:

$$\begin{array}{cc}\hfill D^{*}={arg\; min}_D[& \sum _{v\in \mathcal{V}}{C}_d\left(v\right){\left(D\left(v\right)-\widehat{D}\left(v\right)\right)}^2\hfill \\ \hfill & +\mu \sum _{(v,v^{\prime })\in \mathcal{E}}{W}_{v{v}^{\prime }}{\left(D\left(v\right)-D\left(v^{\prime }\right)\right)}^2].\hfill \end{array}$$

(A24)

where $\widehat{D}\left(v\right)$ is the initial fused disparity estimate.

Proof. 

Eq. (A24) is the energy of a GMRF with data fidelity weighted by confidence $C_d$ and smoothness weighted by W. Setting the gradient to zero yields a sparse linear system $(\mathbf{C}+\mu \mathbf{L})\overrightarrow{d}=\mathbf{C}\widehat{\overrightarrow{d}}$, where $\mathbf{C}=\mathrm{diag}\left(C_d\right)$ and $\mathbf{L}$ is the weighted graph Laplacian. This linear system is the solution to a generalised random walk on $\mathcal{G}$ with absorbing states at high-confidence pixels, establishing the claimed equivalence. □

Corollary A2 

(Edge-aligned propagation). Since $W_{v{v}^{\prime }}$ is small across strong image gradients (by Eq. (A20)), the DRW solution propagates disparity predominantly along iso-intensity contours, formally guaranteeing that depth discontinuities align with photometric edges.

Appendix H.1. Pyramid Construction

Let $D^{\left(0\right)}=D$ denote the full-resolution disparity map and $D^{\left(k\right)}$ the map at pyramid level k, obtained by k successive applications of a Gaussian downsampling operator $\mathbf{S}$:

$$D^{\left(k\right)}=\mathbf{S}{D}^{(k-1)}.$$

(A25)

Each level is upsampled to the next by a bilinear interpolation operator $\mathbf{U}$ and refined by solving a coarse-to-fine version of Eq. (A18).

Appendix H.2. Error Propagation Bound

Theorem A7 

(Multiscale error bound). Let ${ϵ}^{\left(k\right)}={∥D^{\left(k\right)}-D_{\mathrm{true}}^{\left(k\right)}∥}_2$ denote the estimation error at pyramid level k. Under Lipschitz-continuous ${\rho }_d$ and ${\rho }_s$ with constants $L_d$ and $L_s$, the error satisfies

$${ϵ}^{(k-1)}\le c_u{ϵ}^{\left(k\right)}+{\delta }^{(k-1)},$$

(A26)

where $c_u<1$ is a contraction factor depending on the upsampling operator and the smoothness ratio $\lambda L_s/L_d$, and ${\delta }^{(k-1)}$ is the approximation error introduced by downsampling.

Proof. 

The coarse-level energy minimiser $D^{\left(k\right)*}$ satisfies ${∥D^{\left(k\right)*}-D_{\mathrm{true}}^{\left(k\right)}∥}_2\le {ϵ}^{\left(k\right)}$ by assumption. After upsampling and one gradient step of the fine-level energy, the Lipschitz condition gives ${∥D^{(k-1)}-D_{\mathrm{true}}^{(k-1)}∥}_2\le c_u{∥\mathbf{U}{D}^{\left(k\right)}-\mathbf{U}{D}_{\mathrm{true}}^{\left(k\right)}∥}_2+{\delta }^{(k-1)}$, and ${∥\mathbf{U}{D}^{\left(k\right)}-\mathbf{U}{D}_{\mathrm{true}}^{\left(k\right)}∥}_2\le c_u{ϵ}^{\left(k\right)}$ by the bounded operator norm of $\mathbf{U}$, giving Eq. (A26). □

Corollary A3 

(Global error convergence). Unrolling Eq. (A26) over K levels:

$${ϵ}^{\left(0\right)}\le c_u^K{ϵ}^{\left(K\right)}+\sum _{k=0}^{K-1}{c}_u^k{\delta }^{\left(k\right)}.$$

(A27)

For $c_u<1$, the first term vanishes geometrically and the total error is bounded by the cumulative downsampling approximation. Choosing a sufficiently coarse maximum level K ensures that the global optimum is reached efficiently.

Appendix I.1. Depth-Disparity Relationship

Depth and disparity are related by $Z=fb/d$, giving the depth error as a function of disparity error:

$$\delta Z={\displaystyle \frac{\partial Z}{\partial d}}\delta d=-{\displaystyle \frac{fb}{d^2}}\delta d.$$

(A28)

Hence the relative depth error is $\delta Z/Z=-\delta d/d$, and MSE in depth space satisfies

$${\mathrm{MSE}}_Z={\displaystyle \frac{f^2{b}^2}{d^4}}{\mathrm{MSE}}_d.$$

(A29)

Remark A3. 

Equation (A28) shows that depth errors are larger at small disparities (distant surfaces), which is consistent with the observation in the main paper that the Dino scene (with both near and far structures) is harder to reconstruct than Cotton.

Appendix I.2. PSNR as a Reconstruction Fidelity Metric

Proposition A8 

(PSNR monotonicity). $\mathrm{PSNR}\left(d\right)=10{log}_{10}(MA{X}_I^2/\mathrm{MSE}\left(d\right))$ is a strictly decreasing function of $\mathrm{MSE}$ and strictly increasing as depth reconstruction quality improves. A 1 dB increase in PSNR corresponds to a reduction in MSE by a factor of ${10}^{0.1}\approx 1.26$.

Proof. 

Differentiating: $\partial \mathrm{PSNR}/\partial \mathrm{MSE}=-10/(\mathrm{MSE}ln10)<0$, confirming strict decrease. □

Appendix I.3. Diminishing Returns of Depth Sampling

Let $N_d$ be the number of discrete depth planes in the disparity range $[d_{min},d_{max}]$. The quantisation error in the disparity estimate is bounded by $|\delta d_q|\le \Delta d/2$ where $\Delta d=(d_{max}-d_{min})/N_d$. The resulting MSE due to quantisation alone is:

$${\mathrm{MSE}}_q\le {\displaystyle \frac{{(d_{max}-d_{min})}^2}{12N_d^2}}.$$

(A30)

Proposition A9 

(Diminishing returns of sampling). From Eq. (A30), ${\mathrm{MSE}}_q\propto N_d^{-2}$. The marginal gain in MSE reduction per additional depth plane is $\partial {\mathrm{MSE}}_q/\partial N_d\propto -N_d^{-3}$, which decays cubically. Beyond a threshold $N_d^{*}$, the reduction in MSE becomes smaller than the photometric noise floor ${\sigma }^2$, rendering further refinement statistically uninformative.

This result formally justifies the empirical observation in the main paper (Figure 5) that MSE reductions become marginal beyond 11 depth planes.

Figure A1. Combined results summary. (a) Accuracy—efficiency scatter. (b) Per-scene PSNR. (c) MSE vs. depth planes. (d) Average runtime. Together, the panels show that EPI2 achieves near-optimal quality at a fraction of Plane Sweeping’s cost.

Figure A1. Combined results summary. (a) Accuracy—efficiency scatter. (b) Per-scene PSNR. (c) MSE vs. depth planes. (d) Average runtime. Together, the panels show that EPI2 achieves near-optimal quality at a fraction of Plane Sweeping’s cost.

Figure A1 consolidates the main quantitative findings: EPI2 consistently occupies the best accuracy-efficiency regime, validating the proposed spectral epipolar refinement strategy.

Figure A2. Accuracy-efficiency trade-off (PSNR vs. runtime, log scale) for LSG, plane sweeping, EPI1, and EPI2 across Boxes, Dino, and Cotton. Small markers denote individual scenes; large markers denote method means. EPI2 lies on the Pareto frontier, achieving near-peak PSNR at a much lower runtime than plane sweeping.

Figure A2. Accuracy-efficiency trade-off (PSNR vs. runtime, log scale) for LSG, plane sweeping, EPI1, and EPI2 across Boxes, Dino, and Cotton. Small markers denote individual scenes; large markers denote method means. EPI2 lies on the Pareto frontier, achieving near-peak PSNR at a much lower runtime than plane sweeping.

Figure A2 summarizes the accuracy-runtime trade-off. LSG is the fastest ($\approx 19$ s) but least accurate, while Plane Sweeping achieves the highest raw PSNR at prohibitive cost ($\approx 350$ s) [10,14]. EPI1 occupies an intermediate regime. EPI2 reaches near-plane-sweeping accuracy ($32.96$ dB) at $\approx 20$ s, corresponding to a $\sim 17\times$ speedup through anisotropic epipolar refinement rather than exhaustive search [22,24].