Divergent Drivers of Effective (K_eff) and Diffuse (K_d) Attenuation in the Baltic Sea: Quantifying the Roles of TSM and CDOM in Case 2 Waters

1. Introduction

The optical complexity of shallow coastal waters, commonly classified as Case 2 waters, poses a persistent challenge for remote sensing and underwater visibility modelling [1,2]. Unlike open ocean (Case 1) environments, where optical properties are primarily covariant with phytoplankton biomass [3], coastal zones are shaped by independent variations in optically active constituents (OACs), including terrestrial runoff, anthropogenic inputs, and sediment resuspension [4,5,6]. These factors create dynamic optical regimes where scattering often outweighs absorption, complicating the analysis of radiative transfer processes that govern light penetration [1,7,8].

Effective monitoring of these Case 2-water bodies requires accurate vertical profiling of the water column. While passive satellite radiometry provides synoptic coverage [8,9], it is often limited to the first optical depth [8,10] and, in shallow waters, is frequently confounded by bottom reflectance artefacts [11,12]. Active remote sensing, particularly LiDAR (Light Detection and Ranging), offers a potential solution for depth-resolved measurements [13]. However, the accuracy of such systems depends heavily on a robust assessment of the effective attenuation coefficient ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$) [14,15] and its relationship to the passive diffuse attenuation coefficient of downwelling irradiance ($K_{\mathrm{d}}$) [16,17].

Historically, bio-optical models have relied on a geometric simplification, assuming that the effective signal attenuates at approximately twice the rate of the diffuse signal ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$) to account for the two-way photon path [10,13]. While valid in clear, absorption-dominated waters [18,19], this linear assumption frequently breaks down in turbid environments [20,21]. In such waters, the specific angular acceptance of narrow-field sensors results in the “geometric rejection” of scattered photons—photons that are lost to the signal but continue to contribute to the diffuse irradiance field measured by broad-field passive sensors [10,19,22,23].

Although validation studies have examined the relationship between system and diffuse attenuation for LiDAR ($K_{\mathrm{L}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{r}}$) applications [13,17,24], these are predominantly restricted to monochromatic wavelengths (typically 532 nm). These comparisons revealed that $K_{\mathrm{L}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{r}}$ (≈$K_{\mathrm{e}\mathrm{f}\mathrm{f}}$) usually sits distinctively between $K_{\mathrm{d}}$ and the beam attenuation coefficient (c), depending on the field of view (FOV). Conversely, while the partitioning of $K_{\mathrm{d}}$ into constituent drivers i.e., coloured dissolved organic matter (CDOM), suspended particulate matter (SPM), and phytoplankton, is well-established in Baltic/North Sea waters [25,26], these passive-only studies do not account for the geometric complexities of the active signal. Consequently, few studies have bridged this gap to quantify the full spectral variability of the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$ ratio in shallow coastal systems, or explicitly decoupled the influence of specific OACs on this divergence. This gap is important, as applying fixed geometric scaling in Case 2 waters can lead to systematic errors in depth retrieval and water property inversions [12,22,27].

This study addresses these uncertainties by analysing in-situ optical data collected from the southern Baltic Sea, encompassing a diverse range of optical conditions from turbid estuarine to clearer open coastal waters. We aim to: (1) classify the regional optical variability into distinct Optical Water Types (OWTs) using K-Means clustering; (2) quantify the spectral deviation of the observed $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$ratio from theoretical geometric expectations; and (3) determine the specific drivers of attenuation for active versus passive signals using Random Forest (RF) regression. By identifying the divergent control of total suspended matter (TSM) and coloured dissolved organic matter absorption at 440 nm ($a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}\left(440\right)}$) on these coefficients, this study demonstrate that scattering effects drive the ratio significantly beyond theoretical limits and propose a robust empirical correction for retrieving attenuation properties in optically complex, shallow waters.

2. Materials and Methods

2.1. Study Area

The study was conducted along the southern coast of the Baltic Sea, Germany, within the state of Mecklenburg-Vorpommern. The region is characterised by complex optical properties typical of Case 2 waters, ranging from “Mesotrophic/Coastal” to “Highly Turbid/Estuarine” environments. Field campaigns were conducted over two consecutive years: June–July 2022 and July–August 2023. A total of 16 sampling stations were visited, covering a geographic range from Wismar Marina in the west to Dierhagen in the east.

The water depth at the sampling stations ranged from shallow nearshore waters (1.2 m) to deeper coastal stations (4.0 m). A map of the study area and specific station locations is provided in Figure 1. A small hand-operated grab and an underwater camera mounted on a tripod were used to qualitatively determine bottom features. In parallel, Secchi-depth was determined by means of lowering a white disk of 50 cm diameter to determine water depth and transparency.

2.2. Water Sample Analysis

Surface water samples were collected at mid-depths using a water sampler to quantify for key OACs. This is assumed representative of the water column due to the well-mixed and shallow nature of all stations. The water is sampled in two replicates to be averaged. The OACs are quantified for (1) pigments: chlorophyll-a and phaeopigments as proxy to phytoplankton biomass (Chl-a and Phaeo, respectively) in mg m⁻³; (2) TSM in g m⁻³; and (3) $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ in m⁻¹.

Samples were analysed for Chl-a, Phaeo, and CDOM using a Perkin Elmer Lambda 2 double-beam spectrophotometer. To determine pigment concentrations, particulate matter was collected on 45 mm GF/F filters (0.7 µm pore size) and extracted using ethanol. Following the initial Chl-a measurement, the extract was acidified with hydrochloric acid to quantify Phaeo, consistent with the protocol described in [28].

Prior to analysis, both the samples and ultrapure water reference were passed through 0.2 µm Millipore filters. The absorbance was measured and converted to absorption coefficients following the protocols in [29]; the coefficient at 440 nm was used to represent CDOM concentration. To adjust for baseline offsets caused by scattering or salinity differences, the average absorption between 700 and 750 nm was subtracted from each spectrum [30].

TSM was quantified gravimetrically following the protocols in [31,32]. A known volume of water was filtered through pre-weighed 45 mm GF/F filters (0.7 µm pore size). To determine the mass of the retained solids, the filters were oven-dried for a minimum of 24 hours and re-weighed.

2.3. Diffuse Attenuation Coefficient, K_d

Spectral irradiance data were acquired using a Macam Photometrics SR9910 spectroradiometer (Macam Photometrics Ltd., Livingston, UK). The instrument was fitted with an integrating sphere light guide, allowing for the collection of light with a quasi-spherical field of view (FOV) (approximately 360^o). Measurements were recorded across the spectral range of 240 to 800 nm with a bandwidth of 1 nm, expressed in units of Wm⁻²nm⁻¹. Vertical profiles of downward diffuse irradiance were conducted at depths ranging from 1 m to 3 m. Each depth measurement represents the average of two replicates. To account for variations in incident solar radiation during profiling, concurrent surface irradiance was measured using a reference sensor positioned at the water surface. All profiles were measured from the sun-exposed side of the sampling platform to minimize shading artefacts.

Data post-processing included the removal of negative signals and the masking of values outside the visible range (400–750 nm). To mitigate high-frequency noise inherent in the raw measurements, spectral data were smoothed using a 10-point moving average filter. The spectrally resolved diffuse attenuation coefficient, $K_{\mathrm{d}}$(λ) (m⁻¹) was then calculated based on the exponential decay of light with depth, following the Beer-Bouguer-Lambert (BBL) law in Equation 1:

$$K_d\left(\lambda \right)=-{\displaystyle \frac{1}{z}}ln\left({\displaystyle \frac{I_z\left(\lambda \right)}{I_0\left(\lambda \right)}}\right)$$

(1)

where, $K_{\mathrm{d}}\left(\mathsf{\lambda }\right)$ is the diffuse attenuation coefficient at wavelength λ (m⁻¹); $z$ is the depth of the water column (m); $I_{\mathrm{z}}\left(\mathsf{\lambda }\right)$ is the downwelling irradiance at depth $z$; $I_0\left(\mathsf{\lambda }\right)$ is the incident irradiance at the water surface.

2.4. Spectral Reflectance and Target Decay Measurements

Hyperspectral radiometric measurements were conducted using a UDS-1100 spectroradiometer (Spectral Evolution, Haverhill, MA, USA). The instrument covers the spectral range of 320–1100 nm with a sampling bandwidth of 1 nm. A dual-sensor configuration was employed to acquire simultaneous readings of the downwelling solar irradiance ($E_{\mathrm{d}}\left(\lambda \right)$), collected via a cosine-corrected optical diffuser on the surface of the water, and the upwelling radiance ($L_{\mathrm{u}}\left(\lambda \right)$), collected via a fibre optic probe with a fixed 4^o FOV.

Prior to each sampling session, the instrument was calibrated against a white Spectralon reference panel (Labsphere Inc.) to ensure radiometric accuracy. To quantify the depth-dependent attenuation of light, a white reference target was lowered into the water column at incremental depths of 0.5 m. A minimum of five replicate scans were acquired and averaged for each depth step to minimise random noise. Sampling was restricted to stable illumination conditions between 11:00 and 15:00 local time. To mitigate environmental artefacts, measurements were taken on the sun-exposed side of the sampling area to avoid shading, and areas with glint or surface agitation were avoided. At stations influenced by tidal current, sampling coincided with high water slack to minimise turbidity caused by sediment resuspension.

The remote sensing reflectance, $R_{\mathrm{r}\mathrm{s}}$ was calculated as the ratio of upwelling radiance to downwelling irradiance. To isolate the reflectance decay of the target itself, the background signal from the water column (measured without the target) was subtracted from the total signal measured with the submerged target. The isolated target reflectance at a specific depth, $z$ is denoted as $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)$ in Equation 2 and 3:

$$R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)=R_{\mathrm{r}\mathrm{s},\mathrm{ }\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\left(z,\lambda \right)-R_{\mathrm{r}\mathrm{s},\mathrm{ }\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)$$

(2)

$$R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)=\left({\displaystyle \frac{L_{\mathrm{u},\mathrm{ }\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}(z,\lambda )}{E_{\mathrm{d}}\left(\lambda \right)}}\right)-\left({\displaystyle \frac{L_{\mathrm{u},\mathrm{ }\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)}{E_{\mathrm{d}}\left(\lambda \right)}}\right)$$

(3)

where, $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\left(z,\lambda \right)$ is the total reflectance measured with the white target at depth $z$; $R_{\mathrm{r}\mathrm{s},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)$ is the background reflectance of the water column measured without the target; $L_{\mathrm{u},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ and $L_{\mathrm{u},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}$ are the corresponding radiance for surface water with target, and without target, respectively; $E_{\mathrm{d}}\left(\lambda \right)$ is the concurrent surface downwelling irradiance.

As part of quality control, samples PL1 and NHB were excluded from final analysis due to non-physical behaviour (i.e., increasing reflectance with depth), which could be caused by passing turbidity, a highly absorbing or scattering layer, or possibly due to boundary effect at sites which are visible through the bottom [12].

2.5. Spectral Effective Attenuation Coefficient, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$

The derivation of the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ (λ) (m⁻¹) is based on the Beer-Bouguer-Lambert (BBL) extinction law [19,33]. While this fundamental principle describes exponential decay in idealised media, the multiple scattering characteristic of turbid Case 2 waters necessitates an empirical approximation for the return signal [34]. Applying this law to the depth-dependent decay of the measured target signal, the physical model is defined in Equation 4:

$$R_{rs,\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)=R_{\mathrm{r}\mathrm{s},\mathrm{ }\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(\lambda ,0\right)\bullet e^{-K_{eff}\left(\lambda \right)\bullet z}$$

(4)

where, $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)$ is the measured target reflectance at depth $z$; $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\left(\lambda ,0\right)$ represents the extrapolated theoretical reflectance just beneath the surface.

To determine $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$(λ), the measured reflectance values across all depths were linearised using a natural logarithmic transformation. This converts the exponential decay curve into a linear relationship, defined by Equation 5:

$$\mathit{ln}\left(R_{\mathrm{r}\mathrm{s},\mathrm{ }\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)\right)=\mathit{ln}{R}_{\mathrm{r}\mathrm{s},\mathrm{ }\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(\lambda ,0\right)-K_{\mathrm{e}\mathrm{f}\mathrm{f}}\left(\lambda \right)\bullet z$$

(5)

A log-linear least squares regression was applied to this transformed data across the measured spectrum (400–900 nm) for each station. Within this model, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$(λ) corresponds to the negative slope of the resulting regression line. By iteratively fitting this relationship and calculating the gradient for every wavelength, a continuous $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ spectrum was generated. To mitigate high-frequency noise and eliminate spectral spikes inherent in the field data, the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ spectra were smoothed using a 10-point moving average filter.

To ensure the physical validity of the derived attenuation coefficients, the goodness-of-fit was rigorously assessed for each spectral band. The reliability of the regression was quantified using the coefficient of determination (R²), the Root Mean Square Error (RMSE) and standard error for each station. Wavelengths were excluded from the analysis if the regression yielded an R² < 0.85. This strict linearity criterion confirmed the robustness of the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ values, ensuring that the signal was not compromised by the noise floor at greater depths. Furthermore, the maintenance of high linearity (R² < 0.85) suggests that bottom reflectance did not significantly contaminate the slope calculation for the retained stations, as such boundary effects would typically induce non-linear curvature in the log-transformed depth profiles (Table A1 in Appendix A).

2.6. Empirical Relationship Between $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$

To evaluate the theoretical approximation that effective attenuation is roughly double the diffuse attenuation ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$), we compared spectral $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$, and calculated the attenuation ratio ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$). Prior to analysis, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ data in the near-infrared spectral region (750–900 nm) were masked to exclude instability and low signal-to-noise ratios resulting from high water absorption and atmospheric interference.

We assessed three regression approaches to quantify the relationship: (1) a Zero-intercept Linear Model; (2) Linear Model with Intercept; and (3) Robust Linear Regression, applied to mitigate the sensitivity of standard regression to outliers, particularly in the high-attenuation range ($K_{\mathrm{d}}$ > 2 m⁻¹).

The Robust Regression provided the optimal fit, effectively focusing on the primary data cluster. To account for the influence of spectral outliers in the dataset, the robust linear model was fitted using M-estimation (Huber weights) via the MASS package in R [35]. The goodness-of-fit was assessed using the weighted coefficient of determination ($R_w^2$). This metric represents the proportion of variance explained by the model after down-weighting the influence of outliers based on the final robust weights derived from the Iteratively Reweighted Least Squares (IRLS) algorithm [36] presented in Equation 6:

$$R_w^2=1-{\displaystyle \frac{\sum w_i{(y_i-{ŷ}_i)}^2}{\sum w_i{(y_i-{ŷ}_w)}^2}}$$

(6)

where $w_i$ are the final robustness weights, $y_i$ are the observed values, ${ŷ}_i$ are the fitted values, and ${ŷ}_w$ is the weighted mean of the response variable. For the compiled dataset, the model yielded an $R_w^2$ of 0.40.

Finally, a residual analysis comparing the theoretical model and Robust Linear Regression model was performed to evaluate the improvement in fit.

2.7. Machine Learning Model (Random Forest)

To quantify the multivariate influence of water constituents on attenuation, a RF regression model was implemented using the randomForest package in R [37]. This algorithm, based on the method by Breiman [38], was specifically selected for its robustness in handling noisy in-situ measurements and its capability to resolve complex, non-linear interactions between biogeochemical variables without assuming parametric distributions.

$K_{\mathrm{e}\mathrm{f}\mathrm{f}}$  and $K_{\mathrm{d}}$  were designated as the target variables, with Chl-a, Phaeo, TSM and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}\left(440\right)}$  as predictors. To ensure reproducibility, a random seed (set.seed(123)) was initialised, and the dataset was randomly partitioned into a training set (70% of observations) and a testing set (30%). The RF model was trained on the training subset using an ensemble of 100 decision trees (ntree = 100). Model performance was evaluated using Out-of-Bag (OOB) estimates, specifically the RMSE, R², and percent variance explained. Finally, the relative influence of each optical constituent on $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$  and $K_{\mathrm{d}}$  was assessed through variable importance analysis (Increase in Node Purity).

3. Results

3.1. Optical Characterisation and Classification of Water Types

Data were collected from 16 stations along the southern Baltic Sea coast, encompassing a diverse range of optical conditions, reflecting the transition from estuarine to open coastal waters (Figure 1). The observed water depths ranged from 1.2 m to 4.0 m, with Secchi depths varying between 1.2 m and 3.8 m. In half of the stations, the Secchi depth is the same as the water depth, indicating full transparency to the bottom (Table 1).

The concentrations of OACs exhibited significant spatial variability. Chl-a concentrations ranged from 1.21 mg m⁻³ at Graal Müritz (GMB) to 31.3 mg m⁻³ at Schnatermann (SH2). TSM concentrations followed a similar pattern of variability, with the lowest value observed at Rerik (RR2, 0.33 g m⁻³) and the highest at Schnatermann (SH1, 25.3 g m⁻³). $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ ranged from 0.01 m⁻¹ in the clearer coastal waters at Nienhagen (NHB) and Kühlungsborn (KBB) to 0.24 m⁻¹ in the estuarine-influenced waters (SH1).

Based on K-Means clustering of these constituents, three distinct OWTs were identified (Figure 2):

• Clear/Low Turbidity (TSM-dominated): Characterised by high particulate scattering relative to phytoplankton biomass (Cluster means: Chl-a 4.31 mg m⁻³, TSM 12.9 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$  0.04 m⁻¹).
• Mesotrophic/Coastal: Characterised by moderate biological activity and turbidity (Cluster mean: Chl-a 5.85 mg m⁻³, TSM 3.00 g m⁻³, $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$   0.07 m⁻¹).
• Estuarine/High Turbidity: Characterised by elevated organic loads and CDOM influence (Cluster means: Chl-a 26.6 mg m⁻³, TSM 15.4 g m⁻³ and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$  0.20 m⁻¹).

PL1 and NHB were excluded from final attenuation analyses due to non-physical radiative transfer behaviour (i.e., increasing reflectance with depth), resulting in a final dataset of 14 stations.

). Cluster mean concentrations are: Mesotrophic/Coastal (Chl-a 5.85 mg m⁻³, TSM 3.00 g m⁻³ and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.07 m⁻¹); Estuarine/High Turbidity (Chl-a 26.6 mg m⁻³, TSM 15.4 g m⁻³ and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.20 m⁻¹), and Clear/Low Turbidity (TSM-dominated) (Chl-a 4.31 mg m⁻³, TSM 12.9 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.04 m⁻¹).

3.2. Vertical Decay of Spectral Reflectance

The vertical attenuation of the active optical signal was examined through spectral reflectance decay profiles. Figure 3 presents profiles for four representative stations: SH1 (Estuarine), RR1 (Mesotrophic), and KBB and HMB (TSM-dominated). In all cases, apparent spectral reflectance decreased exponentially with depth, though the rate of decay varied spectrally and spatially. The shape of reflectance is distinctly different in the range 400 to 550 nm, in which for SH1 and RR1 (Figure 3a,b), the reflectance magnitude rises gradually, compared to the KBB and HMB (Figure 3c,d), where the slopes are steeper near 450 nm before increasing slowly. Additionally, the magnitude of Reflectance in Figure 3b is considered to be one of the highest, reaching almost 18%, significantly higher than other samples.

3.3. Spectral Comparison of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and K_d

Figure 4 compares the spectral distribution of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$, and it is found that, across all optical environments, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ values were consistently higher than $K_d$values across the visible spectrum, by a factor of approximately 2-3. KBB (Figure 4c) is an exception, that $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ is not significantly higher than $K_{\mathrm{d}}$.

The divergence is most pronounced in the spectral extremes, while minimal and remaining consistent in the mid-spectrum. This difference was quantified using the spectral ratio $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$ (Figure 5). The theoretical geometric expectation for a diffuse light field suggests a ratio of approximately 2 (dashed red line in Figure 5). However, the observed ratios exceeded this theoretical limit significantly, particularly in the middle of the visible spectrum (500–650 nm). While this is true for most stations, KBB (Figure 5c) exhibits lower overall attenuation, with ratios approaching the theoretical geometric expectation.

3.4. Empirical Relationship Between $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$

A robust regression analysis was performed to quantify the relationship between $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$ using data from all 14 valid stations (Figure 6). The derived empirical relationship is given by Equation 7:

$$K_{\mathrm{e}\mathrm{f}\mathrm{f}}={1.71K}_{\mathrm{d}}+1.44$$

(7)

The regression (Pseudo R² = 0.40) reveals a substantial deviation from the theoretical origin-bound relationship ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$). The presence of a positive y-intercept (1.44 m⁻¹) indicates a baseline attenuation for the active signal that persists even as diffuse attenuation approaches zero.

Residual analysis (Figure 7) further differentiates the performance of the robust empirical fit versus the theoretical geometric model. The residuals for the robust regression (Figure 7, left panel) are centred around zero, demonstrating an unbiased fit, although heteroscedasticity is observed at higher $K_{\mathrm{d}}$ values. Conversely, the theoretical model ( Figure 7 , right panel) exhibits a distinct positive bias, with the majority of residuals falling above zero, indicating that the theoretical assumption consistently underestimates the true magnitude of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ in these Case 2 waters.

3.5. Drivers of Attenuation Variability

RF regression analysis in Figure 8 shows the relative influence of different OACs on the variability of the two attenuation coefficients. The importance measures indicate that TSM is the primary driver for both $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$ in this environment. However, the relative importance of absorption-dominated components, specifically $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ and Phaeo, differed between the two coefficients. The influence of $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ and Phaeo was notably higher for $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ than for $K_{\mathrm{d}}$, suggesting that the active signal attenuation is more sensitive to dissolved absorbing components than the downwelling irradiance field in these waters. Chl-a ranks as the second most important variables for $K_d$, but it is the least for $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$.

4. Discussion

4.1. Divergence from Theoretical Geometric Attenuation

A central finding is that the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$ ratio in Case 2 waters significantly exceeds the classical theoretical value of 2. Consequently, assuming a two-way path factor of 2 has consistently underestimated the true attenuation of the active signal [13,39].The assumption $K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$ fails in Case 2 waters because it neglects the angular impact of high particulate scattering. In these environments, the total scattering coefficient (b) often significantly exceeds absorption (a) [10,12].

This high ratio arises because $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ scales with total beam attenuation (c = a + b) due to geometric rejection (scattered light is rejected by the sensor and counted as “loss”) [10,13] , while $K_{\mathrm{d}}$ remains constrained by absorption (a) and the average cosine (the degree of vertical photon alignment) [18].Consequently, in highly turbid waters where scattering dominates (b $\gg$   a ), the divergence between $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$ drives the ratio significantly above 2.

The divergence is most pronounced in the 500-650 nm spectral range (Figure 5), a region that typically corresponds to the “minimum absorption window” in coastal waters [40]. Paradoxically, the lower absorption in this window enhances the survival of photons, allowing them to travel further and undergo more scattering events. For $K_{\mathrm{d}}$, these scattered photons remain within the wide angular acceptance of the spherical collector (oriented to measure the downwelling field) used in this study [17] . However, for $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, which relies on a collimated source and limited detector FOV, these photons are scattered out of the receiver’s path [13,41,42].This “scattering trap” effectively removes photons from the active signal in the specific spectral bands where transmission is theoretically highest [43] , leading to the observed peak in the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_{\mathrm{d}}$ ratio.

The considerable y-intercept (1.44 m⁻¹) from the robust regression (Figure 6) suggests that there is significant active signal attenuation even as $K_{\mathrm{d}}$ approaches zero. While the rate of change (slope) is 1.71, the magnitude of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ is consistently higher due to this baseline offset (intercept). This could be because, in turbid waters, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ aligns more closely with beam attenuation (c; which rejects scattered light) than with diffuse attenuation [10,12,13].Consequently, modelling active attenuation via c or TSM offers a more physically robust approach than deriving it from passive irradiance [44,45].

4.2. The Decoupling of TSM and CDOM Influence

The decoupling of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$ through the distinct influence of OACs on each metric is proved in the RF analysis (Figure 8). For $K_{\mathrm{d}}$, the variable importance ranking follows a standard coastal hierarchy (TSM > Chl-a > CDOM > Phaeo), consistent with established bio-optical models for turbid, Case 2 environments [21,46–48].However, for $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, this ranking shifts significantly, with CDOM rising in importance and Chl-a dropping to the lowest rank (TSM > CDOM > Phaeo > Chl-a).

This divergence stems from the fundamental physical definitions of the coefficients. $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ behaves analogously to c, measuring total light interaction, which is the sum of absorption and total scattering coefficients (a + b), whereas $K_{\mathrm{d}}$ measures downward irradiance decay, which is governed primarily by absorption and backscattering (a + bb).

TSM dominates both scenarios, yet the driving mechanisms differ. For $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, TSM is the primary driver because it dictates the b. In turbid waters, forward scattering (bf) by particles is the single largest component of c, causing immediate geometric rejection (scattering) of the signal [22,49,50] . For $K_{\mathrm{d}}$, TSM dominates because it is the principal contributor to both non-algal particle absorption ( $a_{\mathrm{N}\mathrm{A}\mathrm{P}}$) and backscattering ( ${bb}_{\mathrm{N}\mathrm{A}\mathrm{P}}$) [51].Even when considering only backscattering, the particulate load remains the greatest source of total attenuation variation. In the study area, TSM-dominated examples KBB ( Figure 4c and 5c ) is a unique case that exhibits lower overall attenuation than others, likely because of the lack of absorption $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ (0.01 m⁻¹), reducing the absorption losses in the beam, even if scattering is present.

The most critical insight lies in the reordering of CDOM and Chl-a. For $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, CDOM rises to the second rank because its contribution to total absorption (a) directly amplifies signal loss in the collimated beam. Conversely, Chl-a and Phaeo rank lowest because, relative to the bulk mineral load, their contribution to the total scattering coefficient (b) is minor. The specific ordering of Phaeo > Chl-a suggests that in this dataset, absorption by detrital pigments ( $a_{\mathrm{P}\mathrm{h}\mathrm{a}\mathrm{e}\mathrm{o}}$) exceeds that of living phytoplankton ( $a_{\mathrm{P}\mathrm{h}}$), a scenario common in systems driven by resuspension and organic matter turnover [12].

In contrast, for $K_{\mathrm{d}}$, Chl-a retains a high ranking (second) because it impacts the light field through a dual role: Absorption ( $a_{\mathrm{P}\mathrm{h}}$) and backscattering ( ${bb}_{\mathrm{P}\mathrm{h}}$) [19,48] . This gives Chl-a a “competitive advantage” in importance over CDOM, which acts only as a pure absorber ( $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$) [12,52] . Since $K_{\mathrm{d}}$ relies heavily on the backscattering ratio, the scattering contribution of phytoplankton keeps it statistically more significant than CDOM [12,53].Phaeopigments rank last for $K_{\mathrm{d}}$ as their combined absorption and backscattering signature is negligible compared to the high background of TSM, Chl-a and CDOM.

While the Open Baltic Proper is optically dominated by CDOM [2,54,55], shallow coastal areas (such as in this study) shift towards a TSM-dominated regime [2]. This optical environment is shaped by inorganic suspended matter from coastal erosion, river runoff, and anthropogenic eutrophication [2,56,57]. These inputs, alongside sediment resuspension and wind-wave stirring, significantly elevate scattering (b) and beam attenuation (c) [23,58,59]. Consequently, TSM and Chl-a drive the optical properties in these turbid estuaries, resulting in high diffuse attenuation ($K_{\mathrm{d}}$) and limited light penetration [21,60].Our results reinforce this distinction, demonstrating that even in waters with high dissolved organic loads, attenuation is strictly limited by the geometric scattering of mineral particles. This confirms that the TSM > CDOM ranking for both $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_{\mathrm{d}}$ is a fundamental feature of radiative transfer in these Case 2 waters.

Ultimately, these results confirm the necessity of differentiated bio-optical modelling for Case 2 waters. Algorithms predicting $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ must prioritise the bulk particulate contribution (TSM) to total scattering (b), while models for $K_{\mathrm{d}}$ must accurately partition attenuation between pure absorption ( $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}\left(440\right)}$) and the combined absorption-scattering effects of living and detrital particles (Chl-a, TSM).

4.3. Implications for Optical Monitoring in Case 2 Waters

The breakdown of the linear $K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$ relationship has important implications for remote sensing in shallow coastal waters. Relying on a fixed geometric factor (G) systematically under-corrects for attenuation, leading to depth overestimation [12,22] driven by the high, variable scattering of suspended sediments [27] . Consequently, monitoring strategies must move beyond fixed Apparent Optical Property (AOP)-IOP scaling. While dynamic parameterisation of G is theoretically possible, the challenge of accurately inverting $R_{\mathrm{r}\mathrm{s}}$ for scattering properties in Case 2 waters makes this difficult. A more robust approach is to circumvent scaling entirely by directly retrieving the beam attenuation coefficient (c) via active lidar. Techniques that explicitly solve for c (such as LiDAR RTE inversion [61] , LIF Raman normalisation [62] , and sediment-adaptive bathymetry models [22] ) demonstrate superior robustness by eliminating the uncertainties of approximate geometric factors.

Our results suggest that operational models for these environments require a non-linear approach that explicitly accounts for the background geometric loss (the intercept) and the combined influence of TSM scattering and CDOM absorption ($a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}\left(440\right)}$). The empirical formula ( $K_{\mathrm{e}\mathrm{f}\mathrm{f}}={1.71K}_{\mathrm{d}}+1.44$) optimised based on the 14 sampling stations, is regional, and should be validated with a larger dataset in future studies.

Furthermore, the identification of distinct OWTs in this study supports the development of adaptive algorithms. Rather than applying a “one-size-fits-all” regression, future active sensors could utilise the spectral shape of the return signal to classify the water type (e.g., Mesotrophic vs. TSM-dominated) and dynamically switch to a class-specific attenuation model. This would significantly improve the accuracy of subsurface retrievals in spatially heterogeneous coastal zones.

4.4. Limitations and Future Perspectives

Limiting the study to shallow waters (<4.0 m) risks asymptotic instability and bottom reflectance artefacts [12]. As these boundary effects can invalidate standard reflectance models, we rigorously excluded all data exhibiting increasing reflectance with depth to ensure signal integrity. Despite that, focusing on the shallow domain is a strength, as these areas are often masked in passive satellite products due to bottom contamination. This study demonstrates that active techniques effectively characterise these ‘blind spots’, providing ground-truth where standard sensing fails.

Another limitation is the absence of direct measurements for non-algal particle absorption ($a_{\mathrm{N}\mathrm{A}\mathrm{P}}$) and backscattering ( ${bb}_{\mathrm{N}\mathrm{A}\mathrm{P}}$). While TSM served as a robust proxy due to its strong covariance with NAP in turbid systems [12,21] , this precludes the resolution of mass-specific sediment coefficients [21,51,53].Future campaigns should incorporate comprehensive IOP budgets to more precisely decouple the scattering and absorbing roles of the mineral fraction.

In future outlook, the distinct behaviour of active and passive signals supports a sensor fusion approach. By synergising passive radiometry for OWT classification and active LiDAR for vertical profiling (c), future systems can overcome the fixed geometric scaling limits of Case 2 waters. This combination leverages the strengths of spectral discrimination and depth resolution simultaneously.

5. Conclusions

This study challenges the theoretical assumption ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ ≈ 2 $K_{\mathrm{d}}$) used in bio-optical modelling for shallow, turbid waters. By characterising the optical properties of the southern Baltic Sea, we demonstrated that $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ consistently exceeds $K_{\mathrm{d}}$ by a factor greater than 2, particularly within the 500-650 nm spectral window.

Our analysis yielded three principal insights:

• Failure of Linear Scaling: In scattering-dominated Case 2 waters, the geometric rejection of scattered photons from the collimated active beam leads to a substantial divergence from passive diffuse attenuation. We propose robust empirical correction, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}={1.71K}_{\mathrm{d}}+1.44$ , which accounts for the significant baseline attenuation (intercept) largely ignored by zero-intercept models.
• Decoupled Drivers: While TSM is the primary driver for both coefficients, $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$  exhibits heightened sensitivity to CDOM compared to $K_{\mathrm{d}}$ .This confirms that active systems are more susceptible to beam attenuation losses (c) than to the backscattering-driven diffuse attenuation.
• Implications for Remote Sensing: The identification of distinct OWTs highlights the necessity for adaptive algorithms. Future LiDAR applications in coastal zones should avoid geometric scaling and instead utilise water-type classification to improve the accuracy of subsurface depth and property retrievals.

These findings suggest that operational monitoring in complex waters requires a sensor-fusion approach, leveraging passive radiometry to characterise water type and active LiDAR to resolve vertical structure, independent of classical geometric assumptions.