Automated Photogrammetric Tool for Landslide Recognition and Volume Calculation Using Time-Lapse Imagery

1. Introduction

In recent decades, digital photogrammetry has emerged as a powerful and cost-effective method for 3D reconstruction of objects and surfaces across various scientific and engineering disciplines. Due to advantages such as ease of use and the capability of mounting cameras on drones, its application has expanded to engineering geology. This technology offers an affordable and efficient survey and monitoring solution compared to other terrestrial approaches like laser scanning and radar interferometry. Most photogrammetric applications in this field focus on the morphological 3D reconstruction of slope surfaces and rock masses [1,2,3]. More recently, its use has been extended to landslide displacement monitoring [4,5,6]. The occurrence of sudden collapses and local surface sliding on nearly vertical slopes represents precursor signals of landslide reactivation. These events provide critical information for understanding landslide behavior, assessing stability conditions, and forecasting potential evolution. Conventional techniques, such as topographic surveys, generally fail to capture these events due to excessive deformations associated with collapses and the disruption of optical targets. Moreover, these methods often provide sparse spatial information, limited to the locations of optical prisms. Radar interferometry, while effective for detecting gradual movements, is inadequate for capturing large sudden displacements. Similarly, laser scanner surveys can estimate missing volumes but face challenges in long-term monitoring. In this context, time-lapse close-range photogrammetry offers a viable alternative. It allows for prolonged observation of landslides, automatically obtaining detailed optical and spatial information from images. By comparing and analyzing image sequences from multiple cameras, local slip events and collapsed areas can be quickly identified. The volumes of collapsed material can be estimated based on the relative positions of the cameras. This data can then be used to hypothesize correlations between detachment events and driving factors such as rainfall and piezometric levels. Given the importance of automatic detection and susceptibility mapping for understanding landslide characteristics and risk assessment, numerous intelligent methods based on image analysis have been proposed [7,8,9]. Lei et al. [10] proposed an end-to-end change detection algorithm using a symmetric fully convolutional network. Lv et al. [11] introduced a landslide detection method based on multiscale segmentation and object-based majority voting of images. Convolutional neural network-based methods have been widely adopted in this field [9,12,13]. Additionally, Lu et al. [14] presented an object-oriented change detection method for rapid landslide mapping. While these algorithms can detect landslides over large areas using aerial or satellite images, they typically operate at low temporal resolution and often fail to detect detachments on nearly vertical slopes. In contrast, our approach focuses on terrestrial photo sequences at high spatial and medium-high temporal resolution. Terrestrial photos offer the advantage of adjustable spatial resolution (e.g., by using different lenses or changing the camera distance) and temporal resolution.

In this paper, we introduce a novel algorithm that automatically analyzes and processes sequences of terrestrial images from multiple cameras. Utilizing the Structural Similarity Index Measure (SSIM) algorithm [15] and a series of convolution filters, our program identifies collapse events and determines the collapsed areas. The stereo configuration of the cameras enables the calculation of the 3D shape and volume of the detached mass [16,17,18]. This stereo-camera system has been prototyped and tested in the field at the Perarolo landslide site, where spontaneous slope collapses frequently occur. The system, comprising three cameras and a large spotlight, was installed on the opposite side of the valley. Nearly a year of daily and nightly photo sequences were analyzed. Additionally, three large-scale slope stabilization operations using blasting and excavators at the same site were employed to evaluate the effectiveness of the collapse detection method and the accuracy of volume estimation, which was compared to laser scanner surveys.

2. Methods

2.1. Architecture of the Algorithm

Our program is designed to work with fixed multi-view cameras to perform image acquisition, collapse detection, and calculation of collapsed areas and volumes in a fully automatic and continuous manner, without any third-party intervention or manual operation. As shown in Figure 1, the process consists of two main phases: collapse recognition on the image plane through image comparison at different time steps, and identification and calculation of the collapsed area and volume. Before initiating the collapse recognition phase, wooded and grassy areas, if present, must be excluded from the analysis. The color and texture of these areas can change continuously due to wind and seasonal vegetation growth, hindering accurate comparison between successive images. This operation can be performed using a deep learning mask [19]. It should be noted that this limitation naturally restricts the method’s applicability in fully vegetated areas.

The second pre-processing step involves filtering out areas covered by shadows, which could introduce collapse artifacts, particularly due to changes in sun position over time [20]. A detailed analysis of this aspect will be provided in Section 2.3.

The core of the algorithm uses the Structural Similarity Index Measure (SSIM) to compare pairs of images in sequence, generating a structural similarity map at each time interval [21,22]. To reduce noise caused by natural small variations in color intensity, brightness, and contrast, image convolution filters are applied to the similarity map. We employed a cascade of Gaussian and median convolution filters with different patch sizes for this purpose [23]. Based on the statistics of pixel values in the final structural similarity map, potential collapse areas can be identified, and a unique structural similarity index is obtained. This index is used to determine whether a collapse occurred between the images taken at times t(i) and t(i+1). When the structural similarity index exceeds a certain threshold, the program signals the presence of a collapse and automatically initiates the calculation of the collapsed volume using images from a second camera. This second phase of the process involves comparing two point-clouds generated from the 3D reconstruction of the slope before and after the collapse. The details of this phase will be discussed in Section 2.4.

2.2. Structural Similarity Algorithm Approach

The Structural Similarity Index Measure (SSIM) algorithm is commonly used for assessing image quality and evaluating the performance of image compression algorithms and systems [24]. In this study, we propose using SSIM as a local measure of structural changes to detect collapses in a sequence of images taken from a fixed camera oriented towards the slope. The SSIM calculates the Structural Similarity Index (SSI) for each pixel in the image and generates a Structural Similarity Map (SSM) by comparing a target photo with a reference photo [22,25]. The SSI is derived from the variation of three components between the two images or portions of images x and y: the luminance term, the contrast term, and the structural term, according to the following metric:

$$SSI(x,y)={\left[l\right(x,y\left)\right]}^{\alpha }\bullet {\left[c\right(x,y\left)\right]}^{\beta }\bullet {\left[s\right(x,y\left)\right]}^{\gamma }$$

(1)

where

$$l(x,y)={\displaystyle \frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1}}$$

(2)

$$c(x,y)={\displaystyle \frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2}}$$

(3)

$$s(x,y)={\displaystyle \frac{{\sigma }_{xy}+C_3}{{\sigma }_x{\sigma }_y+C_3}}$$

(4)

and ${\mu }_x,{\mu }_y$, ${\sigma }_x{,\sigma }_y$ and ${\sigma }_{xy}$ are the local means, standard deviations, and cross-covariances for images x and y. The parameters α, β, γ are exponents for the luminance, contrast, and structural terms, respectively.

The Structural Similarity Map (SSM) is a grayscale map (values ranging from 0 to 1) with the same dimensions as the target image in SSIM. It consists of the SSI values for each pixel in the target image. Figure 2 illustrates an example of an SSM applied to two consecutive images of a landslide taken at different time steps: darker pixels indicate the smallest similarity values, representing areas with a higher probability of erosion or collapse, while lighter pixels indicate areas with high SSI values, signifying minimal structural and color changes. At first glance, it is evident that vegetated parts are classified as dark areas, which could lead to erroneous collapse detection. Additionally, due to the complex and changing environmental conditions of the landslide scene, factors such as brightness variation and shadows—primarily caused by clouds, fog, and sun position—can significantly affect the structural and chromatic information of the image. This can result in artifacts in the assessment of collapse areas. Therefore, it was deemed necessary to investigate the impact of these variables on the calculation of the SSI in detail, using sets of images where no collapse events occurred.

2.2.1. Effect of Shadows

Shadow is well known as one of the main sources of error in the analysis of time-lapse aerial and satellite images, and its detection remains a challenging task [20,26,27,28]. Two types of shadows may affect the images: cast shadows, produced by sunlight being blocked by asperities of the slope, which are regular and have a fairly constant periodicity, and shadows produced by clouds, which are generally less sharp and unpredictable. Essentially, there are property-based and model-based methods for identifying and extracting shadows cast on the ground from digital images [27,28]. The first category of methods uses information about colors and textures in the images for shadow segmentation and can be applied to most shadows. The second category uses external models based on timing, latitude, longitude, and digital elevation models to estimate the sun’s position and forecast shadow casting. The latter approach is suitable for analyzing zenithal images and is particularly effective if elevation information can be integrated [29]. In our application, we will employ the first type of shadow detection methods.

To evaluate the influence of shadows cast by the sun on the SSI, we selected a study case involving five sunny days from 27/07/2020 to 31/07/2020, choosing consecutive hourly images taken from 11 AM to 5 PM. For these five sets of images, we computed the SSI using the 3 PM shot as the reference image. In each group of images, the shadow cast on the slope gradually decreases from 11 AM to 3 PM, with the images taken at 4 PM and 5 PM having almost no shadows, as shown in Figure 3.

As shown in Figure 4, the SSI is very high and changes little when the reference image is compared with images with less shadow cast (Δt=−1,+1,+2), while it significantly decreases and changes greatly when the reference image is compared with images with strong shadows. We also found in Figure 4 that as the shadow in the images gradually decreases (Δt=−4,−3,−2,−1), the difference between the SSIM results with and without the shadow filter decreases significantly. Moreover, the results of the five groups of tests maintained the same trend. The overall trend and the standard deviation shown in Figure 4 suggest that shadows have a significant impact on SSIM.

2.2.1. Effect of Illuminance

To evaluate the effect of illuminance on the structural similarity metric, we conducted five groups of image sequence comparison tests using photos with different illuminance levels. Vegetation and shadow filters were applied in these tests. Each group of tests comprised consecutive night images and day images taken within the same time interval. From 23/02/2021 to 28/02/2021, we selected four night-images (taken at 11 PM, 1 AM, 3 AM, 5 AM) and four day-images (taken at 11 AM, 1 PM, 3 PM, 5 PM) as test images to compare with the reference image (taken at 8 AM) in each group. Figure 5 shows an example set of images used in one of these tests. The SSI between the target image and the reference image is illustrated in Figure 6a. It can be observed that the SSI changes little when the reference image is compared with the night images (t=23, 01, 03, 05). This is due to the artificial illuminance provided by the spotlight, which ensures a constant level of brightness. Shadows cast are fixed and limited within the region of interest, since the illuminator is close to the cameras and directed almost along the same line-of-sight. On the other hand, the varying illuminance of the day images (t=11, 13, 15, 17) has a significant effect on SSI due to the substantial changes in sunlight intensity and direction, as discussed earlier. Moreover, the results of the five groups of tests consistently showed the same trend. It is worth mentioning that the reference images of Group 5 have different illuminance levels compared to the reference images of the other groups (see Figure 6b), resulting in a significantly lower SSI for Group 5 compared to the others.

These conclusions highlight the significant impact of illuminance on SSIM. Considering this influence, it is preferable to use images with stable illuminance (e.g., night images) for comparison in collapse detection. Addressing changes in illuminance on the landslide site during daytime monitoring poses a challenging problem, and thus, it is necessary to develop filters to mitigate the influence of illuminance changes, which will be addressed in the next section.

2.3. Image Filters

Several filters are used in our program to enhance the accuracy of the collapse detection phase from images. These filters can be broadly categorized into two groups: image pre-processing filters and noise filters applied directly to the Structural Similarity Map (SSM). The image pre-processing filters include a vegetation filter and a shadow filter. The vegetation filter removes areas covered by vegetation, which were visually identified and sampled from a set of pre-learning images. These sampled areas were used to create a deep learning mask [30]. Similarly, the shadow filter removes areas covered by shadows, which were also visually identified and sampled from a set of pre-learning images. These sampled areas were used to create a color mask. These filters are applied before comparing two images to prevent false-positive identification of collapsed areas due to movement of vegetation and changes in shadows.

The filters applied to the SSM include various image convolution filters, which smooth the noise generated by SSIM to avoid detection of small spots with low SSI and to consolidate detected collapsed areas. In this case, we used multiple median filters with kernel sizes of 23x23 and 7x7 pixels to preserve edges of the SSM [31,32,33], threshold filters [34], and Gaussian filters with a 19x19 kernel and standard deviation of 3 [35,36] for final smoothing. Figure 7 illustrates the specific roles of these filters in image comparison by showing the evolution of the SSM at each step.

2.4. Calculation of Collapsed Volumes

2.4.1. 3D Surface Reconstruction and Disparity Calculation

Once a collapse is detected, the program automatically calculates the collapsed volume. The first step in this phase involves the 3D reconstruction of the slope surface before and after the collapse using fixed calibrated cameras that enable a stereoscopic view of the scene [37,38,39]. Each camera must be calibrated with a series of shots taken from different angles to calculate the intrinsic parameters, including the focal length distortion parameters, as well as the extrinsic parameters, which include the position and orientation of each image [40]. The extrinsic parameters require the system to be geolocalized. This is achieved using fixed points and colored balls specially positioned on the landslide, whose exact positions are determined through a GPS survey [41]. The final image used for calibration is taken from the camera’s operational position to obtain its accurate pose.

For the reconstruction of the scene based on only two images, the principles of stereophotogrammetry were used. Given the large area to be monitored and its distance from the cameras, a larger baseline with two slightly converging lines of sight has been chosen. Using the intrinsic and extrinsic parameters of the two cameras, the stereo parameters can be derived [42,43]. This includes the intrinsic parameters of the individual cameras and a roto-translation matrix that allows conversion from the reference system of the left camera to the reference system of the right camera. This matrix provides the relative position between the two cameras.

The second step is to obtain undistorted and rectified images from the left and right cameras using the stereo camera parameters and considering an epipolar geometry[44,45,46]. To ensure the stereo images are well-aligned and the 3D reconstruction is as accurate as possible, it is necessary to correct for any image oscillations. This correction is achieved by aligning fixed points in the scene using the calibration image as a reference.

The third and final step involves calculating the disparity value of each pixel in the rectified image to obtain the disparity map, which contains the depth information of each pixel in space. Using the disparity map and stereo camera parameters [47,48,49], the 3D position of each point is calculated, resulting in the 3D point cloud.

2.4.2. Point Cloud Comparison and Volume Calculation

After the 3D reconstruction, the next step is to combine the information of the collapse area obtained with SSIM and the point cloud of the entire landslide. As described in the workflow of Figure 8, the first step involves using a median filter with a 10-by-10 kernel to remove the noise generated in the 3D reconstruction stage. To determine the collapse volume, we compute the difference between the point clouds before and after the collapse and apply a threshold filter with a 0.5m threshold. Subsequently, a median filter with a 21-by-21 kernel is used to remove any remaining noise. Finally, we search for continuous areas in the binary map and use the largest area as the approximate collapsed area.

Next, we dilate the boundary of the approximate collapsed area and fill the invalid points within the collapsed area on the depth information map (disparity map or Z-axis coordinate map), as illustrated in Figure 9. We then compare the point clouds again to determine the exact collapsed area and its pixel coordinates. This step is crucial because invalid points in the disparity map can cause some real collapsed areas to go unrecognized; hence, identifying the lost real collapsed areas near the boundaries is essential. Figure 10 shows an example from the Perarolo landslide site.

Based on the pixel coordinates of the collapsed area on the rectified image, we can identify the points on the surface of the collapsed area. As shown in Figure 11, by using all the point clouds on the surface of the collapsed area before and after the collapse, we can create the alpha shape representing the collapsed bodies and easily estimate the collapsed volumes [50,51,52]. This approach also allows us to visualize the spatial shape of the collapsed bodies and calculate their 3D coordinates.

3. Case Study and Results

3.1. Perarolo Landslide Site

The Perarolo Landslide, also known as the Sant’Andrea Landslide, is located in the Cadore area of the Belluno province in northeastern Italy [53]. This area is one of the most geohazard-prone regions in the southern Alpine mountains. The coordinates of the landslide are 46°23′44″N and 12°21′23″E close to the Alpi Carniche. The landslide is part of an older landslide on the southern flank of Mount Zucco, covering approximately 72,000 square meters and extending from an elevation of 490 to 580 meters above sea level.

As shown in Figure 12, the active portion of the landslide is near the left bank of the Boite River. A village is on the right bank along the river, with its central area located downstream relative to the landslide. Additionally, about 100 meters southeast from the bottom of the landslide, there is a road (SP42) that ascends the gentle slope on the southeast side of the landslide. This road features a bend adjacent to the northeast corner of the upper part of the landslide. In the 1980s, a small railway line ran along the upper section of the landslide. However, when landslide activity was detected, the railway was relocated to a tunnel inside the mountain. Furthermore, the Piave River flows southwest on the east side of the landslide body, intersecting with the Boite River approximately 150 meters southeast of the landslide’s base. Behind the village, there is a hillside with a height comparable to that of the landslide, where the cameras used for visual monitoring in this study are installed. As shown in Figure 13, the camera system consists of and three Canon EOS 1300D cameras, each mounted in a covered iron box and equipped with remote connection and a solar charging panel. Moreover, a searchlight is also present for night photos acquisition.

3.2. Image-Based Collapse Detection at the Perarolo Landslide Site

To confirm the capability of our photogrammetric tool based on SSIM to detect collapses, we conducted a test using an image sequence acquired at the Perarolo landslide site from July 2020 to July 2021. One image was taken per day, except during the winter months when the slope was partially or totally covered by snow, and interferometric data and total station data were unavailable. The trend of SSI between the images on day i and i+1 is shown in Figure 14, where sudden changes above the threshold indicate the occurrence of a collapse, in 4 different periods (from 08/07/2020 to 30/09/2020; from 01/10/2020 to 01/12/2020; from 15/02/2021 to 30/04/2021; from 30/04/2021 to 19/07/2021.). We defined that when the SSI is lower than 0.9998, a collapse has occurred between the two images. This threshold is calculated based on the minimum area of detached mass that we aim to detect, depending on the ground pixel size. Considering the camera resolution, the distance from the slope surface, and the mean angle between the line of sight and surface normal, an SSI lower than 0.9998 corresponds to a collapsed area larger than about 7 m². As shown in Figure 14, the program detected 13 distinct collapse events during the monitoring period. These events are consistent with on-site observations and sudden acceleration of displacement rates obtained from radar interferometry just before the events.

The first nine events correspond to small natural collapses, as shown in Figure 15b. The comparison between Figure 15a,b allows the identification of changes in the slope’s condition before and after the event of 28/02/2021. Figure 15c-f illustrate other significant collapses that occurred in the Perarolo landslide area during June-July 2021. As mentioned earlier, a major natural collapse happened on 09/06/2021; the algorithm detected this extreme condition, assigning an SSI of 0.9135 (Figure 15c). Subsequently, on 25/06/2021, an artificial collapse was induced by explosives to restore the safety condition of the slope (SSI = 0.9449, Figure 15d). Finally, in July, anthropic activities were carried out, where a bulldozer reshaped the toe of the landslide. These operations were also correctly detected by the algorithm, with SSIs of 0.9936 and 0.9941 for July 13 and 14, respectively (Figures 15e-f).

For all the cases, the program identified the location of the collapsed area on the original images, as shown in Figure 16. We can assess its accuracy by comparing the original images obtained before and after the events. Since image availability may not be ensured on some days due to bad weather conditions or other technical reasons, we tested the accuracy of our method by comparing pairs of images taken at different time intervals and computing the SSI, essentially estimating a temporal correlation function. For this purpose, we used a daily sequence of 25 images before a collapse event and 25 images after the collapse, from 06/08/2020 to 24/09/2020, to calculate the SSI between images on day i and i+Δt. As shown in Figure 16, among the possible pairs of images, they can be categorized according to the Δt, and grouped into two categories: pairs of images crossing the collapse event (shown in red in Figure 17 and called “with collapse”), and pairs not crossing the event (shown in black and called “without collapse”).

With boxplots using whiskers at 1.5 times the interquartile range, we found that the median of the “with collapse” group fluctuates little with the change of Δt, and there are very few outliers, as shown in Figure 18a. It was found that almost all the SSI values in the “with collapse” group were lower than the threshold value, indicating that the lack of images for several days after a collapse does not affect its detection. On the other hand, in the boxplot of image pairs “without collapse” (Figure 18b), we can see that there are cases where the first quartile (Q1) is below the threshold when Δt≥14 days. When Δt=20 days, the median is also below the threshold. This means that the time interval between two consecutive images should be less than 14 days to ensure a 75% probability of detection. When the interval is 20 days, this probability drops to 50%

3.3. Volume Calculation in Perarolo Landslide Site

Our photogrammetric tool automatically calculates the collapse volume for each detected event. Table 1 illustrates the volumes of collapse events occurring after 09/04/2021, calculated using our photogrammetric tool. To assess the accuracy of this approach, we conducted a couple of laser scanner surveys before and after each of the collapses on 09/06/2021 and 25/06/2021. From the relative point clouds, we measured the collapse volumes using CloudCompare software (see Figure 19), evaluating the distance in vertical direction between the slope surfaces before and after each event. We compared them with the results of our photogrammetric approach, obtaining that the relative errors were found to be 6.6% and 3.1%, respectively.

4. Discussion

The results of the case study at the landslide site demonstrate the capability of our photogrammetric tool in rapid collapse detection and volume calculation. This small-scale time-lapse photogrammetric tool allows for quick collection of collapse information for landslides that continue to collapse, which is crucial for understanding landslide characteristics, back-analysis of collapse causes, and further research toward developing a collapse prediction tool. The fully automatic, non-contact, and low-cost features of this technology make it easily acceptable and suitable for widespread adoption.

However, the results from the tests over multiple days in Section 3.2 have shown that using the current threshold of 0.9998, comparing two images with an interval of more than 14 days may cause the program to fail. It is important to understand the maximum allowable interval between two consecutive images for reliable program operation, especially when natural environmental factors such as fog can prevent image acquisition. In this case, we found that the probability of program failure is 50% when the interval is 20 days. For other probabilities (75% or 90%), we plan to conduct tests with longer image sequences that include a collapse event in the middle, as soon as suitable data becomes available in the future. To further improve the accuracy of collapse detection, it is necessary to study the SSIM specifically for geotechnical materials and utilize machine learning algorithms to identify the most suitable image sequences. These efforts are currently ongoing.

5. Conclusions

This paper presents a new automatic photogrammetric tool designed for rapid collapse detection and calculation of collapsed volume. Collapse detection is achieved through the structural similarity comparison of consecutive image sequences. We investigated the impact of natural environmental factors at the landslide site on the classic image structural similarity algorithm (SSIM) and developed multiple filters within the tool to eliminate noise and optimize detection accuracy. The algorithm for automatic calculation of collapsed volume relies on 3D reconstruction and point cloud comparison. By analyzing image sequences taken over one year at the Perarolo landslide site, along with laser scanner surveys, we validated the tool’s effectiveness under various conditions.

However, further analyses are necessary to refine this tool and obtain precise information on the dynamics of the Perarolo landslide and other camera-instrumented landslide sites. Specifically, ongoing work includes studying the SSIM specific to geotechnical materials and applying machine learning algorithms to enhance the selection of the most suitable image sequences.