Image Processing Systems for Unmanned Aerial Vehicle: State-of-the-art

1. Introduction

Image processing has emerged as a vital field of study with numerous applications in various domains, including computer vision, medical imaging, remote sensing, and robotics [1-7]. Over the years, extensive research has been conducted to develop and improve image processing techniques, algorithms, and methodologies to extract meaningful information from images [8-10]. These studies have contributed valuable insights into various aspects of image processing, such as image enhancement [11], image segmentation image registration [12], and object recognition [13]. Additionally, researchers have investigated the integration of artificial intelligence and machine learning in image processing systems, leading to significant advancements in image classification [14], object detection [15], and image generation [16]. The constant evolution of image processing techniques has paved the way for groundbreaking applications, such as medical image analysis for disease diagnosis and treatment [17], facial recognition for security and authentication purposes [18-20], and satellite image processing for environmental monitoring and disaster management [21]. This introduction aims to provide an overview of the diverse and rapidly evolving landscape of image processing based on the findings of various research papers in the field.

1.1. Unmanned Aerial Vehicle

Unmanned Aerial Vehicles (UAVs), commonly known as drones, have emerged as a transformative technology with a wide range of applications. Numerous research studies have explored various aspects of UAVs considering manufacturing process [22-25], dynamics aspects [26-28], energy management [29] and control system [30] highlighting their potential and challenges. Research on UAV swarm intelligence and its applications in collaborative tasks that enables the UAV to independently make decisions based on shared information [31]. The use of UAVs for agricultural monitoring, emphasizing their role in precision farming and crop management [32]. Moreover, [33] explored the integration of UAVs with artificial intelligence for autonomous navigation and obstacle detection [34]. On the regulatory front, [35] analyzed the legal and ethical considerations surrounding UAV operations, addressing privacy, security, and airspace management. Additionally, [36] examined the use of UAVs in disaster response and humanitarian aid, demonstrating their effectiveness in remote sensing and data collection during emergencies. Challenges of UAV battery technology, aiming to enhance flight endurance and energy efficiency were observed [37]. On the commercial side, [38] studied the impact of drone delivery services on logistics and last-mile delivery solutions. Furthermore, [39] explored the use of UAVs in film-making and media production, showcasing their potential for aerial cinematography. Lastly, the emerging trends and future prospects of UAV technology, pointing towards advancements in swarm intelligence, miniaturization, and increased autonomy [40]. These research findings collectively illustrate the diverse and rapidly evolving landscape of UAVs, underscoring their significance across multiple industries and domains.

Figure 1. FPV Drone.

Figure 1. FPV Drone.

2. Implementation of UAV

This paper analyzes the implementation of UAVs in different sectors around the globe and the technology used to ensure the UAV can achieve the targeted requirements. Other sectors have emphasized defects or flaws to be inspected by the drone. Additional sensors or tools are equipped onto UAVs to scan the damaged structure. Depending on the application sector, UAVs can also use specialized microcontroller based monitoring systems [41-43]. Table 1 shows previous research on inspection drone applications.

Based on the above table, numerous methods were used to enhance drone application on an industrial scale. Under the Industry 4.0 framework, drones have proven significant tools in various industries in recent years. By incorporating UAVs into several industries, it is proven that UAVs are beneficial by reducing the operation cost, the possibility of accidents, and better efficiency [57].

3. FPV Camera

An FPV (First Person View) camera is a cutting-edge device that has revolutionized the world of remote-control hobbies and aerial activities. With its compact design and lightweight construction, the FPV camera offers users a real-time, immersive view from the perspective of their drones, RC cars, or other radio-controlled vehicles [58]. By transmitting live video feeds to specialized goggles or monitors, users can experience the thrill of piloting their vehicles from the inside, providing an adrenaline-packed experience for drone racing enthusiasts and FPV pilots [59]. Moreover, the low latency and high-resolution capabilities of FPV cameras contribute to a remarkable sense of speed and precision during flights or races [60]. As a result, FPV cameras have become an indispensable component in drone racing, freestyle flying, and aerial cinematography, elevating users’ enjoyment, and skill level to unprecedented levels [61]. Prominent brands like DJI, Fat Shark, Foxeer, RunCam, and TBS have been at the forefront of producing top-notch FPV cameras, incorporating the latest technologies to provide an unparalleled FPV experience for enthusiasts [62]. Some of FPV cameras can have a built-in gyroscope (MEMS angular velocity sensor) which makes it possible to provide smooth video and stabilized image [63].

Figure 2. Camera mounted on the UAV.

Figure 2. Camera mounted on the UAV.

Many FPV cameras have been developed recently to improve image quality and ease of use. Field of View (FOV) and lens focal length are vital considerations when choosing a camera [64]. The lens focal length influences the degree of FOV of the camera. Wider FOVs are often achieved by using lenses with shorter focal lengths. Table 2 shows the lens focal length and the approximate FOV for a camera with a 1/3-inch sensor size. FPV cameras would provide clear views both forward and backward. The advantage of flying forward and backward is that the drone does not need to make yaw maneuvers to gain a comprehensive picture of its surroundings [65].

Figure 3. Field of View of the camera.

Figure 3. Field of View of the camera.

The image sensor is the most crucial part of the camera. It creates an electrical signal from the image that the lens sensor has captured. Two imaging sensors commonly used in FPV cameras are charged coupling devices (CCD) and complementary metal oxide semiconductors (CMOS). CCDs and CMOS sensors both rely on metal-oxide-semiconductor (MOS) technology. CMOS sensors using MOSFET (MOS field-effect transistor) amplifiers and CCDs using MOS capacitors. Vacuum tubes of various types are typically used in analog sensors for infrared radiation, whereas flat-panel detectors are used in digital sensors. The CCD is an analog sensor, while CMOS is a digital sensor. The mechanism of the imaging sensor is that light absorbed by the sensors will create a charge, which is subsequently converted into a voltage video signal proportionate to its illumination [66]. Both sensors have pros and cons based on the mission of the quadcopter.

CCD cameras’ wide dynamic range (WDR) capabilities make them excellent in challenging lighting settings. With the proper settings, a decent CCD FPV camera lets the pilot see well, even when looking directly into the sun or pitch-black hours after sunset. Vibration problems do not affect CCD cameras as much as CMOS cameras. This is because CMOS cameras apply a system known as “rolling shutter,” which shoots from top to bottom. If the vibration is there, the picture becomes shaky.

Meanwhile, CCDs are often better suited for robotics applications because they perform better under varying illumination conditions and are less prone to rolling shutter deviations, which can cause image distortion during motion [67]. Besides that, CCD cameras require more power than CMOS cameras, and CCD sensors are considerably more costly. However, CMOS camera is widely used by leading technology companies such as GoPro and DJI, which are well known for their quality and reliable products [68]. Table 3 shows the defect detection method used in the current drones.

(a)

Advantages of CCD Imaging Sensor

Good performance in most lighting circumstances, especially in low light, is one of the benefits of CCD image sensors because the WDR feature adjusts the exposure and the color to be faultless [69]. The video has no vibration effect, and the image contrast is better than CMOS. The resulting image’s color is more natural and has lower noise.

(b)

Advantages of CMOS Imaging Sensor

Low power consumption and low latency are benefits of CMOS. This will result in the image distortion being at a minimum level due to small latency during data transfer [70]. A sharper, higher-resolution image can be obtained using a CMOS imaging sensor. Apart from that, CMOS is less expensive than CCD because the production cost is inexpensive.

3.1. Video Transmitter

A Video transmitter, or VTX, is a gadget attached to the camera and transmits the image in real time from the drone to an FPV receiver over the airways. A secure data transmission can be provided using the onboard system for neural network cryptographic data protection in real-time [75].VTX operates at a frequency of 5.8 GHz but may also broadcast the FPV signal at 900 MHz, 1.3 GHz, or 2.4 GHz, depending on the area. The drawback of employing a camera for a first-person view quadcopter is latency problems. However average delay of 100–200 milliseconds is barely detectable when flying in general [76].

The most crucial factor to consider in choosing VRX is VTX’s frequency. 5.8 GHz is the most used frequency on FPV equipment because this frequency is legal in most parts of the world. The frequency selection is dependable based on the range of the drone’s mission and high data rating [77]. Higher frequency signals carry greater bandwidth, which is advantageous, but they find it much more challenging to get through barriers like buildings and trees. High frequency is suitable for long-range missions, while low frequency is for short-range missions [78].

Programing a transmitter to broadcast on a specific frequency or channel is possible. When flying with other FPV pilots, having more channels that can be configured on the transmitter is helpful since each pilot will fly on a distinct frequency to ensure that the FPV video does not conflict with others. These days, 32 or 40 channel FPV transmitters are the most prevalent. Each transmitter will have a frequency table that lists each channel, band, and matching frequency, as shown in Table 4.

Next, an aspect that needs to be considered in choosing VTX is the output power of VTX because it affects the capability of VTX to transmit the video signal to the receiver. The amount of power transmitted from the transmitter is determined by its output power; commonly, the output power for VTX is 25mW, 200mW, and 600mW. The more extended range was obtained by high output power. However, the VTX might become hot and broken. Aside from that, when flying in an area with a lot of signal reflection (indoor environment), there are better choices than using a high-powered transmitter. Signal interference, or “multipath”, can occur when signals bounce off surfaces, including the floor, ceiling, and walls.

3.2. FPV Receiver

FPV receiver is an essential component in the FPV system that complements the FPV camera by receiving and displaying the live video feed from radio-controlled vehicles, such as drones and RC cars. This receiver acts as a bridge between the vehicle mounted FPV camera and the viewing device, which can be specialized goggles or monitors. It plays a crucial role in ensuring a seamless and real-time transmission of the video feed, enabling users to immerse themselves in the exhilarating experience of piloting their vehicles from a first-person perspective [79]. FPV receivers come in various frequencies, such as 5.8GHz, 2.4GHz, and 1.2GHz, each offering unique advantages and trade-offs in terms of range and signal penetration [80]. The receiver’s ability to handle multiple channels is crucial for racing events, where multiple pilots can simultaneously stream their video feeds [81]. To ensure a reliable and interference-free reception, some receivers are equipped with diverse systems that switch between multiple antennas to find the optimal signal [82]. Advances in FPV receiver technology have contributed significantly to the popularity and growth of FPV racing and other remote-control hobbies, providing users an unparalleled sense of control and excitement [83]. Video transmitters (VTX) send out radio frequencies received by the FPV video receivers, converting those signals into videos that can be viewed on our goggles and screens. FPV receivers are generally in the 5.8 GHz range, and most feature 48 channels total, separated into a few bands with eight channels each.

Figure 4. Process flow of streaming real time image from camera.

Figure 4. Process flow of streaming real time image from camera.

4. Image Filtering

According to Nayagam et al., 2018, digital image processing performs various operations and algorithms on digital images to produce enhanced images. Digital image processing encounters blurred, low-quality, monochrome images, and many more. This is the main reason many methods were created due to these difficulties. The three fundamental steps in image processing are acquiring the input from the source, analyzing and manipulating the image, and generating the enhanced output [84].

One of the valuable filters used in image and video analysis is Gaussian Filter that K.N Sivabalan introduced. The process of blurring a picture using a Gaussian function is called a “Gaussian filter,” sometimes known as “Gaussian blur” (named after mathematician and scientist Carl Friedrich Gauss). A Gaussian low-pass filter blurs specific picture areas and reduces noise (high-frequency components) [85]. The filter is constructed as an odd-sized symmetric kernel (DIP version of a matrix) and passed through each pixel in the region of interest to get the desired result. In processing images with fixed-point arithmetic, using a Gaussian filter increases processing effectiveness and lowers computing costs [86]. However, according to Cabello et al., 2015 heavy computational resources are needed to create a 2D Gaussian Filter for real-time applications. This research compares the processor used to implement a 2D Gaussian Filter. CPU, GPU, and Field programmable gate array (FPGA) were tested to observe the performance of the 2D Gaussian Filter. Fixed-point arithmetic is used to create a 2D Gaussian Filter in FPGA, proven to speed up processing [87].

Gaussian Filter is applied in the defect detection program that was created. It helps reduce the noise of the image captured by the camera mounted on the drone. Gaussian Filter filters the grayscale image; hence it will become blurry. This approach is used since it has an efficiency of 85% in detecting defects in textured and non-textured pictures [84]. Figure 5 shows an image that is filtered using a Gaussian filter.

Gary Bradsky invented OpenCV at Intel in 1999; the initial version was released in 2000. OpenCV is accessible on several operating systems, such as Windows, Linux, OS X, Android, and iOS, and it supports many programming languages, including C++, Python, Java, etc. A package of Python modules called OpenCV-Python was created to solve issues with computer vision. Intel first released OpenCV (Open-Source Computer Vision) as an open-source image and video analysis toolkit. The OpenCV library now contains almost 2500 optimized algorithms in image processing and computer vision. OpenCV is one of the most extensively used computer vision libraries, with many capabilities designed for Intel processors [88].

Approximately 2.5 million programmers have downloaded OpenCV because it is intuitive and easy to learn [89]. Apart from this, the module offered by OpenCV is open source, meaning it is free, and the code is portable. Even though the OpenCV function does not require high complexity of understanding, the designed algorithms are beneficial. Python-OpenCV offers a new alternative for academic research that requires image and video analysis [90]. OpenCV can convert the image into grayscale, blur the image, thresholding the image, and stream the live feed image from the camera [91].

Figure 6. The image is threshold.

Figure 6. The image is threshold.

Table 5. OpenCV library for image and video analysis.

Library	Function
cv2	Display the visual from the camera. Read the image input from the camera. Transform the image into grayscale, blur, and threshold.
NumPy	Arithmetic operations Handling a complex number
Scipy, spatial	Draw an object on the image Measure the size of an object

Table 5. OpenCV library for image and video analysis.

Library	Function
cv2	Display the visual from the camera. Read the image input from the camera. Transform the image into grayscale, blur, and threshold.
NumPy	Arithmetic operations Handling a complex number
Scipy, spatial	Draw an object on the image Measure the size of an object

5. Conclusion

Many sectors have emphasized that the drone could quickly inspect defects or flaws due to additional sensors or tools are equipped onto UAVs to scan the damaged structure. Effectiveness of UAVs in disaster response and relief efforts was proven [92]. UAVs show excellent performance in solving problems faced by several industries. However, difficulties in handling UAVs also were identified, such as photographic quality diminishes in dark environments and UAVs cannot clear debris or other obstructions.

The image sensor is the most crucial part of the camera. The mechanism of the imaging sensor is that light absorbed by the sensors will create a charge, which is subsequently converted into a voltage video signal proportionate to its illumination. Both sensors have their benefits. The CCD sensor shows good performance in low lighting, good image contrast, and the resulting image is more natural with low noise. Meanwhile, CMOS sensor needs low power consumption, produces a high-resolution image, and is inexpensive.

As UAV applications continue to diversify across domains, the insights presented in this paper serve as a foundation for inspiring future innovations and advancements in image processing systems, ultimately shaping the trajectory of UAV technology and its impact on society. As we move forward, it is evident that the synergy between UAVs and image processing will continue to drive innovation and shape the future of various domains. With the emergence of artificial intelligence, machine learning, and deep learning techniques, the potential for UAVs to autonomously interpret and respond to visual data opens up new horizons for applications that were once deemed unattainable.