A Hardware-Accelerated System for the Compression, Transmission, Decoding and Display of 360° Panoramic 4K Video

1. Introduction

Applications such as wide-area surveillance, border security, and port and air-traffic monitoring require high-resolution 360° panoramic imagery acquired by field-mounted cameras to be monitored at a remote center with near-real-time latency. However, 4K-resolution (3840 × 2160) panoramic video with 12-bit color depth (YCbCr 4:2:0) requires a bandwidth on the order of 5 Gbps in uncompressed form, and it is therefore not possible to transmit it directly over the bandwidth-limited 100 Mbps leased lines that are available in practice.

This paper presents an end-to-end system designed and implemented to solve this problem. The system is based on compressing the images on the transmitter side with the hardware HEVC (H.265) encoder of NVIDIA GPUs [1,2,3], transmitting them with the RTP protocol [4] to a receiver system located up to 1000 km away, and decoding them on the receiver side, again with a hardware HEVC decoder, for live display at 5 fps on ten 4K monitors. The complete system was designed to operate under an end-to-end latency constraint of at most 2 s; the measurements performed show that the latency remains below 1 s.

The remainder of the paper is organized as follows. Section 2 presents the system architecture and image acquisition, Section 3 the transmitter system and the shared-memory data structures, Section 4 the bandwidth analysis and compression, Section 5 the receiver system, Section 6 the experimental setup, and Section 7 the results and discussion; Section 8 concludes the paper.

2. System Architecture and Image Acquisition

The system consists of two main subsystems, each comprising hardware and software: the Transmitter System, which reads the images from the camera, compresses them and transmits them to the remote site, and the Receiver System, which decodes the incoming video stream and renders it on the displays. The two systems are connected to each other by a leased network link with a bandwidth of at most 100 Mbps and a length of at most 1000 km. The operation of the end-to-end system is shown in Figure 1.

The camera, which captures high-resolution, 360° panoramic color images at a high rate, does not deliver the full panorama as a single image but as 10 sub-images (sectors), each corresponding to a 36° angular slice. The resolution of each sector is 3840 × 2160 pixels and the color format is 12-bit YCbCr 4:2:0 [5]. The images are produced at 5 frames per second (5 fps). The image-acquisition process is summarized in Figure 2.

3. Transmitter System

A 64-bit Linux operating system runs on the transmitter system. The frame grabber (video capture board), which is installed in the system through a PCI Express x16 interface, is connected directly to the camera. The device driver of the board writes each frame received from the camera — consisting of 10 raw YCbCr 4:2:0 images of 2160 × 3840 pixels together with a header — into the shared memory (System V shared memory) of the operating system, organized as a circular list [6]. Semaphores are used to coordinate the write and read operations on the shared memory.

3.1. Shared-Memory Data Structures

The segment layout of the data structure held in shared memory is given in Table 1, the image header structure in Table 2, and the image buffer structure in Table 3.

3.2. Hardware HEVC Encoding and Transmission over RTP

An application developed in C++ reads the image data in shared memory in parallel, compresses it using the hardware HEVC encoder of the GPU (NVENC) [3], and sends the 10 sectors that make up one frame in parallel over 10 separate channels to the remote receiver system using the RTP protocol [4].

4. Bandwidth Analysis and Compression

The total amount of information held in shared memory for one frame is 124,417,128 bytes, and it consists of the following components:

• Buffer offset: 8 bytes
• Image header × 10: 112 bytes × 10 = 1,120 bytes
• Image buffer × 10: 12,441,600 bytes × 10 = 124,416,000 bytes

The size of a single image buffer is calculated as 2160 × 3840 × 1.5 = 12,441,600 bytes, on the basis of an average of 1.5 bytes per pixel in the YCbCr 4:2:0 format.

Accordingly, the amount of data required for one frame is 124,417,128 bytes = 995,337,024 bits. Since the camera records at 5 fps, the total amount of raw data per second is 5 × 995,337,024 bits = 4,976,685,120 bits, that is, approximately 5 Gbps (≈ 593 MiB/s). Given that the available link rate is at most 100 Mbps, it is not possible to transmit this data without compression.

For this reason, the hardware HEVC (H.265) codec provided by the GPU [1,2] was used and the amount of data was reduced to approximately one hundredth of the raw rate; after compression, 1 s of video is reduced to approximately 47 Mbit, so that the required bandwidth is at most 50 Mbps. The compressed video stream was transmitted to the receiver system using the RTP protocol [4].

5. Receiver System

The video stream arriving from the transmitter system at a rate of at most 50 Mbps is decoded at the receiver system, again using the hardware HEVC decoder of the GPU (NVDEC) [3], and is written into shared memory in the structure given in Table 1 and Table 2 and 3. A separate C++ application running on this system reads the images from shared memory and renders them at 5 fps on ten portrait-oriented 4K monitors. In this way, the 360° 4K panoramic images acquired at the source can be monitored in near real time at the remote site.

6. Experimental Setup

The compression (transmitter) and decoding (receiver) systems were each built on a standard small-form-factor (SFF) PC. The hardware and software components used are summarized in Table 4.

7. Results and Discussion

The measurements performed on the implemented system show that the end-to-end latency, from the acquisition of an image at the camera to its display on the monitors at the remote site, remains below 1 s. This value is clearly below the 2 s latency limit set as the design target. Hardware HEVC encoding reduced the raw data rate from approximately 5 Gbps to at most 50 Mbps (a reduction by a factor of roughly 100), making transmission over the bandwidth-limited 100 Mbps link possible. The use of a separate RTP channel per sector allowed encoding and transmission to be carried out in parallel for all 10 sectors.

8. Conclusions

In this study, an end-to-end system was designed and implemented for the hardware-accelerated compression of 4K-resolution 360° panoramic video, its transmission over a long distance on a link with limited bandwidth, its decoding, and its live display in a multi-screen environment. By combining GPU-based hardware HEVC encoding and decoding, a circular-list data structure in System V shared memory, and parallel per-sector RTP streams, raw data on the order of 5 Gbps was reduced to below 50 Mbps and an end-to-end latency of less than 1 s was achieved. Future work will target support for higher frame rates, improved error resilience (e.g., FEC), and the evaluation of hardware support for next-generation codecs such as AV1.