Evaluating HAS and Low-Latency Streaming Algorithms for Enhanced QoE

1. Introduction

Global internet traffic has exceeded 100 billion GB, and video content represents more than 80% of the total internet traffic in 2021 [1]. Adaptive streaming (HAS) HTTP sends dynamic video streams from the server to the client [2].

The MPEG Dynamic Adaptive Streaming over HTTP (DASH) standard is widely used for video streaming. An MPEG DASH streaming scenario is shown in Figure 1. In MPEG-DASH, the video content is encoded at different video bitrate and divided into small segments. The DASH client downloads the videos segment-by-segment from the server.

The DASH server hosts manifest files that contain information about video segments and quality levels [3]. The user initially requests the manifest file (e.g.,.mpd for DASH) from the server to retrieve information about available video segments, their quality levels, and timing [4].

The ABR algorithm configured on the client side selects the appropriate video segment based on the network and the client side configuration. These algorithms can be classified as throughput, buffer, and hybrid-based approaches. The main aim of the ABR algorithms is to ensure smooth streaming with the initial connection, lowest stalling, and switching events. A quality switch occurs when the two video segments are downloaded with different qualities. Stability is the state when the playback of the media stops because no further segment is available in the client buffer [5,6]. The video sequences are encoded with various bit rates and resolutions. The encoding strategy depends on various parameters, for example, segment size, bitrate, video resolution, and frame rate [7].

Traditional DASH-based ABR algorithms focus on quality of experience metrics such as downloading high quality video content, minimising playback interruptions, and video quality changes [2,8]. However, they often overlook the need to target low-latency streaming. Compared to terrestrial or satellite transmission, Internet broadcast has been shown to have a significant delay between video capture and playback. For video streaming over IP networks, maintaining end-to-end latency is crucial to achieve a user experience comparable to traditional broadcast TV [9]. There are several factors that influence latency, including video capture, encoding, packaging, video delivery through content delivery networks, segment buffering, and decoding. However, a study on Super Bowl 2024 latency found that streaming platforms over the Internet experienced delays averaging up to 70 seconds behind real-time action on the field. In contrast, cable and satellite channels are delivered to homes with an average delay of about five seconds behind the live feed. In the video streaming chain, this delay and other factors negatively impact the quality of experience (QoE) of the video [10].

To this end, low-latency ABR algorithms have been proposed to minimise end-to-end latency [2,11]. However, their performance has not been studied under different client- and server-end configurations. It is crucial to find the right balance between latency and QoE. In the design of traditional ABR algorithms, latency is ignored, and algorithms only simultaneously maximise QoE metrics, while the low-latency algorithms aim to minimise latency while maximising QoE metrics. The aim of this work is to investigate whether the prioritisation of latency by the low-latency algorithms negatively affects other QoE metrics. We analyse the performance of low-latency algorithms and compare it with traditional algorithms using the web-based DASH player, dash.js [12].

This work offers the following contributions.

• We evaluated, compared, and analysed the performance of low-latency and traditional ABR algorithms. The aim of the comparison is to analyse whether low-latency algorithms simultaneously optimise QoE metrics while prioritising minimising latency.
• We conducted extensive experiments to evaluate the effect of varying network conditions, segment durations, and different video data sets on the performance of the algorithms.
• We assess both the strengths and the shortcomings of low-latency algorithms and compare their performance with the traditional ABR algorithms using the DASH player, dash.js.

The paper is structured as follows. Section 2 presents the background work, Section 3 outlines the methodology, Section 4 provides the discussion, and Section 5 concludes the article.

2. Related Work

This section provides details about the current work in the adaptive streaming domain.

2.1. Adaptive Bitrate Algorithms

The goal of ABR algorithms is to maximise the quality of experience for the end user. The paper [13] highlights the ABR algorithms employed in adaptive streaming of HTTP. These algorithms target maximising the video quality, minimising re-buffering, time spent in re-buffering state, and the quality level switching. Bentaleb et al. [14] present a bitrate adaptation scheme that takes advantage of the nature of chunk downloads. This scheme uses a sliding window to assess bandwidth and an online linear adaptive filter for bandwidth prediction. Several ABR algorithms are designed to target low latency. In [15], the authors present a bandwidth-based algorithm that selects the bitrates based on the moving arithmetic mean and the standard deviation of throughput and latency. In [16] by Karagkioules et al., the algorithm aims to minimise latency and select the video quality for the next segment based on online convex optimisation. The algorithm provided by Bentaleb et al. in [17] uses heuristic and learning-based approaches to minimise latency and optimise QoE. The dynamic algorithm proposed by Spiteri et al. [18] is hybrid and employs a throughput algorithm. This algorithm is based on measurement throughput.

In [19], the authors created a data set from various content and video encoders. The data set is based on adaptive bitrate algorithms and performs a subjective evaluation on the data set. The results reveal that there is a correlation between the QoE models and the subjective opinions of the users. This shows that there is a need to improve the existing QoE models and the ABR algorithms.

An evaluation by Rodrigues et al. [20] is performed to find the trade-off between audio and video quality. The subjective evaluation is carried out in MPEG-DASH streaming environment. The results indicate that the proposed method is suitable for objective video quality evaluation in live streaming environments.

The work of O’Hanlon et al. [8] presents the evaluation results of the ABR algorithms with respect to a range of latency targets. The evaluation is performed in the dash.js player and the results reveal that the dynamic algorithm outperforms the low-latency algorithms. Low-latency algorithms achieve higher video quality at the expense of a higher number of playback interruptions. The researchers adjusted the low-latency L2A-LL algorithm and evaluated it under modified settings, revealing promising improvements. A limitation of this work is that the authors do not evaluate algorithms in different video content or investigate the impact of segment duration on algorithm performance. Furthermore, the analysis is based on only three ABR algorithms.

Lyko et al. [2] carry out an interesting evaluation with the low-latency algorithm. The evaluation is performed considering various QoE metrics and bandwidth profiles. In this work, the authors only evaluate low-latency algorithms and do not assess their performance against traditional DASH-based algorithms.

2.2. QoE Factors

There are several factors that affect the quality of the experience that can potentially affect the quality of the video. Lebreton et al. [21] highlight that stalling events and quality switches impact the user experience. The authors in [22] conduct a study on the correlation between media quality and the impact of event stalls on experience quality. The study suggests that if the stall events are very short, they are not noticed by users. The study also mentions that longer stalling events generally impact the user experience the most. The work by Lebreton et al. presented in [23] considers the user quitting ratio while watching videos in an adaptive streaming scenario. The results demonstrate the factors affecting the user persuasion. Another study by Lebreton et al. [24] proposes a method to predict the user quitting ratio while watching videos using adaptive streaming. The results reveal that quality, initial buffering, and stalling impact user behaviour.

Quality switching is also an important factor in quality of experience. The study provided by Babak et al. [22] reveals that frequent switching negatively impacts the overall user experience. The results also confirmed that the users preferred quality switching over stall events. However, as demonstrated in various studies [25,26], switching between quality levels also negatively impacts Quality of Experience. Switches of quality level and switching time affect user persuasion [6,27]. The research work by Allard et al. [28] investigates the trade-off between buffering delays and playback interruptions. The evaluation results reveal that, compared to the rebuffering events, playback interruptions negatively impact QoE. The findings of the study are used to build a model based on two factors: rebuffering and playback interruptions.

3. Experimental Setup

This section provides details about the methodology of the research work in detail.

3.1. Video Dataset

The dataset is acquired from the established DASH database [29]. In the data set quality levels, the configuration between videos is differentiated. It is also required that two sequences have the same configuration. Videos with varying complexities, including high-motion scenes and low-detail content, are used in the evaluation. The source videos and their characteristics are described in Table 1 in detail.

3.2. Segment Length and Quality Representations

The dataset chosen for evaluation is encoded using various segment sizes. The segment ranges from 2 seconds to 10 seconds, as recommended in publication [29]. The consideration of segment length is an important factor in video streaming. The short duration segments like 2 seconds provide more opportunities for the clients to adapt the bitrate. In addition, smaller segment sizes would produce greater overhead, as the client will frequently request the segments. Using longer-duration segment like 10 seconds may reduce the overhead, but the client would have a smaller number of opportunities to adapt the video rate. In case of sudden changes in the throughput, there is a higher probability of playback interruption in the case of a longer duration of the segment. In our evaluation, we consider both short and long duration segments to analyse the performance of the algorithms. The bitrates and quality level ladder are shown in Table 2 thoroughly.

3.3. Evaluation Test Bed

Performance evaluation is carried out using dash.js JavaScript-based tool. The library allows streaming MPEG-DASH media in device browsers. In this section, we discuss the details of the evaluation test bed. The architecture of the test bed is illustrated in Figure 2, and comprises four modules: two computers running Ubuntu, connected via Wi-Fi, simulating a video client and a server. The pre-encoded DASH videos were hosted on an Apache Web server. The key component of the architecture includes the bandwidth shaping and network emulation nodes which are based on Ubuntu utilities. The bandwidth shaping node manages the client’s maximum available bandwidth using the Linux traffic control system (tc) and the hierarchical token bucket (htb), a class-based queuing discipline (qdisc). The encoded videos are stored on the Apache server. The client streams the video sequences using ABR algorithms, which are deployed on the client side. The detailed architecture is shown in Figure 2.

3.4. ABR Algorithms Evaluation

In this work, the ABR algorithms [18,30,31] are considered on the basis of their working principles. These algorithms range from rate-based algorithms to low-latency algorithms. The Dynamic, BOLA, Throughput, L2ALL, and LOL+ algorithms are prototyped in a dynamic adaptive streaming framework called dash.js. This framework references the open-source implementation for the MPEG-DASH standard.

• Throughput: This algorithm makes a decision based on the throughput of the network. The algorithm estimates the throughput and decides which segment to download. This algorithm uses the average throughput of the previous video segment downloaded and decides on the optimal bitrate for the next video segment to be requested from the server [19].
• BOLA (Buffer Occupancy based Lyapunov Algorithm): The BOLA algorithm decides which bitrate to download based on the client buffer. The buffer level is related to the network throughput. This means that this buffer-based algorithm selects a high bitrate in case the buffer fill level is high, and a low bitrate if the buffer level is low. The buffer-based algorithm is chosen by the video streaming provider. The BOLA algorithm is suitable for fluctuation scenarios in the bandwidth [32].
• Dynamic: Dynamic is a hybrid algorithm. This algorithm makes full use of both throughput estimation and buffer levels. This algorithm smoothly switches between BOLA and throughput in real-time streaming. The algorithm addresses the shortcomings of the throughput and buffer-based algorithms [18].
• Learn2Adapt Low Latency (L2A-LL): L2A-LL is an adaptive bitrate (ABR) algorithm based on low latency. This algorithm uses the online convex optimisation principle. The L2A-LL aims to minimise the video latency. Compared to other ABR algorithms, L2A-LL provides a robust adaptation strategy. This algorithm does not require parameter tuning, channel model assumptions, or throughput assessment. These characteristics make L2A-LL ideal for users experiencing variations in the channel during streaming. Another feature of this algorithm is its modular architecture, which takes into account more QoE factors. These factors are categorised as stall, rebuffering, switching, and latency. These QoE factors consider various QoE objectives and streaming scenarios [2,8,33].
• Low on Latency (LOL+): This is a heuristic algorithm that uses learning principles to optimise for the best QoE. In LOL+ each segment boundary is estimated and the highest QoE is predicated. The ABR algorithm which is implemented on a SOM (Self-Organising Map) model which takes into consideration various QoE metrics and network variation. The LOL+ playback speed control module is based on a hybrid algorithm which measures latency and buffer level and manages playback speed. The LOL+, QoE evaluation module is responsible for QoE computation based on metrics such as segment bitrate, switching, rebuffer events, latency and playback speed [33,34].

3.5. Network Simulation

Figure 3, Figure 4, Figure 5, and Figure 6, shows the network profiles used to evaluate the performance of the algorithms. We evaluated the algorithms under different network conditions including (1) gradual changes in the throughput, (2) abrupt moderate variations in the throughput, and (3) abrupt large variations in the throughput. The motivation behind analysing algorithms under these network profiles is that if the algorithm reacts quickly to throughput changes, the algorithm will experience a high number of video rate changes. If the algorithm remains stable during throughput changes, it may respond too slowly to significant fluctuations, which may potentially lead to playback buffering or inefficient bandwidth utilisation. This makes it difficult for algorithms to enhance the user experience under different network conditions.

4. Results and Discussion

This section discusses the results and analysis of the experiments. This also presents the implications of the results in detail.

4.1. Analysis Under Network Profile 1

Here, we will evaluate the performance of the algorithms in different network profiles. First, we will analyse the performance of the algorithms under network profile 1. We start with a bandwidth of 1 Mbps, then increase it to 2 Mbps, and finally to 4 Mbps. Figure. Figure 7, illustrates the algorithms’ response while streaming Big Buck Bunny. The figure shows that LoL+ and dynamic algorithms rapidly increase the bitrate when the bandwidth increases from 1 Mbps to 4 Mbps. The L2A-LL and throughput algorithms delay increasing the bitrate to minimise the risk of interruption. When the bandwidth is increased to 4Mbps, all algorithms instantly increase the bitrate to efficiently use the bandwidth. Figure 8, Figure 9, and, Figure 10, shows that the dynamic algorithm has a similar response when streaming all the videos. Figure 11, illustrates the fact that the dynamic algorithm achieves the highest video rate in all experiments irrespective of the video. The LoL+ algorithm quickly adapts to the bandwidth changes while streaming Big Buck Bunny and Elephant, however, in case of tears of steel and Spark video, it initially increases bitrate but quickly decreases it to avoid playback interruption. The L2A-LL has a similar response to bandwidth changes in all videos, as it cautiously increases bitrates to small changes to avoid buffer overflow. The throughput algorithm carefully increases the bitrate for Big Buck Bunny and Elephant, but more aggressively improves video quality when streaming Tears of Steel and Spark. Figure 11, shows that the traditional DASH algorithms achieved higher bitrates when streaming Tears of Steel and Spark compared to BBB and Elephant, while the LoL + and L2A-LL algorithms achieved similar average bitrates for all experiments. Figure 12, shows that the LoL+ algorithm experienced the lowest number of video rate switches followed traditional DASH algorithms. The L2A-LL algorithm had the highest number of bitrate switches. However, as shown in Figure 7,8,9 and 10, most of these changes were minor and would likely go unnoticed by the user. The major bitrate switches were when the bandwidth encouraged the algorithms to increase the bitrate. In general, the dynamic algorithm exhibited a more consistent response to changes in bandwidth across all videos compared to low-latency algorithms.

4.2. Analysis Under Network Profile 2

First, we will analyse the performance of the algorithms in network profile 2, starting with a bandwidth of 8 Mbps, then dropping it to 500 kbps and finally increasing to 8 Mbps. The aim of evaluating the algorithm under network profile 2, is to analyse their performance under sudden large fluctuations in the throughput. During a significant drop in throughput, it is crucial for ABR algorithms to monitor the available buffer levels and determine how to avoid rebuffering without compromising video quality. However, striking this balance poses a significant challenge for algorithms. Let $\left(B^k\right)$ be the buffer level at the beginning of the download of the $k^{th}$ segment, then the buffer level before the download of the $(k^{th}+1)$ segment will be given by:

$$B^{k+1}=B^k+\tau -\left(\tau \times {\displaystyle \frac{R^k}{T^k}}\right)$$

(1)

Where $\left(R^k\right)$ is the bitrate selected for the $k^{th}$ segment and $\left(T^k\right)$ is the throughput during the download of the $k^{th}$ segment and $\left(\tau \right)$ is the segment duration. When $\left(T^k\right)$ drops, the algorithms must adapt $\left(R^k\right)$ quickly to minimize the risk of $\left(B^{k+1}\right)$ dropping to zero.

Figure 13, illustrates the average bitrates achieved by the algorithms for each video. The figure indicates that the average bitrates downloaded by the algorithms are largely consistent across all videos. However, the L2A-LL algorithm downloaded slightly lower bitrates during the streaming of Tears of Steel and Spark. Overall, the quality of the segment across all algorithms remains similar, with minor differences in these specific cases. Like in the previous experiment, Figure 14, shows that the LoL+ algorithm experienced the lowest number of video rate switches. Most of the fluctuations observed in the other algorithms involved small bitrate changes, which are unlikely to impact the user experience. Figure 15, shows that although the algorithms on average achieved similar video quality, the LoL+ algorithm outperformed the other algorithms by minimising rebuffering events. The LoL+ algorithm encountered only one rebuffering event during the streaming of the Big Buck Bunny video, while it avoided any rebuffering events when streaming the other videos. The rest of the algorithms experienced rebuffering while streaming all the videos. Compared to the low-latency L2A algorithm, the traditional ABR algorithms, Throughput and Dynamic, experienced both a higher frequency and a longer duration of rebuffering events. In network profile 1, where the bandwidth gradually increased and the risk of rebuffering was minimal, the throughput and dynamic algorithms achieved higher bitrates. However, in network profile 2, where there was a sudden drop in throughput, these algorithms struggled to quickly reduce bitrates to prevent rebuffering. In contrast, the LoL+ algorithm efficiently selected higher bitrates without compromising playback by proactively managing the risk of interruptions.

4.3. Impact of Segment Duration

In this section, we will analyse the performance of the algorithms as the segment duration changes. Video streaming services provide different segment durations. For instance, Adobe’s HTTP Dynamic Streaming (HDS) and Apple HTTP Live Streaming (HLS) provide segment durations of four and ten seconds, respectively. Therefore, it is crucial that the algorithms not only adapt the video bitrate efficiently across varying network conditions but also adjust the bitrate seamlessly as the segment durations change to maintain a smooth user experience. Referring to Equation 1, we observe that as the duration of the segment, $\tau$, increases, even a slight mismatch between the selected bitrate and the available throughput can quickly deplete the buffer. Additionally, as the segment duration increases, the throughput is averaged over a longer time. This gives clients fewer opportunities to adapt the video bitrate in response to sudden changes in network conditions. To assess the performance of the algorithms, we analyse their behaviour under both 2-second and 10-second segment durations. For each segment setting, we evaluate and compare their performance across network profiles 3 and 4.

• 2 Seconds Segments
  Here, we will analyse the performance of the algorithms under network profile 3. Figure 16, shows that the BOLA algorithm outperforms the other algorithms by downloading high quality segment while avoiding unnecessary video rate switches. Although the difference in video bitrates between the algorithms is small, the Dynamic algorithm tends to have a slightly lower average bitrate due to experiencing more frequent bitrate switches, where it momentarily drops to a lower bitrate before recovering.
  Next, we compare the performance of the algorithms under network profile 4 where the bandwidth drops from 4Mbps to 1Mbps. As evident in the Figure 17, the BOLA and LoL+ algorithms achieve higher bitrate while avoiding any playback interruptions. The LoL+ algorithm had only a single bitrate switch due to the sudden drop in the bandwidth, and it avoided any unnecessary bitrate switch. The Dynamic and Throughput algorithm are able to download high quality segments when there is no risk of playback interruption. However, when the bandwidth drops suddenly, the algorithms fail to adapt quickly, leading to rebuffering events. The BOLA and low-latency algorithms avoided playback interruptions in both experiments.
• 10 Seconds Segments
  Next, we increase the segment duration to observe how the algorithms adapt to changes in bandwidth. In the first experiment, we analyse the algorithms under network profile 3. Compared to 2 sec segment experiments, we can clearly observe that the algorithms select lower quality segments. This approach is quite intuitive since the larger segment duration in Eq. (1) increase the risk of playback interruption in the event of a mismatch between throughput and bitrate. The low-latency algorithms outperform the traditional algorithms as the segment duration increases. Figure 18, shows that the BOLA and Throughput algorithms experience frequent bitrate switches; however, these are minor fluctuations between neighbouring bitrates.
  In the next experiment, we analyse the performance of the algorithms under network profile 4. Figure 18, shows that, like the previous network profile, the low-latency algorithms outperform traditional algorithms in selecting higher video quality. The BOLA, Dynamic, and throughput algorithms conservative select algorithms. Figure 19, shows that only the Dynamic algorithm is able to avoid any playback interruption, however, at the expense of video quality. Low-latency algorithms achieve higher video quality; however, when throughput drops abruptly, they are unable to prevent the playback buffer from draining. Since the client can only switch bitrates at the start of a segment download, if the throughput drops during the segment download, the bitrate cannot be adjusted mid-segment. Therefore, when streaming longer segments, it is crucial for algorithms to ensure sufficient buffer to prevent rebuffering. However, this often comes at the expense of video quality, creating a trade-off that the becomes a challenge for algorithms.

5. Qualitative Study

This section provides details about the subjective evaluation of the ABR algorithms. This section includes the setup and evaluation procedure of the study.

5.1. Subjective Evaluation Method

The study was carried out to determine the user perception of video quality. The implementation of the study followed the recommendations in [22] and also ITU-T P.910. The Prolific crowd-source platform is used for the evaluation. In the study, 24 video sequences were presented. Of that 4 sequences were of high quality, 4 sequences shown were very low quality. and a total of 16 sequences were presented which are processed using various ABR algorithms. The test sequence has a duration of 1 minute (60 seconds). Each participant was given options to choose from. These were (Bad, Poor, Fair, Good, Excellent). They are required to select the quality of the video according to their perception. The responses are assigned an integer value from (1 to 5). The bad is numbered 1 and the excellent is numbered 5. When test participant watched a sequence, he will respond according to the scale provided and for each sequence 1 MOS score will be recorded.

5.2. Subjective Evaluation Results

Next, we will perform a subjective evaluation analysis for Network Profile 1. The objective of the subjective evaluation is to determine whether the results align with our quantitative analysis. Figure 20, presents the mean opinion scores of the algorithms for Network Profile 1 in various videos. In Network Profile 1, the algorithms initially select a lower quality followed by two incremental quality improvements. As a result, users tend to form an overall impression of the viewing experience based on initial quality. Figure 20, shows that the low-latency algorithms have a better MOS for the BBB and elephant videos. However, their quality drops when streaming Tears of Stell and Spark. The throughput algorithm displays low quality for the BBB and Elephant, whereas its quality improves for Tears of Stell and Spark. This aligns with the results in (Section 4.1) Figure 11, as the performance algorithm streams segments of the BBB and Elephant videos at lower bitrates. The dynamic algorithm delivers a consistent viewing experience maintaining a MOS score of around 3 across all four videos. Figure 21, shows that the dynamic algorithm The dynamic algorithm achieves the best overall viewing experience followed by the throughput algorithm. The low-latency algorithm demonstrates inconsistent performance across different videos, leading to a lower overall QoE compared to traditional algorithms.

6. Conclusions

In this research work, evaluation results of adaptive bitrate algorithms (ABR) are presented. The objective assessment is performed to measure the impact on the video quality. In the experiment, both short- and long-duration segments are considered. In this work, various bandwidth profiles are considered. The MPEG.js environment is used for evaluation. The ABR algorithms are evaluated under various network conditions. The HAS algorithm is compared to the low-latency algorithms. The performance was carried out using QoE metrics like bitrate, buffering events, and buffering duration. The ITU-T guidelines were followed in the objective evaluation. The results demonstrate that the LOL+ and dynamic algorithms performed well when the bandwidth is increased from 1MBIT to 4MBIT. The L2A-LL and throughput algorithms delayed increasing the bitrate, which minimises the chances of playback interruptions. The dynamic algorithm outperformed other algorithms by achieving a high video rate. The evaluation is carried out with longer segment duration (10 seconds). The results demonstrate that when streaming longer segments, it is crucial for algorithms to ensure sufficient buffering to prevent rebuffering. However, this often comes at the expense of video quality, creating a trade-off that becomes a challenge for algorithms. The results provide evidence that there are needs for the framework for video quality in a streaming environment. This research provides a foundation for the building of next-generation video streaming solutions.

In this article a qualitative study is carried out and preliminary results are presented. The purpose of the qualitative study is to collect the opinions of the users and estimate the perception of the users. The results reveal that low-latency algorithms have better mean opinion scores (MOS) for some set of videos. But the quality of the low-latency algorithms drops for one set of videos. These preliminary results align with our quantitative results of the experiment. According to the qualitative study, the dynamic algorithm shows promising viewing experience, maintaining a consistent MOS score across all presented video sequences. According to the results, the low-latency algorithms demonstrate inconsistent performance across all video streams. This leads low-latency algorithms to a lower overall QoE compared to traditional HAS algorithms. In future work, we will extend the experiment and perform further studies considering (a) additional QoE metrics, (b) incorporating more source videos, and (c) deploying CMAF for low-latency computation.