Diagnosis-Effective Sampling of Application Traces

1. Introduction

Distributed tracing (see [1,2,3]) has emerged as a crucial tool for effectively monitoring and troubleshooting native cloud applications (see [4,5,6,7]). It plays a pivotal role in understanding the behavior and performance of applications across distributed environments. Native cloud applications often adopt a microservices architecture, decomposing the application into smaller, loosely coupled services. Each service performs a specific function and communicates with others via Application Programming Interfaces (APIs). Distributed tracing enables inspecting the flow of requests across the microservices, providing insights into how requests are processed and identifying any bottlenecks or issues within the system (see [8]).

Cloud-native applications are designed to be highly scalable and dynamically allocate resources based on demand. Distributed tracing accommodates this dynamic nature by providing visibility into the performance of services as they scale up or down in response to changes in workload. This ensures that performance issues are detected and addressed in real-time, maintaining the application’s reliability and responsiveness. They often utilize containerization and container orchestration via Docker and Kubernetes, respectively. Those technologies enable applications to be deployed and managed efficiently in a distributed environment. Distributed tracing is integrated with container platforms for the tracking of requests as they traverse across containers and pods, providing insights into resource utilization and communication patterns.

Distributed tracing is typically part of a broader observability stack, including metrics and logging (see [2,9]). Integration with metrics allows for correlation between trace data and performance metrics, enabling a more profound analysis of application behavior. Similarly, integration with logging platforms provides context around specific events or errors captured in logs, enhancing the troubleshooting process (see [10,11,12,13,14,15,16,17,18]).

Several critical points are made regarding the complexities and challenges associated with distributed tracing. One is the volume of traces. However, many requests are routine and unremarkable, often called normal requests. Sampling in distributed tracing involves capturing a subset of traces for analysis instead of storing every trace to manage costs. This subset includes "interesting" traces encompassing various events in a distributed architecture. Interest can be linked to a trace latency when traces surpassing a latency threshold are selectively sampled. This allows for the identification of performance bottlenecks and areas needing improvement.

Interest can also be linked to errors. Erroneous traces or exceptions are sampled to investigate and address system reliability and stability issues. It is possible to prioritize requests or services to help determine which traces to sample. High-priority components, such as critical services or specific user interactions, are given preference for sampling. Companies typically adopt one of two strategies for capturing these events.

The common (see [2]) sampling strategy is head-based sampling, in which traces are randomly sampled based on a predefined rate or probability, such as $1\%-2\%$ of all traces. This approach relies on the principle that a sufficiently large dataset will capture the most interesting traces. However, it is worth noting that sampling meaningfully diminishes the value of distributed tracing. While sampling is necessary to manage costs, it can limit the effectiveness of distributed tracing for troubleshooting and debugging purposes. Developers may encounter situations where important traces are missed due to sampling, leading to a loss of trust in the tracing data. Consequently, developers may revert to traditional debugging methods, such as logs, undermining the value of distributed tracing (see [8]). The ideal scenario would involve analyzing the entire set of traces, identifying anomalies, and retaining them for thorough examination.

A more intelligent strategy is called tail-based sampling, when the system awaits completing all spans within a request before determining whether to retain the trace based on its entirety. What truly matters are around $5\%$ of traces that carry anomalies: errors, exceptions, instances of high latency, or other forms of soft errors.

In this paper, we vote for a multi-purpose and hierarchical sampling strategy focusing on the efficiency of application troubleshooting and root cause analysis. We are considering a multi-layered approach to efficiently capture and preserve different types of information within a system (see Figure 1). The first layer enables distributed tracing to track requests or transactions through the application we monitor. The second layer initiates head-based (rate-based) sampling for a general volume reduction. The goal is to obtain a manageable amount of data for further analysis in "Application Performance Monitoring" while retaining a representative sample of the overall system behavior.

The next layer, "Property-Based Sampling," retains information by incorporating sampling, which preserves useful properties like trace type, latency, errors, etc. Trace-type sampling involves categorizing traces based on their characteristics or origins within the system. Latency-based sampling, on the other hand, prioritizes the collection of traces associated with significant delays or performance bottlenecks. Error-based sampling identifies and retains traces associated with errors, exceptions, or other abnormal conditions within the application. Combining those approaches allows the system to store more diverse traces, ensuring that valuable information across different categories and performance metrics is captured. By isolating and storing those traces, developers and system administrators have the necessary data to diagnose and address issues effectively, improving overall system reliability and performance. The "Application Troubleshooting" layer monitors performance degradations by tracing traffic passing through different microservices. The visualization of the tracing traffic is known as the Application Map. It helps visualize malfunctioning services. The detection can be performed by increasing the percentage of erroneous traces or typical latencies. The issues will be analyzed by tracing traffic passing through that specific degraded microservice. The next layer performs noise reduction by rate-based sampling focusing on erroneous traces. It preserves all dominant normal and erroneous traces and removes the rare ones. This approach is acceptable when the problem has already been detected, and the goal is to explain it. The final layer applies rule learning systems to the tracing traffic, revealing recommendations connected to trace errors. The system administrators can inspect the recommendations to accelerate remediation.

Overall, this hierarchical sampling approach efficiently manages trace data by prioritizing information based on its relevance and importance for system analysis and troubleshooting. It balances the need to reduce data volume with the requirement to retain critical details for comprehensive understanding and problem resolution.

One of the main benefits of distributed tracing is improved application performance, resulting in reduced mean time to detect (MTTD) and mean time to repair (MTTR) IT issues. Distributed tracing enables penetration to the bottom of application performance issues faster, often before users notice anything wrong. Upon discovering an issue, tracing can rapidly identify the root cause and address it. It also provides early warning capabilities when microservices are in poor health and highlights performance bottlenecks anywhere. However, this task is only feasible for a few system administrators or site reliability engineers (SREs) due to the large scale of modern distributed environments. The solution is an AI for IT Operations (AIOps) strategy that leverages AI-powered software intelligence to automate developments, service deliveries, and troubleshooting. AIOps is a real game-changer for managing complex IT systems.

In AIOps, where machine learning algorithms analyze vast amounts of data to automate IT operations and decision-making processes, transparency and interpretability are paramount. Trust in AIOps hinges on implementing Explainable AI (XAI) solutions (see [19]). Those techniques enable stakeholders to understand and interpret the decisions made by AI models, build confidence in their reliability, and facilitate human-AI collaboration. By providing insights into how AI algorithms arrive at their conclusions, XAI fosters trust among users, reduces the risk of biased or erroneous outcomes, and enhances the adoption and effectiveness of AIOps solutions. We focus on rule-learning systems and show their efficiency in combination with trace sampling. Rule induction systems, such as RIPPER (Repeated Incremental Pruning to Produce Error Reduction) (see [20]) and C5.0 rules (see [21]), are common examples of XAI solutions.

Figure 2 shows the transformation of erroneous traces into insights based on rule-induction systems. First, we construct features based on traces and their spans. The volume of traces after the sampling will be adequate to carry out that generation procedure. For some systems, the number of features can be rather large. Second, rule-learning methods solve the corresponding classification problem and generate rules that can explain/predict the nature of the group of traces that arrive as erroneous. These learning systems generate understandable rules based on input data. Humans can easily comprehend those rules, which provide clear insights into the decision-making process of the AI model. Unlike black-box machine learning models such as deep neural networks, where it’s often challenging to understand how the model arrives at its predictions, rule induction systems offer transparency. Users can inspect the rules to understand the underlying logic and reasoning behind the rules. This enhances the trustworthiness of the AI system.

This analysis reveals the underlying causes of the errors and potential areas for improvement or mitigation. By interpreting the induced rules, developers and system administrators can gain insights regarding the possible remediation of issues. The rules provide transparent and interpretable explanations of certain errors occurring under specific conditions or scenarios. Armed with insights from the root cause analysis, appropriate measures can be taken to optimize the system, fix bugs, or implement preventive measures to reduce the likelihood of similar errors occurring in the future.

Understanding how error-based sampling in the final stage affects the rules and simplifies the recommendations is one of our goals. It preserves dominant errors while potentially removing rare events across different types. Error-based sampling focuses on data points with higher errors or uncertainties. Certain errors in many datasets may occur more frequently or significantly impact the overall analysis. Rare events in a dataset often contribute to its overall uncertainty, especially if they are outliers or anomalies. These events may skew statistical measures or model predictions, making it challenging to draw reliable conclusions. Error-based sampling may downplay the influence of rare events by sampling them less frequently or excluding them altogether. We show how this reduction in outliers helps to stabilize the analyses and make recommendations more robust to extreme cases.

The paper is organized as follows. Section 2 describes the main trends of trace sampling and rule-induction. Section 3 explains the methods of trace sampling based on properties when the goal is to preserve all available types, latencies, and errors uniformly. Section 4 presents the impact of noise reduction on a rule induction when the goal is to preserve only frequent erroneous traces across all types.

2. Related Work

Various companies and researchers are developing end-to-end distributed tracing technologies in which sampling is one of the core components as the prevailing approach to reducing tracing overheads. Instead of tracing every request, the sampling only captures and persists traces for a subset of requests to the system. To ensure the captured data is useful, sampling decisions are coherent per request – a trace is either sampled in its entirety, capturing the full end-to-end execution, or not at all. Sampling effectively reduces computational overheads; these overheads are only paid if a trace is sampled, so they can be easily reduced by reducing the sampling probability (see [22,23,24,25] with references therein).

Large technology companies such as Google, Microsoft, Amazon, and Facebook often develop methods to monitor and analyze the performance of their own systems and applications. Early tracing systems such as Google’s Dapper (see [26]) and later Facebook’s Canopy (see [27]) make sampling decisions immediately when a request enters the system. This approach, known as head-based sampling, avoids the runtime costs of generating trace data as it occurs uniformly at random, and the resulting data is simply a random subset of requests. In practice, sampling rates can be as low as $0.1\%$ (see [22]).

Tail-based sampling is an alternative to head-based sampling. It captures traces for all requests and only decides whether to keep a trace after it has been generated. While OpenTelemetry (see [28]) offers a tail-based sampling collector, its implementation presents challenges as storing all spans of a trace until the request concludes demands a sophisticated data architecture and solutions. One such end-to-end tracing to enhance distributed system dependability by dynamically verifying and diagnosing correctness and performance issues is proposed in paper [29]. The idea is based on clustering execution graphs to bias sampling towards diverse and representative traces, even when anomalies are rare.

Google has developed distributed tracing technologies, such as Google Cloud Trace and OpenCensus, which allow users to collect and analyze trace data from applications running on Google Cloud Platform and other environments. Each Google Cloud service makes its own sampling decisions. When sampling is supported, that service typically implements a default sample rate or a mechanism to use the parent’s sampling decision as a hint as to whether to sample the span or maximum sampling rate (see [30]). OpenCensus provides "Always," "Never," "Probabilistic," and "RateLimiting" samplers. The last one tries to sample with a rate per time window, which by default is 0.1 traces/second (see [31]).

Azure Application Insights sampling (see [32]) aims to reduce telemetry traffic, data, and storage costs while preserving a statistically correct analysis of application data. It enables three different types of sampling: adaptive sampling, fixed-rate sampling, and ingestion sampling. Adaptive filtering automatically adjusts the sampling to stay within the given rate limit. If the application generates low telemetry, like during debugging or low usage, it doesn’t drop items as long as the volume stays under the limits. The sampling rate is adjusted to hit the target volume as the telemetry volume rises. Fixed-rate sampling reduces the traffic sent from web servers and web browsers. Unlike adaptive sampling, it reduces telemetry at a fixed rate a user decides. Ingestion sampling operates where web servers, browsers, and devices telemetry reaches the Application Insights service endpoint. Although it doesn’t reduce the telemetry traffic sent from an application, it does reduce the amount processed and retained (and charged for) by Application Insights. Microsoft researchers introduced an observability-preserving trace sampling method, denoted as STEAM, based on Graph Neural Networks, which aims to retain as much information as possible in the sampled traces (see [33]).

In 2023, AWS announced the general availability of the tail sampling processor and the group-by-trace processor in the AWS Distro for OpenTelemetry collector (see [34]). Advanced sampling refers to the strategy where the Group By Trace processor and Tail Sampling processor operate together to make sampling decisions based on set policies regarding trace spans. The Group By Trace processor gathers all of the spans of a trace and waits a pre-defined time before moving them to the next processor. This component is usually used before the tail sampling processor to guarantee that all the spans belonging to the same trace are processed together. Then, the Tail Sampling processor samples traces based on user-defined policies.

There are also many other companies developing such technologies. We will mention a few of them. Jaeger is an open-source distributed tracing system developed by Uber Technologies. It is widely used for monitoring and troubleshooting microservices-based applications (see [35]). Zipkin is another open-source distributed tracing system originally developed by Twitter. It helps developers gather data for various components of their applications and troubleshoot latency issues (see [36]). LightStep is a commercial company that offers tracing solutions for monitoring and troubleshooting distributed systems (see [37]). Datadog is a monitoring and analytics platform that offers features for collecting and analyzing traces, logs, and metrics from applications and infrastructure (see [38]). Dynatrace is a software intelligence platform that provides features for monitoring and analyzing the performance of applications and infrastructure, including distributed tracing capabilities (see [39]). New Relic offers a monitoring and observability platform that includes features for collecting and analyzing traces, logs, and metrics from applications and infrastructure (see [40]). AppDynamics is an application performance monitoring solution that provides features for monitoring and troubleshooting applications, including distributed tracing capabilities (see [41]). Honeycomb is an observability platform that offers features for collecting and analyzing high-cardinality data, including traces, to debug and optimize applications (see [42]). Grafana is an open-source analytics and visualization platform that can visualize traces, logs, and metrics collected from various sources, including distributed tracing systems (see [43]). Elastic is the company behind Elasticsearch, Kibana, and other products in the Elastic Stack. Their solutions offer features for collecting, analyzing, and visualizing traces, logs, and metrics (see [44]).

System administrators can no longer perform real-time decision-making due to the growth of large-scale distributed cloud environments with complicated, invisible underlying processes. Those systems require more advanced and ML/AI-empowered intelligent RCA with explainable and actionable recommendations (see [45,46,47,48,49,50,51,52,53] with references therein).

XAI (see [19]) builds user and AI trust, increases solutions’ satisfaction, and leads to more actionable and robust prediction and root cause analysis models. Many users think it is risky to blindly trust and follow AI recommendations and predictions, and they need to understand the foundation of those insights. Many ML approaches, like decision trees and rule-induction systems, have sufficient explainability capabilities for RCA. They can detect and predict performance degradations and identify the most critical features (processes) potentially responsible for the malfunctioning.

In many applications, explainable outcomes can be more valuable than conclusions based on more powerful approaches that act like black boxes. Rule learners are the best if outcomes’ simplicity and human interpretability are superior to the predictive power The list of known rule learners consists of many exciting approaches. We refer to [54] for a more detailed description of available algorithms, their comparisons, and historical analysis. It contains relatively rich references and describes several applications (see also [20,21,55,56,57,58]). Rule learning algorithms have a long history in industrial applications (see [7,14,59,60,61,62,63,64,65,66]). Many such applications utilize classical classifiers like C5.0Rules (see [21]) and JRip. The last one is the Weka implementation of RIPPER.

The recommendations derived from rule-learning systems can be additionally verified regarding uncertainty based on Dempster-Shafer’s theory (DST) of evidence [67]. It is an inference framework with uncertainty modeling, where independent sources of knowledge (expert opinions) can be combined for reasoning or decision-making. In [68], it is leveraged to build a classification method that enables a user-defined rule validation mechanism. These rules are "encoding" expert hypotheses as conditions on the data set features that might be associated with certain classes while making predictions for observations. It provides a “what-if” analysis framework for comprehending the underlying application, as it is applied in [69]. This means that in the current study context, we can apply the same framework to estimate the effectiveness of the trace sampling approaches while comparing the rule validation results in the pre- and post-sampling stages. DST is overlooked in terms of learning algorithms that could be utilized in predictive system diagnostics (see the survey [45], which emphasizes that gap). Therefore, its application in our use case of RCA-effective trace sampling is an additional novelty we introduce.

3. Trace Sampling Based on Type, Duration, and Errors

Modern distributed applications trigger an enormous number of traces. Direct storage of hundreds of thousands of traces heavily impacts users’ budgets. A sampling strategy is a procedure that decides which traces to store for further utilization. We describe a procedure that samples traces based on their types, durations, and/or errors. The final goal is to preserve sufficient information for application troubleshooting without possible distortion.

A trace type can be approximately identified by the root span or the span that arrived at the earliest. More strict identification is based on the analysis of its structure. However, the latest is rather resourceful and will require additional grouping/clustering approaches. The durations of traces (in milliseconds) are available in the corresponding metadata. It is possible to extract the durations for different trace types and estimate the average/typical durations of the corresponding processes. The information on errors is also available in trace metadata. Traces are composed of spans. We have detailed information regarding the spans in the corresponding tags that describe the micro-processes. One of the tags indicates the health of a span. We can assume that the entire trace is erroneous if at least one of its spans has a "True" error label. We suggest a parametric approach for all sampling scenarios, allowing us to modify the required percentage of compression rates accordingly.

3.1. Sampling by a Trace Type

Assume that we know how to determine a trace type. Let M be the number of trace types for a specific period and $N_k,k=1,\cdots ,M$ be the number of traces in each type. This information should be collected/updated for a recent period as the number of trace types and the trace-flow velocity are subject to rapid changes in modern applications. Let N be the number of all traces before the sampling:

$$N=\sum _{k=1}^M{N}_k,$$

and $p_k\left(type\right)=N_k/N,k=1,\cdots ,M$ be the probability of the occurrence of the k-th type. Let $r_k\left(type\right)$ and $N_k^{*}$, $k=1,\cdots ,M$ be the sampling rate (the percentage/fraction of stored traces) and the number of traces after the sampling of the k-th type, respectively:

$$N_k^{*}=r_k\left(type\right)N_k,k=1,\cdots ,M,$$

where $N_k^{*}$ should be rounded to the closest integer. Also, we denote by $N^{*}$ the total number of all traces after the sampling

$$N^{*}=\sum _{k=1}^M{N}_k^{*}.$$

The ratio $r=N^{*}/N$ is the final sampling rate. We propose to compute the sampling rate $r_k$ as inversely proportional to the probability $p_k\left(type\right)$:

$$r_k\left(type\right)=1-{\left(p_k\left(type\right)\right)}^{\alpha },\alpha >0,$$

where $\alpha$ is a parameter a user should determine to satisfy the needed final sampling rate r. Generally, the more frequent a trace type, the lower the corresponding sampling rate $r_k$.

Let us see how the parameter $\alpha$ should be determined if the final user requirement is known:

$$N^{*}=\sum _{k=1}^M{N}_k^{*}=\sum _{k=1}^M{r}_k{N}_k=\sum _{k=1}^M\left(1-{\left(p_k\right)}^{\alpha }\right)N_k=N\sum _{k=1}^M{p}_k\left(1-{\left(p_k\right)}^{\alpha }\right)=N{G}_{type}\left(\alpha \right),$$

where

$$r=N^{*}/N=G_{type}\left(\alpha \right)=\sum _{k=1}^M{p}_k\left(type\right)\left(1-{\left(p_k\left(type\right)\right)}^{\alpha }\right).$$

It can be considered an analog of the Gini index showing the total sampling rate. For a specific dataset of traces, we can compute the values of $G\left(\alpha \right)$ across different $\alpha$ and determine its value to satisfy a user’s requirement. We can try to estimate the proper value of $\alpha$ based on the recent collection of traces (say for the recent 2 hours) and dynamically update those values to meet the requirement.

Let us illustrate how this procedure works for a specific application. Figure 3 shows the distribution of traces across different trace types.

There are types with almost 4000 traces and very rare ones, like with 3 representatives. Overall, we detected 20425 traces distributed across 28 different trace types. Figure 4 and Table 1 reveal how different rates compress traces across types. We experimented with $\alpha =1,0.5,0.25$ and $0.1$. The sampling rate is always smaller for common trace types and larger for rare ones (see Table 1). Smaller values of $\alpha$ correspond to smaller final sampling rates. When $\alpha =1$, we store 17903 traces across all types, $87.7\%$ of all traces. When $\alpha =0.5$, we store 13629 traces, $66.7\%$ of all traces. For $\alpha =0.25$, we store 8882 traces, $43.5\%$. When $\alpha =0.1$, we store 4273 traces, $20.9\%$ of all traces.

Let us inspect some of the sampling rates for specific types. The most frequent type contains 3996 traces with the probability $p=0.2$. The corresponding sampling rate (for $\alpha =1$) is $80.5\%$, and 3215 traces from this class were stored. One of the rarest types contains 3 traces with almost zero probability. The sampling rate is almost $100\%$, and we store two or three traces for different $\alpha$.

The correct value of $\alpha$ that will satisfy a user requirement could be estimated by computing the $G\left(\alpha \right)$ values. Figure 5 explains the procedure. Let us assume that the required sampling rate is $10\%$, meaning we want to store around that percentage of traces. $\alpha =0.044$ provides a sampling rate $r=0.099$.

This algorithm should be applied if trace types carry important information on an application, and both rare and frequent types should be preserved for further analysis. Mathematically, it means no more than, say, 50 trace types with diverse frequencies. Diversity can be inspected via the Gini index or entropy. If those measures are close to zero, the sampling will try to maximize them and store equal portions from all types.

3.2. Sampling by a Trace Duration

In this subsection, we consider the sampling of traces based on their durations. This setup is reasonable if we have a few trace types where traces have almost the same average durations. We would like to keep all traces with some average durations and those with extraordinary short or long ones.

Assume N traces with durations ${\left\{d_k\right\}}_{k=1}^N$. Let $D\left(m\right)$ be the corresponding histogram of durations with m number of bins:

$$D\left(m\right)=\{n_1,\cdots ,n_m\},$$

where $n_s$, $s=1,\cdots ,m-1$ be the number of traces with durations within intervals $[t_{s-1},t_s)$, $s=1,\cdots ,m-1$ and $n_m$ with durations within $[t_{m-1},t_m]$. Let $p_s\left(dur\right)$, $s=1,\cdots ,m$ be the probability that a trace has a duration from the s-th bin:

$$p_s\left(dur\right)=\frac{n_s}{N},s=1,\cdots ,m.$$

We determine the sampling rate $r_s\left(dur\right)$ of a trace with duration within the s-th bin to be inversely proportional to the corresponding probability $p_s\left(dur\right)$:

$$r_s\left(dur\right)=1-{\left(p_s\left(dur\right)\right)}^{\beta },s=1,\cdots ,m,\beta >0,$$

where $\beta$ is the parameter to meet a user’s requirement.

Let $n_s^{*}$ be the number of traces after the sampling in the s-th bin:

$$n_s^{*}=r_s\left(dur\right)n_s.$$

Let $N^{*}$ be the total number of traces after the sampling:

$$N^{*}=\sum _{s=1}^m{n}_s^{*}=\sum _{s=1}^m{r}_s\left(dur\right)n_s=N\sum _{s=1}^m\left(1-{\left(p_s\right)}^{\beta }\right)p_s=N{G}_{dur}\left(\beta \right),$$

where

$$r=N^{*}/N=G_{dur}\left(\beta \right)=\sum _{k=1}^M{p}_k\left(dur\right)\left(1-{\left(p_k\left(dur\right)\right)}^{\beta }\right)$$

is the total sampling rate for different values of parameter $\beta$.

Let us illustrate how this procedure works for the same dataset of traces presented earlier. We will skip the trace types and will look into the trace durations.

Figure 6 shows the distribution of traces with durations in different time intervals. It shows three dominant time intervals where the most traces are concentrated. We can aggressively sample those frequent traces and moderately sample the rare ones outside dense areas. Table 2 shows how the sampling procedure works for different $\beta$ parameter values (we use 7 bins).

The first column, "Time intervals" of Table 2, shows the seven intervals of durations. The intervals have equal lengths, although the bins could be nonuniform. The second column shows the initial number of traces with the durations within the corresponding time intervals. The remaining columns show the numbers after the sampling with the corresponding $\beta$. The last row of Table 2 reveals the final sampling rates. The bigger the value of $\beta$, the more severe the reduction of traces. The value $\beta =0.1$ provides a sampling rate of around $13\%$. Figure 7 visualizes the results of Table 2.

If the requirement is to sample exactly $10\%$ of traces, then Figure 8 shows the parameter $\beta$ estimation based on the values of $G_{dur}\left(\beta \right)$. The red cross indicates that $\beta =0.083$ provides the sampling rate $0.0996$.

Proper histogram construction is a crucial milestone for frequency-based sampling. In general, the problem is the impact of outliers on the bin construction process. Standard procedures use an equidistant split of the data range, and data outliers can enlarge it, resulting in big bins with resolution distortion. The more accurate procedure should involve outlier detection to preserve them in separate bins with further application of the classical procedure for the main part of the data. We consider outlier detection via a modified MAD (median absolute deviation) algorithm (see [70]) with a slight modification. We define upper and lower baselines as $0.9$ and $0.1$ quantiles of data:

$$M\left(up\right)=q_{0.9}\left(data\right),M\left(low\right)=q_{0.1}\left(data\right),$$

where $q_s\left(data\right)$ is the s-th quantile of data with $0\le s\le 1$. We calculate the upper and lower distances:

$$dist\left(up\right)=\left|data\right(up)-M(up\left)\right|,dist\left(low\right)=\left|data\right(low)-M(low\left)\right|,$$

where $data\left(up\right)$ are data points bigger than or equal $M\left(up\right)$ and $data\left(low\right)$ are data points smaller or equal than $M\left(low\right)$. We set upper and lower MADs as

$$MAD\left(up\right)=q_{0.8}\left(dist\left(up\right)\right),MAD\left(low\right)=q_{0.8}\left(dist\left(low\right)\right).$$

Based on the MADs, upper and lower thresholds are defined as:

$$upper=min\left(M\right(up)+2.5MAD(up),max(data\left)\right),$$

and

$$lower=max\left(M\right(low)-2.5MAD(low),min(data\left)\right).$$

All data points lying upper or lower than the corresponding thresholds are assumed to be upper and lower outliers, respectively. We cut outliers from the data and construct the classical histogram for the remaining data with some predefined number of bins (say bins = 5). Then, we calculate the number of lower and upper outliers and append them to the main histogram from left and right, respectively.

Hence, if data have small-value outliers, then the first bin of the histogram contains the number of data points within the interval $\left[data\right(min),upper)$. If data have big-value outliers, the last bin contains the number of data points within the interval $(upper,max(data\left)\right]$. The main part of the histogram relates to data points within the interval $[lower,upper])$. This procedure allows proper construction of the corresponding histograms with a small number of bins. It is worth noting that the main part of the histogram consists of uniform intervals, while the widths of the first and last bins could be different from those bins. Finally, if data have outliers from both sides, the corresponding histogram has $bins+2$ final bins. If data have outliers only from one side, then the final number of bins is $bins+1$. No other bins are added to the classical histogram if data come without outliers.

3.3. Hybrid Approach Based on Types and Durations

The hybrid approach takes into account both trace type and duration information. This algorithm performs accurate sampling if both durations and types are important. Assume that traces are already grouped into trace types. We generate the histogram of durations for each group and apply the procedure described in the previous subsection. We also consider the probability of a trace type. Common types will be sampled more aggressively.

Let

$$H^{\left(k\right)}=\{n_1^{\left(k\right)},\cdots ,n_m^{\left(k\right)}\},k=1,\cdots ,M,$$

be the histogram of durations of the k-th trace type and $n_s^{\left(k\right)}$ be the number of traces in the s-th bin of the k-th type. As above, M is the number of different types, and m is the number of bins in the histogram. Let $N_k$ be the number of traces in the k-th trace type:

$$N_k=\sum _{s=1}^m{n}_s^{\left(k\right)}.$$

Let N be the total number of traces:

$$N=\sum _{k=1}^M{N}_k.$$

Let

$$P^{\left(k\right)}=\{p_1^{\left(k\right)},\cdots ,p_m^{\left(k\right)}\},p_s^{\left(k\right)}=\frac{n_s^{\left(k\right)}}{N_k},k=1,\cdots ,M,$$

and

$$P=\{p_1,\cdots ,p_M\},p_k=\frac{N_k}{N}.$$

We denote the sampling rate of a trace from the s-th bin of the k-th trace type by $r_s^{\left(k\right)}$. It shows the fraction of traces that should be stored. We compute it by the following formula as inversely proportional to the corresponding probabilities:

$$r_s^{\left(k\right)}=1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta },$$

where $\alpha ,\beta \ge 0$ are some unknown parameters to be tuned to meet the requirement for the final sampling rate r.

Let us show how it could be done. Let $N_k^{*}$ be the number of traces in the k-th trace type after the sampling. Let ${n_s^{\left(k\right)}}^{*}$ be the number of traces in the s-th bin and the k-th trace type after the sampling:

$$N_k^{*}=\sum _{s=1}^m{n_s^{\left(k\right)}}^{*}=\sum _{s=1}^m{n}_s^{\left(k\right)}{r}_s^{\left(k\right)}=N_k\sum _{s=1}^m{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right).$$

Let $N^{*}$ be the total number of traces after the sampling across all types and durations:

$$N^{*}=\sum _{k=1}^M{N}_k^{*}=\sum _{k=1}^M{N}_k\sum _{s=1}^m{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right)=NG(\alpha ,\beta ),$$

where

$$r=\frac{N^{*}}{N}=G(\alpha ,\beta )=\sum _{k=1}^M\sum _{s=1}^m{p}_k{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right)$$

is the required final sampling rate that can be accomplished by appropriately selecting parameters $\alpha$ and $\beta$.

First, we will illustrate this procedure when $\alpha =\beta$. Figure 9 reveals the procedure for a specific trace type containing 3996 traces. The left figure shows the scatter plot of durations. The right figure shows the histogram of durations with 7 bins before and after the samplings. The value $\beta =0.1$ corresponds to the sampling that stores 1104 traces. The next value $\beta =0.05$ stores 599 traces. The value $\beta =0.025$ corresponds to the sampling with 315 preserved traces. Finally, the value $\beta =0.01$ stores 131 traces. The last one corresponds to the $3.3\%$ sampling rate. By the way, the probability of this trace type is $0.2$, which also impacts the sampling rates with $\alpha =\beta$ setting.

Figure 9. The hybrid approach for a specific trace type (N2 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces in different bins before and after the samplings corresponding to different values of $\alpha =\beta$.

Figure 9. The hybrid approach for a specific trace type (N2 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces in different bins before and after the samplings corresponding to different values of $\alpha =\beta$.

Figure 10. The hybrid approach for a specific trace type (N17 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces for $\alpha =\beta$.

Figure 10. The hybrid approach for a specific trace type (N17 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces for $\alpha =\beta$.

Figure 11. The hybrid sampling approach for $\alpha =\beta$.

Figure 11. The hybrid sampling approach for $\alpha =\beta$.

Figure 10 shows the hybrid procedure for another trace type with the probability $0.0048$. It contains only 98 traces and is a rare type compared to the previous example. It also corresponds to the setting $\alpha =\beta$. The right figure shows the sampling results for different values of $\beta$. For example, the value $\beta =0.01$ corresponds to the sampling that stores only 7 traces of this specific type. It corresponds to the $7.1\%$ sampling rate. The value $\beta =0.025$ corresponds to the $18.4\%$ sampling rate. The value $\beta =0.05$ stores 31 traces and corresponds to the $31.6\%$ sampling rate. Finally, the value $\beta =0.1$ corresponds to the sampling rate $52\%$.

Figure 11 shows the result of the hybrid approach across different trace types when $\alpha =\beta$. The total sampling rate is $4\%$ for $\alpha =\beta =0.01$. The values $\alpha =\beta =0.1$ correspond to the sampling rate $31\%$. It means that the values of $\alpha$ and $\beta$ can control the sampling rates in a wide range of values. If we need to accomplish a strict requirement, we can turn to the $G(\alpha ,\beta )$ values corresponding to the sampling rates. Figure 12 shows the values of $G(\beta ,\beta )$, where the red cross corresponds to the sampling rate of $10\%$ (the exact value of G is $0.099$) with $\beta =0.029$. We can also tune the values of $\alpha$ and $\beta$ independently. Figure 13 shows the surface of sampling rates corresponding to different values. For example, the total sampling rate of $10\%$ can be accomplished by $\alpha =0.03,\beta =0.022$, or $\alpha =0.026,\beta =0.029$, or $\alpha =0.011,\beta =0.056$, or $\alpha =0.023,\beta =0.035$, or $\alpha =0.01,\beta =0.058$, etc.

Parameter optimization can be performed without the utilization of long historical data. We can take an initially random value for the parameters and, after each hour, verify the actual compression ratio. Then, by increasing or decreasing the values, we achieve the required sampling. This will work especially for dynamic applications when relying on available historical information is impossible.

How, in practice, can we sample a trace? When a specified trace (with known type and duration) is detected with a known sampling rate r, a random variable from $Bernoulli\left(r\right)$ distribution should be generated. This random variable has a boolean outcome with the value 1 that has probability r and value 0 with the probability $1-r$. If the outcome is 1, we store the specified trace; otherwise, we ignore it. This will allow us to store the traces with the required sampling rates in the long run.

Returning to the problem of histogram construction, we refer to a powerful approach known as the t-digest algorithm (see[71]), which addresses several problems in histogram construction. The first concern corresponds to storing time series data with trace durations. For each trace type, we need to store the corresponding time series of durations for further histogram construction, and we need those time series with sufficient statistics. For 28 trace types, as in our experiments, we need to store the 28 time series. The second concern corresponds to the process of histogram construction. Each time when we need those histograms, sorting should be applied to the corresponding data of durations, and the procedure must be repeated each time in case of some new arrivals. The concern is the impact of outliers. An efficient solution to the mentioned problems is the t-digest algorithm. Instead of storing the entire time series data, it stores only the result of data cluster centroids and data counts in each cluster. The efficient merging approach allows for combining different t-digest histograms, making the entire process streaming. The t-digest is very precise while estimating extreme data quantiles (close to 0 and 1), making the procedure robust to outliers.

3.4. The Sampling of Erroneous Traces

In specific frameworks, sampling should preserve other important properties besides durations and types. One such property is the normality/abnormality of a trace, which characterizes microservices performance and could be used for troubleshooting and root-cause analysis. In the described approach, we can’t guarantee a sufficient number of important erroneous traces in the sampled dataset, as there was no requirement to preserve them. Now, we will try to address that requirement by discussing various approaches.

The first approach is the natural modification of the previous one, assuring the existence of enough erroneous traces in each trace type after the sampling. More precisely, we verify the existence of erroneous traces in each trace type. If they exist, we divide the corresponding type into two types containing only normal or erroneous traces. Then, we can apply the approach that has already been considered (sampling by type, duration, or hybrid) to the same dataset but with renewed trace types.

Dividing erroneous traces into additional types by the corresponding error codes is also possible. Assume that trace type "A" has erroneous traces containing error codes "400-Bad Request" and "401-unauthorized". Then, we can divide it into three new trace types "A-we" (without errors), "A-400" (with 400 code), and "A-401" (with 401 code). It is also possible to divide erroneous traces into types by their error codes independently of the types. Say, collect all traces with "400" error codes together in a single type, independently of their original types. Let us show how this approach works.

Assume two trace types: 300 traces of type "A" and 130 of type "B," and we must store only $30\%$ of the traces. The sampling, based only on the trace types, provides the value $\alpha =0.619$. It means storing 60 traces of type "A" and 69 of type "B". Figure 14 illustrate those numbers.

Now, let us also consider erroneous traces. Assume type "A" has 200 normal and 100 erroneous traces. Thus, types "A" and "A*" contain 200 and 100 traces, respectively. Then, assume type "B" has 60 normal and 70 erroneous traces. Thus, "B" and "B*" contain 60 and 70 traces, respectively. Now, instead of two types, there are four ones. The type-based sampling provides the value $\alpha =0.283$, which leads to sampling rates $0.2$, $0.43$, $0.34$, and $0.4$, respectively.

Figure 15 shows the distributions before and after the samplings. As a result, from 260 normal and 170 erroneous traces, the approach sampled 66 and 63, respectively (see the right figure of Figure 15). Now, we can guarantee that the final sampled set also contains erroneous traces across different types. It is possible to apply the hybrid approach that will also consider the duration of the erroneous traces.

The second approach tries to control better the percentage of erroneous traces in the sampled dataset. Let $0<h<1$ be the final required sampling rate. Assume that $h_e$ and $h_n$ are the sampling rates of erroneous and normal traces, respectively:

$$N_n^{*}=h_n{N}_n,$$

and

$$N_e^{*}=h_e{N}_e,$$

where $N_n$ and $N_e$ are the number of normal and erroneous traces before the sampling, respectively, and $N_n^{*}$ and $N_e*$ after the sampling.

Can we also put some requirements on $h_e$? We have

$$h_n\frac{N_n^{*}}{N_n}+h_e\frac{N_e^{*}}{N_e}=h,$$

and

$$h_n=\frac{N}{N_n}\left(h-h_e\frac{N_e}{N}\right).$$

If the requirements on h and $h_e$ lead to the value $0<h_n<1$, then the requirements can be accomplished. Otherwise, if $h_n$ turns out to be a negative number, it is impossible, and we must ask to change the requirements.

Let us return to the previous example with two trace types. Can we sample $30\%$ of traces while preserving $60\%$ of erroneous ones? We put $h=0.3$ and $h_e=0.6$ and try to find appropriate values for $h_n$. Our calculations show that $h_n=0.104$ will work. We will preserve the $60\%$ of erroneous traces and $10\%$ of normal traces by sampling $30\%$ of all traces. Figure 16 illustrates the choices.

4. Troubleshooting of Applications

The ultimate goal of distributed tracing is to monitor application performance, detect malfunctioning microservices, and explain the root causes of problems so that they can be resolved quickly. This is feasible by inspecting the tracing traffic passing through specific microservices, detecting erroneous traces/spans, and trying to explain their origin. Thus, the explainability of ML models is the most crucial property. We consider two highly explainable approaches.

One is RIPPER (see [20]), which is state-of-the-art in inductive rule learning (see [55]). It has important technical characteristics, like supporting missing values, numerical and categorical variables, and multiple classes. We experimented with Weka’s RIPPER implementation, or JRip (see [72]). Application of RIPPER for the inspection of erroneous traces is discussed in [7]. We are not going into such details but show the impact of noise reduction on the rules.

Next is the Dempster-Shafer classifier considered in [68]. It is a far more complex approach but has interesting implications. One important characteristic is the ability to measure the uncertainty of specific rules. It should be interesting to compare the uncertainties of the same rules before and after the samplings.

Tracing traffic passing through a malfunctioning microservice can still contain many traces, even after a series of samplings. In this final stage, we apply the sampling of erroneous traces for reducing the volume and simplifying the application of rule induction methods. We aim to understand how the sampling impacts the rule-generation process regarding precision, recall, or uncertainty. It is worth noting that in this stage, the goal of the sampling is not to preserve all possible types and durations of erroneous traces. Exactly the opposite: the goal should be to remove rare erroneous traces and preserve dominant/common ones, as they probably explain the problems of a microservice. This procedure is very straightforward. We separate normal and erroneous traces and additionally separate them by trace types. Then, in all those groups, we sample at the same rate. As a result, some of the groups (rare ones) can vanish. Trace errors or durations can label the resulting dataset (see [7]). In case of errors, "output=0" corresponds to a normal trace and "output=1" to the erroneous one. We apply RIPPER to explain the origin of erroneous traces.

Let us apply this procedure to a real customer cloud environment. The visualization of a tracing traffic is known as an application map. It shows which microservices malfunction and which traces must be collected for troubleshooting. We collected 5899 traces with some 4917 features that should explain the origin of errors. We have 3145 normal and 2754 erroneous traces. JRip (RIPPER) runs for $11.09$ seconds outputting 13 rules (see Figure 17).

The first 12 rules describe the erroneous traces. they are mostly connected with the type. Only one of the rules is connected with the tag "-annotations-matched". The fraction at the end of the rule shows how many traces were fired by the rule (numerator) and how many misfired (denominator). The first two rules have large coverage and are $100\%$ precise. The others should be connected with the noise in the dataset. We hope that the sampling will remove the noise. However, the accuracy of this classifier is $99.4\%$. The precision and recall of both classes are also bigger than $99\%$.

Figure 18 shows the distribution of normal traces across different types before (the top figure) and after (the bottom figure) the sampling. There are many types containing just a single representative. The sampling preserves only the common groups and removes all rare types. Here, we applied $10\%$ sampling rate. Similarly, Figure 19 shows the distribution of erroneous traces. In both figures, the labels on the horizontal axes show the names of trace types. We identified the trace type by the root span.

Figure 20 reveals the JRip rules after the sampling, which preserved the first two important rules and removed the others connected with the noise. The sampled dataset contains 300 normal and 268 erroneous traces. The classifier applied to the sampling dataset shows $99.1\%$ accuracy. It misclassified 5 traces. The execution time is $0.1$ seconds. As a result, the sampling reduces the execution time, especially for noisy datasets. Moreover, it shortens the list of top recommendations so that users can focus on the most important ones.

As we mentioned before, the sampling is degrading the statistical evidence. The classifiers cannot describe it. Quite the opposite, the accuracy of classifiers increases or remains almost the same. We can refer to the Dempster-Shafer theory and compare the uncertainties of the rules. The theory of belief functions, also referred to as evidence theory or Dempster–Shafer theory is a general framework for reasoning with uncertainty. It offers an alternative to traditional probabilistic theory for the mathematical representation of uncertainty. Figure 21 shows how the sampling affects the uncertainties. It shows the uncertainties of the first two rules before and after the sampling.

5. Conclusions

Distributed tracing is essential for gaining visibility into the complex interactions and dependencies within native cloud applications. By tracing the requests across microservices, containers, and dynamic environments, users can effectively monitor, manage, and troubleshoot their cloud-native applications to ensure optimal performance and reliability. Despite the efforts, adopting this technology encounters several challenges, and one of the most crucial is the volume of data and the corresponding resource consumption needed to handle it.

Sampling is a technique that alleviates the overhead of collecting, storing, and processing vast amounts of trace data. It reduces the volume of data by selectively capturing only a fraction of traces. This approach offers benefits such as decreased latency, reduced resource consumption, and improved scalability. However, sampling introduces challenges, particularly in maintaining representative samples and preserving the accuracy of analysis results. Striking a balance between sampling rate and data fidelity is crucial to ensure effective troubleshooting and performance analysis in distributed systems. Overall, distributed tracing sampling plays a vital role in managing the complexity of distributed environments while optimizing resource utilization and maintaining analytical efficacy.

We explored several approaches for sampling that preserved traces with some specific properties. One such property was the type of trace that described the transaction similarity. The traces of the same type normally should have almost the same structure. The goal of such sampling was to preserve traces across all available types. Another property was the duration of the trace. This approach could or might not be combined with the information regarding the type. Trace durations were important as they described the transaction duration. Similar transactions had some typical/ average durations. Atypical durations indicated a transaction/microservice malfunction. The next important trace characteristic was its normality. Erroneous traces carried important information regarding the problems. Hence, it was natural to try to keep all representatives while monitoring the performance of microservices. The flow of erroneous traces would show which microservices had degraded performance. Further, those traces were critical sources of information for troubleshooting application issues.

We sampled only dominant/common errors at the troubleshooting stage, trying to remove the rare ones. This removed the noise and made explanations more confident. The root cause analysis could be performed by tracing traffic passing through a malfunctioning microservice. Rule learning ML methods could help generate explicit rules that explain the problems and clarify the remediation process. We showed how rule-induction systems like RIPPER solved this problem and provided recommendations system administrators could follow to accelerate the resolution process. We also showed that the sampling could dramatically decrease the time of the program execution and provide more clear recommendations. However, as mentioned before, the sampling degraded the statistics, and sometimes rare but important evidence could escape the analysis.

6. Patents

Poghosyan A., Harutyunyan A., Grigoryan N., Pang C., Oganesyan G., Baghdasaryan D., Automated methods and systems that facilitate root-cause analysis of distributed-application operational problems and failures by generating noise-subtracted call-trace-classification rules. Filed: Oct 1, 2021. Application No.: US 17/492,099. Patent No.: US 11880272 B2. Granted: Jan 23, 2024.

Poghosyan A., Harutyunyan A., Grigoryan N., Pang C., Oganesyan G., Baghdasaryan D., Automated methods and systems that facilitate root cause analysis of distributed-application operational problems and failures. Filed: Oct 1, 2021. Application No.: US 17/491,967. Patent No.: US 11880271 B2. Granted: Jan 23, 2024.

Poghosyan, A., Harutyunyan, A., Grigoryan, N., Pang, C., Oganesyan, G., and Avagyan, K., Methods and systems for intelligent sampling of application traces. Application filed by VMware LLC in 2021. Application No.: US 17/367,490. Patent No.: US 11940895 B2. Granted: Mar 26, 2024.

Poghosyan, A.V., Harutyunyan, A.N., Grigoryan, N.M., Pang, C., Oganesyan, G., and Avagyan, K., Methods and systems for intelligent sampling of normal and erroneous application traces. Application filed by VMware LLC in 2021. Application No.: US 17/374,682. Publication of US 20220291982 A1 in 2022.

Poghosyan, A.V., Harutyunyan, A.N., Grigoryan, N.M., N. M., Pang, C., Oganesyan, G., and Baghdasaryan, D., Automated methods and systems that facilitate root cause analysis of distributed-application operational problems and failures. Application filed by VMware LLC in 2021. Application No.: US 17/491,967 and 17/492,099.

Grigoryan N.M., Poghosyan A., Harutyunyan A.N., Pang C., Nag D.A., Methods and Systems that Identify Dimensions Related to Anomalies in System Components of Distributed Computer Systems using Clustered Traces, Metrics, and Component-Associated Attribute Values. Filed: Dec 12, 2020. Application No.: US 17/119,462. Patent No.: US 11,416,364 B2. Granted: Aug 16, 2022.