Hybrid Taint-Guided Kernel Fuzzing with Selective State Propagation

1. Introduction

Operating-system kernels remain high-risk targets because a single defect can compromise all software layers that depend on them. Despite long-term hardening efforts, recent studies continue to report a growing number of kernel vulnerabilities, particularly in subsystems such as filesystems, networking stacks, and device drivers [1,2]. Coverage-guided greybox fuzzing has become a primary technique for uncovering these flaws at scale, with systems derived from AFL and syzkaller contributing to a steady stream of vulnerability disclosures [3]. However, the kernel’s privileged execution context, extensive interface surface, and complex internal state make systematic testing difficult. In many cases, security-critical code is protected not by shallow control-flow checks but by deep state conditions that are hard to satisfy through coverage feedback alone [4].

To improve exploration of state-dependent behavior, recent work has shifted toward kernel-aware fuzzing techniques that model interfaces, state transitions, and environmental constraints more explicitly. Relation-learning approaches construct dependency graphs over system-call sequences to guide fuzzers toward deeper kernel states [5]. Other systems extract or approximate kernel state variables using static analysis or symbolic reasoning, allowing state changes to serve as feedback signals instead of raw code coverage [6]. Additional efforts focus on kernel-specific execution environments and crash handling, for example by combining fuzzing with symbolic execution or refining crash deduplication using stack-trace clustering [7,8]. Related learning-based studies in security analysis further suggest that data-driven models can capture latent structural patterns and guide exploration even when explicit specifications are incomplete or unavailable [9]. Nevertheless, surveys consistently show that most kernel fuzzers still rely primarily on coverage and struggle to reach state-dependent paths where many vulnerabilities reside [10].

Hybrid fuzzing, which combines coverage-guided mutation with symbolic or concolic execution, has been proposed to overcome such limitations. General-purpose hybrid fuzzers use symbolic execution selectively to bypass hard-to-reach branches and then feed generated inputs back into the fuzzing loop [11,12]. While effective in some contexts, their benefits vary widely across workloads, and symbolic execution often becomes prohibitively expensive when applied to large, stateful programs such as kernels [13]. Studies of hybrid systems further note that their success depends strongly on invocation frequency, path prioritization, and careful coordination between symbolic and mutation-based components [14]. Kernel-oriented hybrids demonstrate similar trade-offs: they can reveal new bugs, but only under strict limits on symbolic exploration and with careful management of kernel execution environments [15].

Taint analysis provides another mechanism for directing exploration toward security-relevant behavior. Static taint analysis has been applied to track how external inputs propagate through kernel data structures, highlighting potential attack surfaces [16]. Dynamic and hybrid taint-guided approaches show that taint information can reduce time spent on irrelevant paths and help distinguish exploitable faults from benign ones [17]. Outside the kernel domain, taint-based guidance has been used to approximate path constraints and reduce reliance on full symbolic reasoning [18]. Recent system-level fuzzers also incorporate taint signals to steer exploration across complex hardware–software interactions [19]. However, in kernel fuzzing, taint analysis is often applied only during triage or exploitability assessment and rarely integrated into the main exploration loop at scale. Several challenges therefore remain unresolved. Coverage-centric feedback is insufficient for routines guarded by complex state variables. Hybrid fuzzers face scalability limits due to symbolic overhead and path explosion, especially on long kernel execution paths. Taint analysis introduces its own practical difficulties: static methods tend to over-approximate in pointer-rich kernel code, while dynamic tracking incurs nontrivial runtime overhead and complicates deployment across kernel versions. As a result, taint-guided fuzzing is typically confined to narrow scenarios and does not systematically guide overall path exploration.

This work investigates how selectively filtered taint information can be integrated into hybrid kernel fuzzing to improve exploration efficiency. We combine subsystem-aware static taint analysis with a dynamic fuzzing loop and introduce a pruning mechanism that removes taint flows unlikely to influence attacker-controlled inputs or key control decisions. Symbolic execution is then applied only to execution segments associated with these pruned taint regions, while the remaining exploration relies on coverage-guided mutation. This design treats symbolic execution as a targeted aid rather than a general search strategy, reducing overhead and limiting path explosion. We evaluate the proposed approach on 18 kernel subsystems and assess its impact on path coverage, bug discovery, and symbolic cost relative to a coverage-only baseline. The results show that selective taint guidance enables the fuzzer to reach more security-relevant paths and uncover memory-safety vulnerabilities while maintaining practical performance, demonstrating a balanced and scalable approach to kernel fuzzing.

2. Materials and Methods

2.1. Sample Scope and Kernel Targets

This study evaluated the taint-guided hybrid fuzzer on 18 Linux kernel subsystems chosen to represent different interface types and state behaviors. The targets included filesystems, memory-management modules, network stacks, device drivers, and IPC components. All subsystems were built from the same kernel version with identical configuration flags to keep testing conditions consistent. For each subsystem, we identified system-call entry points and reachable internal routines that could be affected by external inputs. Test inputs were produced in a controlled environment that varied system-call sequences, argument ranges, and scheduling conditions. Each fuzzing run lasted 48 hours to allow time to reach deeper kernel states while keeping results comparable across subsystems.

2.2. Experimental Design and Control Setup

Three configurations were evaluated: a coverage-guided baseline, a taint-only variant, and the full hybrid fuzzer with selective state propagation. All configurations used the same mutation engine, execution harness, and system-call scheduler so that differences in results could be attributed to taint analysis and hybrid components rather than unrelated factors. In the baseline configuration, no taint information was used and exploration depended only on coverage growth. In the taint-only variant, static taint summaries were used for seed selection, but symbolic execution was disabled. The full hybrid configuration applied symbolic constraint solving only to branches linked to pruned taint regions. All experiments were run under the same hardware and system load, and each configuration was tested three times to reduce random variation.

2.3. Measurement Procedures and Quality Control

We recorded path coverage, taint-relevant path counts, symbolic execution time, and detected faults during each run. Kernel execution traces were collected with a lightweight instrumentation layer that logged branch decisions, state variables connected to taint sources, and propagation edges. To ensure consistent taint tracking, we compared static taint results with hand-annotated samples from four subsystems and confirmed that major dependency paths were correctly captured. All crashes were processed through a deduplication pipeline that grouped faults by stack-trace similarity and reproduced them under controlled replay. Crashes that could not be reproduced twice were removed from analysis. Quality checks also verified that symbolic solvers received only short path fragments rather than complete execution traces, confirming that selective propagation was applied as intended.

2.4. Data Processing and Model Formulation

Coverage values, execution counts, and taint-propagation metrics were aggregated in 30-minute windows to examine progress over time. Symbolic overhead was calculated as the fraction of runtime spent on constraint solving. To study the relationship between taint-guided exploration and bug detection, we fitted a linear model linking taint-relevant coverage${\mathrm{C}}_{\mathrm{t}}$to the number of unique faults B:

$\mathrm{B}=\mathsf{\alpha }+\mathsf{\beta }{\mathrm{C}}_{\mathrm{t}}+\mathsf{\epsilon },$where $\mathsf{\alpha }$ and$\mathsf{\beta }$ were estimated for each subsystem, and $\mathsf{\epsilon }$ represented unexplained variation. We also measured improvement from hybrid execution using the normalized score$\mathrm{I}$:

$\mathrm{I}={\displaystyle \frac{{\mathrm{P}}_{\mathrm{h}}-{\mathrm{P}}_{\mathrm{b}}}{{\mathrm{P}}_{\mathrm{b}}}},$where${\mathrm{P}}_{\mathrm{h}}$ is the number of taint-relevant paths reached by the hybrid fuzzer and ${\mathrm{P}}_{\mathrm{b}}$ is the corresponding number from the baseline. All statistical analyses were performed using Python tools with fixed random seeds to ensure reproducibility.

2.5. Implementation Details and Execution Environment

All experiments were run on a dedicated machine with a 32-core CPU, 128 GB RAM, and hardware virtualization support. The kernel executed inside a lightweight virtualized environment that allowed quick snapshot restoration after each test input, ensuring that every run started from the same initial state. The hybrid symbolic component used a constraint solver integrated at the basic-block level, which enabled selective solving without reconstructing full paths. Static taint analysis relied on a kernel-aware control-flow graph builder that included pointer aliasing rules adapted to kernel memory layouts. The fuzzing harness isolated subsystems to avoid cross-effects during exploration. All seeds, taint summaries, and crash logs were stored in version-controlled repositories for traceability.

3. Results and Discussion

3.1. Coverage Growth and Taint-Relevant Exploration

Across all 18 kernel subsystems, the hybrid fuzzer increased both total branch coverage and the number of taint-relevant executions. Over the 24-hour runs, median edge coverage rose modestly, but the main effect appeared in taint-guided exploration: the hybrid configuration reached about 44% more taint-related paths than the baseline. Several networking and storage subsystems showed larger gains because their internal states depend on multi-step system-call sequences [20].

A similar trend has been reported in studies that refine input scheduling to avoid early stagnation in greybox fuzzing. The curve in our Fig.1 displays a related pattern: the baseline fuzzer stops discovering new taint-relevant edges early, while the hybrid version continues to expand coverage. This indicates that filtering taint information and applying constraints only at selected points helps the fuzzer use its time more effectively than a purely coverage-driven approach [21].

Figure 1. Edge coverage and taint-related path counts of the baseline and hybrid fuzzers over time.

Figure 1. Edge coverage and taint-related path counts of the baseline and hybrid fuzzers over time.

3.2. Bug Discovery Outcomes and Characteristics

The hybrid fuzzer found 13 unique kernel bugs, while the baseline found 7 under identical conditions. The additional bugs mainly came from subsystems where nested pointer structures and multi-step state transitions make unsafe memory operations hard to reach with mutation alone. These included buffer overflows, out-of-bounds reads, and pointer misuse.This observation aligns with prior taint-aware studies. Our Fig.2 shows a similar relationship: subsystems with more taint-relevant coverage also produced more distinct crashes. Unlike methods designed for a single bug class, our approach uses general taint-based signals, yet still detects a wide range of memory faults, suggesting that taint-guided path selection is broadly useful in kernel settings.

Figure 2. Solver time and unique crash cases observed across kernel subsystems under the hybrid setup.

Figure 2. Solver time and unique crash cases observed across kernel subsystems under the hybrid setup.

3.3. Cost of Selective Symbolic Execution and Solver Overhead

One goal of the design was to constrain solver usage so that symbolic execution would not dominate runtime. In the experiments, symbolic solving accounted for 11–18% of total runtime, depending on subsystem complexity. This level is lower than many hybrid fuzzers that run symbolic execution on full paths and often encounter long delays.

Prior evaluations of taint-driven systems also noted that heavy online analysis can slow execution. For instance, GREYONE and related tools report added runtime cost from extensive data-flow tracking. In our case, the selective propagation mechanism limited solver calls to a narrow set of taint-linked branches. As shown in Fig.2, solver time increases at a slower rate than the number of taint-relevant paths, indicating that pruning rules were effective in keeping constraint solving small relative to overall fuzzing effort.

3.4. Comparison with Existing Taint-Guided and Hybrid Fuzzers

Compared with coverage-only fuzzers such as syzkaller-based setups, the hybrid method reduces the gap between code coverage and the reachability of deep state-dependent logic. Prior reports, including UAF-focused fuzzers and directed fuzzers, often rely on vulnerability-specific indicators or target-driven heuristics. These methods work well for selected bug types but may not generalize across kernel modules.

Our approach differs by using static taint analysis to identify fields influenced by untrusted input and applying symbolic reasoning only along these paths. This avoids the heavy state tracking found in full symbolic executors and keeps performance closer to greybox fuzzers. At the same time, the method inherits known limits of static taint analysis, such as incomplete alias handling. Broader evaluations—similar to large-scale tests on UNIFUZZ or LAVA-M—will be useful to assess how the method behaves across more kernels and longer campaigns.

4. Conclusion

This study shows that static taint analysis combined with selective symbolic execution can guide fuzzing toward kernel paths that carry higher security relevance. By reducing unnecessary taint flows and applying constraints only where input-dependent behavior is present, the hybrid fuzzer increases taint-related coverage and uncovers more memory-safety issues than the coverage-only baseline. The added cost of symbolic execution remains controlled, which makes the approach suitable for longer campaigns and for subsystems where deep state interactions are common. These findings suggest that taint-based path selection offers a practical middle ground between fast mutation-driven fuzzing and more precise but expensive symbolic analysis. The method may be useful for large kernel codebases and for testing environments where stateful behaviors limit the reach of greybox fuzzers. However, the approach still depends on the accuracy of static taint analysis and has been tested on a limited set of kernel modules. Further work should improve taint precision, broaden subsystem coverage, and study how the method behaves under larger and more diverse workloads.