Preprint
Article

This version is not peer-reviewed.

ABAF-VP: Validation Planning for Android Bug Mechanisms

Submitted:

26 October 2026

Posted:

27 October 2026

You are already at the latest version

Abstract
Validating a concrete Android bug requires more than choosing a tool category: the rule, inputs, and oracle must decide whether the target mechanism is present on the buggy version and absent on the fixed one. We present ABAF-VP, a validation-planning approach that profiles the target mechanism, filters and ranks candidate plans using mechanism, capability, oracle, and evidence predicates drawn from a 191-record knowledge base, aligns the selected rule to the concrete target, and verifies both artifact versions against an independent two-sided reference. On 100 formal tasks with independent human reference labels, ABAF-VP ranks a valid plan first for 10 of 13 jointly solvable tasks (P@1 = 76.9%), ahead of category, retrieval, and capability baselines; ablation attributes the gain mainly to mechanism matching (−69.2 pp when removed). Target-rule alignment raises paired verification on 60 bug/fix pairs from 5 to 47 (78.3%) and end-to-end task success from 5/100 to 47/100, adding 30.274 s of known machine time (5.5%). Results are bounded by a small solvable subset, manual alignment bindings, and documented capability limits. Within these bounds, validation planning offers an executable, evidence-aware path from Android bug-mechanism knowledge to concrete verification.
Keywords: 
;  ;  ;  ;  ;  ;  

1. Introduction

Android applications fail in mechanism-specific ways: a leaked resource, a missing lifecycle guard, and a mishandled URI grant may share the same coarse bug category, yet each requires different rules, input artifacts, and verification oracles. Developers validating a concrete defect face a planning problem: which tool, rule set, input mode, and independent evidence will verify whether the specific mechanism is present on the buggy version and absent on the fixed one? Existing support answers a different question. Benchmarks such as Ghera [1] and DroidLeaks [2] evaluate tools against known ground truth, and taxonomies describe bug categories; neither tells practitioners which plan is executable and verifiable for their concrete mechanism.
Figure 1 illustrates this gap. In jellyfin-android PR #194 (case FP-08), the fix adds a lifecycle guard around a fragment transaction in WebViewFragment.onSelectServer. A mechanism-agnostic selector ranks the generic rule CommitTransaction first, which matches the category but fails to check the actual mechanism change. The aligned rule FRAGMENT_TXN_LIFECYCLE_GUARD verifies the added guard and confirms the target condition is present on the buggy version and absent on the fixed one. Selecting a plausible rule is not the same as verifying the actual mechanism—across our 60-pair pool, rule–target gaps account for 78.2% of initially failed verifications.
This paper studies validation planning for Android bug mechanisms: given a concrete mechanism, available artifacts, and platform constraints, we produce a plan—tool, version, rule set, input mode, and oracle template—with an independent verification method that can be executed. We present ABAF-VP, a four-stage approach that (1) profiles the target mechanism and its trigger conditions, (2) filters and ranks candidate plans from a 176-plan pool using mechanism, capability, oracle, and evidence predicates, (3) aligns the selected rule to the concrete file, method, or component requiring verification, and (4) executes both artifact versions and applies a differential oracle against an independently established reference. The planning vocabulary derives from ABAF-21, a knowledge base of 191 root-cause records in 21 mechanism categories distilled from a frozen 35-study corpus.
Our evaluation follows a conservative protocol. All reference labels are independently produced by two human annotators; development and test materials are isolated by root-cause cluster, project, and template; recommendations are frozen before scoring; every reported number is recomputed from frozen data. Three research questions organize the results. RQ1 (recommendation): on 13 jointly solvable tasks, ABAF-VP ranks a valid plan first for 10 (P@1 = 76.9%), compared to 4/13 for the strongest retrieval baselines; module ablation attributes the gain primarily to mechanism matching ( − 69.2 pp when removed). RQ2 (paired verification): the unbound planner verifies only 5 of 60 bug/fix pairs, while target-rule alignment raises this to 47/60 (78.3%), including 26 of 36 real-world pairs across 17 projects—the remaining 13 pairs reflect documented capability limits or unresolved cases. RQ3 (execution cost): the aligned system completes end-to-end verification on 47 of 100 tasks with 30.274 s additional known machine time (5.5%) over the unaligned planner.
The contributions are bounded accordingly. First, a reproducible profile specification and a mechanism-driven planner whose recommendation quality is established against four fair baselines on a shared candidate pool. Second, empirical identification of rule–target correspondence as the dominant verification bottleneck, and evidence that target-rule alignment resolves it, raising paired verification from 5/60 to 47/60; in the evaluated extension, rule selection and binding are manually configured, constituting a human-in-the-loop boundary. Third, a rigorous experimental protocol—isolated splits, frozen predictions, independent two-sided references, cluster-aware statistics—whose negative and inconclusive results (a small solvable subset of 13 tasks; two non-significant baseline comparisons; two modules without measurable contribution) are reported alongside the positive findings.
The remainder of the paper is organized as follows. Section 2 positions the work against unified tool evaluation, paired benchmarks, capability recommendation, and validation planning. Section 3 defines the approach and the compared methods. Section 4 describes the data and the experimental protocol. Section 5 answers RQ1–RQ3. Section 6 discusses threats to validity and limitations, and Section 7 concludes.

3. The ABAF-VP Approach

ABAF-VP addresses the following question: given a concrete bug-mechanism task—a target mechanism, available program artifacts, and platform constraints—can we select a verification plan that has an independent verification method and can be executed? The system has four stages (Figure 2): mechanism profiling, candidate filtering and ranking, target-rule alignment, and execution with paired verification. Table 1 defines the terminology: we assign each method and module a formal name, using compact labels B0–B4, A1–A3, and ALIGN only as defined there, including in figures.

3.1. Stage 1: Mechanism Profiling

A validation profile structures a task into the target mechanism, its trigger condition, and required evidence, together with available artifacts and platform context. Profiles draw vocabulary from the ABAF-21 knowledge base (Section 4.1), which supplies mechanism categories, trigger descriptions, and documented tool capabilities. The capability registry is built from public tool documentation, rule implementations, and the development set; it never reads test outputs or reference labels. Each capability predicate records accepted artifacts, language, platform, mechanism labels with synonyms, trigger conditions, required oracle, and estimated execution cost. Listing  shows the profile of task FT-008 (the FP-08 case of Figure 1).
Preprints 240446 i001

3.2. Stage 2: Candidate Filtering and Ranking

Candidate plans are filtered by three-valued predicates: FALSE excludes the candidate; TRUE retains it; UNKNOWN retains the candidate as a conditional recommendation with an explicit evidence gap. Dynamic checks requiring user authorization are never executed automatically.
Surviving candidates are ranked lexicographically by: (1) all hard conditions satisfied; (2) mechanism match quality (exact, partial, category-only, unknown); (3) oracle readiness; (4) strength of prior independent evidence; (5) reproducibility; (6) estimated cost, ascending; and (7) a stable plan identifier as tie-break. Matching uses frozen labels and predicates only. Estimated costs come from development runs and tool documentation; measured test costs are used exclusively for evaluation. The planner outputs a Top-1 and Top-3 with per-criterion match explanations, unmet conditions, suggested oracle, estimated cost, evidence ceiling, and—when the candidate list is empty—an explicit abstention with the reason. Emitting a command is never equated with executing it.

3.3. Stage 3: Target-Rule Alignment

A category-level rule rarely coincides with the concrete location requiring verification. The target-rule alignment module (ALIGN) binds a rule to a concrete file, method, or XML component and produces an executable verification proposition: a target-specific rule whose positive and negative sides can be decided on both artifact versions. In the evaluated extension, rule selection and target binding were manually configured and are recorded as such; the module’s contribution is reported as a system extension with human-configured bindings, not as automated alignment. Listing  shows what such a binding is: a declarative configuration that names the rule, target method, verification proposition, and call/guard vocabularies, rather than ad hoc hard-coded checks.
Preprints 240446 i002

3.4. Stage 4: Execution and Paired Verification

The executor records, for every run, the task, plan, pair, tool, version, rule set, configuration, hashes of inputs and commands, timing, exit status, and locations of logs and reports. A run maps to exactly one immutable input version. Compatibility status, execution status, and target-verification status are recorded separately.
The target verifier receives the target contract and trusted inputs frozen before the run and reports the verification type, target object, scope actually checked, and a three-valued outcome (TRUE/FALSE/UNKNOWN) with its provenance. Four verification types are distinguished: source review, benchmark oracle, static-property check, and runtime reproduction. A differential oracle accepts a pair only when the target condition is present on the buggy artifact and absent on the fixed artifact. A failed run is never counted as a negative target state, a static property change is not claimed as a functional fix, and the independent reference never comes from the verifier’s own judgment.

3.5. Compared Methods and Planning Ablations

All five methods share the candidate pool, task inputs, Top-k definition, and reference labels. Category mapping (B0) maps the legacy taxonomy category to candidate tools without mechanism or oracle information. BM25 retrieval (B1) ranks a fixed corpus of tool and rule documentation with the public task description as query ( k 1 = 1.2 , b = 0.75 , frozen tokenization) [19]. Embedding similarity (B2) ranks the same corpus by cosine similarity over sentence embeddings from a locally hosted general model (sentence-transformers/all-MiniLM-L6-v2 [20], frozen revision); the model is never fine-tuned on test labels. Capability-only matching (B3) applies artifact, language, platform, and runtime compatibility without mechanism, oracle, or evidence information. The full planner (B4) is ABAF-VP as described above. The BM25 and embedding indices contain neither task reference labels nor adjudicated root-cause-to-tool associations.
The three planning ablations (A1–A3, Table 1) each remove one module from B4 and run on the same 100 tasks with the same candidate pool, isolating module contributions under identical conditions.

4. Data and Experimental Protocol

This section describes the knowledge base, the task set, the paired bug/fix cases, and the experimental controls that are common to all three research questions. Table 3 summarizes the resulting assets.

4.1. Knowledge Base and Mechanism Profiles

The ABAF-21 knowledge base was constructed from a frozen corpus of 35 primary studies on Android bug taxonomies and detection. It contains 191 root-cause records with stable registry identifiers, organized into 21 mechanism categories (Table 2); duplicate or ambiguous candidates retain their original identifiers and are linked through mechanism-cluster identifiers rather than being silently merged, so the count of records is preserved by construction. The category assignments were double-annotated during the construction of the knowledge base (Cohen’s κ = 0.877 ). We treat the 191 entries as records with provenance, not as 191 independent mechanisms.
On top of this corpus, two human raters independently annotated 100 Gold profiles. Each profile records the target mechanism, its trigger condition, the required input artifacts, the platform context, the available oracle, and tool applicability. The raters worked from separate files under code names and adjudicated disagreements in a second pass; ten additional non-Gold profiles were used to stabilize the annotation specification before formal annotation began.

4.2. Task Set and Reference Labels

The unit of recommendation is a candidate plan: a combination of a tool, a version, a rule set, an input mode, and an oracle template. After deduplication, the candidate pool contains 176 plans. The task set consists of 120 slots drawn with stratified sampling (domain, then mechanism cluster, then real versus synthetic; random seed 20261011): 20 development tasks used to fix rules, parameters, and the candidate pool, and 100 formal test tasks spanning at least 40 distinct root causes, at least ten mechanism clusters, and three application domains.
For the formal tasks, two human raters independently labeled every task–candidate pair, yielding 100 × 176 = 17 , 600 confirmed reference judgments on a four-way scale (VALID, CONDITIONAL, INVALID, UNKNOWN); candidates that were not examined were never defaulted to INVALID. Of the 100 formal tasks, 13 have at least one independently valid plan and form the jointly solvable subset T = 13 used for recommendation scoring; 47 were confirmed after full candidate review to have no valid plan; and 40 remain unresolved because no physical case input could be obtained for them. The unresolved tasks are retained in all denominators and are never counted as recommendation errors.

4.3. Paired Bug/Fix Cases

Paired verification operates on 60 bug/fix pairs: 36 real-world pairs drawn from 25 open-source projects and 24 benchmark pairs drawn from the Ghera benchmark [1]. Candidates were collected from Ghera, DroidLeaks [2], and verifiable issue/commit pairs; a pair was included only when both versions were obtainable, the target could be localized, the fix was reviewably related to the target mechanism, at least one registered tool had documented testable applicability, an independent oracle could be established, and the inputs were valid. Every exclusion was recorded with its reason. No single project contributes more than 20% of the real cases, and the pool spans at least ten mechanism clusters and three domains. For each pair, a two-sided reference—the target defect is present on the buggy version and absent on the fixed version—was established by two human annotators and frozen before any method output was produced.

4.4. Tools and Compared Methods

Six analysis tools are registered: Android Lint [21], detekt [22], PMD [23], SpotBugs [24], Semgrep/mobsfscan [25], and MobSF [26]. Not every tool is expected to apply to every case; a priori applicability (YES/NO/UNKNOWN) is recorded before any scan, tools are executed only at applicable positions, and a failure to run never changes the a priori denominator.
Five recommendation methods are compared: four baselines and the full system, defined in Section 3 and denoted B0–B4 in tables and figures for compactness. Three planning ablations isolate the contribution of individual planning modules. All methods share the same candidate pool and the same frozen reference labels.

4.5. Isolation, Freezing, and Statistical Protocol

Development and test materials are isolated by root-cause duplicate cluster, by project, and by template group; the 15 legacy Ghera pairs and every case inspected during debugging are treated as development material and never enter the test split. The recommendation side of the system mounts only the public task inputs and the capability registry; Gold labels and test outputs are never mounted. Recommendations are frozen before evaluation, and any correction to a reference label triggers a uniform recomputation for all methods with the original values retained.
All reported confidence intervals are cluster bootstrap intervals over leakage groups (10,000 resamples, seed 20261011); pairwise method comparisons use paired cluster permutation tests with Holm correction. Task planning is budgeted at 20 minutes per task, individual scans time out after 20 minutes, and a pair has a total budget of 40 minutes. All Gold labels and paired references were produced by the two human annotators; automation prepared source packets and empty templates only and never substituted for independent human judgment.
Table 3. Dataset and tool inventory. Root-cause profiles form the frozen mechanism base; the 100 Gold tasks carry independent double annotation; the 60 paired cases comprise 36 real fixes and 24 benchmark cases. Tool versions are the fixed installations used in the execution runs, and all six tools are addressed through absolute installation paths rather than the system PATH.
Table 3. Dataset and tool inventory. Root-cause profiles form the frozen mechanism base; the 100 Gold tasks carry independent double annotation; the 60 paired cases comprise 36 real fixes and 24 benchmark cases. Tool versions are the fixed installations used in the execution runs, and all six tools are addressed through absolute installation paths rather than the system PATH.
Dataset item Value
Root-cause profiles (frozen mechanism base) 191
Gold-annotated tasks (two independent annotators) 100
Development tasks 20
Formal evaluation tasks 100
Paired cases 60 (36 real + 24 benchmark)
Baseline methods (B0–B3) 4
Planning ablations (A1–A3) 3
Tool Installed version
Android Lint 8.9.0
detekt 1.23.8
PMD 7.28.0
SpotBugs 4.10.4
Semgrep/mobsfscan 0.4.3
MobSF container image (latest)

5. Results

5.1. RQ1: Does Mechanism- and Constraint-Driven Planning Improve Recommendation Quality?

Setup.

All five methods rank the same pool of 176 candidate plans for the same 100 formal tasks under frozen reference labels (Section 4.2). Recommendation metrics are scored on the jointly solvable subset T = 13 ; within T, an abstention or run error scores zero. The remaining 87 tasks are not silently dropped: 47 were confirmed to have no valid plan and 40 lack physical case input, both groups reported in the full-task distribution in Section 4.2.

Five-method comparison.

Table 4 and Figure 3(a) report P@1, Success@3, and MRR. The full planner (B4) ranks a valid plan first for 10 of 13 solvable tasks (P@1 = 76.9%), reaches 12/13 on Success@3 (92.3%), and attains MRR = 0.843. The strongest baselines (BM25 retrieval and embedding similarity) reach P@1 = 4/13 (30.8%); category mapping and capability-only matching reach 0/13 and 1/13, respectively. B4 abstains on only 2 of 100 tasks, compared with 24 abstentions for capability-only matching. As an input-filtering sensitivity check, augmenting category mapping with the common input filter raises its P@1 only from 0/13 to 1/13 (with 23 abstentions), so the planner’s advantage does not arise from excluding wrong input types alone.
Paired comparisons on P@1 with Holm correction show B4 significantly ahead of category mapping ( + 76.9 pp, 95% CI [ + 53.8 , + 100 ] , p = 0.0078 ) and capability-only matching ( + 69.2 pp, 95% CI [ + 46.2 , + 92.3 ] , p = 0.0117 ). The margins over BM25 retrieval and embedding similarity are each + 46.2 pp (95% CIs [ + 15.4 , + 69.2 ] and [ + 7.7 , + 76.9 ] , respectively) but do not survive Holm correction ( p = 0.0625 and p = 0.0703 ); with only 13 jointly solvable tasks, the test has limited power, and we report effect sizes without claiming significance.

Planning ablations.

Table 5 isolates the three planning modules on the same 100 tasks and candidate pool. Removing mechanism matching (A1) collapses P@1 from 10/13 to 1/13—the FULL configuration exceeds the ablation by + 69.2 pp, 95% CI [ + 46.2 , + 92.3 ] , Holm p = 0.0117 —the only significant ablation—and all nine tasks for which only the full planner ranked a valid plan first lose their hit. Removing oracle-availability filtering and ranking (A2) changes none of the three metrics. Removing prior-evidence ranking (A3) changes P@1 by + 7.7 pp while reducing Success@3 by 7.7pp, a direction-unstable difference that is not significant (Holm p = 1.0 ). On this task set, mechanism matching is the dominant contributor to recommendation quality; the oracle and evidence modules show no measurable contribution, and we state this as a finding.

Answer to RQ1.

On jointly solvable tasks, mechanism-driven planning improves top-choice accuracy over all four fair baselines, with the ablation attributing the gain primarily to mechanism matching. The evidence is bounded by the small solvable subset ( T = 13 ), and the two retrieval baselines remain within the uncertainty margin after multiplicity correction.

5.2. RQ2: Can the Recommended Plans Verify the Concrete Target on Both Versions?

Setup.

RQ2 evaluates the 60 paired bug/fix cases (Section 4.3). A pair counts as verified only when the selected rule corresponds to the target, the check actually executes, both sides are decidable, and the outcome agrees with the independently frozen reference (defect present on the buggy version, absent on the fixed version). Source review, benchmark oracles, static-property checks, and runtime reproduction are reported as distinct verification types, and a static property change alone is never claimed as a functional fix.

Starting point and bottleneck.

The initial checker verified 1 of 60 pairs. With the full planner’s unbound Top-1 rules, 5 pairs verified directly; of the remaining 55, 43 exhibited a rule–target gap—the selected rule did not correspond to the changed mechanism—10 lay outside the checker scope for the changed mechanism (nine recycling resource-type pairs and FP-09), and 2 remained unresolved. Rule–target gaps thus account for 78.2% of the initially non-passing pairs, identifying rule–target correspondence, rather than ranking, as the dominant bottleneck at verification time. This is consistent with the RQ1 ablation: the planner selects the right kind of tool and rule, but a category-level rule does not by itself decide the concrete target.

Target-rule alignment.

Applying target-rule alignment (ALIGN, Section 3.3) to the 43 gap pairs disposed of all of them: 42 became verified and FP-42 was adjudicated as partial property support, which is not counted as a success. Together with the 5 direct passes, the aligned system verifies 47 of 60 pairs (78.3%; 95% cluster-bootstrap CI [67.2%, 88.1%], 10,000 resamples): 26 of 36 real-world pairs (72.2%) spanning 17 projects, and 21 of 24 benchmark pairs (87.5%). Table 6 summarizes the before/after comparison and the final disposition of all 60 pairs; Figure 4 shows the pair-level state transitions. The terminal states after alignment are legitimate capability boundaries, not silent failures: FP-39 and FP-47 remain unresolved, and the ten out-of-scope pairs remain out of scope. FP-32 is counted as a success under the frozen adjudication decision.

Integrity checks.

All counterexample probes passed, and the recommendation evidence remained unchanged throughout the alignment work. The selected Top-1 plans for the 60 pairs draw on three of the six registered tools—Android Lint (19/30 verified), Semgrep/mobsfscan (20/22), and detekt (8/8)—while PMD, SpotBugs, and MobSF were never the top-ranked applicable plan in this pool; we report this rather than presenting the evaluation as six-tool coverage.

Answer to RQ2.

Recommended plans verify the concrete target on both artifact versions for 47 of 60 pairs once rules are aligned to the target; the residual 13 pairs decompose into ten documented capability limits, one partial-support adjudication, and two unresolved cases, all reported rather than excluded.

5.3. RQ3: Do the Recommended Plans Execute End-to-End, and at What Cost?

Setup.

RQ3 executes the Top-1 plan of every method on all 100 formal tasks—500 method–task receipts backed by 302 unique physical run keys, of which 174 reuse historical runs under the same task, plan, input, and configuration and 128 are new scans (379.0 s of new scan time). The 60 tasks with physical inputs form the executable subset; the 40 tasks without inputs remain in the denominator. End-to-end success requires the same plan version, rules, configuration, and inputs to execute on both sides and to pass independent target verification; a generated command never counts as execution, and a completed run never counts as a verified target.

Execution funnel.

Table 7 and Figure 5 report the funnel. All methods except capability-only matching (76) and the full planner (98) returned a Top-1 for every task. Two-sided execution succeeded on 59–60 of the 60 input-having tasks for category mapping, capability-only matching, and the full planner, and on 45–46 for the two retrieval baselines. End-to-end verified success, however, is rare without target alignment: 0 for category mapping and capability-only matching, 1 each for the retrieval baselines, and 5 for the full planner. The planner’s + 4 to + 5 pp advantage over the baselines is not significant after Holm correction ( p ≈ 0.25 ): with only five effective successes the comparison has very low power, and we report it as such.

Aligned system.

Extending the planner with target-rule alignment raises end-to-end success from 5/100 to 47/100 ( + 42 tasks; 5/60 to 47/60 on the input-having subset), with real-world cases going from 1/36 to 26/36 and benchmark cases from 4/24 to 21/24. The five paired comparisons against B0–B4 are each significant after Holm correction ( p ≈ 5 × 10 − 5 ). This + 42 gain is a property of the system extension—Top-1 selection plus aligned target rules plus verification coverage—and is not attributed to the ranking algorithm alone; the RQ1 comparison is the evidence for ranking quality, and the boundary between the two contributions is kept explicit.

Cost.

Known machine time rises from 552.336 s to 582.610 s, an increment of 30.274 s (5.5%), or 0.721 s per added verified task. We deliberately report known machine time, not full deployment cost: tool runs (120/120), alignment checks (86/86), and AST processing (102/102) are fully timed, but input preparation lacks timers for 48 of 120 input identities, and manual rule selection and binding are recorded as “manual configuration” without a clock.

Answer to RQ3.

The recommended plans execute for real under identical versions, rules, configurations, and inputs, and pass independent target verification at a marginal machine-time cost of about 0.72 s per additionally verified task; the unaligned planner’s end-to-end advantage over baselines is small and statistically inconclusive, while the aligned system’s improvement is large and significant.

6. Discussion

6.1. Interpretation

The three research questions form a chain. Mechanism knowledge pays off first at recommendation time: removing mechanism matching is the only ablation that significantly degrades Top-1 accuracy (RQ1). The bottleneck then shifts: even a well-ranked category-level rule verifies only 5 of 60 pairs, because the rule does not coincide with the concrete target—78.2% of initially non-passing pairs are rule–target gaps (RQ2). Once rules are aligned to their targets, paired verification rises to 47/60 and end-to-end task success to 47/100, at a marginal known machine-time cost of 0.721 s per additional success (RQ3). The practical reading is that mechanism knowledge and target-rule alignment are complementary stages of one pipeline, and that the alignment stage, not the ranking stage, is where verification is currently won or lost.

6.2. Threats to Validity

Internal validity.

The jointly solvable subset is small ( T = 13 ), limiting statistical power: the planner’s margins over the two retrieval baselines ( + 46.2 pp) do not survive Holm correction, and the unaligned planner’s end-to-end advantage in RQ3 ( p ≈ 0.25 ) rests on five effective successes. We report effect sizes and exact p-values rather than rounding into significance. Two planning modules (oracle availability, prior evidence) show no measurable contribution on this task set; this may reflect the task mix rather than a universal property. In the evaluated target-rule alignment, rule selection and binding were manually configured; the RQ2/RQ3 gains characterize a human-in-the-loop system extension, and automating the binding step remains open.

Construct validity.

Verification types are kept distinct: a static property change supports the checked property only and is never reported as a functional fix or runtime reproduction; a failed run is never a negative target state. FP-42’s partial property support is not counted as a success, and FP-32’s inclusion follows a frozen adjudication decision. Cost figures are known machine time: input preparation lacks timers for 48 of 120 input identities and manual configuration is untimed, so true end-to-end cost is higher than reported.

External validity.

The 60 pairs span 25 real-world projects plus the Ghera benchmark and are not 60 independent projects; the 100 tasks are not 100 independent root causes, and cluster correlation is handled only partially by leakage-group resampling. The 191 knowledge-base records carry provenance but are not independent mechanisms. We do not extrapolate beyond the covered domains, mechanism clusters, or tool set.

Reliability and knowledge cost.

The capability registry and profile annotations embody manual effort; two human annotators produced all reference labels with adjudication, and disagreements were retained rather than averaged away. Recommendations were frozen before scoring, and reference corrections triggered uniform recomputation.

6.3. Limitations and Future Work

Forty formal tasks remain without physical inputs, ten pairs remain outside checker scope, and two pairs (FP-39, FP-47) remain unresolved; all are retained in the reported denominators. A per-domain breakdown of recommendation accuracy is not part of the frozen analysis. Automating target-rule binding, instrumenting the untimed cost components, and extending the pool beyond the current domains and six tools are the natural next steps.

7. Conclusions

We studied validation planning for Android bug mechanisms: given a concrete mechanism, available artifacts, and constraints, select a verification plan with an independent verification method that can be executed. ABAF-VP profiles the target mechanism, filters and ranks candidate plans with mechanism, capability, oracle, and evidence predicates, aligns the selected rule to the concrete target, and verifies both artifact versions against an independently established reference.
On frozen evidence, the planner ranks a valid plan first for 10 of 13 jointly solvable tasks (P@1 = 76.9%), ahead of all four fair baselines, with mechanism matching as the dominant contributor ( − 69.2 pp when removed). Target-rule alignment raises paired verification from 5/60 to 47/60 (78.3%), including 26/36 real-world pairs across 17 projects, and raises end-to-end task success from 5/100 to 47/100 at an increment of 30.274 s of known machine time (5.5%). The boundaries are integral to the result: the solvable subset is small, two baseline comparisons are not significant after correction, alignment bindings are manually configured, and 13 pairs end in documented capability limits or unresolved states. Within these bounds, the evidence supports validation planning as a workable path from Android bug-mechanism knowledge to executable, verifiable checks.

Author Contributions

Conceptualization, Z.X. and W.L.; methodology, Z.X.; software, Z.X.; validation, Z.X. and Y.W.; formal analysis, Z.X.; investigation, Z.X. and Y.W.; data curation, Z.X.; writing—original draft preparation, Z.X.; writing—review and editing, Y.W. and W.L.; visualization, Z.X.; supervision, W.L.; project administration, W.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data supporting the reported results are available on request from the corresponding author.

Acknowledgments

During the preparation of this study, the authors used Claude Code (Anthropic) for data screening and GPT (OpenAI) for the design of the paper draft. The manuscript was written and polished by the human authors, who have reviewed and edited all output and take full responsibility for the content of this publication.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Mitra, J.; Ranganath, V.P. Ghera: A Repository of Android App Vulnerability Benchmarks. In Proceedings of the 13th Int’l Conf. on Predictive Models and Data Analytics in Software Engineering (PROMISE), 2017; pp. 43–52. [Google Scholar] [CrossRef]
  2. Liu, Y.P.; Wang, J.; Wei, L.L.; Xu, C.; Cheung, S.C.; Wu, T.Y.; Yan, J.; Zhang, J. DROIDLEAKS: A Comprehensive Database of Resource Leaks in Android Apps. Empir. Softw. Eng. 2019, 24. [Google Scholar] [CrossRef]
  3. Xiong, Y.H.; Xu, M.Q.; Su, T.; Sun, J.L.; Wang, J.; Wen, H.; Pu, G.G.; He, J.F.; Su, Z.D. An Empirical Study of Functional Bugs in Android Apps. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023. [Google Scholar] [CrossRef]
  4. Chen, S.; Zhang, Y.X.; Fan, L.L.; Li, J.M.; Liu, Y. AUSERA: Automated Security Vulnerability Detection for Android Apps. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2022. [Google Scholar] [CrossRef]
  5. Mazuera-Rozo, A.; Escobar-Velásquez, C.; Espitia-Acero, J.; Vega-Guzmán, D.; Trubiani, C.; Linares-Vásquez, M.; Bavota, G. Taxonomy of Security Weaknesses in Java and Kotlin Android Apps. J. Syst. Softw. 2022, 187. [Google Scholar] [CrossRef]
  6. Pauck, F.; Bodden, E.; Wehrheim, H. Do Android Taint Analysis Tools Keep Their Promises? In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018. [Google Scholar] [CrossRef]
  7. Reaves, B.; Bowers, J.; Gorski, S.A., III; Anise, O.; Bobhate, R.; Cho, R.; Das, H.; Hussain, S.; Karachiwala, H.; Scaife, N.; et al. *droid: Assessment and Evaluation of Android Application Analysis Tools. ACM Comput. Surv. 2016, 49. [Google Scholar] [CrossRef]
  8. Arzt, S.; Rasthofer, S.; Fritz, C.; Bodden, E.; Bartel, A.; Klein, J.; Le Traon, Y.; Octeau, D.; McDaniel, P. FlowDroid: Precise Context, Flow, Field, Object-sensitive and Lifecycle-aware Taint Analysis for Android Apps. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014; pp. 259–269. [Google Scholar] [CrossRef]
  9. Li, L.; Bissyandé, T.F.; Papadakis, M.; Rasthofer, S.; Bartel, A.; Octeau, D.; Klein, J.; Traon, Y.L. Static Analysis of Android Apps: A Systematic Literature Review. Inf. Softw. Technol. 2017, 88. [Google Scholar] [CrossRef]
  10. Sadeghi, A.; Bagheri, H.; Garcia, J.; Malek, S. A Taxonomy and Qualitative Comparison of Program Analysis Techniques for Security Assessment of Android Software. IEEE Trans. Softw. Eng. 2017, 43. [Google Scholar] [CrossRef]
  11. Acar, Y.; Backes, M.; Bugiel, S.; Fahl, S.; McDaniel, P.; Smith, M. SoK: Lessons Learned from Android Security Research for Appified Software Platforms. In Proceedings of the IEEE Symposium on Security and Privacy (S&P), 2016; pp. 433–451. [Google Scholar] [CrossRef]
  12. Mahmud, J.; Pandey, S.; Silva, N.D.; Dipongkor, A.K.; Wu, J.; Chaparro, O.; Fazzini, M.; Moran, K. Automated Program Repair for UI-Centric Android Bugs: How Far Are We? Proc. ACM Softw. Eng. 2026, 3, 4143–4167. [Google Scholar] [CrossRef]
  13. Rua, R.; Ali, K. Towards LLMinating Performance Anti-Patterns in Android Apps. In Proceedings of the 2026 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW); IEEE, 2026; pp. 17–24. [Google Scholar] [CrossRef]
  14. Lachgar, M.; Kharroubi, F.; Ayoujil, A.; Achbarou, O.; Tourad, M.C. MobileSec: A microservices-based intelligent platform for automated Android APK vulnerability analysis and remediation. SoftwareX 2026, 35, 102897. [Google Scholar] [CrossRef]
  15. Chen, M.; Tan, T.; Pan, M.; Li, Y. PacDroid: A Pointer-Analysis-Centric Framework for Security Vulnerabilities in Android Apps. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE); IEEE, 2025; pp. 2803–2815. [Google Scholar] [CrossRef]
  16. Santos, G.; Bispo, J.; Mendes, A. Detecting Resource Leaks on Android with Alpakka. In Proceedings of the 18th ACM SIGPLAN International Conference on Software Language Engineering, 2025; ACM; pp. 199–211. [Google Scholar] [CrossRef]
  17. Zhou, Z.; Chen, R.; Chen, J.; Liu, Y.; Wei, L. Characterizing Bugs in Login Processes of Android Applications: An Empirical Study. In Proceedings of the 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC); IEEE, 2025; pp. 25–36. [Google Scholar] [CrossRef]
  18. Yang, H.; Huang, R.; Fang, Y.; Zhang, B.; Guo, J.; Wu, Z.; Mi, X. Okara: Detection and Attribution of TLS Man-in-the-Middle Vulnerabilities in Android Apps with Foundation Models. In Proceedings of the Lecture Notes in Computer Science; Information Security and Privacy; Springer Nature Singapore, 2026; pp. 225–270. [Google Scholar] [CrossRef]
  19. Robertson, S.; Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 2009, 3, 333–389. [Google Scholar] [CrossRef]
  20. Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019; pp. 3982–3992. [Google Scholar] [CrossRef]
  21. Android Developers. Improve Your Code with Lint Checks. 2025. Available online: https://developer.android.com/studio/write/lint (accessed on 18 October 2026).
  22. detekt: Static Code Analysis for Kotlin. 2025. Available online: https://detekt.dev.
  23. PMD: An Extensible Multilanguage Static Code Analyzer. 2024. Available online: https://pmd.github.io.
  24. SpotBugs: Find Bugs in Java Programs. 2024. Available online: https://spotbugs.github.io.
  25. Semgrep: Lightweight Static Analysis for Many Languages. 2025. Available online: https://semgrep.dev.
  26. Mobile Security Framework (MobSF). 2025. Available online: https://github.com/MobSF/Mobile-Security-Framework-MobSF.
Figure 1. Motivation for target-specific paired verification. A real lifecycle-guard repair in jellyfin-android (FP-08, PR #194) illustrates the difference between selecting a rule and checking the changed mechanism. Code excerpts reproduce WebViewFragment.kt lines 187-192 (buggy) and 188-197 (fixed) with only the outermost block indent removed; long lines are wrapped for display and repeat their source number, the fixed ... elides the lifecycle-checked replaceFragment body on lines 192-195, and every number is the source line. The original Top-1 rule was CommitTransaction; the aligned source-level rule FRAGMENT_TXN_LIFECYCLE_GUARD checks the lifecycle guard in WebViewFragment.onSelectServer and records PRESENT on V_bug and ABSENT on V_fix. Across the fixed 60-pair pool, the frozen unbound B4 succeeds on 5 pairs and B4 with the aligned rule extension succeeds on 47. The bottom comparison concerns B4 before and after the extension; the category and keyword selectors are conceptual alternatives. The 43 rule-target gaps account for 78.2% of the 55 initially non-passing pairs.
Figure 1. Motivation for target-specific paired verification. A real lifecycle-guard repair in jellyfin-android (FP-08, PR #194) illustrates the difference between selecting a rule and checking the changed mechanism. Code excerpts reproduce WebViewFragment.kt lines 187-192 (buggy) and 188-197 (fixed) with only the outermost block indent removed; long lines are wrapped for display and repeat their source number, the fixed ... elides the lifecycle-checked replaceFragment body on lines 192-195, and every number is the source line. The original Top-1 rule was CommitTransaction; the aligned source-level rule FRAGMENT_TXN_LIFECYCLE_GUARD checks the lifecycle guard in WebViewFragment.onSelectServer and records PRESENT on V_bug and ABSENT on V_fix. Across the fixed 60-pair pool, the frozen unbound B4 succeeds on 5 pairs and B4 with the aligned rule extension succeeds on 47. The bottom comparison concerns B4 before and after the extension; the category and keyword selectors are conceptual alternatives. The 43 rule-target gaps account for 78.2% of the 55 initially non-passing pairs.
Preprints 240446 g001
Figure 2. Four-stage ABAF-VP system overview. (1) Mechanism profiling structures a task into target mechanism, trigger condition and required evidence. (2) B4 ranks verification plans from a 176-candidate pool using mechanism, capability, oracle and evidence information. (3) ALIGN binds a target-specific rule to a concrete file, method or XML component, producing an executable verification proposition. In the evaluated extension, rule selection and target binding were manually configured. (4) Both artifacts are checked and a differential oracle evaluates whether the target condition is present on the buggy artifact and absent on the fixed artifact; the recorded state and timing are retained. The ABAF-21 knowledge base supplies Stages 1 and 2. Source-level property evidence retains its recorded scope and is not a runtime-failure claim.
Figure 2. Four-stage ABAF-VP system overview. (1) Mechanism profiling structures a task into target mechanism, trigger condition and required evidence. (2) B4 ranks verification plans from a 176-candidate pool using mechanism, capability, oracle and evidence information. (3) ALIGN binds a target-specific rule to a concrete file, method or XML component, producing an executable verification proposition. In the evaluated extension, rule selection and target binding were manually configured. (4) Both artifacts are checked and a differential oracle evaluates whether the target condition is present on the buggy artifact and absent on the fixed artifact; the recorded state and timing are retained. The ABAF-21 knowledge base supplies Stages 1 and 2. Source-level property evidence retains its recorded scope and is not a runtime-failure claim.
Preprints 240446 g002
Figure 3. Recommendation quality and planning-module ablation (RQ1). The five-method comparison uses the existing 100-task evaluation; P@1, Success@3 and MRR are scored on the jointly solvable subset T = 13. (a) Method comparisons; MRR is multiplied by 100. The dark outline identifies B4. Brackets denote significant Holm-adjusted paired comparisons on P@1: B4 versus B0 (p = 0.0078) and B3 (p = 0.0117). B4 versus B1 and B2 has p = 0.0625 and 0.0703, respectively. (b) Changes in P@1 after removing a module, relative to FULL. Only point estimates and Holm-adjusted exact permutation p-values are reported; no confidence interval is drawn on the T = 13 subset. Removing mechanism matching reduces P@1 by 69.2 percentage points (Holm p = 0.0117). Removing oracle adaptation changes neither P@1 nor Success@3. Removing prior evidence raises P@1 by 7.7 points while reducing Success@3 by 7.7 points; the P@1 difference is not significant. The FULL P@1 value is a reference annotation, not a line on the difference axis.
Figure 3. Recommendation quality and planning-module ablation (RQ1). The five-method comparison uses the existing 100-task evaluation; P@1, Success@3 and MRR are scored on the jointly solvable subset T = 13. (a) Method comparisons; MRR is multiplied by 100. The dark outline identifies B4. Brackets denote significant Holm-adjusted paired comparisons on P@1: B4 versus B0 (p = 0.0078) and B3 (p = 0.0117). B4 versus B1 and B2 has p = 0.0625 and 0.0703, respectively. (b) Changes in P@1 after removing a module, relative to FULL. Only point estimates and Holm-adjusted exact permutation p-values are reported; no confidence interval is drawn on the T = 13 subset. Removing mechanism matching reduces P@1 by 69.2 percentage points (Holm p = 0.0117). Removing oracle adaptation changes neither P@1 nor Success@3. Removing prior evidence raises P@1 by 7.7 points while reducing Success@3 by 7.7 points; the P@1 difference is not significant. The FULL P@1 value is a reference annotation, not a line on the difference axis.
Preprints 240446 g003
Figure 4. Paired-validation outcome and the target-rule extension (RQ2). The fixed task pool comprises 36 real-world pairs from 25 projects and 24 benchmark pairs. Before ALIGN, unbound B4 verifies 5 pairs (direct passes); 53 pairs are not applicable, of which 43 are rule-target gaps and 10 lie outside the checker scope for the changed mechanism, and 2 are unresolved. ALIGN adds target-specific rules and bindings for the rule-target gaps: 42 of those 43 pairs become verified (17 benchmark, 25 real-world). Together with the 5 direct passes this yields 47/60 (78.3%; 95% leakage-group bootstrap CI [67.2%, 88.1%], 10,000 resamples). After ALIGN, 3 pairs remain unverified (FP-39 and FP-47 unresolved, FP-42 partial property support) and the 10 out-of-scope pairs remain out of scope. Success is 26/36 on real-world pairs, spanning 17 successful projects, and 21/24 on benchmark pairs. Pair-level transitions are 53 not applicable to 43 rule-target gaps plus 10 inapplicable, then 42 to verified, 1 to partial support and 10 unchanged; 5 direct passes stay verified and 2 unresolved stay unresolved. Band heights are proportional to these counts. The grouping is for target-specific validation, not an evaluation of alert populations. Results retain the frozen FP-32 closeout decision.
Figure 4. Paired-validation outcome and the target-rule extension (RQ2). The fixed task pool comprises 36 real-world pairs from 25 projects and 24 benchmark pairs. Before ALIGN, unbound B4 verifies 5 pairs (direct passes); 53 pairs are not applicable, of which 43 are rule-target gaps and 10 lie outside the checker scope for the changed mechanism, and 2 are unresolved. ALIGN adds target-specific rules and bindings for the rule-target gaps: 42 of those 43 pairs become verified (17 benchmark, 25 real-world). Together with the 5 direct passes this yields 47/60 (78.3%; 95% leakage-group bootstrap CI [67.2%, 88.1%], 10,000 resamples). After ALIGN, 3 pairs remain unverified (FP-39 and FP-47 unresolved, FP-42 partial property support) and the 10 out-of-scope pairs remain out of scope. Success is 26/36 on real-world pairs, spanning 17 successful projects, and 21/24 on benchmark pairs. Pair-level transitions are 53 not applicable to 43 rule-target gaps plus 10 inapplicable, then 42 to verified, 1 to partial support and 10 unchanged; 5 direct passes stay verified and 2 unresolved stay unresolved. Band heights are proportional to these counts. The grouping is for target-specific validation, not an evaluation of alert populations. Results retain the frozen FP-32 closeout decision.
Preprints 240446 g004
Figure 5. End-to-end verification effectiveness and known machine time (RQ3). Bars show verified tasks out of all 100 evaluation tasks; the left axis counts verified tasks, not execution volume, so baselines that run their tools but reach no target-specific verdict have short bars. The orange line shows known machine time, including recorded input preparation, on a separate zero-based axis; the short horizontal marks beside the baseline points mark the baseline methods (B0–B3), whose time buys tool runs rather than target-specific judgments. Extending B4 with aligned target rules and verification coverage raises successes from 5 to 47, adding 42 tasks. Recorded machine time rises from 552.336 to 582.610 seconds, an increment of 30.274 seconds (5.5%), or 0.721 seconds per added verified task. The five new paired comparisons are significant after Holm correction (p rounded to 5 × 10−5 for each). This comparison evaluates a system extension with different target-rule coverage. Time values exclude untimed manual binding and development; preparation timers are available for 72/120 input identities. The figure therefore reports known machine time, not full deployment cost.
Figure 5. End-to-end verification effectiveness and known machine time (RQ3). Bars show verified tasks out of all 100 evaluation tasks; the left axis counts verified tasks, not execution volume, so baselines that run their tools but reach no target-specific verdict have short bars. The orange line shows known machine time, including recorded input preparation, on a separate zero-based axis; the short horizontal marks beside the baseline points mark the baseline methods (B0–B3), whose time buys tool runs rather than target-specific judgments. Extending B4 with aligned target rules and verification coverage raises successes from 5 to 47, adding 42 tasks. Recorded machine time rises from 552.336 to 582.610 seconds, an increment of 30.274 seconds (5.5%), or 0.721 seconds per added verified task. The five new paired comparisons are significant after Holm correction (p rounded to 5 × 10−5 for each). This comparison evaluates a system extension with different target-rule coverage. Time values exclude untimed manual binding and development; preparation timers are available for 72/120 input identities. The figure therefore reports known machine time, not full deployment cost.
Preprints 240446 g005
Table 1. Formal names of the compared methods, the alignment module, and the planning ablations. The compact labels are used in tables and figures.
Table 1. Formal names of the compared methods, the alignment module, and the planning ablations. The compact labels are used in tables and figures.
Label Formal name Definition
B0 Category mapping Legacy taxonomy category mapped to candidate tools; fixed-ID tie-break
B1 BM25 retrieval BM25 over tool/rule documentation; query is the public task description
B2 Embedding similarity Sentence-embedding cosine similarity over the same corpus
B3 Capability-only matching Artifact/language/platform/runtime compatibility only
B4 ABAF-VP (full planner) Mechanism- and constraint-driven planning, all modules on
ALIGN Target-rule alignment Stage 3: binds a rule to a concrete file/method/component
A1 −Mechanism ablation B4 without mechanism compatibility filtering and ranking
A2 −Oracle ablation B4 without oracle-availability filtering and ranking
A3 −Evidence ablation B4 without prior-evidence strength ranking
Table 2. The ABAF-21 knowledge base: 21 mechanism categories and 191 root-cause records. One representative root cause is shown per category; the full record list retains provenance per entry.
Table 2. The ABAF-21 knowledge base: 21 mechanism categories and 191 root-cause records. One representative root cause is shown per category; the full record list retains provenance per entry.
ID Category #RCs Representative root cause
C01 Activity & Fragment Lifecycle 7 State loss / transaction loss during commit
C02 Callback Bugs 3 Unhandled callbacks / missing callbacks
C03 General Programming Errors 23 Missing or incorrectly specified conditional checks
C04 Android-Related Errors 10 Android framework API misuse
C05 Third-Party Library Bugs 5 Library bugs / limitations
C06 Data Storage Security 12 External / internal storage exposure
C07 Data Encryption Security 10 Insecure AES / DES encryption
C08 Data Access Security 17 Exported Activity / Service / Provider / Receiver
C09 Communication Security 9 Invalid certificate / hostname verification
C10 Configuration Security 6 Backup / debugging enabled in the manifest
C11 App State & UI Bugs 16 UI operations outside the main thread
C12 API & Compatibility Bugs 9 API misuse; incompatible APIs
C13 Resource Management 16 Missing / incorrect string resources
C14 Permission-Based Security 11 Unimplemented runtime permissions (API 23+)
C15 Device Orientation Bugs 1 Incorrect handling of orientation changes
C16 Inter-Component Communication 5 Nonexistent targets; null / missing intents
C17 Memory Bugs 9 Use-after-free
C18 Process & Thread Bugs 6 Concurrency; race conditions
C19 Back & Up Navigation 2 Incorrect back/up navigation implementation
C20 WebView-Related Bugs 11 Misaligned WebView lifecycles
C21 Performance Issues 3 Lengthy operations on the main thread
Total 191
Table 4. Recommendation accuracy of the five methods on the T = 13 tasks that have a confirmed common solution. P@1 and Success@3 are reported as counts and percentages; MRR is the mean reciprocal rank over the same tasks. The last two columns give B4 minus each baseline on P@1 (percentage points) and the Holm-adjusted exact permutation p-value over the four baseline comparisons. B0: category mapping; B1: BM25; B2: embedding similarity; B3: capability-only compatibility filtering; B4: full method.
Table 4. Recommendation accuracy of the five methods on the T = 13 tasks that have a confirmed common solution. P@1 and Success@3 are reported as counts and percentages; MRR is the mean reciprocal rank over the same tasks. The last two columns give B4 minus each baseline on P@1 (percentage points) and the Holm-adjusted exact permutation p-value over the four baseline comparisons. B0: category mapping; B1: BM25; B2: embedding similarity; B3: capability-only compatibility filtering; B4: full method.
P@1 Success@3 B4 − baseline
Method n / 13 (%) n / 13 (%) MRR Δ P@1 (pp) Holm p
B0 0 0.0% 0 0.0% 0.0145 + 76.9 0.0078
B1 4 30.8% 11 84.6% 0.5526 + 46.2 0.0625
B2 4 30.8% 9 69.2% 0.5253 + 46.2 0.0703
B3 1 7.7% 3 23.1% 0.1949 + 69.2 0.0117
B4 10 76.9% 12 92.3% 0.8429 – –
Table 5. Planning-module ablation on the same T = 13 tasks. Each row removes one module from the full configuration (FULL): A1 is mechanism matching, A2 is oracle adaptation, and A3 is prior evidence. The last three columns give FULL minus the ablated configuration on P@1, the 95% group-bootstrap confidence interval of that difference, and the Holm-adjusted exact permutation p-value over the three comparisons.
Table 5. Planning-module ablation on the same T = 13 tasks. Each row removes one module from the full configuration (FULL): A1 is mechanism matching, A2 is oracle adaptation, and A3 is prior evidence. The last three columns give FULL minus the ablated configuration on P@1, the 95% group-bootstrap confidence interval of that difference, and the Holm-adjusted exact permutation p-value over the three comparisons.
P@1 Success@3 FULL − configuration
Configuration n / 13 (%) n / 13 (%) MRR Δ P@1 (pp) 95% CI (pp) Holm p
FULL 10 76.9% 12 92.3% 0.8429 – – –
−A1 1 7.7% 2 15.4% 0.1652 + 69.2 [ + 46.2 , + 92.3 ] 0.0117
−A2 10 76.9% 12 92.3% 0.8429 0.0 [ 0.0 , 0.0 ] 1.0000
−A3 11 84.6% 11 84.6% 0.8675 − 7.7 [ − 23.1 , 0.0 ] 1.0000
Table 6. Target-specific paired verification over the 60 pairs (36 real fixes and 24 benchmark cases). (a) Successful pairs for Unbound B4 and for B4 + ALIGN, i.e., before and after the target-rule alignment correction; a success requires a rule–target correspondence, bilateral execution, decidable evidence on both sides, and agreement with the independent reference. (b) Disposition of the 60 pairs under B4 + ALIGN: 43 pairs had a rule–target mismatch, of which 42 became MATCHED and one was adjudicated as PROPERTY_SUPPORT and is not counted as a success.
Table 6. Target-specific paired verification over the 60 pairs (36 real fixes and 24 benchmark cases). (a) Successful pairs for Unbound B4 and for B4 + ALIGN, i.e., before and after the target-rule alignment correction; a success requires a rule–target correspondence, bilateral execution, decidable evidence on both sides, and agreement with the independent reference. (b) Disposition of the 60 pairs under B4 + ALIGN: 43 pairs had a rule–target mismatch, of which 42 became MATCHED and one was adjudicated as PROPERTY_SUPPORT and is not counted as a success.
Metric Unbound B4 B4 + ALIGN Δ
Successful pairs 5/60 (8.3%) 47/60 (78.3%) +42
Real cases 1/36 (2.8%) 26/36 (72.2%) +25
Benchmark cases 4/24 (16.7%) 21/24 (87.5%) +17
Real projects covered 1 17 +16
Disposition of the 60 pairs n Share
Previously successful, retained 5 8.3%
Rule–target mismatch, 43 pairs
   corrected to MATCHED 42 70.0%
   PROPERTY_SUPPORT (adjudicated, not counted) 1 1.7%
NOT_APPLICABLE (outside implemented checker scope) 10 16.7%
UNRESOLVED 2 3.3%
Total 60 100.0%
Table 7. End-to-end execution and known machine time for the five frozen methods and for the B4+ALIGN extension. Top-1 emitted counts the plans returned per 100 tasks, with abstentions in parentheses; the 40 tasks without a physical input remain in the /100 denominator. Bilateral execution and verified success are reported over the 60 tasks that have a physical input. A task without input cannot succeed, so the /60 success count equals the /100 count for every method. Known machine time covers timed machine work only (tool execution, input preparation, rule checking, and AST parsing) and excludes manual rule binding and development. The p-value refers to the B4+ALIGN versus B4 comparison over 89 leakage clusters with Holm correction over the five new comparisons.
Table 7. End-to-end execution and known machine time for the five frozen methods and for the B4+ALIGN extension. Top-1 emitted counts the plans returned per 100 tasks, with abstentions in parentheses; the 40 tasks without a physical input remain in the /100 denominator. Bilateral execution and verified success are reported over the 60 tasks that have a physical input. A task without input cannot succeed, so the /60 success count equals the /100 count for every method. Known machine time covers timed machine work only (tool execution, input preparation, rule checking, and AST parsing) and excludes manual rule binding and development. The p-value refers to the B4+ALIGN versus B4 comparison over 89 leakage clusters with Holm correction over the five new comparisons.
Method Top-1
emitted
(/100)
Bilateral
executed
(/60)
Verified
success
(/100)
Verified
success
(/60)
Known machine
time (s)
B0 100 59 0 0 602.271
B1 100 45 1 1 637.671
B2 100 46 1 1 590.434
B3 76 (24 abstained) 60 0 0 410.976
B4 98 (2 abstained) 60 5 5 552.336
B4+ALIGN 98 (2 abstained) 60 47 47 582.610
Δ (B4+ALIGN − B4) 0 0 +42 +42 +30.274 (5.5%)
Increment of B4+ALIGN over B4 Value
Real cases 1/36 → 26/36 (+25)
Benchmark cases 4/24 → 21/24 (+17)
Known time per new success 0.721 s
Holm-adjusted p 5.0 × 10 − 5
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.