Submitted:
26 October 2026
Posted:
27 October 2026
You are already at the latest version
Abstract
Validating a concrete Android bug requires more than choosing a tool category: the rule, inputs, and oracle must decide whether the target mechanism is present on the buggy version and absent on the fixed one. We present ABAF-VP, a validation-planning approach that profiles the target mechanism, filters and ranks candidate plans using mechanism, capability, oracle, and evidence predicates drawn from a 191-record knowledge base, aligns the selected rule to the concrete target, and verifies both artifact versions against an independent two-sided reference. On 100 formal tasks with independent human reference labels, ABAF-VP ranks a valid plan first for 10 of 13 jointly solvable tasks (P@1 = 76.9%), ahead of category, retrieval, and capability baselines; ablation attributes the gain mainly to mechanism matching (−69.2 pp when removed). Target-rule alignment raises paired verification on 60 bug/fix pairs from 5 to 47 (78.3%) and end-to-end task success from 5/100 to 47/100, adding 30.274 s of known machine time (5.5%). Results are bounded by a small solvable subset, manual alignment bindings, and documented capability limits. Within these bounds, validation planning offers an executable, evidence-aware path from Android bug-mechanism knowledge to concrete verification.
Keywords:
android bug mechanisms
; validation planning
; mechanism-driven recommendation
; target-rule alignment
; paired verification
; static analysis tools
; automated software quality assurance
1. Introduction
Android applications fail in mechanism-specific ways: a leaked resource, a missing lifecycle guard, and a mishandled URI grant may share the same coarse bug category, yet each requires different rules, input artifacts, and verification oracles. Developers validating a concrete defect face a planning problem: which tool, rule set, input mode, and independent evidence will verify whether the specific mechanism is present on the buggy version and absent on the fixed one? Existing support answers a different question. Benchmarks such as Ghera [1] and DroidLeaks [2] evaluate tools against known ground truth, and taxonomies describe bug categories; neither tells practitioners which plan is executable and verifiable for their concrete mechanism.
Figure 1 illustrates this gap. In jellyfin-android PR #194 (case FP-08), the fix adds a lifecycle guard around a fragment transaction in WebViewFragment.onSelectServer. A mechanism-agnostic selector ranks the generic rule CommitTransaction first, which matches the category but fails to check the actual mechanism change. The aligned rule FRAGMENT_TXN_LIFECYCLE_GUARD verifies the added guard and confirms the target condition is present on the buggy version and absent on the fixed one. Selecting a plausible rule is not the same as verifying the actual mechanism—across our 60-pair pool, rule–target gaps account for 78.2% of initially failed verifications.
This paper studies validation planning for Android bug mechanisms: given a concrete mechanism, available artifacts, and platform constraints, we produce a plan—tool, version, rule set, input mode, and oracle template—with an independent verification method that can be executed. We present ABAF-VP, a four-stage approach that (1) profiles the target mechanism and its trigger conditions, (2) filters and ranks candidate plans from a 176-plan pool using mechanism, capability, oracle, and evidence predicates, (3) aligns the selected rule to the concrete file, method, or component requiring verification, and (4) executes both artifact versions and applies a differential oracle against an independently established reference. The planning vocabulary derives from ABAF-21, a knowledge base of 191 root-cause records in 21 mechanism categories distilled from a frozen 35-study corpus.
Our evaluation follows a conservative protocol. All reference labels are independently produced by two human annotators; development and test materials are isolated by root-cause cluster, project, and template; recommendations are frozen before scoring; every reported number is recomputed from frozen data. Three research questions organize the results. RQ1 (recommendation): on 13 jointly solvable tasks, ABAF-VP ranks a valid plan first for 10 (P@1 = 76.9%), compared to 4/13 for the strongest retrieval baselines; module ablation attributes the gain primarily to mechanism matching (pp when removed). RQ2 (paired verification): the unbound planner verifies only 5 of 60 bug/fix pairs, while target-rule alignment raises this to 47/60 (78.3%), including 26 of 36 real-world pairs across 17 projects—the remaining 13 pairs reflect documented capability limits or unresolved cases. RQ3 (execution cost): the aligned system completes end-to-end verification on 47 of 100 tasks with 30.274 s additional known machine time (5.5%) over the unaligned planner.
The contributions are bounded accordingly. First, a reproducible profile specification and a mechanism-driven planner whose recommendation quality is established against four fair baselines on a shared candidate pool. Second, empirical identification of rule–target correspondence as the dominant verification bottleneck, and evidence that target-rule alignment resolves it, raising paired verification from 5/60 to 47/60; in the evaluated extension, rule selection and binding are manually configured, constituting a human-in-the-loop boundary. Third, a rigorous experimental protocol—isolated splits, frozen predictions, independent two-sided references, cluster-aware statistics—whose negative and inconclusive results (a small solvable subset of 13 tasks; two non-significant baseline comparisons; two modules without measurable contribution) are reported alongside the positive findings.
The remainder of the paper is organized as follows. Section 2 positions the work against unified tool evaluation, paired benchmarks, capability recommendation, and validation planning. Section 3 defines the approach and the compared methods. Section 4 describes the data and the experimental protocol. Section 5 answers RQ1–RQ3. Section 6 discusses threats to validity and limitations, and Section 7 concludes.
2. Background and Related Work
2.1. Android Bug Taxonomies and Mechanism Knowledge
Empirical studies have organized Android defects by root cause and mechanism. Xiong et al. built a taxonomy of functional bugs with symptoms and oracles over 399 instances and evaluated test-generation capabilities [3]. AUSERA organizes Android security vulnerabilities into five main categories and fifty types, comparing detection tools per type [4]. Mazuera-Rozo et al. derived a CWE-guided taxonomy of security weaknesses in Java and Kotlin apps from 400 fix commits [5]. DroidLeaks couples a database of 292 real resource leaks with mechanism characteristics and an evaluation of eight detectors [2]. These works establish that taxonomy, dataset, and tool analysis combine fruitfully; the ABAF-21 knowledge base integrates such sources across defect domains.
2.2. Unified Tool Evaluation and Paired Benchmarks
A second line of work evaluates analysis tools under shared ground truth. Pauck et al. introduced ReproDroid and the AQL query language to verify whether six taint-analysis tools meet machine-readable expectations [6]. Reaves et al. audited seven Android analysis tools for availability, deployability, and output fidelity [7]. DroidBench [8] and Ghera [1] provide benchmark cases with documented expectations, and DroidLeaks adds real-world fix pairs for resource leaks [2]. Our work reuses such benchmarks as sources of paired cases and borrows their discipline of machine-checkable expectations, but addresses a different question: not how a tool scores on a fixed benchmark, but which plan should be selected for a concrete mechanism and whether it can verify that mechanism on both artifact versions.
2.3. Systematic Reviews and Comparative Taxonomies
Li et al. systematically reviewed 124 studies on static analysis of Android apps along goals, techniques, sensitivities, Android features, and evaluation scale [9]. Sadeghi et al. produced a taxonomy and qualitative comparison of analysis techniques for Android security assessment [10]. Following SoK methodology [11], we present our corpus as a frozen, bounded integration with recorded provenance, not as an exhaustive survey of Android research.
2.4. Recent Joint Analyses and the Remaining Gap
Recent work continues combining categories, datasets, and tools: automated program repair for UI-centric bugs couples a taxonomy of repair obstacles with datasets and multiple repair tools [12]; performance anti-pattern work organizes eleven categories and compares static analysis tools [13]; MobileSec integrates multi-category APK security analysis over DroidBench and Ghera in a microservices platform [14]; PacDroid compares pointer-analysis-centric vulnerability detection across frameworks and benchmarks [15]; Alpakka revisits resource-leak detection on DroidLeaks [16]; a study of login-process bugs separates root causes, symptoms, and trigger conditions [17]; and Okara links detection to mechanism-level attribution for TLS vulnerabilities [18].
Within this landscape, the specific combination we target—selecting per concrete task an executable plan (tool, version, rule set, input mode, oracle template) conditioned on the target mechanism and constraints, then verifying that mechanism on both artifact versions against an independently established reference—is not addressed by the compared works, whose evaluation units are tools on fixed benchmarks or categories over datasets. Our incremental literature search (2025–2026, 37 included publications) supports this positioning within its recorded bounds; we state the gap as a property of the compared set, not as a claim of universal novelty.
3. The ABAF-VP Approach
ABAF-VP addresses the following question: given a concrete bug-mechanism task—a target mechanism, available program artifacts, and platform constraints—can we select a verification plan that has an independent verification method and can be executed? The system has four stages (Figure 2): mechanism profiling, candidate filtering and ranking, target-rule alignment, and execution with paired verification. Table 1 defines the terminology: we assign each method and module a formal name, using compact labels B0–B4, A1–A3, and ALIGN only as defined there, including in figures.
3.1. Stage 1: Mechanism Profiling
A validation profile structures a task into the target mechanism, its trigger condition, and required evidence, together with available artifacts and platform context. Profiles draw vocabulary from the ABAF-21 knowledge base (Section 4.1), which supplies mechanism categories, trigger descriptions, and documented tool capabilities. The capability registry is built from public tool documentation, rule implementations, and the development set; it never reads test outputs or reference labels. Each capability predicate records accepted artifacts, language, platform, mechanism labels with synonyms, trigger conditions, required oracle, and estimated execution cost. Listing shows the profile of task FT-008 (the FP-08 case of Figure 1).

3.2. Stage 2: Candidate Filtering and Ranking
Candidate plans are filtered by three-valued predicates: FALSE excludes the candidate; TRUE retains it; UNKNOWN retains the candidate as a conditional recommendation with an explicit evidence gap. Dynamic checks requiring user authorization are never executed automatically.
Surviving candidates are ranked lexicographically by: (1) all hard conditions satisfied; (2) mechanism match quality (exact, partial, category-only, unknown); (3) oracle readiness; (4) strength of prior independent evidence; (5) reproducibility; (6) estimated cost, ascending; and (7) a stable plan identifier as tie-break. Matching uses frozen labels and predicates only. Estimated costs come from development runs and tool documentation; measured test costs are used exclusively for evaluation. The planner outputs a Top-1 and Top-3 with per-criterion match explanations, unmet conditions, suggested oracle, estimated cost, evidence ceiling, and—when the candidate list is empty—an explicit abstention with the reason. Emitting a command is never equated with executing it.
3.3. Stage 3: Target-Rule Alignment
A category-level rule rarely coincides with the concrete location requiring verification. The target-rule alignment module (ALIGN) binds a rule to a concrete file, method, or XML component and produces an executable verification proposition: a target-specific rule whose positive and negative sides can be decided on both artifact versions. In the evaluated extension, rule selection and target binding were manually configured and are recorded as such; the module’s contribution is reported as a system extension with human-configured bindings, not as automated alignment. Listing shows what such a binding is: a declarative configuration that names the rule, target method, verification proposition, and call/guard vocabularies, rather than ad hoc hard-coded checks.

3.4. Stage 4: Execution and Paired Verification
The executor records, for every run, the task, plan, pair, tool, version, rule set, configuration, hashes of inputs and commands, timing, exit status, and locations of logs and reports. A run maps to exactly one immutable input version. Compatibility status, execution status, and target-verification status are recorded separately.
The target verifier receives the target contract and trusted inputs frozen before the run and reports the verification type, target object, scope actually checked, and a three-valued outcome (TRUE/FALSE/UNKNOWN) with its provenance. Four verification types are distinguished: source review, benchmark oracle, static-property check, and runtime reproduction. A differential oracle accepts a pair only when the target condition is present on the buggy artifact and absent on the fixed artifact. A failed run is never counted as a negative target state, a static property change is not claimed as a functional fix, and the independent reference never comes from the verifier’s own judgment.
3.5. Compared Methods and Planning Ablations
All five methods share the candidate pool, task inputs, Top-k definition, and reference labels. Category mapping (B0) maps the legacy taxonomy category to candidate tools without mechanism or oracle information. BM25 retrieval (B1) ranks a fixed corpus of tool and rule documentation with the public task description as query (, , frozen tokenization) [19]. Embedding similarity (B2) ranks the same corpus by cosine similarity over sentence embeddings from a locally hosted general model (sentence-transformers/all-MiniLM-L6-v2 [20], frozen revision); the model is never fine-tuned on test labels. Capability-only matching (B3) applies artifact, language, platform, and runtime compatibility without mechanism, oracle, or evidence information. The full planner (B4) is ABAF-VP as described above. The BM25 and embedding indices contain neither task reference labels nor adjudicated root-cause-to-tool associations.
The three planning ablations (A1–A3, Table 1) each remove one module from B4 and run on the same 100 tasks with the same candidate pool, isolating module contributions under identical conditions.
4. Data and Experimental Protocol
This section describes the knowledge base, the task set, the paired bug/fix cases, and the experimental controls that are common to all three research questions. Table 3 summarizes the resulting assets.
4.1. Knowledge Base and Mechanism Profiles
The ABAF-21 knowledge base was constructed from a frozen corpus of 35 primary studies on Android bug taxonomies and detection. It contains 191 root-cause records with stable registry identifiers, organized into 21 mechanism categories (Table 2); duplicate or ambiguous candidates retain their original identifiers and are linked through mechanism-cluster identifiers rather than being silently merged, so the count of records is preserved by construction. The category assignments were double-annotated during the construction of the knowledge base (Cohen’s ). We treat the 191 entries as records with provenance, not as 191 independent mechanisms.
On top of this corpus, two human raters independently annotated 100 Gold profiles. Each profile records the target mechanism, its trigger condition, the required input artifacts, the platform context, the available oracle, and tool applicability. The raters worked from separate files under code names and adjudicated disagreements in a second pass; ten additional non-Gold profiles were used to stabilize the annotation specification before formal annotation began.
4.2. Task Set and Reference Labels
The unit of recommendation is a candidate plan: a combination of a tool, a version, a rule set, an input mode, and an oracle template. After deduplication, the candidate pool contains 176 plans. The task set consists of 120 slots drawn with stratified sampling (domain, then mechanism cluster, then real versus synthetic; random seed 20261011): 20 development tasks used to fix rules, parameters, and the candidate pool, and 100 formal test tasks spanning at least 40 distinct root causes, at least ten mechanism clusters, and three application domains.
For the formal tasks, two human raters independently labeled every task–candidate pair, yielding confirmed reference judgments on a four-way scale (VALID, CONDITIONAL, INVALID, UNKNOWN); candidates that were not examined were never defaulted to INVALID. Of the 100 formal tasks, 13 have at least one independently valid plan and form the jointly solvable subset used for recommendation scoring; 47 were confirmed after full candidate review to have no valid plan; and 40 remain unresolved because no physical case input could be obtained for them. The unresolved tasks are retained in all denominators and are never counted as recommendation errors.
4.3. Paired Bug/Fix Cases
Paired verification operates on 60 bug/fix pairs: 36 real-world pairs drawn from 25 open-source projects and 24 benchmark pairs drawn from the Ghera benchmark [1]. Candidates were collected from Ghera, DroidLeaks [2], and verifiable issue/commit pairs; a pair was included only when both versions were obtainable, the target could be localized, the fix was reviewably related to the target mechanism, at least one registered tool had documented testable applicability, an independent oracle could be established, and the inputs were valid. Every exclusion was recorded with its reason. No single project contributes more than 20% of the real cases, and the pool spans at least ten mechanism clusters and three domains. For each pair, a two-sided reference—the target defect is present on the buggy version and absent on the fixed version—was established by two human annotators and frozen before any method output was produced.
4.4. Tools and Compared Methods
Six analysis tools are registered: Android Lint [21], detekt [22], PMD [23], SpotBugs [24], Semgrep/mobsfscan [25], and MobSF [26]. Not every tool is expected to apply to every case; a priori applicability (YES/NO/UNKNOWN) is recorded before any scan, tools are executed only at applicable positions, and a failure to run never changes the a priori denominator.
Five recommendation methods are compared: four baselines and the full system, defined in Section 3 and denoted B0–B4 in tables and figures for compactness. Three planning ablations isolate the contribution of individual planning modules. All methods share the same candidate pool and the same frozen reference labels.
4.5. Isolation, Freezing, and Statistical Protocol
Development and test materials are isolated by root-cause duplicate cluster, by project, and by template group; the 15 legacy Ghera pairs and every case inspected during debugging are treated as development material and never enter the test split. The recommendation side of the system mounts only the public task inputs and the capability registry; Gold labels and test outputs are never mounted. Recommendations are frozen before evaluation, and any correction to a reference label triggers a uniform recomputation for all methods with the original values retained.
All reported confidence intervals are cluster bootstrap intervals over leakage groups (10,000 resamples, seed 20261011); pairwise method comparisons use paired cluster permutation tests with Holm correction. Task planning is budgeted at 20 minutes per task, individual scans time out after 20 minutes, and a pair has a total budget of 40 minutes. All Gold labels and paired references were produced by the two human annotators; automation prepared source packets and empty templates only and never substituted for independent human judgment.
Table 3.
Dataset and tool inventory. Root-cause profiles form the frozen mechanism base; the 100 Gold tasks carry independent double annotation; the 60 paired cases comprise 36 real fixes and 24 benchmark cases. Tool versions are the fixed installations used in the execution runs, and all six tools are addressed through absolute installation paths rather than the system PATH.
Table 3.
Dataset and tool inventory. Root-cause profiles form the frozen mechanism base; the 100 Gold tasks carry independent double annotation; the 60 paired cases comprise 36 real fixes and 24 benchmark cases. Tool versions are the fixed installations used in the execution runs, and all six tools are addressed through absolute installation paths rather than the system PATH.
| Dataset item | Value |
|---|---|
| Root-cause profiles (frozen mechanism base) | 191 |
| Gold-annotated tasks (two independent annotators) | 100 |
| Development tasks | 20 |
| Formal evaluation tasks | 100 |
| Paired cases | 60 (36 real + 24 benchmark) |
| Baseline methods (B0–B3) | 4 |
| Planning ablations (A1–A3) | 3 |
| Tool | Installed version |
| Android Lint | 8.9.0 |
| detekt | 1.23.8 |
| PMD | 7.28.0 |
| SpotBugs | 4.10.4 |
| Semgrep/mobsfscan | 0.4.3 |
| MobSF | container image (latest) |
5. Results
5.1. RQ1: Does Mechanism- and Constraint-Driven Planning Improve Recommendation Quality?
Setup.
All five methods rank the same pool of 176 candidate plans for the same 100 formal tasks under frozen reference labels (Section 4.2). Recommendation metrics are scored on the jointly solvable subset ; within T, an abstention or run error scores zero. The remaining 87 tasks are not silently dropped: 47 were confirmed to have no valid plan and 40 lack physical case input, both groups reported in the full-task distribution in Section 4.2.
Five-method comparison.
Table 4 and Figure 3(a) report P@1, Success@3, and MRR. The full planner (B4) ranks a valid plan first for 10 of 13 solvable tasks (P@1 = 76.9%), reaches 12/13 on Success@3 (92.3%), and attains MRR = 0.843. The strongest baselines (BM25 retrieval and embedding similarity) reach P@1 = 4/13 (30.8%); category mapping and capability-only matching reach 0/13 and 1/13, respectively. B4 abstains on only 2 of 100 tasks, compared with 24 abstentions for capability-only matching. As an input-filtering sensitivity check, augmenting category mapping with the common input filter raises its P@1 only from 0/13 to 1/13 (with 23 abstentions), so the planner’s advantage does not arise from excluding wrong input types alone.
Paired comparisons on P@1 with Holm correction show B4 significantly ahead of category mapping (pp, 95% CI , ) and capability-only matching (pp, 95% CI , ). The margins over BM25 retrieval and embedding similarity are each pp (95% CIs and , respectively) but do not survive Holm correction ( and ); with only 13 jointly solvable tasks, the test has limited power, and we report effect sizes without claiming significance.
Planning ablations.
Table 5 isolates the three planning modules on the same 100 tasks and candidate pool. Removing mechanism matching (A1) collapses P@1 from 10/13 to 1/13—the FULL configuration exceeds the ablation by pp, 95% CI , Holm —the only significant ablation—and all nine tasks for which only the full planner ranked a valid plan first lose their hit. Removing oracle-availability filtering and ranking (A2) changes none of the three metrics. Removing prior-evidence ranking (A3) changes P@1 by pp while reducing Success@3 by 7.7pp, a direction-unstable difference that is not significant (Holm ). On this task set, mechanism matching is the dominant contributor to recommendation quality; the oracle and evidence modules show no measurable contribution, and we state this as a finding.
Answer to RQ1.
On jointly solvable tasks, mechanism-driven planning improves top-choice accuracy over all four fair baselines, with the ablation attributing the gain primarily to mechanism matching. The evidence is bounded by the small solvable subset (), and the two retrieval baselines remain within the uncertainty margin after multiplicity correction.
5.2. RQ2: Can the Recommended Plans Verify the Concrete Target on Both Versions?
Setup.
RQ2 evaluates the 60 paired bug/fix cases (Section 4.3). A pair counts as verified only when the selected rule corresponds to the target, the check actually executes, both sides are decidable, and the outcome agrees with the independently frozen reference (defect present on the buggy version, absent on the fixed version). Source review, benchmark oracles, static-property checks, and runtime reproduction are reported as distinct verification types, and a static property change alone is never claimed as a functional fix.
Starting point and bottleneck.
The initial checker verified 1 of 60 pairs. With the full planner’s unbound Top-1 rules, 5 pairs verified directly; of the remaining 55, 43 exhibited a rule–target gap—the selected rule did not correspond to the changed mechanism—10 lay outside the checker scope for the changed mechanism (nine recycling resource-type pairs and FP-09), and 2 remained unresolved. Rule–target gaps thus account for 78.2% of the initially non-passing pairs, identifying rule–target correspondence, rather than ranking, as the dominant bottleneck at verification time. This is consistent with the RQ1 ablation: the planner selects the right kind of tool and rule, but a category-level rule does not by itself decide the concrete target.
Target-rule alignment.
Applying target-rule alignment (ALIGN, Section 3.3) to the 43 gap pairs disposed of all of them: 42 became verified and FP-42 was adjudicated as partial property support, which is not counted as a success. Together with the 5 direct passes, the aligned system verifies 47 of 60 pairs (78.3%; 95% cluster-bootstrap CI [67.2%, 88.1%], 10,000 resamples): 26 of 36 real-world pairs (72.2%) spanning 17 projects, and 21 of 24 benchmark pairs (87.5%). Table 6 summarizes the before/after comparison and the final disposition of all 60 pairs; Figure 4 shows the pair-level state transitions. The terminal states after alignment are legitimate capability boundaries, not silent failures: FP-39 and FP-47 remain unresolved, and the ten out-of-scope pairs remain out of scope. FP-32 is counted as a success under the frozen adjudication decision.
Integrity checks.
All counterexample probes passed, and the recommendation evidence remained unchanged throughout the alignment work. The selected Top-1 plans for the 60 pairs draw on three of the six registered tools—Android Lint (19/30 verified), Semgrep/mobsfscan (20/22), and detekt (8/8)—while PMD, SpotBugs, and MobSF were never the top-ranked applicable plan in this pool; we report this rather than presenting the evaluation as six-tool coverage.
Answer to RQ2.
Recommended plans verify the concrete target on both artifact versions for 47 of 60 pairs once rules are aligned to the target; the residual 13 pairs decompose into ten documented capability limits, one partial-support adjudication, and two unresolved cases, all reported rather than excluded.
5.3. RQ3: Do the Recommended Plans Execute End-to-End, and at What Cost?
Setup.
RQ3 executes the Top-1 plan of every method on all 100 formal tasks—500 method–task receipts backed by 302 unique physical run keys, of which 174 reuse historical runs under the same task, plan, input, and configuration and 128 are new scans (379.0 s of new scan time). The 60 tasks with physical inputs form the executable subset; the 40 tasks without inputs remain in the denominator. End-to-end success requires the same plan version, rules, configuration, and inputs to execute on both sides and to pass independent target verification; a generated command never counts as execution, and a completed run never counts as a verified target.
Execution funnel.
Table 7 and Figure 5 report the funnel. All methods except capability-only matching (76) and the full planner (98) returned a Top-1 for every task. Two-sided execution succeeded on 59–60 of the 60 input-having tasks for category mapping, capability-only matching, and the full planner, and on 45–46 for the two retrieval baselines. End-to-end verified success, however, is rare without target alignment: 0 for category mapping and capability-only matching, 1 each for the retrieval baselines, and 5 for the full planner. The planner’s to pp advantage over the baselines is not significant after Holm correction (): with only five effective successes the comparison has very low power, and we report it as such.
Aligned system.
Extending the planner with target-rule alignment raises end-to-end success from 5/100 to 47/100 ( tasks; 5/60 to 47/60 on the input-having subset), with real-world cases going from 1/36 to 26/36 and benchmark cases from 4/24 to 21/24. The five paired comparisons against B0–B4 are each significant after Holm correction (). This gain is a property of the system extension—Top-1 selection plus aligned target rules plus verification coverage—and is not attributed to the ranking algorithm alone; the RQ1 comparison is the evidence for ranking quality, and the boundary between the two contributions is kept explicit.
Cost.
Known machine time rises from 552.336 s to 582.610 s, an increment of 30.274 s (5.5%), or 0.721 s per added verified task. We deliberately report known machine time, not full deployment cost: tool runs (120/120), alignment checks (86/86), and AST processing (102/102) are fully timed, but input preparation lacks timers for 48 of 120 input identities, and manual rule selection and binding are recorded as “manual configuration” without a clock.
Answer to RQ3.
The recommended plans execute for real under identical versions, rules, configurations, and inputs, and pass independent target verification at a marginal machine-time cost of about 0.72 s per additionally verified task; the unaligned planner’s end-to-end advantage over baselines is small and statistically inconclusive, while the aligned system’s improvement is large and significant.
6. Discussion
6.1. Interpretation
The three research questions form a chain. Mechanism knowledge pays off first at recommendation time: removing mechanism matching is the only ablation that significantly degrades Top-1 accuracy (RQ1). The bottleneck then shifts: even a well-ranked category-level rule verifies only 5 of 60 pairs, because the rule does not coincide with the concrete target—78.2% of initially non-passing pairs are rule–target gaps (RQ2). Once rules are aligned to their targets, paired verification rises to 47/60 and end-to-end task success to 47/100, at a marginal known machine-time cost of 0.721 s per additional success (RQ3). The practical reading is that mechanism knowledge and target-rule alignment are complementary stages of one pipeline, and that the alignment stage, not the ranking stage, is where verification is currently won or lost.
6.2. Threats to Validity
Internal validity.
The jointly solvable subset is small (), limiting statistical power: the planner’s margins over the two retrieval baselines (pp) do not survive Holm correction, and the unaligned planner’s end-to-end advantage in RQ3 () rests on five effective successes. We report effect sizes and exact p-values rather than rounding into significance. Two planning modules (oracle availability, prior evidence) show no measurable contribution on this task set; this may reflect the task mix rather than a universal property. In the evaluated target-rule alignment, rule selection and binding were manually configured; the RQ2/RQ3 gains characterize a human-in-the-loop system extension, and automating the binding step remains open.
Construct validity.
Verification types are kept distinct: a static property change supports the checked property only and is never reported as a functional fix or runtime reproduction; a failed run is never a negative target state. FP-42’s partial property support is not counted as a success, and FP-32’s inclusion follows a frozen adjudication decision. Cost figures are known machine time: input preparation lacks timers for 48 of 120 input identities and manual configuration is untimed, so true end-to-end cost is higher than reported.
External validity.
The 60 pairs span 25 real-world projects plus the Ghera benchmark and are not 60 independent projects; the 100 tasks are not 100 independent root causes, and cluster correlation is handled only partially by leakage-group resampling. The 191 knowledge-base records carry provenance but are not independent mechanisms. We do not extrapolate beyond the covered domains, mechanism clusters, or tool set.
Reliability and knowledge cost.
The capability registry and profile annotations embody manual effort; two human annotators produced all reference labels with adjudication, and disagreements were retained rather than averaged away. Recommendations were frozen before scoring, and reference corrections triggered uniform recomputation.
6.3. Limitations and Future Work
Forty formal tasks remain without physical inputs, ten pairs remain outside checker scope, and two pairs (FP-39, FP-47) remain unresolved; all are retained in the reported denominators. A per-domain breakdown of recommendation accuracy is not part of the frozen analysis. Automating target-rule binding, instrumenting the untimed cost components, and extending the pool beyond the current domains and six tools are the natural next steps.
7. Conclusions
We studied validation planning for Android bug mechanisms: given a concrete mechanism, available artifacts, and constraints, select a verification plan with an independent verification method that can be executed. ABAF-VP profiles the target mechanism, filters and ranks candidate plans with mechanism, capability, oracle, and evidence predicates, aligns the selected rule to the concrete target, and verifies both artifact versions against an independently established reference.
On frozen evidence, the planner ranks a valid plan first for 10 of 13 jointly solvable tasks (P@1 = 76.9%), ahead of all four fair baselines, with mechanism matching as the dominant contributor (pp when removed). Target-rule alignment raises paired verification from 5/60 to 47/60 (78.3%), including 26/36 real-world pairs across 17 projects, and raises end-to-end task success from 5/100 to 47/100 at an increment of 30.274 s of known machine time (5.5%). The boundaries are integral to the result: the solvable subset is small, two baseline comparisons are not significant after correction, alignment bindings are manually configured, and 13 pairs end in documented capability limits or unresolved states. Within these bounds, the evidence supports validation planning as a workable path from Android bug-mechanism knowledge to executable, verifiable checks.
Author Contributions
Conceptualization, Z.X. and W.L.; methodology, Z.X.; software, Z.X.; validation, Z.X. and Y.W.; formal analysis, Z.X.; investigation, Z.X. and Y.W.; data curation, Z.X.; writing—original draft preparation, Z.X.; writing—review and editing, Y.W. and W.L.; visualization, Z.X.; supervision, W.L.; project administration, W.L. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
The data supporting the reported results are available on request from the corresponding author.
Acknowledgments
During the preparation of this study, the authors used Claude Code (Anthropic) for data screening and GPT (OpenAI) for the design of the paper draft. The manuscript was written and polished by the human authors, who have reviewed and edited all output and take full responsibility for the content of this publication.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Mitra, J.; Ranganath, V.P. Ghera: A Repository of Android App Vulnerability Benchmarks. In Proceedings of the 13th Int’l Conf. on Predictive Models and Data Analytics in Software Engineering (PROMISE), 2017; pp. 43–52. [Google Scholar] [CrossRef]
- Liu, Y.P.; Wang, J.; Wei, L.L.; Xu, C.; Cheung, S.C.; Wu, T.Y.; Yan, J.; Zhang, J. DROIDLEAKS: A Comprehensive Database of Resource Leaks in Android Apps. Empir. Softw. Eng. 2019, 24. [Google Scholar] [CrossRef]
- Xiong, Y.H.; Xu, M.Q.; Su, T.; Sun, J.L.; Wang, J.; Wen, H.; Pu, G.G.; He, J.F.; Su, Z.D. An Empirical Study of Functional Bugs in Android Apps. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023. [Google Scholar] [CrossRef]
- Chen, S.; Zhang, Y.X.; Fan, L.L.; Li, J.M.; Liu, Y. AUSERA: Automated Security Vulnerability Detection for Android Apps. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2022. [Google Scholar] [CrossRef]
- Mazuera-Rozo, A.; Escobar-Velásquez, C.; Espitia-Acero, J.; Vega-Guzmán, D.; Trubiani, C.; Linares-Vásquez, M.; Bavota, G. Taxonomy of Security Weaknesses in Java and Kotlin Android Apps. J. Syst. Softw. 2022, 187. [Google Scholar] [CrossRef]
- Pauck, F.; Bodden, E.; Wehrheim, H. Do Android Taint Analysis Tools Keep Their Promises? In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018. [Google Scholar] [CrossRef]
- Reaves, B.; Bowers, J.; Gorski, S.A., III; Anise, O.; Bobhate, R.; Cho, R.; Das, H.; Hussain, S.; Karachiwala, H.; Scaife, N.; et al. *droid: Assessment and Evaluation of Android Application Analysis Tools. ACM Comput. Surv. 2016, 49. [Google Scholar] [CrossRef]
- Arzt, S.; Rasthofer, S.; Fritz, C.; Bodden, E.; Bartel, A.; Klein, J.; Le Traon, Y.; Octeau, D.; McDaniel, P. FlowDroid: Precise Context, Flow, Field, Object-sensitive and Lifecycle-aware Taint Analysis for Android Apps. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014; pp. 259–269. [Google Scholar] [CrossRef]
- Li, L.; Bissyandé, T.F.; Papadakis, M.; Rasthofer, S.; Bartel, A.; Octeau, D.; Klein, J.; Traon, Y.L. Static Analysis of Android Apps: A Systematic Literature Review. Inf. Softw. Technol. 2017, 88. [Google Scholar] [CrossRef]
- Sadeghi, A.; Bagheri, H.; Garcia, J.; Malek, S. A Taxonomy and Qualitative Comparison of Program Analysis Techniques for Security Assessment of Android Software. IEEE Trans. Softw. Eng. 2017, 43. [Google Scholar] [CrossRef]
- Acar, Y.; Backes, M.; Bugiel, S.; Fahl, S.; McDaniel, P.; Smith, M. SoK: Lessons Learned from Android Security Research for Appified Software Platforms. In Proceedings of the IEEE Symposium on Security and Privacy (S&P), 2016; pp. 433–451. [Google Scholar] [CrossRef]
- Mahmud, J.; Pandey, S.; Silva, N.D.; Dipongkor, A.K.; Wu, J.; Chaparro, O.; Fazzini, M.; Moran, K. Automated Program Repair for UI-Centric Android Bugs: How Far Are We? Proc. ACM Softw. Eng. 2026, 3, 4143–4167. [Google Scholar] [CrossRef]
- Rua, R.; Ali, K. Towards LLMinating Performance Anti-Patterns in Android Apps. In Proceedings of the 2026 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW); IEEE, 2026; pp. 17–24. [Google Scholar] [CrossRef]
- Lachgar, M.; Kharroubi, F.; Ayoujil, A.; Achbarou, O.; Tourad, M.C. MobileSec: A microservices-based intelligent platform for automated Android APK vulnerability analysis and remediation. SoftwareX 2026, 35, 102897. [Google Scholar] [CrossRef]
- Chen, M.; Tan, T.; Pan, M.; Li, Y. PacDroid: A Pointer-Analysis-Centric Framework for Security Vulnerabilities in Android Apps. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE); IEEE, 2025; pp. 2803–2815. [Google Scholar] [CrossRef]
- Santos, G.; Bispo, J.; Mendes, A. Detecting Resource Leaks on Android with Alpakka. In Proceedings of the 18th ACM SIGPLAN International Conference on Software Language Engineering, 2025; ACM; pp. 199–211. [Google Scholar] [CrossRef]
- Zhou, Z.; Chen, R.; Chen, J.; Liu, Y.; Wei, L. Characterizing Bugs in Login Processes of Android Applications: An Empirical Study. In Proceedings of the 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC); IEEE, 2025; pp. 25–36. [Google Scholar] [CrossRef]
- Yang, H.; Huang, R.; Fang, Y.; Zhang, B.; Guo, J.; Wu, Z.; Mi, X. Okara: Detection and Attribution of TLS Man-in-the-Middle Vulnerabilities in Android Apps with Foundation Models. In Proceedings of the Lecture Notes in Computer Science; Information Security and Privacy; Springer Nature Singapore, 2026; pp. 225–270. [Google Scholar] [CrossRef]
- Robertson, S.; Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 2009, 3, 333–389. [Google Scholar] [CrossRef]
- Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019; pp. 3982–3992. [Google Scholar] [CrossRef]
- Android Developers. Improve Your Code with Lint Checks. 2025. Available online: https://developer.android.com/studio/write/lint (accessed on 18 October 2026).
- detekt: Static Code Analysis for Kotlin. 2025. Available online: https://detekt.dev.
- PMD: An Extensible Multilanguage Static Code Analyzer. 2024. Available online: https://pmd.github.io.
- SpotBugs: Find Bugs in Java Programs. 2024. Available online: https://spotbugs.github.io.
- Semgrep: Lightweight Static Analysis for Many Languages. 2025. Available online: https://semgrep.dev.
- Mobile Security Framework (MobSF). 2025. Available online: https://github.com/MobSF/Mobile-Security-Framework-MobSF.
Figure 1.
Motivation for target-specific paired verification. A real lifecycle-guard repair in jellyfin-android (FP-08, PR #194) illustrates the difference between selecting a rule and checking the changed mechanism. Code excerpts reproduce WebViewFragment.kt lines 187-192 (buggy) and 188-197 (fixed) with only the outermost block indent removed; long lines are wrapped for display and repeat their source number, the fixed ... elides the lifecycle-checked replaceFragment body on lines 192-195, and every number is the source line. The original Top-1 rule was CommitTransaction; the aligned source-level rule FRAGMENT_TXN_LIFECYCLE_GUARD checks the lifecycle guard in WebViewFragment.onSelectServer and records PRESENT on V_bug and ABSENT on V_fix. Across the fixed 60-pair pool, the frozen unbound B4 succeeds on 5 pairs and B4 with the aligned rule extension succeeds on 47. The bottom comparison concerns B4 before and after the extension; the category and keyword selectors are conceptual alternatives. The 43 rule-target gaps account for 78.2% of the 55 initially non-passing pairs.
Figure 1.
Motivation for target-specific paired verification. A real lifecycle-guard repair in jellyfin-android (FP-08, PR #194) illustrates the difference between selecting a rule and checking the changed mechanism. Code excerpts reproduce WebViewFragment.kt lines 187-192 (buggy) and 188-197 (fixed) with only the outermost block indent removed; long lines are wrapped for display and repeat their source number, the fixed ... elides the lifecycle-checked replaceFragment body on lines 192-195, and every number is the source line. The original Top-1 rule was CommitTransaction; the aligned source-level rule FRAGMENT_TXN_LIFECYCLE_GUARD checks the lifecycle guard in WebViewFragment.onSelectServer and records PRESENT on V_bug and ABSENT on V_fix. Across the fixed 60-pair pool, the frozen unbound B4 succeeds on 5 pairs and B4 with the aligned rule extension succeeds on 47. The bottom comparison concerns B4 before and after the extension; the category and keyword selectors are conceptual alternatives. The 43 rule-target gaps account for 78.2% of the 55 initially non-passing pairs.

Figure 2.
Four-stage ABAF-VP system overview. (1) Mechanism profiling structures a task into target mechanism, trigger condition and required evidence. (2) B4 ranks verification plans from a 176-candidate pool using mechanism, capability, oracle and evidence information. (3) ALIGN binds a target-specific rule to a concrete file, method or XML component, producing an executable verification proposition. In the evaluated extension, rule selection and target binding were manually configured. (4) Both artifacts are checked and a differential oracle evaluates whether the target condition is present on the buggy artifact and absent on the fixed artifact; the recorded state and timing are retained. The ABAF-21 knowledge base supplies Stages 1 and 2. Source-level property evidence retains its recorded scope and is not a runtime-failure claim.
Figure 2.
Four-stage ABAF-VP system overview. (1) Mechanism profiling structures a task into target mechanism, trigger condition and required evidence. (2) B4 ranks verification plans from a 176-candidate pool using mechanism, capability, oracle and evidence information. (3) ALIGN binds a target-specific rule to a concrete file, method or XML component, producing an executable verification proposition. In the evaluated extension, rule selection and target binding were manually configured. (4) Both artifacts are checked and a differential oracle evaluates whether the target condition is present on the buggy artifact and absent on the fixed artifact; the recorded state and timing are retained. The ABAF-21 knowledge base supplies Stages 1 and 2. Source-level property evidence retains its recorded scope and is not a runtime-failure claim.

Figure 3.
Recommendation quality and planning-module ablation (RQ1). The five-method comparison uses the existing 100-task evaluation; P@1, Success@3 and MRR are scored on the jointly solvable subset T = 13. (a) Method comparisons; MRR is multiplied by 100. The dark outline identifies B4. Brackets denote significant Holm-adjusted paired comparisons on P@1: B4 versus B0 (p = 0.0078) and B3 (p = 0.0117). B4 versus B1 and B2 has p = 0.0625 and 0.0703, respectively. (b) Changes in P@1 after removing a module, relative to FULL. Only point estimates and Holm-adjusted exact permutation p-values are reported; no confidence interval is drawn on the T = 13 subset. Removing mechanism matching reduces P@1 by 69.2 percentage points (Holm p = 0.0117). Removing oracle adaptation changes neither P@1 nor Success@3. Removing prior evidence raises P@1 by 7.7 points while reducing Success@3 by 7.7 points; the P@1 difference is not significant. The FULL P@1 value is a reference annotation, not a line on the difference axis.
Figure 3.
Recommendation quality and planning-module ablation (RQ1). The five-method comparison uses the existing 100-task evaluation; P@1, Success@3 and MRR are scored on the jointly solvable subset T = 13. (a) Method comparisons; MRR is multiplied by 100. The dark outline identifies B4. Brackets denote significant Holm-adjusted paired comparisons on P@1: B4 versus B0 (p = 0.0078) and B3 (p = 0.0117). B4 versus B1 and B2 has p = 0.0625 and 0.0703, respectively. (b) Changes in P@1 after removing a module, relative to FULL. Only point estimates and Holm-adjusted exact permutation p-values are reported; no confidence interval is drawn on the T = 13 subset. Removing mechanism matching reduces P@1 by 69.2 percentage points (Holm p = 0.0117). Removing oracle adaptation changes neither P@1 nor Success@3. Removing prior evidence raises P@1 by 7.7 points while reducing Success@3 by 7.7 points; the P@1 difference is not significant. The FULL P@1 value is a reference annotation, not a line on the difference axis.

Figure 4.
Paired-validation outcome and the target-rule extension (RQ2). The fixed task pool comprises 36 real-world pairs from 25 projects and 24 benchmark pairs. Before ALIGN, unbound B4 verifies 5 pairs (direct passes); 53 pairs are not applicable, of which 43 are rule-target gaps and 10 lie outside the checker scope for the changed mechanism, and 2 are unresolved. ALIGN adds target-specific rules and bindings for the rule-target gaps: 42 of those 43 pairs become verified (17 benchmark, 25 real-world). Together with the 5 direct passes this yields 47/60 (78.3%; 95% leakage-group bootstrap CI [67.2%, 88.1%], 10,000 resamples). After ALIGN, 3 pairs remain unverified (FP-39 and FP-47 unresolved, FP-42 partial property support) and the 10 out-of-scope pairs remain out of scope. Success is 26/36 on real-world pairs, spanning 17 successful projects, and 21/24 on benchmark pairs. Pair-level transitions are 53 not applicable to 43 rule-target gaps plus 10 inapplicable, then 42 to verified, 1 to partial support and 10 unchanged; 5 direct passes stay verified and 2 unresolved stay unresolved. Band heights are proportional to these counts. The grouping is for target-specific validation, not an evaluation of alert populations. Results retain the frozen FP-32 closeout decision.
Figure 4.
Paired-validation outcome and the target-rule extension (RQ2). The fixed task pool comprises 36 real-world pairs from 25 projects and 24 benchmark pairs. Before ALIGN, unbound B4 verifies 5 pairs (direct passes); 53 pairs are not applicable, of which 43 are rule-target gaps and 10 lie outside the checker scope for the changed mechanism, and 2 are unresolved. ALIGN adds target-specific rules and bindings for the rule-target gaps: 42 of those 43 pairs become verified (17 benchmark, 25 real-world). Together with the 5 direct passes this yields 47/60 (78.3%; 95% leakage-group bootstrap CI [67.2%, 88.1%], 10,000 resamples). After ALIGN, 3 pairs remain unverified (FP-39 and FP-47 unresolved, FP-42 partial property support) and the 10 out-of-scope pairs remain out of scope. Success is 26/36 on real-world pairs, spanning 17 successful projects, and 21/24 on benchmark pairs. Pair-level transitions are 53 not applicable to 43 rule-target gaps plus 10 inapplicable, then 42 to verified, 1 to partial support and 10 unchanged; 5 direct passes stay verified and 2 unresolved stay unresolved. Band heights are proportional to these counts. The grouping is for target-specific validation, not an evaluation of alert populations. Results retain the frozen FP-32 closeout decision.

Figure 5.
End-to-end verification effectiveness and known machine time (RQ3). Bars show verified tasks out of all 100 evaluation tasks; the left axis counts verified tasks, not execution volume, so baselines that run their tools but reach no target-specific verdict have short bars. The orange line shows known machine time, including recorded input preparation, on a separate zero-based axis; the short horizontal marks beside the baseline points mark the baseline methods (B0–B3), whose time buys tool runs rather than target-specific judgments. Extending B4 with aligned target rules and verification coverage raises successes from 5 to 47, adding 42 tasks. Recorded machine time rises from 552.336 to 582.610 seconds, an increment of 30.274 seconds (5.5%), or 0.721 seconds per added verified task. The five new paired comparisons are significant after Holm correction (p rounded to 5 × 10−5 for each). This comparison evaluates a system extension with different target-rule coverage. Time values exclude untimed manual binding and development; preparation timers are available for 72/120 input identities. The figure therefore reports known machine time, not full deployment cost.
Figure 5.
End-to-end verification effectiveness and known machine time (RQ3). Bars show verified tasks out of all 100 evaluation tasks; the left axis counts verified tasks, not execution volume, so baselines that run their tools but reach no target-specific verdict have short bars. The orange line shows known machine time, including recorded input preparation, on a separate zero-based axis; the short horizontal marks beside the baseline points mark the baseline methods (B0–B3), whose time buys tool runs rather than target-specific judgments. Extending B4 with aligned target rules and verification coverage raises successes from 5 to 47, adding 42 tasks. Recorded machine time rises from 552.336 to 582.610 seconds, an increment of 30.274 seconds (5.5%), or 0.721 seconds per added verified task. The five new paired comparisons are significant after Holm correction (p rounded to 5 × 10−5 for each). This comparison evaluates a system extension with different target-rule coverage. Time values exclude untimed manual binding and development; preparation timers are available for 72/120 input identities. The figure therefore reports known machine time, not full deployment cost.

Table 1.
Formal names of the compared methods, the alignment module, and the planning ablations. The compact labels are used in tables and figures.
Table 1.
Formal names of the compared methods, the alignment module, and the planning ablations. The compact labels are used in tables and figures.
| Label | Formal name | Definition |
|---|---|---|
| B0 | Category mapping | Legacy taxonomy category mapped to candidate tools; fixed-ID tie-break |
| B1 | BM25 retrieval | BM25 over tool/rule documentation; query is the public task description |
| B2 | Embedding similarity | Sentence-embedding cosine similarity over the same corpus |
| B3 | Capability-only matching | Artifact/language/platform/runtime compatibility only |
| B4 | ABAF-VP (full planner) | Mechanism- and constraint-driven planning, all modules on |
| ALIGN | Target-rule alignment | Stage 3: binds a rule to a concrete file/method/component |
| A1 | −Mechanism ablation | B4 without mechanism compatibility filtering and ranking |
| A2 | −Oracle ablation | B4 without oracle-availability filtering and ranking |
| A3 | −Evidence ablation | B4 without prior-evidence strength ranking |
Table 2.
The ABAF-21 knowledge base: 21 mechanism categories and 191 root-cause records. One representative root cause is shown per category; the full record list retains provenance per entry.
Table 2.
The ABAF-21 knowledge base: 21 mechanism categories and 191 root-cause records. One representative root cause is shown per category; the full record list retains provenance per entry.
| ID | Category | #RCs | Representative root cause |
|---|---|---|---|
| C01 | Activity & Fragment Lifecycle | 7 | State loss / transaction loss during commit |
| C02 | Callback Bugs | 3 | Unhandled callbacks / missing callbacks |
| C03 | General Programming Errors | 23 | Missing or incorrectly specified conditional checks |
| C04 | Android-Related Errors | 10 | Android framework API misuse |
| C05 | Third-Party Library Bugs | 5 | Library bugs / limitations |
| C06 | Data Storage Security | 12 | External / internal storage exposure |
| C07 | Data Encryption Security | 10 | Insecure AES / DES encryption |
| C08 | Data Access Security | 17 | Exported Activity / Service / Provider / Receiver |
| C09 | Communication Security | 9 | Invalid certificate / hostname verification |
| C10 | Configuration Security | 6 | Backup / debugging enabled in the manifest |
| C11 | App State & UI Bugs | 16 | UI operations outside the main thread |
| C12 | API & Compatibility Bugs | 9 | API misuse; incompatible APIs |
| C13 | Resource Management | 16 | Missing / incorrect string resources |
| C14 | Permission-Based Security | 11 | Unimplemented runtime permissions (API 23+) |
| C15 | Device Orientation Bugs | 1 | Incorrect handling of orientation changes |
| C16 | Inter-Component Communication | 5 | Nonexistent targets; null / missing intents |
| C17 | Memory Bugs | 9 | Use-after-free |
| C18 | Process & Thread Bugs | 6 | Concurrency; race conditions |
| C19 | Back & Up Navigation | 2 | Incorrect back/up navigation implementation |
| C20 | WebView-Related Bugs | 11 | Misaligned WebView lifecycles |
| C21 | Performance Issues | 3 | Lengthy operations on the main thread |
| Total | 191 | ||
Table 4.
Recommendation accuracy of the five methods on the tasks that have a confirmed common solution. P@1 and Success@3 are reported as counts and percentages; MRR is the mean reciprocal rank over the same tasks. The last two columns give B4 minus each baseline on P@1 (percentage points) and the Holm-adjusted exact permutation p-value over the four baseline comparisons. B0: category mapping; B1: BM25; B2: embedding similarity; B3: capability-only compatibility filtering; B4: full method.
Table 4.
Recommendation accuracy of the five methods on the tasks that have a confirmed common solution. P@1 and Success@3 are reported as counts and percentages; MRR is the mean reciprocal rank over the same tasks. The last two columns give B4 minus each baseline on P@1 (percentage points) and the Holm-adjusted exact permutation p-value over the four baseline comparisons. B0: category mapping; B1: BM25; B2: embedding similarity; B3: capability-only compatibility filtering; B4: full method.
| P@1 | Success@3 | B4 − baseline | |||||
|---|---|---|---|---|---|---|---|
| Method | (%) | (%) | MRR | P@1 (pp) | Holm p | ||
| B0 | 0 | 0.0% | 0 | 0.0% | 0.0145 | 0.0078 | |
| B1 | 4 | 30.8% | 11 | 84.6% | 0.5526 | 0.0625 | |
| B2 | 4 | 30.8% | 9 | 69.2% | 0.5253 | 0.0703 | |
| B3 | 1 | 7.7% | 3 | 23.1% | 0.1949 | 0.0117 | |
| B4 | 10 | 76.9% | 12 | 92.3% | 0.8429 | – | – |
Table 5.
Planning-module ablation on the same tasks. Each row removes one module from the full configuration (FULL): A1 is mechanism matching, A2 is oracle adaptation, and A3 is prior evidence. The last three columns give FULL minus the ablated configuration on P@1, the 95% group-bootstrap confidence interval of that difference, and the Holm-adjusted exact permutation p-value over the three comparisons.
Table 5.
Planning-module ablation on the same tasks. Each row removes one module from the full configuration (FULL): A1 is mechanism matching, A2 is oracle adaptation, and A3 is prior evidence. The last three columns give FULL minus the ablated configuration on P@1, the 95% group-bootstrap confidence interval of that difference, and the Holm-adjusted exact permutation p-value over the three comparisons.
| P@1 | Success@3 | FULL − configuration | ||||||
|---|---|---|---|---|---|---|---|---|
| Configuration | (%) | (%) | MRR | P@1 (pp) | 95% CI (pp) | Holm p | ||
| FULL | 10 | 76.9% | 12 | 92.3% | 0.8429 | – | – | – |
| −A1 | 1 | 7.7% | 2 | 15.4% | 0.1652 | 0.0117 | ||
| −A2 | 10 | 76.9% | 12 | 92.3% | 0.8429 | 0.0 | 1.0000 | |
| −A3 | 11 | 84.6% | 11 | 84.6% | 0.8675 | 1.0000 | ||
Table 6.
Target-specific paired verification over the 60 pairs (36 real fixes and 24 benchmark cases). (a) Successful pairs for Unbound B4 and for B4 + ALIGN, i.e., before and after the target-rule alignment correction; a success requires a rule–target correspondence, bilateral execution, decidable evidence on both sides, and agreement with the independent reference. (b) Disposition of the 60 pairs under B4 + ALIGN: 43 pairs had a rule–target mismatch, of which 42 became MATCHED and one was adjudicated as PROPERTY_SUPPORT and is not counted as a success.
Table 6.
Target-specific paired verification over the 60 pairs (36 real fixes and 24 benchmark cases). (a) Successful pairs for Unbound B4 and for B4 + ALIGN, i.e., before and after the target-rule alignment correction; a success requires a rule–target correspondence, bilateral execution, decidable evidence on both sides, and agreement with the independent reference. (b) Disposition of the 60 pairs under B4 + ALIGN: 43 pairs had a rule–target mismatch, of which 42 became MATCHED and one was adjudicated as PROPERTY_SUPPORT and is not counted as a success.
| Metric | Unbound B4 | B4 + ALIGN | |
|---|---|---|---|
| Successful pairs | 5/60 (8.3%) | 47/60 (78.3%) | +42 |
| Real cases | 1/36 (2.8%) | 26/36 (72.2%) | +25 |
| Benchmark cases | 4/24 (16.7%) | 21/24 (87.5%) | +17 |
| Real projects covered | 1 | 17 | +16 |
| Disposition of the 60 pairs | n | Share | |
| Previously successful, retained | 5 | 8.3% | |
| Rule–target mismatch, 43 pairs | |||
| corrected to MATCHED | 42 | 70.0% | |
| PROPERTY_SUPPORT (adjudicated, not counted) | 1 | 1.7% | |
| NOT_APPLICABLE (outside implemented checker scope) | 10 | 16.7% | |
| UNRESOLVED | 2 | 3.3% | |
| Total | 60 | 100.0% | |
Table 7.
End-to-end execution and known machine time for the five frozen methods and for the B4+ALIGN extension. Top-1 emitted counts the plans returned per 100 tasks, with abstentions in parentheses; the 40 tasks without a physical input remain in the /100 denominator. Bilateral execution and verified success are reported over the 60 tasks that have a physical input. A task without input cannot succeed, so the /60 success count equals the /100 count for every method. Known machine time covers timed machine work only (tool execution, input preparation, rule checking, and AST parsing) and excludes manual rule binding and development. The p-value refers to the B4+ALIGN versus B4 comparison over 89 leakage clusters with Holm correction over the five new comparisons.
Table 7.
End-to-end execution and known machine time for the five frozen methods and for the B4+ALIGN extension. Top-1 emitted counts the plans returned per 100 tasks, with abstentions in parentheses; the 40 tasks without a physical input remain in the /100 denominator. Bilateral execution and verified success are reported over the 60 tasks that have a physical input. A task without input cannot succeed, so the /60 success count equals the /100 count for every method. Known machine time covers timed machine work only (tool execution, input preparation, rule checking, and AST parsing) and excludes manual rule binding and development. The p-value refers to the B4+ALIGN versus B4 comparison over 89 leakage clusters with Holm correction over the five new comparisons.
| Method | Top-1 emitted (/100) |
Bilateral executed (/60) |
Verified success (/100) |
Verified success (/60) |
Known machine time (s) |
|---|---|---|---|---|---|
| B0 | 100 | 59 | 0 | 0 | 602.271 |
| B1 | 100 | 45 | 1 | 1 | 637.671 |
| B2 | 100 | 46 | 1 | 1 | 590.434 |
| B3 | 76 (24 abstained) | 60 | 0 | 0 | 410.976 |
| B4 | 98 (2 abstained) | 60 | 5 | 5 | 552.336 |
| B4+ALIGN | 98 (2 abstained) | 60 | 47 | 47 | 582.610 |
| (B4+ALIGN − B4) | 0 | 0 | +42 | +42 | +30.274 (5.5%) |
| Increment of B4+ALIGN over B4 | Value | ||||
| Real cases | 1/36 → 26/36 (+25) | ||||
| Benchmark cases | 4/24 → 21/24 (+17) | ||||
| Known time per new success | 0.721 s | ||||
| Holm-adjusted p | |||||
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.