Evaluating Security Claims in LLM Systems: A Survey of Adversarial Attacks and Defenses

1. Introduction

Large language models are increasingly embedded in applications that retrieve private documents, maintain memory, invoke tools, delegate tasks to other agents, and share computing resources across users. Indirect prompt injection demonstrates how untrusted content can become control input [1,2], and agent benchmarks show that the resulting influence can extend to tool calls and external state [3,4]. Retrieval poisoning, model backdoors, and serving side channels further extend the attack surface to persistent knowledge, software supply chains, and shared infrastructure [5,6,7].

The literature already includes extensive surveys and taxonomies of LLM security. Broad reviews organize attacks, defenses, alignment, metrics, and guardrails [8]. More focused studies examine jailbreak robustness and operations [9,10], multimodal jailbreaking [11], prompt-pipeline defenses and evaluation [12,13], model extraction [14], agentic systems [15,16], supply-chain security [17], and operational deployment [18,19]. Collectively, these surveys provide complementary views of attack and defense mechanisms, system components, and lifecycle stages.

This survey covers adversarial research on LLMs and related systems, including attacks on model behavior, protected data, retrieval, memory, tools, computational resources, and infrastructure. Foundational adversarial-learning studies are included when they directly inform modern LLM security. Generic hallucination, bias, misinformation, ordinary software faults, and beneficial uses of LLMs for cyber defense fall outside the scope unless they involve an explicit adversarial mechanism. The underlying collection is curated and supplemented with targeted literature, so the analysis provides a conceptual synthesis rather than a population-level or PRISMA-style systematic review.

Building on these surveys, we examine how experimental conditions bound the resulting security claims. Defense studies often report attack success rate (ASR) against a known attack set or refusal rates, but output-only measures may capture only part of the relevant security consequence. A harmful answer may establish a policy failure. In contrast, refusal rates do not establish whether a tool call was authorized, a stored memory was poisoned, or a shared cache disclosed a prompt [4,7,20]. Results also depend on the prompt distribution, model family, query budget, detector, and evaluation oracle. Defense-aware re-evaluations have bypassed protections that performed well under their original static tests [21,22]. These findings do not imply that the defenses are generally ineffective, but show that the meaning of a result depends on the evaluated attacker, system configuration, information and authority available to the defense, and measured consequence. We therefore ask:

Under exactly what adversarial conditions has an LLM security mechanism been shown to remain effective, and what system consequence does that evidence establish?

To answer this question, we build on multidimensional threat models for adversarial machine learning [23,24,25] to develop a common procedure for interpreting experimental evidence across the adversarial settings covered by this survey.

The paper makes four contributions:

1.

A paired threat and defense model: The threat tuple records entry and control surface, attacker knowledge and access, adaptivity, objective, persistence, impact, and budget. The defense tuple records the control’s observable information, security function, and enforceable response. Together, the tuples specify the evaluated attack conditions and defensive capabilities.

2.

A method for bounding claims: An attack-defense assurance matrix records whether the evidence addresses each relevant condition without assigning an overall score to the defense. A bounded claim grammar specifies the attacker, system version, consequence, cost, and oracle associated with a result, while an evidence ladder from L0 to L5 specifies empirical evidence requirements for a fixed security claim and threat model.

3.

A cross-stack synthesis: We apply the method to a curated collection of studies. The synthesis identifies seven recurring patterns, including mismatches between defensive observation and enforcement, differences between static and defense-aware evidence, and asymmetric attacker and defender costs.

4.

A practical procedure: An eight-step procedure and a worked comparison guide researchers and evaluators in determining the scope of support provided by a reported result.

2. Threat and Defense Models

We represent an adversarial setting by the threat tuple $\mathcal{T}$ and its controls by the defense tuple $\mathcal{D}$:

$$\mathcal{T}=\langle E,K,A,O,P,I,B\rangle ,\mathcal{D}=\langle V,F,R\rangle .$$

(1)

The threat tuple specifies the entry and control surface E, attacker knowledge and access K, adaptivity A, objective O, persistence P, impact I, and budget B. The defense tuple specifies the observable view V, security function F, and enforceable response R. Figure 1 illustrates their relationship. These tuples do not make all experiments formally comparable. Instead, they expose assumptions needed to determine whether an evaluated attacker and defense support the stated security claim. Table 1 summarizes the question associated with each dimension and the interpretive error it is intended to prevent.

2.1. Entry and Control Surfaces

The entry dimension contains named surfaces without imposing an order. Direct interaction $E_{\mathrm{dir}}$ includes prompts, suffixes, and multi-turn conversation. Externally controlled context $E_{\mathrm{ctx}}$ includes webpages, emails, files, multimodal content, tool outputs, and conflicts introduced during prompt assembly. Persistent application state $E_{\mathrm{state}}$ includes retrieval-augmented generation (RAG) corpora, vector indexes, memory, caches, and logs. The model-production path $E_{\mathrm{build}}$ covers pretraining, fine-tuning, preference data, adapters, model editing, and distribution hubs. Delegated execution $E_{\mathrm{exec}}$ includes tools, APIs, code interpreters, other agents, and protocols, while the service substrate $E_{\mathrm{infra}}$ includes schedulers, batching, KV and prefix caches, GPUs, network traces, and tenant-isolation mechanisms. A campaign may traverse several surfaces, and the labels imply neither sequence nor severity.

This decomposition avoids reducing every attack to a “prompt” or “model” category. An injected email is expressed in language but enters through $E_{\mathrm{ctx}}$ [1], whereas PoisonedRAG changes $E_{\mathrm{state}}$ without modifying the current request [5]. A backdoored tool-use model spans $E_{\mathrm{build}}$ and $E_{\mathrm{exec}}$ [26], while timing and cache attacks operate at $E_{\mathrm{infra}}$ and may disclose prompts without altering model output [7,27,28].

2.2. Attacker Capability and Adaptation

Knowledge and access define a capability profile rather than a scalar level. The profile records repeated prompt and output queries ($k_q$) and observation of sessions or state ($k_o$). Model access may encompass a model family, surrogate, or score ($k_s$), or extend to gradients, activations, parameters, or the training procedure ($k_g$). Other components describe control over data stores or deployment artifacts ($k_d$), control over tools or the external environment ($k_t$), and infrastructure placement or co-residency ($k_i$). Each component retains its own scope and constraints because, for example, gradient access does not confer write access to a corpus, and co-residency does not reveal model weights. Capability profiles are therefore only partially ordered.

The label "black-box" describes what information an attacker can directly access, but limited internal access does not necessarily weaken the attacker. Repeated queries can compensate for this limitation by providing feedback that supports tree-based search, fuzzing, best-of-N sampling over N trials, reinforcement learning, or human-guided refinement [29,30,31]. Attackers may also reduce uncertainty through surrogate models or by identifying the deployed guardrail before tailoring an attack [32,33]. Therefore, evaluations should report the resources available to the attacker, including query and model-call budgets, wall-clock effort, parallelism, surrogate access, and human intervention.

Adaptivity is a separate property of how the attack develops. It has several modes that can coexist and do not form a ranking:

• $a_{\mathrm{static}}$: a fixed attack set is replayed without feedback.
• $a_{\mathrm{iter}}$: prompts or actions are revised using target responses.
• $a_{\mathrm{transfer}}$: attacks are optimized on a surrogate, a related model, or a previous version.
• $a_{\mathrm{def}}$: the attacker knows or infers the defense and optimizes against its decision rule.
• $a_{\mathrm{joint}}$: the objective combines target behavior, defense evasion, cost, and semantic constraints.
• $a_{\mathrm{state}}$: the attacker reacts across turns, memory, tools, retries, or operational telemetry.

Many automated jailbreaks are iterative without being defense-aware. Robust Prompt Optimization (RPO) and prompt adversarial tuning incorporate adversarial objectives into defense optimization [34,35], and DataSentinel trains a detector through a minimax game against adaptive prompt injection [36]. More demanding re-evaluations use $a_{\mathrm{def}}$ or $a_{\mathrm{joint}}$, where the attacker explicitly incorporates the defense into the attack objective or search procedure. For example, the attacker may optimize gradients through the defense, use its outputs as reinforcement-learning rewards, and search for inputs that achieve the attack goal while evading the defense [21,22]. Stateful adaptation is a distinct capability rather than simply a stronger form of access: an attacker may rely only on black-box feedback but exploit information carried across repeated interactions, such as persistent session state, rate-limit behavior, cached responses, or timing differences, to refine later attack attempts.

2.3. Objectives, Persistence, Budgets, and Impact

Attack objectives include policy bypass, integrity violation, confidentiality loss, privacy inference, availability loss, unauthorized capability use, control-flow manipulation, external action, model theft, and persistent compromise. Their persistence may be limited to one turn or extend across multiple turns, sessions, memories, corpora, model versions, users, or the supply chain. Backdoors and retrieval poisoning are particularly important because insertion and activation are separated in time [5,37,38]. Similarly, multi-turn and trajectory attacks accumulate influence through interactions that single-turn benchmarks cannot represent [39,40].

Budget B includes query count, tokens, latency, compute, model access, human time, poisoning volume, and acceptable detectability. Best-of-N and many-shot attacks show how query budget and context capacity can increase attacker capability: repeated sampling provides more opportunities to obtain a successful output, while longer contexts let the attacker supply many examples or instructions in a single interaction [41,42]. Denial-of-service attacks exploit a different resource asymmetry: relatively inexpensive requests trigger disproportionately costly inference or processing on the defender’s side. This exhausts compute, memory, or service capacity [43,44,45].

Impact describes what the attack ultimately causes, such as incorrect model behavior, information disclosure, service disruption, financial loss, or physical harm. Figure 2 distinguishes six evidence classes. A detector or evaluator response without target compromise is $I_{\mathrm{sig}}$, behavioral deviation is $I_{\mathrm{beh}}$, and a safety or policy violation in generated content is $I_{\mathrm{pol}}$. Unauthorized disclosure, corrupted retrieval, or persistent state modification constitutes $I_{\mathrm{asset}}$, whereas an unauthorized tool selection or argument constitutes $I_{\mathrm{act}}$. External actions, cross-agent propagation, cross-user effects, and measurable service failures fall under $I_{\mathrm{ext}}$. These classes identify evidentiary boundaries rather than levels of harm because, for example, disclosure of sensitive data may be more consequential than a reversible tool action. This classification prevents harmful-text ASR from being interpreted as evidence of action-level security. AgentDojo evaluates task utility and security cases in tool environments [4]. Task Shield and MELON check whether actions remain aligned with user intent [46,47], while IPIGuard constrains execution via a tool-dependency graph [48]. Hence, the defended consequence determines both the relevant oracle and the required enforcement mechanism.

2.4. Defense View, Function, and Response

Defenses are often described by deployment location, such as an input filter, output filter, guard model, or gateway, but location alone does not determine whether a defense can stop a particular attack. The examples below distinguish limitations in V from limitations in R. A user-input filter does not see a document retrieved later, while an output moderator observes text only after a tool call may have executed. A model-internal detector may observe hidden states without knowing the requesting principal or protected object. A detector may alert or abstain, whereas an authorization layer may deny an action. When a detector lacks the authority to stop the action, another component remains responsible for enforcement. The roles represented by F include prevention, separation, detection, verification, authorization, hardening, repair, isolation, resource control, or recovery.

Two examples illustrate this distinction. A semantic guard can inspect the meaning and intent of natural-language inputs, but it cannot by itself determine whether a user is authorized to access a particular object. In contrast, an object-level allowlist may consider only a tool name, user identity, and object identifier, yet it can directly block access to unauthorized objects even if an injected instruction has manipulated the model. Neither control is universally stronger: the semantic guard better detects harmful or suspicious language, whereas the allowlist enforces a concrete authorization boundary. Section 4 applies this defense model to specific defense mechanisms, and Section 5 compares it with the threat tuple.

2.5. Composition, Equivalence, and Partial Ordering

The dimensions of $\mathcal{T}$ are distinct for analysis but are not assumed to be statistically independent. Identical text can instantiate different threats because a user-supplied sentence enters at $E_{\mathrm{dir}}$, a retrieved webpage at $E_{\mathrm{ctx}}$, and a stored copy also occupies $E_{\mathrm{state}}$. The latter cases cross application trust boundaries and can reach other users. Conversely, different encodings can be security-equivalent when access, objective, persistence, and impact do not change, which prevents typographic variation from multiplying attack classes.

Attacker access cannot be ranked on a single scale because different access types enable different actions. Gradient access can help optimize an adversarial suffix but does not allow the attacker to modify a retrieval corpus. Write access to that corpus can make an attack persist across future queries without revealing model logits or gradients. Likewise, access to tools or the external environment can allow an attacker to cause direct harm without knowing the model’s internals. Comparisons should therefore record the attacker’s specific privileges rather than reducing them to white-box, gray-box, or black-box labels.

Threat tuples also compose because an attacker can poison a corpus at $E_{\mathrm{state}}$, wait for retrieval into $E_{\mathrm{ctx}}$, induce a tool call at $E_{\mathrm{exec}}$, and leave a trace in a cache or audit log at $E_{\mathrm{infra}}$. Retaining the complete path instead of selecting a maximum class reveals where controls can intervene. A broad security claim should therefore state which feasible paths its controls interrupt and which residual consequences isolation or recovery can bound.

Table 2 makes the claim boundary concrete. A classifier can contribute to an indirect-injection defense, but its output does not show that no action occurred. A cryptographic protocol can establish confidentiality under a leakage model, but it cannot show that the requesting principal was authorized. An attack-resistant model can shut one path to harm, yet the deployment stays insecure if a tool or retrieval layer breaks the invariant on its own. For this reason we evaluate matched attack-defense pairs and do not rank techniques globally.

3. Attack Taxonomy Through the Threat Model

Figure 3 organizes the attack families into a tree with representative mechanisms, carriers, or targets as subbranches, while Table 3 maps the families to the threat-model dimensions that distinguish them. The categories intentionally overlap because, for example, a multimodal image can carry an indirect injection into an agent, with modality serving as the entry representation and agent hijacking as the impact path. Each subsection below retains only work that shows a distinct mechanism, exposes an assumption mismatch, or supplies evidence used in later sections.

3.1. Jailbreaks as Search under Access and Budget

Jailbreak attacks can be viewed as attempts to find an input or interaction strategy that causes the model to violate its intended restrictions, subject to the attacker’s available access and query budget. Early work linked such failures to conflicts between a model’s helpfulness and safety objectives [68], while universal suffix attacks showed that carefully optimized token sequences could transfer across prompts and models [49]. Later black-box attacks improved jailbreaks by repeatedly testing and refining prompts, attack strategies, or multi-turn conversations based only on model responses [29,30,31,69]. Other work showed that the decoding process itself can be manipulated to increase the chance of unsafe outputs [70]. Query budget and context length also affect attacker capability: best-of-N attacks increase the chance of success by generating many candidate outputs, whereas many-shot attacks use long contexts to provide many examples or instructions [41,42]. These methods therefore differ not only in prompt wording, but also in what the attacker can observe, how many attempts they can make, and how they adapt subsequent attempts.

Papers in this family assume different signals and costs. BEAST replaces gradients with beam search to lower GPU cost [71]. Query-efficient black-box and fuzzing methods use target feedback [72,73,74]. AdvPrefix, reparameterization, and ASTRA change the objective, transfer, or reuse of strategies [75,76,77]. Equal prompt counts do not make these comparable, because they hide surrogate calls, parallel samples, and retained strategy libraries. A defensible evaluation states the objective, stopping rule, target and surrogate calls, restart policy, decoding settings, and whether failed samples stay in the denominator.

Multi-turn attacks turn conversation history into attack state. Few-shot demonstrations, analogy chains, Crescendo-style escalation, dialogue-aware tree search, risk-aware planning, and trajectory optimization all spread influence over turns [39,78,79,80,81,82]. An evaluation should keep failed turns, refusal feedback, truncation, and service-side rate limits. Collapsing a successful trajectory into one final prompt helps transfer analysis, but it no longer measures the same persistence or observation process.

Prompt-content attacks can use very different mechanisms. Some attacks manipulate meaning through role-playing, fictional scenarios, persuasion, complex reasoning setups, or by reframing the jailbreak as an inference-time alignment problem [83,84,85,86,87]. Others change how the instruction is represented, for example by encoding it in ASCII art, ciphers, transliteration, low-resource languages, or carefully rewritten text designed to evade detection [88,89,90,91,92]. These differences help describe how an attack is expressed, but they do not necessarily imply different attacker capabilities. The more important questions are whether the attack requires different access, feedback, or adaptation, and whether it produces a different impact. Taxonomies based only on prompt form can become outdated as models change, whereas attacker capability and the feedback process remain more stable dimensions for comparison.

Transfer attacks provide evidence about whether an attack developed under one access setting also works under another. Papernot et al., for example, considered attackers that can query a target model for predicted labels but obtain gradients only from a locally trained substitute model [93]. The attack is optimized on the substitute and then tested on the target, so successful transfer shows that the adversary does not need direct gradient access to the target. The same distinction matters for LLMs. An attack optimized on an open-weight model can be evaluated against a closed API to test whether it transfers across models. If it succeeds, this may reflect a weakness shared by both models, although it can also result from properties of the evaluation setup that both models happen to share. If it fails, however, the result shows only that this particular attack did not transfer; it does not show that the closed model would resist an attack optimized directly against its own feedback. Deployment constraints must also be reported. For example, a many-shot jailbreak that requires more context, queries, or cost than a deployed service permits demonstrates a model vulnerability under those experimental conditions, but not necessarily a practical attack against that service. Attack evaluations should therefore report both the access used to construct the attack and the resources required to execute it.

Mechanistic studies examine where safety-related behavior is represented inside the model and how changes to internal components affect that behavior. For example, pruning or low-rank edits can alter safety responses, while harmful and benign prompts can produce partially separable hidden-state patterns. Cross-layer analyses can identify internal components or processing paths associated with unsafe behavior [94,95,96,97]. These internal patterns can also be used directly in attacks or defenses. In activation editing, the adversary modifies hidden representations to strengthen or weaken refusal behavior. Representation-space attacks search for internal states that induce unsafe outputs, and compliance-direction methods identify directions in activation space associated with following rather than refusing harmful requests [98,99,100]. However, separating harmful from benign inputs in representation space is not enough to demonstrate robustness. The model may still fail against new attack objectives or cause harmful actions when connected to tools or external systems [101].

Benchmarks have improved measurement. HarmBench compares 18 red-teaming methods on 33 LLMs [102], and JailbreakBench provides an open robustness pipeline [103]. Comparative studies and suites broaden attack, model, and safeguard coverage [104,105,106,107,108,109]. Work on confusion, small-model risk, the attack-filter arms race, and watermark measurement shows that the chosen target and oracle can change the conclusion [110,111,112,113]. StrongREJECT shows that simply detecting the absence of a refusal or relying on a safety classifier can incorrectly count weak, irrelevant, or incomplete responses as successful jailbreaks [114]. JailMeter uses a stricter criterion by checking whether the response actually addresses the malicious request and provides enough information to satisfy it [115]. These benchmarks strengthen measurement of $I_{\mathrm{pol}}$ but do not provide evidence of $I_{\mathrm{act}}$ or $I_{\mathrm{ext}}$.

Jailbreak defenses remain important because bypassing a model’s safety policy can enable later attacks, such as information disclosure or unauthorized tool use. However, a policy violation, $I_{\mathrm{pol}}$, does not necessarily mean that the attacker accessed sensitive information, performed an unauthorized action, or caused another downstream security failure. A model may generate prohibited text without accessing sensitive data or external tools. Conversely, an agent may perform an unauthorized action even when its text output appears harmless [3,4]. Jailbreak evaluations should therefore be complemented by separate tests of whether sensitive information can be exposed and whether the system can perform actions beyond the user’s authority.

3.2. Prompt Injection and Context-Mediated Control

Prompt injection occurs when a system fails to separate control from data. Direct injection mixes untrusted user data with application instructions. Indirect injection imports attacker-controlled content through a channel the system chooses [1,2]. A formal evaluation across attacks, defenses, models, and tasks found existing protections insufficient and called for stronger optimization-based attacks [50]. Studies of safety-filter evasion later showed that character-level perturbations and adversarial rewrites can bypass prominent guards while preserving the harmful meaning or intended effect of the original prompt [32]. Controlled-release prompting exploits a capacity gap between prompt guards and the larger models they protect [116]. The attack encodes a harmful request so a lightweight filter cannot reliably interpret it, while the target model can still recover and follow it.

Defenses can act at several stages. Some methods identify and remove parts of the input that appear to contain injected instructions, while others classify external content before it reaches the model [117,118]. Another approach reverses the attack strategy by prompting the model to resist malicious instructions without additional training [119]. Evaluation can also be strengthened by testing attacks within realistic application contexts and by using diverse synthetic instructions [120,121]. PromptGuard combines several protections, including input filtering, structured prompting, output checks, and response revision [122]. These defenses matter most when evaluation preserves the origin of each piece of content and the exact form the model receives it in during deployment.

What a guard can detect depends on where it is placed in the system. A guard that sees only the user’s request cannot detect malicious instructions introduced later through retrieved documents or tool outputs. A guard that inspects the complete prompt may detect such an instruction, but it may still be unable to decide whether the resulting action is authorized. PromptLocate and causal attribution methods go beyond simply classifying content as malicious by identifying which part of the context influenced the model and whether that influence affected a tool decision [123,124]. This shifts the evaluation from whether the text looks unsafe to whether untrusted content can influence security-sensitive actions.

Direct and indirect prompt injection differ in who supplies the malicious content and whether the user intended to provide it. In a direct injection attack, the user controls the input, so the application can often restrict the requested task or reject suspicious requests. In an indirect injection attack, a benign user asks the system to process content that was created or modified by someone else [1]. Simply blocking the user may penalize the wrong person, while trusting external content may let an attacker influence the system. Evaluation should preserve the content’s origin, how the application was supposed to use it, and why it was included in the model’s context. Otherwise, a failure of trust or authorization can be reduced to a simple question of whether the text looks malicious.

The final prompt presented to the model can change substantially during execution. The application may reorder, shorten, or rewrite system instructions, developer policies, user input, retrieved documents, memory, tool descriptions, and tool results. A guard evaluated only on isolated text may therefore miss the actual combination of instructions that influences the model. Formal benchmarks provide a common basis for evaluation [50], while environments with tool use show how the same injected instruction can affect tool actions in the evaluated environments [3,4]. A successful injection in a protected system may need to evade a classifier, remain present after retrieval, override competing instructions, and cause the model to issue a plausible tool request. Testing only the target model therefore does not establish the risk in systems with several layers of protection. Testing only the detector can have the opposite problem: an attack may evade detection but no longer influence the target model. Evaluation should therefore measure attack success, defense evasion, preservation of the intended malicious instruction, and resulting system impact under the same resource constraints.

3.3. Persistent Data, Models, and Supply Chains

Retrieval poisoning targets what the system will later treat as evidence. PoisonedRAG corrupts knowledge retrieval [5], and blocker documents attack availability by occupying retrieval positions [53]. RAG integrity depends jointly on admission and ranking. An attacker who can publish a document may not control the vector store, but can still shape content so that ingestion and retrieval favor it. A store administrator can instead alter chunks or metadata after admission. These actors need different controls and different poisoning budgets. Evaluation should separate the probability of retrieval from the probability that a retrieved payload changes the answer or action.

Memory extends the effect of an attack beyond the current interaction because malicious content written into memory can influence later users or tasks. For example, MemoryGraft causes the system to store malicious experience templates that it may later retrieve and reuse. Its main quantitative results measure how often poisoned memories are retrieved, rather than how often they lead to harmful actions [20]. Sleeper memory poisoning separates the attack into three stages: inducing a memory write, retrieving that memory later, and using it to influence model behavior, all without requiring direct access to the memory store [125]. MPBench further distinguishes several ways to write memory, including explicit updates, policy-driven updates, compaction, and converting past experience into reusable procedures [126]. These attacks show that security depends not only on the malicious text itself, but also on who can store information and when that information is later reused. SafetyMem similarly uses semantic and episodic memory as part of the defense, making the source of stored information and permission to modify memory important security considerations [127].

The distinction between different goals of training-time attacks predates LLMs. Biggio et al. inject malicious training examples to reduce an SVM’s overall validation performance [128]. BadNets pursues a different goal: the model behaves normally on clean inputs but produces attacker-chosen behavior when a specific trigger is present [129]. These studies therefore distinguish attacks that broadly reduce model performance from attacks that preserve normal behavior while introducing a hidden, trigger-based failure. They establish this distinction in earlier machine learning systems, but their empirical results may not transfer directly to LLMs.

LLM training pipelines add several insertion routes. Instruction tuning with human feedback and direct preference optimization create post-training control surfaces [130,131]. Virtual prompt injection embeds conditional behavior during instruction tuning [37]. EmbedX manipulates embeddings, and embedding-indistinguishable poisoning exploits trust in pretrained artifacts [6,132]. BadChain poisons in-context chain-of-thought demonstrations and needs no training-data or weight access [38]. Manipulated pretraining can strengthen membership inference [133], and benign-looking fine-tuning can erode safety without a conventional trigger [54,134]. MetaDefense targets fine-tuning jailbreaks before and during generation [135]. Different insertion routes call for artifact provenance, behavioral promotion tests, and rollback, in addition to code scanning [17,55].

Backdoor evaluation must distinguish knowledge of the trigger from knowledge of the model. A detector tested only against a disclosed token trigger may perform well simply because the trigger is easy to search for. Unknown semantic, contextual, or activation-space triggers pose a different problem. BAIT evaluates black-box inversion across a broad range of models and architectures [136], whereas Patcher begins with an observed failure and attempts to localize and repair it [137]. The two works address complementary questions: whether hidden behavior can be elicited and whether an observed behavior can be removed. Neither establishes that a model contains no undiscovered backdoor. Post-training modification further complicates provenance. Sleeper Agents shows that backdoor behavior can persist through the safety-training procedures it evaluates [138], while BackdoorAlign deliberately uses secret-trigger safety examples to protect models against harmful fine-tuning [139]. Because conditional behavior can arise in both attacks and defenses, benign performance alone is insufficient as a criterion for promotion.

3.4. Privacy, Theft, Availability, and Substrate Attacks

Privacy attacks target different kinds of information, so no single leakage measure captures them all. Training data extraction asks whether an attacker can recover memorized training content from the model [56]. Membership inference asks a different question: whether a particular record, or even an example supplied in context, was used by the system [57,140,141]. Reliable membership evaluation requires comparing with nonmembers, carefully selecting the population, running repeated trials, and controlling for false positives. For example, Shokri et al. trained shadow models to learn patterns in classifier outputs that reveal membership [142]. Prompt stealing targets another asset: the hidden prompts or instructions that form part of an application’s intellectual property [58]. Prompt Obfuscation protects these instructions by replacing them with functionally equivalent representations, so its evaluation must consider both resistance to recovering the original prompt and preservation of model utility [143]. MEXTRA targets yet another source of private information: stored interactions that an agent retrieves during later tasks. It attempts to recover these interactions through black-box queries and tool use [144]. These goals are not interchangeable. Preventing verbatim extraction does not necessarily prevent membership inference, and protecting training data does not guarantee that stored user interactions remain confidential. Privacy can also leak through signals beyond generated text, as side-channel attacks show [145]. Finally, defenses based on obfuscation, selective learning, or unlearning may reduce leakage at the cost of changes in model utility, so both privacy and performance must be measured [146,147].

Model extraction can target several different properties of a system. Functional cloning aims to reproduce the model’s behavior, prompt stealing tries to recover hidden instructions, architectural extraction infers internal design choices such as layer or embedding dimensions, and parameter recovery seeks the model weights themselves. In production settings, attacks have recovered components such as a model’s embedding projection [59]. Earlier work by Tramer et al. also showed that prediction APIs can support model extraction and that hiding confidence scores does not remove every extraction method [148]. A survey of model extraction organizes these targets and the defenses proposed against them [14]. The appropriate defense depends on what is being protected. Rate limits and restricted outputs can make cloning through repeated queries harder, but they do not help if an attacker gains direct access to model files or parameters. Watermarking can help identify a stolen model without preventing the theft itself. Private inference addresses a different problem by protecting client inputs from the server under a defined protocol, but it does not necessarily hide all information about the server’s model from the client. Evaluation should therefore clearly state what information or component is protected and what access the attacker has.

Availability attacks try to make a system consume excessive time or resources. They can do this by inducing unusually long generation or reasoning, creating repeated agent loops, triggering costly retrieval, or exploiting how requests are batched and scheduled [44,45,60,61]. Their impact should therefore be measured through outcomes such as response delay, reduced throughput, GPU usage, energy consumption, and monetary cost under a clearly specified request load, rather than through the harmful meaning of model outputs. Let $c_a>0$ denote the attacker’s cost and $c_d$ the incremental defender cost relative to a stated baseline. Both costs should specify the resource and measurement interval, and the ratio $c_d/c_a$ is dimensionless only when their units agree. A denial-of-wallet attack is especially effective when a relatively cheap request forces the defender to spend much more, producing a large $c_d/c_a$ ratio even when the service restricts or filters incoming requests. Long reasoning traces, maximum-length outputs, repeated tool calls, repeated retrieval, and costly defensive checks can all increase this imbalance. NaturalSloth and CRABS specifically exploit generation and computation costs [43,44]. These attacks may affect the slowest requests or deprive other users of resources without greatly changing the median response time, so median latency alone is not sufficient for evaluation.

Side channel attacks show that information can leak through system behavior even when the model’s text output reveals nothing sensitive. Prompt caching can change response time depending on whether a prefix has been seen before [28], shared inference can expose information through timing [7], and cache behavior or GPU partitioning can create measurable hardware signals [27,62]. These results show that analyzing LLM security must also account for scheduling, caching, and resource isolation. Evaluating such attacks requires more than observing a timing difference. The attacker must show that the measured signal reveals information about a victim despite noise from other workloads, network delay, batching, hardware variation, and prior knowledge. Leakage should therefore be measured by how accurately the attacker can infer the protected information, or by how much additional information the signal provides compared with an appropriate baseline. Defenses should be tested under the same shared workload and system conditions, and should account for the cost of disabling cache reuse or increasing isolation. CacheSolidarity evaluates this tradeoff for prefix reuse [149].

3.5. Multimodal Carriers and Cross-Agent Propagation

Multimodal systems create additional ways to deliver and hide harmful instructions through text formatting, images, audio, and combinations of different modalities. FigStep places harmful instructions in images or visual text to bypass safeguards that focus mainly on ordinary text prompts [64]. Other attacks directly modify images to influence both visual representations and the language generated from them [65]. Jailbreak-AudioBench shows that audio language models require separate safety evaluation because attacks can exploit properties specific to spoken input [66]. Multilingual and multimodal red teaming similarly tests whether safeguards that work for one language or representation remain effective when the same intent is expressed differently [150]. Other attacks exploit how visual information affects the model’s reasoning, allowing image content to steer the generated response toward unsafe behavior [151,152]. Related work uses multimodal models to generate text attacks, shows that apparently harmless images can carry malicious instructions, and uses an LLM to plan attacks against protected text-to-image systems [153,154,155]. Defenses have also begun to target these interactions: ReCon uses safety-related concepts to counter malicious visual influence, while SafePTR identifies and removes harmful multimodal signals before restoring the useful information needed for the task [156,157].

Using images or audio does not necessarily change the underlying security goal. The threat model still distinguishes whether malicious content is supplied by the user, retrieved from an external source, stored for later use, or passed between agents, and whether the attack changes only the model’s output or also causes an unsafe action. Multimodal defenses can fail when the guard and target receive different information. For example, FigStep’s low-contrast variants bypass OCR-based screening while remaining effective against the evaluated vision-language models [64]. For audio, transcription can discard information retained by end-to-end models [66], so a transcription-only guard should not be assumed to observe all information available to the target. Evaluation should therefore report the exact input each control receives and the information available after modality fusion. It should also test defense-aware transformations that preserve the interpretation driving the target’s response. A single overall multimodal ASR does not establish whether these conditions hold.

Propagation attacks are dangerous because a compromise can spread beyond the first affected agent. For example, Agent Smith shows how a malicious image can propagate through interactions among connected agents [67]. Evaluation should therefore measure how often the attack spreads to another agent, how many additional agents each compromised agent can affect, how long the spread continues before containment, whether the attacker gains additional privileges later, and whether the attack crosses user or tenant boundaries. Defenses should also consider agent identity, message origin, restrictions on delegated permissions, limits on repeated interactions, and mechanisms to isolate compromised agents. A benchmark that resets the system after a single recipient is affected cannot measure whether an attack persists and spreads through the wider agent network.

4. Defense Taxonomy: View, Function, and Response

Using the defense model in Section 2, we classify each defense by $(V,F,R)$: the information it can observe, the security function it performs, and the response it can enforce. Figure 4 presents a tree of security functions and representative mechanisms, while Table 4 specifies their observable views, enforceable responses, and limitations. When the required view is absent, even a strong benchmark result does not transfer to the corresponding attack path.

4.1. View and Input Mediation

A defense can protect a property only if it can observe the information needed to make the relevant decision or enforce a rule that prevents the unsafe action. The filter and authorization examples in Section 2 illustrate these limits. The same distinction applies to cache isolation, which can prevent data reuse across tenants but does not protect the integrity of a retrieval corpus. A separate gate placed before the action can still block it even when an earlier defense fails.

Giving a defense more information is not always better. A central guard that receives private prompts, retrieved records, tool arguments, and execution results becomes another component that must be trusted with sensitive data and may itself introduce privacy risks. Private inference and information-flow tracking address this problem by reducing plaintext exposure or restricting how information moves between users and components [167,168,171,172]. The design question is therefore how much information a defense needs to make its decision and how to protect that information.

Some defenses assume that adversarial inputs are more sensitive to small transformations than benign inputs. SmoothLLM perturbs an input several times and combines the resulting model behavior, while PARDEN asks a model to repeat its output and uses the discrepancy between the original and repeated text for detection [158,159]. These methods can be applied without retraining the underlying model, but they work only if adversarial and benign inputs respond differently enough to these transformations. Their decision rules must limit false alarms while accounting for the additional inference cost. An adaptive attacker may instead search for inputs that remain effective after perturbation or deliberately exploit the additional computation required by the defense.

Structural defenses address the problem differently. StruQ separates trusted instructions from untrusted data and fine-tunes the model to respect this distinction [160]. This provides more protection than simply placing visible delimiters around untrusted text because both the input structure and the model are designed around separate roles. However, the protection depends on the application assembling the prompt correctly, identifying all untrusted content, and the model continuing to respect the separation. If the application removes this structure before sending the prompt to the model, or mistakenly treats a tool result as a trusted instruction, the protection is lost. Evaluation should therefore inspect the actual input the model receives rather than only the data exposed through the application interface. Information about where content came from must also survive operations such as chunking, summarization, translation, and transfer between tools. If an untrusted document loses its source label during one of these steps, later components may mistakenly treat it as trusted. A text-only attack success rate does not measure whether this source information remains intact throughout the system.

4.2. Detection, Hardening, and Repair

Detectors observe different forms of evidence. Gradient Cuff probes the refusal-loss landscape [161], WildGuard and multilingual guards classify policy-relevant text [162,163], and HiddenDetect and cognition-based detection use internal signals [109,164]. Multilingual Collaborative Defense trains a shared safety prompt across languages, while X-Guard combines translation with evaluation [173,174]. These methods do not enforce tool authorization, and their detection evidence remains conditional on the attacker’s access to the signal and on transfer beyond the languages, models, and attacks tested. Detection also produces an operating curve rather than a binary property because low thresholds block ambiguous benign requests, whereas high thresholds leave a wide evasion region. Attack success should therefore be reported at fixed clean false-positive rates or along a security-utility curve. Choosing a threshold after observing the attack set requires a separate validation set, and queryable accept/reject behavior lets an iterative attacker infer the boundary and become defense-aware.

Adversarial hardening changes the training distribution and not only the runtime decision. RPO optimizes defensive prompts against jailbreaks, PAT alternates attack and defense prompts, DPP trains a defensive prefix, and DataSentinel uses a minimax prompt-injection objective [34,35,36,165]. When the inner attack is competitive, these methods provide more direct evidence for resistance to optimized attacks than training on a static attack list. The supported claim remains limited to the optimized carrier, objective, model, and budget. A stronger attacker, a new representation, or an action-level objective can void a general claim while the measured result stays correct. Other data and prompt methods differ in how directly they model the attacker. In-context adversarial games and AGD alternate attack and defense, while pre-synthesized jailbreak-like instructions broaden guard training. Template-based in-context learning adds a defensive suffix, and length-transfer work tests whether short adversarial training protects against longer attacks [175,176,177,178,179]. They help only to the extent that the generated or optimized inner attack spans the deployed carrier, model, and policy.

Internal and decoding-time controls use other observables. Layer editing, concept manipulation, and self-defense act on behavior or representations [180,181,182]. Essence-oriented detection and meta-operation reasoning look for signals that depend less on surface form [183,184]. Trajectory kinetics, adaptive steering, and gradient-controlled decoding intervene at the layer or token level [185,186,187]. Stress tests should go beyond a development-suite mean. Jailbreak-Zero trades attack success against risk coverage, diversity, and distribution fidelity [188], and ensemble and keyword-bias analyses reveal correlated shortcuts [189,190].

Repair begins after evidence of compromise. BAIT searches for hidden target behavior through inversion [136], Patcher uses a failure case and model access to localize and edit unsafe behavior [137], and weak-to-strong distillation transfers cleaner behavior under stated data assumptions [166]. Repair evidence should include the triggering case, nearby variants, unrelated utility, alternate triggers, and persistence after later fine-tuning or merging because passing the original case alone invites patch overfitting. Foundational backdoor defenses also show that access assumptions matter. Spectral Signatures filters poisoned training examples by separating them in learned representations [191], while Neural Cleanse reconstructs candidate triggers from a trained classifier and clean examples [192]. These methods motivate inspection and repair but do not cover semantic LLM triggers, so a claim should state whether the defender has the training set, model internals, clean examples, or only a witnessed failure.

4.3. Enforcement, Isolation, and Recovery

Agent defenses can evaluate whether a proposed action aligns with the user’s goal before letting it proceed. Task Shield checks whether an action is relevant to that goal [46]. MELON masks the user request and compares the resulting next actions to test whether the original request is still driving execution [47]. AttriGuard reduces the influence of external content and measures whether the proposed tool call changes [124]. These methods consider how the task and surrounding context influence an action rather than judging isolated text alone. However, they may require additional model calls and rely on model outputs that can vary across runs, creating new evasion opportunities.

Authorization provides a separate mechanism to enforce what the system can do. IPIGuard restricts tool execution using a dependency graph [48]. More generally, the execution layer can check task-specific tool permissions, argument schemas, and ownership of the requested object. It can also enforce narrowly scoped credentials and require fixed policy checks or human approval for sensitive actions. These controls may not detect a prompt injection, but they can still block an unauthorized action even if the model follows the injected instruction.

Verification asks whether an action appears consistent with the user’s intent, whereas authorization asks whether the user is permitted to perform that action on the specified object under the current conditions. A request can seem reasonable while still violating ownership rules, spending limits, geographic restrictions, or separation-of-duties rules. Conversely, a language-based verifier may reject a legitimate request that a precise authorization rule could safely permit. Strong systems therefore use semantic verification to estimate risk and explicit authorization rules to enforce permissions.

Isolation limits the damage when prevention fails. Tool sandboxes restrict access to files, networks, and processes. Narrowly scoped credentials limit which resources can be reached, while tenant and cache isolation reduce the chance that information crosses between users. GIFT tracks information ownership within the GPU serving layer and reports the resulting throughput cost [167]. CacheSolidarity studies how prompt caching can retain some reuse while reducing leakage between users [149]. These mechanisms can show that damage is contained under a particular deployment and threat model, but they do not show that the model itself is aligned or resistant to malicious instructions.

Resource controls address attacks that try to exhaust compute or service capacity. Token limits can restrict visible output length, but they may not limit internal reasoning, retrieval, retries, or repeated tool use. A more complete budget must account for all work triggered by a request, including retries, delegated agents, additional safety checks, and asynchronous tasks. PD³F combines request scoring, scheduling, and early termination for local models under controlled attack and user workloads [170]. Production evaluations should also consider changing request rates, worst-case latency, fairness across users, and attackers that adapt when thresholds are introduced.

Recovery requires more than detecting or blocking attacks. Systems that store persistent state need records of what changed, who caused each write, previous versions of corpora and prompts, revocable credentials, rollback for models and adapters, and enough logging to reconstruct an incident. A recovery claim should show that malicious state has been removed, affected indexes or memories have been rebuilt, compromised credentials have been revoked, and the original attack path no longer succeeds. Detecting an attack without removing its effects can leave the system compromised. Rolling back changes without preserving evidence can also make it harder to understand the incident and prevent it from happening again.

4.4. Defense Composition

The attack paths reviewed here motivate complementary controls at multiple system layers. Input filters can reject known attacks early, structured context can help separate trusted instructions from untrusted data, and provenance can identify where external content originated. Authorization checks can block prohibited actions, sandboxing can limit the damage of a successful attack, output checks can reduce information disclosure, and rate limits and resource budgets can protect availability. Logs and records of state changes support investigation and recovery. Together, these controls provide defense in depth. In LLM systems, however, safety judgments made by the model or a guard must ultimately be enforced by system controls that can permit, restrict, or block the resulting action.

Figure 5 illustrates three attack paths that cross several layers of an LLM system. Each path can be interrupted at multiple points, but only if a defense can observe the relevant information and has the authority to act there. Model refusal, for example, may help prevent harmful behavior during generation, but it cannot replace restricted credentials during tool execution or tenant isolation in the serving infrastructure.

5. Attack-Defense Assurance Matrix

The matrix evaluates whether the available evidence actually tests the conditions required by a security claim. For an attack class a and a defense d, we define the coverage vector

$$\mathbf{M}(a,d)=(M_E,M_K,M_A,M_P,M_I,M_V,M_R),$$

(2)

where each component records matched, partial, or untested coverage for the corresponding dimension of Eq. (1). The objective, resource budget, and system version are held fixed when making the comparison. Matched means that the relevant condition is directly represented in the evaluation. Partial means that only part of the condition, or an indirect proxy for it, is tested. Untested means that the evaluation does not test that condition in a way that supports the claim. Coverage does not imply that the defense succeeds: a well-matched evaluation may instead show that the defense fails. Hence, we keep the dimensions separate rather than combining them into a single score. If a security claim depends on an untested dimension, the available evidence is insufficient to support the full claim, even when the other dimensions match.

A simple example illustrates the vector. Suppose a detector is evaluated only against fixed adversarial suffixes supplied directly by the user. The experiment covers the direct input surface, a static attack, and the detector’s ability to recognize the corresponding attack signal. It does not, by itself, establish that the detector prevents the harmful outcome that the policy is intended to stop. If the attacker is not allowed to adapt the suffix to the detector, then $M_A$ remains untested for any claim of resistance to adaptive attacks. Similarly, if the detector is intended to protect an agent but cannot observe tool arguments, then its observable view is only partially represented. If it also cannot restrict the resulting tool call, evidence for an enforceable response is incomplete. An allowlist provides a different form of protection: even if the model follows an injected instruction, the execution layer can still block an unauthorized tool or object by independently checking whether the requested action is permitted.

Figure 6 summarizes the available evidence for evaluated settings rather than ranking attacker capability. The static, iterative, transfer, and defense-aware rows describe how the attack was constructed, whereas the persistent-state and cross-stack rows describe its time horizon and system scope. The columns identify the stage at which a control observes or enforces a boundary, and four letters encode the strength and consistency of the evidence.

• S: direct, consequence-matched control evaluations reported across independent studies within the named setting.
• P: narrower, conflicting, proxy-based, or attack-only evidence.
• R: a gap observed in at least two independent primary studies within the named setting; it does not imply that every control of that type fails.
• I: insufficient or heterogeneous evidence for a stable assignment; it should not be interpreted as evidence of safety.

Each of the 36 matrix cells has a corresponding entry in an evidence ledger. The entry records the supporting studies with source locators, the basis for the assignment, the boundary of what the evidence establishes, and any counterevidence. A separate codebook defines the assignment rules for S, P, R, and I.

For example, the cell at the intersection of the defense-aware row and the action column is marked R. The ledger cites two independent evaluations, by Zhan et al. and He et al., that report degradation of existing agent defenses under adaptive attacks [22,124]. It also notes that AttriGuard remains resilient to the optimization-based attacks evaluated in its study. The R label therefore denotes a recurring evaluated weakness in the named designs, not a general failure of action-level controls. By contrast, the static-row cell in the same column is marked S because several action controls were evaluated directly in simulated tool environments [46,47,48,124]. The contrast shows why evidence from static evaluations alone cannot support a claim of robustness against defense-aware attacks.

Persistent-state serving is marked P because cache-poisoning experiments link retained entries to corrupted later responses and evaluate targeted mitigations, but they do not establish general remediation across tenants [193]. Cross-stack model evidence is likewise marked P. Environment-poisoned training traces and backdoored tool-use models connect learned behavior to later actions. Nevertheless, they do not demonstrate a broadly effective defense across the composed system [26,194]. These assignments therefore reflect limited but relevant evidence, while avoiding stronger claims that the evaluated defenses are generally effective.

Table 5. Representative attack-defense coverage and conditions that remain untested or only partially established.

Attack class	Plausible controls	What the control sees	Representative evidence	Condition still open
Static jailbreak	Perturbation, repetition, refusal detectors	User input, output, or refusal landscape	Diverse attacks across several models, with utility evaluated for several defenses [158,159,161]	Defense-aware optimization, policy drift, and version drift
Adaptive jailbreak	Adversarial prompt or model training, layered moderation	Gradients or attack corpus, plus input and output	Explicit inner maximization and selected adaptive evaluations [34,35,165]	Independent attack budgets and transfer to changed or proprietary stacks
Direct injection	Structured queries, detection, localization	User-data boundary or assembled prompt	Multi-model benchmarks and adaptive detector training [36,50,160]	Trusted prompt assembly, semantic encodings, and utility on ambiguous tasks
Indirect injection	Goal verification, counterfactual replay, tool graph	Full context, user goal, and tool trace	AgentDojo tasks and action-level defenses [4,46,47,48,124]	Adaptive long-horizon attacks, real credentials, and irreversible actions
RAG and memory poisoning	Provenance, ACLs, robust retrieval, state review	Corpus origin, retrieval ranking, and memory writes	Targeted corruption and retrieval-jamming demonstrations [5,53]	Cross-user persistence, deletion completeness, and multi-stage adaptation
Backdoor and supply chain	Scanning, signatures, promotion tests, repair	Weights, targets, lineage, and failure examples	BAIT across 153 LLMs, post-hoc repair, and distillation [136,137,166]	Process-compromised attackers, unknown trigger families, and composed adapters
Privacy and model theft	Rate limits, output restriction, DP, private inference	Query trace, output, and encrypted execution	Production extraction and parameter recovery, together with HE and FHE prototypes [56,59,168]	Composition across repeated sessions and accuracy-latency trade-offs at scale
DoS and denial of wallet	Budgets, scheduling, circuit breakers, loop limits	Request cost, queue state, and token and tool traces	Controlled-load evaluation across six models and several attack mechanisms [44,45,170]	Distributed production traffic and attacker adaptation to schedulers
Tenant and side channel	Isolation, cache policy, information-flow tracking, confidential compute	Cache reuse, latency, and ownership flow	CacheSolidarity efficiency trade-offs and GIFT throughput overhead [149,167]	Provider-specific caches, network noise, and co-resident adaptive attackers

Table 5. Representative attack-defense coverage and conditions that remain untested or only partially established.

Attack class	Plausible controls	What the control sees	Representative evidence	Condition still open
Static jailbreak	Perturbation, repetition, refusal detectors	User input, output, or refusal landscape	Diverse attacks across several models, with utility evaluated for several defenses [158,159,161]	Defense-aware optimization, policy drift, and version drift
Adaptive jailbreak	Adversarial prompt or model training, layered moderation	Gradients or attack corpus, plus input and output	Explicit inner maximization and selected adaptive evaluations [34,35,165]	Independent attack budgets and transfer to changed or proprietary stacks
Direct injection	Structured queries, detection, localization	User-data boundary or assembled prompt	Multi-model benchmarks and adaptive detector training [36,50,160]	Trusted prompt assembly, semantic encodings, and utility on ambiguous tasks
Indirect injection	Goal verification, counterfactual replay, tool graph	Full context, user goal, and tool trace	AgentDojo tasks and action-level defenses [4,46,47,48,124]	Adaptive long-horizon attacks, real credentials, and irreversible actions
RAG and memory poisoning	Provenance, ACLs, robust retrieval, state review	Corpus origin, retrieval ranking, and memory writes	Targeted corruption and retrieval-jamming demonstrations [5,53]	Cross-user persistence, deletion completeness, and multi-stage adaptation
Backdoor and supply chain	Scanning, signatures, promotion tests, repair	Weights, targets, lineage, and failure examples	BAIT across 153 LLMs, post-hoc repair, and distillation [136,137,166]	Process-compromised attackers, unknown trigger families, and composed adapters
Privacy and model theft	Rate limits, output restriction, DP, private inference	Query trace, output, and encrypted execution	Production extraction and parameter recovery, together with HE and FHE prototypes [56,59,168]	Composition across repeated sessions and accuracy-latency trade-offs at scale
DoS and denial of wallet	Budgets, scheduling, circuit breakers, loop limits	Request cost, queue state, and token and tool traces	Controlled-load evaluation across six models and several attack mechanisms [44,45,170]	Distributed production traffic and attacker adaptation to schedulers
Tenant and side channel	Isolation, cache policy, information-flow tracking, confidential compute	Cache reuse, latency, and ownership flow	CacheSolidarity efficiency trade-offs and GIFT throughput overhead [149,167]	Provider-specific caches, network noise, and co-resident adaptive attackers

5.1. Assumption Mismatches

Four mismatch patterns recur when the matrix is applied to specific defense families. In each case, the defense may suit the mechanism it targets, but the evaluation assumes a system model that does not match the deployment. Exposing the mismatch does not require a new attack algorithm; it requires changing the conditions under which the defense is tested.

1) The defense assumes the carrier is visible, but the system assembles or retains it elsewhere. Persistent artifacts illustrate this mismatch. PoisonedRAG and blocker documents manipulate retrieval through corpus content [5,53]. Backdoors place conditional behavior in learned parameters through instruction tuning, while reusable demonstrations can carry context poisoning across later interactions [37,38]. A runtime filter observes only what reaches its inspection point and may not observe where the harmful state was inserted or retained. Assurance therefore requires controls over admission, provenance, promotion, and recovery, followed by replay after the relevant state change.

2) The defense assumes the attacker cannot adapt to its decision, but accept-and-reject behavior provides feedback. DataSentinel explicitly formulates an adaptive minimax game [36], while character-level and adversarial evasion studies show why detector-aware evaluation matters [32]. Accuracy on a balanced benchmark is insufficient for deployment because attack prevalence may be low and false positives can disrupt normal workflows. A production claim therefore requires calibration at the expected base rate, repeated-query resistance, full-pipeline context, and attacks that must both evade the detector and achieve the harmful objective. The same concern applies to wrappers such as SmoothLLM, PARDEN, and Gradient Cuff, which rely on repeated generations, inconsistency, or refusal signals [158,159,161]. Repetition increases both defender cost and the feedback available to an adaptive attacker. Matched evaluation should therefore include wrapper-aware attacks, fixed false-positive operating points, and defender cost under adversarial inputs. Optimized defenses such as RPO, PAT, and DPP strengthen the evidence by incorporating attacks during defense development [34,35,165], but questions remain about the independence of training and evaluation attacks, comparable optimization budgets, and transfer across model families.

3) The defense assumes the output is the final consequence, but tools may act before or independently of it. Task Shield, MELON, AttriGuard, and IPIGuard reason over user goals, contextual interventions, causal influence, or dependency structure [46,47,48,124]. These mechanisms align more closely with $I_{\mathrm{act}}$ than output moderation does because they evaluate or constrain the action path itself. However, their evidence remains bounded by simulated environments, selected tool schemas, additional model-call cost, and the level of adaptive access granted to the attacker. End-to-end evaluation should therefore include malformed arguments, object-level authorization, retries, compromised tool outputs, and gates for irreversible actions.

4) The defense assumes a one-shot state, but the attacker can persist across interactions or tenants. An attacker may write memory, poison a corpus, persist in model weights, or retry across accounts. Serving-isolation studies such as CacheSolidarity and GIFT address part of this problem by evaluating protection together with cache reuse, latency, or throughput [149,167]. Their security oracles are necessarily tied to the infrastructure they protect, so semantic attack success rate is not the relevant metric. The remaining questions concern transfer across serving frameworks, provider-specific cache policies, noisy co-tenancy, and attackers that learn the conditions under which isolation is enforced.

The converse mismatch occurs when the evaluation grants the attacker more privilege than the intended deployment permits. Gradient access, direct vector-store writes, or co-residency may fit supply-chain or insider threat models but can overstate risk when attributed to an ordinary Internet user. Studies should therefore report feasibility separately from conditional impact. The matrix does not treat a stronger attacker as inherently more informative; it asks whether the evaluated attacker corresponds to a credible principal, capability set, and system boundary.

6. Bounded Security Claims

A statement such as “Defense X is robust” omits the attacker, system version, consequence, cost, and evaluation oracle that bound the experiment. Two studies may therefore describe a defense as robust while evaluating materially different conditions. The following claim grammar makes those conditions explicit for a defense d:

For system version s, defense d achieved the reported prevention or detection rate for consequence I at surface E against objective O, knowledge and access K, adaptivity A, persistence P, and budget B, with utility U, cost C, and stated uncertainty under oracle J.

Each field is independently testable and prevents a specific interpretive error. Version s prevents a result from silently transferring to a changed API or guard stack, while consequence I prevents harmful text from serving as a proxy for tool or state compromise. The fields E, K, A, and P define the evaluated adversarial region, and budget B exposes resource asymmetries. Utility U and cost C reveal whether apparent security follows from excessive refusal or repeated checking. Oracle J records whether success was determined by a keyword, classifier, LLM judge, human reviewer, cryptographic verification, or deterministic system assertion. Any field absent from the source remains unreported rather than being inferred.

The grammar also identifies results that should not be compared directly. A defense evaluated on a large static suite and one evaluated on a smaller defense-aware suite address different attacker models, so pooling their ASR values can cause the larger static evaluation to dominate despite testing a weaker adversary. Likewise, a RAG study of answer manipulation and an agent study of unauthorized tool calls may share taxonomy dimensions without sharing a common outcome scale.

6.1. Worked Comparison

A related-work discussion may correctly state that a detector such as DataSentinel and a graph constraint such as IPIGuard both “defend against prompt injection,” even though the claims supported by their evaluations are different. Table 6 instantiates the tuples using the properties discussed in Section 2 to Section 5. The purpose is to compare claim structure, not to restate or rank the reported results.

Each column supports a different bounded claim. For the detector, the evidence supports detection performance on the evaluated attacks and models under adaptive training. It does not, by itself, establish prevention of unauthorized actions because the detector produces a signal rather than an independently enforced control. For the graph constraint, the evidence concerns restricting unauthorized tool actions in simulated environments with reported utility, but it does not establish protection over tool arguments, object ownership, production credentials, or irreversible actions. Section 5 treats these as separate conditions in an end-to-end evaluation. A full-stack claim therefore requires evidence of both detection and enforcement under a consistent attacker model and consequence.

7. Adaptive Evaluation and the Evidence Ladder

A static suite asks whether an implementation stops a specified set of inputs. A defense-aware evaluation asks a different question: whether the attack objective remains achievable after the attacker adapts the carrier, optimization target, timing, or system path to the defense. The evidence ladder distinguishes which of these questions an experiment addresses.

7.1. Evidence Levels

The ladder in Figure 7 describes the type and scope of evidence for one fixed security claim and threat model. Levels L0 to L5 organize a progression of evidence requirements for that bounded claim, not a universal quality scale. A higher numerical label does not imply a better defense or a stronger paper overall, and comparisons are meaningful only for the same or closely matched claims and threat models. Some evidence is complementary rather than strictly ordered. Formal or theoretical guarantees may strongly support a narrowly defined property without fitting this empirical progression, as in a proof under a specified leakage model that makes no claim about malicious tools.

• L0, assertion: a mechanism and intended security claim, with no adversarial experiment.
• L1, static replication: known attacks, fixed models, and a defined metric for resistance to a named suite.
• L2, diversity: several attack families, models, and seeds, together with relevant clean utility, provide evidence that the result is not specific to one setup.
• L3, generalization: unseen attacks or models, transfer, ablation, and comparable budgets provide evidence beyond conditions already used to develop the defense.
• L4, defense-aware: an attacker with knowledge of the defense adapts directly to it under a realistic or comparable attack budget, using queries, gradients, surrogates, or human analysis as permitted by the threat model.
• L5, system assurance: L4 plus realistic authority, persistent state where relevant, direct consequence evidence, operational cost, containment, and remediation replay.

L1 to L3 mainly expand empirical coverage and generalization across attacks and models. L4 changes the attacker assumption by requiring explicit knowledge of and adaptation to the defense under a stated attack budget. L5 adds system conditions and consequence evidence through realistic authority, persistent state where relevant, and an oracle that inspects the security-relevant state rather than only a response label. These requirements apply only where they are relevant to the stated claim and do not make L5 a universal endpoint. Increasing the number of static prompts alone does not satisfy the diversity or generalization requirements of L2 or L3. A well-designed defense-aware test addresses an adaptive claim more directly, regardless of whether the attack succeeds, while L1 evidence may be entirely adequate for a static filter whose stated claim is limited to resistance against a named suite or inexpensive triage.

Nasr et al. illustrate the effect of defense-aware adaptation by optimizing gradient descent, reinforcement learning, random search, and human-guided attacks against 12 defenses, bypassing most of them at above 90% ASR under the reported configurations [21]. Zhan et al. similarly adapted attacks to eight agent defenses and exceeded 50% ASR across the evaluated defense-agent combinations on InjecAgent, although the same threshold did not hold for every AgentDojo result [22]. These dated experimental results should not be interpreted as properties of current services. Instead, they provide L4 evidence about how conclusions can change when attacks adapt to defenses that performed well under earlier evaluation conditions. Guard evasion can also transfer through surrogate importance rankings and character-level channels [32], while guard identification can reduce uncertainty before adaptation [33].

7.2. Adaptive Re-Evaluation and Experimental Design

These results motivate checks for four evaluation risks: 1) models and detectors may overfit a fixed attack set, 2) repeated queries may erode protection that depends on defense secrecy, 3) low ASR may reflect degraded benign utility, and 4) the evaluator may fail to recognize attack success. StrongREJECT and JailMeter address the final issue for harmful-content tasks, but model-based evaluators remain attack surfaces and do not necessarily establish changes in system state [114,115].

An interpretable security statement requires more than ASR. A defense evaluation should specify the attacker’s knowledge, query and token budgets, access to gradients, logits, and surrogates, and adaptation procedure and stopping rule. It should also report wall-clock and monetary cost alongside the target and guard versions, sampling configuration, and transfer or unseen conditions. Outcome reporting should include utility and false-positive curves, evaluator agreement with human judgments or deterministic oracles, the system state needed to establish impact, and whether remediation survives exact replay. Section 6 organizes these elements into a bounded claim.

Experimental design should begin from a defense disclosure package. At a minimum, the red team needs the decision interface, observable accept and reject behavior, model and guard query limits, and the security objective. White-box tests with additional disclosure may include prompts, thresholds, surrogate data, logits, or gradients. Keeping every detail secret tests one operational setting, but it cannot support a claim against a knowledgeable adversary. Results should therefore be stratified by disclosure level rather than averaged together.

For an end-to-end claim, the attack objective should cover the full defended pipeline. In a simple input-gated system, let x be an attacker-controlled input, f the target model, and g a binary admission rule with $g\left(x\right)=1$ indicating acceptance. When oracle J uses one for the specified attack success and zero otherwise, joint success requires $g\left(x\right)=1\wedge J\left(f\right(x\left)\right)=1$ on the same trial. Requests rejected before reaching the target count as unsuccessful joint attacks. Optimizing only f tests the unguarded model, while evading g alone may yield benign outputs. Joint attacks must also respect semantic and resource constraints. Agent settings require evaluating the resulting action or state change, while availability settings require measuring defender cost. Even without gradients, rejection sampling, surrogate ranking, tree search, reinforcement learning, and human analysis can search for joint success [21,32].

Budget parity requires more than equal query counts. A target-model call and a cheap classifier call can differ by orders of magnitude in cost. Parallel best-of-N sampling changes wall-clock time without changing the total number of calls, and a defender may invoke several models for a single request. Reporting tokens, model calls by type, elapsed time, hardware, and monetary cost allows readers to reconstruct both attacker and defender expenditure. For stochastic targets, evaluations should also report seeds or repeated trials together with confidence intervals.

Adaptive testing should proceed sequentially. It should first reproduce the paper’s static baseline, then vary the model, prompt distribution, encoding, and seed, test transfer from independent surrogates, and finally disclose the defense and allow a red-team adaptation period. This sequence separates implementation failure from assumption failure and shows which forms of knowledge change the result. If the defender retrains after observing the new attack, evaluation should continue as a repeated game and end with a held-out red-team strategy rather than stopping after the defender’s final update.

Table 7. Minimum adaptive-evaluation protocol by defense type.

Defense type	Attacker disclosure	Adaptation mechanism	Primary outcome oracle	Required collateral evidence
Input/output guard	Interface, threshold behavior, policy classes	Query search, surrogate transfer, character/semantic mutation	Joint evasion and target success	Precision-recall or false-positive curve, target and guard cost
Prompt or model hardening	Defensive prompt/training objective as appropriate	Independent optimization with equal or larger budget	Harmful completion with fulfillment-aware judge	Clean helpfulness, unseen attacks/models, prompt or weight version
RAG/memory control	Admission, ranking, provenance, and write interfaces	Adaptive documents, ranking manipulation, repeated writes	Retrieval, answer/action effect, persistent state	Clean recall, poisoning budget, cross-user reach, rollback
Agent/action gate	Tool schema, approval logic, observable failures	Context and trajectory search, retry, argument mutation	Deterministic action and external-state assertion	Benign task completion, extra calls, irreversible-action containment
Serving/isolation	Scheduler/cache policy or learned trigger behavior	Load shaping, co-tenancy, timing inference	Tail latency, leakage, fairness, ownership violation	Throughput, utilization, false isolation, noisy-network tests
Artifact repair	Detection/repair method and promotion gate	Alternate triggers, semantic neighborhoods, post-repair tuning	Trigger success and clean/safety regression	Model utility, repeated repair cost, lineage and rollback

Table 7. Minimum adaptive-evaluation protocol by defense type.

Defense type	Attacker disclosure	Adaptation mechanism	Primary outcome oracle	Required collateral evidence
Input/output guard	Interface, threshold behavior, policy classes	Query search, surrogate transfer, character/semantic mutation	Joint evasion and target success	Precision-recall or false-positive curve, target and guard cost
Prompt or model hardening	Defensive prompt/training objective as appropriate	Independent optimization with equal or larger budget	Harmful completion with fulfillment-aware judge	Clean helpfulness, unseen attacks/models, prompt or weight version
RAG/memory control	Admission, ranking, provenance, and write interfaces	Adaptive documents, ranking manipulation, repeated writes	Retrieval, answer/action effect, persistent state	Clean recall, poisoning budget, cross-user reach, rollback
Agent/action gate	Tool schema, approval logic, observable failures	Context and trajectory search, retry, argument mutation	Deterministic action and external-state assertion	Benign task completion, extra calls, irreversible-action containment
Serving/isolation	Scheduler/cache policy or learned trigger behavior	Load shaping, co-tenancy, timing inference	Tail latency, leakage, fairness, ownership violation	Throughput, utilization, false isolation, noisy-network tests
Artifact repair	Detection/repair method and promotion gate	Alternate triggers, semantic neighborhoods, post-repair tuning	Trigger success and clean/safety regression	Model utility, repeated repair cost, lineage and rollback

7.3. Statistical and Evaluator Validity

When a security experiment reports the strongest attack found across prompts, temperatures, seeds, or checkpoints, it supports worst-case discovery. The reported uncertainty should nevertheless account for this selection process. Confidence intervals over sampled prompts capture variation within one configuration, but not uncertainty from model versions, hidden guard changes, evaluator error, or the search over configurations. Therefore, papers should distinguish sampling uncertainty within a configuration from sensitivity across configurations.

Detector results require prevalence-aware reporting. For a fixed detector and population, let $\pi$ be attack prevalence, $\mathrm{TPR}$ the probability of flagging an attack, and $\mathrm{FPR}$ the probability of flagging benign input. Precision, or positive predictive value $\mathrm{PPV}$, follows from Bayes’ rule:

$$\mathrm{PPV}={\displaystyle \frac{\pi \mathrm{TPR}}{\pi \mathrm{TPR}+(1-\pi )\mathrm{FPR}}}.$$

(3)

Equation (3) applies when the probability of an alert is nonzero. It explains why false positives can dominate alerts at low attack prevalence, even when accuracy on a balanced benchmark is high. Rates estimated on a different population need not transfer to deployment.

Harmful-response judges similarly require calibration across successful, refused, irrelevant, and partially compliant outputs. StrongREJECT addresses fulfillment-aware harmfulness [114], while JailMeter uses human-labeled instances to improve evidence extraction and reports evaluator limitations [115]. Neither substitutes for deterministic oracles when the relevant outcome is object authorization, tool arguments, external state change, latency, or cache leakage.

The evaluator itself can also become part of the attack surface, a limitation acknowledged by JailMeter [115]. If the same judge selects attack prompts and scores the final results, optimization may exploit judge-specific artifacts and inflate apparent success. A stronger design uses a separate selection signal, a hidden final evaluator, human adjudication of disagreements, and deterministic state assertions wherever possible. Evaluator prompts, model versions, temperatures, thresholds, and refusal-handling rules should be included in the reproducibility record.

8. From Model Failure to System Compromise

This section applies the framework to surfaces for which output-level evidence is most incomplete. For each surface, the analysis identifies the boundary crossed by the attack, the information and authority required by a defense, and the oracle needed to determine whether that boundary held.

8.1. RAG and Memory

RAG introduces at least four boundaries: document admission, indexing, retrieval, and prompt assembly. Admission asks whether the source and principal are trusted. Indexing asks whether chunking and embedding preserve provenance. Retrieval asks whether an attacker can influence ranking or crowd out relevant evidence. Assembly asks whether the system treats retrieved language as data or as authority. PoisonedRAG attacks knowledge integrity, blocker documents attack availability, and indirect injection in the wild exploits the path from external content into prompt context [5,52,53].

SafeRAG makes attack placement explicit. It injects attacks into the knowledge base, retrieved context, or filtered context, and evaluates misleading noise, conflicting evidence, covert advertising, and refusal-inducing warnings [195]. Its retrieval and answer-quality measures expose failures at specific components, but they do not measure resource exhaustion. Denying a useful answer and denying computational resources require different oracles, even when both are described as DoS.

A RAG defense should measure poisoning budget, retrieval rate, answer manipulation, source attribution, cross-user reach, persistence after deletion, and clean retrieval utility. Input-only filters are structurally incomplete because the attack carrier may enter after the filter. Retrieval-augmented jailbreak defenses can update safety examples without retraining [196], but their own databases become protected state and therefore require provenance and poisoning tests. Long-context results introduce the same concern. Increasing context length changes both the attack surface and the evidence available to the model, so safety should be tested on the assembled long context rather than inferred from short-input evaluations [197].

RAG compromise can be decomposed into stages. Let r be the event that a malicious item is retrieved, c the event that the item influences model behavior, and i the event that the specified security impact occurs. Writing $Pr(·)$ for probability, the chain rule gives

$$Pr(r\cap c\cap i)=Pr\left(r\right)Pr(c\mid r)Pr(i\mid r\cap c).$$

(4)

The conditional terms in Eq. (4) require conditioning events with positive probability. This factorization assumes no independence and describes the specified retrieval-mediated path, not a compromise that bypasses retrieval. Admission and ranking defenses target the retrieval term, structured prompting and instruction separation target the conditional influence term, and action gates target the conditional impact term. Reporting only the final answer obscures which control failed or succeeded and whether a different downstream action would remain exposed.

Memory requires the same stage separation. MPBench measures malicious writes and later behavioral influence conditioned on write success, while sleeper-memory evaluation also separates retrieval from later use [125,126]. Such conditional outcomes should not be reported as unconditional compromise rates. Evaluations should preserve intermediate memory state and test whether deletion removes both the stored payload and any derived summaries, procedures, or other retained effects.

Louck’s TMA-NM preprint provides an enforcement example. Origin-bound memory authority and action-bound authorization constrain consequential actions even when stored text is persuasive [198]. Its machine checking applies to a bounded model, and the protection depends on correct origin labeling, trustworthy authorization, and independent corroboration. Free-text answer bias and cross-agent shared memory fall outside its main guarantee. The result therefore supports a bounded action-integrity claim rather than a general conclusion that memory poisoning is solved.

Provenance must survive transformation. Documents may be chunked, embedded, deduplicated, summarized, translated, re-ranked, and combined before generation. A source label attached only to the original file is ineffective if that information is lost when derived chunks enter the assembled context. The system should therefore bind each retained span to its source, principal, ingestion time, policy, and integrity state. For high-risk actions, it should require corroboration from independent trusted sources and should not allow one retrieved item to supply both factual content and executable authority.

Deletion should also be treated as a security experiment, not just data maintenance. Removing a malicious source may leave embeddings, caches, summaries, conversation memory, fine-tuning examples, or downstream copies. A persistent-poisoning benchmark should first trigger the attack, then apply the documented remediation, and finally replay the original and adaptive attack variants across affected users. The relevant outcome is whether the malicious influence is removed within the stated retention and propagation boundary, not merely whether the original source disappears from its folder.

8.2. Agents, Tools, and Authority

An agent crosses from probabilistic interpretation into deterministic authority. Secure operation therefore depends on a chain of boundaries: source identity, instruction origin, planning, tool selection, argument construction, credential scope, object authorization, execution, and outcome verification. Prompt injection becomes harmful when untrusted content acquires authority at one of these transitions.

Empirical catalogs of application failures and reasoning-model vulnerabilities show that the weak point may lie in orchestration, guard logic, tool mediation, or model behavior rather than in a single prompt class [199,200]. The Agent Index likewise documents substantial variation in the technical and safety features of deployed systems [201]. This variation argues against evaluating an abstract “agent” without recording its memory, tools, planning loop, approval rules, and monitoring path.

AgentDojo provides task and security cases in an executable environment [4], while InjecAgent focuses on indirect injection in tool-integrated agents [3]. Task Shield checks whether an action contributes to the user goal [46]. MELON tests action independence after masking the user request [47]. AttriGuard uses counterfactual observation replay and reports resistance to its optimization-based attacks [124]. IPIGuard constrains execution through a planned tool-dependency graph [48]. These mechanisms provide complementary semantic, causal, and structural controls rather than forming a single scale of strength.

ASB extends agent evaluation to memory poisoning, mixed attacks, and a training-free Plan-of-Thought backdoor [202]. Its threat models include access to designated malicious tools, so an assurance comparison should preserve that capability rather than reducing the experiment to prompt access alone. An unauthorized tool call is also not equivalent to verified damage to an external object.

The main remaining gap is production authority. The cited agent benchmarks rely on simulated tools, whereas deployed workflows introduce credential scopes, shared records, network egress, and asynchronous effects. An L5 evaluation therefore needs object-level authorization outside the model, transaction controls, approval for irreversible operations, and before-and-after evidence over authoritative state. Back-Reveal illustrates another insertion route: a fine-tuned backdoored agent leaks memory through retrieval-tool arguments under semantic triggers [26]. Testing the tool software alone would miss a compromise that originates in the model.

Higher-realism test environments do not require real customer credentials. They can use seeded canary secrets, explicit object ownership, transactional state, scoped mock credentials, egress controls, delayed side effects, and rollback. The outcome oracle can then compare authoritative state before and after execution. This is more reliable than asking an LLM judge whether a simulated action appeared harmful, while avoiding the ethical risks of production testing.

Multi-agent systems add delegation and propagation. Agent Smith shows rapid spread from a poisoned image in simulated pairwise interactions among vision-language agents [67]. The result supports a propagation threat model, not a measured production outbreak. Relevant controls include identity, signed messages, capability attenuation, cross-hop provenance, loop bounds, and quarantine. Evaluation should record both propagation and any increase in authority across hops.

Timing creates a separate boundary. ChronosAttack changes only the arrival order of authentic tool responses under bounded delays and shows that asynchronous observation order can alter final decisions [203]. Unlike prompt injection, the attacker does not modify content. A matched defense must therefore bind decisions to synchronized evidence, test consistency across arrival orders, or delay commitment until required observations are complete.

Plans also create time-of-check to time-of-use gaps. A verifier may approve an abstract plan before the model changes arguments, receives a malicious tool response, or delegates to another agent. Approval should therefore bind the exact tool, typed arguments, principal, target object, and expiration time, with renewed authorization whenever relevant fields change after replanning. This is especially important when the language model serves as both planner and policy interpreter.

Tool interfaces create bidirectional attack surfaces. Tool outputs may inject instructions into later reasoning, while tool inputs may carry secrets or attacker-controlled payloads into external systems. A safe wrapper should preserve output provenance, cap returned content, remove active content where possible, and limit the authority that tool output can influence. On the outbound path, data-loss prevention and object authorization should inspect the serialized arguments because the model’s explanation does not establish that the transmitted data are appropriate.

Agent guardrails also differ in temporal scope. AGrail adapts safety checks from experience, while structural monitoring and deployment frameworks instrument trajectories rather than only final text [204,205,206]. Rapid protection frameworks emphasize operational integration [207]. Evaluations of these designs should report whether the guard state itself can be poisoned and whether monitoring actually prevents an action rather than merely flagging it.

8.3. Model and Application Supply Chains

The LLM supply chain includes data sources, preprocessing code, base models, adapters, quantizers, prompt templates, agent frameworks, tool packages, container images, and serving configuration [17]. Trust is transitive across these components. A signed application may load an unsigned adapter, and a verified model may execute through a compromised tool dependency. A threat model should therefore identify which organization produces, reviews, distributes, and activates each artifact.

Artifact integrity and behavioral integrity are related but distinct. Hashes establish byte-level consistency, while verified signatures can bind an artifact to a signing identity. Neither establishes that the publisher, training process, or resulting behavior is trustworthy. Behavioral promotion tests can reveal known unsafe behavior, but they cannot rule out unknown triggers. Stronger assurance therefore combines provenance, reproducible builds where feasible, static scanning, model and adapter inspection, adversarial behavioral testing, staged deployment, monitoring, and rollback. Pickle-based model poisoning further shows that packaging formats and loader behavior belong to the security boundary [55].

Fine-tuning services introduce another principal. A customer, contractor, or automated data pipeline may submit examples that erode safety or install conditional behavior. Promotion gates should compare the resulting model with the exact base version, measure both task gains and security regressions, inspect data lineage, and test plausible trigger families. Auditable fine-tuning and inference aim to reduce trust in proprietary execution [208], but verifying that a computation was performed correctly does not establish that the specified objective or data policy was safe.

Malice in Agentland connects operational injection to model production. A compromised webpage or tool response can steer a trace-collecting teacher, whose demonstrations then backdoor a fine-tuned agent [194]. This threat differs from direct dataset modification and from distributing an already backdoored model. The protected boundary therefore includes both the collection environment and the trace-admission policy. Benign task performance after training does not rule out trigger-conditioned exfiltration.

Incident response also requires dependency reachability. If an adapter, dataset, or library is revoked, operators must identify every derived model, index, deployment, and cached artifact that depends on it. A software bill of materials is necessary but insufficient because learned behavior may persist after removing the original artifact. A corresponding behavioral provenance record should therefore track parent checkpoints, data and adapter versions, promotion tests, and known failure signatures so that affected descendants can be identified and re-evaluated.

8.4. Service and Substrate

Serving combines users for efficiency through batching, caching, shared GPUs, schedulers, and shared base models. S-LoRA shows the efficiency incentive to host thousands of adapters on common infrastructure [209]. That capability also makes adapter identity, memory ownership, and cross-tenant scheduling into assurance concerns. Prompt-cache audits show that hit and miss latency can reveal whether a prefix was processed before [28]. Timing and local cache studies reconstruct or distinguish private activity [7,27], and MIG partitioning does not remove all cross-partition signals [62]. Attacker capability differs across these results. PromptPeek reconstructs prompts through shared KV-cache timing with ordinary client access, under assumptions about serving policy and tokenizer [210]. MoEcho exploits input-dependent mixture-of-experts execution through hardware side channels and needs platform-specific co-location, shared-library, or performance-counter access [211]. They are not interchangeable examples of a remote API attack.

Cache integrity is a separate concern from cache leakage. Cache Me, Catch You demonstrates inference-time prefix and multimodal collisions and semantic-cache poisoning, then evaluates namespace separation, secure serialization and hashing, and semantic checks [193]. These results concern the studied implementations and versions. A cache-isolation claim should name both confidentiality and integrity requirements and test stale entries and invalidation as well as timing leakage.

Defenses show the trade-off between security and utility. CacheSolidarity monitors cross-user reuse, selectively isolates suspicious prefixes, and reports higher reuse and lower latency than complete isolation [149]. GIFT tracks ownership flows on GPUs, and both GIFT and its confidential-computing extension report 4–10.7% throughput overhead in vLLM and DistServe under the same latency constraints, and they exclude side channels, denial of service, and inference-result integrity [167]. These studies use system metrics rather than text ASR, illustrating why a unified framework must retain domain-specific oracles.

Multi-tenancy exists above and below the GPU. API gateways share rate limits and logs, vector stores may share indexes, prefix and KV caches reuse model state, batchers combine requests, and accelerators share memory systems and interconnects. Isolation at one layer does not imply isolation at the next. A provider can separate GPU partitions and still leak through a shared prompt cache, or partition caches and still mix principals in a retrieval index. An assurance case should list every reuse mechanism and its ownership rules.

Timing leakage depends on configuration. Continuous batching, speculative decoding, quantization, network routing, and other users’ load can hide or amplify a signal. A lab demonstration with controlled co-residency shows that leakage is possible. Deployment claims also need measurements under realistic jitter, workloads, and attacker scheduling. Network-level prompt and trait leakage in local research agents shows that traffic patterns can stay informative even when message contents are protected [63]. Availability defenses interact with fairness. A scheduler that deprioritizes suspected expensive requests can protect aggregate throughput and starve legitimate long-context or accessibility use. Per-user quotas can be evaded through distributed accounts, and global caps can cause collateral denial of service. Evaluation should report benign tail latency by workload class, attacker amplification, false throttling, and recovery after a burst. Resource controls should also bound agent loops and tool fan-out, which token limits may not capture.

Telemetry closes the response loop but creates sensitive data. Prompts, retrieved content, tool arguments, and cache identifiers help detection and forensics, and they are also a high-value privacy target. Logs need minimization, access control, tenant separation, retention limits, integrity protection, and tested deletion. Private or confidential inference does not give end-to-end confidentiality if plaintext is later copied into a central monitoring system.

9. Deployment Archetypes and Assurance Cases

The same defense can support different claims in different deployments. Table 8 applies the threat and defense tuples to five common archetypes. These are worked assurance cases rather than reference architectures. Each illustrates what additional evidence is required as a model gains access to data, persistent state, external authority, or shared infrastructure. Frontier-risk assessment likewise emphasizes that capabilities, safeguards, evaluations, and incident evidence evolve together [212]. That breadth motivates assurance planning, but system-specific claims still require the measurable invariants below.

9.1. Application-facing Deployments

The public chat endpoint is the narrowest archetype, although its threat model is often understated. A user has prompt and output access with repeated queries ($k_q$) and may exploit sampling, session history, and multiple accounts. A defense stack may combine model alignment, prompt and output guards, and abuse limits. HarmBench, JailbreakBench, StrongREJECT, and JailMeter provide increasingly structured evidence for harmful-content outcomes [102,103,114,115]. The matched claim remains $I_{\mathrm{pol}}$: resistance to defined policy-violating output under a stated cumulative budget. The assurance case should therefore treat the model and guard as one stochastic service and report per-attempt and per-session success, parallel or repeated sampling, model and policy versions, guard thresholds, and clean refusal cost. Best-of-N and many-shot attacks show how query and context budgets alter the threat [41,42], while defense-aware re-evaluations show that a hidden or static guard cannot support an adaptive claim [21]. No conclusion about tools, private retrieval, or cross-tenant isolation follows unless those components are present and evaluated.

Enterprise retrieval assistants add principals, protected corpora, ingestion pipelines, retrieval, and citations. Their central invariant is not merely “answer safely.” It is that users receive only information they are authorized to access and that untrusted documents cannot change control policy. The attack path may involve malicious public content, a compromised internal source, ranking manipulation, indirect injection, or stale memory. PoisonedRAG, blocker-document work, and real-world indirect injection each expose part of this path [5,52,53]. Controls must therefore compose across the full pipeline. Admission authenticates sources, ACLs filter candidates before ranking, chunk-level provenance survives prompt assembly, retrieved text is treated as data rather than authority, disclosure checks use the requesting principal, and persistent state is versioned for deletion. Evaluation should seed canary records owned by different principals, allow an adaptive attacker to publish or modify documents within the stated role, and measure retrieval, influence, disclosure, and remediation separately. Clean answer quality must also be measured, because aggressive source rejection can appear secure simply by disabling useful retrieval.

Tool-using and delegated agents add $I_{\mathrm{act}}$ and $I_{\mathrm{ext}}$ consequences. Semantic defenses such as Task Shield, MELON, and AttriGuard attempt to verify intent or causal influence, while IPIGuard constrains execution structure [46,47,48,124]. The assurance case should place these controls before the point of no return and bind approvals to exact typed actions. Least-privilege credentials, object-level checks, transaction caps, idempotency, and sandboxing can remain effective even when the model is manipulated. Testing should expose the agent to untrusted documents and tool outputs, allow repeated planning and retries, include malicious or compromised tools, and preserve state across delegation. The relevant success condition is an unauthorized state transition rather than an unsafe explanation. Benign task completion should be measured in the same environment because repeated verification may block complex workflows or multiply operational cost. AgentDojo and InjecAgent provide executable foundations [3,4], but production claims require explicit principals, credential scopes, delayed effects, and remediation tests.

9.2. Model Hosting and Shared Infrastructure

Hosted fine-tuning moves the adversary into the model-production pipeline. A customer may control training examples or adapters, but should not be able to compromise other tenants, loader integrity, or downstream consumers. Virtual prompt injection, benign-data fine-tuning attacks, and poisoned model artifacts illustrate different persistent insertion channels [6,37,54]. A promotion gate therefore needs both artifact provenance and behavioral testing. It should use safe serialization, isolate training jobs, bind training data and code to the resulting artifact, compare the output with the exact parent version, search for suspicious targets or triggers, run safety and utility regressions, and stage deployment. BAIT, Patcher, and weak-to-strong defense provide examples of inversion, repair, and distillation-based mitigation [136,137,166]. The residual claim must still acknowledge unknown trigger families and compromise of the production process. Rollback evidence must also cover derived adapters and deployments rather than only the originally uploaded artifact.

Shared inference trades isolation for utilization. Prefix reuse, KV caches, continuous batching, GPU partitioning, and scheduling can create leakage or interference even when generated outputs comply with policy [7,27,28,62]. The relevant invariant must therefore quantify acceptable cross-principal leakage and service degradation. Defenses may disable sharing, partition resources, monitor reuse, track ownership, or enforce scheduling controls, but each choice affects throughput and latency. A matched evaluation needs an adaptive co-tenant, realistic workload noise, controlled victim secrets, repeated probes, and multiple placement conditions. It should report inference advantage over prior knowledge, attacker queries and elapsed time, benign throughput, tail latency, fairness, and false-isolation cost. CacheSolidarity and GIFT make these trade-offs explicit [149,167]. Availability and confidentiality should also be evaluated together, because an attacker may shape load to increase co-residency or amplify timing signals.

Across all five archetypes, assurance grows by composition rather than by replacing model-level evidence. Harmful-content benchmarks remain useful for an agent, but they cover only one layer of the system. Cryptographic inference may protect prompts in a shared service, while object authorization, resource isolation, and fairness remain separate requirements. The framework makes these boundaries explicit and prevents strong evidence in one deployment archetype from generalizing to another without the additional matched evidence the new system boundary requires.

10. Private and Verifiable Inference

Cryptographic and trusted-execution mechanisms reduce what infrastructure can observe, but they introduce a different trusted computing base and performance envelope. IRON provides private transformer inference using cryptographic protocols [171]. BOLT and Comet improve the efficiency of private transformer and LLM inference through protocol and sparsity optimizations [172,213]. EncryptedLLM and THOR use GPU-accelerated fully homomorphic or homomorphic encryption for transformer inference [168,214]. Power-Softmax targets secure inference over encrypted data [215], while zkGPT addresses verifiable inference rather than confidentiality alone [169].

These mechanisms do not replace prompt, retrieval, or agent controls. Encryption can protect data from the inference provider under a specified protocol, but it does not determine whether a user is authorized to retrieve a document or invoke a tool. Verifiability can establish that a committed computation was executed correctly even when the encoded policy itself is unsafe. TEEs and confidential GPUs change the trust boundary rather than eliminating it. GIFT-CC, for example, retains a trusted monitor and excludes side-channel attacks from its threat model [167]. Evaluation should therefore report the adversary role, leakage function, setup assumptions, security parameter, supported sequence length and model size, approximation-induced accuracy loss, latency, throughput, communication, memory, and composition with application controls.

The observable view of the defense exposes a tension between semantic controls, which often require plaintext context, and private inference, which seeks to conceal that context. Possible compositions include client-side guards, encrypted policy evaluation, split trust, minimal metadata exposure, and post-decryption action gating. An assurance claim should state which party can observe prompts, retrieved content, policy decisions, intermediate values, and outputs.

Private-inference systems should be compared only under equivalent functionality. Sequence length, batch size, model width, numerical precision, activation functions, and supported decoding strategy can dominate cost. A protocol evaluated for a single forward pass does not automatically provide a practical autoregressive service, and a system that accelerates linear layers may still incur substantial cost in nonlinear operations or communication. Accuracy should be measured after all approximations rather than inherited from the plaintext baseline. These functional differences, more than a single speedup figure, determine whether two systems provide comparable service.

Verifiability addresses a different question: whether the server executed the committed model and computation. zkGPT points toward verifiable inference [169], while auditable fine-tuning work addresses proprietary training and inference workflows [208]. The proved statement should include preprocessing, model version, decoding, and guard logic whenever those stages fall within the claimed scope. Any excluded stage remains outside the guarantee. A proof of one forward pass does not establish that the application used the proved output correctly, nor does it establish that the model satisfies a safety policy.

A compositional evaluation should therefore distinguish four channels: confidentiality of client inputs from the server, confidentiality of model parameters from the client, integrity of the executed computation, and authorization of application actions after inference. These channels require separate assessment, so this survey treats cryptographic and confidential-computing defenses as components of a broader assurance case rather than as universal secure-LLM solutions.

11. Benchmarks, Metrics, and Evaluator Reliability

Table 9 summarizes widely used evaluation resources and distinguishes the inferences each resource supports from those that require additional evidence. This distinction should determine the scope of claims derived from benchmark results.

ASR is not a universal metric because its denominator may include all prompts, only non-refused prompts, successful generations, or attempts that pass an initial filter. Success may also be determined by keyword matching, a classifier, the target model, an LLM judge, or a human reviewer, under different budgets and sampling temperatures. Statistical analyses of jailbreak success show that sampling and aggregation procedures must be specified because a single observed rate is not an intrinsic property of a model [217]. Cross-paper rankings based on ASR are therefore unreliable when these choices differ or remain unreported.

For a binary success criterion with resolved outcomes, let $n>0$ be the number of prespecified evaluation units, j their index, and $y_j\in \{0,1\}$ the verdict from oracle J for unit j, with one denoting attack success and zero otherwise. The empirical rate is $\widehat{\mathrm{ASR}}=n^{-1}{\sum }_{j=1}^n{y}_j$, where a unit may be an attempt, task, or session but must remain consistent within the reported rate. Unresolved outcomes should be reported explicitly rather than silently assigned zero or removed.

The direction of conditioning matters as much as ASR. The proportion of harmful outputs among accepted requests measures a guard differently from the proportion of attacks that eventually succeed after repeated attempts. An adversary with N trials may achieve a substantially higher cumulative success rate than single-query ASR suggests, as best-of-N evaluations make explicit [41]. If trials are independent and have the same per-trial success probability p, the probability of at least one success is $1-{(1-p)}^N$. Adaptive or correlated trials do not generally satisfy these assumptions, so per-session and budgeted success should be measured directly when they fail. For injection and agent evaluations, the denominator should distinguish attack delivery, detector passage, model influence, tool proposal, authorization, and execution. This stage-wise accounting reveals whether a defense blocks the attack carrier or only a particular downstream consequence. AgentDojo provides a useful basis through its joint security and utility tasks [4], although production extensions require persistent state and deterministic checks on external objects.

Utility should be measured in terms appropriate to the protected function. Chat-safety evaluations require false-refusal and helpfulness measures, agent evaluations require benign task completion and unnecessary-action blocking, and RAG evaluations require retrieval recall and answer grounding. Serving evaluations should report latency percentiles, throughput, and fairness, while private-inference evaluations should report approximation error, communication, and cost. A security-utility curve is therefore more informative than performance at a single selected threshold.

LLM judges enable evaluation at scale but remain susceptible to error and adversarial manipulation. StrongREJECT shows that apparent jailbreak success can be inflated by outputs that do not actually fulfill the malicious request [114]. JailMeter improves evidence extraction and identifies evaluator-targeted attacks and multimodality as limitations [115]. For system tests, deterministic assertions are preferable whenever available: exact object ownership, tool-argument equality, database state differences, network egress, or secret canaries. Timing and cache measurements also require statistical treatment. Human review is most useful for ambiguous intent and harm, but it does not replace observable system state.

Reproducibility is complicated by model drift. Proprietary endpoint names do not identify a stable artifact, and guard stacks may change without notice. Evaluations should record the date, API and policy versions, sampling settings, system prompt, tools, corpus, cache state, defense thresholds, and judge version. Releasing attack and defense code helps, but it is insufficient when hidden prompts or commercial middleware materially affect the result. Reproducibility and validity should therefore be reported separately. A perfectly reproducible benchmark can measure the wrong consequence, while a difficult-to-reproduce production observation can still reveal a real vulnerability. The former needs a stronger oracle or threat model; the latter needs timestamped evidence, independent confirmation, and a bounded claim.

Cross-model averages can also hide concentration. A defense may work on refusal-tuned chat models and fail on reasoning or multimodal models, while an attack may dominate only one model family. Results should therefore report per-model distributions, confidence intervals, and worst-case outcomes under a fixed budget. Model selection should include open artifacts for adaptive white-box testing and closed systems for deployed-stack evaluation when the claimed scope requires both. Neither should be treated as a universal proxy for the other.

12. Cross-Cutting Findings

The following patterns synthesize the curated collection and matrix ledger. They are qualitative findings rather than prevalence estimates and follow from the models in Section 2 and the evidence reviewed in Section 3 to Section 11.

1. Observability mismatch: Attackers may control documents, training artifacts, or shared-service observations that fall outside a semantic guard’s view. An evidentiary gap arises when no evaluated control observes or constrains the relevant transition. This does not itself predict failure, because authorization or isolation may still enforce the invariant without classifying the payload.

2. Authority mismatch: A control may observe an attack without having authority to stop it. An output moderator cannot reverse an executed tool call, and an alert is not equivalent to enforcement. Semantic verification can inform risk assessment, whereas deterministic authorization decides which principal may act on a protected object.

3. Static versus defense-aware evidence: Defense claims are bounded by the adversary used during development and evaluation. RPO, PAT, DPP, and DataSentinel strengthen the adaptive inner loop [34,35,36,165], but a new carrier, objective, or system path may still fall outside that loop. Defense-aware re-evaluations illustrate how conclusions can change when the attacker explicitly targets the defense [21,22]. System scope and adversarial strength are separate axes. ASB spans prompts, memory, and tools, while its tool-use criterion remains tied to the benchmark’s designated actions [202]. Sleeper-memory poisoning adds defense-aware optimization and separates write, retrieval, and use [125]. Conversely, optimization-heavy jailbreak studies may use strong adaptive search while still measuring only generated text. Neither broader scope nor stronger adaptation alone constitutes a realism score.

4. Persistence and state: Backdoors, RAG poisoning, safety memory, and cross-agent propagation make retained state central to assurance [5,37,67,127]. AgentDojo does not cover successive user tasks without resetting the agent’s context [4]. Persistent evaluation should instead preserve accumulation, cross-user exposure, write provenance, remediation, and replay across system updates.

5. Model-level versus system-level evidence: Alignment may reduce the probability that a malicious instruction is followed, but it cannot enforce database ownership, payment limits, cache tenancy, or cryptographic confidentiality. System controls can enforce these boundaries without necessarily recognizing harmful language. Secure deployment therefore requires an explicit handoff from model-level judgment to system-level enforcement.

6. Evaluator reliability and utility accounting: Judges and detectors can themselves be attacked, and their outputs are not equivalent to system state. Utility and cost also depend on the protected function. A guard requires false-positive and helpfulness measurements; an agent defense requires benign task completion; retrieval controls require retrieval and answer quality; and isolation or private-inference mechanisms require latency, throughput, communication, and fairness. PARDEN reports true- and false-positive operating points [159], Task Shield reports security together with task utility [46], and CacheSolidarity and GIFT quantify serving overhead [149,167]. These measures are not interchangeable, but each helps determine whether apparent security is achieved by disabling useful behavior or shifting cost elsewhere. Reproducibility should likewise distinguish artifact availability, exact environment reconstruction, and independent replication, especially for proprietary endpoints whose hidden moderation and versions may change.

7. Attacker and defender cost asymmetry. Defense cost is itself part of the attack surface. Repetition, ensemble judging, masked re-execution, and counterfactual attribution can multiply model calls [47,124,159]. A defense may therefore reduce semantic ASR while increasing denial-of-wallet or queue-exhaustion risk.

No composite score follows from these patterns. Adaptive testing, transfer, utility, system impact, and reproducibility are distinct dimensions, and strength on one does not compensate for an unsupported claim on another.

13. Research Agenda

The following research directions address gaps identified by the cross-cutting findings.

13.1. Matched Evaluation and Trustworthy Oracles

Findings 3 and 6 motivate machine-readable reporting of the threat tuple, defense tuple, model and system version, resource budget, and impact oracle for every leaderboard result. Scores should be stratified by adaptivity and consequence rather than pooled into a single ASR, and independent attack teams should receive sufficient time and access to study the defense. Because attackers may also target the evaluator or monitoring pipeline, benchmarks need protections against evaluator-specific overfitting. These can include hidden or rotating human-validated test sets, multiple evaluator families, deterministic system oracles where available, disagreement reporting, and attacks designed specifically to probe the evaluator. Each defense result should specify whether the attack targeted the model, detector, judge, or their composition.

13.2. System Invariants and Persistent Evaluation

Findings 1, 2, 4, and 5 motivate security invariants that remain enforceable outside the model. Untrusted content should not raise privilege or modify authorization policy. Every tool argument should remain bound to a user goal and principal, and irreversible actions should pass a deterministic gate. Memory writes should be attributable and revocable, while delegated capabilities should not increase without explicit authorization. Task Shield, MELON, AttriGuard, and IPIGuard advance enforceable action semantics [46,47,48,124]. Corresponding benchmarks should preserve state across turns and users, support poisoned documents and memories, compose multiple tools and agents, and expose remediation interfaces. Evaluation should determine whether remediation removes the vulnerability from memory, corpora, caches, and dependent agents rather than only whether the original triggering prompt stops working.

13.3. Cost, Provenance, and Reproducibility

Findings 5, 6, and 7 motivate joint accounting of token, query, energy, latency, memory, communication, and human-review costs for attackers and defenders. Agent loops and expensive safety escalations require resource invariants. At the same time, scheduler and caching defenses should be tested together with semantic controls because mitigation at one layer may shift cost or risk to another. Model and adapter provenance should connect to promotion tests, signed artifacts, runtime attestation, and behavioral regression. Private-inference systems should expose interfaces that preserve object authorization and tool policy without defeating confidentiality, and evaluations should identify which parties control keys, code, model weights, guard policies, and audit logs. Because closed systems evolve, researchers should publish timestamped transcripts, query budgets, prompt and tool schemas, independent labels, and sensitivity analyses for plausible version changes. When exact reproduction is impossible, claims should describe a dated service configuration rather than an entire model family.

14. Practical Assurance Procedure

A deployment-oriented evaluation can apply the framework in eight steps.

1.

Define the protected assets, principals, trust boundaries, and prohibited state transitions.

2.

Identify entry surfaces across prompts, external context, persistent stores, artifacts, tools, services, and infrastructure.

3.

Specify attacker knowledge, access, budget, adaptivity, and persistence for each scenario.

4.

Define the impact oracle, preferring deterministic state evidence over response labels where possible.

5.

Map each defense’s observable view, security function, and enforceable response.

6.

Run static baselines, transfer tests, and defense-aware attacks under comparable budgets.

7.

Measure clean utility, false positives, latency, throughput, cost, and human escalation.

8.

Reproduce successful attack paths, verify remediation, and retain version-bound regression tests.

The output is not a claim that an LLM is safe. It is a bounded assurance case of the form defined in Section 6. For a stated attacker, system version, resource budget, and set of consequences, the evaluation records which named attack paths the tested controls prevented or contained, together with utility costs, remaining gaps, and residual risk.

15. Conclusion

LLM security research reports attack success and defense effectiveness, but these results must be interpreted in the context in which they were obtained. The threat tuple specifies entry, access, adaptivity, objective, persistence, impact, and budget, while the defense tuple specifies what a control can observe and enforce. The assurance matrix records whether the evidence addresses these conditions, and the claim grammar states the boundaries of each result. The evidence ladder identifies empirical requirements for a fixed security claim and threat model rather than ranking papers or defenses. Across models, retrieval, memory, agents, supply chains, privacy, availability, and infrastructure, the framework explains why static input filtering does not establish agent security. Rather than treating a model or defense as universally secure, it identifies the attacker, system version, resource budget, consequence, cost, utility, and residual conditions to which the evidence applies. Claims against adaptive attackers require tests that allow adaptation to the defense and measure the stated consequence. Security evidence supports an asserted system property most directly when the evaluated attacker, observable boundary, enforcement mechanism, and outcome oracle match that property.