Benchmarking the Personalization Capabilities of Large Language Models

1. Introduction

Communication as defined by the seminal work of Lasswell (Lasswell (1948)) characterizes any communicative act in five variables: who says what to whom, through which channel, with what effect, and at what time. Within this framework personalization can be located as the act of varying the message, that is, the what, while conditioning on (and holding fixed) the speaker, receiver, channel, and time. The act has two parties whose interests need not initially align; a sender, who selects the message with the goal of inducing some action from the receiver, and a receiver, who has independent preferences and chooses whether to act. The study of how senders shape messages to induce receiver action has been a long tradition in multiple fields, including, psychology, economics, and marketing, beginning with the Yale Communication and Attitude Change program (Hovland et al. (1953)) and continuing through the Elaboration Likelihood Model (Petty and Cacioppo (1986)) and the formal signaling-game treatment of Bayesian Persuasion (Kamenica and Gentzkow (2011)).

So far, machine learning research on personalization has approached this problem as learning a policy to retrieve and rank items from a fixed inventory of candidates. Recommender systems rank items from a catalog against user-history signals (Ricci et al. (2010)); advertising platforms optimize the selection, and targeting of ads among a fixed pool of pre-authored creatives (Choi et al. (2020)); persona-based dialogue systems condition response generation on explicit persona representations while remaining restricted to comparatively narrow conversational domains (Zhang et al. (2018)).

Large language models transition this personalization from a retrieval and ranking problem to a generative one. LLMs can generate a continuum of message variants for a given (speaker, receiver, channel, time) tuple, conditioned on whatever attributes of the receiver can be inferred from the available context. A growing body of work has applied LLMs to generative-personalization in marketing (Matz et al. (2024)), education (Sharma et al. (2025); Tasdelen and Bodemer (2025)), and human-AI interaction (Chen et al. (2024)). These applied results raise the question of how well current LLMs perform personalization in the sense the classical literature studies it: as a sender selecting what to say in order to induce a specific receiver action. While some work exists for measuring LLM personalization, however, it measures a very different property compared to the personalization talked about in psychology and economics literature (Kamenica and Gentzkow (2011); Lasswell (1948)). LaMP (Salemi et al. (2024)) evaluates personalized text generation conditioned on a user’s history of past interactions. PersoBench (Afzoon et al. (2024)) measures persona consistency in open-domain dialogue. PersonaConvBench (Li et al. (2025)) scores persona-grounded conversational quality. PersonaLens (Zhao et al. (2025)) evaluates assistant behavior under declared user preferences. PersonaMem (Jiang et al. (2025)) measures long-horizon recall of user attributes across sessions. In these works, the recipient of the LLM generated message is the same user that the model is serving, and the optimization target is one-party: the alignment between the model’s output and the preferences of the user who issued the prompt, in the same sense that RLHF aligns an assistant to its user. The two-party question, in which sender and receiver have independent objectives and the sender’s success is measured by whether the receiver acts, has primarily been investigated through randomized human-subject experiments in which participants are exposed to human- or LLM-generated persuasive messages and evaluated based on subsequent shifts in attitudes, agreement, or behavioral intentions (Durmus et al. (2024);Matz et al. (2024)). Such studies are tied to a specific methodology, require weeks of execution, and cannot be re-run against a new model on demand. Consequently, comparisons between LLMs and human experts for two-party personalization remain specific to individual studies and are not directly comparable across systems. Therefore, there is a need for a formal model of personalization which accounts for both the sender and receiver of the message, sufficient to support a reproducible and automated benchmark applicable to arbitrary generative systems.

This formulation requires an empirical setting where receiver actions are observed and logged against the specific messages that induced them, the messages are authored at the receiver level rather than at the segment level, and human-authored ground truth at known successful induced actions are available at scale. Sales outreach artifacts provides a good testbed to measure this as receiver actions (replying, scheduling a call, closing a deal) are routinely logged against the specific outreach that induced them (Terho et al. (2022a)). A sales outreach is drafted one-to-one by a Sales Development Representative (SDR) for a specific prospect. In an in-house study conducted with a Fortune 100 enterprise, we observed that personalized SDR outreach achieved approximately seven times the click-through rate of templated outreach when promoting the same products to comparable prospect groups.. Furthermore, the sales funnel produces a layered set of human-authored artifacts at known successful transitions, namely outreach emails that secured a call, call transcripts that secured deal discussions, and post-deal customer success stories that document the content that closed the deal. Together, these properties make sales a natural empirical instantiation of the two-party formulation where each artifact in the funnel serves as ground truth at a different stage of the same (seller, prospect, product) tuple, enabling stage-specific evaluation of generative personalization.

We develop our framework SDR-Arena (illustrated in Figure 1) on this empirical setting and list our contributions are as follows:

• We adapt Bayesian Persuasion to generative agents, recovering personalization as informational alignment between an agent’s generated content and the receiver-specific content implicit in the ground-truth sales outreach artifact.
• We construct SDR-Bench, a public corpus of 6,279 customer success stories from approximately 200 enterprises across 22 industries, each paired with the seller, prospect, product, and historical timestamp required to evaluate whether an agent can predict the strategic content of the deal-closing pitch.
• We release SDR-Arena, an evaluation framework that operationalizes the formalization on SDR-Bench and on proprietary sales artifacts; to prevent future data leakage, where an agent retrieves the very success story it is being asked to predict, SDR-Arena serves agents a frozen view of the public web at the historical timestamp of each evaluation instance.
• We apply the framework to proprietary sales-email and sales-transcript corpora from a Fortune 100 tech company and a mid-sized healthcare firm, comprising approximately 115,000 filtered outreach emails and 5435 outreach calls by 124 SDRs labeled by whether they induced a successful receiver action.
• We validate the framework through field deployment with 12 professional sales development representatives across our partner enterprises and a gold-standard exercise with senior SDRs from five enterprises

Figure 1. Overview of SDR-Arena showcasing how LLM generated output is compared with artifacts like Sales Emails, Transcripts & Success Stories to benchmark their personalization capability.

Figure 1. Overview of SDR-Arena showcasing how LLM generated output is compared with artifacts like Sales Emails, Transcripts & Success Stories to benchmark their personalization capability.

Across frontier LLMs and open-source deep-research agents, including STORM (Shao et al. (2024)), ODR (LangChain (2025)), GPT-4o (OpenAI (2023)), Claude Sonnet 4.6 (Anthropic (2026)) and Qwen-2.5 ((Yang et al. (2024)), we observe a consistent personalization plateau. Alignment scores cluster in the 30 to 43 percent range, and on the tech-firm cohort no model statistically separates successful from unsuccessful outreach. Specialized agents such as STORM reach the upper end of the range, but at one to two orders of magnitude greater inference cost; standard LLMs with temporally constrained search occupy a more compute-efficient frontier.

2. Problem Formulation

We formalize the empirical sales in the form of generative personalization by first describing the Sales Development Lifecycle, then casting personalization as a Bayesian Persuasion task where the generated outreach is aimed at inducing specific actions from the recipients within the sales funnel.

2.1. Sales Development Lifecycle

A sales journey (illustrated in Figure 2) begins with Sales Development Representatives (SDRs) researching prospective accounts to identify needs and budget signals, then sending tailored outreach emails to schedule an initial call. The call further develops the prospect’s needs and progresses toward deal closure, with some opportunities materializing into deals and others not. Following a successful closure, the workflow often culminates in a Customer Success Story: a publicly documented case study published by the Seller company showcasing how their products helped a customer overcome key challenges. Released as web articles, these stories validate the partnership by documenting the transition from a ‘pain state’ to a ‘success state’ (Terho et al. (2022b)). Examples include success stories from Oracle, Salesforce, and Adobe.

Figure 2. Sales Journey: From Prospecting to Outreach to Call and eventual Deal Closure leading to Success Story publication.

Figure 2. Sales Journey: From Prospecting to Outreach to Call and eventual Deal Closure leading to Success Story publication.

2.2. Bayesian Persuasion Formulation

We adapt the Bayesian Persuasion framework of Kamenica (Kamenica and Gentzkow (2011)) to model personalized outreach as a signaling game between a Sender (the SDR or the LLM agent that replaces the SDR) and a Receiver (the prospect). This framework naturally aligns for personalization because it acknowledges that the receiver enters the interaction with a prior belief about their own needs, and the sender’s role is to provide a signal, the personalized message, that updates the receiver’s posterior in favor of action.

The unobserved receiver state $\omega$ represents the latent compatibility between the receiver’s requirements and the sender’s product. We decompose $\omega$ at time t as a tuple ${\omega }_t=\{n_i,w_i\}$ where n denotes the receiver’s explicit needs (functional requirements, current pain points) and w denotes their latent wants (strategic goals, avenues for value generation). The receiver chooses an action $a\in \{0,1\}$, where $a=1$ represents a successful transition to the next stage of the sales funnel (e.g., an email leading to a call, or a call leading to deal discussions), while $a=0$ represents a failure to progress.

Following Kamenica (Kamenica and Gentzkow (2011)), the Receiver is treated as a rational Bayesian agent with a prior belief $\mu$ over $\omega$. The Receiver takes action $a=1$ if and only if their expected utility $u_R$, conditional on their belief, exceeds a reservation threshold $\tau$:

$$\mathbb{E}[u_R(a=1,{\omega }_t,\xi )\mid \mu ]\ge \tau$$

The Sender’s utility $u_S$ is aligned with the Receiver acting $a=1$. Thus, the Sender’s goal is to deliver a signal that updates the receiver’s posterior belief such that the above condition is satisfied. Note that the utility function is also subject to exogeneous factors $\xi$ (timing, organizational urgency, prior context, noise) that are independent of the sender’s signal but contribute to the receiver’s utility. The sender’s signal can shift $\mu$ in favor of acting; it cannot control $\xi$. Consequently, even an optimal signal is not guaranteed to induce $a=1$, and the receiver’s action is best understood as a probabilistic outcome whose likelihood the sender attempts to maximize.

The key adaptation for our setting is that the sender does not observe ${\omega }_t$ directly. The sender (in our case an LLM agent $\Phi$) operates on an observable context $W_t$ (state of the world at time t) consisting of public information available at time t , from which the latent state must be inferred. From this inference, the agent generates an outreach O, which we represent structurally as a list of pitch points. Each pitch point is a specific argument linking the seller’s product to one of the receiver’s inferred needs or wants through a particular value proposition. Formally, the agent’s policy is a mapping:

$$O=\{p{p}_1,p{p}_2,\dots ,p{p}_k\},\widehat{O}=\{{\widehat{pp}}_1,\dots ,{\widehat{pp}}_n\}=\Phi (W_t,P\mid \widehat{\omega })$$

where $\widehat{\omega }$ is the agent’s inferred receiver state, $\widehat{O}$ is the outreach generated by agent $\Phi$ and P is the product. The generated list serves as the informational signal intended to update the receiver’s posterior such that the probability of the desired stage transition ($a=1$) is maximized.

2.3. Evaluation Methodology and Proxy

The ideal objective of the Sender is to generate an outreach O that maximizes the Receiver’s expected utility: $O^{*}={arg\; max}_O\mathbb{E}[u_R(a=1,\omega )\mid O]$. Direct evaluation of this objective is intractable as the receiver’s utility function $u_R$ and the true state $\omega$ are unobservable to the agent (and to the benchmark). We therefore replace the unobservable utility with an observable proxy - the informational overlap between the agent’s generated content and the content implicit in a human-authored message that is known to have induced the desired action.

We utilize a dataset of successful historical outreach attempts across different engagement stages (Emails, Calls, and Success Stories). Let $\mathcal{D}={\left\{(C_i,{\omega }_i,O_i^{*})\right\}}_{i=1}^N$ be a set of ground truth examples where $O_i^{*}$ is a human-authored message that successfully induced action $a=1$ (moving to the next funnel stage).Success in email is defined by an outreach $O_i^{*}$ that led to a call; a call success implies $O_i^{*}$ led to deal discussions; and a success story implies $O_i^{*}$ led to deal closure. Because each $O_i^{*}$ resulted in a positive outcome, it empirically satisfies the receiver’s utility threshold and can be treated as a sample from the set of utility-maximizing messages for the corresponding (sender, receiver, product, time) tuple. We extract a list of ground truth pitch points $V^{*}$ from $O_i^{*}$ and define the Weighted Coverage Score (WCS) as the semantic alignment between the predicted pitch points $\widehat{O}$ and the ground truth points $V^{*}$:

$$\mathrm{Weighted}\mathrm{Coverage}\mathrm{Score}=\mathcal{S}(\widehat{O},V^{*})$$

where $S(·,·)$ is a semantic alignment scoring function (defined in Section 3.1).

This allows us to rigorously benchmark Agent performance by measuring the Relevance Alignment between the predicted pitch points (where the value proposition is embedded) against those extracted from the successful ground truth. This serves as a tractable proxy for the Bayesian persuasion objective: a higher matching score implies the Agent has successfully identified the winning strategy that induces the desired action.

WCS is, by construction, a lower bound on personalization quality. It captures the what component of personalization — whether the agent has correctly inferred the receiver’s decision-relevant needs and wants and identified the value-generating arguments that historically induced the desired action while abstracting away the how (style, tone, formatting). An agent that achieves high WCS has demonstrated that it can recover the receiver-specific strategic content of a known successful message; an agent that achieves low WCS has not, regardless of how well-written its output is.

2.4. Dataset Construction

To evaluate generative personalization in real-world settings, we construct two complementary datasets: (i) SDR-Bench, a large-scale public benchmark derived from enterprise sales artifacts, and (ii) a private enterprise dataset containing real sales outreach emails and downstream sales outcomes from two organizations. Together, these datasets provide both reproducible public evaluation and high-fidelity validation on real-world personalized communication.

SDR-Bench Dataset

To enable reproducible public benchmarking of generative personalization systems, we construct SDR-Bench, a large-scale corpus of publicly available enterprise sales narratives and customer success stories. We targeted approximately 12,000 global enterprises with revenues exceeding $1B, identifying sitemaps for 8,298 organizations and collecting 117,000 candidate URLs using heuristics tailored to common success-story paths (e.g., /customer-stories, /case-study).

A multi-stage filtering pipeline (Table 3) removed non-text formats, generic landing pages, articles lacking verifiable publication dates or identifiable product solutions, and anonymized stories where the customer organization was not explicitly named (e.g., “a large food products company”). This process yielded a final corpus of 6,279 success-story articles spanning 22 industries. Distributions of companies and stories by industry are shown in Figure 4 and Appendix Figure 6. Detailed construction steps are provided in Appendix A.1.

Private Enterprise Dataset

To validate our theoretical proxy, we require settings where the ground-truth message $O_i^{*}$ and its successful outcome $(a=1)$ are explicitly observed. We therefore collaborated with two enterprises—a Fortune 100 technology company and a mid-sized healthcare firm—to collect real human-authored sales outreach paired with downstream prospect actions.

For the Fortune 100 company, we collected approximately 100k outreach emails authored by 124 SDRs and 5,435 sales call transcripts over a two-year period (2023–2025), identifying 13,236 instances in which the outreach successfully induced a sales call. For the healthcare firm, we analyzed 24,506 outreach emails, of which 354 resulted in a scheduled sales call. These successful outreach instances serve as observed realizations of optimal messages ($O^{*}$) in our relevance-alignment framework. Table 1 summarizes the dataset construction pipeline.

Table 1. Processing of Enterprise Sales Email Data.

Metric / Artifact Category	Healthcare	Tech
Number of SDRs	3	124
Total Emails Collected	48,150	609,191
Deduplication	31,034	186,379
Sales Outreach Emails	24,506	90,809
Sales Call scheduled	354	13,236
Golden dataset handpicked	400	400

Table 1. Processing of Enterprise Sales Email Data.

Metric / Artifact Category	Healthcare	Tech
Number of SDRs	3	124
Total Emails Collected	48,150	609,191
Deduplication	31,034	186,379
Sales Outreach Emails	24,506	90,809
Sales Call scheduled	354	13,236
Golden dataset handpicked	400	400

Human Personalization Strategies

To characterize the qualitative structure of expert personalization, we analyzed the strategies employed by SDRs across both datasets (Figure 3). The three most common strategies were: (i) industry-based personalization, tailoring content to sector-specific trends and pain points; (ii) persona-based personalization, adapting the value proposition to the recipient’s organizational role; and (iii) activity-based personalization, leveraging behavioral signals such as webinar attendance or prior engagement.

Appendix Figure 5 provides qualitative examples showing how SDRs adapt the same product positioning differently across recipients with distinct inferred needs ($n_i$) and wants ($w_i$).

Figure 3. Distribution of count of strategies across a random subset of 34,000 emails.

Figure 3. Distribution of count of strategies across a random subset of 34,000 emails.

Figure 4. Distribution of count of companies by Industry Type.

Figure 4. Distribution of count of companies by Industry Type.

3. SDR-Arena

We introduce SDR-Arena, a scalable framework designed to systematically benchmark LLM-based agents on generative personalization over sales outreach artifacts. To ensure a rigorous and valid evaluation, the arena utilizes an isolated environment that provides agents access to a Historical Internet Simulator ($W_t$). The arena serves as a standardized testbed for comparing diverse agentic workflows, ranging from complex research pipelines to simple tool-use configurations. We evaluate two primary configurations on SDR-Bench and our Enterprise Dataset:

• LLMs + Web Search: A baseline equipping frontier models with standard search tools to measure the marginal utility of agentic workflows against simpler tool-use capabilities.
• Deep Research Agents: Specialized agents that produce comprehensive research via multi-turn conversation and broad search retrieval over the internet (LangChain (2025); Shao et al. (2024)).

Historical Internet Simulator: This environment prevents “future leakage” by enforcing a strict temporal boundary, ensuring agents only synthesize information that was publicly available at the simulated time of the sales interaction. The system enforces the $W_t$ boundary by passing search_start_date and search_end_date parameters to the BrightData SERP API (Bright Data (2026)). By restricting results to time t, we ensure that the generated pitch points are constructed solely from context that would have been accessible to a human researcher at the time of the original sales event, preventing the model from ‘cheating’, where an agent might mistakenly find the successful outcome of a deal that hasn’t happened yet in the simulation.

3.1. Evaluation Framework

We define each evaluation instance as a tuple $(S,C,P,t)$, where S is the seller, C is the prospect, P represents the products, and t is the historical timestamp. The tuple is extracted from each sales artifact individually. Please refer Appendix A.4 for examples.

Implementation: The agent is prompted to act as a sales representative for S pitching P to C using the time-restricted search tool. The resulting output $\widehat{O}=\{{\widehat{pp}}_1,\dots ,{\widehat{pp}}_n\}$ is a set of personalized pitch points intended to address the inferred needs and strategic goals of the prospect. We employ an LLM-based semantic judge to extract ground truth pitch points $V^{*}$ from the historical sales artifact. We use the raw content of the sales artifact and employ GPT-4o OpenAI (2023) to perform an ontological extraction of ‘Pitch Points.’ Each pitch point is required to follow a strict triad structure: Product/Service → Specific Pain Point → Value Proposition/Mechanism. To ensure the pitch points are grounded, the extraction model was instructed to provide and validate pitch points with exact ‘evidence quotes’ from the source text for every claim. An expert study verified the precision and recall of this extraction to be $0.92$ and $0.97$ respectively, showing strong alignment with expert judgment. Refer to App. Section A.8 for more details.

For each $pp\in V^{*}$, the judge evaluates whether the agent’s output $\widehat{O}$ successfully covered the point. Performance is measured by the Coverage Score, defined as the fraction of ground truth strategic value propositions successfully recovered by the agent. We employ a Coverage Judge relying on a 5-point Likert scale that grades Sales Effectiveness and Factual Precision, ranging from 0 (Miss / Irrelevant) through 1 (Marketing Fluff), 2 (Topic Match), 3 (Implied / Soft Match), 4 (Strong Sales Argument), up to 5 (Strategic Bullseye) - A perfect extraction that captures the exact pain point of the recipient and the specific mechanism the product provides to address it. The full scoring rubric and the judge prompt are in the Appendix.

Weighted Coverage Score (WCS): This metric normalizes the Likert-scale evaluations into a percentage representing the agent’s completeness in capturing the winning sales logic. For a given success story with N ground truth pitch points, let $s_i\in \{0,\dots ,5\}$ be the score assigned by the judge for the i-th point. The WCS is calculated as:

$$\mathrm{WCS}=\left({\displaystyle \frac{{\sum }_{i=1}^N{s}_i}{5N}}\right)\times 100\%$$

A score of 100% implies that the agent successfully predicted every critical deal-winning argument with maximum specificity. It is important to note that this is a prediction task rather than a retrieval task: the ground truth serves as a future artifact, and agents must predict these winning points using only historical data available at time t. This metric measures the semantic alignment (Section 2.3) of agent outputs against these future artifacts resulting in a realistic back testing scenario.

4. Results & Analysis

We evaluate models across two categories: frontier LLMs augmented with the temporally-restricted SDR-Arena web-search tool, comprising Claude Sonnet 4.6, GPT-4o, GPT-4o-mini, GPT-5.4, GPT-5.4-mini and QWEN-2.5-72B and deep research agents, STORM and ODR, both built on QWEN-2.5-72B. These configurations are evaluated across three corpora. The first is a public corpus of curated customer success stories partitioned by industry: Technology, Manufacturing, Energy, and IT, with an aggregate set of 180 stories. The second is a corpus of transcripts of sales calls from a company exceeding $10B in revenue. The third is a corpus of human-authored enterprise sales emails, divided into two cohorts: a Healthcare company with under $1B in revenue and a Technology company exceeding $10B in revenue, each containing 200 successful and 200 unsuccessful emails.

4.1. Discussion of Empirical Findings

We observe several notable trends. First, Claude Sonnet 4.6 leads all models with an aggregate WCS of 55.8 on the public success story dataset, representing a meaningful gap above the next best model, GPT-5.4-mini at 44.63. Despite this it only recovers roughly half of the strategic content of the human-authored success story, indicating a clear personalization plateau across all agent families. Second, frontier LLMs are a more cost-efficient alternative to deep research agents. Claude Sonnet 4.6 achieves the highest WCS at an inference cost comparable to ODR ( $0.270 vs.  $0.250), while surpassing it by more than 20 WCS points.

The Enterprise Sales Email cohorts (Table 2) reveal a sector-dependent pattern. In the Healthcare cohort, models more consistently assign higher scores to successful outreach than unsuccessful outreach (e.g., STORM: $32.27$ vs. $22.46$), suggesting they capture personalization cues relevant to specialized, high-stakes sectors. In the Technology cohort, however, this pattern inverts or collapses: several models score unsuccessful emails comparably to or higher than successful ones (e.g., STORM: $43.15$ vs. $39.24$), indicating that models generate coherent but strategically shallow content insufficient to drive real-world revenue in competitive markets.

Table 2. Ground truth artifact evaluation. Values reported as Coverage.

	Enterprise Sales Emails					Success Stories					Average Cost per Query
Model	Healthcare (<1B)		Tech (>10B)		Other	Tech.	Mfg.	Energy	IT	Agg.	Prompt	Completion	Cost
	Unsucc	Succ	Unsucc	Succ	Sales Call Transcripts	(30)	(30)	(30)	(30)	(180)	Tokens	Tokens	($)
STORM-QWEN-2.5	22.46	32.27	43.15	39.24	30.43	43.40	41.59	39.24	44.60	42.51	∼29k	∼6.3k	∼0.135
ODR-QWEN-2.5	15.40	30.41	38.51	39.82	22.55	30.16	30.95	35.28	35.33	33.53	∼66k	∼8.6k	∼0.250
QWEN2.5-72B (WEB)	32.11	36.72	39.53	36.43	25.34	32.09	36.75	37.04	38.21	36.84	∼5.6k	∼0.25k	∼0.002
Claude Sonnet 4.6 (WEB)	59.89	60.89	64.08	61.33	34.52	52.6	60.5	54	54.9	55.8	∼79.2k	∼2.2k	∼0.270
GPT-4o (WEB)	35.71	40.62	47.44	45.17	—	33.50	38.93	32.26	36.01	35.42	∼9.2k	∼0.4k	∼0.027
GPT-4o-mini (WEB)	36.16	39.14	48.51	45.63	—	33.30	38.54	37.07	38.05	37.46	∼12.2k	∼0.6k	∼0.002
GPT-5.4-mini (WEB)	39.28	43.67	54.64	52.66	—	40.99	46.02	41.18	45.39	44.63	∼7.6k	∼0.7k	∼0.009
GPT-5.4 (WEB)	39.57	48.71	53.80	53.02	—	38.21	45.46	42.90	46.67	44.32	∼12.1k	∼0.9k	∼0.044

Table 2. Ground truth artifact evaluation. Values reported as Coverage.

	Enterprise Sales Emails					Success Stories					Average Cost per Query
Model	Healthcare (<1B)		Tech (>10B)		Other	Tech.	Mfg.	Energy	IT	Agg.	Prompt	Completion	Cost
	Unsucc	Succ	Unsucc	Succ	Sales Call Transcripts	(30)	(30)	(30)	(30)	(180)	Tokens	Tokens	($)
STORM-QWEN-2.5	22.46	32.27	43.15	39.24	30.43	43.40	41.59	39.24	44.60	42.51	∼29k	∼6.3k	∼0.135
ODR-QWEN-2.5	15.40	30.41	38.51	39.82	22.55	30.16	30.95	35.28	35.33	33.53	∼66k	∼8.6k	∼0.250
QWEN2.5-72B (WEB)	32.11	36.72	39.53	36.43	25.34	32.09	36.75	37.04	38.21	36.84	∼5.6k	∼0.25k	∼0.002
Claude Sonnet 4.6 (WEB)	59.89	60.89	64.08	61.33	34.52	52.6	60.5	54	54.9	55.8	∼79.2k	∼2.2k	∼0.270
GPT-4o (WEB)	35.71	40.62	47.44	45.17	—	33.50	38.93	32.26	36.01	35.42	∼9.2k	∼0.4k	∼0.027
GPT-4o-mini (WEB)	36.16	39.14	48.51	45.63	—	33.30	38.54	37.07	38.05	37.46	∼12.2k	∼0.6k	∼0.002
GPT-5.4-mini (WEB)	39.28	43.67	54.64	52.66	—	40.99	46.02	41.18	45.39	44.63	∼7.6k	∼0.7k	∼0.009
GPT-5.4 (WEB)	39.57	48.71	53.80	53.02	—	38.21	45.46	42.90	46.67	44.32	∼12.1k	∼0.9k	∼0.044

Pre-Training Leakage Is Not Driving WCS

A complementary concern is that publicly indexed success stories in SDR-Bench may have appeared in LLM pre-training corpora, inflating WCS through memorization rather than genuine inference. To probe this, we partition SDR-Bench by article publication date and re-evaluate on pre-2024 vs. post-2024 cohorts; since GPT-4o’s training cutoff sits between Q4-2023 and early 2024, post-2024 stories are unlikely to have been seen during pre-training. We observe negligible WCS differences across the split (STORM: $0.42$ vs. $0.43$; GPT-4o: $0.36$ vs. $0.36$). The absence of pre-training-era inflation indicates that performance on SDR-Bench reflects context-conditioned synthesis, not retrieval of memorized content.

4.2. Human Alignment and Validation

To ensure that our automated metrics reliably reflect real-world quality, we conducted expert studies calibrating our Coverage Judge against independent human raters, and confirming practical utility through deployment with professional sales representatives.

To show that the Coverage Judge follows human judgement, we conducted a human study on 20 success stories (80 model responses across STORM, ODR, GPT, and Qwen). Three independent human annotators, blinded to model identities, scored coverage following the exact protocol of our LLM-based Coverage Judge.

The study yields three convergent signals on Judge fidelity. (i) The Judge tracks human scores with strong rank correlation (Spearman’s $\rho =0.7435$, $p<0.0001$), holding across models (ODR: $0.7768$; GPT-4o: $0.7575$; QWEN-2.5: $0.7330$; STORM: $0.6963$). (ii) The Judge preserves model ordering: both human- and Judge-graded WCS rank STORM > GPT-4o > ODR, so absolute-score differences do not distort comparative conclusions. (iii) The Judge is systematically more conservative than human raters (STORM: $48.06$ vs. $55.29$; GPT-4o: $37.94$ vs. $46.04$; ODR: $31.10$ vs. $41.69$), ruling out score inflation and establishing WCS as a rigorous lower bound that tracks human intuition at scale.

We show that the WCS-based ranking transfers to expert SDR judgment in two field studies with 12 senior SDRs from the partner enterprises whose data appears in this paper. The first measures per-pitch usefulness - whether any individual model-generated pitch point would be used verbatim in real outreach, and the second measures strategy-level overlap between model output and SDR-authored gold standards. Together they probe the two granularities at which an automated score can mismatch expert judgment: per-point quality and overall strategic match.

• Per-pitch usefulness: Twelve SDRs used GPT-4o + SDR-Arena to generate pitch points for $200+$ new prospect companies inside their normal outbound pipeline, with each SDR auditing the model output on accounts they were actively working. For every generated pitch point, the SDR rated, on a binary criterion, whether it both (a) reflected genuine understanding of the prospect’s pain points and (b) was usable in outreach without rewriting; 48.2% of pitch points met both criteria.The field rate corresponds to roughly half of agent output being expert-grade in live deployment, with the remaining points being factually accurate but strategically generic, directly consistent with the personalization plateau identified above.
• Gold-Standard Alignment:We asked senior SDRs ($\ge 10$ years of industry experience and $\ge 5$ years at the firm) from five enterprises to independently author reference “gold-standard” strategies to pitch 30 products to 5 prospects each. The SDRs were not shown any model output during the exercise, so the reference strategies are an independent expert read of what should be pitched. We then computed the overlap between the gold-standard strategy and the outputs of the four benchmarked models, and correlated this expert-overlap score against the corresponding automated WCS on SDR-Bench. Across models, expert overlap and WCS track at Pearson $r=0.816$. The WCS-based ranking therefore transfers to senior-SDR judgment without re-tuning the rubric per enterprise, supporting WCS as a calibrated proxy for whether an agent has identified the strategic content a domain expert would pitch.

Together, these studies establish that our pipeline’s outputs are both factually grounded and meaningful in live sales contexts.

We also evaluate a closed source deep research agent, GEMINI-2.5-PRO-DR on a separate 25-story subset. Its Deep Research API does not expose temporal-restriction parameters and its higher inference cost precludes broader evaluation. On this subset, it achieves a WCS of 62.63. Notably, the margin between this score and that of Claude Sonnet 4.6 with web search remains narrow, further underscoring that frontier LLMs with web search constitute a cost-effective alternative to deep research agents.

5. Conclusion

In this work, we introduced SDR-Arena, the first comprehensive framework for benchmarking the generative personalization capabilities of Large Language Models. By grounding our evaluation in the Bayesian Persuasion framework, we transitioned from subjective assessments of "quality" to a rigorous measure of Relevance Alignment. Our experiments utilize SDR-Bench—a novel, high-fidelity corpus of over 6,200 success stories—and a unique enterprise-scale dataset of successful sales outreach to quantify how effectively LLMs can synthesize winning strategic arguments.

Our findings reveal a significant “personalization plateau.”, showing a substantial gap remains between AI-generated outreach and human-level strategic proficiency.

By releasing SDR-Arena, we provide the research community with the tools necessary to study autonomous personalization while strictly controlling for data leakage. As LLMs continue to move into high-stakes business operations, we hope this framework serves as a foundation for developing AI agents that are not only persuasive but verifiably aligned with the nuanced needs of their human recipients.