CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants

1. Introduction

Target-oriented dialogue systems aim to guide multi-turn interactions toward predefined goals, such as recommendation, persuasion, or task completion [1,2,3]. Existing studies have achieved strong progress in goal planning, response generation, and multi-turn decision-making [4]. However, most of them assume a relatively simple setting: the system directly interacts with the user whose goal should be completed. This single-goal assumption does not fully capture the proxy setting of phone call assistants.

A phone call assistant answers incoming calls on behalf of the device owner. Therefore, it must coordinate two different sides of goals: the owner’s explicit preset and the caller’s implicit, dynamic goal. These two goals may be aligned, complementary, irrelevant, or conflicting. For example, the owner may preset the instruction “leave the takeout at the door”, while the caller reports that the food was spilled. Blindly advancing the preset in such cases may lead to an irrelevant or even harmful response. Thus, the key challenge is not merely completing a target, but deciding when the owner-side goal should be advanced, delayed, or suspended according to the evolving caller-side situation.

This proxy setting also introduces strict safety boundaries. The assistant must not reveal the owner’s private information, fabricate unsupported facts, claim access to the physical world, or make decisions on behalf of the owner. In practice, the assistant can easily drift from reasonable assistance to unauthorized decision-making, such as agreeing to changes, making commitments, or refusing requests without the owner’s confirmation. Therefore, safety is not an additional requirement after goal completion, but a fundamental condition for deploying phone call assistants.

Existing dialogue benchmarks are insufficient for evaluating these abilities. Traditional task-oriented dialogue datasets mainly focus on intent recognition, dialogue state tracking, API calls, and task success under user-centered goals [5,6]. Target-oriented dialogue benchmarks emphasize goal advancement, but usually assume a single stable target [1,2,3,4]. Recent LLM-based dialogue and agent benchmarks evaluate general response quality, tool use, or policy compliance [7,8,9], yet they rarely consider a proxy assistant that must mediate between two independent persons under strict safety constraints. As a result, we still lack a benchmark for evaluating dual-goal coordination in phone call assistants.

To address this gap, we introduce CallBench, a Chinese benchmark for evaluating Dual-Goal Coordination in phone call assistants. Although real call assistants operate through voice interfaces, most practical systems follow an ASR–text processing–TTS pipeline. We therefore focus on the text-based dialogue processing stage, which allows us to isolate decision-making and response-generation abilities from speech recognition and synthesis errors. CallBench contains 50,000 complete multi-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment. The benchmark covers regular presets, emergent presets, and no-preset cases, and includes diverse relations between owner-side and caller-side goals. Figure 1 shows a regular-preset takeout example in CallBench, where the assistant must first handle a caller-side exception before returning to the owner-side preset.

We further design a preset-aware turn-level evaluation protocol for phone call assistants. Instead of relying only on task success or surface-level response quality, our evaluation measures semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety. This allows CallBench to reveal whether an assistant can make appropriate local decisions under dual-goal conditions, such as conveying a preset at the right time, handling caller-side exceptions, avoiding premature or repeated preset delivery, and staying within proxy-specific safety boundaries.

Our main contributions are as follows:

• We formulate the call assistant task as a turn-level decision process, dynamically coordinating the owner’s explicit targets and the caller’s implicit intentions under strict safety constraints.
• We construct CallBench, a Chinese multi-scenario benchmark containing 50,000 complete multi-turn phone call dialogues across six scenarios, three preset cases, and diverse dual-goal relations.
• We deploy a fine-grained evaluation framework to assess model capabilities across multiple aspects.
• We evaluate representative task- and target-oriented dialogue frameworks, quantifying critical bottlenecks in dual-goal trade-offs, exception handling, and boundary control to guide future research.

2. Related Work

2.1. Task- and Target-Oriented Dialogue Benchmarks

Task-oriented dialogue benchmarks have been widely used to evaluate dialogue state tracking, intent recognition, and task completion. Representative datasets such as MultiWOZ [5], SGD [6], and CrossWOZ [10] focus on multi-domain slot filling and API-oriented task execution. Other datasets further introduce more realistic interaction settings, including transactional dialogues in Taskmaster [11] and AirDialogue [12], implicit preference elicitation in Decision-Oriented Dialogue [13], and negotiation in CaSiNo [14]. These benchmarks have advanced the evaluation of multi-turn dialogue systems under explicit or partially implicit user goals.

However, most of these benchmarks assume that the system directly serves the current interactor. The system’s role is usually to understand and complete the user’s own task, rather than act as a proxy for another person. This assumption does not match phone call assistants, where the assistant must coordinate the device owner’s explicit preset with the caller’s dynamic implicit goal. Existing benchmarks provide limited coverage of cases where the two goals are irrelevant, complementary, or conflicting, and they rarely treat proxy-specific constraints such as privacy protection and unauthorized decision-making as central evaluation criteria.

2.2. Policy-Constrained Agentic Dialogue Evaluation

Recent LLM-agent benchmarks extend dialogue evaluation beyond text generation to tool use, policy compliance, and multi-turn decision-making [15,16]. For example, $\tau$-bench [8] and ${\tau }^2$-bench [17] evaluate agents in customer-service environments with policy documents and controlled interactions. ABCD [18] aligns customer requests with institutional policies and action sequences, while ToolTalk [19] studies safety issues in tool-use dialogues, especially under irreversible actions. Related work such as PAChat [20] further explores multi-speaker and personalized communication settings.

These benchmarks cover important aspects of policy following, action correctness, and communication structure, but they still differ from the proxy setting of phone call assistants. Their constraints are usually defined by institutions or tools, while the interaction remains largely between a user and a service system. In contrast, a phone call assistant mediates between two independent natural persons: the device owner and the caller. It must decide when to advance, delay, or suspend the owner’s preset while respecting the caller’s current need and strict safety boundaries. CallBench targets this gap by evaluating dual-goal coordination under phone-call-specific constraints, including goal conflict handling, preset timing, privacy protection, and decision-boundary compliance.

3. Problem Definition

Current real-time call assistant systems are commonly built as a pipeline consisting of ASR, text processing, and TTS. This paper focuses on the text processing stage. We assume that the system has already obtained the textual form of the caller’s utterance and needs to generate the next response of the call assistant. Speech recognition, speech synthesis, endpoint detection, and other speech-side issues are beyond the scope of this work. Under this setting, we define the task of Dual-Goal Coordination for real-time call assistants.

Given the dialogue history at turn t:

$$H_t=\{(u_1,a_1),(u_2,a_2),\dots ,(u_{t-1},a_{t-1})\},$$

where $u_i$ denotes the caller’s utterance at turn i, and $a_i$ denotes the assistant’s response at turn i. The current input includes the caller’s latest utterance $u_t$, a scene label $s\in \mathcal{S}$, and the owner’s preset pool ${\mathcal{P}}^o$, where:

$$\mathcal{S}=\{\mathrm{takeout},\mathrm{delivery},\mathrm{taxi},\mathrm{work},\mathrm{life},\mathrm{harassment}\}.$$

The owner’s preset pool ${\mathcal{P}}^o$ contains goal-related information predefined by the device owner. To align with practical phone call assistant systems under deployment, we model ${\mathcal{P}}^o$ as a structured preset interface rather than an arbitrary set of instructions. Each preset can be represented as:

$$p_i^o=(q_i^o,r_i^o),$$

where $q_i^o$ denotes the trigger expression or trigger condition, and $r_i^o$ denotes the corresponding owner-specified response. Each preset slot may also be left empty.

The preset pool contains two types of slots: regular preset slots and emergent preset slots. Regular preset slots correspond to default owner instructions for high-frequency service calls. In our setting, regular preset slots are only defined for takeout, delivery, and taxi. Each of these scenes has at most one regular preset slot. For a regular preset, the trigger expression $q_i^o$ is predefined by the system, and the owner only specifies the response $r_i^o$.

A regular preset is available whenever the current service scene has a non-empty regular slot. If the caller’s utterance matches the default trigger query, the assistant may directly convey the owner-specified response. If the utterance does not yet match the default trigger query, the assistant should not treat the preset as irrelevant. Instead, it may ask for missing information, confirm the caller’s situation, or guide the conversation toward a state where the preset can be naturally conveyed. However, if the caller raises an exception or a conflicting request, the assistant should delay or suspend the preset rather than force it.

Emergent preset slots correspond to user-defined conditional instructions. For an emergent preset, both the trigger expression $q_i^o$ and the response $r_i^o$ are specified by the owner. Unlike regular presets, emergent presets may be activated in any of the six scenarios, but only when the current caller utterance matches the user-defined trigger expression.

At turn t, the assistant retrieves the owner-side explicit goal available in the current context:

$$g_t^o=\mathrm{R}\mathrm{E}\mathrm{T}\mathrm{R}\mathrm{I}\mathrm{E}\mathrm{V}\mathrm{E}\mathrm{P}\mathrm{R}\mathrm{E}\mathrm{S}\mathrm{E}\mathrm{T}(H_t,u_t,s,{\mathcal{P}}^o).$$

The retrieval process checks the preset pool according to the dialogue history, caller utterance, and current scene. For takeout, delivery, and taxi, the assistant first checks the regular preset slot associated with the current scene. If the slot is empty, the assistant then checks whether any emergent preset is triggered by the current utterance. For work, life, and harassment, the assistant only checks whether the current caller utterance matches any user-defined emergent trigger condition. If no preset is available, then $g_t^o=⌀$.

This setting gives rise to three preset cases. A regular preset case occurs when the current service scene has a non-empty regular preset slot. An emergent preset case occurs when a user-defined trigger condition is matched by the caller’s utterance. A no-preset case occurs when no owner-side preset is available or activated in the current turn. Since every regular slot and emergent slot can be empty, no-preset cases may appear in any scenario.

In contrast to the owner-side explicit goal, the caller’s goal at the current turn is denoted as $g_t^c$. This goal is usually not explicitly provided before the call, but needs to be understood from the dialogue history, current utterance, and scene information. The caller-side implicit goal may be aligned with, complementary to, irrelevant to, or conflicting with the current owner-side goal.

This setting gives rise to a turn-level dual-goal coordination problem in real-time call assistants. At each turn, the system needs to determine the relationship between the current owner-side explicit goal $g_t^o$ and the caller-side implicit goal $g_t^c$, and generate the assistant response:

$$a_t=f(H_t,u_t,s,g_t^o,g_t^c;\mathcal{C}),$$

where $\mathcal{C}=\{C_{\mathrm{priv}},C_{\mathrm{dec}},C_{\mathrm{fact}},C_{\mathrm{phys}}\}$ denotes the set of safety boundary constraints that the call assistant must follow. Specifically, $C_{\mathrm{priv}}$ denotes the privacy protection constraint, requiring the system not to disclose the owner’s personal information, location, contact information, or other sensitive content; $C_{\mathrm{dec}}$ denotes the decision-boundary constraint, requiring the system not to make commitments, agreements, refusals, or choices on behalf of the owner; $C_{\mathrm{fact}}$ denotes the factuality constraint, requiring the system not to fabricate information that is not provided in the dialogue context or the owner’s presets; and $C_{\mathrm{phys}}$ denotes the non-embodiment constraint, requiring the system not to present itself as an embodied agent with direct sensory access or physical agency in the real world, such as claiming that it can see an object, check a location, or physically handle an item. The response $a_t$ should advance or complete the owner-side goal at an appropriate time, while also properly addressing the caller’s current request. When the two goals conflict, when the current information is insufficient, or when the conversation is ready to end, the system should adjust its response according to the current context rather than mechanically pushing the original owner-side goal.

Under this definition, a call assistant should be evaluated not only by the naturalness and fluency of its responses, but also by its ability to correctly understand the caller’s current request, activate and advance the owner’s preset goal at the right time, handle deviation or conflict between the two goals, and consistently obey the safety boundaries.

4. Benchmark Construction

Real phone calls often contain private information and are difficult to release as public evaluation data. To enable controlled and reproducible evaluation, we construct CallBench, a Chinese benchmark for Dual-Goal Coordination in phone call assistants. In total, CallBench contains 50,000 multi-turn Chinese phone call dialogues across six incoming-call scenarios, covering different preset cases, caller-side goals, owner-side goals, and dual-goal relations.

4.1. Scenario and Preset Configuration

CallBench covers the six incoming-call scenarios defined above, spanning short service calls and more open-ended interpersonal calls. In takeout, delivery, and taxi, calls are usually short and revolve around a concrete service task, such as delivering an order, placing a package, or coordinating a ride. In work, life, and harassment, caller intents are more diverse, requiring the assistant to retain useful information, respond cautiously, or terminate the call safely.

For each scenario, we define possible caller roles, call purposes, and conversation states. In takeout, delivery, and taxi, callers may report normal arrival, address ambiguity, item damage, failed delivery, or ask for confirmation. In work and life, callers may ask for the owner, leave a message, provide new information, or request a decision. In harassment, callers may engage in telemarketing, repeatedly ask irrelevant questions, try to obtain private information, or pressure the assistant into making unauthorized commitments.

Following the preset cases defined in the problem definition, CallBench includes regular presets, emergent presets, and no-preset instances. Regular presets are included only in takeout, delivery, and taxi, where stable owner instructions commonly exist. Emergent presets and no-preset instances may appear in any of the six scenarios. This design allows the benchmark to cover both scene-level default instructions and turn-level conditional owner goals, while also testing cases where no owner-side goal is available.

4.2. Dual-Goal Relation Design

A central property of the benchmark is that each dialogue instance contains both an owner-side explicit goal and a caller-side implicit goal. The owner-side goal is represented by the preset pool, while the caller-side goal is reflected through the caller’s role, motivation, and utterances. We explicitly construct different relations between the two goals, including alignment, complementarity, irrelevance, and conflict.

In aligned cases, the caller-side goal already supports advancing the owner-side goal, such as a delivery person reporting arrival under a leave-at-door instruction. In complementary cases, the caller-side goal does not directly trigger the preset, but can help establish the missing condition for advancing it. The assistant should respond to the caller-side situation and guide the dialogue toward a state where the owner preset can be naturally conveyed. In irrelevant cases, the caller-side goal is unrelated to the owner preset, so forcing the preset into the turn would be unnatural or premature. In conflict cases, the caller-side goal blocks or contradicts advancement of the owner-side goal, such as reporting a spilled order, a damaged package, or an inability to complete the service as originally planned.

This design allows the benchmark to evaluate not only whether the owner-side goal can be completed, but also whether it is advanced timely. In many cases, a desirable response should delay, suspend, or ignore the owner preset when context makes direct goal execution inappropriate.

4.3. Dialogue Construction

We construct complete multi-turn phone call dialogues through a structured role-play generation process. For each dialogue, we first prepare a scenario description that specifies the scene, caller role, caller-side goal, preset case, owner target response, trigger query if available, and possible abnormal conditions. The caller-side goal describes why the caller makes the call and what the caller wants to achieve in the conversation. For example, in the takeout scene, the caller-side goal may be to report that the food was spilled and ask how to handle the order.

We use Doubao-Seed-1.6 [21] to implement two role-playing agents, one acting as the caller and the other acting as the phone call assistant. The caller agent is given the caller role, caller-side goal, and abnormal conditions, and is instructed to behave like a real caller in a phone conversation. The assistant agent is given the dialogue history, scene label, owner preset information, trigger condition, and safety constraints, and is asked to respond as a phone call assistant answering on behalf of the owner.

Since the caller-side goal and the owner-side preset are designed from two different perspectives, they are not always naturally aligned. Within each scene, we combine different caller-side goals with different owner presets, so that the generated calls can cover diverse dual-goal relations.

Under each configuration, the two role-playing agents generate multiple complete calls with the temperature set to $1.0$. In this way, the goal relation is reflected through the actual dialogue flow rather than only assigned as a static label. During generation, we encourage short spoken utterances and avoid written expressions, so that the dialogues better resemble real phone calls. After a dialogue is generated, we build the owner preset pool for this dialogue according to its preset case. In addition, we also randomly insert silence turns where the caller utterance is marked as "caller no response" to reflect irregularities in real phone calls.

CallBench stores each generated sample as a complete multi-turn dialogue. During evaluation, assistant responses are assessed turn by turn within the full dialogue context. For each assistant response, the evaluator considers the preceding dialogue history, the current caller utterance, the scene label, and the owner preset pool. This setting preserves the temporal context of the phone call while allowing fine-grained assessment of goal timing, caller-side issue handling, and natural conversation closing.

4.4. Quality Control

We apply quality control throughout the dialogue construction process. The automatic checking modules are implemented with Doubao-Seed-2.0-lite [22]. At each assistant turn, the generation pipeline tracks whether the caller utterance matches the trigger query, whether the owner-side target has already been conveyed, and whether advancing the preset would conflict with the caller’s current request or the scene condition. These checks help avoid premature preset delivery, repeated conveyance, and inappropriate goal advancement in conflict cases.

Candidate assistant responses are validated before being added to the dialogue history. A response is rejected if it fails to follow the latest caller utterance, ignores the previous context, repeats information unnecessarily, or sounds unlike a phone conversation. We also filter responses that violate the safety boundaries of phone call assistants, including unauthorized decisions on behalf of the owner, fabricated information, privacy leakage, and claims of direct access to the physical world. For example, the assistant should not claim that it can see a parcel, inspect food damage, identify a vehicle, or handle an item in the real world.

If a generated assistant response fails these checks, it is regenerated. Dialogues with invalid role behavior, inconsistent goal relations, severe safety violations, or unnatural conversation flow are removed. In addition, we manually inspect 10% of the generated dialogues at the dialogue level using the same criteria. A dialogue receives a score of 1 only when no issue is found, 0.5 when it contains minor imperfections that do not affect the main goal or safety, and 0 when it contains severe errors or violates core safety rules. The inspected subset achieves an average score of 0.93.

4.5. Data Statistics

CallBench contains 50,000 complete multi-turn Chinese phone call dialogues across six high-level scenarios and we split it into training, validation, and test sets with a ratio of 8:1:1. Each dialogue is associated with a scene label, caller-side goal, preset case and the owner preset pool.

The dataset is approximately balanced across scenarios. For takeout, delivery, and taxi, regular-preset, emergent-preset, and no-preset instances are distributed with an approximate ratio of 3:1:1. For work, life, and harassment, where no regular preset slots are defined, the data is approximately balanced between emergent-preset and no-preset instances. Detailed information is listed in Table 1.

5. Evaluation Protocol

5.1. Turn-Level Multi-Dimensional Evaluation

CallBench evaluates phone call assistants at the turn level. For each assistant turn, the evaluator receives the dialogue history $H_t$, current caller utterance $u_t$, scene label s, preset metadata, and the assistant response. The goal is to judge whether the response appropriately handles the caller-side need, the owner-side preset, and the safety boundaries of phone call assistants. Since preset timing is context-dependent and may allow multiple acceptable responses, we do not define a single gold action for each turn. For example, when the caller reports an exception, the assistant may either ask for more details or state that the issue will be relayed to the owner, as long as it does not force the preset prematurely or make unauthorized decisions. We therefore adopt a rubric-based automatic judge for scalable evaluation. We use Qwen3.7-Max [23] as the judge, scoring each applicable dimension as 0, $0.5$, or 1, indicating severe issue, minor issue and no issue. The rubric covers semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety.

5.2. Evaluation Dimensions

The evaluation covers both general response quality and dual-goal coordination ability.

Semantic Understanding measures whether the assistant correctly understands the current caller utterance and provides a logically relevant response. A response receives a low score when it answers the wrong question, ignores the caller’s request, or misunderstands the current situation.

Context Use measures whether the assistant correctly uses the dialogue history. This dimension is important in multi-turn calls, where the assistant must avoid asking repeated questions, contradicting previous information, or forgetting that the owner-side goal has already been conveyed.

Active Guidance measures whether the assistant guides the conversation at the right time while coordinating the owner-side preset with the caller-side goal. In regular-preset cases, the assistant should guide the caller toward a state where the preset can be naturally conveyed, but should not ignore caller-side exceptions or constraints. After the preset is conveyed, it should stop unnecessary guidance and move toward confirmation or closing. In emergent-preset cases, the assistant should not induce the caller to say the trigger condition, but should respond naturally to the current caller-side goal. For no-preset cases, this dimension is not applied.

Response Quality evaluates the linguistic and conversational quality of the response. The response should be natural, concise, colloquial, and suitable for a phone call. It should not be overly written, unnecessarily verbose, repetitive, mechanically templated, or grammatically awkward.

Preset Compliance measures whether the assistant follows the owner-side preset at the appropriate time. For regular presets, the assistant should convey the owner target response when the dialogue context is ready, but should not force the preset when the caller reports an exception or conflict. For emergent presets, the assistant should convey the target response only when the current caller utterance matches the trigger query. If the trigger is not matched, premature delivery of the preset is penalized. For no-preset cases, this dimension is not applied.

Dialogue Rhythm evaluates whether the assistant manages the progression and closing of the call appropriately, with assessment criteria varying across distinct scenarios. The assistant should respond to the current caller utterance in a meaningful way, close the call only after the key information has been exchanged and the caller has no further requirement, and avoid unnecessarily prolonging the conversation after the caller-side issue has been handled and the owner-side goal has been completed. For silence turns, the assistant should handle the lack of caller input reasonably, such as checking whether the caller is still available.

Safety evaluates whether the assistant obeys the safety constraint set $\mathcal{C}$ defined in the problem definition. This includes avoiding privacy leakage, unauthorized decision-making, unsupported factual claims, and false claims of real-world access. Unlike other dimensions, safety is scored only as 0 or 1. Any violation of $\mathcal{C}$ results in a safety score of 0.

5.3. Score Aggregation

For each turn, the judge outputs both an overall score and scores for the applicable dimensions. The overall score is assigned on a three-level scale: 1 indicates that the response has no major issue, $0.5$ indicates that it contains minor or partial issues, and 0 indicates a severe error or a violation of core requirements. Dimensions that are not applicable to the current preset case are excluded. We report both the overall score and the scores of individual dimensions.

6. Experiments

6.1. Experimental Settings

We evaluate different task-oriented dialogue [24] methods on the test split of CallBench, including Direct Prompting (DP), ReAct [25], SimpleTOD [26], DivTOD [27], AutoTOD [28], ProCoT [29], EnPL [30], and ChatSOP [31]. We do not train or fine-tune any baseline on CallBench; all methods are evaluated in an inference-only setting. For methods that originally involve training, fine-tuning, or learned modules, we replace training-dependent components with prompt-based adaptation using the same observable input. This keeps the comparison focused on how different dialogue-control paradigms coordinate the two goals, rather than on additional supervised training. Each system receives the same input, including dialogue history, current caller utterance, scene label, safety constraints, and the owner preset associated with the current instance, which isolates response-level dual-goal coordination from upstream preset selection. Unless otherwise specified, all experiments use Doubao-Seed-1.6 with temperature 0 as the backbone model. We additionally conduct experiments with Qwen3.5-27B [32] to examine backbone robustness, with results reported in Appendix A.

6.2. Main Results

Table 2 reports the main results on CallBench. The results show that existing task- and target-oriented dialogue methods do not transfer directly to phone call assistants. Although these methods are designed for structured task completion or goal progression, they generally underperform ReAct on the overall score, and some even fall behind the simple direct-prompting baseline. This suggests that current dialogue methods still lack a comprehensive solution for phone call assistants: they may perform well on certain aspects, but struggle to balance goal coordination, response quality, dialogue rhythm, and safety simultaneously. These results demonstrate the value of CallBench, whose multi-dimensional evaluation reveals weaknesses that would be overlooked by overall scores alone.

6.3. Results for Regular Preset

Table 3 presents the results on regular-preset cases across the three service scenarios. These cases most directly test dual-goal coordination, since the owner-side preset remains available throughout the call while the caller-side goal may evolve, remain incomplete, or conflict with it. We report both LLM-judge scores on the full turn-level subset and human ratings on sampled turn-level instances. For human evaluation, we sample 250 turn-level instances from each scenario and ask annotators to follow the same scoring rubric as the LLM judge, assigning 1, $0.5$, or 0 according to whether the response has no issue, minor issues, or severe errors. ReAct achieves the strongest average performance under both evaluation settings, while methods with higher LLM-judge scores generally also receive higher human ratings. To examine the relation between automatic and human evaluation, we compute rank correlation over the 24 method-scenario pairs in Table 3. The automatic and human scores show a positive association, with Kendall’s ${\tau }_b=0.764$. This result is used as a sanity check rather than a direct inter-annotator agreement measure, since the scores are aggregated and the human ratings are collected on sampled turn-level instances. At the same time, the absolute scores remain far from saturated, indicating that existing methods still struggle with coordination between two independent goals.

6.4. Error Analysis

Figure 2 reports method-wise error rates for the six most frequent diagnostic error types, selected by their method-averaged rates across all evaluated methods. The most common error is safety issue, with an average rate of 10.63%, suggesting that existing assistants still struggle to maintain proxy boundaries when responding on behalf of the owner. In addition, dialogue-level errors are also frequent: repetitive reply and improper closing reach 6.68% and 6.57% on average, respectively, indicating that many assistants still have difficulty controlling dialogue rhythm in phone calls, either repeating similar expressions or ending the call at an inappropriate time. Meanwhile, preset-related errors remain prominent. Over-following the preset reaches 6.18% on average, while repeated preset delivery reaches 5.23%. These errors capture different failures in owner-side goal handling: applying the preset despite an unsuitable caller-side situation, or unnecessarily repeating it after delivery. Irrelevant answers account for 6.20% on average, showing that assistants may produce locally plausible but contextually inappropriate responses when caller-side requests deviate from the preset flow. These results suggest that existing methods fail less because of surface-level fluency and more because of turn-level decision-making under dual-goal conditions, especially safety boundaries, dialogue rhythm, and preset timing under noisy, evolving caller-side contexts.

7. Conclusions

We present CallBench, a Chinese benchmark for dual-goal coordination in phone call assistants. The benchmark targets a realistic proxy setting where the assistant must answer calls for the device owner, coordinate the owner’s explicit target with the caller’s implicit goal, and obey strict safety boundaries. We formalize structured owner preset pools with regular, emergent, and no-preset cases, and construct 50,000 multi-turn dialogues across six incoming-call scenarios. We propose a turn-level evaluation protocol measuring semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety. Our evaluation shows existing task-oriented dialogue methods still struggle, suggesting that effective phone call assistants must reason about two independent goals under strict proxy constraints and make reliable decisions throughout the call. We hope CallBench will support future research on reliable call assistants in realistic settings.