Configuring and Diagnosing Conversational Agents for Process Mining Tasks

1. Introduction

Process mining connects event data with questions about control flow, conformance, performance, resources, fairness, and redesign [1,2]. LLMs are attractive interfaces for such questions because they can explain logs, models, queries, and improvement options in natural language [3,4,5]. Yet process mining answers can be plausible while being wrong: they may assume the wrong case notion, ignore timestamp ordering, misuse a formal model, or overstate a fairness or optimization claim. A natural reaction is to ask whether multiple specialized agents can do better than a single LLM call.

In this paper, a conversational agent is not a tool-using autonomous system. The same underlying LLM is reused under different role prompts—an “event log interpreter”, a “conformance checker”, an “optimization consultant”, and so on—and a selector decides which persona contributes next to a shared transcript before a final synthesis prompt produces the answer. Agents do not use tools like Python or SQL; they differ only in the role description that frames their reading of the conversation. As shown in Figure 1, this turns one LLM call into a structured deliberation among personas, but it also adds cost, noise, and weak handoffs. The design problem is therefore to configure roles and routing, and, just as importantly, to know whether any given configuration is actually helping.

We argue that the conversation produced by such a system is itself event data: ordered handoffs between roles, with intermediate contributions that can be scored and aggregated. This makes it natural to analyze agent runs the way process mining analyzes organizational behavior as a directed social network of actors and handoff relations [6,7]. Doing so lets us ask not only whether a configuration produces good final answers, but which roles are useful, which handoffs are weak, and whether the same configuration behaves the same way under different base models. The diagnostic information that emerges can then be used to revise routing constraints, rather than redesigning agents by intuition.

This paper makes three contributions. First, it defines process-mining-focused conversational configurations expressed as portable role sets and routing constraints, so that the same persona definitions can be instantiated in different conversational-agent frameworks, including Microsoft AutoGen and X.AI Grok 4.20, without rewriting the agents. Second, it proposes a diagnostic methodology based on Social Network Analysis of recorded agent traces, combining final-answer quality, average role performance, and average handoff performance into a single view of how a configuration behaves. Third, it provides an open source conversational agent implementation that runs these configurations, records their traces, computes the SNA diagnostics, and uses them to constrain weak handoffs in revised configurations.

The empirical findings should be read with these contributions in mind. On 57 process mining questions from PM-LLM-Benchmark [5], agentic configurations do not reliably outperform direct answering; the strongest result we observe is that an SNA-guided pruning of weak handoffs improves a verification-heavy configuration, although equal wins and losses against the no-agent baseline mean this is suggestive rather than conclusive. We also observe that the same role set behaves quite differently under different base models, with sparse early stopping under one model and denser deliberation under others. Together, these results support a diagnostic rather than a performance-driven reading of multi-agent process mining: configurations should be evaluated through their traces, not only their final answers.

The rest of the paper is organized as follows. Section 2 reviews LLMs for process mining, conversational agent frameworks, LLM-as-judge evaluation, and SNA. Section 3 describes the configurations, trace analysis, benchmark, and evaluation procedure. Section 4 reports results in terms of the three contributions. Section 5 presents the implementation, and Section 6 discusses implications and limitations.

2. Related Work

LLMs have recently been studied as process mining assistants. Existing work translates process mining artifacts into text, evaluates direct answering and query generation, and discusses benchmarks and evaluation strategies [3,4]. PM-LLM-Benchmark is closest to our setting because it provides process mining questions and LLM-based grading, but it does not evaluate conversational agent configurations or SNA-based trace diagnosis [5].

Conversational agent frameworks are now used in both research and industry. AutoGen, developed at Microsoft, supports configurable conversable agents and group-chat orchestration [8]; CAMEL and MetaGPT show how role assignment can structure complex LLM tasks [9,10]. Industry-facing systems also expose agent composition and execution traces. Figure 2 illustrates a Grok 4.20 instantiation of the balanced process mining configuration. Our focus is narrower than general tool-using agents: we study role configurations, routing, and trace-based diagnosis for process mining question answering.

LLM-as-judge methods enable scalable grading of open-ended answers, while also introducing biases such as verbosity and position effects [11,12]. We use judge scores as proxy evidence. SNA provides the diagnostic lens: it studies actors and relations [13], and process mining has used event logs to discover handover-of-work relations [6,7]. We transfer this idea from human performers to LLM agent roles.

3. Process Mining Configurations and Evaluation Methodology

This section presents the conversational framework, the three role configurations under study, the SNA measures applied to recorded traces, and the benchmark used to evaluate them.

3.1. Conversational Framework and Configurations

The evaluated system is a centralized, selector-mediated conversational framework. A configuration defines the available agent personas, their role descriptions, the maximum number of iterations, whether only one agent may speak per iteration, and optional forbidden handoffs. Agents do not call external tools and do not communicate directly. Instead, each selected agent reads the original process mining task and the accumulated transcript, then appends one intermediate contribution.

At each iteration, the selector receives the task, available agents, previous answers, the iteration number, and routing constraints. It returns either an agent name or FINAL. If an agent is selected, its answer is added to the transcript; if FINAL is selected, or the iteration budget is reached, a final synthesis prompt produces the benchmark answer. The no-agent baseline skips this loop and answers the original prompt directly.

We study three complete process-mining-oriented configurations. The artifact-pipeline configuration follows a parse-and-audit structure, with roles for artifact mapping, trace reasoning, control-flow semantics, performance analysis, fairness/compliance review, and answer precision. The balanced configuration covers common process mining task families: event log interpretation, process modeling, conformance checking, querying, hypothesis generation, fairness review, and optimization. The verification-heavy configuration emphasizes robustness through parsing, domain reasoning, formal checking, counterexample search, data auditing, bias auditing, and benchmark-style answer judging.

The configurations are expressed as portable role templates and routing constraints rather than as framework-specific code. Hence, the same role set can be instantiated in different conversational-agent frameworks, such as AutoGen-style group chats or industrial interfaces such as Grok. In this paper, the configuration file is the main experimental object: changing roles or handoff constraints changes the effective conversational system. Table 1 details the roles.

3.2. Trace Representation and SNA Measures

Each benchmark execution produces a trace that is interpreted as a directed social network. The actors are conversational roles. For each question, the judged sequence contains START, zero or more selected agent nodes, and COMPLETE. Consecutive nodes define directed handoffs. For example,

$$\mathtt{START}\to \mathtt{artifact}\_\mathtt{parser}\to \mathtt{domain}\_\mathtt{reasoner}\to \mathtt{COMPLETE}$$

records that the selector first chose the parser, then the domain reasoner, and then stopped.

Each judged node v has a role label and a score $s\left(v\right)$ on the 1 to 10 rubric. The COMPLETE score measures final-answer quality. Intermediate node scores measure the judged usefulness of selected roles. For role a, with occurrences $V_a$, the mean role score is

$${\overline{s}}_a={\displaystyle \frac{1}{|V_a|}}\sum _{v\in V_a}s\left(v\right).$$

(1)

Edges are scored by the target node after a handoff. For all observed transitions from role a to role b, denoted $E_{a,b}$, the mean handoff score is

$${\overline{s}}_{a,b}={\displaystyle \frac{1}{|E_{a,b}|}}\sum _{(v_i,v_{i+1})\in E_{a,b}}s\left(v_{i+1}\right).$$

(2)

Thus, an edge score answers: when the conversation moves from a to b, how good is the next contribution on average? These scores are diagnostic associations, not causal estimates, because task type, transcript context, and selector behavior also influence the result.

The analysis therefore reports three complementary quantities: final output quality from COMPLETE, role-level performance from node means, and handoff-level performance from edge means. We also inspect counts, standard deviations, number of observed edges, graph density, and category-level role usage. Low-scoring repeated handoffs are candidates for routing constraints. This is how the pruned verification-heavy configuration is obtained: it keeps the same roles but excludes four weak transitions identified in the initial SNA.

3.3. PM-LLM-Benchmark and Judge

The evaluation uses 57 questions from PM-LLM-Benchmark [5], a benchmark for assessing LLM answers to process mining tasks expressed through textual logs, models, constraints, queries, and analysis goals. The questions are grouped into eight categories: i event log interpretation, covering activities, cases, timestamps, resources, and data quality; ii conformance and anomaly analysis, covering deviations from rules or models; iii process model generation, covering formal or semi-formal behavior descriptions; iv querying and model description, covering SQL-style logic and model explanations; v hypothesis generation, covering diagnostic hypotheses and evidence plans; vi fairness, covering group comparisons and responsible interventions; vii object-centric and advanced representations, covering richer process views; and viii optimization, covering bottlenecks, capacity, rework, and redesign.

All final answers and intermediate agent nodes are scored by openai/gpt-5.4 on a 1.0 to 10.0 process mining rubric. We report means, population standard deviations, call counts, graph summaries, and paired differences over the 57 common questions. Paired p values use a simple two-sided normal approximation and are descriptive. The complete evaluation and diagnosis workflow is summarized in Figure 3.

3.4. Configuration Rationale and Research Questions

The three configurations in Table 1 are intended to explore different sources of potential benefit: stable parse-and-audit behavior, broad process mining specialization, and verification-oriented review. The pruned variant is intended to explore whether SNA-based diagnostics can inform a configuration’s routing without changing its roles.

In this spirit, we organize the empirical study around four exploratory research questions. RQ1 examines whether the proposed configurations tend to behave differently from direct answering in terms of final-answer quality. RQ2 examines which roles and handoffs appear to be associated with higher or lower judged quality in the recorded traces. RQ3 examines whether excluding handoffs flagged as low-performing by the SNA diagnostics is associated with changes in final-answer quality for a given configuration. RQ4 examines the extent to which a fixed configuration yields comparable social networks across different base models. The questions are framed as diagnostic rather than confirmatory: with 57 benchmark questions and judge-based scoring, the evidence we report is best read as descriptive support for or against a hypothesis, not as a definitive test.

All experiments are conducted via OpenRouter using openai/gpt-5.4-mini, x-ai/grok-4.20, and qwen/qwen3.6-35b-a3b with reasoning disabled. The empirical tables (Table 2, Table 3 and Table 4) report all three base models side by side, so the same observations can be cross-checked across them. For readability, our discussion of RQ1 through RQ3 concentrates on the openai/gpt-5.4-mini columns, and the cross-model comparison for RQ4 concentrates on the balanced configuration; the corresponding figures focus on these slices accordingly. We made this choice for narrative clarity rather than because the other combinations behave very differently, and the qualitative picture across the remaining configuration and model combinations is broadly similar. The complete set of runs, judged traces, and SNA summaries is available in the project repository (see Section 5).

4. Results

We report the results in three parts, addressing the four research questions in turn.

4.1. Effectiveness of Process Mining Configurations

This subsection addresses RQ1 and the first contribution. The overall picture from Table 2 is that, on these 57 questions, none of the conversational configurations clearly dominates direct answering. The no-agent baseline turns out to be a competitive reference point. On openai/gpt-5.4-mini, the balanced and artifact-pipeline configurations sit close to direct answering on average, the unpruned verification-heavy configuration sits slightly below it, and the pruned verification-heavy configuration is the highest scoring of the five with somewhat lower variance. On x-ai/grok-4.20 and qwen/qwen3.6-35b-a3b, direct answering is the strongest reference, and the pruned verification-heavy configuration tracks closely behind it on x-ai/grok-4.20. We therefore read the results as suggestive rather than conclusive: adding role-specialized personas does not, by itself, reliably translate into better final answers, but it does not appear to be uniformly harmful either when routing is reasonable. Looking at the per-category columns, the configurations differ more from each other on the harder categories (notably the formal modeling category c03 and the object-centric category c07) than on the easier ones, which is consistent with the idea that conversational deliberation matters most when the underlying task is itself harder for a single call.

4.2. SNA-Based Diagnosis of Roles and Handoffs

This subsection addresses RQ2 and RQ3 and illustrates the second contribution. The SNA diagnostics provide two complementary views of a configuration: a node-level view, which asks how individual roles score when they are selected, and an edge-level view, which asks how the next contribution scores after a particular handoff. Both views are informative, and they tend to highlight different kinds of issues.

At the node level (Table 3), reviewer-oriented roles tend to receive comparatively high intermediate scores when they are selected. On openai/gpt-5.4-mini, this shows up in the counterexample hunter, the bias and impact auditor, and the data consistency auditor within the verification-heavy configuration, and in the hypothesis generator and query analyst within the balanced configuration. Broader generative roles such as the process modeler are also selected often, but their scores are more variable. This view is useful on its own: it helps locate where a configuration is reliably contributing and where its variance comes from, independently of the final answer. The category usage in Table 4 adds a useful overlay, showing that certain roles are activated mainly for the categories they target (for example, the fairness reviewer concentrates on c06), while others such as the artifact parser and the answer precision auditor appear broadly across the benchmark.

The edge-level view turns out to be more actionable. In the unpruned verification-heavy configuration, a small set of handoffs recurs across questions and consistently produces lower-scoring next contributions, even when the roles on either end are individually reasonable. Treating those transitions as forbidden, while keeping the same role set, produces the pruned configuration; the resulting handoff graph (Figure 4) is smaller and more focused, and the associated final-answer mean reported in Table 2 is modestly higher with somewhat lower variance on openai/gpt-5.4-mini. The category-level usage of the pruned configuration in Table 4 indicates that the routing changes preserve the original coverage rather than concentrating effort on a smaller part of the benchmark.

Reading the two views together brings out a recurring pattern: a strong intermediate contribution does not automatically translate into a strong final answer, because the synthesis step still has to integrate that contribution and the conversation has to reach synthesis in a useful state. The edge-level diagnostic is helpful precisely here. It points to transitions that tend to leave the conversation in a worse state, rather than to roles that need to be redesigned. We report the pruning result as an illustration of this diagnostic workflow rather than as a causal claim, since the weak handoffs were identified and re-evaluated on the same benchmark.

4.3. Configuration Portability Across Base Models

This subsection addresses RQ4. Comparing runs of the same balanced configuration across the three OpenRouter base models (Figure 5, with supporting counts and role scores in Table 3 and Table 4) suggests that a configuration file is only a partial description of the resulting conversational system. In Figure 5, node and edge colors encode the average judge score (green for higher means, red for lower means), so the visual texture of each network reflects not just its routing pattern but also the quality of the contributions it elicits. With openai/gpt-5.4-mini the selector tends to stop early and yields a sparse network in which most balanced roles are activated only a handful of times across the 57 questions. With x-ai/grok-4.20 and qwen/qwen3.6-35b-a3b, the same roles and the same constraints lead to denser networks with many more role activations and a richer mix of higher and lower scoring transitions. Final-answer quality also varies across models, as can be read off Table 2, but the more interesting observation for our purposes is structural: the social network induced by a configuration is shaped jointly by the role set and by how a particular base model plays the selector role. This argues for treating configurations as portable templates whose behavior should be re-diagnosed when the underlying model changes, rather than assumed, and it is one reason we report SNA summaries alongside final-answer scores throughout the paper.

5. Implementation and Availability

The framework is publicly available at https://github.com/fit-alessandro-berti/agents-sna under GPL-3.0. It consists of a Python package and a single-file browser tool that share the same configuration format, so a configuration designed in one can be loaded in the other without modification.

• Repository structure.

The repository follows the four steps of Figure 3. The package src/agents_sna contains the orchestrator with the selector, agent, and final-synthesis prompts; a CLI entry point agents-sna; a benchmark_runner that executes a configuration over a local PM-LLM-Benchmark checkout; an agent_judge that scores saved benchmark artifacts with an LLM-as-a-judge; and an evaluation_network module that aggregates judged traces into the SNA statistics used in this paper. Configurations live in configs/, benchmark outputs in benchmark_runs/, judged node sequences in agent_evaluations/, and unit tests in tests/. A shared OpenRouter client lets the base model and any extra payload (e.g., temperature, reasoning) be set from a single flag.

• Configuration format.

A configuration is a self-contained JSON object with four fields. The agents list holds role personas, each given by a name and a description that becomes the system prompt for that role; an empty list reduces the system to direct answering. The integer max_iterations bounds deliberation including the final synthesis call, and the boolean single_agent_per_iteration switches between one-agent-per-turn and multi-agent regimes. The list excluded_handoffs contains pairs {from, to} that the selector is forbidden to take, and is the field used to express the SNA-guided routing constraints discussed in Section 4.2.

• Pipeline modules.

Given a configuration and a prompt, the orchestrator runs the selector-mediated loop of Section 3.1: each iteration sends the selector the original prompt, the previous agent answers, and the allowed next agents, and the loop ends when the selector returns FINAL or the iteration budget is exhausted. Full request bundles and a compact conversation trace can be exported to JSON. The benchmark runner replays this orchestration over every PM-LLM-Benchmark question, supports concurrent execution and retries on transient OpenRouter failures, and skips already answered questions. The agent judge reads the saved artefacts of a run and produces, for each question, a list of judged nodes with a 1 to 10 score and an explanation; the first node is always START and the last COMPLETE, which gives the final-answer score. The evaluation network module turns a folder of judged traces into the directed social network used in this paper, computing node and edge counts, mean scores, standard deviations, a category-by-role usage table in LaTeX, and a Graphviz drawing in which colour encodes mean scores and edge thickness encodes handoff frequency.

• Browser tool.

Figure 6 shows agent_chatbot.html, a self-contained browser tool that exercises the same configuration format interactively. The left panel holds the OpenRouter credentials, the base model, and a JSON editor with a validate button and built-in presets corresponding to the configurations evaluated in this paper, including the pruned verification-heavy variant. The centre panel is a chat composer that shows the prompt, an expandable run details block listing each selector decision and agent contribution, and the final answer. The right panel maintains a live handoff network across the session, with counters for prompts, roles, and edges, a Render graph button that draws the network in-browser through a Graphviz WebAssembly build, and a read-only DOT-source view that can be copied for offline use. The tool therefore acts both as a usability front end for the configurations and as an inspector for the same SNA diagnostics computed in batch by the Python pipeline.

6. Discussion and Conclusion

We studied selector-mediated conversational agent configurations for process mining and proposed an SNA-based methodology to diagnose them from recorded traces. The agents are role-prompted personas on a shared LLM without external tools, so any value they add must come from how the conversation is structured. We evaluated three role sets on PM-LLM-Benchmark and used node- and edge-level judge scores to identify weak handoffs, which were then removed to obtain a pruned configuration.

Three findings stand out. First, agentic configurations are not automatically better than direct answering: the no-agent baseline is competitive, and only the pruned configuration improved over it on average, suggestively rather than conclusively on 57 questions. Second, role and handoff usefulness are distinct: strong intermediate reviewers do not guarantee strong final answers, so synthesis and routing matter at least as much as role design. Third, the same configuration file behaves differently across base models, producing sparse early-stopping networks under one and denser deliberation under others, so role sets are not portable in isolation from selector behavior.

Two practical points follow. Direct answering should remain a strong baseline, and personas should be added only when they contribute a distinct angle such as formal checking, counterexample search, or fairness review. Conversational agent frameworks should also expose traces, configurations, and handoff statistics as first-class artifacts so that routing can be revised on observed evidence rather than intuition.

Several limitations qualify these results. Judge scores are proxies with possible model-specific biases; the pruning was diagnosed and re-evaluated on the same benchmark and should be re-tested on held-out splits; we did not measure cost, latency, run-to-run variance, or human agreement; and tool-less agents mean that “formal” and “conformance” roles remain LLM personas rather than computational checkers. Future work should combine SNA-based diagnosis with human validation, explore tool-augmented personas (e.g., PM4Py or SQL engines), and study category-specific routing and cost-aware optimization. The broader takeaway is that agent conversations can themselves be treated as event data: both a source of answers and evidence for improving the system that produced them.