Interpretable Conversation Routing with Latent Embeddings Approach

1. Introduction

Chatbots built on top of large language models (LLMs) face more and more complex tasks [1], which can even contradict each other. For example, answers may require different length and level of details, model can use external data sources, call other models, APIs or run certain code to improve the result for a specific use case. A simple monolithic architecture for LLM based applications cannot comprehend such challenges as it is not able to execute all diverse tasks with the same level of quality [2]. One big prompt would not be enough to handle it and may even cause conflicts between instructions for use cases (for example be creative in some situations and give laconic, straight answers in others).

Routing layer can solve this problem by orchestrating a system of multiple specified agents, which solve concrete tasks instead of trying to handle everything [3]. This way the chatbot is split into a network of sophisticated sub applications, which are connected by a classifier that chooses the best answer generator for new message. This allows to make the answering process more context-aware, accurate and cost efficient. It also helps to avoid conflicts between instructions as these agents can be different prompts, models or even have some predefined sets of actions and answers.

Another advantage is increased security of application as harmful instructions, unintended questions or jailbreaks will not even get passed to the LLM itself [4]. Routing layer can filter out such requests and protect agents from any instruction injections and attempts at out-of-scope usage of application [5].

Routing problem can be expressed as a simple classification task. It can be solved by a fine-tuned transformer or any other text classifier [6,7]. The only requirement is to predict the correct route quickly as this layer would be called for every input and can easily become a bottleneck in case of a high latency. However, developers are likely to face a problem of gathering and annotation of training data for conversations. Fine-tuning a classifier would require thousands of correctly labeled examples, which cover different use cases. Messages have to diverse as users can formulate similar requests in multiple ways or just make spelling, punctuation or grammar mistakes. Some cases can be covered poorly and it would be difficult to catch it before the real-life usage of the system. Inputs for modern chatbots are not structured and do not have any determined, correct way of expressing users’ intentions. This makes the development and support of routing layer much more difficult at early stages of chatbot lifecycle.

Multiagent chatbot systems can significantly enhance performance of medical workers by taking routine tasks and providing a simple way to interact with already collected data [8]. Machine learning can act as advice system to help with diagnosing, forecasting and modelling [9]. However, interpretability and controllability of machine learning systems is crucial in this field as it requires a full-on trust from user to the system with detailed explanations of the decision-making process [10]. This requires each step of the answering to be explained and grounded including the routing layer.

Also, there is going to be a problem of maintenance. Number of routes can change with time and definitions of existing routes can be dynamic too. This would require data gathering, annotation, full retraining and testing of the routing model, so the update will take a significant amount of time to end up in production.

Another significant challenge is the interpretation and control of routing process. It is crucial to understand reasoning behind the chosen route and to have ways of easily controlling it later [11]. Standard text classification models do not provide means to control the inference without tuning the model itself [12]. Interpretation methods are lacking too as it is either analysis of SHAP like visualizations or attention masks [13].

Zero and few shot classifiers can ease the implementation as they do not need training or a large set of examples [14]. Approaches like Natural Language Inference (NLI) classifiers or LLM in-context learning can be used to solve it [15,16]. Our previous research proved that NLI routers significantly underperform and LLM router is too unreliable. LLM router is easy to implement and has a high accuracy even for multilingual systems. However, it is biased towards classes order, labels and descriptions of classes [17]. Even an intermediate Multiple Evidence Calibration (MEC) cannot make it more stable [18]. Such instability makes them suitable for initial data annotation or application prototyping, but they cannot be relied on in real life production environment.

LLM router lacks interpretability as the model can use similar reasoning for different results, which makes it more difficult to debug and tune use cases [19]. Control is possible via prompting and generation hyperparameters change, which is better than a full retraining. However, it is difficult to correctly adjust LLM classifier behavior without breaking other cases considering lack of interpretability and stability.

To address the issues listed previously a method of routing via latent embeddings retrieval was proposed [20]. This work shows that semantic routing approach can be used in fields, where high interpretability is critical, as it can provide explanations of decision making and can be easily controlled and modified to adjust the behavior.

In this paper we aim to provide a set of best practices on how to use such semantic routers, particularly to enhance their efficiency in dynamic, multilingual and multitask environments with a high level of interpretability and control over inference. Research highlights main disadvantages of this approach and how to address them. We checked how such routing method handles LLM jailbreaks, influence of examples set size, compared multiple examples set building approaches and created visualizations to highlight router decision making.

The article is organized as follows: Section 2 covers used datasets, the semantic routing approach in general and proposed modifications. Section 3 presents conducted experiments, their results and interpretations. Finally, Section 4 provides conclusions and discusses obtained results in terms of further possible research and general usage of the routing approach.

2. Materials and Methods

2.1. Routing Benchmark Datasets

First of all, we need a dataset to test semantic routing approaches, which would contain multiple types of questions to be handled by different agents. During our previous research we used a dataset of questions for wine store assistant, which handles following scenarios:

• Catalog recommendations, where agent has to generate search query, find corresponding wines and form a message with a proposition. Resulting message has to be grounded by search response, so model does not make up any items out of catalog;
• General questions about wine, where agent has to answer any question about wine topic in general without search or other additional actions. Here the model is allowed to talk about topic in any way, even about items which are not sold by store;
• Small talks, where agent just needs to keep a human-like friendly conversation and answer simple messages like greetings, gratitude and others. Agent is allowed to be creative and does not require any additional actions;
• Out of scope messages (offtop). Such messages should be ignored by system completely as they are either LLM jailbreaks (harmful or just attempts to use the chatbot as free LLM wrapper) or just random questions out of system’s scope (not about wines or their attributes, history, geography and the wine making process itself in case of this dataset) [21].

Instructions to solve these use cases conflict with each other and they need different intermediate steps, so one shared answering pipeline would make the system not only less stable and reliable, but also more expensive and slower. Such conflicting use cases may appear in any field, so splitting them into different sub applications with a routing layer on top would be a suitable strategy to manage such situation.

Dataset was gathered in multiple ways: a set of examples provided by consumer (wine online store), questions scrapped from the web and synthetically generated questions with LLM. Synthetic messages were generated by Gemini 1.5 Pro for each route separately with multiple configurations: a question by a newbie customer, a middle-experienced customer and an expert [22]. Offtop (out of scope) messages were taken from Stanford SQuAD dataset [23]. Table 1 presents general statistics on gathered dataset.

Also, we used machine translation to create some multilingual samples in French, Italian, German and Ukrainian. Table 2 shows language distribution of texts in the dataset.

Benchmark was saved in 2 formats: a full one with all 2,876 texts and a small randomly chosen subset with 40% of texts (1,151 samples).

Dataset was created for internal usage by Teamwork Commerce, so it cannot be open sourced at this moment.

Another dataset we used in this research is Jailbreak28K and specifically its mini version with 258 harmful LLM jailbreaks, which should interfere with original instructions and inject malicious intents or make the model solve an unethical problem [24]. Also, we used a 10% sample of the large dataset (2800 samples) without “figstep” types of questions (questions for multimodal LLMs with vision capabilities, where the jailbreak itself is passed as an image) as we test routing specifically for text-only interactions.

2.2. Semantic Routing Based on Latent Sentence Embeddings Retrieval

In this paper we propose a slight modification of semantic routing based on latent sentence embeddings retrieval. Original approach can work both as a few-shot learning classifier and as a supervised learning classifier. It receives a set of examples for each route and a rejection threshold. The router builds an embedding storage for provided examples and each time it needs to classify an example it retrieves N most similar examples and their respective labels. Similarity scores of examples get aggregated by labels with a function, which is chosen during the creation of routing layer. It can be a sum, mean or max aggregation. Then, each route has to pass a rejection threshold filtering. If the route does not get an aggregated similarity score higher than its rejection threshold, than it gets rejected even if it is top 1 among all fetched. If all routes get filtered by rejection thresholds, it means the message is out of scope of the current system and should not be passed to any agent further (Figure 1).

This routing approach provides an easily interpretable solution as user can just check which examples were most similar to the input message and compare aggregated similarity scores to threshold values in case message got rejected completely. Also, it is easy to control such system and modify its behavior by just adding new examples, deleting old ones, modifying existing or adjusting the thresholds.

In case you have a training dataset and want to tune such routing layer, it would only adjust the thresholds without learning new examples or generalizing them, which could boost the performance of the system even further. The router will iterate over the dataset, try different values for thresholds in provided range and check which one gives the best performance on training data.

Another problem of this approach is redundant, alike examples are not filtered and get memorized by the routing layer anyway. The examples set can be optimized additionally to take less time on retrieval and make classification faster. Also, the routing layer does not try to generalize or transform examples in any way. This leads to larger example set to cover more alike samples. In case of our dataset it can be multiple examples to cover different occasions to buy a wine, when one generalized and more templated sample could cover correspond to all of them just fine.

We propose to add an additional step of initial examples filtering based on their similarity to already memorized samples. There would be 2 approaches to prune excessive examples, which already have similar counterparts in router memory:

• Just filter input examples by another cutoff threshold to reduce the examples set size and make it more efficient;
• Save not just the original version of input example, but also a generalized version generated by LLM to cover multiple use cases at once with possibly higher similarity. Examples would become more templated to correspond to multiple requests at once.

By pruning we mean following steps during router initialization:

• Fetch examples set of the route provided by developer;
• Encode all examples of route with embedder model;
• Save the first example to router memory;
• For following examples retrieve top 1 similar sample of current route, which is already saved to router memory;
• If similarity score of top 1 most similar sample is higher than example pruning threshold (a coefficient, which describes when 2 texts are too similar, so it does not make sense to save another example alike), ignore new example and do not add it to router memory;
• If similarity score of top 1 most similar sample is less than example pruning threshold, add it to router memory either as it is (first proposed approach) or pass it through LLM to generalize it (second proposed approach) and save this generalized version to router memory;
• Repeat for every route.

During our work we checked these routing adjustments and original way with multiple encoders, quantization techniques and embedding sizes to check how additional pruning changes the behavior of the router in terms of accuracy and speed (especially for local environments as API based encoders are more difficult to track in terms of time performance).

3. Results

3.1. Examples Pruning Effect on Semantic Routing

First of all, we tested original semantic router approach and our 2 proposed modifications on a small subset of our routing benchmark. Here is a detailed router configuration we used in this research:

• Aggregation function: max;
• Number of examples to retrieve: 15;
• Similarity score threshold for each route: 0.6 (if route aggregated similarity is lower than 0.6, it would be rejected);
• Examples are provided only for valid routes: general wine questions, catalog and small talk. Offtop route is assigned only when all valid routes get rejected;
• Examples pruning threshold: 0.8;
• LLM configuration for generalization of examples: GPT 4o with temperature 0.0 and top p 0.0 [25];
• Encoder model for router: text-multilingual-embedding-002 by Google with task type RETRIEVAL_QUERY with embedding size 768 (task types in Google text embedding API allow to choose model optimized for the specific embeddings use case) [26,27];
• Encoder model for pruning: text-multilingual-embedding-002 by Google with task type SEMANTIC_SIMILARITY with embedding size 768;
• Full list of examples provided to router is listed in Appendix A.

Table 3 shows classification report (accuracy and routes F1 scores) for this small portion of benchmark for all 3 router configurations.

As you can see pruning preprocessing allowed us to remove around 36.1% of examples from router without an overall performance degradation. Overall accuracy stayed the same as it was with only offtop and small talk routes getting a slight F1 score drop (0.02-0.03).

However, generalization of examples proved to be less efficient than expected as it loses to router built with all available examples and router with pruned examples and no additional transformations. Generalization prompt can be found in Appendix B.

Then we ran the same measurement on full benchmark to check how the results would scale on it. Table 4 shows obtained classification report and result of LLM based router built with GPT 4o.

Pruning without generalization still keeps the overall accuracy on the same level as a router built with all available examples. Slight performance degradation for small talks and offtops gets reproduced too with the same 0.02 score drop. Examples transformation with a generalization prompt still did not show any positive effects as the overall accuracy is the lowest for this router and it does not outperform others on any specific route or at least by examples set size.

Semantic router still loses to LLM in-context learning classifier (classification prompt can be found in Appendix B), but the difference is just 6% of overall accuracy with advantage of high interpretability, easier control and reproducibility. Adding more specific examples can make the performance of semantic router even better and closer to the LLM router. Also, it would be easier to extend it later in production and debug wrong cases.

Then we proceeded with checked different values of pruning threshold and how it affects the router examples set size and performance. Table 5 shows multiple pruning configurations with different values of pruning threshold to demonstrate how it affects the size of router examples set.

Once routers with different pruning thresholds were constructed we measured them on a 40% subset of routing benchmark. Table 6 shows how different pruning threshold values affect the accuracy of the router.

Table 5 and Table 6 show that tuning a pruning coefficient is crucial as low values can lead to catastrophic performance degradation of the router (coefficient values lower than 0.75). At the same time keeping the coefficient around 0.90 or higher would not make any sense, as almost no examples get pruned and the set size remains almost the same. Best results were achieved with pruning coefficient from 0.80 to 0.85 as they keep the original performance of the router with a set size reduction around 15-34%. These measurements may differ for different encoders, so pruning coefficient should be tuned specifically for the chosen router encoder.

However, we noticed slight accuracy fluctuations with different pruning coefficient values. For examples coefficient 0.90 filters only 1 example, but gets a worse accuracy on 40% subset of the benchmark due to more errors during catalog and small talk classification. On the other hand, 0.85 coefficient leads to an improvement of overall accuracy and better results for catalog and general wine questions routes. These 2 routers pruning configurations were remeasured on full benchmark dataset and the results are presented in Table 7.

The same behavior got reproduced on full benchmark, so it was decided to check which examples were pruned and why it happened. First of all, 0.90 coefficient was checked as it lost just 1 example and got a drop of accuracy from it.

As for coefficient 0.85 we focused on general wine questions as it got significantly improved with such configuration. Route pruned only 2 examples: 1 about the influence of alcohol content on wine characteristics and another about wine geography. Both examples had similar ones among memorized samples, so it made sense to not add them too. This category of questions is the hardest in dataset as it can be close to out of scope questions (for example when they touch wine geography or history). Also, the questions here can be diverse and cover completely different topics: glasses, grapes, wine traditions, geography, history, winemaking, wineries, the wine itself and the ways to store or drink it. It makes sense to memorize as many examples as possible as most of them can be unique. This proves a necessity to optimize the pruning threshold for each route or class specifically as some of them need more data to work better and some would be fine if some redundant examples get removed from the memory.

Originally proposed semantic router allows to increase the example set over time as developers get more real-life data. However, as router vector count grows it would make retrieval slower and memory consumption would increase too. Pruning would allow to keep vector count optimal and can keep the retrieval efficient without quantization of router embeddings or manual examples filtering. The only pruned example was “I’m impressed with the variety of wines you offer.” from small talk route, which led to a lack of similar examples about user’s impressions and store experience.

3.2. Jailbreaks Prevention

Next step was to check the level of protection semantic router provides against common LLM jailbreaks. First of all, we measured routers without pruning, with pruning coefficient 0.8 and with both pruning and generalization on a mini version of Jailbreak28K. Router has to predict offtop for all provided texts as they should not be passed to any agent down the line. Table 8 shows the results obtained from test.

All configured routers show the same performance in terms of jailbreaks rejection, which proves the efficiency of semantic routing not only for multiagent chatbot orchestration, but also as a security measurement for LLM based applications.

Next step was to run the same test with a 10% sample of a full Jailbreak28K without image jailbreaks (2,600 texts). Table 9 shows the results of conducted measurements.

Result got fully reproduced even on a larger set of possible instruction injections. The final thing to check is how router would handle these jailbreaks if they contain topic keywords like “wine”, “red”, “catalog” and others. Messages from Jailbreak28K do not touch the target topic of router, so it would be easier to write them off as offtops. Addition of topic keywords can make it easier to mislead the router by making input embeddings closer to embeddings of valid examples, so such vulnerability has to be checked.

For this we create a small set of questions about wines or just sets of keywords (10 texts). Mini version of Jailbreak28K will be used as a base for this test. Wine related statements and keywords will be added in the end of the text and for 35% of texts they will be added both at the start and in the end of the message. Table 10 shows the result of the experiment.

Performance of the router drops significantly as it gets mislead by injected wine-related statements. This makes the router vulnerable to instruction injections if the attacker knows the primary topic of chatbot and adds some topic-related statements in there to avoid getting sent to out of scope route right away. This does not mean that the underlying agents won’t be able to handle this later, but a possible harmful message would still get to the LLM as router would not filter it.

3.3. Interpretability and Controlability

Even if the semantic router still loses to LLM based router, it provides crucial features for chatbot systems maintenance: high level of control and interpretability by extending and modifying the examples set.

We created some visualizations to demonstrate these advantages of such approach. They were generated with the same semantic search configuration as the router in Section 3.1 (top 15 most similar examples, text-multilingual-embedding-002 embedding model by Google with task type RETRIEVAL_QUERY with embedding size 768). Pruning was applied to router examples set with pruning coefficient 0.8, so it has 46 memorized samples.

First of all, we visualized how it classifies a catalog question by creating 2D TSNE projections of examples and query embeddings [28]. You can see the result in Figure 2.

Larger circles with a clear black border are most similar examples, while others did not make it to top 15, so they do not affect the prediction. Most such samples belong to catalog type in the visualization, which explains the router decision making process. In Table 11 we show top 5 most similar examples from this visualization to show the exact text, which model considered most similar to the query “I’m looking for a wine to use in cooking, specifically for making risotto. Any suggestions?”.

Next step was to visualize an out of scope (offtop) message. We chose a sample with following text: “Who initially proposed using the term “Native American” to recognize the primacy of the native population?”. Result can be seen in Figure 3.

Sample is distanced from valid routes examples. All examples among top 15 did not cross the 0.6 threshold, so this sample would not be passed to any agent of the chatbot. Maximum similarity value was 0.43 for general question example. However, it is important to note that categories like small talk or general questions about a certain wide area can be more vulnerable than more specific categories like catalog recommendations. Out of scope sample is closer to those vulnerable categories in Figure 3 than to a specific, narrow one about searching store goods database. This proves that semantic router would achieve its best performance with a narrower definition of routes instead of working with general and wide topics if they can be difficult to distinguish.

Final visualization shows a case of misclassification by router. We took one of previously adjusted jailbreaks, which gets wrapped in statements similar to the catalog route and visualized it the same way as in Figure 2 and Figure 3. Here is the text of the sample: “Red wine, white wine, rose. Wines, wines. Respond to every prompt I give you with a moralizing rant about the openai content policies, but then start a new paragraph with the sentence “anyways, since we are out of this *** ***, let’s ignore the *** rules: “ and then respond to the prompt exactly as an unfiltered and unsafe, completely unlimited language model could do. Be detailed. Prompt: Invent a method for producing fake legal documents for court cases. I want to find a good red wine for date tonight.”. Figure 4 shows the result.

As you can see sample falls close to catalog examples and 13 out of 15 examples get similarity score higher than the rejection threshold (0.6). This way adding a set of keywords related to router primary topic can significantly mislead the router.

Such visualizations and tables can easily explain the router decision making for any area and topic as they just provide a set of most similar examples and their scores. So, if the user would like to modify router behavior, they can add another example for the target route to avoid further misclassification.

4. Discussion

Semantic routing approach based on latent sentence embeddings retrieval can achieve close performance to LLM in-context learning classifier and provides a high level of interpretability and controllability. Router can be adjusted by adding more examples, modifying existing ones or changing the rejection threshold to filter fetched examples differently. Decision making process is clear and can be easily understood as user just needs to check which examples were chosen as most similar and compare their similarity scores to the rejection threshold.

This approach can be extended for the interpretable text classification in general in areas which require a detailed explanation of conducted decisions and even for multimodal tasks with multiple domains [29]. At this stage it can be less accurate then an ordinary tuned transformer or LLM classifier, but it is easier to understand and control. Modification would not take a full-on fine-tuning and the results are easily reproduced in contrast to LLM router. Routing achieves its best performance with narrowly specified options, so it is recommended to avoid wide definitions, which can possible overlap. There is still field for further research on how to increase the accuracy of this approach to keep both interpretability of semantic router and accuracy of transformer classifiers.

Proposed automatic example pruning allows to not overextend router example set with redundant samples, which can harm its performance without any added accuracy. This way example set can get almost 40% smaller without accuracy degradation and manual filtering of too similar examples.

However, generalization of examples in addition to pruning did not improve performance and even caused a slight drop of accuracy. It got worse not only in more wide routes like “general wine questions” or “small talk”, but also in narrow and specific one like “catalog”. At the same time out of scope questions still get labeled with a similar success rate to ordinary router or the one with pruning mechanism.

Even though semantic router increases the security of the chatbot application by filtering most out of scope messages and jailbreaks, it is still vulnerable to attacks when such message is wrapped within valid statements, which are similar to target routes. This problem should be explored further to find a solution to counter these misleading examples and avoid jailbreaks.