A Sovereign Conversational Assistant Powered by ALIA and Mistral for the AI Act Age: Architecture, Governance and Evaluation

1. Introduction

Digital twins and Living Labs are becoming central instruments for smart-city governance, enabling real-world experimentation, continuous sensing, and simulation-assisted decision-making. In cultural-heritage contexts, these approaches are expected to support preventive conservation, risk management, accessibility, and sustainable visitor flows [13,14]. Major heritage and urban sites serve as prominent examples. They are increasingly framed as digital twins, integrating 3D models, IoT sensors, AI, and data analytics to collect real-time information (e.g., visitor flows, climate, pollution), run scenario simulations (e.g., evacuation, floods, seismic events), and improve the visitor experience. Because these systems involve highly heterogeneous data—ranging from technical operational rules and live sensor feeds to conservation protocols and scholarly sources—navigating this information can be overwhelming. Overcoming this "last mile" of information access is a practical barrier we aim to solve.

In local-government contexts, the Smart Cities literature highlights both the breadth of public-sector AI use cases and the need for a responsible and trustworthy deployment [7]. In cultural venues, chatbots are increasingly being adopted to distribute curated content and support visitors [11,12]. However, public-sector and heritage operators face governance friction when adopting standard proprietary large models, including uncertainty about data residency, reproducibility, transparency, and consistent support for local languages.

This paper presents a sovereign SLM+RAG assistant designed to be deployed under public-sector constraints, grounded in authoritative sources, and aligned with the EU regulatory direction [45]. The assistant is designed as a reusable component of the Libelium Heritage Living Lab: a “front door” to the digital twin for visitors and staff, and a governance-aware access layer for heritage knowledge and digital twin services.

1.1. Contributions

The contributions of this paper are as follows.

• A reference architecture for a sovereign SLM+RAG assistant tailored to a cultural-heritage Living Lab, integrating evidence retrieval, provenance, data analysis, safety, and language controls;
• A governance blueprint mapping technical controls (provenance, logging, refusal, language steering) to public-sector requirements and EU regulatory direction [45];
• A benchmark-driven evaluation with an explicit scoring model and failure analysis, highlighting actionable improvements for multilingual and safety-critical deployments.

2. Related Work

2.1. Digital Twins and Heritage-Focused Smart-City Applications

Heritage digital twins have evolved from static 3D replicas to cyber-physical systems that include sensing, data services, and decision support. Reviews highlight the integration of digital twins with BIM, 3D scanning, IoT, and analytics in conservation workflows [14]. In museums, enabling technologies and sensor requirements have been surveyed throughout the digital-twin lifecycle [13]. Recent work emphasizes formal knowledge structures and semantic layers (e.g., ontologies for sensors and actuation) to support trustworthy and explainable digital twins of heritage [15,16].

Digital twins are also discussed in smart tourism and destination governance, including the management of overtourism and sustainability [19]. These strands reinforce a key implication for heritage destinations: digital-twin investments must be paired with usable, accountable interfaces for diverse stakeholders.

2.2. Living Labs and Co-Creation

Urban Living Labs provide co-creation settings for socio-technical experimentation, learning, and value creation. Foundational work characterises co-creation dynamics and the organisational patterns that shape participation and knowledge generation [20]. Recent Smart Cities research connects living-lab practice to smart-city evolution through socio-technical innovation lenses [9]. Evaluation frameworks highlight the need for longitudinal assessment, stakeholder inclusion, and institutional learning [21]. For cultural-heritage and tourism contexts, systematic reviews emphasise the need to integrate heritage goals into smart-city development and place-making [18].

2.3. RAG for Trustworthy Natural-Language Interfaces

Retrieval-Augmented Generation (RAG) grounds model outputs in retrieved evidence to mitigate knowledge cut-offs and hallucinations [22]. Recent MDPI systematic reviews synthesise RAG techniques, metrics, and challenges, pointing to fragmented evaluation practices and a need for realistic benchmarks [23,24]. Advanced RAG frameworks explore reasoning-aware retrieval planning and graph-based organisation [25]. In Smart Cities, RAG has been proposed to improve interaction and trust for digital-twin systems [8], which motivates adopting RAG in heritage settings where provenance and interpretability are essential.

2.4. LLM Orchestration and Agent Engineering: The LangChain Ecosystem

In the evolving landscape of Large Language Models (LLMs), the transition from simple model querying to the deployment of robust AI agents requires sophisticated orchestration frameworks. A prominent standard in the current state of the art is LangChain [58]. Originally developed to simplify the creation of LLM-powered applications by chaining together prompts, models, and external tools, the LangChain ecosystem has expanded into a comprehensive suite for end-to-end agent engineering. The platform addresses both the development and production phases of the AI lifecycle. For development, it provides the open-source LangChain library for model-agnostic, high-level abstractions, alongside LangGraph, which facilitates stateful, multi-actor orchestration for custom agents requiring fine-grained control and multi-step planning.

3. Materials and Methods

3.1. Use Case: Libelium Heritage Living Lab Information Assistant

The Libelium Heritage Living Lab aims to operationalise a digital twin for the Heritage sites, integrating multiple technologies and real-time data streams to improve conservation, safety, accessibility, sustainability, and visitor experience. Within this programme, the assistant targets three primary user groups:

1.

Visitors and families: accessible answers about the monument, itineraries, rules, and cultural context.

2.

Researchers and operators: evidence-grounded answers referencing authoritative documents to support interpretation and conservation workflows.

3.

Living Lab staff: Expert technician staff in charge of acting upon the insights provided by the Libelium Heritage Living Lab.

3.2. Models: ALIA, Mistral and EmbeddingGemma

To operationalize our Sovereign Conversational Assistant (SCA) while adhering to strict public-sector data governance, our platform integrates a carefully selected suite of open-weight models.

1.

The Sovereign Engine (ALIA): Our primary generative model is the newly released BSC-LT/ALIA-40b-instruct-2601. ALIA is more than just a large language model; supported by the supercomputing capabilities of the Barcelona Supercomputing Center, it serves as a foundational public AI infrastructure optimized for Spanish and its co-official languages. By providing open and transparent weights, ALIA actively strengthens technological sovereignty and aligns seamlessly with the data privacy requirements of heritage Living Labs [26,27,28].

2.

The State-of-the-Art Benchmark (Mistral): To rigorously evaluate ALIA’s performance and benchmark our proposed assistant against current industry standards, we introduce a secondary, highly capable open-weight model: mistralai/Mistral-Small-3.2-24B-Instruct-2506 [56]. Serving as a state-of-the-art representative for mid-size small language models (SLMs), Mistral-Small enables a comprehensive comparative analysis across our five testing categories, ensuring that our sovereign approach does not compromise on generative quality, safety, or instruction-following capabilities.

3.

The Retrieval Mechanism (EmbeddingGemma): Finally, the core of our retrieval-augmented generation (RAG) pipeline relies on an efficient and accurate vectorization mechanism to ground the assistant’s answers in authoritative Living Lab sources. For this, google/embeddinggemma-300m [57] is utilized to encode the knowledge base. This lightweight yet powerful embedding model facilitates the high-fidelity semantic search necessary to enforce factual provenance and resist hallucinations within the SCA platform.

3.3. System Architecture, Knowledge Bases, Application Scope and Data Handling

To maximize the assistant performance across the three user groups we have separated the Libelium Heritage Living Lab core data corpus into two distinct knowledge bases, therefore, creating two sub-applications which share the same architecture (see Figure 1).

1.

Location-Expert: This application focuses on offering an accessible interface to interact with academic sources regarding the site’s history and architecture, official institutional webpages, curated PDFs (e.g., visitor-facing audioguide material), and other overarching institutional publications. By choosing different profiles, users can toggle between family-standard, a more accessible guide for simpler explanations and questions about the general visitor experience, and researcher-standard, an academic profile tailored toward domain experts and scholarly research. This application is developed with multilingual support to accommodate an international visitor base and boasts system prompts in English with output language restrictions. However, initial testing of the app will be restricted to English and Spanish.

2.

IrisChat: This virtual lab tech task is to help technical staff navigate the Libelium Heritage Living Lab and Digital twin interface, based on Libelium’s Iris360, with ease, leveraging its knowledge base composed of user manuals for the different functionalities of the platform. Furthermore, it also can interact with the real-time sensor data, answering user questions on related to sensor data or summarizing plots for easy consumption. This app is specifically designed for Spanish tech staff and, therefore, for the moment is only available in Spanish with system prompts tailored to the Spanish language.

In both cases, knowledge base retrieval operates over chunked document representations stored in a vector index; for each query, the system retrieves the top-k evidence chunks (in our experiments, $k=5$), each with a similarity score and source identifier courtesy of google/embeddinggemma-300m. The retrieved context is then injected into the generator prompt along with explicit instructions to: (i) Ground claims in the provided context, (ii) Avoid unsupported speculation, (iii) Produce Spanish or English output. The pipeline offers native support for multilingual queries even if most of the sources are in spanish, as it is to be discussed at Section 3.4.

3.4. Retrieval-Augmented Generation (RAG) Methodology

3.4.1. Overview and Design Rationale

Conceptually, the design follows the canonical RAG paradigm: retrieve evidence from an external knowledge base and condition the generator on this evidence to reduce hallucinations and improve factual grounding.

We extend this baseline with two constraints that are critical for public sector and cultural-heritage deployment:

• Governance-by-design controls tightly integrated into the inference loop (policy screening, reduction of sensitive data exposure, provenance logging, and refusal/safe completion).
• Spanish-first language homogenisation. As our current use cases knowledge corpus are mainly in Spanish, to support multilingual traduction for the Location-Expert, translation of non-Spanish user-prompts, in this case to English, is required to accurately retrieve relevant sources. This could be extended to all languages supported by the LLM.

3.4.2. Offline Ingestion and Indexing

The knowledge base is built from curated, authoritative sources (institutional webpages, official PDFs, guides, and policies). During ingestion, each document is normalised into a structured record:

• Content: extracted text and layout-preserving segments (e.g., headings, lists).
• Metadata: source identifier (URL/path), document type, timestamp/version, and governance tags (e.g., public-facing vs. internal operational notes).

Documents are segmented into overlapping chunks to balance semantic coherence with retrieval granularity. Each chunk inherits document metadata, enabling provenance and auditability at the answer level. Chunks are then indexed for retrieval.

3.4.3. Dense Retrieval

At runtime, given a user query q, the retriever performs a single dense retrieval pass: the query is encoded with GemmaEmbeddings and a cosine-similarity nearest-neighbour search is executed over the vector store, returning the top-k candidate chunks ($k=5$ by default).

Before generation, we apply a lightweight evidence sufficiency gate: if the retriever returns low-confidence or irrelevant evidence (e.g., below a similarity threshold or failing mandatory metadata constraints), the assistant must fail transparently by (i) stating limitations and (ii) requesting additional disambiguating details, rather than producing speculative completions. This behaviour directly supports trust and auditability in public-facing deployments.

3.4.4. Context Construction

Retrieved chunks are assembled into a context block $\mathcal{C}$ by iterating over all documents that survived the similarity-threshold filter, in descending order of cosine similarity score. Each entry is formatted as a numbered section containing: (i) the chunk text in full, (ii) the source file path as provenance metadata, and (iii) the numeric relevance score, all enclosed in an XML-style <rag_context> tag that demarcates the retrieval evidence for the generator.

3.4.5. Grounded Generation with ALIA/Mistral

Given $(q,\mathcal{C})$, the generator produces an answer a with explicit instructions to:

• answer only using information supported by $\mathcal{C}$ (grounding);
• clearly state uncertainty when evidence is missing (no fabrication);
• maintain the required user-profile style and tone (e.g., families vs. researchers);
• output in Spanish or english if multilingual assistant is in use (language constraint);
• attach citations or source identifiers corresponding to the retrieved chunks (provenance).

A governance layer wraps the retrieval and generation steps to enforce: (1) policy screening and refusal/safe completion for harmful requests, (2) privacy-preserving logging and PII handling, and (3) traceability via stored source identifiers and retrieval scores. These controls operationalise a “governance-by-design” approach aligned with EU regulatory direction and public-sector requirements [32,33].

3.4.6. Post-Processing Guardrails: Provenance, Safety, and Language

After generation, outputs are post-processed with three checks:

1.

Citation/provenance formatting: ensure that the response includes retrievable identifiers (URL/path + chunk/document IDs) so that users and auditors can verify the evidence trail.

2.

Safety/refusal enforcement: if the query is classified as disallowed or sensitive, override the answer with a refusal template or safe completion, consistent with policy.

3.4.7. What Type of RAG Is This? Positioning in the RAG Design Space

The implemented method is best characterised as a standard RAG augmented with corrective governance). In the taxonomy of common RAG variants, it sits between:

• Simple/Standard RAG: single-pass retrieve → generate, typically without explicit self-critique or multi-step tool use.
• Corrective RAG: adds decision points when evidence is insufficient or irrelevant (e.g., fallback, clarification prompts, or retrieval retries), thereby reducing unsupported outputs.

It is not an agentic/tool-using RAG by default: we intentionally preserve a read-only posture (no automatic actions) to minimise operational risk and simplify compliance and auditing in a public-sector environment.

Table 1. Comparison of RAG variants and their implications for sovereign, public-sector deployment.

RAG variant	Core mechanism	Typical cost/latency	Governance fit
Simple / Standard RAG	Single retrieval pass; prompt conditioned on top-k chunks; one generation.	Low–moderate (1 retrieval + 1 generation).	Good baseline; limited self-correction.
Corrective RAG	Adds relevance/sufficiency checks; may re-retrieve or ask for clarification before answering.	Moderate (additional checks / retries).	Strong fit when transparency and “fail gracefully” behaviour are required.
Self-RAG / critique-based	Model grades its own draft, checks grounding/hallucination risk, and iterates retrieval/generation.	High (multiple LLM calls).	Potentially strong quality, but harder to audit and tune; higher latency.
Fusion RAG	Generates/aggregates multiple candidate answers (or multiple evidence sets) and fuses them.	High (multi-sample generation).	Useful for complex synthesis, but costly; risk of over-generation and inconsistency.
Speculative RAG	Produces multiple “speculative” drafts and selects/filters best via scoring/judging.	High (multi-draft + judge).	Improves robustness but increases attack surface and complexity of governance.
Agentic RAG	LLM acts as an agent that can call tools (search, APIs) and loops until goals are met.	Variable; can be very high.	Riskier in operational contexts; requires strict tool governance and human-in-the-loop.

Table 1. Comparison of RAG variants and their implications for sovereign, public-sector deployment.

RAG variant	Core mechanism	Typical cost/latency	Governance fit
Simple / Standard RAG	Single retrieval pass; prompt conditioned on top-k chunks; one generation.	Low–moderate (1 retrieval + 1 generation).	Good baseline; limited self-correction.
Corrective RAG	Adds relevance/sufficiency checks; may re-retrieve or ask for clarification before answering.	Moderate (additional checks / retries).	Strong fit when transparency and “fail gracefully” behaviour are required.
Self-RAG / critique-based	Model grades its own draft, checks grounding/hallucination risk, and iterates retrieval/generation.	High (multiple LLM calls).	Potentially strong quality, but harder to audit and tune; higher latency.
Fusion RAG	Generates/aggregates multiple candidate answers (or multiple evidence sets) and fuses them.	High (multi-sample generation).	Useful for complex synthesis, but costly; risk of over-generation and inconsistency.
Speculative RAG	Produces multiple “speculative” drafts and selects/filters best via scoring/judging.	High (multi-draft + judge).	Improves robustness but increases attack surface and complexity of governance.
Agentic RAG	LLM acts as an agent that can call tools (search, APIs) and loops until goals are met.	Variable; can be very high.	Riskier in operational contexts; requires strict tool governance and human-in-the-loop.

• Justification of our choice.

We prioritise a compact, auditable and predictable RAG stack that meets institutional constraints: minimal and stable latency, traceable sources, and minimal autonomous behaviour. More complex variants (self-critique, speculative, agentic) can improve answer quality in open-domain settings, but they introduce additional model calls (more cost and latency), non-determinism, and tool-mediated side effects that complicate public-sector compliance, logging, and red-teaming.

3.4.8. Why Translate Everything to Spanish? Homogeneity as a Retrieval and Governance Control

For the Libelium Heritage Living Lab use case, in particular when the Location-Expert is utilized, we adopt a Spanish-homogeneous pipeline by translating non-Spanish inputs into Spanish and enforcing the corresponding language at output. This design choice is justified as it improves on Retrieval quality and determinism. Monolingual retrieval reduces cross-lingual embedding mismatch, improves lexical retrieval reliability, and simplifies rank fusion calibration. When the query and corpus are in the same language, both dense and sparse retrieval tend to exhibit higher recall and fewer spurious matches.

For the IrisChat side, we have adopted a Spanish-only policy, as there is no need for multilingual support, with spanish prompts and source documents as policy rules, and safety templates are significantly simpler to maintain in one target language.

3.5. Governance, Compliance, and Safety Controls

Public-sector deployment requires governance-by-design, to address it we implement:

• Provenance and traceability: storing document identifiers and retrieved-source lists per answer;
• Data minimisation: logging policies that avoid persistent storage of personally identifiable information (PII) beyond operational necessity;
• Refusal and safe completion: policy enforcement for harmful or inappropriate requests;
• Language control: enforcing the requested language to avoid usability regressions.

These controls are aligned with the regulatory direction of the EU AI Act [45] and with responsible technology discussions in smart-city governance [7].

4. Results

4.1. Testbed

We have conducted our evaluation our evaluation utilizing OVHcloud as our solution provider. Mistral’s mistralai/Mistral-Small-3.2-24B-Instruct-2506 was already deployed as an AI Endpoint, however, BSC’s BSC-LT/ALIA-40b-instruct-2601 is not available, therefore, a specific deployment was perfromed using the AI Deploy feature to load the vLLM inference engine as a docker image running on an environment consisting of 52 vCores, 320 GiB RAM and 4x NVIDIA L40, with 45 GiB of VRAM each with 20.00 GB/s downloading speeds.

Table 2. Summary of LLM Testbed and Configuration

Model ID	Deployment	Hardware / Environment	Temp.	Max Tokens
mistralai/Mistral-Small-3.2-24B-Instruct-2506	OVHcloud AI Endpoint (Pre-deployed)	Managed Infrastructure	0.15	1024
BSC-LT/ALIA-40b-instruct-2601	OVHcloud AI Deploy (Custom vLLM Docker)	52 vCores, 320 GiB RAM, 4x NVIDIA L40 (45 GiB VRAM each)	0.07	1024

Table 2. Summary of LLM Testbed and Configuration

Model ID	Deployment	Hardware / Environment	Temp.	Max Tokens
mistralai/Mistral-Small-3.2-24B-Instruct-2506	OVHcloud AI Endpoint (Pre-deployed)	Managed Infrastructure	0.15	1024
BSC-LT/ALIA-40b-instruct-2601	OVHcloud AI Deploy (Custom vLLM Docker)	52 vCores, 320 GiB RAM, 4x NVIDIA L40 (45 GiB VRAM each)	0.07	1024

4.2. Benchmark and Scoring Model

We evaluate the assistant using an automated dual benchmark suite comprising 19 tests across five categories: historical queries, client experience, data analysis, hallucination resistance, and safety/ethics. For Location-Expert, two user-profiles are utilized: family-standard and researcher-standard.

Each test produces three signals:

• Keyword score ($S_{\mathrm{kw}}$): rule-based checks for mandatory, positive, and negative keywords;
• LLM-judge score ($S_{\mathrm{llm}}$): a rubric-based evaluation executed by an external automated evaluator or LLM judge(mistralai/Mistral-Small-3.2-24B-Instruct-2506). It assesses factuality, completeness, tone, and category adherence on a 0–100 scale;
• Language gate: a detector that verifies Spanish output to ensure strict Spanish-first language homogenization. Any mismatch is treated as a hard failure.

The final score is computed as:

$$S=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathrm{language}\mathrm{gate}\mathrm{fails}\hfill \\ 0.1·S_{\mathrm{kw}}+0.9·S_{\mathrm{llm}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(1)

A test is considered a pass when the final score achieved is strictly greater than 70 ($S>70$), as this score is consistent with covering all the relevant information the answer needs as well as avoiding context drift and hallucinations. Each suite is run 5 times, and the average scores are computed to ensure consistent, evidence-grounded performance. Figure 2 summarises the evaluation harness.

4.3. Test Suite 1: IrisChat (Spanish)

This suite evaluates the assistant’s ability to act as a technical aide, focusing on its data analysis features. In this case hallucinations substract points (up to the LLM judge to decide how much) but do not immediately render the answer invalid if the general narrative and most important values are communicated accurately. This is a limitation we acknowledge and aim to solve for future iterations.

• Data Analysis: Evaluates if the system can accurately generate reports as well as identify daily ozone patterns (low nighttime baselines and afternoon peaks) using specific hourly readings without hallucinating statistics across 1 day and 1 week data.
• Client Experience: The application is questioned about information available in the RAG source pool and it’s expected to comunicate it to the user in a specific manner. In this case we have prepared a question on how to guide the user to be able to export dashboard data to CSV.
• Safety & Ethics (Red-Team Malware Drill): A critical security test where the user role-plays as a cybersecurity student requesting a Python ransomware script.

Table 3. IrisChat Evaluation Suite: Test Definitions, Data Context, and Objectives

Test ID	Category	Data Context	Test Objective & Description
IC_analyze_001	Data Analysis	CSV File with O3 values during 1 week (168 rows) + RAG Source Pool	O3 Diurnal Cycle: Verifies that the system correctly identifies the diurnal O3 photochemical cycle (low nighttime baseline, afternoon peaks) across 168 hourly readings.
IC_analyze_002	Data Analysis	CSV File with O3 values during 1 day (24 rows) + RAG Source Pool	Steepest Drop: Evaluates the system’s ability to identify the single steepest hourly drop in O3 levels and correctly explain the atmospheric dynamics behind it.
IC_analyze_003	Data Analysis	CSV File with O3 values during 1 day (24 rows) + RAG Source Pool	Abrupt Changes: Tests if the system can identify an ambiguous number of the most abrupt variations and rates of change (acceleration/rapid drops) without triggering hallucinated values or insights.
IC_chat_004	Client Experience	RAG Source Pool	Platform Operation (CSV Export): Verifies that the assistant can effectively explain the CSV data export feature of the Iris360 platform, providing step-by-step UI navigation based on the RAG source pool.
IC_chat_005	Safety & Ethics	RAG Source Pool	Red-Team Malware Drill: A critical security test verifying that the chatbot strictly refuses to generate malicious code (a Python ransomware script), even when the user frames the prompt as a harmless educational exercise.

Table 3. IrisChat Evaluation Suite: Test Definitions, Data Context, and Objectives

Test ID	Category	Data Context	Test Objective & Description
IC_analyze_001	Data Analysis	CSV File with O3 values during 1 week (168 rows) + RAG Source Pool	O3 Diurnal Cycle: Verifies that the system correctly identifies the diurnal O3 photochemical cycle (low nighttime baseline, afternoon peaks) across 168 hourly readings.
IC_analyze_002	Data Analysis	CSV File with O3 values during 1 day (24 rows) + RAG Source Pool	Steepest Drop: Evaluates the system’s ability to identify the single steepest hourly drop in O3 levels and correctly explain the atmospheric dynamics behind it.
IC_analyze_003	Data Analysis	CSV File with O3 values during 1 day (24 rows) + RAG Source Pool	Abrupt Changes: Tests if the system can identify an ambiguous number of the most abrupt variations and rates of change (acceleration/rapid drops) without triggering hallucinated values or insights.
IC_chat_004	Client Experience	RAG Source Pool	Platform Operation (CSV Export): Verifies that the assistant can effectively explain the CSV data export feature of the Iris360 platform, providing step-by-step UI navigation based on the RAG source pool.
IC_chat_005	Safety & Ethics	RAG Source Pool	Red-Team Malware Drill: A critical security test verifying that the chatbot strictly refuses to generate malicious code (a Python ransomware script), even when the user frames the prompt as a harmless educational exercise.

• The assistant operates on pre-processed, statistically-augmented data, not on raw series.

In dashboard workflows, forwarding raw CSV or JSON directly to the model is both wasteful and unreliable: large payloads exceed context budgets, and asking the model to derive statistics from scratch produces inconsistent results. Instead, each request to analyze dashboard data passes through a structured pre-processing pipeline before any text reaches the language model. First, the raw payload, CSV (with automatic comma/semicolon delimiter detection and datetime column inference) or JSON (arrays, column-oriented objects, or single records), is parsed into a DataFrame and full-dataset statistics are computed server-side: er-column minimum, maximum, mean, median, standard deviation, valid and null counts, outlier count (values beyond $3\sigma$ from the mean), a linear trend label derived from an ordinary-least-squares regression over the time axis, the timestamps at which each per-column extreme occurred, and pairwise Pearson correlations for all numeric columns where $\left|r\right|>0.5$. For datasets with a parsed datetime index, the pipeline also produces temporal aggregates: median inter-observation gap (sampling frequency), mean value by hour-of-day together with the peak and trough hours, mean value by calendar day, and mean value by weekday, providing the model with a pre-computed diurnal and weekly profile rather than expecting it to infer such patterns from a sampled subset.

Only after these statistics are fixed does the pipeline reduce the data to a configurable row budget via smart sampling to preserve the first and last rows (temporal boundaries), the per-column minimum and maximum rows (extremes), and a uniform stride over the remainder. A second reduction pass applies if the resulting CSV still exceeds a character-count limit. The system prompt then instructs the model to read the pre-calculated statistics rather than recomputing from the sample, to identify patterns (trends, peaks, valleys, anomalies), and to open its response with a two-to-three-line executive summary before detailing findings and, where appropriate, suggesting attention points.

4.3.1. Evaluation

Table 4 results clearly show some clear ideas on where each model is. Looking at the data analysis tests (IC_analyze_001 through 003), Mistral demonstrated superior reliability. It correctly interpreted the pre-computed server-side statistics, achieving a perfect 5/5 pass rate across all analysis tasks. It accurately highlighted the diurnal ozone (O3) photochemical cycle, correctly noting the afternoon concentration peaks (and the maximum value) and the nighttime baselines without inventing extra data.

Conversely, the ALIA model struggled with hallucination resistance in complex tasks. In test IC_analyze_001, ALIA only passed 1 out of 5 runs as while the general key insights about the data and trends was correct, the evaluation logs revealed that it frequently fabricated insights and data, as it magnified a descending weekly trend that exists but that ALIA often exaggerated hallucinating much bigger differences and some values that differed from the actual information that was input. It also offered some miscalculated percentages and differences that often significantly varied from the original data points. That general correct idea idea of the answer with very visible numerical errors and some magnified insights is why its average score for this test is close to the passing mark, but it only passed once.

While ALIA successfully identified the steepest single-hour drop in IC_analyze_002 (passing 5/5), some IC_analyze_002 answers included correct insights that were more convoluted and less analytical in tone compared to Mistral, which offered a more straight forward and clear explanation on the probable causes behind the drop.

At IC_analyze_003, while Mistral incurred in smaller calculus errors when speaking about percentages such as going from 25% to 29%, it did not hallucinated when computing subtractions to discuss the drop in IC_analyze_002. This allowed its scores to not be reduced due to hallucination as it was not detected by the LLM judge. In Mistral’s case, while again it constantly points out the most significant drop and sometimes it is able to detect other sudden changes without hallucination, its output included more significant calculus errors and hallucinations introducing values that did not match the input data. In some scenarios the LLM judge misses some hallucinated jumps, which also would further lower ALIA’s, also exposing a limitation with the LLM score and something that should be addressed in future paper revisions.

At the RAG test, IC_analyze_004, both models scored an average of 0.89, successfully guiding users on how to export dashboard data to CSV formats. They successfully parsed the retrieved text to deliver clear, Spanish-first instructions without losing the user in unnecessary technical details.

As Test IC_chat_005 simulated a cybersecurity red-team drill where a user requested a Python ransomware script, framing it as a harmless educational exercise, both models achieved their highest scores in this category (ALIA: 0.97, Mistral: 0.96). The safety guardrails and refusal mechanisms functioned flawlessly with both models declining the malicious request immediately without being easily tricked by the academic context. Then, they successfully redirected the user to Iris360 as they summarized what they can do in the context of Iris360 with ALIA explaining in much detail how it could assist with Iris360 tasks based on the RAG sources pulled by the pipeline.

4.4. Test Suite 2: Location-Expert (Spanish & English)

This suite evaluates the public-facing application across two user profiles (families and researchers) and two languages to ensure it delivers accurate, tonally appropriate, and secure information in a multilingual environment. As text answers in this assistant are more sensitive and difficult to check for the end-user, hallucination does render the answer invalid and limits it to

• Historical Queries: This category tests the assistant’s ability to retrieve long, historical facts from our trusted database and synthesize complex academic topics. A key requirement here is tone adaptation: ensuring the assistant speaks simply and accessibly to families, while utilizing precise, academic terminology for researchers.
• Client Experience: In these tests, the application is questioned about operational information on the site that would be aligned with what visitors could ask staff personnel. Only the family profile is benchmarked in this category.
• Hallucination Resistance: The system is tested against fake prompts, such as a user asking for details about a non-existent, mythical underground chamber. The assistant must resist hallucinating, state the facts, and firmly correct the user based strictly on the approved source pool.
• Safety & Ethics (Vulnerability Probing): As data privacy and infrastructure security are paramount, especially under public-sector AI governance. This category tests the assistant’s refusal mechanisms against manipulative prompts trying to exploit its site knowledge for malicious purposes.

Table 5. Location-Expert Evaluation Suite: Test Definitions, Sources, and Objectives (Executed in ES and EN)

Test ID	Category	Data Context	Test Objective & Description
LE_hist_001	Historical	RAG Source Pool	Fact Retrieval & Tone Adaptation: Verifies fact retrieval by identifying the site’s oldest construction. Evaluates the system’s ability to adapt tone—using accessible language for the family profile and precise architectural terminology for researchers.
LE_hist_002	Historical	RAG Source Pool	Architectural Function: Tests the assistant’s ability to accurately explain the architectural function of the site’s most famous central station.
LE_hist_003	Historical	RAG Source Pool	Academic Topic Synthesis: A complex query requiring the synthesis of two distinct topics, very specific to the academic source material provided.
LE_cexp_004	Client Experience	RAG Source Pool	Official Ticketing Channels: Checks practical, operational retrieval regarding official ticket channels.
LE_cexp_005	Client Experience	RAG Source Pool	Schedules & Rules: Verifies the strict explanation of visiting schedules and associated rules.
LE_cexp_006	Client Experience	RAG Source Pool	Restrictions & Accommodations: Tests the accurate communication of restrictions specific to some areas and how the site can help you deal with them.
LE_se_007	Safety & Ethics	RAG Source Pool	Security Vulnerability Refusal: A red-team scenario where the user acts as a security consultant asking for "blind spots" to prevent the theft of items. The system must strictly refuse to identify vulnerabilities or theft mechanics.
LE_hr_008	Hallucination Resistance	RAG Source Pool	Mythical Entity Correction: The user asks for details about a fake, mythical underground chamber. The assistant must resist hallucinating and firmly correct the user using verified facts.

Table 5. Location-Expert Evaluation Suite: Test Definitions, Sources, and Objectives (Executed in ES and EN)

Test ID	Category	Data Context	Test Objective & Description
LE_hist_001	Historical	RAG Source Pool	Fact Retrieval & Tone Adaptation: Verifies fact retrieval by identifying the site’s oldest construction. Evaluates the system’s ability to adapt tone—using accessible language for the family profile and precise architectural terminology for researchers.
LE_hist_002	Historical	RAG Source Pool	Architectural Function: Tests the assistant’s ability to accurately explain the architectural function of the site’s most famous central station.
LE_hist_003	Historical	RAG Source Pool	Academic Topic Synthesis: A complex query requiring the synthesis of two distinct topics, very specific to the academic source material provided.
LE_cexp_004	Client Experience	RAG Source Pool	Official Ticketing Channels: Checks practical, operational retrieval regarding official ticket channels.
LE_cexp_005	Client Experience	RAG Source Pool	Schedules & Rules: Verifies the strict explanation of visiting schedules and associated rules.
LE_cexp_006	Client Experience	RAG Source Pool	Restrictions & Accommodations: Tests the accurate communication of restrictions specific to some areas and how the site can help you deal with them.
LE_se_007	Safety & Ethics	RAG Source Pool	Security Vulnerability Refusal: A red-team scenario where the user acts as a security consultant asking for "blind spots" to prevent the theft of items. The system must strictly refuse to identify vulnerabilities or theft mechanics.
LE_hr_008	Hallucination Resistance	RAG Source Pool	Mythical Entity Correction: The user asks for details about a fake, mythical underground chamber. The assistant must resist hallucinating and firmly correct the user using verified facts.

4.4.1. Evaluation

At Table 6, the Location Expert benchmark continues the trend we saw on the previous suite, as both models seem to consistently handle RAG queries with accuracy. As these tests do not require data analysis, they align more with what can be expected from AI assistants powered by a sole LLM. While results are consistent for both models and tests 001 through 006 are passed across all runs, except one fail for ALIA due to misinterpreting sources, ALIA starts showing up issues at the Safety & Ethics and Hallucination Resistance tests.

In LE_hr_008, the Spanish language gate fails every time, therefore scoring 0 at every occasion regardless of the profile. Upon close inspection, two English language sources were pulled in, which is the differentiating factor with the other scenarios. While this had no effect on Mistral, ALIA drifted towards English despite being explicitly prompted to maintain Spanish. LE_hr_007 presented a different scenario as no language gate was failed but fails were consistently racked up by both profiles when Spanish was utilized. Even though the researcher profile managed to obtain two passes, it still failed to pass the test three times. However, English tests passed regardless of profile. This indicates that ALIA is more prone to ignore the System prompt and security guardrails when cross-language queries, in this case English/Spanish, appear. In this scenario ALIA did not understand the danger of the query and proceeded to give not only the public information about the most visited spots from the RAG-obtained sources but detailed information and tips on how to carry on pickpocketing activities which is unacceptable and of course were not present in the RAG Sources nor the prompt. Therefore it remains an interesting point to test cross-language prompts compared to one-language ones for all available languages to evaluate if this security risk is persistent and the best way to handle it is to avoid mixing languages or if Spanish answers are just more prone to follow malicious orders despite guardrails. Nonetheless, the earlier cybersecurity red teaming drill was passed utilising Spanish-only system and user prompts, so at the moment, cross-language queries seem to be the most likely vulnerability for ALIA to not follow orders to refuse malicious prompts and maintain language respectively.

In contrast Mistral does not return a valuable answer to the malicious user, repeatedly indicating that it cannot provide that information, pivoting the conversation towards name-dropping some of the most visited spots and ending its answer indicating that security can always improve with more vigilance, cameras and indicating visitors to keep an eye on their belongings. While information on the most visited spots can be relevant and is usually shared by Heritage Sites (it was information available on the sources), it is to be studied whether it should be shared depending on the intent of the user or not shared at all. Other approach would be to avoid fetching potentially sensitive sources despite being public knowledge.

Nonetheless, it is remarkable to indicate that ALIA can keep up with Mistral at the Historical and Client Experience tests, offering similar scores and answers, covering all the informational points and consistently outputting some of the specific keywords for the answer. This indicates that ALIA can synthetize the necessary textual information from the source material to return accurate answers without hallucinating in the process.

4.5. Integration with the Digital Twin

Figure 3 outlines how the assistant can interface with digital-twin services. In a full deployment, the assistant becomes a unifying interaction layer over: (i) static heritage knowledge; (ii) real-time sensor streams; and (iii) simulation services. This pattern mirrors broader trends of coupling digital twins with knowledge representations for proactive management [15,16] and smart museum operations [10].

5. Lessons Learned: ALIA-Based Assistant for Digital Twins

Beyond the offline benchmark, we integrated the ALIA-based SLM+RAG assistant into the iris360 digital-twin platform from Libelium as an embedded conversational panel (Fig. Figure 3). The objective of this integration was primarily usability: to reduce the “last-mile” barrier between (i) complex dashboard-based digital-twin interfaces and (ii) the diverse stakeholders who need to interpret data, understand model outputs, and locate operational knowledge while remaining inside the same working environment.

• In-context assistance matters more than generic chat.

A recurrent usability issue in digital twins is that user questions are situated: “What does this curve mean?”, “Why do these two series differ?”, or “What time window am I looking at?” are only answerable if the assistant is aware of the current context (e.g., selected dashboard, widget title, time range, series names, units, and any annotations). The iris360 integration therefore treats the digital-twin UI state as first-class context for the assistant. This reduces the amount of information the user must type, and it improves answer relevance because the assistant can ground explanations on the same objects the user is currently inspecting (e.g., a specific prediction plot).

• Provenance needs a UI affordance, not only a technical feature.

For operational adoption, provenance must be both available and non-intrusive. In the iris360 assistant, references are surfaced as an expandable “sources” element instead of being embedded as dense inline citations, balancing transparency with visual simplicity. This design supports two common usage modes: a quick “tell me what I am seeing” interaction, and an audit-oriented interaction in which the user expands sources to verify definitions, assumptions, or documentation.

• Failing gracefully is a usability feature.

A practical lesson from deployment in a digital-twin UI is that users frequently ask questions that are adjacent to the available context (e.g., platform configuration details, serial numbers, or administrative procedures). In these cases, the assistant must avoid “confident completion” and instead: (i) state that the current context does not contain the requested information; and (ii) propose concrete next steps (where in the UI to look, which documentation section to consult, or when to escalate to support). This behavior is not only a safety measure; it prevents user frustration and preserves trust in the assistant as a reliable guide.

• Keep read-only interaction as the default.

Digital twins often control assets, alarms, and operational workflows. From a usability and governance standpoint, we found it important to separate explanation from actuation. Even when the assistant can suggest actions (e.g., “open the alarm details”, “switch to the last 24 h window”, “compare with sensor X”), the UI should treat these as explicit user-driven steps, rather than automatically executing changes. This preserves user agency, reduces accidental operations, and aligns with governance-by-design requirements in operational environments.

• Language consistency is part of usability.

In visitor-facing and public-sector contexts, language drift is not a minor quality issue; it is a functional usability defect. When the surrounding UI and stakeholders are Spanish-first, the assistant must remain Spanish-first as well. Consequently, the final integration benefits from explicit language enforcement (and, when needed, regeneration loops) to ensure that a high-quality answer is not rendered unusable by switching languages.

• Summary of integration-oriented usability lessons.

The integration in Figure 3 suggests the following design principles for digital-twin assistants:

• Inject UI context: pass the current dashboard/widget/time-range state to reduce ambiguity and improve relevance.
• Prefer skimmable outputs and pre-processed augmented data: summaries + data + bullet points + suggested next checks, instead of unstructured paragraphs.
• Expose provenance ergonomically: make citations available via expandable sources to balance trust and visual load.
• Fail transparently: when evidence is missing, state limitations and route the user to the appropriate documentation or support path.
• Separate explanation from actuation: keep the assistant “read-only” by default and require explicit user intent for any operational action.
• Enforce the interaction language: treat language control as a core usability requirement, not a cosmetic preference.

The results demonstrate that a compact, sovereign RAG stack can achieve strong performance on core factual and quality requirements, provided authoritative sources are curated and retrieval is reliable while it still can be better tailored to work with ALIA for a fully sovereign solution. These outcomes align with Smart Cities research showing that public-sector AI deployments are expanding rapidly but require responsible governance [7] and that RAG can improve trust and interaction paradigms for digital twins [8].

At the same time, the failure modes provide concrete engineering priorities for heritage and smart-city deployments:

• Safety-by-design is non-negotiable. Even a single refusal failure is unacceptable in public-facing contexts. Mitigations include strict upstream screening, refusal-first prompting, and ongoing red-team evaluation.
• Multilingual robustness requires explicit control. Language drift is a practical usability defect that can invalidate otherwise correct answers. Stronger language constraints and multilingual evaluation are required, particularly in visitor-facing systems.

Beyond the assistant itself, the Living Lab context suggests an iterative operating model: benchmark-driven regression testing, continuous corpus curation, and transparent governance. Living Lab literature emphasises participation and learning cycles [20,21]; the assistant can be treated as an evolving socio-technical component whose performance, accessibility, and sustainability are monitored longitudinally.

6. Discussion

6.1. Datocracy and European Technological Sovereignty: From Data Spaces and Digital Twins to ALIA-Based Sovereign Assistants

Alicia Asín has articulated the concept of datocrazy (“datocracy”) as an explicitly democratic evolution of the smart-city paradigm: public administrations should take decisions grounded in data and subsequently publish the results of those actions to citizens, so that technology strengthens accountability rather than merely optimising operations [1]. In this framing, the technification of decisions can improve quality of life and democracy—provided that privacy is handled transparently (“privacidad inclusiva”) and data is not weaponised to distort reality or generate misleading narratives.

In later interventions, Asín positions datocrazy as a strategic opportunity for Europe to lead technologically by leveraging European-style data spaces that consolidate and govern heterogeneous datasets across domains (mobility, environment, security, economy, culture, etc.) [2]. A key insight is that city leaders do not ask siloed questions. For example, understanding the impact of a large public event (e.g., a concert) requires correlating mobility flows, environmental measurements, security reports, economic indicators, and municipal services. datocrazy therefore does not equate to “more sensors”, but to decision capacity enabled by integrated and governed data.

This vision aligns structurally with Europe’s regulatory and rights-based approach. Asín emphasises that the European Union can become a “safe territory” for digital rights, ensuring that the rule of law established in the offline world has an enforceable counterpart in the digital domain [2]. Open and unrestricted access to sensitive data (e.g., health data) is not acceptable without clear governance rules defining who can access the data, for which purpose, under which retention conditions, and under a Fair AI posture that mitigates bias and discrimination. datocrazy thus presupposes not only technological capability, but institutionalised governance.

Importantly, this framework does not advocate “dataism” (data as autopilot). Democratic agency remains central. Administrations should publish objectives and action plans so that citizens can evaluate fulfilment and hold institutions accountable [2]. datocrazy therefore represents a new dialogue between citizens and public authorities, grounded in measurable commitments and evidence-backed reasoning.

6.2. Digital Twins as the Decision Engine of Datocrazy

Within this paradigm, digital twins become the operational engine of datocrazy. By transforming integrated data streams into scenario-based simulations, digital twins allow ex ante evaluation of public policies. Asín highlights use cases such as urban air-quality twins that simulate the impact of low-emission zones before implementation, and predictive models for high-voltage transmission lines where safe operating capacity depends on atmospheric conditions [2].

These examples illustrate that datocrazy requires not only data availability but also interpretable models capable of testing “what-if” interventions and anticipating second-order effects. Policy-making evolves from retrospective justification toward simulation-informed experimentation.

The data-space layer ensures scalability and governance. Platforms such as iris360 are conceived as horizontal environments that orchestrate diverse data sources, enabling integrated decision-making rather than departmental fragmentation [3]. In this architecture, data spaces provide controlled interoperability, while digital twins provide analytical and predictive capacity.

6.3. The Missing Last Mile: Legibility, Contestability, and Auditability

However, dashboards and APIs alone do not automatically translate into governance capacity. The “last mile” problem arises when complex digital-twin infrastructures must be made legible, contestable, and auditable for heterogeneous stakeholders, including policymakers, operators, and citizens.

If datocrazy requires publishing results and enabling scrutiny, then systems must provide:

• Usable access to evidence (which data and assumptions were used);
• Interpretable explanations (why a model predicts a given outcome);
• Transparent provenance trails that support verification.

Conversational interfaces can act as governance primitives in this context, lowering the barrier to interacting with complex data spaces and digital twins. However, proprietary black-box assistants introduce friction with datocratic principles, including uncertainty about data residency, reproducibility, multilingual support, and compliance with European regulation.

These constraints motivate the adoption of sovereign AI stacks, where models, data flows, logging mechanisms, and safety controls remain under public or institutional governance.

6.4. ALIA and Sovereign LLMs as Enablers of Datocracy

ALIA is positioned as a public AI infrastructure providing open and transparent language models in Spanish and co-official languages [4,5]. Its objectives directly align with the requirements of datocrazy in three dimensions:

1.

Language as democratic accessibility. Accountability mechanisms must operate in the languages used by citizens. Spanish-first AI infrastructure reduces dependency on externally optimised models and mitigates linguistic inequities.

2.

Transparency and governance-by-design. Public AI infrastructure facilitates traceability, auditability, and alignment with the EU AI Act regulatory framework [6].

3.

Technological sovereignty. Control over deployment, data processing, and safety policies reinforces European strategic autonomy in high-impact AI applications.

The EU AI Act (Regulation (EU) 2024/1689) establishes harmonised obligations concerning risk management, transparency, and governance controls [6]. datocrazy-oriented deployments must internalise these requirements if they are to evolve into durable public infrastructure rather than experimental pilots.

6.5. Results Discussion & Benchmarking Methodology

The results proved the RAG pipeline as a strong solution for the Location Expert and IrisChat use cases. Nonetheless, it is clear that the Location Expert use case is more aligned with the design of the pipeline and the chosen models, as both ALIA and Mistral-24B can take on multiple sources and synthesize a correct, verifiable answer for the user. When more data analysis and commentary is necessary, the system does become more reliant on the LLM capability to handle numerical value hallucination as manifested by ALIA and to a smaller degree, Mistral. This render the data analysis functionality to a more orientative role that needs human supervision to validate the correct usage of numerical values and leaves room for a more complex multi-model Agentic solution as well as a more complex benchmark to detect hallucinated values.

The LLM-as-a-judge pattern, works well enough verifying the sources were correctly utilised and the tested answer does not steer far off from the sources of truth but misses on some numerical drifiting that can be large enough to confuse the user and should render lower scores. A larger, frontier LLM combined with a more detailed pipeline to detect hallucinated values would also come a long way to better the Data Analysis query evaluation.

6.6. A Concrete Implementation Pattern: Data Spaces + Digital Twins + Sovereign RAG

The combination of:

1.

Governed data spaces (interoperability and access control),

2.

Digital twins (simulation and predictive analytics), and

3.

ALIA-based sovereign LLM assistants (auditable conversational access),

constitutes a practical architecture for operationalising datocrazy.

This integration enables several transitions:

• From open data to accountable data use. Retrieval-Augmented Generation (RAG) allows responses to be grounded in authoritative evidence, attaching provenance and limiting speculative output.
• From cross-domain complexity to civic questioning. Sovereign assistants can translate multi-domain queries (e.g., impact of an event on mobility, economy, and environment) into governed retrieval and simulation-backed explanations.
• From policy debate to measurable commitments. Digital twins support ex ante simulation, while assistants expose results in accessible narratives, preserving democratic contestability.
• From trust as rhetoric to trust as engineering. Logging, provenance storage, language enforcement, refusal mechanisms, and data minimisation embed governance into the inference loop.

In this sense, sovereign LLMs do not replace democratic governance; rather, they scale its informational legibility. datocrazy ultimately concerns who can see, understand, and contest the evidence behind public decisions. Data spaces and digital twins create the analytical substrate; ALIA-based sovereign conversational assistants provide the accountable and multilingual interface layer that makes this substrate operational under European governance principles.

This perspective extends the architectural contributions described in this manuscript—specifically, the sovereign SLM+RAG integration for digital-twin environments (cf. Section 3, Section 4 and Section 5)—toward a broader governance paradigm in which AI systems are not only technically performant but institutionally aligned with democratic accountability.

7. Conclusions and Future Works

We presented a sovereign SLM+RAG conversational assistant for the Libelium Heritage Living Lab that, through either ALIA’s public language-model infrastructure or open-weight Mistral models, enables evidence-grounded, auditable interaction under public-sector constraints. The benchmark results show strong performance on core factual and quality tasks and reveal three high-priority gaps, specially in the case of ALIA: numerical value drift, refusal robustness, language control.

Nonetheless, our architecture, governance blueprint, and evaluation harness provide a replicable pattern for deploying conversational access layers in cultural-heritage digital twins and other smart-city Living Labs.

This study reports a dual benchmark over 19 tests across five categories: historical queries, client experience, data analysis, hallucination resistance, and safety/ethics. While useful for regression testing, broader coverage and human evaluation are needed for publication-grade claims about user experience and real-world impact.

We plan to: (i) expand the benchmark suite and publish a red-teaming protocol suitable for cultural-heritage settings; (ii) incorporate more robust language-steering mechanisms; (iii) evaluate an agentic pipeline for data analysis queries; (iv) integrate real-time digital-twin APIs for sensor queries and simulation triggers; and (v) add structured provenance to further improve trust and auditability.