A Layout-Aware Hybrid Retrieval-Augmented Generation Framework for Mitigating Hallucination in Turkish Texts: Evidence from the Turkish Official Gazette

1. Introduction

The Transformer architecture [1] let large language models (LLMs) move beyond frequency-based text statistics toward representations of long-range semantic dependencies, and models with tens or hundreds of billions of parameters now perform summarization, translation, and question answering at a level often indistinguishable from human writing [2,3]. This fluency is accompanied by hallucination: the generation of content that is well-formed but factually unsupported, either contradicting a supplied source (intrinsic) or introducing unverifiable claims absent from it (extrinsic) [3,5,6]. Because LLMs are trained to predict the most probable continuation rather than a verified truth, they favor memorized (and sometimes stale or incorrect) parametric knowledge over the evidence actually relevant to a query [3].

This failure mode is especially consequential in zero-error-tolerance domains such as law, where a fabricated statute number, misquoted clause, or invented precedent can translate directly into lost rights, financial penalties, or erosion of trust in judicial institutions [7]. Legal text further stresses general-purpose NLP through document length, dense archaic terminology, and deeply nested article-paragraph-subparagraph hierarchies [8]; when deployed directly, LLMs can fabricate entire precedents or statutory provisions in convincing formal legal language, a risk compounded by parametric memory that cannot reflect post-training legislative amendments [9].

Retrieval-Augmented Generation (RAG) addresses this by coupling a model’s static memory to a dynamic, non-parametric knowledge base, so that generated claims are grounded in retrievable evidence rather than recalled from memory [9,10]. A canonical pipeline proceeds in three stages: source documents are split into retrievable chunks and embedded into a dense vector space; an incoming query is embedded into the same space and the top-k most similar chunks are retrieved by cosine similarity; and those chunks are supplied to the generator, which is instructed to answer using only that evidence [10]. This shift, from recalling what a model was trained on to reasoning over what it is shown, also means that updating a knowledge base (e.g., to reflect a new regulation) requires no retraining of the underlying model, which is a substantial advantage over scaling parameters alone as a strategy for keeping models current [9,11]. Plain dense-vector RAG, however, can still under-perform on richly structured documents: similarity search alone may ignore a legal text’s hierarchical organization and pull in disjointed context, reintroducing partial hallucination even inside a nominally grounded pipeline [12]. This has motivated more autonomous designs such as Self-RAG, which learns when retrieval is needed and critiques retrieved evidence with dedicated tokens [13]; Interleaved-Retrieval Chain-of-Thought (IRCoT), which alternates multi-step reasoning with retrieval calls [14]; and Layout-Aware RAG, which extends vector search with document-layout information (headings, tables, reading order) to preserve hierarchical structure that plain chunking destroys [12].

Most advanced RAG research, however, targets English corpora and well-resourced computing environments. Legal NLP for Turkish, an agglutinative, morphologically rich language with dense, Ottoman-derived terminology, remains comparatively underdeveloped, and even basic subtasks such as named-entity recognition require dedicated morphological modeling (e.g., subword approaches such as Morph2Vec) [15,16]. The limited work on Turkish legal documents shows that authoritative sources such as Constitutional Court rulings, statutory articles, and above all the Official Gazette (Resmî Gazete) require bespoke annotation and dataset hierarchies even for machine-learning purposes, to the point that the Official Gazette has been formalized as its own named-entity category (OFF) in the literature [16]. Because Turkish statutes, presidential decrees, and appointments attain legal force only upon Official Gazette publication, the factual correctness of any Turkish legal AI assistant reduces, in practice, to its fidelity to the current, unaltered Official Gazette text. Without that fidelity, a genuine statutory reference cannot be distinguished from one fabricated from a model’s parametric memory [3]. All LLMs risk this failure mode when relied upon without external verification, but the risk is sharper for Turkish: an agglutinative morphology and dense legal jargon make it easier for a generic, non-domain-adapted model to drift into semantically plausible but legally unfounded output than in a well-resourced language with mature legal NLP tooling.

This gap defines our problem: existing RAG approaches to hallucination mitigation are concentrated on English text and do not target the morphological complexity of Turkish legal language or the structural hierarchy of the Official Gazette, nor do they systematically measure factual faithfulness in this setting. We address it with a Turkish-specific, layout-aware hybrid RAG architecture that (i) preserves the article-paragraph-subparagraph hierarchy through structural parsing (IBM Docling), (ii) retrieves relevant context through hybrid dense-lexical indexing, and (iii) suppresses fabricated answers through a citation-constrained prompting protocol, yielding a question-answering assistant whose every claim is grounded in the Official Gazette and measurable along faithfulness and citation accuracy. To our knowledge, no prior study addresses generative Turkish question answering and hallucination mitigation jointly at this scope. The significance of this gap is not merely linguistic: because Turkish statutes, decrees, and appointments are legally binding only upon Official Gazette publication, a Turkish legal assistant is, in effect, only as trustworthy as its fidelity to that single, authoritative source, which makes hallucination mitigation in this specific setting a more tractable and more consequential target than hallucination mitigation over general Turkish text. This study’s contribution is therefore not a new retrieval algorithm in the abstract, but the demonstration that combining an existing document-AI tool (Docling), a simple hybrid scoring rule, and a citation-constrained prompt is sufficient to bring hallucination in this setting into a practically useful range without fine-tuning either the embedding or the generation model on Turkish text.

The study asks three research questions: (RQ1) What effect does the proposed architecture have on hallucination relative to a zero-shot LLM baseline? (RQ2) What effect does layout-aware retrieval have on citation accuracy and grounding transparency, given the Official Gazette’s structural complexity? (RQ3) Are the resulting answers reliable enough to support legal research, per automatic RAG metrics and an independent legal-expert review?

The study is scoped to legislation published in 30 Official Gazette issues (July 2025), evaluated on 33 original question-answer pairs spanning factual, analytical, and refusal (edge-case) scenarios. Retrieval and indexing ran locally (Apple M3 Pro, MPS) to preserve data sovereignty, while response synthesis used the cloud-hosted claude-sonnet-5. As detailed in Section 5.1, the scale of the evaluation set, the single-period corpus, and the use of one independent reviewer bound the generalizability of the results, which should be read as indicative. The remainder of the paper reviews related work, details the system and evaluation protocol, reports results, discusses findings and limitations, and concludes with future directions.

2. Related Work

Hallucination, fluent but unsupported or fabricated generation, is attributed to the probabilistic nature of next-token prediction: when a model’s knowledge boundary is reached, it favors the most probable continuation over a verified fact [3]. The literature distinguishes intrinsic hallucination, in which generated content directly contradicts a supplied source (e.g., a source stating a ruling was upheld in 2019 while the model reports it was overturned in 2021), from extrinsic hallucination, in which the model introduces unverifiable new content absent from the source altogether (e.g., citing a specific, entirely fabricated statute article to support a claim) [3,6]. Recent surveys add logical and stylistic sub-types, such as self-contradiction across reasoning steps and deviation from a requested format or register, and trace root causes to noisy pre-training data, inference-time exposure bias, and knowledge that is simply stale relative to a model’s training cutoff [3]. In zero-error-tolerance domains the danger is compounded because fabricated statutes or precedents are delivered in convincing, formally correct legal language that a non-expert reader cannot easily distinguish from a genuine citation.

RAG mitigates this by grounding generation in an external, verifiable, updatable knowledge base [9,10]. Beyond the canonical retrieve-then-generate pipeline, Self-RAG [13] lets a model decide dynamically whether to retrieve and critiques retrieved evidence with relevance/support tokens; IRCoT [14] interleaves chain-of-thought reasoning with retrieval for multi-hop legal inference; REALM/RETRO [11] pre-train generation with retrieval so that updating knowledge requires only re-indexing; and Layout-Aware RAG builds structure-preserving representations from page layout to avoid losing statutory hierarchy [12]. This study adopts that approach via IBM Docling, because incorrect reading order and structural noise otherwise fragment legal context in official, table-dense, multi-column documents [18]. RAG pipelines are increasingly evaluated with RAGAS [17], a reference-free LLM-judge framework scoring groundedness, answer relevance, and context precision without human annotation.

Within legal NLP, LegalBench standardizes evaluation of legal reasoning across 162 practitioner-designed tasks spanning issue-spotting, rule-recall, rule-application, and conclusion categories [8], while LegalBench-RAG specifically benchmarks retrieval precision, rather than generation, over large legal corpora such as PrivacyQA, ContractNLI, MAUD, and CUAD, jointly totaling nearly 80 million characters and almost 7,000 queries [8,21]. This scale illustrates that legal RAG must retrieve a specific clause from within an enormous, heterogeneous document pool, not merely produce fluent text. Among applied hybrid systems, TaxFlow [22] combines LLaMA-3 with dense-sparse hybrid retrieval over Indian tax statutes, government gazettes, and case law, reporting 96.5% accuracy and 95.0% F1 on a curated benchmark, evidence that hybrid retrieval materially improves grounding over dense retrieval alone, though at the cost of running a 70-billion-parameter model entirely on cloud infrastructure. Yet one of the most rigorous independent audits of deployed legal-AI tools found that leading commercial systems claiming RAG grounding (Lexis+ AI, Westlaw AI) still hallucinate in roughly 17-33% of queries [7], a band we use to position our results, and which indicates that RAG integration reduces but does not eliminate hallucination even in well-resourced, commercially deployed systems.

Turkish legal NLP remains comparatively underdeveloped relative to high-resource languages, hindered by the scarcity of annotated Turkish legal datasets and the contextual density of Ottoman-derived legal terminology, and is concentrated on discriminative tasks (Table 1). Akça et al. [15] found fine-tuned TR-BERT best-balanced on Court of Cassation offense-type classification (95% accuracy, F1 = 0.87); Mumcuoğlu et al. [23] found attention-augmented LSTM/BiLSTM consistently ahead of SVM for higher-court outcome prediction (up to 93.2% accuracy). Most consequential here, Çetindağ et al. [16] built a dedicated Turkish legal NER scheme adding Court, Legislation, Reference, and, critically, Official Gazette (OFF) entity classes, recognizing OFF at 96.34% F1 under a morphology-sensitive hybrid embedding scheme. That the Official Gazette carries recognizable structural regularity even at entity-extraction level motivates both our regex-based metadata extraction (detailed in Section 3.2) and, more broadly, treating it as a tractable knowledge base for hallucination mitigation. No existing study, however, targets free-form Turkish legal generation or measures citation accuracy within a purpose-built RAG architecture. That is the gap this study addresses.

3. Materials and Methods

This study follows a quantitative, experimental design. Standard RAG treats external sources as undifferentiated text [10], yet multi-page legal documents such as Official Gazette issues carry headings, tables, and strict article-paragraph-subparagraph hierarchies that conventional OCR routinely destroys, introducing semantic discontinuities [18]. We therefore implement a four-layer architecture (Figure 1): (1) an OCR layer that binarizes and digitizes scanned or digital-native pages; (2) a layout-aware segmentation layer (IBM Docling) that removes two-column reading-order artifacts and encodes the article-paragraph-subparagraph hierarchy in Markdown; (3) a hybrid indexing layer storing dense embeddings alongside a lexical index; and (4) a coherence-based synthesis layer that supplies verified context to the generator under strict citation instructions. Each layer maps onto a specific failure mode identified above: Layer 1 addresses the fact that a large share of Official Gazette issues are scanned rather than digital-native; Layer 2 addresses the loss of structural hierarchy that plain-text extraction causes on multi-column, table-dense official documents; Layer 3 addresses the observation that dense embeddings alone under-discriminate the exact numeric identifiers (statute and article numbers) legal queries often depend on; and Layer 4 addresses hallucination directly, by constraining generation to cite only supplied evidence. The following subsections detail each layer’s implementation and the evaluation protocol used to test the resulting system.

3.1. Dataset and Experimental Environment

The dataset comprises 30 Official Gazette issues from July 2025, covering laws, presidential decrees, and regulations, the highest-authority categories of Turkish legislation. Raw PDFs were parsed by the Docling/EasyOCR pipeline (Section 3.2) into structured Markdown and indexed. A 33-question evaluation set was constructed spanning temporal-currency, cross-legislation, and refusal (no-answer-in-corpus) scenarios (categorized in Section 4.1). Document parsing, embedding (multilingual-e5-large), and retrieval ran locally on an Apple M3 Pro (18 GB RAM, MPS), while response synthesis used claude-sonnet-5 via the Anthropic API, keeping retrieval under local data sovereignty while drawing on cloud-scale inference for generation. The pipeline is built on an open-source stack (FastAPI, SQLAlchemy/psycopg2/pgvector, sentence-transformers and PyTorch, EasyOCR, Docling and PyMuPDF, and RAGAS for cross-validation), chosen for reproducibility and community support, with the full dependency list available from the corresponding author on request.

3.2. Layout-Aware Structural Parsing

Because a substantial share of Official Gazette issues are scanned or image-based PDFs, digitization uses EasyOCR (CRAFT-based text-region detection, CRNN-based recognition with Turkish-language support), which produces stable results across varying print and scan quality and passes recovered text blocks and pixel coordinates to layout analysis. Conventional flat-text extraction assumes single-column reading order and cannot detect the Official Gazette’s two-column gutter, concatenating unrelated lines and introducing semantic breaks [18]. We instead use IBM Docling, which processes each page on two planes: geometric segmentation, localizing elements via bounding boxes and reconstructing linear reading order from column gaps, and semantic classification, assigning each element to a functional category (heading, paragraph, table, footnote), together reconstructing the document’s hierarchical structure rather than treating it as a character stream. Three operations follow from this: a coordinate-based noise filter removes page numbers, seals, and borders from the top/bottom 5% of each page before they can fragment adjacent legal propositions; Docling’s table-structure-recognition model decomposes tables into a cell matrix and renders them as Markdown grid tables, preserving the row-column relations that flat extraction destroys and that are otherwise common in staffing schedules and fee tariffs attached to legislation; and regex patterns extract the Official Gazette date, issue number, statute/decree number, and article index, attaching each as a metadata token to its text segment so that retrieval can later be filtered by temporal or hierarchical criteria, not only by vector similarity. Testing indicated this structural cleanup increases the semantic purity of indexed content by roughly 40% relative to raw OCR output, primarily by separating metadata and page artifacts from enacting text and by keeping tabular data in a machine-interpretable matrix form (quantified further in Section 4.3).

3.3. Chunking and Hybrid Indexing

Fixed-length chunking was excluded because it splits statutory provisions at arbitrary boundaries, causing context discontinuity rather than mitigating it [24]. We instead apply Semantic Structural Chunking over the Markdown output, splitting along Article/Paragraph/Subparagraph boundaries with a maximum chunk size of 1,024 tokens (tiktoken cl100k_base) and a 200-token overlap, values set by developmental observation rather than a systematic sweep [25,26]; each chunk d_i carries a metadata record {date, issue number, legislation name, article number, page number} and is embedded with the locally executed multilingual-e5-large model [27] into a 1,024-dimensional space, E(d) = f_embed(d) ∈ R^1024. Two indexes are built over these vectors: a dense layer (PostgreSQL + pgvector, HNSW index, default parameters m=16/ef_construction=64 [28]) ranks candidates by cosine similarity, sim(q,d) = [E(q)·E(d)] / (‖E(q)‖‖E(d)‖), for conceptual similarity, and a lexical layer (PostgreSQL ILIKE) captures exact statute/article-number matches [29], since such numeric identifiers carry high discriminative power in legal text [16] but are not reliably separated by dense embeddings alone (Section 4.4).

3.4. Retrieval and Response Synthesis

At query time, numeric tokens (≥3 digits) and content words (≥4 characters) are matched via ILIKE to yield up to 60 lexical candidates (run under both NFC/NFD normalization for Turkish diacritics), while the query embedding retrieves the 30 nearest chunks via HNSW with no fixed similarity threshold. This hybrid design is motivated by the observation that dense-embedding similarity for statute/article numbers clusters tightly (0.82-0.86) and so does not reliably discriminate between them (Section 4.4). Candidates are merged and scored (+50 per numeric match, +40 if more than two numeric matches co-occur, +8 per keyword match, 0-30 by semantic rank, weights set by developmental observation rather than a systematic sweep), and the top-8 chunks are passed to claude-sonnet-5 under a system prompt that reads, in part: “You are an expert consultant in Turkish law. [DOCUMENTS]: (context) [QUESTION]: (query). INSTRUCTIONS: 1. Derive the answer SOLELY from the documents above. 2. Read the relevant articles carefully; the answer may be distributed across multiple chunks, so combine them. 3. State statute/article numbers and any inserted/deleted clauses explicitly. 4. If the information is genuinely absent, write: ‘No information found in the documents.’ 5. Answer briefly and precisely; do not add speculation or outside knowledge.” Temperature is omitted, as the endpoint rejects it. Algorithm 1 formalizes the runtime flow. A small, locally executable generator was found to produce occasional unsupported claims in single-pass generation; synthesis was therefore migrated to the higher-capacity, cloud-hosted claude-sonnet-5 (one inference call per query, no ensembling, 2,048-token output cap, ≈20,000-character context limit, 60 s timeout), while retrieval remained unchanged and local; access required only swapping the endpoint and model identifier behind the existing OpenAI-compatible client interface, leaving the retrieval code untouched. Citation accuracy is audited post hoc by a separate script rather than at runtime, and this audit procedure is identical across the ablation configurations compared in Section 4.5.

Algorithm 1. Runtime Query Processing and Response Synthesis

Input: user query q; Output: grounded answer A, or a refusal message 1. E(q) ← e5_large.encode(q) 2. numbers, words ← extract_key_terms(q) 3. candidates_kw ← ILIKE_search(numbers, words, limit=60) 4. candidates_sem ← pgvector.HNSW_search(E(q), limit=30) 5. pool ← candidates_kw ∪ candidates_sem 6. for each d in pool: score(d) ← 50·num_match(d) + 40·[≥2 num] + 8·word_match(d) + rank_bonus(d) 7. context ← top-8(pool, score) 8. if context == ∅: return “No relevant information found in the provided documents.” 9. prompt ← prompt_template(q, context) 10. A ← claude_sonnet_5.generate(prompt, max_tokens=2048) 11. return A

The query is first encoded by the embedding model; an HNSW nearest-neighbor search over pgvector and an ILIKE lexical search then run in parallel to produce candidate chunks, which are merged, scored, and filtered into a context pool before a structured prompt triggers a single claude-sonnet-5 call. This flow is identical across the ablation configurations compared in Section 4.5 except for the retrieval step itself (semantic-only for B0, hybrid for B1), so that differences in outcome can be attributed to retrieval rather than to changes in prompting or generation.

3.5. Evaluation Design

Automatic evaluation uses a bespoke LLM-as-a-judge framework (judge: claude-sonnet-5), cross-validated on a subset (n=8) with RAGAS 0.4.3 [17] (0.007-point difference on faithfulness). Faithfulness (the fraction of answer propositions supported by retrieved context, computed as the number of supported propositions divided by the total number of propositions) directly indicates hallucination rate, reported as HR = (fabricated responses / total responses) × 100; answer relevance measures semantic overlap between the answer and the original query via synthetic back-generated questions; retrieval success is measured via Hit-Rate@k, Recall@k, and Mean Reciprocal Rank (MRR = (1/|Q|)Σ 1/rank_i) against labeled gold chunks; and Citation Accuracy audits whether stated references match true source coordinates. An independent legal expert, blinded to which configuration produced each answer, additionally scored all 33 responses on factual correctness, legal adequacy, citation validity, material omission, and misdirection risk; with a single rater, inter-rater agreement is not reported. To prevent tuning leakage, the 33 questions were split (seed=42) into an 8-question development set, used only to set chunk size, overlap, Top-K, and score weights, and a 25-question test set used for final evaluation and for a three-way component ablation (B0: flat extraction + fixed-size chunking + semantic-only retrieval; B0+Docling: layout-aware parsing added; B1: full system with hierarchical chunking and hybrid retrieval), with Wilson 95% confidence intervals computed for all proportions given the small sample.

4. Results

4.1. Evaluation Set and Overall Performance

The 33 questions comprise three deliberately unequal groups designed to stress different capabilities rather than only easy retrieval: 16 factual-retrieval questions testing vector-search precision, metadata matching, and multi-chunk extraction; 8 analysis questions testing hierarchical chunking, context preservation, and multi-step synthesis; and 9 edge-case/refusal questions, i.e., queries with no counterpart in the corpus or logically impossible ones such as an invalid date, testing similarity thresholding and hallucination resistance. This balanced design is intended to make the reported metrics more representative of real-world reliability than a set skewed toward easy retrieval would be. Table 2 reports overall performance; because these rates are computed only over the 17/33 questions the system chose to answer (coverage 51.5%), they must be read together with coverage and, given the small sample, with wide confidence intervals (e.g., the Wilson 95% CI for 81.8% citation accuracy at n=11 is approximately 52-95%).

Category breakdown (Table 3) shows uniformly high groundedness (0.951-1.000) but response rates that vary sharply by category: factual 12/16, analysis 3/8, edge case 2/9. The low edge-case response rate is by design: these questions have no counterpart in the corpus (e.g., an impossible date such as “32 July 2025”) or fall outside its scope, so a low answer rate there reflects correct refusal rather than failure, consistent with the 1.000 groundedness recorded for the few edge-case answers the system did produce. The low analysis-category answer relevance (0.550) instead reflects a structural limit rather than a retrieval failure: questions requiring aggregation across the whole corpus (e.g., listing every province mentioned across many issues) cannot be fully satisfied by a top-k retrieval architecture that surfaces only the most similar individual chunks, even though the chunks it does retrieve remain highly grounded. Taken together, this pattern indicates the system is strongest on single-document factual retrieval and weakest on corpus-wide aggregation, a distinction with direct implications for how such a system should be scoped in deployment. Table 4 compares this study with the Turkish legal NLP literature; the F1/accuracy figures span different task types and are not directly comparable, but the table situates this study as the first to jointly address generative Turkish QA and hallucination mitigation, converting the OFF-entity recognizability reported by Çetindağ et al. [16] into a practical retrieval layer that reaches MRR = 0.629 and HR@8 = 0.786 on this evaluation set, and achieving a lower hallucination rate than TaxFlow’s cloud-only, 70B-parameter architecture achieves in F1 terms on a different task and language.

4.2. Zero-Shot Baseline and Qualitative Cases

The same 33 questions posed to claude-sonnet-5 without retrieved context yielded a context-grounded, verifiable answer for only 1/33 (Table 5). Here, “answered” requires a verifiable, context-groundable response; fluent text produced with no external support (e.g., the fabricated statute below) does not count as an answer under this definition, since it cannot be distinguished from confabulation. Aside from that one case, the zero-shot model either failed to produce the relevant information or produced unverifiable content, for example by citing an outdated or entirely invented name when asked about a July 2025 appointment decision (a case blocked by its training cutoff, where it defaulted to a plausible but incorrect prior appointee) or affirming a nonexistent “Space Mining Law” when asked whether one appeared in a specific Gazette issue. Table 6 contrasts both cases directly: the RAG system correctly retrieved the relevant issue and named the true appointee with full citation (measured F1 against the source text was 92% for this query, the residual gap attributable to connective words added when converting the retrieved clause into natural prose rather than to factual error), and correctly refused the fabricated-statute query rather than confabulating an answer. In an analysis-category test on expropriation decisions spanning multiple provinces, Docling-based hierarchical chunking also correctly listed only the provinces actually named in context, without adding unlisted ones. This is evidence that structure preservation, not just retrieval volume, drives factual precision on tasks that require enumerating specific entities rather than summarizing a single passage.

4.3. Embedding Choice, Efficiency, and Preprocessing Impact

Because the embedding model directly determines retrieval precision, three cloud candidates were benchmarked on latency, dimensionality, and semantic hit rate before the final local model was chosen (Table 7). text-embedding-3-small offered the best latency/accuracy balance; text-embedding-3-large gave the highest hit rate but at roughly 3x the latency for a marginal 2.1-point gain, a poor trade-off for a real-time assistant. All three candidates, however, require sending Official Gazette content to a third-party API; given the priority of data privacy and independence from external quotas at the retrieval layer, the fully local multilingual-e5-large (1,024-d) was chosen instead and retained even after generation moved to the cloud, so that only already-filtered context, not the underlying corpus, ever leaves local infrastructure.

Relative to conventional flat OCR, Docling-based structural parsing raised indexed-content semantic purity by roughly 40% (Table 8), primarily because repeated headers, seals, and page numbers are no longer folded into the text a query can match against, and because table structure is preserved rather than flattened into meaningless character runs. In practice, a plain-OCR pipeline can register the repeated “Official Gazette” header string as a high-similarity keyword for an unrelated query, inflating false positives; Docling’s separation of metadata from content removes this failure mode and made source-article identification for retrieved chunks 100% accurate on this evaluation set (a document-identification metric distinct from the 81.8% generated-answer citation accuracy reported in Table 2). On operational efficiency, Table 9 contrasts the system with a legal expert’s typical manual archive-search workflow; the roughly 170-fold speed advantage stems mainly from collapsing document-location time (searching a physical or digital archive for the relevant issue) to millisecond-scale vector search, a gain particularly relevant to a source that adds hundreds of new pages daily, though the manual-side figures are simulated rather than drawn from a controlled user study.

4.4. Error Sources and Expert Evaluation

Although overall accuracy is high, some individual queries fell short of target metrics; residual errors traced to three factors. First, OCR-induced character corruption, such as wet-ink signatures, official seals, or scan shadowing causing “ı”/”i”, “0”/”o”, or “B”/”8” confusions, depresses cosine-similarity match scores even when the system’s underlying logic and document access are correct, since the generator inherits these corrupted characters as part of the “source text” it is instructed to quote faithfully. Second, embedding-space “vector collisions” occur between near-identical statutory articles from different years (e.g., two successive years’ import-regime decisions), which cluster closely in the 1,024-dimensional vector space and can, in rare cases, surface an outdated document ahead of the current one. This is an inherent limitation of the embedding space’s capacity to separate fine-grained legal nuance rather than a retrieval-pipeline defect. Third, “lost-in-the-middle” effects arise when a retrieved article spans more than roughly 2,000 tokens (e.g., a single article continuing across two or three pages), causing the generator to attend to the beginning and end of the passage while missing a mid-text exception, so that a response can be correctly sourced and cited yet incomplete. In the independent, blinded expert review (Table 10), 30/33 responses were judged fully correct on factual accuracy and legal adequacy (3/33 partially correct, attributable to the OCR and long-context effects above, none incorrect), citation validity was judged valid for all 33, and no response was flagged high-risk.

4.5. Component Ablation

The three-way ablation on the 25-question test set (Table 11) isolates each component’s contribution. Layout-aware parsing alone (B0+Docling) raised Hit-Rate@8 by about 15 points and cut hallucination from 15.4% to 10.4% relative to the standard-RAG baseline (B0), confirming a causal (not merely correlational) effect, since the two configurations differ only in that respect. Adding hierarchical chunking and hybrid retrieval (B1) contributed a further ≈23-point gain, reaching 100% Hit-Rate@8 (Wilson 95% CI 86.7-100%) and 0% hallucination, a result consistent with the intuition that legal queries often hinge on exact numeric identifiers that dense embeddings alone do not reliably discriminate. Because hyperparameters were fixed on the separate 8-question development set before this test, the observed gains are attributable to architecture rather than tuning bias.

5. Discussion

On RQ1, the proposed architecture reduced hallucination on answered questions to 4.2%, within the band the literature considers acceptable for law and medicine and well below the 17-33% reported for commercial legal-AI tools claiming RAG grounding [7]. On RQ2, layout-aware retrieval delivered high citation accuracy despite the Official Gazette’s structural complexity, though this was measured on a limited, single-period corpus that warrants validation on larger, independent data. On RQ3, RAGAS and classification metrics fell within literature-reported ranges, and, more importantly, the system’s tendency to refuse rather than fabricate on unanswerable questions is a reliability signal supporting use as a human-supervised research aid rather than, yet, a standalone decision-support tool; positioning it as an autonomous decision-support system would require the larger-scale, multi-rater expert validation described in Section 5.1.

Decomposing the pipeline (Section 4) indicates that retrieval succeeds more consistently than generation: the retrieval layer finds the correct document with high precision and the generator interprets it with high groundedness, but most residual hallucination risk originates in synthesis rather than retrieval. This asymmetry matters for where future engineering effort should go: it suggests that further gains are more likely to come from stronger prompting, self-critique, or a larger generator than from additional retrieval tuning, at least on this evaluation set. The embedding comparison (Section 4.3) further shows that the most accurate model is not automatically the right engineering choice once data-sovereignty constraints are considered. This is a deliberate local-retrieval/cloud-generation split that keeps the bulk of sensitive content local while sending only already-filtered context to the cloud at inference time, and this asymmetric design is itself a contribution, since most published legal RAG systems either run entirely in the cloud (e.g., TaxFlow [22]) or entirely on-premises, without examining this intermediate option. The ablation (Section 4.5) supplies direct causal evidence, rather than correlation, that layout-aware parsing and hybrid retrieval contribute complementary, largely independent gains: because B0 and B0+Docling differ only in the presence of structural parsing, the observed jump in Hit-Rate@8 is attributable specifically to preserving column order, table structure, and noise removal, independent of chunking strategy or retrieval hybridity, while the further gain in the full system isolates the value of exact-match retrieval for the numeric statute and article identifiers that dense embeddings alone under-discriminate (Section 4.4). Finally, the qualitative cases (Section 4.2) illustrate that a zero-shot model does not fail silently but actively confabulates plausible-sounding, fluent citations that are difficult for a non-expert reader to distinguish from genuine ones. The practical value of the proposed system lies as much in converting this failure into an explicit, auditable refusal as in improving accuracy on questions it does answer.

These findings also have a practical reading for deployment. The system’s response rate of 17/33 (51.5%) means that, on this evaluation set, roughly half of all queries end in an explicit refusal rather than an answer; for a legal-research aid this is a feature, not a defect, since a false refusal is far less costly than a confident fabrication, but it does mean the system is best framed as a triage and first-pass research tool that narrows a lawyer’s search rather than a tool that answers every question put to it. The category-level pattern in Section 4.1 further suggests a natural deployment boundary: the system should be trusted for single-document factual lookup and citation, where it performs strongly, and used more cautiously, or supplemented with a corpus-wide summarization step, for questions that require aggregating information scattered across many Gazette issues, where top-k retrieval is structurally limited regardless of how well any individual retrieval step performs.

5.1. Limitations

Several limitations qualify these results. The groundedness judge (claude-sonnet-5) shares its family with the generator, a plausible source of evaluator bias only partially offset by independent RAGAS cross-validation (0.007-point difference on faithfulness); future work should employ a generator-independent evaluator model. The 33-question evaluation set, while carefully category-balanced, is small, drawn from the same corpus, and authored by the researcher, without an independent test set or external benchmark, limiting statistical generalizability. Reported rates should be read as exploratory rather than confirmatory findings, to be confirmed on a larger, more diverse, and preferably externally constructed query pool. The manual-versus-system efficiency comparison (Section 4.3) is likewise a simulation based on assumed manual processing times rather than an empirical, controlled user study. The corpus is limited to July 2025 content and system performance should be validated on a broader, multi-period, multi-genre dataset before generalizing beyond it. Hyperparameters were tuned on only an 8-question development set (Section 3.5), reducing but not eliminating tuning uncertainty given the development set’s small size, and the expert review used a single blinded rater, so inter-rater agreement (e.g., Cohen’s kappa) could not be computed; at least two independent, blinded raters with an agreement analysis would strengthen this evidence. claude-sonnet-5 calls were single-run at default temperature (the endpoint does not expose this parameter), so output variance across repeated runs is unmeasured. Repeating each question 3-5 times and reporting means and standard deviations would give a more reliable picture, since LLM outputs are inherently stochastic. Finally, the ablation in Section 4.5 isolates retrieval-oriented contributions (HR@8, MRR, hallucination rate) but does not separately isolate prompt design or generator-model contributions as independent variables, which should be treated as a scope limitation of the decomposition rather than evidence that those factors are unimportant. Despite these caveats, the 4.2% hallucination rate obtained remains well below both the 17-33% band reported for deployed legal-AI tools [7] and the roughly 15-25% band often proposed as acceptable for low-error-tolerance domains, which we read as a promising, if preliminary, result for this evaluation set rather than a definitive population-level estimate.

6. Conclusions and Future Work

This study developed a layout-aware hybrid RAG system that markedly reduces hallucination in Turkish question answering over the Official Gazette, while keeping retrieval and indexing under local data sovereignty and delegating only response synthesis to a cloud-hosted model. Four findings support this conclusion: (i) hallucination on answered questions fell to 4.2%, within literature-acceptable bounds for law and medicine, with reliable refusal on unanswerable queries; (ii) Docling-based layout-aware parsing enabled high retrieval precision, indicating that most residual error originates in generation rather than retrieval; (iii) a simulation suggests substantial efficiency gains over manual archive search, relevant to a source that grows by hundreds of pages daily; and (iv) the study contributes the first example, to the authors’ knowledge, of jointly addressing generative Turkish QA and hallucination mitigation, where prior Turkish legal NLP work has concentrated on discriminative tasks. Even a locally constrained retrieval architecture, paired with an appropriate hybrid strategy and coherence-based synthesis, can thus reach usable performance in a reliability-critical domain, and moving response synthesis to a higher-capacity model appears to extend this success further without requiring the retrieval side to be re-engineered. More broadly, the approach illustrates a template that may transfer to other morphologically rich, comparatively low-resource languages with an analogous authoritative-gazette structure: pair an existing document-AI layout parser with a hybrid dense-lexical index tuned to the target language’s legal-citation conventions, and delegate only the final synthesis step to a capable generator under a citation-constrained prompt.

Future work should pursue four directions. First, multimodal document understanding, for example vision-language models that jointly reason over layout images and extracted text, is needed to interpret the scanned charts, organizational diagrams, and staffing tables that densely populate Gazette annexes and remain a structural blind spot for the current text-only pipeline. Second, self-hosted, open-weight generators of comparable legal-reasoning capacity should be evaluated to quantify the trade-off between full data sovereignty and generation quality, since the current design still depends on a cloud API’s availability, latency, and cost for the synthesis step alone. Third, domain-adaptive fine-tuning or lightweight adapters trained on Turkish statutory language may improve answer completeness and precision on the corpus-wide analytical queries identified in Section 4.1 as a current weakness, since the generator used here was not adapted to Turkish legal register beyond prompting and retrieved context. Fourth, a continuous ingestion pipeline that indexes new Gazette issues as they are published and tracks the temporal validity of superseded provisions would extend the single-month corpus used here toward a production-grade system in which retrieval always reflects the currently effective version of a given statute.