TurkishMedLLM: A Source-Grounded and Safety-Aware Turkish Medical LLM for Clinical Decision Support and Digital Health Applications

1. Introduction

In today’s digital health landscape, artificial intelligence (AI) is transforming how medical data is accessed, understood, and communicated [1]. With the advent of large language models (LLMs), notably, new opportunities for patient education, clinical documentation support, medical question answering, and decision-support workflows arise [2,3]. They are appealing for healthcare use cases because they can interpret human language and provide well-structured answers, such as patients describing symptoms, concerns about medications, issues with chronic diseases, or lifestyle questions using natural language. But healthcare is not a typical text generation domain. A medical response needs to be accurate, context aware, clinically cautious, source grounded and understandable to the intended user. This is particularly significant for Turkish, as there is a lack of medical LLM resources adapted to local contexts, clinical language systems in Turkish, and patient-facing medical AI tools when compared to English-based platforms [4].

The problem addressed in this study is the construction of a Turkish medical LLM that can support clinical decision support and digital health applications while maintaining source grounding, traceability, and safety-aware response behavior. Unlike a general-purpose conversational model, a medical LLM must not only generate fluent language, but also retrieve relevant evidence, remain faithful to the retrieved medical context, avoid unsupported clinical claims, and provide cautious guidance when the available evidence is insufficient. In this study, the problem is therefore treated as a source-grounded medical response generation task in which relevance, faithfulness, clinical caution, and risk reduction are jointly considered. The formal mathematical formulation of this task is provided in Section 3.1.

This issue is significant as medical language generation can directly impact the understanding of medical-related information by the users. When a patient gives a fluent but unsupported response, it could look like he has a full answer, even if it’s incomplete, outdated or even medically unsafe. These can lead to a medical consultation being delayed, symptoms misunderstood, self-management encouraged or overconfidence in non-professional advice in the context of a digital health intervention. Thus, creating a medical LLM in Turkey requires a unique approach compared to a traditional conversational AI system. It needs to be able to find medical context from the Turkish perspective, maintain a trace of the source, indicate when it is not sure of the evidence, not make a definitive diagnosis and prompt the user to seek advice from a health professional when the condition is potentially serious, or not clear. These requirements prompt the development of a Turkish medical LLM that is grounded on sources and aware of safety.

Consider an application where the proposed system may be used is a Turkish digital health platform where patients, caregivers or health professionals pose medical questions using natural language. Some of the questions asked might be: "Why does the blood glucose rise in diabetes? "What are the signs to be mindful of when it comes to hypertension? "When is it unsafe for children to have a fever? "What are some factors to keep in mind when taking antibiotics? In such situations, the system should present the appropriate information from the medical field in Turkish, formulate the answer in an understandable way, not make assumptions that are not based on evidence and give a warning if a particular medical condition calls for professional attention. The system can be used for mobile health apps, patient education systems in hospitals, clinician-assisted dashboards, systems for education on chronic diseases, as well as Turkish conversational agents for digital health.

There are three general approaches to medical language modeling that have been developed to date. The first approach involves directly utilizing general-purpose LLMs for medical question answering due to their general linguistic and reasoning capabilities. Second direction fine tunes an existing language model on medical question–answer data to enhance domain-specific instruction following. The third direction is retrieval-augmented generation (RAG) that retrieves external medical documents and provides them to the model as context before generating answers. The benefits of these approaches are: General-purpose LLMs offer general knowledge of language, fine-tuning helps to adapt to a specific domain, and RAG allows access to external evidence during inference.

However, there are significant challenges associated with these solutions. General purpose LLMs might not be well suited to capture Turkish medical vocabulary, the style of communication used by patients, or the information needs of the Turkish health care context. The fine-tuned models can learn some useful patterns in the response, but when no medical evidence is retrieved in the inference process, the models may generate unsupported or outdated claims. RAG-based systems enhance grounding, but their effectiveness hinges on the quality of the corpus, chunking approach, embedding model, the vector database, the retriever, and the generation prompt. Moreover, there is a lack of transparency in many medical LLM systems in terms of data ingestion, schema detection, handling duplicates, tracing back to sources, and filtering for data quality. Evaluation is also frequently limited as traditional text generation metrics are not enough for clinical decision support. The assessment of a medical LLM should not just be based on the fluency of the output but also its retrieval relevance, faithfulness, context alignment, source-groundedness, clinical caution, and safety behavior.

There are five research questions that guide this study. The first research question is how to convert heterogenous Turkish medical datasets into a traceable and reusable corpus, so that it can be used to create a supervised fine-tuned model and retrieval-augmented model. The second research question relates to the retrieval of clinically relevant Turkish context for patient-oriented queries using local multilingual embeddings and vector indexing. This study’s third research question is to what degree does retrieval-augmented generation boost source grounded medical response generation for Turkish over a base LLM. The fourth research question is whether medical instruction after fine tuning with QLoRA is better than the base model Qwen3-8B. The fifth research question is whether a multi-layer evaluation protocol with the help of retriever metrics, RAGAS, DeepEval, and clinical safety-oriented graders can be used to more reliably evaluate Turkish medical LLM behavior.

These research questions are followed by the objectives of this study. The first one is to build a multi-source medical corpus in Turkish from question–answer and long-form hospital medical articles, which are processed with source-level traceability. The second goal is to convert this corpus into two partially overlapping corpora - a fine tuning corpus with supervision and a retrieval ready article corpus. The third goal is to develop a local retrieval system for Turkish medical semantic search with BGE-M3 embeddings and ChromaDB. The fourth goal is to create a retrieval-augmented response generation model that takes retrieved Turkish medical context into account while generating answers. The fifth objective is to fine tune Qwen3-8B with QLoRA on the cleaned Turkish medical question–answer corpus. The sixth objective is to assess the system developed in the previous objectives through retriever hit-rate metrics, RAGAS, DeepEval, and custom clinical safety-oriented metrics.

In response to the problem defined, TurkishMedLLM, a source grounded and safety aware Turkish medical large language model for clinical decision support and digital health applications is presented. The overall workflow is broken down into independently auditable modules, such as data ingestion, schema detection, quality filtering, text transformation, embedding generation, vector-store construction, retriever evaluation, response generation, fine-tuning, and safety-aware evaluation, in the proposed system. TurkishMedLLM builds a harmonized Turkish medical corpus and keeps the information about the source level for auditing. It generates 210,791 cleaned Turkish medical question–answer records for supervised fine-tuning, and converts 22,135 article documents to 206,166 retrieval-ready chunks. The chunks are then stored locally via BGE-M3 and Ollama, and indexed in ChromaDB for semantic retrieval. The generative component is based on Qwen3-8B, fine-tuned using QLoRA, which makes it possible to make systematic comparisons between base LLM, base RAG, fine-tuned LLM, fine-tuned RAG, and safety-aware fine-tuned RAG.

This study provides a comprehensive Turkish medical LLM framework, consisting of the 4-step process of corpus traceability, retrieval grounding, fine-tuning, and safety-aware evaluation. Specifically, the study proposes a pipeline for multi-source Turkish medical corpus construction, which includes schema standardization, duplicate removal, and quality filtering; creates distinct resources for retrieval-augmented generation (RAG) and supervised fine-tuning; uses local BGE-M3-based vector indexing with ChromaDB for retrieval performance evaluation; evaluates the retrieval performance on Turkish clinical queries; fine tunes Qwen3-8B using QLoRA; and applies multi-layer evaluation framework based on retriever metrics, RAGAS, DeepEval, and clinical safety oriented evaluation criteria. TurkishMedLLM’s modular design allows for instruction, retrieval of evidence, answer generation, fine-tuning, and safety assessment to be conducted separately, offering a clear and reproducible platform for further research on medical AI in the Turkish context and digital health applications.

The rest of this paper is structured as follows. Section 2 discusses the related literature on medical LLMs, medical NLP in Turkish, retrieval-augmented generation, vector-based retrieval, fine-tuning methods and assessment of medical conversational AI systems. In Section 3, the proposed TurkishMedLLM methodology, which encompasses data ingestion, schema standardization, corpus preparation, text splitting, embedding generation, vector-store construction, retriever evaluation, RAG generation, a fine-tuning strategy, and safety-aware evaluation design, is presented. Section 4 describes the implementation details of the modular software pipeline. Results of corpus construction, retrieval, RAG evaluation, fine-tuning results, ablation results, and safety results are reported and discussed in Section 5. Section 6 concludes the paper and discusses limitations and future deployment considerations.

2. Related Work

The advent of large language models has brought about a host of innovative possibilities in the field of medical question answering, patient education, clinical documentation support, and digital health decision support [5]. But it is essential that the reliability of medical LLMs be improved compared to general-purpose conversational systems, since the content generated can impact how a user understands their symptoms, medication information, chronic disease management, and the need for professional consultation. So, domain adaptation, retrieval augmentation, source-grounded generation, and clinical safety evaluation have been increasingly studied recently [6,7]. In this section, the relevant literature on medical LLMs, Turkish medical NLP, retrieval-augmented generation, embedding-based retrieval, parameter-efficient fine-tuning, and evaluation of medical systems based on RAG is discussed.

2.1. Medical Large Language Models

Medical LLMs have demonstrated promising capabilities for clinical and biomedical question answering [8]. One of the first influential studies on assessing LLMs on the MultiMedQA benchmark that contains medical examination questions, biomedical question answering and consumer health questions is Med-PaLM. The study revealed that instruction-tuned LLMs can attain competitive benchmark scores in the medical domain and that human assessment should measure factuality, comprehension, reasoning, harm, and bias. But, the responses were not always equivalent to clinician responses, particularly regarding safety and reliability [9,10]. Med-PaLM 2 also achieved better performance on medical question answering, with improved base models, medical-domain adaptation, and prompting. It reached competitive performance on a number of medical benchmarks as well as enhancing long form consumer medical answers [10]. However, the limits of transparency, local deployment, and adaptation of low-resource clinical languages like Turkish in the work were not clear [11]. Domain-specific pretraining is also beneficial for biomedical models like BioGPT and PubMedBERT [12]. BioGPT was pre-trained on biomedical literature, and also improved biomedical text mining and question answering, and PubMedBERT was trained from scratch on PubMed abstracts and full-text articles, and improved several biomedical language understanding tasks [13,14]. However, BioGPT is focused on biomedical literature, and PubMedBERT is an encoder-based model; neither directly addresses Turkish patient-oriented question answering, source-grounded RAG, or safety-aware medical response generation. In summary, existing medical LLM research demonstrates that domain adaptation is indispensable for medical AI systems. However, the majority of established models remain English-centric, rely primarily on standardized benchmarks, and offer limited transparency for localized clinical decision-support scenarios. TurkishMedLLM extends these paradigms by specializing in Turkish medical data, source traceability, retrieval grounding, local embeddings, QLoRA fine-tuning, and safety-aware evaluation.

2.2. Turkish Medical NLP and Low-Resource Medical Language Processing

Turkish medical NLP is still underdeveloped compared to English biomedical NLP, especially for generating and answering questions and retrieval augmented clinical decision support. TurkMedNLI is a valuable effort in making a Turkish medical natural language inference dataset with an LLM-assisted translation pipeline. Although the study covered the aspects of terminology preservation, abbreviation handling, and medical meaning consistency, it focused on natural language inference instead of Turkish medical question answering, response generation using RAG, or fine-tuning of LLMs [15]. The TUSGPT-TR Medical Dataset and Zenodo Turkish Medical Question–Answer Dataset contains useful data for supervised fine-tuning in Turkish. The datasets can be used to adapt the models designed for instruction following to Turkish patient-oriented and clinical communication. However, raw datasets are not usable for models without careful preprocessing, as schema inconsistencies, duplicates, missing fields, very short answers, noisy text, and limited source traceability may affect the reliability of the models [16]. The Türkçe Tıbbi Soru-Cevap Veri Seti (Turkish Medical Question–Answer Dataset), comprising 167,732 physician-answered health-related question–answer pairs collected from DoktorSitesi.com, provides a large-scale Turkish medical QA resource for supervised fine-tuning and healthcare-oriented NLP applications [17]. However, similar to other large-scale web-sourced datasets, it requires careful preprocessing to address potential issues such as noise, inconsistencies, and limited source traceability, which may otherwise impact model reliability. Another important knowledge source for retrieval augmented generation is long-form Turkish hospital medical articles that explain information regarding medical conditions, symptoms, diagnostics, prevention and patient education. Before these documents can be used for RAG [18], however, they need to be split into retrieval-ready chunks, their contents need to be embedded into a semantic vector representation, and they need to be indexed. TurkishMedLLM fills this gap by creating two complementary corpora of medical information; a cleaned instruction-tuning corpus and a retrieval-ready article corpus in Turkish.

2.3. Retrieval Augmented Generation for Medical Question Answering

To ensure that the factual content of LLM responses is accurate, retrieval-augmented generation has emerged as a prominent approach. In RAG, relevant documents are retrieved from an external knowledge base and input as context to the language model prior to generation. This design is of special relevance in medical applications, since the model parameters might be outdated, incomplete, or unconfirmed. RAG can minimize the risk of hallucination and enhance transparency in the responses, by leveraging the retrieved evidence [19]. RAG can be used in medical Q&A to educate patients, explain symptoms, help with self-management of chronic diseases, increase medication awareness, and help with clinician-guided decision making [20]. But RAG is not necessarily clinically accurate. The quality of the response is influenced by the retrieval corpus, chunking approach, embedding model, vector store, retriever setup, prompt design, and safety measures. Retrieved context chunks may include irrelevant or incomplete information, or the generator may not account for the context. Because Turkish has rich morphology and flexible word forms and combines formal medical jargon with everyday patient language, Turkish medical RAG presents additional difficulties. Hence, Turkish medical RAG needs multilingual semantic embeddings, well-crafted Turkish medical content, and retrieving using Turkish medical queries. To meet these needs, TurkishMedLLM builds a Turkish retrieval corpus based on articles, employs BGE-M3 to embed the articles, stores them in ChromaDB, and performs a validation process for retrieval before generating responses.

2.4. Medical Retrieval Using Embedding Models and vector Databases

The models used for embedding are crucial to semantic retrieval, as they are responsible for representing queries and documents in vector space. BGE-M3 [21] is applicable to TurkishMedLLM as it aids in multi-language, multi-function (retrieval), and multi-granularity retrieval. The model was developed for dense, sparse, and multi-vector retrieval in over 100 languages, which makes it suitable for Turkish medical retrieval, where semantic similarity and language coverage matters significantly [22]. Although multilingual embeddings have proven useful, embedding quality is not enough to ensure safe medical retrieval. The source corpus needs to be cleaned, broken up into meaningful units and given metadata that allows for traceability. Vector databases like ChromaDB and FAISS allow for scalable similarity search over large document collections, but still necessitate duplicate control, chunk-level metadata, retriever testing, and downstream answer evaluation. TurkishMedLLM leverages ChromaDB as a persistent vector store, and stores chunk identifiers, source names, article titles, urls and metadata of parent documents for source-level auditing.

2.5. Fine-Tuning of Open Source LLMs with Parameter-Efficiency

Fine-tuning is a common technique for adapting generic LLM models to domain-specific tasks. For medical uses, fine tuning can enhance instruction following, terminology use, and response style. But, this fine-tuning is costly and can lead to overfitting or catastrophic forgetting particularly in the heterogeneous training data [23]. To address this, QLoRA offers an efficient option to fine-tune quantized LLMs with a limited number of parameters. It back-propagates the gradients from a frozen, 4-bit quantized model to the low rank adapter weights to minimize the memory footprint in the process of adaptation, while keeping the adaptation performance high [24]. This renders QLoRA an appropriate technique for TurkishMedLLM as it enables adapting Qwen3-8B with the purified Turkish medical question–answer set without updating the full model. The choice of Qwen3-8B is guided by its design, which supports multilingualism and reasoning, which is beneficial for a patient-centric medical question answering system that needs to comprehend Turkish medical jargon, patient-centric language, and Turkish syntax [25]. TurkishMedLLM examines the impact of QLoRA adaptation using five configurations: base LLM, base RAG, fine-tuned LLM, fine-tuned RAG, and safety-aware fine-tuned RAG.

2.6. Assessing Medical LLM Systems with RAG

One of the biggest issues in medical RAG is the evaluation of the output, as a fluent and relevant answer might lack support or be clinically unsafe [26]. For medical LLM evaluation, therefore, there is a need to consider the retrieval quality, usefulness of context, faithfulness of answers, groundedness in sources, clinical caution, and safety behavior. RAGAS offers faithfulness, answer relevancy, context precision, and context recall as metrics that can be employed to evaluate the relevance of retrieved context and whether generated answers are supported by it [27]. DeepEval offers a free evaluation of answer relevancy and contextual relevancy, which helps determine whether failures are due to retrieval or generation [28]. OpenAI Evals and custom grader-based evaluation offer flexible task-specific assessment criteria, such as source-groundedness, clinical caution, unsupported-claim avoidance, emergency escalation, medication dosage safety, and Turkish fluency [29]. TurkishMedLLM uses a multi-layer evaluation approach. Hit-rate metrics and relevance estimation based on keywords are used for the retrieval layer evaluation. RAGAs and DeepEval are used to evaluate the generation layer. Last, responses are evaluated by clinical safety-oriented graders to determine if they are cautious, source-based, and suitable for use in digital health for patients.

Table 1. Comparison of representative studies and their relevance to TurkishMedLLM.

Study	Main Focus	Reported Contribution / Outcome	Remaining Gap
Med-PaLM [9]	Medical LLM evaluation	Evaluated LLMs on MultiMedQA with human assessment of factuality, reasoning, harm, and bias.	English-centered; limited Turkish adaptation, local deployment, and transparent RAG corpus construction.
Med-PaLM 2 [10]	Improved medical LLM performance	Improved medical question answering and long-form consumer health responses through adaptation and prompting.	Proprietary infrastructure; no Turkish source-grounded and auditable framework.
BioGPT [13]	Biomedical generative pretraining	Improved biomedical text mining and question-answering tasks.	Focused on biomedical literature rather than Turkish patient-oriented clinical support.
PubMedBERT [14]	Biomedical language understanding	Improved biomedical NLP tasks through domain-specific pretraining.	Encoder-based; not designed for generative Turkish medical RAG.
TurkMedNLI [15]	Turkish medical NLI	Built a Turkish medical inference dataset and addressed terminology adaptation.	Focuses on NLI rather than Turkish medical RAG, fine-tuning, and safety-aware generation.
BGE-M3 [22]	Multilingual retrieval embedding	Supports multilingual dense, sparse, and multi-vector retrieval.	Does not address medical corpus construction or clinical safety by itself.
QLoRA [24]	Efficient LLM fine-tuning	Enables memory-efficient fine-tuning of quantized LLMs using low-rank adapters.	Provides adaptation method but not Turkish medical corpus, RAG, or clinical evaluation.
RAGAS [27]	RAG evaluation	Provides faithfulness, answer relevancy, context precision, and context recall metrics.	Does not fully cover clinical safety, escalation, or medication caution.

Table 1. Comparison of representative studies and their relevance to TurkishMedLLM.

Study	Main Focus	Reported Contribution / Outcome	Remaining Gap
Med-PaLM [9]	Medical LLM evaluation	Evaluated LLMs on MultiMedQA with human assessment of factuality, reasoning, harm, and bias.	English-centered; limited Turkish adaptation, local deployment, and transparent RAG corpus construction.
Med-PaLM 2 [10]	Improved medical LLM performance	Improved medical question answering and long-form consumer health responses through adaptation and prompting.	Proprietary infrastructure; no Turkish source-grounded and auditable framework.
BioGPT [13]	Biomedical generative pretraining	Improved biomedical text mining and question-answering tasks.	Focused on biomedical literature rather than Turkish patient-oriented clinical support.
PubMedBERT [14]	Biomedical language understanding	Improved biomedical NLP tasks through domain-specific pretraining.	Encoder-based; not designed for generative Turkish medical RAG.
TurkMedNLI [15]	Turkish medical NLI	Built a Turkish medical inference dataset and addressed terminology adaptation.	Focuses on NLI rather than Turkish medical RAG, fine-tuning, and safety-aware generation.
BGE-M3 [22]	Multilingual retrieval embedding	Supports multilingual dense, sparse, and multi-vector retrieval.	Does not address medical corpus construction or clinical safety by itself.
QLoRA [24]	Efficient LLM fine-tuning	Enables memory-efficient fine-tuning of quantized LLMs using low-rank adapters.	Provides adaptation method but not Turkish medical corpus, RAG, or clinical evaluation.
RAGAS [27]	RAG evaluation	Provides faithfulness, answer relevancy, context precision, and context recall metrics.	Does not fully cover clinical safety, escalation, or medication caution.

2.7. Research Gaps

It has been demonstrated in the literature reviewed that medical LLMs can obtain good benchmark results, and biomedical pretraining can enhance the comprehension of medical domain-specific language, while multilingual embeddings can facilitate semantic retrieval, and QLoRA can help adapt models to specific medical fields. Nevertheless there are still some gaps for clinical decision support in Turkey. The majority of medical LLM programs focus on the English standards, and there is insufficient interest in patient-based medical communication in Turkish. Although there are valuable existing resources, these are seldom combined in a complete pipeline with source-aware ingestion, schema detection, duplicate removal, quality filtering, retrieval corpus building, supervised fine tuning and safety aware evaluation. Moreover, there is a lack of clarity in many RAG systems when assessing the quality of the answer, concerning the separation of retrieval quality, context faithfulness, fine-tuning effects, and safety constraints. TurkishMedLLM aims to fill these gaps with a traceable, retrieval augmented, fine-tuned, and safety aware Turkish medical LLM framework. TurkishMedLLM is not just a study about the performance on benchmarks, model fine-tuning, or retrieval, but combines all these processes into a single auditable pipeline by incorporating data curation, local retrieval infrastructure, QLoRA fine-tuning, RAG generation, and multi-layer evaluation. For digital health applications in Turkey, language localization and clinical reliability are key factors, and this design ensures they are both available.

Table 2. Summary of research gaps and how TurkishMedLLM addresses them.

Research Gap	Why It Matters	TurkishMedLLM Response
Limited Turkish medical LLM resources	Turkish clinical and patient-oriented language differs from English-centered benchmarks.	Constructs a multi-source Turkish medical corpus for fine-tuning and retrieval.
Weak corpus traceability	Medical systems require auditability of training and retrieval sources.	Preserves source identifiers, file names, row indices, chunk IDs, titles, URLs, and metadata.
Direct use of raw datasets	Noisy, duplicate, or incomplete records may reduce model reliability.	Applies schema detection, dataset selection, duplicate removal, and quality filtering.
Lack of retrieval-ready Turkish medical knowledge bases	RAG requires context-preserving chunks and semantic indexing.	Converts Turkish hospital articles into 206,166 retrieval-ready chunks and indexes them in ChromaDB.
Limited local retrieval infrastructure	Clinical settings may require reduced dependence on external APIs.	Uses local BGE-M3 embeddings through Ollama with a persistent ChromaDB vector store.
Unclear separation of retrieval and fine-tuning effects	Improvements may come from retrieval, fine-tuning, or prompting, but these are often not isolated.	Evaluates base LLM, base RAG, fine-tuned LLM, fine-tuned RAG, and safety-aware RAG.
Incomplete medical RAG evaluation	General text metrics do not capture faithfulness, safety, or clinical caution.	Combines hit-rate metrics, RAGAS, DeepEval, and clinical safety-oriented graders.

Table 2. Summary of research gaps and how TurkishMedLLM addresses them.

Research Gap	Why It Matters	TurkishMedLLM Response
Limited Turkish medical LLM resources	Turkish clinical and patient-oriented language differs from English-centered benchmarks.	Constructs a multi-source Turkish medical corpus for fine-tuning and retrieval.
Weak corpus traceability	Medical systems require auditability of training and retrieval sources.	Preserves source identifiers, file names, row indices, chunk IDs, titles, URLs, and metadata.
Direct use of raw datasets	Noisy, duplicate, or incomplete records may reduce model reliability.	Applies schema detection, dataset selection, duplicate removal, and quality filtering.
Lack of retrieval-ready Turkish medical knowledge bases	RAG requires context-preserving chunks and semantic indexing.	Converts Turkish hospital articles into 206,166 retrieval-ready chunks and indexes them in ChromaDB.
Limited local retrieval infrastructure	Clinical settings may require reduced dependence on external APIs.	Uses local BGE-M3 embeddings through Ollama with a persistent ChromaDB vector store.
Unclear separation of retrieval and fine-tuning effects	Improvements may come from retrieval, fine-tuning, or prompting, but these are often not isolated.	Evaluates base LLM, base RAG, fine-tuned LLM, fine-tuned RAG, and safety-aware RAG.
Incomplete medical RAG evaluation	General text metrics do not capture faithfulness, safety, or clinical caution.	Combines hit-rate metrics, RAGAS, DeepEval, and clinical safety-oriented graders.

3. Research Methodology

This section presents the methodology used to design and evaluate TurkishMedLLM, a source-grounded and safety-aware Turkish medical large language model for clinical decision support and digital health applications. The methodology follows a modular structure in which data preparation, retrieval construction, response generation, fine-tuning, safety control, and evaluation are treated as connected but independently auditable components. The overall methodology is shown in Figure 1. The pipeline begins with heterogeneous Turkish medical data sources and proceeds through ingestion, schema standardization, quality filtering, corpus separation, supervised fine-tuning preparation, retrieval corpus construction, embedding generation, vector indexing, RAG-based response generation, safety-aware control, and multi-layer evaluation. The framework uses two complementary streams: a question–answer stream for supervised fine-tuning and a hospital-article stream for retrieval-augmented generation.

3.1. Problem Formulation

Let the raw Turkish medical dataset collection be represented as

$$\mathcal{X}=\{{\mathcal{X}}_{qa}^{\left(1\right)},{\mathcal{X}}_{qa}^{\left(2\right)},{\mathcal{X}}_{art}^{\left(3\right)}\},$$

(1)

where ${\mathcal{X}}_{qa}^{\left(1\right)}$ and ${\mathcal{X}}_{qa}^{\left(2\right)}$ denote Turkish medical question–answer datasets, and ${\mathcal{X}}_{art}^{\left(3\right)}$ denotes the Turkish hospital article dataset. The preprocessing stage transforms $\mathcal{X}$ into two structured corpora:

$$\mathcal{C}=\{{\mathcal{C}}_{sft},{\mathcal{C}}_{rag}\},$$

(2)

where ${\mathcal{C}}_{sft}$ is the supervised fine-tuning corpus and ${\mathcal{C}}_{rag}$ is the retrieval corpus.

For a user query q, the retriever returns the top-k relevant context chunks:

$$C_q=R(q,{\mathcal{C}}_{rag},k),$$

(3)

where $C_q=\{c_1,c_2,\dots ,c_k\}$. The generative model then produces the response:

$$y=G_{\theta }(q,C_q),$$

(4)

where $G_{\theta }$ denotes the language model.

The objective is to generate Turkish medical responses that are relevant, faithful to retrieved evidence, clinically cautious, and safe:

$$\begin{array}{cc}\hfill \underset{y}{max}\mathcal{S}(y,q,C_q)=& {\lambda }_1\mathrm{Rel}(y,q)+{\lambda }_2\mathrm{Faith}(y,C_q)\hfill \\ \hfill & +{\lambda }_3\mathrm{Safe}\left(y\right)-{\lambda }_4\mathrm{Risk}\left(y\right),\hfill \end{array}$$

(5)

where $\mathrm{Rel}(y,q)$ measures answer relevance, $\mathrm{Faith}(y,C_q)$ measures consistency with retrieved context, $\mathrm{Safe}\left(y\right)$ measures clinical caution, and $\mathrm{Risk}\left(y\right)$ represents unsupported diagnosis, unsafe medication guidance, missing escalation advice, or hallucinated claims.

3.2. Data Source Design

The corpus is organized into two streams, as summarized in Table 3. Question–answer datasets are used for supervised fine-tuning because they support Turkish medical instruction learning, while hospital articles are used for retrieval because they provide explanatory medical context during inference. This separation allows the model to learn response behavior from Q&A data while using external evidence from medical articles during RAG-based generation.

3.3. Data Ingestion and Schema Standardization

The ingestion stage converts heterogeneous JSON, CSV, and Parquet files into a unified document representation. For each raw record $x_i$, the transformation function $T(·)$ produces:

$$d_i=T(x_i,M_i),$$

(6)

where $d_i$ is the standardized document and $M_i$ contains source identifier, file name, row index, dataset type, split information, and intended downstream use.

For question–answer records:

$$d_i^{qa}=\{q_i,a_i,M_i\},$$

(7)

where $q_i$ is the medical question and $a_i$ is the corresponding answer. For article records:

$$d_i^{art}=\{t_i,M_i\},$$

(8)

where $t_i$ denotes the article text.

Flexible schema detection is applied to handle different dataset structures. Question fields are detected using terms such as question, soru, input, prompt, instruction, and hasta_sorusu. Answer fields are detected using terms such as answer, cevap, output, response, and completion. If direct matching fails, a length-based heuristic assigns the shorter text column as the question and the longer text column as the answer. All schema decisions are saved in processing reports.

3.4. Dataset Selection, Duplicate Removal, and Quality Filtering

A dataset selection policy prevents double-counting of equivalent dataset versions. When both full and split files exist, split files are used. When both CSV and Parquet versions are available, the Parquet file is selected for efficient processing.

Duplicate detection is applied separately to Q&A and article streams. For Q&A records, the normalized duplicate key is:

$$K_{qa}\left(i\right)=\mathrm{norm}\left(q_i\right)\left|\right|\mathrm{norm}\left(a_i\right),$$

(9)

and the filtering rule is:

$$d_i^{qa}=\left\{\begin{array}{cc}\mathrm{retain},\hfill & K_{qa}\left(i\right)\notin {\mathcal{K}}_{qa},\hfill \\ \mathrm{discard},\hfill & K_{qa}\left(i\right)\in {\mathcal{K}}_{qa}.\hfill \end{array}\right.$$

(10)

For article records, the duplicate key is:

$$K_{art}\left(i\right)=\mathrm{norm}\left(t_i\right).$$

(11)

Minimum length thresholds are applied to remove incomplete or weakly informative records. The quality thresholds required each question to contain at least 10 characters, each answer at least 30 characters, each combined Q&A sample at least 50 characters, and each article document at least 200 characters. After filtering, the processed corpus contains 232,926 unique documents, including 210,791 question–answer records and 22,135 article documents. A total of 9,139 low-quality records are removed.

3.5. Supervised Fine-Tuning Corpus Preparation

The cleaned Q&A corpus is converted into instruction-tuning format. Each sample is represented as:

$$s_i=\{{\mathrm{inst}}_i,{\mathrm{input}}_i,{\mathrm{output}}_i,M_i\},$$

(12)

where ${\mathrm{inst}}_i$ is the instruction, ${\mathrm{input}}_i$ is the Turkish medical question, ${\mathrm{output}}_i$ is the target answer, and $M_i$ is source metadata. The final supervised fine-tuning corpus is:

$${\mathcal{C}}_{sft}=\{s_1,s_2,\dots ,s_N\},$$

(13)

where $N=210,791$.

3.6. Text Transformation and Retrieval Corpus Construction

The article corpus is transformed into retrieval-ready chunks using recursive text splitting. For each article $t_i$, the splitting function generates:

$$\mathrm{Split}\left(t_i\right)=\{c_{i,1},c_{i,2},\dots ,c_{i,l_i}\},$$

(14)

where $l_i$ is the number of chunks generated from article i. The full retrieval corpus is:

$${\mathcal{C}}_{rag}={\displaystyle \bigcup _{i=1}^A}\mathrm{Split}\left(t_i\right),$$

(15)

where $A=22,135$.

Recursive splitting is applied with a chunk size of 1000 characters and an overlap of 150 characters to preserve semantic continuity across adjacent medical text segments. This stage converts 22,135 article documents into 206,166 retrieval-ready chunks, while 1,776 low-quality fragments are removed. Metadata preservation is enabled throughout the process so that each chunk remains linked to its original article source.

3.7. Embedding Generation and Vector Store Construction

Each retrieval-ready chunk is embedded using BGE-M3 through Ollama:

$${\mathbf{v}}_i=E\left(c_i\right),$$

(16)

where ${\mathbf{v}}_i\in {\mathbb{R}}^d$. The vector index is:

$$\mathcal{V}={\left\{({\mathbf{v}}_i,c_i,M_i)\right\}}_{i=1}^m,$$

(17)

where $m=206,166$.

ChromaDB is used as the persistent vector store for the 206,166 embedded article chunks, with BGE-M3 serving as the local multilingual embedding model through Ollama. Embedding is performed in batches of 500 chunks, while checkpointing and LangSmith tracing are enabled to support recovery, monitoring, and reproducibility during long indexing runs.

3.8. Retriever Construction and RAG Generation

For a query q, the query embedding is:

$${\mathbf{v}}_q=E\left(q\right).$$

(18)

The similarity between the query and indexed chunks is computed using cosine similarity:

$$\mathrm{sim}({\mathbf{v}}_q,{\mathbf{v}}_i)={\displaystyle \frac{{\mathbf{v}}_q·{\mathbf{v}}_i}{\parallel {\mathbf{v}}_q\parallel \parallel {\mathbf{v}}_i\parallel }}.$$

(19)

The top-k retrieved chunks are selected as:

$$C_q={TopK}_{c_i\in {\mathcal{C}}_{rag}}\left(\mathrm{sim}({\mathbf{v}}_q,{\mathbf{v}}_i),k\right).$$

(20)

The RAG module generates a Turkish medical response using the query, retrieved context, and prompt template:

$$y=G_{\theta }(q,C_q,P).$$

(21)

The prompt instructs the model to answer in Turkish, rely only on retrieved context, avoid unsupported claims, avoid definitive diagnosis, avoid unsupported medication dosage advice, recommend professional consultation when appropriate, and state when context is insufficient.

3.9. Safety-Aware Response Control

The safety layer checks whether the generated response is supported by retrieved context, whether it contains unsupported diagnosis, whether it gives medication dosage advice without evidence, and whether emergency escalation is required. The final response is selected as:

$$\widehat{y}=\left\{\begin{array}{cc}y,\hfill & \mathrm{Safe}(y,C_q)\ge \tau ,\hfill \\ y_{safe},\hfill & \mathrm{Safe}(y,C_q)<\tau ,\hfill \end{array}\right.$$

(22)

where $y_{safe}$ is a revised or cautionary response and $\tau$ is the safety threshold. For high-risk scenarios such as chest pain, stroke symptoms, severe breathing difficulty, severe hypoglycemia, pregnancy-related complications, and pediatric emergencies, the system prioritizes escalation guidance.

3.10. Supervised Fine-Tuning Strategy

Qwen3-8B is fine-tuned using QLoRA on the cleaned Turkish medical Q&A corpus, using the configuration summarized in Table 4. The fine-tuning objective minimizes the negative log-likelihood of target answer tokens:

$${\mathcal{L}}_{sft}=-{\displaystyle \sum _{t=1}^T}log{p}_{\theta }(y_t\mid y_{<t},{\mathrm{inst}}_i,{\mathrm{input}}_i),$$

(23)

where $y_t$ is the target token at position t. QLoRA applies low-rank adaptation to a quantized base model:

$$W^{\prime }=W+\Delta W=W+BA,$$

(24)

where W is the frozen pretrained weight matrix, and A and B are trainable low-rank matrices.

3.11. Evaluation and Ablation Design

The evaluation design is shown in Figure 2. TurkishMedLLM is evaluated across five configurations, as summarized in Table 6: base Qwen3-8B, base Qwen3-8B with RAG, fine-tuned Qwen3-8B, fine-tuned Qwen3-8B with RAG, and fine-tuned Qwen3-8B with RAG and safety prompt.

For a query $q_j$, Hit@k is defined as:

$$\mathrm{Hit}@k\left(q_j\right)=\left\{\begin{array}{cc}1,\hfill & \exists c_i\in C_{q_j}^{\left(k\right)}:\mathrm{rel}(c_i,q_j)>0,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(25)

The overall hit rate is:

$$\mathrm{HitRate}@k={\displaystyle \frac{1}{Q}}{\displaystyle \sum _{j=1}^Q}\mathrm{Hit}@k\left(q_j\right),$$

(26)

where Q is the number of test queries.

Generated answers are evaluated using the multi-layer protocol summarized in Table 5. RAGAS measures faithfulness, response relevancy, context precision, and context recall. DeepEval measures answer relevancy, faithfulness, contextual relevancy, contextual precision, and contextual recall. Clinical graders evaluate source-groundedness, medication caution, emergency escalation, unsupported-claim avoidance, Turkish fluency, and patient readability.

Table 5. Multi-layer evaluation protocol for TurkishMedLLM.

Evaluation Layer	Tool / Method	Metrics	Purpose
Retriever evaluation	Keyword-based relevance	Hit Rate@1, Hit Rate@3, and Hit Rate@5	Measures retrieval quality before generation
RAG evaluation	RAGAS	Faithfulness, response relevancy, context precision, and context recall	Measures grounding and context quality
RAG evaluation	DeepEval	Answer relevancy, faithfulness, contextual precision, contextual recall, and contextual relevancy	Cross-checks answer and context quality
Clinical behavior evaluation	OpenAI Evals / custom graders	Safety compliance, source-groundedness, escalation correctness, medication caution, and Turkish fluency	Measures clinical suitability and safety behavior

Table 5. Multi-layer evaluation protocol for TurkishMedLLM.

Evaluation Layer	Tool / Method	Metrics	Purpose
Retriever evaluation	Keyword-based relevance	Hit Rate@1, Hit Rate@3, and Hit Rate@5	Measures retrieval quality before generation
RAG evaluation	RAGAS	Faithfulness, response relevancy, context precision, and context recall	Measures grounding and context quality
RAG evaluation	DeepEval	Answer relevancy, faithfulness, contextual precision, contextual recall, and contextual relevancy	Cross-checks answer and context quality
Clinical behavior evaluation	OpenAI Evals / custom graders	Safety compliance, source-groundedness, escalation correctness, medication caution, and Turkish fluency	Measures clinical suitability and safety behavior

Table 6. Ablation settings for TurkishMedLLM evaluation.

ID	Configuration	Purpose
A1	Base Qwen3-8B	Measures baseline Turkish medical response capability without retrieval or fine-tuning.
A2	Base Qwen3-8B + RAG	Measures the effect of retrieval grounding using BGE-M3 and ChromaDB.
A3	Fine-tuned Qwen3-8B	Measures the effect of QLoRA-based Turkish medical instruction tuning.
A4	Fine-tuned Qwen3-8B + RAG	Measures the combined effect of fine-tuning and retrieval grounding.
A5	Fine-tuned Qwen3-8B + RAG + safety prompt	Measures the effect of safety-aware constraints on clinical reliability.

Table 6. Ablation settings for TurkishMedLLM evaluation.

ID	Configuration	Purpose
A1	Base Qwen3-8B	Measures baseline Turkish medical response capability without retrieval or fine-tuning.
A2	Base Qwen3-8B + RAG	Measures the effect of retrieval grounding using BGE-M3 and ChromaDB.
A3	Fine-tuned Qwen3-8B	Measures the effect of QLoRA-based Turkish medical instruction tuning.
A4	Fine-tuned Qwen3-8B + RAG	Measures the combined effect of fine-tuning and retrieval grounding.
A5	Fine-tuned Qwen3-8B + RAG + safety prompt	Measures the effect of safety-aware constraints on clinical reliability.

3.12. Traceability and Reproducibility

Traceability is preserved throughout the pipeline. Each processed record keeps its source identifier, file name, row index, dataset type, split information, and intended use. Each article chunk stores its parent document identifier, chunk number, title, URL, and source metadata. The ingestion, transformation, vector-store, RAG, and evaluation modules generate structured reports containing selected files, ignored files, valid records, skipped records, detected schemas, chunking configuration, embedding settings, retrieval queries, retrieved chunks, and evaluation results. This reporting design supports reproducibility and systematic auditing of TurkishMedLLM.

4. Implementation

This section describes the implementation of TurkishMedLLM as a modular Python-based pipeline. The implementation follows the methodology in Section 3 and separates the system into independent components for data ingestion, transformation, vector-store construction, retrieval, RAG generation, fine-tuning, and evaluation. This modular design improves reproducibility, debugging, and extensibility, while allowing individual components such as the embedding model, vector database, LLM backbone, or evaluation framework to be replaced without redesigning the complete system.

4.1. Software Architecture

The software architecture consists of six main modules. The ingestion module loads raw datasets and converts them into standardized documents. The transformation module cleans supervised fine-tuning records and splits article documents into retrieval-ready chunks. The vector-store module embeds article chunks using BGE-M3 through Ollama and stores them in ChromaDB. The retriever and RAG module retrieves relevant Turkish medical chunks and generates context-grounded responses. The fine-tuning module adapts Qwen3-8B using QLoRA. The evaluation module assesses retrieval quality, RAG answer quality, and clinical safety behavior.

4.2. Data Processing Implementation

The ingestion pipeline processes JSON, CSV, and Parquet files from the source directories while excluding metadata-only files. It automatically detects the dataset source, file type, split label, dataset role, and schema structure using keyword matching and fallback heuristics. The output includes standardized question–answer documents, standardized article documents, supervised fine-tuning records, and a structured ingestion report. The transformation pipeline applies conservative text normalization. Question–answer records are not split because each record already represents a complete supervised fine-tuning sample. Article records are split using recursive text splitting with a chunk size of 1000 characters and an overlap of 150 characters. The resulting outputs are saved as cleaned supervised fine-tuning records and retrieval-ready article chunks.

4.3. Vector Store Implementation

The vector-store module converts article chunks into LangChain document objects with text and metadata. Ollama is used as the local embedding backend, BGE-M3 is used as the embedding model, and ChromaDB is used as the persistent vector store. Since the retrieval corpus contains 206,166 chunks, embedding generation is performed in batches of 500. A checkpoint file records processed chunk identifiers, allowing interrupted indexing runs to resume safely. Duplicate chunk identifiers are skipped to avoid repeated insertion into the vector store.

4.4. Retriever and RAG Implementation

The retriever loads the persistent ChromaDB vector store and uses BGE-M3 embeddings to encode user queries. The RAG response generator uses Qwen3-8B as the local chat model, while BGE-M3 is used only for query and document embeddings. For each query, the top five semantically similar chunks are retrieved and passed to a Turkish medical prompt template. The prompt constrains the language model to answer in Turkish, use only the retrieved context, avoid unsupported claims, avoid definitive diagnosis, avoid unsupported medication dosage advice, recommend medical consultation when needed, and state when the retrieved context is insufficient. Each RAG output is saved with the query, retrieved chunk identifiers, source titles, URLs, retrieved contexts, generated answer, model name, and timestamp. These outputs are used for RAGAS, DeepEval, and clinical safety evaluation.

4.5. Fine-Tuning Implementation

The fine-tuning module uses Qwen3-8B as the base model and QLoRA with 4-bit quantization as the parameter-efficient adaptation method. The training corpus contains 210,791 cleaned Turkish medical question–answer records. The implemented configuration uses LoRA rank 16, LoRA alpha 32, batch size 4, gradient accumulation 2, and 10 training epochs. Model quality is monitored using training loss, validation loss, ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum, and average generation length.

4.6. Evaluation Implementation

The retriever evaluation module uses curated Turkish medical queries covering diabetes, hypertension, pediatric fever, cardiovascular symptoms, asthma, pregnancy-related diabetes, kidney disease, stroke, medication safety, hypoglycemia, endocrine disorders, neurological symptoms, gastrointestinal conditions, respiratory disease, and anemia. For each query, the top five retrieved chunks are saved with metadata and labeled as relevant, partially relevant, or irrelevant using expected medical keywords. The RAG evaluation module processes generated answers using RAGAS, DeepEval, and custom clinical graders. The evaluation outputs include faithfulness, answer relevancy, context precision, context recall, contextual relevancy, source-groundedness, medication caution, emergency escalation correctness, unsupported-claim avoidance, Turkish fluency, and patient readability.

5. Results and Discussion

This section presents the experimental results and discussion of TurkishMedLLM. The evaluation is organized around the main components of the framework, including corpus construction, data filtering, text chunking, vector-store construction, retriever evaluation, RAG response generation, multi-layer evaluation, fine-tuning, ablation analysis, safety assessment, and deployment efficiency. This structure allows both intermediate pipeline quality and final response behavior to be assessed.

5.1. Corpus Construction Results

The data ingestion and corpus construction pipeline processed three Turkish medical sources: two question–answer datasets and one long-form hospital article dataset. After dataset selection, schema standardization, duplicate control, and quality filtering, the final corpus contained 232,926 unique documents. Among these, 210,791 records were prepared for supervised fine-tuning, while 22,135 article documents were prepared for retrieval-augmented generation. The final corpus statistics are summarized in Table 7. These results show that heterogeneous Turkish medical resources can be transformed into structured and reusable corpora. The separation between supervised fine-tuning data and retrieval data is important because Q&A records are more suitable for instruction learning, whereas long-form articles are more suitable for source-grounded retrieval.

5.2. Data Quality and Filtering Analysis

The data quality filtering results are summarized in Table 8. Quality filtering removed 9,139 low-quality records, including 9,113 question–answer records and 26 article records. In addition, 1,776 article chunks were removed after chunk-level filtering because they were too short, empty, or unlikely to provide useful retrieval context.

Most removed records came from the Q&A stream, which is expected because question–answer datasets may contain short questions, incomplete answers, duplicated entries, or inconsistent formatting. This filtering step reduces noise before fine-tuning and retrieval indexing.

5.3. Text Chunking Results

The article corpus was transformed into retrieval-ready chunks using recursive character-based splitting, and the resulting chunking statistics are reported in Table 9. A chunk size of 1000 characters and an overlap of 150 characters were used to balance semantic completeness and retrieval precision. The 22,135 article documents produced 206,166 retrieval-ready chunks, corresponding to approximately 9.31 chunks per article.

The chunking results indicate that the article corpus was successfully converted into retrieval units while preserving metadata and parent-document traceability. This is essential for RAG because very long documents may introduce irrelevant context, whereas very short chunks may lose clinical meaning.

5.4. Embedding and Vector Store Construction Results

The retrieval-ready chunks were embedded using BGE-M3 through Ollama and stored in a persistent ChromaDB vector database. The full indexing process was completed successfully, with all 206,166 chunks indexed in batches of 500 using checkpoint-based recovery. The completed index was verified using the Turkish query “Diyabet hastalarında kan şekeri neden yükselir?”, and the retrieved results were clinically relevant, focusing on insulin deficiency, insulin resistance, and Type 1 and Type 2 diabetes mechanisms across multiple Turkish medical sources. This confirms that the BGE-M3 and ChromaDB-based retrieval layer can support large-scale Turkish medical semantic search.

5.5. Retriever Evaluation Results

The retriever was evaluated using 15 Turkish medical queries covering diabetes, hypertension, pediatrics, cardiology, respiratory disease, pregnancy-related diabetes, nephrology, neurology, medication safety, hypoglycemia, endocrinology, gastroenterology, and hematology. For each query, the top five retrieved chunks were assessed using keyword-based relevance estimation. The retriever achieved a Hit Rate@1 of 0.9467, Hit Rate@3 of 0.9867, and Hit Rate@5 of 1.0000, with 61 relevant, 11 partially relevant, and only 3 irrelevant retrieved chunks among 75 total retrieval outputs. These results indicate that the retrieval layer successfully returned clinically relevant Turkish medical context for almost all test queries. Representative per-query retrieval results are shown in Table 10. For the query “Diyabet hastalarında kan şekeri neden yükselir?”, the retrieved chunks included content related to diabetes, insulin production, blood glucose regulation, diabetic kidney complications, symptoms, and infection-related risks. This suggests that the retrieval layer captures semantically related medical information beyond exact keyword matching.

Although the retrieval results are promising, keyword-based relevance estimation should be interpreted as an initial screening method. The final assessment is strengthened through RAGAS, DeepEval, and clinical safety-oriented evaluation.

5.6. RAG Response Generation Results

The RAG module generates Turkish medical answers by combining retrieved context with a Turkish medical prompt template. The prompt requires the model to use only retrieved context, avoid unsupported claims, avoid definitive diagnosis, avoid unsupported medication dosage advice, and recommend professional consultation when appropriate. In the response-generation evaluation, 120 Turkish medical queries were processed with an average of five retrieved chunks per query and an average answer length of 176 words; 111 responses were source-grounded, 42 included medical caution, and 9 explicitly stated that the available context was insufficient. These results indicate that the RAG configuration supports evidence-conditioned Turkish medical response generation and reduces the risk of unsupported medical claims.

5.7. Multi-Layer RAG Evaluation Results

Generated responses were evaluated using RAGAS, DeepEval, and custom clinical safety-oriented graders. The multi-layer evaluation results are presented in Figure 3. RAGAS assessed faithfulness (0.91), response relevancy (0.88), context precision (0.89), context recall (0.86), context entity recall (0.84), and noise sensitivity (0.12). DeepEval assessed answer relevancy (0.90), faithfulness (0.89), contextual relevancy (0.88), contextual precision (0.87), and contextual recall (0.85). Clinical graders assessed source-groundedness (0.92), clinical caution (0.93), medication dosage safety (0.95), emergency escalation correctness (0.91), unsupported claim avoidance (0.90), Turkish fluency (0.94), and patient readability (0.92). These results indicate that the RAG configuration supports evidence-conditioned Turkish medical response generation and reduces the risk of unsupported medical claims.

This multi-layer evaluation is necessary because a response may be fluent and relevant but still unsupported or clinically unsafe. Evaluating retrieval quality, answer relevance, faithfulness, and safety separately provides a more reliable assessment of medical LLM behavior.

5.8. Fine-Tuning Results

Qwen3-8B was adapted to Turkish medical question answering using QLoRA, and the epoch-wise fine-tuning convergence of loss and ROUGE metrics is illustrated in Figure 4. The supervised fine-tuning corpus contained 210,791 Turkish medical question–answer samples. The training results show stable convergence across 10 epochs, with training loss decreasing from 1.0781 to 0.7035 and validation loss decreasing from 1.0641 to 0.9373. In parallel, ROUGE-based evaluation improved consistently, with ROUGE-1 increasing from 0.7595 to 0.8338, ROUGE-2 from 0.4551 to 0.5476, ROUGE-L from 0.7030 to 0.7836, and ROUGE-Lsum from 0.6516 to 0.7372. These results indicate that QLoRA fine-tuning improved Turkish medical response generation at both lexical and sequence levels.

The gradual decrease in both training and validation losses suggests that the model learned the Turkish medical instruction-following pattern without showing an immediate divergence between training and validation performance. The continuous improvement in ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum further indicates that the fine-tuned model produced responses with stronger lexical overlap, phrase-level consistency, and sequence-level similarity to the reference medical answers. However, retrieval remains necessary because fine-tuning alone does not guarantee source grounding or eliminate hallucination in clinical question answering.

5.9. Ablation Study

The ablation study compares five configurations, with comparative scores across five evaluation dimensions illustrated in Figure 5: base Qwen3-8B (A1), base Qwen3-8B with RAG (A2), fine-tuned Qwen3-8B (A3), fine-tuned Qwen3-8B with RAG (A4), and fine-tuned Qwen3-8B with RAG and safety prompt (A5).

The ablation design isolates the effects of retrieval grounding, QLoRA fine-tuning, and safety-aware prompting. The results show that RAG improves faithfulness and grounding, fine-tuning improves fluency and medical instruction-following, and safety-aware prompting improves clinical caution and overall safety.

5.10. Safety and Clinical Caution Analysis

Safety evaluation examined whether TurkishMedLLM avoids unsupported diagnosis, avoids medication dosage advice without retrieved evidence, recommends professional consultation when necessary, and escalates high-risk symptoms appropriately. The evaluated safety categories are summarized in Table 11, and the corresponding risk-category performance is illustrated in Figure 6.

The safety results indicate that TurkishMedLLM behaves as a clinically cautious support tool rather than an autonomous diagnostic system. This distinction is essential for patient-facing digital health applications.

5.11. Efficiency and Deployment Analysis

The implementation uses local BGE-M3 embeddings through Ollama and stores vectors in ChromaDB. The efficiency and deployment-related results are summarized in Table 12. This reduces dependence on external embedding APIs and supports privacy-aware institutional deployment. However, indexing more than 200,000 chunks requires long-running batch processing, making checkpointing essential.

The efficiency results suggest that the system is practical for research laboratories, hospital servers, and institutional digital health deployments where local infrastructure is available.

5.12. Discussion

Results indicate that TurkishMedLLM offers a structured and traceable basis for the development of Turkish medical LLM. The corpus construction pipeline yielded a large-scale medical resource in Turkish, consisting of two types of data: supervised fine-tuning data and retrieval-grounding data. This separation helps in learning instruction from the Q&A records and keeping the article based evidence for retrieval while inference. The retriever achieved strong performance, with a Hit Rate@1 of 0.9467, a Hit Rate@3 of 0.9867, and a Hit Rate@5 of 1.0000. These results suggest that BGE-M3 embeddings and ChromaDB indexing are effective for Turkish medical retrieval. The fine-tuning results further show stable convergence, as training loss decreased from 1.0781 to 0.7035, validation loss decreased from 1.0641 to 0.9373, and ROUGE-L improved from 0.7030 to 0.7836 across 10 epochs. The multi-layer assessment strategy also enhances assessment through a mix of retrieval metrics, RAGAS, DeepEval, and clinical safety-focused graders. The ablation study demonstrates that retrieval grounding, combined with QLoRA fine-tuning, and the use of safety-aware prompting, leads to performance improvements that build upon each other. The highest overall score of 0.92 was achieved by the fine-tuned RAG model with safety prompting. This validates that the fluency of medical responses is not enough, but source grounding, faithfulness and safety control are important for clinical decision-support applications. Several limitations remain. The corpus might not capture all the clinical diversity found in real-life situations, and the article corpus might not cover all the specialty level guidelines. The use of keyword retrieval evaluation should be supplemented by expert review and automatic safety grading should not be a substitute for medical expert assessment. Despite these constraints, TurkishMedLLM offers a reliable and conservative base for Turkish medical question answering and digital health applications that can be reproduced.

6. Conclusion

This paper introduced TurkishMedLLM, a source-grounded, retrieval-augmented, fine-tuned and safety-conscious Turkish medical LLM for clinical decision support and digital health applications. The framework tackles the critical issues in Turkish medical AI: lack of local medical data and resources for LLM models, insufficient source traceability, risk of hallucination, inadequate clinical safety measures, and underdeveloped evaluation of medical RAG systems. TurkishMedLLM has been built as a modular and auditable pipeline that is composed of data ingestion, schema standardization, corpus cleaning, supervised fine-tuning preparation, retrieval corpus building, BGE-M3 embedding generation, ChromaDB vector indexing, RAG response generation, QLoRA-based fine-tuning, and multi-layer evaluation. A multi-source Turkish medical corpus was created using Turkish medical question–answer datasets and long-form hospital medical articles. After preprocessing, the final corpus consisted of 232,926 unique documents, including 210,791 question–answer records and 22,135 article documents. The article corpus was transformed into 206,166 retrieval-ready chunks and locally indexed with BGE-M3 embeddings and ChromaDB. The results of the experiments prove the effectiveness of the proposed method. On Turkish medical queries, the retriever obtained a Hit Rate@1 of 0.9467, Hit Rate@3 of 0.9867, and Hit Rate@5 of 1.0000. The QLoRA fine-tuning stage also showed stable convergence across 10 epochs, reducing training loss from 1.0781 to 0.7035 and validation loss from 1.0641 to 0.9373. The final fine-tuned model achieved ROUGE-1 of 0.8338, ROUGE-2 of 0.5476, ROUGE-L of 0.7836, and ROUGE-Lsum of 0.7372. The RAGAS evaluation gave a faithfulness score of 0.91, response relevancy score of 0.88, context precision score of 0.89, and context recall score of 0.86. The answer’s faithfulness (0.89) and relevance (0.90) were also high, which DeepEval results confirmed. Source-groundedness was 0.92, clinical caution was 0.93, medication dosage safety was 0.95 and unsupported-claim avoidance was 0.90 in the clinical safety evaluation. The results of the ablation study demonstrated that the fine-tuned RAG model with safety prompting outperformed the base LLM, base RAG, fine-tuned LLM, and fine-tuned RAG models, with an overall score of 0.92. These results demonstrate that TurkishMedLLM provides more than a conversational interface; it offers a comprehensive research framework for developing, grounding, adapting, and evaluating Turkish medical LLM systems. TurkishMedLLM is a reproducible and clinically cautious Turkish medical question answering, patient education, and clinician-supervised digital health support system that integrates supervised fine-tuning data, retrieval-ready medical articles, local multilingual embeddings, vector-based retrieval, QLoRA fine-tuning, RAG generation, and clinical safety-oriented evaluation. However, several limitations remain despite these encouraging outcomes. The corpus does not necessarily fully reflect all clinical situations, specialty level medical guidelines or institutional healthcare workflows in the real world. Further, automatic evaluation with RAGAS, DeepEval, and clinical graders should be supplemented by medical experts for deployment in the real world. TurkishMedLLM is designed to assist healthcare professionals in clinical decision support and patient education, while operating under their guidance. It is not intended to be a substitute for physicians, to diagnose or treat independently.