SumLLM: Performance Evaluation and the Judgment of Large Language Models in Bengali Abstractive News Summarization

1. Introduction

Extractive and abstractive techniques are the two primary categories of summarization methods based on the choice and arrangement of information. Like highlighting text, extractive summarizing involves selecting the most pertinent phrases from a text using particular characteristics and combining them to create a summary. In contrast, abstractive summarization mimics the process of composing a summary from an individual’s thoughts by producing new phrases[1]. Bangla abstractive summarization, the task of generating concise summaries that capture the core ideas of original texts, has historically faced multiple obstacles. These include the scarcity of high-quality annotated datasets and the limitations of conventional evaluation techniques, which rely heavily on reference-based metrics that often underestimate the generative capacity of modern Large Language Models[2]. Recent analyses of model behavior further indicate that generative systems can inherit various forms of underlying skew that influence output quality and reliability[3]. However, recent advances in neural sequence-to-sequence models with attention mechanisms have brought significant improvements in abstractive summarization for Bangla, although progress remains constrained by noisy datasets and limited evaluation standards[4].

Motivated by the rapid evolution of generative LLMs in text summarization [5] and the need for more reliable evaluation methods, this study systematically evaluates the performance of six advanced LLMs on zero-shot Bangla abstractive summarization using the ’Bengali Abstractive News Summarization’ (BANS) dataset[6]. To ensure a fair and unbiased comparison between LLM-generated summaries, we adopt a fully reference-free evaluation strategy using LLMs as calibrated judges. We introduce a hybrid evaluation methodology that integrates traditional automatic metrics, such as ROUGE[7] and BERTScore[8], with an innovative LLM-as-a-Judge framework. This approach addresses the inherent bias and quality issues present in human-written references, offering a more reliable and scalable assessment of summarization.

In this work, we benchmark six recent open source LLMs: GPT-4[9], Llama-3.1-8B[10], Mixtral-8x22B-Instruct-v0.1[11], Gemma-2-27B[12], DeepSeek-R1[13], and Qwen3-30B-A3B[14] under zero-shot conditions. Addressing these challenges, our key contributions are:

• On our Bengali News (BANS) dataset, this is the first comprehensive study evaluating LLMs specifically for summarization tasks in the Bengali language within the newspaper domain.
• We propose the first reference-free, LLMs-as-Judges evaluation framework for Bangla News summarization that eliminates dependence on noisy BANS summaries and provides a scalable, reproducible, and human-aligned evaluation method. Also, We publicly release all prompts, evaluation rubrics, and LLM-judge protocol.
• We conduct the largest zero-shot performance benchmark of six state-of-the-art LLMs on the full 19,096-article BANS dataset. No previous work evaluates summarization at this scale in Bangla. We demonstrated that zero-shot LLMs can be rival fine-tuned models, achieving high-quality summaries even without task-specific training.
• Empirical results show that modern LLMs frequently outperform human-written summaries under reference-free evaluation, establishing zero-shot LLMs as a strong baseline for low-resource summarization.
• We provide the first systematic comparative analysis of summary length, model behavior, and linguistic quality across multiple LLMs in Bengali.

2. Literature Review

Text summarization refers to the process of condensing lengthy documents into shorter versions that retain the essential information and general meaning. This task is increasingly important for efficient information consumption, given the abundance of textual data in various domains. Recent progress in large language models (LLMs) opens promising avenues for abstractive summarization of Bangla by leveraging their strong natural language generation capabilities.

Abrar et al. [15] evaluates the effectiveness of nine state-of-the-art large language models (LLMs) in summarizing consumer health queries in Bengali. The models were tested using the BanglaCHQ-Summ dataset, which contains 2,350 annotated health-related queries and their summaries. The evaluation focused on zero-shot learning, where the models generate summaries without task-specific fine-tuning. The results showed that zero-shot LLMs can deliver high-quality summaries competitive with fine-tuned models. Mixtral-8x22b-Instruct achieved the best scores in ROUGE-1 and ROUGE-L, while Bangla T5 excelled in ROUGE-2. The study highlights the potential of advanced LLMs to provide scalable, effective summarization for healthcare queries in under-resourced languages like Bangla, improving healthcare accessibility and response efficiency.

Rony et al.[16] evaluates the performance of large language models (LLMs) in summarizing Bangla texts, focusing on news article summarization. It uses two popular datasets and five different LLMs alongside human evaluations. The results show that GPT-4 performs well in generating concise and informative summaries in a zero-shot setting. The study also highlights that previous research underestimated human performance and the few-shot capabilities of language models due to low-quality references. High-quality summaries created by student writers were used for more reliable human assessment. The models were evaluated both qualitatively and quantitatively, using metrics like ROUGE and BLEU, demonstrating significant improvements over earlier methods. Sultana et al.[17] focused on abstractive summarization of news articles, with the BanglaT5 model being the top performer. Tanjila et al.[18] created the Bengali ChartSumm dataset, a benchmark dataset designed to help researchers summarize Bengali chart images into descriptive text. The BenLLM-Eval benchmark[19] evaluates the performance of large language models (LLMs) on natural language processing tasks in Bengali.The study assesses three LLMs, GPT-3.5, LLaMA-2-13b-chat, and Claude-2, across seven Bengali NLP tasks. Results show mixed results, with zero-shot LLMs performing above state-of-the-art fine-tuned models. Talukder et al.[20] proposed an abstractive summarization approach using sequence-to-sequence Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Their model employs a bidirectional LSTM encoder-decoder architecture with attention mechanisms to capture contextual relationships within Bengali text. This design allows the model to generate new sentences that effectively summarize the original content, rather than simply extracting existing sentences. Fabbri et al.[21] critically examine the effectiveness of existing automatic metrics to evaluate neural text summarization systems. Recognizing that widely used metrics like ROUGE and BLEU often fail to align with human judgments, the authors conduct a large-scale, systematic comparison of 14 evaluation metrics using outputs from 23 neural summarization models on the CNN/DailyMail dataset. Their methodology involves collecting both expert and crowd-sourced human ratings on key summary quality dimensions coherence, consistency, fluency, and relevance providing a robust benchmark for metric validation. Ahmed et al.[22] uses GPT-based Codex models for project-specific code summarization tasks using few-shot training techniques. It shows that fine-tuned Codex models produce more accurate, context-aware summaries, capturing code snippet functionality and intent better. This approach offers a scalable solution to improve code documentation, maintenance, and comprehension in real-world software development environments.

3. Methodology

In this investigation, we systematically examine the zero-shot summarization capabilities of modern Large Language Models (LLMs) for Bangla news articles, with a primary focus on overcoming the limitations inherent in existing evaluation frameworks. Our methodology is designed to critically assess model performance while addressing the fundamental issue of unreliable reference summaries. We used 6 LLMs for this study.

3.1. Experimental Dataset

The Bengali Abstractive News Summarization (BANS) dataset [6], collected from the online news portal bangla.bdnews24.com[23], has articles and news summaries with 19,096 each. Table 1 demonstrates the overview of the BANS dataset.

We utilized the full test set of the BANS dataset, comprising 19,096 Bangla news articles spanning multiple domains, including politics, sports, business, and entertainment. Each article is accompanied by a single abstractive summary written in human language. To ensure robustness of the evaluation, we implemented strict pre-processing, including unicode normalization, removal of special characters, and consistent sentence segmentation in all texts. Table 2 shows sample articles and their corresponding human-written summaries from the BANS dataset.

3.2. Model Selection and Configuration

We selected six state-of-the-art LLMs representing diverse architectural approaches and capability levels. Using these six open-source models, we evaluated and compared the performance of zero-shot LLMs on the BANS dataset.

• GPT-4: A powerful language model in the OPENAI GPT series excels in multistep reasoning, delivering accurate output in summarizations, text generation, and question answers, with a notable strength in tasks that require factual consistency.
• Llama-3.1-8B: This compact model from Meta has 8 billion parameters, 40 attention heads, a vocabulary of 128,000 tokens, and supports 8,192-token context windows. It offers competitive summarization quality through effective alignment with task instructions.
• Mixtral-8x22B-Instruct-v0.1: Mistral AI uses eight specialized components with 22 billion parameters. Mixtral combines a mixture-of-experts (MoE) design with large instruction-tuned capacity, activating only a subset of experts per input.
• Gemma-2-27B: Gemma-2-27B is a 27 billion multilingual transformer tailored for instruction following and content summarization.
• DeepSeek-R1-Distill-Llama-70B: DeepSeek-R1-Distill-Llama-70B is a large-scale distilled variant of the Llama-70B model, optimized for efficiency without sacrificing performance. The model delivers factually consistent and fluent outputs in multilingual settings.
• Qwen3-30B-A3B: Qwen3-30B-A3B is a high-capacity 30 billion-parameter multilingual model with strong optimization for Asian languages. Its scale and fine-grained instruction tuning allow it to generate precise summaries with strong semantic relevance.

3.3. Prompt Design and Evaluation

We designed a zero-shot framework to represent the performance of Large Language Models effectively on the BANS dataset. This method shows how well these models generalize to domain-specific tasks without fine-tuning, providing insights into their capabilities in summarizing Bengali News. Initially, without fine-tuning LLMs, this makes a large summary or out-of-context answers for some particular models like Qwen3 and Deepseek-R1. With the prompt design techniques, this generalized and concise summary has more context than before. Initial experiments revealed that some models (e.g., Gemma-2, Mixtral) defaulted to English outputs. To ensure Bangla summaries, we explicitly instructed the model in the prompt to respond in Bangla. Finally, we used the following organized prompt in to obtain responses from the LLMs to generate precise responses from the input queries:

3.4. Traditional Metric Assessment

For comparative purposes with prior work, we also computed standard metrics, including ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (RL), and BERTScore, in a zero-shot setting. For this, we evaluate the performance among all open-source large language models on the BANS dataset. This assessment indicates how well the LLMs can perform on Bangla summarizing.

3.5. LLMs-as-a-Judge Evaluation Framework

We introduce a novel LLM-as-a-Judge framework to overcome the fundamental limitation of unreliable references. This approach directly assesses the quality of the summary against the source article, bypassing the need for human-written references. The framework operates as follows:

3.5.1. Judge Selection and Calibration

The summaries are independently evaluated by LLMs, which assess each summary based on three criteria: faithfulness, coherence, and relevance. We selected our six diverse LLMs as judges: Deepseek-R1, Gemma-2, GPT-4, Llama-3.1, Mixtral, and Qwen3. To calibrate these judges and establish a benchmark, our proposed LLMs also evaluated ground truth summaries based on the defined criteria. The LLM judges were then calibrated on these human summaries to ensure consistent and reliable rating behavior.

3.5.2. Evaluation Criteria

Judges assessed each summary along three dimensions using detailed rubrics:

• Faithfulness (Factual Consistency): Measures the alignment between summary content and facts from the source article. The summary represents the actual or main text whether it is trustworthy or misleading information [24].
• Coherence (Readability): Coherence assesses logical flow, grammatical correctness, and overall clarity. It contains claims directly supported by the source document[25].
• Relevance (Informativeness): Relevance evaluates the inclusion of key information and exclusion of irrelevant details. The summary should condense the source by including only its essential information [25].

3.5.3. Evaluation Protocol

For each article-summary pair, judges received the source article, the model-generated summary, detailed evaluation guidelines, and a structured response format. Judges provided numerical ratings with justifications. The final score for each summary was calculated as the average across all three metrics, thereby mitigating the bias of relying on a single model.