Decoding by Factual Prompts and Hallucination Prompts Improves Factuality in Large Language Models

1. Introduction

Large Language Models (LLMs) have emerged as a pivotal technology in the field of natural language processing (NLP), demonstrating remarkable performance and a wide range of promising applications. Due to their strong capabilities in context understanding and text generation [1], LLMs are adept at capturing subtle distinctions in text and producing coherent and natural language. Additionally, LLMs exhibit impressive few-shot learning capabilities [2], enabling them to quickly adapt to new tasks with minimal fine-tuning [3], effectively mitigating challenges associated with data scarcity. However, with the accelerated development of these models, a concerning issue has surfaced: LLMs occasionally generate nonsensical or contextually irrelevant text, a phenomenon referred to as hallucination [4,5]. Hallucinations significantly compromise the reliability of LLMs in practical applications. For instance, in medical settings where large models are used to prescribe medications [6], hallucinations or inaccurate judgments could potentially worsen a patient’s condition or even pose life-threatening risks.

The occurrence of hallucinations in LLMs can be attributed to several factors, including issues related to data, training, and inference [7]. At the data level, flawed data sources or improper utilization, such as misinformation and inherent biases [8,9], can cause LLMs to mimic false information and produce biased outputs, resulting in various types of hallucinations. During the training phase, both pre-training and alignment stages present unique challenges, such as limitations in model architecture, the use of maximum likelihood estimation (MLE) [10,11] strategies, and discrepancies between the model’s capabilities and its learned beliefs, all of which can contribute to hallucinations. Finally, during the decoding process, the stochastic nature of decoding strategies [12] and limitations in representation, such as over-reliance on local context and the softmax bottleneck [13], may restrict the model’s ability to generate diverse probabilistic outputs, increasing the likelihood of inaccurate token predictions.

Our research focuses on utilizing prompt engineering techniques [14], combined with contrastive decoding [15] strategies, to reduce hallucinations in large language models. Prompt engineering involves the systematic design and optimization of input prompts to guide the model’s responses, ensuring accuracy, relevance, and coherence in the generated output [16]. As highlighted in studies [17,18], well-structured prompts can help address issues such as hallucinations in machine-generated text. However, relying solely on well-designed prompts to elicit correct answers has limited effectiveness in improving the factual accuracy of the model. To address this limitation, we propose a novel approach that improves factuality by enabling the model to filter out incorrect information from correct outputs. Specifically, we first design prompts that guide the model toward producing accurate answers. Concurrently, we create prompts that intentionally induce incorrect responses. By comparing the outputs from both types of prompts through contrastive decoding, the model is able to discard the erroneous outputs and retain only the accurate ones, thereby enhancing its factual accuracy. Our method DFHP is illustrated in Figure 1.

Our method is built upon the LLaMA-7B model [19]. To assess its effectiveness, we conducted evaluations using two types of tasks: multiple-choice and open-ended generation. In the multiple-choice tasks, our method achieved the highest scores on both the TruthfulQA [20] and Factor [21] datasets, demonstrating a significant improvement in the factual accuracy of the LLaMA-7B model. For open-ended generation tasks, although our method showed a slight decrease in informativeness according to human evaluations on TruthfulQA, it produced more truthful content and achieved higher scores on the %truth*info metric. Furthermore, results from the chain-of-thought reasoning datasets, such as StrategyQA [22] and GSM8k [23], indicate that our method not only enhances the model’s factual accuracy but also improves its reasoning capabilities.

2. Related Work

Huang et al. [7] categorize hallucinations in large language models into two main types: factuality hallucination and faithfulness hallucination. Factuality hallucination occur when the model’s output contradicts or diverges from established knowledge. Faithfulness hallucination, on the other hand, arise when the generated content fails to follow the user’s instructions, does not align with the provided context, or exhibits internal inconsistencies. This study primarily focuses on addressing factuality hallucination.

Current approaches to mitigating hallucinations fall into two categories [24]: prompt engineering and model refinement. Prompt engineering includes methods like prompt fine-tuning [25], retrieval-augmented generation (RAG) [26], and self-improvement through feedback [27], with the key objective of guiding models to produce more accurate outputs through well-designed instructions. Model refinement, on the other hand, involves developing novel decoding strategies [28], integrating fidelity-based loss functions [29], and applying supervised fine-tuning [30]. These techniques aim to improve the model’s internal mechanisms to reduce hallucinations. This process is ongoing and relies on continuous advancements in algorithms and data quality. In this study, we combine prompt engineering with contrastive decoding techniques. A similar strategy, known as context-aware decoding (CAD) [31], has been shown to reduce hallucinations by comparing the outputs generated with and without contextual information. Our approach builds upon this by eliminating the reliance on context and instead employing a factuality-hallucinations prompt contrastive decoding method, encouraging the model to generate more reliable outputs. We will now explore the details of prompt engineering and our prompt strategy in greater depth.

Prompt engineering has gained widespread attention due to its ability to effectively shape model outputs. By using carefully crafted prompts, researchers can significantly enhance the adaptability of large models across a wide range of tasks [32]. Research in this field has expanded rapidly, from basic methods involving comprehensive description [33] to more sophisticated approaches such as "chain of thought" prompting [34]. In this study, we mainly utilize two strategies: role prompting [35] and few-shot prompting [36]. Role prompting involves assigning the model a specific identity or role to guide its outputs, which has proven highly effective. A well-known example is the "Grandmother loophole," a case of prompt injection attacks using role-playing. Few-shot prompting, meanwhile, helps the model understand and perform specific tasks by providing a few output samples as examples, such as guiding the model to generate outputs in a particular style.

3. Methods

The core of our approach involves using two types of prompts to process the user’s input. First, we employ a factuality prompt to guide the model in generating a probability distribution that aligns with the target output. Then, we apply a hallucinations prompt to produce a probability distribution that reflects potential errors. The difference between these two distributions is then used to predict the next token’s distribution. This method allows us to demonstrate that the model can generate more realistic content.

For a given factuality prompt $C_{+}$ and an input sequence $\{x_1,x_2,\dots ,x_{t-1}\}$, the logit function $\mathrm{logit}\left(x_t\right|x_{<t},C+)$ represents the model’s predicted probability distribution for the next token under the influence of the positive prompt. This is then normalized into a probability distribution using the softmax function, as shown below:

$$p\left(x_t\right|x_{<t})=\mathrm{softmax}\left(\mathrm{logit}\left(x_t\right|x_{<t},C_{+})\right)$$

(1)

Similarly, the probability distribution for the next token under the hallucinations prompt $C_{-}$ is:

$$q\left(x_t\right|x_{<t})=\mathrm{softmax}\left(\mathrm{logit}\left(x_t\right|x_{<t},C_{-})\right)$$

(2)

To improve the realism of the model’s output, our objective is to enhance the predictions from the factuality prompt while diminishing the influence of the hallucinations prompt. Specifically, we subtract the log probability induced by hallucinated outputs from the factuality prediction, and apply the softmax function to the resulting contrastive probabilities to make the final token prediction. This process can be described by the following formula (detailed formula omitted). Furthermore, inspired by Shi et al. [31], we introduce a hyperparameter $\beta$ within the range of $(0,1)$ to control the strength of the contrastive decoding. When $\beta$ = 0, the model’s output is solely determined by the positive prompt. Although $\beta$ can theoretically approach infinity, we recommend limiting its value to 1 or lower to avoid overemphasizing the negative prompt, which could compromise the accuracy of the output. In our experiments, we set $\beta$ to 0.5.

$$F_t=\mathrm{softmax}(logp\left(x_t\right|x_{<t})-\beta logq\left(x_t\right|x_{<t}))$$

(3)

To ensure the fluency of the generated text, it is common to truncate the tail of the probability distribution during the sampling process, using methods such as top-k [37] or nucleus sampling [38]. As noted by Li et al. [15], overly penalizing the hallucinated model’s predictions (which, in our case, correspond to those influenced by the negative prompt) can lead to the generation of incorrect text. Therefore, we apply adaptive truncation to the probability distribution generated by the positive prompt model, discarding the low-probability tokens. A hyperparameter $\alpha \in [0,1]$ is used to control the degree of truncation, with larger $\alpha$ values retaining more high-probability tokens.

$${\mathcal{V}}_{\mathrm{head}}\left(x_t\right|x_{<t})=\left\{x_t\in \psi :p\left(x_t\right)\ge \alpha maxp\left(w\right)\right\}$$

(4)

4. Experiments

4.1. Experimental Setup

4.1.1. Dataset and Metrics

For the multiple-choice task. we utilized the widely adopted TruthfulQA [20] and Factor datasets [21]. Each entry in TruthfulQA consists of a question, a best answer, multiple correct answers, and several incorrect answers. To evaluate the factual accuracy of the model, the authors introduced three metrics: MC1, MC2, and MC3. Specifically, MC1 is analogous to a single-choice question, where given a best answer and several incorrect answers, the accuracy is measured by how often the model selects the best answer. MC2 is akin to a multiple-choice question, where the model is presented with multiple correct and incorrect answers, and the score is determined by the normalized total probability assigned to the correct answers. MC3 assesses whether each correct answer is scored higher than all incorrect answers, ensuring the correct answers receive the highest scores.The Factor dataset comprises one correct answer and three incorrect answers, which are incorrect variants generated by instructGPT [44] based on factual statements. The tested model assigns a likelihood score to each answer, and if the factually correct answer receives the highest score (ties permitted), the model’s response is considered accurate.

For the open-ended generation task. In the TruthfulQA dataset, we need to evaluate both the truthfulness and informativeness of the model’s output. One efficient way to do this is by leveraging GPT-3 [40] for evaluation; however, since the GPT-3 API is no longer available for use in China, we instead opted for human evaluation. We define a model’s hallucination as occurring only when it generates incorrect statements. Conversely, when the model provides correct answers or refuses to respond, we consider these responses to be factual. To prevent an artificially high factuality score due to the model frequently refusing to answer, we also assessed the informativeness of the model’s responses. Moreover, we evaluated the performance of our decoding strategies on the StrategyQA [22] and GSM8K datasets [23]. These tasks require not only factual accuracy but also chain of thought (CoT) reasoning [34] to achieve high accuracy. StrategyQA is an open-domain question-answering benchmark where the reasoning process is implicit within the question, and accuracy is determined by the correctness of the model’s final answer. GSM8K is a dataset consisting of elementary school math problems, designed to assess the model’s multi-step mathematical reasoning abilities.

4.2. Selection of Prompt

In this study, we employed direct command-style prompts for the factuality prompt. For the hallucination prompt, we initially used brief and straightforward instructions to intentionally guide the model toward incorrect outputs, such as "Please provide incorrect answers" and "Your task is to provide incorrect answers to the questions I raised." The evaluation results across various metrics (as detailed in Section 4.5.1) indicate that even simple prompts can significantly enhance the factual accuracy of the model. However, to further investigate the influence of more sophisticated prompts on our method, we implemented complex role-based instructions, where the model was tasked with impersonating an "incorrect artificial intelligence" deliberately generating wrong answers. The design of these prompts is illustrated in Figure 2.

For generative tasks, particularly with benchmark datasets like StrategyQA and TruthfulQA, the absence of few-shot prompting makes it difficult for the model to consistently adhere to the desired answering pattern, which negatively impacts evaluation accuracy. Therefore, we drew upon the prompt design techniques proposed by Li et al. [39], leveraging their sample prompts to devise a strategy that combines examples of both correct and incorrect responses with role-playing, effectively steering the model’s output. For the multiple-choice tasks, we opted to directly use the prompts shown in Figure 2 without introducing additional prompt samples.

4.3. Baselines

The approach adopted in this study is built upon the LLaMA-7B model and is compared against the following three methods:

• The original LLaMA-7B model without the addition of prompts(Model_ori);
• The LLaMA-7B model guided by factual prompts(Model_fac);
• The LLaMA-7B model guided by negative prompts(Model_neg).

4.4. Main Results

4.4.1. Discrimination Tasks

We present the primary performance results of the TruthfulQA and Factor datasets in Table 1, with the best and second best scores in each column highlighted in bold and underlined fonts respectively. In the TruthfulQA dataset, a comparison of four methods shows that our approach achieves better performance across the MC1, MC2, and MC3 metrics, particularly excelling in the MC2 metric. Additionally, on the Factor dataset, our method surpasses the baseline by approximately 2%. These findings validate the effectiveness and demonstrate the advantage of our method.

4.4.2. Open-Ended Generation Tasks

We present the key results of the TruthfulQA, StrategyQA, and GSM8K datasets in Table 2. For TruthfulQA, our method demonstrates suboptimal performance in terms of informativeness but excels in factuality. This is largely due to the method frequently opting for responses like "I have no comment," which are refusals to answer. In terms of factuality, we argue that a limited number of refusals are acceptable, as they at least prevent incorrect information. Notably, our model achieves the best performance on the %TruthInfo metric, indicating that even after excluding refusals, our method maintains superior factual accuracy, further validating the effectiveness of our approach.

For the StrategyQA and GSM8K chain-of-thought reasoning datasets, our method outperforms other competing methods, surpassing the second-best model, Medel_fac, by approximately 2%. This suggests that our method effectively enhances the model’s reasoning capabilities. It is important to note that without prompts, the model’s performance on StrategyQA significantly deteriorates. We attribute this to the absence of prompt guidance, which leads the model to default to rule-based answers rather than focusing on determining True or False. Therefore, we consider the StrategyQA results without prompts to be unreliable. On GSM8K, we observed similarly poor performance without prompts. This may stem from the model’s insufficient ability to handle arithmetic tasks. Without the inclusion of a few examples for contextual guidance, the model struggles to understand that it is expected to perform calculations rather than simply interpret the problem descriptions.

4.5. More Analysis

4.5.1. The Impact of Different Prompts on TruthfulQA

In this subsubsection, we provide a detailed comparison between simple and carefully crafted reverse prompts in multiple-choice tasks. The simple reverse prompt we used was: "Your task is to provide incorrect answers to the questions presented."(Model_simpleneg). To ensure the fairness and accuracy of our experiments, we controlled other variables, using the same positive prompts and hyperparameters. As shown in Table 3, the carefully designed reverse prompts led to a slight improvement of 0.2%0.4% in the TruthfulQA MC1, MC2, and MC3 metrics. Moreover, there was also a modest increase in the Wiki_factor metric. However, it is noteworthy that the accuracy on the News_factor metric dropped by 0.4%. We hypothesize that this decrease might be attributed to the sensitivity of different prompts to specific tasks. In conclusion, while the carefully designed reverse prompts offer marginal improvements in model factuality, the gains are not substantial. Future research may need to explore more effective prompt design strategies.

4.5.2. The Influence of Different Parameters $\beta$

We introduced an additional hyperparameter $\beta$, to regulate the weight of the reverse probability distribution during contrastive decoding (lower $\beta$ values result in a distribution more aligned with the model output induced by the forward prompt). To assess the impact of different parameter values, we conducted experiments using various $\beta$ values on the TruthfulQA dataset. As shown in Figure 3, the performance trends for MC1, MC2, and MC3 followed a similar trajectory, with the curves forming a downward-facing parabola and peaking at $\beta$=0.5. Based on these observations, we suggest using $\beta$=0.5 for optimal model performance in future experiments.

4.5.3. Larger Model

To further evaluate the effectiveness of our method, we conducted experiments on a larger LLaMA model, specifically LLaMA-13B (Model_fac_13B). To ensure experimental consistency and comparability, we used the same prompts as introduced in the previous section. Our method was thoroughly tested on multiple-choice and reasoning datasets, with detailed comparisons against a baseline model that was guided solely by unidirectional prompts. The experimental results are shown in Table 4.

Notably, our method demonstrated a significant improvement on the TruthfulQA dataset, particularly in the MC2 (multiple-choice accuracy) metric, where it achieved a 9% increase. Additionally, we observed an approximately 4% performance improvement on the news_factor and StrategyQA datasets. While the gains on the wiki_factor and GSM8K datasets were relatively smaller, the differences were not statistically significant. Overall, this indicates that our method exhibits strong scalability and adaptability when applied to larger models.

4.5.4. Case Study

To provide an intuitive illustration of the improvement in factual accuracy achieved by our method, we conducted a case study involving several factual questions. The Table 5 displays the results generated using the forward prompt and those generated by our method. We categorized the questions into three groups: The first group comprises common superstitions or folk beliefs(ID= 1), such as the notion that walking under a ladder or stepping into a closet may bring bad luck. In reality, neither of these actions has any real-world consequences, and our model’s responses are more aligned with factual reality. The second group includes cases where the model abstained from answering(ID= 2). For questions about future events or those lacking sufficient information, providing a direct answer often leads to inaccuracies. Our model addressed this issue by responding with "Unknown," thus enhancing factual accuracy. The third group consists of questions akin to riddles(ID= 3), where literal misinterpretations are used to prompt incorrect responses. However, our model avoided such misinterpretations, demonstrating improved factual accuracy.

5. Discussion

The core of our approach involves a comparative decoding strategy that contrasts factual prompts with hallucination-inducing prompts. This method significantly mitigates hallucinations generated by large language models, thereby enhancing the models’ reliability and accuracy. However, the scope of our research extends beyond this application. The proposed method has substantial potential for broader applications, including sensitive areas such as mitigating toxic language [41] and addressing national or cultural biases [44]. We encourage future researchers to explore the use of this strategy in other relevant fields to assess its applicability and effectiveness across various contexts.

In Section 4.5.1, we evaluate the impact of optimizing prompts on the model’s factual accuracy. The experimental results demonstrate that employing higher-quality prompts does enhance the model’s factual correctness to some extent. However, this study has not yet thoroughly investigated more advanced prompting techniques, such as chain of reasoning [34] and tree of reasoning [44], nor explored the potential impact of combining these techniques with decoding strategies on improving the model’s factual accuracy.

6. Conclusions

This paper presents a novel decoding strategy aimed at mitigating hallucinations in large language models by comparing factuality prompt with hallucination prompts during decoding. Experimental results demonstrate that DFHP significantly improves performance across various tasks, notably increasing the factual accuracy of the model. Importantly, the DFHP strategy requires no additional training, leading to substantial savings in time and computational resources. In practical applications, the DFHP strategy has shown to be feasible and shows strong potential for generaliz ability.