HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor

1. Introduction

Large Language Models (LLMs) have become a cornerstone technology of the new generation of artificial intelligence, making previously unattainable applications like automatic programming increasingly commonplace. However, the safety concerns of LLMs remain a significant challenge. To enhance LLM safety, researchers have employed alignment training to instill human preferences—training models to refuse rather than execute harmful instructions [1]. Recent studies [2,3], however, have revealed a critical limitation in this approach: LLMs’ safety appears to be predominantly controlled by refusal behavior, but only generalizes to the initial tokens of their refusal responses. This vulnerability is particularly concerning as it enables attackers to easily manipulate model responses by simply inserting affirmative prefixes, effectively bypassing safety measures to elicit harmful responses from safety-aligned models, which is known as prefix injection attack [4,5].

Unfortunately, prefix injection attacks are almost inevitable due to several factors: 1) Although safety-trained models can refuse harmful instructions, the mere act of beginning a response with an affirmative prefix is not inherently harmful; 2) Some training tasks may require models to follow user instructions to begin responses with specific prefixes (e.g., JSON format), further increasing the risk of prefix injection; 3) Attackers can enhance the success rate of prefix injection through adversarial techniques [5,6]; 4) Most critically, when models are white-box or providers support assistant prefilling, attackers can directly modify LLM response prefixes. Consequently, as long as model safety relies on refusal prefixes, the risk of prefix injection jailbreak attacks remains inevitable. This leads to our core research question: Can the safety of LLMs be enhanced by reducing their reliance on refusal prefixes?

To address this challenge, we introduce HumorReject, an innovative data-driven approach that employs humor as an indirect refusal strategy to deflect harmful instructions. Our choice of humor as the core solution offers two distinct advantages: 1) it provides a way to generate harmless responses without explicit refusal, and 2) it preserves natural conversational flow even when faced with injected affirmative prefixes, as the seamless transition to humorous deflection maintains contextual coherence. Specifically, we constructed a HumorReject preference dataset comprising 200 harmful and 200 benign samples. We demonstrate that by applying the existing alignment algorithm [7] with merely 10 epochs of fine-tuning on this dataset, we can fundamentally enhance model safety, even on previously unsafeguarded LLMs [8]. As illustrated in Figure 1, our approach proves remarkably effective - even when directly prefilled with an affirmative prefix, the model successfully evades harmful instructions by generating witty, humorous responses, establishing HumorReject as a compelling alternative to traditional refusal training.

To thoroughly evaluate our approach, we formulate six key research questions (RQs): RQ 1: How effectively does HumorReject decouple safety from refusal prefix? RQ 2: How effectively does HumorReject defend against prefix injection attacks? RQ 3: Beyond prefix injection, do other types of attacks still pose threats to model safety? RQ 4: Does the HumorReject approach introduce new security risks? RQ 5: Does HumorReject affect the model’s performance on benign inputs? RQ 6: Why did previous humorous LLMs not demonstrate good safety? We will address these questions in § Section 4.1 through § Section 4.6. These questions guide our comprehensive evaluation of HumorReject’s effectiveness and resilience.

While our approach presents a significant advancement in enhancing LLM safety, we acknowledge certain limitations. Firstly, investigating whether LLMs can possess a sense of humor akin to humans is beyond the scope of this paper, as it pertains to broader philosophical and cultural considerations. Instead, we utilize humorous rejection solely as a strategy to decouple safety from refusal prefixes. Secondly, while humorous replies provide harmless responses without direct refusal, we recognize that humor is not the only possible solution for this decoupling. Future research may explore alternative rejection strategies that could complement or surpass the effectiveness of humor in ensuring model safety.

In summary, the main contributions of this paper are as follows:

• We propose a novel indirect refusal strategy based on humorous responses, which can effectively decouple LLMs’ safety from refusal prefixes (§Section 4.1), significantly lowering the risk of prefix injection attacks;
• We construct and publicly release the HumorReject preference dataset of 400 samples, and demonstrate that using existing alignment algorithm [7] with just 10 epochs of fine-tuning on this dataset can fundamentally enhance the safety of the previously unsafeguarded Mistral-7B-instruct-v0.1 model (§Section 4.2). This effective result indicates that existing alignment algorithms are sufficient for producing highly safe models—innovations at the data level are even more fundamental than the alignment algorithm itself in achieving effective LLM safety;
• Beyond prefix injection attacks, we conduct extensive security evaluations of the HumorReject model through various attack vectors, including mismatched generalization attacks (§Section 4.3) and our novel adaptive attack “HumorDAN” (§Section 4.4). Our experimental results demonstrate the model’s robust resistance against these diverse attack strategies;
• We also perform in-depth analysis of model usability and find that previous defense methods suffer from over-defense issues: 1) models generate refusals even for benign inputs [2,11], and 2) response quality significantly deteriorates under harmful context conditions [12]. HumorReject training effectively avoid these problems (§Section 4.5).

Through this research, we aim to enhance the safety of current LLMs and provide new perspectives for future work in this direction.

2. Related Work

2.1. LLM Alignment

Aligning LLMs with human preferences has predominantly relied on supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) [13]. Direct Preference Optimization (DPO) [14] streamlines this process by directly optimizing for human preference scores. ORPO [7] unifies SFT and preference alignment into a single process, while KTO [15] introduces a human-aware loss function based on prospect theory for binary feedback.

Despite these advancements, LLMs remain vulnerable to jailbreak attacks. To mitigate such risks, researchers have recently explored novel training approaches. The Circuit Breaker [12] prevents harmful outputs by redirecting internal representations to an orthogonal space. To combat "shallow safety alignment" where safety measures only appear in initial tokens, Qi et al. [2] augment training data with harmful instruction-response pairs followed by refusals. Decoupled Refusal Training (DeRTa) [11] enables models to refuse harmful content at any position through combined Maximum Likelihood estimation and Reinforced Transition Optimization. While these methodologies demonstrate remarkable effectiveness in enhancing LLM safety beyond traditional alignment approaches, our analysis in §Section 4.5 reveals their susceptibility to over-defense issues.

2.2. Jailbreak Attacks

Numerous studies [5,6,10,16,17,18,19,20] have demonstrated LLMs’ vulnerability to jailbreak attacks that exploit their training and alignment weaknesses to elicit harmful outputs. Wei et al. [4] categorized these attacks into two types: competing objectives and mismatched generalization.

Competing objectives attacks emerge when a model’s pretraining and instruction-following objectives conflict with its safety objectives. A notable example is prefix injection [5,6], where attackers use affirmative prefixes to misalign model behavior, exploiting the tension between instruction-following and safety constraints.

Mismatched generalization attacks [21,22,23,24] occur when inputs fall outside the model’s safety training distribution while remaining within its pretraining scope. For instance, CodeAttack [22] elicits harmful outputs by prompting responses in out-of-distribution formats like code blocks, highlighting the limitations of current safety measures against diverse attack vectors.

2.3. LLM with Humor

Recent research on LLMs has increasingly focused on their ability to understand and generate humor. In the realm of humor generation, Zhong et al. [25] introduced the Creative Leap-of-Thought (CLoT) paradigm, which significantly enhances LLMs’ humor generation capabilities. Similarly, Tikhonov et al. [26] explored multi-step reasoning for generating stand-up jokes, further advancing the humor creation abilities of LLMs. Chen et al. [27] constructed a large-scale Chinese humor response dataset and designed auxiliary tasks to improve pre-trained language models’ ability to generate humorous responses. Vikhorev et al. [28] developed the CleanComedy corpus, employing data filtering and model training to create non-toxic, humorous content while analyzing the effectiveness and limitations of these methods. Additionally, Chen et al. [29] proposed a multi-stage curriculum learning framework that significantly enhances LLMs’ ability to generate puns by optimizing both pun structure preferences and humor preferences. Overall, these works demonstrate that LLMs have already developed preliminary capabilities in humor understanding and generation.

3. HumorReject Training

In this section, we will introduce the training details of HumorReject, including training dataset construction (§ Section 3.1) and training settings (§ Section 3.2).

3.1. Training Dataset Construction

Overview. We construct a preference dataset for HumorReject alignment training, which comprises both harmful and benign instructions. Specifically, we extract 200 samples from the AdvBench [5] dataset as harmful instructions and 200 samples from the Alpaca [30] dataset as benign instructions. As illustrated in Figure 2, for harmful instructions, the chosen response is a humorous reply generated by GPT-4o prompted for humor, while the rejected response is from an uncensored LLM (we use Mistral-7B-instruct-v0.1). Conversely, for benign instructions, the chosen response is the standard reply from the uncensored LLM, whereas the rejected response is a humorous reply generated by GPT-4o. The construction of this preference dataset aligns with and reflects our training objectives:

1) For harmful instructions, we aim to train the model to generate humorous responses instead of harmful ones, thereby decoupling safety from refusal prefixes.

2) For benign instructions, we want the model to maintain its normal functionality and avoid unnecessary humorous responses.

3) Through this preference dataset, we can effectively guide the model to learn when humor is appropriate as a safety mechanism and when it should provide standard responses.

Non-Refusing but Harmless Humorous Responses. To ensure that humorous responses do not resemble refusal-based prefixes, we prompt GPT-4o to generate responses that are humorous but free of explicit rejections or negative tones. These responses are designed to defuse potential harm while maintaining a natural conversational flow. To verify their harmlessness, we use the Claude-3.5-sonnet [31] model as the safety judge, ensuring that all generated responses are thoroughly safe. Any response flagged as harmful is regenerated until it fully complies with harmlessness standards.

3.2. Training Settings

We employ the ORPO [7] alignment algorithm for training our models. The training is conducted on 4 NVIDIA RTX 4090 GPUs with a batch size of 4 and a learning rate of 5e-5, utilizing a cosine learning rate scheduler. We adopt Low-Rank Adaptation (LoRA) [32] with rank 8 for efficient parameter-efficient fine-tuning. The training process spans 10 epochs and is implemented using the open-source LLaMA Factory [33] codebase. We conducted training on two instruct-tuned LLMs: Llama3-8B-instruct [9] and Mistral-7B-instruct-v0.1 [8] to producing their HumorReject variants.

4. Empirical Studies

This section presents empirical studies designed to address the key research questions outlined in §Section 1. We first detail our experimental setup, including the test dataset, baseline defense strategies, and evaluation metrics.

Test Dataset. We constructed a test dataset containing 100 harmful instructions by selecting 50 samples each from AdvBench [5] and HarmBench [34] to evaluate safety performance against harmful inputs. Additionally, each instruction in the test set is paired with a positive prefix (e.g, "Sure, here is the method to make a bomb"), which serves as the target for prefix injection attacks.

Baseline Defense. In addition to the vanilla model, we incorporate three different defense training strategies as baselines: Circuit Breaker (CB) [12], Qi et al.’s augmentation [2] (in our paper we refer to it as DeepAug), and DeRTa [11]. CB employs a forgetting mechanism to make the model unlearn how to effectively execute harmful instructions, DeepAug utilizes data augmentation, and DeRTa proposes a Reinforced Transition Optimization training method to enable model rejection at any position.

Evaluation Metrics. We design three different judge prompts, namely Safety Judge, Rejection Judge, and Humor Judge. The Safety Judge is used to determine whether the model generates harmful instructions, the Rejection Judge is used to determine whether the model generates refusal prefixes, and the Humor Judge is used to determine whether the model generates humorous replies. Based on these judges, we define three metrics: Safety Rate, Humor Rate, and Reject Rate, each ranging from 0% to 100%, representing the proportion of responses classified as safe, humorous, and refusal-based, respectively. In our experiments, we utilize Claude-3.5-sonnet [31] to implement these judge models.

4.1. RQ1: How Effectively Does HumorReject Decouple Safety from Refusal Prefix?

To assess whether HumorReject enables the model to generate humorous and harmless responses to harmful instructions instead of refusal statements, we evaluated it using the 100 harmful instructions from our test dataset.

As shown in Table 1, the baseline model, LLaMA3-8B-instruct Vanilla, exhibited a high Reject Rate of 96% and a Safety Rate of 97%, indicating a strong reliance on refusal prefixes when responding to harmful instructions. Besides, it did not generate any humorous responses (Humor Rate of 0%). In contrast, the HumorReject model achieved a remarkable Humor Rate of 95%, showcasing its ability to produce humorous and harmless replies. The Reject Rate plummeted to 2%, demonstrating a minimal dependence on traditional refusal prefixes. Additionally, the Safety Rate remained at 100%, ensuring that all responses were safe and devoid of harmful content. These results affirm that HumorReject successfully decouples safety mechanisms from refusal prefixes by leveraging humor.

4.2. RQ2: How Effectively Does HumorReject Defend Against Prefix Injection Attacks?

In §Section 4.1, we initially validated the safety of the HumorReject model against direct harmful instruction inputs. Building on this foundation, we now assess the robustness of HumorReject against prefix injection attacks.

We conducted experiments on two models: Llama3-8B-instruct and Mistral-7B-instruct-v0.1. We tested five distinct types of prefix injection attacks—GCG [5], AutoDAN [6], Template [10], Prefill, and Template+Prefill—and employed five defense strategies for each attack type, including Vanilla, Circuit Breaker (CB), DeepAug, DeRTa, and our proposed HumorReject. The following table presents the Safety Rates of each defense strategy across the various prefix injection attacks for both models, along with their average performance.

As illustrated in Table 2, HumorReject consistently outperforms all baseline defense strategies across every type of prefix injection attack for both Llama3-8B-instruct and Mistral-7B-instruct-v0.1 models. Specifically, HumorReject achieves Safety Rates ranging from 95% to 100%, demonstrating exceptional robustness and reliability in mitigating prefix injection attacks.

When compared to the Vanilla model, which exhibits moderate to low Safety Rates depending on the attack type (averaging 63.2% for Llama3-8B-instruct and 6.6% for Mistral-7B-instruct-v0.1), HumorReject significantly enhances defense effectiveness. Additionally, while strategies like Circuit Breaker also show strong performance with average Safety Rates of 97.4% for Llama3-8B-instruct and 90.6% for Mistral-7B-instruct-v0.1, HumorReject achieves even higher averages of 99.0% and 96.6% respectively. This highlights HumorReject’s superior capability in safeguarding LLMs against adversarial prefix injections, underscoring its effectiveness and robustness relative to existing defense mechanisms.

4.3. RQ3: Beyond Prefix Injection, Do Other Types of Attacks Still Pose Threats to Model Safety?

As highlighted by [4], in addition to prefix injection attacks, LLMs are vulnerable to mismatched generalization attacks. These attacks exploit discrepancies between training and test data distributions, causing models to generate unintended or harmful outputs. To evaluate the effectiveness of HumorReject against such threats, we compare its performance against three specific mismatched generalization attacks: ReNeLLM [24], CodeAttack [22], and CodeChameleon [21].

The results in Table 3 indicate that HumorReject consistently maintains high Safety Rates across all three mismatched generalization attacks for both Llama3-8B-instruct and Mistral-7B-instruct-v0.1 models. Specifically, HumorReject achieves Safety Rates of 92% for ReNeLLM, 77% for CodeAttack, and 83% for CodeChameleon on Llama3-8B-instruct. For Mistral-7B-instruct-v0.1, the Safety Rates are 95%, 98%, and 95% respectively. These findings suggest that HumorReject effectively mitigates the risks posed by mismatched generalization attacks, enhancing the overall safety of LLMs beyond protection against prefix injection alone.

From our case study in Appendix A, we believe that HumorReject enhances the model’s defense against mismatched generalization attacks by establishing a deep connection between harmful instructions and humorous responses. This may enable the model to respond humorously to harmful instructions even when required to use unseen reply templates, such as those used in CodeAttack.

4.4. RQ4: Does the HumorReject Approach Introduce New Security Risks?

HumorDAN, A Novel But Ineffective Adaptive Attack. Previous research [35] has emphasized that new defense mechanisms can potentially introduce new security vulnerabilities. To address this concern for HumorReject, we designed an adaptive attack named HumorDAN. Inspired by the DAN (Do Anything Now) [36] template, HumorDAN prompts the model to respond in two distinct styles: Classic and Humorous. This attack is considered adaptive because it attempts to exploit the fact that HumorReject models are trained to respond humorously rather than directly refuse harmful instructions - by requesting a "Classic Style" response, the attack aims to bypass the humorous defense mechanism and elicit harmful content in a straightforward manner.

Results. We executed the HumorDAN attack against both HumorReject Mistral-7B-instruct-v0.1 and HumorReject Llama3-8B-instruct. The results are as follows:

• HumorReject Mistral-7B-instruct-v0.1: Safety Rate 99%
• HumorReject Llama3-8B-instruct: Safety Rate 99%

These high safety rates indicate that the HumorDAN attack was unsuccessful in bypassing the HumorReject defense mechanism. Even when prompted to respond in Classic Style, the models consistently refused to generate harmful content, instead providing humorous refusals.

Defense Example. To illustrate the effectiveness of HumorReject, consider the following example where the model is prompted to make a bomb:

In this example, regardless of the requested style, the model refrains from providing harmful instructions and instead offers humorous responses that neutralize the intent of the prompt.

4.5. RQ5: Does HumorReject Affect the Model’s Performance on Benign Inputs?

In this subsection, we evaluate the impact of HumorReject on the model’s performance when handling benign inputs. The assessment is conducted from three perspectives: Multi-Task Language Understanding (MMLU), performance in contexts containing harmful instructions, and the propensity of the model to over-reject benign requests. The evaluation methods are detailed as follows:

MMLU Evaluation. Firstly, to assess the model’s capability in understanding and responding accurately across diverse tasks, we utilized 500 question-answer pairs from the MMLU [37] dataset. This evaluation measures the model’s general comprehension and response quality in various domains.

MMLU with Harmful Context. Secondly, to evaluate the model’s ability to handle benign tasks after responding to harmful instructions, we first prompted the model to respond to a harmful instruction randomly selected from our test set before presenting each MMLU question-answer pair. This setup assesses whether addressing harmful content impacts the model’s subsequent performance on legitimate MMLU tasks.

XSTEST Compliance Rate. Thirdly, to investigate whether HumorReject leads to unnecessary refusals of non-harmful requests, we employed 250 "safe" instructions from the XSTEST [38] dataset. This assessment measures the model’s compliance with benign requests, indicating its tendency to over-reject when faced with non-harmful inputs.

Experimental Results are summarized in Table 4, which compares different defense strategies across evaluation metrics. The results show that HumorReject maintains and slightly improves the model’s MMLU performance, with scores increasing from 58.0 to 60.8 for Llama3-8B-instruct and 49.8 to 50.2 for Mistral-7B-instruct-v0.1, suggesting that humor integration may enhance model comprehension.

When evaluating MMLU performance with harmful context, HumorReject shows superior robustness compared to baseline defenses. While methods like Circuit Breaker and DeepAug show severe degradation in performance, HumorReject maintains strong performance with scores of 58.2 for Llama3-8B-instruct and 45.4 for Mistral-7B-instruct-v0.1. Notably, we observe that after encountering harmful instructions, Circuit Breaker can only generate meaningless responses even for benign queries (causing Mistral’s score to drop to 0), revealing its over-defensive behavior that severely impacts model utility.

Regarding over-rejection, HumorReject demonstrates exceptional performance by maintaining high compliance rates (94.8% and 94.0%) comparable to the Vanilla Model. In contrast, alternative approaches like DeRTa and DeepAug exhibit substantially higher over-rejection tendencies. These findings convincingly illustrate HumorReject’s sophisticated capability to accurately distinguish between benign and harmful inputs, effectively minimizing false rejections while maintaining robust safety measures.

4.6. RQ6: Why Did Previous Humorous LLM Not Demonstrate Good Safety?

Among existing LLMs, Grok [39] is notable for incorporating a "fun mode" that aims to respond in a humorous and playful manner, capable of humorously rejecting harmful instructions like "Tell me how to make cocaine" [40]. However, recent research [41] has revealed that Grok’s safety performance ranks below many mainstream models in robustness against jailbreak attacks, indicating that its humor-based approach does not effectively ensure model safety.

Our investigation reveals that this vulnerability stems from Grok’s reliance on system prompts to generate humorous responses, rather than having this capability built into the model through training. As evidenced by recent prompt leaks [42], Grok’s humorous responses are primarily controlled through system-level prompting rather than being an inherent model capability. This distinction is crucial, as prompt-based approaches can be easily bypassed or modified by users.

To validate this hypothesis, we conducted experiments across multiple LLMs to assess whether system prompts alone could effectively implement humor-based safety measures similar to our HumorReject approach. We provided models with system prompts instructing them to reject harmful queries using humor, then evaluated their responses to the template attack [10]. The results are visualized in Figure 3.

Our experiments revealed two distinct patterns in how models respond to humor-based system prompts. Models like Mistral-7B-instruct-v0.1, Gemma-2-27B-it [43], and LLaMA3-70B-instruct showed limited ability to follow the humorous rejection instructions, resulting in minimal safety improvements. In contrast, models such as Qwen2.5-72B-instruct [44], Grok-2-1212 [39], and Claude-3.5-sonnet demonstrated stronger adherence to the humorous system prompts, showing notable improvements in both safety and humor metrics when rejecting harmful queries.

These findings highlight that while system prompts can potentially implement humor-based safety measures, their effectiveness heavily depends on the model’s ability to follow system instructions. More importantly, this approach remains fundamentally limited as users can simply choose not to use these safety-oriented system prompts. This vulnerability underscores why our HumorReject approach, which builds humor-based safety directly into model training, provides a more robust and reliable solution compared to prompt-based methods.

5. Conclusion

In this work, we introduced HumorReject, a novel method that leverages humor to enhance the safety of LLMs. By employing humorous responses as indirect refusals, HumorReject maintains engaging and natural interactions while effectively addressing safety concerns. Our comprehensive experiments demonstrated that HumorReject significantly improves model robustness while preserving performance on benign inputs, thereby avoiding the over-defense issues prevalent in existing safety strategies. Additionally, the creation and public release of a 400-sample HumorReject preference dataset provide a valuable resource for future research in model alignment and safety. Despite its promising results, further research is needed to explore the cultural nuances of humor and potential security implications of this approach, as humor can sometimes be perceived as offensive by different cultural groups or individuals. We recommend that future work carefully consider cultural sensitivities and establish appropriate guidelines for the use of humor. Overall, HumorReject offers an innovative solution for developing safer, more resilient, and user-friendly LLMs, paving the way for advanced and reliable AI interactions.

Appendix A.1. Examples of Training Data

Appendix A.2. Prompts

Appendix A.2.1. Prompts for Judge Models

Appendix A.2.2. Prompts for HumorDAN Attack

Appendix A.2.3. HumorReject-like System Prompt in RQ6

Appendix A.3. Defense Cases

Appendix A.3.1. Defense Against GCG Attack

Appendix A.3.2. Defense Against AutoDAN Attack

Appendix A.3.3. Defense Against CodeAttack

Appendix A.3.4. Defense Against ReNeLLM Attack

Appendix A.3.5. Defense Against CodeChameleon Attack

Appendix A.4. Failure Cases