Prompt Sensitivity and Bias Amplification in Aligned Video Diffusion Models

1. Introduction

Text-to-video generation has progressed rapidly in recent years, largely driven by diffusion-based models that extend text-to-image generation with explicit temporal modeling and large-scale training strategies [1,2]. Beyond improvements in visual fidelity, recent studies indicate that video diffusion models learn structured motion patterns and temporally consistent representations that are useful for spatiotemporal understanding, suggesting their broader role as general generative systems rather than tools limited to visual synthesis [3]. As these models are increasingly deployed in creative and decision-support applications, their stability under realistic prompt variation has become a critical concern. Minor changes in wording, context, or descriptive emphasis can lead to noticeable differences in generated content, particularly when prompts involve people, social roles, or other sensitive attributes.

Alignment tuning has therefore emerged as a standard post-training technique for guiding diffusion models toward desired behavior, including safety compliance, preference satisfaction, and instruction following. Recent alignment approaches span preference-based optimization using paired comparisons, reward-guided objectives based on lightweight feedback signals, and tuning-free or low-cost methods that adjust generation trajectories without full retraining [4,5]. Survey studies have organized these methods according to training objectives, feedback sources, and evaluation practices, while also identifying unresolved trade-offs between controllability, generalization, and robustness [6]. Importantly, recent evidence shows that alignment tuning can systematically reshape the response of video diffusion models to textual inputs, altering how demographic attributes are stabilized over time and revealing a transition from preference optimization to temporally persistent social bias [7]. This finding highlights that alignment does not merely suppress undesirable outputs, but can fundamentally change how sensitive models are to prompt formulation. A growing body of work has documented that diffusion models can reproduce and amplify social biases present in training data. In text-to-image generation, multiple studies report consistent demographic skews related to gender, ethnicity, and occupation, and propose standardized auditing frameworks to measure these effects in generated samples [8,9]. More recent work expands bias analysis beyond fixed demographic categories by introducing counterfactual prompt designs, adaptive measurement strategies, and concept-level explanations [10]. Bias mitigation techniques have also been explored, including training-time rebalancing and inference-time controls that aim to reduce demographic imbalance while preserving generation quality [11,12]. Other studies analyze how common prompt patterns activate demographic attributes in widely used diffusion models, revealing systematic associations between language cues and visual emphasis [13]. However, most of this literature focuses on static images. In video generation, biased attributes may persist across frames, reinforcing stereotypical representations through temporal consistency and repeated visual exposure. Only recently have studies begun to examine how alignment tuning interacts with social bias in video diffusion models. Existing analyses suggest that bias introduced or stabilized during alignment can propagate across diffusion steps and frames, leading to increased consistency of demographic patterns over time [14]. These findings imply that alignment tuning may alter prompt sensitivity rather than uniformly reducing bias. In parallel, robustness studies show that small prompt perturbations, such as spelling variations or minor rephrasing, can produce measurable changes in diffusion model behavior [15]. Other work reports that common, non-adversarial prompts may trigger systematic generation failures, indicating that instability is not limited to adversarial inputs [16]. Prompt-based attack studies further demonstrate that the prompt channel remains a practical source of vulnerability in diffusion systems [17]. Nevertheless, most robustness analyses focus on semantic accuracy or safety violations and rarely evaluate how bias distributions shift under prompt variation, particularly in video diffusion settings. Despite this progress, several limitations remain. Many bias evaluations rely on fixed prompt templates and do not examine how small semantic or lexical changes influence demographic outcomes at scale, especially in aligned video diffusion models. Alignment research typically reports average improvements in preference scores or instruction adherence, but provides limited analysis of prompt sensitivity as an independent property. As a result, it remains unclear whether alignment tuning reduces or amplifies output variability under realistic prompt noise. Although attribution and interpretability methods for diffusion models have advanced, they are seldom integrated into bias evaluation pipelines, limiting the ability to link prompt components with visual emphasis in generated videos [18,19]. These issues are particularly consequential for video generation, where bias may manifest not only in appearance but also in actions, roles, camera framing, and temporal focus.

To address these gaps, this study investigates prompt sensitivity and bias amplification in aligned video diffusion models using a controlled prompt perturbation framework. Paired prompts are constructed to preserve semantic intent while introducing minimal lexical or contextual variation. A large set of video samples is generated and evaluated using complementary metrics, including demographic skew ratios, attribute co-occurrence patterns, and saliency-based attribution that links prompt elements to visual emphasis. By comparing aligned and unaligned models under identical perturbation conditions, the analysis isolates how alignment tuning affects the relationship between prompt sensitivity and bias variability. This study provides a systematic evaluation approach that reveals bias instability not captured by single-prompt audits and clarifies the conditions under which alignment tuning may unintentionally increase bias variance in response to minor prompt changes, offering practical guidance for the responsible deployment of video diffusion models.

2. Materials and Methods

2.1. Sample and Study Scope

This study examines text-to-video diffusion models that generate short video clips from natural language prompts. Two model versions were used: an alignment-tuned model and an unaligned baseline. Both models share the same architecture and pretraining data, differing only in the alignment stage. In total, 26,700 videos were generated for analysis. The prompts describe everyday human-related scenes, such as appearance, occupations, and common activities. All videos were produced using the same resolution, frame rate, and inference settings. Prompts were written in English and kept descriptive and neutral, except where demographic information was required for measurement.

2.2. Experimental Design and Control Setup

A paired prompt design was used to evaluate sensitivity to prompt changes. Each base prompt was matched with a perturbed version that introduced small wording or contextual changes while keeping the intended meaning unchanged. The alignment-tuned model was treated as the experimental group, and the unaligned model was used as the control group. Both models were tested with identical prompt pairs and fixed random seeds. This setup ensures that differences in outputs are related to alignment tuning rather than random variation. Experiments were conducted across multiple prompt categories to reduce topic-specific bias.

2.3. Measurement Procedures and Quality Control

Bias outcomes were evaluated using three complementary measures. First, demographic skew ratios were calculated based on the relative frequency of visual attributes linked to predefined demographic groups. Second, attribute co-occurrence statistics were computed to examine how demographic features appear together with roles, actions, or visual context. Third, visual saliency attribution was applied to identify image regions most influenced by prompt content. Quality control included manual inspection of randomly selected samples, removal of incomplete or corrupted videos, and comparison between automated measurements and human annotations on a validation subset. All procedures were applied consistently across models.

2.4. Data Processing and Model Formulation

Generated videos were processed to extract frame-level features using a fixed pretrained vision encoder. These features were then aggregated over time to obtain clip-level representations. Prompt sensitivity was defined as the normalized change in bias measures between paired prompts. For a bias metric$\mathrm{B}$, sensitivity was calculated as

$\mathrm{S}={\displaystyle \frac{|{\mathrm{B}}_{\mathrm{perturbed}}-{\mathrm{B}}_{\mathrm{base}}|}{{\mathrm{B}}_{\mathrm{base}}+\mathsf{ϵ}}},$

where $\mathsf{ϵ}$ is a small constant. Bias variation was further examined using a linear regression model,

${\mathrm{B}}_{\mathrm{i}}={\mathsf{\beta }}_0+{\mathsf{\beta }}_1{\mathrm{A}}_{\mathrm{i}}+{\mathsf{\beta }}_2{\mathrm{P}}_{\mathrm{i}}+{\mathsf{\epsilon }}_{\mathrm{i}},$

where ${\mathrm{A}}_{\mathrm{i}}$​indicates whether alignment tuning was applied, ${\mathrm{P}}_{\mathrm{i}}$ represents prompt perturbation, and ${\mathsf{\epsilon }}_{\mathrm{i}}$ is the error term. This formulation separates the effects of alignment and prompt variation.

2.5. Statistical Analysis

Statistical tests were used to compare prompt sensitivity and bias variation between the two model versions. Mean differences were evaluated using two-sided tests with standard significance thresholds. Confidence intervals were estimated using bootstrap resampling over prompt pairs. Effect sizes were reported to support interpretation of the results. All analyses followed the same processing pipeline and used fixed random seeds to ensure reproducibility. The analysis focused on robustness and stability rather than performance optimization.

3. Results and Discussion

3.1. Prompt-Level Noise Leads to larger Output Variation After Alignment

Across the 26,700 generated videos, paired prompts that differed only in minor wording still produced clear differences in demographic composition and role depiction. For neutral scenes, these differences were limited. In contrast, for socially sensitive prompts, the aligned model showed noticeably larger fluctuations in demographic skew ratios and higher variability across repeated generations. This behavior suggests that alignment strengthens the influence of text conditioning. When prompts contain sensitive cues, even indirectly, small wording changes can propagate through the generation process and result in visible changes in who appears and how actions are portrayed [20]. Similar sensitivity has been reported in diffusion stress analyses, where changes in the optimization or conditioning regime lead to higher loss variance and slower convergence, indicating increased responsiveness to small perturbations (Figure 1).

3.2. Bias Amplification Appears Under Routine Rewording Rather Than Single Prompts

When bias is evaluated using single, fixed prompts, aligned and unaligned models may appear similar. The paired-prompt analysis reveals a different pattern. Alignment increases prompt sensitivity, so two prompts with the same intended meaning can lead to different demographic outcomes. In this study, socially sensitive prompt groups showed a 34.1% higher sensitivity after alignment, together with wider dispersion in attribute co-occurrence measures, such as identity–occupation and identity–activity pairs. These results suggest that bias amplification is driven by instability across prompt variants rather than by a consistently higher bias level. From a practical perspective, a mitigation strategy that reduces average bias for one wording may still fail under common paraphrases, which better reflects how users interact with generative systems [21].

3.3. Safety Constraints and content Stability Show A Shared Trade-Off

Saliency attribution analysis indicates that aligned models place more concentrated emphasis on identity-related visual regions when prompts include weak but socially loaded context cues. This concentration can increase variability across prompt neighbors. The observation aligns with broader findings in generative safety research, where stronger constraints often trade off against content fidelity and stability. In video diffusion safety editing, ablation studies show that removing preservation constraints reduces safety quality and also harms semantic and temporal consistency [22,23]. This demonstrates that constraint design reshapes model behavior rather than uniformly improving outcomes (Figure 2).

3.4. Implications for Evaluation and Mitigation Under Alignment

Taken together, the results indicate that alignment should be evaluated not only by average bias levels but also by local robustness around prompts. Two models may show similar mean demographic skew while differing substantially in stability under minor rewording. For reliable assessment, evaluation protocols should include neighborhood-based tests, such as paired prompts and paraphrase sets, and should report variance-based measures alongside mean bias scores [24]. Mitigation strategies should also distinguish between constraint strength and conditioning stability. Without this separation, stronger alignment may reduce explicit unsafe outputs while increasing sensitivity to ordinary prompt variation, which can amplify bias in everyday use.

4. Conclusions

This study examined how prompt sensitivity and bias behavior change in aligned video diffusion models using large-scale paired prompt experiments. The results show that alignment tuning can improve instruction following, but it can also make models more responsive to small prompt changes in socially sensitive settings. As a result, demographic representation and attribute patterns vary more across similar prompts. The analysis indicates that alignment alters how textual cues influence generation, rather than consistently reducing bias across outputs. A key contribution of this work is the focus on robustness around prompt neighborhoods instead of bias measured from single prompts, which reveals instability that is not captured by average metrics. These findings are relevant for real-world use of video diffusion models in media generation and human-centered applications, where prompt rewording is common. The study is limited to a specific group of diffusion models and prompt structures, and it does not examine longer temporal narratives or interactive inputs. Future research should evaluate a wider range of models and video lengths, and develop mitigation methods that balance alignment goals with stability under normal prompt variation.