Scale Collapse, Model Disagreement, and False Precision: A Multi-Model Audit of LLM Behavioral Interview Scoring

1. Introduction

1.1. Background: The Rise of AI in Hiring

Artificial intelligence is rapidly transforming hiring practices. An estimated 99% of Fortune 500 companies now use AI-powered tools somewhere in their recruitment pipeline, and 90% of large employers use automated systems to filter job applications [1]. SHRM’s 2024 Talent Trends Survey found that among organizations using AI for HR, 64% apply it to recruitment, interviewing, and hiring — the leading use case [2]. Products like HireVue, Pymetrics, and numerous startup offerings promise to evaluate candidates at scale using Large Language Models (LLMs). The appeal is clear: consistent evaluation, reduced human bias, faster throughput, and lower cost per assessment.

The behavioral interview — in which candidates describe past experiences demonstrating competencies such as leadership, conflict resolution, and problem-solving — is a cornerstone of hiring at major technology companies including Google, Meta, and Amazon. These companies employ structured rubrics to evaluate behavioral answers on ordinal scales, making them a natural target for LLM-based automation.

1.2. The Missing Audit

Despite rapid deployment, a critical gap exists: no systematic reliability audit has been conducted on LLM behavioral interview evaluators. This stands in stark contrast to other high-stakes AI applications:

• Medical AI systems require FDA clearance and clinical validation before deployment.
• Financial AI models undergo rigorous backtesting and stress testing.
• Criminal justice risk assessment tools face increasing scrutiny and auditing requirements.

Yet AI hiring tools — which directly affect people’s livelihoods — operate with minimal reliability benchmarking. The few existing studies focus on single-model evaluations or narrow prompt engineering without addressing fundamental questions of measurement reliability.

1.3. Research Questions

We address three core questions:

1.

Scale utilization: Do LLM evaluators effectively use the full range of hiring rating scales, or does the scale degenerate under automated evaluation?

2.

Reliability: How consistent are LLM hiring ratings across repeated evaluations (test–retest) and across different models (inter-rater)?

3.

Validity: Do LLM ratings align with community expectations for behavioral interview answer quality?

1.4. Contributions

1.

The first multi-model behavioral interview scoring benchmark, spanning 9 frontier models, 3 model families, 3 companies, 2 seniority levels, and 10 behavioral questions, with up to 50 repeated samples per evaluation cell.

2.

Five empirically validated reliability failures in LLM interview scoring: scale collapse, test–retest unreliability, inter-model disagreement, question-type sensitivity, and expectation gap.

3.

A model robustness scorecard demonstrating that stability (self-consistency) and consensus (inter-model agreement) are orthogonal dimensions — no single model excels on all reliability axes.

4.

A minimum reliability checklist for deploying LLM interview evaluators, informed by our empirical findings.

5.

Open-source evaluation framework enabling reproducible auditing of LLM interview scoring systems.

1.5. Paper Organization

Section 2 reviews related work. Section 3 describes our experimental design. Section 4 presents our five core findings. Section 5 introduces the model robustness scorecard. Section 6 proposes the reliability checklist and discusses implications. Section 7 acknowledges limitations. Section 8 concludes.

2. Related Work

2.1. LLM-as-Judge

The paradigm of using LLMs as evaluators has gained significant traction. [3] introduced MT-Bench and Chatbot Arena for evaluating conversational AI using LLM judges. [4] proposed G-Eval, demonstrating that GPT-4 evaluations correlate with human judgments for text generation quality. However, these works focus on general NLG evaluation rather than domain-specific, high-stakes assessment.

Recent studies have identified systematic biases in LLM judges, including leniency bias [5], position bias [3], and verbosity bias [6]. Our work extends this line of inquiry to the hiring domain, where such biases have direct consequences for candidates and employers.

2.2. AI in Hiring

Automated hiring systems have evolved from keyword-matching resume screeners to sophisticated multi-modal assessment platforms. HireVue (now part of the broader AI hiring ecosystem) processes millions of video interviews annually. Academic work has explored automated interview scoring [7,8] and feedback generation [9].

However, the reliability of these systems remains underexplored. [10] surveyed AI hiring tools and found a lack of validation evidence. [11] highlighted fairness concerns in AI hiring but did not conduct systematic reliability auditing. Our work fills this gap with the first multi-model reliability benchmark specifically for behavioral interview evaluation.

2.3. Evaluation Reliability in Psychology

The psychometric tradition provides a rigorous framework for evaluating measurement instruments. Key reliability dimensions include:

• Test–retest reliability: Consistency of scores across repeated measurements [12].
• Inter-rater reliability: Agreement among different raters evaluating the same subject [13].
• Internal consistency: Coherence of items within a scale [14].

We adapt these classical reliability frameworks to the LLM evaluation context, treating each model as a “rater” and each repeated API call as a “retest.”

2.4. Regulatory Landscape

The regulatory environment for AI hiring is rapidly evolving:

• NYC Local Law 144 requires annual bias audits for automated employment decision tools [15].
• Illinois AI Video Interview Act mandates disclosure when AI analyzes video interviews [16].
• EU AI Act classifies AI hiring tools as “high-risk,” requiring conformity assessments [17].
• EEOC guidance addresses AI-related discrimination in employment decisions [18].
• NIST AI Risk Management Framework provides standards for high-risk AI applications [19].

Our work provides empirical evidence directly relevant to these regulatory requirements, demonstrating specific failure modes that auditing frameworks should address.

2.5. Positioning

Our work differs from prior studies in several key respects: (1) we evaluate multiple frontier models from different vendors rather than a single model; (2) we focus specifically on behavioral interview evaluation, a high-stakes domain with structured rubrics; (3) we apply psychometric reliability frameworks (test–retest, inter-rater agreement) rather than focusing solely on accuracy; (4) we conduct the evaluation at scale (141 cells, up to 50 samples each) to enable robust statistical analysis.

3. Experimental Design

3.1. Answer Source: Community Consensus Baseline

We use 10 behavioral interview question–answer pairs from the awesome-behavioral-interviews GitHub repository, a curated resource with over 8,000 stars — one of the most popular behavioral interview preparation resources on the internet. We selected this source deliberately: these answers represent community consensus on what constitutes a “good” behavioral interview answer. With 8,000+ stars, they reflect a broad “sounds reasonable” crowd endorsement, making them an ideal baseline for testing whether LLM evaluators agree with community expectations.

Table 1. Question categories covered in the benchmark.

Category	Questions	N
Vulnerability	Failure, conflict, pressure, tough feedback, manager disagreement	5
Leadership	Led a team, difficult decision	2
Initiative	Above and beyond	1
Epistemic	Handle not knowing the answer	1
Introduction	Tell me about yourself	1

Table 1. Question categories covered in the benchmark.

Category	Questions	N
Vulnerability	Failure, conflict, pressure, tough feedback, manager disagreement	5
Leadership	Led a team, difficult decision	2
Initiative	Above and beyond	1
Epistemic	Handle not knowing the answer	1
Introduction	Tell me about yourself	1

3.2. Model Matrix

We evaluate 9 models from 3 major LLM provider families, covering the full capability-cost spectrum (Table 2). Flagship models receive 10 samples per cell due to higher cost; lightweight and mid-tier models receive 50 samples for richer distribution estimates. All models were accessed via LiteLLM with temperature $=0$ (or minimum available) for evaluation consistency.

3.3. Evaluation Dimensions

Each answer is evaluated under 6 configurations (3 companies × 2 levels):

• Companies: Google, Meta, Amazon — representing distinct but overlapping behavioral interview cultures.
• Levels: Mid-level, Senior — reflecting different expectation bars.

This yields 141 unique evaluation cells (question × company × level × model), as not all models were evaluated under all configurations due to cost optimization. The primary evaluation slice is Google, Senior, which has full 11-model coverage.

3.4. Prompt Design

We use a unified evaluation prompt across all models to ensure comparability:

You are an interviewer at {company}.

Evaluate the candidate’s behavioral answer for a {level} Software Engineer role.

Question: {question}

Candidate answer: {answer}

Output MUST be valid JSON:

{"feedback": "<~200 words>", "rating": "<No Hire | Leaning No Hire |

{Leaning Hire | Hire | Strong Hire>"}

The 5-point ordinal scale maps to: No Hire (1), Leaning No Hire (2), Leaning Hire (3), Hire (4), Strong Hire (5).

3.5. Repeated Sampling and Test–Retest Protocol

Each evaluation cell receives 10–50 independent API calls. For test–retest analysis, we conducted 27 separate experimental runs across different dates, allowing us to compute within-run and across-run agreement rates.

3.6. Metrics

We compute the following reliability metrics:

• Cell entropy (Shannon entropy in nats): Measures rating distribution spread within a cell. Zero entropy = deterministic output.
• Cross-question expected score SD: Standard deviation of expected scores across questions for a given (company, level, model). Measures a model’s sensitivity to question content.
• Pairwise Spearman $\rho$: Rank correlation of per-question expected scores between model pairs. Measures inter-model agreement on relative question difficulty.
• Test–retest agreement rate: Proportion of multi-replicate groups where all replicates yield the same rating.
• Ordinal pstdev: Population standard deviation of ordinal scores (1–5) within replicate groups.

In addition to these descriptive metrics, we apply formal statistical tests to quantify significance and effect size:

• Friedman test (${\chi }^2$): Non-parametric repeated-measures ANOVA testing whether models produce significantly different ratings across questions.
• Kendall’s W: Coefficient of concordance measuring how consistently models rank-order questions from easiest to hardest.
• ICC(2,1): Intraclass correlation coefficient (two-way random, single measures) quantifying test–retest reliability for each model.
• Krippendorff’s $\alpha$ (ordinal): Inter-model agreement metric suitable for ordinal data with any number of coders.
• Wilcoxon signed-rank test: Paired non-parametric test comparing flagship vs. lightweight model tiers on matched questions.

4. Results: Five Reliability Failures

4.1. Finding 1: Scale Collapse — A 5-Point Scale Used as a 2-Point Scale

The most striking finding is the systematic collapse of the 5-point hiring scale. Across all 4,600 ratings (141 evaluation cells):

• “No Hire” accounts for 0.0% of ratings — the lowest category is never assigned.
• “Strong Hire” accounts for only 0.43% — nearly absent.
• “Leaning Hire” dominates at 77.2%, with “Leaning No Hire” at 14.0% and “Hire” at 8.4%.
• 62% of cells have zero entropy — the model produces the same rating every single time.
• 90–100% of ratings concentrate in a single bin for the majority of cells.

Figure 1. Scale collapse — number of rating bins used per evaluation cell.

Figure 1. Scale collapse — number of rating bins used per evaluation cell.

Figure 2. Expected rating score by question and model (Google, Senior slice).

Figure 2. Expected rating score by question and model (Google, Senior slice).

Formal information-theoretic analysis confirms the severity: Shannon entropy is 0.707 nats against a maximum of 1.609 nats for a uniform 5-category distribution, yielding a normalized entropy of only 0.439. The effective number of categories ($exp\left(H\right)$) is 2.03 — a 5-point scale is functioning as a 2-point scale. The Gini-Simpson diversity index of 0.378 further corroborates that fewer than 2 categories carry meaningful probability mass.

Implication. The 5-point scale degenerates into a de facto binary judgment (Leaning No Hire vs. Leaning Hire) under LLM evaluation. There is zero headroom to distinguish exceptional candidates from mediocre ones, and no mechanism to flag clearly unqualified candidates. Any selection system built on these ratings is operating with far less discriminative power than its scale implies.

4.2. Finding 2: Test–Retest Unreliability — 10× Spread in Consistency

Figure 3. Test–retest agreement rate by model.

Figure 3. Test–retest agreement rate by model.

Test–retest reliability varies by an order of magnitude across models (Table 3). GPT-5.4 achieves perfect 100% agreement in its best runs (identical ratings across all replicates). At the other extreme, Gemini Flash and Claude Haiku drop to 10–20% agreement, with ordinal pstdev of 0.37–0.42 — nearly a full grade of jitter on a 5-point scale.

Formal ICC(2,1) analysis (two-way random, single measures) quantifies the gap (Table 4). ICC values range from 0.852 (gpt-5.4-pro) to 0.160 (claude-haiku-4-5) — a 5.3× spread. Only one model (gpt-5.4-pro) exceeds the conventional 0.75 threshold for “good” reliability [20]. Four models fall below the 0.50 “fair” threshold, with claude-haiku-4-5 approaching chance-level consistency. Notably, gpt-5.4-mini — the highest-volume model in our benchmark (50 samples/cell) — achieves only 0.342 despite having the most data to stabilize.

Implication. A candidate evaluated by Gemini Flash could receive “Leaning Hire” on one submission and “Leaning No Hire” on the next, purely from stochastic variation. Single-shot LLM scores are unreliable for low-consistency models. Repeatability must be benchmarked and reported before deployment.

4.3. Finding 3: Inter-Model Disagreement — Swapping Models Reshuffles Rankings

Figure 4. Pairwise inter-model rank correlation (Google, Senior).

Figure 4. Pairwise inter-model rank correlation (Google, Senior).

Pairwise Spearman rank correlations of per-question expected scores reveal striking disagreement patterns (Table 5). The most surprising finding: same-brand variant agreement ($\rho =0.25$–$0.31$) can be worse than cross-vendor agreement ($\rho =0.52$–$0.87$). Upgrading from GPT-5.4 to GPT-5.4-pro — ostensibly a capability improvement within the same product line — can reshuffle candidate rankings more than switching to an entirely different vendor’s model.

Three formal tests confirm that inter-model disagreement is both statistically significant and substantively large:

1.

Friedman test (${\chi }^2=35.56$, $p=2.8\times {10}^{-5}$, $k=9$ models, $n=10$ questions): Models produce significantly different rating distributions. Mean rank analysis reveals gpt-5.4-pro as the harshest rater (mean rank $=2.05$) and claude-haiku-4-5 as the most lenient (mean rank $=7.9$) — a 3.9× gap in rank position.

2.

Kendall’s $W=0.639$ (${\chi }^2=51.79$, $p<0.001$): Models show moderate concordance on question ranking — they partly agree on which questions are “harder” versus “easier,” but agreement is far from complete. A W of 0.639 indicates roughly 64% shared ranking variance, leaving 36% attributable to idiosyncratic model-specific difficulty calibration.

3.

Krippendorff’s $\alpha =0.523$ (ordinal, 9 coders × 10 units): Inter-model agreement falls below the conventional 0.667 threshold for tentative conclusions [21] and far below the 0.800 threshold required for reliable coding. This means LLM ratings would not pass inter-rater reliability standards in any social-science content-analysis study.

Furthermore, vulnerability-category questions (failure, conflict, pressure) show the highest cross-model standard deviation (up to 0.58), serving as the most potent amplifiers of inter-model disagreement. GPT-5.4-pro rates “tough feedback” and “conflict” questions at 100% “Leaning No Hire” (expected score $=2.0$), while GPT-5.4-mini rates the identical questions at 100% “Leaning Hire” (expected score $=3.0$) — same brand, one full ordinal grade apart.

Implication. Swapping model versions is not a neutral operation. It is functionally equivalent to swapping the entire scoring rubric, with unpredictable effects on candidate rankings.

4.4. Finding 4: Question-Type Sensitivity — 2-Grade Swings from Question Framing

Figure 5. Cross-question score volatility by model (Google, Senior).

Figure 5. Cross-question score volatility by model (Google, Senior).

The same candidate answer receives dramatically different ratings depending on question type:

• “Tell me about yourself” (Google, Senior): 8 out of 11 models assign 100% “Leaning No Hire” (expected score $=2.0$).
• “How do you handle not knowing the answer?”: Gemini Flash assigns 98% “Hire” (expected score $=4.02$).

This represents a 2-full-ordinal-point difference (2.0 vs. 4.0) for the same candidate, purely based on question framing. Cross-question expected score standard deviation ranges from 0.075 (GPT-5.4-mini, near-flat) to 0.684 (Gemini Flash, highly volatile) — a 9× gap in sensitivity.

Implication. LLM “harshness” is strongly governed by question type. Cross-question candidate comparison requires normalization or stratification. A candidate who excels at “how do you handle X” questions but struggles with “tell me about a time you failed” may be rated identically to a candidate with the opposite profile — or dramatically differently, depending on which model is used.

4.5. Finding 5: Expectation Gap — Community “Best Answers” Score Below Hire

Figure 6. Flagship vs. lightweight model ratings by question.

Figure 6. Flagship vs. lightweight model ratings by question.

The 10 answers used in this experiment come from a repository with 8,000+ GitHub stars — representing broad community endorsement. One would reasonably expect these answers to achieve at least “Hire” (4.0 on the 5-point scale).

Table 6. Actual weighted-average ratings per question.

Question (abridged)	Flagship Avg	Lightweight Avg
Handle not knowing the answer	3.23	3.41
Difficult decision	3.13	3.13
Led a team	3.00	3.14
Above and beyond	2.80	2.92
Conflict with teammate	2.73	3.08
Disagreement with manager	2.73	3.14
Time you failed	2.40	2.89
Under pressure	2.30	2.86
Tough feedback	2.23	2.76
Tell me about yourself	2.18	2.47

Table 6. Actual weighted-average ratings per question.

Question (abridged)	Flagship Avg	Lightweight Avg
Handle not knowing the answer	3.23	3.41
Difficult decision	3.13	3.13
Led a team	3.00	3.14
Above and beyond	2.80	2.92
Conflict with teammate	2.73	3.08
Disagreement with manager	2.73	3.14
Time you failed	2.40	2.89
Under pressure	2.30	2.86
Tough feedback	2.23	2.76
Tell me about yourself	2.18	2.47

Zero out of 10 questions reach the “Hire” threshold (4.0) for either flagship or lightweight models. The best-performing question barely exceeds “Leaning Hire” (3.0). Flagship models are consistently harsher than lightweight models (all 10 differences negative, mean $\Delta =-0.31$).

A Wilcoxon signed-rank test confirms that the flagship–lightweight gap is statistically significant: $W=0$, $z=-2.80$, $p=0.005$ ($n=10$ matched question pairs, all 10 differences non-zero and negative). The mean flagship rating (2.67) is 0.31 ordinal points below the mean lightweight rating (2.98). This is not random fluctuation — larger, more “capable” models are systematically harsher evaluators, a finding with direct implications for organizations that assume upgrading model tier improves evaluation quality.

This expectation gap admits two non-mutually-exclusive explanations:

1.

The internet is not the interview room — Even highly upvoted community BQ answers may not meet actual interview “Hire” standards. Star count reflects “sounds reasonable” crowd consensus, not calibrated interview quality.

2.

SOTA LLMs are still unreliable for BQ evaluation — Current frontier models may be systematically too harsh or unable to reliably distinguish strong behavioral answers from mediocre ones.

Implication. Neither crowd-sourced BQ prep materials nor LLM evaluators should be trusted at face value without validation against human interviewer ground truth.

4.6. Summary of Formal Statistical Tests

Table 7 consolidates the formal statistical tests applied across the five findings. All tests point to the same conclusion: LLM behavioral interview ratings exhibit statistically significant unreliability across every measured dimension — scale utilization, test–retest consistency, inter-model agreement, and tier-level bias.

5. Model Robustness Scorecard: Stability and Consensus Are Orthogonal

Figure 7. Model robustness radar chart.

Figure 7. Model robustness radar chart.

Aggregating all metrics per model reveals that stability (self-consistency) and consensus (agreement with other models) are independent dimensions (Table 8).

The key insight: GPT-5.4 is “confidently idiosyncratic” — it produces near-deterministic outputs (entropy $=0.05$) but agrees least with other models ($\rho =0.49$). Conversely, Gemini Flash is “volatile but mainstream” — it has the highest cross-question variance (SD $=0.68$) yet ties for the highest inter-model consensus ($\rho =0.79$).

No single model dominates all robustness dimensions. Choosing a model for hiring evaluation requires explicit trade-offs between self-consistency and inter-model alignment — a decision that most current deployments do not make transparently.

6. Discussion

6.1. A Minimum Reliability Checklist for LLM Interview Evaluators

Based on our findings, we propose that any deployment of LLM interview evaluators should meet the following minimum reliability requirements (Table 9).

Currently, no model in our benchmark simultaneously meets all five requirements:

• GPT-5.4-mini comes closest (4/5), failing only on R5 (no human ground truth available).
• Several models fail R1, R2, or R3 individually.

6.2. Implications for AI Hiring Policy

For NYC Local Law 144 compliance. The law requires annual bias audits, but our results suggest that reliability auditing (test–retest, inter-model) should be equally mandatory. A system that gives different ratings to the same candidate on different days fails a basic fairness standard regardless of aggregate bias statistics.

For EU AI Act compliance. High-risk AI systems must demonstrate “accuracy, robustness, and cybersecurity.” Our test–retest results (10–20% agreement for some models) and inter-model disagreement ($\rho =0.25$ for same-brand variants) directly challenge robustness claims.

For EEOC guidance. If a candidate’s hiring outcome depends more on which model version is used ($\rho =0.25$ between GPT-5.4 and GPT-5.4-pro) than on their actual answer quality, this constitutes a form of arbitrary discrimination that current anti-discrimination frameworks are ill-equipped to address.

6.3. Implications for LLM-as-Judge Research

Our findings extend the LLM-as-Judge literature in several important ways:

1.

Domain-specific failure modes: Scale collapse appears to be particularly severe in the hiring domain, where the 5-point scale has only 2 meaningful categories. This may not manifest in general NLG evaluation tasks.

2.

Cross-version instability: The finding that same-brand model variants ($\rho =0.25$) disagree more than cross-vendor pairs ($\rho =0.87$) has implications for any system that relies on a specific model checkpoint. Model updates can silently invalidate evaluation pipelines.

3.

Orthogonality of stability and consensus: The independence of self-consistency and inter-model agreement is a novel finding that suggests current model selection practices (choosing the “most capable” model) may be misguided for evaluation tasks.

6.4. Implications for Job Seekers

1.

Do not assume fairness: The same answer can receive ratings differing by 2 full grades depending on model choice and question type.

2.

Resubmission may yield different results: For low-consistency models, re-taking the same assessment can change outcomes purely from stochastic variation.

3.

Community preparation materials may not calibrate correctly: High-star GitHub answers still score below “Hire” under LLM evaluation.

6.5. Why This Matters for the United States

AI-assisted hiring is projected to affect over 100 million job applications annually in the United States [22]. The reliability failures documented in this paper — scale collapse masking discriminative power, test–retest jitter exceeding one full grade, inter-model disagreement rivaling random assignment — have direct consequences for labor market efficiency and worker welfare. Unreliable AI hiring tools waste employer resources on false-positive screenings and systematically disadvantage candidates through arbitrary scoring variation. Our findings provide the empirical foundation for evidence-based regulation of AI hiring tools, supporting the goals of the EEOC, NIST AI RMF, and state-level AI hiring legislation.

7. Limitations

1.

Answer source: Our answers come from a community-curated GitHub repository rather than real interview recordings. While this provides a useful community-consensus baseline, real candidate answers may exhibit different characteristics (nervousness, digressions, varied articulation quality) that could affect LLM evaluation patterns.

2.

No human ground truth: We lack parallel human interviewer ratings for the same answers. While Finding 5 (Expectation Gap) provides indirect calibration evidence, direct human–LLM agreement measurement remains future work.

3.

Behavioral interviews only: Our audit covers behavioral (competency-based) interview questions only. Technical interviews, system design interviews, and case study formats may exhibit different reliability patterns.

4.

Model snapshot: LLM capabilities change with model updates. Our results reflect model behavior at the time of experimentation (March 2026). Findings should be periodically revalidated as models evolve.

5.

Single answer per question: We evaluate one community-sourced answer per question. A more comprehensive benchmark would include multiple answers of varying quality per question to test discriminative validity.

6.

Prompt sensitivity: While we use a standardized prompt, different prompt designs could yield different reliability profiles. Our findings represent one specific — but realistic — evaluation prompt template.

8. Conclusion

We present the first systematic reliability audit of LLM behavioral interview scoring, evaluating 9 frontier models across 141 evaluation cells. Our five core findings — scale collapse, test–retest unreliability, inter-model disagreement, question-type sensitivity, and expectation gap — collectively demonstrate that current LLM interview evaluators fail basic psychometric reliability standards.

The most concerning finding is that these failures are not merely theoretical: they occur at the frontier of LLM capability, across all three major model families, and on the most widely-used behavioral interview preparation materials on the internet. Scale collapse means the rating instrument lacks discriminative power. Test–retest jitter means outcomes are partially random. Inter-model disagreement means outcomes depend on vendor choice. Question-type sensitivity means outcomes depend on question framing. The expectation gap means the evaluator’s standards may not reflect human judgment.

Any LLM-based interview scoring system that reports only point estimates without cross-question variance, inter-model rank correlation, and test–retest metrics is masking real noise with false precision. We urge practitioners, policymakers, and researchers to adopt the minimum reliability checklist proposed in this paper before deploying LLM evaluators in hiring contexts.

Appendix A.1. System Configuration

• Python Version: 3.11+
• LLM Integration: LiteLLM library
• Async Processing: asyncio with max 5 concurrent requests
• Experiment Dates: March 25–27, 2026

Appendix A.2. Model Access

All models accessed via LiteLLM unified API. API keys stored in environment variables; no secrets committed to repository.

Appendix A.3. Data and Code Availability

The complete evaluation framework, analysis scripts, and raw experimental data are available at the project repository. The answer source is publicly available at https://github.com/ashishps1/awesome-behavioral-interviews.

Appendix B.1. Full Rating Distribution by Cell

See viz_rating_stats_summary.csv in supplementary materials for complete per-cell rating distributions across all 141 evaluation cells.

Appendix B.2. Full Pairwise Spearman Correlation Matrix

See reliability_pairwise_spearman.csv for all model-pair correlations stratified by company and level.

Appendix B.3. Test–Retest Detail

See reliability_rep_per_group.csv and reliability_rep_summary_by_run_model.csv for detailed replicate-level agreement data.