Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark

1. Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in software development [1,2,3], which includes their ability to detect and resolve bugs. Various benchmarks are designed to evaluate these capabilities, such as QuixBugs [4], HumanEvalFix [1], BugsInPy [5] and MDEVAL [6].

The most widely acknowledged bugfix benchmark is probably HumanEvalFix, a component of the HumanEvalPack [1]. In this benchmark, each of the 164 canonical solutions from the original HumanEval code generation benchmark [7] has been manually corrupted by introducing a bug. The task of the LLM is to repair the buggy function. The modified output is then verified using the original test cases.

Although reliable reports of benchmark results are essential to compare models, it is not uncommon to see differing benchmark results for the same model throughout studies [8,9,10]. In the broader community, researchers can struggle to reproduce the officially reported benchmark results. The difference between reported and reproduced results can be minor, but it can also be significant. This not only complicates comparison of models but also questions the validity of the currently published benchmark results.

In this paper, we address the issue of inconsistent benchmark results on the HumanEvalFix benchmark. We look for discrepancies in a broad range of results published in the literature, considering models with scores reported in two or more studies. To identify the underlying reasons of these discrepancies, we conduct our own evaluations using 11 models of various sizes and tunings: DeepSeekCoder (1.3B and 6.7B, base and instruct) [3], CodeLlama (7B and 13B, base and instruct) [2], and CodeGemma (2B base, 7B base and instruct) [11]. Our findings are as follows:

• We reproduce 16 of the 32 scores reported for the evaluated models. This implies that many of the discrepancies in reported results can be attributed to using different, or possibly incorrect evaluation settings.
• We quantify the impact of modifying evaluation settings individually. For instance, maximum generation length, sampling strategy, 4-bit quantization, and prompt template choice significantly influence the results, whereas precision and 8-bit quantization do not.
• We identify instances of possibly misreported results, likely due to confusion between base and instruction-tuned models.

2. Method

We begin by introducing the benchmark we focus on, HumanEvalFix, as well as the evaluation framework commonly used to run it. Then we identify the discrepancies across reported results by conducting a literature review of the benchmark scores. Finally, we analyze GitHub issues about unsuccessful reproduction of published results to determine possible reasons for these discrepancies and our experimental setup.

2.1. Benchmarking with HumanEvalFix

The authors of the HumanEvalFix benchmark have introduced a bug into each of the 164 canonical solutions in the HumanEval benchmark, thereby turning it into a bugfixing benchmark. These bugs involve missing logic, extra logic, or incorrect logic. The benchmarked LLM is prompted with the incorrect function, along with unit tests, and is tasked with repairing the function. The output code is then validated using the same unit tests found in the original HumanEval benchmark. Since the Python version of this benchmark is the most widely used, we will also focus on this variant.

For conducting our evaluations, we use the Code Generation LM Evaluation Harness [12] framework, provided by the BigCode Project. This framework includes multiple benchmark implementations, such as HumanEval, MBPP, and QuixBugs, as well as the HumanEvalFix benchmark. Most of these benchmarks evaluate functional correctness of code outputs using the pass@k metric, which indicates the percentage of outputs that pass the tests at least once in k attempts. Users of the framework can adjust a range of evaluation settings, such as the maximum generation length, batch size, and sampling parameters.

2.2. Review of Reported Benchmark Results

We conduct a review of 18 papers that contain results of the HumanEvalFix benchmark [1,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28], focusing on their reported HumanEvalFix results. These papers report benchmark results for a total of 117 different models (including variants in sizes and tunings). Out of the 117 models mentioned, 26 appeared in at least two papers. These models form the basis of our investigation. Their reported results are visualized in Figure 1, as well as in Table 1 with all relevant citations.

Six out of 26 models have identical reported results across studies (e.g. InstructCodeT5+, StarCoder). This suggests that the initially published result was cited by other authors without trying to reproduce it. For 4/26 models, results differ only slightly (e.g. OctoCoder, Llama3-Inst-8B). Small differences are accepted in many cases as they can be due to minor differences in configuration settings. However for most of the models, more precisely in 16/26 cases, the results differ significantly (e.g. CodeLlama, DeepSeekCoder). For example, the reported results for DeepSeekCoder-Instruct (1.3B) range from $9.1\%$ to $48.9\%$. Similarly, for CodeLlama-Instruct (13B), six benchmark results are reported, ranging from $15.2\%$ to $30.5\%$, nearly evenly spread. According to one reported result, DeepSeekCoder-Instruct (33B) should be one of the top models for bugfixing with an exceptionally high score of $81.0\%$, yet two other studies report a much lower score of $47.5\%$.

As the benchmark dataset size is exactly 164, the benchmark results should be percentages that are calculated through dividing by 164. Since in most cases this is an infinitely long fraction, reported values are usually rounded or floored to one or two decimals. Notably, we have observed that in certain cases, the reported result cannot be obtained either by flooring or rounding. For example, the reported pass@1 score of $44.9\%$ (for DeepSeekCoder-Inst 6.7B) cannot be correct: if 73 fixes were successful, the score would be $44.51\%$, and if 74 fixes were successful, it would be $45.12\%$. Throughout the reported scores across the analyzed papers (from Figure 1 and Table 1), 34 reports have such an unobtainable score out of 85. Considering the unique scores only, 20 scores have this property out of 57. Such scores are hard to explain. They could be a result of a simple mistake, such as inaccurate rounding/flooring, or a more severe one, such as confusion with a different benchmark score.

2.3. Experimental Setup

The goal of our experiments is twofold. First, we examine the effect of a wide range of settings on the results of models mentioned in multiple studies. Then, we try to reproduce results reported in the literature by finding out the evaluation settings they could have used. In this section, we determine the experimental setup for the evaluations.

To explore potential causes of differences in benchmark results, we reviewed the reported issues about unsuccessful attempts at reproducing published results in the GitHub repository of the benchmarking framework1. At the time of investigation, 145 issues were published (47 open and 98 closed)2. Thirteen out of the 145 issues (6 open and 7 closed) were about unsuccessful reproduction of officially reported results. We analyzed the issues that were either closed or had helpful answers to them (10 in total), and found these causes:

• A differently tuned version of the model was used (e.g. the base model instead of its instruction-tuned variant), causing a large difference in benchmark result. (2 issues)
• The wrong variant of the same benchmark was used: MBPP instead of MBPP+, and HumanEval instead of its unstripped variant. (2 issues)
• The temperature was set incorrectly: while generating one sample for each prompt, the temperature $T=0.8$ turned out to be an improperly large value. (1 issue)
• An incorrect prompt was used, causing more than $6\%$ of a decrease in pass@1 performance. (1 issue)
• The reproduced results were different by only a few percentage points compared to the officially reported ones. Such a discrepancy was considered negligible, as this can be due to minor variations in evaluation settings, minor updates in model versions, or in hardware configurations. (4 issues)

We chose 11 models for evaluation, most of which have multiple reported results across papers. We evaluated these models on the HumanEvalFix benchmark, in different settings. The models used are: DeepSeekCoder (1.3B and 6.7B), DeepSeekCoder-Instruct (1.3B and 6.7B), CodeLlama (7B and 13B), CodeLlama-Instruct (7B and 13B), CodeGemma (2B and 7B) and CodeGemma-Instruct (7B). In our experiments, we consider these evaluation settings and values:

Prompt template

The evaluation framework defaults to the instruct prompt template, with only the context and instruction (the program and the instruction to fix it). The default prompt template lacks model-specific formatting suggested by model authors. We evaluate models both with their suggested prompt template as well as the default instruct setting.

Sampling vs. greedy decoding

The default behavior in the framework is not to use greedy decoding, but to apply sampling with the temperature $T=0.2$. Alongside greedy decoding, we conduct experiments using sampling with temperatures $T=0.2$ and $T=0.8$.

Limiting generation length

The default value of 512 tokens is insufficient, as it can lead to unfinished programs and reduced benchmark scores. We conduct our experiments using lengths of 512 and 2048. Since the max length generation parameter considers both the prompt and the generated output, it must be large enough to prevent cutting the output short. We found 2048 to be a good choice on this benchmark, as tokenizers typically fit the inputs within 1024 tokens, leaving enough space for the generated output.

Precision and quantization

The majority of LLMs are released using bf16 precision, making it the preferred choice for precision. We evaluate models using all three common precision formats: fp16, fp32, and bf16. While quantization is a useful technique to lower memory usage when running models, it’s generally best to use models without it. In addition to our tests without quantization, we also run experiments using 4-bit and 8-bit quantization (with fp16 precision).

To assess the effect of each parameter setting individually, we select a configuration as the baseline. Then we compare the evaluation results of this baseline configuration with those obtained when only one setting differs. In the baseline setting, the prompt format follows the recommendations of the model’s authors, no sampling is applied, the maximum generation length is set to the sufficiently large value of 2048, and bf16 precision is used without quantization.

Next, we attempt to reproduce the results reported in the literature. To do this, we run an exhaustive grid search for each model, with all possible combinations of the evaluation settings. As we have 2 options for prompt, 3 for sampling and temperature, 2 for generation length, and 5 for precision and quantization, we evaluate every single model in 60 settings.

While our main focus is on instruction-tuned models – as these are more frequently evaluated throughout studies –, we also take results of base models into consideration to assess whether they reproduce results originally attributed to their instruction-tuned counterparts.

The experiments are performed on A100 GPUs with 40GB VRAM each. We set max_memory_per_gpu to the value of auto, which distributes the model on the assigned GPUs. We utilize only as many GPUs as required to load the model and execute the benchmark, which is typically just one. Where multiple GPUs were used, we also measured whether modifying the number of assigned GPUs has an influence on results; we have not seen any variation in benchmark results when increasing the number of utilized GPUs.

3. Experimental Results

In this section, we present the results of our evaluations. First, we examine the impact of each evaluation parameter individually, compared to the baseline configuration. Next, we perform an exhaustive grid search for each model and assess whether the reported results can be reproduced. All evaluation results are summarized in Table 2.

3.1. The Effect of Individual Evaluation Settings

We measure the effect of individual evaluation settings by comparing the baseline configuration against variants in which evaluation parameters are modified individually. The effect of each parameter change on the benchmark scores is analyzed in isolation. The impact of these modifications is visualized in Figure 2.

3.1.1. Temperature: Greedy Evaluation and Sampling

Considering all models, sampling with temperature T=$0.2$ has a slight impact on average performance: $-0.95\%$. The change in performance is more significant when using T=$0.8$: $+7.94\%$. However, by only considering the instruction-tuned models, this change is less notable: $+0.40\%$ (with T=$0.2$) and $+3.63\%$ (with T=$0.8$).

For the baseline of greedy decoding, the standard deviation of scores across precision and quantization settings is 1.58 for all models and 1.98 for instruction-tuned models. When we switch to sampling at T=$0.2$, the standard deviation rises to 1.94 ($+22.99\%$) for all models and 2.48 ($+25.03\%$) for the instruction-tuned ones. Increasing the temperature further to T=$0.8$ increases standard deviation even more, to 2.37 ($+50.40\%$) for all models and 3.03 ($+52.78\%$) for the instruction-tuned models.

3.1.2. Maximum Generation Length

In the case of DeepSeekCoder-Instruct (1.3B) (evaluated using fp16), we see a notable decrease in performance ($21.34\%\to 16.46\%$) when limiting the length of generation to the default value (to 512). This is a significant effect with a relative drop of $22.87\%$. The effect also applies more generally: across all models, the average performance drop is $20.63\%$. Similarly, for the instruction-tuned models, this drop is $21.37\%$.

3.1.3. Using an Incorrect Prompt Template

Considering the instruction-tuned models, switching from the suggested template to the default instruct prompt template resulted in average relative drop of $7.26\%$ in model performance. This phenomenon is not observable when base models are also included in the calculation: the average performance even increases by $2.54\%$.

3.1.4. Precision and Quantization

By switching from bf16 to fp32 or fp16 precision, performance drops by $2.22\%$ (fp32) and $1.59\%$ (fp16) for all models, and increases by $1.61\%$ (fp32) and $2.82\%$ (fp16) for instruction-tuned models. Similarly, 8-bit quantization seems to have a rather small effect with changes of $-0.95\%$ (all models) and $+3.63\%$ (instruction-tuned). However, using 4-bit quantization decreases overall scores by a significant margin: $-13.33\%$ for all models and $-8.47\%$ for the instruction-tuned ones.

3.2. Reproducibility of Results in Existing Research

We attempt to reproduce existing results in the literature by trying out all possible combinations of the experimental settings. Our main goal is to identify potential misconfigurations in the original setups. All results from our evaluations as well as scores from existing studies can be seen in Table 2.

3.2.1. CodeLlama

The results reported in studies for CodeLlama-Instruct (7B) range from $15.2\%$ to $28.0\%$. This is in line with our evaluations, in which this model achieves scores from $14.02\%$ to $25.61\%$. We reproduced all of the reported scores with one exception, indicating that the differences in reported results are indeed caused by variations in the evaluation settings. For CodeLlama-Instruct (13B), reported results range from $15.2\%$ to $30.5\%$. Our evaluation yielded considerably lower scores: $10.37\%$ to $23.17\%$. We reproduced 4 out of 6 results.

The base CodeLlama (13B) model has two reported scores across papers: $6.1\%$, which we reproduced, and an unexpectedly high score of $33.1\%$. The latter score is very close to the upper range of scores reported for CodeLlama-Instruct (13B).

3.2.2. DeepSeekCoder

The reported results for the DeepSeekCoder (1.3B and 6.7B) base models go up as high as $16.4\%$ and $45.4\%$ for the 1.3B and 6.7B variants, respectively. These scores are unusually high, especially when considering our own evaluations, which never exceed $11.59\%$ (1.3B) and $35.37\%$ (6.7B).

DeepSeekCoder (1.3B) for example is measured to have both $1.2\%$ and $16.4\%$ pass@1 score, which represents more than a 10X difference. We evaluated this model and its instruction-tuned variant and reproduced both reported results: $1.2\%$ was reproduced using the base model, while the score of $16.4\%$ was only reproducible with the instruction-tuned variant. Using the 6.7B base model, we reproduced two of the reported results, but not the very high score of $45.4\%$. Its instruction-tuned variant however, has yielded multiple results close to this value, with the closest being $45.12\%$.

For the instruction-tuned 1.3B model, one of three values was reproduced. The score of $9.1\%$ might come from the base-model, as some of our evaluations yielded values close to it. Our evaluation results of the 6.7B instruction-tuned model range from $31.71\%$ and $52.44\%$. These are partially in line with the reported results, with two very close results.

3.2.3. CodeGemma

Our evaluations of the CodeGemma (2B) base model attained up to $12.20\%$ for the 2B model with one of the two reported scores reproduced. Using the 7B base model, values go up to $17.07\%$ with one exact match and one near match to the reported scores; the third score was significantly higher. In our evaluations of CodeGemma-Instruct (7B), we reproduced one of the two reported results.

4. Discussion

The results of our reproducibility experiments highlight several possible reasons for the discrepancies between results in the literature. They also point to some strategies that could help prevent these discrepancies.

In some cases, we see surprisingly large scores for base models. We think this is due to confusing the base model with its instruction-tuned variant. It is crucial to ensure that the intended model variant is used, not one with different tuning. Furthermore, it is necessary to confirm that the correct model name and type is specified when reporting evaluation results.

The pass@1 scores do not change significantly when switching from greedy evaluation to sampling, especially for instruction-tuned models. However, the standard deviation increases, especially with higher temperature values. Thus, we observe that greedy evaluation should be preferred over sampling when calculating the pass@1 performance on this task.

Restricting the models to a short generation length affects the results negatively, resulting in large drops in performance. In order to obtain accurate results, a proper limit should be chosen to avoid cutting possibly correct generations. In the case of our benchmark of focus, HumanEvalFix, 2048 is such a value, but this might differ for other benchmarks.

Using the proper prompt template (defined by model authors) is necessary to ensure stable and reliable evaluation results. Without it, the evaluation may not reflect accurate results. This only applies to instruction-tuned models, as base models were not fine-tuned to process such templates.

We noticed only a slight variation when modifying precision settings or using 8-bit quantization, suggesting these factors do not strongly account for differing results across papers. However, 4-bit quantization does have a notable effect.

5. Conclusion

In this paper, we highlighted the issue of inconsistent benchmark results in the literature on the HumanEvalFix benchmark, one of the most widely used benchmarks for evaluating bugfixing. We found that for most of the models mentioned in multiple papers, the reported benchmark results can vary significantly. We conducted evaluations to determine the effect of various evaluation settings, and to reproduce different scores reported for the same models, in order to uncover the reasons for these inconsistencies.

Through a series of emprirical evaluations, using multiple models in different sizes and tunings, we identified the set of factors influencing benchmark results. Our experiments revealed that factors such as the prompt template, maximum generation length and decoding strategies had a notable influence on benchmark results, whereas precision and 8-bit quantization did not. We have also found cases where results were likely misattributed between instruction-tuned and base models. We wish to emphasize the importance of using appropriate evaluation settings and including these settings in the studies to ensure reliable and reproducible results.