Preprint
Article

This version is not peer-reviewed.

Which Way do Small-Sample Confidence Intervals for a Difference of Means Fail? Coverage, Directional Error and the Signed Skewness Balance

Submitted:

16 November 2026

Posted:

18 November 2026

You are already at the latest version

Abstract
Background: The difference between two arithmetic means remains a central target in small independent-group studies, yet total interval coverage does not say on which side an interval fails. We evaluated coverage, the side of noncoverage and interval width for 95% intervals, and tested a predeclared hypothesis about the direction of failure. Methods: A predeclared simulation study (plan dated before the first run) compared Welch, percentile-bootstrap, basic-bootstrap, bootstrap-t and BCa intervals in 147 settings with 10,000 simulated datasets and 1,999 separate-group resamples each. Seven population families (including left-skewed and opposed-skew designs, so that direction is not built into the design), seven sample-size allocations from (5,5) to (100,100) and variance ratios 1, 4 and 10 were crossed. The primary outcomes were coverage, the probability L that the interval lies entirely below the true difference, the probability U that it lies entirely above it, and their difference. The predeclared hypothesis was that the sign of L-U follows the sign of Γ, the standardised third cumulant of the difference in sample means. Earlier exploratory stages are reported separately and not pooled. Results: For Welch intervals the sign of L-U agreed with the sign of Γ in 66 of 68 determinate settings (97.1%); none of 78 settings with Γ=0 showed imbalance beyond three Monte Carlo standard errors. For right-skewed lognormal data with variance ratio 1 the failure was on the lower side when the small group was Group 1 (+14.4 percentage points) and on the upper side when it was Group 2 (-14.2). With opposed skew in the two groups, imbalance was +14.4 points at n=5 and +5.8 at n=100 per group, against about zero for same-shape skew. The first-order size of the imbalance was over-predicted (slope 0.56 for groups of at least 30). Worst-case Welch coverage was 68.1% (varying-shape lognormal) and 80.9% (fixed-shape lognormal). Bootstrap-t halved the median imbalance at group sizes of 30 and above (median ratio to Welch 0.53) but not at group sizes of 10 or below. Percentile, basic and BCa intervals under-covered even for normal data at n≤10 (85.6–92.2% across such settings). Conclusions: In small samples the side on which an interval fails is systematic and predictable in sign from the balance of skewness between groups weighted by their contribution to the variance of the difference, whereas its size is not captured by the first-order term. No procedure was uniformly reliable. Studies should report lower- and upper-side noncoverage, not total coverage alone.
Keywords: 
;  ;  ;  ;  ;  ;  ;  
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.