The question of how many deaths were averted by interventions during the COVID-19 pandemic carries important implications for policies going forward. Given that the interventions of lockdowns and mass vaccination carry acknowledged downsides, it is important to balance these against potential upsides, which were the putative reasons for health officials to implement these policies. Several attempts have been made to quantify the impact of mass vaccination on averted mortality both globally, and in specific countries/regions, using different methodologies. Here, we examine the assumptions of these models and look for areas of improvement, while understanding that simplifications are inherent in model building. We find that several assumptions greatly overstate the degree of averted mortality due to vaccination and perform an empirical analysis of country level data in Europe to test if vaccination was associated with lower excess mortality. We show a positive and statistically significant correlation between number of vaccine doses given and 2022 excess mortality, calling into question estimates of positive averted mortality due to mass vaccination. Our results show several assumptions which work to systematically overstate the level of averted mortality from Covid-19 vaccines. This work aims to improve epidemiological modeling on the impact of vaccination, and to ground the public health response to infectious diseases in robust and rigorous analysis.