(1) Background: Breast cancer is the leading malignancy worldwide and in Ghana it has a poor overall survival. However, approximately 50% of cases are early stage disease and with advances in breast cancer treatment and improvement in survival, quality of life (QOL) becomes as im-portant as treatment of the disease. (2) Methodology: This was a cross-sectional study of survivors who had breast conserving surgery (BCS), mastectomy only (M) and mastectomy with breast re-construction (BRS) from 2016 to 2020 at a tertiary hospital in Ghana, comparatively assessing their QOL using EORTC QLQ C-30 and EORTC QLQ BR-23. (3) Results: The study participants had an overall Global Health Status (GHS) median score of 83.3 [IQR: 66.7-91.7] with no significant differences between the surgery types. The BRS group had a statistically significant lower median scores for the Functional scale (82.8 and 51.0) and the highest scores for the Symptomatic scale (15.7 and 16.5). Body image was significantly lowest for the BRS group 83.3 [68.8-91.7] and highest 100 [91.7-100] for the BCS group (p < 0.001). (4) Conclusion: There is a need to develop support systems tailored at improving the QOL of breast cancer survivors taking into consideration the type of surgery performed.