The ravage of COVID-19 is not merely limited to taking its toll with half a million fatalities. It has halted the world economy, disrupting normalcy of lives with supervening severity than any other global catastrophe of the last few decades. The majority of the vaccine discovery attempts are still on trial, making early detection and containment the only feasible redress. The existing diagnostic technique with high accuracy has the setbacks of being expensive and sophisticated, requiring skilled individuals for specimen collection and screening resulting in lower outreach. So, methods excluding direct human intervention are much sought after, and artificial intelligence-driven automated diagnosis, especially with radiography images, captures the researchers' interest. This survey marks a detailed inspection of the deep-learning-based automated detection of COVID-19 works done to date, methodical challenges along with probable solutions, and scopes of future exploration in this arena. We also provided a comparative quantitative analysis of the performance of 315 deep models in diagnosing COVID-19, Normal, and Pneumonia from x-ray images. Our results show that Densenet201 model with Quadratic SVM classifier performs the best (accuracy: 98.16\%, sensitivity: 98.93\%, specificity: 98.77\%) and maintains high accuracies in other similar architectures as well. This proves that even though radiography images might not be conclusive for radiologists, but it is so for deep learning algorithms for detecting COVID-19. We hope this extensive review will provide a comprehensive guideline for researchers in this field.