Prediction of protein-ligand binding affinities is crucial for computational drug discovery. A number of deep learning approaches have been developed in recent years to improve the accuracy of such affinity prediction. While the predicting power of these models have advanced to some degrees depending on the dataset used for training and testing, the effects of the quality and quantity of the underlying data have not been thoroughly examined. In this study, we employed erroneous datasets and data subsets of different sizes, created from one of the largest databases of experimental binding affinities, to train and evaluate a deep learning system based on convolutional neural networks. Our results show that data quality and quantity do have significant impacts on the performance of trained models. Depending the variations in data quality and quantity, the performance differences could be comparable to or even larger than those observed among different deep learning approaches. This implies that continued accumulation of high-quality affinity data is important for improving deep learning models to better predict protein-ligand binding affinities.