Crack inspection is important to monitor the structural health of pavement structures and to facilitate an easier rehabilitation process. Currently, pavement crack inspection is conducted manually, which is inefficient and costly at the same time. For solving the problem, this work has developed a robotic system for automated data collection and analysis in real-time. The robotic system navigates on the pavement and collects visual images from the surface. A deep learning-based semantic segmentation framework named RCDNet was proposed and implemented on the onboard computer of the robot to identify cracks from the visual images. Simulation results show that the deep learning model obtained 96.29% accuracy for predicting the images. The proposed robotic system was tested on both indoor and outdoor environments and was observed that it can complete inspecting a 3m × 2m grid within 10 minutes and a 2.5m × 1m grid within 6 minutes. This outcome shows that the proposed robotic method can drastically reduce the time of manual inspection. Furthermore, a severity map based on the results from visual images was also generated to provide an idea of which locations should be paid more attention to repair in a test grid.