Existing vehicle travel time prediction applications face two primary challenges: modeling complex road network structures and handling irregular spatio=emporal traffic state propagation. To address these challenges, this paper proposes a novel travel time prediction method, GMTP, based on vehicle GPS data. The proposed GMTP method improves the original attention mechanism ,by introducing shared attention mechanism with high computational efficiency, and integrates the graph attention network GATv2 and the long sequence language model BERT. This enables the adaptive analysis of dynamic correlations between road segments across broad spatial and temporal dimensions. The pre-training process of the model consists of two blocks. In the first block, a segment interaction pattern-enhanced graph attention network is employed to convert road network structural features and interaction semantics into road segment representation vectors. In the second block, a traffic congestion-aware trajectory encoder maps these road segment representations to trajectory representation vectors incorporating traffic time characteristics for encoding. Additionally, two self-supervised tasks are designed: adaptive masked trajectory reconstruction task and trajectory contrastive learning, which aims to enhance the model’s accuracy and robustness. Finally, the model is fine-tuned on two large-scale real-world trajectory datasets to validate its effectiveness. Experimental results demonstrate that the proposed method generalizes well across different cities and adapts to heterogeneous trajectory datasets. Moreover, key evaluation metrics such as MAE, MAPE, and RMSE are significantly reduced, and computational efficiency is improved.