Transcriptome-Based Subtype Discovery in Breast Cancer Using Variational Representation Learning

Breast cancer is a heterogeneous disease whose molecular complexity complicates therapeutic stratification. Although PAM50 is a widely used molecular subtyping framework, discrete subtype assignments may underrepresent transitional molecular states and within-subtype heterogeneity. We therefore developed a biologically interpretable representation-learning framework that projects bulk RNA-seq profiles into nonlinear latent spaces and applies unsupervised clustering to investigate additional tumor structure. The pipeline integrates a variational autoencoder (VAE), t-SNE and UMAP visualization, agglomerative hierarchical clustering, and functional and immune characterization of the resulting clusters. The source dataset contained 19,131 transcriptomic profiles. An analytical cohort of 8,749 unique PAM50-labeled tumor and normal samples was used for downstream characterization, and label-informed oversampling generated 54,544 training records for VAE optimization. The VAE objective and clustering remained unsupervised, but the sampling scheme used known labels to balance representation during training. Three latent dimensions (Z = 10, 20, and 40) were benchmarked against a linear PCA baseline. Across internal analyses, the Z = 20 representation achieved the highest agreement with the reference labels (ARI = 0.5543) and identified seven clusters not resolved as clearly by PCA. These included a cluster spanning HER2-enriched and Luminal B samples, a highly proliferative cluster, and a B-cell/Treg-enriched group with a maximum inferred regulatory T-cell fraction of 0.711. The Z = 10 configuration yielded the highest prognostic discrimination among the tested representations (C-index = 0.553 versus 0.467 for the PAM50-based comparator), corresponding to an absolute difference of 0.086 and a relative increase of 18.4%. These internally derived findings suggest that nonlinear representations may capture complementary biological and prognostic structure, but independent-cohort and spatial or single-cell validation are required before clinical interpretation.