S-BGM: Stable Self-Evolving LLMs via Cognitive Bipartite Graph Modeling

1. Introduction

To alleviate the reliance on human supervision, self-evolving frameworks have emerged as a promising paradigm in which LLMs autonomously generate training data and derive reward signals to guide their own optimization [1,2,3,4]. A typical self-evolving framework generally comprises two roles: a Questioner responsible for generating training questions and a Solver tasked with answering them [5]. These two components are trained in an alternating fashion to facilitate co-evolution, as illustrated in Figure 1(a). Existing self-evolving models are often subject to a critical limitation: as training iterations progress, the Solver’s performance initially improves rapidly but soon plateaus and may even degrade, a phenomenon commonly referred to as model collapse [6,7]. Prior work has sought to address this issue from several perspectives, including introducing external data [8,9], designing reliable verification signals [10,11], and improving the diversity of training data [12,13].

However, existing approaches for mitigating model collapse generally concentrate on a single role (the Questioner or the Solver), overlooking the evolving dynamics of the holistic Questioner-Solver system. This narrow focus addresses only the symptoms rather than the underlying cause, leading to the challenges of the short-term and separate training reward. In the general self-evolving paradigm, each role is optimized only according to the immediate reward in the current iteration, without considering how this update will affect the long-term evolution of the coupled system. For instance, upon discovering that the current Solver is uncertain about calculus problems, the Questioner may generate more difficult calculus questions in order to maximize its short-term reward. This behavior may far exceed the Solver’s capability upper bound, leading to model collapse. In addition, the Questioner and Solver are rewarded as separate components instead of a whole. Optimized against fixed counterparts, the Questioner exploits momentary uncertainty instead of exploring broadly, and the Solver fits a static distribution, producing rote answers without diverse reasoning [14]. Consequently, as shown in Figure 1(b) and (c), the edges in the initial state denote broad exploration across mathematical domains, which collapses into concentrated edges in model collapse.

In this paper, we argue that self-evolving training should incorporate dynamic signals to depict stability at the system level, rather than focusing solely on role-level rewards. A straightforward strategy is to store and utilize all historical checkpoints in a general fashion [15]; however, this leads to prohibitive resource costs and yields disconnected rewards. Differently, we propose modeling the dynamics of the self-evolving system through a bipartite graph, where the two disjoint vertex sets represent the semantic distribution of questions generated by the Questioner and the problem-solving ability of the Solver. Edges across the two node sets depict how the Questioner and the Solver interact globally, allowing the coupled behavior of the two roles to be analyzed as a unified objective. When a question is generated and a response is produced for that question, the corresponding nodes are connected based on this interaction, with the edge weight recording the interaction frequency. During self-evolving training, new interactions strengthen or create edges, while less frequent interactions gradually weaken or remove edges, allowing the graph topology to track the long-term evolution of the coupled system. From this perspective, model collapse can be identified as a topological concentration phenomenon in this bipartite interaction graph. As shown in Figure 1(b,c), the initially broad interaction topology gradually collapses into a small subset of states, reflecting repetitive question generation and increasingly narrow Solver behavior. A stable self-evolving system in Figure 1(d) instead maintains a distributed interaction topology, where diverse question patterns continue to induce varied Solver behaviors throughout the co-evolution process.

Building on this insight, we propose S-BGM, a framework for Stable self-evolving LLMs based on Bipartite Graph Modeling. First, we formalize self-evolving as a coupled Questioner-Solver system from a mathematical bipartite graph view. After that, a Cognitive Bipartite Graph is constructed by discretizing the Questioner’s question space into semantic clusters and the Solver’s response space into uncertainty intervals, such that each interaction induces an edge across the two partitions. To quantify system stability from a holistic perspective, we propose the structural entropy to measure the dispersion of interaction weights over the bipartite graph topology, thereby reflecting whether the Questioner-Solver system remains broadly coupled or collapses into a concentrated interaction pattern. Such entropy is added into the training signal of the Questioner and Solver to encourage each role to optimize its local reward while avoiding excessive concentration of the global interaction structure. The Cognitive Bipartite Graph is updated by a sliding window that ensures newly observed interactions strengthen the corresponding edges, while outdated interactions are removed. Thus, the graph continuously tracks the evolution of the coupled system for stabilizing self-evolving training. Our main contributions are summarized as follows:

• As far as we know, we are the first to propose model self-evolving systems as a Cognitive Bipartite Graph to facilitate long-term interactions in a holistic system view.
• We propose S-BGM, a novel self-evolving paradigm integrated with the proposed structural entropy loss and dynamic graph topology update scheme, providing system-level stability signals instead of traditional role-level rewards.
• Comprehensive evaluations across mathematical and general reasoning benchmarks show that S-BGM reliably enhances end-task performance, maintains curriculum learning dynamics, and enables more stable long-horizon self-evolution than representative baselines.

2. Preliminary Knowledge and Analysis

2.1. Cognitive Bipartite Graph Modeling for Self-evolving System

A typical self-evolving system is composed of two roles: a questioner model ${\mathcal{M}}_Q$ and a solver model ${\mathcal{M}}_S$. To characterize the interaction dynamics between two roles, the system is modeled as a Cognitive Bipartite Graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{W})$, where the vertex set consists of two disjoint partitions:

• ${\mathcal{V}}_Q=\{v_1,\dots ,v_m\}$ (Question-Side Semantic States of ${\mathcal{M}}_Q$): Each node $v_i\in {\mathcal{V}}_Q$ represents a state in the question-generation space of ${\mathcal{M}}_Q$, characterizing the semantic distribution of questions produced by the Questioner.
• ${\mathcal{V}}_S=\{u_1,\dots ,u_n\}$ (Solver-Side Problem-Solving States of ${\mathcal{M}}_S$): Each node $u_j\in {\mathcal{V}}_S$ represents a behavioral state in the response space of ${\mathcal{M}}_S$, characterizing how the Solver responds to generated questions.

Each edge $(v_i,u_j)\in \mathcal{E}$ represents an observed interaction between the Questioner and the Solver, where a question-side state $v_i$ is associated with a response-side state $u_j$. The edge weight $w_{ij}$ quantifies how frequently interactions between the question-side state $v_i$ and the response-side state $u_j$ are observed during self-evolving training. Under this abstraction, the evolving state of the system is fully represented by the bi-adjacency matrix $\mathbf{B}\in {\mathbb{R}}^{m\times n}$ throughout the training process, where ${\mathbf{B}}_{ij}=w_{ij}$ for observed interactions $(v_i,u_j)\in \mathcal{E}$ and ${\mathbf{B}}_{ij}=0$ otherwise.

A straightforward strategy is to treat each question or answer as an independent node. However, this causes the graph to grow with the number of samples, resulting in a sparse and unreliable structure. To address this, microscopic interactions are discretized by projecting them onto a finite set of macroscopic cognitive states. On the ${\mathcal{M}}_Q$ side, semantic-based discretization is applied: the high-dimensional embedding space of generated questions is partitioned into m representative semantic regions, and each question q is mapped to a vertex $v_i\in {\mathcal{V}}_Q$ by nearest-centroid proximity. On the ${\mathcal{M}}_S$ side, uncertainty-based discretization is applied: the consensus ratio across multiple Solver samples serves as a proxy for response uncertainty and is discretized into n intervals, each corresponding to a vertex $u_j\in {\mathcal{V}}_S$ that represents a distinct confidence level. Edge weights $\mathcal{W}$ reflect the frequency of recent interactions between these macroscopic states, so the bi-adjacency matrix $\mathbf{B}$ compactly represents the current system interaction status.

2.2. Understanding Model Collapse via Cognitive Bipartite Graph

Our research begins with an examination of the correlation between model collapse and the evolving topology of the cognitive bipartite graph. This study conducts five iterations of self-evolution within the R-Zero framework [5] on Qwen3-4B-Base [16]. Then, our study validate the Solver’s performance on the popular mathematical datasets used in Section 4.1. As illustrated in the left panel of Figure 2, R-Zero exhibits a typical model collapse, where the performance initially improves rapidly but soon plateaus and finally decreases. To understand model collapse from the perspective of the interaction graph $\mathcal{G}$, we analyze the normalized weighted-degree ratios of nodes at iterations 0 and 5. In the middle and right panels of Figure 2, the x-axis sorts nodes by descending normalized weighted-degree ratio, while the y-axis shows each node’s proportion of the total degree mass. After five iterations, both the Solver side ${\mathcal{V}}_S$ and the Questioner side ${\mathcal{V}}_Q$ show clear topological concentration. On the Solver side, the distribution becomes more skewed, indicating that interactions increasingly focus on a small number of response states. This suggests that the Solver becomes increasingly overconfident during training. On the Questioner side, the curve also becomes more concentrated: only a few nodes carry most of the weight, while most nodes are rarely visited. This indicates that the Questioner gradually focuses on a limited set of question types, reducing the diversity of the generated curriculum. These results show that model collapse can be measured as topological concentration in the Cognitive Bipartite Graph, motivating a topology-aware global signal to regulate self-evolving training. Please refer to Appendix E for the detailed theoretical proof.

3. Methodology

Figure 3 presents the overview of our proposal, including the construction of the Cognitive Bipartite Graph, the training of the Questioner and Solver under the constraint of structural entropy.

3.1. Cognitive Bipartite Graph Construction

Based on Section 2, the self-evolving system is modeled as a cognitive bipartite graph $\mathcal{G}=({\mathcal{V}}_Q\cup {\mathcal{V}}_S,\mathcal{E},\mathcal{W})$, where the two partitions characterize the evolving interactions between the two roles.

Questioner Node Mapping via Semantic Clustering. To represent the Questioner-side node set ${\mathcal{V}}_Q$, we discretize the continuous question space into semantic clusters, mapping semantically similar questions to the same node so that each node captures a coarse-grained question type rather than a single problem. Following previous work [13], we focus on mathematical reasoning and adopt the training set of the MATH dataset [17] as the reference corpus $\mathcal{D}$, which provides a fixed semantic coordinate system. Each problem in $\mathcal{D}$ is embedded by an embedding model $E(·)$, and K-means clustering is then applied to obtain m cluster centers $\mathcal{C}=\{c_1,c_2,\dots ,c_m\}$, corresponding to distinct mathematical sub-topics (e.g., number theory, calculus, probability). Each generated question q is then mapped to its nearest cluster center (node) by cosine similarity: $v^{*}\left(q\right)=arg{max}_{v_i\in \mathcal{C}}cos(E\left(q\right),c_i)$.

Solver Node Mapping via Uncertainty Estimation. On the Solver side ${\mathcal{V}}_S$, responses with similar uncertainty are assigned to the same node, capturing the Solver’s confidence. For a question q, the Solver draws R responses ${\left\{a_j\right\}}_{j=1}^R$. Following [3,5,8], the consensus answer $\tilde{a}$ is obtained by majority voting, yielding the consensus ratio $r\left(q\right)={\displaystyle \frac{1}{R}}{\sum }_{j=1}^R\mathbf{1}[a_j=\tilde{a}],$ where $\mathbf{1}$ is the indicator function. Larger $r\left(q\right)$ indicates higher confidence, smaller $r\left(q\right)$ denotes higher uncertainty. The response is then mapped to the node $u^{*}\left(q\right)=u_{min(\lfloor n·r(q)\rfloor +1,n)},$ with n the number of Solver-side nodes, thereby grouping responses with similar consensus ratios.

Cognitive Bipartite Graph Initialization. The cognitive bipartite graph is initialized as a stable reference state for subsequently evolved models, grounded in the vanilla Questioner and Solver. A set of L interactions are further sampled, in which each question is mapped to a Questioner-side node $v_i\in {\mathcal{V}}_Q$ via semantic clustering, and the corresponding response is mapped to a Solver-side node $u_j\in {\mathcal{V}}_S$ via uncertainty estimation. Starting from an all-zero adjacency matrix ${\mathbf{B}}^{\left(0\right)}\in {\left\{0\right\}}^{m\times n}$, the entry at $(v_i,u_j)$ is updated as ${\mathbf{B}}_{ij}^{\left(0\right)}\leftarrow {\mathbf{B}}_{ij}^{\left(0\right)}+1$. After processing all L interactions, the resulting bipartite graph ${\mathcal{G}}^{\left(0\right)}$ represents the initial stable state of the system.

3.2. Graph-Stabilized Questioner Training

The Questioner ${\mathcal{M}}_Q$ is trained with two signals: a role-specific reward measuring each question’s local usefulness, and a stability control mechanism applied to the normalized GRPO advantage, weighted by the Cognitive Bipartite Graph’s structural entropy.

3.2.1. Role-Specific Optimization Reward

Following prior works [5,9], role-specific rewards are employed to encourage the Questioner to generate questions that are both informative for the Solver and sufficiently diverse. For each generated question $q_i$, the Solver samples R candidate answers ${\left\{a_{i,j}\right\}}_{j=1}^R$. The pseudo-label ${\tilde{a}}_i$ is obtained via majority voting, while the consensus ratio ${\widehat{p}}_i$ is computed as an empirical estimate of the Solver’s confidence. Given that an effective question should ideally lie close to the Solver’s ability boundary, an uncertainty reward centered at ${\widehat{p}}_i=0.5$ is introduced: $r_{\mathrm{unc}}\left(q_i\right)=exp\left(-{\displaystyle \frac{{({\widehat{p}}_i-0.5)}^2}{2{\sigma }^2}}\right),$ where $\sigma$ controls the tolerance around the target uncertainty level. To discourage redundant question generation, a historical question embedding pool is maintained by ${\mathcal{P}}_{\mathrm{emb}}={\left\{E\left(q_k\right)\right\}}_{k=1}^M$. For each newly generated question $q_i$, its maximum cosine similarity $s_i$ to the pool ${\mathcal{P}}_{\mathrm{emb}}$ is computed, and the repetition penalty is subsequently defined as: $r_{\mathrm{rep}}\left(q_i\right)=max(0,s_i-\tau ),$ where $\tau$ is the similarity threshold. For valid questions, the final role-specific optimization reward is

$$\begin{array}{c}\hfill r_Q\left(q_i\right)=max\left(0,r_{\mathrm{unc}}\left(q_i\right)-r_{\mathrm{rep}}\left(q_i\right)\right).\end{array}$$

(1)

3.2.2. Structural Entropy for System Stability Estimation

Bipartite Graph Update. During self-evolving training, the sum of edge weights of in the cognitive bipartite graph is kept at size L and is updated with a sliding window to track recent interactions. For each generated interaction (i.e., edge) consisting of a Questioner-side node $v_i$ and a Solver-side node $u_j$, the corresponding entry in the adjacency matrix is updated as ${\mathbf{B}}_{ij}\leftarrow {\mathbf{B}}_{ij}+1$. At the same time, the oldest interaction $(v_{i^{\prime }},u_{j^{\prime }})$ is removed by performing the update ${\mathbf{B}}_{i^{\prime }{j}^{\prime }}\leftarrow {\mathbf{B}}_{i^{\prime }{j}^{\prime }}-1$. The resulting graph ${\mathcal{G}}^{\left(t\right)}$ summarizes the latest L interactions and supports the system stability estimation.

Structural Entropy Calculation. We define structural entropy to quantify the dispersion of interaction weights across the bipartite graph. It measures whether Questioner–Solver interactions are broadly distributed over the state space or concentrated on a limited subset of states. This distribution pattern serves as an indicator of the system’s current structural status, and thereby provides a global stability signal. Inspired by [18], let $d_i$ be the weighted degree of node i, $\mathrm{vol}\left(\mathcal{G}\right)={\sum }_i{d}_i$ be the graph volume, and $\mathcal{T}$ be the partitioning tree of height K to represent a hierarchical clustering of the vertices. The K-dimensional structural entropy of $\mathcal{G}$ is defined as the minimum structural entropy $H^{\mathcal{T}}\left(\mathcal{G}\right)$ achieved across all possible partitioning trees of height K:

$$\begin{array}{ccc}\hfill H^{\mathcal{T}}\left(\mathcal{G}\right)& =-\sum _{\alpha \in \mathcal{T},\alpha \ne \lambda }{\displaystyle \frac{g_{\alpha }}{\mathrm{vol}\left(\mathcal{G}\right)}}{log}_2{\displaystyle \frac{V_{\alpha }}{V_{{\alpha }^{-}}}},H\left(\mathcal{G}\right)\hfill & \hfill =\underset{\mathcal{T}:\mathrm{height}\left(\mathcal{T}\right)=K}{min}{H}^{\mathcal{T}}\left(\mathcal{G}\right)\end{array}$$

(2)

where $\alpha$ is a non-root node in the partitioning tree, $\lambda$ is the root of $\mathcal{T}$, ${\alpha }^{-}$ is the parent of $\alpha$, $V_{\alpha }$ is the volume of the vertex subset associated with $\alpha$, and $g_{\alpha }$ is the total weight of edges leaving that subset. A high-entropy graph indicates that Questioner-Solver interactions are broadly distributed across diverse states, suggesting that the system operates in a normal manner. Conversely, a low-entropy graph suggests that interactions are concentrated on a small subset of states, denoting that the system has a tendency to collapse. More details about structural entropy are provided in Appendix F.

System Stability Estimation. Let t denote the training time step, and let $H\left({\mathcal{G}}^{\left(t\right)}\right)$ denote the structural entropy at step t. A naïve strategy is to maximize $H\left({\mathcal{G}}^{\left(t\right)}\right)$, which may over-regularize the system by forcing interactions to remain uniformly dispersed, which can suppress useful learning signal. Instead of pursuing maximum entropy, we anchor the objective to the initial graph structure and allow its entropy to decay gradually throughout training. The initial graph ${\mathcal{G}}^{\left(0\right)}$ is treated as a stable reference state, and its entropy $H\left({\mathcal{G}}^{\left(0\right)}\right)$ defines a training target that decays slowly over time:

$$H_{\mathrm{target}}^{\left(t\right)}=H\left({\mathcal{G}}^{\left(0\right)}\right)-\gamma log(1+t),e_t=H_{\mathrm{target}}^{\left(t\right)}-H\left({\mathcal{G}}^{\left(t\right)}\right),$$

(3)

where $\gamma \ge 0$ controls the rate of entropy decay. The error term $e_t$ measures the deviation between the current structural entropy and the target entropy. A positive $e_t$ indicates that the current interaction topology is over-concentrated, suggesting a stronger need for stability regulation. A direct way to regulate the system is to react to this instantaneous error. However, relying only on $e_t$ may overlook persistent drift. If the graph entropy remains below the target threshold for multiple steps, such one-step proportional feedback cannot fully capture the long-term cumulative instability. Therefore, we introduce a proportional-integral (PI) control mechanism that combines the current entropy error with its accumulated historical values:

$$\begin{array}{c}\hfill {\alpha }_t=\mathrm{clip}\left(K_p{e}_t+K_i{\displaystyle \sum _{s=1}^{t-1}}{e}_s,0,1\right),\end{array}$$

(4)

where $K_p=1$ is the proportional gain for reacting to the current entropy gap, $K_i=0.01$ is the integral gain for accumulating persistent deviations over previous steps, and $\mathrm{clip}(·,0,1)$ constrains the coefficient to $[0,1]$. As a result, ${\alpha }_t$ provides an adaptive scaling factor that reflects the strength of stability regulation required at training step t. Please refer to Appendix G for more details.

3.2.3. Training Objective Function

The role-specific optimization reward $r_Q$ and the estimated stability coefficient ${\alpha }_t$ are incorporated into the standard GRPO [19] function. For each sampled group of size G, the role-specific reward $r_Q^i$ of the i-th generated question is first normalized within the group to obtain the GRPO advantage ${\widehat{A}}_i$, and the estimated stability coefficient ${\alpha }_t$ is then applied to the normalized advantage:

$${\widehat{A}}_i={\displaystyle \frac{r_Q^i-\mathrm{mean}(r_Q^1,\dots ,r_Q^G)}{\mathrm{std}(r_Q^1,\dots ,r_Q^G)+{ϵ}_{\mathrm{norm}}}},{\widehat{A}}_i^{\prime }=(1-{\alpha }_t){\widehat{A}}_i.$$

(5)

The subsequent process follows the standard GRPO. When structural entropy falls below the target threshold, indicating excessive concentration of interactions, ${\alpha }_t$ is increased to suppress the effective advantage magnitude and penalize the model update. Please refer to Appendix G.4 for more details.

3.3. Graph-Stabilized Solver Training

After the Questioner is updated, the Solver is also trained with two objectives. First, the updated Questioner is utilized to generate a candidate pool ${\mathcal{P}}_{\mathrm{candidate}}={\left\{q_i\right\}}_{i=1}^N$ as a training set for the Solver. Following previous work [20], for each candidate question, the current Solver produces m answers, from which a majority-vote pseudo-label ${\tilde{a}}_i$ and a consensus ratio ${\widehat{p}}_i$ are derived as described above. A question is retained only when its consensus ratio lies near the Solver’s uncertainty boundary: $\left|{\widehat{p}}_i-{\displaystyle \frac{1}{2}}\right|\le \delta ,$ where $\delta$ is the tolerance threshold that controls the width of the retained uncertainty region. This filtering step removes questions that are already too easy for the Solver, as well as questions whose pseudo-labels are too unreliable [20,21]. The Solver ${\mathcal{M}}_S$ is then trained on the resulting dataset ${\mathcal{D}}_S=\left\{(q_i,{\tilde{a}}_i)\right\}$ with GRPO. For each retained question $q_i$, the Solver samples R candidate answers ${\left\{a_{i,j}\right\}}_{j=1}^R$ and receives a binary verifiable reward, termed the role-specific optimization reward, based on the pseudo-label obtained through majority voting: $r_S(a_{i,j};q_i)=\mathbf{1}[a_{i,j}={\tilde{a}}_i].$ Meanwhile, the cognitive bipartite graph is updated with newly generated Questioner-Solver interactions, and the stability scaling factor ${\alpha }_t$ is computed as described in Section 3.2.2. The Solver is therefore optimized with the standard pseudo-label-based correctness reward, while its normalized GRPO advantage is further scaled by the system-level stability signal.

3.4. Iterative Co-Evolution Paradigm

S-BGM forms a complete self-evolving loop by alternating between Questioner and Solver optimization. At each iteration, the Solver is kept fixed while the Questioner is optimized with the role-specific reward $r_Q$ to generate questions that are both informative and diverse. Meanwhile, every valid Questioner-Solver interaction is mapped onto a cognitive bipartite graph, whose latest structural entropy is used to estimate the stability coefficient ${\alpha }_t$. This coefficient modulates the normalized GRPO advantage, so the Questioner update follows standard GRPO while being regularized by the system’s current stability signal. After the Questioner update, it produces a new candidate curriculum for Solver training. The Solver generates multiple responses per question; majority voting then yields pseudo-labels and consensus ratios for uncertainty-boundary filtering. The retained questions constitute the Solver training set, and the Solver is subsequently optimized with pseudo-label supervision under the same graph-based stability estimation. See Algorithm A1 for details.

4. Experiments

4.1. Experimental Setup

Datasets. Evaluation is conducted on seven popular mathematical benchmarks and three general reasoning benchmarks. The mathematical benchmarks include GSM8K [22], MATH-500 [17], AMC, Minerva Math [23], OlympiadBench [24], AIME-2024, and AIME-2025, spanning from basic to competition-level mathematics. The general reasoning benchmarks are MMLU-Pro [25], SuperGPQA [26] and BBEH [27]. Pass@1 accuracy with greedy decoding is reported for all benchmarks, with the exception of AMC and AIME, where mean@32 is used following prior work [5].

Baselines. Our method is compared against representative self-evolving methods including Absolute Zero [4], SPICE [9], R-zero [5], R-Few [8], SPIRAL [28]. Details are summarized in Appendix C.

Implementation Details. Experiments are conducted using Qwen3-4B-Base and Qwen3-8B-Base [16] as base models. Self-evolving training is run for 5 iterations using GRPO algorithm [19]. For the cognitive bipartite graph, the number of semantic clusters is $m=100$ and the number of uncertainty intervals is $n=10$. The embedding model is Qwen3-Embedding-0.6B. Unless otherwise specified, the structural entropy is configured with $K=2$. More details and prompts are provided in Appendix B.

4.2. Main Results

Table 1 shows that S-BGM achieves the best aggregate performance across both model scales. The gains are especially pronounced on mathematical reasoning benchmarks, including competition-style tasks such as AMC, Olympiad, and AIME. On general reasoning benchmarks, S-BGM remains competitive, achieving strong results on SuperGPQA and BBEH while maintaining comparable performance on MMLU-Pro. These results suggest that stabilizing the Questioner-Solver interaction benefits self-evolving training. By discouraging the interaction topology from collapsing into narrow question or response patterns, S-BGM helps preserve a more diverse and informative self-generated curriculum. This stability directly enhances mathematical reasoning and demonstrates strong generalization ability. Overall, the improvements in both Math AVG and Overall AVG indicate that graph-based stability modulation provides benefits beyond role-specific optimization.

4.3. Analysis of the Correlations between Model Collapse and Structural Entropy

To examine the relationship between model collapse and structural entropy, we construct a cognitive bipartite graph and compute its structural entropy after each iteration, as shown in Figure 4. The left panel shows that R-Zero (blue curve) degrades after Iteration 3, coinciding with a sharp drop in structural entropy. In contrast, S-BGM achieves a steadily increasing Math AVG and follows a smoother entropy changing trajectory. The middle and right panels further indicate that, from Iteration 0 to 5, the normalized weighted-degree distributions of S-BGM become moderately concentrated from both the Solver and Questioner sides. On the Solver side, although the top-ranked node still becomes higher, it is relatively slight compare to Figure 2, indicating S-BGM effectively suppresses the overconfidence caused by majority voting. On the Questioner side, the distribution over most nodes remains close to the initial one, suggesting that S-BGM helps maintain diverse question-generation states.

4.4. Evolution of Question Difficulty

To examine how question difficulty evolves during S-BGM training, 200 questions are sampled from the Questioner at each iteration to construct five evaluation sets ${\left\{{\mathcal{D}}_{\mathrm{Iter}k}\right\}}_{k=1}^5$, with reference labels provided by GPT-4o. Solvers from different iterations are then evaluated on these cross-iteration datasets. As shown in Table 2, for a fixed Solver, pass rates generally decrease on later-iteration question sets, indicating that the generated problems become progressively harder. For a fixed question set, later Solvers usually achieve higher pass rates, suggesting that the increased difficulty remains learnable rather than noisy. The matched entries stay close to 50%, showing that the Questioner continues to generate questions near the Solver’s uncertainty boundary. Overall, S-BGM produces a progressive and learnable difficulty schedule that supports effective self-evolving training.

4.5. Generalization Analysis of the Performance Gains

A critical question is whether S-BGM can sustain performance gains over multiple self-evolving iterations across different model scales. Figure 5 compares S-BGM, R-Zero, and the base models over five iterations on Qwen3-4B-Base and Qwen3-8B-Base. R-Zero improves in the early stage but later plateaus or collapses, with the degradation being most pronounced on Math AVG. On Qwen3-4B-Base, it peaks around Iteration 3 and drops sharply toward the base-model level by Iteration 5; a similar but milder decline is observed on Qwen3-8B-Base. In contrast, S-BGM maintains a consistently upward trajectory across both model scales on both Math AVG and Overall AVG. These results indicate that structural-entropy-based regularization stabilizes the multi-iteration self-evolving process. While S-BGM and R-Zero perform comparably in the early stage, their gap widens in later iterations, suggesting that S-BGM better mitigates accumulated training instability.

4.6. Ablation Study

To validate the contribution of each component, we conduct ablation studies in Table 3. For role-specific rewards, removing either the uncertainty reward or the repetition penalty hurts performance, indicating that boundary-aware question generation and diversity constraints are both important. Removing uncertainty boundary filtering for Solver training also causes degradation, suggesting that overly easy or overly difficult questions provide less useful training signals. For system-level stabilization, removing the entire structural entropy module or replacing PI control with the instantaneous error $e_t$ underperforms the full method. This confirms that global topology regulation and accumulated entropy deviations help produce a more reliable stability signal.

4.7. Hyperparameter Analysis

Figure 6 reports the sensitivity of S-BGM to the structural entropy dimension K, sliding window size L, number of semantic clusters m, and number of uncertainty intervals n. The best performance is achieved with $K=2$, $L=1024$, and $(m,n)=(100,10)$, suggesting that moderate structural resolution is important for stable entropy estimation. For K, one-dimensional entropy may be insufficient to capture the coupling structure, while deeper partitions can introduce estimation noise. For L, a small window is sensitive to short-term fluctuations, whereas a large window retains outdated interactions and reduces responsiveness to recent changes. For $(m,n)$, too few states merge distinct interaction patterns, while too many states sparsify $\mathbf{B}$ and make entropy estimation less reliable.

5. Conclusions

In this work, we propose S-BGM to study model collapse in self-evolving LLMs from a system-stability perspective. S-BGM models the coupled Questioner–Solver interaction as a Cognitive Bipartite Graph and uses structural entropy to measure whether the interaction topology collapses into a narrow set of states. By incorporating this graph-level stability signal into training, S-BGM combines role-specific capability improvement with system-level stabilization. Extensive experiments show that S-BGM improves aggregate performance and stabilizes multi-iteration self-evolution.

Appendix B.1. Training Hyperparameters

We detail the hyperparameters introduced in the training procedure and their roles.

• $L=1024$: The length of the sliding window used to maintained the cognitive bipartite graph.
• $\sigma =0.01$: Bandwidth of the Gaussian uncertainty reward $r_{\mathrm{unc}}$ (see Section 3.2). It controls the decay intensity of the reward when the empirical consensus ratio ${\widehat{p}}_i$ deviates from the target uncertainty boundary of $0.5$.
• $\tau =0.6$: Cosine similarity threshold for the repetition penalty $r_{\mathrm{rep}}$ (see Eq. (1)). When the maximum similarity $s_i$ between a newly generated question and the historical question pool ${\mathcal{P}}_{\mathrm{emb}}$ exceeds this threshold, a penalty term $s_i-\tau$ is incurred to suppress the generation of redundant questions.
• $R=10$: Number of samples used for the Solver’s majority voting. For each generated question, the Solver produces R candidate answers to compute the pseudo-label ${\tilde{a}}_i$ and the consensus ratio ${\widehat{p}}_i$, balancing pseudo-label reliability with computational sampling overhead.
• $M=2000$: Capacity of the historical question embedding pool ${\mathcal{P}}_{\mathrm{emb}}$. This pool is maintained using a sliding-window mechanism to calculate $r_{\mathrm{rep}}$, ensuring that the questions generated by the Questioner maintain continuous semantic diversity during the self-evolution process.
• $N=8000$: Total number of candidate questions generated by the updated Questioner in each iteration. These questions form the candidate pool ${\mathcal{P}}_{\mathrm{candidate}}$, which is subsequently filtered via consistency checks to construct the final training set ${\mathcal{D}}_S$ for the Solver.
• $\gamma =1\times {10}^{-4}$: Decay rate of the target structural entropy trajectory $H_{\mathrm{target}}^{\left(t\right)}$ (see Eq. (3)). It governs the allowable reduction in structural entropy over training steps, reflecting a tolerance for the gradual concentration of interaction patterns as the system moves from broad exploration to focused learning.
• $K_p=1$: Proportional gain of the PI controller for structural entropy regulation (see Eq. (4)). it provides an immediate corrective response to the current entropy deviation $e_t$ by dynamically adjusting the stability coefficient ${\alpha }_t$ to rectify trends drifting away from the target trajectory.
• $K_i=0.01$: Integral gain of the PI controller for structural entropy regulation (see Eq. (4)). It eliminates steady-state errors by accumulating historical entropy deviations $\sum e_s$, allowing the system to capture and correct long-term systemic collapse risks that may be invisible in a single time step.
• $\delta =0.25$: Filtering bandwidth for the Solver’s curriculum learning. Only questions satisfying $|{\widehat{p}}_i-0.5|\le \delta$ are retained in the training set ${\mathcal{D}}_S$, aiming to exclude tasks that are either too simple for the Solver or possess highly unreliable pseudo-labels.

Appendix B.2. Questioner Training

The Questioner is optimized with GRPO using the following hyperparameters:

• Global Batch Size: 128
• Learning Rate: $1\times {10}^{-6}$
• Weight Decay: $1\times {10}^{-2}$
• KL Penalty Coefficient (${\lambda }_{KL}$): $1\times {10}^{-2}$
• Max Steps: 5
• Number of Rollouts: 4
• Rollout Temperature: 1.0
• Rollout Top-p: 0.99

Appendix B.3. Solver Training

The Solver is optimized with GRPO using the following hyperparameters:

• Global Batch Size: 128
• Learning Rate: $1\times {10}^{-6}$
• Weight Decay: $1\times {10}^{-2}$
• KL Penalty Coefficient (${\lambda }_{KL}$): $1\times {10}^{-2}$
• Max Steps: 15
• Number of Rollouts: 5
• Rollout Temperature: 1.0
• Rollout Top-p: 0.99

Appendix B.4. Prompt Templates

This section presents all prompts used in S-BGM training and analysis. All prompt remain identical to R-Zero [5].

Appendix D.1. Analysis of More Self-Evolving Iterations

To further examine the long-horizon behavior of S-BGM, we extend the self-evolving process to more iterations on Qwen3-4B-Base, as shown in the left and middle panels of Figure A1. The results show that S-BGM continues to improve substantially in the early iterations and consistently outperforms the base model. After around five iterations, the performance curve becomes relatively stable, with only marginal additional gains. This suggests that S-BGM can prevent the rapid performance collapse observed in conventional self-evolving methods, but the benefit of further iterations gradually saturates. This result is reasonable because reinforcement learning mainly serves as a mechanism to elicit and refine the latent capabilities of the base model, rather than indefinitely expanding its intrinsic capacity [36,37,38]. In the early stages, self-evolving RL can effectively activate underutilized reasoning behaviors through increasingly informative training signals. However, once most learnable signals from the self-generated curriculum have been absorbed, further iterations provide diminishing returns because the achievable performance is increasingly constrained by the inherent limitations of the base model. Therefore, the plateau after several iterations should not be interpreted as model collapse, but rather as the natural saturation of RL-based capability elicitation under self-generated supervision.

Figure A1. Additional analysis of extended self-evolving iterations and PI-control hyperparameters.

Figure A1. Additional analysis of extended self-evolving iterations and PI-control hyperparameters.

Appendix D.2. Hyperparameter Analysis of PI Control

The right panel of Figure A1 analyzes the sensitivity of S-BGM to the PI-control parameters $K_p$ and $K_i$. Overall, the best performance is obtained around $K_p=1.0$, while both smaller and larger values lead to lower performance. When $K_p$ is too small, the controller responds weakly to entropy deviations, making stability regulation insufficient. When $K_p$ is too large, the controller may overreact to short-term entropy fluctuations and suppress useful updates. The results are relatively stable across different values of $K_i$, with $K_i=0.01$ achieving the best or near-best performance in most settings. This indicates that a moderate integral term is helpful for capturing persistent entropy deviations without introducing excessive accumulated correction. Together, these results support the effectiveness of the PI-control design and show that S-BGM performs best when the stability signal reacts sufficiently to topology drift while avoiding overly aggressive update suppression.

Appendix G.1. Basic Principle of Proportional-Integral Control

Proportional-integral (PI) control is a classical feedback mechanism designed to drive a controlled variable $y\left(t\right)$ toward a prescribed target $y_{\mathrm{tar}}$ [42,43]. At time step t, the tracking error is defined as

$$\begin{array}{c}\hfill e_t=y_{\mathrm{tar}}-y\left(t\right).\end{array}$$

(A18)

The PI control law specifies the control input as a weighted combination of the instantaneous error and the accumulated historical error:

$$\begin{array}{c}\hfill u_t=K_p{e}_t+K_i{\displaystyle \sum _{s=1}^{t-1}}{e}_s,\end{array}$$

(A19)

where $K_p>0$ is the proportional gain and $K_i\ge 0$ is the integral gain.

The two components play complementary roles. The proportional term $K_p{e}_t$ provides an immediate response to the current deviation, so larger errors induce stronger control. However, proportional control alone can leave a non-zero steady-state error under persistent disturbances. The integral term $K_i{\sum }_s{e}_s$ accumulates historical deviations; even when each instantaneous error is small, a persistent bias increases the integral response over time until the steady-state deviation is reduced. Combining the two terms therefore provides both responsiveness and improved long-horizon tracking accuracy.

Appendix G.2. Control Loop in S-BGM

In S-BGM, the structural entropy $H\left({\mathcal{G}}^{\left(t\right)}\right)$ of the Cognitive Bipartite Graph is used as the system-level stability signal. The target trajectory $H_{\mathrm{target}}^{\left(t\right)}$ defines the expected entropy level at training step t:

$$\begin{array}{c}\hfill H_{\mathrm{target}}^{\left(t\right)}=H\left({\mathcal{G}}^{\left(0\right)}\right)-\gamma log(1+t),\end{array}$$

(A20)

where $\gamma \ge 0$ controls the rate of entropy decay. The adaptive scaling factor ${\alpha }_t$ represents the strength of stability regulation. The error signal is

$$\begin{array}{c}\hfill e_t=H_{\mathrm{target}}^{\left(t\right)}-H\left({\mathcal{G}}^{\left(t\right)}\right),\end{array}$$

(A21)

where a positive $e_t$ indicates that the current interaction topology is more concentrated than the target trajectory. The PI controller computes

$$\begin{array}{c}\hfill {\alpha }_t=\mathrm{clip}\left(K_p{e}_t+K_i{\displaystyle \sum _{s=1}^{t-1}}{e}_s,0,1\right).\end{array}$$

(A22)

The resulting stability coefficient scales the normalized GRPO advantage by

$$\begin{array}{c}\hfill {\widehat{A}}_i^{\prime }=(1-{\alpha }_t){\widehat{A}}_i,\end{array}$$

(A23)

and thereby adjusts the effective update magnitude in GRPO. The intended stabilizing effect is

$$\begin{array}{c}\hfill {\alpha }_t\uparrow \Rightarrow \parallel \Delta \theta \parallel \downarrow \Rightarrow \Delta {\pi }_{\theta }\downarrow \Rightarrow \Delta \mathbf{B}\downarrow \Rightarrow \Delta H\left(\mathcal{G}\right)\downarrow .\end{array}$$

(A24)

We next justify this mechanism and clarify its scope. The scaling is not intended to directly maximize structural entropy or force a uniform interaction topology. It acts as a stability regulation mechanism: when the graph indicates excessive concentration, it suppresses large policy updates, while the role-specific rewards and subsequent sampling continue to determine which outputs are preferred.

Appendix G.3. Theoretical Analysis

Lemma A4 

(Lipschitz continuity of structural entropy under smoothed edge distributions). Let $\mathbf{B},{\mathbf{B}}^{\prime }\in {\mathbb{R}}_{\ge 0}^{m\times n}$ be two bi-adjacency matrices maintained by a sliding window of size L. Since some edges may have zero weight, we define a smoothed normalized edge distribution with a fixed $\zeta >0$:

$$\begin{array}{c}\hfill {\tilde{p}}_{ij}={\displaystyle \frac{B_{ij}+\zeta }{{\sum }_{a,b}(B_{ab}+\zeta )}},{\tilde{p}}_{ij}^{\prime }={\displaystyle \frac{B_{ij}^{\prime }+\zeta }{{\sum }_{a,b}(B_{ab}^{\prime }+\zeta )}}.\end{array}$$

(A25)

Because the window contains L interactions, each entry is uniformly lower bounded by

$$\begin{array}{c}\hfill {\tilde{p}}_{ij},{\tilde{p}}_{ij}^{\prime }\ge p_{min}:={\displaystyle \frac{\zeta }{L+mn\zeta }}.\end{array}$$

(A26)

Then the structural entropy computed from the smoothed graph distributions satisfies

$$\begin{array}{c}\hfill \left|H\left(\mathcal{G}\right)-H\left({\mathcal{G}}^{\prime }\right)\right|\le C_H{\parallel \tilde{\mathbf{p}}-{\tilde{\mathbf{p}}}^{\prime }\parallel }_1,\end{array}$$

(A27)

where $C_H>0$ is a constant depending only on $m,n,L$, and ζ.

Proof 

(Proof sketch). For a fixed partitioning tree $\mathcal{T}$, the structural entropy $H^{\mathcal{T}}\left(\mathcal{G}\right)$ is a finite sum of terms determined by smoothed normalized edge weights and induced subset volumes. The smoothing removes the logarithmic singularity at zero: all probabilities lie in $[p_{min},1]$, so each logarithmic component is Lipschitz because $|plogp-p^{\prime }log{p}^{\prime }|$ is bounded by a constant multiple of $|p-p^{\prime }|$ on this interval. Summing over at most $mn$ edge states yields a finite Lipschitz constant depending on $|log{p}_{min}|$. Since $H^K\left(\mathcal{G}\right)$ is the minimum over a finite collection of partitioning trees of height K, and the pointwise minimum of functions sharing a common Lipschitz constant is also Lipschitz, the stated bound follows.    □

Effect of stability scaling on the GRPO update. Let $g_{\mathrm{pg}}$ denote the gradient contribution of the GRPO policy-improvement surrogate before applying the KL penalty. Since the advantage enters this surrogate linearly, replacing ${\widehat{A}}_i$ by ${\widehat{A}}_i^{\prime }=(1-{\alpha }_t){\widehat{A}}_i$ gives

$$\begin{array}{c}\hfill g_{\mathrm{pg}}^{\prime }=(1-{\alpha }_t)g_{\mathrm{pg}}.\end{array}$$

(A28)

Thus the stability coefficient directly scales the update component that moves the policy toward higher-reward rollouts, while leaving the within-group ranking induced by the role-specific reward unchanged. The KL term continues to anchor the policy to ${\pi }_{{\theta }_{\mathrm{old}}}$, so increasing ${\alpha }_t$ reduces the magnitude of destabilizing policy movement rather than changing the reward preference itself.

Proposition 1: Suppression of structural drift. Assume that, within a local training region, the normalized interaction distribution has bounded sensitivity to the policy-improvement update:

$$\begin{array}{c}\hfill \mathbb{E}\left[\parallel \Delta {\mathbf{p}}^{\left(t\right)}{\parallel }_1\right]\le L_{\pi }\eta \parallel g_{\mathrm{pg}}^{\prime }\parallel ,\end{array}$$

(A29)

where $L_{\pi }>0$ summarizes the local sensitivity of generation, semantic assignment, uncertainty binning, smoothing, and sliding-window aggregation. Combining this sensitivity bound with Lemma A4 yields

$$\begin{array}{c}\hfill \mathbb{E}\left[\left|H\left({\mathcal{G}}^{(t+1)}\right)-H\left({\mathcal{G}}^{\left(t\right)}\right)\right|\right]\le C_H{L}_{\pi }\eta (1-{\alpha }_t)\parallel g_{\mathrm{pg}}\parallel .\end{array}$$

(A30)

Equivalently, for $\rho :=C_H{L}_{\pi }\eta \parallel g_{\mathrm{pg}}\parallel$,

$$\begin{array}{c}\hfill \mathbb{E}\left[|\Delta H^{\left(t\right)}|\right]\le \rho (1-{\alpha }_t).\end{array}$$

(A31)

This result should be viewed as a drift-control statement rather than a convergence theorem: larger ${\alpha }_t$ lowers an upper bound on the amount by which a single update can perturb the graph structure. It does not by itself prove that entropy will increase. Instead, it prevents large updates from further amplifying collapse once the system enters an unhealthy regime, preserving a more stable condition under which role-specific rewards, curriculum filtering, and subsequent sampling can recover interaction diversity.

Proposition 2: Response guarantee under persistent collapse. Suppose that, over an interval $[t_0,t_0+T]$, the system persistently stays below the target trajectory so that $e_t\ge \delta >0$ at every step. Then the integral contribution satisfies

$$\begin{array}{c}\hfill K_i{\displaystyle \sum _{s=t_0}^{t_0+T}}{e}_s\ge K_i\delta T.\end{array}$$

(A32)

Thus, before clipping, the stability scaling strength grows at least linearly with the duration of the collapse signal.

Proof. 

Since $e_s\ge \delta >0$ for every $s\in [t_0,t_0+T]$, monotonicity of summation gives ${\sum }_{s=t_0}^{t_0+T}{e}_s\ge \delta T$. Multiplying by $K_i\ge 0$ proves the claim.    □

Proposition 2 explains the role of the integral term in our setting. For slow collapse across multiple self-evolution iterations, each instantaneous deviation may be small, so pure proportional feedback would only impose limited suppression. The integral term accumulates these persistent deviations and progressively increases the stability scaling strength, slowing down further structural drift and giving the role-specific objectives and future sampling rounds an opportunity to restore diversity. This property is aligned with the long-horizon iterative training regime of S-BGM.

Appendix G.4. The Final Loss Function of GRPO

The design of ${\widehat{A}}_i^{\prime }=(1-{\alpha }_t){\widehat{A}}_i$ is important because applying a group-level coefficient directly to rewards would be canceled by GRPO’s within-group normalization. By scaling the normalized advantage instead, the stability signal remains active in the gradient. Since $(1-{\alpha }_t)$ is non-negative, this scaling preserves the relative ranking and signs of samples within each group. Its role is therefore not to change which outputs are preferred by the role-specific reward, but to adjust the update magnitude according to the global system state. When the interaction topology is stable, ${\alpha }_t$ remains small and the update is close to standard GRPO. When the graph entropy indicates excessive concentration, ${\alpha }_t$ increases and suppresses the update, preventing the system from further amplifying unstable interaction patterns. The final loss function is consistent with the standard GRPO:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{GRPO}}\left(\theta \right)=-{\displaystyle \frac{1}{G}}{\displaystyle \sum _{i=1}^G}min\left({\displaystyle \frac{{\pi }_{\theta }\left(x_i\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(x_i\right)}}{\widehat{A}}_i^{\prime },\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(x_i\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(x_i\right)}},1-ϵ,1+ϵ\right){\widehat{A}}_i^{\prime }\right)+\beta \mathrm{KL}({\pi }_{\theta }\parallel {\pi }_{{\theta }_{\mathrm{old}}}).\end{array}$$

(A33)

where $\theta$ denotes the trainable parameters of the current policy model, ${\pi }_{\theta }$ is the current policy, and ${\pi }_{{\theta }_{\mathrm{old}}}$ is the old policy used for importance-ratio estimation. $x_i$ denotes the i-th sampled output in a group, which corresponds to a generated question in Questioner training or a generated response in Solver training. G is the group size, $ϵ$ is the clipping threshold in GRPO, and $\beta$ controls the strength of the KL regularization term. The term $\mathrm{KL}({\pi }_{\theta }\parallel {\pi }_{{\theta }_{\mathrm{old}}})$ penalizes excessive deviation from the old policy, while ${\widehat{A}}_i^{\prime }$ is the stability-scaled advantage defined above.