SCOPE-RL: Reinforcement Learning with Executable Step-Level Verification for Mathematical Reasoning

1. Introduction

Large language models (LLMs) have recently demonstrated remarkable reasoning abilities across a wide range of complex tasks [1,2,3,4,5,6,7] and programming [6,8,9,10,11,12]. In particular, mathematical reasoning has emerged as a central testbed for evaluating and improving the multi-step reasoning capabilities of LLMs, where Chain-of-Thought (CoT) reasoning [5] enables models to explicitly decompose a problem into a sequence of intermediate derivations. Recent reasoning-oriented models, including Qwen-3 [13], DeepSeek-R1 [14], Seed-1.5 Thinking [15], and GPT-5 Thinking [7], further demonstrate the effectiveness of extended reasoning and reinforcement learning for solving challenging mathematical problems. Despite these advances, mathematical CoT generation remains fundamentally autoregressive and can be viewed as a token-level Markov Decision Process (MDP) [16,17,18,19], where each reasoning decision is conditioned on the previously generated trajectory. This autoregressive nature is particularly problematic in mathematical reasoning: once an incorrect algebraic manipulation, arithmetic calculation, equation transformation, or logical deduction is introduced into the derivation, subsequent reasoning steps are conditioned on the erroneous intermediate state, causing the error to propagate and potentially amplify throughout the remainder of the solution [20,21]. Consequently, even when the overall reasoning trajectory appears coherent, a locally invalid mathematical step can render the final solution incorrect. Existing CoT reasoning therefore lacks an explicit mechanism for verifying the validity of intermediate mathematical transformations before they are incorporated into subsequent derivations.

A number of approaches have been proposed to mitigate this limitation. One line of work introduces external search mechanisms that explore multiple candidate reasoning trajectories [22,23,24]. Such methods broaden exploration and allow promising continuations to be selected post hoc, but the improvement is largely supplied by an external controller rather than being internalized by the policy. Moreover, repeated model queries introduce substantial inference overhead [25], while the reliability of the search remains dependent on the quality of the underlying scoring mechanism [26]. Another line of work encourages reflection, backtracking, or self-correction through reinforcement learning [16,17,27,28]. Although these methods allow the model to reconsider previous decisions, the resulting reflection is typically expressed in free-form natural language and is therefore difficult to verify objectively. Similarly, explicit planning methods [8,17] provide useful structural guidance, but a plan primarily specifies how to proceed rather than whether an individual mathematical transformation is correct.

Reinforcement learning has also become a central paradigm for improving LLM reasoning, evolving from reinforcement learning from human feedback (RLHF) [19] toward reinforcement learning with verifiable rewards (RLVR), where task correctness itself can provide an objective learning signal. GRPO [29] and its recent variants [16,17,18,28,30,31,32,33] have shown strong performance by optimizing groups of sampled responses without requiring a separately learned value model. However, these objectives are primarily outcome-based: the reward is determined by the final answer, leaving the internal reasoning trajectory largely unsupervised [33]. Process reward models (PRMs) provide finer-grained supervision by assigning credit to intermediate steps [34,35], but typically rely on step-level annotations or separately trained evaluators. This leaves open how to obtain an intermediate supervision signal that is both directly verifiable and inexpensive to incorporate into RL training.

Mathematical problem solving provides a natural opportunity for such verification. Human solvers rarely trust a derivation solely because it produces a plausible final answer; they routinely check intermediate results through algebraic equivalence, substitution, boundary behavior, admissible ranges, special cases, and back-substitution [36,37]. Many of these operations have explicit mathematical semantics and can be evaluated deterministically. More importantly, they can be applied directly to the reasoning step where a transformation occurs. For example, two algebraic expressions can be tested for symbolic equivalence, a proposed solution can be substituted back into an equation, and an approximation can be evaluated at a boundary point before it is propagated through the remainder of the solution. However, merely introducing additional mathematical conditions is insufficient. A condition may be mathematically correct while providing little information about the validity of the particular reasoning transformation being examined. For instance, a property of the original problem may remain true regardless of whether the model performs an incorrect approximation. Such a condition cannot serve as an effective detector of reasoning errors. A useful intermediate check should therefore satisfy two properties: it should be executable, so that its outcome can be determined through symbolic or numerical operations, and it should be discriminative, so that it accepts a valid transformation while rejecting plausible local errors.

Motivated by this observation, we introduce Step-wise Check Optimization for Process-Enhanced Reinforcement Learning (SCOPE-RL), a reinforcement learning framework that equips intermediate mathematical reasoning with executable, step-level verification as shown in Figure 1b. Instead of representing verification conditions as unrestricted natural-language statements, SCOPE-RL associates intermediate reasoning steps with typed mathematical checks, including symbolic equivalence, substitution consistency, equation residuals, boundary evaluation, and sign or range verification. Each candidate check is first evaluated on a correct reference derivation and then tested against controlled local counterexamples. Only checks that accept the reference transformation while rejecting a sufficient proportion of corrupted alternatives are retained, providing an explicit mechanism for filtering mathematically valid but nondiscriminative conditions.

During reinforcement learning, SCOPE-RL executes these checks directly on intermediate reasoning states. Verified transformations receive positive process credit, detected mathematical inconsistencies receive negative credit, and unsupported or inapplicable checks provide no process signal. This step-level reward is combined with the standard final-answer correctness reward under GRPO [29]. Importantly, process verification and outcome correctness are optimized separately: a correct answer retains its correctness reward even when an intermediate check fails, while an incorrect answer receives no additional outcome credit merely for identifying its own inconsistency. Consequently, SCOPE-RL introduces explicit intermediate supervision without requiring additional LLM rollout sampling or altering the underlying group-relative policy optimization procedure.

Our main contributions are summarized as follows:

• Executable step-level verification. We propose SCOPE-RL, a framework that directly evaluates intermediate mathematical transformations using typed symbolic and numerical checks, providing a programmatically verifiable process signal beyond final-answer correctness.
• Counterexample-based check certification. We introduce a certification procedure that requires each mathematical check to accept a valid reference transformation while rejecting controlled local perturbations, filtering out conditions that are correct but ineffective for detecting reasoning errors.
• Verification-guided reinforcement learning. We integrate step-level programmatic verification into GRPO through an additional process reward while preserving final-answer correctness as an independent outcome objective, providing dense intermediate supervision without additional model rollout sampling.

2. Preliminaries and Related Work

2.1. Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) is a widely used reinforcement learning algorithm for training large language models with verifiable rewards. Building upon Proximal Policy Optimization (PPO) [38], GRPO removes the need for a separately learned value function and instead estimates relative advantages from multiple responses sampled for the same query. This design substantially simplifies reinforcement learning for reasoning models and has been adopted in a number of recent RL-based reasoning systems.

Formally, let Q denote the set of training questions, ${\pi }_{{\theta }_{\mathrm{old}}}$ the policy used for rollout generation, and ${\left\{{\mathfrak{o}}_i\right\}}_{i=1}^N$ a group of N candidate responses sampled for a question $q\in Q$. Let ${\pi }_{{\theta }_{\mathrm{ref}}}$ denote a fixed reference policy. The GRPO objective can be written as

$$\begin{array}{cc}\hfill & J_{\mathrm{GRPO}}\left(\theta \right)={\mathbb{E}}_{q\sim Q,{\left\{{\mathfrak{o}}_i\right\}}_{i=1}^N\sim {\pi }_{{\theta }_{\mathrm{old}}}}\hfill \\ \hfill & [{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{t=1}^{|{\mathfrak{o}}_i|}}\min \left({\displaystyle \frac{{\pi }_{\theta }({\mathfrak{o}}_i^t\mid q)}{{\pi }_{{\theta }_{\mathrm{old}}}({\mathfrak{o}}_i^t\mid q)}}{A}_i,clip\left({\displaystyle \frac{{\pi }_{\theta }({\mathfrak{o}}_i^t\mid q)}{{\pi }_{{\theta }_{\mathrm{old}}}({\mathfrak{o}}_i^t\mid q)}},1-ϵ,1+ϵ\right)A_i\right)\hfill \\ \hfill & -\beta D_{\mathrm{KL}}({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}})],\hfill \end{array}$$

(1)

where $ϵ$ controls the clipping range of the importance ratio and $\beta$ determines the strength of the KL regularization toward the reference policy.

Instead of estimating advantages with a critic, GRPO standardizes rewards within each response group:

$$A_i={\displaystyle \frac{r_i-\mu }{\sigma }},\mu ={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{r}_j,\sigma =\sqrt{{\displaystyle \frac{1}{N}}\sum _{j=1}^N{(r_j-\mu )}^2},$$

(2)

where $r_i$ is the scalar reward assigned to response ${\mathfrak{o}}_i$.

In mathematical reasoning tasks, a response typically consists of a chain-of-thought trajectory followed by a final answer, while the verifiable reward is determined primarily by whether the final answer is correct. Consequently, all tokens in a trajectory are usually optimized using an advantage derived from the same terminal outcome. This provides an effective and scalable training signal, but it does not distinguish among intermediate reasoning steps that contribute differently to the final prediction. In particular, trajectories with substantially different reasoning quality may receive identical rewards as long as they arrive at the same answer. This motivates methods that introduce more informative supervision into the reasoning trajectory while retaining the scalability of outcome-based RL.

2.2. Reinforcement Learning for Mathematical Reasoning

Reinforcement learning has emerged as an important paradigm for improving the mathematical reasoning capabilities of large language models. Early approaches largely relied on reinforcement learning from human feedback (RLHF) [19], whereas recent reasoning models increasingly adopt reinforcement learning with verifiable rewards (RLVR), where mathematical correctness can be evaluated automatically using reference answers or rule-based verification. GRPO [29] and subsequent variants [16,17,18,28,30,31,32,33] have demonstrated that such outcome-based signals can substantially improve mathematical problem-solving ability without requiring a learned value model.

A fundamental limitation of standard RLVR is that supervision is typically available only at the end of the generated solution. The final reward is propagated across the entire response, despite the fact that different portions of a reasoning trajectory may vary considerably in correctness, informativeness, and contribution to the final answer. As a result, outcome-based optimization provides limited information about which intermediate reasoning decisions should be reinforced or corrected.

A complementary direction therefore focuses on introducing process-level supervision. Process reward models (PRMs) estimate the quality of individual reasoning steps and provide denser credit signals during training or inference [34,35]. Although effective, PRMs generally require step-level supervision, synthetic labeling pipelines, or separately trained reward models, which introduces additional annotation and modeling cost. Other approaches encourage more structured reasoning through intermediate plans, reflection, decomposition, or explicit reasoning stages [8,16,27]. These mechanisms can improve the organization of the generated trajectory, but the intermediate structures are often used primarily as reasoning scaffolds rather than as independently verifiable optimization signals.

More broadly, these studies highlight a central challenge in reinforcement learning for mathematical reasoning: how to obtain dense and reliable supervision over intermediate reasoning without introducing expensive human annotation or an additional learned verifier. An appealing solution is to derive such signals directly from information already available during training and to use them to differentiate the learning signal across different parts of the reasoning trajectory. This is the setting considered by SCOPE-RL.

3. Approach of SCOPE-RL

We introduce Step-wise Check Optimization for Process-Enhanced Reinforcement Learning (SCOPE-RL), a two-stage reinforcement learning framework that improves mathematical reasoning by explicitly verifying intermediate reasoning steps. The central idea is to replace free-form reasoning constraints with executable mathematical checks whose correctness can be assessed through symbolic and numerical operations shown in Figure 2. Unlike approaches that evaluate whether generated constraints are consistent with the final answer, SCOPE-RL directly verifies the mathematical transformations performed during reasoning, providing explicit supervision over intermediate steps.

Given a mathematical problem q, the policy generates a structured reasoning trajectory $o=\left({\left\{(s_t,c_t)\right\}}_{t=1}^T,\widehat{y}\right),$ where $s_t$ denotes the t-th reasoning step, $c_t$ represents an executable mathematical check associated with that step, and $\widehat{y}$ is the final answer. Each check belongs to a predefined family of mathematical verification operators, including symbolic equivalence, substitution, equation residual, boundary evaluation, and range consistency. This structured representation enables intermediate reasoning to be evaluated through deterministic mathematical operations rather than subjective language-model judgments.

The framework consists of two consecutive stages. First, we construct a certified step-check corpus and perform supervised fine-tuning to initialize the desired reasoning structure. Second, we optimize the resulting policy through reinforcement learning, using programmatically verified intermediate steps to construct an additional process reward. Importantly, process verification is decoupled from final-answer correctness, allowing the model to learn mathematically consistent reasoning without introducing incentives for unnecessary self-rejection or penalizing correct answers solely because of their self-verdicts.

3.1. Certified Step-Check Warm-Up

A pretrained language model does not naturally express its reasoning in a form that can be directly evaluated by symbolic or numerical verifiers. We therefore first establish the desired stepwise verification behavior through supervised fine-tuning.

We construct a warm-up dataset from mathematical problems paired with reference solutions. For each problem $q_i$, let $\{s_{i,1}^{*},s_{i,2}^{*},\dots ,s_{i,T_i}^{*}\}$ denote the sequence of intermediate reasoning steps in its reference solution, and let $y_i$ denote the corresponding ground-truth answer. For each intermediate step, we generate candidate mathematical checks and convert them into a restricted executable representation. Specifically, we define a verifier library $\mathcal{V}=\left\{V_{\mathrm{eqv}},V_{\mathrm{sub}},V_{\mathrm{res}},V_{\mathrm{bnd}},V_{\mathrm{rng}}\right\},$ where the operators respectively evaluate symbolic equivalence, substitution correctness, equation residuals, boundary conditions, and sign or range consistency.

Each candidate check is represented as a typed mathematical specification containing an operator, the relevant mathematical expressions, variable assignments or domains, and an optional numerical tolerance. For example, the transformation

$${\displaystyle \frac{x^2}{x^2+a^2}}=1-{\displaystyle \frac{a^2}{x^2+a^2}}$$

(3)

can be associated with a symbolic-equivalence check. Similarly, a boundary condition can be represented as

$$V_{\mathrm{bnd}}\left({\displaystyle \frac{x^2}{x^2+a^2}},x=a,{\displaystyle \frac{1}{2}}\right).$$

(4)

To ensure consistent verification semantics, each operator returns one of three possible states:

$$V(c,s)\in \{1,0,\perp \},$$

(5)

where 1 indicates that the mathematical check is satisfied, 0 indicates a detected violation, and ⊥ denotes an unexecutable or inapplicable check. For transition-based checks, the verifier additionally takes the preceding reasoning state as input. The distinction between verification failure and non-verifiability prevents unsupported mathematical expressions from being incorrectly classified as reasoning errors.

Constraint Certification.  Although a mathematical check may be syntactically executable, it does not necessarily provide useful information about the correctness of a reasoning step. For instance, a condition may hold for both valid and invalid derivations, making it ineffective for error detection. We therefore introduce a certification procedure that evaluates both the validity and discriminative capability of candidate checks.

For each reference step $s_{i,t}^{*}$, a candidate check $c_{i,t}$ must first satisfy $V(c_{i,t},s_{i,t}^{*})=1.$ For transition-based checks, this condition is evaluated using the corresponding reference transition. We then construct a set of M local counterexamples ${\tilde{\mathcal{S}}}_{i,t}={\left\{{\tilde{s}}_{i,t}^{\left(m\right)}\right\}}_{m=1}^M,$ by applying controlled mathematical perturbations to the reference step. Depending on the structure of the reasoning operation, perturbations include coefficient modifications, sign flips, arithmetic-operator replacement, incorrect substitutions, denominator removal, and inequality-direction changes. Only perturbations confirmed to introduce an invalid mathematical transformation are retained as counterexamples.

We define the counterexample rejection score as

$$d_{i,t}={\displaystyle \frac{1}{M}}{\displaystyle \sum _{m=1}^M}\mathbb{I}\left[V\left(c_{i,t},{\tilde{s}}_{i,t}^{\left(m\right)}\right)=0\right],$$

(6)

where $\mathbb{I}[·]$ denotes the indicator function. A candidate check is retained only when

$$V(c_{i,t},s_{i,t}^{*})=1\mathrm{and}{d}_{i,t}\ge {\tau }_d,$$

(7)

where ${\tau }_d$ is a predefined discrimination threshold. The first requirement ensures consistency with a valid reference derivation, while the second requires the check to reject a sufficient proportion of local counterexamples. Consequently, mathematically valid but nondiscriminative conditions are excluded from the training corpus. This certification establishes validity on the reference derivation and empirical discrimination against the specified perturbation distribution, without assuming that every possible reasoning error can be detected. After certification, the warm-up corpus is constructed as

$${\mathcal{D}}_{\mathrm{SFT}}={\left\{\left(q_i,{\left\{(s_{i,t}^{*},c_{i,t}^{*})\right\}}_{t=1}^{T_i},y_i\right)\right\}}_{i=1}^n.$$

(8)

Each solution is serialized into a structured sequence using explicit <step>, <check>, and <answer> tags, allowing reasoning steps and their associated verification conditions to be extracted independently. We then minimize the standard sequence negative log-likelihood

$${\mathcal{L}}_{\mathrm{SFT}}\left(\theta \right)=-{\mathbb{E}}_{(q,o)\sim {\mathcal{D}}_{\mathrm{SFT}}}\left[\log {\pi }_{\theta }(o\mid q)\right].$$

(9)

This produces an initial policy ${\pi }_{{\theta }_0}$ that learns to generate structured mathematical reasoning together with executable checks. While supervised fine-tuning establishes the required representation, the subsequent reinforcement learning stage further encourages generated checks to remain executable, discriminative, and consistent with intermediate reasoning.

3.2. Reinforcement Learning with Stepwise Verification

The warm-up stage initializes the policy to produce structured reasoning and associated mathematical checks. However, supervised learning alone does not ensure that checks generated for unseen problems remain mathematically informative. We therefore introduce a verification-based reinforcement learning objective that explicitly rewards valid intermediate reasoning. Following the standard GRPO setting, for each question q, we sample a group of N responses from the old policy:

$${\left\{o_i\right\}}_{i=1}^N\sim {\pi }_{{\theta }_{\mathrm{old}}}(·\mid q),$$

(10)

where each response is represented as

$$o_i=\left({\left\{(s_{i,t},c_{i,t})\right\}}_{t=1}^{T_i},{\widehat{y}}_i\right).$$

(11)

All verification signals are computed from the generated trajectories and deterministic mathematical operations, without introducing additional language-model sampling beyond the standard GRPO group.

Programmatic Step Verification.  For each generated reasoning step $s_{i,t}$, we extract its associated mathematical check $c_{i,t}$ and evaluate the corresponding mathematical condition using the programmatic verifier:

$$g_{i,t}=V(c_{i,t},s_{i,t})\in \{0,1,\perp \}.$$

(12)

For checks involving transformations between consecutive reasoning states, such as symbolic equivalence or equation preservation, the verification operation is defined as

$$g_{i,t}=V\left(c_{i,t},s_{i,t-1},s_{i,t}\right).$$

(13)

Unlike final-answer consistency checks, these operations explicitly evaluate the mathematical validity of intermediate transformations. For example, consider an incorrect approximation

$${\displaystyle \frac{x^2}{x^2+a^2}}\approx {\displaystyle \frac{x^2}{a^2}},a>0,$$

(14)

applied over an interval containing $x=a$. Evaluating the two expressions at this boundary gives

$${\displaystyle \frac{a^2}{a^2+a^2}}={\displaystyle \frac{1}{2}},{\displaystyle \frac{a^2}{a^2}}=1.$$

(15)

The discrepancy violates the required boundary consistency condition. Under a verification tolerance smaller than the observed discrepancy, the verifier therefore returns

$$g_{i,t}=0.$$

(16)

This directly identifies an invalid intermediate approximation rather than inferring its correctness from the final answer.

Online Diagnostic Filtering.  Although the policy is initialized with certified checks, it may still generate trivial or nondiscriminative conditions during reinforcement learning. To prevent such behavior, we evaluate the diagnostic quality of each generated check through local counterexample testing. For an executable check $c_{i,t}$, we construct M controlled perturbations of the associated reasoning step and compute

$${\widehat{d}}_{i,t}={\displaystyle \frac{1}{M}}{\displaystyle \sum _{m=1}^M}\mathbb{I}\left[V\left(c_{i,t},{\tilde{s}}_{i,t}^{\left(m\right)}\right)=0\right].$$

(17)

When the corresponding mathematical transformation cannot be parsed or valid counterexamples cannot be constructed, the diagnostic score is treated as unavailable. We then define an activity indicator

$$m_{i,t}=\left\{\begin{array}{cc}1,\hfill & g_{i,t}\ne \perp \mathrm{and}{\widehat{d}}_{i,t}\ge {\tau }_d,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(18)

Thus, only executable and sufficiently discriminative checks contribute to the process reward. Checks that cannot be parsed, evaluated, or shown to possess diagnostic value remain inactive. This filtering mechanism prevents the policy from obtaining verification rewards by generating mathematically irrelevant conditions that are trivially satisfied.

Stepwise Process Reward.  We assign a signed verification score to every reasoning step:

$$r_{i,t}^{\mathrm{step}}=\left\{\begin{array}{cc}+1,\hfill & m_{i,t}=1\mathrm{and}{g}_{i,t}=1,\hfill \\ -1,\hfill & m_{i,t}=1\mathrm{and}{g}_{i,t}=0,\hfill \\ 0,\hfill & m_{i,t}=0.\hfill \end{array}\right.$$

(19)

A verified correct transformation receives positive credit, a detected mathematical inconsistency receives negative credit, and an inactive or unverifiable check contributes zero reward.

The trajectory-level process reward is then defined as

$$r_i^{\mathrm{proc}}={\displaystyle \frac{1}{T_i}}{\displaystyle \sum _{t=1}^{T_i}}{r}_{i,t}^{\mathrm{step}}.$$

(20)

Importantly, the reward is normalized by the total number of reasoning steps rather than the number of successfully verified steps. Therefore, a trajectory cannot obtain a high process reward by attaching a small number of easy checks to an otherwise unverified derivation. Instead, maximizing the process reward requires both sufficient verification coverage and consistency across intermediate transformations.

By construction, the process reward satisfies

$$-1\le r_i^{\mathrm{proc}}\le 1.$$

(21)

Outcome Reward..  Final-answer correctness remains an independent optimization target. For each response $o_i$, we define

$$r_i^{\mathrm{ans}}=\mathbb{I}\left[{\widehat{y}}_i=y\right],$$

(22)

where y denotes the ground-truth answer. We deliberately avoid modifying the outcome reward according to the model’s self-verdict. In particular, correct answers retain their full correctness reward even when a generated intermediate check fails, while incorrect answers receive no additional outcome credit merely for identifying an inconsistency. This design prevents the policy from being rewarded for unnecessary self-rejection and ensures that final-answer correctness remains the primary learning objective.

Total Reward.  The final reward combines outcome correctness with stepwise verification:

$$R_i=r_i^{\mathrm{ans}}+{\beta }_{\mathrm{proc}}{r}_i^{\mathrm{proc}},$$

(23)

where ${\beta }_{\mathrm{proc}}$ controls the relative contribution of process supervision.

Unlike objectives that combine multiple auxiliary incentives, the proposed reward contains only two components: final-answer correctness and programmatically verified intermediate reasoning. No additional self-rejection, verdict-gating, or response-length rewards are introduced.

This simplified reward formulation allows the contribution of step-level mathematical verification to be studied independently of other reward-shaping mechanisms.

Group-Relative Policy Optimization.  We optimize the policy using the standard GRPO objective. Given a response group with rewards ${\left\{R_i\right\}}_{i=1}^N$, the group-relative advantage is computed as

$$A_i={\displaystyle \frac{R_i-{\mu }_R}{{\sigma }_R+{ϵ}_A}},$$

(24)

where

$${\mu }_R={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{R}_j,{\sigma }_R=\sqrt{{\displaystyle \frac{1}{N}}\sum _{j=1}^N{(R_j-{\mu }_R)}^2}.$$

(25)

For the t-th token in response $o_i$, we define the importance ratio ${\rho }_{i,t}\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}}.$ The policy objective is

$$\begin{array}{cc}\hfill \mathcal{J}\left(\theta \right)={\mathbb{E}}_{q,\left\{o_i\right\}}\left[{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \frac{1}{|o_i|}}{\displaystyle \sum _{t=1}^{|o_i|}}\right(& \min ({\rho }_{i,t}\left(\theta \right)A_i,\hfill \\ \hfill & clip\left({\rho }_{i,t}\left(\theta \right),1-{ϵ}_c,1+{ϵ}_c\right)A_i)\hfill \\ \hfill & -{\beta }_{\mathrm{KL}}{D}_{\mathrm{KL}}\left({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}\right)\left)\right],\hfill \end{array}$$

(26)

where ${ϵ}_c$ is the clipping threshold and ${\beta }_{\mathrm{KL}}$ controls regularization toward the reference policy. The KL term is evaluated at the corresponding token context. The underlying GRPO optimization remains unchanged. The principal difference lies in the construction of the per-response reward, which incorporates direct mathematical verification of intermediate reasoning while preserving the standard rollout sampling budget.

3.3. Discussion

Two properties distinguish SCOPE-RL from conventional outcome-only reinforcement learning and approaches based on free-form reasoning constraints.

First, the supervision is grounded in executable mathematical operations. Rather than evaluating a condition only against the final answer, SCOPE-RL directly checks intermediate mathematical transformations, allowing inconsistencies to be detected at the step where they arise. The certification procedure further ensures that retained checks are not merely syntactically valid but possess measurable discrimination against local counterexamples. Consequently, the process reward reflects both mathematical consistency and the diagnostic capability of the verification conditions.

Second, outcome correctness and process verification are explicitly separated. This avoids introducing reward incentives that favor self-rejection or reduce the correctness reward solely because a trajectory violates one of its own generated checks. To formalize this property, we establish a sufficient condition under which correct responses are always preferred over incorrect responses in terms of their total reward.

Proposition 1. Let the total reward be

$$R=r^{\mathrm{ans}}+{\beta }_{\mathrm{proc}}{r}^{\mathrm{proc}},$$

(27)

where

$$r^{\mathrm{ans}}\in \{0,1\},r^{\mathrm{proc}}\in [-1,1].$$

(28)

If

$$0\le {\beta }_{\mathrm{proc}}<{\displaystyle \frac{1}{2}},$$

(29)

then every correct response receives a strictly greater total reward than every incorrect response.

Proof. For any response with a correct final answer, we have

$$R_{\mathrm{correct}}=1+{\beta }_{\mathrm{proc}}{r}^{\mathrm{proc}}\ge 1-{\beta }_{\mathrm{proc}}.$$

(30)

For any response with an incorrect final answer,

$$R_{\mathrm{incorrect}}={\beta }_{\mathrm{proc}}{r}^{\mathrm{proc}}\le {\beta }_{\mathrm{proc}}.$$

(31)

Since ${\beta }_{\mathrm{proc}}<1/2$, it follows that

$$1-{\beta }_{\mathrm{proc}}>{\beta }_{\mathrm{proc}}.$$

(32)

Therefore,

$$R_{\mathrm{correct}}>R_{\mathrm{incorrect}},$$

(33)

which completes the proof.

This proposition establishes that, under the specified reward coefficient, process supervision cannot reverse the reward ordering between correct and incorrect final answers. It does not guarantee that every intermediate step is verified or that the optimized policy always produces correct reasoning; rather, it ensures that the additional verification signal cannot make an incorrect outcome more rewarding than a correct one.

Furthermore, the three-state verification mechanism distinguishes detected mathematical violations from unsupported checks. A failed executable condition provides explicit negative feedback, whereas an unexecutable or inapplicable condition contributes no process reward. This prevents the limitations of symbolic parsing or numerical verification from being conflated with reasoning errors.

Finally, the framework does not assume that all mathematical transformations can be verified programmatically. Its effectiveness depends on the coverage and reliability of the executable verifier library. By explicitly measuring verification coverage and counterexample rejection, SCOPE-RL provides a framework in which the quality of intermediate reasoning supervision can be evaluated independently of downstream answer accuracy.

4. Experiment

LLM/VLM usage.  Our backbone suite covers five representative models spanning different training paradigms and model families. Specifically, we consider Qwen3-4B-Instruct-2507, Qwen3-4B-Adaptive (No-Think), and Qwen3-4B-Base [13], which provide a controlled comparison among instruction-tuned, non-deliberative, and pretrained variants within the same architecture. We further include Llama-3.1-8B-Instruct [39] and Qwen2.5-7B-Base [40] to broaden the evaluation across distinct architectures and parameter scales. This diverse backbone selection enables us to study the robustness of SCOPE-RL under different levels of post-training, reasoning behavior, and model capacity. Beyond text-based reasoning, we further evaluate the generalizability of SCOPE-RL in the multimodal setting, using Qwen2.5-VL-7B [40] and MiMo-VL-Instruct-2508 [41] as the vision-language backbone.

Experimental Settings.  SCOPE-RL is trained in two stages. We first perform supervised fine-tuning on 10K OpenThoughts mathematical problems, where reference solutions are decomposed into intermediate steps and paired with certified executable checks using the <step>, <check>, and <answer> format. We then optimize the warmed-up policy with GRPO, sampling 8 responses per prompt. The rollout batch size is 640 and the global batch size is 160. We use AdamW with a learning rate of $1\times {10}^{-6}$, weight decay $1\times {10}^{-2}$, GRPO clipping coefficient $0.2$, and KL coefficient $1\times {10}^{-2}$. Training uses temperature $1.0$ and top-$p=1.0$, while evaluation uses temperature $0.6$ and top-$p=0.95$. The maximum prompt length is set to 2,048 tokens for all models. During training, the maximum response length is 4,096 tokens for all models except Qwen3-4B-Instruct-2507, for which it is set to 10,000 tokens. During evaluation, we increase the maximum response length to 8,192 tokens for the other models and 15,000 tokens for Qwen3-4B-Instruct-2507.

For executable verification, we use symbolic equivalence, substitution, equation residual, boundary, and sign/range checks, with a numerical tolerance of ${10}^{-6}$ for non-symbolic comparisons. Each candidate check is certified using $M=8$ controlled local counterexamples generated by coefficient changes, sign flips, operator replacement, incorrect substitutions, denominator removal, or inequality reversal. Perturbations that remain mathematically valid or cannot be verified are discarded. A check is retained only if it accepts the reference transformation and rejects at least ${\tau }_d=0.6$ of the valid counterexamples. The same criterion is applied online during RL; checks that are unparsable, unexecutable, or insufficiently discriminative contribute no process reward.

The process reward assigns $+1$ to verified transformations, $-1$ to detected violations, and 0 to inactive checks, and is averaged over all reasoning steps. The total reward is

$$R_i=r_i^{\mathrm{ans}}+{\beta }_{\mathrm{proc}}{r}_i^{\mathrm{proc}},$$

(34)

where we set ${\beta }_{\mathrm{proc}}=0.3$. No additional self-rejection, verdict-gating, structural, or length reward is used.

Data Construction and Evaluation Suite.  For the mathematical reasoning experiments, we adopt the data construction protocol of [42] and use training samples from DAPO [30] and DeepscaleR [43]. To assess the domain generality of SCOPE-RL, we further extend the training and evaluation pipeline to multimodal reasoning [16], rather than restricting the analysis to text-only mathematics. For a fair comparison, all baselines use the same warm-start initialization. The only difference is that, in SCOPE-RL, the check component is self-generated by the policy and is further improved through verification-guided reinforcement learning.

We evaluate mathematical reasoning performance on AIME24, AIME25, MATH500, AMC23, Minerva, and Olympiad, reporting accuracy averaged over 64 independent runs. Our multimodal evaluation spans both general-purpose visual-language understanding and specialized scientific reasoning. Specifically, MathVista [44], MathVision [45], MathVerse [46], and EMMA [47] are used to assess broad multimodal reasoning.

4.1. Experimental Results

As shown in Table 1, SCOPE-RL consistently achieves the best performance across five backbone configurations and six mathematical reasoning benchmarks, covering a total of 30 model–benchmark combinations. Overall, the results demonstrate that the proposed method provides stable and generalizable improvements across models with substantially different reasoning capabilities, including instruction-tuned models, adaptive no-think variants, base models, and models from different architecture families. Compared with vanilla GRPO, SCOPE-RL improves the average performance by 3.12, 3.93, 5.37, 1.72, and 4.88 points on Qwen3-4B-Instruct-2507, Qwen3-4B-Adaptive (No-Think), Qwen3-4B-Base, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Base, respectively. These consistent gains indicate that the effectiveness of SCOPE-RL is not tied to a particular model initialization or reasoning format.

The improvements are especially evident on more challenging reasoning benchmarks. For Qwen3-4B-Base, SCOPE-RL improves GRPO by 7.4 points on AIME24 and 5.4 points on AIME25, while also achieving gains of 6.9 points on Math500 and 4.8 points on Minerva. Similarly, on Qwen2.5-7B-Base, SCOPE-RL yields improvements of 3.3 and 4.4 points on AIME24 and AIME25, respectively, and achieves a substantial 9.8-point gain on AMC23. Notably, the method also remains effective under the Qwen3-4B-Adaptive (No-Think) setting, where it improves GRPO by 6.7 points on AMC23 and 5.1 points on Olympiad. These results suggest that SCOPE-RL can provide meaningful benefits not only for strong reasoning-oriented models but also for weaker or less explicitly deliberative policies.

Beyond comparisons with vanilla GRPO, SCOPE-RL also consistently outperforms stronger RL-based baselines, including HET-DAPO, SRPO, PTA-GRPO, and HICRA-GRPO. For example, on Qwen3-4B-Instruct-2507, SCOPE-RL improves upon the strongest competing result by 1.4 points on both AIME24 and AIME25, while maintaining additional gains on Math500, AMC23, Minerva, and Olympiad. A similar trend can be observed on Qwen3-4B-Base, where SCOPE-RL surpasses the strongest competing baseline across all six benchmarks by approximately 0.9–1.6 points. This consistent advantage over both standard GRPO and more competitive RL variants indicates that the gains are not merely due to stronger reinforcement learning relative to a weak baseline, but instead reflect the effectiveness of the proposed learning strategy itself. Taken together, the results in Table 1 demonstrate that SCOPE-RL provides robust, consistent, and broadly transferable improvements for mathematical reasoning across different model families, reasoning settings, and benchmark difficulty levels.

As shown in Table 2, SCOPE-RL consistently achieves the best performance across both multimodal backbone models and all four reasoning benchmarks, demonstrating that the proposed method generalizes effectively beyond text-only mathematical reasoning. On MiMO-VL-Instruct-2508, SCOPE-RL improves vanilla GRPO by 8.3 points on MathVista, 7.4 points on MathVision, 3.6 points on MathVerse, and 3.4 points on EMMA, corresponding to an average improvement of 5.68 points. The gains are particularly notable on MathVista and MathVision, indicating that SCOPE-RL can substantially enhance visually grounded mathematical reasoning. Similar improvements are observed on Qwen2.5-VL-7B-Instruct, where SCOPE-RL surpasses GRPO by 2.0, 4.3, 1.6, and 2.4 points on the four benchmarks, respectively, yielding an average gain of 2.58 points. Importantly, SCOPE-RL also consistently outperforms stronger RL baselines such as ORZ and HICRA-GRPO across all evaluated settings. For example, compared with HICRA-GRPO, SCOPE-RL achieves gains of 1.3, 1.3, 1.2, and 1.2 points on MiMO-VL-Instruct-2508, while maintaining clear advantages on Qwen2.5-VL-7B-Instruct. These results suggest that the benefits of SCOPE-RL are not restricted to language-only reasoning, but transfer reliably to multimodal reasoning tasks that require joint visual understanding and multi-step problem solving.

4.2. Ablation Study

As shown in Table 3, the ablation results demonstrate that each component of SCOPE-RL contributes positively to the final performance. The complete SCOPE-RL framework achieves the highest average accuracy of 82.5 across the four mathematical reasoning benchmarks, outperforming vanilla GRPO by 4.2 points. Removing the Step-Check Warm-up causes the largest performance degradation, reducing the average score from 82.5 to 79.7, which highlights the importance of providing reliable step-level supervision before reinforcement learning. Removing Counterexample Certification also leads to a substantial 2.2-point decrease in average performance, suggesting that explicitly certifying potentially incorrect intermediate reasoning plays an important role in improving the reliability of step-level feedback. Online Diagnostic Filtering provides an additional benefit, as removing it decreases the average score from 82.5 to 80.8, indicating that filtering unreliable or ambiguous diagnostic signals is important for maintaining effective optimization. Similarly, removing the Negative Step Reward reduces the average performance to 81.4, showing that explicitly penalizing incorrect reasoning steps complements positive verification signals and improves credit assignment. The Verified-Step Normalization variant achieves 81.7, which is the strongest ablated configuration but still underperforms the full SCOPE-RL framework by 0.8 points. Overall, these results indicate that the components of SCOPE-RL are complementary rather than redundant: step-check warm-up establishes a reliable reasoning foundation, counterexample certification and online diagnostic filtering improve the quality of step-level supervision, and negative step rewards further strengthen fine-grained credit assignment, leading to the best performance when combined in the complete framework.

4.3. Effect of Warm-Start Strategy

To further investigate whether the improvement of SCOPE-RL mainly comes from supervised warm-starting or from the proposed certified step-check initialization, we conduct a controlled comparison on Qwen3-4B-Instruct-2507. Specifically, we compare three initialization strategies while keeping the subsequent reinforcement learning stage unchanged: (1) no supervised warm-up, (2) standard reasoning warm-up using conventional reference solutions, and (3) our Certified Step-Check Warm-up using structured <step>-<check>-<answer> trajectories. Both supervised variants use the same 10K OpenThoughts mathematical problems, so the comparison isolates the effect of the proposed step-level verification structure rather than the amount of supervised data.

As shown in Table 4, standard reasoning warm-up improves the average accuracy from 79.7 to 80.8, confirming that supervised initialization itself provides a useful starting point for reinforcement learning. However, our Certified Step-Check Warm-up further increases the average performance to 82.5, outperforming standard reasoning warm-up by 1.7 points on average.

The advantage is particularly clear on the more challenging AIME benchmarks. Compared with standard reasoning warm-up, the proposed initialization improves AIME24 from 72.3 to 74.5 and AIME25 from 63.8 to 66.5, corresponding to gains of 2.2 and 2.7 points, respectively. Consistent improvements are also observed on Math500 and AMC23.

These results indicate that the benefit of SCOPE-RL cannot be explained solely by additional supervised fine-tuning. While conventional reasoning warm-up improves the initial reasoning capability of the policy, the proposed warm-start explicitly exposes the model to executable and discriminative intermediate checks before reinforcement learning. This initialization better aligns the policy with the subsequent verification-guided objective, enabling the model to more effectively exploit step-level process supervision during reinforcement learning.

4.4. Performance on Test-Time Scaling

Figure 3 evaluates how effectively SCOPE-RL leverages additional test-time computation. As the number of sampled responses increases from $K=1$ to $K=16$, both GRPO and SCOPE-RL achieve consistent performance improvements across all six benchmarks, confirming the general effectiveness of test-time compute scaling. More importantly, SCOPE-RL consistently outperforms GRPO under every compute budget, demonstrating that its advantage is robust to different numbers of sampled reasoning trajectories. The gains are particularly evident on challenging benchmarks such as AIME 2024, AIME 2025, and Olympiad, where exploring additional reasoning paths is more beneficial. Although performance gradually saturates on relatively easier benchmarks such as Math500, SCOPE-RL maintains a clear advantage at larger values of K. Moreover, on several benchmarks, SCOPE-RL achieves performance comparable to or better than GRPO while using fewer samples, indicating a more favorable accuracy–computation trade-off. Overall, these results demonstrate that SCOPE-RL not only improves single-sample reasoning accuracy but also scales more effectively with additional test-time computation.

4.5. Sensitivity Analysis

As shown in Figure 4, the number of sampled rollouts has a clear influence on the final reasoning performance of both SCOPE-RL and GRPO. Across all six benchmarks, increasing the rollout number from 4 to 8 generally leads to consistent performance improvements, while further increasing it to 10 provides only marginal gains and in several cases results in a slight performance decrease. For SCOPE-RL, the best or near-best performance is typically achieved with eight sampled rollouts, including on Math500, AIME 2024, AIME 2025, AMC 2023, and Olympiad. A similar saturation trend can also be observed for GRPO, suggesting that excessively increasing the rollout budget does not necessarily translate into better optimization. More importantly, SCOPE-RL consistently outperforms GRPO under every rollout setting and on all benchmarks, indicating that its advantage is robust to the choice of sampling budget. The performance gap is particularly evident on AIME 2024, AIME 2025, AMC 2023, Minerva, and Olympiad, where SCOPE-RL maintains substantially higher accuracy across the entire range of rollout numbers. Overall, these results suggest that a moderate rollout budget provides a favorable trade-off between training efficiency and reasoning performance, with eight sampled rollouts offering a strong default configuration for SCOPE-RL.

4.6. Verification Quality Analysis

Beyond final-answer accuracy, we directly examine whether SCOPE-RL improves the quality of intermediate verification. For each generated reasoning step, we record the verifier state $g_{i,t}\in \{1,0,\perp \}$ and the online counterexample rejection score ${\widehat{d}}_{i,t}$. We report the executable rate, verification coverage, check pass rate, unsupported-check rate, and mean counterexample rejection score.

As shown in Table 5, reinforcement learning substantially improves the verifiability of generated reasoning trajectories. The fraction of executable checks increases from 78.6% after step-check warm-up to 91.3% after SCOPE-RL, while effective verification coverage rises from 61.8% to 80.7%. At the same time, the proportion of unsupported or inapplicable checks decreases from 21.4% to 8.7%. Among active checks, the pass rate improves from 84.7% to 93.1%, indicating that the generated reasoning steps become increasingly consistent with their executable mathematical conditions. Finally, the mean counterexample rejection score increases from 0.69 to 0.81, suggesting that the learned checks become more diagnostic rather than merely easier to satisfy.

5. Discussion

Our results suggest that explicitly verifying intermediate mathematical transformations provides a complementary source of supervision to conventional outcome-based reinforcement learning. Standard RLVR methods such as GRPO primarily optimize final-answer correctness, assigning the same trajectory-level reward to all reasoning tokens regardless of where a mathematical error occurs. In contrast, SCOPE-RL introduces an additional process signal by executing typed mathematical checks directly on intermediate reasoning steps. This enables the training objective to distinguish locally valid transformations from detected inconsistencies while retaining final-answer correctness as the primary optimization target.

A key observation is that executability alone is insufficient for useful process supervision. A mathematical condition may be perfectly valid yet provide little information about whether a particular transformation is correct. SCOPE-RL therefore evaluates not only whether a check can be executed, but also whether it can discriminate a valid reference transformation from controlled local perturbations. The consistent degradation observed when counterexample-based certification or online diagnostic filtering is removed indicates that the effectiveness of process supervision depends critically on the diagnostic quality of the verification signal rather than simply on the presence of additional structured outputs.

The results also highlight the importance of signed step-level feedback. Rewarding verified transformations alone provides a useful positive signal, but explicitly penalizing detected mathematical violations yields further improvements. This suggests that identifying where a reasoning trajectory becomes mathematically inconsistent can provide information that is not available from terminal correctness alone. At the same time, unsupported or unexecutable checks receive zero reward rather than being treated as failures. This three-state design is important because symbolic parsing and numerical verification are necessarily incomplete, and failures of the verifier should not automatically be interpreted as failures of the underlying reasoning.

Another important design choice is the separation between process verification and outcome correctness. SCOPE-RL does not gate the final-answer reward according to the model’s own intermediate checks. A correct answer therefore retains its outcome reward even if one of its generated checks fails, while an incorrect answer cannot obtain additional correctness credit merely by detecting its own inconsistency. This avoids creating incentives for unnecessary self-rejection and preserves the reward ordering between correct and incorrect outcomes when the process-reward coefficient satisfies the condition established in Proposition 1. As a result, executable verification acts as an auxiliary source of process supervision rather than replacing the original task objective.

The ablation results further show that the certified step-check warm-up is important for stable optimization. Pretrained language models do not naturally produce intermediate reasoning in a form that can be reliably parsed and evaluated by symbolic or numerical operators. Supervised initialization therefore provides the structural interface required by the verifier, while reinforcement learning subsequently improves the consistency and usefulness of the generated checks. Moreover, normalizing the process reward by the total number of reasoning steps rather than only the number of active checks encourages verification coverage across the trajectory and prevents the policy from receiving a high process reward by attaching a small number of easy checks to otherwise unverified reasoning.

Despite these advantages, SCOPE-RL has several limitations. Its effectiveness depends on the coverage and reliability of the executable verifier library, while counterexample-based certification only ensures discrimination against the perturbations considered during training. In addition, symbolic parsing may fail on informal or complex reasoning steps. Finally, SCOPE-RL is primarily suited to domains with explicit executable semantics and is less applicable to open-ended reasoning tasks that cannot be deterministically verified.

These limitations suggest several promising directions for future work. Expanding the verifier library with theorem provers, computer algebra systems, numerical solvers, or other external tools could increase verification coverage. Learned verifiers may also be combined with executable checks for transformations that cannot be handled deterministically, while retaining programmatic verification whenever possible. Another direction is to adaptively select verification operators according to the type and uncertainty of each reasoning step, thereby allocating stronger verification to steps that are more likely to propagate errors. More broadly, our results indicate that intermediate supervision is most useful when it is not merely dense, but executable, localized, and demonstrably diagnostic.

6. Conclusions

We introduced SCOPE-RL, a reinforcement learning framework for mathematical reasoning that augments outcome-based RLVR with executable step-level verification. SCOPE-RL associates intermediate reasoning steps with typed mathematical checks and certifies these checks using valid reference transformations and controlled local counterexamples, while online diagnostic filtering suppresses trivial or nondiscriminative verification signals during reinforcement learning. The resulting process reward provides positive credit for verified transformations, negative credit for detected inconsistencies, and no reward for unsupported checks, and is combined with final-answer correctness under the standard GRPO objective without additional rollout sampling. Experiments across multiple language models and mathematical reasoning benchmarks show that SCOPE-RL consistently improves over GRPO and other strong RLVR baselines, while ablation studies confirm the importance of structured step-check initialization, counterexample-based certification, online filtering, and signed step-level supervision. Overall, these results demonstrate that executable intermediate verification provides an effective and practical way to introduce reliable process supervision into reinforcement learning, enabling reasoning models to produce solutions that are not only more accurate but also more mathematically consistent and verifiable.