Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering

Source-first digest for checked paper rank 42, rank_id p029.

• Routing status: success
• PDF extraction: not used

Motivation / Background

Knowledge-intensive QA is an awkward target for RL. Math and code can often use deterministic reward functions, but factual QA responses can mix a correct answer with unsupported claims in the same reasoning trace. Outcome-level rewards see only the final answer, while sentence-level factuality rewards usually require NLI models, LLM judges, retrieval, or knowledge-verification services for every generated sentence.

CorVer targets that gap. It turns Wikipedia subject-object co-occurrence into a process reward: extract one factual triplet from each generated sentence, query an Infini-gram Wikipedia index for the subject and object content words, convert the count into a small sentence reward, and align that sentence reward back to the generated tokens. Figure 1 frames the paper's motivation: factual QA lacks the cheap verification loop that has made RL easier for math and code.

The key caveat is already visible in the motivation: CorVer is not a fact checker. A co-occurrence count can be a useful directional signal, but it cannot prove the predicate is correct. The rest of the paper argues that this weak but cheap signal is useful enough when it is calibrated, bounded, and combined with a response-level answer reward.

Claims And Evidence

Scores are support-from-paper scores, not independent reproduction scores. Claims with single-model diagnostics or asymmetric baseline configurations are capped below 5 even when the reported results are strong.

Core Technical Idea

CorVer has three reward channels:

• A response-level string-match judge reward for whether the answer is correct.
• A response-level format reward for the required think / answer template.
• A sentence-level co-occurrence reward, aligned back to the tokens in that sentence.

Figure 2 is the compact view: generated text is split into sentences, a 0.5B QuCo extractor pulls a subject-object relation triplet, the subject and object content words are counted in a Wikipedia Infini-gram index, and the resulting sentence reward is combined with GRPO.

For a factual sentence \(s_i\), CorVer extracts a first valid triplet, discards the relation for the main query, reduces the head and tail to content words, and counts bounded-window co-occurrences:

The count becomes a bounded auxiliary reward:

Table 1. Four-tier co-occurrence reward map. The concrete bins come from the appendix implementation section and the human calibration audit. Table 1 is important because it shows how deliberately small the co-occurrence term is.

The per-token return adds the sentence reward only to tokens that align to that sentence:

That is the main design decision. A response can get a positive final-answer reward while unsupported sentences receive a local negative shaping term, and two sentences in one completion can receive different local advantages.

Method Details

The paper uses a local English Wikipedia 20231101 Infini-gram index with about 6.4M articles and 5.5B tokens. Counts are bounded to a 1,000-token inter-clause window, so \(c_i\) measures passage-like position-level co-occurrence, not document-level co-occurrence. The extractor is QuCo-extractor-0.5B, a Qwen2.5-0.5B-Instruct triplet model. For each sentence, the implementation keeps only the first valid head-relation-tail triplet whose head and tail are non-empty and non-pronominal.

The reward scale is intentionally conservative. The response judge maps GOOD, BAD, and NA to \(+2.0, -1.0, -1.0\), the format reward is \(\pm 1.0\), and \(\lambda_{\mathrm{f}}=\lambda_{\mathrm{j}}=\lambda_{\mathrm{c}}=1.0\). The source text notes that the maximum co-occurrence contribution for a typical completion is about \(0.3\), an order of magnitude smaller than the judge reward swing of \(3.0\). This is why CorVer is best read as a local shaping signal, not a replacement correctness label.

Training uses LoRA with rank 128 and alpha 256, \(G=16\) generations per prompt, prompt batch 24, max completion length 1024, and 100 GRPO steps for the canonical runs. The method trains directly from the raw instruction-tuned model rather than from an SFT cold start. The paper reports that small 3B/4B models needed mastered anchor questions mixed into the self-filtered learning-zone pool, while models at least 8B did not.

The evaluation setup is summarized in Table 2. Training prompts come only from NQ-Open train and WebQuestions, so TriviaQA, PopQA, SimpleQA, and TruthfulQA are out-of-distribution for the RL prompt source.

Table 2. Experimental setup. Table 2 is included to make the scope of the reported claims explicit.

Calibration And Reward Robustness

The main validation question is whether \(c_i\) is directionally meaningful. Figure 5 and Table 3 report the human audit: correctness rises monotonically from 24.0% for \(c_i=0\) to 81.0% for \(c_i \geq 20\).

Table 3. Five-bucket calibration audit. The two thresholds used by the reward map sit at visible precision transitions: \(c_i=5\) and \(c_i=20\).

Figure 6 checks whether the zero-count penalty is fragile. The full recipe is retrained with zero-count penalties from -0.1 to -1.0 on Llama-3.2-3B-Instruct; both weaker and stronger penalties underperform the canonical -0.3 setting.

The qualitative case study in Table 9 shows the same idea at the sentence level: zero co-occurrence can catch plausible but wrong entities that a GPT-4o-mini offline judge labels correct.

Experiments And Results

Main Accuracy Results

Table 4 compresses the main table into model-level averages and the two exceptions noted in the source. CorVer improves every Raw cell in the four-model main comparison; compared with the four prior factuality-RL baselines, it wins 18 of 20 model-benchmark cells under the paper's feasible baseline configurations.

Table 4. Main result summary. The average deltas in Table 4 are computed from the source table. The paper's own wording emphasizes that CorVer improves every Raw cell and beats prior baselines in 18 of 20 cells, with the two baseline wins both on Qwen3-8B and within a point.

Scaling Across Six Models

Figure 3 and Table 5 address generality. The scaling heatmap reports positive CorVer-minus-Raw gains in all 30 cells across six instruction-tuned models and five benchmarks.

Table 5. Full scaling summary. Table 5 preserves the Raw and CorVer values behind the 30 positive cells.

Cost And Feasibility

At the canonical rollout density, the reward loop issues about:

sentence-level reward calls per training run. Figure 4 and Table 6 show why the paper cares so much about per-call cost.

Table 6. Training wall-clock comparison. Table 6 is not an equal-budget comparison; it reflects the paper's argument that neural-verifier baselines cannot be run at the same rollout configuration without prohibitive cost.

What Drives The Gain

Table 7 holds the base model fixed at Llama-3.1-8B-Instruct and removes components. The full configuration beats every ablation on every benchmark. The most important comparison is A3 versus full: A3 keeps the same QuCo scalar magnitude but removes per-token sentence alignment, and it recovers only part of the gain.

Table 7. Component ablation. Table 7 supports the claim that CorVer needs both the response-level judge and the token-aligned sentence reward, especially outside TriviaQA.

Table 8 compares alternative triplet aggregation rules. The canonical "first valid triplet" rule is not the most exhaustive factual checker, but it is the best training signal in the reported experiment: Min catches more within-sentence issues but collapses length and accuracy, while RelCheck adds relation tokens and cost but underperforms.

Table 8. Triplet aggregation variants. Table 8 explains a practical choice: a weaker, first-triplet proxy can train better than a harsher multi-triplet rule because the policy can exploit the latter by shortening outputs.

The paper's qualitative Figure 7 has no external graphics asset, so I reconstruct its content as Table 9. Each row is a single sentence where the co-occurrence reward assigns \(r_i^{\mathrm{c}}=-0.3\) and the offline GPT-4o-mini judge incorrectly says the sentence is correct.

Table 9. Reconstructed case-study figure. The source caption says these are three single-sentence examples from the zero-frequency bucket; all were independently verified as factually incorrect by a human annotator.

Coverage, Refusals, And Checkpoints

Table 10 is the paper's most important limitation analysis. If CorVer mainly rescued rare entities, gains should be largest in Q1. Instead, gains are larger for better-covered PopQA entities, especially Q3 and Q4.

Table 10. PopQA popularity-quartile gains. Table 10 shows that the reward works best where the corpus has enough coverage to distinguish correct from incorrect co-occurrences.

For Qwen3-8B, Table 11 shows a large reduction in refusal rates. The paper checks the subset of questions that Raw refused but CorVer attempted and reports nontrivial correctness on that subset, which supports the interpretation that the model is recovering some factual recall rather than just guessing everywhere.

Table 11. Qwen3-8B refusal decomposition. This diagnostic is useful but narrow: the source text says the Llama family shows the opposite NA-rate shift, so this is not a universal mechanism.

Figure 8 supports the uniform step-100 checkpoint choice. On a longer Llama-3.2-3B CorVer run, the largest jump comes in the first 100 steps; step 200 is higher but costs twice as much, and later checkpoints drift slightly down.

Practical Takeaways

• The reusable idea is not "co-occurrence equals truth"; it is "a cheap, bounded, calibrated corpus statistic can be useful as a dense auxiliary reward when paired with an ordinary answer reward."
• The strongest empirical support is breadth: all 30 Raw-vs-CorVer cells improve across six models and five benchmarks.
• The strongest engineering support is cost: at roughly 120k sentence-level reward calls per run, avoiding neural verifier calls changes what configurations are feasible.
• The most important limitation is predicate blindness. A subject and object can co-occur in Wikipedia while the generated relation is wrong.
• Corpus coverage is a real constraint. The PopQA analysis shows larger gains on better-covered entities, so this is not a magic fix for the rarest facts.
• The "first triplet" rule is pragmatic. It misses some multi-fact sentence errors, but harsher aggregation changed model behavior in worse ways.
• The paper validates CorVer only under GRPO. PPO, REINFORCE, DPO-style preference construction, larger web indexes, and combinations with retrieval or NLI rewards are future-work claims, not demonstrated results.

Limitations And Residual Risk

The reward is a corpus-grounded proxy, not a claim-level verifier. It ignores predicate semantics in the main query, keeps only one triplet per sentence, depends on English Wikipedia coverage, and uses a lenient substring-plus-alias evaluator for absolute accuracies. The relative comparisons use the same grader, so the tables are still meaningful for within-paper comparisons, but absolute correctness should be read cautiously.

I did not use raw PDF extraction. I did not consult latex_flattened/main.flattened.tex because the numeric tables needed for this digest were recoverable from paper.md; the HTML-rowspan tables were awkward, but not malformed enough to require fallback recovery.