LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

Source-first digest for checked paper rank 9, rank_id p013.

• Routing status: success
• PDF extraction: not used

Motivation / Background

The paper starts from a concrete failure mode in current VLMs: semantically identical text can produce different behavior depending on whether it is presented as tokens or rendered as an image. The authors call this carrier sensitivity. In the motivating experiment, the same questions are evaluated under normal text input and under a rendered-text-as-image protocol; accuracy drops for all four tested VLMs, including Qwen3.5-9B dropping from 68.95 to 60.25 and LLaVA-OV1.5-8B dropping from 53.52 to 41.84.

The paper argues that this is not just OCR weakness. It measures hidden-state distance between a text input and its rendered-image counterpart, then shows a monotonic relationship: samples in the closest distance bin lose 7.75 points, while samples in the farthest bin lose 21.23 points. Figure 1 is the central motivation figure because it connects the observed accuracy drop, the pairwise cross-modal distance, and LoMo's claimed 14.2% reduction in average distance.

LoMo is the proposed data-side fix. Instead of changing the VLM architecture, it takes a text-only training example, renders a local span as an image, perturbs that image, and inserts it back into the surrounding text. The model must then answer from a text -> visual -> text input while the supervision target remains unchanged. The design is summarized in Figure 2.

Claims And Evidence

Scores are support-from-paper scores, not independent reproduction scores. C1 is capped at 4 because the distance analysis is correlational even though the trend is clean. C7 is capped at 3 because the pure-text gains are small and one metric regresses.

Core Technical Idea

LoMo is a data transformation, not a new model. Given a text-only supervised example \((x, a)\), it selects a middle span \(x_{\text{mid}}\), renders that span into an image, applies a visual perturbation, and returns a mixed-carrier input whose answer is still \(a\).

The important supervision signal is that the target answer is unchanged, so the model must learn to treat a rendered local text span as a semantic continuation of the surrounding token text. Figure 2 makes the mechanism concrete: choose a coherent span, render it, optionally distort it, and insert it into the original position.

The paper also rewrites the SFT objective to expose the intended alignment pressure. Standard SFT optimizes:

LoMo instead optimizes the same answer under the transformed carrier:

Taking expectation over \(a \sim p_\theta(\cdot \mid x)\) turns the second term into a KL-style agreement pressure:

The claim is not that LoMo explicitly minimizes a separate KL loss in implementation. The claim is that training on semantically equivalent mixed-carrier inputs turns the ordinary answer loss into an implicit pressure for text and rendered-text carriers to agree.

Method Details

Span Localization, Rendering, And Distortion

LoMo's pipeline is intentionally simple:

1. If the prompt has at most three sentences, render the whole text as the target span. Otherwise, split the input with a formula-aware chunker. 2. Treat text and formula blocks as atomic units so equations are not cut in half:

3. Select the middle one-third of the block sequence as \(x_{\text{mid}}\), keeping prefix and suffix in text. 4. Render math spans with a LaTeX renderer and non-math spans with a text renderer. If LaTeX rendering fails, fall back to text rendering rather than dropping the sample. 5. Trim excess margins and sample one of Clean, Rotate, Blur, ShadowOrStain, or Wave as the final perceptual distortion.

The appendix algorithm can be summarized as:

Input: text-only instance (x, a)
If SentenceCount(x) <= 3:
  (x_pre, x_mid, x_suf) = (empty, x, empty)
Else:
  C = FormulaAwareChunk(x)
  (x_pre, x_mid, x_suf) = ExtractMiddle(C)
If ContainsMath(x_mid):
  I = LaTeXRender(x_mid), falling back to TextRender
Else:
  I = TextRender(x_mid)
I = TrimMargin(I)
I' = one of Clean, Rotate, Blur, ShadowOrStain, Wave
Return ((x_pre, I', x_suf), a)

Training Setup

The experiments use two open VLM backbones: LLaVA-OneVision-1.5-8B-Base and Qwen3.5-9B-Base. The training pool is the LLaVA-OneVision 1.5 SFT corpus: 2M multimodal instruction examples plus 2M text-only instruction examples. Standard SFT trains on the same 4M examples without rendering; LoMo renders 50% of the text-only examples and keeps the optimizer, learning-rate schedule, number of steps, and data scale matched.

Implementation details are conventional SFT rather than a new architecture: one node with 8 NVIDIA H200 GPUs, maximum sequence length 32,768, maximum image resolution 2,560,000 pixels, FlashAttention 2, bf16, DeepSpeed ZeRO Stage 1, sequence packing, AdamW, one epoch, frozen vision tower, and updates to the language model plus multimodal projector.

Evaluation Protocols And Metrics

The paper evaluates each benchmark two ways. Standard Evaluation uses the original image plus text question. Rendered Evaluation renders the whole question as an image and uses that rendered image in place of text. Because the linguistic content is identical, the rendered protocol isolates carrier sensitivity.

For internal alignment, the paper uses MIR as a set-level visual/text token distribution gap and a paired distance:

This metric is important because Figure 4 shows that LoMo improves both task accuracy and representation alignment as the data scale grows.

Experiments And Results

Main Benchmark Results

Table 1 is the main evidence for the paper's performance claims. Under Standard Evaluation, LoMo improves the average by +2.68 for LLaVA-OV1.5-8B and +2.82 for Qwen3.5-9B. Under Rendered Evaluation, the average gains grow to +18.86 and +11.92, respectively. This is the strongest support for the claim that LoMo specifically attacks carrier sensitivity.

Table 1. Main results across 13 multimodal benchmarks under two evaluation protocols. Original caption: Standard Evaluation feeds the original multimodal inputs (image + text question) to the model. Rendered Evaluation renders the entire text question as a single image. \(\Delta\) denotes the absolute change of LoMo over Standard SFT.

The radar figure gives the compact visual version of the Standard Evaluation results. It is useful because it shows that the average gain is not concentrated in a single category: LoMo improves most axes for both backbones, with small regressions on LLaVA MMMU and Qwen MathVista/CC-OCR.

Representation Alignment And Data Scale

Figure 4 is the main evidence that LoMo's benefits track alignment, not only downstream accuracy. At the 4M scale, LoMo's average accuracy gain reaches +2.68, MIR is 0.122 lower than Standard SFT, and paired cross-modal distance drops to 0.49 while Standard SFT rises to 0.57.

Component Ablations

Table 2 separates the pieces of LoMo. Full-question rendering improves the average from 40.88 to 42.07, but LoMo without perceptual distortion reaches 43.10 and full LoMo reaches 43.56. The paper's interpretation is that span localization and interleaving are the dominant ingredients, while perceptual distortion adds another smaller gain.

Table 2. Component ablation of LoMo on LLaVA-OV1.5-8B. Original caption: Full-Text Rendering renders the entire input as images without Structure-Aware Span Localization or Perceptual Distortion. PD: Perceptual Distortion.

Rewrite Ratio, Position, And Matched Exposure

Table 3 shows that rewriting some text-only data is consistently useful, but rewriting all of it is not best. The average peaks at a 50% rewrite ratio, 43.56, then falls to 42.68 at 100%, suggesting that LoMo needs a mixed diet of normal text and interleaved text-image examples.

Table 3. Quantitative results of different rewrite ratios on LLaVA-OV1.5-8B. Original caption: The rewrite ratio controls the fraction of text-only training samples reformatted into text-image interleaved sequences with LoMo. \(\Delta\) denotes the average improvement over the Standard SFT baseline.

Table 4 tests where the rendered span should sit. Middle placement is best at 43.56; prefix, suffix, and multi-span are all lower. This supports the method rationale that a text-image-text structure forces fusion between visual and textual carriers.

Table 4. Quantitative results of different rendering positions on LLaVA-OV1.5-8B. Original caption: Prefix renders the first one-third of the prompt, Middle renders the central one-third, Suffix renders the last one-third, and Multi-Span renders two short spans.

Table 5 addresses a potential confound: LoMo converts some text-only samples into image-bearing samples, so perhaps the benefit comes from more visual exposure. The matched 1:1 image:text comparison still gives +2.45 average points over Standard SFT, close to the original +2.68, so this concern is partially controlled.

Table 5. Controlled comparison under different image-bearing to text-only sample ratios on LLaVA-OV1.5-8B. Original caption: The original setting uses LoMo's natural 3:1 ratio after rewriting, while the matched setting controls the effective ratio back to 1:1 to match Standard SFT.

Pure-Text Sanity Check

Table 6 is a narrow safety check. LoMo slightly improves the average on both backbones, +0.28 and +0.58, but the changes are small and not uniformly positive. The most important reading is "no obvious pure-text collapse" rather than a strong pure-text improvement claim.

Table 6. Pure-text capability sanity check. Original caption: Standard SFT and LoMo are evaluated on five pure-text benchmarks covering general knowledge, mathematical reasoning, code generation, and instruction following.

Practical Takeaways

• The most reusable idea is the data interface: convert part of a text-only instruction into a rendered, perturbed image while leaving the answer unchanged. This makes text-pixel equivalence part of the supervised task.
• The strongest empirical result is under Rendered Evaluation. The paper's Standard Evaluation gains are useful, but the much larger rendered gains show where LoMo is most targeted.
• The method is cheap at inference because it changes only training data. Its cost is in rendering and SFT data construction.
• The ablations argue against two weak explanations: full-text rendering is not enough, and a matched image:text ratio still improves. Middle local substitution appears to matter.
• The main limitation is scope. LoMo is validated only during SFT on two 8B-9B backbones; behavior during pretraining, RL post-training, and larger-model scaling is left open.

The paper's limitations section says LoMo is applied only during SFT, uses a block-level middle-span heuristic, and is evaluated under compute constraints on two backbones. Those caveats matter because the method is promising as a data recipe, but not yet shown as a universal pretraining or post-training principle.