Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

Source-first digest for checked paper rank 6, rank_id p012.

• Routing status: success, but the Markdown manifest reports six missing main-text includes, so the main body was recovered from LaTeX source files rather than PDF extraction.
• PDF extraction: not used

Motivation / Background

This paper asks whether VLM spatial-reasoning accuracy reflects structured 3D understanding or shortcut use from natural image statistics. The target shortcut is simple: in many photographs, objects that are farther away on the ground plane also appear higher in the image. A VLM can therefore answer "farther" questions by leaning on vertical image position instead of representing distance as its own spatial axis.

The authors call this failure mode vertical-distance entanglement. It matters for embodied and robotic settings because a model that appears good on standard spatial benchmarks may still fail when the natural correlation between "higher in the image" and "farther in 3D" is weakened, reversed, or synthetically controlled. Figure 1 summarizes the paper's thesis: many VLMs follow the perspective shortcut, while stronger spatial models separate axes and remain correct on counter-heuristic cases.

The geometric reason the shortcut exists is the standard ground-plane projection relation. For a camera at height \(H_c\) with focal length \(f\), a ground-plane point at depth \(Z\) projects to:

As \(Z\) grows, \(v_{\mathrm{img}}\) approaches the horizon, so farther ground-plane objects move upward in the image. The paper's core concern is that VLMs often internalize this correlation as if it were the depth relation itself.

Claims And Evidence

Scores are support-from-paper scores, not independent reproduction scores. C3-C6 are capped below 5 because the evidence is strong within the paper but still diagnostic/correlational rather than causal proof of the full training mechanisms.

Core Technical Idea

The paper combines two diagnostics:

• A behavioral split that classifies depth questions as perspective-consistent or counter-heuristic depending on whether the farther object appears higher in the image.
• A representation probe that uses swapped question pairs to test whether the model's hidden-state deltas form separable horizontal, vertical, and distance axes.

The behavioral split asks whether the shortcut appears in outputs. The probing method asks whether the shortcut is geometrically present in representation space. This distinction is important: two models can have similar benchmark accuracy but very different internal spatial structure.

The consistent/counter categorization is visualized in Figure 2. This figure is referenced by the main text as the operational definition behind the real-benchmark and SpatialTunnel splits.

Method Details

Model And Data Setup

The experiments cover Molmo-7B-O-0924, NVILA-Lite-2B, Qwen2.5-VL-3B-Instruct, RoboRefer-2B-SFT, and Qwen3-VL-235B-A22B-Instruct. For Molmo, NVILA, and Qwen2.5-VL, the authors train spatial fine-tuning variants at 80k, 400k, 800k, and 2M samples. The sampled spatial data mix comes from SAT, RoboSpatial, SPAR-7M, RefSpatial, and PRISM. RoboRefer is treated as a depth/spatially supervised reference model sharing the NVILA base family, and Qwen3-VL-235B is treated as a very large-scale reference model.

Consistent-Counter Split

For real-image benchmarks, each depth-related example is grouped as:

• Consistent: the farther object has a smaller image \(y\)-coordinate, meaning it is higher in the image.
• Counter: the farther object appears lower.
• Ambiguous: the vertical center difference is too small to support a clean split.

The split is a behavioral test for whether models rely on the elevation cue. If a model truly reasons about 3D depth, consistent and counter accuracy should be close. Large positive gaps imply shortcut dependence.

SpatialTunnel

SpatialTunnel is a synthetic Blender benchmark built around a single-point-perspective corridor. Two objects are placed at fixed depths while their angular positions on the tunnel cross-section are swept independently. This creates a \(16 \times 16\) grid of positions where vertical image location changes without changing the true depth order. Figure 3 shows this design.

For each image, the model answers binary depth questions such as whether one object is closer or farther than the other. The probability-based scoring rule extracts the first-token logits for Yes and No:

The correctness score is \(v=p\) when the ground-truth answer is Yes, and \(v=1-p\) when it is No. Reported metrics are mean correctness \(v\), consistent correctness \(v_{\text{cons}}\), counter correctness \(v_{\text{ctr}}\), and the shortcut gap:

Contrastive Probing

Figure 6 summarizes the representation-level probe. For each VQA example, the authors build a minimally swapped question pair, such as "Is A left of B?" vs. "Is B left of A?", so the ground-truth spatial relation flips while the image and object identities remain fixed. They extract the final-token hidden state at an intermediate layer and compute the delta vector:

For each spatial axis, deltas from opposing categories are sign-corrected so they point in a canonical direction:

Axis coherence is then the mean pairwise cosine similarity over the sign-corrected set:

The vertical-distance entanglement index compares perspective-aligned and perspective-opposing mean deltas:

Positive VD-EI means the hidden representation aligns above with far and below with close.

Experiments And Results

Real Benchmark Evidence

Table 1 shows that real-image benchmarks are dominated by perspective-consistent examples. This supports C2: a model can receive a lot of credit on aggregate depth performance while repeatedly failing the rarer counter cases.

Table 1. Distribution of consistent, counter, and ambiguous examples. The skew toward consistent examples mirrors natural perspective statistics.

Table 2 is the main real-image evidence for C1. Every listed model family has lower counter accuracy than consistent accuracy. The most striking example is Qwen2.5-VL-3B with 2M spatial samples: 60.9% consistent vs. 24.0% counter on EmbSpatial-Bench.

Table 2. Accuracy on consistent vs. counter examples. Values are accuracies from depth-related examples in EmbSpatial-Bench and CV-Bench-3D. The gap persists across architectures, scales, and fine-tuning levels.

SpatialTunnel Evidence

Table 3 confirms the same pattern in a controlled synthetic environment. Because SpatialTunnel balances the geometry and decouples vertical position from depth, the positive \(\Delta\) values are stronger evidence that the shortcut is model-intrinsic rather than merely a property of evaluation-set skew.

Table 3. Consistent vs. counter correctness on SpatialTunnel. \(v\) is mean correctness, \(v_{\text{cons}}\) is correctness on consistent samples, \(v_{\text{ctr}}\) is correctness on counter samples, and \(\Delta\) is their difference.

The Molmo heatmaps in Figure 4 and Figure 5 show the spatial pattern behind the aggregate scores. Consistent cells become easier with scaling, while counter cells remain harder and are especially degraded around intermediate fine-tuning steps.

Benchmark Accuracy Is Not Enough

Table 4 broadens the evaluation across EmbSpatial, CV-Bench, and BLINK. The pattern is not a clean "more spatial fine-tuning means better spatial understanding" story. For example, NVILA 2M reaches 93.8 on CV-3D Depth but only 62.9 on BLINK Spatial Relation, while Qwen 2M is 78.3 on BLINK Spatial Relation but 52.2 on CV-3D Distance.

Table 4. Performance across spatial benchmarks. Fine-tuned variants fluctuate across benchmark formats. RoboRefer and Qwen3-VL-235B are more consistently strong.

Representation-Level Evidence

Figure 7 links counter accuracy to distance coherence. The paper reports that \(\mathrm{Coh}_D\) computed on SpatialTunnel correlates with counter accuracy on EmbSpatial-Bench and CV-Bench-3D with Spearman \(\rho=0.759\) and \(\rho=0.804\), respectively, both with \(p<10^{-3}\). Figure 8 shows that within the NVILA family, RoboRefer occupies the desirable region of high distance coherence and lower VD-EI.

The PCA visualization in Figure 9 makes the same point visually. Molmo 2M, NVILA 2M, and Qwen 2M separate horizontal and vertical deltas better than distance deltas; RoboRefer and Qwen3 show much cleaner separation of all three axes.

Table 5. Axis coherence and VD-Entanglement Index. Distance coherence is the weakest axis across the reported models. RoboRefer has the highest coherence on all three axes and lower VD-EI than the NVILA scaling variants.

Additional Evidence From Source Appendices

The appendix extends the cue analysis from vertical position to object size. Figure 10 shows that Molmo and NVILA degrade when the farther object becomes larger than the nearer object, while Qwen remains near chance and therefore cannot be read as robust. This supports C7: high average depth accuracy can reflect multiple correlated visual cues, not just true 3D reasoning.

Figure 11 is useful as an externality check on the probing metric: the absolute \(\mathrm{Coh}_D\) values differ between synthetic SpatialTunnel and real EmbSpatial-Bench, but the relative ordering within model families is largely preserved.

Practical Takeaways

• The main reusable idea is the split between behavior and representation: benchmark accuracy says whether a model answered correctly, while contrastive probing asks whether it encoded spatial axes cleanly.
• The highest-value diagnostic is distance coherence. In this paper, \(\mathrm{Coh}_D\) is consistently the weakest axis and tracks counter-heuristic robustness better than aggregate accuracy alone.
• Spatial fine-tuning is not automatically enough. Some fine-tuned variants improve benchmark scores while maintaining or increasing shortcut gaps.
• RoboRefer and Qwen3-VL-235B are the strongest evidence that better spatial representations are possible, but the paper does not isolate a single causal ingredient; RoboRefer differs in both data scale and supervision, and Qwen3 differs heavily in model scale.
• The limitation to remember is that SpatialTunnel is synthetic and diagnostic. It isolates one failure mode well, but it is not a full proxy for embodied 3D competence.
• For future model evaluation, the paper argues for reporting counter-heuristic splits and representation structure, not only standard benchmark averages.