MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

Evaluating the impact of reasoning complexity across all 37 models reveals a consistent performance degradation as the number of hops increases, confirming that compositional spatial reasoning remains a fundamental challenge for current VLMs. Both MCQ accuracy and Acc@50IoU exhibit steep declines from 1-hop to 3-hop.

Crucially, this degradation is exacerbated under ego-centric evaluation, which requires additional perspective transformation. The widening performance gap between ego-centric and exo-centric views at higher hops suggests that perspective-taking compounds with multi-step reasoning, creating a multiplicative rather than additive difficulty.

Average performance by hop count. Performance degrades consistently as reasoning hops increase.

Beyond a simple performance drop, ego-centric evaluation fundamentally alters capability visibility. Under exo-centric conditions, our Acc@50IoU metric clearly distinguishes grounding-capable models (e.g., Qwen3-VL) from those lacking native localization (e.g., InternVL-3.5) despite their similar MCQ accuracies.

Conversely, ego-centric evaluation acts as an evaluation blind spot: it suppresses even strong grounding models to a 20–25% floor, completely masking these capability gaps. This compression highlights the necessity of evaluating perspectives jointly and reinforces Acc@50IoU as an essential metric to expose disparities invisible to conventional MCQ evaluation.

Model performance scatter plots. Ego-centric evaluation masks grounding differences visible in exo-centric conditions.

While reasoning models consistently outperform instant models, this advantage diminishes under multi-hop pressure. The initial MCQ gain of up to +8 pp at 1-hop narrows considerably by 3-hop.

Critically, even with extended thinking, reasoning models plummet to sub-20% MCQ accuracy and sub-10% Acc@50IoU on 3-hop ego-centric tasks. This convergence toward the performance floor reveals that inference-time reasoning (e.g., chain-of-thought) yields diminishing returns as compositional steps accumulate. MultihopSpatial thus exposes a capability ceiling that test-time compute alone cannot resolve.

Reasoning vs. Instant models. The reasoning advantage shrinks with increasing hop count.

While proprietary models hold a slight MCQ advantage in ego-centric tasks—indicating stronger perspective-taking—open-weight models consistently dominate in Acc@50IoU. This reversal stems from a distinct asymmetry in visual grounding capabilities.

Proprietary models exhibit high variance: while the Gemini-3 series demonstrates robust localization, others (e.g., GPT-5.2, Claude) lack native grounding, significantly dragging down the group average. Conversely, open-weight families (e.g., Qwen3-VL, GLM-4.6V) maintain consistently high grounding performance, forming a dense cluster at 60%+ in exo-centric Acc@50IoU.

Open-weight vs. Proprietary models across hop count and viewpoint.

Comparing specialized spatial reasoning models (SRMs) against comparable general-purpose VLMs (≤10B) reveals a counterintuitive trend: SRMs consistently underperform across all metrics and hop counts.

This shortfall primarily stems from the robust native grounding inherent in general models (e.g., Qwen3-VL), which most SRMs lack. These findings indicate that current SRMs, typically fine-tuned on single-step QA, fail to generalize to multi-hop scenarios, suggesting that general-purpose VLMs provide a stronger foundation for compositional spatial understanding.

Spatial Reasoning Models vs. General Models (≤10B).

A striking disconnect exists between answer selection and spatial localization: most models fall far below the y=x line, indicating that high MCQ accuracy does not entail accurate grounding. Across all 37 models, the average ungrounded ratio reaches 59%, meaning more than half of correctly answered questions lack proper spatial localization.

This ratio varies dramatically by category: proprietary instant models exhibit 93% ungrounded accuracy, while open-weight reasoning models achieve the lowest at 43%. At the extreme, models such as Gemma-3-IT, Molmo2, and Claude-Sonnet-4.5 exceed 98% ungrounded, effectively answering through shortcuts without any spatial understanding.

Grounded (Acc@50IoU) vs. Ungrounded accuracy per model. Hatched regions indicate ungrounded responses.

Grounding Gap: MCQ Acc. vs. Acc@50IoU.

Ungrounded Ratio by Model Category.

Analyzing model scale across three open-weight families reveals that scaling the language backbone alone yields limited gains. While MCQ accuracy shows modest, saturating improvements, Acc@50IoU largely plateaus (e.g., Qwen3-VL) or remains near zero (e.g., Gemma-3-IT).

A notable exception is InternVL-3.5 at 38B, which exhibits a sharp grounding jump from 5.2% to 27.4%. Crucially, this leap coincides with a significant upgrade in its vision encoder (from 300M to 6B), whereas models utilizing fixed, small vision encoders (e.g., Qwen3-VL’s 400M) saturate early. This demonstrates that multi-hop spatial reasoning depends critically on scaling the capacity of visual spatial representations, not solely the language component.

Effect of model scale on MCQ Accuracy (top) and Acc@50IoU (bottom) across overall and per-hop settings.

Analyzing error rates across spatial tag combinations reveals that while reasoning models consistently outperform instant models, multi-tag compositions remain a critical bottleneck. Specifically, the Position-Relation (P-R) combination incurs significantly higher errors than single-tag settings, highlighting the severe difficulty of jointly handling positional localization and relational comparison.

Even specialized spatial reasoning models, which achieve the lowest overall errors, exhibit non-trivial failures on these complex compositions. This underscores that true compositional spatial reasoning remains an unresolved challenge across the current VLM ecosystem.

Average error rates (%) by tag combination across model categories. A, R, and P denote Attribute, Relation, and Position.