MICCAI 2026

Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision–Language Models

Mashrafi Monon¹ Umaima Rahman^1,2 Asif Hanif¹ Numan Saeed¹ Mohammad Yaqub¹

¹ Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE ² New York University Abu Dhabi, Abu Dhabi, UAE

Paper 🤗 Dataset Code

Dataset Scale

9,077

QA Pairs

1,601

CT Volumes

Spatial Categories

VLMs Benchmarked

Model Performance

43.69%

Best Accuracy (CT-Chat)

34%

Avg. Accuracy (8 VLMs)

Abstract

Do 3D medical VLMs truly understand volumetric space — or do they rely on language priors? We introduce CT-SpatialVQA, a benchmark of 9,077 clinically grounded QA pairs from 1,601 CT volumes that demands explicit spatial reasoning. Eight state-of-the-art models average only 34% accuracy, often below random chance, revealing a critical gap between linguistic fluency and volumetric spatial understanding.

CT-SpatialVQA Benchmark

9,077 spatially grounded QA pairs from 1,601 CT volumes with 95% human consensus

Spatially-Grounded QA Design

LLM-based generation, independent validation, and medical human auditing across 6 spatial dimensions

Comprehensive Evaluation

8 medical VLMs benchmarked with LLM-as-Jury and text metrics, revealing consistent failure modes

Method

Example CT scan with radiology report and QA pairs

Example. A CT scan, its paired radiology report, and representative spatially-grounded QA pairs.

Model prediction failures on spatial reasoning

Spatial reasoning failures. All three models answer incorrectly on spatially grounded questions despite varying question types.

Dataset

🤗 Access on HuggingFace

A single CT volume shown across all three anatomical planes:

Axial

Coronal

Sagittal

Six Spatial Reasoning Categories

Laterality & Bilateral Symmetry

Is the consolidation area in the right or left lung?

In the right lung.

Longitudinal (Vertical) Position

Is the pleural effusion located in the upper or lower thorax?

In the lower thorax, predominantly basal.

Anterior-Posterior (Depth) Relations

Where is the cystic lesion relative to the ascending aorta?

In the anterior neighborhood of the ascending aorta.

Medial-Lateral Orientation (Centricity)

Is the hilar mass centrally or peripherally located?

Centrally located at the hilum.

Adjacency & Containment

Where are the space-occupying lesions relative to the ribs?

Anteriorly at the sternal junctions of the 2nd and 3rd ribs.

Spatial Extent & Boundaries

Does the ground-glass opacity cross the lobar fissure?

Yes, it extends across the fissure into the adjacent lobe.

Results

Metric	CT-Chat	MERLIN	Med3DVLM	M3D	RadFM	VILA-M3	MedGemma	MedEvalKit	Avg.
LLM-based Evaluation (Accuracy %)
LLM as Judge (Gemini)	44.68	29.85	34.55	33.04	33.36	29.49	39.30	39.96	35.53
LLM as Judge (GPT)	41.50	29.45	31.82	31.09	31.40	28.43	40.55	39.44	34.21
LLM as Judge (Qwen)	45.27	30.34	32.20	31.21	32.09	27.81	37.33	41.89	34.79
LLM as Jury	43.69	28.85	31.44	30.57	31.49	28.20	38.24	40.16	34.08
Text Similarity Metrics
SBERT Cosine Sim.	0.562	0.503	0.399	0.412	0.585	0.471	0.581	0.579	0.512
BLEU	3.29	8.11	0.56	1.08	16.45	2.51	3.07	3.46	4.82
ROUGE-L	0.130	0.300	0.198	0.204	0.372	0.273	0.167	0.158	0.225
METEOR	0.230	0.256	0.068	0.077	0.311	0.114	0.237	0.252	0.193

Category-level performance (LLM-as-Jury). Sharpest drops in vertical and depth-oriented reasoning.

Answer length distribution. Questions, ground-truth answers, and model predictions across all 8 VLMs.

Citation

@article{monon2026ctspatialvqa, title = {Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision--Language Models}, author = {Monon, Mashrafi and Rahman, Umaima and Hanif, Asif and Saeed, Numan and Yaqub, Mohammad}, journal = {arXiv preprint arXiv:2605.08787}, year = {2026} }