Planning with the Views

This could be an interesting paper by Li Fei Fei and Leonidas Guibas

From the abstract:

"Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We call this capability view planning, requiring 

(1)understanding how a single action transforms the view, and 

(2) composing many such transformations across multi-turn plans to identify a target view.

We probe both abilities in our proposed ViewSuite, a 3D point-cloud environment on real ScanNet scenes.

Across 13 frontier VLMs, a critical planning gap emerges: they possess basic view-action knowledge but fail to compose it across multi-turn plans, with the gap widening as viewpoint distance grows.

To close this gap, we propose an iterative framework that alternates self-exploration with view graph distillation.

The key insight is that all exploration trajectories, regardless of their outcome, collectively form a view graph that compactly captures how viewpoints connect across a scene.

Distilling this graph into diverse supervised tasks reshapes the policy distribution and overcomes the sparse rewards that stall pure RL.

This improves Qwen2.5-VL-7B from 2.5% to 47.8% on interactive view planning, surpassing GPT-5.4 Pro (18.5%) and Gemini 3.1 Pro (21.4%). Self-exploration emerges as a promising path toward VLMs that can actively reason and plan in 3D space. Code and Data are at this https URL."

[2605.29563] Planning with the Views