Vision

Language

World

Model

Vlwm

Vision Language World Model — VLWM: A Deep Dive into the Language-Based World Model for High-Level Planning

https://arxiv.org/html/2509.02722v1

Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the Vision Language World Model (VLWM), a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions. The VLWM learns both an action policy and a dynamics model, which respectively facilitates reactive system-1 plan decoding and reflective system-2 planning via cost minimization. The cost evaluates the semantic distance between the hypothetical future states given by VLWM roll-outs and the expected goal state, and is measured by a critic model that we trained in a self-supervised manner. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations, where system-2 improves the Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmark.

Figure 2:Overview of VLWM. (a) VLWM is a JEPA-style world model that predict abstract representation of future world states, instead of generating noisy and high-volume raw observations. (b) Given video contexts, VLWM’s prediction target is a structured textual representation of the unobserved future. It includes goal and interleaved action (A) world state changes (Δ​S), all extracted automatically. © VLWM can infer possible goals from the context, and interpret them with current initial state and the expected final state. It supports both fast reactive system-1 plan generation and reflective system-2 reasoning based on cost minimization.

Figure 3:System-2 planning of VLWM. (a): the critic is trained in a self-supervised manner, assigning lower cost to valid progress, while assigning higher cost for adding irrelevant distractors or shuffling the steps. (b): VLWM generates candidate action sequences and simulates their future state transitions. A critic evaluates the resulting state trajectories given the goal, and the planner selects the lowest-cost plan.

This paper introduces the Vision Language World Model (VLWM), a new foundation model for high-level planning that uses natural language to represent and predict world dynamics. Here are the core insights from the paper:

Language as a World Model: The VLWM is a novel approach that uses language as an abstract representation of the world, rather than predicting raw pixels. This makes the model more efficient, interpretable, and allows for better integration with the reasoning capabilities of large language models (LLMs).

Two-Stage Data Processing: To train the VLWM, the paper proposes a two-stage data processing pipeline. First, it compresses videos into a “Tree of Captions” to create a hierarchical summary. Then, it uses an LLM with a “Self-Refine” process to extract structured goal-plan descriptions from these captions. This method efficiently creates high-quality training data from large-scale video datasets.

Dual Planning Systems: The VLWM features two modes for planning:

• System-1 (Reactive Planning): A fast, direct approach that generates a plan as a text sequence. This is suitable for simple, short-horizon tasks.
• System-2 (Reflective Planning): A more sophisticated “planning with reasoning” mode that generates multiple possible plans and uses a “critic” model to evaluate them. The critic, trained through self-supervision, selects the optimal plan by minimizing the cost, which represents the semantic distance to the goal. This allows for a form of internal trial-and-error.

State-of-the-Art Performance: The VLWM achieves state-of-the-art results on several benchmarks. It shows significant improvements in Visual Planning for Assistance (VPA) and outperforms strong baselines in RoboVQA.

Human-Preferred Plans: The authors introduced “PlannerArena,” a human evaluation framework, to assess the quality of generated plans. In these evaluations, VLWM’s System-2 generated plans were preferred over those from other leading models and even over the ground-truth plans from existing datasets, which highlights the shortcomings of current benchmarks.

Self-Supervised Critic for Reasoning: A key component of the System-2 planning is the critic model, which is trained in a self-supervised manner to differentiate between good and bad plans. The critic’s effectiveness is also demonstrated independently in tasks like goal achievement detection.