World Models: Interview Questions and Answers

World Model Fundamentals

Q1 [Basic] What is a world model and how does it differ from a model-free agent?

Q: How is a world model defined, and what distinguishes a model-based agent that uses one from a model-free agent?

A: A world model is an internal representation that an agent uses to simulate the dynamics of its environment: given a current state and an action, it predicts the next state and the associated reward. The term originates from cognitive science (Craik, 1943) but was formalized in the reinforcement learning context by Ha & Schmidhuber (2018). The key distinction from model-free agents is that a world model enables counterfactual reasoning — the agent can imagine the consequences of actions without executing them in the real environment, allowing planning in “imagination” rather than requiring every policy update to come from costly real interaction.

The architecture of a world model typically decomposes into three components: a representation model (encoder) that maps high-dimensional observations to compact latent states, a transition model that predicts the next latent state given an action, and an optional decoder that reconstructs observations from latent states. The policy and value function then operate on latent states rather than raw observations, drastically reducing the dimensionality of the input to the decision-making component.

Model-based agents are substantially more sample-efficient than model-free agents because each real transition serves double duty: the world model extracts dynamics information from it, and can subsequently generate many synthetic transitions for additional policy training. The fundamental trade-off is that world model errors compound when planning over multiple steps — a model that is 1% inaccurate per step produces a 14% error after 15 steps — and inaccurate models in rarely-visited regions of state space can mislead the planner into exploiting model errors rather than achieving genuine task performance.

Q2 [Basic] What is the Ha & Schmidhuber World Models architecture?

Q: Describe the Visual, Memory, and Controller decomposition in Ha & Schmidhuber (2018) and explain the significance of training a policy inside a learned world model.

A: Ha & Schmidhuber (2018) proposed the first end-to-end learned world model architecture, decomposing the agent into three components. The Visual model (V) is a Variational Autoencoder (VAE) that compresses each raw observation frame into a compact latent vector $z_t$, discarding irrelevant visual details and retaining a low-dimensional representation of the scene. The Memory model (M) is a Mixture Density Network-Recurrent Neural Network (MDN-RNN) that models temporal dynamics in latent space: given $z_t$ and action $a_t$, it predicts a probability distribution over the next latent state $z_{t+1}$ as a mixture of Gaussians, capturing the inherent stochasticity of transitions. The Controller (C) is a simple linear mapping from the concatenation of $z_t$ and the RNN hidden state $h_t$ to actions, trained by CMA-ES (Covariance Matrix Adaptation Evolution Strategy) rather than gradient descent.

The key experiment demonstrated that the Controller could be trained entirely inside the learned world model — the policy was optimized on imagined rollouts generated by V and M, then deployed in the real environment without further adaptation. On CarRacing-v0 and ViZDoom, this “dream training” approach achieved competitive performance while dramatically reducing the number of required real environment interactions.

The architecture revealed two important limitations that motivated subsequent work. First, the VAE’s pixel reconstruction objective may discard task-relevant features that do not aid reconstruction but are critical for reward — the encoder optimizes for visual fidelity, not for preserving information relevant to action selection. Second, the MDN-RNN’s mixture-of-Gaussian transitions do not scale to complex, highly multi-modal dynamics, and the controller’s linear form limits the expressiveness of the policy.

Q3 [Basic] How does model-based planning in imagination work?

Q: Describe how a world model enables planning through imagined rollouts, and what are the two main planning paradigms?

A: Planning with a world model proceeds by performing imagined rollouts: starting from the current latent state, the agent iteratively applies the transition model to simulate future states and uses the reward model to estimate returns, entirely within the latent space. The action sequence that maximizes the expected imagined return is then selected, bypassing the need for real environment interaction during planning.

Two main planning paradigms exist. Explicit planning (model predictive control, MPC) generates candidate action sequences, simulates each forward using the world model, evaluates expected returns, and executes the first action of the best sequence — then replans at every step. The Cross-Entropy Method (CEM) and MPPI (Model Predictive Path Integral) are common samplers used in this paradigm. This approach makes no assumptions about the policy form and can incorporate uncertainty estimates from ensemble models, but it is computationally expensive (many forward passes per decision step) and does not amortize the planning computation across similar states.

Amortized planning (Dreamer-style) trains a parametric policy using backpropagation through the world model’s differentiable transition function. The actor’s gradient is computed analytically through imagined multi-step trajectories rather than estimated from Monte Carlo policy gradient samples, yielding a dense, low-variance learning signal at every imagined time step. The computational cost of planning is paid once during training and then recovered as fast policy inference at test time. The critical requirement is that the transition model be differentiable with respect to the action input, which holds for continuous state-action spaces but requires special handling (e.g., straight-through gradient estimators) for discrete latent states.

Q4 [Advanced] What is the Dyna architecture and how does it relate to modern world models?

Q: Describe the Dyna framework, its core design choices, and trace the conceptual lineage from Dyna to modern neural world model approaches.

A: Dyna (Sutton, 1991) is the foundational model-based RL framework that introduced the principle of using a learned environment model to generate synthetic experience for policy improvement. The Dyna agent alternates between two phases: real interaction (collecting transitions, storing them in a replay buffer, and updating the model), and simulated experience generation (sampling from the model and updating the policy/value function as if those samples were real transitions). This amortizes the cost of real interaction by multiplying each real experience into multiple model-generated transitions, improving sample efficiency without altering the policy update mechanism.

The critical design parameter is the ratio of real to simulated transitions — the “planning ratio” $k$. With a perfect model, increasing $k$ monotonically improves policy quality; with an imperfect model, high $k$ amplifies model errors and degrades performance. This bias-variance trade-off in Dyna motivated the development of uncertainty-aware planning (MBPO, MOPO, MOREL), which limits imagination rollout length or pessimistically penalizes high-uncertainty model regions to prevent model exploitation.

Modern neural world models (Dreamer, MuZero, TD-MPC) are best understood as high-capacity differentiable extensions of Dyna: the tabular model is replaced by a neural network with a continuous latent state space, the tabular policy update is replaced by gradient-based actor-critic optimization, and planning occurs in learned latent space rather than in symbolic or tabular state space. The Dyna framework also anticipates the offline model-based RL literature: if the replay buffer is a static fixed dataset (no new real interaction), Dyna becomes offline MBRL, with model accuracy on the data distribution being the limiting factor for policy quality. TD-MPC2 (Hansen et al., 2024) extends this lineage with a value-equivalent latent model and joint model-policy training, achieving strong performance across locomotion, manipulation, and game tasks simultaneously.

Latent Dynamics Modeling

Q5 [Basic] What role does the latent space play in a world model?

Q: Why do world models operate in a latent space rather than in observation space, and what properties make a latent space useful for planning?

A: The latent space in a world model serves as a compressed, abstract representation of the environment state that retains the information necessary for predicting future states and rewards, while eliminating the computational expense of operating in raw observation space. By learning dynamics in a compact latent space rather than in pixel space, the world model avoids predicting irrelevant visual details (background textures, lighting variation) that consume model capacity without influencing task performance, and enables planning at a rate several orders of magnitude faster than pixel-level generation.

A useful latent space for world model planning should satisfy three properties, which are often in tension. Sufficiency: the latent state should retain all task-relevant information from the observation, so that reward and the consequences of actions can be predicted accurately. Predictability: the latent transition dynamics should be smooth and tractable to learn — abrupt, discontinuous transitions in latent space make the transition model inaccurate. Compactness: the space should be low-dimensional enough for efficient planning — high-dimensional latent spaces increase the complexity of the transition model and reduce the effectiveness of gradient-based planners.

These properties can be enforced by different learning objectives. VAE-based reconstruction (RSSM in Dreamer) rewards sufficiency (the decoder must recover the observation from the latent state) but may reward visual richness over task relevance. Contrastive objectives (SimCLR, CURL applied to RL) reward predictability and compactness but do not guarantee that task-relevant information is preserved. Value-equivalence objectives (MuZero) require only that the latent space accurately predict reward and value, achieving the strongest form of task relevance but making the representation uninterpretable and tied to a specific reward signal.

Q6 [Advanced] How does the Recurrent State Space Model work and why does it combine deterministic and stochastic components?

Q: Describe the RSSM’s architectural decomposition, explain the purpose of each component, and detail how the model is trained and used for imagination.

A: The Recurrent State Space Model (RSSM), introduced in PlaNet (Hafner et al., 2019), is the core latent dynamics component shared by PlaNet and the entire Dreamer series. It decomposes the latent state into two components with complementary roles. The deterministic recurrent state $h_t$ is computed by a GRU from $h_{t-1}$ and action $a_{t-1}$: $h_t = f_\phi(h_{t-1}, z_{t-1}, a_{t-1})$. This component carries long-range temporal information across many time steps without stochastic bottlenecks — the GRU’s hidden state accumulates a compressed history of past observations and actions, enabling the model to infer hidden state variables (e.g., velocity from successive positions, game mode from recent events). The stochastic state $z_t$ is sampled from a distribution conditioned on $h_t$: during training, the posterior $q_\phi(z_t | h_t, o_t)$ incorporates the current observation; during imagination (no observation access), the prior $p_\phi(z_t | h_t)$ generates the stochastic state from the recurrent history alone.

The stochastic component models irreducible uncertainty: even with perfect knowledge of history and action, the next state may not be fully determined (physical noise, multi-modal outcomes, partial observability). By explicitly representing this uncertainty as a learned distribution, the RSSM can generate diverse imagined trajectories that reflect the range of possible futures rather than a single deterministic prediction.

The model is trained end-to-end via the ELBO: $\mathcal{L} = \mathbb{E}_{q}[\log p(o_t | h_t, z_t)] - \beta \cdot D_{\text{KL}}(q_\phi(z_t | h_t, o_t) \| p_\phi(z_t | h_t))$. The reconstruction term trains the decoder, and the KL term trains the prior to predict the posterior — this is the imagination training signal. At inference, only the prior is used to generate $z_t$ from $h_t$, enabling multi-step imagined rollouts without accessing real observations. The separation between $h_t$ (deterministic, trained by BPTT through the GRU) and $z_t$ (stochastic, trained by variational inference) is crucial: it prevents the KL training from disrupting the temporal credit assignment through the recurrent state.

Q7 [Advanced] How do world models handle partial observability?

Q: What is partial observability, why does it pose a challenge for world models, and how does the RSSM’s recurrent state address it?

A: Partial observability occurs when the observation $o_t$ does not fully determine the true underlying state $s_t$ of the environment — sensors have limited range, objects are occluded, and relevant state variables (e.g., the velocity of an off-screen object, the current game phase encoded implicitly in score history) are not directly visible. A world model that treats each observation independently as a sufficient state representation will fail systematically in partially observable settings: its predictions will be inaccurate whenever the current observation is ambiguous about the true state.

The correct treatment requires maintaining a belief state — a distribution over possible underlying states given the full observation history $o_{1:t}$ — rather than conditioning on only $o_t$. In the POMDP (Partially Observable Markov Decision Process) formalism, the belief state $b_t = p(s_t | o_{1:t}, a_{1:t-1})$ is the minimal sufficient statistic for predicting future observations and rewards. Computing the exact belief state is intractable in continuous spaces, requiring approximate inference.

The RSSM implements approximate belief state tracking via the recurrent deterministic state $h_t$: by processing the sequence of observations through a GRU, $h_t$ accumulates a compressed summary of observation history that serves as an approximate belief state. The model can learn to integrate multiple past observations to infer hidden variables — for example, on Flickering Atari variants (where observations are randomly masked with probability 0.5), Hafner et al. (2019) demonstrate that the RSSM’s recurrent architecture substantially outperforms feedforward models that condition on single observations, because the recurrent model can integrate information across unmasked frames to reconstruct the true game state. This recurrent belief integration is equivalent to learned variational filtering in a POMDP, replacing exact Bayesian inference with a deterministic function learned by backpropagation through time.

Q8 [Advanced] What are the trade-offs between pixel-space and latent-space prediction in world models?

Q: Compare pixel-level prediction and latent-state prediction as world model approaches, including value-equivalent models, and explain when each is preferable.

A: The choice of prediction target — raw observation pixels, a learned continuous latent state, discrete tokens, or value-relevant quantities only — fundamentally shapes a world model’s capability, computational cost, and failure modes.

Pixel-space prediction (autoregressive video generation as in IRIS, GameGAN) produces human-interpretable imagined trajectories and provides a natural auxiliary loss that prevents the representation from collapsing or discarding perceptual information. However, predicting a 64×64 RGB frame requires generating 12,288 output values per step, most of which (background texture, irrelevant object details) carry no task-relevant information and waste model capacity. Autoregressive pixel generation compounds errors: a mistake in an early pixel token biases the distribution of all subsequent tokens. Despite these costs, pixel-level models are useful when interpretability matters and when downstream reward prediction requires fine-grained visual details.

Continuous latent-space prediction (RSSM in Dreamer, SLAC) operates in compact representations ($\leq 1024$ values per state in DreamerV2), enabling fast imagination rollouts and differentiable planning. The representation is more abstract — the model predicts in a space where dynamics are learned to be smooth and predictable — but it depends on the encoder being high quality. If the encoder discards task-relevant information (the classic “VAE learns to encode style over structure” failure mode), the transition model learns dynamics over an uninformative latent space. Latent-space models are preferred for sample efficiency: DreamerV3 achieves competitive performance on diverse benchmarks while requiring far fewer model parameters per imagination step than pixel-level counterparts.

Value-equivalent prediction (MuZero) predicts neither pixels nor a general-purpose latent state, but specifically reward, value, and policy logits — the minimal quantities needed for MCTS-based planning. This maximally compresses the representation toward decision-relevant information and eliminates reconstruction entirely, but makes the latent space opaque and requires a well-defined reward signal for training. MuZero is strictly preferable when planning quality is the only objective and observation reconstruction is unnecessary; it is unsuitable for tasks requiring world model interpretability or downstream transfer to new reward functions.

PlaNet and Dreamer Series

Q9 [Basic] What is PlaNet and how does it plan in latent space?

Q: Describe PlaNet’s RSSM-based world model, its Cross-Entropy Method planning approach, and how it differs from DreamerV1’s amortized actor-critic.

A: PlaNet (Hafner et al., 2019, “Learning Latent Dynamics for Planning from Pixels”) introduced the RSSM and demonstrated for the first time that an agent can solve continuous control tasks from raw pixel observations by planning entirely within a learned latent state space — without ever planning in pixel space or training an explicit policy network.

PlaNet’s planning uses Model-Predictive Control (MPC) with the Cross-Entropy Method (CEM): at every environment step, CEM samples $J$ candidate action sequences of length $H$, evaluates each by imagining the corresponding RSSM rollout and summing predicted rewards, selects the top-$K$ sequences by predicted return, refits a Gaussian distribution over action sequences to these elite samples, and repeats for several iterations. The first action of the converged best sequence is executed; the entire planning procedure runs again at the next step. This approach requires no explicit policy network: decision-making emerges from test-time optimization guided by the world model.

The RSSM is trained purely from pixel observations via the ELBO: the encoder maps observations to posterior latent states, the transition model predicts the prior, and an image decoder and reward predictor are jointly trained with the KL divergence regularization term. At planning time, only the transition model and reward predictor are used — the decoder is not queried.

On six continuous control tasks from the DeepMind Control Suite, PlaNet achieves competitive performance with only 2,000 environment episodes, matching or exceeding model-free baselines that require 50× more data. The key limitation is that CEM planning is expensive at test time: evaluating $J$ candidate sequences per step scales poorly to longer horizons and complex tasks. DreamerV1 directly addresses this by replacing CEM with a parametric actor network trained via backpropagation through imagined trajectories — moving the planning compute from test time to training time.

Q10 [Basic] What is DreamerV1 and how does it train a policy entirely in imagination?

Q: Describe DreamerV1’s training procedure and the evidence that planning in imagination can match or exceed model-free performance on continuous control tasks.

A: DreamerV1 (Hafner et al., 2020, “Dream to Control: Learning Behaviors by Latent Imagination”) introduced the first world model capable of learning from raw pixel observations and training a competitive policy solely through imagined trajectories. The training alternates between three phases. First, the RSSM, image decoder, reward predictor, and discount predictor are jointly trained on real environment transitions stored in a replay buffer, minimizing the ELBO loss. Second, imagined rollouts of length $H = 15$ are generated by unrolling the RSSM prior from latent states stored in the replay buffer — no new real environment interaction is used during this phase. Third, the actor (policy) and critic (value function) are trained purely on these imagined trajectories using straight-through gradients for the actor and $\lambda$-returns for the critic.

The actor is trained to maximize the $\lambda$-return of imagined trajectories via reparameterized gradients through the differentiable transition model: $\nabla_\phi \mathbb{E}_{p_\phi}[V_\lambda(z_{1:H})]$, where the expectation is over imagined trajectories generated by the policy acting in the learned model. This provides dense gradient signal at every imagined step, in contrast to model-free policy gradient which obtains sparse gradients from sparse real rewards. The critic is updated via temporal difference learning on imagined rollouts, bootstrapping value estimates at the end of each horizon.

On the DeepMind Control Suite (DMControl) across 20 continuous control tasks, DreamerV1 matches or exceeds state-of-the-art model-free agents (SAC, D4PG) in asymptotic performance while using 5× fewer real environment steps. This established imagined rollouts as a viable and efficient training paradigm and initiated the Dreamer line of research.

Q11 [Advanced] What improvements did DreamerV2 introduce over DreamerV1?

Q: Describe DreamerV2’s categorical latent representations and KL balancing, explain the motivation for each, and summarize its performance on Atari.

A: DreamerV2 (Hafner et al., 2020, “Mastering Atari with Discrete World Models”) extended Dreamer to the Atari benchmark suite — discrete action, image-based games with complex, non-smooth dynamics. Two principal innovations addressed known weaknesses of the continuous Gaussian latent representation in DreamerV1.

Categorical latent representations replaced the Gaussian stochastic state $z_t \sim \mathcal{N}(\mu, \sigma^2)$ with a product of $K$ categorical distributions, each with $L$ classes: $z_t = \text{concat}[\text{cat}(\pi_1), \ldots, \text{cat}(\pi_K)]$, where $K = 32$, $L = 32$, giving a $32 \times 32 = 1024$-dimensional binary vector. Straight-through gradients enable backpropagation through the discrete argmax sampling. Categorical latents are better suited to modeling discrete structure in environments: where the world has a finite number of semantically distinct states (game modes, discrete collision events, binary object presence), categorical distributions represent them as point masses rather than overlapping Gaussian blobs. The authors found that categorical latents reduce the reconstruction loss faster and produce better-calibrated uncertainty estimates than Gaussians on Atari.

KL balancing addressed training instability arising from the KL divergence term in the ELBO. With a high-capacity encoder, the posterior $q(z_t | h_t, o_t)$ can learn much faster than the prior $p(z_t | h_t)$, causing the prior to perpetually lag behind and failing to train the model’s imagination capability. KL balancing applies asymmetric gradient coefficients: the prior receives gradient weight $\alpha = 0.8$ and the posterior receives $1 - \alpha = 0.2$, ensuring the prior updates faster than the posterior and learns to predict the posterior’s distribution accurately. On 55 Atari games at 200M environment steps, DreamerV2 achieves human-level performance on 45 games and matches the data efficiency of model-free Rainbow, establishing for the first time that a model-based agent can compete with top model-free methods on Atari.

Q12 [Advanced] How does DreamerV3 achieve general-purpose world modeling across diverse domains?

Q: Describe the three technical contributions of DreamerV3 that enable domain-agnostic hyperparameters, and summarize its cross-domain performance.

A: DreamerV3 (Hafner et al., 2023, “Mastering Diverse Domains with World Models”) addressed the central limitation of prior Dreamer variants: every benchmark domain required separate hyperparameter tuning, limiting the framework’s claim to generality. The goal was a single fixed hyperparameter configuration that works across tasks spanning seven orders of magnitude in reward scale — from proprioceptive robot control (rewards in $[-1, 1]$) to Minecraft diamond collection (rewards in $[0, 1024]$).

Symlog predictions apply a symmetric log transformation to all predicted quantities (reward, value, return targets): $\text{symlog}(x) = \text{sign}(x) \cdot \ln(|x| + 1)$. This compresses large reward scales and expands small ones uniformly, so that prediction error in reward space is commensurate regardless of domain. The inverse transformation $\text{symexp}$ recovers the original scale for action selection. Symlog eliminates the need for reward normalization hyperparameters entirely.

Free bits impose a minimum threshold $B$ on the per-dimension KL divergence: $\mathcal{L}_{\text{KL}} = \max(B, D_{\text{KL}}(q \| p))$. Without this, in early training when the prior is uninformative and the posterior encodes little information, the KL term is trivially satisfied (both are nearly uniform) and the model receives no gradient to improve the prior’s imagination capability. Setting $B = 1$ nat ensures the model is always pushed to improve its prior, preventing posterior collapse and stabilizing early training across diverse task difficulties.

Percentile-based return normalization scales the actor’s gradient by the 5th–95th percentile range of returns observed in the current imagination batch: $\hat{g} = g / \max(1, S_{95} - S_5)$. This makes gradient magnitude domain-invariant: tasks with large return variance (Minecraft) and tasks with small return variance (continuous control) produce comparably-scaled actor updates. On benchmarks spanning DMControl, Atari, Crafter, BSuite, Minecraft, and memory tasks, DreamerV3 with fixed hyperparameters achieves state-of-the-art or competitive performance across all domains simultaneously — including the first reinforcement learning agent to collect a diamond in Minecraft from scratch without human demonstrations.

Q13 [Advanced] How is the actor-critic trained in Dreamer’s latent space?

Q: Describe the mechanics of Dreamer’s imagination-based actor-critic, including gradient flow, return estimation, and the advantage of this approach over model-free policy gradient.

A: Dreamer’s actor-critic framework operates entirely within the world model’s latent space, never accessing real observations during policy optimization. The actor $\pi_\phi(a_t | z_t)$ maps latent states to action distributions; the critic $V_\psi(z_t)$ estimates discounted future returns from latent states. Both are updated using synthetic experience from imagined $H$-step trajectories.

Imagined rollouts begin from latent states $z_t$ sampled from the replay buffer (not from random initialization), ensuring the trajectories start from state-space regions with real data support. At each imagination step, the actor samples an action, the RSSM deterministic update computes $h_{t+1} = f_\phi(h_t, z_t, a_t)$, and the prior samples $z_{t+1} \sim p_\phi(z_{t+1} | h_{t+1})$. The reward predictor $\hat{r}_{t+1} = r_\psi(h_{t+1}, z_{t+1})$ provides a reward estimate, and the discount predictor $\hat{\gamma}_{t+1} = \gamma_\psi(h_{t+1}, z_{t+1})$ predicts episode termination probability.

The actor is trained via straight-through reparameterized gradients through the $\lambda$-return: $\mathcal{L}_\phi = -\mathbb{E}_{p_\phi, \pi_\phi}[V_\lambda(z_{1:H})]$, where $V_\lambda(z_t) = \hat{r}_t + \hat{\gamma}_t [(1 - \lambda) V_\psi(z_{t+1}) + \lambda V_\lambda(z_{t+1})]$ recursively combines bootstrapped value estimates with imagined rewards. For discrete latent states (DreamerV2/V3), straight-through gradients approximate $\nabla_\phi \mathbb{E}[\cdot]$ by passing gradients through the argmax as if it were an identity. The critic is updated via temporal difference with the same $\lambda$-returns as targets: $\mathcal{L}_\psi = \mathbb{E}[(V_\psi(z_t) - \text{sg}(V_\lambda(z_t)))^2]$, where $\text{sg}$ is stop-gradient.

The advantage over model-free policy gradient (REINFORCE, PPO) is the gradient quality: every imagined action receives an analytical gradient through the differentiable transition and reward models, yielding dense credit assignment across the entire $H$-step imagined trajectory. Model-free methods estimate this gradient from stochastic reward samples, producing high-variance estimates that require large batch sizes and many environment interactions to overcome.

Q14 [Advanced] When does world-model planning outperform model-free baselines?

Q: Under what task conditions do world models provide the largest performance advantage over model-free agents, and what conditions lead to world models underperforming?

A: World models provide the largest advantage over model-free baselines in tasks where two conditions hold simultaneously: real environment interaction is expensive (the cost of collecting data is high relative to the cost of model queries) and transition dynamics are learnable (the environment follows patterns that a neural network can accurately approximate from limited data). On the DMControl Suite, where physics are smooth, deterministic, and continuous, DreamerV3 achieves equivalent asymptotic performance to model-free SAC while using 5–20× fewer environment steps. In robotics manipulation and locomotion tasks, where real robot interaction is slow, expensive, and physically risky, the sample efficiency advantage of world model-based methods is practically significant.

World models provide diminishing advantage on tasks with stochastic, multi-modal, or rapidly changing dynamics that are hard to model accurately. On Atari, the advantage narrows compared to DMControl: game mechanics involve abrupt discrete transitions (score multipliers, invincibility phases, game-over conditions) that the RSSM’s smooth Gaussian or categorical dynamics model captures imperfectly, increasing imagined-versus-real trajectory divergence. Similarly, multi-player games with adversarial opponents present dynamics that depend on the opponent’s policy, which changes as the agent improves — a non-stationary distribution that makes the learned world model perpetually out of date.

World models also underperform when reward is extremely sparse (e.g., binary success/failure with episode horizon > 100 steps): imagined trajectories of length 15–50 steps may never reach the reward, providing no training signal for the actor and critic despite accurate dynamics modeling. Hierarchical world models that plan over temporally abstract subgoals (option frameworks, latent goal-conditioned models) are designed to address this, but remain an active research problem. Finally, model exploitation — where the planner discovers actions that receive high imagined reward but low real reward by exploiting model inaccuracies — is a systematic failure mode that requires uncertainty quantification and pessimistic planning (MOPO, MOREL) to mitigate.

Predictive & Self-Supervised Approaches

Q15 [Basic] What is predictive coding and how does it relate to the world model framework?

Q: Describe the predictive coding hypothesis, its computational formulation, and how it maps onto modern world model architectures.

A: Predictive coding (Rao & Ballard, 1999) is a computational neuroscience theory of perception proposing that the brain is a hierarchical prediction machine: higher cortical areas maintain generative models that continuously predict the activity of lower areas, and only prediction errors — the residuals between top-down predictions and bottom-up sensory inputs — are propagated upward. The core insight is that most neural processing is devoted to generating and refining predictions, not to passively encoding sensory input; perception is the process of minimizing prediction error across the cortical hierarchy.

This maps directly to the world model framework: the brain’s hierarchical generative model corresponds to the world model, the prediction error propagation corresponds to the model update signal, and perception corresponds to the process of updating the model to minimize prediction error given the current observation. Architectures such as PredNet (Lotter et al., 2017) made this correspondence explicit by building neural networks with the cortical hierarchy structure: each layer predicts the representation at the layer below, and only prediction errors propagate upward rather than raw activations. This biologically motivated inductive bias encourages each layer to learn increasingly abstract, temporally stable representations.

The connection to modern world models is formal: the RSSM’s KL divergence term $D_{\text{KL}}(q(z_t | h_t, o_t) \| p(z_t | h_t))$ measures the discrepancy between what the imagination predicts (prior $p$) and what the observation reveals (posterior $q$) — precisely the prediction error signal in predictive coding. DreamerV3’s free bits modification, which maintains a minimum non-zero KL, corresponds to predictive coding models that prevent degenerate solutions by requiring non-zero residual errors at each level. The predictive coding lens provides a unifying normative interpretation of world model training: the agent learns to predict its own sensory future, and the prediction error is both the learning signal and the information the agent needs to update its beliefs.

Q16 [Advanced] How does MuZero build a value-equivalent world model without reconstructing observations?

Q: Describe MuZero’s three learned functions, the value-equivalence principle, and how MCTS planning integrates with the learned model at inference time.

A: MuZero (Schrittwieser et al., 2020) learns a world model that is explicitly optimized for planning quality rather than observation fidelity, using three learned functions. The representation function $h_\theta$ maps the observation history to an initial latent state: $s^0 = h_\theta(o_{1:t})$. The dynamics function $g_\theta$ predicts the next latent state and immediate reward given a latent state and action: $(s^{k+1}, r^k) = g_\theta(s^k, a^k)$. The prediction function $f_\theta$ maps a latent state to a policy distribution and value estimate: $(\mathbf{p}^k, v^k) = f_\theta(s^k)$. There is no decoder — the latent space is never required to reconstruct observations, and $s^k$ has no enforced correspondence to any observation.

The value-equivalence principle defines what it means for two latent states to be equivalent: they are equivalent if they produce the same predictions of cumulative reward and value under all possible action sequences. This means MuZero’s latent space discards information irrelevant to planning (visual texture, background color, non-goal-relevant objects) and retains only information that affects future reward — a more targeted compression than reconstruction-based models, which must represent all observable aspects of the environment.

At inference time, MuZero uses MCTS to plan: given the current observation, $h_\theta$ encodes the initial latent state $s^0$, and the MCTS algorithm expands a search tree by calling $g_\theta$ for state transitions, $f_\theta$ for policy and value estimates at each node, and a UCB selection criterion (PUCT) to balance exploration and exploitation. After a fixed number of simulations, the action with the highest visit count is selected. All three functions are trained end-to-end from real game transitions using bootstrapped value targets ($n$-step returns with the MCTS value as the target) and policy distillation from MCTS visit count statistics. MuZero achieves superhuman performance simultaneously on Go, Chess, Shogi, and all 57 Atari games — the first algorithm to do so with no domain-specific feature engineering.

Q17 [Advanced] How does JEPA differ from generative world models?

Q: Describe the Joint Embedding Predictive Architecture proposed by LeCun (2022), explain why it avoids pixel-level prediction, and summarize the empirical evidence for JEPA-style representations.

A: The Joint Embedding Predictive Architecture (JEPA, LeCun, 2022) proposes a principled departure from generative world models that predict future observations (pixels, tokens). JEPA instead predicts the latent representation of a future or target observation from the latent representation of a context observation, without decoding back to observation space. Formally, given a context $x$ (e.g., a past video frame) and a target $y$ (a future frame), JEPA trains encoders $s_x = E_x(x)$, $s_y = E_y(y)$, and a predictor $P$ that maps context representations to target representations: $\hat{s}_y = P(s_x, z)$, where $z$ is an optional latent variable capturing prediction uncertainty. The training objective minimizes $\| \hat{s}_y - \text{sg}(s_y) \|^2$ in representation space, without any reconstruction penalty in pixel space.

The key motivation is that pixel-level prediction forces the model to allocate capacity to modeling irrelevant stochastic details — exact lighting variation, texture randomness, motion blur — that do not affect high-level understanding or action selection. LeCun argues that a model trained to predict in an abstract, potentially non-invertible representation space can develop representations that discard unpredictable low-level details and focus on high-level semantic content that is more useful for downstream tasks. Unlike masked autoencoders (MAE), which are trained to reconstruct masked patches, JEPA’s targets are latent representations rather than pixels, allowing the encoder to develop any representation that makes prediction easy.

V-JEPA (Bardes et al., 2024) demonstrated this principle on video: a model trained to predict masked spatiotemporal regions of video in representation space — without any pixel reconstruction — achieves state-of-the-art linear probing performance on action recognition (Kinetics) and scene classification, substantially outperforming masked video autoencoders trained with pixel-level reconstruction objectives. This provides empirical validation that the JEPA objective induces more semantically meaningful representations than reconstruction-based alternatives.

Q18 [Advanced] How does IRIS use transformer-based discrete tokenization for world modeling?

Q: Describe IRIS’s discrete autoencoder and transformer-based transition model, explain how imagination proceeds in token space, and compare its performance to RSSM-based models.

A: IRIS (Micheli et al., 2023, “Transformers are Sample-Efficient World Models”) combines discrete visual tokenization with an autoregressive transformer transition model to enable a world model that reasons in compressed discrete observation space rather than in an abstract continuous latent space. The architecture has two components trained sequentially.

A discrete autoencoder (VQVAE-based) maps each $64 \times 64$ observation frame to a grid of $K$ discrete tokens, each selected from a codebook of size $V$. For IRIS, $K = 16$ tokens per frame (from a $4 \times 4$ spatial grid) with $V = 512$ codebook entries. The encoder learns patch-level visual abstractions; the decoder reconstructs the frame from the $K$ tokens. Crucially, the tokenization is spatial and compositional: different patches of the frame are encoded independently, allowing the model to represent local visual structure.

A GPT-style autoregressive transformer then models the joint sequence of observation tokens and actions. Given a history of frames (as token sequences) and actions, the transformer predicts the next frame’s token sequence one token at a time, followed by a scalar reward token. At training time, the transformer is trained on real trajectories with teacher forcing; at imagination time, it generates complete future observation token sequences autoregressively, which are decoded back to pixels by the VQVAE decoder for reward prediction and interpretability. The actor-critic is then trained on imagined pixel-decoded trajectories.

Compared to RSSM-based models, IRIS operates in a more interpretable, compositional latent space (decoded tokens are visual patches) at the cost of autoregressive generation overhead ($K$ forward passes per imagined frame vs. one for the RSSM). On the Atari 100K benchmark (100,000 environment steps), IRIS substantially outperforms DreamerV2 and achieves performance close to EfficientZero, demonstrating that transformer-based discrete world models are a competitive alternative to recurrent continuous-state models for sample-efficient RL.

Evaluation & Frontiers

Q19 [Basic] How is world model quality evaluated?

Q: What metrics are used to measure world model prediction accuracy, and how does model quality relate to downstream policy performance?

A: World model evaluation proceeds along two axes that are correlated but not equivalent: model quality (how accurately does the model predict future states and rewards?) and downstream performance (does the model enable better policy learning than model-free alternatives?).

Model quality metrics assess open-loop prediction accuracy on held-out environment transitions. Multi-step prediction error (MSE or LPIPS perceptual similarity between predicted and true future frames at horizons $t = 1, 5, 15, 50$ steps) quantifies how quickly the imagined trajectory diverges from the real environment. Reward prediction accuracy (Pearson correlation or MSE between predicted and true reward) measures the model’s ability to evaluate imagined action consequences. FVD (Fréchet Video Distance) and FID measure the distributional similarity between generated and real observation sequences, capturing both per-frame fidelity and temporal coherence. These metrics are computed with the model’s encoder held fixed while imagining from a fixed latent state, isolating transition model quality from policy behavior.

Downstream performance is evaluated by training a policy using the world model and measuring the resulting agent on real environment tasks. Sample efficiency curves (cumulative episode return vs. number of real environment steps) and asymptotic performance (final return after a fixed interaction budget) are the primary metrics on standard benchmarks: DMControl Suite, Atari 100K (100K real steps), Crafter (open-world survival), NetHack, and Minecraft. The separation between model quality and downstream performance is important: Hafner et al. (2019) show that models with worse pixel reconstruction quality can achieve better downstream performance if their latent dynamics are more accurate, motivating the shift toward value-equivalent objectives in MuZero and TD-MPC. This decoupling means that model quality metrics alone are insufficient for evaluating world models intended for control.

Q20 [Advanced] What are the principal failure modes of learned world models?

Q: Categorize the main failure modes of learned world models, explain the mechanism of each, and describe the mitigation strategies in the literature.

A: Learned world models fail in four principal ways with distinct mechanisms and mitigations.

Compounding prediction error is the most fundamental failure mode: errors in the transition model accumulate over imagined rollout steps because each step’s prediction is conditioned on the previous step’s (potentially erroneous) output. A transition model with per-step error $\epsilon$ accumulates error $O(\epsilon H)$ over an $H$-step rollout — imagined trajectories progressively diverge from the real environment. Mitigation: short imagination horizons (DreamerV1 uses $H = 15$; extending beyond 50 steps rarely improves performance), $\lambda$-return discounting that weights near-future imagined states more heavily than distant ones, and ensemble models that detect disagreement as a proxy for compounding error.

Model exploitation (also called hallucinated reward) occurs when the planning algorithm discovers action sequences that receive high predicted reward from the model but low actual reward — the optimizer finds adversarial inputs to the model rather than genuinely good policies. This is a form of reward hacking applied to the learned model rather than the true reward function. Mitigation: uncertainty-aware planning that penalizes high-uncertainty model regions (MOPO penalizes imagined rewards by $\lambda \cdot \hat{\sigma}(s, a)$, where $\hat{\sigma}$ is the model’s uncertainty estimate); constraining imagination rollouts to remain close to the replay buffer distribution (MBPO limits rollout length to $k$ steps from real states).

Non-stationarity and distribution shift between the model’s training distribution and the states visited by the improving policy: as the policy improves, it visits regions of state space for which the world model has little training data, causing accuracy to drop precisely where the policy now operates. Mitigation: online model learning with continuous replay buffer updates, aggressive data collection interleaved with imagination, and pessimistic value estimation in low-data model regions.

Partial observability artifacts: if the observation does not fully characterize the underlying state, the model may learn spurious correlations between visual features and transitions rather than true causal structure, producing systematic prediction errors in novel observation configurations. Mitigation: recurrent state representations (RSSM) that integrate observation history to form an approximate belief state, multi-step consistency regularization, and auxiliary losses tying the latent representation to task-relevant state features (e.g., proprioceptive state reconstruction as an auxiliary task in image-based world models).

Q21 [Advanced] What are the open problems and research frontiers in world model research?

Q: What are the most significant unsolved challenges in building general, accurate, and scalable world models, and what are the active research directions addressing each?

A: Several fundamental open problems define the research frontier for world models.

Compositional generalization is perhaps the deepest challenge: current world models learn dynamics for the specific state-action combinations seen during training but fail to compose known sub-dynamics in novel configurations — a model trained on rooms with chairs and tables cannot extrapolate to a room with chairs on tables. Object-centric and graph-structured world models (Slot-Attention, C-SWM, DreamerV3 with object-centric encoders) attempt to address this by learning separate dynamics for individual objects, enabling combinatorial generalization over object compositions. These approaches remain limited to simple synthetic environments with few objects; scaling to complex real-world scenes is an active research direction.

Long-horizon planning remains constrained by compounding error: reliable planning beyond 50 steps is not achieved by any current world model. Hierarchical world models that plan at multiple levels of temporal abstraction — a high-level model plans over subgoals at coarse time granularity while a low-level model handles moment-to-moment transitions — are a principled approach, but learning the hierarchy itself (what constitutes a useful subgoal) remains unsolved.

Scalable world model training follows an empirical scaling question: DreamerV3 showed that larger world models improve performance on diverse tasks, but the relationship between model scale, data scale, and policy quality is not characterized with the precision of LLM scaling laws. Whether world models exhibit similarly predictable scaling behavior is an open empirical question — one with significant implications for the compute-optimal training recipe for embodied agents.

Causally faithful world models: current world models learn statistical correlations between states and actions but do not explicitly represent causal structure. They cannot answer “what would have happened under a different action sequence?” (counterfactual) or “which aspect of the state caused the reward increase?” (attribution). Causal representation learning (Schölkopf et al., 2021) and structural causal models provide a theoretical framework for causal world models, but integrating causal structure learning with the high-capacity neural network architectures used in practice remains an open research problem with substantial implications for interpretability, robustness, and transfer.

Integration with language and prior knowledge: real-world tasks require understanding language instructions, common-sense physical priors, and semantic context beyond what any agent can learn from environment interaction alone. Integrating pre-trained vision-language models as world model encoders or as auxiliary supervision — using semantic embeddings from CLIP or LLaVA to regularize the latent space — is an active frontier for making world models more generalizable and data-efficient in low-data regimes.

Quick Reference

Resources

• Ha and Schmidhuber, World Models (2018)
• Hafner et al., Learning Latent Dynamics for Planning from Pixels (PlaNet, 2019)
• Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination (DreamerV1, 2020)
• Hafner et al., Mastering Atari with Discrete World Models (DreamerV2, 2020)
• Hafner et al., Mastering Diverse Domains with World Models (DreamerV3, 2023)
• Schrittwieser et al., Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero, 2020)
• LeCun, A Path Towards Autonomous Machine Intelligence (JEPA, 2022)
• Bardes et al., V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video (2024)
• Micheli et al., Transformers are Sample-Efficient World Models (IRIS, 2023)
• Sutton, Dyna, an Integrated Architecture for Learning, Planning, and Reacting (1991)
• Rao and Ballard, Predictive Coding in the Visual Cortex (1999)
• Lotter et al., Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (PredNet, 2017)
• Hansen et al., TD-MPC2: Scalable, Robust World Models for Continuous Control (2024)
• Schölkopf et al., Toward Causal Representation Learning (2021)

See Also

• Model-Free Reinforcement Learning: Interview Questions and Answers