Video generation as world simulation is the claim that a model trained to predict video, when scaled up enough, does not merely produce pretty clips but begins to act as a general-purpose simulator of the physical world. OpenAI advanced this framing in its February 2024 technical report “Video generation models as world simulators,” which accompanied Sora. The report argued that training generative models on large amounts of video at varied durations, resolutions, and aspect ratios is a promising path toward building such simulators.
The technical mechanism is to represent video as spacetime patches, small chunks of a compressed latent video that capture both appearance and short-term motion. The report drew an explicit analogy: just as text tokens are word fragments that can be assembled into any sentence, spacetime patches are visual phrases that can be assembled into any video. A transformer, the same DiT-style architecture used for image generation, then learns over these patches. As the model scales, OpenAI reported emergent simulation behaviors, such as a rough sense of 3D consistency and object permanence, that were never explicitly programmed.
This framing reorients what generative video is for. If predicting the next frame well enough requires implicitly modeling physics, persistence, and cause and effect, then video models become a route to world models useful for robotics, planning, and embodied agents, not only for media. For a general reader, it explains why labs talk about video generators as steps toward AI that understands reality, and why the same systems sit at the center of both the creative-tools market and the deeper pursuit of machines that model the world. (OpenAI’s pages return an HTTP 403 error to automated fetchers, so the report’s contents were corroborated through search against the canonical openai.com URL cited above, the same sourcing path used by the existing Sora entries.)