Google’s new video generation AI model Lumiere uses a new diffusion model called Space-Time-U-Net, or STUNet, that figures out where things are in a video (space) and how they simultaneously move and change (time). Ars Technica reports this method lets Lumiere create the video in one process instead of putting smaller still frames together. Lumiere starts with creating a base frame from the prompt. Then, it uses the STUNet framework to begin approximating where objects within that frame will move to create more frames that flow into each other, creating the appearance of seamless motion. Lumiere also generates 80 frames compared to 25 frames from Stable Video Diffusion. Admittedly, I am more of a text reporter than a video person, but the sizzle reel Google published, along with a pre-print scientific paper, shows that AI video generation and editing tools have gone from uncanny valley to near realistic in just a few years. It also establishes Google’s tech in the space already occupied by competitors like Runway, Stable Video Diffusion, or Meta’s Emu. Runway, one of the first mass-market text-to-video platforms, released Runway Gen-2 in March last year and has started to offer more realistic-looking videos. Runway videos also have a hard time portraying movement. Google was kind enough to put clips and prompts on the Lumiere site, which let me put the same prompts through Runway for comparison.
Full story : Google demos Lumiere, a space-time diffusion model for realistic AI videos.
