Choosing video-generation models: shots, control and usable footage
Which video-generation system fits my shot, and how do I compare systems on footage I can actually use?
Choose a video model for the shot you need. A beautiful opening frame is not enough: the subject must stay recognizable, the action must finish, and the clip must cut cleanly against its neighbors. Start with image-generation selection if you first need a reference image or storyboard frame.
The main text explains what to look for and how to measure it, independent of any product. Current services, model IDs, open-weight releases and benchmark links are in Current options. No paid generations were run for this article.
Key points
- Name your inputs first: text only, an image to animate, start and end frames, existing footage to edit, or audio to match. Support for each varies by model and by interface.
- Video models generate whole clips in a compressed space-time latent; cost and memory grow with duration, resolution and frame rate, which is why clips are short and extensions drift.
- Judge motion, not stills: temporal consistency, motion plausibility, action order, camera control, reference fidelity and audio sync are separate axes.
- Distribution metrics (FVD) and dimension-wise suites (VBench, VBench-2.0) diagnose different failures; preference leaderboards measure appeal. None replaces watching your own shots.
- Compare cost per accepted second: a cheaper service per generated second can be more expensive per second you keep.
- Treat code, weights, extra encoders and hosted access as separate artifacts with separate terms.
Decide what you already have
| Starting point | Operation | Best when | Watch for |
|---|---|---|---|
| A written scene | Text-to-video | Exploratory mood shots | Wrong subject design; action out of order |
| An approved still | Image-to-video | Product or character shots where appearance is fixed | The subject changing shape as it moves |
| First and last frames | Interpolation between keyframes | Shots that must start and end in known compositions | Implausible paths between the frames |
| Existing footage | Video-to-video editing, restyling, reframing | Keeping an actor’s real movement | Unrequested changes; flicker |
| A generated clip | Extension | Longer shots | Gradual drift in identity and lighting |
| Dialogue or sound needs | Joint audio–video generation | Speech, effects, ambience in one pass | Lip-sync and timing errors |
A control called “reference” may steer identity, style or a boundary frame depending on the product. Read its definition, and confirm you have the rights to the reference assets you upload.
How video generators work, briefly
Plain version. A video generator works like an image generator that denoises a whole stack of frames at once, so that each frame agrees with its neighbors. To make that affordable it compresses the video in both space and time first.
Precise version. Video diffusion models extend image diffusion to a space-time volume, denoising all frames of a clip jointly so that temporal coherence is learned rather than enforced afterwards. [1] Practical systems compress the clip with a video autoencoder into a latent that is smaller in height, width and time, run a denoiser (increasingly a transformer that attends across space and time) in that latent, and decode. [2, 3] Conditioning on text, an input image, or keyframes enters the denoiser alongside the noisy latent. The site’s DLCV 12A covers these designs, including a worked calculation of latent size and attention cost.
What the mechanism implies for selection:
- Duration is expensive. Attention cost grows quickly with the number of latent frames and pixels, so services cap clip length and resolution, and longer shots are built by extension. Each extension conditions on the previous clip’s end, so small errors accumulate as drift.
- Image-to-video constrains appearance, not physics. The input image fixes the first frame; plausible contact, occlusion and object permanence still depend on what the model learned.
- Edits re-generate. As with images, editing typically re-synthesizes the clip under new conditioning; check every region that should not change.
- Settings are part of the result. Resolution, frame rate, duration, sampling steps, prompt rewriting and seed all change the output; record them.
Capability axes
| Axis | Question | How to test |
|---|---|---|
| Temporal consistency | Do faces, objects, clothing and backgrounds stay the same? | A slow orbit around a detailed subject |
| Motion plausibility | Are contacts, weight, occlusion and collisions believable? | A hand picking up a glass; a ball bouncing |
| Instruction and order | Do the requested actions happen, in order? | “Opens the door, then turns to camera” |
| Camera control | Can you specify pans, dollies, focal changes? | One camera move named in the prompt or controls |
| Reference fidelity | Does an input image or character survive motion? | Animate your approved product still |
| Duration, resolution, frame rate | Native limits and quality at your delivery format | Generate at the size and rate of your edit |
| Audio | Speech, effects and sync, if supported | A short spoken line with a visible action |
| Editability | Can you extend, loop, reframe or edit without breaking continuity? | Extend one clip twice and compare ends |
| Speed and cost | Waiting time and spend per accepted second | Track over the whole trial |
How video is evaluated
- FVD (Fréchet Video Distance) extends the FID idea to video: it compares feature statistics of many generated and real clips. It measures distribution-level quality and needs many samples; it cannot tell you whether one shot followed its prompt. [4]
- VBench decomposes quality into separate dimensions, such as subject consistency, motion smoothness, temporal flicker and spatial relationships, each with its own automatic evaluator, so you can see which failure a model has. [5]
- VBench-2.0 targets intrinsic faithfulness: human fidelity, controllability, creativity, physics and commonsense. It is a different suite; its scores are not interchangeable with VBench totals. [6]
- Preference leaderboards rank systems by blind pairwise comparisons; with and without audio they answer different questions. [7, 8]
DLCV 12A’s section on evaluating video generators discusses these metrics and human evaluation. Whatever you quote, save the suite version, model and provider IDs, task (text- or image-to-video), audio setting, resolution, frame rate, duration and sampling budget.
Review your own shots
Write a shot brief with subject, action, camera movement, lighting, framing and required sound. Separate mandatory details from preferences. Then review every candidate with the same sheet:
- Instruction: Does the requested action occur, in the right order?
- Continuity: Do faces, objects, clothing and backgrounds remain consistent?
- Motion: Are contacts, occlusion and camera movement plausible?
- Sound: If supported, do speech and effects match the visible event?
- Editability: Are the opening and ending usable beside the neighboring shots?
Watch the full clip at playback speed, then step through suspicious transitions frame by frame; a sharp still can hide an object changing shape halfway through. Change one major variable per iteration. For a product shot, establish shape and motion first and refine atmosphere afterwards. Save the prompt, input assets, model and version, settings, outputs and rejection reasons. Assemble accepted shots in an editor and review continuity across cuts: extending one clip does not plan a story’s screen direction, lighting or sound.
Compare the cost of usable footage
Cost per accepted second is all generation spending for a shot, including discarded attempts, divided by the seconds you actually keep in the edit. State whether upscaling, audio, storage and human editing are included, and report a failure if nothing passes rather than a misleading price. The hypothetical log below shows why the per-generated-second price can mislead:
Code · Python · 13 lines
# Hypothetical trial log for one shot brief: (clip seconds, generation cost, seconds kept in the edit).
# The numbers are made up to show the arithmetic; they are not prices of any service.
trials = {
"service A": [(8, 1.20, 0), (8, 1.20, 5), (8, 1.20, 0), (8, 1.20, 6)],
"service B": [(5, 0.40, 0), (5, 0.40, 0), (5, 0.40, 3), (5, 0.40, 0),
(5, 0.40, 0), (5, 0.40, 4), (5, 0.40, 0), (5, 0.40, 0)],
}
for name, log in trials.items():
generated = sum(s for s, _, _ in log)
spent = sum(c for _, c, _ in log)
kept = sum(k for _, _, k in log)
print(f"{name}: generated {generated} s, kept {kept} s, spent {spent:.2f},"
f" {spent / generated:.3f} per generated s, {spent / kept:.3f} per accepted s")Code · 2 lines
service A: generated 32 s, kept 11 s, spent 4.80, 0.150 per generated s, 0.436 per accepted s
service B: generated 40 s, kept 7 s, spent 3.20, 0.080 per generated s, 0.457 per accepted sService B is almost half the price per generated second and still more expensive per second kept. With a handful of trials the difference is not significant; the point is which ratio to compute. Keep resolution, duration, source assets, budget and acceptance rules identical across candidates, and record waiting time as well as generation time.
Running models locally
Downloadable weights do not guarantee that a workstation can run your workload. Memory and time depend on resolution, frame count, precision, the set of model components (text encoders, the video autoencoder, the denoiser) and the runtime. Offloading components to CPU memory reduces GPU memory pressure but adds transfers and latency. Start with the repository’s documented configuration, then measure your target settings.
For reproducibility, save the model-card URL and revision, the checkpoint filename, code revision, precision or quantization, sampler and step count. A distilled or quantized checkpoint is a separate experiment from the full release. Code, weights, extra encoders and hosted access can each carry different terms (licensing layers).
Begin with one hosted service for a bounded shot; consider local weights when customization, data location or sustained volume justifies operating them. Reuse the same trial shots after every upgrade, following the model-evaluation guide.
Current options, checked 6 November 2026
Representative examples, not a complete list or a tested ranking. Pin the access surface (which API, which platform) as well as the model name.
Hosted services and models
| Family or service | Current IDs in official documentation | First thing to test |
|---|---|---|
| Google Gemini Omni Flash | gemini-omni-1.2-flash is the Gemini API’s current video model, for generation and conversational video editing; the Gemini API’s Veo 3.1 preview endpoints were shut down on 22 October 2026 [9] | Whether iterative edits preserve the subject and earlier decisions |
| Google Veo 3.1 on Google Cloud | GA veo-3.1-generate-001 and veo-3.1-fast-generate-001; preview veo-3.1-lite-generate-001; sound generation supported [10] | Motion, dialogue and audio–video sync together |
| Runway API catalog | Runway gen4.5, gen4_turbo and the editing model aleph2, plus models from other companies such as Kling kling_4, ByteDance seedance2_5, MiniMax hailuo3 / h3_max, Alibaba wan3 [11] | The exact endpoint’s inputs and edit preservation |
| Luma Ray 3.2 | ray-3.2: text- and image-to-video with start/end frames or keyframe anchors, extension, editing, reframing, optional HDR, 5 or 10 s clips [12] | An edit whose unchanged content must survive |
| xAI Grok Imagine Video 1.5 | grok-imagine-video-1.5 (text and image input) [13] | Motion and prompt order on a short brief |
A service can host another company’s model, and a consumer app and an API may expose different controls; compare the product you will actually use.
No longer available. OpenAI’s deprecations page lists the Videos API and the Sora 2 models as removed from the API on 24 September 2026. Sora belongs in historical comparisons only. [14]
Which “Muse”? Meta’s announcement of Muse Image and Muse Video (7 July 2026) describes Muse Video as a preview with native audio and names audio–video sync and fast-motion physics as open gaps; the current Meta Model API catalog lists no Muse Video endpoint, so no public API ID or checkpoint is assumed here. [15, 16] Microsoft Research’s Muse (WHAM) is an unrelated research model of a video game that generates game visuals and controller actions. [17]
Downloadable weights
| Release | Why inspect it | Terms |
|---|---|---|
| Wan2.2: T2V-A14B, I2V-A14B, TI2V-5B | Separate text, image and joint text–image conditioned checkpoints [18] | Repository Apache-2.0; check the selected weights and dependencies |
| HunyuanVideo and HunyuanVideo-1.5 | Public inference code; T2V, I2V and distilled variants [19, 20] | Tencent Hunyuan Community License [19] |
LTX-2: ltx-2-19b-dev, ltx-2-19b-distilled | Audio–video inference and training tools; full and distilled checkpoints [21] | Code Apache-2.0; weights under the LTX-2 Community License [21] |
| CogVideoX1.5: 5B T2V, 5B I2V | Explicit text- and image-to-video cards [22] | CogVideoX license on the checkpoint |
| Mochi 1 preview | Another open text-to-video checkpoint [23] | Apache-2.0 on the model card; preview status |
Benchmark entry points
| Link | Contributes | Boundary |
|---|---|---|
| VBench project and leaderboard [5, 24] | Dimension-level quality with evaluation code | Record the suite, task and generation settings |
| VBench-2.0 and protocol [6] | Fidelity, controllability, creativity, physics, commonsense | Not interchangeable with VBench totals |
| Artificial Analysis text-to-video / image-to-video and methodology [7, 8] | Blind pairwise preferences, separate audio and silent evaluations | Benchmark version and evaluator protocol |
Further reading
- Ho et al., Video Diffusion Models: the starting point for joint space–time denoising. [1]
- Blattmann et al., Stable Video Diffusion: latent video diffusion and the role of data curation at scale. [2]
- Wan team, Wan: Open and Advanced Large-Scale Video Generative Models: a detailed technical report on an open-weight video model family. [3]
- Huang et al., VBench: why per-dimension evaluation beats one aggregate score. [5]
- Unterthiner et al., FVD: the distribution metric and its assumptions. [4]
References
Accessed 6 November 2026 unless noted:
- Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi and David J. Fleet, “Video Diffusion Models,” NeurIPS, 2022. Paper.
- Andreas Blattmann et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, 2023. Paper.
- Team Wan et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” arXiv:2503.20314, 2025. Report.
- Thomas Unterthiner et al., “Towards Accurate Generative Models of Video: A New Metric & Challenges,” arXiv:1812.01717, 2018. Paper.
- Ziqi Huang, Yinan He, Jiashuo Yu, et al., “VBench: Comprehensive Benchmark Suite for Video Generative Models,” CVPR, 2024. Project.
- Dian Zheng, Ziqi Huang, Hongbo Liu, et al., “VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness,” arXiv, 2025. Paper, protocol.
- Artificial Analysis, “Video Generation Benchmarking Methodology,” 2026. Methodology.
- Artificial Analysis, “AA-Video-T2V” and “AA-Video-I2V” leaderboards, 2026. Text-to-video, image-to-video.
- Google, “Video generation in the Gemini API” and “Gemini Omni Flash,” 2026. Video API and preview shutdown, Omni guide, deprecations.
- Google Cloud, “Veo 3.1,” 2026. Model IDs, launch stages and access.
- Runway, “Available AI Models,” API documentation. Model catalog.
- Luma AI, “Models,” Luma Agents documentation. Model guide.
- xAI, “Grok Imagine Video 1.5,” API documentation. Model page.
- OpenAI, “Deprecations,” API documentation. Videos API and Sora 2 shutdown.
- Meta, “Introducing Muse Image and Muse Video,” 7 July 2026. Announcement.
- Meta, “Models,” Meta Model API. Model catalog.
- Katja Hofmann, “Introducing Muse: Our first generative AI model designed for gameplay ideation,” Microsoft Research, 19 February 2025. Article.
- Wan team, “Wan2.2.” Repository.
- Tencent Hunyuan, “HunyuanVideo” and its community license. Repository, license.
- Tencent Hunyuan, “HunyuanVideo-1.5,” 2025. Model card, inference instructions.
- Lightricks, “LTX-2,” 2026. Model card, repository, model license.
- Z.ai / THUDM, “CogVideoX1.5-5B,” 2024. T2V card and license, I2V card.
- Genmo, “Mochi 1 preview,” 2024. Model card.
- Vchitect, “VBench Leaderboard.” Leaderboard.