Choosing image-generation models: capabilities, evidence and a fair trial
Which image-generation system fits my deliverable, and how do I test it before committing?
Choose an image system by the image you need to deliver. A convincing concept sketch, a product photo with an unchanged label, and a diagram with exact text need different checks. The most attractive sample in a gallery may not answer your question.
The main text explains capabilities and evaluation in terms that stay true as models change. Specific products, model IDs, open-weight releases and benchmark links are in Current options. No generation experiment or ranking is reported here.
Key points
- Name the operation first: text-to-image, instruction editing, masked inpainting/outpainting, reference-guided generation, or structure-controlled generation. Systems differ more by operation than by overall “quality”.
- Most current systems iterate from noise in a compressed latent space; a few generate discrete image tokens. Those mechanisms explain common failures such as drift in “unchanged” regions during edits.
- Judge a system on capability axes that match the deliverable: prompt adherence, text rendering, edit preservation, identity consistency, resolution, control, speed and cost, and usage rights.
- Automatic metrics and leaderboards measure narrow things: FID compares distributions, CLIP-based scores measure text–image agreement, compositional benchmarks count objects and relations, preference arenas measure taste. None certifies your use.
- Run a small acceptance sheet with your own briefs and assets, keep the failures, and compare cost per accepted asset.
- Check the license of the exact weights and the hosted terms; consider provenance metadata for published images.
First decide which operation you need
| Operation | Starts from | Typical use | What to inspect |
|---|---|---|---|
| Text-to-image | A written description | Concepts, illustrations | Composition, object count, spatial relations |
| Instruction editing | An image and an instruction | “Make the jacket red” | Requested change and everything that should not change |
| Inpainting / outpainting | An image plus a mask or a larger canvas | Remove an object; extend a background | Seams, lighting continuity, mask boundaries |
| Reference-guided | One or more reference images | A consistent character or product across a set | Identity, proportions, small details such as logos |
| Structure-controlled | A sketch, edge map, depth map or pose | Layout-exact compositions | Faithfulness to the control signal |
| Upscaling | A low-resolution image | Final delivery size | Invented detail, texture artifacts |
A feature called “reference” can mean style guidance, identity preservation or a boundary frame depending on the product; read its definition. Structure control became widespread through methods such as ControlNet, which adds a trainable conditioning branch to a frozen text-to-image model so that an edge map, depth map or pose can steer layout. [1]
How image generators work, briefly
Plain version. Most modern generators start from random noise and remove it step by step, guided by your text, until an image remains. To save computation they work on a compressed version of the image and decode it at the end.
Precise version. Latent diffusion trains an autoencoder that maps images to a smaller latent grid, then trains a denoising network in that latent space, conditioned on text through cross-attention; generation runs the denoiser for a number of steps and decodes the result. [2] Many recent systems replace the convolutional denoiser with a transformer and train with rectified flow, a formulation that connects data and noise along a straight line. [3] A different family generates discrete image tokens, either one at a time or by predicting masked tokens in parallel, as in Google Research’s Muse. [4] Some current products generate and edit images inside a multimodal language model’s conversation, which makes multi-turn editing natural. The site’s ML 08 and ML 09 explain these mechanisms in depth.
What the mechanism implies for selection:
- Edits re-generate. A common editing approach adds noise to the input image and denoises it toward the new instruction; the amount of noise trades faithfulness to the original against freedom to change. [5] That is why “unchanged” areas can drift, and why mask-based inpainting or dedicated edit models preserve more.
- Text rendering varies by system and script. Letters are small, exact details, and some releases name text rendering as an explicit training focus (see Qwen-Image under Current options). Test your actual script and language, including Traditional Chinese, at the final display size.
- Seeds are not portable. A seed fixes the starting noise for one model, sampler and resolution. The same seed in another system, or another version, is a different experiment.
- Steps and guidance are part of the result. Fewer sampling steps are faster; the guidance scale trades prompt adherence against diversity and naturalness (classifier-free guidance in ML 08). Record these settings with every output.
Capability axes and how to test each
| Axis | What it means | A quick test |
|---|---|---|
| Prompt adherence | Correct objects, counts, attributes and spatial relations | “Three red cups left of a blue bowl”: count and position |
| Text rendering | Exact characters, spelling, punctuation | Your real headline, in your language, at final size |
| Edit preservation | Only the requested region changes | Edit one object; diff the rest of the image |
| Identity consistency | Same subject across images and edits | Five scenes of one character or product |
| Style range and control | Can it match a brand style or a reference? | One brief in three specified styles |
| Resolution and aspect | Native sizes; quality of upscaling | Generate at your delivery size and aspect ratio |
| Speed and cost | Time and spend per image at your settings | Measure across the whole trial, including failures |
| Rights and provenance | Weight license, hosted terms, content credentials | Read the license file and terms for your intended use |
What benchmarks and metrics can tell you
Plain version. Each metric checks one narrow property. Use them to shortlist, never to decide.
- FID (Fréchet Inception Distance) compares the statistics of many generated images with many real images in a pretrained network’s feature space. It measures distribution-level realism and diversity, needs thousands of samples, and says nothing about whether a particular image matches its prompt. [6]
- CLIP-based scores such as CLIPScore measure agreement between an image and a text using a pretrained image–text model; they are cheap, but like FID they give a holistic score that is unsuited to checking individual objects, counts or positions. [7, 8]
- Compositional benchmarks (object, count, attribute and relation checks scored by detectors or question-answering models) target prompt adherence directly; their scores depend on the benchmark version and the judge model. [8, 9, 10]
- Human preference leaderboards rank systems by pairwise votes. They capture overall appeal, not exact text, edit preservation or rights. [11, 12]
ML 08’s section on evaluating generative models covers FID and its successors. Whatever the source, record the benchmark version, model ID, quality setting, resolution, sample count and evaluator, and keep provider-reported claims separate from independent comparisons.
Run a small acceptance sheet
Use the same brief, input assets, delivery size and budget for every candidate.
| Trial | What to inspect |
|---|---|
| A simple scene with specified objects and positions | Missing objects, spatial relations, composition |
| A poster with your exact text | Spelling, Traditional Chinese characters, punctuation, legibility |
| An edit confined to one object | The requested change and unintended changes elsewhere |
| A reference-led set of related assets | Subject identity, style consistency, useful variation |
| An export in the final format | Dimensions, transparency if required, visible artifacts |
Keep failed outputs, not only the best sample. Record cost, retries, editing time and pass/fail per trial. Cost per accepted asset divides all spending, including failures, by the number of assets that passed; report human cleanup separately if you do not price it. A slower system that needs fewer repairs can be cheaper overall. The model-evaluation guide explains why a handful of trials gives only a rough estimate.
Running models locally
A downloadable model runs inside a pipeline: text encoders, the denoiser, the image decoder, and often extra components for control or upscaling. Node-based workflow tools let you wire these together and save the whole graph, which is the best record of how an image was made. The tool is a workbench, not the model: changing the model changes what the same graph can produce, and some tools also offer paid cloud nodes, so a local interface does not prove every step ran locally. [13]
Estimate memory for the whole pipeline, not just the main weight file: text encoders, decoder, intermediate tensors, output resolution and batch size all count. Reduced precision and moving components between devices trade memory for speed. Start with a documented configuration, measure peak memory and time, and add custom nodes only when they solve a clear need, since they bring executable code and dependencies.
Rights and provenance
Image models carry the same layered terms as language models: code, weights and hosted service can each have different licenses, and one family can mix permissive and non-commercial terms (licensing layers). Check the exact checkpoint’s license and any extra components it needs. If you publish images, consider attaching provenance metadata: the C2PA standard defines Content Credentials that record where a piece of content came from and how it was edited. [14]
A practical starting route
For occasional illustrations, begin with a hosted interface and a clear brief. For repeatable asset production, test an API or a saved workflow. For local operation or customization, evaluate an open-weight release against your hardware and license needs. Keep exact text and precise layout in a conventional design tool when that gives more reliable control. When still images pass, use them as deliberate inputs to motion work: continue with choosing video-generation models.
Current options, checked 6 November 2026
Representative examples, not a complete list or a ranking. Record the exact model ID or checkpoint revision with every output: an alias, a dated snapshot and a product-interface name are different records.
Hosted image systems
| Family | Current IDs or versions found in official documentation | First trial to run |
|---|---|---|
| OpenAI GPT Image | Catalog lists gpt-image-2.6-sunburst (detailed work) and gpt-image-2.6-flare (fast iteration), plus 2.5 models with dated snapshots such as gpt-image-2.5-sunburst-2026-09-08; gpt-image-1 was shut down on 23 October 2026 [15, 16, 17] | Revise one object while preserving the scene |
| Google Gemini image models | gemini-nano-banana-2.1 (Nano Banana 2.1, an update to Nano Banana 2) and the stable gemini-3.1-flash-image (Nano Banana 2) and gemini-3-pro-image [18, 19] | A sequence of conversational edits; check identity drift |
| Midjourney | Explicit version selection; check the version page for the current default [20] | Concepts for one brief; effort to refine one |
| Adobe Firefly | Adobe and partner models in one interface [21] | Finish an asset inside the design workflow you will use |
| Meta Muse Image | muse-image-1.0 for generation and multi-turn editing [22] | A reference-led edit; inspect identity and unchanged regions |
| Others with official model pages | Microsoft MAI-Image-2.6, xAI Grok Imagine Image 2.0, ByteDance Seedream 4.0 [23, 24, 25] | Same acceptance sheet |
Firefly and some API platforms host partner models; a model available inside a product is not necessarily trained by that product’s company. Runway’s API catalog, for example, lists image models from several companies, including Seedream 5 variants. [26] Subscription and API access can carry different entitlements.
Downloadable weights
| Release | What the primary source establishes | Selection implication |
|---|---|---|
| Qwen-Image and Qwen-Image-Edit-2509 | Apache-2.0 generation model with a text-rendering focus; a separate editing checkpoint [27, 28] | Generation and editing checkpoints are not interchangeable |
| FLUX.2 [klein] 4B / 9B | 4B under Apache-2.0; 9B open weights under non-commercial terms [29, 30] | Do not apply one size’s license to the family |
| Stable Diffusion 3.5 Large | Stability AI Community License [31] | Read its conditions; it is not Apache-2.0 |
| SDXL base 1.0 | Separate model card and license [32] | Match the workflow’s base/refiner requirements |
| PixArt-Σ XL-2-1024-MS | Diffusion-transformer release with an inference pipeline [33] | Measure the whole pipeline on your hardware |
| HunyuanImage-3.0 | Tencent community license; generation and Instruct variants [34] | Keep the variants distinct |
| Janus-Pro-7B | Unified understanding/generation research release [35] | Test generation separately from understanding |
Workflow tool. ComfyUI is a widely used node-based workbench and backend; its repository is GPL-3.0, and it offers optional paid API nodes. [13]
Two different “Muse” names. Meta Muse Image is the hosted line above. Google Research’s Muse (2023) is a research method for masked generative image transformers, read alongside the official MaskGIT implementation; it is not evidence of a callable Google Muse API or a downloadable checkpoint. [4, 36]
Benchmark entry points
| Resource | Useful evidence | Check before comparing |
|---|---|---|
| GenEval / GenEval 2 [8, 9] | Objects, counting, attributes and relations | Benchmark version, judge, per-skill results |
| T2I-CompBench / ++ [10] | Compositional attributes and relationships | Edition, categories, evaluation model |
| DPG-Bench [37] | Dense-prompt alignment through questions | Prompt set, sampling, evaluator |
| Arena text-to-image leaderboard [11] | Pairwise preference rankings | Model variant, date, vote count and uncertainty |
| Artificial Analysis image models and methodology [12] | Preference quality, price, generation time | Generation versus editing, provider, settings |
Further reading
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models: the design most current generators descend from. [2]
- Esser et al., Scaling Rectified Flow Transformers: the transformer and rectified-flow recipe behind many recent models. [3]
- Zhang et al., ControlNet: how structural control is added to a text-to-image model. [1]
- Meng et al., SDEdit: why editing by re-noising trades faithfulness for freedom. [5]
- Ghosh et al., GenEval: an object-focused way to measure prompt adherence. [8]
References
Accessed 6 November 2026 unless noted:
- Lvmin Zhang, Anyi Rao and Maneesh Agrawala, “Adding Conditional Control to Text-to-Image Diffusion Models,” ICCV, 2023. Paper.
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer, “High-Resolution Image Synthesis With Latent Diffusion Models,” CVPR, 2022. Paper.
- Patrick Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML (PMLR 235), 2024. Paper.
- Huiwen Chang et al., “Muse: Text-To-Image Generation via Masked Generative Transformers,” arXiv, 2023. Project, paper.
- Chenlin Meng et al., “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations,” arXiv:2108.01073, 2021. Paper.
- Martin Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017 (introduces FID). Paper.
- Jack Hessel et al., “CLIPScore: A Reference-free Evaluation Metric for Image Captioning,” EMNLP, 2021. Paper.
- Dhruba Ghosh, Hanna Hajishirzi and Ludwig Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” 2023. Paper, evaluation code.
- A. Kamath et al., “GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation,” 2025. Official data and evaluation.
- K. Huang et al., “T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation,” NeurIPS, 2023. Benchmark and ++ release.
- Arena, “Text-to-Image Leaderboard,” live resource. Leaderboard.
- Artificial Analysis, “Image Model Comparisons” and “Image Generation Benchmarking Methodology.” Comparisons, methodology.
- ComfyUI contributors, “ComfyUI.” Repository and license.
- Coalition for Content Provenance and Authenticity (C2PA), “C2PA Specifications.” Overview, specification 2.2.
- OpenAI, “Image generation.” API guide.
- OpenAI, “Models” and “Create image” API reference. Catalog, GPT-Image-2.6 Sunburst, supported model IDs and snapshots.
- OpenAI, “Deprecations.” Shutdown schedule.
- Google, “Gemini Nano Banana 2.1,” developer documentation. Model page.
- Google, “Gemini API models” and “Gemini 3.1 Flash Image.” Catalog, Gemini 3.1 Flash Image, deprecations.
- Midjourney, “Version.” Version documentation.
- Adobe, “Adobe Firefly.” Product and partner models.
- Meta, “Muse Image,” 2026. Model IDs, image-generation guide, announcement.
- Microsoft AI, “MAI-Image-2.6.” Model page.
- xAI, “Grok Imagine Image 2.0,” API documentation. Model ID and capabilities.
- ByteDance Seed, “Seedream 4.0,” official model page. Release and access.
- Runway, “Available AI Models,” API documentation. Model catalog.
- Qwen team, “Qwen-Image,” 2025. Model card.
- Qwen team, “Qwen-Image-Edit-2509,” 2025. Model card.
- Black Forest Labs, “How does FLUX licensing work?” License overview.
- Black Forest Labs, “FLUX.2 [klein],” 2026. 4B model card, 9B model card.
- Stability AI, “Stable Diffusion 3.5 Large,” 2024. Model card and license.
- Stability AI, “Stable Diffusion XL base 1.0,” 2023. Model card.
- PixArt team, “PixArt-Sigma-XL-2-1024-MS,” 2024. Model card.
- Tencent Hunyuan, “HunyuanImage-3.0,” 2025. Model card.
- DeepSeek-AI, “Janus-Pro-7B,” 2025. Model card.
- Google Research, “MaskGIT,” official implementation, 2022. Code and paper links.
- TencentQQGYLab, “ELLA — DPG-Bench,” 2024. Prompts and evaluation instructions.