Choosing models: reading benchmarks, licenses and evidence
How do I read the evidence about a model and choose one for my task?
A model is one component of a working system. A leaderboard can help you shortlist candidates, but your tools, instructions, data and acceptance criteria decide whether a candidate is useful. This guide teaches how to read the evidence: what a benchmark number actually measures, why it can mislead, how much it can be trusted, and what you are allowed to do with the model you pick.
The main text is written to stay true as models change. Current model names, version links, benchmark entry points and a reading list of organizations and people are in Current options. Read this after choosing agent tools if you are selecting a model for an agent.
Key points
- You never choose “a model”; you choose an artifact: a pinned hosted version behind an API, or a specific checkpoint with a runtime and precision.
- “Open” has to be checked separately for code, weights, hosted service and training data. A model can be open-weight without being open source.
- A benchmark score is the output of a task set, a harness, a scorer and an aggregation rule. Change any of them and the number changes.
- Public test sets leak into training data. Prefer evidence from fresh, time-stamped or private tasks.
- Small evaluations are noisy: 21 of 30 tasks solved is consistent with anything from roughly 52% to 83%.
- “Solved in one of k tries” (pass@k) and “solved in all k tries” (pass^k) answer opposite questions; unattended agents need the second.
- Your own small, fixed evaluation set beats any leaderboard for the final decision.
What you are actually choosing
A model name on a leaderboard is a family label. What your system will call is more specific:
- A hosted version. Providers expose models through identifiers. Some identifiers are aliases that move to newer snapshots over time; others are pinned snapshots whose behavior should not change until retirement. Log the identifier the API reports back, not only the one you requested, and note the provider’s retirement date.
- A downloadable checkpoint. A repository revision, a file format, a precision or quantization, a chat template, and the serving runtime together determine behavior. A 4-bit conversion is a different experiment from the original 16-bit release.
- A system configuration. Reasoning-effort settings, tool definitions, retry policy and token budgets change results as much as the model choice. A benchmark that reports a model with one agent scaffold says little about the same model in a different one.
Openness is a property of each layer

Plain version. “Open model” is ambiguous. Ask what exactly is open, and for what use.
Precise version. Four layers carry separate terms:
- Code. Inference and training code, and client tools, are software. The Open Source Definition sets the criteria for an open-source license, such as free redistribution, access to source, permission for derived works, and no discrimination against fields of endeavor. Apache-2.0 and MIT are common examples. [1]
- Weights. Model parameters are distributed under their own license, which may be a standard permissive license, a custom “community” license with conditions (for example on use or on very large deployments), or a non-commercial license. A family can mix these: one size permissive, another non-commercial. [2]
- Hosted service. When you call an API, the provider’s terms decide data retention, use of inputs, rate limits and regions, independently of any weight license.
- Training data. Often only partially documented. The Open Source Initiative’s Open Source AI Definition 1.0 requires, beyond code and parameters, data information detailed enough for a skilled person to build a substantially equivalent system; most “open-weight” releases do not meet that bar. [3]
So open-weight (you can download and run the parameters, under some license) is not the same as open source in the OSI sense. Read the license file attached to the exact checkpoint, and the model card for intended use and limitations. [2, 4]
How a benchmark produces a number
Plain version. A benchmark is an exam: a set of tasks, rules for taking it, a way of marking answers, and a way of adding up the marks. Each of those choices shapes the score.
Precise version. Every reported score is the output of five components:
| Component | What it is | What can change the number |
|---|---|---|
| Task set | The problems, with a version or date window | Different split, newer release, removed or fixed items |
| Harness | How the model is prompted, which tools and scaffold it gets, how many steps | Agent framework, tool-calling mode, step and token limits |
| Sampling | Temperature, reasoning effort, number of attempts | More attempts or more thinking raise success and cost |
| Scorer | Unit tests, exact match, a human vote, or a model acting as judge | Lenient or strict matching; judge bias toward length or style |
| Aggregation | Mean over tasks, pass@k, an Elo-style rating, a weighted index | Weighting of categories; how ties and failures count |
Benchmarks also differ in kind:
| Kind | How it scores | Example (see Current options) | What it does not establish |
|---|---|---|---|
| Static, test-based | Hidden tests pass or fail | SWE-bench: patches must pass repository tests [5] | Code quality beyond the tests; performance in your repository |
| Live, time-windowed | Problems published after a date | LiveCodeBench collects new contest problems with release dates [6] | Results across different date windows are not comparable |
| Agentic environment | Final state of an interactive environment | Terminal and tool-use suites | A model-only ability independent of the scaffold |
| Pairwise preference | People choose the better of two outputs | Arena-style leaderboards | Factual correctness; success in your workflow |
| Aggregated index | Weighted combination of many evaluations | Commercial and academic indexes such as HELM | A timeless ranking: the components and weights change |
Before quoting a number, write down the suite and version, the harness and its version, the model identifier, the sampling settings and the number of attempts. Two numbers that differ in any of these are not directly comparable.
Contamination: when the exam leaks
Plain version. If a model saw the exam questions during training, a high score may measure memory rather than ability.
Precise version. Large models are trained on web-scale data, and public benchmark items, or close paraphrases, can end up in that data. The effect is measurable. When researchers wrote a fresh set of grade-school math problems matched in style and difficulty to a popular public set, some model families scored up to 8 percentage points lower on the fresh set, and the gap correlated with how likely each model was to reproduce items from the public set; many frontier models, however, showed little sign of overfitting. [7] LiveCodeBench addresses the problem by time-stamping each problem and evaluating models on problems published after their training cutoff. [6] The site’s ML 10 section on contamination covers the underlying idea.
Practical consequences:
- Prefer results on time-windowed, regularly refreshed or private tasks.
- Be suspicious of a model that is much stronger on an old public benchmark than on a newer one testing the same skill.
- Keep your own evaluation set private and do not paste it into hosted tools that may train on inputs.
How much a number can be trusted
Plain version. A score from a few dozen tasks is a rough estimate. Two models a few points apart on a small set may not differ at all.
Precise version. If a system solves of independent tasks, the observed rate is . Its sampling uncertainty shrinks only with the square root of . The Wilson score interval is a well-behaved approximate 95% interval for even with small : [39]
where is the observed success rate, is the number of tasks, and for 95% coverage. Evaluation methodology papers make the same point for language-model benchmarks and give formulas for comparing two models on the same questions. [8]
Repeated attempts. Agents are stochastic, so run each task several times. Two summaries answer opposite questions. With attempts on a task of which pass:
where pass@ estimates the probability that at least one of attempts passes (introduced for code generation [9]), and pass estimates the probability that all attempts pass (introduced to measure agent reliability [10]). Both are averaged over tasks.
Code · Python · 27 lines
from math import comb, sqrt
def pass_at_k(n, c, k):
"""Unbiased estimate of P(at least one of k attempts passes), from n attempts with c passes."""
return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n, c, k):
"""Estimate of P(all k attempts pass), from the same n attempts."""
return comb(c, k) / comb(n, k)
def wilson(successes, trials, z=1.96):
"""Approximate 95% interval for a success rate (Wilson score interval)."""
p = successes / trials
centre = (p + z * z / (2 * trials)) / (1 + z * z / trials)
half = z * sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials)) / (1 + z * z / trials)
return centre - half, centre + half
n, c = 10, 7 # one task, ten attempts, seven passed
for k in (1, 3, 5):
print(f"k={k}: pass@k={pass_at_k(n, c, k):.3f} pass^k={pass_hat_k(n, c, k):.3f}")
for solved, total in ((21, 30), (210, 300)):
lo, hi = wilson(solved, total)
print(f"{solved}/{total} tasks: {solved / total:.0%} solved, 95% interval {lo:.0%} to {hi:.0%}")Code · 5 lines
k=1: pass@k=0.700 pass^k=0.700
k=3: pass@k=0.992 pass^k=0.292
k=5: pass@k=1.000 pass^k=0.083
21/30 tasks: 70% solved, 95% interval 52% to 83%
210/300 tasks: 70% solved, 95% interval 65% to 75%
Use pass@ when a human will pick the best of several drafts; use pass (or simply the per-run success rate with its interval) when the system must work unattended every time.
Metrics that help you operate a system
Task success rate is the fraction of attempts that pass an acceptance test written before reading any output. Include constraints such as preserving unrelated files or returning usable citations.
Cost per accepted result divides all spending, including failed attempts and retries, by the number of accepted outputs. Report human review time separately or with an explicit rule. If nothing passes, report that, not an infinite or misleading cost.
Latency needs a boundary: time to first token says when output begins; end-to-end time says when an accepted artifact is ready. Report typical and slow-tail values over repeated runs, not the best demonstration.
Reliability is the per-run success rate and its interval, or pass, as above.
Running a model yourself: a memory estimate
Parameter count is not a memory budget. Two terms dominate on a GPU: the weights, roughly the parameter count times bytes per parameter, and the key–value (KV) cache, which stores attention keys and values for every token of every active sequence:
where is the number of layers, the number of key–value heads, the head dimension, the tokens per sequence, the number of concurrent sequences, the bytes per stored value, and the factor 2 counts keys and values. KV-cache memory grows with context length and concurrency, which is why serving systems manage it carefully. [11] For a hypothetical 8-billion-parameter model (32 layers, 8 KV heads, head dimension 128):
Code · Python · 18 lines
GiB = 2**30
# Assumed architecture of a hypothetical 8-billion-parameter decoder.
params, layers, kv_heads, head_dim = 8e9, 32, 8, 128
def weights_gib(bits):
return params * bits / 8 / GiB
def kv_cache_gib(tokens, sequences, bits=16):
# 2 tensors (keys and values) per layer, per KV head, per token.
return 2 * layers * kv_heads * head_dim * tokens * sequences * bits / 8 / GiB
for bits in (16, 4):
print(f"weights at {bits:>2}-bit: {weights_gib(bits):5.1f} GiB")
for tokens, seqs in ((8_000, 1), (32_000, 1), (32_000, 8)):
print(f"KV cache, {tokens:,} tokens x {seqs} sequence(s): {kv_cache_gib(tokens, seqs):5.1f} GiB")Code · 5 lines
weights at 16-bit: 14.9 GiB
weights at 4-bit: 3.7 GiB
KV cache, 8,000 tokens x 1 sequence(s): 1.0 GiB
KV cache, 32,000 tokens x 1 sequence(s): 3.9 GiB
KV cache, 32,000 tokens x 8 sequence(s): 31.2 GiBQuantizing the weights shrinks the first term; long contexts and many simultaneous users grow the second, which can exceed the weights. Runtime overhead and activations add more. Measure on your hardware at your intended context length and concurrency; a model that loads can still be too slow for interactive use. ML 11 explains quantization itself.
Build your own evaluation
- Collect 20–50 real tasks with known checks: ordinary work, an ambiguous request, a missing dependency, a tool failure, and a few hard cases.
- Write acceptance tests first. Prefer executable checks; where a rubric is needed, keep it short and calibrate any model-based judge against human judgments.
- Fix the protocol: prompt, tools, budget, retry policy, number of attempts, and the exact model identifiers.
- Run, then read the failures, not only the score. Failure categories tell you what to change.
- Report uncertainty. If two candidates’ intervals overlap heavily, decide on cost, latency, data terms or operational fit instead of declaring a winner.
- Rerun the same set after every model or prompt change. Keep tuning cases separate from judging cases.
Start with one hosted model through a tool you understand. Try an open-weight alternative when local operation, customization, data location or serving economics gives you a concrete reason, and revisit the decision when the workload changes. For visual work, continue to image-generation selection or video-generation selection.
Current options, checked 6 November 2026
These lists identify artifacts to investigate. They are not exhaustive, not a ranking, and not reproduced benchmark results. Providers rename, add and retire models often; open the catalog before relying on a row.
Hosted model families and version pages
| Provider | Current models in the official catalog | What to check |
|---|---|---|
| OpenAI | GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna [12] | Reasoning effort, tool support, latency and total task cost |
| Anthropic | Claude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5 [13] | Model IDs are pinned snapshots; effort setting; platform-specific IDs |
Gemini 4 Argon (gemini-4-argon, stable), Gemini 3.8 Flash, 3.6 Flash, 3.5 Flash-Lite [14] | Stable, new or preview status; deprecated IDs that will be rerouted | |
| Meta | muse-spark-1.3 and muse-wonder-1.0 (preview) [15] | API models versus the Muse app and Muse Code agent; Standard versus Contributor data-use mode |
| xAI | Grok 4.7 (grok-4.7) and model catalog [16] | Enable and budget search separately when current information matters |
Added in this revision: Google’s catalog lists gemini-4-argon as a stable flagship reasoning model. [14]
Meta names need an access check. Muse is a consumer agent and Muse Code a terminal coding agent. Muse Wonder Full is available only in Muse Code on qualifying subscriptions, not through the Model API; Muse Spark Open and Muse Glimmer are open-weight releases without Model API IDs. [15, 17, 18, 19]
Downloadable checkpoints
| Family | Checkpoint or download page | What to check |
|---|---|---|
| OpenAI gpt-oss | Official repository (Apache-2.0) [20] | Released checkpoint, reasoning format, serving runtime |
| Meta Muse | Muse-Spark-Open-816B-A42B, Muse-Glimmer-30B [18, 19] | Self-hosted or provider-hosted; not Meta Model API names |
| Qwen | Qwen3.8-27B, Qwen3.8-Flash-Next, Qwen3.8-2.4T-A95B [21] | The cards list different licenses; a shared family name does not imply equal terms |
| DeepSeek | DeepSeek-V4.1-Flash-1017 [22] | Dated checkpoint, precision, tool parser, inference settings |
| Google Gemma | gemma-4-31B-it [23] | Instruction-tuned artifact and its terms |
| Mistral AI | Mistral-Small-4-119B-2603 [24] | Release-specific deployment and tool-use instructions |
| Moonshot AI | Kimi-K2.5 [25] | Intended agent harness and thinking settings |
| Z.ai | GLM-5 [26] | Official chat template and serving instructions |
Benchmark entry points
| Resource | Question it asks | Record with any result |
|---|---|---|
| SWE-bench leaderboards and Verified dataset [5, 27] | Do generated patches pass the repository’s tests? | Dataset split, agent scaffold and its version |
| Terminal-Bench (site reports version 4.0) and task registry [28, 29] | Can an agent complete executable terminal tasks? | Benchmark version, agent, budgets |
| LiveCodeBench [6] | Can a model solve contest problems from a chosen time window? | Date window and sampling settings |
| Berkeley Function-Calling Leaderboard V4 [30] | Can a model choose and construct tool calls under the suite’s rules? | Native versus prompted function calling |
| Arena [31, 32] | Which response do sampled voters prefer? Agent Mode uses a different design. | Leaderboard category and date |
| HELM [33] | How does a model behave across many scenarios and metrics? | Scenario set and release |
| Artificial Analysis and methodology [34] | How do models compare on specified quality, speed and price measurements? | Index and methodology version, effort setting |
Organizations and people worth following. Teams build the field; these are entry points into different strands of work, not a ranking of credit. Dario Amodei and Daniela Amodei (Anthropic leadership) [35]; Demis Hassabis (Google DeepMind) [36]; Andrej Karpathy, for practical explanations of neural networks and language-model implementation [37]; Fei-Fei Li, for visual recognition, ImageNet and spatial intelligence (Stanford HAI) [38]. Read their technical work and the actual artifacts alongside interviews; prominence does not validate an unrelated product claim.
Further reading
- Jimenez et al., SWE-bench: a model of how to build an execution-graded benchmark from real tasks. [5]
- Jain et al., LiveCodeBench: time-windowed evaluation as a contamination defense. [6]
- Zhang et al., A Careful Examination of LLM Performance on Grade School Arithmetic: measuring overfitting with a fresh matched test set. [7]
- Miller, Adding Error Bars to Evals: statistics for reporting and comparing evaluation results. [8]
- Yao et al., τ-bench: agent evaluation with pass^k reliability. [10]
- Open Source Initiative, Open Source AI Definition 1.0: what “open” would require for an AI system. [3]
References
Primary documentation, papers and model cards, accessed 6 November 2026 unless noted:
- Open Source Initiative, “The Open Source Definition.” Definition.
- Black Forest Labs, “How does FLUX licensing work?” (an example of different licenses within one model family). License overview.
- Open Source Initiative, “The Open Source AI Definition 1.0.” Definition.
- Hugging Face, “Model Cards,” Hub documentation. Documentation.
- Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” ICLR, 2024. Paper, evaluation harness.
- Naman Jain et al., “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code,” arXiv:2403.07974, 2024. Paper, project, leaderboard.
- Hugh Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS Datasets and Benchmarks Track, 2024. Paper.
- Evan Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations,” arXiv:2411.00640, 2024. Paper.
- Mark Chen et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374, 2021. Paper.
- Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,” arXiv:2406.12045, 2024. Paper.
- Woosuk Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP, 2023. DOI, preprint.
- OpenAI, “Models.” Model catalog.
- Anthropic, “Models overview.” Model IDs and platform comparison.
- Google, “Gemini API models.” Model catalog, Gemini 4 Argon.
- Meta, “Models,” Meta Model API. Model identifiers and access.
- xAI, “Grok 4.7 Developer Guide” and “Models.” Grok 4.7, catalog.
- Meta, “Introducing Muse: The World’s First Personal AI Agent Built for Everyone,” 2026. Announcement.
- Meta, “Muse Spark Open: Get the model.” Released checkpoints.
- Meta, “Muse Glimmer: Get the model.” Released weight variants.
- OpenAI, “gpt-oss.” Repository and license.
- Qwen team, “Qwen3.8.” Official release collection.
- DeepSeek-AI, “DeepSeek-V4.1-Flash-1017.” Model card.
- Google, “gemma-4-31B-it.” Model card.
- Mistral AI, “Mistral-Small-4-119B-2603.” Model card.
- Moonshot AI, “Kimi-K2.5.” Model card.
- Z.ai, “GLM-5.” Model card.
- SWE-bench maintainers, “Official Leaderboards” and “SWE-bench Verified.” Results, dataset card.
- Terminal-Bench maintainers, benchmark and leaderboard. Official site.
- Harbor, “Dataset registry.” Versioned benchmark tasks.
- Berkeley Function-Calling Leaderboard maintainers, “BFCL V4.” Leaderboard.
- Arena, “Frequently asked questions.” Methodology.
- Arena, “How to use Agent Mode.” Evaluation design.
- Stanford CRFM, “Holistic Evaluation of Language Models (HELM).” Framework.
- Artificial Analysis, “AI Model & API Providers Analysis” and “Methodology.” Comparisons, methodology.
- Anthropic, “Leadership.” Official biographies.
- Google DeepMind, “About.” Organization.
- Andrej Karpathy, personal website. Teaching and projects.
- Stanford HAI, “Fei-Fei Li.” Profile.
- Edwin B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference,” Journal of the American Statistical Association, 22(158), 209–212, 1927. DOI.