Benchmark · Coding
LiveCodeBench
Definition
- Short name
- LiveCodeBench
- Category
- Coding
- Introduced
- 2024
- Metric
- pass@1 (%) within a stated problem date window
- Direction
- Higher is better
- Reference check
- Confirmed against the paper or official site
Health review
As of
Top models cluster near the ceiling; small differences are noise.
A 2026 study of saturation across 60 benchmarks rated LiveCodeBench highly saturated (saturation index 0.77): its top models’ scores are compressed relative to evaluation uncertainty.
Source: arxiv.org — source for this health review (external site)
Caveats
- Problems carry release dates and scores are computed within a date window, so results are only comparable within the same window and release version.Source: livecodebench.github.io — source for caveat 1 on LiveCodeBench (external site)
- Artificial Analysis removed LiveCodeBench from its Intelligence Index in v4.0 (January 2026), without stating a reason, and has retired it from active reporting.Source: artificialanalysis.ai — source for caveat 2 on LiveCodeBench (external site)
- On Artificial Analysis’s LiveCodeBench leaderboard, which it now lists as a legacy evaluation, the three highest scores are 89.6–91.7%.Source: artificialanalysis.ai — source for caveat 3 on LiveCodeBench (external site)
Results on file
3 results in 2 comparability groups (3 vendor-reported). Results are only charted against others in the same group — the same benchmark and the same variant as named by the source. Last retrieved .
LiveCodeBench · v6
2 results
Not directly comparable
These are vendor-reported numbers from 2 different organisations. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Qwen3.8-27B | Not stated by source | 90.3%No CI reported | conditions not stated | Vendor-reportedAlibaba Cloud (Qwen)huggingface.co — source for Qwen3.8-27B on LiveCodeBench, v6 (external site) | Show excerpt for Qwen3.8-27B on LiveCodeBench, v6“LiveCodeBench v6 | 90.3 | 83.9 | 89.6 | -- | 88.8 (first col = Qwen3.8-27B)” Retrieved |
| Gemma 4 31B | Not stated by source | 80.0%No CI reported | instruction-tuned model; no other conditions stated | Vendor-reportedGoogle DeepMindhuggingface.co — source for Gemma 4 31B on LiveCodeBench, v6 (external site) | Show excerpt for Gemma 4 31B on LiveCodeBench, v6“| LiveCodeBench v6 | 80.0% |” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.
LiveCodeBench · Problems 2024-10-01 to 2025-02-01
1 result
Not directly comparable
These are vendor-reported numbers. Each source chose its own settings — reasoning effort, tools, harness or scaffold, number of attempts, grader — so a longer bar is a claim made under different conditions, not evidence that one model is better. None of these results has been reproduced by HARBAN.
Only one result is on file for this variant, so there is nothing to chart it against.
| Model | Model detail | Score | Settings (as stated) | Source | Evidence |
|---|---|---|---|---|---|
| Llama 4 Maverick | Not stated by source | 43.4%No CI reported | 0-shot; problems dated 10/01/2024-02/01/2025; instruction-tuned model | Show excerpt for Llama 4 Maverick on LiveCodeBench, Problems 2024-10-01 to 2025-02-01“LiveCodeBench (10/01/2024-02/01/2025) 0 pass@1 33.3 27.7 32.8 43.4” Retrieved |
- Vendor-reported:
- Published by the organisation that built the model. Not independently reproduced.