HumanEval
In plain wordsA set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code.
A code-generation benchmark of 164 Python problems composed by hand. For each, a model is shown a function's signature and a docstring stating what the function should do, and must write the body; the code is then run against hidden unit tests, 7.7 per problem on average, and results are reported as pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.; each problem also comes with a reference solution. Writing the problems by hand, rather than collecting them from the web, was meant to keep them out of training data. At release, OpenAI's Codex solved 28.8% of problems with one sample (70.2% when allowed 100 samples), while GPT-3 solved none. HumanEval was the standard code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs. metric of 2021–2023 and was saturated by 2024, with frontier models above 90%. EvalPlus's HumanEval+ (2023) added many more tests per problem and showed that some 'passing' solutions were in fact wrong.
Our small code model reaches 85% pass@1 on HumanEval but only 60% on HumanEval+.
Origin
Introduced by Mark Chen, Jerry Tworek, Heewoo Jun and colleagues at OpenAI in 'Evaluating Large Language Models Trained on Code' (July 2021), the paper describing Codex, the model behind the first version of GitHub CopilotGitHub Copilot, prop. n.: An AI assistant from GitHub, built into programmers' code editors, that suggests code as they type, answers questions about code and can carry out coding tasks on its own..
Usage note
Its problems are short, self-contained functions, so HumanEval says little about real software engineering; repository-level benchmarks such as SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents. replaced it as the headline coding test. Google's MBPP (2021), with 974 crowd-sourced basic Python problems, is often reported alongside it.
Related
- pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.
- code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs.
- SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- GitHub CopilotGitHub Copilot, prop. n.: An AI assistant from GitHub, built into programmers' code editors, that suggests code as they type, answers questions about code and can carry out coding tasks on its own.
- coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed.
Sources
- Chen, M., Tworek, J., Jun, H. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374. arxiv.org/abs/2107.03374
- Liu, J., Xia, C. S., Wang, Y. & Zhang, L. (2023). Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. NeurIPS 2023.
Mentioned in
SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents., agentic workflowagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs., CodexCodex, prop. n.: A name OpenAI has used twice: first for a 2021 AI model that wrote code and powered the original GitHub Copilot, then from 2025 for its family of coding agents., pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code., ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining.