code execution
In plain wordsLetting an AI write a small program and actually run it — to do math, analyze a spreadsheet or test its own code — then use the real output instead of guessing.
The capability, offered to a model as a tooltool, n.: Any outside capability an AI model is allowed to use — a web search, a calculator, a code runner, a database or an app — described to the model so it can ask for it by name., to run code it has written — typically Python or shell commands — and receive the output, errors and any files produced. It gives language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. exact computation, data analysis and charting, file conversion and the ability to test software, and lets coding agentscoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed. run builds and test suites. Code can also serve as an agent's entire action format: in the CodeAct approach (Xingyao Wang et al., 2024) the model emits executable Python instead of JSON tool callstool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on., composing many operations, loops and conditionals in one step, which the authors found raised success rates by up to 20 percentage points. Because model-written code may be wrong or steered by prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., execution normally happens in a sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed. — a container, micro-VM or WebAssembly runtime with limited file-system, network and time access.
Turn on code execution so the model computes the totals rather than estimating them.
Origin
Program-aided language models (PAL; Luyu Gao et al., 2022) and Program of Thoughts (Wenhu Chen et al., 2022) had models write code for an interpreter to run. OpenAI's Code Interpreter for ChatGPT (2023, later renamed Advanced Data Analysis) brought the idea to the public, and model providers later offered server-side code-execution tools in their APIs.
Usage note
Distinct from code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs., which only writes code; execution closes the loop by running it and feeding back the results.
Related
- sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed.
- tool usetool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work.
- coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed.
- code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs.
- program synthesisprogram synthesis, n.: Getting a computer to write a program automatically from a description of what the program should do — a logical specification, a few input–output examples or a request in plain language.
- agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act.
- code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs.
Sources
- Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H. & Ji, H. (2024). Executable Code Actions Elicit Better LLM Agents. ICML 2024. arXiv:2402.01030. arxiv.org/abs/2402.01030
- Gao, L. et al. (2022). PAL: Program-aided Language Models. arXiv:2211.10435 (ICML 2023).
Mentioned in
coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed., sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed., tool usetool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work., ChatGPT pluginsChatGPT plugins, prop. n.: Add-ons, launched in 2023 and retired in 2024, that let ChatGPT use outside services — web browsing, a code runner, travel booking or shopping sites — while answering a user., code modecode mode, n.: A way for AI agents to use tools by writing a short program that calls them, instead of asking for each tool one step at a time., program-aided language modelprogram-aided language model, n.: A technique in which an AI solves a math or logic problem by writing a small computer program and letting a computer run it, instead of doing the arithmetic in its head., recursive language modelrecursive language model, n.: A way of running an AI language model so it can handle enormous documents by writing small programs to search and split them, and calling copies of itself on the pieces.