instruction-based image editing
In plain wordsChanging a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand.
Image editing in which a model takes an input image and a natural-language instruction and produces the edited image, changing what the instruction specifies while preserving everything else, such as identity, layout and lighting. Early text-guided editing manipulated a diffusion model's internals for each image — inverting it to noise and regenerating under an altered prompt, or editing cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt. maps (prompt-to-prompt). InstructPix2Pix (2022) instead trained a conditional diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. on a synthetic dataset of before-and-after pairs created with a language model and a text-to-image model, so it could edit in a single pass without per-image optimization. Since 2025 the task has been dominated by large multimodal generators — GPT-4o image generation, Nano BananaNano Banana, prop. n.: The nickname, later Google's official branding, for its Gemini image models, a viral hit in 2025 for editing photos from simple instructions while keeping people looking like themselves., FLUX.1 Kontext and similar models — that accept several reference images and multi-turn instructions. Hard problems include preserving unedited regions and identities across repeated edits, following spatial instructions precisely and evaluating edits, often with vision-language-model judgesLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers..
Instruction-based editing turned a two-hour retouch into one sentence: remove the scaffolding.
Origin
Tim Brooks, Aleksander Holynski and Alexei Efros of UC Berkeley introduced InstructPix2Pix in a paper posted in November 2022 (CVPR 2023), framing editing as following written instructions. The approach became a mainstream consumer feature with natively multimodal image models in 2025.
Related
- inpaintinginpainting, n.: Filling in a missing or erased part of a picture so it blends seamlessly with the rest — removing a stranger from a photo, or repairing a damaged one.
- image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout.
- Nano BananaNano Banana, prop. n.: The nickname, later Google's official branding, for its Gemini image models, a viral hit in 2025 for editing photos from simple instructions while keeping people looking like themselves.
- FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely.
- text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds.
- denoising diffusion implicit modeldenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture.
- multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
Sources
- Brooks, T., Holynski, A. & Efros, A. A. (2023). InstructPix2Pix: Learning to Follow Image Editing Instructions. CVPR 2023. arXiv:2211.09800. arxiv.org/abs/2211.09800
Mentioned in
text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout., inpaintinginpainting, n.: Filling in a missing or erased part of a picture so it blends seamlessly with the rest — removing a stranger from a photo, or repairing a damaged one., Nano BananaNano Banana, prop. n.: The nickname, later Google's official branding, for its Gemini image models, a viral hit in 2025 for editing photos from simple instructions while keeping people looking like themselves., denoising diffusion implicit modeldenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture.