pelican on a bicycle
In plain wordsA joke test in which an AI model is asked to draw, by writing code, a pelican riding a bicycle; comparing the results became a popular, lighthearted way to judge each new model.
An informal benchmark in which a language model is asked to produce an SVG (Scalable Vector Graphics) image of a pelican riding a bicycle — a text-only task requiring the model to write drawing code for a scene it cannot see. The test probes spatial reasoning, composition and code generation at once, and its absurd subject was chosen so that ready-made examples were unlikely to exist in training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system.. The results, often comic (detached wheels, pelicans hovering beside the frame), became a fixture of commentary on model launches. Its creator presents it as a joke that nonetheless tracks overall model quality surprisingly well, and has warned that its fame invites targeted optimizationbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks., suggesting other animal-and-vehicle pairings as checks.
Forget the leaderboard — how's its pelican on a bicycle?
Origin
Created by the British programmer Simon Willison, who rolled it out in a blog post of 25 October 2024 comparing the output of 16 models given the prompt 'Generate an SVG of a pelican riding a bicycle'. He has since run it on most major model releases, and it became one of the best-known informal AI tests.
Related
- vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- Will Smith eating spaghettiWill Smith eating spaghetti, n.: A grotesque 2023 AI-generated video of actor Will Smith eating spaghetti that became a meme and an informal yardstick: each new video model is judged by how well it redoes the scene.
- benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.
- jagged intelligencejagged intelligence, n.: The uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
Sources
- Willison, S. (2024). Pelicans on a bicycle. Simon Willison's Weblog, 25 October 2024. simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle
Mentioned in
one-shotone-shot, v.: Slang for an AI completing a whole task correctly on the first try from a single request — 'it one-shotted the app' — with no corrections or follow-up needed., vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests., Simon WillisonSimon Willison, prop. n.: A British programmer and blogger, co-creator of the Django web framework, whose widely read writing on language models coined 'prompt injection' and the 'lethal trifecta'., Will Smith eating spaghettiWill Smith eating spaghetti, n.: A grotesque 2023 AI-generated video of actor Will Smith eating spaghetti that became a meme and an informal yardstick: each new video model is judged by how well it redoes the scene.