style transfer
In plain wordsRepainting a photo in the style of an artwork — your street as Van Gogh might have painted it — by keeping the photo's content and borrowing the painting's colors, strokes and textures.
The rendering of the content of one image in the visual style of another. In neural style transfer both are described by features of a pretrained convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. (classically VGG): content by the activations of a deeper layer, and style by the correlations between feature channels — Gram matrices — at several layers, which capture texture while discarding layout. The original method optimizes the pixels of an output image by gradient descent to match the content features of one image and the style statistics of the other, taking minutes per image. Feed-forward networks trained with the same perceptual lossesperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one. (2016) made it real-time, and adaptive instance normalization (2017) handled arbitrary styles in one pass — an idea later reused in StyleGANStyleGAN, prop. n.: An image-generating network from NVIDIA, famous for photorealistic faces of people who do not exist, that sets coarse traits such as head angle and face shape separately from fine details such as coloring and skin texture.. Today diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. perform style transfer through prompts, reference-image adapters or fine-tuning.
The app's style transfer turned every holiday snap into a passable woodcut.
Origin
Leon Gatys, Alexander Ecker and Matthias Bethge introduced neural style transfer in 'A Neural Algorithm of Artistic Style' (posted August 2015). Justin Johnson and colleagues made it real-time in 2016, and phone apps such as Prisma (2016) turned it into a mass phenomenon. Earlier work on texture synthesis and 'image analogies' pursued the same goal without neural networks.
Usage note
Style transfer raises the questions that surround AI artAI art, n.: Images, music or other artworks made with the help of AI — from early computer drawing programs to today's prompt-based image generators — and the debate over whether and how such work counts as art.: an artist's style is generally not protected by copyright, but imitating living artists with models trained on their work is contested, and tools such as Glaze and NightshadeNightshade, prop. n.: A free tool for artists that subtly alters their images so that AI image generators trained on them without permission learn the wrong things — associating 'dog' with pictures of cats, say. try to disrupt style mimicry.
Related
- perceptual lossperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one.
- image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout.
- AI artAI art, n.: Images, music or other artworks made with the help of AI — from early computer drawing programs to today's prompt-based image generators — and the debate over whether and how such work counts as art.
- StyleGANStyleGAN, prop. n.: An image-generating network from NVIDIA, famous for photorealistic faces of people who do not exist, that sets coarse traits such as head angle and face shape separately from fine details such as coloring and skin texture.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- DeepDreamDeepDream, prop. n.: A 2015 Google program that exaggerates whatever patterns an image-recognition network detects in a picture, producing hallucinatory images full of eyes, dog faces and swirls.
Sources
- Gatys, L. A., Ecker, A. S. & Bethge, M. (2015). A Neural Algorithm of Artistic Style. arXiv:1508.06576. arxiv.org/abs/1508.06576
- Johnson, J., Alahi, A. & Fei-Fei, L. (2016). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. ECCV 2016.
Mentioned in
AI artAI art, n.: Images, music or other artworks made with the help of AI — from early computer drawing programs to today's prompt-based image generators — and the debate over whether and how such work counts as art., feature mapfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears., generative artgenerative art, n.: Art in which the artist invents a process — dice rolls, software, written instructions — and lets that process make the work; the phrase now also loosely covers pictures from AI image generators., image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., StyleGANStyleGAN, prop. n.: An image-generating network from NVIDIA, famous for photorealistic faces of people who do not exist, that sets coarse traits such as head angle and face shape separately from fine details such as coloring and skin texture., DeepDreamDeepDream, prop. n.: A 2015 Google program that exaggerates whatever patterns an image-recognition network detects in a picture, producing hallucinatory images full of eyes, dog faces and swirls., perceptual lossperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one., VGGNetVGGNet, prop. n.: An influential 2014 image-recognition network from Oxford, built by stacking many identical small pattern detectors, which showed that making networks deeper by itself improves accuracy.