ControlNet

A method for steering a pretrained text-to-imagetext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds. diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. with a guide image that fixes the layout, while the original network stays untouched. The base model is frozen; a trainable duplicate of its encoder takes the guide — a human-pose skeleton, a depth or segmentation map, an edge map, a scribble — and adds its features to the frozen network's skip connections and middle block through 1 × 1 convolutions whose weights and biases are initialized to zero ('zero convolutions'). These contribute nothing at first, so the combined model starts out behaving exactly like the original and control is phased in only as training finds useful weights, which keeps early noisy gradients from damaging the pretrained backbone. Training works with under 50,000 conditioning pairs or over a million. Separate ControlNets can be stacked and combined with fine-tunes such as LoRAlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves., and they made pose- and layout-faithful generation routine in the Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it. ecosystem. Lighter adapters, such as T2I-Adapter and IP-Adapter for reference images, follow similar ideas.

Feed the pose skeleton through ControlNet and all four dancers land exactly where you drew them.