Skip to content
Last Dictionary
Plane
Inference & EfficiencyHardware & ComputeDeep Learning

N:M sparsity

nounAlso: 2:4 sparsity, 2:4 structured sparsity, semi-structured sparsity, fine-grained structured sparsity

In plain wordsA tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.

  1. A semi-structured pattern of weight sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation. in which at most N of every M consecutive weights along a matrix's input dimension are nonzero, most commonly 2:4, meaning 50% sparsity with exactly two nonzeros in each block of four. The regular pattern lets hardware store the matrix compactly with small per-block index metadata and skip the zeros during matrix multiplication: NVIDIA's sparse tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., introduced with its Ampere architecture in 2020, double peak math throughput for 2:4-sparse operands. Models are pruned to the pattern by magnitude or second-order criteria and then fine-tuned, or pruned in one shot, as SparseGPT does for 2:4 and 4:8 patterns. Accuracy loss is usually larger than for unstructured pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well. at the same density but much smaller than for removing whole channels. End-to-end speedups are often well below twofold, because only the matrix multiplications benefit.

    We pruned the feed-forward weights to 2:4 and got a 1.3x speedup on the A100 after a short fine-tune.

Origin

Popularized by NVIDIA's A100 GPU (2020), whose Ampere sparse tensor cores supported 2:4 'fine-grained structured sparsity', described by Asit Mishra and colleagues (2021); Aojun Zhou and colleagues studied training N:M-sparse networks from scratch (ICLR 2021).

Related

See also
  • sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
  • pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
  • tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
  • model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
  • GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.

Sources

  1. Mishra, A. et al. (2021). Accelerating Sparse Deep Neural Networks. arXiv:2104.08378.
  2. Zhou, A. et al. (2021). Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. ICLR 2021.
  3. Frantar, E. & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. arXiv:2301.00774. arxiv.org/abs/2301.00774

Mentioned in

pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well., sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.