EfficientNet
In plain wordsA family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing.
A family of convolutional networksconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. (B0 to B7) produced by compound scaling. A small baseline found by neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them., built from mobile inverted-bottleneck blocks with depthwise convolutionsdepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step., squeeze-and-excitation and SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models. (Swish) activations, is enlarged by increasing depth, width and input resolution together according to fixed ratios raised to a single compound coefficient, instead of scaling one dimension at a time. The authors found balanced scaling far more efficient: EfficientNet-B7 reached 84.3% ImageNet top-1 accuracy while being 8.4 times smaller and 6.1 times faster at inference than the best previous convolutional network. EfficientNetV2 (2021) improved training speed. Compound scaling influenced later thinking about how to allocate compute across a model's dimensions (scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.).
EfficientNet-B3 gave the best accuracy we could fit within the phone's latency budget.
Origin
Introduced by Mingxing Tan and Quoc Le of Google Brain in 'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks' (arXiv, May 2019; ICML 2019).
Related
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
- depthwise separable convolutiondepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step.
- SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
Sources
- Tan, M. & Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv:1905.11946; ICML 2019. arxiv.org/abs/1905.11946
Mentioned in
MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., squeeze-and-excitation networksqueeze-and-excitation network, n.: An image-network design whose add-on blocks let the network decide, for each picture, which of its internal feature channels to turn up and which to turn down.