squeeze-and-excitation network

A convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. design, and the add-on block it introduced, that lets a network reweight its own feature channels for each input. The squeeze-and-excitation (SE) block 'squeezes' each channel of a feature mapfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears. to one number by global average poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions., passes the resulting vector through a small bottleneck of two fully connected layersfully connected layer, n.: A layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer. (a ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. then a sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.) to compute a weight between 0 and 1 for every channel ('excitation'), and multiplies each channel by its weight. Channels useful for the current image are thus boosted and others damped, a form of channel-wise attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. or gating, at small cost (for ResNet-50, about 0.3% more computation and 10% more parameters). SE blocks can be inserted into existing architectures such as ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI. and InceptionInception, prop. n.: A family of Google image-recognition networks, the first called GoogLeNet, whose building blocks examine each patch of an image at several scales at once while keeping computation low., and are part of MobileNetV3MobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy. and EfficientNetEfficientNet, prop. n.: A family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing..

Adding SE blocks to every ResNet stage improved top-1 accuracy by about a point.