focal loss
In plain wordsA way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases.
A modification of cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. loss for classification that multiplies each example's loss by a modulating factor (1 − p_t)^γ, where p_t is the probability the model assigns to the true class and γ ≥ 0 is a focusing parameter. The factor shrinks toward zero as p_t approaches one — with γ = 2, an example classified with p_t = 0.99 keeps only a ten-thousandth of its cross-entropy loss — so in a heavily imbalanced problem the gradient comes mainly from the few hard or rare examples, not the mass of easy background cases; γ = 0 recovers ordinary cross-entropy, and an optional class weight α_t adds conventional rebalancing. It was designed for dense, one-stage object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., where a detector scores on the order of 100,000 candidate locations per image but only a handful contain objects; its authors found γ = 2 with α = 0.25 worked best. It has since been adopted for segmentation, medical imaging and other problems with class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., and studied as a way to improve calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time..
Switching to focal loss let the defect detector learn from the rare scratches instead of the endless clean background.
Origin
Introduced by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollár of Facebook AI Research in 'Focal Loss for Dense Object Detection' (ICCV 2017), together with the RetinaNet detector, which matched or exceeded the accuracy of two-stage detectors while running at one-stage speed. The paper received the conference's best student paper award.
Usage note
Focal loss reweights examples by difficulty, not by class frequency, so it complements rather than replaces resampling. Because it discourages confident predictions, its output probabilities can become underconfident and may need recalibration before being used as probabilities.
Related
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
- label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
Sources
- Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P. (2017). Focal Loss for Dense Object Detection. Proc. ICCV 2017.
Mentioned in
Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science.