mode connectivity

The empirical finding that solutions found by training the same architecture several times (for example, from different random seeds) are usually not isolated basins walled off by high loss: low-loss paths link them, often as simple as a quadratic Bézier curve or a two-segment polygonal line, and training and test performance barely change along the way. Linear mode connectivity is the stronger property that the straight line between two solutions has no loss barrier; it generally fails between independently trained networks but holds between runs that share an early segment of training, and can be recovered for independent runs by permuting one network's hidden units to align with the other's, exploiting permutation symmetry (as in Git Re-Basin). Mode connectivity helps explain why weight averaging works (see exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original.), underlies model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training. and task arithmetictask arithmetic, n.: Editing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed. for fine-tuned models that share a pretrained starting point, and informs the lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket., whose tickets emerge once training becomes stable in this linear sense.