Tagged optimizers
2 write-ups.
Hill Climbing, Backpropagation and NEAT on the Same XOR Problem
I expected backpropagation to learn XOR with the least work. At learning rate 0.5 it needed 9,268 to 11,124 network runs where a random-nudge hill climber needed 1,512 to 1,648. Only on a two-output full adder did it win, needing about a third of the hill climber's work.
Error Feedback, Gradient Compression, and Why Adam Breaks It
Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.