Scaling the learning rate by progress

Dopamine neurons do not signal how wrong an animal is. They signal how much better or worse things turned out than expected: a reward prediction error [1]. I wanted to try the crudest possible version of that idea in a neural network. Instead of a fixed step size, scale each update by how much the loss just improved.

The rule

Let $L_t$ be the loss on step $t$ and $\eta$ the base learning rate. The update is

$$\theta_{t+1} = \theta_t - \eta\, s_t\, \nabla_\theta L_t, \qquad s_t = \operatorname{clip}\big(1 + 2\,(L_{t-1} - L_t),\; 0.1,\; 3\big).$$

When the loss falls, $s_t > 1$ and the step grows. When it rises, the step shrinks. The objective is unchanged; only the step size adapts. This places the rule in an old family of heuristics that adapt the learning rate from the recent loss trajectory, such as Jacobs' delta-bar-delta [2] and the "bold driver" schedule [3].

The name is loose. A real reward prediction error compares an outcome to a learned prediction, $\delta = r + \gamma V(s') - V(s)$, while $L_{t-1} - L_t$ compares it only to the previous step. That makes noisy: on a single example, consecutive losses differ mostly by chance. An exponential moving average of the loss, , would be a closer analogue to a prediction and a better signal to try next.

Setup

• Task: reach a goal on a 50 × 50 grid with 15% of cells blocked at random.
• Model: a NumPy MLP, 4 inputs (offset to goal and normalized position), 8 sigmoid hidden units, 4 outputs (one per move).
• Training: supervised, not reinforcement learning. Each state on an episode's path is labeled with the direction toward the goal. Moves are chosen ε-greedily with ε = 0.1.
• Budget: 20,000 episodes per method, base learning rate 0.08, one seed.

Results

Overall, there is no detectable difference. A two-proportion test on 20,000 episodes each gives $z \approx 1.03$, $p \approx 0.31$.

Late in training, there is a gap. On the last 1,000 episodes, $z \approx 2.90$, $p \approx 0.004$. Three things weaken this:

• the window was chosen after looking at the curves;
• consecutive episodes from one training run are not independent samples;
• it comes from a single seed.

So the honest reading is: no overall effect, plus one late-training gap that is worth trying to reproduce and is not yet evidence of anything.

The supervised labels also cap what either method can learn. The target is "move toward the goal," which walks into obstacles, so a perfect learner of these labels still fails often. That likely explains much of why both methods sit near 20%.

Next

1. Run 10 or more seeds per method and report the late-training success rate with a confidence interval.
2. Replace $L_{t-1} - L_t$ with the smoothed version above.
3. Compare against standard adaptive optimizers (Adam, and a plain learning-rate decay) so a positive result means something.
4. Use labels that can be learned perfectly, such as the first move of a shortest path, so the ceiling stops hiding differences.

References

1. Schultz, Dayan and Montague. A Neural Substrate of Prediction and Reward. Science, 1997.
2. Jacobs. Increased Rates of Convergence Through Learning Rate Adaptation. Neural Networks, 1988.
3. Battiti. Accelerated Backpropagation Learning: Two Optimization Methods. Complex Systems, 1989.