Uniform Smoothing Is Better Than Exponential Smoothing

TensorBoard’s only smoothing option is exponential smoothing.

That’s a shame, because it’s worse than uniform smoothing on the things that actually matter: it’s biased, it’s hard to reason about intuitively, and it throws away your ability to tell signal from noise.

The historical justification for exponential smoothing — its computational efficiency — isn’t actually important anymore: O(k) memory is trivial, and uniform smoothing can be computed in O(n) time overall. This matters far less than the statistical issues exponential smoothing has.

Systemic Bias and Unintuitive Behavior

Consider the following Tensorboard graphs (smoothing = 0.95):

The smoothed signal is systematically biased — in this case biased upward, since the overall metric is trending downward. The faster the signal changes, the stronger the bias, which means it is often severe at the beginning of the curve (since metrics often move extremely rapidly at the beginning of training).

This bias is also not very intuitive — your brain doesn’t readily understand the influence of outliers 100 steps away, and how much (or how little) this affects the current smoothed value.

Which raises the core question: why do you want to be upweighting the most recent iterations? Your model is usually improving at an extremely modest rate batch-to-batch, so the statistical error introduced by batch sizes is likely far more important than capturing the extremely small improvement your model just made.

Uniform Filtering

In contrast, a uniform window is trivial to understand — your smoothed curve is the average of its nearest k neighbors. If your outlier or first few batches are outside the window, they have no influence. If they’re within the window, they have 1/k influence.

Not only is this easy to understand intuitively (and, ultimately, when you’re looking at a graph you’re only reasoning intuitively — you’re not crunching numbers), this means your signal genuinely smooths out faster (for a given effective sample size), since it isn’t upweighting the most recent 1-10 iterations. Since the unweighted sample mean is the best unbiased estimator of a (stationary) mean, this is the most reasonable, default option when the signal is changing slowly (relative to the batch-to-batch statistical noise).

Exponential moving averages only really win when the statistical noise from your batch size is small relative to the movement of the signal. This is plausibly true at the very beginning of training, but it is also precisely when smoothing is not really helpful in the first place — the entire point of smoothing is to reduce statistical error from small sample sizes, so EMA winning when this error is minor isn’t a glowing endorsement.

Let’s compare this image to the graph above:

Just as smooth, but far less biased.

It’s worth noting that the uniform window is still biased, but this bias is locally isolated, which, for a monotonic curve, reduces it. You can also center your windows (i.e. averaging past and future data) and reduce the bias even more (at the cost of “leaking” future measurements, which may or may not be a concern for you). This is a real advantage in the kinds of monotonic curves common in machine learning.

Moreover, a uniform filter gives a more sensible notion of “sample size”, since each point on the smoothed curve is simply an average. This lets us overlay standard errors or 95% confidence intervals, which helps distinguish meaningful motion from statistical noise.

This is important because all types of smoothing make the curve look “smoother” than they should, since your brain is eager to treat the smoothed points as independent when they’re actually highly correlated. This means your intuition for how smooth your loss curve is, is unreliable, and being able to render standard errors on the graph gives you that missing anchor.

It is especially important when comparing multiple runs against each other, since your own intuition about the standard error of your loss metric is likely not good. With a rendered 95% CI you can fiddle with the smoothing parameter until the interval is narrow enough to make a sound comparison (at which point you know precisely how polluted your estimate is with old datapoints). If you’re using exponential smoothing, all you can do is eyeball it.

The reasons exponential smoothing became the default are primarily performance or algorithmic complexity, neither of which matters in the modern world. The fact that it produces biased, unintuitive signals is significantly more important. Uniform smoothing fixes this by reducing bias, being intuitive, and giving you statistical anchors.

The only real downside is Tensorboard doesn’t support it.