Guide

VMAF Mean, Harmonic Mean, 1% Low and QBVE Compared

By Dirk Hildebrandt, CTO of Wavelet Beam

VMAF scores every single frame. To compare encoders you need one number per stream, and how you pool the frame scores decides what you see. This guide compares the common pooling methods with one worked example and shows which one exposes the quality drops viewers actually notice.

Why pooling matters

A 10 minute video at 50 fps has 30,000 frames and therefore 30,000 VMAF values. Encoder shootouts, bitrate ladder decisions and convex hull calculations all need a single value per stream. The standard choice is the arithmetic mean. The problem: a viewer does not experience an average. A few seconds of visible blocking in an action scene are noticed, no matter how perfect the rest of the stream is.

A worked example

Take three encodes of the same 30,000 frame video. All frames score VMAF 94, except for a number of frames in a demanding scene that collapse to VMAF 72:

• Stream A: no collapse, all frames at 94.
• Stream B: 300 frames, 6 seconds, collapse to 72.
• Stream C: 1,364 frames, about 27 seconds, collapse to 72.

Harmonic mean as computed by libvmaf: N divided by the sum of 1 / (VMAF + 1), minus 1.

Arithmetic mean

The mean of Stream C drops by just 1 point, from 94 to 93, although almost half a minute of the video is clearly damaged. On a typical results sheet, 93 still reads as excellent. The mean blends the drops with the high-performing frames, and static, perfect scenes statistically mask the failures in high-motion scenes. There is a second problem: no systematic error intervals have been published for VMAF, so a delta of 1 point between two encoders is hard to judge at all.

Harmonic mean

The harmonic mean weighs low values a little more, which is why it is often recommended as the better average. In the example it moves from 93.00 to 92.72 for Stream C. That is more honest, but still far too small a change to reflect 27 seconds of visible artifacts. It remains an average of all frames.

Minimum and 1% low

The minimum VMAF and the "1% low", the score of the worst one percent of frames, were introduced to spot exactly these drops. They show how deep a dip was. They do not show how often and how long quality dropped: Streams B and C get identical values, although C is damaged more than four times longer. Constant fluctuations of quality make an image look restless, an effect we call quality pumping. Neither minimum nor 1% low can quantify it.

QBVE: failure energy below a floor

QBVE (Quadratic Bad VMAF Energy) only counts frames below a quality floor T and squares their distance to it. It works like a gated MSE: frames above the floor contribute nothing, so excellent frames cannot hide a failing one. Squaring makes a deep drop weigh far more than a small fluctuation, and summing over all frames makes the duration count. In the example, QBVE separates all three streams clearly: 0, 4.41 and 20.05.

For high-end UHD content in HEVC we use a floor of 93. Lower resolutions show different artifact perception, so the lower the resolution, the lower the threshold: for 1080p we assume an initial threshold of 72. The floor can be adapted, as long as it stays the same within one test series.

Recommendations for practice

• Report the mean, but never decide on the mean alone.
• Add a measure of failures: at least the share of frames below a floor, better QBVE.
• Use the same floor and the same reference across the whole test series.
• If denoising is involved, measure the denoised workflow against the denoised source. Against a noisy original, removed noise counts as an error. See why denoise before encoding.
• Use failure based metrics to steer encoding. In per-shot encoding, QBVE works as a control instance instead of a brute force search.