Subject: [video-codec] Command-line options for encoder comparisons

From: Thomas Daede <[email-masked]>
Date: Wed, 25 Mar 2015 16:19:48 -0700

I am updating the scripts that AreWeCompressedYet uses. I would like to
select what rate control options are used in the comparison. This is for
purposes of absolute comparison between different codecs, and to
indicate when a particular performance point is reached. This is not for
adding tools to an existing codec. There are a couple of different rate
control modes that we can use to compare:

Constant Quantizer:
This is not as simple as it might sound. Should referenced frames still
be allowed a boost in this mode? Should adaptive quantization be
allowed? Can quantization matrices be switched? I tried to come up with
a sane way to compare different codecs with a constant quantizer and was
not successful.

Constant Quality:
Here the codecs are set to use all features available within temporal
constraints. For stored video, this means full lookahead (or two pass,
if that's what the codec uses). For realtime, this means no lookahead,
but otherwise all quality-oriented rate control is allowed. Example rate
control parameters:

VP9 parameters:
vpxenc --codec=vp9 --cq-level=$x --end-usage=q

x264 parameters:
x264 --crf=$x

x265 parameters:
x265 --crf=$x

Constant Bitrate:
This is the hardest category to compare, as it means that all of the CBR
implementations need to be set to have the same buffer constraints. It
is an important use case, so I think we will need to figure out a way to
test it. I'd be okay with adding this one later, though.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 30 Mar 2015 05:16:38 +0200

I know I'm being contrarian....

what's the model you are trying to construct, that we perform
comparisions on?

Is it one where you generate a PSNR/rate graph for each codec, you need
one parameter to vary to make a plot, and you want to have a parameter
that has roughly the same influence on all codecs?

> a sane way to compare different codecs with a constant quantizer and was

While I believe that constant quantizer is useless, I don't see a
problem here. If you need a single set of points, ask that there be a
rule for deriving all the q values from other q values, and model each
rule as a different codec; keep the one that performs best.

In the compare-codecs tool, see the codecs "vp8-mpeg" and "vp8-mpeg-1d"
for examples - one uses three constant quantizers, the other one wraps
the three into one parameter and a formula.

> vpxenc --codec=vp9 --cq-level=$x --end-usage=q

What's the value of this comparision, given that there's no uniformity
among the codecs on what the "cq-level" / "crf" means?

It might work well if we only use it to generate a spread of encoding
results that can be plotted on a bitrate/quality graph, but will not
work at all if we start saying "for x from 15 to 35, do this" - since
there's no uniformity in the range or scaling form for the "quality"
parameter.

> test it. I'd be okay with adding this one later, though.

This is the default mode (ie the one I've been spending the most time
with) on compare-codecs. I'd argue that we should be setting some kind
of constraint on what we're doing (such as "encoding process uses no
more than X MB of RAM), but otherwise leave each codec to optimize
within those constraints - for my tests, I've not put any controls on
buffering strategies so far.

The argument I'd make here is that setting a rate should produce at most
(roughly) that rate - that is, we need to have some means of removing
configurations where the bitrate grows beyond reasonable.

Most codecs (especially in 1-pass) have problems hitting their bitrate
targets on the short test clips; production usage usually assumes that
bitrate can be averaged over longer periods. (Note: I'm not sure this is
actually a reasonable assumption for a DASH environment.)

> 
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: Mohammed Raad <[email-masked]>
Date: Mon, 30 Mar 2015 12:24:46 +0300

It's interesting that you mention a DASH environment at the end - shouldn't
this use case form the basis of a test scenario?

-- 
Mohammed Raad, PhD.
Partner
RAADTECH CONSULTING
P.O. Box 113
Warrawong
NSW 2502 Australia
Phone: +61 414451478
Email: [email-masked]

---

From: Thomas Daede <[email-masked]>
Date: Mon, 30 Mar 2015 10:19:09 -0700

> that has roughly the same influence on all codecs?

Yes, that's the model I was trying to construct. Don't worry about being
contrarian, I don't have any strong opinions about this.

> the three into one parameter and a formula.

Thanks - I looked in compare-codecs before posting, but didn't realize
what those were for until now.

> there's no uniformity in the range or scaling form for the "quality"

Right, I was planning to crop the graphs to a certain bitrate range
afterwards. This would be less important if the rate control was always
precise in hitting the specified file size, which I imagine it is for 2
pass.

Am I correct in assuming that the vp9 cq-level command will give
identical quality to target-bitrate in 2 pass mode?

> buffering strategies so far.

Right, I looked at compare-codecs, but you also allow 2 pass. This is
essentially having an unlimited buffer, and as far as I am aware, is
equivalent to "constant quality". I guess what I was going for in this
section was for realtime streaming where you have to hit a bitrate
target in a very short window with little or no lookahead.

> configurations where the bitrate grows beyond reasonable.

Youtube seems to use libvpx's cq mode, which is constant quality, but
enforces a maximum bitrate over some window. Is that what you're
thinking of? It seems the most reasonable for a DASH-like scenario.