Subject: [video-codec] Test sequences, automated and otherwise

From: Monty Montgomery <[email-masked]>
Date: Fri, 23 Jan 2015 13:45:02 -0800

Hi everyone,

Testing discussions are already happening in private and elsewhere.
The basic points are usually pretty similar and I expect they're going
to keep coming up, so I'm going to try centralizing it here.

The Daala team is using a small number of fairly short test sequences
in an automated testing system ('Are We compressed Yet?) mostly as a
way of doing regression testing.  It's also used to sanity check small
improvements that are unlikely to be apparent in visual testing.  The
various graphs we've shown in demos and the like are coming from AWCY.

One point that's been raised several times is that we're not testing
reasonable or practical bitrates.  Or, rather, that our range includes
the typical useful rates, but we also go way up to insanely high rates
no one would ever use.  That's mainly because AWCY is meant as a
sanity/regression check, and many subtle bugs we've had in the past
have only popped out at near-lossless quality levels.  Automated
testing should report these rates, but we don't intend to use them in
evaluation of relative practical performance.

Another point that keeps coming up is that we should come up with a
standard set of sequences and rates for trading and evaluating first-
and probably second-line evaluations.  Full agreement there, though
our own frontline tetsing right now is only using a handful of clips
of a few seconds each (though our available library is much larger).

Monty

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Jan 2015 12:57:39 -0800

> evaluation of relative practical performance.

To sharpen this point a little bit, it's probably useful to distinguish
between testing for performance evaluation and testing for regression,
continuous integration etc.

For the purpose of continuous integration, we should, as you
suggest, test all sorts of parameter sets, including insane ones,
as well as how the decoder handles bogus encodings, etc. For
the purpose of performance evaluation/comparison, I would suggest
that it would be useful to try to define a set of strawman scenarios
which we would use. I'm sure others have different opinions, but
for me the scenarios of most interest are basically videoconferencing,
so roughly speaking:

- Some set of sizes like:
  - QCIF
  - VGA
  - 1080p

- 30fps.
- Bit rates on the order of 200kps - 5ish Mbps, depending on the size
(I could be a bit off here).

Presumably we want some streaming use cases, but I have a less good
handle on what typical streaming is...

-Ekr

> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Jan 2015 13:14:23 -0800

Following up to myself: what i'm hoping here is to get a consensus list of
the domains of interest so that we can then evaluate our progress against
them.

-Ekr

---

From: Monty Montgomery <[email-masked]>
Date: Mon, 26 Jan 2015 17:09:51 -0800

Right, and our use of AWCY and test clips being mostly regression
testing at the moment (and specific to our particular glass box), I
wasn't proposing the particulars what we're doing right now as the
starting point.  Rather, it could be, but used a different way, not as
we're using it.

Monty

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 27 Jan 2015 09:17:21 +0000

Since a particularly interesting comparison will be HEVC, should we
consider some JCT-VC standard sequences and operating points?

We can redefine the classes of sequences to be the most relevant for
initial experiments. For example:

Class 1: Video Conferencing
  a) 360p 30Hz 0.1-1Mbps
  b) 720p 30Hz 0.2-2Mbps
  c) 720p 60Hz 0.3-3Mbps
  d) 1080p 30Hz 0.4-4Mbps
  e) 1080p 60Hz 0.6-6Mbps
  f) 2160p 30Hz 1.2-12Mbps

Class 2: Video Streaming
  a) 360p 30Hz 0.2-2Mbps
  b) 720p 30Hz 0.3-3Mbps
  c) 1080p 24Hz 0.6-6Mbps
  d) 2160p 24Hz 2-20Mbps

1b and 2c may be the most interesting operating points for many.

We can also consider other classes like professional applications, screen
content / wireless display, etc. with relatively higher rates.

Mo

On 1/26/15, 8:09 PM, Monty Montgomery <[email-masked]> wrote:

Monty

---

From: Ralph Giles <[email-masked]>
Date: Tue, 27 Jan 2015 10:14:52 -0800

On Tue, Jan 27, 2015 at 1:17 AM, Mo Zanaty (mzanaty) <[email-masked]> wrote:

>   ...

I think 1080p 60Hz at least should be on the list as well. This is
important for game video and other action/interactive use, as your
say, and already deployed by YouTube with current codecs.

 -r

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Sat, 31 Jan 2015 16:05:48 -0500

> say, and already deployed by YouTube with current codecs.
For what it's worth, in defense of this, I've also begun uploading 1080p60 content onto one of the YouTube channels I maintain due to the existing support (source is 1080i60, but looks good and smooth after running yadif=1 on it).
-- 
Libre Video
http://librevideo.org