Subject: [video-codec] An opensource project for codec comparision

From: Harald Alvestrand <[email-masked]>
Date: Fri, 09 Jan 2015 15:20:26 +0100

Part of the problem of figuring out what video codecs are fit for what
purpose is measurements.
Unless we can measure something, it's very hard to know if we have
achieved it.

We know that the problem of measuring something that relates to the
quality of video encoding is hard.
But we think that it's worth the effort to make an open-source platform
that lets us make the
measurements we can, and - by virtue of being open source - allows us to
at least agree on what
we are measuring.

Such a platform can never be anything but a work-in-progress, but this
seems as good a time as any
for calling attention to the fact that it's been made public.

The platform we've made is available here:

https://github.com/google/compare-codecs

The results of the test runs generated by the platform are available here:

http://compare-codecs.appspot.com/

Comments are most welcome!

Harald

-- 
Surveillance is pervasive. Go Dark.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 09 Jan 2015 06:35:03 -0800

> Comments are most welcome!

I'm very glad to see something like this. A couple of questions:

1. Are you planning on adding any objective quality metrics besides PSNR?

2. The Bjontegaard calculation appears to fit a single cubic to an 
arbitrary number of points. When comparing small changes using many 
sample points, this will be highly sensitive to noise at very low and 
very high qualities (i.e., exactly the least interesting places). The 
following JCTVC document discusses the issue: 
http://phenix.it-sudparis.eu/jct/doc_end_user/documents/6_Torino/wg11/JCTVC-F270-v1.zip

One proposal in that document is to use Piecewise Cubic Hermite 
Polynomial Interpolation (PCHIP)... there's Matlab code at 
<http://www.ligo-wa.caltech.edu/~cheryl.vorvick/etmx2007/OldStuffFromPCLaptop/MATLAB6p1/toolbox/matlab/polyfun/pchip.m> 
that can be used for validation. Although, if there are many sample 
points in a curve, even piecewise linear interpolation can provide a 
more accurate result than a single cubic.

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 09 Jan 2015 16:12:35 +0100

> 1. Are you planning on adding any objective quality metrics besides PSNR?

Yes, definitely. The requirement is that the measurement can be made by
an opensource tool that we can integrate (compatible license). Your SSIM
tool should fit the bill :-)

Even with PSNR, there are a number of tweaks I'd like to make visible:
per-Y/U/V PSNR, median per-frame PSNR vs overall clip PSNR, for instance
... these things tend to point out "odd" results.

That's why the results component is represented as a key/value set :-)

> more accurate result than a single cubic.

The "Size AVG" column is in fact piecewise linear interpolation; the
reason for giving both is that it gives some place to check if BD-PSNR
has gone haywire (really BD-Rate, which is the metric given).

I've wondered about the choice of cubic for BD-PSNR; the datasets at the
moment are all a maximum of 4 points, so cubic (dimension = points - 1)
seems appropriate, but with more points, cubic seems like a more random
choice. The paper actually suggests going with 8 points and a degree-6
polynomial as one possible approach.

Another thing I would like to do is to include a side-by-side clip
player into the site, so that we could actually show the clips; this
could be the basis for adding some subjective evaluation into the mix -
never a really bad thing. But of course that bloats the storage
requirements by quite a large factor!

> 
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 09 Jan 2015 07:23:53 -0800

> has gone haywire (really BD-Rate, which is the metric given).

Ah, I'd missed that, thanks.

> polynomial as one possible approach.

Well, the problem with higher degree polynomials is that you run into 
precision problems. Degree 6 is about the highest you can fit into 
64-bit doubles with any usefulness, but even that has issues sometimes. 
If you need more degrees of freedom to describe something, piecewise 
interpolation seems the safer route.

> requirements by quite a large factor!

Perhaps combined with the median per-frame idea, you could just show an 
image or two. Not as good as video, of course, but a lot smaller.