Subject: Re: [video-codec] Test sequences, automated and otherwise

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Feb 2015 09:17:10 +0100

Coming in very late on thread....

sequences need to be grouped in multiple dimensions. I'd suggest that
one dimension be:

- Talking heads (still camera, little movement)
- Cinematic material (slow camera moves, dramatic cuts, sudden moves)
- Amateur video (camera shake, sudden moves)
- Artificially generated (no camera shake, color gradients smoother than
normal, no inter-frame movement blur)

Another dimension should be available bandwidth

- 0.1 Mbits, 1 Mbits, 10 Mbits, 100 Mbits

I've come to the conclusion that varying resolution should be considered
one tool for achieving lossy compression, and should not be a primary
dimension - YMMV.
(that said, existing source material has resolutions, and a codec should
make sure to handle all resolutions well.)

A third dimension should be available computing resources for encode and
decode:

- Infinite encode resources available (test style)
- Encode 10x slower than real time (production environments)
- Encode faster than real time (interactive + overhead)

- Decode in real time, stalls acceptable
- Decode in real time, stalls not acceptable

A fourth dimension is about the use of lookahead:

- Any lookahead, including multipass, is acceptable
- Lookahead of a second or two is acceptable
- Lookahead of 100 ms is acceptable
- No lookahead at all is permissible (minimum delay)

I'm running some codec comparisons on compare-codecs.appspot.com (open
source project) - incorporating Derf's clip set is high on my TODO list.

Den 27. jan. 2015 10:17, skrev Mo Zanaty (mzanaty):
> Since a particularly interesting comparison will be HEVC, should we
> consider some JCT-VC standard sequences and operating points?
> 
> We can redefine the classes of sequences to be the most relevant for
> initial experiments. For example:
> 
> Class 1: Video Conferencing
>   a) 360p 30Hz 0.1-1Mbps
>   b) 720p 30Hz 0.2-2Mbps
>   c) 720p 60Hz 0.3-3Mbps
>   d) 1080p 30Hz 0.4-4Mbps
>   e) 1080p 60Hz 0.6-6Mbps
>   f) 2160p 30Hz 1.2-12Mbps
> 
> Class 2: Video Streaming
>   a) 360p 30Hz 0.2-2Mbps
>   b) 720p 30Hz 0.3-3Mbps
>   c) 1080p 24Hz 0.6-6Mbps
>   d) 2160p 24Hz 2-20Mbps
> 
> 1b and 2c may be the most interesting operating points for many.
> 
> We can also consider other classes like professional applications, screen
> content / wireless display, etc. with relatively higher rates.
> 
> 
> Mo
> 
> 
> On 1/26/15, 8:09 PM, Monty Montgomery <[email-masked]> wrote:
> 
> Right, and our use of AWCY and test clips being mostly regression
> testing at the moment (and specific to our particular glass box), I
> wasn't proposing the particulars what we're doing right now as the
> starting point.  Rather, it could be, but used a different way, not as
> we're using it.
> 
> Monty
> 
> On Sun, Jan 25, 2015 at 1:14 PM, Eric Rescorla <[email-masked]> wrote:
>> Following up to myself: what i'm hoping here is to get a consensus list
>> of
>> the domains of interest so that we can then evaluate our progress against
>> them.
>>
>> -Ekr
>>
>>
>> On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla <[email-masked]> wrote:
>>>
>>>
>>> On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery <[email-masked]>
>>> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> Testing discussions are already happening in private and elsewhere.
>>>> The basic points are usually pretty similar and I expect they're going
>>>> to keep coming up, so I'm going to try centralizing it here.
>>>>
>>>> The Daala team is using a small number of fairly short test sequences
>>>> in an automated testing system ('Are We compressed Yet?) mostly as a
>>>> way of doing regression testing.  It's also used to sanity check small
>>>> improvements that are unlikely to be apparent in visual testing.  The
>>>> various graphs we've shown in demos and the like are coming from AWCY.
>>>>
>>>> One point that's been raised several times is that we're not testing
>>>> reasonable or practical bitrates.  Or, rather, that our range includes
>>>> the typical useful rates, but we also go way up to insanely high rates
>>>> no one would ever use.  That's mainly because AWCY is meant as a
>>>> sanity/regression check, and many subtle bugs we've had in the past
>>>> have only popped out at near-lossless quality levels.  Automated
>>>> testing should report these rates, but we don't intend to use them in
>>>> evaluation of relative practical performance.
>>>
>>>
>>> To sharpen this point a little bit, it's probably useful to distinguish
>>> between testing for performance evaluation and testing for regression,
>>> continuous integration etc.
>>>
>>> For the purpose of continuous integration, we should, as you
>>> suggest, test all sorts of parameter sets, including insane ones,
>>> as well as how the decoder handles bogus encodings, etc. For
>>> the purpose of performance evaluation/comparison, I would suggest
>>> that it would be useful to try to define a set of strawman scenarios
>>> which we would use. I'm sure others have different opinions, but
>>> for me the scenarios of most interest are basically videoconferencing,
>>> so roughly speaking:
>>>
>>> - Some set of sizes like:
>>>   - QCIF
>>>   - VGA
>>>   - 1080p
>>>
>>> - 30fps.
>>> - Bit rates on the order of 200kps - 5ish Mbps, depending on the size
>>> (I could be a bit off here).
>>>
>>>
>>> Presumably we want some streaming use cases, but I have a less good
>>> handle on what typical streaming is...
>>>
>>> -Ekr
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Another point that keeps coming up is that we should come up with a
>>>> standard set of sequences and rates for trading and evaluating first-
>>>> and probably second-line evaluations.  Full agreement there, though
>>>> our own frontline tetsing right now is only using a handful of clips
>>>> of a few seconds each (though our available library is much larger).
>>>>
>>>> Monty
>>>>
>>>> _______________________________________________
>>>> video-codec mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/video-codec
>>>
>>>
>>
> 
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
> 
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 25 Feb 2015 06:36:20 -0800

> one tool for achieving lossy compression, and should not be a primary

Can you say a little bit more about how you would go about comparing the 
results of two encoders that chose to encode a sequence (or even 
individual frames within a given sequence) at different resolutions?

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Feb 2015 16:17:24 +0100

> individual frames within a given sequence) at different resolutions?

I think comparision tools that care about what the codecs do rather than
what the result is are problematic - so I'm not sure whether this is
different from all the other choices encoders can make.

The overall process would have to return an image suitable for display -
for simplicity of testing, I would make the output resolution the same
as the input resolution.

We've experimented with this sort of resolution change in the WebRTC
pipeline - encoding in a lower resolution saves both CPU and bandwidth.

(Just for fun, I integrated such a rescaling into the compare-codecs
pipeline that encodes using the H.261 encoder, to allow comparing H.261
with its very limited available resolutions to other codecs on HD-sized
clips - it "works", for some value of "work".

psnr values of 35 dB where x264 achieves 40 dB - it seems psnr isn't
particularly sensitive to the resulting blurriness).

---

From: Thomas Daede <[email-masked]>
Date: Wed, 25 Feb 2015 09:39:56 -0800

> normal, no inter-frame movement blur)

Rather than "Artificially Generated", I would suggest the following
categories:
 - Screensharing - 10fps
 - Video game streaming - 60fps, 1080p

I think it's okay for sequences like the Blender open movie projects to
be in cinematic material. I'm also not totally sure it makes sense to
separate cinematic and amateur video. I think it is unlikely that any
video hosting site is going to try to treat the two differently.

The problem with all of these dimensions is that we now have 5*4*3*2*4 =
480 different combinations to try. I think we should probably limit
ourselves to sensible combinations here.

---

From: Thomas Daede <[email-masked]>
Date: Wed, 25 Feb 2015 09:44:39 -0800

> consider some JCT-VC standard sequences and operating points?

After looking at the JCT-VC standard sequences, I don't feel that they
are sufficient for our testing. However, I do think that they are a good
way to compare to the published results of other standards, so we should
include them as a category.

The JCT-VC test set only includes three clips for video conferencing,
which I don't think is sufficient. It also specifies a intra frame
period of 1 second, which is not reasonable.

It does not feature any 2160p test sequences either, which is a bit strange.

---

From: Yushin Cho <[email-masked]>
Date: Wed, 25 Feb 2015 10:18:51 -0800

> ourselves to sensible combinations here.
Why do we multiply the # of cases of each category, instead of adding?
I believe that encoder will encode video clips from each category
independently to other categories.

Thanks,
Yushin

---

From: Yushin Cho <[email-masked]>
Date: Wed, 25 Feb 2015 10:27:42 -0800

I think, for all video *encoding* testing, we want to assume the input and
output resolution will be identical.
(One possible exception I can think of is : If we ever support a resolution
scalability,
then lower resolution video can be additionally decoded from the same coded
bitstream.)

Meanwhile, the encoder designers can do whatever they want inside encoder
and possibly its paired decoder,
for example encode at down sampled resolution then decode, upsample to
original input resolution,
and add pepper and salt noises on top of it.

-yushin

-- 
Thanks,
Yushin

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 25 Feb 2015 12:05:28 -0800

> particularly sensitive to the resulting blurriness).

Yes, it's well-known that PSNR loves low-passing. It's not the only 
metric that's going to have these problems. FastSSIM will probably be 
similarly blind. Fixable problems, maybe, but I don't want to get in the 
business of designing my own metrics. I'm not even sure there's good 
data on human preferences for when one should downsample, but I haven't 
spent any time looking.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 25 Feb 2015 21:39:25 +0000

This is perhaps getting into charter bashing, but I think we will need
some early milestone (close to requirements) for an evaluation criteria
document that represents the workgroup consensus on comparative testing
methodology and selection of solution candidates or specific tools. The
set of test sequences will only be one small part of that. Metrics will be
a very important part of that. While I agree designing new metrics should
probably be beyond the scope of proposed deliverables, I think we likely
need a thorough evaluation and discussion of various metrics and reach
some consensus on how proposed solutions and tools will be measured and
adopted.

Mo

On 2/25/15, 3:05 PM, Timothy B. Terriberry <[email-masked]> wrote:

---

From: Thomas Daede <[email-masked]>
Date: Wed, 25 Feb 2015 14:11:16 -0800

To start the discussion, here is a brief overview of the four metrics we
currently use in Daala. The reference code is in the tools/dump_*.c
files in the Daala repository. Note that all of these metrics are
applied to the luma plane only.

## PSNR

PSNR is a traditional signal quality metric, measured in decibels. It is
directly drived from mean square error (MSE), or its square root (RMSE).
The formula used is:

20 * log10 ( MAX / RMSE )

or, equivalently:

10 * log10 ( MAX^2 / MSE )

which is the method used in the dump_psnr.c reference implementation.

## PSNR-HVS-M

The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the image,
weights the coefficients, and then calculates the PSNR of those
coefficients. Several different sets of weights have been considered.
The weights used by the dump_pnsrhvs.c tool have been found to be the
best match to real MOS scores.

## SSIM

SSIM (Structural Similarity Image Metric) is a still image quality
metric introduced in 2004. It computes a score for each individual
pixel, using a window of neighboring pixels. These scores can then be
averaged to produce a global score for the entire image. The original
paper produces scores ranging between 0 and 1.

For the metric to appear more linear on BD-rate curves, the score is
converted into a nonlinear decibel scale:

-10 * log10 (1 - SSIM)

## Fast Multi-Scale SSIM

Multi-Scale SSIM is SSIM extended to multiple window sizes. This is
implemented by downscaling the image a number of times, and computing
SSIM over the same number of pixels, then averaging the SSIM scores
together. The final score is converted to decibels in the same manner as
SSIM.