Subject: Re: [video-codec] Test sequences, automated and otherwise

From: Thomas Davies <[email-masked]>
Date: Mon, 02 Mar 2015 10:52:44 +0000

The "Class A" JCT-VC sequences were originated at 4k or 8k and cropped 
down for reasons of compute time. The 8k-originated ones (Nebuta and 
SteamLocomotive) are very noisy and were shot with an early camera and 
sensor. There wasn't much good 4k at the time HEVC started, and it would 
certainly be good to get more.

regards

Thomas

On 25/02/15 17:44, Thomas Daede wrote:
> The JCT-VC test set only includes three clips for video conferencing, 
> which I don't think is sufficient. It also specifies a intra frame 
> period of 1 second, which is not reasonable. It does not feature any 
> 2160p test sequences either, which is a bit strange. 
> _______________________________________________ video-codec mailing 
> list [email-masked] 
> https://www.ietf.org/mailman/listinfo/video-codec

---

From: Mohammed Raad <[email-masked]>
Date: Mon, 2 Mar 2015 15:25:00 +0200

Hi,

Can you provide some data regarding how well these metrics correlate with
subjective measurements? I expect each will be more suitable for different
types of content, but it would be interesting to know how these perform.

BR,

On Thu, Feb 26, 2015 at 12:11 AM, Thomas Daede <[email-masked]> wrote:

> To start the discussion, here is a brief overview of the four metrics we
> currently use in Daala. The reference code is in the tools/dump_*.c
> files in the Daala repository. Note that all of these metrics are
> applied to the luma plane only.
>
> ## PSNR
>
> PSNR is a traditional signal quality metric, measured in decibels. It is
> directly drived from mean square error (MSE), or its square root (RMSE).
> The formula used is:
>
> 20 * log10 ( MAX / RMSE )
>
> or, equivalently:
>
> 10 * log10 ( MAX^2 / MSE )
>
> which is the method used in the dump_psnr.c reference implementation.
>
> ## PSNR-HVS-M
>
> The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the image,
> weights the coefficients, and then calculates the PSNR of those
> coefficients. Several different sets of weights have been considered.
> The weights used by the dump_pnsrhvs.c tool have been found to be the
> best match to real MOS scores.
>
> ## SSIM
>
> SSIM (Structural Similarity Image Metric) is a still image quality
> metric introduced in 2004. It computes a score for each individual
> pixel, using a window of neighboring pixels. These scores can then be
> averaged to produce a global score for the entire image. The original
> paper produces scores ranging between 0 and 1.
>
> For the metric to appear more linear on BD-rate curves, the score is
> converted into a nonlinear decibel scale:
>
> -10 * log10 (1 - SSIM)
>
> ## Fast Multi-Scale SSIM
>
> Multi-Scale SSIM is SSIM extended to multiple window sizes. This is
> implemented by downscaling the image a number of times, and computing
> SSIM over the same number of pixels, then averaging the SSIM scores
> together. The final score is converted to decibels in the same manner as
> SSIM.
>
> On 02/25/2015 01:39 PM, Mo Zanaty (mzanaty) wrote:
> > This is perhaps getting into charter bashing, but I think we will need
> > some early milestone (close to requirements) for an evaluation criteria
> > document that represents the workgroup consensus on comparative testing
> > methodology and selection of solution candidates or specific tools. The
> > set of test sequences will only be one small part of that. Metrics will
> be
> > a very important part of that. While I agree designing new metrics should
> > probably be beyond the scope of proposed deliverables, I think we likely
> > need a thorough evaluation and discussion of various metrics and reach
> > some consensus on how proposed solutions and tools will be measured and
> > adopted.
> >
> > Mo
> >
> > On 2/25/15, 3:05 PM, Timothy B. Terriberry <[email-masked]> wrote:
> >
> > Harald Alvestrand wrote:
> >> psnr values of 35 dB where x264 achieves 40 dB - it seems psnr isn't
> >> particularly sensitive to the resulting blurriness).
> >
> > Yes, it's well-known that PSNR loves low-passing. It's not the only
> > metric that's going to have these problems. FastSSIM will probably be
> > similarly blind. Fixable problems, maybe, but I don't want to get in the
> > business of designing my own metrics. I'm not even sure there's good
> > data on human preferences for when one should downsample, but I haven't
> > spent any time looking.
> >
> > _______________________________________________
> > video-codec mailing list
> > [email-masked]
> _______________________________________________
> video-codec mailing list
> [email-masked]

-- 
Mohammed Raad, PhD.
Partner
RAADTECH CONSULTING
P.O. Box 113
Warrawong
NSW 2502 Australia
Phone: +61 414451478
Email: [email-masked]

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 02 Mar 2015 08:52:47 -0800

Mohammed Raad wrote:
> Can you provide some data regarding how well these metrics correlate
> with subjective measurements? I expect each will be more suitable for
> different types of content, but it would be interesting to know how
> these perform.

There are a number of studies available (on of the advantages of not 
designing our own metrics).

The original PSNR-HVS-M paper [1] conducted experiments using additive 
Gaussian noise and spatially-correlated Gaussian noise (in regions with 
various masking properties), and got the following Spearman correlations 
(Kendall correlations showed similar results):

MS-SSIM:    0.406
PSNR:       0.537
PSNR-HVS-M: 0,984

Of course, Gaussian noise is not terribly representative of compression 
artifacts.

The original Fast SSIM paper [2] used the LIVE image database (JPEG and 
JPEG2000 compression artifacts, white noise, Gaussian blur, and 
fast-fading channel noise), and showed these Spearman rank-order 
correlations:

PSNR:         0.8755
SSIM:         0.9244
MS-SSIM:      0.9425
Fast MS-SSIM: 0.9409

They also compared with the LIVE video database (MPEG 2 compression, 
H.264 compression, H.264 compressed bit streams under packet loss, and 
H.264 compressed bit streams with bit errors):

PSNR:         0.3684
SSIM:         0.4953
MS-SSIM:      0.7593
Fast MS-SSIM: 0.6991

I expect the numbers are so much lower than the image case primarily 
because of the packet loss and bit error conditions, but the authors did 
not break out the numbers by type of distortion, so it is difficult to 
say. Comparing with the numbers below, it also seems clear there is a 
significant temporal effect that none of these still-image metrics can 
capture.

The TID 2008 database (maintained by the authors of PSNR-HVS-M) has 
human rankings for a number of different distortion types, and 
comparisons of various metrics against them can be found [3]:
              JPEG    JPEG 2000  JPEG                 JPEG 2000
                                 transmission errors  t. errors
PSNR         0.899   0.8255     0.7646               0.7769
SSIM         0.8994  0.8888     0.8216               0.8395
MS-SSIM:     0.935   0.9706     0.8747               0.8585

The same authors have a new expanded TID 2013 database, with more 
results [4]. Sadly, they no longer break out results by the individual 
type of distortion, but group them into subsets. Aside from the "Full" 
subset, only the "Actual" subset includes both JPEG and JPEG 2000 
compression artifacts ("Simple" and "Color" also include JPEG 
compression). Only "Exotic" includes transmission errors, along with 
many other distortions completely unlike what would be produced by a 
codec. Spearman rank-order correlations:

             Actual  Simple  Full
PSNR:       0.8246  0.9134  0.6395
SSIM:       0.7877  0.8371  0.6370
MS-SSIM:    0.8871  0.9053  0.7872
PSNR-HVS-M: 0.9175  0.9379  0.6246

The results here are by no means exhaustive. The algorithms here are 
also by no means always the best ones in the studies cited, and new 
studies usually come with a new algorithm that purports to be better. 
There have been a number of these since 2011. But these are the metrics 
we've been using and have some familiarity with.

[1] http://enpub.fulton.asu.edu/resp/vpqm/vpqm2007/papers/399.pdf
[2] http://live.ece.utexas.edu/publications/2011/chen_rtip_2011.pdf
[3] http://hdrvdp.sourceforge.net/reports/2.0/quality_tid2008/index.html
[4] http://ponomarenko.info/tid2013.htm

---

From: Monty Montgomery <[email-masked]>
Date: Mon, 2 Mar 2015 12:59:56 -0800

I agree with Yushin for practical reasons specific to testing.  The
various objective metrics all have somewhat tentative relationships to
actual human perception, and can only be used with awareness of their
limitations.  The primary limitation being the fact that different
classes of artifacts are not penalized consistently, and so the
metrics cannot be used as a black box to compare the relative
worthiness of completely different techniques or codecs.

Although multiscale encoding is undeniably useful, it further
complicates testing by technique into the mix that the objective
metrics are known to fail particularly badly at grading.  They handle
blurring only slightly better than they handle color*.

We may need to account for that and come up with a way to grade it
anyway.  But it will be work, and there's a good case to be made
against designing or modifying our own metrics.  It's a task
empirically demonstrable to be harder than building the codec itself.

Monty

(*which is to say, they don't take chroma into account at all)

---

From: Thomas Daede <[email-masked]>
Date: Mon, 02 Mar 2015 14:19:48 -0800

Do you know if the 1-second keyframe interval was also for reasons of
compute time? (It allows each GOP to be encoded in parallel). It
unfortunately seems a bit short for many streaming applications, and
also prevents rate control from being tested.

At the moment, I run all codecs with rate control on in their constant
quality mode. Perhaps this is not the right thing to do.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 3 Mar 2015 02:19:43 +0000

Harald will probably disagree, but the convention is to disable rate
control for comparing objective metrics. Rate control is normally an
application-specific configuration of the encoder which can add noise or
bias if comparing different rate controls. Rate control strategies can
often be abstracted to apply to any modern codec (with adaptive
quantizers), so it is best to remove this source of noise/bias if you
can¹t guarantee identical (not default) strategies across the comparison.
Note that some GOP structures often include some basic rate controls by
default, like different QP offsets for hierarchical B/P frames. This can
also be a source of noise/bias unless configured identically across the
comparison.

Mo

On 3/2/15, 5:19 PM, Thomas Daede <[email-masked]> wrote:
At the moment, I run all codecs with rate control on in their constant
quality mode. Perhaps this is not the right thing to do.

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Tue, 3 Mar 2015 08:47:25 +0000

No, I suspect it was to be more appropriate for broadcast applications, which usually have more frequent intra for channel switching.

Back in the day, 1/2 second or 12 frame GOPs were deemed essential for MPEG-2 digital TV to mimic analogue switching times, but I think that is long gone - now audio is switched first and then the video when possible, to give an illusion of immediacy. I think low bit rate channels could easily have intra periods of 2 or 3 seconds.

I agree it's short for streaming applications, and could be longer. However, including a decent proportion of intra in the metrics is probably good.

Regards

Thomas

-----Original Message-----
From: Thomas Daede [mailto:[email-masked]] 
Sent: Monday, March 02, 2015 10:20 PM
To: Thomas Davies (thdavies)
Cc: [email-masked]
Subject: Re: [video-codec] Test sequences, automated and otherwise

Do you know if the 1-second keyframe interval was also for reasons of compute time? (It allows each GOP to be encoded in parallel). It unfortunately seems a bit short for many streaming applications, and also prevents rate control from being tested.

At the moment, I run all codecs with rate control on in their constant quality mode. Perhaps this is not the right thing to do.

On 03/02/2015 02:52 AM, Thomas Davies wrote:
> sensor. There wasn't much good 4k at the time HEVC started, and it 
> would certainly be good to get more.
> 
> regards
> 
> Thomas
>

---

From: Mohammed Raad <[email-masked]>
Date: Tue, 3 Mar 2015 11:10:50 +0200

At MPEG we use this to determine the codec performance for the "random
access" case. I don't know why the 1 second interval was chosen and don't
think that its relevant in many use cases, but MPEG codecs are aimed at
satisfying a broad range of applications and so this test case has been
maintained. Members of the potential IETF activity may find that this is an
irrelevant test case.

Regards,

On Tue, Mar 3, 2015 at 10:47 AM, Thomas Davies (thdavies) <
[email-masked]> wrote:

> No, I suspect it was to be more appropriate for broadcast applications,
> which usually have more frequent intra for channel switching.
>
> Back in the day, 1/2 second or 12 frame GOPs were deemed essential for
> MPEG-2 digital TV to mimic analogue switching times, but I think that is
> long gone - now audio is switched first and then the video when possible,
> to give an illusion of immediacy. I think low bit rate channels could
> easily have intra periods of 2 or 3 seconds.
>
> I agree it's short for streaming applications, and could be longer.
> However, including a decent proportion of intra in the metrics is probably
> good.
>
> Regards
>
> Thomas
>
> unfortunately seems a bit short for many streaming applications, and also
> prevents rate control from being tested.
>

---

From: Mohammed Raad <[email-masked]>
Date: Tue, 3 Mar 2015 14:34:27 +0200

Just a note on the JCT-VC sequences, not all will be available for use
outside of MPEG based on the copyright terms under which these sequences
were provided to MPEG.

Regards,

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 15 Mar 2015 11:29:46 +0100

> quality mode. Perhaps this is not the right thing to do.

I don't think that's even possible; when I looked at the VP8 code, I saw
that if you turn on constant quality mode, rate control will disable
itself - the code path for rate control is simply never reached.

Other codecs may have other ways to deal with such a combination of
parameters.

One easy test to see how this works is to encode a clip four times:

1 - Rate A, quality X
2 - Rate B, quality X
3 - Rate A, quality Y
4 - Rate B, quality Y

Most likely, the results will be pairwise equal, and you will then know
which setting overrides the other.

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 15 Mar 2015 11:32:02 +0100

Den 03. mars 2015 03:19, skrev Mo Zanaty (mzanaty):
> control for comparing objective metrics.

Oh no, I completely agree that this is the convention. I just don't
agree that it's useful :-)

 Rate control is normally an

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 15 Mar 2015 11:36:39 +0100

Forking the subject....

Den 03. mars 2015 10:10, skrev Mohammed Raad:
> access" case. I don't know why the 1 second interval was chosen and
> don't think that its relevant in many use cases, but MPEG codecs are
> aimed at satisfying a broad range of applications and so this test case
> has been maintained. Members of the potential IETF activity may find
> that this is an irrelevant test case.

The reasons for having non-initial I-frames that I know of:

- To allow seeking in stored media. To jump to a specific point, you
locate the last I-frame before the point you want to go to and run the
decoder from that point to the point where you really want to go.

With DASH being a prevalent mode of delivering stored media, and each
DASH chunk starting with an I-frame, 5 seconds would be realistic for this.

- To allow recovery from a known good state in streaming media.
For interactive two-way media, you can explicitly ask for an I-frame and
have the headend produce it; for other media, such as broadcast (the
original use case for half-second intervals, as mentioned above), more
complex schemes are needed (I've heard of technologies for storing a
reference frame in some other form and transferring it out of band so
that the decoder can initialize its state and start decoding without an
I-frame, but I don't know if these are used in practice.)

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sun, 15 Mar 2015 13:26:02 +0000

Another reason for testing with periodic intra is the recent rise in RTP mixers which switch rather than transcode media. Setting the intra interval to the average active speaker switching time gives a good estimate of real-world codec efficiency in these topologies. 

Mo

On Mar 15, 2015, at 6:36 AM, Harald Alvestrand <[email-masked]> wrote:

Forking the subject....

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sun, 15 Mar 2015 07:06:42 -0700

> Another reason for testing with periodic intra is the recent rise in RTP mixers which switch rather than transcode media. Setting the intra interval to the average active speaker switching time giv…

Do either you or Harald have some statistics with which to estimate that 
time?

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sun, 15 Mar 2015 16:54:47 +0000

I don't have good stats, but I'm sure it's much longer than all current test sequences, which tend to be only a few seconds due to uncompressed video size constraints. 

It is important to understand intra efficiency during a stream switch, recovery, join, random access, etc. Perhaps we can just test All-Intra configs as a separate data point for this rather than mixing periodic intra. 

Mo

On Mar 15, 2015, at 10:07 AM, Timothy B. Terriberry <[email-masked]> wrote:

> Another reason for testing with periodic intra is the recent rise in RTP mixers which switch rather than transcode media. Setting the intra interval to the average active speaker switching time giv…

Do either you or Harald have some statistics with which to estimate that time?

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 15 Mar 2015 18:47:22 +0100

Hi,

Based on my hands-on-experience, RTP mixer should not depend on periodic 
Intras for working, instead they should send an RTCP FIR/PLI request to 
the new active speaker and wait until the I-Frame is received before 
switching the RTP stream. In fact, in my mcu I configure the encoders to 
not send periodic i frames at all and only send them by request (on 
received RTCP FIR/PLI).

I agree with Harald on the other use cases for non-initial I-frames, 
especially the seeking on storing media one. But I think there could be 
other alternatives that provide viable solutions while not having a big 
impact on quality/bitrate as Iframes have.

BTW, not sure if it is the right place/thread, but as an implementator, 
I find it quite important to be able to tell if a frame is 
iframe/recovery/seekable without to deep diving into the encoded frame 
bitstream.

Best regards
Sergio

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sun, 15 Mar 2015 18:01:47 +0000

Regarding indication of intra frames, draft-berger-avtext-framemarking proposes a header extension for intra and other critical info for codec agnostic switching. It will be discussed in avtext. 

Mo

On Mar 15, 2015, at 1:47 PM, Sergio Garcia Murillo <[email-masked]> wrote:

Hi,

Based on my hands-on-experience, RTP mixer should not depend on periodic Intras for working, instead they should send an RTCP FIR/PLI request to the new active speaker and wait until the I-Frame is received before switching the RTP stream. In fact, in my mcu I configure the encoders to not send periodic i frames at all and only send them by request (on received RTCP FIR/PLI).

I agree with Harald on the other use cases for non-initial I-frames, especially the seeking on storing media one. But I think there could be other alternatives that provide viable solutions while not having a big impact on quality/bitrate as Iframes have.

BTW, not sure if it is the right place/thread, but as an implementator, I find it quite important to be able to tell if a frame is iframe/recovery/seekable without to deep diving into the encoded frame bitstream.

Best regards
Sergio

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sun, 15 Mar 2015 11:14:13 -0700

> Regarding indication of intra frames, draft-berger-avtext-framemarking proposes a header extension for intra and other critical info for codec agnostic switching. It will be discussed in avtext.

Right, naturally most codecs include this up-front (in Daala it is the 
very first bit of the packet). But the other feedback I've received for 
MCUs is that it is often useful to know these things without decrypting 
the stream, which makes the header extension an attractive solution. Of 
course, if we know the header extension will be used, we can possibly 
avoid redundantly transmitting some of that data in the codec.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sun, 15 Mar 2015 19:53:52 +0000

Encrypted streams were the primary driver, especially new proposed architectures where media is encrypted end-to-end with no decryption keys at switching mixers. 

The codec must still have its own intra indication for internal use, since it can't rely on any system layer info like RTP headers. But it doesn't necessarily have to make this easy to parse for non-decoders if there are more efficient ways to signal this internally, while the system layer handles the job of providing easily parsed critical info. 

Mo

On Mar 15, 2015, at 2:14 PM, Timothy B. Terriberry <[email-masked]> wrote:

> Regarding indication of intra frames, draft-berger-avtext-framemarking proposes a header extension for intra and other critical info for codec agnostic switching. It will be discussed in avtext.

Right, naturally most codecs include this up-front (in Daala it is the very first bit of the packet). But the other feedback I've received for MCUs is that it is often useful to know these things without decrypting the stream, which makes the header extension an attractive solution. Of course, if we know the header extension will be used, we can possibly avoid redundantly transmitting some of that data in the codec.

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 15 Mar 2015 22:06:44 +0100

(side note: I *hate* the current setting of Thunderbird where it doesn't
wrap lines before replying - it interferes destructively with Mo's
Outlook settings that send plain text with extremely long lines...
apologies for the broken result.)

> It is important to understand intra efficiency during a stream switch, recovery, join, random access, etc. Perhaps we can just test All-Intra configs as a separate data point for this rather than m…

By "All-Intra", do you mean streams with only I-frames? I think they are
probably too unrealistic to tell us much except the still image
performance. Even adding 3 p-frames after the i-frame will
*significantly* increase compression of the video stream.

But it would indeed be nice to have specific data on how the encoding
efficiency varies with recovery point interval. There should be some
point where the average efficiency is not significantly affected by
regular I-frames, because the I-frame size is small compared to the
accumulated p-frames (but the bumpy bitrate caused by large I-frames
could still cause interesting effects in bandwidth-constrained situations.)

---

From: Mohammed Raad <[email-masked]>
Date: Mon, 16 Mar 2015 09:14:46 +0200

Are there any use-cases besides web streaming and communication that is
being considered for this effort? If people are thinking of using a new
codec for something like digital cinema then an all intra case makes sense.

Mohammed

On Sun, Mar 15, 2015 at 11:06 PM, Harald Alvestrand <[email-masked]>
wrote:

> > I don't have good stats, but I'm sure it's much longer than all current
> test sequences, which tend to be only a few seconds due to uncompressed
> video size constraints.
> >
> > It is important to understand intra efficiency during a stream switch,
> recovery, join, random access, etc. Perhaps we can just test All-Intra
> configs as a separate data point for this rather than mixing periodic intra.
> >
>
> >> Another reason for testing with periodic intra is the recent rise in
> RTP mixers which switch rather than transcode media. Setting the intra
> interval to the average active speaker switching time gives a good estimate
> of real-world codec efficiency in these topologies.
> >

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 16 Mar 2015 12:46:11 +0000

I don't think we should restrict any use case, but the focus should be to optimize performance in common cases.

To clarify my all-intra comment, I did indeed mean a test config of all intra frames (like jct-vc), but the primary purpose of those results would be a separate data point for general intra efficiency in all use cases, not specifically for an all-intra use case, which I agree is rather rare outside some professional applications.

Mo

On Mar 16, 2015, at 3:14 AM, Mohammed Raad <[email-masked]<mailto:[email-masked]>> wrote:

Are there any use-cases besides web streaming and communication that is being considered for this effort? If people are thinking of using a new codec for something like digital cinema then an all intra case makes sense.

Mohammed

On Sun, Mar 15, 2015 at 11:06 PM, Harald Alvestrand <[email-masked]<mailto:[email-masked]>> wrote:

> Mo
>
>
>
> On Mar 15, 2015, at 10:07 AM, Timothy B. Terriberry <[email-masked]<mailto:[email-masked]>> wrote:
>
> [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
video-codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/video-codec

--
Mohammed Raad, PhD.
Partner
RAADTECH CONSULTING
P.O. Box 113
Warrawong
NSW 2502 Australia
Phone: +61 414451478
Email: [email-masked]<mailto:[email-masked]>
_______________________________________________
video-codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/video-codec

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 16 Mar 2015 11:38:44 -0700

Mo Zanaty (mzanaty) wrote:
> To clarify my all-intra comment, I did indeed mean a test config of all
> intra frames (like jct-vc), but the primary purpose of those results
> would be a separate data point for general intra efficiency in all use

I agree, particularly because there are a number of coding tools which 
one would expect to help primarily on intra frames, and it makes it much 
easier to measure their performance when not cluttered by the noise of 
two orders of magnitude more inter frames (though it's still important 
to test with inter frames to make sure they don't have some deleterious 
effect on prediction).

Personally, I think it's better to have a wide variety of still images 
on which to measure intra performance, rather than coding an existing 
video test sequence with all-intra. The main reason being that 
individual frames in a video clip are highly correlated, but the space 
of images is very large. It's a better use of testing resources to use 
uncorrelated images to explore that space as widely as possible. That's 
what informed the construction of the four still-image subsets in 
draft-daede-netvc-testing Section 5.2.