Subject: [rtcweb] More H.264 vs VP8 tests

From: Bo Burman <[email-masked]>
Date: Sat, 22 Jun 2013 13:41:08 +0000

Hi all, 

We have had a look at Google's comparison between VP8 and H.264 constrained baseline that was posted on April 3rd (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This post contains, as the one mentioned above (and if the attachments make it to the list), information on the exact tools and options used for encoding and should thus be repeatable by anyone interested.

As was already stated by others on this list, one major problem is that Google's test involves the rate control mechanism. Typically codecs are measured with rate control turned off, since it acts as a huge noise on the measurement. Instead we propose to compare the codecs using fixed qp-levels. The qp-level is the quantization parameter that affects the rate/distortion tradeoff. Comparing using fixed qp-levels is what has been used when benchmarking HEVC against H.264 in the JCT-VC standardization, for instance. We are going to select a codec (essentially bit stream format), not a rate control mechanism: Once the codec is selected you can choose whatever rate control mechanism you wish.

We used Google's excellent framework as the baseline and changed the parameter settings in order to make it possible to measure using fixed qp. We used the same sequences, but limited them to the first 10 seconds since they varied from 10 seconds to minutes; this also eased computation time.

We used two H.264 encoder implementations: X264, which is an open-source codec that can operate in everything from real-time to slow, and JM which is the reference implementation that was used to develop H.264. JM is very slow but attempts to be very efficient in terms of bits per quality. The results were as follows:

X264 baseline vs VP8: H.264 wins with 1%
JM baseline vs VP8: H.264 wins with 4%

Running times: 
X264: 1 hour 3 minutes
VP8: 2 hours 0 minutes
JM: order of magnitude slower

It is interesting to note that the measurements are more stable in the new test; the variance of the percentages for the sequences is now around 70, down from around 700 in Google's test of April 3rd.  We believe this is due to the removal of the rate controller, which acts like noise on the measurements. 

We also tried setting H.264 to constrained high (no interlace and no B-pictures, compared to high). The results were then:

X264 constrained high vs VP8: H.264 wins with 25%
JM constrained high vs VP8: H.264 wins with 24%

We also note that the script that Google provided to calculate the rate differences ("BD-rate") does not give exactly the same numbers as the JCT-VC-way of calculating BD-rate. The main difference is that the JM score for constrained high is better (around 29%) if the JCT-VC way of calculating BD-rate is used. 

In summary we think that proper testing can conclude that there is no clear performance advantage to any codec between VP8 and H.264 baseline. When comparing VP8 against H.264 constrained high on the other hand, it seems like there is an advantage for H.264 constrained high.

The attached file includes the files necessary to reproduce the test.

Best Regards,

Bo Burman

---

From: Leon Geyser <[email-masked]>
Date: Sat, 22 Jun 2013 18:54:21 +0200

Hi Bo,

Thanks for the additional testing done.

I am not 100% why we have to test with fixed qp-levels when WebRTC is going
to be used over the internet. I would assume that WebRTC would be used over
the internet...
My understanding of rate-control might be completely wrong. Why test modes
that might not even be used in real-world scenarios?

On 22 June 2013 15:41, Bo Burman <[email-masked]> wrote:

> Hi all,
>
> We have had a look at Google's comparison between VP8 and H.264
> constrained baseline that was posted on April 3rd (
> http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This
> post contains, as the one mentioned above (and if the attachments make it
> to the list), information on the exact tools and options used for encoding
> and should thus be repeatable by anyone interested.
>
> As was already stated by others on this list, one major problem is that
> Google's test involves the rate control mechanism. Typically codecs are
> measured with rate control turned off, since it acts as a huge noise on the
> measurement. Instead we propose to compare the codecs using fixed
> qp-levels. The qp-level is the quantization parameter that affects the
> rate/distortion tradeoff. Comparing using fixed qp-levels is what has been
> used when benchmarking HEVC against H.264 in the JCT-VC standardization,
> for instance. We are going to select a codec (essentially bit stream
> format), not a rate control mechanism: Once the codec is selected you can
> choose whatever rate control mechanism you wish.
>
> We used Google's excellent framework as the baseline and changed the
> parameter settings in order to make it possible to measure using fixed qp.
> We used the same sequences, but limited them to the first 10 seconds since
> they varied from 10 seconds to minutes; this also eased computation time.
>
> We used two H.264 encoder implementations: X264, which is an open-source
> codec that can operate in everything from real-time to slow, and JM which
> is the reference implementation that was used to develop H.264. JM is very
> slow but attempts to be very efficient in terms of bits per quality. The
> results were as follows:
>
> It is interesting to note that the measurements are more stable in the new
> test; the variance of the percentages for the sequences is now around 70,
> down from around 700 in Google's test of April 3rd.  We believe this is due
> to the removal of the rate controller, which acts like noise on the
> measurements.
>
> We also tried setting H.264 to constrained high (no interlace and no
> B-pictures, compared to high). The results were then:
>
> We also note that the script that Google provided to calculate the rate
> differences ("BD-rate") does not give exactly the same numbers as the
> JCT-VC-way of calculating BD-rate. The main difference is that the JM score
> for constrained high is better (around 29%) if the JCT-VC way of
> calculating BD-rate is used.
>
> In summary we think that proper testing can conclude that there is no
> clear performance advantage to any codec between VP8 and H.264 baseline.
> When comparing VP8 against H.264 constrained high on the other hand, it
> seems like there is an advantage for H.264 constrained high.
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Mon, 24 Jun 2013 11:33:21 +0000

On 2013-06-22 18:54, Leon Geyser wrote:
> I am not 100% why we have to test with fixed qp-levels when WebRTC is
> going to be used over the internet. I would assume that WebRTC would be
> used over the internet...
> My understanding of rate-control might be completely wrong. Why test
> modes that might not even be used in real-world scenarios?

Hi Leon (responding for Bo who is enjoying his vacation),

you are absolutely right, these codecs are going to be used with 
rate-controllers when deployed in the real world. However, we still do 
not think it is a good idea having rate control turned on when comparing 
them. The reason is that the rate controller influences the quality so 
much that it is very hard to get any comparable signal if you include 
them. Let me give you one example; let's say that you use the same 
codec, but two different rate controllers. Rate control A is very 
liberal with its bits in the beginning of the sequence, whereas rate 
control B is very frugal. On a video conferencing type sequence, where 
the camera is static and not much is happening, rate control A will win 
by a huge margin over rate control B. The reason is that the first frame 
will be encoded very well and this data can be used during the remainder 
of the sequence. Rate control B will not be able to stand a chance, even 
though it has more bits for the remaining images.  And this is with the 
same video codec!

We could of course try to make sure that the two rate controllers have 
similar settings; however, since they are not identical they cannot be 
made to work exactly the same. Even small changes in rate control 
settings could have a huge effect on the end result.

Rather than trying to make two pieces of software work the same, it is 
much easier to just shut off the rate controllers. This gives a direct 
measurement of the relative strengths of the codecs. This is also the 
reason why organizations such as JC-TVC have used this method to compare 
their codecs.

Best Regards,
Stefan

>
> On 22 June 2013 15:41, Bo Burman <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>     constrained baseline that was posted on April 3rd
>     (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html).
>     This post contains, as the one mentioned above (and if the
>     attachments make it to the list), information on the exact tools and
>     options used for encoding and should thus be repeatable by anyone
>     interested.
>
>     As was already stated by others on this list, one major problem is
>     that Google's test involves the rate control mechanism. Typically
>     codecs are measured with rate control turned off, since it acts as a
>     huge noise on the measurement. Instead we propose to compare the
>     codecs using fixed qp-levels. The qp-level is the quantization
>     parameter that affects the rate/distortion tradeoff. Comparing using
>     fixed qp-levels is what has been used when benchmarking HEVC against
>     H.264 in the JCT-VC standardization, for instance. We are going to
>     select a codec (essentially bit stream format), not a rate control
>     mechanism: Once the codec is selected you can choose whatever rate
>     control mechanism you wish.
>
>     parameter settings in order to make it possible to measure using
>     fixed qp. We used the same sequences, but limited them to the first
>     10 seconds since they varied from 10 seconds to minutes; this also
>     eased computation time.
>
>     We used two H.264 encoder implementations: X264, which is an
>     open-source codec that can operate in everything from real-time to
>     slow, and JM which is the reference implementation that was used to
>     develop H.264. JM is very slow but attempts to be very efficient in
>     terms of bits per quality. The results were as follows:
>
>     It is interesting to note that the measurements are more stable in
>     the new test; the variance of the percentages for the sequences is
>     now around 70, down from around 700 in Google's test of April 3rd.
>       We believe this is due to the removal of the rate controller,
>     which acts like noise on the measurements.
>
>     We also note that the script that Google provided to calculate the
>     rate differences ("BD-rate") does not give exactly the same numbers
>     as the JCT-VC-way of calculating BD-rate. The main difference is
>     that the JM score for constrained high is better (around 29%) if the
>     JCT-VC way of calculating BD-rate is used.
>
>     In summary we think that proper testing can conclude that there is
>     no clear performance advantage to any codec between VP8 and H.264
>     baseline. When comparing VP8 against H.264 constrained high on the
>     other hand, it seems like there is an advantage for H.264
>     constrained high.
>
>     [email-masked] <mailto:[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 25 Jun 2013 00:00:11 +0200

First of all, Bo, thanks for running the tests, and for making the 
results fully open!

> though it has more bits for the remaining images.  And this is with the

This would seem to me to be an argument for including the rate 
control.... what we are looking for is the codec where people can 
demonstrate the best performance for our specific situations, not the 
codec that is best able to fulfil a set of conditions that are 
inappropriate for Real Life.

For the example you give, the bit division between the I-frame/key-frame 
and the subsequent frames is not just influenced by the chosen Q value - 
it's also influenced by codec design.

What will happen in a video conference scenario if someone uses this 
rate packing too much is that the I-frame takes a long time to transmit, 
because the available bit rate is in the ramp-up phase; if we wanted to 
make realistic scenario settings, we should probably draw constraints 
around the delay in delivering the second frame under specified 
bandwidth conditions.

But on the other hand, people may find a delay to the second frame quite 
acceptable, and your "rate control A" is indeed the one that is most 
reasonable to use.

> reason why organizations such as JC-TVC have used this method to compare

And it's going to be discussed there.... I'll be late for the Berlin 
IETF meeting because I have to be at the MPEG meeting in Vienna to 
present Google's formal submission of VP8 to MPEG. The use or non-use of 
variable QP will probably be an item of debate .... again.

The IETF has not started down the path of defining test conditions under 
which it wishes to measure codec performance; I think we do have rough 
consensus that the technical differences are not so great as to make a 
decisive difference between the proposed codecs - while the IPR 
differences are all too well known to us, and do matter enough to make a 
decisive difference to a lot of us.

                Harald

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 25 Jun 2013 12:17:26 +0200

Again - thanks for releasing this openly!

I ran the scripts (with a few tweaks; you run on a system where sh is 
bash, not dash, for instance), and got the same numbers within +/- 0.5% 
(probably some binary version skew); we may have disagreements on the 
parameters to use, but we agree on the numbers those parameters produce.

(I have since modified the Google framework to include a script that 
pulls in the sources for the needed binaries and compiles them - if you 
want to make 100% sure people are working from the same sources, you may 
want to rebase to a newer version of the comparision toolkit.)

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 25 Jun 2013 14:56:30 +0000

> acceptable, and your "rate control A" is indeed the one that is most

I think that the rate control should be omitted from this kind of 
comparisons since it adds a lot of noise to the results and since - as 
far as I have understood - the rate controller is rather independent of 
the codec (you could use very similar rate controllers for VP8 and h.264).

But I agree to the main point you make further below, so perhaps this is 
a debate that can be saved for another discussion.

> decisive difference to a lot of us.

I agree, and our tests underline, that the performance differences are 
not so great as to make any decisive difference between the VP8 and 
h.264 baseline - their performance is very similar.

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 25 Jun 2013 14:59:30 +0000

> parameters to use, but we agree on the numbers those parameters produce.

Great. We spent quite some time trying to find the best parameter 
settings, but it is quite complex and we may have missed something - 
feedback is welcome.

> want to rebase to a newer version of the comparision toolkit.)

We'll look into that.

---

From: John Koleszar <[email-masked]>
Date: Tue, 25 Jun 2013 08:36:18 -0700

> JM constrained high vs VP8: H.264 wins with 24%
[...]

At one point, running the libvpx implementation of VP8 in fixed qp
mode would effectively limit the encoder to only a single reference
frame. I don't recall if it's still the case. Another unexpected
behavior for people trying this mode is that making libvpx bump up
against the lower quantizer without setting an approximately correct
target bitrate can also put the encoder into its "over quant" mode,
where it'll throw away additional residual trying to hit the target
bitrate. It doesn't have anything like the x264 ipratio (though I see
you set this to 1, but this is something that bites people trying
fixed-qp). This mode isn't really supported by libvpx, so I'd be
careful about taking these results as representative of the very best
VP8 can do.

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 26 Jun 2013 15:35:08 +0000

Hi,

On Jun 25, 2013, at 0:00, Harald Alvestrand <[email-masked]> wrote:
> This would seem to me to be an argument for including the rate control.... what we are looking for is the codec where people can demonstrate the best performance for our specific situations, not the codec that is best able to fulfil a set of conditions that are inappropriate for Real Life.

I'd agree with this argument *if* we actually had some idea of what congestion control algorithm(s) RMCAT will propose. If we did, looking at congestion-controlled output quality would make a lot of sense.

But, we don't know what RMCAT will recommend, and - with my RMCAT chair hat on - progress is slow, because participation is not exactly stellar. RTCWEB folks with an interest in this topic should start participating in RMCAT. Pretty please.

But, I digressed. Since we don't know what algorithm RMCAT will recommend, comparing different codec/congestion-control pairs makes it very hard to figure out whether quality differences are due to the codec, or due to the congestion controller.

So as a baseline, it would make sense to eliminate the congestion controller from this comparison, at least initially. Once we have some candidate congestion controllers on the table in RMCAT, comparing encoder-controller pairs makes sense, e.g., VP8 with controller A against MPEG with controller A, VP8 with controller B against MPEG with controller B, etc.

Because in the end, as you say, congestion-controlled output quality is what matters in the real world. So IMO any final decision on a codec needs to wait until we know what the congestion controller is doing.

Lars

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 26 Jun 2013 17:41:20 +0000

Rate control is typically a different beast from congestion control, and any coupling is typically very loose. Rate control is primarily adaptive quantization to optimize perceptual quality based on content statistics, with little or no coupling to transport statistics. Any such coupling is often loose, like a target average bit rate over coarse timescales inappropriate for congestion control like seconds or longer. Constraining rate control to also perform congestion control over tight timescales like a frame time will result in suboptimal video quality, because content statistics are inherently uncorrelated with transport statistics, so forcing any coupling degrades perceptual quality. Hence why congestion control is often implemented as a pacer/shaper outside the encoder rate control, with a slower feedback loop to adapt the target rate of the encoder rate control less frequently than congestion controlled packet transmission.

Anyone interested in congestion control or its coupling to rate control should participate in RMCAT.

Mo

On Jun 26, 2013, at 11:35 AM, "Eggert, Lars" <[email-masked]> wrote:

Hi,

Lars

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 26 Jun 2013 17:50:51 +0000

On 2013-06-26 17:35, Eggert, Lars wrote:
>> demonstrate the best performance for our specific situations, not
>> the codec that is best able to fulfil a set of conditions that are
> I'd agree with this argument *if* we actually had some idea of what
> congestion control algorithm(s) RMCAT will propose. If we did,
> looking at congestion-controlled output quality would make a lot of
> sense.
>
> But, we don't know what RMCAT will recommend, and - with my RMCAT
> chair hat on - progress is slow, because participation is not exactly
> stellar. RTCWEB folks with an interest in this topic should start
> participating in RMCAT. Pretty please.
>
> But, I digressed. Since we don't know what algorithm RMCAT will
> recommend, comparing different codec/congestion-control pairs makes
> it very hard to figure out whether quality differences are due to the
> codec, or due to the congestion controller.
>
> So as a baseline, it would make sense to eliminate the congestion
> controller from this comparison, at least initially. Once we have
> some candidate congestion controllers on the table in RMCAT,
> comparing encoder-controller pairs makes sense, e.g., VP8 with
> controller A against MPEG with controller A, VP8 with controller B
> against MPEG with controller B, etc.
>
> Because in the end, as you say, congestion-controlled output quality
> is what matters in the real world. So IMO any final decision on a
> codec needs to wait until we know what the congestion controller is
> doing.

If this was only about video quality I would agree, but as things stand 
now I would say that the data we have available indicate that VP8 and 
h.264 baseline are comparable regarding video quality, and that possible 
MTI selection would come down to other things (like licensing, current 
deployment etc.)

>
> Lars _______________________________________________ rtcweb mailing
> list [email-masked] https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 26 Jun 2013 21:24:16 +0000

On 2013-06-25 17:36, John Koleszar wrote:
>> attachments make it to the list), information on the exact tools
>> and options used for encoding and should thus be repeatable by
>> anyone interested.
>>
>> codecs are measured with rate control turned off, since it acts as
>> a huge noise on the measurement. Instead we propose to compare the
>> parameter that affects the rate/distortion tradeoff. Comparing
>> using fixed qp-levels is what has been used when benchmarking HEVC
>> against H.264 in the JCT-VC standardization, for instance. We are
>> going to select a codec (essentially bit stream format), not a rate
>> control mechanism: Once the codec is selected you can choose
>> whatever rate control mechanism you wish.
>>
>> We used Google's excellent framework as the baseline and changed
>> the parameter settings in order to make it possible to measure
>> using fixed qp. We used the same sequences, but limited them to the
>> first 10 seconds since they varied from 10 seconds to minutes; this
>> also eased computation time.
>>
>> X264 baseline vs VP8: H.264 wins with 1% JM baseline vs VP8: H.264
>> wins with 4%
>>
>> Running times: X264: 1 hour 3 minutes VP8: 2 hours 0 minutes JM:
>> order of magnitude slower
>>
>> We believe this is due to the removal of the rate controller, which
>> acts like noise on the measurements.
>>
>> We also tried setting H.264 to constrained high (no interlace and
>> no B-pictures, compared to high). The results were then:
>>
>> X264 constrained high vs VP8: H.264 wins with 25% JM constrained
>> high vs VP8: H.264 wins with 24%
>>
> [...]
>
> frame. I don't recall if it's still the case.

We have stepped through the decoding of the .webm files produced using 
our parameter settings and they indeed use several reference frames, so 
no, this seems not to be the case.

> Another unexpected
> bitrate.

We have not seen this happen either. The target bitrate we set is 
ridiculously high so we do not see why the encoder should throw away 
anything that would cost bits, rather it would do the opposite.

> It doesn't have anything like the x264 ipratio (though I
> see you set this to 1, but this is something that bites people
> trying fixed-qp). This mode isn't really supported by libvpx, so I'd
> be careful about taking these results as representative of the very
> best VP8 can do. _______________________________________________
> rtcweb mailing list [email-masked]

---

From: Stefan Håkansson LK <[email-masked]>
Date: Thu, 27 Jun 2013 11:55:44 +0000

> On Sat, Jun 22, 2013 at 6:41 AM, Bo Burman <[email-masked]>
...
> where it'll throw away additional residual trying to hit the target

To make sure that this is not an issue we checked the bit rate for a 
sequence compressed with our settings. Then we encoded the same sequence 
three times; once with --target-bitrate set to well below this bit rate, 
once with --target-bitrate set to this bit rate, and once with 
--target-bitrate set to well above this bit rate. In all three cases we 
ended up with exactly the same bit rate and PSNR. In fact the resulting 
.webm files were bit-identical. So I think we can rule out the 
possibility that setting the target bitrate incorrectly can affect the 
results.

Again, we have put a lot of effort into selecting these parameters and 
those for X264/JM, but since this is a complex issue we welcome feedback.

---

From: Stefan Håkansson LK <[email-masked]>
Date: Sat, 29 Jun 2013 13:17:58 +0000

On 6/25/13 12:17 PM, Harald Alvestrand wrote:
> (probably some binary version skew);

I re-run the scripts on another computer with another OS today, and I 
get exactly the same results as Bo sent out. I noted however that if the 
input clips are not cut at 10s (but used in their entire length) the 
results get slightly different, but within +/- 0.5%. Can this be the 
reason why you get slightly different numbers?

> we may have disagreements on the