Subject: [video-codec] A few comments on draft-midtskogen-netvc-chromapred

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 20 Jul 2016 02:58:34 -0700

How useful are very large values of "a"?

With a range of +/- 2**23 in Q16, it seems you can scale luma values by 
as much as 128. As a result, when subsampling the prediction for 4:2:0, 
you require independent quantization and clipping steps for all four 
input pixels, and then a second quantization and rounding step for the 
average. If you restricted the range of "a" to +/- 2**21, you could 
still expand luma by a factor of 32, but introduce less rounding noise 
into the averaging process in most cases, and clip less often. Since 
that also removes the non-linear operations applied to each individual 
value of yr(i, j), it lets you save 3 of the 4 multiplies, too, i.e.,

c(i, j) = clip(
     a*(yr(2*i, 2*j) + yr(2*i+1, 2*j) + yr(2*i, 2*j+1) + yr(2*i+1, 2*j+1)
     + b >> (16 + 2)).

In a similar vein, it seems possible to use a scaled y'(i, j) (i.e., 
without the final right shift by two) without overflowing an unsigned 
32-bit integer for any of the sums in Figure 3, assuming 8-bit input and 
at most 64x64 blocks. All you'd have to do is remove the "+ 2" in the 
shift at the end of the above equation.

For 10-bit input you would still have to downscale in your average to 
keep these intermediate calculations in 32-bit, and 12-bit input already 
would not fit.

It also seems like the prediction is biased, since the calculation of 
c(i, j) from yr(i, j) does not include a rounding offset. Simply adding 
(1 << 15) to "b" before clipping should remove this bias. Correctly 
rounding the division used to compute "a" would also slightly reduce the 
error, but that effect should be much smaller (and rounding in the 
calculation of SSyy, SScc, and SSyc is not helpful).

Just as an editorial note, I think there are a couple of mistakes in the 
formulas in Section 4.

I think you meant SSyc = YCsum - ((Ysum * Csum) >> 2*log2(N))
(instead of SSyc = YCsum - ((YCsum * YCsum) >> 2*log2(N))).

And then also b = ((Csum << 16) - a*Ysum) >> 2*log2(N)
(instead of b = ((Csum << 16) - a*YCsum) >> 2*log2(N)).