Subject: [video-codec] Low-latency rate control constraints

From: Thomas Daede <[email-masked]>
Date: Tue, 21 Jul 2015 13:11:50 +0200

At the Monday NETVC session, it was suggested that CBR mode with a fixed
buffer size was not a very good representation of what rate control for
video conferencing would do.

Are there suggestions of a better model? Keep in mind that this is only
for relatively short (~15s) clips.

Another option is just to not specify CBR bounds, so that this test
would be like the high latency test but with lookahead and 2 pass
constraints.

---

From: Keith Winstein <[email-masked]>
Date: Wed, 22 Jul 2015 01:40:25 -0700

Hello Thomas,

It seems like constraining the bitrate variation by enforcing a limited
receiver-side VBV buffer is kind of an indirect way to get at what you
really care about. Most receivers can probably buffer at least >5 megabytes
(so >5 seconds of video) with no problem. If the document wants to specify
a model for low-latency video, I think the constraint might be better
expressed in terms of latency directly.

Here would be my proposal for an idealized model for what video
conferencing would require from the video coder:

(a) The source video arrives at 30 fps, and each frame is given an "encode
time" of n/30 seconds.
(b) After being encoded, the coded frames are appended to a sender-side
FIFO.
(c) The sender-side FIFO is drained (and delivered to the receiver) at a
particular link rate given in bits per second.
(d) At the receiver, each frame is given a "display time" of the earliest
moment it can be shown to the user given when its coded representation
finishes being delivered, and any rules of the format about frame
reordering (i.e. the presentation timestamp of MPEG). The receiver doesn't
attempt isochronous presentation of the frames; it just shows each frame at
the earliest possible moment.

For the first 8 seconds, the link rate "(c)" is 4 megabits/sec, and the
coder is informed of this. At t=8 seconds, the link rate "(c)"
instantaneously changes to 500 kilobits/sec. The coder learns of this
change at t=8.25 seconds.

The figure of merit is a combination of (1) visual fidelity (e.g. PSNR,
SSIM, etc.) of the coded frames relative to the source frames, and (2) the
maximum difference between the "encode time" and "display time" of any
frame, taken over all the frames in the video.

Best regards,
Keith

On Tue, Jul 21, 2015 at 4:11 AM, Thomas Daede <[email-masked]> wrote:

> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 22 Jul 2015 13:09:45 +0000

I agree encoder latency should be specified directly in low latency testing. IMO, this has nothing to do with bitrates or rate control strategies. It is simply constraining the encoder to zero structural delay when encoding each frame. That is, frame in, frame out, no reordering or lookahead.

Rate control constraints may be necessary in some testing to reflect transport channel constraints. But this should not be conflated with low latency as in zero structural delay.

Mo (as individual)

On 7/22/15, 4:40 AM, video-codec on behalf of Keith Winstein <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Hello Thomas,

It seems like constraining the bitrate variation by enforcing a limited receiver-side VBV buffer is kind of an indirect way to get at what you really care about. Most receivers can probably buffer at least >5 megabytes (so >5 seconds of video) with no problem. If the document wants to specify a model for low-latency video, I think the constraint might be better expressed in terms of latency directly.

Here would be my proposal for an idealized model for what video conferencing would require from the video coder:

(a) The source video arrives at 30 fps, and each frame is given an "encode time" of n/30 seconds.
(b) After being encoded, the coded frames are appended to a sender-side FIFO.
(c) The sender-side FIFO is drained (and delivered to the receiver) at a particular link rate given in bits per second.
(d) At the receiver, each frame is given a "display time" of the earliest moment it can be shown to the user given when its coded representation finishes being delivered, and any rules of the format about frame reordering (i.e. the presentation timestamp of MPEG). The receiver doesn't attempt isochronous presentation of the frames; it just shows each frame at the earliest possible moment.

For the first 8 seconds, the link rate "(c)" is 4 megabits/sec, and the coder is informed of this. At t=8 seconds, the link rate "(c)" instantaneously changes to 500 kilobits/sec. The coder learns of this change at t=8.25 seconds.

The figure of merit is a combination of (1) visual fidelity (e.g. PSNR, SSIM, etc.) of the coded frames relative to the source frames, and (2) the maximum difference between the "encode time" and "display time" of any frame, taken over all the frames in the video.

Best regards,
Keith

On Tue, Jul 21, 2015 at 4:11 AM, Thomas Daede <[email-masked]<mailto:[email-masked]>> wrote:
At the Monday NETVC session, it was suggested that CBR mode with a fixed
buffer size was not a very good representation of what rate control for
video conferencing would do.

_______________________________________________
video-codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/video-codec

---

From: "Arild Fuldseth (arilfuld)" <[email-masked]>
Date: Wed, 22 Jul 2015 16:43:39 +0000

All,

My concern is that mandating CBR and/or allowing for 2-pass encoding with input-dependent GOP/QP structures would encourage people to spend a lot of effort on non-normative encoding techniques. Also, it would be more difficult to determine whether or not any differences is due to normative- or non-normative techniques.

I would recommend that netvc candidate encoders should at least support the same fixed (input-independent) GOP structure as JCT-VC test conditions, and that this should be the primary mode of testing both for low-delay  and high-delay.

This has the additional advantage is that it would be possible to do an apple-to-apple comparison with HM reference software.

Regards,

Arild

From: video-codec [mailto:[email-masked]] On Behalf Of Mo Zanaty (mzanaty)
Sent: 22. juli 2015 15:10
To: [email-masked]; Thomas Daede
Cc: [email-masked]
Subject: Re: [video-codec] Low-latency rate control constraints

Mo (as individual)

Hello Thomas,

Best regards,
Keith

---

From: Yushin Cho <[email-masked]>
Date: Fri, 24 Jul 2015 10:00:56 -0700

Fully agree on Arild's point.

A Video codec w/o rate control would be useless as someone mentioned,
but it will be very hard to evaluate the codec's true coding performance if
different rate controllers are used.
If our test includes CBR mode, I expect that a block level rate control
will be required (probably tuned for subjective quality),
which has dependency on the codec, then a evaluation would hardly avoid the
dependency on the particular rate control algorithm.

-yushin

On Wed, Jul 22, 2015 at 9:43 AM, Arild Fuldseth (arilfuld) <
[email-masked]> wrote:

>  All,
>
>
>
> My concern is that mandating CBR and/or allowing for 2-pass encoding with
> input-dependent GOP/QP structures would encourage people to spend a lot of
> effort on non-normative encoding techniques. Also, it would be more
> difficult to determine whether or not any differences is due to normative-
> or non-normative techniques.
>
>
>
> I would recommend that netvc candidate encoders should at least support
> the same fixed (input-independent) GOP structure as JCT-VC test conditions,
> and that this should be the primary mode of testing both for low-delay  and
> high-delay.
>
>
>
> This has the additional advantage is that it would be possible to do an
> apple-to-apple comparison with HM reference software.
>
>
>
> Regards,
>
>
>
> Arild
>
>
>
> *From:* video-codec [mailto:[email-masked]] *On Behalf Of *Mo
> Zanaty (mzanaty)
> *Sent:* 22. juli 2015 15:10
> *To:* [email-masked]; Thomas Daede
> *Cc:* [email-masked]
> *Subject:* Re: [video-codec] Low-latency rate control constraints
>
>
>
> I agree encoder latency should be specified directly in low latency
> testing. IMO, this has nothing to do with bitrates or rate control
> strategies. It is simply constraining the encoder to zero structural delay
> when encoding each frame. That is, frame in, frame out, no reordering or
> lookahead.
>
>
>
> Rate control constraints may be necessary in some testing to reflect
> transport channel constraints. But this should not be conflated with low
> latency as in zero structural delay.
>
>
>
> Mo (as individual)
>
>
>
> On 7/22/15, 4:40 AM, video-codec on behalf of Keith Winstein <
> [email-masked] on behalf of [email-masked]> wrote:
>
>
>
> Hello Thomas,
>
>
>

-- 
Thanks!
Yushin

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 26 Jul 2015 20:18:47 +0200

+1

Anything different than zero latency is useless for video conferencing.
Obviously no second pass neither, and rate control should be instantly
applied.

>From my practical experience with non-video conferencing encoders I
typically have to set vbv buffers size to exactly one frame based on
current bitrate in order to get a real time behavior.

I would think twice about CBR on a power frame basis, as it could be
helpful for bandwidth estimation algorithms, although redundant/recovery
info may be a better usage of the extra bitrate.

Best regards
Sergio