Subject: [video-codec] NETVC encoding/decoding delay

From: Adam Roach <[email-masked]>
Date: Thu, 7 Apr 2016 14:47:58 -0300

[as participant]

In today's meeting, Jesup brought up an issue that Maire had raised at 
our previous session: the section on Video Conferencing (section 2.2 in 
the current version) calls out a "preferable" end-to-end delay value of 
100ms. I think the discussion at today's meeting may have introduced 
some confusion on this point.

To be clear, this section of the document talks about "two-way video and 
audio transmission for communication in real-time." For this use case, 
there is a dramatic drop-off in system usability that starts around 
200ms. Keeping in mind that this data also must traverse a network that 
frequently will add on the order of 100ms or thereabouts, it's pretty 
clear that 100ms encoding/decoding delay per direction is well outside 
of ideal.

I'll also note that specifying 320ms as a "maximum acceptable" would 
result in an experience that is only barely passable even without 
network delay, and likely to become actually unusable for most users 
once you send it across a network:

So, for conversational cases, I would argue that the "maximum 
acceptable" value may be in the 100ms range. The preferable delay is 
clearly much smaller than that.

/a

---

From: Jose Alvarez <[email-masked]>
Date: Thu, 7 Apr 2016 18:07:10 +0000

Dear Mo,

The end-to-end delay of 320 ms value came from section 10.1 of reference [2]<https://www.itu.int/rec/T-REC-G.1091-201410-I/en> Recommendation ITU-T G.1091: Quality of Experience requirements for telepresence services, 2014. Certainly your data shows that this value is not adequate and we should consider adjusting it.

Best Regards,

José Roberto Alvarez
Futurewei Technologies, Inc.
2330 Central Expressway, Santa Clara, CA 95050
Email: [email-masked]
Tel: 408-330-5443 | Mbl: 408-829-5219

----------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI or any of its subsidiaries,
which is intended only for the person or entity whose address is listed above. Any use of the information
contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination)
by persons other than the intended recipient(s) isprohibited. If you receive this e-mail in error, please notify
the sender by phone or email immediately and delete it.

From: Adam Roach [mailto:[email-masked]]
Sent: Thursday, April 07, 2016 10:48 AM
To: [email-masked]
Cc: Randell Jesup; Maire Reavy; Timothy B. Terriberry; Jose Alvarez; Filippov Alexey
Subject: NETVC encoding/decoding delay

[as participant]

In today's meeting, Jesup brought up an issue that Maire had raised at our previous session: the section on Video Conferencing (section 2.2 in the current version) calls out a "preferable" end-to-end delay value of 100ms. I think the discussion at today's meeting may have introduced some confusion on this point.

To be clear, this section of the document talks about "two-way video and audio transmission for communication in real-time." For this use case, there is a dramatic drop-off in system usability that starts around 200ms. Keeping in mind that this data also must traverse a network that frequently will add on the order of 100ms or thereabouts, it's pretty clear that 100ms encoding/decoding delay per direction is well outside of ideal.

I'll also note that specifying 320ms as a "maximum acceptable" would result in an experience that is only barely passable even without network delay, and likely to become actually unusable for most users once you send it across a network:

So, for conversational cases, I would argue that the "maximum acceptable" value may be in the 100ms range. The preferable delay is clearly much smaller than that.

/a

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 7 Apr 2016 18:26:41 +0000

For interactive applications like video conferencing, the standard practice for video codecs is zero structural delay (internal to the codec, ignoring transport and other system delays). I think our requirements should echo this standard practice.

We can cite useful informative references for  acceptable end-to-end delays, but we clearly can't impose or meet any such requirements within our charter since the codec is only one factor in overall system end-to-end delays.

Regards,
Mo (as participant)

On Apr 7, 2016, at 3:07 PM, Jose Alvarez <[email-masked]<mailto:[email-masked]>> wrote:

Dear Mo,

Best Regards,

Jos? Roberto Alvarez
Futurewei Technologies, Inc.
2330 Central Expressway, Santa Clara, CA 95050
Email: [email-masked]<mailto:[email-masked]>
Tel: 408-330-5443 | Mbl: 408-829-5219

To: [email-masked]<mailto:[email-masked]>

[as participant]

/a
_______________________________________________
video-codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/video-codec

---

From: Adam Roach <[email-masked]>
Date: Thu, 7 Apr 2016 15:27:33 -0300

To be clear, the 320 figure from G.1091 is consistent with the data from 
G.114: 320ms is approximately where the the R-value drops below "users 
satisfied." The issue at play here is that both documents (G.1092 and 
G.114) deal with total system end-to-end delay. But this is a total 
number that represents a budget, and the budget must be shared by 
capture, encoding, packetization, network transit, dejittering, 
decoding, and playout. Grabbing the entire budget amount for encoding 
and decoding requires an assumption that the remaining steps will take 0 ms.

/a

On 4/7/16 15:07, Jose Alvarez wrote:
>
> Dear Mo,
>
> The end-to-end delay of 320 ms value came from section 10.1 of 
> reference [2] <https://www.itu.int/rec/T-REC-G.1091-201410-I/en> 
> Recommendation ITU-T G.1091: Quality of Experience requirements for 
> telepresence services, 2014. Certainly your data shows that this value 
> is not adequate and we should consider adjusting it.
>
> Best Regards,
>
> *José Roberto Alvarez*
>
> *Futurewei Technologies, Inc. *
>
> *Email: [email-masked]*
>
> *Tel: 408-330-5443 | Mbl: 408-829-5219*
>
>
> contained herein in any way (including, but not limited to, total or 
> partial disclosure, reproduction, or dissemination)
>
> by persons other than the intended recipient(s) isprohibited. If you 
> receive this e-mail in error, please notify
>
> *From:*Adam Roach [mailto:[email-masked]]
> *Sent:* Thursday, April 07, 2016 10:48 AM
> *To:* [email-masked]
> *Cc:* Randell Jesup; Maire Reavy; Timothy B. Terriberry; Jose Alvarez; 
> Filippov Alexey
> *Subject:* NETVC encoding/decoding delay
>
> [as participant]
>
> our previous session: the section on Video Conferencing (section 2.2 
> in the current version) calls out a "preferable" end-to-end delay 
> value of 100ms. I think the discussion at today's meeting may have 
> introduced some confusion on this point.
>
> To be clear, this section of the document talks about "two-way video 
> and audio transmission for communication in real-time." For this use 
> case, there is a dramatic drop-off in system usability that starts 
> around 200ms. Keeping in mind that this data also must traverse a 
> network that frequently will add on the order of 100ms or thereabouts, 
> it's pretty clear that 100ms encoding/decoding delay per direction is 
> well outside of ideal.
>

---

From: Martin Thomson <[email-masked]>
Date: Thu, 7 Apr 2016 15:57:17 -0300

As a user who frequently enjoys a network delay in the order of 320ms, I
can confirm that this is highly unpleasant.

On 7 April 2016 at 15:27, Adam Roach <[email-masked]> wrote:

> G.114) deal with total system end-to-end delay. But this is a total number
> that represents a budget, and the budget must be shared by capture,
> encoding, packetization, network transit, dejittering, decoding, and
> playout. Grabbing the entire budget amount for encoding and decoding
> requires an assumption that the remaining steps will take 0 ms.
>
> /a
>
>
> The end-to-end delay of 320 ms value came from section 10.1 of reference
> [2] <https://www.itu.int/rec/T-REC-G.1091-201410-I/en> Recommendation
> ITU-T G.1091: Quality of Experience requirements for telepresence services,
> 2014. Certainly your data shows that this value is not adequate and we
> should consider adjusting it.
>
>
>
> Best Regards,
>
>
>
> *José Roberto Alvarez*
>
> *Email: [email-masked] <[email-masked]>*
>
> *From:* Adam Roach [mailto:[email-masked] <[email-masked]>]
> In today's meeting, Jesup brought up an issue that Maire had raised at our
> previous session: the section on Video Conferencing (section 2.2 in the
> current version) calls out a "preferable" end-to-end delay value of 100ms.
> I think the discussion at today's meeting may have introduced some
> confusion on this point.
>
> there is a dramatic drop-off in system usability that starts around 200ms.
> Keeping in mind that this data also must traverse a network that frequently
> will add on the order of 100ms or thereabouts, it's pretty clear that 100ms
> encoding/decoding delay per direction is well outside of ideal.
>
> result in an experience that is only barely passable even without network
> delay, and likely to become actually unusable for most users once you send
> it across a network:
>
>
> So, for conversational cases, I would argue that the "maximum acceptable"
> value may be in the 100ms range. The preferable delay is clearly much
> smaller than that.
>
> /a
>
>
>
> [email-masked]

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Thu, 7 Apr 2016 21:17:55 +0000

I don’t think a time delay is a requirement that a codec can fulfil all on its own, so this requirement could not be directly applied as it stands. What is important is that if you have a system with a specific latency requirement, down to 100ms or below, is that when you’ve worked out what contribution to the latency you can allow a codec to have, you can readily design an encoder to meet that. For low delay that means no reordering of pictures, but that is trivial for a conventional hybrid codec. More importantly I think it means supporting bitstream features that allow for fine-grained rate control so that you can meet really tight buffer constraints with single-pass encoding.

In practice I think that will mean supporting some kind of sub-frame quantisation method so you can hit a bit rate target for a frame or even parts of a frame. We probably want such mechanisms to allow for visual optimization and region-of-interest coding too.

Best regards

Thomas

From: video-codec [mailto:[email-masked]] On Behalf Of Martin Thomson
Sent: Thursday, April 07, 2016 7:57 PM
To: Adam Roach
Cc: Filippov Alexey; [email-masked]; Maire Reavy; Timothy B. Terriberry; Randell Jesup; Jose Alvarez
Subject: Re: [video-codec] NETVC encoding/decoding delay

As a user who frequently enjoys a network delay in the order of 320ms, I can confirm that this is highly unpleasant.

On 7 April 2016 at 15:27, Adam Roach <[email-masked]<mailto:[email-masked]>> wrote:
To be clear, the 320 figure from G.1091 is consistent with the data from G.114: 320ms is approximately where the the R-value drops below "users satisfied." The issue at play here is that both documents (G.1092 and G.114) deal with total system end-to-end delay. But this is a total number that represents a budget, and the budget must be shared by capture, encoding, packetization, network transit, dejittering, decoding, and playout. Grabbing the entire budget amount for encoding and decoding requires an assumption that the remaining steps will take 0 ms.

/a

Best Regards,

José Roberto Alvarez

[as participant]

/a

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 11 Apr 2016 03:37:59 +0200

Den 07. april 2016 23:17, skrev Thomas Davies (thdavies):
> I don’t think a time delay is a requirement that a codec can fulfil all
> on its own, so this requirement could not be directly applied as it
> stands. What is important is that if you have a system with a specific
> latency requirement, down to 100ms or below, is that when you’ve worked
> out what contribution to the latency you can allow a codec to have, you
> can readily design an encoder to meet that. For low delay that means no
> reordering of pictures, but that is trivial for a conventional hybrid
> codec. More importantly I think it means supporting bitstream features
> that allow for fine-grained rate control so that you can meet really
> tight buffer constraints with single-pass encoding.

I think this hits the nail on the head.
"No forward references" = "zero structural delay", but in addition the
*implementation* of the codec has to be capable of meeting a budget
constraint.

One limit is that the encoder has to have enough CPU or hardware to
encode one frame in less than one frame interval under ~all
circumstances. Another one is that it's able to clock it out on the wire
in less than one frame interval.

> 
>  
> 
> In practice I think that will mean supporting some kind of sub-frame
> quantisation method so you can hit a bit rate target for a frame or even
> parts of a frame. We probably want such mechanisms to allow for visual
> optimization and region-of-interest coding too.

I tried implementing a measurement tool for delay imposed by bufffering
into a limited-bandwidth pipe (accounting for the delay imposed by, for
instance, over-large i-frames) as part of compare-codecs.appspot.com. It
showed that for commercially available codecs, this wasn't a big issue -
it's a good tool to have in the toolbox, though.

> 
>  
> 
> Best regards
> 
>  
> 
> Thomas
> 
>  
> 
>  
> 
> *From:*video-codec [mailto:[email-masked]] *On Behalf Of
> *Martin Thomson
> *Sent:* Thursday, April 07, 2016 7:57 PM
> *To:* Adam Roach
> *Cc:* Filippov Alexey; [email-masked]; Maire Reavy; Timothy B.
> Terriberry; Randell Jesup; Jose Alvarez
> *Subject:* Re: [video-codec] NETVC encoding/decoding delay
> 
>  
> 
>     telepresence services, 2014. Certainly your data shows that this
>     value is not adequate and we should consider adjusting it.
> 
>      
> 
>     Best Regards,
> 
>      
> 
>     *José Roberto Alvarez*
> 
>     *Email: [email-masked]
>     <mailto:[email-masked]>*
> 
>     *To:* [email-masked] <mailto:[email-masked]>
>     *Cc:* Randell Jesup; Maire Reavy; Timothy B. Terriberry; Jose
>     Alvarez; Filippov Alexey
>     In today's meeting, Jesup brought up an issue that Maire had raised
>     at our previous session: the section on Video Conferencing (section
>     2.2 in the current version) calls out a "preferable" end-to-end
>     delay value of 100ms. I think the discussion at today's meeting may
>     have introduced some confusion on this point.
> 
>     network that frequently will add on the order of 100ms or
>     thereabouts, it's pretty clear that 100ms encoding/decoding delay
>     per direction is well outside of ideal.
> 
> [email-masked] <mailto:[email-masked]>

---

From: Filippov Alexey <[email-masked]>
Date: Fri, 15 Apr 2016 13:49:33 +0000

Dear Colleagues,

First of all, I’d like to thank all of you for your comments and excuse me for a delay with my reply to them.

I’d like to note that currently, the end-to-end delay is specified in the requirements draft, i.e. it includes all the stuff listed by Mr. Roach (capture, encoding, packetization, network transit, dejittering, decoding, and playout). Why? I guess that we should give codec developers as much freedom as possible. What do I mean? For example, there are, at least, two ways of how a bit-stream produced by a video codec can be protected:
- The codec itself can provide some packet loss protection mechanisms (e.g., redundant slices or an embedded error concealment mechanism).
- The bit-stream can be protected using transport-level mechanisms such as FEC or packer resending.
So, the codec developers should make a decision by themselves what of the ways is preferable for their codec. Thus, our requirement is to keep the total end-to-end delay less that the specified values. However, the way of how to perform the redistribution between different components of this delay is the task of codec developers. On the other hand, our document is codec-related, and we could recommend the codec developers to use some mechanisms that guarantee the support of low delay configuration (e.g., fine-grained rate control mentioned by Mr. Davies).
Thus, I propose to keep both variants in the requirements draft:
1. We add mechanisms proposed by experts and able to guarantee low delay.
2. We keep the maximum and preferable delay values for the low-delay configuration so that the codec developers could have a chance to convince the NETVC committee that their codec integrated into the system will meet end-to-end delay requirement for all the described use cases.

The maximum value of the end-to-end delay specified in the requirements draft can be reduced up to 280 ms that corresponds to “user satisfied”

--
Best regards,
Alexey Filippov

From: Thomas Davies (thdavies) [mailto:[email-masked]]
Sent: Friday, April 08, 2016 12:18 AM
To: Martin Thomson; Adam Roach
Cc: Filippov Alexey; [email-masked]; Maire Reavy; Timothy B. Terriberry; Randell Jesup; Jose Alvarez
Subject: RE: [video-codec] NETVC encoding/decoding delay

Best regards

Thomas

/a

Best Regards,

José Roberto Alvarez

[as participant]

/a