Subject: [rtcweb] Transcoding

From: John Leslie <[email-masked]>
Date: Sun, 12 Jan 2014 15:56:08 -0500

Karl Stahl <[email-masked]> wrote:
> 
> [1] Transcoding VP8 ?> H.264 is very CPU intensive. Chrome with VP8 on old
> laptops with 1 core 1.5 Ghz x86 is not even usable. Core i5 CPU is fine. 
> 
> Even the master of transcoding, ACME Packet/Oracle said at the last WebRTC
> conference that they will not support transcoding, leaving that to others
> real heavy DSP cabinets.

   (I will write an analysis _after_ the straw poll closes; but it looks
like that analysis will state we should consider transcoding as an
alternative to one-or-more MTI codecs.)

   Transcoding _seems_ to be almost universally rejected; but this is
the first clear description of _why_ it's rejected.

   Clearly, some hardware that we'd like to see participating in webRTC
isn't capable of transcoding at line speed. Thus, I fully agree that we
should reject transcoding as a requirement for _all_ webRTC devices.

   But other hardware will surely be capable of transcoding at line speed.
It might be painful, but we _could_ specify a "negotiate transcoding"
function which decided that one end does all the transcoding -- or that
a middleman does all the transcoding.

   (Of course, part of the pain is the transcoding delay -- but IMHO
folks are pretty tolerant of lip-sync mismatch of several hundred
milliseconds: it's the mismatch of one second or more which is
unbearable.)

   It appears _very_ likely that several browsers will support both
H.264 and VP8 natively; and others will be able to specify a low-
latecy middleman. Thus, we can hope for the normal case to be no
obvious transcoding delay...

   You may flame at will...

--
John Leslie <[email-masked]>

---

From: "cb.list6" <[email-masked]>
Date: Sun, 12 Jan 2014 13:12:30 -0800

> obvious transcoding delay...

I am not a codec expert.

But, it is worth noting that transcoding, to the best of my knowledge,
substantially reduces quality. Any compression creates an approximation.
Transcoding is an approximation of an approximation, which leads to bad
quality.  So it is expensive, complex, and poor quality.

CB
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: cowwoc <[email-masked]>
Date: Sun, 12 Jan 2014 19:23:21 -0500

That hasn't been my experience. I am no codec expert but the transcoding 
quality I've seen was as good as the original (to the naked eye).

My main objection to transcoding is the computational cost of doing so. 
In my experience, it isn't viable to do transcoding in software on 
mobile devices (at almost any resolution) and on desktops you will run 
into problems at 720p or higher. Even if it's doable for a simple P2P 
situation, consider that for an N:N chat, the computational cost goes 
through the roof. In short, transcoding severely limits scalability and 
increases the financial cost of any solution.

Gili

On 12/01/2014 4:12 PM, cb.list6 wrote:
>
>
> On Jan 12, 2014 12:56 PM, "John Leslie" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> > Karl Stahl <[email-masked] <mailto:[email-masked]>> 
> wrote:
> > >
> substantially reduces quality. Any compression creates an 
> approximation. Transcoding is an approximation of an approximation, 
> which leads to bad quality.  So it is expensive, complex, and poor 
> quality.
>
> CB
> > --
> > John Leslie <[email-masked] <mailto:[email-masked]>>
> > [email-masked] <mailto:[email-masked]>

---

From: Ron <[email-masked]>
Date: Mon, 13 Jan 2014 15:36:31 +1030

> quality.  So it is expensive, complex, and poor quality.

That must be why youtube transcodes *everything* uploaded to it.
Clearly they want to provide an expensive, complex, poor quality service
that nobody will ever use.

I'm glad the best of your knowledge has cleared that mystery up for us.

  Crosseyed and painless,
  Ron

---

From: "Cavigioli, Chris" <[email-masked]>
Date: Mon, 13 Jan 2014 11:02:54 +0000

Yes, transcoding consumes significant compute resources, draws power and generates heat.  With good technology, that can be somewhat mitigated, but even if you had the world's largest supercomputer on the head of a pin, transcoding introduces speech frame buffer delays that are irreducible.  Add to those codec latencies, another set of latencies for jitter buffer in the receiver side and then also another jitter buffer in the network for the transcoder's receiver side ... then add extra buffer delays when audio and video don't sync up and you slip to the next timeslot.  In speakerphone configurations, you add extra delay for acoustic echo cancellation.  Then you add the latencies going in the return path.  Before you know it, you have significantly noticeable mouth-to-ear latencies.  Once these the round-trip delay gets to a certain level, people begin to double-talk and it makes a conversational dialog very difficult, especially when you have a multiparty call to discuss a contentious topic.

Imagine you are a service provider.  If WebRTC becomes widespread, then service providers will need to invest capex to purchase & maintain transcoders to support "free" calls.  Does this make financial business sense?

Will WebRTC gain a reputation for poor user experience due to round-trip delay?

We may argue about H.264 and VP8 today, but in the blink of an eye, we'll be arguing about H.265 and VP9 ... but wait ... there will still be H.264 and VP8 infrastructure and devices out there.  So you can either chose to transcode an ever-growing list of evolving codecs ... or better ... use OFFER/ANSWER to negotiate transcoder-free operation so that each session will narrow-down to a common codec for all participants from a constrained list of mandatory codecs.  Thus WebRTC systems must support both VP8 and H.264 today, then also VP9 and H.265 in the future ... as well as the voice channel with Opus, G.711, AMR, AMR-WB and probably eventually also EVS, G.722, etc.

Chris Cavigioli
Intel Corp, Marketing & Product Planning, Graphics/Multimedia, Mobile and Communications Group (MCG)
+1 (415) 254-4545 mobile
+1 (408) 653-8348 desk
+1 (408) 884-2400 fax
This e-mail may contain confidential and privileged material for the sole use of the intended recipient(s).  Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

From: rtcweb [mailto:[email-masked]] On Behalf Of cb.list6
Sent: Sunday, January 12, 2014 1:13 PM
To: John Leslie
Cc: [email-masked]
Subject: Re: [rtcweb] Transcoding

On Jan 12, 2014 12:56 PM, "John Leslie" <[email-masked]<mailto:[email-masked]>> wrote:
>
> Karl Stahl <[email-masked]<mailto:[email-masked]>> wrote:
> >

I am not a codec expert.

But, it is worth noting that transcoding, to the best of my knowledge, substantially reduces quality. Any compression creates an approximation. Transcoding is an approximation of an approximation, which leads to bad quality.  So it is expensive, complex, and poor quality.

CB
> --
> John Leslie <[email-masked]<mailto:[email-masked]>>
> [email-masked]<mailto:[email-masked]>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 13 Jan 2014 12:09:13 +0000

Comments on transcoding based on testing in lab and research. 

*  In side by side comparison, single encode/decode and encode/transcode/decode we can see that the untranscoded video stream can be 20-35% lower bitrate and give the same visual quality.  The tests are performed in a video call between two endpoints at 1 -2 mb/s bitrate, both setups used the same 720p60 HDMI video source. For this test we used high quality cameras, I have not tried the same setup with low cost cameras at lower resolution, or lower bitrate.
* Lip-sync should be within 50 ms, this is the conclusion from a Norwegian based ph.d student and also recommendation from EBU.  (http://tech.ebu.ch/docs/r/r037.pdf) 
* Advanced media resilience techniques like LTRF, disposable frames and more are hard to map between different video codecs, so transcoding will likely reduce robustness for packet loss. 
* Transcoding adds delay, between 60 - 200 ms  (depending on packet loss and implementation). This has an impact since camera to screen latency should be below 300 ms to get a smooth conversation between to participants in a video call. Test on people shows that glass to glass latency should be below 330 ms, to avoid latency to be noticed.  

All in all we should avoid transcoding for video conferencing use cases to get to a good user experience. 

Youtube and other streaming services has the benefit of reliable transport (HTTP) and also latency requirements in the range of seconds and not milliseconds. The differences in requirements make it difficult  to compare streaming with requirements for low latency video conferencing.   

Cheers 

-Espen 

-----Original Message-----
From: rtcweb [mailto:[email-masked]] On Behalf Of Ron
Sent: 13. januar 2014 06:07
To: [email-masked]
Subject: Re: [rtcweb] Transcoding

On Sun, Jan 12, 2014 at 01:12:30PM -0800, cb.list6 wrote:
> Transcoding is an approximation of an approximation, which leads to 
> bad quality.  So it is expensive, complex, and poor quality.

That must be why youtube transcodes *everything* uploaded to it.
Clearly they want to provide an expensive, complex, poor quality service that nobody will ever use.

  Crosseyed and painless,
  Ron

---

From: Ron <[email-masked]>
Date: Mon, 13 Jan 2014 22:47:08 +1030

On Mon, Jan 13, 2014 at 11:02:54AM +0000, Cavigioli, Chris wrote:
>
> transcoding introduces speech frame buffer delays that are irreducible.

Why would transcoding a separate video stream introduce speech delays?

I fear your streams of consciousness may have crossed here :)

> or better ... use OFFER/ANSWER to negotiate transcoder-free operation so
> that each session will narrow-down to a common codec for all participants
> from a constrained list of mandatory codecs.

The idea of *a* mandatory codec is that offer/answer will never fail
though it may well be able to select something even better sometimes.

You seem to be suggesting that we make every possible codec mandatory
and then introduce a negotiation system to try and find one from that
set that people actually did implement?

Inconceivable?

  Ron

---

From: Ron <[email-masked]>
Date: Mon, 13 Jan 2014 23:03:38 +1030

Hi Espen,

> Comments on transcoding based on testing in lab and research. 

Thanks for contributing those here.

> to get to a good user experience.

I don't think that anybody is arguing that avoiding it where possible
is a Bad Thing.  That's why there's a negotiation scheme.

Just that doing it where it can't be avoided for whatever reason is
hardly an end-of-the-world scenario as some people seem to want to
paint it.  There are even some cases where doing it would give you a
*better* experience, such as where there is a bandwidth bottleneck
somewhere in the transmission path to one or more participants.

  Cheers,
  Ron

---

From: "Karl Stahl" <[email-masked]>
Date: Mon, 13 Jan 2014 13:46:52 +0100

> Why would transcoding a separate video stream introduce speech delays?

--- Real life: That is exactly why I said: 
" Chrome with VP8 on old laptops with 1 core 1.5 Ghz x86 is not even usable.
"

CPU load shows 60-100% usage, and you now and then see a good video, BUT
AUDIO is more than 5s, FIVE SECONDS DELAY!

Turning off video, Opus audio is perfect (200 ms delay or so).

Transcoding these advanced Videocodecs is a very poor option... Simply too
heavy!

/Karl

-----Ursprungligt meddelande-----
Från: rtcweb [mailto:[email-masked]] För Ron
Skickat: den 13 januari 2014 13:17
Till: [email-masked]
Ämne: Re: [rtcweb] Transcoding

> or better ... use OFFER/ANSWER to negotiate transcoder-free operation 
> so that each session will narrow-down to a common codec for all 
> participants from a constrained list of mandatory codecs.

The idea of *a* mandatory codec is that offer/answer will never fail though
it may well be able to select something even better sometimes.

You seem to be suggesting that we make every possible codec mandatory and
then introduce a negotiation system to try and find one from that set that
people actually did implement?

Inconceivable?

  Ron

---

From: 'Ron' <[email-masked]>
Date: Mon, 13 Jan 2014 23:48:36 +1030

> Transcoding these advanced Videocodecs is a very poor option... Simply too

It's not quite clear to me what you are actually arguing here, but you
realise that VP8+Opus in Chrome is ...   not transcoding?

Which would seem to leave the only valid part of your statement as just:

 "these advanced Videocodecs is a very poor option... Simply too heavy!"

Perhaps you would be interested in Hervé's benchmarks of H.261 then?

Unless you subscribe to the opinion some people have expressed that you
just don't deserve video on that device and would be better off without
any option for it at all?

  Ron

---

From: "Karl Stahl" <[email-masked]>
Date: Mon, 13 Jan 2014 15:02:44 +0100

I'm just informing that transcoding VP8-H.264 is not anything that should be
considered doing in software (and this huge CPU eating was only for half -
the VP8 side).

Such transcoding would (and consider also doing 1000s in a networks gateway)
require very powerful hardware support for both H.264 and VP8. 

On this list, hardware support for H.264 in devices has been discussed and
requested API's for. We won't see such support also for VP8 just for the
idea of Transcoding.

I am arguing that we should not even consider any selection that requires
VP8 <-> H.264 transcoding.

(Not liking the H.261 is just because it not good enough quality to be an
interesting option. That was not in this argumentation.)

/Karl

-----Ursprungligt meddelande-----
Från: rtcweb [mailto:[email-masked]] För 'Ron'
Skickat: den 13 januari 2014 14:19
Till: [email-masked]
Ämne: Re: [rtcweb] Transcoding

On Mon, Jan 13, 2014 at 01:46:52PM +0100, Karl Stahl wrote:
> CPU load shows 60-100% usage, and you now and then see a good video, 
> BUT AUDIO is more than 5s, FIVE SECONDS DELAY!
> 
> Transcoding these advanced Videocodecs is a very poor option... Simply 
> too heavy!

Unless you subscribe to the opinion some people have expressed that you just
don't deserve video on that device and would be better off without any
option for it at all?

  Ron

---

From: 'Ron' <[email-masked]>
Date: Tue, 14 Jan 2014 01:12:47 +1030

> requested API's for. We won't see such support also for VP8 just for the

Hardware support for VP8 already exists today.  People who own H.264 patents
just don't like to talk about it very much.

> VP8 <-> H.264 transcoding.

Then it seems odd that you would rank as acceptable:

 All entities MUST support at least one of H.264 and VP8

Since that option essentially guarantees such transcoding would be needed
for interop between some devices.

> interesting option. That was not in this argumentation.)

Ah, then maybe you'd instead be interested in the quality metrics that
Hervé posted instead.  Or the examples that Maik and others put up.

Or maybe someone could set up a blind test that all the people who
claimed "oh noes teh bad" could take for themselves.  I suspect it
would be quite amusing to see some hard numbers on how many might
actually pick the difference on a single pass viewing.

  Ron

---

From: "Karl Stahl" <[email-masked]>
Date: Mon, 13 Jan 2014 16:18:51 +0100

My Acceptable to that question had:
b.	Do you have any objections to this option, if so please summarize
them:

* Only if codec slots also are mandated: “MUST”*

That of course indicated that the non-supported codec, would be supported by
downloadable one like Cisco's H.264 offering, THUS NO TRANSCODING REQUIRED

/Karl

PS Just informing/warning about non workable solutions....

-----Ursprungligt meddelande-----
Från: rtcweb [mailto:[email-masked]] För 'Ron'
Skickat: den 13 januari 2014 15:43
Till: [email-masked]
Ämne: Re: [rtcweb] Transcoding

On Mon, Jan 13, 2014 at 03:02:44PM +0100, Karl Stahl wrote:
> On this list, hardware support for H.264 in devices has been discussed 
> and requested API's for. We won't see such support also for VP8 just 
> for the idea of Transcoding.

> (Not liking the H.261 is just because it not good enough quality to be 
> an interesting option. That was not in this argumentation.)

Ah, then maybe you'd instead be interested in the quality metrics that Hervé
posted instead.  Or the examples that Maik and others put up.

Or maybe someone could set up a blind test that all the people who claimed
"oh noes teh bad" could take for themselves.  I suspect it would be quite
amusing to see some hard numbers on how many might actually pick the
difference on a single pass viewing.

  Ron

---

From: John Leslie <[email-masked]>
Date: Wed, 15 Jan 2014 09:11:03 -0500

> Comments on transcoding based on testing in lab and research. 

   :^)

> *  In side by side comparison, single encode/decode and
>    encode/transcode/decode we can see that the untranscoded video
>    stream can be 20-35% lower bitrate and give the same visual quality...

   This tradeoff will be acceptable in many cases, IMHO.

> * Lip-sync should be within 50 ms, this is the conclusion from a
>   Norwegian based ph.d student and also recommendation from EBU...

   The EBU document discusses HDTV programming (and indeed recommends
audio no more than 5 msec early or 15 msec late). They claim to base
this on failure of lip-sync becoming "perceptable to 50% of observers".

   The 50 msec number seems plausible, but I have no basis to judge.

   Lip-sync can always be accomplished by delaying one or the other:
this gives us a trade-off (which I'd prefer to be under application
control).

> * Advanced media resilience techniques like LTRF, disposable frames
>   and more are hard to map between different video codecs, so
>   transcoding will likely reduce robustness for packet loss. 

   (discussed elsewhere)

> * Transcoding adds delay, between 60 - 200 ms  (depending on packet
>   loss and implementation).

   I have no basis to judge those numbers; but they seem high.

>   This has an impact since camera to screen latency should be below
>   300 ms to get a smooth conversation between to participants in a
>   video call. Test on people shows that glass to glass latency
>   should be below 330 ms, to avoid latency to be noticed.  

   Do you have a source for that?

   My gut-feel is that folks will tolerate at least 200 msec glass-to-
glass, but have problems when mouth-to-ear exceeds 150 msec. Alas I
don't have a source for that... :^(

> cases to get to a good user experience. 

   I quite agree it's worth "avoiding" -- but that's not at all the
same as saying we should try to prevent it.

---

From: John Leslie <[email-masked]>
Date: Wed, 15 Jan 2014 09:32:07 -0500

Cavigioli, Chris <[email-masked]> wrote:
> 
> Yes, transcoding consumes significant compute resources, draws power
> and generates heat.

   All true...

> With good technology, that can be somewhat mitigated, but even if you
> had the world's largest supercomputer on the head of a pin, transcoding
> introduces speech frame buffer delays that are irreducible.

   Not sure what "speech frame buffer delays" means; but yes, transcoding
introduces video delays.

> Add to those codec latencies, another set of latencies for jitter
> buffer in the receiver side and then also another jitter buffer in
> the network for the transcoder's receiver side ... then add extra
> buffer delays when audio and video don't sync up and you slip to
> the next timeslot.

   Those latencies really shouldn't be blamed on transcoding.

> In speakerphone configurations, you add extra delay for acoustic echo
> cancellation.

   Agreed that echo-cancellation adds delay -- but again, that shouldn't
be blamed on transcoding. (BTW, I'm not a fan of echo-cancellation in
best-effort-Internet telephony.)

> Then you add the latencies going in the return path. Before you know
> it, you have significantly noticeable mouth-to-ear latencies.

  Agreed it's really easy to get noticeable mouth-to-ear latency.

> Once these the round-trip delay gets to a certain level, people begin
> to double-talk and it makes a conversational dialog very difficult,
> especially when you have a multiparty call to discuss a contentious
> topic.

   I have experienced such latencies. It's really painful. But recall
that there was useful earth-to-moon conferencing -- so I don't think
we should forbid such latencies: it feels like an application-level
consideration.

> Imagine you are a service provider.

   That's easy! ;^)

> If WebRTC becomes widespread, then service providers will need to
> invest capex to purchase & maintain transcoders to support "free"
> calls.

   Actually, we don't need to...

> Does this make financial business sense?

   In some cases, yes. In the other cases, we let someone else do the
investing.

> Will WebRTC gain a reputation for poor user experience due to

   Not if there are a majority of cases where delay is no worse than
the PSTN.

> We may argue about H.264 and VP8 today, but in the blink of an eye,
> we'll be arguing about H.265 and VP9 ... but wait ... there will still
> be H.264 and VP8 infrastructure and devices out there.

   Yes!!!

> So you can either chose to transcode an ever-growing list of evolving
> codecs ... or better ... use OFFER/ANSWER to negotiate transcoder-free
> operation so that each session will narrow-down to a common codec for
> all participants

   Exactly!

> from a constrained list of mandatory codecs.

   Good lord no! We should never limit the choice to MTI codecs!

> Thus WebRTC systems must support both VP8 and H.264 today, then also
> VP9 and H.265 in the future ... as well as the voice channel with
> Opus, G.711, AMR, AMR-WB and probably eventually also EVS, G.722, etc.

   (I don't get it... Why would any sane engineer want to go _there_?

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Wed, 15 Jan 2014 15:59:18 +0000

Comments inline. 

-----Original Message-----
From: John Leslie [mailto:[email-masked]] 
Sent: 15. januar 2014 15:11
To: Espen Berger (espeberg)
Cc: [email-masked]
Subject: Re: [rtcweb] Transcoding

   :^)

>   Norwegian based ph.d student and also recommendation from EBU...

   The EBU document discusses HDTV programming (and indeed recommends audio no more than 5 msec early or 15 msec late). They claim to base this on failure of lip-sync becoming "perceptable to 50% of observers".

EBE>> The section I read was overall lip-sync at playout time should not exceed -40 to +60 ms offset. This is also in line with the PH.d work I have seen. 

   Lip-sync can always be accomplished by delaying one or the other:
this gives us a trade-off (which I'd prefer to be under application control).
EBE>> Robust lip-sync should happen at the playout step, to handle various latency differences between audio and video related to transport, jitter buffer handling, decode or playout.  

   (discussed elsewhere)

   I have no basis to judge those numbers; but they seem high.
EBE>> I do not have a break down of what take time, but encoding of 1080p60 video takes a fair amount of time even on modern  CPU architecture. The longer delays are typically to handle receiver based packet loss and/or packet out-of-order, which is impacting your jitter buffer handling.  My point with transcoding overhead is packet re-ordering might have a higher impact in latency than the actual CPU time to do a decode+encode. 

   My gut-feel is that folks will tolerate at least 200 msec glass-to- glass, but have problems when mouth-to-ear exceeds 150 msec. Alas I don't have a source for that... :^(

EBE>> Unfortunately,  I cannot find an online source for my statement. 

> cases to get to a good user experience.

   I quite agree it's worth "avoiding" -- but that's not at all the same as saying we should try to prevent it.

EBE>> Nothing is preventing us from doing transcoding here, it's more that transcoding has impact.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 16 Jan 2014 10:40:51 +0900

> we should forbid such latencies: it feels like an application-level

Re. earth-to-moon conferencing: Just because it worked for carefully 
selected and well-trained astronauts and ground personnel doesn't mean 
it will be easy for everyday users :-(.

Regards,   Martin.

---

From: Ron <[email-masked]>
Date: Thu, 16 Jan 2014 17:46:41 +1030

On Thu, Jan 16, 2014 at 10:40:51AM +0900, "Martin J. Dürst" wrote:
> selected and well-trained astronauts and ground personnel doesn't
> mean it will be easy for everyday users :-(.

Wait.  You have to be a carefully selected astronaut to get a CB or
Amateur Radio licence, or to use a satellite phone?

Why did nobody tell me this before??

I completely agree that calling the customer service center of the
monopoly telco here is a world of pain and fail[1] since they moved
their call centers offshore and put them behind a poor quality VoIP
link -- but they reassure us that it's perfectly acceptable.  And
they should know right?  They're the well-trained experts at this.

  Floating in a most peculiar way,
  Ron

[1]  When the line quality of your support call center is worse
     than the phone I'm calling to report a serious line fault on
     then the message in the subtext is at least crystal clear!