Subject: [rtcweb] Some thoughts on optional audio codecs

From: Bo Burman <[email-masked]>
Date: Mon, 15 Jul 2013 15:15:19 +0000

Regarding the previous discussion on optional audio codecs in the (currently expired) draft on RTCWEB audio codecs (https://datatracker.ietf.org/doc/draft-ietf-rtcweb-audio/):

I think most parties involved in WebRTC work, myself included, hope and believe that it will be ubiquitous and easy to include real-time media conversation functionality in nearly any web context. Since it will be that easy, it can be expected that most web developers need not be, and thus will not be, media specialists or very knowledgeable about codecs.

The definition of RTCWEB MTI codecs ensures that communication is possible since at least one codec will always be found, but it is not possible to claim the resulting communication to be optimum for every possible context.

Even if WebRTC will be close to ubiquitous, there will for quite some time likely be a desire to reach real-time media domains and devices that were not originally designed for and thus are not optimized for use with WebRTC. A communication device that is not designed solely for WebRTC use will likely include functionality and codecs also for its "native" domain.

Any added cost of not being able to use existing "native" codecs will vary both in amount and where the cost has to be taken. Eliminating it is indeed an optimization, but the total cost savings may still be significant.

With the current design and to my understanding, it will be the browser vendor's choice to add optional codecs, including any "native" domain codecs.  The choice may possibly be delegated to individual web developers making use of WebRTC functionality. A browser vendor will arguably have to know each target platform to some extent, but it can hardly be assumed that a web developer knows the capabilities of all devices that will use the WebRTC-enabled site unless the browser can provide the needed information. There is a risk that "native" codecs in devices are not well handled, unless the motivations and methods to make use of them are better specified.

While any audio codecs besides the MTI ones are clearly optional, I believe the suggested text addition on optional audio codecs to the RTCWEB audio draft in Ticket #12 (http://trac.tools.ietf.org/wg/rtcweb/trac/ticket/12#) to be too brief considering the above.

In that draft, I would prefer something more in line with:

"If other suitable audio codecs are available to the browser to use, it is recommended that they are also included in the offer in order to maximize the possibility to establish the session without the need for audio transcoding".

Assuming that the browser vendor (or web developer) is sufficiently concerned with codecs to read the audio codecs draft (or the corresponding RFC to-be), the above text may, as a start, give some added guidance why non-MTI codecs may be desirable to consider in addition to the MTI ones.

Cheers,
Bo

Multimedia Technologies
Ericsson Research
Färögatan 6
SE-164 80, Kista, Sweden
www.ericsson.com

---

From: Enrico Marocco <[email-masked]>
Date: Tue, 16 Jul 2013 16:35:26 +0200

On 7/15/13 5:15 PM, Bo Burman wrote:
> "If other suitable audio codecs are available to the browser to use,
> it is recommended that they are also included in the offer in order
> to maximize the possibility to establish the session without the need
> for audio transcoding".

Yes, in fact this is happening already:
https://hacks.mozilla.org/2013/01/firefox-development-highlights-h-264-mp3-support-on-windows-scoped-stylesheets-more/

If people feel SHOULD/RECOMMENDED to be too strong, I'm sure we can find
the right keyword in RFC 6919..

Enrico

---

From: "Bogineni, Kalyani" <[email-masked]>
Date: Tue, 16 Jul 2013 13:02:07 -0400

We support the following wording proposal from Bo Burman.

Regards,
Kalyani Bogineni
Verizon

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bo Burman
Sent: Monday, July 15, 2013 11:15 AM
To: [email-masked]
Subject: [rtcweb] Some thoughts on optional audio codecs

Cheers,
Bo

Multimedia Technologies
Ericsson Research
Färögatan 6
SE-164 80, Kista, Sweden
www.ericsson.com
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Adam Roach <[email-masked]>
Date: Tue, 16 Jul 2013 15:07:00 -0500

> https://hacks.mozilla.org/2013/01/firefox-development-highlights-h-264-mp3-support-on-windows-scoped-stylesheets-more/

You have misread that article. What that article says is that Mozilla is 
adding select platform-supplied codecs to Firefox for non-WebRTC uses.

Our implementation of audio codecs for WebRTC continues to support PCMU, 
PCMA, Opus, and nothing else; our implementation use VP8 exclusively for 
video.

/a

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 17 Jul 2013 09:44:43 +0200

> adding select platform-supplied codecs to Firefox for non-WebRTC uses.

Sure, didn't mean to imply that it is happening for WebRTC.

> PCMA, Opus, and nothing else; our implementation use VP8 exclusively for 

And that's pretty visible also from the outside. What's not visible is
whether at some point in time Mozilla will consider taking advantage of
OS/platform-provided encoding/decoding capabilities also for WebRTC. One
may be led to assume yes, as you're doing it for static content already.

But maybe different conditions apply here, and I don't expect them to be
discussed on this mailing list (even if it would be very, very
interesting!).

Enrico

---

From: "Hutton, Andrew" <[email-masked]>
Date: Wed, 17 Jul 2013 08:45:47 +0000

We appear to have been around this loop a number of times the text suggested here is exactly what was suggested by Andrew Allen back in January and I for one supported it them and still do - See http://www.ietf.org/mail-archive/web/rtcweb/current/msg06121.html.

Not sure there was a definitive conclusion to that particular consensus call.

Andy

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Bogineni, Kalyani
> Sent: 16 July 2013 18:02
> To: 'Bo Burman'; [email-masked]
> Cc: Bogineni, Kalyani
> Subject: Re: [rtcweb] Some thoughts on optional audio codecs
> 
> "If other suitable audio codecs are available to the browser to use, it
> is recommended that they are also included in the offer in order to
> maximize the possibility to establish the session without the need for
> audio transcoding".
> 
> Regards,
> Kalyani Bogineni
> Verizon
> 
> Regarding the previous discussion on optional audio codecs in the
> (currently expired) draft on RTCWEB audio codecs
> (https://datatracker.ietf.org/doc/draft-ietf-rtcweb-audio/):
> 
> I think most parties involved in WebRTC work, myself included, hope and
> believe that it will be ubiquitous and easy to include real-time media
> conversation functionality in nearly any web context. Since it will be
> that easy, it can be expected that most web developers need not be, and
> thus will not be, media specialists or very knowledgeable about codecs.
> 
> The definition of RTCWEB MTI codecs ensures that communication is
> possible since at least one codec will always be found, but it is not
> possible to claim the resulting communication to be optimum for every
> possible context.
> 
> Even if WebRTC will be close to ubiquitous, there will for quite some
> time likely be a desire to reach real-time media domains and devices
> that were not originally designed for and thus are not optimized for
> use with WebRTC. A communication device that is not designed solely for
> WebRTC use will likely include functionality and codecs also for its
> "native" domain.
> 
> Any added cost of not being able to use existing "native" codecs will
> vary both in amount and where the cost has to be taken. Eliminating it
> is indeed an optimization, but the total cost savings may still be
> significant.
> 
> With the current design and to my understanding, it will be the browser
> vendor's choice to add optional codecs, including any "native" domain
> codecs.  The choice may possibly be delegated to individual web
> developers making use of WebRTC functionality. A browser vendor will
> arguably have to know each target platform to some extent, but it can
> hardly be assumed that a web developer knows the capabilities of all
> devices that will use the WebRTC-enabled site unless the browser can
> provide the needed information. There is a risk that "native" codecs in
> devices are not well handled, unless the motivations and methods to
> make use of them are better specified.
> 
> While any audio codecs besides the MTI ones are clearly optional, I
> believe the suggested text addition on optional audio codecs to the
> RTCWEB audio draft in Ticket #12
> (http://trac.tools.ietf.org/wg/rtcweb/trac/ticket/12#) to be too brief
> considering the above.
> 
> "If other suitable audio codecs are available to the browser to use, it
> is recommended that they are also included in the offer in order to
> maximize the possibility to establish the session without the need for
> audio transcoding".
> 
> Assuming that the browser vendor (or web developer) is sufficiently
> concerned with codecs to read the audio codecs draft (or the
> corresponding RFC to-be), the above text may, as a start, give some
> added guidance why non-MTI codecs may be desirable to consider in
> addition to the MTI ones.
> 
> Cheers,
> Bo
> 
> Multimedia Technologies
> Ericsson Research
> Färögatan 6
> SE-164 80, Kista, Sweden
> www.ericsson.com

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 18 Jul 2013 08:25:29 -0400

Yes, this text has been around for a while. I also support it.

...Paul

>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Hutton, Andrew
>Sent: Wednesday, July 17, 2013 4:46 AM
>To: Bogineni, Kalyani; 'Bo Burman'; [email-masked]
>We appear to have been around this loop a number of times the text
>suggested here is exactly what was suggested by Andrew Allen back in
>January and I for one supported it them and still do - See
>http://www.ietf.org/mail-archive/web/rtcweb/current/msg06121.html.
>
>> it is recommended that they are also included in the offer in order to
>> Behalf Of Bo Burman
>> I think most parties involved in WebRTC work, myself included, hope
>> and believe that it will be ubiquitous and easy to include real-time
>> media conversation functionality in nearly any web context. Since it
>> will be that easy, it can be expected that most web developers need
>> not be, and thus will not be, media specialists or very knowledgeable
>about codecs.
>>
>> use with WebRTC. A communication device that is not designed solely
>> for WebRTC use will likely include functionality and codecs also for
>> its "native" domain.
>>
>> With the current design and to my understanding, it will be the
>> browser vendor's choice to add optional codecs, including any "native"
>> domain codecs.  The choice may possibly be delegated to individual web
>> provide the needed information. There is a risk that "native" codecs
>> in devices are not well handled, unless the motivations and methods to
>> it is recommended that they are also included in the offer in order to

---

From: 邓灵莉/Lingli Deng <[email-masked]>
Date: Fri, 19 Jul 2013 13:33:42 +0800

We also support this proposal.

In particular, I think it makes sense to allow the codec choice be possibly delegated to individual web developers making use of WebRTC functionality.
Therefore, it would be better for them to be informed of available options and make the decision accordingly.
While it is not realistic to assume that a browser could always implement a best choice for every potential setting, to include other available codec in the offer would open the door to a win-win outcome.

BR
Lingli

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Paul Coverdale
发送时间: 2013年7月18日 20:25
收件人: 'Hutton, Andrew'; 'Bogineni, Kalyani'; 'Bo Burman'; [email-masked]
主题: Re: [rtcweb] Some thoughts on optional audio codecs

...Paul

---

From: "Lijing (Jessie, Technology Introduction & Standard and Patent Dept)" <[email-masked]>
Date: Mon, 22 Jul 2013 01:34:55 +0000

I fully agree with the idea to " use existing "native" codecs ". 

In addition, from a web developer's perspective, in order to make use of the "native" codecs, the browser should provide codec APIs to open the local codec capabilities of the device to applications and developers. It really make sense.

Best Regards
Lijing

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Bo Burman
发送时间: 2013年7月15日 23:15
收件人: [email-masked]
主题: [rtcweb] Some thoughts on optional audio codecs

Cheers,
Bo

Multimedia Technologies
Ericsson Research
Färögatan 6
SE-164 80, Kista, Sweden
www.ericsson.com

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 21 Jul 2013 18:38:58 -0700

On Sun, Jul 21, 2013 at 6:34 PM, Lijing (Jessie, Technology Introduction &
Standard and Patent Dept) <[email-masked]> wrote:

> In addition, from a web developer's perspective, in order to make use of
> the "native" codecs, the browser should provide codec APIs to open the
> local codec capabilities of the device to applications and developers. It
> really make sense.
>

I don't understand what this means.

-Ekr

---

From: "Lijing (Jessie, Technology Introduction & Standard and Patent Dept)" <[email-masked]>
Date: Mon, 22 Jul 2013 03:42:17 +0000

Sorry，I’ve confused two concepts. What I want to express is that,
Firstly, the browser may be able to make use of “native” codec APIs of the “hard” device(e.g. mobile phone), and include the “native” codecs in the SDP O/A messages.
In addition, for application flexibility, the browser should provide codec capability APIs to developers. Then,  the developers can choose proper codecs according to the specific scenario.

So there should be two APIs. The first is provide by the device and used by the browser. The second is provided by the browser and used by the developer.

Best Regards,
Lijing

发件人: Eric Rescorla [mailto:[email-masked]]
发送时间: 2013年7月22日 9:39
收件人: Lijing (Jessie, Technology Introduction & Standard and Patent Dept)
抄送: Bo Burman; [email-masked]
主题: Re: [rtcweb] 答复: Some thoughts on optional audio codecs

On Sun, Jul 21, 2013 at 6:34 PM, Lijing (Jessie, Technology Introduction & Standard and Patent Dept) <[email-masked]<mailto:[email-masked]>> wrote:
I fully agree with the idea to " use existing "native" codecs ".

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 21 Jul 2013 20:46:57 -0700

On Sun, Jul 21, 2013 at 8:42 PM, Lijing (Jessie, Technology Introduction &
Standard and Patent Dept) <[email-masked]> wrote:

>  Sorry，I’ve confused two concepts. What I want to express is that,****
>
> Firstly, the browser may be able to make use of “native” codec APIs of the
> “hard” device(e.g. mobile phone), and include the “native” codecs in the
> SDP O/A messages.****
>
> In addition, for application flexibility, the browser should provide codec
> capability APIs to developers. Then,  the developers can choose proper
> codecs according to the specific scenario. ****
>
> ** **
>
> So there should be two APIs. The first is provide by the device and used
> by the browser.
>

Well, devices may or may not do this, but that's kind of out of scope for
this WG.

The second is provided by the browser and used by the developer.

Yes, this seems not unreasonable...

-Ekr