Subject: [rtcweb] My Opinion: Why I think a negotiating protocol is a Good Thing

From: Harald Alvestrand <[email-masked]>
Date: Tue, 18 Oct 2011 13:43:54 +0200

(Apologies for the length of this note - but I wanted to follow this 
argument a bit deeply)

In the discussions about RTCWEB signalling, I’ve had a change of heart.
Or perhaps it’s just a change of terminology.

The high bit is this:
I believe we need to standardize a negotiation protocol as part of RTCWEB.
This needs to be described as a protocol, and cannot usefully be 
described as an API.

This note is to explain to my fellow WG members what led me to this 
conclusion, and - in some detail - what I think the words in the above 
paragraph mean.
Those who don't want to read a long message can stop here.
----------------------------------------------------------------------------

The context of RTCWEB is the well known trapezoid:

+-----------+ +-----------+
| Web | | Web |
| | Signalling | |
| |-------------| |
| Server | path | Server |
| | | |
+-----------+ +-----------+
/ \
/ \ Proprietary over
/ \ HTTP/Websockets
/ \
/ Proprietary over \
/ HTTP/Websockets \
/ \
+-----------+ +-----------+
|JS/HTML/CSS| |JS/HTML/CSS|
+-----------+ +-----------+
+-----------+ +-----------+
| | | |
| | | |
| Browser | ------------------------- | Browser |
| | Media path | |
| | | |
+-----------+ +-----------+

or even the triangle, where the triangle is formed when the two Web 
severs on top are collapsed into one.

A design criterion for RTCWEB has been that it should be possible to 
write applications on top of RTCWEB simply - that is, without deep 
knowledge about the world of codecs, RTP sessions and the like.
Another design criterion is that interworking should be possible - which 
means that SOMEWHERE in the system, deep knowledge about the world of 
codecs, RTP sessions and the like must be embedded; we can’t just 
simplify our options until everything’s simple.

There’s one place in the ecosystem where this knowledge HAS to be - and 
that is within the browser, the component that takes care of 
implementing the codecs, the RTP sessions and the related features. If 
we can avoid requiring embedding it twice, that’s a feature.

It used to be that I was a believer in APIs - that we should make the 
API the “king”, and describe the way you generate an RTP session as “you 
turn this knob, and get this effect”.
After looking at the problem of Web applications that don’t have domain 
knowledge for a while, I’ve concluded that this doesn’t work. There’s 
the need for one browser to communicate with the other browser, and if 
the intermediate components are to have the ability to ignore the 
details, what’s communicated has to be treated like a blob - something 
passed out of one browser’s API and into another browser’s API, 
unchanged by the application - because the application must be able to 
ignore the details.

OK, now we have the API with blobs. We also have to make some 
assumptions about how those blobs are transported, who’s responsible for 
acting on them, and so on. And we have to make sure different browsers 
implement the blob the same way - that is, it has to be standardized.
What’s more - we DO want to enable applications that are NOT simple. 
Including gateways, which are not browsers. So applications must be free 
to look inside the blob - break the blob boundary - when they need to. 
So this pulls in the same direction as the need for interoperability - 
the format, semantics and handling rules for these blobs has to be 
specified. In some detail.

So we have:
- a data format
- a transmission path
- a set of handling rules, in the form of states, processes and timers

This doesn’t look like an API any more. This looks like a protocol. 
We’ve got experience in describing protocols - but it becomes much 
easier to apply that experience when we call it a protocol.

Let’s do that.

But - you did not address my point...
----------------------------------------------
No, I didn’t. But here are some points that might bear watching.

“We shouldn’t mandate a new protocol on the wire”.
We’re not. We’re specifying one, and mandating it for a specific point: 
The browser/JS interface.
We can have applications that parse it locally using a JS library; in 
that case, the protocol runs between the browser and the local JS 
application.
We can have applications that pass the blobs to a server for gatewaying 
into something else; in that case, the protocol runs between the browser 
and the server.
We can have applications that pass the blobs (possibly via a very simple 
server) to another browser, unchanged; in that case, the protocol runs 
between the two browsers.

In no case does the browser need to know where the other end is; all it 
cares about is that the messages flow according to protocol.

“We should build knobs and let JS libraries take care of what’s on top 
of them”.
Well - we could. But it violates the idea of only having the domain 
knowledge present once.
If you have to have a browser that implements codec A, and a JS library 
that knows how to control codec A, you are *requiring* knowledge in two 
places. That’s not optimal.
Things may change in the future - once downloadable codecs actually 
work, downloadable negotiation blobs may be a reasonable counterpart. 
But we’re not there now.

“We should use existing protocol X”
We could. But then, we’d buy into all the baggage of protocol X - both 
the things it requires us to do and the things that it won’t let us do. 
Doesn’t sound too good to me.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 18 Oct 2011 14:03:11 +0200

2011/10/18 Harald Alvestrand <[email-masked]>:
> There’s the need for one browser to communicate with the other browser, and
> if the intermediate components are to have the ability to ignore the
> passed out of one browser’s API and into another browser’s API, unchanged by
> the application - because the application must be able to ignore the
> details.
>
> OK, now we have the API with blobs. We also have to make some assumptions
> about how those blobs are transported, who’s responsible for acting on them,
> and so on. And we have to make sure different browsers implement the blob
> the same way - that is, it has to be standardized.

Is not that covered win ROAP draft? AFAIK ROAP defines how the SDP are
communicated from the JS to the RTCweb stack and viceversa.

> Including gateways, which are not browsers. So applications must be free to
> look inside the blob - break the blob boundary - when they need to. So this
> pulls in the same direction as the need for interoperability - the format,
> semantics and handling rules for these blobs has to be specified.

Of course they must be specificied, but do you mean that such
specification must be a single one?

In order to understand what you are proposing let me ask a question:

I use SIP over WebSocket, I just need RTCweb API's to be defined. Does
your proposal mean that I won't be able to use SIP over WebSocket and
instead I must use some specific protocol between the JS client and
the server? I hope the answer is not...

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Neil Stratford <[email-masked]>
Date: Tue, 18 Oct 2011 13:20:04 +0100

On Tue, Oct 18, 2011 at 12:43 PM, Harald Alvestrand <[email-masked]>wrote:

> This needs to be described as a protocol, and cannot usefully be described
> as an API.
>

I strongly disagree. We need an API, not a protocol. This is a fundamentally
different problem to traditional RTC due to the presence of the javascript
execution engine that can implement the protocol.

> A design criterion for RTCWEB has been that it should be possible to write
> applications on top of RTCWEB simply - that is, without deep knowledge about
> the world of codecs, RTP sessions and the like.
>

Which can be achieved even with a low-level javascript API by leveraging
library code. We do not need to embed every bit of intelligence into the
browser to enable that to happen.

> codecs, RTP sessions and the like must be embedded; we can’t just simplify
> our options until everything’s simple.
>
> that is within the browser, the component that takes care of implementing
> the codecs, the RTP sessions and the related features. If we can avoid
> requiring embedding it twice, that’s a feature.
>

Agreed - the browser handles RTP and codecs, and the javascript should
handle negotiation and protocol state machines.

> It used to be that I was a believer in APIs - that we should make the API
> the “king”, and describe the way you generate an RTP session as “you turn
> this knob, and get this effect”.
> knowledge for a while, I’ve concluded that this doesn’t work. There’s the
> need for one browser to communicate with the other browser, and if the
> intermediate components are to have the ability to ignore the details,
> what’s communicated has to be treated like a blob - something passed out of
> one browser’s API and into another browser’s API, unchanged by the
> application - because the application must be able to ignore the details.
>

The application can equally ignore what a library does if it wants to, but
it's not tied to one library, it can pick the one most suited to the task in
hand.

> semantics and handling rules for these blobs has to be specified. In some
> detail.
>

This is the problem with passing blobs through an API. If we had a low-level
API we wouldn't be passing blobs.

Neil

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 18 Oct 2011 14:20:50 +0200

> the server? I hope the answer is not...

After re-reading your mail, I'm less afraid:

-----------------

We’re not. We’re specifying one, and mandating it for a specific
point: The browser/JS interface.

We can have applications that pass the blobs to a server for
gatewaying into something else; in that case, the protocol runs
between the browser and the server.

We can have applications that pass the blobs (possibly via a very
simple server) to another browser, unchanged; in that case, the
protocol runs between the two browsers.
-----------------

So line 3 allows me to sleep tonigh :)

However, I don't fully understand why ROAP is not enough. ¿Does not it
specify the *signaling* communication between the JS and the browser
(RTCweb stack)?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 18 Oct 2011 14:34:49 +0200

If "negotiation protocol" means something that resembles ROAP (i.e. more 
or less a wrapper on SDP o/a that enables negotiation of 
session/streams), then I agree. I really like that the application does 
not need to bother, all knowledge of state, codecs, RTP, .... can be 
limited to the browser as long as the app passes the blobs on.

On 10/18/2011 01:43 PM, Harald Alvestrand wrote:
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 18 Oct 2011 14:39:58 +0200

On 10/18/11 14:20, Iñaki Baz Castillo wrote:
> 2011/10/18 Iñaki Baz Castillo<[email-masked]>:
To be clear: I believe that ROAP is a good candidate specification for 
the "blob protocol".
I support the ROAP approach; in this note, I'm trying to lay out my 
thinking for why I support it.

                Harald

---

From: Tim Panton <[email-masked]>
Date: Tue, 18 Oct 2011 13:53:36 +0100

On 18 Oct 2011, at 12:43, Harald Alvestrand wrote:

> (Apologies for the length of this note - but I wanted to follow this argument a bit deeply)
> 
> This needs to be described as a protocol, and cannot usefully be described as an API.
> 
> This note is to explain to my fellow WG members what led me to this conclusion, and - in some detail - what I think the words in the above paragraph mean.
> or even the triangle, where the triangle is formed when the two Web severs on top are collapsed into one.
> 
> A design criterion for RTCWEB has been that it should be possible to write applications on top of RTCWEB simply - that is, without deep knowledge about the world of codecs, RTP sessions and the like.
> Another design criterion is that interworking should be possible - which means that SOMEWHERE in the system, deep knowledge about the world of codecs, RTP sessions and the like must be embedded; we can’t just simplify our options until everything’s simple.

I've generally found that "simplify our options until everything’s simple" is a very good policy for Version one of anything. 
But instead we have a complex set of PSTN-interop centric requirements, it is no surprise that we will come up with a complex PSTN centric solution.

Sadly this will be such a straight-jacket that no other new uses will flourish.

> 
> There’s one place in the ecosystem where this knowledge HAS to be - and that is within the browser, the component that takes care of implementing the codecs, the RTP sessions and the related features. If we can avoid requiring embedding it twice, that’s a feature.

Actually it isn't a universally desired feature. The web is full of multiple ways of manipulating the same object.
an HTML table can be specified in HTML, manipulated in a DOM via javascript, and styled in CSS . 
The writer of each of those bits of code needs to understand what a table is or does at least partially.

> 
> It used to be that I was a believer in APIs - that we should make the API the “king”, and describe the way you generate an RTP session as “you turn this knob, and get this effect”.
> After looking at the problem of Web applications that don’t have domain knowledge for a while, I’ve concluded that this doesn’t work. There’s the need for one browser to communicate with the other browser, and if the intermediate components are to have the ability to ignore the details, what’s communicated has to be treated like a blob - something passed out of one browser’s API and into another browser’s API, unchanged by the application - because the application must be able to ignore the details.

Extend that argument a bit further - the codecs  at each end can agree about their mutual compatibility very simply - without involving the rest of
the stack. All they have to do is expose a method on the decoder that can check the self-description sent by the encoder.

Instead we are making a monolithic codec/network/negotiation/(signalling soon) hunk of code instead of splitting out the parts and (for 
example ) exposing the codecs as objects with behaviours.

> 
> OK, now we have the API with blobs. We also have to make some assumptions about how those blobs are transported, who’s responsible for acting on them, and so on. And we have to make sure different browsers implement the blob the same way - that is, it has to be standardized.
> What’s more - we DO want to enable applications that are NOT simple. Including gateways, which are not browsers. So applications must be free to look inside the blob - break the blob boundary - when they need to. So this pulls in the same direction as the need for interoperability - the format, semantics and handling rules for these blobs has to be specified. In some detail.

We have an API with blobs only because we chose to stick with the ugliness that is SDP. Once you take that decision, the rest follows.

> This doesn’t look like an API any more. This looks like a protocol. We’ve got experience in describing protocols - but it becomes much easier to apply that experience when we call it a protocol.

Ok, lets design a protocol (if we must), but let's base it on the real capabilities of browsers, web coders abilities and the real needs of web users, 
not constantly looking over our shoulders at a legacy protocol that barely works in the environment it was designed for.

And will this new protocol have a standard API ?

Tim.

---

From: Dzonatas Sol <[email-masked]>
Date: Tue, 18 Oct 2011 08:42:13 -0700

On Tue, Oct 18, 2011 at 4:43 AM, Harald Alvestrand <[email-masked]>wrote:

> ------------------------------**------------------------------**
> ----------------
>
> This doesn’t look like an API any more. This looks like a protocol. We’ve
> got experience in describing protocols - but it becomes much easier to apply
> that experience when we call it a protocol.
>

I like the idea that expands the FILE: protocol such that it would work with
an audio-only codec already available. I guess that would be a blob instead
of an image.

> ------------------------------**----------------
> We’re not. We’re specifying one, and mandating it for a specific point: The
> browser/JS interface.
> We can have applications that parse it locally using a JS library; in that
> case, the protocol runs between the browser and the local JS application.
> into something else; in that case, the protocol runs between the browser and
> the server.
> “We should build knobs and let JS libraries take care of what’s on top of
> them”.
> Things may change in the future - once downloadable codecs actually work,
> downloadable negotiation blobs may be a reasonable counterpart. But we’re
> not there now.
>
> We could. But then, we’d buy into all the baggage of protocol X - both the
> things it requires us to do and the things that it won’t let us do. Doesn’t
> sound too good to me.
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>

---

From: Wolfgang Beck <[email-masked]>
Date: Tue, 18 Oct 2011 19:54:18 +0200

> The context of RTCWEB is the well known trapezoid:
I'll nag you again about my draft where there is only one single web
server and one JS client that runs on both browsers.
If you can make sure that all parties in a call use the same RTCWEB
server and -client, you don't need to standardize a protocol.
We startet RCTWEB to overcome the slow innovation pace associated with
protocol standardization.
If RTCWEB relies on a standardized protocol for its key functionality,
we have gained very little.

> Another design criterion is that interworking should be possible
I'm not convinced that we need this, at least not at the signaling
level of RTCWEB clients and servers.

Wolfgang Beck

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 18 Oct 2011 20:00:44 +0200

On 10/18/11 19:54, Wolfgang Beck wrote:
> On Tue, Oct 18, 2011 at 1:43 PM, Harald Alvestrand<[email-masked]>  wrote:
>
>
I disagree with both of your assertions.

1) we have to support both the case of one web server and the case of 2 
web servers.

2) when there is only one web server, one of the browsers is Firefox and 
the other one is Chrome, the JS needs to have a standard means of 
communication with the browser. This means a standard.
I have explained in my long note why I think it makes sense to describe 
this as a protocol.
> level of RTCWEB clients and servers.
I agree that we disagree.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 18 Oct 2011 20:16:03 +0200

2011/10/18 Harald Alvestrand <[email-masked]>:
> 2) when there is only one web server, one of the browsers is Firefox and the
> other one is Chrome, the JS needs to have a standard means of communication
> with the browser. This means a standard.
> I have explained in my long note why I think it makes sense to describe this
> as a protocol.

If that just involves communication between the JS and the browser
than IMHO that's an API rather than a protocol.
However, if such signaling between JS and the browser (the RTCweb
stack) mantains state, then I could agree on naming it "protocol".

Anyhow, the point here is: are you advocating for having a default
signaling protocol in-the-wire or not? It's a bit unclear from your
first mail given the ammount of signaling fragments in the whole
RTCweb picture.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 18 Oct 2011 20:29:18 +0200

On 10/18/11 20:16, Iñaki Baz Castillo wrote:
> 2011/10/18 Harald Alvestrand<[email-masked]>:
ROAP does.
> first mail given the ammount of signaling fragments in the whole
I'm advocating a single protocol between the Javascript and the Web browser.

It's a viable option for some applications to run that protocol from one 
browser to the other browser over a mechanism of their choice; many 
applications will make other choices, including those who choose to 
implement SIP in Javascript.

---

From: Wolfgang Beck <[email-masked]>
Date: Tue, 18 Oct 2011 20:29:42 +0200

> 1) we have to support both the case of one web server and the case of 2 web
> servers.
>
If we look at SIP, the trapezoid model hasn't been too sucessful.
Depressingly few SIP providers do interconnection, after all those
years.
Instead, even simple SIP clients support more than one SIP account so
people can reach their contact on different SIP providers.

> with the browser. This means a standard.
The chat function of stackexchange.com has Firefox and Chrome users.
I'm not aware of any standardized signalling protocol to exchange
the capabilities between the browsers. Instead the JS chat client
queries the browser's capabilities locally and sends them in a
non-standard format
to the web server.

Wolfgang Beck

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 18 Oct 2011 20:43:34 +0200

> I'm advocating a single protocol between the Javascript and the Web browser.

Well, so instead of managing a simple JS API you mean a JS API with
memory and state, so let's call it "protocol". Then I strongly think
that what you suggest is the same as ROAP. And I do agree with such
proposal.

> implement SIP in Javascript.

There is a risk here. If the specs and the API are based on that
signaling protocol, maybe they just fullfit the requirements of that
protocol and forget other possible solutions. Just wondering. But
there will be people ensuring that does not happen ;)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 18 Oct 2011 13:59:51 -0700

On Oct 18, 2011, at 5:53 , Tim Panton wrote:

> We have an API with blobs only because we chose to stick with the ugliness that is SDP.

One of the most compelling argument to me to stick with SDP is that I have a hard time imagining anyone creating a replacement for it in less than a year or two.

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 18 Oct 2011 14:13:42 -0700

> Depressingly few SIP providers do interconnection, after all those

Interesting - most the people I seem to deal with need to send SIP from one domain to another. Who are you thinking of that does not need to do interconnect?

---

From: Wolfgang Beck <[email-masked]>
Date: Wed, 19 Oct 2011 07:21:27 +0200

> Interesting - most the people I seem to deal with need to send SIP from one domain to another. Who are you thinking of that does not need to do
There is a need that people with an account at provider A can call
people at provider B. I have several SIP accounts with various
providers and I don't know for sure which one would be usable for
communicating with you. It's not like email.

What I see is SIP providers that use the PSTN for interconnection. We
might see a lot more SIP interconnection with LTE and VoLTE, which
will
look very much like the PSTN. RCS ('Rich Communication Suite')
standardized presence, chat and file transfer for some years. When it
came to interconnection, providers could only agree on the least
common denominator, RCS-e. Even in the PSTN you lose many features
with interconnection.

Wolfgang Beck

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 19 Oct 2011 06:05:21 +0000

> One of the most compelling argument to me to stick with SDP is that I have a hard time imagining anyone creating a replacement for it in less than a year or two. 

Well... there are already replacements for it in other protocols, but that's not the issue for me - for me one problem is you're _guaranteeing_ we won't see a replacement for it because you're putting it into the browser itself.  Instead of treating the browser as a media/RTP library with an API, you're treating it as an active application agent with a protocol. 

How many RTP libraries do you know of that use SDP offer/answer protocol as their API?  

-hadriel

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 19 Oct 2011 06:15:25 +0000

On Oct 19, 2011, at 1:21 AM, Wolfgang Beck wrote:

> What I see is SIP providers that use the PSTN for interconnection. 

I think you need to get out more. :)
It's true it's not like email, but provider interconnection with SIP is big business.
Maybe you mean for more than voice calls?  IM and video are not nearly as big for SIP interconnection.

-hadriel

---

From: <[email-masked]>
Date: Wed, 19 Oct 2011 08:01:52 +0000

Hi,

>How many RTP libraries do you know of that use SDP offer/answer protocol as

This confuses me too in the direction where the RTC-Web work seems to be heading. There are countless SIP user agents and XMPP clients that implement SDP offer/answer or the Jingle equivalent. But I've not  heard SDP used as an *API* towards the RTP/media libraries. So, when I think about the RTP/media API that the browser should expose to the Javascript app, I'm surprised to see SDP on that level.

I see that it may be useful to have a higher level API that provides ICE and SDP offer/answer type of functions to developers who want that. But I'm not sure if that should be a real W3C standard API or could it just be Javascript libraries. 

Having said that, I'm not going to write a "signaling" or API draft. I can live with any outcome that does not limit app developers too much. So for me at this point it would be valuable input to understand what we are *loosing* if the API were based on SDP and we would have something like ROAP. I've heard it "limits innovation" but I haven't seen concrete examples of that. So if Hadriel is going to produce a draft on these issues, it would be great if it had something like that.  

Markus

---

From: Binod PG <[email-masked]>
Date: Wed, 19 Oct 2011 20:37:10 +0530

> This doesn’t look like an API any more. This looks like a protocol. We’ve got experience in describing protocols - but it becomes much easier to apply that experience when we call it a protocol.

I think, it is a move in the right direction. Having some sort of definition around how the negotiation would look like
is far better than leaving everything to interpretation of the web developer. This is especially good for people who
develop gateway libraries, since they will now have something to actually translate.

What exactly you mean by defining "transmission path"?

thanks,
Binod.

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 19 Oct 2011 18:45:29 -0600

I agree SDP is not used as an RTP API. I looked at a few RTP API and they looked like "here is your compressed media and it's time stamp" and "send this compressed media with following time stamp to this IP, port. I don' think any one want that to be the level of API that gets exposed by the browser. For one thing, the performance issues would likely not work. 

One question that comes up is when a browser adds support for a new CODEC, should the Java script code of a given web sight need to change to take advantage of it. If your answer to that is at least in some cases, No, then it means you need an API that allows the browser to do the negotiation of the codecs. Today SDP (and the mapping of it in Jingle), are pretty much the only games in town for that sort of negotiation. You can argue if we should use SDP or invent something new. I'd love to hear your opinion on that. If we decide we are going to use SDP, I think you end up with a API at roughly the level of what is the W3C WEBRTC draft today and something around the level of protocol as described in ROAP. 

A third level of api that one could do - and I think Dan York, may have been arguing for this but I'm not sure - is and API where there is an DOM object that represent all the communications and in the implies case you just tell it who to connect to and it is done. Imagine if the HTML5 Video tag has something like rtc attribute of someone you wanted to communicate with and the browser set up a video session with them. For example: <video  rtc="jingle:[email-masked]" />. WIth this sort of API, JS could still be used to manipulate the dom object , add streams, pause them, get attributes etc. My read of folks involved was this sort of approach had been pretty much rejected this. 

Like you, I care more about we have something that works and does not limit innovation. I don't really care about how we do it but I want it done soon not 5 years from now. I wrote up ROAP with JDR because I think that it splits the middle ground as is by far the most acceptable thing to people. Largely that is based on my feelings from the room after the last IETF.

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 19 Oct 2011 18:46:10 -0600

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 20 Oct 2011 03:13:35 +0000

[breaking your email up into chunks, 'cause my response is getting long]

> I agree SDP is not used as an RTP API. I looked at a few RTP API and they looked like "here is your compressed media and it's time stamp" and "send this compressed media with following time stamp t…

Yeah when I said it I was kinda thinking "well not really 'RTP' library, because many of those are basically pseudo-sockets", but I was thinking like a media library (libraries that tie the codecs and RTP and SRTP and ICE layers together and provide an API to an app above).

> One question that comes up is when a browser adds support for a new CODEC, should the Java script code of a given web sight need to change to take advantage of it. If your answer to that is at least in some cases, No, then it means you need an API that allows the browser to do the negotiation of the codecs.

I don't think you do.  At least for most "codecs", all a Browser needs to give/be-given is all the a=rtmp and a=fmtp lines, or maybe even only the portions after the PT numbers, as a list/table of strings and their payload-type numbers, per audio/video stream.  I know there are some other codec-specific attributes now and then (T.38 comes to mind, but it was arguably not your typical "codec" anyway)... but in general wouldn't rtpmap+fmtp contents do it?

There are a lot of other attributes, of course, but not per new codec.  Right?

> Today SDP (and the mapping of it in Jingle), are pretty much the only games in town for that sort of negotiation. You can argue if we should use SDP or invent something new. I'd love to hear your opinion on that. If we decide we are going to use SDP, I think you end up with a API at roughly the level of what is the W3C WEBRTC draft today and something around the level of protocol as described in ROAP. 

Let's assume we have to use SDP - I don't actually think we do, but just for argument's sake.  That still doesn't mean we have to embed the offer/answer model in the Browser, or even use it anywhere at all for pure web-apps.  The offer/answer model has some very specific semantics, which are actually restrictive.  For example, the offer/answer model assumes a symmetric media-line model: if you send one audio and one video m-line, the answer can only have one audio and one video m-line.

As an example of something that cannot be accomplished because of this: imagine a Web-application which allows the Browser to communicate with a TelePresence (TP) system.  TP systems have multiple cameras, screen displays, microphones, and speakers.  A PC-based Browser typically only has a single microphone and camera, but can display multiple video feeds separately and can render-mix the incoming audio streams.  Thus, a Browser to TP system would produce an asymmetric media stream model: multiple video streams from the TP system to the Browser, and one video stream from the Browser to the TP system, and the same for audio.  Each TP audio and video stream is an independent m-line RTP session and has unique attributes to indicate position (left/center/right), which a Browser could in theory display on the left/center/right of your browser window.  Doing that is currently not possible with SDP offer/answer; not only because the SDP attributes aren't yet defined, but because the offer/answer model assumes a symmetric number of media-lines (m= lines).  Clearly if and when SDP is changed to handle TelePresence cases, Browsers could be subsequently upgraded to handle it as well sometime after; but they wouldn't need to if the Browser hadn't been involved in SDP and offer/answer to begin with.

The offer/answer model is an ok protocol between VoIP systems, but it's not a good *API* between an application and its library. 

-hadriel

---

From: Roman Shpount <[email-masked]>
Date: Wed, 19 Oct 2011 23:54:06 -0400

One more reason not to use SDP is that there are non-negotiated CODEC
parameters and media stream parameters that generally come from some sort of
configuration, and not from SDP. Examples would be if audio stream should
run noise suppression, automatic gain control, echo canceller, in iLBC codec
that would be running enhancer, in SILK this would be complexity and
expected packet loss percent. It would make sense to provide some mechanism
to specify this.

As far as data format for media description is concerned we can have
something like this:

List of Media Stream Formats. Each contains:

   -     Media Type
   -     Transport (RTP vs SRTP vs DTLS-SRTP)
   -     List of name/value network attributes (ptime, crypto etc)
   -     List of name/value configuration attributes (AGC, EC, NS)
   -     List of formats. Each contains:

   - RTP payload type
   - Name (PCMU, PCMA, etc)
   - Clock Rate (8000, 16000)
   - Number of channels (mono, stereo, surround)
   - List of name/value network attributes (usedtx, maxaveragebitrate,
   bitrate)
   - List of name/value configuration attributes (iLBC enchancer, SILK
   complexity)

This can be easily mapped to/from SDP, and will provide a easy to work
structure for non-SDP based solutions.
_____________
Roman Shpount

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 20 Oct 2011 04:43:22 +0000

> A third level of api that one could do - and I think Dan York, may have been arguing for this but I'm not sure - is and API where there is an DOM object that represent all the communications and in…

Well I don't reject that - I reject that the browser has it hard-coded in.  I hope that some smart Javascript developer will create a library that enables something like what you describe.  But I think it's a role for a JS library.  Because in my opinion one of the enablers to success of the web as an application platform has been that the Browsers are relatively dumb really, and all the logic/smarts is in the code they execute from JS/Flash/whatever, because only the web-app knows what the web-app wants.  If we could have the Browser not involved in RTCWeb at all including not even doing media, I would argue in favor of that... and arguably RTCWeb would have never been created because there'd be nothing for us to have to standardize.  Obviously we have to involve the Browser for media+RTP, so now we're on our slippery-slope to standardized signaling.

> Like you, I care more about we have something that works and does not limit innovation.
> I don't really care about how we do it but I want it done soon not 5 years from now. I wrote up ROAP with JDR because I think that it splits the middle ground as is by far the most acceptable thing to people.

But clearly you do care how we do it - I mean we all care one way or another. :)
There's nothing wrong with that. None of us have a crystal ball about the right answer for this thing, and it's clearly not obvious or else we'd have broader agreement one way or another by now.

> Largely that is based on my feelings from the room after the last IETF. 

I get the feeling that this may be one of those rare (and exciting!) times in the IETF when the people who physically attend the IETF meetings are neither a majority of, nor a representative sample of, the people active on the mailing list.

-hadriel

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 20 Oct 2011 22:35:00 -0600

> I don't think you do.  At least for most "codecs", all a Browser needs to give/be-given is all the a=rtmp and a=fmtp lines, or maybe even only the portions after the PT numbers, as a list/table of…

I'm happy to ignore T.38 as a "special case" but new codecs, video codecs in particular, need to define new parameters. Lets say a given codec defines some new parameters that show up in some normal spot in SDP, I don't see how the JS app will be able to negotiation without understanding what the new parameters mean. Lets say some new VP9 video codec defines a new maxFluffyDepth and the browser reports that it support 666. The other other side offers a maxFlufflyDepth=Yes. What does the Javascript code select? It might be that the based on these you even use a different parameter, say fluffyDepth=-6ft , to specify the interoperable solution. 

> As an example of something that cannot be accomplished because of this: imagine a Web-application which allows the Browser to communicate with a TelePresence (TP) system.  TP systems have multiple…

Hmm - this example sort leads to the direction I think about this. I imagine we both agree that eventually mmusic will define an SDP attribute to help position the video feeds. Clue will slow this down from happening but once clue figures out it need to specify some geometry, it will end taking that to  mmusic. Having a different solution defined in something the browsers are using (or in a protocol like TIP) will just complicate making it work in SDP in a way that continues to be possible to gateway to shat SDP does. It would be easier to just add it in one place and call it done.

---

From: Stefan Håkansson <[email-masked]>
Date: Fri, 21 Oct 2011 08:41:13 +0200

On 10/20/2011 05:13 AM, Hadriel Kaplan wrote:
> As an example of something that cannot be accomplished because of
> this: imagine a Web-application which allows the Browser to
> communicate with a TelePresence (TP) system.  TP systems have
> multiple cameras, screen displays, microphones, and speakers.  A
> PC-based Browser typically only has a single microphone and camera,
> but can display multiple video feeds separately and can render-mix
> the incoming audio streams.  Thus, a Browser to TP system would
> produce an asymmetric media stream model: multiple video streams from
> the TP system to the Browser, and one video stream from the Browser
> to the TP system, and the same for audio.  Each TP audio and video
> stream is an independent m-line RTP session and has unique attributes
> to indicate position (left/center/right), which a Browser could in
> theory display on the left/center/right of your browser window.
> Doing that is currently not possible with SDP offer/answer; not only
> because the SDP attributes aren't yet defined, but because the
> offer/answer model assumes a symmetric number of media-lines (m=
> lines).  Clearly if and when SDP is changed to handle TelePresence
> cases, Browsers could be subsequently upgraded to handle it as well
> sometime after; but they wouldn't need to if the Browser hadn't been
> involved in SDP and offer/answer to begin with.
I think this is doable with the current toolbox. There is an API that 
treats outgoing and incoming MediaStreams (over the same PeerConnection) 
separately, so nothing stops having an asymmetric setup. In addition, 
you could have several PeerConnections if that would help.

Each incoming MediaStream has an unique label, so it is quite simple to 
associate the received MediaStream with the right display 
(left/center/rigth).

The role of the SDP o/a (or ROAP) is IMO only to enable that those 
MediaStreams are sent as RTP streams between peers and can be assembled 
correctly to MediaStreams at the receiving peer (typically a MediaStream 
consists of several RTP streams, e.g. audio+video).

---

From: Hadriel Kaplan <[email-masked]>
Date: Fri, 21 Oct 2011 07:11:53 +0000

> I'm happy to ignore T.38 as a "special case" but new codecs, video codecs in particular, need to define new parameters. Lets say a given codec defines some new parameters that show up in some norma…

I'm not sure you're following what I meant, or else I'm not following you.  
If a new codec VP9 needed to indicate a "maxFluffyDepth" and it supports a value of "666", then when JS calls a getter on the API for available codec info, in the array-list of returned codecs it sees something like this:

returnedArrayOfLocalVideoCodecs = [
	{ "payload-type": 99, "name": "H263-2000", "clock" : 90000, "fmtp": "CIF=4;QCIF=2;F=1;K=1" },
	{ "payload-type": 100, "name": "VP9", "clock": 64000, "fmtp": "maxFluffyDepth=666" },
	...
]

(that's not actually exactly what I would do, BTW, but just for sake of simplicity)

JS could, if it so wished, only use entries in that array it understood.  OR, it could use any/all of them.  How it uses them exactly is app-specific.  It could stringify this into JSON, etc.

But let's just say it wants to turn this into SDP and use offer/answer.  Lucky for us there's a sdpForDummies.js library which does that, so this is all in that library.  The library knows nothing about codec VP9, but it knows how to produce attributes and such using the fields defined in the API, so it creates this:

[some SDP lines missing because I'm lazy]

m=video 1234 RTP/AVPF 99 100
a=rtpmap:99 H263-2000/90000
a=fmtp:99 CIF=4;QCIF=2;F=1;K=1
a=rtpmap:100 VP9/64000
a=fmtp:100 maxFluffyDepth=666

Then let's say it gets an SDP answer back of this:
m=video 5678 RTP/AVPF 100
a=rtpmap:100 VP9/64000
a=fmtp:100 maxFluffyDepth=Yes

So sdpForDummies.js understands the SDP offer/answer model, but not the codec chosen.  But it does what it does in every case: it parses it, sees the fmtp stuff and treats that as a opaque string, and calls a Browser API setter function like 
	MediaStream.setRemoteCodec(1, 100, 64000, "maxFluffyDepth=Yes"); 
	// the first arg=1 is the array index of codecs for "VP9"

You can say "well isn't that like an answer/offer just with SDP parsed out as Javascript integral data types?"  And I would say no it's not - the Browser's maintaining no offer/answer state, it's not handling forking, it's not handling reversion in case of failure, it's not parsing protocol messages (ie, SDP), and it's not making decisions unbeknownst to the JavaScript.  In short: it's an API rather than a protocol. 

Will there be some future case that could require the JS to change when the Browser does something new?  Of course.  But JS developers are incredibly fast at changing code and getting it deployed/used compared to timescales of Browser changes.  If the JS app developer wants to enable some new doohicky that really did new things differently in an incompatible way, then the JS developer will do it.  We don't need to tell them how to do their job.  We need to *let* them do their job, and their job is code.

-hadriel
p.s. as a side benefit, doing this might also keep us honest in MMUSIC about how we encode things and handle SDP offer/answer consistently across codecs.  :)