Subject: Re: [rtcweb] Low Level Javascript API Proposal avail on the webrtc list

From: Harald Alvestrand <[email-masked]>
Date: Sat, 01 Oct 2011 09:28:36 +0200

On 09/30/2011 10:57 PM, Iñaki Baz Castillo wrote:
> 2011/9/30 Tim Panton<[email-masked]>:
>>> Not in case you implement pure SIP on top of JavaScript/WebSocket :)
>> So the question is - do you think that you _could_ implement SIP on top of JavaScript/WebSocket
>> and use the RTP/media api as described ? We tried to make them orthogonal, but I think Neil feels that
>> to support forking you may need additional features or flags to the calls described in the proposal.
> I don't know (yet) if a WebRTC client can establish different media
> sessions with different peers at the same time (it seems difficult as
> there is supposed to be a single microphone). If not, then at
> javascript level I can decide which session to enable and which one to
> pause (I hope). Also, if I make an outgoing call from the webbrowser
> and receive multiple 180/183 with different SDP, I could choose which
> one to render so I just establish the media session with one of them.
The scenarios-and-requirements document requires that this be supported.

---

From: Matthew Kaufman <[email-masked]>
Date: Tue, 04 Oct 2011 19:29:01 -0700

On 9/28/2011 9:12 AM, Roman Shpount wrote:
> Few questions that I have in regard to this:
>
>
> 3. How do you process multiple answers for the same dialog?
>
> 4. What do we do with multiple answers from a forked dialog? Is it 
> even supported?
>

This stuff is exactly what you'll need to add yet more code to solve if 
you try to bake SDP offer-answer into the browser to turn it 90% of the 
way into a SIP phone.

And exactly what you *don't* need browser code to solve if it is done 
elsewhere (for instance, the server might be handling the entire SIP and 
SDP exchange and resolve forking issues at that end without involving 
the client at all.)

Matthew Kaufman

---

From: Matthew Kaufman <[email-masked]>
Date: Tue, 04 Oct 2011 19:37:48 -0700

On 9/30/2011 12:46 PM, Iñaki Baz Castillo wrote:
> 2011/9/28 Tim Panton<[email-masked]>:
>> I just want to ask folks to take a look at a proposal Neil Stratford wrote for the
>> Low Level Javascript API and sent to the webrtc list.
>>
>> It  is based on his and my experience of web based audio plugins and on a
>> proposal of Cullen's a while back.
>>
>> http://lists.w3.org/Archives/Public/public-webrtc/2011Sep/0106.html
>>
>> We'd appreciate any feedback either here or on the webrtc list as appropriate.
> Hi, the API in that document is just valid for a few specific
> scenarios (make a call, receive a call and maybe putting a session on
> hold).

Totally disagree. The API in that document can be used to build every 
single VoIP application you can build with SIP *plus a bunch more*.

This is like claiming that DirectSound is "just valid for a few specific 
scenarios" when in fact it is a low-level audio API for Win32 systems 
that can be used to build pretty much anything that wants to play sound.

>   That is a very limited subset of what any VoIP protocol can
> offer, so I don't think it's a good idea for rtcweb to assume a custom
> protocol like that for inclusion in WebRTC JS API (as native JS code
> within the browser).

I think you're saying that the JS API should be some existing protocol 
standard like SIP and SDP here, yes?

It is much easier to discuss if we keep "the wire protocol" and "the JS 
API" separate... and realize that just like my above Win32 example, a 
"low level" JS API could in fact be used in conjunction with *any* wire 
protocol (that you can send/receive within the browser context) plus 
server behavior.

This is also true of the "high level" JS API proposals... it is just 
more annoying, as you need to do things like trick them into generating 
an SDP offer, then decode it locally or send it to a server for 
processing, and then kill that and start the actual connection you 
wanted while lying to the browser about what the other end has claimed.

I'd rather have the API that lets me do what I want directly.

>
> Of course, a simple API like that would be useful for lot of people
> offering basic media services to the web site users (basic scenarios
> in which there is no call transfer, parallel forking and so).

Yes. And it would also be useful for people offering all types of 
complex telephony services, including call transfer, parallel forking, 
and the like... as well as all types of complex *non-telephony* RTC 
applications, like games.

Matthew Kaufman

---

From: Roman Shpount <[email-masked]>
Date: Tue, 4 Oct 2011 23:19:28 -0400

On Tue, Oct 4, 2011 at 10:29 PM, Matthew Kaufman
<[email-masked]>wrote:

>
> This stuff is exactly what you'll need to add yet more code to solve if you
> try to bake SDP offer-answer into the browser to turn it 90% of the way into
> a SIP phone.
>
> elsewhere (for instance, the server might be handling the entire SIP and SDP
> exchange and resolve forking issues at that end without involving the client
> at all.)
>

Let me rephrase my questions:

3. If I generate an offer using this API and then send it to the web server,
which will deal with SIP end of things, and will get multiple answers in the
same dialog, how do I process those answers via this API? What I need is an
ability to provide an answer update to the connection. Should I just remove
old streams and candidates and add a new ones?

4. Same thing with an offer that created multiple dialogs. I want to clone
the connection and add streams and candidates to it. I don't think this is
possible with the current API.

Few new questions:

5. What is the priority order for generating candidates when multiple
STUN/TURN servers are specified? How would this be affected if we also have
proxies? I am not sure this is part of any specification so far and probably
should be part of RTC spec defined by IETF

6. How do we deal with CODEC negotiated parameters? It is possible to have
codec level parameter that needs to be negotiated via offer/answer exchange.
There are also codec level parameters that are affected by stream level
parameters (like iLBC mode=30 that only makes sense with ptime dividable by
30). Is it going to be implemented in JavaScript? This might not be the best
idea, since this will require JavaScript to have a detailed knowledge about
each codec implementation provided by the browser. For third party call
control, it would be more robust to provide a way to return actual codec
parameters selected by the browser so that they can be returned in the
answer.
_____________
Roman Shpount

---

From: Matthew Kaufman <[email-masked]>
Date: Tue, 04 Oct 2011 21:25:31 -0700

On 10/4/2011 8:19 PM, Roman Shpount wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>     This stuff is exactly what you'll need to add yet more code to
>     solve if you try to bake SDP offer-answer into the browser to turn
>     it 90% of the way into a SIP phone.
>
>     And exactly what you *don't* need browser code to solve if it is
>     done elsewhere (for instance, the server might be handling the
>     entire SIP and SDP exchange and resolve forking issues at that end
>     without involving the client at all.)
>
>
> 3. If I generate an offer using this API and then send it to the web 
> server, which will deal with SIP end of things, and will get multiple 
> answers in the same dialog, how do I process those answers via this 
> API? What I need is an ability to provide an answer update to the 
> connection. Should I just remove old streams and candidates and add a 
> new ones?

How about I rephrase?

If I generate an offer by writing SDP code in C++ using Win32 APIs to 
allocate sockets and send and receive packets, and I send a SIP packet 
that gets multiple answers in the same dialog, how do I process those 
answers using the Win32 API? See how silly that sounds?

Seriously though, if you need an additional local endpoint because of 
forking, you need to create a second connection object, which will then 
give you an onCandidates() callback, as I understand this API proposal. 
The other way to do it is to have the connection object be able to bind 
to multiple possible far endpoints, but that's not exactly what this API 
says, and it isn't clear that it is needed. (Because it isn't how other 
SIP devices use their network stacks to solve this problem either).

>
> 4. Same thing with an offer that created multiple dialogs. I want to 
> clone the connection and add streams and candidates to it. I don't 
> think this is possible with the current API.

I don't see why you can't create a second connection object (or extend 
this API to allow cloning), and add streams and candidates to that 
connection object. Should be possible to add the same stream to two 
different connection objects, too.

> STUN/TURN servers are specified? How would this be affected if we also 
> have proxies? I am not sure this is part of any specification so far 
> and probably should be part of RTC spec defined by IETF

Lots of ways to handle this... but yes, the current version of this 
proposed specification doesn't expose any sort of prioritization.
>
> 6. How do we deal with CODEC negotiated parameters? 

I believe that the codec parameters are exposed by the codec object, at 
least that's how I read this spec.

> It is possible to have codec level parameter that needs to be 
> negotiated via offer/answer exchange.

Maybe. Or maybe there's an offer-answer at the server that picks the 
right answer and then sends that down as a setting to the browser.

> There are also codec level parameters that are affected by stream 
> level parameters (like iLBC mode=30 that only makes sense with ptime 
> dividable by 30).

That's just a bug with how RTP works.

> Is it going to be implemented in JavaScript?

I suppose it might need to be, if you really need to select 
connection-level things to match codec-level things and vice versa. Or 
control it up at the server and then send the right settings down.

> This might not be the best idea, since this will require JavaScript to 
> have a detailed knowledge about each codec implementation provided by 
> the browser. For third party call control, it would be more robust to 
> provide a way to return actual codec parameters selected by the 
> browser so that they can be returned in the answer.

Except that we often don't want the codec parameters to be "selected by 
the browser". We want to know what the browser can do and just set them 
a particular way.

I've already suggested that the format of the "detailed knowledge" can 
actually just reuse the parameter strings that are already defined for 
use in SDP.

Matthew Kaufman

---

From: Roman Shpount <[email-masked]>
Date: Wed, 5 Oct 2011 02:21:59 -0400

On Wed, Oct 5, 2011 at 12:25 AM, Matthew Kaufman
<[email-masked]>wrote:

> allocate sockets and send and receive packets, and I send a SIP packet that
> gets multiple answers in the same dialog, how do I process those answers
> using the Win32 API? See how silly that sounds?
>
> forking, you need to create a second connection object, which will then give
> you an onCandidates() callback, as I understand this API proposal. The other
> way to do it is to have the connection object be able to bind to multiple
> possible far endpoints, but that's not exactly what this API says, and it
> isn't clear that it is needed. (Because it isn't how other SIP devices use
> their network stacks to solve this problem either).
>
>
I think you are completely missing the point: If I build similar thing using
Win32 API, I would re-initialize codec and  RTP states and re-do the ICE
candidate selection based on the new answer. This particular API has methods
to add and remove streams and candidates, but no method to start a
re-negotiation. Don't assume something is silly just because you did not
understand it.

> I don't see why you can't create a second connection object (or extend this
> API to allow cloning), and add streams and candidates to that connection
> object. Should be possible to add the same stream to two different
> connection objects, too.
>
>
Each connection object has an associated local port on which it is listening
for incoming media and ICE STUN messages. What I need is to be able to reuse
the same port to connect to multiple different sets of remote candidates.
Remote IP/port should be sufficient to disambiguate different streams.

> It is possible to have codec level parameter that needs to be negotiated
> via offer/answer exchange.
>
>
> Maybe. Or maybe there's an offer-answer at the server that picks the right
> answer and then sends that down as a setting to the browser.
>
>
> 30).
> have a detailed knowledge about each codec implementation provided by the
> browser. For third party call control, it would be more robust to provide a
> way to return actual codec parameters selected by the browser so that they
> can be returned in the answer.
>
>
> Except that we often don't want the codec parameters to be "selected by the
> browser". We want to know what the browser can do and just set them a
> particular way.
>
> actually just reuse the parameter strings that are already defined for use
> in SDP.
>
>
Once again, you are missing the point completely. Part of the problem here
is that typically codec information in offer/answer exchange does not
provide the list of all the codec level parameters supported by the codec.
The fact that, for instance SILK implementation supports FEC or DTX does not
have to be present in the offer to be present in the answer. This means
Codec object will need to expose a more information then typically provided
in SDP (ie all the supported codec parameters vs just the desired one). This
means RTC will need to define fields for this object for each codec it will
support and cannot simply reuse MMUSIC/PAYLOAD standards.

This also brings about the whole new problem -- asymmetric codecs and codec
parameters. There are a lot of valid use cases when you want to use
different codecs in different directions (different hardware optimized
codecs in different mobile devices). There are cases when different payload
types are used for the same codecs in different directions (It can also be
considered an RTP bug). You might want to use different video resolutions
due to device camera and display limitations. You might want to use
different codec bandwidths due to difference in up and down stream network
capacity. There is no way to handle this now. I would suggest that the
better model for third party call control or inbound call would be get the
offer from remote side, process it on the server or in javascript, pass it
to the RTC stack in the browser, get the answer from the RTC stack in the
browser, process it (if needed) in javascript or on the server and send it
to the remote side.

Yet another codec negotiation deficiency that I see here is that you cannot
specify multiple codecs. It is fairly typical to have audio codec, comfort
noise codec, and DTMF to be used at the same time for a single stream.

Final comment about the draft is that it is missing a lot of parameters
about codecs. It is missing things like media type, transport type, clock
rates, payload types, crypto attributes, etc. It would be interesting to see
which attributes would end up in which of the objects.
______________
Roman Shpount

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 5 Oct 2011 11:31:36 +0200

> exchange and resolve forking issues at that end without involving the client

No please. The client CAN be intelligent, and that is how SIP usually
works (intelligence in endpoints rather than in servers).

You can do that if you want (you can build a custom
SIP<-->EasySimpleJsonProtocol gateway in server side), but please
don't mandate it. Let me to deal with SDP's in client side (at
JavaScript level), please. Others could choose what to do, but don't
force me to create a ugly application protocol gateway just because
some people prefer to make web clients as "simple as possible".

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 5 Oct 2011 11:41:03 +0200

> standard like SIP and SDP here, yes?

No, I've *never* tried to mandate a specific signaling protocol,
never. I just what my choice to also be feasible, and that is SIP over
WebSocket and full intelligence in client side (in the JS
application). Others can use whatever they want.

> It is much easier to discuss if we keep "the wire protocol" and "the JS API"
> separate... and realize that just like my above Win32 example, a "low level"
> JS API could in fact be used in conjunction with *any* wire protocol (that
> you can send/receive within the browser context) plus server behavior.

That's a good approach. Please take a look to other thread in which
I've suggested a pseudo-code showing a theorical WebRTC JS API.

> This is also true of the "high level" JS API proposals... it is just more
> annoying, as you need to do things like trick them into generating an SDP
> offer, then decode it locally or send it to a server for processing, and
> then kill that and start the actual connection you wanted while lying to the
> browser about what the other end has claimed.
>

Yes, most of the people assumes that the intelligence must be in the
web server and can only assume some easy JSON message exchange between
web client and server ("the client is stupid, the server is clever").

Ok, let me say that within next months (hopelly) we have:

- Web browsers have powerful JavaScript stack for coding client applications.
- WebSocket for building realtime and bidirectional protocols.
- RtcWeb for media sessions.

So we have *all the tools* for creating a rich multimedia client
application (as if it would be a Qt desktop application). Now, if you
prefer to leave some logic processing into the web server, that's ok,
but please let me using all the power of the client side.

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Neil Stratford <[email-masked]>
Date: Wed, 5 Oct 2011 10:52:49 +0100

On Wed, Oct 5, 2011 at 10:31 AM, Iñaki Baz Castillo <[email-masked]> wrote:

> you
> SDP

I don't think that anyone is saying that we want to mandate a simple Json
signalling protocol - just that you have the option of coding it all in
javascript at the client side, or at the server side if you wanted to. The
important thing in my view is not to bake SDP into the JS API with the
browser in such a way that we can only build something that looks like SIP.

By SDP 'client side at JavaScript level' are you proposing that the JS API
accepts SDP directly as a parameter, or that you parse the SDP in JS and set
the ICE connection and codecs up as appropriate? (Which is my preference.)

---

From: Neil Stratford <[email-masked]>
Date: Wed, 5 Oct 2011 12:00:12 +0100

On Wed, Oct 5, 2011 at 7:21 AM, Roman Shpount <[email-masked]> wrote:

>
> On Wed, Oct 5, 2011 at 12:25 AM, Matthew Kaufman <
> [email-masked]> wrote:
>
>> How about I rephrase?
>>
> I think you are completely missing the point: If I build similar thing
> using Win32 API, I would re-initialize codec and  RTP states and re-do the
> ICE candidate selection based on the new answer. This particular API has
> methods to add and remove streams and candidates, but no method to start a

I'm sure if it is required we can add the ability to re-start ICE
negotiation with a new set of farCandidates once the JS decides what to do
as a result of the fork. (I'm not a great supporter of the need for forking,
but if there are real use cases then lets add it.)

...

> support and cannot simply reuse MMUSIC/PAYLOAD standards.

I'm not sure I understand why this is such a problem. XMPP/Jingle defines
it's own mapping, which is a common VoIP protocol that is widely supported.
I'd much rather that the JS API is as expressive as possible in codec
specific matters - for example even to the point of providing callbacks to
the JS on certain codec level events, such as decoding a DTMF digit.

> browser, process it (if needed) in javascript or on the server and send it

Again, if it's required I don't see why we can't support asymmetric codecs
in the low level API. In my view a lot of this codec manipulation is much
better controlled by the JS which has application specific knowledge about
how to adapt to given situations than the browser itself. How would you
signal to the browser to dynamically lower framerate rather than image size,
or colour depth depending on the application use case? Giving the
application direct control over codec parameters seems a better option.

> noise codec, and DTMF to be used at the same time for a single stream.

Agreed - this is an oversight. Though I don't see any mention of DTMF in the
current PeerConnection draft either. I would propose in the low level API
that we add a DTMF codec with a callback on events and a function to
transmit. How would you do this in the PeerConnection API? How would you
specify to generate in-band DTMF on the ULAW codec rather than send an
RFC2833 digit?

> which attributes would end up in which of the objects.

Agreed. These all need to be added - if you have suggestions it would be
great to include them.

Neil

---

From: Roman Shpount <[email-masked]>
Date: Wed, 5 Oct 2011 13:20:50 -0400

> the JS on certain codec level events, such as decoding a DTMF digit.
What we need is some way to expose all possible codec parameter combinations
to JavaScript API. MMUSIC/PAYLOAD specifications for codec typically define
CODEC parameters and how they are encoded in SDP/MIME content type, but they
fall short of defining a way to list all the capabilities. For instance,
there is no way for SILK codec to specify valid bit rates range for a given
sampling rate. This means JavaScript would need intimate knowledge of each
codec implementation in the browser in order to implement codec negotiation.
If a different codec implementation is provided by the browser (for instance
SILK codec that supports more restricted bit rates for the same sampling
rate), JavaScript will not be able to produce valid negotiation results. As
far as I can see, there are two possible solutions: better uniform
description of codec capabilities to JavaScript API or delegating at least
some portion of codec negotiation to actual codec implementation.

> application direct control over codec parameters seems a better option.

I believe in current API we provide one codec with one set of parameters to
the API. We should at least be able to provide two codec groups -- one for
transmit/another to receive, with two sets of parameters. Even if we handle
all the negotiations in JavaScript, we should be able to provide the result
for both directions to the stream (may be with default value for the second
codec group to be the same as first since this is a much more common).
______________
Roman Shpount

---

From: Colin Perkins <[email-masked]>
Date: Thu, 6 Oct 2011 23:11:21 +0100

On 5 Oct 2011, at 18:20, Roman Shpount wrote:
> On Wed, Oct 5, 2011 at 7:00 AM, Neil Stratford <[email-masked]> wrote:
> I'm not sure I understand why this is such a problem. XMPP/Jingle defines it's own mapping, which is a common VoIP protocol that is widely supported. I'd much rather that the JS API is as expressive as possible in codec specific matters - for example even to the point of providing callbacks to the JS on certain codec level events, such as decoding a DTMF digit.
>  
> What we need is some way to expose all possible codec parameter combinations to JavaScript API. MMUSIC/PAYLOAD specifications for codec typically define CODEC parameters and how they are encoded in SDP/MIME content type, but they fall short of defining a way to list all the capabilities.

I'm not convinced that exposing all possible codec parameter combinations is an appropriate goal. There is simply too much flexibility present with many of the codecs, especially in combination, for the signalling to be reasonable if all the flexibility is exposed. Better, I think, for the browser to expose a set of known-reasonable combinations, and parameter ranges, from which the most appropriate choice is picked.

> For instance, there is no way for SILK codec to specify valid bit rates range for a given sampling rate. This means JavaScript would need intimate knowledge of each codec implementation in the browser in order to implement codec negotiation. If a different codec implementation is provided by the browser (for instance SILK codec that supports more restricted bit rates for the same sampling rate), JavaScript will not be able to produce valid negotiation results. As far as I can see, there are two possible solutions: better uniform description of codec capabilities to JavaScript API or delegating at least some portion of codec negotiation to actual codec implementation.
> 
> Again, if it's required I don't see why we can't support asymmetric codecs in the low level API. In my view a lot of this codec manipulation is much better controlled by the JS which has application specific knowledge about how to adapt to given situations than the browser itself. How would you signal to the browser to dynamically lower framerate rather than image size, or colour depth depending on the application use case? Giving the application direct control over codec parameters seems a better option.
> 
> I believe in current API we provide one codec with one set of parameters to the API. We should at least be able to provide two codec groups -- one for transmit/another to receive, with two sets of parameters. Even if we handle all the negotiations in JavaScript, we should be able to provide the result for both directions to the stream (may be with default value for the second codec group to be the same as first since this is a much more common).
> ______________
> Roman Shpount
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

-- 
Colin Perkins
http://csperkins.org/

---

From: Saul Ibarra Corretge <[email-masked]>
Date: Sat, 8 Oct 2011 11:18:43 +0200

> I'm not convinced that exposing all possible codec parameter combinations is an appropriate goal. There is simply too much flexibility present with many of the codecs, especially in combination, fo…

Maybe capabilities could be expressed in terms of "codecs" and "profiles". A profile is a set of properties attached to a codec under a given name, lets say codec H264 with "HD" profile or "SD" profile. Having the ability to make a custom profile would be nice, for those who need/want to have a more fine grained control over it.

-- 
Saúl Ibarra Corretgé
AG Projects