Subject: [rtcweb] ROAP Example Application

From: Cullen Jennings <[email-masked]>
Date: Thu, 20 Oct 2011 20:08:18 -0600

I liked Dan Yorks rant on what we need and it made me want to show some simple code using an interface like the current W3C PeerConnection API along with ROAP. Assuming the ROAP objects got passed in and out of the Peer Connection object, here is a short little example I copied that shows how simple it is to write a web application that sets up an audio / video connection between two browsers. It assumes a simple web server that that can pass messages from alice to bob and visa versa. Both alice and bob just short poll the web server to get messages from their mailbox and post messages to the others mailbox. The web server does not transform the messages in any way. Clearly this example is wrong in some ways, lacks authentication, and various error handling but I think it illustrates the basics of the interfaces. 

var ui_log = function(msg) {
    var t = document.createTextNode(msg);
    var d = document.createElement("div");
    $(d).append(t);

    $("#logwindow").append(d);
};

// simple wrapper around XHR. The cool kids use JQuery here, but
// we want a self-contained example
var ajax = function(params) {
    var xhr = new XMLHttpRequest();
    xhr.open(
	params.type || "GET",
	params.url,
	true);
    if(params.contentType){
	xhr.setRequestHeader("Content-Type",params.contentType)
    }
    if(params.data){
	xhr.send(params.data);
    }
    else{
	xhr.send();
    }
    xhr.onreadystatechange = function() {
	if(xhr.readyState === 4) {
	    if(xhr.status === 200) {
		if(params.success){
		    params.success(xhr.responseText);
		}
	    }
	    else {
		if (params.error) {
		    params.error(xhr.responseText);
		}
	    }
	}
    }
};

var ROAPClient = function(username, peer, start_call) {
    var poll_timeout = 1000; // ms
    
    var log = function(msg) {
	console.log("LOG (" + username + "): " + msg);
	ui_log("LOG (" + username + "): " + msg);
    };
    
    var signaling = function(msg) {
	msg.dest = peer;

	log("Sending: " + JSON.stringify(msg));

	ajax({
		 url : "/msg/",
		 type:"POST",
		 contentType:"application/json",
		 data: JSON.stringify(msg),
	     });
    };

    var poll_success = function(msg) {
	var js = JSON.parse(msg);
	log("Received message " + JSON.stringify(js));
	
	pc.processSignalingMessage(js);
    };
        
    var poll_error = function(msg) {
	setTimeout(poll, poll_timeout);
    };

    var poll = function() {
	ajax({
		 url: "/msg/" + username + "/",
		 success:poll_success,
		 error:poll_error
	     });
    };

    var pc = new PeerConnection({}, signaling);
    
    if (start_call) {
	pc.addStream();
    }

    // Start polling
    poll();
};

var ROAPTest = function() {
    var caller = new ROAPClient("alice", "bob", true);
    var callee = new ROAPClient("bob", "alice");
};

---

From: Matthew Kaufman <[email-masked]>
Date: Thu, 20 Oct 2011 23:46:42 -0700

> I liked Dan Yorks rant on what we need and it made me want to show some simple code using an interface like the current W3C PeerConnection API along with ROAP. Assuming the ROAP objects got passed…

Thanks for the example.

However, in addition to illustrating the basics, it also illustrates 
that ICE parameters and codec SDP parameters are now inextricably 
intertwined within this proposal, where in fact there is a good argument 
that not only should they be kept separate but that in fact they are the 
responsibility of different objects within the JavaScript API.

Matthew Kaufman

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 21 Oct 2011 13:00:12 +0200

On 10/21/2011 08:46 AM, Matthew Kaufman wrote:
>> I liked Dan Yorks rant on what we need and it made me want to show 
>> some simple code using an interface like the current W3C 
>> PeerConnection API along with ROAP. Assuming the ROAP objects got 
>> passed in and out of the Peer Connection object, here is a short 
>> little example I copied that shows how simple it is to write a web 
>> application that sets up an audio / video connection between two 
>> browsers. It assumes a simple web server that that can pass messages 
>> from alice to bob and visa versa. Both alice and bob just short poll 
>> the web server to get messages from their mailbox and post messages 
>> to the others mailbox. The web server does not transform the messages 
>> in any way. Clearly this example is wrong in some ways, lacks 
>> authentication, and various error handling but I think it illustrates 
>> the basics of the interfaces.
>>
>>
>
> Thanks for the example.
>
> intertwined within this proposal, where in fact there is a good 
> argument that not only should they be kept separate but that in fact 
> they are the responsibility of different objects within the JavaScript 
> API.

As I've noted before, they are more entangled than one might like them 
to be; if we don't support the extension to use a single RTP session, we 
don't know how many ICE sessions we need to set up until we know at 
least how many top level media types we want to support. If we want to 
support multiple RTP sessions for other reasons, we need similar control 
of the number of ICE sessions set up.

It illustrates that while a lot of the concerns of codecs and ICE 
sessions are reasonably separate, the functions that manipulate RTP 
sessions need access to both.

                     Harald

---

From: Tim Panton <[email-masked]>
Date: Fri, 21 Oct 2011 12:13:20 +0100

On 21 Oct 2011, at 12:00, Harald Alvestrand wrote:

> 
> As I've noted before, they are more entangled than one might like them to be; if we don't support the extension to use a single RTP session, we don't know how many ICE sessions we need to set up until we know at least how many top level media types we want to support. If we want to support multiple RTP sessions for other reasons, we need similar control of the number of ICE sessions set up.

Who is 'we' ? Surely the decision of how many top level media types to support _must_ be taken at the 
application layer. The application has to make screen real-estate available for things like video, keypads etc.

> 
> It illustrates that while a lot of the concerns of codecs and ICE sessions are reasonably separate, the functions that manipulate RTP sessions need access to both.

And in my view the javascript needs access to _all_ 3 of them.

Tim.

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 21 Oct 2011 07:13:57 -0700

On 10/21/11 4:00 AM, Harald Alvestrand wrote:
>>> to the others mailbox. The web server does not transform the 
>>> messages in any way. Clearly this example is wrong in some ways, 
>>> lacks authentication, and various error handling but I think it 
>>> illustrates the basics of the interfaces.
>>>
>>>
>>
>> Thanks for the example.
>>
>> they are the responsibility of different objects within the 
>> JavaScript API.
>
> to be;

This is just an argument that RTP is actually a poor choice as well, but 
I'm afraid it is too late for that.

> if we don't support the extension to use a single RTP session, we 
> support multiple RTP sessions for other reasons, we need similar 
> control of the number of ICE sessions set up.

But none of these examples actually shows that there's a requirement to 
combine the two into a single blob.

An easy solution to these examples, for instance, would be to exchange 
the codec-choosing blobs *first* and then, based on the results, examine 
the "encoder" and "decoder" objects to see how many transport streams 
they need attached. Or, even easier, when you wire the media 
encoding/decoding objects to the transport object, the transport object 
can know what it will need to be transporting and, once it receives an 
internal event that the codec and RTP parameters have been selected, it 
can fire an event to JavaScript containing the ICE setup blob.
> sessions need access to both.
Actually this suggests to me that there's *three* blobs. One for codec 
parameters, one for RTP, and one for ICE.

Matthew Kaufman

ps. I understand the desire to have the codec selecting and codec 
parameter parts of this work in such a way that a pair of new browsers 
that have new codecs can automatically upgrade to the latest and 
greatest, even if the web site in between doesn't know about it... but 
that argument doesn't appear to apply to the RTP or ICE, which suggests 
that the "opaque blob" model isn't even required for 2/3rds of this.

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 21 Oct 2011 07:22:03 -0700

On Fri, Oct 21, 2011 at 7:13 AM, Matthew Kaufman
<[email-masked]> wrote:
>>>> I liked Dan Yorks rant on what we need and it made me want to show some
>>>> simple code using an interface like the current W3C PeerConnection API along
>>>> with ROAP. Assuming the ROAP objects got passed in and out of the Peer
>>>> Connection object, here is a short little example I copied that shows how
>>>> simple it is to write a web application that sets up an audio / video
>>>> connection between two browsers. It assumes a simple web server that that
>>>> can pass messages from alice to bob and visa versa. Both alice and bob just
>>>> short poll the web server to get messages from their mailbox and post
>>>> messages to the others mailbox. The web server does not transform the
>>>> messages in any way. Clearly this example is wrong in some ways, lacks
>>>> authentication, and various error handling but I think it illustrates the
>>>> basics of the interfaces.
>>>>
>>>>
>>>
>>> Thanks for the example.
>>>
>>> However, in addition to illustrating the basics, it also illustrates that
>>> ICE parameters and codec SDP parameters are now inextricably intertwined
>>> within this proposal, where in fact there is a good argument that not only
>>> should they be kept separate but that in fact they are the responsibility of
>>> different objects within the JavaScript API.
>>
>> As I've noted before, they are more entangled than one might like them to
>> be;
>
> This is just an argument that RTP is actually a poor choice as well, but I'm
> afraid it is too late for that.
>
>> if we don't support the extension to use a single RTP session, we don't
>> know how many ICE sessions we need to set up until we know at least how many
>> top level media types we want to support. If we want to support multiple RTP
>> sessions for other reasons, we need similar control of the number of ICE
>> sessions set up.
>
> An easy solution to these examples, for instance, would be to exchange the
> codec-choosing blobs *first* and then, based on the results, examine the
> "encoder" and "decoder" objects to see how many transport streams they need
> attached. Or, even easier, when you wire the media encoding/decoding objects
> to the transport object, the transport object can know what it will need to
> be transporting and, once it receives an internal event that the codec and
> RTP parameters have been selected, it can fire an event to JavaScript
> containing the ICE setup blob.

Note that Jingle in some cases does sort of the opposite of this:
The caller offers combined media encoding/decoding objects  and
transport objects (in a session-initiate) and the callee sends
back only transport objects (in a transport-info) and then the
combined media encoding objects and transport objects
(in a session-accept).

XEP-0167 has a good example of this.

-Ekr

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 21 Oct 2011 19:37:13 +0200 (CEST)

Thought experiment:

If one had a connection object that supported an underlying protocol that could send one or more encrypted flows of both real-time media and reliable data, what information would you need to exchange between the two browser endpoints in order to ensure that any pair of browsers with any future audio or video codec could use the best mutually-supported codecs?

(And note that if you had this, you could even send that information over the peer-to-peer data channel itself, once established.. Or you could send it up to a server... Your choice as programmer.)

Now, what does it take to swap out the protocol underneath that connection object with ICE, DTLS-SRTP and TCP-over-UDP (or equivalent) for the data?

If we can successfully do this, we will have achieved a much cleaner software architecture within the browser, separated the connection setup from the media setup, enabled the substitution of alternative transports, etc.

Matthew Kaufman

Sent from my iPad, on a plane

---

From: Ted Hardie <[email-masked]>
Date: Fri, 21 Oct 2011 12:01:58 -0700

On Fri, Oct 21, 2011 at 10:37 AM, Matthew Kaufman <[email-masked]
> wrote:

> Thought experiment:
>
> If one had a connection object that supported an underlying protocol that
> could send one or more encrypted flows of both real-time media and reliable
> data, what information would you need to exchange between the two browser
> endpoints in order to ensure that any pair of browsers with any future audio
> or video codec could use the best mutually-supported codecs?
>
>
I think I am missing your point slightly.  Do you mean the connection object
talks to an abstraction layer that manages how the flows are sent, and the
swapping occurs in what it manages, or do you mean the connection object
should be able to swap out whether it is talking to ICE, DTLS-SRTP, or
TCP-over-UDP transparently?  or something else?

regards,

Ted

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 21 Oct 2011 16:32:17 -0700

On 10/21/2011 12:01 PM, Ted Hardie wrote:
> On Fri, Oct 21, 2011 at 10:37 AM, Matthew Kaufman 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Thought experiment:
>
>     If one had a connection object that supported an underlying
>     protocol that could send one or more encrypted flows of both
>     real-time media and reliable data, what information would you need
>     to exchange between the two browser endpoints in order to ensure
>     that any pair of browsers with any future audio or video codec
>     could use the best mutually-supported codecs?
>
>
> I think I am missing your point slightly.  Do you mean the connection 
> object talks to an abstraction layer that manages how the flows are 
> sent, and the swapping occurs in what it manages, or do you mean the 
> connection object should be able to swap out whether it is talking to 
> ICE, DTLS-SRTP, or TCP-over-UDP transparently?  or something else?

I mean "what if the connection object was sitting on top of something 
like RTMFP", in other words a protocol that can trivially be asked to 
open a secure (encrypted and authenticated) NAT-traversing session 
between your browser and another browser and then can deliver multiple 
parallel flows of partially-reliable media and fully-reliable data 
(prioritized appropriately)  to the far end.

Then the next question I asked is, IF you had a connection object like 
that, what is the minimum you need to get the 
latest-and-greatest-codec-just-works behavior that Cullen, et. al. have 
described?

Matthew Kaufman

---

From: Hadriel Kaplan <[email-masked]>
Date: Sat, 22 Oct 2011 01:12:13 +0000

I think what you're getting at is: if we threw out SDP and re-designed how/what devices indicated/negotiated media-plan stuff, we wouldn't end up with SDP again - neither in syntax nor semantics nor even architecture, as we'd likely separate the higher-layer info portion, transport portion, and the pure codec-type negotiation stuff into separate components; and we'd probably move the codec negotiation into the media-path itself potentially, for example.  Is that what you mean?

I don't disagree with that, and it's been raised many times in the SIP-related working groups over the years.  We know SDP isn't "clean" in terms of either syntax or semantics, but obviously in the SIP world it was way too late to change, and folks now use the info all being together in the SIP signaling-plane to their advantage.

I think though you can sort of emulate the separation in RTCWeb, even with the SDP offer/answer and ROAP.  You create a MediaStream with a track of "data" type, put it in a PeerConnection, and issue the SDP Offer.  Then when the data connection is open to the peer Browser/device, you create your audio/video tracks/streams and add them in, and when the new Offer is issued by the Browser you send the ROAP over the data connection to the peer, who does likewise for the Answer.  That might work.

-hadriel

On Oct 21, 2011, at 7:32 PM, Matthew Kaufman wrote:

>> On Fri, Oct 21, 2011 at 10:37 AM, Matthew Kaufman <[email-masked]> wrote:
>> Thought experiment:
>> 
>> I think I am missing your point slightly.  Do you mean the connection object talks to an abstraction layer that manages how the flows are sent, and the swapping occurs in what it manages, or do you mean the connection object should be able to swap out whether it is talking to ICE, DTLS-SRTP, or TCP-over-UDP transparently?  or something else?
> 
> I mean "what if the connection object was sitting on top of something like RTMFP", in other words a protocol that can trivially be asked to open a secure (encrypted and authenticated) NAT-traversing session between your browser and another browser and then can deliver multiple parallel flows of partially-reliable media and fully-reliable data (prioritized appropriately)  to the far end.
> 
> Then the next question I asked is, IF you had a connection object like that, what is the minimum you need to get the latest-and-greatest-codec-just-works behavior that Cullen, et. al. have described?
> 
> Matthew Kaufman
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Randell Jesup <[email-masked]>
Date: Fri, 21 Oct 2011 23:25:02 -0400

> I think what you're getting at is: if we threw out SDP and re-designed how/what devices indicated/negotiated media-plan stuff, we wouldn't end up with SDP again - neither in syntax nor semantics no…

Actually, I think Matthew is thinking of something else entirely (though 
you're right that SDP is not what we'd do with a clean piece of paper - 
but that would be a multi-year effort, and in the end unless there was a 
major driving adopter, it would fall by the wayside like SDP-ng (and for 
that matter, cap-neg)).

> I think though you can sort of emulate the separation in RTCWeb, even with the SDP offer/answer and ROAP.  You create a MediaStream with a track of "data" type, put it in a PeerConnection, and issu…

This isn't what Matthew was thinking of - he wanted to know if with 
these tools, it would ease negotiating codecs that didn't exist when the 
web-app was written (especially with a low-level API to the browser and 
its codecs).

What you describe is simply a way for the JS App to do "renegotiations" 
without using the "high path" through the servers.  Eminently doable, 
but not fundamentally different.  It may make renegotiations faster.  No 
reason an app can't do that, and I have some services built on WebRTC in 
mind that may depend on this usage.

>>> On Fri, Oct 21, 2011 at 10:37 AM, Matthew Kaufman<[email-masked]>  wrote:
>>> Thought experiment:
>>>

-- 
Randell Jesup
[email-masked]

---

From: Hadriel Kaplan <[email-masked]>
Date: Sun, 23 Oct 2011 04:33:30 +0000

On Oct 21, 2011, at 11:25 PM, Randell Jesup wrote:

> This isn't what Matthew was thinking of - he wanted to know if with these tools, it would ease negotiating codecs that didn't exist when the web-app was written (especially with a low-level API to the browser and its codecs).
> 
> What you describe is simply a way for the JS App to do "renegotiations" without using the "high path" through the servers.  Eminently doable, but not fundamentally different.  It may make renegotiations faster.  No reason an app can't do that, and I have some services built on WebRTC in mind that may depend on this usage.

Well one really good reason to do it would be to be able to get a session on/off-hold (ie, start/stop RTP using SDP direction attributes), or add/remove streams, without clipping and without loading the Web Server down passing ROAP messages back/forth.  That might be almost necessary in some applications.  For example one where you have a lot of separate video sessions up on a screen and change which one is active at any given time by a mouseover/click.  Or a game application, where you might have a direct data channel open between all participants anyway, and want to do audio/video whenever the users are in the same area of the game or by team-commands or whatever.

-hadriel