Subject: [rtcweb] Interaction between MediaStream API and signaling

From: Stefan Hakansson LK <[email-masked]>
Date: Fri, 30 Mar 2012 10:59:39 +0200

The JS API has deals with MediaStreams (this is what you send and 
receive using PeerConnection from an application perspective).

A browser receiving RTP streams, needs side info to be able to assemble 
those RTP streams into MediaStreams in a correct way. The current model 
is that this is signaled using SDP exchanges (where Haralds MSID 
proposal would tell which MediaStream an RTP stream belongs to).

As I brought up at the mike yesterday, I think we may have a race 
condition for the responder.

For the initiator side browser, this is clear: once an (PR-)ANSWER is 
received, the responder has received the SDP, and hence can map incoming 
RTP streams into MediaStreams.

But for the responder side this is less clear to me. Imagine 
applications where the responder just mirrors the initiator - if one of 
the parties adds a MediaStream to PeerConnection, the other end would 
add the corresponding MediaStream.

This can happen any time in the session, so ICE can very well be up and 
running. One example could be that the data channel is used for text 
chat, when one side clicks a button to start video. And the application 
can have asked for permission to use all input devices earlier, so no 
user interaction may be involved.

In this situation the responder's (added) RTP streams can very well 
arrive before the ANSWER if I understand correctly. I think we need to 
find a way to handle this. One way is to add an "ACK" that indicates to 
the responder that the initiator has received the ANSWER, but I'm not 
sure that is the best way.

Stefan

---

From: Randell Jesup <[email-masked]>
Date: Fri, 30 Mar 2012 17:39:31 -0400

On 3/30/2012 4:59 AM, Stefan Hakansson LK wrote:
> A browser receiving RTP streams, needs side info to be able to 
> assemble those RTP streams into MediaStreams in a correct way. The 
> current model is that this is signaled using SDP exchanges (where 
> Haralds MSID proposal would tell which MediaStream an RTP stream 
> belongs to).
>
> received, the responder has received the SDP, and hence can map 
> incoming RTP streams into MediaStreams.
>
> applications where the responder just mirrors the initiator - if one 
> of the parties adds a MediaStream to PeerConnection, the other end 
> would add the corresponding MediaStream.
>
> This can happen any time in the session, so ICE can very well be up 
> and running. One example could be that the data channel is used for 
> text chat, when one side clicks a button to start video. And the 
> application can have asked for permission to use all input devices 
> earlier, so no user interaction may be involved.
>
> arrive before the ANSWER if I understand correctly. 

Yes.  Just like in SIP.  And so when you send an OFFER (or modified 
re-OFFER), you must be ready to receive data per that offer even if no 
ANSWER has been received - just like in SIP.  And if its a re-offer, you 
need to accept the old, and accept the new (though you could probably 
use reception of obviously new-OFFER media to turn off 
decoding/rendering old-OFFER in preparation for the ANSWER).

The flip side of this is the responder has to infer when the sender 
switches over to the result of the ANSWER from the media.  For example:

A                                      B
<--- H.261 --->
re-OFFER(VP8) --->
<-- ANSWER(VP8) (delayed in reception)
<-----------VP8            (A should infer that B ANSWERed and accepted VP8)
  ----------> H.261
<-- ANSWER(VP8) (received)
<--------VP8----------> (B should infer by reception of VP8 that ANSWER 
was received)

(Personally, I hate inferences, but without a 3 (or 4) way handshake, 
you have to).  If you switches of codecs are staged, then this isn't 
(much) of a problem.  Either leave old codec on the list, or leave it on 
the list until accept, and then re-OFFER to remove the un-used codec.

One problem is what to do in the switchover window when you might get a 
mixture of old and new media, especially if you moved them to different 
ports and so can't count on RTP sequence re-ordering to un-mix them; in 
the past I dealt with that (and long codec-switch times) by locking out 
codec changes for a fraction of a second after I do one.  Not a huge 
deal, however.

My apologies if I've missed something in JSEP; I've been heads-down 
enough in Data Channels and bring-up that I could have a disconnect here 
and be saying something silly.

> I think we need to find a way to handle this. One way is to add an 
> "ACK" that indicates to the responder that the initiator has received 
> the ANSWER, but I'm not sure that is the best way.

If you need to know that, you need a SIP-style ACK.

-- 
Randell Jesup
[email-masked]

---

From: Stefan Hakansson LK <[email-masked]>
Date: Sat, 31 Mar 2012 07:01:45 +0200

On 03/30/2012 04:26 PM, Jim Barnett wrote:
> Doesn't the basic model require that the offer be ready to receive media
> as soon as it sends the offer?  If that's the case then the offer must
> be ready and able to map RTP streams to media streams (using the mapping
> specified in the SDP for its offer) as soon as it sends the offer.

Yes, the basic SDP o/a model requires that the offerer is ready to 
receive media as soon as the offer is sent. But as I tried to explain 
(but obviously failed to) is that there is an additional constraint 
here: the JS application deals with MediaStreams, which often has more 
than one track, e.g. one video and one audio track. A track in turn 
would correspond to an RTP stream. And you can have several concurrent 
MediaStreams in action.

Say now that the offerer, before getting an ANSWER with info, receives 
three RTP streams; two video and one audio. The offerer would in this 
situation not have a clue how these belong together. One could be video 
from a front facing camera (that is supposed to be displayed on a large 
video screen), the other video RTP stream could contain the video from 
the rear facing camera and is supposed to together with the audio RTP 
stream be one MediaStream that is supposed to be played in sync in a 
thumbnail video display.

Without the offerer having the info about what these RTP streams 
represent it can't do anything sensible.

Stefan

>
> - Jim
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Stefan Hakansson LK
> Sent: Friday, March 30, 2012 5:00 AM
> To: [email-masked]
> Subject: [rtcweb] Interaction between MediaStream API and signaling
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Stefan Hakansson LK <[email-masked]>
Date: Sat, 31 Mar 2012 07:17:47 +0200

> the list until accept, and then re-OFFER to remove the un-used codec.

I think I understand what you mean, and this would work fine as long as 
you just switch codecs that are used in already set-up MediaStreams.

But if A in this case, as part of re-OFFERING the session, not only 
offers a new codec (VP8) for the already flowing video but also adds a 
new outgoing video stream (e.g. front cam), and then (without receiving 
the ANSWER - delayed in reception) starts receiving VP8 video it could 
not really know if this VP8 video is new video from the responders front 
cam or just a new codec for the existing (back cam) video from the 
responder to the sender.

> and be saying something silly.

Actually I don't think this is very JSEP related; it is the generic 
problem that the browser receiving RTP streams need some side info about 
them before being able to do anything sensible with them.

> If you need to know that, you need a SIP-style ACK.

As explained, I do think we need to know that.

---

From: Stefan Hakansson LK <[email-masked]>
Date: Sat, 31 Mar 2012 07:41:03 +0200

> cam or just a new codec for the existing (back cam) video from the

This may have been a very bad example. Probably you can tell them apart 
on the SSRC. But even so, the A browser won't know what the VP8 stream 
(if it has an unknown SSRC) represents without receiving the ANSWER.