Subject: [rtcweb] Questions to consider when analyzing the signaling proposals

From: Magnus Westerlund <[email-masked]>
Date: Tue, 24 Jan 2012 09:37:04 +0100

WG,

At the interim we are attempting to come to some conclusions on the
signaling proposals. The WG chairs are aware that parts of the arguments
between the proponents will likely be related to API more than the
general functionality. We have discussed this with the W3C WEBRTC WG
chairs, to enable a constructive session we do need to allow such
discussion. But please remember that the formal decision power regarding
any questions for the API is in the W3C, this WG can only recommend or
formulate requirements on the API for them to consider.

It is also important that we don't forget important aspects of the
signaling and ensure that it functions correctly. Therefore I have put
together this email which contains a number of questions or issues
related to signaling that is likely important. The reason for this is to
provide food for thought in the evaluation of the proposals by WG
participants. I also recommend the proponents to try to answer how their
proposals handles the listed questions or issues.

There is no particular order among the issues, they are alphabetically
numbered to help you keep track of them. I also encourage the WG
participants to submit additional items for discussion. If you want to
discuss a particular item, please change the subject line so that one
keep track of the threads.

A) Glare handling: How does the signaling handle situations where both
end-points initiate negotiation, updates or re-negotiate at the same
time (i.e. messages pass each other in-flight between the peers)?

B) Can media clipping occur at the initial media establishment? Can it
occur at re-negotiations? Initially this shouldn't happen as long as
media configuration is in place prior to ICE promotes any candidate
pairs. However, at media re-negotiation this could occur. And when you
split the transport setup from media negotiation it becomes an issue again.

C) Does the glare handling result in periods where either side is
prevented from updating offers? For example is it possible to send a new
offer immediately after sending an answer to the peer's Offer? This
becomes important as we are likely having declarative media stream
information, like the Media Stream ID mapping to SSRCs for each media
stream an end-point intends to send.

D) Is it possible to provide additional ICE candidate's after the
initial offer? What level of complexities does that incur?

E) Is it possible to establish transport session which completes ICE
processing without negotiating any media, or at least without forcing
the user to assign media resource to it (not having to run getUserMedia
in the API)? Some versions of WhatWG API did establish a data channel by
default, which could be one option but other possibilities do exist.

F) How does one support forking, and specifically can one deal with;

 - F1) Parallel forking? I.e. deal with multiple answers to
       a given offer and establish multiple flows?

 - F2) Sequential/Serial Forking? i.e. receive one answer and
       later get a second answer to the same initial offer and
       use that to replace the first?

G) How does the proposal attempt to improve the time until media is
negotiated and can flow between the peers.

H) What explicit indication or other hints can the upper layer get from
the proposal get that this might be a suitable time to perform a
re-negotiation or is it automatically triggered by the browser core?

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 24 Jan 2012 11:27:22 +0100

In accordance with the chair's request, I'm starting a new thread on 
this issue. There will be more.

> time (i.e. messages pass each other in-flight between the peers)?
I believe that this question has an underlying assumption that this is a 
well-formed question for all cases.

In the case of a specification where the offer/answer state machine is 
not part of the standard implemented in the browser, I believe this 
question is not well-formed.

Instead, it needs to be broken down further:

- Is it clear which set of configuration-changing API calls on each side 
leads to a state where communication happens?
- Is it clear which set of configuration-changing API calls on each side 
can lead from a given configuration state to another given configuration 
state?
- Is it clear which set of configuration-changing API calls start, stop 
and change media flows, and when their effects happen?
- Are there sequencing constraints between these API calls on the two 
sides that can lead to non-communication if the sequencing constraints 
are not obeyed?

As an extreme example, consider a case where, for JSEP supporting a 
peer-to-peer chat, a centralized application takes people's createOffer 
SDP blobs, computes an appropriate SetLocalDescription SDP blob and a 
SetRemoteDescription SDP blob for them, and transmits those to each 
participant. As long as nobody starts sending media until the 
configurations have been installed on each side, this should work fine, 
no matter what the sequence of calls is - and there is no possibility of 
glare, since any time something is renegotiated, it is mediated by the 
central processor - the local app never makes independent decisions to 
do SetLocal or SetRemote.

                    Harald

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 24 Jan 2012 14:20:46 +0100

Thanks Harald for splitting out a new name.

Comments as individual

> question is not well-formed.

Agreed, and indeed my intention was to indicate areas well worth
thinking over in more details. And yes, there are differences between
the proposals that have different set of issues or considerations.

> sides that can lead to non-communication if the sequencing constraints 

Highly relevant considerations. But there is a clear overlap with B)
(Media clipping).

> do SetLocal or SetRemote.

To my understanding this can not cause GLARE in the sense that offers
from the different participants don't occur due to a single central
nodes decides.

However, I think your bullet:
> - Is it clear which set of configuration-changing API calls start,
> stop and change media flows, and when their effects happen?

Is applicable and causing a situation where things aren't clear and
communication fails unless some ordered sequence happens.

One alternative:

1. Server(S) orders all Clients(C) to halt media transmission.
2. S awaits sufficient time, or get reports that the last fragment of
media has been delivered or is considered lost by all Cs.
3. S sends out new local and remote configurations.
4. S awaits acknowledge from all Cs.
5. Sends out a command that media transmission can be started.

Another alternative:
If the extended media halting is to be avoided, one have to take care to
install local configurations that are both capable of the previous
remote configuration and the new remote configuration. When that has
occurred then a new remote configuration can be configured. This to
avoid that any outstanding media packets gets dropped at reception due
to failure to meat the configuration for incoming packets.

This is to my knowledge deeply encoded in SDP Offer/Answer in that at
the point of sending any Offer the reception must be ready to receive
what is being offered. To me this appears to spill over into JSEP also.
In fact it appears to prevent both the browser core and the JS
application manipulating the Offer and Answer from being non-aware of
this and keeping track of the state. The browser core will need to
always emit offers that has the property of being compliant with the
currently set LocalDescription if any. If the JS forgets about it
several things appear to possible to occur.

 - The Local description is reduced so that the configuration for the
currently received  media stream prior the peers remote description
being updated are not possible to decode when SetLocalDescription is done.

 - SetRemoteDesription is called at the Peer prior to the originating
side having done SetLocalDesciption, thus making the peers new media
configuration arrive prior to the local side being configured.

>From that perspective I think Section 5.4 of the JSEP draft 00 is way to
unspecified to avoid communication interruptions.

I also think it is a bit misleading to say that JSEP manages to decouple
the browser core from the Offer/Answer state machine.

Cheers

Magnus Westerlund

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Jan 2012 17:44:21 +0100

> split the transport setup from media negotiation it becomes an issue again.

The experience we have had when doing SIP-like negotiation is that an 
important thing causing clipping  (or rather, errors that have the net 
result of clipping) is the lack of crypto keys at the initiator for 
decoding incoming traffic.

This occurs at the sender when the following things occur:

- Initiator sends an offer with his encryption keys and ICE candidates
- Initiator starts listening for ICE exchanges on his proposed ports
- Responder sends an answer with his encryption keys and ICE candidates
- Responder starts *and completes* ICE negotiation
- Responder starts sending media
- Response (SDP answer) arrives at sender.

The result is that the initiator doesn't have the crypto keys to decrypt 
the responder's media.

This will of course happen only when signalling messages are slow and 
media is fast; a common case for that is when the two ends are in the 
same office while the signaling server is on another continent.

One solution suggested in ROAP was the "OK" message, which went from 
initiator to responder, and delaying the sending of media until you see 
an "OK"; this will of course not transfer across a SIP gateway, since 
SIP has no corresponding message.

If keys are negotiated inband, as with DTLS-SRTP, this problem 
disappears, since the recipient will know when the sender knows the 
crypto keys (the negotiation has completed).

If the key exchange was one where the responder suggested the key that 
the initiator should use for encryption, and vice versa, the problem 
would also not occur (the decrypting party would always know the key 
before the encrypting party), but that was not our reading of the RFCs.

Note: I think the same problem can also occur if the responder is free 
to add codecs, track descriptions and so on in the answer that are not 
in the offer, but this has not been a clearly discernible problem in our 
deployments yet.

                     Harald

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 25 Jan 2012 18:13:34 +0100

2012/1/25 Harald Alvestrand <[email-masked]>:
> The result is that the initiator doesn't have the crypto keys to decrypt the
> responder's media.
>
> This will of course happen only when signalling messages are slow and media
> is fast; a common case for that is when the two ends are in the same office
> while the signaling server is on another continent.

Maybe I'm wrong, but in case the sender sends an offer with with his
encryption keys, then the SDP answer MUST also contain encryption keys
(am I wrong?).

If so, the sender could just ignore any RTP received before the SDP answer.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Jan 2012 18:29:50 +0100

On 01/25/2012 06:13 PM, Iñaki Baz Castillo wrote:
> 2012/1/25 Harald Alvestrand<[email-masked]>:
Yes, the loss of those packets is what causes the clipping.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 25 Jan 2012 18:44:16 +0100

> Yes, the loss of those packets is what causes the clipping.

Well, then this is just an audio processing question. It's not hard at
all to alter an audio wave by applying a volume linear mask from 0 to
100% during the first ~50ms after receiving the first valid RTP packet
(once the encryption keys have been received). That would remove the
audio clipping. Not hard at all.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Jan 2012 18:56:44 +0100

> audio clipping. Not hard at all.
It's a bit harder if those packets were the first parts of the first 
I-frame on a video connection.

---

From: Randell Jesup <[email-masked]>
Date: Wed, 25 Jan 2012 09:59:11 -0800

> crypto keys (the negotiation has completed).

Another mark in favor of DTLS-SRTP :-)

> before the encrypting party), but that was not our reading of the RFCs.

I believe you are correct; I vaguely remember discussion over this years 
ago in AVT or one of the related IETF groups, and there was some 
security-based reason for it.  Perhaps because it increases the 
vulnerability to MITM attacks.

> in the offer, but this has not been a clearly discernible problem in our

In that case I would guess the responder might wait until it got a 
response from the initiator indicating reception (3-way handshake) 
before sending.  We've used that trick in the past to deal with certain 
routers set up with a DMZ device; if traffic arrives before the 
initiator starts sending on that port (i.e. media from responder beats 
the OK), then the router will send the media to the DMZ and lock in that 
mapping.  Ouch.  Only reasonable solution was to wait to send media 
until ACK.  It did help with some issues like clipping, and users didn't 
get tripped up by it because we didn't show the other person or change 
to the in-call UI until we were ready to send.

The wonders of annoying routers...

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Wed, 25 Jan 2012 10:07:07 -0800

> I-frame on a video connection.

Or the "Hel" of "Hello?" - quite annoying, and that's a tough case when 
a use case is "click to answer" - you don't have the 0.1-0.5s delay for 
the answerer to get a handset to their mouth.  When I refer to 
"clipping" in this context, that's the sort of thing I'm referring to 
(and Harald's as well).

-- 
Randell Jesup
[email-masked]

---

From: Justin Uberti <[email-masked]>
Date: Wed, 25 Jan 2012 15:35:48 -0500

On Wed, Jan 25, 2012 at 1:07 PM, Randell Jesup <[email-masked]>wrote:

>>> 2012/1/25 Harald Alvestrand<harald@alvestrand.**no<[email-masked]>
>>> >:
>>>
> Or the "Hel" of "Hello?" - quite annoying, and that's a tough case when a
> use case is "click to answer" - you don't have the 0.1-0.5s delay for the
> answerer to get a handset to their mouth.  When I refer to "clipping" in
> this context, that's the sort of thing I'm referring to (and Harald's as
> well).

Right. Note that this is not fully solved by the OK message mentioned
above, since even if the answerer waits to get the OK before transmitting,
the "Hel" may be already lost.

The best approach I've seen so far to deal with this is to allow the
transport and encryption functionality to be brought up while waiting for
the answerer to respond, so that transmission can begin immediately upon
answer.
(e.g. using trickle candidates and DTLS-SRTP)

>
>
> --
> Randell Jesup
> [email-masked]
>
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Jan 2012 21:48:32 +0100

On 01/25/2012 09:35 PM, Justin Uberti wrote:
>
>
> On Wed, Jan 25, 2012 at 1:07 PM, Randell Jesup <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>             2012/1/25 Harald Alvestrand<[email-masked]
>             <mailto:[email-masked]>>:
>
>                     If so, the sender could just ignore any RTP
>                     received before the SDP
>                     answer.
>
>     Or the "Hel" of "Hello?" - quite annoying, and that's a tough case
>     when a use case is "click to answer" - you don't have the 0.1-0.5s
>     delay for the answerer to get a handset to their mouth.  When I
>     refer to "clipping" in this context, that's the sort of thing I'm
>     referring to (and Harald's as well).
>
>
> above, since even if the answerer waits to get the OK before 
> transmitting, the "Hel" may be already lost.
>
> transport and encryption functionality to be brought up while waiting 
> for the answerer to respond, so that transmission can begin 
> immediately upon answer.
Note - this is something that is strongly application dependent.

If the answerer has no expectation of privacy of location, like a call 
center, it makes perfect sense to do the whole offer/answer/crypto/so-on 
exchange while waiting for the human to react to the ring tone.

If the answerer has expectation of privacy of location, or even of 
presence, he may not want to send *any* packet (including ICE probes) in 
the direction of the caller until he's thought carefully about whether 
or not he has sufficient proof that the caller is someone who he's 
comfortable with knowing his network location, and made his choice known 
by hitting "answer" or "reject".

In the first case, clipping should be a non-problem; machines are 
usually faster than humans.

In the second case, the "...lo" problem can only reasonably be solved by 
shaping the dialog so that the user doesn't think he can start speaking 
until everything's set up.

In both cases, much can be done if we expose to the application the 
state of media flow (not ready / ready / flowing).

---

From: Randell Jesup <[email-masked]>
Date: Wed, 25 Jan 2012 13:17:36 -0800

>> <mailto:[email-masked]>> wrote:
[SNIP]
> by hitting "answer" or "reject".

Correct - the use-case discussed before was someone calling their 
estranged wife and sniffing the network traffic to figure out where 
she's hiding from him (to use a particularly nasty example).  I won't 
repeat it all; it's in the archives, but it may make sense to review.

> until everything's set up.

Right; that's largely how we did it - it's easier when you have visible 
UI, and doubly so when there's an expectation of seeing video from the 
far end.

> state of media flow (not ready / ready / flowing).

Yes.

-- 
Randell Jesup
[email-masked]

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 25 Jan 2012 22:18:11 +0100

> this context, that's the sort of thing I'm referring to (and Harald's as

Ok, that's not what I call "clipping" in the audio/music world ;)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 25 Jan 2012 23:31:17 +0100

Hi Harald,

I may have missunderstood something, but in your use-case the crypto keys were sent already in the first answer, together with the ICE candidates.

So, when the ICE negotiation etc has finished, and the "final" answer is sent, the offerer already has the keys.

>From my experience, clipping during call setup it not a big real life problem.

TECHNICALLY, clipping during session modification is more difficult, especially if the change affects IP, CPU resources. In the SIP world it is said that an offerer must be able to handle media both using the old media parameters, and the new ones offered, but it is not always possible. But, I am not sure whether it's a real life problem either.

You also said:

"One solution suggested in ROAP was the "OK" message, which went from
initiator to responder, and delaying the sending of media until you see
an "OK"; this will of course not transfer across a SIP gateway, since
SIP has no corresponding message."

In SIP you do have the precondition mechanism (which you could use to not even alarm the remote user until the ICE negotiations etc have completed).

Or, you could simply use the direction attributes. Keep the connection "inactive" until you're ready to receive media.

How well it would map with ROAP, I haven't checked :)

Regards,

Christer

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Iñaki Baz Castillo [[email-masked]]
Sent: Wednesday, January 25, 2012 11:18 PM
To: Randell Jesup
Cc: [email-masked]
Subject: Re: [rtcweb] Clipping (Re: Questions to consider when analyzing the signaling proposals)

--
Iñaki Baz Castillo
<[email-masked]>
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Justin Uberti <[email-masked]>
Date: Wed, 25 Jan 2012 23:12:43 -0500

On Wed, Jan 25, 2012 at 4:17 PM, Randell Jesup <[email-masked]>wrote:

>>> <mailto:[email-masked]**>> wrote:
>>>
>>>  [SNIP]
>
>
Or even before playing the ring tone.

> estranged wife and sniffing the network traffic to figure out where she's
> hiding from him (to use a particularly nasty example).  I won't repeat it
> all; it's in the archives, but it may make sense to review.

Right; I recall the solution suggested for this case was to only use TURN
candidates until the call had been accepted, although I don't think it's
been scrutinized.

> UI, and doubly so when there's an expectation of seeing video from the far
> end.
>
>

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 27 Jan 2012 10:45:16 -0700

Here are the answers for ROAP to the questions Magnus sent.  As Magnus asked, if you want to respond to one of these, please change the subject line so that we can keep track of the threads. 

At a high level both sides realize that each has tried to change the SDP at the same time, one side “wins” and that side’s SDP attempted change is used and the losing side responds to it - the losing side is informed of an error so it can retry their change at an appropriate time. At all points both sides have a consistent view of the evolving state of the SDP offer/answer pair which is critical for there not to be confusion about matching m-lines in SDP. Each side needs to implement the same algorithm for determining the winner; the specific algorithm in ROAP is designed to both work for federated use cases and to be interoperable with SIP without state on the SIP gateway.

The only way media clipping can occur at initial establishment is if the callee decides to send media before a valid ICE candidate has been found.  If the systems wishes to make sure that ICE gets set up an early as possible, probably the best thing to do is to use an initial offer with only a data channel possibly marked as inactive and full muxing, so that that channel can the be used for the media. This will require some thought about ensuring address privacy and making sure there is a hint in the API that restricts ICE to relayed candidates until the caller has answered (or otherwise approved revealing local IP addresses).

RFC 3264 prohibits sending offers at certain times, such as when you have an offer outstanding. ROAP maintains those invariants. The support for final answers allows the callee side to have the necessary control to indicate if it needs to keep the offer outstanding or not. This is designed both to meet use cases for WebRTC and to be possible to gateway to SIP without a media gateway. It may turn out that we want to define some other set of messages that can say things that are “safe” when the offer is outstanding, as below (see question D) with ICE candidates, but a large class of changes can’t be treated this way without breaking the 3264 model.

Yes - using an updated offer but as discussed above, there are limits of when that can be sent. It’s pretty clear we could extend ROAP to allow additional ICE candidates at any phase by adding a CANDIDATE message that can only contain the new candidates. We haven’t done it because we’re uncertain how much additional benefit there is.

I think the question you’re asking here is whether I can arrange to complete ICE for all media streams without actually having to send or receive any media. From a ROAP perspective, this is principally an SDP question: the answerer sends a tentative answer with all the streams marked as “inactive” but provides ICE candidates. This will cause the offerer and the answerer to do ICE right away, and then when the callee decides to accept the phone call, a new final answer is sent with the streams marked as active. Note that as ROAP is separate from the signaling, the application could be designed such that this can all happen before the UI alerts the callee in any way. 

The above should work even without bundling, but obviously if it’s performance you’re after then you also want bundling, so we need to make sure that the bundle draft supports bundling even if (a) the streams are inactive and (b) I eventually reject one or more of the streams. I think this is probably true, but we need to check it.  The advantage of this is that you then don’t need to worry about doing unnecessary ICEs for media channels which you don’t end up using. If someone offers Audio and Video and we don’t bundle, then I have to do ICE on both, even if I eventually only take Audio. However, if we know we’re bundling here, then the ICE cost is the same regardless.

The answer to question G also sheds some light on this question

We can either choose Same as SIP or not allow this. I don’t care and it does not change ROAP. 

Yes, this is legal, same as in SIP. This is just the simple case of the final answers replaces any existing provisional answers. 

So, as answered in (E) above, you can get ICE going prior to having the callee agree to answer, which helps a lot.

There are a number of optimizations available beyond that:

1) TURN can start whenever the JS application wishes including starting when the page is loaded or when a user opens a dialog to select which other party they wish to communicate with. [Though we probably need some new API support for this]. 

2) The offerer can offer only a data channel then later add voice and video, thus allowing parallelism between the ICE setup and the authorization dialog. Obviously you’d want to use bundle here, which imposes a requirement on the bundle draft that you be able to negotiate use of bundle even if initially you only had one stream and to add streams to the bundle later. The current bundle draft looks like it supports this.

Both of these optimizations are designed to set up a speculative channel that can then be quickly upgraded to a media channel with a signaling RT (actually one-way signaling and then media flows immediately in the opposite direction.) Note that you can actually start (2) even if the client hasn’t indicated they want to make a call, if you’re going to be really aggressive. This only makes sense, of course, if there is a high probability that they will call this particular person.

In most cases, using this type of approach would allow the callee to start sending media as soon as the user had accepted the call with no media clipping. And once the callee had accepted the call, we would have the one way latency of notifying the caller of this and then the caller can send media. 

Currently ROAP assumes an API where the browser calls a callback in the JS to tell it a new offer is ready and pass the new offer to the JS application. 

An alternative design would be that the callback only indicates that something has changed and that the application needs to prompt the browser core to generate a new offer. [Note that we’ll need to be careful with the API design to make sure that odd stuff doesn’t happen if the application is slow about requesting the offer, so the events pile up.] This would be a little closer to how JSEP works and still work fine in ROAP, since it would put the state changes at the control of the app, so it seems like a fine alternative to consider.