Subject: [rtcweb] Fwd: [MMUSIC] SDP work needed for WebRTC stuff

From: Matthew Kaufman <[email-masked]>
Date: Tue, 11 Dec 2012 15:25:42 -0800

Adding RTCWEB again, thanks to new Thunderbird feature that has apparently bitten several of us now.

Matthew Kaufman

(Sent from my iPhone)

Begin forwarded message:

> From: Matthew Kaufman <[email-masked]>
> Date: December 11, 2012, 3:16:27 PM PST
> To: [email-masked]
> Subject: Re: [MMUSIC] SDP work needed for WebRTC stuff
> 
> On 12/8/2012 3:21 PM, Eric Rescorla       wrote:
>> +rtcweb
>> 
>> On Sat, Dec 8, 2012 at 2:41 PM, Cullen Jennings (fluffy) <[email-masked]> wrote:
>>> 
>>> I was looking over everything that needs to be completed to finish a fist cut of the WebRTC related work. There are a handful of big SDP problems that are currently blocking some of the WebRTC work and I'd like to figure out how to make some progress on them.
>>> 
>>> Let me loosely characterize them as
>>> 
>>> 1) If we have several video streams, how do theses map up to             1 or more m lines.
>>> 
>>> 2) if we are doing port multiplexing, what does the SDP look like (the bundle problem)
>>> 
>>> 3) How do we map the RTCWeb track and stream label concepts to identifiers in SDP
>>> 
>>> 3) SDP for application running over SCTP/DTLS
>> 
>>> I don't want to speak for all the various chairs but I am under the impression that most of chairs of related groups in W3C and IETF believe these are issues that need to be resolved primarily in the MMUSIC WG and that they impact             both WebRTC and CLUE as well as the general long term use of SDP in SIP and other protocols.
>>> 
>>> I'd like to get some discussion going on how we can make some progress on these. I don't think we are going to solve these in 20 minutes of discussion at an IETF meeting so I think we probably need some interim (virtual or face to face) to sort this out.
>>> 
>>> Thoughts?
>> 
>> Wow, I'm totally confused here.
>> 
>> I had assumed that the SDP-related issues were going to be the main
>> topics at the WebRTC/RTCWEB interim in January. Is that not the case?
>> 
>> IMO the lack of clarity around how to             encode various media
>> configurations into SDP is the major thing blocking progress here. In
>> particular, Firefox has opted not to implement multiplexing of media
>> streams over the same transport flow (whether of the bundle or
>> multiple m-line variety) until the SDP for it is well-defined. The
>> same thing applies to the question of             how to map multiple m-lines to
>> incoming MediaStreams/Tracks.
>> 
>> We really need to cover these issues             in the interim.
>> 
>> -Ekr
> 
> We only need to cover these issues soon if RTCWEB continues to insist that SDP is a good idea for an API *and* that changes to SDP are necessary in order for RTCWEB to then be successful.
> 
> I would posit that if SDP as used in (and referenced from) RFC3264 was the appropriate API surface for RTCWEB then NO ADDITIONAL WORK in mmusic would be required in order to ship RTCWEB 1.0.
> 
> Matthew Kaufman
> 
> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmusic

---

From: Roman Shpount <[email-masked]>
Date: Tue, 11 Dec 2012 19:02:33 -0500

+1.

I would argue that if you want something you can inter-operate with you are
better off with an API that clearly matches the capabilities of the system.
Building an API which uses under-specified text blobs (in this case SDP)
for interfaces usually ends up producing things that only offer check box
interoperability. I grant you it is much easier to specify or implement,
since it gives you a lot more implementation freedom, but your normally end
up with something that only work with another instance of the same
implementation.

BTW, using SDP from RFC3264 is not an option. This specification does not
define SDP. It defines a negotiation framework. SDP is defined in RFC 2327,
which is of no value since all it does is defines the basic structure. You
need RFC 3551 to describe how SDP is used for audio and video data. Then
you need another RFC for ICE. Another for AVPF. And the list goes on
(bundle, SCTP, specifying a port for RTCP, SRTP, DTLS-SRTP). Each of those
RFCs will specify some aspects that we need but will bring a number of
things that has no relevance. So, you would need to write another document
that describes how the features that we need are implemented using all of
those RFCs. I think we are better of just describing how we can control the
features that we need via the dedicated API methods and let JavaScript
application map it to the text blob, which can be SDP or can be just a JSON
object.
_____________
Roman Shpount

On Tue, Dec 11, 2012 at 6:25 PM, Matthew Kaufman <[email-masked]> wrote:

> *From:* Matthew Kaufman <[email-masked]>
> *Date:* December 11, 2012, 3:16:27 PM PST
> *To:* [email-masked]
> *Subject:* *Re: [MMUSIC] SDP work needed for WebRTC stuff*
>
>> I was looking over everything that needs to be completed to finish a fist
>> cut of the WebRTC related work. There are a handful of big SDP problems
>> that are currently blocking some of the WebRTC work and I'd like to figure
>> out how to make some progress on them.
>>
>  I don't want to speak for all the various chairs but I am under the
>> impression that most of chairs of related groups in W3C and IETF believe
>> these are issues that need to be resolved primarily in the MMUSIC WG and
>> that they impact both WebRTC and CLUE as well as the general long term use
>> of SDP in SIP and other protocols.
>>
>> I'd like to get some discussion going on how we can make some progress on
>> these. I don't think we are going to solve these in 20 minutes of
>> discussion at an IETF meeting so I think we probably need some interim
>> (virtual or face to face) to sort this out.
>>
>> Thoughts?
>>
>
> We only need to cover these issues soon if RTCWEB continues to insist that
> SDP is a good idea for an API *and* that changes to SDP are necessary in
> order for RTCWEB to then be successful.
>
> I would posit that if SDP as used in (and referenced from) RFC3264 was the
> appropriate API surface for RTCWEB then NO ADDITIONAL WORK in mmusic would
> be required in order to ship RTCWEB 1.0.
>
> Matthew Kaufman
>
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 12 Dec 2012 05:46:00 +0000

Hi Roman,

>BTW, using SDP from RFC3264 is not an option. This specification does not define SDP. It defines a negotiation framework. SDP is defined in RFC 2327, which is of no value 
>since all it does is defines the basic structure. You need RFC 3551 to describe how SDP is used for audio and video data. Then you need another RFC for ICE. Another for 
>AVPF. And the list goes on (bundle, SCTP, specifying a port for RTCP, SRTP, DTLS-SRTP). Each of those RFCs will specify some aspects that we need but will bring a number 
>of things that has no relevance. So, you would need to write another document that describes how the features that we need are implemented using all of those RFCs.

I assume that is what JSEP is trying to do.

Of course, there is no need to re-write everything in JSEP. For example, I still think the negotiation part shall be RFC3264.

>I think we are better of just describing how we can control the features that we need via the dedicated API methods and let JavaScript application map it to the text blob, which can be SDP or can be just a JSON object.

The proposal previously known as the Microsoft proposal, you mean?

Regards,

Christer

Matthew Kaufman

(Sent from my iPhone)

Begin forwarded message:

Thoughts?

-Ekr

Matthew Kaufman

---

From: "Hutton, Andrew" <[email-masked]>
Date: Wed, 12 Dec 2012 11:39:48 +0000

See below.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Christer Holmberg
> Sent: 12 December 2012 05:46
> To: Roman Shpount; Matthew Kaufman
> Cc: [email-masked]
> Subject: Re: [rtcweb] Fwd: [MMUSIC] SDP work needed for WebRTC stuff
> 
> Hi Roman,
> 
> >BTW, using SDP from RFC3264 is not an option. This specification does
> not define SDP. It defines a negotiation framework. SDP is defined
> in RFC 2327, which is of no value
> >since all it does is defines the basic structure. You need RFC 3551 to
> describe how SDP is used for audio and video data. Then you need
> another RFC for ICE. Another for
> >AVPF. And the list goes on (bundle, SCTP, specifying a port for RTCP,
> SRTP, DTLS-SRTP). Each of those RFCs will specify some aspects that we
> need but will bring a number
> >of things that has no relevance. So, you would need to write another
> document that describes how the features that we need are implemented
> using all of those RFCs.
> 

Agree that this is what JSEP is intended for and there is quite a bit of work to do to make JSEP in to a clear authoritative description of SDP usage for RTCWEB. 

Even without all the SDP enhancements needed (E.g. Bundle, SCTP/DTLS, Multiple audio/video streams) JSEP has to explain how the API uses SDP to meet the requirements and what the split of responsibilities is between the browser and the application.  For example adding a new stream during an existing session is not currently explained neither is how the sess-version in the o-line is maintained etc.  

Andy

---

From: Roman Shpount <[email-masked]>
Date: Wed, 12 Dec 2012 17:30:58 -0500

On Wed, Dec 12, 2012 at 12:46 AM, Christer Holmberg <
[email-masked]> wrote:

> Of course, there is no need to re-write everything in JSEP. For example, I
> still think the negotiation part shall be RFC3264.
>

I actually not 100% convinced that this should be the case (I know that
consensus is against me). Main issues that I have against basing the API on
RFC 3264 are:

1. This forces to negotiate transport parameters (ice candidates,
encryption keys) every time encoding parameters are changed. I do not see a
reason why we need to run ice state machine and re-negotiate encryption to
add or remove a bundled video stream or change encoding. To do this
efficiently we need to have a side channel and additional operations to RFC
3264

2. Dealing with parameters which are not negotiated via RFC 3264 is
cumbersome. For instance, how would you specify the complexity for OPUS
encoder? You would most likely need some other side channel.

3. Offer answer is not the only available scenario. One thing that was
discussed that each game or conference participant generates an offer,
keeps it active for the duration of the conference and sends it to the
server, where it is cached. When new participant joins the conference, it
gets cached offers for all the other participants from the server and
generates answers to them. This removes an extra signaling hop but
impossible with current API.

> The proposal previously known as the Microsoft proposal, you mean?

Microsoft proposal was closer to what I want but not quite done either.

Periodically I feel like we are all members of the wheel reinvention
comity, we all know what the shape should be, but cannot agree on the
color. The features we need are:

1. Ability to negotiate the transport parameters, ie generate the list of
ICE candidates, select encryption type (SRTP or DTLS-SRTP), get the ICE
candidates from remote side and run the ICE state machine (and if needed
DTLS negotiation). Once this is done we end up with a transport channel. If
we do this right we should be able to do forking: generate an offer and run
an ICE state machine for multiple answers to set up multiple connections
for the same offer. I would argue that all the required data parameters and
the require API for this is pretty clear.

2. Setup a mapping from connection, packet type (SCTP, RTP, RTCP) and  SSRC
to a RTP or RTCP sinc, from there, based on payload to codec decoder and
from there to media track. We should have an API which allows to change
these assignments on the fly without re-negotiating communication channels.

3. Setup a mapping from media track to codec encoder to RTP/RTCP sinc and
from there to a connection. Once again, we should have an API which allows
to change these assignments on the fly without re-negotiation communication
channels or processing of inbound packets.

If we organize the API like this it would take a lot less time
to standardize, will be a lot more flexible and will remove a lot of
dependencies from other working groups. I would also argue that this will
allow us to build something that has a much higher chance of interoperating
with real deployments then something based on text blobs defined across
four working groups for conflicting and unrelated purposes.

P.S. I am aware that this is fairly late in the game and Chrome/Firefox
will or are already released with something completely different. I would
still argue that it is better to build something that is clean and usable
later in the game, then keep on building kludges (like various hints in
multiple places).
_____________
Roman Shpount

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 13 Dec 2012 11:08:30 +0100

On 12/12/2012 11:30 PM, Roman Shpount wrote:
>
> On Wed, Dec 12, 2012 at 12:46 AM, Christer Holmberg 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Of course, there is no need to re-write everything in JSEP. For
>     example, I still think the negotiation part shall be RFC3264.
>
>
> I actually not 100% convinced that this should be the case (I know 
> that consensus is against me).

I'm not 100% convinced either, but I don't find your arguments compelling.

> Main issues that I have against basing the API on RFC 3264 are:
>
> encryption keys) every time encoding parameters are changed. I do not 
> see a reason why we need to run ice state machine and re-negotiate 
> encryption to add or remove a bundled video stream or change encoding. 
> To do this efficiently we need to have a side channel and additional 
> operations to RFC 3264
I don't know that this is the case. If RFC 3264 renegotiation is done 
without changing the crypto and transport parameters from what's 
currently in force, I can't see why the ICE and crypto state machines 
should change.
> cumbersome. For instance, how would you specify the complexity for 
> OPUS encoder? You would most likely need some other side channel.
Doesn't OPUS do that in-band?
If complexity needs to be specified before starting an OPUS connection, 
that is needed when OPUS is used in SIP too - so we're reducing this to 
a problem that needs to be solved anyway.
> server, where it is cached. When new participant joins the conference, 
> it gets cached offers for all the other participants from the server 
> and generates answers to them. This removes an extra signaling hop but 
I'm not sure it's impossible. I'd like to try to build it first and see 
where it fails.

One issue with such a scenario (in any model) is that ICE candidates 
decay quickly, due to NAT timeouts and so on. You need to refresh them 
before use - and that's independent of offer/answer.

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 13 Dec 2012 14:32:53 +0000

Hi,

>>I actually not 100% convinced that this should be the case (I know that consensus is against me).
>
>> 1. This forces to negotiate transport parameters (ice candidates, encryption keys) every time encoding parameters are changed. I do not see a reason 
>> why we need to run ice state machine and re-negotiate encryption to add or remove a bundled video stream or change encoding. To do this efficiently we need to have a side channel and additional operations to RFC 3264
>
> I don't know that this is the case. If RFC 3264 renegotiation is done without changing the crypto and transport parameters from what's currently in force, I can't see why the ICE and crypto state machines should change.

I agree.

The parameters do need to be present in any new SDP offer/answer, but if the values haven't changed it's just copies of the previous values.

>> 3. Offer answer is not the only available scenario. One thing that was discussed that each game or conference participant generates an offer, keeps 
>> it active for the duration of the conference and sends it to the server, where it is cached. When new participant joins the conference, it gets cached offers for all the other participants from the server and generates answers to them. 
>> This removes an extra signaling hop but impossible with current API.
> I'm not sure it's impossible. I'd like to try to build it first and see where it fails.

With SIP on the wire this "caching of offers" is probably not going to work, but of course if you use offer/answer with some other protocol on the wire...

In any case, I don't really see any harm in the server returning an answer to the offer, and then sending a new offer to the participant when additional participant(s) join the conference.

Regards,

Christer

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 13 Dec 2012 06:52:09 -0800

> Doesn't OPUS do that in-band?

I think his point was that this is a parameter which does not need to be 
negotiated at all: one side can decide it unilaterally and it has little 
effect on the other. This is what the constraints/settings API in the 
W3C is designed to tackle (and the W3C is where it should be tackled). 
You can call those APIs a "side channel", but I do not think they should 
be part of the same control surface for the same reasons that people 
regret having codec negotiation and transport setup intermingled in SDP.

---

From: Roman Shpount <[email-masked]>
Date: Thu, 13 Dec 2012 15:49:51 -0500

On Thu, Dec 13, 2012 at 9:32 AM, Christer Holmberg <
[email-masked]> wrote:

> reason
> >> why we need to run ice state machine and re-negotiate encryption to add
> or remove a bundled video stream or change encoding. To do this efficiently
> we need to have a side channel and additional operations to RFC 3264
> >
> without changing the crypto and transport parameters from what's currently
> in force, I can't see why the ICE and crypto state machines should change.
>
> I agree.
>
> The parameters do need to be present in any new SDP offer/answer, but if
> the values haven't changed it's just copies of the previous values.

In case I need to add video, I would create a new offer, keep all the
transport parameters from the previous offer, change the codec and media
streams, set this as a local description. Now I send this to a remote
party. First of all, there is no guarantee that remote party will give me
the answer with the same transport parameters, so I should be prepared to
renegotiate the transport. Second, I am not 100% I can just keep the
transport parameters from the previous offer and set it to local
description and just expect this to work. From my point of view it would be
much cleaner if transport settings and codec settings were separate. It
might not seem like a big deal, but these extra transport re-negotiations
can lead to audible and visible glitches when new media streams are setup
or turn down.

> discussed that each game or conference participant generates an offer, keeps
> >> it active for the duration of the conference and sends it to the
> generates answers to them.
> With SIP on the wire this "caching of offers" is probably not going to
> work, but of course if you use offer/answer with some other protocol on the
> wire...
>
> In any case, I don't really see any harm in the server returning an answer
> to the offer, and then sending a new offer to the participant when
> additional participant(s) join the conference.
>

In this particular case SIP is not important. Not every application has
interoperability requirements. Online game can very well only need to
communicate with other instances of the same game and nothing else. In any
case, would not it be nice to have transport parameters cached (advertised
on some rendezvous server), and get each new party to open a data
connection and then negotiate the rest of the codec parameters over this
peer-to-peer connection. Yes, you can work around this by bouncing
everything over the central server, but this can reduce number of round
trips during the conference setup or make the round trips to be between the
conference participants and not between participant to server to
participant.
_____________
Roman Shpount

---

From: Roman Shpount <[email-masked]>
Date: Thu, 13 Dec 2012 15:53:49 -0500

On Thu, Dec 13, 2012 at 9:52 AM, Timothy B. Terriberry <
[email-masked]> wrote:

> effect on the other. This is what the constraints/settings API in the W3C
> is designed to tackle (and the W3C is where it should be tackled). You can
> call those APIs a "side channel", but I do not think they should be part of
> the same control surface for the same reasons that people regret having
> codec negotiation and transport setup intermingled in SDP.
>
>
Not that it is terribly important right now, but current Chromium and
Firefox code, which is based on VoiceEngine does not allow setting any of
the non-negotiated or negotiated codec specific parameters, such as encoder
complexity, at all.

In any case, application developer should be able to change these
non-negotiated codec parameters, change codecs used to transmit,
enable/disable comfort noise, etc at any time without going through
offer/answer or adding or removing media tracks.
_____________
Roman Shpount

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 14 Dec 2012 09:59:08 +0000

Hi,

> In case I need to add video, I would create a new offer, keep all the transport parameters from the previous offer, change the codec and media streams, set this as a local description. Now I send this 
> to a remote party. First of all, there is no guarantee that remote party will give me the answer with the same transport parameters, so I should be prepared to renegotiate the transport.

Sure. But, the remote party could as well send you a new offer, which would trigger renegotiation.

(I don't think the remote party will change the parameters simply because it can - there needs to be a reason for it)

> Second, I am not 100% I can just keep the transport parameters from the previous offer and set it to local description and just expect this to work.

IF the transport parameters change you will need to indicate that no matter what.

> From my point of view it would be much cleaner if transport settings and codec settings were separate. It might not seem like a big deal, but 
> these extra transport re-negotiations can lead to audible and visible glitches when new media streams are setup or turn down.
> 
> In this particular case SIP is not important. Not every application has interoperability requirements. Online game can very well only need to communicate with other 
> instances of the same game and nothing else. In any case, would not it be nice to have transport parameters cached (advertised on some rendezvous server), and 
> get each new party to open a data connection and then negotiate the rest of the codec parameters over this peer-to-peer connection. 

Based on the consensus, obviously not :)

Regards,

Christer