Subject: Re: [rtcweb] Review of draft-ietf-rtcweb-jsep-07

From: Kiran Kumar Guduru <[email-masked]>
Date: Mon, 21 Jul 2014 02:35:09 +0000 (GMT)

It seems there is a small mis-understanding here. I agree there is a need for parsing and editing SDP by JS application for now.

I am not saying that constraints will do everything.

My intention is, to find the possibilities to extend the API of existing RTCPeerConnection and proposed RTCRTPSender/Receiver to add the attributes required by JS application.

For now, I proposed solution for one such kind of problem, i.e., prioritizing codecs.

setSdPAttribute (key, value) is just an example on top of my head, which might solve this. But we can really figure out this after getting the proposed API (RTCRTPSender/Receiver ) in draft.

Regards,

Kiran

------- Original Message -------

Sender : Paul Kyzivat<[email-masked]>

Date : Jul 19, 2014 22:35 (GMT+09:00)

Title : Re: [rtcweb] Review of draft-ietf-rtcweb-jsep-07

Kiran,

My need won't be satisfied by plausible constraints.
For instance, to interwork with CLUE it will be necessary to add an
"a=group:clue" and include in it those m-lines that are to be "clue
controlled". And that is needed in both offer and answer processing.

Thanks,
Paul

On 7/18/14 10:41 PM, Kiran Kumar Guduru wrote:
> Dear Paul,
>
> My comment is only regarding the comments on section 6 of the JSEP draft.
>
> The sdp mangling described in section 6 may result in various error
> scenarios.
>
> In that section, it is described like most of the parameters can be
> modified thorugh constraints.
>
> The required modification, I found, which cannot not be done by the
> constraints is "Remove or reorder of codecs (m=)"
>
> In order to avoid the SDP mangling at Java Script application level, I
> submitted a draft for changing the codec preferences
> http://tools.ietf.org/html/draft-guduru-rtcweb-codec-preferences-01.
> (Please review it).
>
> If this proposal is acceptable to the group, I hope we can remove
> section 6 from JSEP draft.
>
> (Expecting that we can find alternatives for adding extra attributes
> like CLUE as identified by you, for example by extending the
> proposed RTCRTPSender/Receiver by adding a generic setSdPAttribute (key,
> value) method).
>
> Thanks,
>
> Kiran.
>
>
>
> -------Original Message--------
> Sent: Paul Kyzivat
> Date: Fri, 18 Jul 2014 14:32:24 -0400
> Subject: [rtcweb] Review of draft-ietf-rtcweb-jsep-07
>
> I just did my first thorough read-through of JSEP in quite some time. I
> came away with a number of concerns:
>
> Section 1.1:
>
> When describing offers, modification by the application is mentioned:
>
>      JSEP's handling of session descriptions is simple and
>      straightforward.  Whenever an offer/answer exchange is needed, the
>      initiating side creates an offer by calling a createOffer() API.  The
>      application *optionally modifies that offer*, and then uses it to set
>      up its local config via the setLocalDescription() API.
>
> but when describing answers it is not:
>
>      When the call is accepted, the callee uses the createAnswer() API to
>      generate an appropriate answer, applies it using
>      setLocalDescription(), and sends the answer back to the initiator
>      over the signaling channel.  When the offerer gets that answer, it
>      installs it using setRemoteDescription(), and initial setup is
>      complete.
>
> And Section 6 only talks about changing offers, not answers. It is
> equally important to be able to modify answers. (More on this in later
> comment on section 6.)
>
> Section 4.1.4:
>
>      The only difference between a provisional and final answer is that
>      the final answer results in the freeing of any unused resources that
>      were allocated as a result of the offer.  As such, the application
>      can use some discretion on whether an answer should be applied as
>      provisional or final, and can change the type of the session
>      description as needed.  For example, in a serial forking scenario, an
>      application may receive multiple "final" answers, one from each
>      remote endpoint.  The application could choose to accept the initial
>      answers as provisional answers, and only apply an answer as final
>      when it receives one that meets its criteria (e.g. a live user
>      instead of voicemail).
>
> Issue: in this forking case, until the final selection is made it is
> unclear which one will need ICE completed. Only when a setlocal is done
> on one of the answers will ICE begin to be performed. Then, if another
> answer is provisionally set, won't that stop ICE for the prior one? And
> won't it require new candidates? What if one of the earlier ones is
> eventually chosen as final?
>
> And what if ICE completes for one of the candidates? Can't media start
> to flow? Then what if a different candidate is set as the final answer?
>
> Section 4.1.4.1:
>
> I question the following:
>
>      ...  While it is good practice to send an immediate
>      response to an "offer", in order to warm up the session transport and
>      prevent media clipping, the preferred handling for a web application
>      would be to create and send an "inactive" final answer immediately
>      after receiving the offer.  Later, when the called user actually
>      accepts the call, the application can create a new "sendrecv" offer
>      to update the previous offer/answer pair and start the media flow.
>
> This is very unfriendly when receiving calls that might be forked. By
> immediately "answering" a call before knowing if the user will accept
> it, you preempt the possibility that it will be answered on some other fork.
>
> For instance, if a call could come to your browser, or be sent to an
> answering service, and your broswer is on but you aren't present to
> accept the call, then the call will be accepted and then dropped rather
> than sent to the answering machine.
>
> So this technique should not be used if there is any possibility that
> the request could be coming from a source that might try other
> possibilities.
>
> Section 5.2.1:
>
>      When createOffer is called for the first time, the result is known as
>      the initial offer.
>
> By this definition, if a peer connection initially *received* an offer
> and sent an answer, and then it later sends an offer then that offer
> would be considered an initial offer.
>
> I think perhaps what is intended is:
>
>      When createOffer is called before setLocal has been called with
>      an offer,  the result is known as the initial offer.
>
> The following doesn't support the "balanced" BUNDLE policy:
>
>      Once all m= sections have been generated, a session-level "a=group"
>      attribute MUST be added as specified in [RFC5888].  This attribute
>      MUST have semantics "BUNDLE", and MUST include the mid identifiers of
>      each m= section.  The effect of this is that the browser offers all
>      m= sections as one BUNDLE group.  However, whether the m= sections
>      are bundle-only or not depends on the BUNDLE policy.
>
> Section 5.2.2 says:
>
>      o  If any MediaStreamTracks have been added, and there exist recvonly
>         m= sections of the appropriate media type with no associated
>         MediaStreamTracks, or rejected m= sections of any media type,
>         those m= sections MUST be recycled, and a local MediaStreamTrack
>         associated with these recycled m= sections until all such existing
>         m= sections have been used.  This includes any recvonly or
>         rejected m= sections created by the preceding paragraph.
>
> This fails to say anything about codec compatibility. SDP O/A requires
> the answer to be a subset of the offer in terms of PT/codec
> configuration. You should not use the same m-line unless there is a
> desire to use the same settings for the new track as are currently in
> use in the other direction.
>
> Section 5.3.1:
>
>      When createAnswer is called for the first time after a remote
>      description has been provided, the result is known as the initial
>      answer.
>
> By this definition, if a peer connection initially sent an offer and
> received an answer, and then it later receives an offer then the answer
> to that first *received* offer would be considered an Initial Answer.
>
> I think perhaps what is intended is:
>
>      When createAnswer is called before setLocal has been called with
>      an offer,  the result is known as the initial answer.
>
> When specifying the mapping of local tracks to m-lines in a received
> offer, there is again no discussion of codec compatibility. It is quite
> possible that the m-line chosen by the algorithm in this section won't
> have compatible codec attributes, but some other m-line will.
>
> Sections 5.3.2, 5.3.3, and 5.4-5.7:
>
> Are these empty because the content is yet to be written?
>
> Section 6:
>
> Other likely changes are addition of extra attributes and maybe other
> parameters. For instance, CLUE will want to add another grouping construct.
>
> Thanks,
> Paul
>
>

---

From: Kevin Dempsey <[email-masked]>
Date: Mon, 21 Jul 2014 09:34:21 +0100

Would the browser be expected to act differently with "a=group:clue"? If
not, then that is really part of the signalling and not part of the
createXxx/setXxxDescription interface.

The SDP passed to setXxxDescription and the SDP sent as part of the
signalling (if SDP is used) do not need to be identical.

On 21 July 2014 03:35, Kiran Kumar Guduru <[email-masked]> wrote:

>  It seems there is a small mis-understanding here. I agree there is a
> need for parsing and editing SDP by JS application for now.
>
> My intention is, to find the possibilities to extend the API of existing
> RTCPeerConnection and proposed RTCRTPSender/Receiver to add the attributes
> required by JS application.
>
> setSdPAttribute (key, value) is just an example on top of my head, which
> might solve this. But we can really figure out this after getting the
> proposed API (RTCRTPSender/Receiver ) in draft.
>
> Regards,
>
> Kiran
>
>
>
> ------- *Original Message* -------
>
> *Sender* : Paul Kyzivat<[email-masked]>
>
> *Date* : Jul 19, 2014 22:35 (GMT+09:00)
>
> *Title* : Re: [rtcweb] Review of draft-ietf-rtcweb-jsep-07
>
>
> Kiran,
>
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>

---

From: Christian Groves <[email-masked]>
Date: Wed, 23 Jul 2014 09:56:28 +1000

The browser would delay the establishment of media that are part of the 
"a=group:clue" group until a CLUE exchange has taken place and those 
media have been selected via a CLUE configure message. I'm not sure if 
that fits your criteria for acting differently?

Christian

On 21/07/2014 6:34 PM, Kevin Dempsey wrote:
> Would the browser be expected to act differently with "a=group:clue"? 
> If not, then that is really part of the signalling and not part of the 
> On 21 July 2014 03:35, Kiran Kumar Guduru <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     It seems there is a small mis-understanding here. I agree there is
>     a need for parsing and editing SDP by JS application for now.
>
>     My intention is, to find the possibilities to extend the API of
>     existing RTCPeerConnection and proposed RTCRTPSender/Receiver to
>     add the attributes required by JS application.
>
>     setSdPAttribute (key, value) is just an example on top of my head,
>     which might solve this. But we can really figure out this after
>     getting the proposed API (RTCRTPSender/Receiver ) in draft.
>
>     Regards,
>
>     Kiran
>
>     *Sender* : Paul Kyzivat<[email-masked]
>     <mailto:[email-masked]>>
>
>     > Section 4.1.4.1 <http://4.1.4.1>:
>     >
>     [email-masked] <mailto:[email-masked]>

---

From: Justin Uberti <[email-masked]>
Date: Tue, 22 Jul 2014 23:18:15 -0400

On Tue, Jul 22, 2014 at 7:56 PM, Christian Groves <
[email-masked]> wrote:

> "a=group:clue" group until a CLUE exchange has taken place and those media
> have been selected via a CLUE configure message. I'm not sure if that fits
> your criteria for acting differently?
>
> The browser doesn't understand a=group:clue. How could it know that it
needs to wait for CLUE messaging to complete?

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 23 Jul 2014 08:17:32 -0400

> signalling (if SDP is used) do not need to be identical.

No, there is no need for the *browser* to act differently.

So yes, I suppose the app can manage the pair of SDPs used for SIP O/A 
separately from the SDPs that are passed through JSEP to control the 
browser. That will be annoying, and complicated, not so much for the 
first O/A but for subsequent ones.

	Thanks,
	Paul

>     existing RTCPeerConnection and proposed RTCRTPSender/Receiver to add
>     the attributes required by JS application.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 23 Jul 2014 08:23:11 -0400

> that fits your criteria for acting differently?

I haven't thought about it much, but I think there are a couple of ways 
to deal with that. One is to simply hide those from the browser until 
the desired time. Another is to mark them as inactive for the browser 
until configured.

A more difficult problem is that CLUE uses one-way media streams, while 
jsep forces opposite direction ones to be merged into a single sendrecv 
m-line. I don't know how to fix that.

	Thanks,
	Paul

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 23 Jul 2014 13:18:47 +0000

> m-line. I don't know how to fix that.

Some time back I argued a lot that rtcweb should also use also use 
one-way m-lines. Perhaps that ship has sailed, but I think it is a 
cleaner solution (especially as the app at one end can at any time 
change a lot of settings related to a MediaStreamTrack being sent - and 
then perhaps it is unsuitable to use the same codec settings etc. as the 
reverse stream any more). Naturally there would need to be a way to 
support senrecv m-lines when interoperating with legacy equipment.

---

From: Justin Uberti <[email-masked]>
Date: Wed, 23 Jul 2014 09:19:36 -0400

On Wed, Jul 23, 2014 at 8:23 AM, Paul Kyzivat <[email-masked]> wrote:

> I haven't thought about it much, but I think there are a couple of ways to
> deal with that. One is to simply hide those from the browser until the
> desired time. Another is to mark them as inactive for the browser until
> configured.
>

If you can demonstrate a specific problem that this causes, we can
re-evaluate this. But we need to ensure that a simple phone call ends up
with a single m=audio line.

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 24 Jul 2014 00:14:03 -0400

On 7/23/14 9:19 AM, Justin Uberti wrote:
>
>
>
> On Wed, Jul 23, 2014 at 8:23 AM, Paul Kyzivat <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     I haven't thought about it much, but I think there are a couple of
>     ways to deal with that. One is to simply hide those from the browser
>     until the desired time. Another is to mark them as inactive for the
>     browser until configured.
>
>     A more difficult problem is that CLUE uses one-way media streams,
>     while jsep forces opposite direction ones to be merged into a single
>     sendrecv m-line. I don't know how to fix that.
>
>

The fundamental problem with this is that AFAICT it makes it impossible 
to build a WebRTC client that can interop with a CLUE server. I guess it 
would be possible with a special purpose media gateway that demuxed the 
clue media from sendrecv m-lines to sendonly & recvonly m-lines.

As another clue person said today, CLUE chose to use sendonly and 
recvonly m-lines so we could use the sendonly ones to provide SDP 
descriptions of what is available to be *sent*. (With sendrecv m-lines 
this describes only what is desired to be received.) Clue use cases are 
likely to be highly asymmetric, so it is essential to be able to do this.

Clue does have a solution for the simple phone, but it probably isn't a 
solution that would be satisfactory to rtcweb. The first O/A on a CLUE 
session assumes it doesn't know what it might be connecting to, and so 
makes an offer that is likely to work with "legacy" devices. Generally 
this would be one sendrecv audio and one sendrecv video, though YMMV. 
There are a few other things in there that allow the two ends to 
discover if they both can do clue after the first O/A.

If both do clue, then there is another O/A that provides sendonly 
m-lines for the offerer. And there will be a 3rd that provides sendonly 
m-lines by the other end. We consider this tolerable for telepresence.

The first O/A will also include DTLS/SCTP m-line that is intended to 
carry the CLUE data channel. (Assumed to be rejected by a "legacy" 
device.) While we don't require it, bundling is very attractive for 
clue. So if ICE has succeeded for that m-line, the subsequent O/As can 
add all those sendonly m-lines in the same bundle, so no new ice will be 
required.

	Thanks,
	Paul

---

From: Justin Uberti <[email-masked]>
Date: Thu, 24 Jul 2014 13:35:40 -0400

On Thu, Jul 24, 2014 at 12:14 AM, Paul Kyzivat <[email-masked]>
wrote:

> The fundamental problem with this is that AFAICT it makes it impossible to
> build a WebRTC client that can interop with a CLUE server. I guess it would
> be possible with a special purpose media gateway that demuxed the clue
> media from sendrecv m-lines to sendonly & recvonly m-lines.
>
> As another clue person said today, CLUE chose to use sendonly and recvonly
> m-lines so we could use the sendonly ones to provide SDP descriptions of
> what is available to be *sent*. (With sendrecv m-lines this describes only
> what is desired to be received.) Clue use cases are likely to be highly
> asymmetric, so it is essential to be able to do this.
>
> makes an offer that is likely to work with "legacy" devices. Generally this
> would be one sendrecv audio and one sendrecv video, though YMMV. There are
> a few other things in there that allow the two ends to discover if they
> both can do clue after the first O/A.
>
> If both do clue, then there is another O/A that provides sendonly m-lines
> for the offerer. And there will be a 3rd that provides sendonly m-lines by
> the other end. We consider this tolerable for telepresence.
>
> The first O/A will also include DTLS/SCTP m-line that is intended to carry
> the CLUE data channel. (Assumed to be rejected by a "legacy" device.) While
> we don't require it, bundling is very attractive for clue. So if ICE has
> succeeded for that m-line, the subsequent O/As can add all those sendonly
> m-lines in the same bundle, so no new ice will be required.
>

I think this would work fine with the new APIs being provided.

In the initial offer, sendrecv audio, video, and data m= line will be
offered, with the option to bundle.
In the second offer, the OfferToReceiveVideo:X option can be used to create
sendonly m= lines from the offerer.
In the third offer, the OfferToReceiveVideo:Y option can be used to create
sendonly m= lines from the answerer.

Did I overlook something?

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 24 Jul 2014 21:14:25 -0400

On 7/24/14 1:35 PM, Justin Uberti wrote:

>     The fundamental problem with this is that AFAICT it makes it
>     impossible to build a WebRTC client that can interop with a CLUE
>     server. I guess it would be possible with a special purpose media
>     gateway that demuxed the clue media from sendrecv m-lines to
>     sendonly & recvonly m-lines.
>
>     descriptions of what is available to be *sent*. (With sendrecv
>     m-lines this describes only what is desired to be received.) Clue
>     use cases are likely to be highly asymmetric, so it is essential to
>     be able to do this.
>
>     Clue does have a solution for the simple phone, but it probably
>     isn't a solution that would be satisfactory to rtcweb. The first O/A
>     on a CLUE session assumes it doesn't know what it might be
>     connecting to, and so makes an offer that is likely to work with
>     "legacy" devices. Generally this would be one sendrecv audio and one
>     sendrecv video, though YMMV. There are a few other things in there
>     that allow the two ends to discover if they both can do clue after
>     the first O/A.
>
>     m-lines for the offerer. And there will be a 3rd that provides
>     sendonly m-lines by the other end. We consider this tolerable for
>     telepresence.
>
>     clue. So if ICE has succeeded for that m-line, the subsequent O/As
>     can add all those sendonly m-lines in the same bundle, so no new ice
>     will be required.
>

Which new APIs? Do they work with the m-line management rules in 
jsep-07? Do you mean just the ones that allow OfferToReceive with a number?

> In the second offer, the OfferToReceiveVideo:X option can be used to
> create sendonly m= lines from the offerer.

Doesn't OfferToReceive generate *recvonly* m-lines. (It hardly makes 
sense for it to create *sendonly* lines.)

> In the third offer, the OfferToReceiveVideo:Y option can be used to
> create sendonly m= lines from the answerer.
>

I think so, or else I'm missing something.

	Thanks,
	Paul

---

From: Justin Uberti <[email-masked]>
Date: Thu, 24 Jul 2014 22:35:00 -0400

On Thu, Jul 24, 2014 at 9:14 PM, Paul Kyzivat <[email-masked]> wrote:

> Which new APIs? Do they work with the m-line management rules in jsep-07?
> Do you mean just the ones that allow OfferToReceive with a number?

Yes, the OfferToReceiveX APIs.

> Doesn't OfferToReceive generate *recvonly* m-lines. (It hardly makes sense
> for it to create *sendonly* lines.)

It can do either, as Eric pointed out in the presentation on Wednesday. If
you specify a number to receive that is less than the expected number of
m-lines, then the remainder will be marked as sendonly.

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 25 Jul 2014 08:12:38 -0400

On 7/24/14 10:35 PM, Justin Uberti wrote:

> On Thu, Jul 24, 2014 at 9:14 PM, Paul Kyzivat <[email-masked]

> It can do either, as Eric pointed out in the presentation on Wednesday.
> If you specify a number to receive that is less than the expected number
> of m-lines, then the remainder will be marked as sendonly.

OK. I had seen that mentioned in JSEP, but it was new to me and I hadn't 
really grokked it. It does sound like it might work.

The whole point of using these is to describe the characteristics of 
media we are intending to send, which may not all be the same. So the 
client may need to do more editing of these m-lines to get them as 
desired. I think that is allowed.

It is also the case that there might be a lot of these, as 
*possibilities*, though in practice the other end will accept only a few 
of them. That of course assumes that the other end *can* choose which 
ones it wants to use. If the other end is an RTCWEB client I don't see 
how that will work. It can use OfferToReceiveVideo:X to cause recvonly 
lines, but not which of the sendonly lines are chosen.

This gets back to the other issue I raised about needing to consider 
codecs when matching MSTs to m-lines.

I guess at this point what we need are some carefully worked clue 
examples, tailored for what a WebRTC client might be expected to do.

	Thanks,
	Paul

---

From: Jonathan Lennox <[email-masked]>
Date: Fri, 25 Jul 2014 13:32:11 +0000

On Jul 25, 2014, at 8:12 AM, Paul Kyzivat <[email-masked]> wrote:

> OK. I had seen that mentioned in JSEP, but it was new to me and I hadn't really grokked it. It does sound like it might work.

If I understood the JSEP presentation correctly, in the current APIs you can generate *either* sendonly or recvonly m-lines for an offer, but you can’t generate *both*.  You either get sendrecv and recvonly, or sendrecv and sendonly.  Is that correct?

(Presumably you could fake it by putting the sendrecv lines on hold, though?)

Also, is there any way to generate inactive m-lines?