Subject: [rtcweb] summary of issues raised at W3C meeting

From: Matthew Kaufman <[email-masked]>
Date: Fri, 2 Nov 2012 18:21:13 +0000

During the one and a half days of WEBRTC meeting and half day of Media Capture Task Force meeting at this week's W3C TPAC, a large number of issues came up, some of which were resolved. These fall into several categories, such as issues for the W3C WG, for other W3C WGs, for the IETF RTCWEB WG, other IETF WGs, and in some cases span more than one. Rather than attempt to categorize each of these items, I will simply list them below for 1) others present to expand upon where I missed or mis-stated something and 2) others to categorize appropriate and/or turn into action items (drafts, etc.)

-- begin list --

Collections of streams within an RTCPeerConnection are currently an array. This leads to the risk that someone will remember the array index, which would prevent removing streams. Solution is to replace this with a different type of collection and accessor. (Likewise, outside of WEBRTC, there needs to be a corresponding change for arrays of tracks within streams)

 Both streams and tracks need both "id" and "label" fields. "id" is guaranteed to be unique and will be used as the "msid" in the SDP. "label" is human-readable.

 AGC and AEC settings will have local effect only, set using constraints.

 There is no API to reject an incoming track (and thus reject from the offer/answer exchange without modifying SDP directly)... even if there were at the track level, this wouldn't map directly to the m= sections of the SDP.

It was suggested that content labels to be added a parameter on a track. Default of none. If specific content type is set then label is set in a new m= section with a=content: set to match the label.

 SDES or not still needs to be decided at IETF. Also things like MTI cipher suites for both SDES and DTLS-SRTP.

 There is apparently a desire to get certain kinds of constraints to the far end of an RTCPeerConnection... treating it as a pipe. So that if a constraint of (say) resolution is placed on a video track that is playing out that constraint would be sent somehow (SDP? RTCP? Application?) to the other end and affect the sending track and ultimately the source device.

 There is a desire to do trickle ICE. This requires a way to signal the desire to do trickle ICE in the SDP in a way that is backwards-compatible with existing RFC3264 O/A and existing non-trickle ICE implementations, while still generating fully 3264-compatible SDP for the trickle case (or updating 3264 and pointing JSEP at the update).

 There is a need to write down what MUST NOT be changed in the SDP that comes out of createOffer (and createAnswer?) before it is put in to setLocalDescription, and what MUST BE ACCEPTED IF CHANGED. This includes complexities such as reassignment of ports in m= line. The rest is permissive to change (see below).

SDP changes that aren't compatible either because they are in the "what MUST NOT be changed" list or in the (default) permissive list but not supported by this browser must be reported as an error. That error must include an error object that has an sdpLineNumber field.

The ICE state machine in the draft is poorly labeled... need a better drawing.

There is a desire to merge the icegatheringchange event into the onicecandidate event.

Rename iceconnect state to iceconnected.

Still open the be decided in media capture task force is what device enumeration we are allowed to do before and after user consent is received.

Stats - Several alternatives were proposed, general consensus to use a javascript structure with some limited hierarchy vs. flat dictionary w/ string keys encoding structure into the strings. Document the ICE stats as an example to test this.

DTMF - Decided to wrap mediastream in a decorator using peerconnection to do the creation/wrapping (so it learns about the peerconnection), NOT add a dtmf api on the peerconnection.

Open issue - at receiver, do all audio streams that have DTMF in the SDP come out decorated as above or undecorated?

Identity proposal was discussed - General opinion was that a hybrid approach ("# 4 = (1+3)") with peeridentity attribute is appropriate. Open issue is how to distinguish between have and don't have identity cases.

There was a desire to propose new names for onaddstream events and maybe others.

Open issue: how does createOffer decide to offer a data channel - not resolved

 Still need to resolve bundle at the IETF.

Still need to resolve media stream ID at the IETF.

Need to specify what happens if ssrc is not provided or media stream ID is not provided in received SDP. Is this an error?

We need a way to set priority for each track/data channel (3 or 4 levels) *and* have that enforced in the network subsystem of UA *and* optionally diffserv mark *and* get the diffserv markings back via an API (and include in SDP?)

createOffer, createAnswer, setLocalDescription, setRemoteDescription all cause callbacks. There is ambiguity created in the state machine if any of these can be called while any other is in the process of acting. Queuing must be implemented to ensure that execution of any of these is prevented until the callback is delivered from the previous call to any of these.

Rollback APIs must be added for both setLocalDescription(offer) and setRemoteDescription(offer) in order to get back to the stable state from which the same or opposite might be called.

Need to strengthen language to show that createOffer() is mandatory to call because without createOffer() you cannot have some of the SDP that MUST be provided to setLocalDescription()

Desire to change the guarantee of how long createOffer() is guaranteed to have its output accepted by setLocalDescription() from "in callback" to "15 seconds" (or something similar to that)

Discussion of changing the defaults for starting ICE from "all" to "none", resolved to staying with "all" and adding contraints (see below).

Decision to add a new constraint on RTCPeerConnection() to ask it to "preheat" a specified number of ICE candidate... the default constraint is 0. Therefore, this also clarifies than candidate gathering will not start at "new RTCPeerConnection" and instead waits until the constraint is set OR some undefined point in the future which still needs clarification (at createOffer or setLocalDescription, but neither is currently a good answer).

Decision to merge the concept of constraints and settings to only have constraints. Constraints always replace previous constraints. (Some interpretations were to set them to the intersection of request and previous.)

 getUserMedia() currently returns nothing, and has async callbacks on success or failure. This is a problem because the callee side of an RTCPeerConnection must either 1) not connect the camera immediately when a call is signaled, and thus respond with recvonly (and send the information to the caller necessary to have it pre-attached to the correct audio and video tags (because the information required to populate the ssrc and msid and send a provision answer doesn't exist), thus preventing playout at the caller even if RTP packets were to be sent to it), or 2) pre-allocate the camera and/or microphone while waiting for a call and tie up the resource. Proposed solution is to change getUserMedia() to return a MediaStream immediately which is a source of no sound and no video... if/when user consent is received this is delivered as an event on that stream, likewise failure to gain consent is an event on that stream. There was some discussion against the idea of returning the MediaStream synchronously in favor of a callback delivering the "empty" MediaStream which was not resolved.

-- end list --

Matthew Kaufman (and compiled with the assistance of Dan Burnett and Martin Thomson)

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 02 Nov 2012 16:32:15 -0400

Matthew Kaufman <[email-masked]> wrote:
> Open issue - at receiver, do all audio streams that have DTMF in the  
>  SDP come out decorated as above or undecorated?

I don't remember all the details of this discussion, but I'm not sure  
I understand this one. There is currently no W3C API for _receiving_  
DTMF, and no use case driving one, so I don't know what you would  
decorate a remote stream you are receiving with.

> We need a way to set priority for each track/data channel (3 or 4   
> levels) *and* have that enforced in the network subsystem of UA   
> *and* optionally diffserv mark *and* get the diffserv markings back   
> via an API (and include in SDP?)

To be clear, the DSCP is probably an IETF matter (and certainly  
including it in the SDP is). Enforcement in the UA's network subsystem  
might be, depending on how much of that rmcat needs to standardize.

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 02 Nov 2012 16:37:25 -0400

> Rollback APIs must be added for both setLocalDescription(offer) and setRemoteDescription(offer) in order to get back to the stable state from which the same or opposite might be called.

Certainly there is a need to rollback.
Conversely there should also be a need to commit.

Assume a prior O/A was successful and corresponding media streams are in 
operation. Then an offer is generated with the intent to change this 
state. For example, assume the new offer adds one new media stream and 
drops one that is currently in use.

 From the time of the offer until it's known that the answer has been 
accepted, resources need to be assigned to both the old media stream and 
the new one. If the answer is successfully received, then the changes 
can be committed - which means that the resources associated with the 
dropped stream can be reclaimed. If there is an error, then there is 
need to do a rollback - which means that the resources associated with 
the stream that was being added can be reclaimed.

What doesn't work (at least not well) is to assume at the time of the 
new offer that it will succeed, reclaiming resources from the stream 
being dropped. If that is done and then a rollback is needed, the 
resources would need to be reclaimed and reinitialized. There is no 
guarantee that they will be available, and even if they were this would 
cause all sorts of glitches. It might then also require ICE again, 
making more of a mess.

	Thanks,
	Paul

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Fri, 2 Nov 2012 20:43:21 +0000

On Nov 2, 2012, at 2:32 PM, "Timothy B. Terriberry" <[email-masked]>
 wrote:

> I don't remember all the details of this discussion, but I'm not sure I understand this one. There is currently no W3C API for _receiving_ DTMF, and no use case driving one, so I don't know what you would decorate a remote stream you are receiving with.

Consider the case where a PSTN gateway calls a browser - or at least the browser gets an offer with the media stream that will later be used for DTMF. The browser still wants to be able to send DTMF to the PSTN on that audio track so seems you might need something to be able to do that.

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 2 Nov 2012 21:11:20 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Paul Kyzivat
> Sent: Friday, November 2, 2012 8:37 PM
> To: [email-masked]
> Subject: Re: [rtcweb] summary of issues raised at W3C meeting - Rollback
> 
> > Rollback APIs must be added for both setLocalDescription(offer) and
> setRemoteDescription(offer) in order to get back to the stable state from
> which the same or opposite might be called.
> 

Yes.

> operation. Then an offer is generated with the intent to change this state.
> For example, assume the new offer adds one new media stream and drops
> one that is currently in use.
> 
> the new one. If the answer is successfully received, then the changes can be
> committed - which means that the resources associated with the dropped
> stream can be reclaimed.

That's the difference between PRANSWER and ANSWER, I'm afraid.

Standard comment #22 applies here as well.

Matthew Kaufman

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 2 Nov 2012 21:13:15 +0000

> Behalf Of Timothy B. Terriberry
> Sent: Friday, November 2, 2012 8:32 PM
> To: [email-masked]; [email-masked]
> Subject: Re: [rtcweb] summary of issues raised at W3C meeting
> 
> I don't remember all the details of this discussion, but I'm not sure I
> understand this one. There is currently no W3C API for _receiving_ DTMF,
> and no use case driving one, so I don't know what you would decorate a
> remote stream you are receiving with.

I think I was confusing MediaStream (unidirectional) with the RTP media streams (bidirectional).

Of course in the future we might add receipt-of-DTMF too... I can think of JavaScript apps I might want to do that in, using RTCWEB, outside of browsers.

> To be clear, the DSCP is probably an IETF matter (and certainly including it in
> the SDP is). Enforcement in the UA's network subsystem might be,
> depending on how much of that rmcat needs to standardize.

Exactly.

Matthew Kaufman

---

From: Matthew Kaufman <[email-masked]>
Date: Fri, 2 Nov 2012 21:16:34 +0000

Cullen Jennings (fluffy):
> 
> Consider the case where a PSTN gateway calls a browser - or at least the
> browser gets an offer with the media stream that will later be used for
> DTMF. The browser still wants to be able to send DTMF to the PSTN on that
> audio track so seems you might need something to be able to do that.

No, I think Tim's right... there's two audio tracks, and only the "outgoing" one needs DTMF and that's not the one on the stream that "pops out" of the RTCPeerConnection.

Of course this is also the other driver for getUserMedia allowing you to get a null media stream... you might want to get a call from a device that expects you to enter some DTMF but you don't have -- or want to provide consent to -- a microphone.

Matthew Kaufman

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 Nov 2012 21:50:07 +0000

> Of course this is also the other driver for getUserMedia allowing you to get a null media stream... you might want to get a call from a device that expects you to enter some DTMF but you don't have…

We didn't work out how we would make a truly fake stream.  In the
synchronous proposal, any fakeness in the stream ends when the user
permits or denies access to the input device.  We should probably fix
that.

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 03 Nov 2012 23:09:27 +0100

> permits or denies access to the input device.  We should probably fix

There's another proposal (not on the list so far) that proposes adding a 
constraint "source-type" and a constraint "source-id" for doing things 
like selecting a screengrab, a browser tab or other synthetic objects.

Should we have a "source-type: synthetic" and a "source-id: silence"? 
The user shouldn't need to be prompted for that.

Or is that too much magic?

> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Adam Bergkvist <[email-masked]>
Date: Mon, 5 Nov 2012 13:29:52 +0100

On 2012-11-02 19:21, Matthew Kaufman wrote:
> During the one and a half days of WEBRTC meeting and half day of
> Media Capture Task Force meeting at this week's W3C TPAC, a large
> number of issues came up, some of which were resolved. These fall
> into several categories, such as issues for the W3C WG, for other W3C
> WGs, for the IETF RTCWEB WG, other IETF WGs, and in some cases span
> more than one. Rather than attempt to categorize each of these items,
> I will simply list them below for 1) others present to expand upon
> where I missed or mis-stated something and 2) others to categorize
> appropriate and/or turn into action items (drafts, etc.)
>
> -- begin list --
[...]
> Both streams and tracks need both "id" and "label" fields. "id" is
> guaranteed to be unique and will be used as the "msid" in the SDP.
> "label" is human-readable.

If no-one objects, I'll go ahead and edit this into the specs.

/Adam

---

From: Stefan Hakansson LK <[email-masked]>
Date: Mon, 5 Nov 2012 14:46:51 +0100

> If no-one objects, I'll go ahead and edit this into the specs.

(Personal view): I think you can go ahead.

Stefan