Subject: [rtcweb] Microsoft tells W3C and IETF what we are doing no signs of offering real world interoperability

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 6 Aug 2012 16:43:01 +0000

I see that Microsoft decided to wait until the W3C and IETF were close to done before putting together a proposal that, if accepted, would explode most the current works and create maximal delay on this work. 

http://blogs.skype.com/en/2012/08/customizable_ubiquitous_real_t.html

---

From: Luca De Cicco <[email-masked]>
Date: Mon, 6 Aug 2012 19:16:49 +0200

I think that the proposed "customizable response to changing network
quality" is a good idea.
Wiring the adaptation logic into webrtc would make it less flexible to
different needs in particular
applications. Of course a default adaptation logic should be implemented in
the browser.

Cheers,
--
Luca De Cicco, PhD
Politecnico di Bari
Dipartimento di Elettrotecnica ed Elettronica
Via Re David, 200 - Bari - ITALY
Office: +39 080 596 3851

On Mon, Aug 6, 2012 at 6:43 PM, Cullen Jennings (fluffy)
<[email-masked]>wrote:

>
> I see that Microsoft decided to wait until the W3C and IETF were close to
> done before putting together a proposal that, if accepted, would explode
> most the current works and create maximal delay on this work.
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Rohit Puri <[email-masked]>
Date: Mon, 6 Aug 2012 12:27:32 -0700

Based on our experiences at TenHands Inc. where we are trying to build a RT
video-centric collaboration service, the goal cited in MSFT proposal (
http://html5labs.com/cu-rtc-web/cu-rtc-web.htm), namely:

"*Flexibility in its support of popular media formats and codecs as well as
openness to future innovation*—A successful standard cannot be tied to
individual codecs, data formats or scenarios. They may soon be supplanted
by newer versions, which would make such a tightly coupled standard
obsolete just as quickly. The right approach is instead to to support
multiple media formats and to bring the bulk of the logic to the
application layer, enabling developers to innovate. "

sounds like a great idea.

I have not gone through the MSFT proposal yet and do not consider myself to
be a browser expert, but if it were possible to have flexibility in the
webRTC framework to the extent that a client developer could use their own
"algorithms" in the media pipeline, that would be awesome. Concretely, some
algorithms that come to mind include (a) pre-processing of captured media
(b) any choice of codecs (b) bandwidth adaptation (c) post-processing of
decoded media (d) CPU adaptation.  There is probably many more.

On Mon, Aug 6, 2012 at 10:16 AM, Luca De Cicco <[email-masked]> wrote:

> applications. Of course a default adaptation logic should be implemented
> in the browser.
>
> Cheers,
> --
> Luca De Cicco, PhD
> Politecnico di Bari
> On Mon, Aug 6, 2012 at 6:43 PM, Cullen Jennings (fluffy) <[email-masked]
> > wrote:
>
>>

-- 
Thanks and best regards,

Rohit Puri ([email-masked])
Software Development, TenHands Inc. (www.tenhands.net)
My TenHands URL: http://tenhands.net/[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 06 Aug 2012 21:43:52 +0200

On 08/06/2012 09:27 PM, Rohit Puri wrote:
> Based on our experiences at TenHands Inc. where we are trying to build 
> a RT video-centric collaboration service, the goal cited in MSFT 
> proposal (http://html5labs.com/cu-rtc-web/cu-rtc-web.htm), namely:
>
> "*Flexibility in its support of popular media formats and codecs as 
> well as openness to future innovation*---A successful standard cannot 
> be tied to individual codecs, data formats or scenarios. They may soon 
> be supplanted by newer versions, which would make such a tightly 
> coupled standard obsolete just as quickly. The right approach is 
> instead to to support multiple media formats and to bring the bulk of 
> the logic to the application layer, enabling developers to innovate. "
>
The WG agrees with you that multiple codecs should be supported, but the 
WG long ago came to consensus that there needs to be a 
mandatory-to-implement codec to prevent interoperability failure.
That consensus has been rough at times, but I believe it is still the 
consensus.

See section 2.3 of draft-ietf-rtcweb-overview.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 6 Aug 2012 20:00:37 +0000

On Aug 6, 2012, at 1:27 PM, Rohit Puri wrote:

> Based on our experiences at TenHands Inc. where we are trying to build a RT video-centric collaboration service, the goal cited in MSFT proposal (http://html5labs.com/cu-rtc-web/cu-rtc-web.htm), namely:
> 
> "Flexibility in its support of popular media formats and codecs as well as openness to future innovation—A successful standard cannot be tied to individual codecs, data formats or scenarios. They may soon be supplanted by newer versions, which would make such a tightly coupled standard obsolete just as quickly. The right approach is instead to to support multiple media formats and to bring the bulk of the logic to the application layer, enabling developers to innovate. "
> 

and every proposal from any company or individual that I can recall being sent to theses WGs has had exactly that property. So it is sort of disappointing to see Microsoft present it as if their proposal was somehow different in this regards.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 6 Aug 2012 22:19:43 +0200

Note also that pre- and post-processing of media are not covered by this
proposal, those concepts are in the domain of MediaStream (and
post-processing is possible today through various methods).

In fact, I think the primary novel features of this proposal are:
- SessionDescriptions are true objects, instead of wrappers around SDP
- Additional control over the ICE Agent

However, no use cases have yet been outlined that require this
functionality.

On Mon, Aug 6, 2012 at 10:00 PM, Cullen Jennings (fluffy)
<[email-masked]>wrote:

> > Based on our experiences at TenHands Inc. where we are trying to build a
> RT video-centric collaboration service, the goal cited in MSFT proposal (
> > "Flexibility in its support of popular media formats and codecs as well
> as openness to future innovation—A successful standard cannot be tied to
> and every proposal from any company or individual that I can recall being
> sent to theses WGs has had exactly that property. So it is sort of
> disappointing to see Microsoft present it as if their proposal was somehow
> different in this regards.

---

From: Rohit Puri <[email-masked]>
Date: Mon, 6 Aug 2012 13:49:34 -0700

Having not been actively involved in the effort and realizing that i might
be late to the party, i do not mean to cause distraction. Regardless, I
want to voice out the following high-level opinion based on what we have
been learning at TenHands Inc. (this is one of the reasons for us to
continue with a plugin implementation for our service).

If from the media pipeline perspective, it were possible to have an
architecture like for example, gstreamer, wherein there is a concept of a
processing chain for media and it is possible for the client developer to
provide proprietary processing blocks that suit the use case for their
application better than the default implementation, that would be great.

Perhaps, the MSFT proposal does not talk about this, but two of the tenets
they lay out upfront, namely (a) *Customizable response to changing network
quality and (b) **Flexibility in its support of popular media formats and
codecs as well as openness to future innovation* certainly sound appealing.

On Mon, Aug 6, 2012 at 1:19 PM, Justin Uberti <[email-masked]> wrote:

> On Mon, Aug 6, 2012 at 10:00 PM, Cullen Jennings (fluffy) <
> [email-masked]> wrote:
>
>>
>> a RT video-centric collaboration service, the goal cited in MSFT proposal (

-- 
Thanks and best regards,

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Mon, 06 Aug 2012 16:07:04 -0500

> However, no use cases have yet been outlined that require this

I doubt you'll ever see a use case that *requires* SessionDescription 
objects, since the identical behavior can be achieved (in most cases) by 
handling SDP directly. As has been previously discussed, though, using 
objects would make life easier for JS developers, and might lead to more 
natural solutions to problems that have been recently posted on this 
list (like inspection of offers to determine which parts should be 
accepted and which should not).

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: [email-masked] | SIP: [email-masked] | Skype: kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at www.digium.com & www.asterisk.org

---

From: "Jim Barnett" <[email-masked]>
Date: Mon, 6 Aug 2012 14:11:42 -0700

As I recall, we recently discussed the idea of an API abstraction on top
of the SDP and rejected it.  I think that the reasoning was that it was
an unnecessary complication.

- Jim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Kevin P. Fleming
Sent: Monday, August 06, 2012 5:07 PM
To: [email-masked]
Subject: Re: [rtcweb] Microsoft tells W3C and IETF what we are doing no
signs of offering real world interoperability

On 08/06/2012 03:19 PM, Justin Uberti wrote:
> Note also that pre- and post-processing of media are not covered by 
> this proposal, those concepts are in the domain of MediaStream (and 

--
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: [email-masked] | SIP: [email-masked] | Skype:
kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA Check us out at
www.digium.com & www.asterisk.org
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Mon, 06 Aug 2012 16:14:24 -0500

> an unnecessary complication.

Indeed... and yet if it turns out that the abstraction is useful 
somebody will produce one and publish it as a JS library :-)

---

From: Randell Jesup <[email-masked]>
Date: Mon, 06 Aug 2012 17:16:47 -0400

> applications. Of course a default adaptation logic should be implemented

Already proposed by me (and we've said here that we will have a way to 
do this from the application).  Here's my proposed API, which I think 
covers exactly this and what you want:

https://www.w3.org/Bugs/Public/show_bug.cgi?id=15861

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Mon, 06 Aug 2012 17:21:40 -0400

On 8/6/2012 4:49 PM, Rohit Puri wrote:
> architecture like for example, gstreamer, wherein there is a concept of
> a processing chain for media and it is possible for the client developer
> to provide proprietary processing blocks that suit the use case for
> their application better than the default implementation, that would be
> great.

See the MediaStream Processing API proposal:
https://dvcs.w3.org/hg/audio/raw-file/tip/streams/StreamProcessing.html

Note that this was partly an alternative to the Web Audio API, which has 
been adopted, and they plan to integrate some aspects of MediaStream 
Processing into Web Audio, but MediaStream Processing is still relevant, 
especially for video (and will be more so once JS workers can access WebGL).

-- 
Randell Jesup
[email-masked]

---

From: Matthew Kaufman <[email-masked]>
Date: Mon, 6 Aug 2012 21:51:55 +0000

> - SessionDescriptions are true objects, instead of wrappers around SDP

More important, there is no SDP Offer/Answer state machine within the browser. A developer who desires SDP Offer/Answer semantics may implement that in Javascript or server-side. This is as opposed to JSEP, which includes a particular SDP O/A state machine within the browser, thus requiring it to be interoperable with other SDP O/A state machines both within other vendor browsers as well as existing SIP+SDP equipment and software.

> However, no use cases have yet been outlined that require this functionality.

I believe that there are several use cases, including interoperability use cases, where our proposed solution is more likely to be able to meet all edge cases.

An example would be recovery from call setup in the face of a browser page reload... a case where the state of the browser must be reinitialized, leading to edge cases where it becomes impossible with JSEP for a developer to write Javascript that behaves properly in all cases (because without an offer one cannot generate an answer, and once an offer has been generated one must not generate another offer until the first offer has been answered, but in either case there is no longer sufficient information as to how to proceed). While SDP O/A is a useful abstraction for trunk interconnection of peer endpoints, we do not believe it is a good choice for the Javascript API surface.

Likewise, baking in one ICE implementation is likely to lead to interoperability failures when it is found that it has subtle incompatibilities with other deployed applications, whereas our proposal would allow the developer of the web site or supporting Javascript library to code around these issues once RTC-capable browsers are already fielded.

In addition to these differences that fall out of not baking in the full ICE and SDP O/A state machines, our proposal provides several other capabilities, including hooks that allow a developer to customize their application's response to changing network conditions... an area which is currently completely unaddressed in the current WebRTC draft.

I wouldn't be surprised if the existing use-case document(s) is (are) inadequate to describe situations where, for instance, a developer might wish to prioritize video quality over frame rate, or drop video in order to continue audio, but that just means that we all need to provide more input on those documents as well.

Matthew Kaufman

---

From: Justin Uberti <[email-masked]>
Date: Tue, 7 Aug 2012 02:11:27 +0200

On Mon, Aug 6, 2012 at 11:51 PM, Matthew Kaufman
<[email-masked]>wrote:

> More important, there is no SDP Offer/Answer state machine within the
> browser. A developer who desires SDP Offer/Answer semantics may implement
> that in Javascript or server-side. This is as opposed to JSEP, which
> includes a particular SDP O/A state machine within the browser, thus
> requiring it to be interoperable with other SDP O/A state machines both
> within other vendor browsers as well as existing SIP+SDP equipment and
> software.
>

While there is a simple state machine, with JSEP you can drive the state
machine to any state you want, easily. To say that it bakes in SDP
offer/answer is inaccurate; if you don't want to use offer/answer you can
just set the local or remote description at any time (followed by
reapplying the existing remote/local description to complete the sequence).

This could of course be streamlined, but this could be done as an evolution
of the existing API.

> I believe that there are several use cases, including interoperability use
> cases, where our proposed solution is more likely to be able to meet all
> edge cases.
>
> An example would be recovery from call setup in the face of a browser page
> reload... a case where the state of the browser must be reinitialized,
> leading to edge cases where it becomes impossible with JSEP for a developer
> to write Javascript that behaves properly in all cases (because without an
> offer one cannot generate an answer, and once an offer has been generated
> one must not generate another offer until the first offer has been
> answered, but in either case there is no longer sufficient information as
> to how to proceed). While SDP O/A is a useful abstraction for trunk
> interconnection of peer endpoints, we do not believe it is a good choice
> for the Javascript API surface.
>
> Likewise, baking in one ICE implementation is likely to lead to
> interoperability failures when it is found that it has subtle
> incompatibilities with other deployed applications, whereas our proposal
> would allow the developer of the web site or supporting Javascript library
> to code around these issues once RTC-capable browsers are already fielded.

> In addition to these differences that fall out of not baking in the full
> ICE and SDP O/A state machines, our proposal provides several other
> capabilities, including hooks that allow a developer to customize their
> application's response to changing network conditions... an area which is
> currently completely unaddressed in the current WebRTC draft.
>

As others have pointed out, these kinds of hooks have been discussed but
haven't yet made it into the current API draft.

>
> I wouldn't be surprised if the existing use-case document(s) is (are)
> inadequate to describe situations where, for instance, a developer might
> wish to prioritize video quality over frame rate, or drop video in order to
> continue audio, but that just means that we all need to provide more input
> on those documents as well.
>
>
> Matthew Kaufman
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 07 Aug 2012 10:27:29 +0900

> http://html5labs.com/cu-rtc-web/cu-rtc-web.htm), namely:

I'm surprised to see this called a W3C Submission when it is not (yet?) 
listed at http://www.w3.org/Submission/ (and it's not yet the 7th in the 
areas of the world where most of the involved people are :-).

Regards,   Martin.

---

From: Randell Jesup <[email-masked]>
Date: Mon, 06 Aug 2012 21:46:15 -0400

On 8/6/2012 5:51 PM, Matthew Kaufman wrote:
> In addition to these differences that fall out of not baking in the
> full ICE and SDP O/A state machines, our proposal provides several
> other capabilities, including hooks that allow a developer to
> customize their application's response to changing network
> conditions... an area which is currently completely unaddressed in the
> current WebRTC draft.

We've said repeatedly that we need such capability, and I have a 
proposed JS API for just that:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=15861

The other side of this is in the hands of the Media Capture Task Force, 
where we've been discussing the best way to modify a MediaStream/Track 
after they're created.  There are two rough proposals, one to re-use the 
constraints API for getUserMedia(), and the other I've proposed 
(verbally only so far, no draft API) uses events and lets them "bubble 
up" the chain of MediaStreams/Tracks from a consumer to a source (which 
could be a remote source, and the application could signal that request 
(such as a resolution or frame-rate change) across to the remote source 
of the stream.  This is relevant even without a remote connection.

> inadequate to describe situations where, for instance, a developer
> might wish to prioritize video quality over frame rate, or drop video
> in order to continue audio, but that just means that we all need to
> provide more input on those documents as well. Matthew Kaufman

I think most of those concerns are covered in what I describe above. 
(For example, the application could override the automatic allocation of 
bits and close a video stream, change the resolution, etc.)

I'd love to your input on these proposals, API suggestions, 
alternatives, etc!  I'm glad you're willing to help flesh out them out 
and bring them to completion, and get them into the spec drafts ASAP.

-- 
Randell Jesup
[email-masked]