Subject: [rtcweb] Resolution negotiation - a contribution

From: Harald Alvestrand <[email-masked]>
Date: Thu, 12 Apr 2012 10:46:00 +0200

Hello folks,

after the IETF, I've attempted to gather together information about 
resolution negotiation and what we might need to do there.

I pulled together the paragraphs below - this may want to be 
incorporated in an I-D on "SDP usage in RTCWEB", or it may want to go 
somewhere else. Chairs' guidance is appreciated.

At the moment, I have it in xml2rfc format, so it should be easy to do 
cut-and-paste on it, but don't want to push it as a separate I-D unless 
that's what the chairs desire.
Note - there are some QUERY sections in there - I'm unsure what those 
should be resolved to.

Comments?

                          Harald

2.  Requirements

    The relevant requirement for video resolution negotiation from the
    RTCWEB use cases document
    [I-D.ietf-rtcweb-use-cases-and-requirements] is:

    o  F25 The browser SHOULD use encoding of streams suitable for the
       current rendering (e.g. video display size) and SHOULD change
       parameters if the rendering changes during the session.

    The resulting need is:

    o  To negotiate a maximum spatial resolution for all videos at call
       setup time

    o  To negotiate a maximum temporal resolution ("frame rate") across
       all videos at call setup time

    o  To indicate the desire of the recipient for a particular spatial
       or temporal resolution on a particular video source

    o  To indicate the intent of the sender to send a video source in a
       particular spatial or temporal resolution

    This document does not mention other requirements.

3.  SDP negotiation of video resolution

    An RTCWEB client MUST support negotiation of resolution using the
    "imageattr" attribute, as documented in [RFC6236].

    An RTCWEB client MUST support a SAR value of 1.0 (square pixels), and
    MAY choose to support only the 1.0 value of the "sar" attribute.

    The interpretation of the negotiation is that any video stream in the
    m= line containing the a=imageattr attribute will have a resolution
    within the bounds established by the negotiation.

    An RTCWEB client MUST support negotiation of the "a=framerate"
    attribute, as specified in [RFC4566] section 6.  Note that this is an
    upper bound on framerate; there is no lower bound negotiated.

    These requirements are necessary and sufficient for establishing the
    maximum spatial and temporal resolution for all videos at call setup
    time.  These bounds MAY be renegotiated over the course of the call,
    but MUST NOT be renegotiated to render any currently transmitted
    video stream out of bounds; the sending video stream MUST first be
    changed to be of a resolution that fits within both the old and the
    new bounds, or halted. <<QUERY: Is this necessary or sufficient
    restriction???>>

    An RTCWEB client MUST support per-SSRC requests for video
    resolutions, as described in
    [I-D.lennox-mmusic-sdp-source-selection].  This satisfies the
    requirement to indicate the desire of the recipient for a particular
    spatial or temporal resolution.

    We assume that the media sent from a sender to a receiver contains
    enough information inside the media format to tell what the
    resolution and framerate is. <<QUERY: Do we need to extend RFC 5576
    with an "imageattr" or "framerate" attribute in the forward direction
    instead?>>

4.  Non-SDP negotiation of video resolution

    An RTCWEB client MAY support the COP mechanism
    [I-D.westerlund-avtext-codec-operation-point] to negotiate the
    resolution of video within the limits established by the SDP
    negotiation without the need for additional SDP exchanges.

5.  Relation to WebRTC API constraint

    It is intended that the resolution negotiation be influenced by the
    constraints set by the application of either mandatory or optional
    constraints at the WebRTC API, as registered in the registry
    established by [I-D.burnett-rtcweb-constraints-registry].

    The following relationships hold for all attributes that the
    implementation intends to satisfy (note that the constraints listed
    here have NOT been registered yet):

    video-min-height >= value of imageattr y= xyrange lower bound

    video-max-height <= value of imageattr y= xyrange upper bound

    video-min-framerate is not represented in SDP

    video-max-framerate <= value of a=framerate attribute

    video-min-aspect-ratio <= value of imageattr "par=" prange lower
    bound

    video-max-aspect-ratio >= value of imageattr "par=" prange upper
    bound

    The implementation is free to increase "min" values or decrease "max"
    values (make requirements more restrictive) and add "step" in order
    to fit with its implementation restrictions. <<NOTE - check the value
    of the direction of those assumptions>>.

    Constraints specified at PeerConnection creation time are reflected
    as SDP-wide values.  Constraints specified when creating a
    MediaStream or attaching a MediaStream to a PeerConnection are
    reflected as ssrc-specific values.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 12 Apr 2012 02:09:45 -0700

> new bounds, or halted. <<QUERY: Is this necessary or sufficient

Is this restriction intended to be imposed on the sender only? I.e., can 
the recipient still attempt renegotiation (using SDP source selection) 
of a smaller resolution than is currently allowed (e.g., because it's 
CPU-bound)?

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 12 Apr 2012 14:47:45 +0200

On 04/12/2012 11:09 AM, Timothy B. Terriberry wrote:
> Is this restriction intended to be imposed on the sender only? I.e., 
> can the recipient still attempt renegotiation (using SDP source 
> selection) of a smaller resolution than is currently allowed (e.g., 
> because it's CPU-bound)?

What do you think it should say?

What I was thinking about was the silly state you can get into if you 
have a running 1024x768 stream, based on negotiating "width between 1000 
and 1500", and want to negotiate max resolution down to 480x360 by 
saying "width between 400 and 500".

The possibilities that may make sense on the sender side:

- Negotiate width between 400 and 1500, then change to 480x360, then 
negotiate width 400-500
- Stop the stream, negotiate width between 400 and 500, then turn on 
stream at new size
- Keep sending 1024x768 until negotiation is complete, then change at 
your leisure
- Switch to 480x360 even though it's not allowed by current parameters, 
then renegotiate

The two first look ugly, but are (as I understand) compatible with 
current standards. The two last ones seem to break the "only send what 
you have negotiated" model.

If the negotiation is receiver-initiated, it makes more sense (just 
thinking on the fly here) to say that it can be started at any time, and 
that the sender is free to change resolution to the new params at any 
time after he's decided the offer is acceptable. Perhaps he MUST do so 
before sending the answer?

(This is one particular example of Cullen's "what are the steps to 
change a codec" scenario.....)

                  Harald

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 12 Apr 2012 12:19:14 -0400

On Thu, Apr 12, 2012 at 4:46 AM, Harald Alvestrand <[email-masked]> wrote:
> I pulled together the paragraphs below - this may want to be incorporated in
> an I-D on "SDP usage in RTCWEB", or it may want to go somewhere else.
> Chairs' guidance is appreciated.
>
> Note - there are some QUERY sections in there - I'm unsure what those should
> be resolved to.
>
> Comments?
>
>                         Harald
>
> 2.  Requirements
>

It is not clear if this is "per session" or "per browser" and that
should be made clear. Here is why it makes a difference:

Per session maximum video limits can give rise to a "race to the
bottom" in the case of a mixed multiparty conference.

Suppose, for example, you have 4 parties. One is on a low powered
phone with a low bandwidth connection, and
all they can handle is SIF at 384 kbps and 15 fps. The other three
have the power to handle 1048p at > 1 Mbps and 30 fps.

Restriction of all uses to the capability of the least-capable
participant is, for example, not
likely to go over well in mixed RTCWEB / CLUE Telepresence sessions.

This is a very common situation, and the solution is to do this negotiation
"per browser." Of course, in practice, this generally requires
middleware (4.3.3. in
draft-ietf-rtcweb-use-cases-and-requirements ).

Regards
Marshall

> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 12 Apr 2012 18:51:52 +0200

On 04/12/2012 06:19 PM, Marshall Eubanks wrote:
> On Thu, Apr 12, 2012 at 4:46 AM, Harald Alvestrand<[email-masked]>  wrote:
>> Hello folks,
>>
> should be made clear.
Since this is negotiation, it has to be observable across the wire, 
which means that it's negotiated per SDP session - that is, per 
PeerConnection pair, which has an SDP session between them. I don't see 
how it could be otherwise.
>   Here is why it makes a difference:
>
> have the power to handle 1048p at>  1 Mbps and 30 fps.
>
I break this up differently than  you do..... we might be talking about 
the same thing still.

for the mesh conferencing situation, each participant has a direct 
connection to the "weakling".
That session will need to negotiate down to what's available. It's up to 
the app to decide whether it produces multiple streams at different 
resolutions or not (and up to the browser to figure out if it can 
support that).

For the centralized scenario, either the central unit provides 
downsampling (or stripping, if the end nodes are doing some kind of AVC 
or simulcast), or the "race to the bottom" you worry about will happen.

The word "session" has become one of the words I try to avoid using 
without a qualifier....

>>    new bounds, or halted.<<QUERY: Is this necessary or sufficient
>>    restriction???>>
>>
>>    resolution and framerate is.<<QUERY: Do we need to extend RFC 5576
>>    video-min-height>= value of imageattr y= xyrange lower bound
>>
>>    video-max-height<= value of imageattr y= xyrange upper bound
>>
>>    video-max-framerate<= value of a=framerate attribute
>>
>>    video-min-aspect-ratio<= value of imageattr "par=" prange lower
>>    bound
>>
>>    video-max-aspect-ratio>= value of imageattr "par=" prange upper
>>    bound
>>
>>    to fit with its implementation restrictions.<<NOTE - check the value

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 12 Apr 2012 18:19:51 +0000

Hi Harald,
Thanks for this strawman.  I believe it should work, but I fail to see how
a two dimensional negotiation requirement (negotiating max values for
framerate and image size--which, in turn, also has two-dimensional
properties) leads to better interop than a one dimensional negotiation
(pixels per second processing requirement).  Let alone a one dimensional
negotiation with a fragmented value space and constraints imposed an
maxima and minima for both resolution and frame rates to values that
represent common operation points (such as MPEG-style levels).  Actually,
I fail to see ANY positive aspect, except perhaps that dimension/framerate
may be a bit more intuitive than processing power demands.  This goes for
both the API and the protocol.
One can argue a lot about the value of MPEG's profiles in today's software
implementation world (and people do, myself included).  But the level
concept appears to make sense to me.
As a side note, something along the lines of CLUE's simultaneous
transmission sets concept (or however that thing is currently called; I
lost track) would also be beneficial for a system that is inherently
capable of displaying multiple video signals, such as  a web browser.
Regards,
Stephan

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 12 Apr 2012 21:08:06 +0200

On 04/12/2012 08:19 PM, Stephan Wenger wrote:
> (pixels per second processing requirement).
Stephan,

I do not see this (or the requirement from the use-cases document) first 
and foremost a decoder complexity negotiation; it is a negotiation of 
how much data it is useful to send, given the recipient's intended use 
of that data.

For instance, in Google+ Hangouts, we use a similar function to make 
sure the participants who are not currently displayed in large format on 
anyone's screen only send the data needed to display the "thumbnail"; 
this represents a huge saving in bandwidth at our central servers.

It just happens that the decoding complexity of the VP8 codec is 
sufficiently linear with the number of pixels displayed that we don't 
see any advantage in encumbering the negotiation of that codec with 
extra features like "profiles" - but that's a discussion for PAYLOAD, 
not RTCWEB.

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 12 Apr 2012 21:13:38 +0000

>how much data it is useful to send, given the recipient's intended use

Then such a negotiation should be executed in addition.  Decoder cycle
requirement do matter in practical implementations.

>this represents a huge saving in bandwidth at our central servers.

Oh, absolutely.  I'm all in favor of resolution adaptation based on
receiver (population) requirements.  But only after an upper limit is
established (that could be based on many factors, including screen size,
connectivity, and computational resources (sic!).

>extra features like "profiles" - but that's a discussion for PAYLOAD,

True, agreed, and not what I was arguing about.

Stephan

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 13 Apr 2012 09:45:53 +0200

On 04/12/2012 11:13 PM, Stephan Wenger wrote:
>
> On 4.12.2012 12:08 , "Harald Alvestrand"<[email-masked]>  wrote:
>
Feel free to propose language that captures this requirement. As noted, 
my I-D fragment doesn't.

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 16 Apr 2012 21:40:09 +0000

Hi all,

For context: Harald and myself have been at odds for a while now about the
lack of support for a code point in the VP8 payload that can be used to
negotiate a maximum decoder/bitstream complexity.  Specifically, Harald
(and other VP8 payload folks) suggested that generic mechanisms, such as
the a=framerate attribute of RFC4566 in conjunction with the picture size
aspect of the imageattr of RFC 6236 can be used, at least in the rtcweb
context.  However, as far as I understood our argument, these two
mechanisms in combination are not meant as a limit for decoder complexity
(in terms of samples/sec processing rate), but rather as an indication,
from receiver to sender, of an upper bound of what is "useful to send".
See the email below.  To me, it's quite obvious that an indication of
"useful to send" includes "my decoder can handle this"; however, it could
be more restrictive in that factors other than decoder horsepower could
also be at play, such as screen size, user interface settings, and so on.

I believe that the combination of what can be signaled using the above
mechanisms should be sufficient for rtcweb.  However, I also believe that
it is insufficient for general purpose use, mostly because it requires the
support of RFC 6236, which is not exactly a widely deployed technology.
Further, the a=framerate attribute is not a particularly useful attribute
these days anymore, because variable frame rates, at least for software
encoding/decoding, are the norm.

In previous posts on the payload list (in response to the VP8 payload
WGLC), I have commented on the practical shortcomings of the (lack of)
complexity negotiation, and suggested that this needs to be fixed.

Two options:

1) codify Harald's mechanism (based on a=framerate and imageattr in the
VP8 payload draft, at MUST strength.  "In a declarative context, a
prospective media sender supporting this (VP8 payload) specification MUST
support RFC 4566 a=framerate and RFC6236 imageattr, and MUST include code
points according to both mechanisms to identify the properties of the
media stream.  In an offer-answer context, both offerer and answerer
receiver supporting this VP8 payload specification MUST support
a=framertate and imageattr, and MUST include both in their respective
offer/answer messages, so to identify an operation point that will not
overload the media decoder's capabilities.

The issue with this approach, IMO, is that we are dealing here with three
individual code points (framerate, horizontal and vertical picture size),
where a single code point ought to be sufficient for determining whether a
décor is capable of decoding a stream, at least from a complexity
viewpoint).

2) include, in the V8 payload, a negotiable SDP code point indicating the
complexity of a stream, in units of samples per second processing
requirements or a derivative thereof (such as: levels as used in the MPEG
world).  For example, the VP8 payload could include a single, optional,
negotiable parameter "SamplePerSecond".  If SamplePerSecond were absent in
the SDP, a value of xxxxx must be inferred.  (a sensible value for xxxxx
could be, for example 9216000, which is the number of samples per second
for VGA resolution at 30 Hz).  If SamplePerSecond is present in a
declarative context, it indicates the minimum processing requirements a
decoder must support in order to successfully decode the stream.  In a
symmetric offer-answer context, SamplePerSecond can be used to "dial down"
the complexity of the stream to a value that both encoder and decoder can
support.

My preference is obviously the second proposal, but I'm willing to help
fleshing out either or both of them, just not today :-)

Regards,
Stephan

---

From: "Yuepeiyu (Roy)" <[email-masked]>
Date: Tue, 17 Apr 2012 02:23:42 +0000

Hi Harald and Stephen,

First of all, the "SamplePerSecond" in the option 2 should be the maximal sample per second of the stream, because of the variable frame rates.

Second, if we are talking about signaling of decoder complexity point of view, 
	1) I don't see much difference between option 1 and 2, if option 2 is to signal the real number of "sample per second". 
	2) I believe a derivative of "SamplePerSecond" (still option 2) would make more sense, since to some extent, less values would be seen as a guideline for the encoder to produce the media stream. This is meaningful especially for the declarative context where the encoder has fewer information of the decoder's capability.

Just my 2 cents.
Roy
-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Harald Alvestrand
发送时间: 2012年4月17日 8:32
收件人: Stephan Wenger
抄送: [email-masked]
主题: Re: [payload] VP8 payload, decoder processing capabilities (was Re: [rtcweb] Resolution negotiation - a contribution)

Responding strictly on the payload list, and regarding the VP8 payload 
format only:

The a=framerate attribute is a maximum framerate only. Since the maximum 
framerate is the one that places the most stress on the decoder, the 
ability to use lower framerates is irrelevant to the question of 
restricting the decoder load.

Apart from that, I believe we agree on what we disagree about; the input 
of other participants would be valuable.

                 Harald

On 04/16/2012 11:40 PM, Stephan Wenger wrote:
> On 4.13.2012 00:45 , "Harald Alvestrand"<[email-masked]>  wrote:
>

_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 19 Apr 2012 09:46:40 +0200

When discussing the issue of resolution negotiation locally, we found a 
quandary:

We have two different negotiation needs:
- Maximum resolution that's possible to use in a given context
- The preferred resolution for a given context

The current imageattr= spec (RFC 6236) doesn't specify clearly whether 
the result of the negotiation is to be considered mandatory or advisory 
(it uses the term "wishes" a lot), but the description reads most 
consistently if one assumes that it is a "MUST" type request.

One interpretation is that the highest functionality available in the 
spec is the "limit", while the highest q value is the "preference", thus:

recv [x=320,y=200,q=1.0] [x=200:1024,y=160:768,q=0.1]

would indicate that the receiver is capable of 200x160 to 1024x768, but 
wants 320x200.

This maps reasonably to the capabilities spec's talk about "mandatory" 
vs "optional" capabilities.

Does this interpretation make sense?

                       Harald

On 04/12/2012 10:46 AM, Harald Alvestrand wrote:
> cut-and-paste on it, but don't want to push it as a separate I-D 
> unless that's what the chairs desire.

---

From: "Roni Even" <[email-masked]>
Date: Thu, 19 Apr 2012 11:18:33 +0300

Hi Harald,
The reason for the language was that this is really what the receiver wished
to receive. The issue is that at least for H.264 there is no mandatory
requirement for the encoder to support all resolutions within the supported
level. The receiver must be able to crop and scale.
 
Roni Even

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Harald Alvestrand
> Sent: Thursday, April 19, 2012 10:47 AM
> To: [email-masked]
> Subject: Re: [rtcweb] Resolution negotiation - mandatory / advisory
> resolutions
>

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 19 Apr 2012 10:47:40 +0200

On 04/19/2012 10:18 AM, Roni Even wrote:
> to receive.
The strength of his wish may be the issue here....
>   The issue is that at least for H.264 there is no mandatory
I couldn't parse that.

Does the sender have to (MUST) send a resolution within the boundaries 
of the "recv" spec, or can he send something outside it? (ie if the 
receiver sends recv [x=320,y=200], can the sender send something with 
x=320,y=240, or is he completely constrained?)

What does the receiver have to be able to do?

                    Harald

---

From: "Roni Even" <[email-masked]>
Date: Thu, 19 Apr 2012 12:56:24 +0300

Hi Harald,
Look at example 4.2.1. The sender of the media will reject the recv value in
the offer and may offer other send options. Typically they should be larger
than the rejected value in order to allow cropping and/or scaling down to
the right size. (Cropping may provide better quality than scaling up).

Roni

> From: Harald Alvestrand [mailto:[email-masked]]
> Sent: Thursday, April 19, 2012 11:48 AM
> To: Roni Even
> Cc: [email-masked]
> > The reason for the language was that this is really what the receiver
> > wished to receive.
> > requirement for the encoder to support all resolutions within the
> > supported level. The receiver must be able to crop and scale.
> I couldn't parse that.
> 
> >> When discussing the issue of resolution negotiation locally, we
> found
> >> a
> >> quandary:
> >>
> >> The current imageattr= spec (RFC 6236) doesn't specify clearly
> >> whether the result of the negotiation is to be considered mandatory
> >> or advisory (it uses the term "wishes" a lot), but the description
> >> reads most consistently if one assumes that it is a "MUST" type
> request.
> >>
> >> would indicate that the receiver is capable of 200x160 to 1024x768,
> >> but wants 320x200.
> >>
> >>> incorporated in an I-D on "SDP usage in RTCWEB", or it may want to
> >>> go somewhere else. Chairs' guidance is appreciated.
> >>>
> >>> Note - there are some QUERY sections in there - I'm unsure what
> >>> those should be resolved to.
> >>>
> >>> Comments?
> >>>
> >>>                           Harald
> >>>
> >>> 2.  Requirements
> >>>
> >>>     o  To indicate the intent of the sender to send a video source
> >>> in
> >> a

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 19 Apr 2012 13:17:48 +0200

> the right size. (Cropping may provide better quality than scaling up).

Thank you - that helps a bit.

I interpret the example in 4.2.1 (second alternative) as consisting of 4 
messages; in an offer/answer exchange, this would look like:

1) Alice -> Bob: OFFER
     a=imageattr:97 send [x=800,y=640,sar=1.1,q=0.6] [x=480,y=320] \
                    recv [x=330,y=250]
2) Bob -> Alice: ANSWER
     a=imageattr:97 recv [x=800,y=640,sar=1.1] \
                    send [x=[320:16:640],y=[240:16:480],par=[1.2-1.3]]
3) Alice -> Bob: OFFER (re-invite, probably)
     a=imageattr:97 send [x=800,y=640,sar=1.1] \
                    recv [x=336,y=256]
4) Bob -> Alice: ANSWER
     a=imageattr:97 recv [x=800,y=640,sar=1.1] \
                    send [x=336,y=256]

So after message 2, Bob cannot send anything, since he can't reproduce 
what Alice asked for, right?

And after message 2, Alice can only send 800x640 with a sar of 1.1, 
right? It's forbidden for Alice to send a noncompensated image, or a 
640x480 image?

Again, it's the use of the words "wishes" and "desires" that I want to 
be sure I understand correctly - as used in RFC 6236, they both mean 
"demand" / "require" / "MUST". Right?

---

From: "Roni Even" <[email-masked]>
Date: Thu, 19 Apr 2012 16:25:14 +0300

Hi Harald,
This is a good question, my view will be that since Alice gave a preference
like profile and level (and maybe bandwidth) in the original offer than Bob
can send any video that will be within this level. The image attribute says
that Alice prefers to get something specific to help with processing (note
that Alice can limit the received video by using max-mbps and max-fs for
H.264). It may be good to discuss it with others from MMUSIC.

The usage of the image attribute in my view is more relevant to allow the
receiver to ask for a specific image size since this is not defined in H.264
(in H.263 you can specify the common specific resolutions and frame rate
using the optional parameters in RFC 4629). This was a requirement to allow
a receiver to ask for a specific size.
The usage of "wish" was because H.264 encoder is not required to support
scale to a specific resolution and a compliant decoder must support
everything within the negotiated profile and level. So when a receiver asks
for a size it can happen only if the encoder can do it but if the encoder
cannot it is still compliant.

I think that one reason for no scaling requirements on encoder is to support
simple video cameras that support one resolution and send an H.264 stream
(no scaling/cropping capabilities) and the decoder must be able to display
it on any output device (monitor).

Roni

> Sent: Thursday, April 19, 2012 2:18 PM
> To: Roni Even
> Cc: [email-masked]
> > Look at example 4.2.1. The sender of the media will reject the recv
> > value in the offer and may offer other send options. Typically they
> > should be larger than the rejected value in order to allow cropping
> > and/or scaling down to the right size. (Cropping may provide better
> quality than scaling up).
> 
> I interpret the example in 4.2.1 (second alternative) as consisting of
> 4 messages; in an offer/answer exchange, this would look like:
> 
> 1) Alice -> Bob: OFFER
> >>> The reason for the language was that this is really what the
> >>> receiver wished to receive.
> >> Does the sender have to (MUST) send a resolution within the
> >> boundaries of the "recv" spec, or can he send something outside it?
> >> (ie if the receiver sends recv [x=320,y=200], can the sender send
> >> something with x=320,y=240, or is he completely constrained?)
> >>
> >>>> spec is the "limit", while the highest q value is the
> "preference",
> >>>> thus:
> >>>>
> >>>>> after the IETF, I've attempted to gather together information
> >>>>> about resolution negotiation and what we might need to do there.
> >>>>>
> >>>>> At the moment, I have it in xml2rfc format, so it should be easy
> >>>>> to
> >>>> do
> >>>>>      The relevant requirement for video resolution negotiation
> >>>>> from
> >> the
> >>>>>      o  F25 The browser SHOULD use encoding of streams suitable
> >>>>> for
> >> the
> >>>>>      o  To negotiate a maximum spatial resolution for all videos
> >>>>> at
> >>>> call
> >>>>>         setup time
> >>>>>
> >>>>>      o  To indicate the intent of the sender to send a video
> >>>>> source in
> >>>> a

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 24 Apr 2012 16:25:13 +0200

Harald, WG,

This is posted as individual. When it comes to this topic I will not
engage in any chair activities because I do have an alternative proposal
based on
https://datatracker.ietf.org/doc/draft-westerlund-avtext-codec-operation-point/.

My proposal has some significant differences in how it works compared to
Harald's. I will start with a discussion of requirements, in addition to
Harald's, then an overview of my proposal, and ending with a discussion
of the differences between the proposals. This is quite long but I do
hope you will read it to the end.

Requirements
------------

Let's start with the requirements that Harald have written. I think they
should be formulated differently. First of all I think the requirements
are negotiated, indicated etc in the context of a peer connection.

Secondly, when it comes to maximum resolution and maximum frame-rate,
there are actually three different limits that I think are important;
Highest spatial resolution, highest frame-rate, and maximum complexity
for a given video stream. The maximum complexity is often expressed as
the number of macroblocks a video codec can process per second. This is
a well-established complexity measure, used as part of standardized
video codec's "level" definitions since the introduction of H.263 Annex
X in 2004. As this is basically a joint maximum requirement on the
maximum amount of pixels per frame times the maximum frame-rate, there
exist cases where this complexity figure actually is the constraining
factor forcing a sender or receiver to either request higher frame rate
and lower resolution or higher resolution and a lower frame rate.

The requirements should also be clearer in that one needs to handle
multiple SSRCs per RTP session, including multiple cameras or audio
streams (microphones) from a single end-point, where each stream can be
encoded using different parameter values.

I also think it is important that we consider what additional encoding
property parameters that would make sense for WebRTC to have a
possibility to dynamically negotiate, request and indicate.

Some additional use cases that should be considered in the requirements:

1) We have use cases including multi-party communication. One way to
realize these is to use a central node and I believe everyone agrees
that we should have good support for getting this usage to work well.
Thus in a basic star topology of different participants there is going
to be different path characteristics between the central node and each
participant.

1A) This makes it necessary to consider how one can ensure that the
central node can deliver appropriate rates. One way is to de-couple the
links by having the central node perform individual transcodings to
every participant. A simpler non-transcoding central node, only
forwarding streams between end-points, would have to enforce the lowest
path characteristics to all. If one don't want to transcode at the
central node and not use the lowest path characteristics to all, one
need to consider either simulcast or scalable video coding. Both
simulcast and scalable video coding result in that at least in the
direction from a participant to the central node one needs to use
multiple codec operation points. Either one per peer connection, which
is how I see simulcast being realized with today's API and
functionality, or using an encoding format supporting scalable coding
within a single peer connection.

1B) In cases where the central node has minimal functionality and
basically is a relay and an ICE plus DTLS-SRTP termination point (I
assume EKT to avoid having to do re-encryption), there is a need to be
able to handle sources from different participants. This puts extra
requirements on how to successfully negotiate the parameters. For
example changing the values for one media source should not force one to
renegotiate with everyone.

2) The non-centralized multiparty use case appears to equally or more
stress the need for having timely dynamic control. If each sender has a
number of peer-connections to its peers, it may use local audio levels
to determine if its media stream is to be sent or not. Thus the amount
of bit-rate and screen estate needed to display will rapidly change as
different users speaks. Thus the need for minimal delay when changing
preferences are important.

3) We also have speech and audio including audio-only use cases. For
audio there could also exist desire to request or indicate changes in
the audio bandwidth required, or usage of multi-channel.

4) Adaptation to legacy node from a central node in a multi-party
conference. In some use cases legacy nodes might have special needs that
are within the profiles a WebRTC end-point is capable of producing. Thus
the central node might request the nodes to constrain themselves to
particular payload types, audio bandwidth etc to meet a joining session
participant.

5) There appear to exist a need for expressing dynamic requests for
target bit-rate as one parameter. This can be supported by TMMBR
(RFC5104) but there exist additional transport related parameters could
help with the adaptation. These include MTU, limits on packet rate, and
amount of aggregation of audio frames in the payload.

Overview
--------

The basic idea in this proposal is to use JSEP to establish the outer
limits for behavior and then use Codec Operation Point (COP) proposal as
detailed in draft-westerlund-avtext-codec-operation-point to handle
dynamic changes during the session.

So highest resolution, frame-rate and maximum complexity are expressed
in JSEP SDP. Complexity is in several video codecs expressed by profile
and level. I know that VP8 currently doesn't have this but it is under
discussion when it comes to these parameters.

During the session the browser implementation detects when there is need
to use COP to do any of the following things.

A) Request new target values for codec operation, for example due to
that the GUI element displaying a video has changed due to window resize.

B) Indicate when the end-point in its role as sender change parameters.

In addition to just spatial resolution and video frame rate, I propose
that the following parameters are considered as parameters that could be
dynamically possible to indicate and request.

Spatial resolution (as x and y resolution), Frame-rate, Picture Aspect
ratio, Sample Aspect Ratio, Payload Type, Bit-rate, Token Bucket Size
(To control burstiness of sender), Channels, Sampling Rate, Maximum RTP
Packet Size, Maximum RTP Packet Rate, and Application Data Unit
Aggregation (to control amount of audio frames in the same RTP packet).

Differences
-----------

A) Using COP and using SDP based signaling for the dynamic changes are
two quite different models in relation to how the interaction happens.

For COP this all happens in the browser, normally initiated by the
browser's own determination that a COP request or notification is
needed. Harald's proposal appears to require that the JS initiate a
renegotiation. This puts a requirement on the implementation to listen
to the correct callbacks to know when changes happens, such as window
resize. To my knowledge there are not yet any proposals for how the
browser can initiate a JSEP renegotiation.

Thus COP has the advantage that there is no API changes to get browser
triggered parameter changes. W3C can select too but are not required to
add API methods to allow JS to make codec parameter requests.

The next thing is that COP does not require the JS application to have
code to detect and handle re-negotiation. This makes it simpler for the
basic application to get good behavior and they are not interrupted nor
do they need to handle JSEP & Offer/Answer state machine lock-out
effects due to dynamic changes.

How big impact these API issues have are unclear as W3C currently appear
not to have included any discussion of how the browser can initiate a
offer/answer exchange towards the JS when it determines a need to change
parameters.

But I am worried that using SDP and with an API that requires the
application to listen for triggers that could benefit from a codec
parameter renegotiation. This will likely only result in good behavior
for the JS application implementors that are really good and find out
what listeners and what signaling tricks are needed with JSEP to get
good performance. I would much rather prefer good behavior by default in
simple applications, i.e. using the default behavior that the browser
implementor have put in.

B) Using the media plane, i.e. RTCP for this signaling lets it in most
case go directly between the encoding and the decoding entity in the
code. There is no need to involve the JS nor the signaling server. One
issue of using JSEP and SDP is that the state machine lock-out effects
that can occur if one has sent an Offer. Then that browser may not be
able to send a new updated Offer reflecting the latest change until the
answer has been properly processed. COP doesn't have these limitations.
It can send a new parameter request immediately, only limited by RTCP
bandwidth restrictions. Using the media plane in my view guarantees that
COP is never worse than what the signaling plane can perform at its best.

C) As the general restrictions are determined in the initial
negotiation, COP doesn't have the issue that in-flight media streams can
become out of bounds. Thus no need for a two phase change of signaling
parameters.

D) Relying on draft-lennox-mmusic-sdp-source-selection-04 has several
additional implications that should be discussed separately. The draft
currently includes the following functionalities.

  D1) It contains a media stream pause proposal. This makes it subject
to the architectural debate currently ongoing in dispatch around
draft-westerlund-avtext-rtp-stream-pause-00 which is a competing
proposal for the same functionality.

  D2) The inclusion of max desired frame rate in a SSRC specific way

  D3) Extending the image attribute to be per SSRC specific expression
of desired resolutions.

  D4) Expressing relative priority in receiving different SSRCs.

  D5) Providing an "information" on a sent SSRC

  D6) Indication if the media sender is actively sending media using the
given SSRC.

Is it a correct observation that only D2 and D3 are required for the
functionality of Resolution negotiation?

E) The standardization situation is similar for both proposals. They are
both relying on Internet drafts that are currently individual
submissions. Both are partially caught in the architectural discussion
which was held in Paris in the DISPATCH WG around Media pause/resume
(draft-westerlund-avtext-rtp-stream-pause-00) and Media Stream Selection
(draft-westerlund-dispatch-stream-selection-00) on what the most
appropriate level of discussion are. This discussion will continue on
the RAI-Area mailing list.

F) As seen by the discussion on the mailing list the imageattr
definitions may not be a 100% match to what is desired with Harald's
proposal. I believe that COP's are more appropriate especially the
"target" values possibility. In addition these are still open for
adjustment and if they don't match WebRTC's requirements.

I would also like to point out that I believe this functionality is also
highly desirable for CLUE and that their requirements should be taken
into account. I do think that this is one of the aspects where having
matching functionality will make it much easier to have WebRTC to CLUE
interworking.

Thanks for reading all the way here!

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 24 Apr 2012 18:43:43 +0200

I'm happy to see this - there are a number of quibbles I want to make, 
but overall, I like it.

My big worry is the IPR issue - I think that negotiation of video size 
during a call is likely to be "MUST implement", and I don't want to 
require implementation of a protocol where people have filed non-RF IPR 
claims against it if there are viable alternatives.

               Harald

On 04/24/2012 04:25 PM, Magnus Westerlund wrote:
> do they need to handle JSEP&  Offer/Answer state machine lock-out

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 24 Apr 2012 09:47:30 -0700

> claims against it if there are viable alternatives.

Particularly when those claims come from the author's employer.

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 24 Apr 2012 11:13:46 -0700

Magnus' proposal for only using SDP to set the limits of behavior makes
sense to me.  Clearly SDP is not potentially viable for rapid adjustment.

The question is really about what information it is useful
for a receiver or sender to provide.  For example, I would question 
whether an SVC sender necessarily needs to send a message saying 
it is adjusting its sending rate, assuming that the new rate is within 
the range negotiated.  

In any case, this really is not an RTCWEB-specific problem, and comimg
up with different solutions in different IETF WGs seems highly undesirable.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 24 Apr 2012 22:05:16 +0200

> sense to me.  Clearly SDP is not potentially viable for rapid adjustment.
I would make that "potentially not viable" - if the RTT between clients 
and server is very low, and the server processing time is very short, 
there can be scenarios where sub-500-ms renegotiates are reasonable to 
expect even with server-mediated renegotiation, and there's always the 
potential of SDP exchange over a data channel, once that's commonly 
implemented. But it will be trivially easy for something to get 
implemented in a nonoptimal fashion, and for this to take quite a bit 
longer.
> up with different solutions in different IETF WGs seems highly 
I would love to not invent mechanisms. All the mechanisms so far 
suggested have been previously proposed elsewhere, but only the 
m-line-level a=imageattr spec (RFC 6236) has been published as a 
standars-track RFC.

---

From: Justin Uberti <[email-masked]>
Date: Tue, 24 Apr 2012 23:20:19 -0400

> hope you will read it to the end.

Maybe I misunderstand what you are saying, but the application will
definitely know when the browser size changes, and can then trigger a JSEP
renegotiation by calling createOffer and shipping it off.

I have the opposite concern - how can the browser know when the application
makes a change, for instance to display a participant in a large view
versus a small view? This may be possible if using a <video/> for display,
but I don't think it will be possible when using WebGL for rendering, where
the size of the view will be dictated only by the geometry of the scene.

> simple applications, i.e. using the default behavior that the browser

The issue here is that this behavior needs to be sufficiently specified, or
else we will have many different behaviors, none of which can be fully
overridden by the app. I worry this will make life hard for people trying
to develop sophisticated apps.

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 25 Apr 2012 16:35:18 +0200

On 2012-04-25 05:20, Justin Uberti wrote:
> On Tue, Apr 24, 2012 at 10:25 AM, Magnus Westerlund
> <[email-masked] <mailto:[email-masked]>>
> wrote:
> 
> definitely know when the browser size changes, and can then trigger a
> JSEP renegotiation by calling createOffer and shipping it off.

Yes, they can but that requires them to write code to do this. With COP
the browser can do it and it works well for all applications, not only
the ones that have implementors that are aware of the need to do this.

I am proposing a mechanism that makes the default behavior if you care
nothing about controlling or optimizing this behavior in your JS to be
good. In addition JS that wants to control it can affect the behavior
through generic mechanisms, such as controlling the video element. And
if clear need is shown and some functionality is missing, W3C can create
an API to provide explicit control.

> 
> I have the opposite concern - how can the browser know when the
> application makes a change, for instance to display a participant in a
> large view versus a small view? This may be possible if using a <video/>
> for display, but I don't think it will be possible when using WebGL for
> rendering, where the size of the view will be dictated only by the
> geometry of the scene.

What I understand from my colleagues is that all rendering of video have
to go through a video element, even if not displayed directly. Thus the
application can control the size of the video element used and possibly
consumed by WebGL to be included in rendering. Thus the application will
have control over this even when COP is being used.

Can you be more specific on any particular usage that you think would
have issues?

> The issue here is that this behavior needs to be sufficiently specified,
> or else we will have many different behaviors, none of which can be
> fully overridden by the app. I worry this will make life hard for people
> trying to develop sophisticated apps.

I agree that it needs to be clearly specified. But COP appears to be
less prone to complete failures by using target values. Then at least it
will receive video that can be used, even if not optimal. imageattr can
result in failures in the negotiation requiring at minimal a second
round potentially no agreed on acceptable resolution then that is
clearly sub-optimal. That is even without considering involving JS
mainipulating this.

Cheers

Magnus Westerlund

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 25 Apr 2012 18:13:54 +0200

On 04/25/2012 04:35 PM, Magnus Westerlund wrote:
>> <[email-masked]<mailto:[email-masked]>>
>> wrote:
>>
It's not at all clear to me that the browser being in control is right 
for all cases.
Renegotiating has a cost, and some of the costs are known only to the 
application.

We may possibly handle this via constraints like 
"VideoEnumRenegotiationTendency" or something like that - when you get 
the proposal together more, I'd like to see the controls you're proposing.
>> large view versus a small view? This may be possible if using a<video/>
We have on the "list of things that need doing" the connection of a 
mediastream directly to a canvas, a WebGL element or being captured to a 
file, so don't gamble too much on the video element to stay in the path.

Example code for capturing video to a canvas (taken from a real app):

        video.src = webkitURL.createObjectURL(s);
        canvas.getContext("2d").drawImage(video, 0, 0, 300, 300, 0, 0, 
300, 300);

I am not sure where the size controls go. I'm not even sure the 
developer knows.

> clearly sub-optimal. That is even without considering involving JS
I certainly hope that we can avoid the need for SDP manipulation of JS 
in all of these proposals.

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 30 Apr 2012 11:13:35 +0200

>         canvas.getContext("2d").drawImage(video, 0, 0, 300, 300, 0, 0, 

According to my Colleague Per-Erik you can use video.width and
video.height to set the resolution the video element should have.

see
http://dev.w3.org/html5/spec/the-video-element.html#the-video-element

Thus the code you presented should be something like.

video.src = webkitURL.createObjectURL(s);
video.width = 300;
video.height = 300;
canvas.getContext("2d").drawImage(video, 0, 0, 300, 300);

That would set the video objects size to a 300, 300 video resolution.
The drawImage call I have removed the part which selects a cropping of
the source video to automatically show the full video in the specified
area.

The adaptation of the video will of course not happen instantly so until
that happens the video will be scaled to the specified width and height.
And the actual delivered resolution will depend on the COP request and
what the sender support.

> in all of these proposals.

Looking forward to a bit more detailed explanation of how that is
accomplished in relation to the API.

Cheers

Magnus Westerlund

---

From: Justin Uberti <[email-masked]>
Date: Mon, 30 Apr 2012 20:02:59 -0700

On Mon, Apr 30, 2012 at 2:13 AM, Magnus Westerlund <
[email-masked]> wrote:

> >>> <[email-masked]<mailto:[email-masked]
> >>
> >>> wrote:
> >>>

I think this is a reasonable way to approach the problem as long as a
<video/> element stays in the path. But as Harald mentions we have already
considered several usages where a <video/> would not be in the path, i.e.
connecting a MediaStream to a file, or perhaps as input to another
PeerConnection.

Perhaps we need to provide hooks directly on the MediaStream to allow the
resolution to be specified, and these hooks are invoked automatically when
the MediaStream is connected to a <video/>.