Subject: Re: [rtcweb] Changing video size (Re: use case:)

From: Cullen Jennings <[email-masked]>
Date: Tue, 5 Jul 2011 16:30:20 -0600

Agree with folks on this thread about this being a way for the receiver to tell the send it's preferences - Perhaps the receiver might indicate that it would like 1080P video but the best the sender can do is 720P because that is the size of camera they have.  In this case the sender does what the sender wants ignoring the receiver. There are other cases where say the receiver can not decode a bitstream above 768 kbps but the sender would typically send a stream at a much higher rate than this. In this case the sender needs to do what the receiver wants. You might have an old browser that support vp8 and newer version of the browser that does both vp8 and vp9 and you use vp8 when that is all you have but prefer vp9 if both sides support it. I'm not sure exactly how to phrase all this but roughly speaking we need a way for the sender and receiver to agree on a format that meets the constraints of both and somewhat optimizes the desiree format within these constraints. 

On Jun 22, 2011, at 12:06 AM, Stefan Håkansson LK wrote:

> All,
>  
> this RFC could perhaps be useful for this purpose of signaling display size: http://www.rfc-editor.org/rfc/rfc6236.txt
>  
> Stefan
> 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Bert Greevenbosch
> Sent: den 22 juni 2011 07:56
> To: Justin Uberti
> Cc: [email-masked]
> Subject: Re: [rtcweb] Changing video size (Re: use case:)
> 
> Hi Justin, all,
>  
> Yes, that is a second issue. Even if the receiving end can signal its current display size, the streaming source might not be able to provide it. Therefore flexibility would be needed for the source on how to handle the display size information.
>  
> Best regards,
> Bert
>  
>  
> From: Justin Uberti [mailto:[email-masked]] 
> Sent: 22 June 2011 11:59
> To: Bert Greevenbosch
> Cc: Cullen Jennings; Aron Rosenberg; [email-masked]
> Subject: Re: [rtcweb] Changing video size (Re: use case:)
>  
> Given the fact that this request for a video size change is an optimization (either of resources or user experience), I think that SHOULD is probably the strongest label we can put on it. Consider the case of a middlebox; if it has no capacity to scale the stream, due to compute or codec restrictions, there's not much it can do with this request.
>  
> 
> However, whether we can cater for this also depends on the actual technical solution for negotiation and which codec to use. If adding video size negotiation will not increase complexity too much, I see no reason not to do it. However, if it adds much complexity to the solution, we might need to consider whether it is worth it. Therefore, maybe it should not be a requirement, but something weaker; i.e. it could be phrased as a SHOULD rather than a MUST.
> 
> Best regards,
> Bert
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Cullen Jennings
> Sent: 22 June 2011 00:11
> To: Aron Rosenberg
> Cc: [email-masked]
> Subject: Re: [rtcweb] Changing video size (Re: use case:)
> 
> 
> On Jun 20, 2011, at 1:30 , Harald Alvestrand wrote:
> 
> > On 06/17/11 21:38, Aron Rosenberg wrote:
> >> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings <[email-masked]> wrote:
> >>
> >> Video Size Change.
> >>
> >> Alice and Bob are in a video call in their browsers and have negotiate a high resolution video. Bob decides to change the size of the windows his browser is displaying video to a small size.  Bob's browser should be able to regenerate the video codec parameters with Alice's browser to change the resolution of video Alice sends to match the smaller size.
> >>
> >>
> >> Changing compression due to a user change in window size is usually not done since it has a bad long term effect on video quality. In almost all the major video calling systems, video resolution is a function of current bandwidth (which changes as the rate control system detects differences) and/or constrained by the physical device, but not a function of a user dragging the window size. At an equivalent bitrate, it is better to compress a larger resolution and display it smaller (higher PSNR), then compress a smaller resolution if you are within the codec's effective bitrate for that larger resolution.
> 
> Consider two different video flows using the same codec ...
> 
> Stream A is 640 x 480 at 1mbps which is this scaled to 320x240 and displayed
> 
> Stream B is 320 x 240 at 1mbps which is displayed at 320x240.
> 
> My experience has been with modern video codecs that stream B will look better than stream A. As well as looking better, it will typically also have a better PSNR. There's a bunch of reasons why which are probably not worth going into here but give it a try and you will see what I mean. The key point is both streams where 1mbps. If stream B was sent at 256 kbps, then A would look better.
> 
> 
> 
> 
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>  
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Randell Jesup <[email-masked]>
Date: Tue, 05 Jul 2011 22:55:15 -0400

Resend from the right address...

On 7/5/2011 6:30 PM, Cullen Jennings wrote:
> Agree with folks on this thread about this being a way for the receiver to tell the send it's preferences - Perhaps the receiver might indicate that it would like 1080P video but the best the sender can do is 720P because that is the size of camera they have.  In this case the sender does what the sender wants ignoring the receiver. There are other cases where say the receiver can not decode a bitstream above 768 kbps but the sender would typically send a stream at a much higher rate than this. In this case the sender needs to do what the receiver wants.

These sorts of issues are at least partly addressed in the O/A section 
of the H.264 spec (RFC 6184, update to RFC 3984).  Obviously that only 
applies to H.264.  There are also issues about limitations due to the 
other direction; in particular look at level-asymmetry-allowed and what 
happens if it's not used.

As mentioned, there are issues both of initial negotiation (as implied 
here) and in-call changes.

I think a) the two sides should indicate what they support receiving in 
negotiation, and b) indicate their initial preferred resolution/etc in 
negotiation.  In-call, there should be a mechanism for changing 
preference without a full call renegotiation (I suggest an RTCP-FB 
message), though one could fall back on a full new SDP exchange.

> You might have an old browser that support vp8 and newer version of the browser that does both vp8 and vp9 and you use vp8 when that is all you have but prefer vp9 if both sides support it.

That's just classic SDP and Offer/Answer:

New:
       m=video 1234 RTP/AVP 98 99 100
       a=rtpmap 98 VP9/90000
       a=rtpmap 99 VP8/90000
       a=rtpmap 100 H264/90000

Old:
       m=video 1234 RTP/AVP 99 100
       a=rtpmap 99 VP8/90000
       a=rtpmap 100 H264/90000

> I'm not sure exactly how to phrase all this but roughly speaking we need a way for the sender and receiver to agree on a format that meets the constraints of both and somewhat optimizes the desiree format within these constraints.

RFC 6236 certainly looks like a reasonable option to consider; in fact 
I'd consider it to be the
defacto choice barring some problematic limitation.  It allows asymmetry 
within the limits of the codec
negotiation; if RFC 6184 is used for H.264 then level asymmetry is allowed.

There *might* be an argument for also describing the portions of the 
decoded window that aren't visible
(per the previous conversation).

My biggest issue is that they assume it can only be changed via UPDATE 
or re-INVITE.  Maybe that's
reasonable, but I have to say I'd lean towards solutions that are direct 
end-to-end in-call to cut overhead
and lower reaction time.

   Randell

>>>> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings<[email-masked]>  wrote:
>>>>
>>>> Video Size Change.
>>>>

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Tue, 05 Jul 2011 22:51:24 -0400

New:

Old:

   Randell

-- 
Randell Jesup
[email-masked]

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 6 Jul 2011 22:04:56 -0600

On Jul 5, 2011, at 8:55 PM, Randell Jesup wrote:

> RFC 6236 certainly looks like a reasonable option to consider; in fact I'd consider it to be the
> defacto choice barring some problematic limitation.  

yah, make sense to me

> There *might* be an argument for also describing the portions of the decoded window that aren't visible

I worry about too much "optimization" :-)

> 
> My biggest issue is that they assume it can only be changed via UPDATE or re-INVITE.  Maybe that's
> reasonable, but I have to say I'd lean towards solutions that are direct end-to-end in-call to cut overhead
> and lower reaction time.

It one of the arguments people make in favor of the "low path". Or perhaps H.245 :-)

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 11 Jul 2011 16:01:56 +0200

On 06/22/11 05:59, Justin Uberti wrote:
> Given the fact that this request for a video size change is an 
> optimization (either of resources or user experience), I think that 
> SHOULD is probably the strongest label we can put on it. Consider the 
> case of a middlebox; if it has no capacity to scale the stream, due to 
> compute or codec restrictions, there's not much it can do with this 
> request.
There might be some things it can do, even if it can neither recode nor 
do SVC-style tricks.
For instance, if it's a distribution point (one incoming video stream, 
many outgoing copies), it can keep track of the maximal video size 
requested by any of its downstreams, and signal the upstream to send a 
resolution that is no higher than the largest resolution requested.

I agree that we can't demand that the sender send exactly the resolution 
requested; in *many* cases, it's reasonable to send both higher and 
lower resolutions than the requested ones.

>
>
>     However, whether we can cater for this also depends on the actual
>     technical solution for negotiation and which codec to use. If
>     adding video size negotiation will not increase complexity too
>     much, I see no reason not to do it. However, if it adds much
>     complexity to the solution, we might need to consider whether it
>     is worth it. Therefore, maybe it should not be a requirement, but
>     something weaker; i.e. it could be phrased as a SHOULD rather than
>     a MUST.
>
>     Best regards,
>     Bert
>
>
>     From: [email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>]
>     On Behalf Of Cullen Jennings
>     Sent: 22 June 2011 00:11
>     To: Aron Rosenberg
>     Cc: [email-masked] <mailto:[email-masked]>
>     >> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     >>
>     >> Video Size Change.
>     >>
>     >> Alice and Bob are in a video call in their browsers and have
>     negotiate a high resolution video. Bob decides to change the size
>     of the windows his browser is displaying video to a small size.
>      Bob's browser should be able to regenerate the video codec
>     parameters with Alice's browser to change the resolution of video
>     Alice sends to match the smaller size.
>     >>
>     >>
>     >> Changing compression due to a user change in window size is
>     usually not done since it has a bad long term effect on video
>     quality. In almost all the major video calling systems, video
>     resolution is a function of current bandwidth (which changes as
>     the rate control system detects differences) and/or constrained by
>     the physical device, but not a function of a user dragging the
>     window size. At an equivalent bitrate, it is better to compress a
>     larger resolution and display it smaller (higher PSNR), then
>     compress a smaller resolution if you are within the codec's
>     effective bitrate for that larger resolution.
>
>     My experience has been with modern video codecs that stream B will
>     look better than stream A. As well as looking better, it will
>     typically also have a better PSNR. There's a bunch of reasons why
>     which are probably not worth going into here but give it a try and
>     you will see what I mean. The key point is both streams where
>     1mbps. If stream B was sent at 256 kbps, then A would look better.
>
>
>
>
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>