Subject: [rtcweb] use case:

From: Cullen Jennings <[email-masked]>
Date: Fri, 17 Jun 2011 12:09:45 -0600

If people agree with the following use cases, I was hoping they might get added to draft-holmberg-rtcweb-ucreqs.

Video Size Change.

Alice and Bob are in a video call in their browsers and have negotiate a high resolution video. Bob decides to change the size of the windows his browser is displaying video to a small size.  Bob's browser should be able to regenerate the video codec parameters with Alice's browser to change the resolution of video Alice sends to match the smaller size. 

Call Fedex

Alice uses her web browser with a  service something like Skype to be able to phone PSTN numbers. Alice calls 1-800-gofedex. Alice should be able to hear the initial prompts from the fedex IVR and when the IVR says press 1, there should be a way for Alice to navigate the IVR. 

NIC Change

Alice is on her notebook computer that is plugged in to 1G ethernet and has 802.11 wireless interface. Alice is in a call talking with Bob and decides to unplug here notebook computer and walk down to a different room. The call should continue as Alice unplugs from the fat network interface and moves to a slower one. The media codecs may end up renegotiating to select a different bandwidth consumption on the wireless network. 

QoS Marking 

Alice's browser is on a computer behind a common residential router that supports prioritization of traffic . The browser should be able to take advantage of the capabilities of the  router to prioritize voice and video appropriately. 

Cullen <in my individual contributor role>

---

From: Aron Rosenberg <[email-masked]>
Date: Fri, 17 Jun 2011 12:38:12 -0700

On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings <[email-masked]> wrote:

>
> Video Size Change.
>
> Alice and Bob are in a video call in their browsers and have negotiate a
> high resolution video. Bob decides to change the size of the windows his
> browser is displaying video to a small size.  Bob's browser should be able
> to regenerate the video codec parameters with Alice's browser to change the
> resolution of video Alice sends to match the smaller size.
>
>
> Changing compression due to a user change in window size is usually not
done since it has a bad long term effect on video quality. In almost all the
major video calling systems, video resolution is a function of current
bandwidth (which changes as the rate control system detects differences)
and/or constrained by the physical device, but not a function of a user
dragging the window size. At an equivalent bitrate, it is better to compress
a larger resolution and display it smaller (higher PSNR), then compress a
smaller resolution if you are within the codec's effective bitrate for that
larger resolution.

Aron Rosenberg
LifeSize, A Division of Logitech

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 20 Jun 2011 10:30:34 +0200

On 06/17/11 21:38, Aron Rosenberg wrote:
> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     Video Size Change.
>
>     Alice and Bob are in a video call in their browsers and have
>     negotiate a high resolution video. Bob decides to change the size
>     of the windows his browser is displaying video to a small size.
>      Bob's browser should be able to regenerate the video codec
>     parameters with Alice's browser to change the resolution of video
>     Alice sends to match the smaller size.
>
>
> Changing compression due to a user change in window size is usually 
> not done since it has a bad long term effect on video quality. In 
> almost all the major video calling systems, video resolution is a 
> function of current bandwidth (which changes as the rate control 
> system detects differences) and/or constrained by the physical device, 
> but not a function of a user dragging the window size. At 
> an equivalent bitrate, it is better to compress a larger resolution 
> and display it smaller (higher PSNR), then compress a smaller 
> resolution if you are within the codec's effective bitrate for that 
> larger resolution.
Interesting info, but I don't know if it's true in all cases.

If the overall constraint on a conferencing system is bandwidth, the 
system might find the saving in average bandwidth saving of encoding 
from a base image of 140x200 for "small" windows rather than "HD" 
(720x1024?) worth considering, even though the sending of a complete 
I-frame to reinitialize the video stream might take some effort.

I like having the use case of "change recipient video window size, and 
as a result change sender bandwidth usage"; we might argue about the 
exact phrasing of the use case.

                    Harald

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 20 Jun 2011 06:52:44 -0700

Applications requiring (or making desirable) adequate response of the video
codec to changes of rendering window sizes are a perfect scenario for the
use of spatial scalability.  One avoids the waste of sending bits for a high
resolution picture that cannot be rendered at high resolution, but has
freedom to start sending high resolution, without the delay involved in
sending an I frame, at any time.

Scalable video is also a useful tool to combat losses and to adapt
seamlessly to changes in the channel bandwidth (to combat congestion, for
example).

Stephan

From:  Harald Alvestrand <[email-masked]>
Date:  Mon, 20 Jun 2011 10:30:34 +0200
To:  Aron Rosenberg <[email-masked]>
Cc:  <[email-masked]>
Subject:  [rtcweb] Changing video size (Re:  use case:)

 On 06/17/11 21:38, Aron Rosenberg wrote:
>> browser is displaying video to a small size.  Bob's browser should be able to
>> regenerate the video codec parameters with Alice's browser to change the
> Changing compression due to a user change in window size is usually not done
> since it has a bad long term effect on video quality. In almost all the major
> video calling systems, video resolution is a function of current bandwidth
> (which changes as the rate control system detects differences) and/or
> constrained by the physical device, but not a function of a user dragging the
> window size. At an equivalent bitrate, it is better to compress a larger
> resolution and display it smaller (higher PSNR), then compress a smaller
> resolution if you are within the codec's effective bitrate for that larger
> resolution.
>  
>  
>  
 
 If the overall constraint on a conferencing system is bandwidth, the system
might find the saving in average bandwidth saving of encoding from a base
image of 140x200 for "small" windows rather than "HD" (720x1024?) worth
considering, even though the sending of a complete I-frame to reinitialize
the video stream might take some effort.
 
 I like having the use case of "change recipient video window size, and as a
result change sender bandwidth usage"; we might argue about the exact
phrasing of the use case.
 
                    Harald

_______________________________________________ rtcweb mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Randell Jesup <[email-masked]>
Date: Mon, 20 Jun 2011 10:11:24 -0400

On 6/20/2011 4:30 AM, Harald Alvestrand wrote:
>> system detects differences) and/or constrained by the physical 
>> device, but not a function of a user dragging the window size. At 

Don't use "most current systems" as a guide.  Just because they don't do 
it doesn't mean it's a bad idea (it might be, but more proof than their 
not doing it is needed).

I would assert that in general, it's better to scale before encoding 
than to scale after decoding; and one scale operation is generally 
better than two (at camera->encoder, and decoder->display), given a 
constant bitrate.

> exact phrasing of the use case.

The more general issue here is one of communication: events such as 
window resize are of interest to the sender, and may cause the sender to 
change what they're sending.    Events could include bandwidth changes, 
decode resolution changes, decode framerate changes  (probably not 
common, but perhaps for managing battery use), exposure (decode partly- 
or fully-obscured, or perhaps the user has zoomed in on one portion of 
the video), etc.  How the encoder reacts to that is largely up to the 
encoder/app (and input from the decoder app on what it wants); 
transmission of the information to allow decision-making is what I'm 
concerned with.

Some of these can/should be spoken to in use-cases.

-- 
Randell Jesup
[email-masked]

---

From: Varun Singh <[email-masked]>
Date: Tue, 21 Jun 2011 09:45:43 +0200

In case of RTP, RTCP carries network level reception statistics and video/voice metrics (RTCP XR etc). However, there is a no standardized congestion control for RTP (http://tools.ietf.org/html/draft-gharai-avtcore-rtp-tfrc-00 uses packet loss and delay not some of the other cues mentioned in the mail).

Unless we identify a congestion control, I would suggest that the use-case capture that we need an extensible mechanism to carry this type of information and then the sender implements its own congestion control (best to its ability).

-- 
Varun Singh

On Monday, June 20, 2011 at 4:11 PM, Randell Jesup wrote:

> > > On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings <[email-masked] (mailto:[email-masked]) 
> [email-masked] (mailto:[email-masked])
> 
> [email-masked] (mailto:[email-masked])
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 21 Jun 2011 11:48:52 +0200

I just want to highlight that changing video display window size is 
captured already in the very first use case ("Simple Video Communication 
Service") in 
http://datatracker.ietf.org/doc/draft-holmberg-rtcweb-ucreqs/?include_text=1. 

In the preparation of the document we also discussed putting a 
requirement on that the receiving end should explicitly signal the size 
or the required codec parameters to the sender but concluded at that 
time that more discussion is needed.

My personal view is that this kind of functionality should be there. 
After all the suitable codec settings depend a lot on e.g. the display 
size. But I'm not sure that a requirement is needed. There must be a 
signaling channel of some kind available to set up streams (to negotiate 
codecs and codec settings). I think it can be re-used for this purpose.

Setting it up this way there could initially be simpler but standard 
compliant implementations that does not support this, with support added 
later - with the web application being unaffected.

//Stefan
On 2011-06-20 16:11, Randell Jesup wrote:
>>> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings<[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 21 Jun 2011 17:26:34 +0200

> Unless we identify a congestion control, I would suggest that the use-case capture that we need an extensible mechanism to carry this type of information and then the sender implements its own cong…
I think we need to separate the two requirements:

- response to network conditions (congestion control)
- response to recipient system conditions such as window size, CPU load, 
window being obscured, and so on.

Even though the possible actions to take might be the same, the two are 
fundamentally different; responses to end system events concern only the 
sender and the recipient, while responses to congestion events have the 
requirement that the behavior be globally sensible - that is, works well 
with the algorithms deployed by other users of the network.

                          Harald

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 21 Jun 2011 17:35:54 +0200

+1
On 2011-06-21 17:26, Harald Alvestrand wrote:
> [email-masked]

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 21 Jun 2011 09:11:15 -0700

On Jun 20, 2011, at 1:30 , Harald Alvestrand wrote:

>> Changing compression due to a user change in window size is usually not done since it has a bad long term effect on video quality. In almost all the major video calling systems, video resolution is a function of current bandwidth (which changes as the rate control system detects differences) and/or constrained by the physical device, but not a function of a user dragging the window size. At an equivalent bitrate, it is better to compress a larger resolution and display it smaller (higher PSNR), then compress a smaller resolution if you are within the codec's effective bitrate for that larger resolution.

Consider two different video flows using the same codec ...

Stream A is 640 x 480 at 1mbps which is this scaled to 320x240 and displayed

Stream B is 320 x 240 at 1mbps which is displayed at 320x240. 

My experience has been with modern video codecs that stream B will look better than stream A. As well as looking better, it will typically also have a better PSNR. There's a bunch of reasons why which are probably not worth going into here but give it a try and you will see what I mean. The key point is both streams where 1mbps. If stream B was sent at 256 kbps, then A would look better.

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 21 Jun 2011 09:16:28 -0700

>> with the algorithms deployed by other users of the network.

+1 It seems to me that what resolution video you send often is the largest size that is smaller than several constraints. The constraints include what you have bandwidth to transports, what the receiver is capable of displaying, and the desires of the two users involved. No one argues we should send 1080P video to a systems that is only capable of decoding 720P. I think what I was trying to get at in the use case is that these things depend on the desires of both the sending and receiving systems and they change dynamically during the session.

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 21 Jun 2011 09:23:41 -0700

What Cullen writes below matches my experience.  And, its not even limited
to "modern" codecs.  I would suggest that the observed behavior is true
for any video codec that has sub-pel motion compensation, i.e. any video
compression standard developed since H.263 (1998).
Stephan

On 6.21.2011 09:11 , "Cullen Jennings" <[email-masked]> wrote:

>[...]
>My experience has been with modern video codecs that stream B will look
>better than stream A. As well as looking better, it will typically also
>have a better PSNR. There's a bunch of reasons why which are probably not
>worth going into here but give it a try and you will see what I mean. The
>key point is both streams where 1mbps. If stream B was sent at 256 kbps,
>then A would look better.
>
>
>
>

---

From: Bert Greevenbosch <[email-masked]>
Date: Wed, 22 Jun 2011 03:19:42 +0000

Hi Cullen, all,

<cite>

</cite>

Can I ask what you mean with "both streams where 1mbps"? Do you refer to the capacity of the channel (i.e. how much data the channel can transport), or to the actually codec bitrate (i.e. how much data is sent from the streaming server).

In the first case, assuming that the size of stream B is considerably less than the size of stream A, the quality of Stream B can be expected to be higher than the quality of Stream A, since a higher capacity of the channel results in less packet loss. 

In the second case, assuming both streams are coded at the same bitrate, the quality of Stream B can also be expected to be higher than the quality of Stream A, due to the availability of more bits per pixel (though similar packet loss).

However, if we reduce the capacity of the channel, it becomes more complicated. I think this highly depends on which codec is being used. For example, I am not sure whether  Stream B with 25% channel capacity would look better than Stream A (after scaling) with 100% channel capacity. I think this highly depends on the codec and scaling algorithm, and saturation of the channel.

To come to the question on whether negotiation of the video size is needed: I feel that it could be a useful feature. Even if the channel allows streaming the 640x480 version without congestion problems now, where is the guarantee that it will not happen later? Probably there will be multiple video streams (at least partially) using the same path. If congestion happens later, the codec bitrate of multiple streams may need to be reduced. How to know from which streams the bitrate needs to be reduced? It seems to be better to prevent this and use a reduced codec bitrate from the start for those streams that are known not to require a higher bitrate.

However, whether we can cater for this also depends on the actual technical solution for negotiation and which codec to use. If adding video size negotiation will not increase complexity too much, I see no reason not to do it. However, if it adds much complexity to the solution, we might need to consider whether it is worth it. Therefore, maybe it should not be a requirement, but something weaker; i.e. it could be phrased as a SHOULD rather than a MUST.

Best regards,
Bert

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Cullen Jennings
Sent: 22 June 2011 00:11
To: Aron Rosenberg
Cc: [email-masked]
Subject: Re: [rtcweb] Changing video size (Re: use case:)

_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Justin Uberti <[email-masked]>
Date: Tue, 21 Jun 2011 20:59:19 -0700

Given the fact that this request for a video size change is an optimization
(either of resources or user experience), I think that SHOULD is probably
the strongest label we can put on it. Consider the case of a middlebox; if
it has no capacity to scale the stream, due to compute or codec
restrictions, there's not much it can do with this request.

> However, whether we can cater for this also depends on the actual technical
> solution for negotiation and which codec to use. If adding video size
> negotiation will not increase complexity too much, I see no reason not to do
> it. However, if it adds much complexity to the solution, we might need to
> consider whether it is worth it. Therefore, maybe it should not be a
> requirement, but something weaker; i.e. it could be phrased as a SHOULD
> rather than a MUST.
>
> Best regards,
> Bert
>
>
> better than stream A. As well as looking better, it will typically also have
> a better PSNR. There's a bunch of reasons why which are probably not worth
> going into here but give it a try and you will see what I mean. The key
> point is both streams where 1mbps. If stream B was sent at 256 kbps, then A
> would look better.
>
>
>
>

---

From: Bert Greevenbosch <[email-masked]>
Date: Wed, 22 Jun 2011 05:55:52 +0000

Hi Justin, all,

Yes, that is a second issue. Even if the receiving end can signal its current display size, the streaming source might not be able to provide it. Therefore flexibility would be needed for the source on how to handle the display size information.

Best regards,
Bert

From: Justin Uberti [mailto:[email-masked]]
Sent: 22 June 2011 11:59
To: Bert Greevenbosch
Cc: Cullen Jennings; Aron Rosenberg; [email-masked]
Subject: Re: [rtcweb] Changing video size (Re: use case:)

Given the fact that this request for a video size change is an optimization (either of resources or user experience), I think that SHOULD is probably the strongest label we can put on it. Consider the case of a middlebox; if it has no capacity to scale the stream, due to compute or codec restrictions, there's not much it can do with this request.

Best regards,
Bert

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Cullen Jennings
Sent: 22 June 2011 00:11
To: Aron Rosenberg
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [rtcweb] Changing video size (Re: use case:)

On Jun 20, 2011, at 1:30 , Harald Alvestrand wrote:

>> On Fri, Jun 17, 2011 at 11:09 AM, Cullen Jennings <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> Video Size Change.
>>

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb
_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 22 Jun 2011 08:06:28 +0200

All,

this RFC could perhaps be useful for this purpose of signaling display size: http://www.rfc-editor.org/rfc/rfc6236.txt

Stefan

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bert Greevenbosch
Sent: den 22 juni 2011 07:56
To: Justin Uberti
Cc: [email-masked]
Subject: Re: [rtcweb] Changing video size (Re: use case:)

Hi Justin, all,

Best regards,
Bert

Best regards,
Bert