Subject: [rtcweb] Support of video with different resolutions

From: Christer Holmberg <[email-masked]>
Date: Wed, 19 Dec 2012 15:55:01 +0000

Hi,

Section 4.3.3.1 (description of the 'Video conferencing system with
central server' use-case) of the use-case-and-requirements draft
contains a note, describing different mechanisms for providing
video streams with different resolution.

                "Note: This use-case adds requirements on support for fast stream switches F7, on encryption of media and on ability to
                traverse very restrictive FWs. There exist several solutions that enable the server to forward one high resolution and several low
                resolution video streams: a) each browser could send a high resolution, but scalable stream, and the server could send just the
                base layer for the low resolution streams, b) each browser could in a simulcast fashion send one high resolution and one low
                resolution stream, and the server just selects or c) each browser sends just a high resolution stream, the server transcodes into
low resolution streams as required."
If we want to support content sent in different resolutions, the question is whether we need to mandate the browser to support a specific
mechanism.

As shown in the note, there are basically 3 mechanism: SVC, simulcast and "local transcoding".

The "local transcoding" alternative doesn't put any requirements on
browsers. The SVC and simulcast alternatives would.

However, the "local transcoding" alternative will consume lots of resources,

and as far as I know none of the proposed MTI video codecs support SVC.

Some people have indicated support for simulcast in the past, but it would

be nice to get a wider view.

Opinions?

Regards,

Christer

---

From: Bernard Aboba <[email-masked]>
Date: Fri, 21 Dec 2012 07:40:43 -0800

Christer said: 

"If we want to support content sent in different resolutions, the question is whether we need to mandate the browser to support a specific

mechanism.  As shown in the note, there are basically 3 mechanisms: SVC, simulcast and “local transcoding”."
[BA] One of the reasons that "no MTI codec proposal supports SVC" is that by definition mandates apply to a wide range of situations, from a browser on a mobile device to a desktop scenario involving exchange of multiple streams at high bandwidth.  This leads to mandates focusing on the "lowest common denominator", which may make it difficult mandate browser support for encoding of SVC or simulcast. 

SVC supports multiple scalability mechanisms:  temporal, spatial and quality.  Temporal is the most widely implemented in software (at least for H.264/SVC), although there is an open source implementation that supports all modes.  In hardware, support for spatial or quality scaling is not common.  As a result, a browser on a mobile device could have difficulty complying with a mandate to support spatial scaling within the encoder. 

Although cameras on mobile devices are growing increasingly capable, unlimited usage LTE plans are still not ubiquitous, which puts a damper on sending of simulcast from mobile devices.
Given the above, it seems hard to mandate any of the above options.

---

From: Gunnar Hellström <[email-masked]>
Date: Sat, 22 Dec 2012 00:27:41 +0100

On 2012-12-21 16:40, Bernard Aboba wrote:
> SVC supports multiple scalability mechanisms: temporal, spatial and 
> quality.  Temporal is the most widely implemented in software (at 
> least for H.264/SVC), although there is an open source implementation 
> that supports all modes.  In hardware, support for spatial or quality 
> scaling is not common. As a result, a browser on a mobile device could 
> have difficulty complying with a mandate to support spatial scaling 
> within the encoder.
>
This is sad. For good usability of video, maintained frame rate is 
usually much more important than maintained spatial resolution. E.g. for 
sign language or lip reading usage with a single person in image, a 
frame rate under 20 fps introduces loss of language contents, and 
requires the users to try to fill in the gaps by imagination, while 
spatial resolution reduction down to QCIF causes much less harm to 
language perception possibilities.

Gunnar

---

From: Bernard Aboba <[email-masked]>
Date: Fri, 21 Dec 2012 17:02:25 -0800

A minimum frame rate can be assured in SVC by having the base layer support the required resolution (e.g.  30 fps), while having additional layers enable higher frame rates (e.g. An additional 30 fps, bringing the total to 60 fps). Some chipsets do support temporal scaling, and so the situation for temporal scaling is considerably better.

On Dec 21, 2012, at 15:28, "Gunnar Hellström" <[email-masked]> wrote:

> This is sad. For good usability of video, maintained frame rate is usually much more important than maintained spatial resolution. E.g. for sign language or lip reading usage with a single person in image, a frame rate under 20 fps introduces loss of language contents, and requires the users to try to fill in the gaps by imagination, while spatial resolution reduction down to QCIF causes much less harm to language perception possibilities.
> 
> Gunnar
> 
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Alex Eleftheriadis <[email-masked]>
Date: Sun, 23 Dec 2012 01:50:19 -0500

Benard, I don't believe that it is true that temporal is the most widely implemented in software. Vidyo introduced, and has been shipping,  software-based temporal+scalable SVC since 2008 (with deployments including Google+ Hangouts). What other implementation are you referring to that would tip the scale towards temporal only?

--Alex

On Dec 22, 2012, at 1:27 AM, Gunnar Hellström <[email-masked]<mailto:[email-masked]>> wrote:

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Thu, 27 Dec 2012 16:40:07 +0000

> This is sad. For good usability of video, maintained frame rate is usually much more important than maintained spatial resolution. E.g. for sign language or lip reading usage with a single person i…

Hmm - I think the trade off is a bit more complicated. The is both a minimum number of pixels for the hand + a minimum frame rate - if you get under either of theses , SL does not work real well. Does anyone have pointers to research that shows what theses  are? I've see this type of study in the past but can't find it.

---

From: Randell Jesup <[email-masked]>
Date: Mon, 31 Dec 2012 11:55:06 -0500

On 12/21/2012 6:27 PM, Gunnar Hellström wrote:
>> scaling is not common. As a result, a browser on a mobile device 
>> could have difficulty complying with a mandate to support spatial 
>> scaling within the encoder.
>>
> usually much more important than maintained spatial resolution. E.g. 
> for sign language or lip reading usage with a single person in image, 
> a frame rate under 20 fps introduces loss of language contents, and 

Strong agreement. High (and consistent!) frame rate is critical to VRS 
and sign-language - and for hearing people, it 'feels' much more like 
talking to someone if the video framerate is kept >20 (and preferably 
 >25), with perfect lip-sync. This is based on the experience I and 
Maire Reavy had at Worldgate, where we provided videophones for VRS 
usage. 30FPS QCIF works pretty well for sign-language, and better 
generally than 20FPS CIF, even though CIF is 4x the number of pixels.

Note that there's no reason a browser in a mobile device couldn't 
maintain 30fps, and if congestion indicates it needs to reduce bandwidth 
too much for 'good quality' 30FPS, it can instead reduce the resolution. 
The only time I'd value resolution over framerate is when viewing 
graphics or the equivalent. SVC certainly isn't needed here for 1-1 
communication; a main advantage of SVC is to allow a node in the network 
(i.e. conf. server/MCU) to subset down a stream for one recipient while 
providing higher-resolution streams to others while avoiding 
decode/re-encode on the server. Of course, this was the use-case that 
started this.

In a corollary, frame rate should not drop when there's a lot of 
movement, or should only drop momentarily. (And I mean very momentarily, 
like dropping a single frame when there's a big jump in motion from the 
last frame, then resuming 30FPS at higher quantization or higher bitrate.)

However, Christer's original question said:
"However, the “local transcoding” alternative will consume lots of 
resources, and as far as I know none of the proposed MTI video codecs 
support SVC. "

This is I believe is at least partially incorrect. From 
blog.webmproject.com, 1/27/2012, "VP8 Codec SDK "Duclair" Released":

    This release introduces substantial new VP8 encoder features that
    are especially useful for real-time use cases such as live streaming
    and videoconferencing.

      * Temporal scalability produces a video stream that can be
        decimated to different frame rates, with independent rate
        targeting for each substream.
      * Multiframe postprocessing can make visual quality more
        consistent in the presence of frames that are of substantially
        different quality than the surrounding frames, as in the
        temporal scalability case and in some forced keyframe scenarios.
      * Multiple-resolution encoding enables simultaneous encoding of
        the same content at different resolutions, resulting in much
        faster encoding than processing them separately.

This is both multicast and temporal scalability.

-- 
Randell Jesup
[email-masked]

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 31 Dec 2012 10:51:38 -0800

> This is both multicast and temporal scalability.

[BA] The above seems to suggest support for temporal, quality and spatial scalability. 
Did you really mean "multicast" (since that's out of scope of WebRTC)?

---

From: Randell Jesup <[email-masked]>
Date: Mon, 31 Dec 2012 16:50:34 -0500

> Did you really mean "multicast" (since that's out of scope of WebRTC)?

I meant sending multiple streams at different resolutions at the same 
time (as opposed to SVC spacial enhancement layers).

Overloaded terms.

-- 
Randell Jesup
[email-masked]