Subject: [rtcweb] Is support for non square pixels MTI

From: Cullen Jennings <[email-masked]>
Date: Thu, 25 Aug 2016 07:20:40 -0600

I was trying to figure out from the specs what the story is on WebRTC support for non square pixels (for example CIF resolution video is often not quite square pixels).

Thoughts on what the specs should say on this ? Anyone know what browser do today?

Thanks Cullen

---

From: Randell Jesup <[email-masked]>
Date: Wed, 31 Aug 2016 11:27:06 -0400

> Thoughts on what the specs should say on this ? Anyone know what browser do today?

Yeah... ugh.  So most cameras *now* are square pixels.  Non-square pixel 
support would allow things like 640x240 (or 320x480) at 2:1/1:2 ratios 
as well.  Some much older cameras are the 10:11-ish ratios that also 
apply to BT.601/etc (see 
https://en.wikipedia.org/wiki/Pixel_aspect_ratio), and HDV/HDCAM (1.3 -- 
1440x1080).

So we should be defining how MediaStreamTracks and PeerConnections deal 
with this.  Must all MSTs be square pixels always?  (and what if the 
source just isn't, must it be converted to square at the input point 
(typically GetUserMedia)?  Can PeerConnection carry a non-square-pixel 
stream (separate from the MST requirement) for use outside of the W3 
environment?  If so, how is it signaled?  (I suspect there's some way to 
do so in the SDP, haven't checked).

If non-square are allowed in MSTs, we need to define some property of 
the stream so the <video> element can correctly display it.

There's work no matter which we select, though likely less in total if 
we mandate square pixels everywhere - though it loses some capability.  
And I think we need to say something (here and/or in the W3).  Passing 
data without any indication as the ratio (if it's not fixed as square) 
is just a mess, and where we are today.

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email [email-masked]!  Way too much spam

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Wed, 31 Aug 2016 17:53:16 +0200

On 31/08/2016 17:27, Randell Jesup wrote:
> So we should be defining how MediaStreamTracks and PeerConnections 
> deal with this.  Must all MSTs be square pixels always?  (and what if 
> the source just isn't, must it be converted to square at the input 
> point (typically GetUserMedia)?  Can PeerConnection carry a 
> non-square-pixel stream (separate from the MST requirement) for use 
> outside of the W3 environment?  If so, how is it signaled?  (I suspect 
> there's some way to do so in the SDP, haven't checked). 

At least in H264, pixel aspect ratio is handled internally by the 
encoder/decoder and signaled on the sar-understood, sar-supported sdp 
fmtp attributes. Not sure if VP8 has something similar.

Best regards

Sergio

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Wed, 31 Aug 2016 17:17:43 +0000

The RFC 6236 stuff allows defines sar parameter that allows us to signal non square pixels in SDP for all types of video codecs. It ends up with SDP like 

a=imageattr:97 send [x=720,y=576,sar=1.1]

JSEP requires support for the 6236 signaling ( see https://tools.ietf.org/html/draft-ietf-rtcweb-jsep-15 section 3.6.2) but it's not clear if the a browser is required to implement receiving video with a sar other than 1.0 ? 

> On Aug 31, 2016, at 9:53 AM, Sergio Garcia Murillo <[email-masked]> wrote:
> 
>> So we should be defining how MediaStreamTracks and PeerConnections deal with this.  Must all MSTs be square pixels always?  (and what if the source just isn't, must it be converted to square at the input point (typically GetUserMedia)?  Can PeerConnection carry a non-square-pixel stream (separate from the MST requirement) for use outside of the W3 environment?  If so, how is it signaled?  (I suspect there's some way to do so in the SDP, haven't checked). 
> 
> At least in H264, pixel aspect ratio is handled internally by the encoder/decoder and signaled on the sar-understood, sar-supported sdp fmtp attributes. Not sure if VP8 has something similar.
> 
> Best regards
> 
> Sergio
> 
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 31 Aug 2016 18:12:39 +0000

Note that capability negotiation (e.g. SDP) and actual media format (current RTP packets) may have different SARs, which may even change dynamically if multiple are negotiated. (Similar to how negotiated and actual resolutions may also differ, even dynamically.)

Mo

On Aug 31, 2016, at 1:17 PM, Cullen Jennings (fluffy) <[email-masked]> wrote:

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 31 Aug 2016 18:42:27 +0000

H.264 bitstreams indicate their actual SAR in SPS VUI aspect_ratio_idc, which is not used by the encoder or decoder but can be passed to a renderer.

VP8 bitstreams do not indicate their actual SAR, presumably because 1:1 is assumed, or higher systems layers (e.g. webm container) are expected to convey this. 

Mo

Best regards

Sergio