Subject: Re: [video-codec] Charter issues from BoF

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 04 Dec 2012 17:10:38 -0800

Spencer Dawkins wrote:
> I'm not seeing a problem yet for this proposed working group, but I am
> remembering that "the Internet" covers a LOT of different points in the
> "speed/delay/reliability" space, from interconnected data centers to
> environments that are slightly better-connected than DTNRG networks :-)
> ... it would be great to be clear on what environments are being targeted.

This is good advice. My primary interest is public, consumer-level, 
best-effort networks (i.e., where people run browsers). I can try to add 
some language to the charter to that effect if no one objects to that 
being the focus.

> program. I'd love for a chartered videocodec working group to have a
> good plan for liaison relationships, and I say that without reference to
> any other IETF working group having, or not having, a good plan for
> liaison relationships :-)

I've been approached by a couple of people from the MPEG IVC effort who 
think that we can do better than liaison-style collaboration, but I'm 
not sure what the best approach is here, yet. I will try to encourage 
them to post their thoughts on the list.

> I'd suggest the proposed working group community discuss whether it's
> worth producing a video codec that is known to not be RF, early in the
> process.

There is some difficulty in deciding what "known to not be RF" means. 
Some people would take the existence of an IPR disclosure---any IPR 
disclosure---that does not make that IPR available under RF terms to be 
an indication that the work is not RF... and in the case that the 
disclosure is by the author of the technique in question, this is very 
compelling. Other people are willing to investigate the real risk 
associated with those disclosures, evaluate the effectiveness of 
design-arounds (which do not incur an obligation to update that 
disclosure), etc.

Experience with Opus suggests to me that this should be handled by WG 
consensus on a case-by-case basis, instead of with an iron-clad rule in 
the charter. This was discussed a fair bit at the bar BoF in Vancouver, 
and the "developed under the IPR rules of the IETF" language that's 
currently in the charter was strongly suggested to me as the best way to 
phrase this.

Speaking personally, I would not support publishing as an RFC any codec 
that I did not believe was RF.

> I'm not saying that I have an opinion about "one codec or many", I'm
> saying it would be helpful to understand whether one codec could meet
> the need for a codec that's "optimized for the Internet", before
> deciding whether to produce more than one codec.

I think it can. The biggest challenge comes in deciding whether or not 
to support coding tools that are not robust in the face of packet loss. 
This includes, for example, anything which makes decisions about what 
values to read from the bitstream based on the decoded, reconstructed 
pixels. In the presence of loss, making one of those decisions 
incorrectly would corrupt the rest of the frame.

To illustrate this, one of the items mentioned in the VP9 presentation 
included scoring candidate motion vectors by matching the 
(already-decoded) edge pixels from neighboring blocks to the reference 
frame at the same offset. I believe this is okay in its current form, 
because the codebook used to decode the motion vector residual has no 
dependency on the value of the MV being used as a predictor. But if we 
wanted to use a scheme that _did_ depend on that value, we'd have to 
give up the ability to look at those pixels in neighboring blocks. I've 
had in mind for a while now a coding scheme which allows you to code a 
MV relative to two different predictors, but unlike the normal approach 
of "code which predictor is closer, and then code an offset relative to 
that predictor," doesn't introduce any redundancy in the coding. This 
depends on exactly what those two predictors are (consider the case 
where they're the same, for example), so it wouldn't be compatible with 
this VP9 technique for deriving the list of predictors.

But even in this case, we could pick which of the two approaches works 
better on average, or even allow both techniques to be used, and switch 
between them. Video frames have enough bits to make this kind of 
switching at the frame or even smaller levels quite practical, and you 
could pick up quite a lot of alternate algorithms like this before it 
made sense to give up all the things that _are_ common in the two use 
cases and have two completely separate codecs.

> If I'm understanding what other people are saying, I'm not hearing "need
> old technology where patents have expired" as a strategy for video
> codecs, but It might be helpful for the proposed working group community
> to be clear on how much new technology people are planning to define,
> early on.

I think we should be doing both things. If you look carefully at some of 
the drafts that have already been submitted, you'll see some of the 
references are _very_ old. There's a lot of research out there that's 
been around for a long time, but never made it into a widely deployed 
codec. Sometimes that's for very good reasons. Sometimes it's because 
design decisions which were important at the time have become less so as 
Moore's law has changed the best operating point on the chip area/clock 
speed/power usage frontier. Sometimes it's because the technique 
introduced practical problems which are solvable, but no one ever 
attempted to solve them.

We (Mozilla) are also exploring ways we can actually publish solid 
information about why we think the the codec will be royalty-free (once 
it is finished). As you may know, it is normally not in our interest to 
do so (it is basically mapping out your defense, or at least part of it, 
to any potential litigants), but it may wind up being worth the trade-off.

> I hope this is helpful. And again, I think the important thing is that
> the proposed working group community take Cullen's suggestions
> seriously, more than what the community decides about them.

I think this (and Cullen's feedback generally) has been very helpful. I 
hope to submit an updated version of the charter incorporating a bunch 
of the feedback received thus far before next week.

---

From: "Syed, Yasser" <[email-masked]>
Date: Wed, 5 Dec 2012 01:18:22 +0000

Was anyone involved in some of the AVS discussions in the past?  It wasn't
RF, but I believe there was some tradeoff discussions in terms of what IPR
made it in or out and how it affected end video quality. It seems like a
similar effort in the past that did result in an output document.

Yasser

On 12/4/12 6:10 PM, "Timothy B. Terriberry" <[email-masked]> wrote:

>_______________________________________________
>video-codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/video-codec

---

From: Spencer Dawkins <[email-masked]>
Date: Thu, 06 Dec 2012 12:07:40 -0600

Hi, Tim,

Thanks for your reply. Where you are seems, to me, to be well within the 
boundaries of the kind of decisions we rely on working groups to make, 
and rely on working group chairs to call consensus for.

On one specific point, and this is as much for your amusement as for 
your information ...

> design-arounds (which do not incur an obligation to update that

So, right. We rely on IETF working groups to make decisions about what's 
acceptable from an IPR standpoint, in specific cases.

One of my favorite quotes of all time is from the 1968 NATO Software 
Engineering (conveniently located at 
http://www.scrummanager.net/files/nato1968e.pdf):

"Bauer: The concept seems to be clear by now. It has been defined 
several times by examples of what it is not."

Your response is very clear at a detailed level. My comment was at the 
50,000-foot level - I'm betting there are IPR situations that you and 
other working group participants would agree are not even remotely "RF", 
and I'm wondering if it's helpful for working group participants to know 
whether they expect to proceed if that's the way things play out.

> currently in the charter was strongly suggested to me as the best way to

Yes, exactly - not that this is nailed down in the charter, but that the 
working group has given the question some thought. Perhaps that's a 
reasonable group activity; perhaps it's better left to individual 
contemplation.

> that I did not believe was RF.

This is the kind of thought I'm suggesting :-)

I'm glad my comments were mostly useful. IMO, one of the most important 
things IAB members do is to help IETF participants charter new work.

Spencer

---

From: Rob Glidden <[email-masked]>
Date: Mon, 10 Dec 2012 13:06:03 -0800

> think that we can do better than liaison-style collaboration [...]

Indeed, at the last MPEG meeting, decisions were taken to approach IETF 
about potential collaboration in RF codec standardization and to make 
publicly available the IVC Test Model (ITM) v 3.0.

Internet Video Coding (aka "IVC") is a proposal for an RF codec standard 
under active exploration by MPEG.

IVC is roughly based on a patent-scrubbed version of AVS and partly 
inspired by an RF codec activity at Sun called OMC.

IVC is a work-in-progress, and improvements in the test code base and 
additional/alternative/better technologies (if RF vetted) are welcome.

There is a lot to discuss about how collaboration could proceed so I 
won't try to put it all in one email.   But in the spirit of "rough 
consensus and working code" and links below, here's a start.

Rob

MPEG 102 Meeting Resolutions, section 15.2 Internet Video Codec:
http://www.itscj.ipsj.or.jp/sc29/open/29view/29n13049c.htm

MPEG Liaison Statement to IETF RTCWeb WG and IETF video-codec BoF on the 
progress of IVC and WebVC:
https://datatracker.ietf.org/liaison/1212/

IVC public repository:
- URL: http://wg11.sc29.org/ivcpublic/repos/
- Read-Only User: ivcpublic
- Read-Only Password: ivcpublic
- (posted on IVC AHG alias)

IVC Ad Hoc group alias:
- http://mpeg.chiariglione.org/meetings/shanghai12/shanghai_ahg.htm

ISO, SC29, MPEG procedures:
- http://www.itscj.ipsj.or.jp/sc29/29w7proc.htm, in particular:
- ISO/IEC Directives Part 1: Procedures for the technical work (9th 
edition, 2012)
- ISO/IEC Directives Supplement - Procedures specific to JTC 1 (3rd 
edition, 2012)
- Common Patent Policy for ITU-T/ITU-R/ISO/IEC
- Guidelines for Implementation of the Common Patent Policy for 
ITU-T/ITU-R/ISO/IEC

Ad Hoc Group rules:
- SC29: Section 1.14, ISO/IEC Directives, Part 1 Supplement --- 
Procedures specific to JTC 1
- MPEG: http://mpeg.chiariglione.org/ad-hoc_groups.php

------------------------------------------------------------------------

On 12/4/2012 5:10 PM, Timothy B. Terriberry wrote:
> best-effort networks (i.e., where people run browsers). I can try to 
> add some language to the charter to that effect if no one objects to 
> that being the focus.
>
> I've been approached by a couple of people from the MPEG IVC effort 
> who think that we can do better than liaison-style collaboration, but 
> I'm not sure what the best approach is here, yet. I will try to 
> encourage them to post their thoughts on the list.
>
> disclosure---that does not make that IPR available under RF terms to 
> be an indication that the work is not RF... and in the case that the 
> consensus on a case-by-case basis, instead of with an iron-clad rule 
> in the charter. This was discussed a fair bit at the bar BoF in 
> Vancouver, and the "developed under the IPR rules of the IETF" 
> language that's currently in the charter was strongly suggested to me 
> as the best way to phrase this.
>
> Speaking personally, I would not support publishing as an RFC any 
> codec that I did not believe was RF.
>
> to support coding tools that are not robust in the face of packet 
> loss. This includes, for example, anything which makes decisions about 
> what values to read from the bitstream based on the decoded, 
> reconstructed pixels. In the presence of loss, making one of those 
> decisions incorrectly would corrupt the rest of the frame.
>
> give up the ability to look at those pixels in neighboring blocks. 
> I've had in mind for a while now a coding scheme which allows you to 
> code a MV relative to two different predictors, but unlike the normal 
> approach of "code which predictor is closer, and then code an offset 
> relative to that predictor," doesn't introduce any redundancy in the 
> coding. This depends on exactly what those two predictors are 
> (consider the case where they're the same, for example), so it 
> wouldn't be compatible with this VP9 technique for deriving the list 
> of predictors.
>
> better on average, or even allow both techniques to be used, and 
> switch between them. Video frames have enough bits to make this kind 
> of switching at the frame or even smaller levels quite practical, and 
> you could pick up quite a lot of alternate algorithms like this before 
> it made sense to give up all the things that _are_ common in the two 
> use cases and have two completely separate codecs.
>
> I think we should be doing both things. If you look carefully at some 
> of the drafts that have already been submitted, you'll see some of the 
> design decisions which were important at the time have become less so 
> as Moore's law has changed the best operating point on the chip 
> area/clock speed/power usage frontier. Sometimes it's because the 
> technique introduced practical problems which are solvable, but no one 
> ever attempted to solve them.
>
> information about why we think the the codec will be royalty-free 
> (once it is finished). As you may know, it is normally not in our 
> interest to do so (it is basically mapping out your defense, or at 
> least part of it, to any potential litigants), but it may wind up 
> being worth the trade-off.
>
> I think this (and Cullen's feedback generally) has been very helpful. 
> I hope to submit an updated version of the charter incorporating a 
> bunch of the feedback received thus far before next week.