Subject: Re: [video-codec] Charter issues from BoF - targets

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:28:08 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 5. Be clear about targets for coding efficiency.
> 
> Speaking personally, as long as we have a significant advantage over existing royalty-free options (e.g., Theora and VP8), then it is worthwhile publishing the result. Some people think we should strive for much more (i.e., significantly better than HEVC), and I think that's great, but if we were merely "competitive" with HEVC, I wouldn't count this as a failure. Greg Maxwell has language to this effect in his requirements draft. Should there be similar language in the charter as well?

I think we need to be clear on "when we are done" or we will never be done, thus, I think it is critical do decide this in the charter. I don't care what it says, it could say better than YUV sent as UTF-8 encoded floats in XML for all I care. I just want it to be clear when it is done. 

Any realist view of when it will be done would certainly suggest that what it will be competing against will be at least HEVC quality. I suspect that to realistically compete in the market place is needs to at similar power, computation and visual quality levels, it needs to not use more than 25% more bandwidth that HEVC or AVC. That might be a more realistic bar to set in the charter. 

The important thing to me is we know when we are done. My experience with video codecs is certainly that more time always bring more improvements and there is always some more improvements just down the road. The charter needs a way of answering when it is "good enough" to publish.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:28:43 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 9. Use signaling to have fine grained enablement of features.
> 
> Regardless of any IPR implications, this feels like a technical discussion. I.e., there are implications about interoperability, profiling, and testing here. You don't want more than 8...10 of these, or you add an enormous burden if you want to test all combinations exhaustively. I'm not saying this is a bad idea, just that there's a lot of details to work through (beyond the obvious "what features should the flags affect?"). If someone has an idea of something useful we can say in the charter on this subject, I'm all ears.

I'm not suggesting we would test ever possible combination of features - I think we would test some logical subset we cared about. Even with no features like this, it's hard to test all possible things that could happen given the complex code paths in a video codec. With a codec like VP8, just picking the input parameters you are going to test with already make a space that is way too large to test.  I'm not convinced this will make it a much harder but I agree not all possible features sets would be tested. 

Just consider the vector of enabled features an input parameter :-)

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:28:47 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 7. Have a strategy for achieving RF.
> 
> The most important part of this strategy is already specified in the charter, namely that the codec be developed "Under the IPR rules of the IETF." I discussed that in a little bit more detail at the BoF.

Send a separate email along lines of what I meant on this

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:28:54 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 6. Decide if you are doing one codec or many.
> 
> I think we should do one codec.

I don't think that is realistic if we want this implemented in hardware. There is just too many tradeoffs and we end up with more of a family of codecs with negotiated options.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:29:04 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 8. Be clear if the WG is creating new technology or selecting existing
>> technology.
> 
> Given the existing technology I'm aware of, I have no problem saying we're going to be creating something new here. That might preclude the possibility of the JCTVC offering the world HEVC royalty-free via the IETF, but that proposition seems so vanishingly unlikely that it won't keep me up at night.

What I really meant is are we going to more or less take VP.next from google or are we going to try and do work on the mailing list to create the technology. Note I could care less about where the codec came from, I only care about outcome. The reason I raise this is that if half the people think we are doing one thing and the other half thinks we are doing the other, it's a long slow road to get to consensus. I prefer bring small incremental bits of technology to the WG and we get something working together that is better than what would have happened otherwise but I can easily live with we test VP.next and if it is good enough, we publish it.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:29:08 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 10. Have a clear idea how to get test results to inform WG decisions.
> 
> Fortunately, we're in a much better position with video than we were with audio. The objective metrics are more useful... everyone knows they're still flawed in various ways, but you can still make a lot of progress relying on such metrics (see the various ITU/MEPG efforts that rely on them exclusively). PSNR measured on a single, short clip comparing mostly similar algorithms actually correlate with human observer ratings pretty well [1]. Most comparisons between different codecs rely on them, too.
> 
> By contrast, in audio optimizing for SNR is actively harmful, and even more advanced metrics like PEAQ are essentially blind to things like transients, which are one of the most important source of artifacts for a transform codec. So if we wanted useful results, we didn't have a lot of options other than relying on human listening tests.
> 
> Humans are still the ultimate gold standard for video, of course. At least for the purposes of getting a technique adopted, I think we can just say the burden of proof lies on the person proposing the technique. If the visual improvement is large enough, even if the objective metrics say it looks worse, it shouldn't actually take much to convince people it's a good idea.

I'm pretty skeptical about PSNR but I was not really pushing on how we did testing here. I was more focused on how we will get any test data at all. 

Saying no feature goes in without test data in the charter would be a good way to address this. I'm sure there are others ways but what I want to avoid is a bunch of technical proposal which seem like they might be OK idea but actually don't gain much. We need some reasonable way to have enough information to decide if a feature is worth adding or not.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:29:15 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 4. Sort out preferred licensing terms early.
> 
> We (Xiph.Org/Mozilla) obviously have some strong preferences here. You can look at our Opus disclosures to see what they are. Given the strong reactions against discussing such issues on the list with Opus, I'm hesitant to specify what those terms should be in the charter.

I don't believe we will ever sort them out if we don't do it at this stage and given they delayed opus for a year or more and still are not really resolved, I think it would be worth spending the time now to figure it out.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:29:20 +0000

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 
>> 3. Better plan for Liasons with other SDOs - particularly existing Joint *
> 
> So, I agree with Stephan's comments during the BoF that trying to set up something like the JCTVC with _both_ the ITU and ISO would be a multi-year effort in and of itself, making it somewhat impractical. But I certainly agree the current language in the charter on this subject could be improved. What text would people _like_ to see here?

I'd like to see it honestly describe what we intend to do. I would summarize that as "nothing" but we will let others know what we are up to.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:29:35 +0000

inline 

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[email-masked]> wrote:

>> 2. Be clear about "optimized for the internet"
> 
> This was left somewhat vague because we're still getting feedback on precisely what "the internet" (i.e., the IETF) thinks is important, but there are certainly a number of specific examples of ways we could improve on existing codecs:
> 
> 1) Fast/flexible congestion control (e.g., resolution changes without keyframes, etc.)

I don't think we need resolution changes without keyframes if that costs in overall average bandwidth - I think we need smooth transition of nitrates over a 2 order of magnitude range. 

> 2) Simplify the interaction of packet loss recovery with reference frame re-ordering (should we even use such re-ordering... if we don't, the interaction is _very_ simple).

> 3) "Fast channel switching", i.e., the ability to join broadcast streams without waiting for a keyframe.

I may not understand with 2 and 3 but these seem to be better phrased in the what we will do 

> 4) Support for screen captures (remote services, desktop sharing, etc.), which may require special coding tools, non-subsampled chroma (a feature notably lacking from VP8), etc.

I'm in favor of a codec for displaying desktop screen captures or slides but I think that is a pretty different set of requirements

> 
> These are just some of the things that immediately spring to mind given the <video> tag and WebRTC use-cases. I'm sure there's many more. I'm planning to meet with Janardhan Iyengar to discuss the interaction of video and transport on Thursday during the 15:10 session (others are welcome to join us: location currently TBD, but send e-mail and I'll make sure to let you know).
> 
> We can certainly list some of these examples in the charter, but I think it's somewhat premature to say those are the only things we're going to do, or even that we're going to do all of them. There was some criticism that "optimized for the internet" was ill-defined for Opus, but if you look at the actual result, you can see that it informed a lot of decisions, resulting in a codec that operates quite a bit differently from most audio codecs: <http://www.ietf.org/mail-archive/web/rtcweb/current/msg05205.html>.

I'm still confused on how this work is going to result in something more optimized for the internet than say VP8, or H.264

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 15 Jan 2013 15:54:37 +0100

> I don't think that is realistic if we want this implemented in hardware. There is just too many tradeoffs and we end up with more of a family of codecs with negotiated options.
To me, what distinguishes "one codec" from "many codecs" is that with 
one codec, it's possible to write one decoder that can decode all valid 
bitstreams. (that's not the same thing as "one decoder MUST decode all 
bitstreams, even though that's what we aimed for with VP8 - options, 
features, frame sizes and frame rates all conspire to make some 
bitstreams that not all decoders can handle - at least not in real time).