Subject: [video-codec] Charter issues from BoF

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 06 Nov 2012 11:26:25 -0500

I wanted to follow up on the "lessons learned" that Cullen presented  
during the BoF,  
<http://www.ietf.org/proceedings/85/slides/slides-85-videocodec-7.pdf>, which  
he said he wished had been addressed in the charter.

> 1. Don't use source code as a normative spec

I think there's broad agreement that this is a good idea this time  
around, but in the codec WG we specified this in a Guidelines document  
(RFC 6569). The proposed charter says that we'll use that document as  
the starting point for the guidelines for this effort, making changes  
as appropriate. We can certainly list this as one such change right now.

> 2. Be clear about "optimized for the internet"

This was left somewhat vague because we're still getting feedback on  
precisely what "the internet" (i.e., the IETF) thinks is important,  
but there are certainly a number of specific examples of ways we could  
improve on existing codecs:

1) Fast/flexible congestion control (e.g., resolution changes without  
keyframes, etc.)
2) Simplify the interaction of packet loss recovery with reference  
frame re-ordering (should we even use such re-ordering... if we don't,  
the interaction is _very_ simple).
3) "Fast channel switching", i.e., the ability to join broadcast  
streams without waiting for a keyframe.
4) Support for screen captures (remote services, desktop sharing,  
etc.), which may require special coding tools, non-subsampled chroma  
(a feature notably lacking from VP8), etc.

These are just some of the things that immediately spring to mind  
given the <video> tag and WebRTC use-cases. I'm sure there's many  
more. I'm planning to meet with Janardhan Iyengar to discuss the  
interaction of video and transport on Thursday during the 15:10  
session (others are welcome to join us: location currently TBD, but  
send e-mail and I'll make sure to let you know).

We can certainly list some of these examples in the charter, but I  
think it's somewhat premature to say those are the only things we're  
going to do, or even that we're going to do all of them. There was  
some criticism that "optimized for the internet" was ill-defined for  
Opus, but if you look at the actual result, you can see that it  
informed a lot of decisions, resulting in a codec that operates quite  
a bit differently from most audio codecs:  
<http://www.ietf.org/mail-archive/web/rtcweb/current/msg05205.html>.

> 3. Better plan for Liasons with other SDOs - particularly existing Joint *

So, I agree with Stephan's comments during the BoF that trying to set  
up something like the JCTVC with _both_ the ITU and ISO would be a  
multi-year effort in and of itself, making it somewhat impractical.  
But I certainly agree the current language in the charter on this  
subject could be improved. What text would people _like_ to see here?

> 4. Sort out preferred licensing terms early.

We (Xiph.Org/Mozilla) obviously have some strong preferences here. You  
can look at our Opus disclosures to see what they are. Given the  
strong reactions against discussing such issues on the list with Opus,  
I'm hesitant to specify what those terms should be in the charter.

> 5. Be clear about targets for coding efficiency.

Speaking personally, as long as we have a significant advantage over  
existing royalty-free options (e.g., Theora and VP8), then it is  
worthwhile publishing the result. Some people think we should strive  
for much more (i.e., significantly better than HEVC), and I think  
that's great, but if we were merely "competitive" with HEVC, I  
wouldn't count this as a failure. Greg Maxwell has language to this  
effect in his requirements draft. Should there be similar language in  
the charter as well?

> 6. Decide if you are doing one codec or many.

I think we should do one codec.

> 7. Have a strategy for achieving RF.

The most important part of this strategy is already specified in the  
charter, namely that the codec be developed "Under the IPR rules of  
the IETF." I discussed that in a little bit more detail at the BoF.

> 8. Be clear if the WG is creating new technology or selecting existing
> technology.

Given the existing technology I'm aware of, I have no problem saying  
we're going to be creating something new here. That might preclude the  
possibility of the JCTVC offering the world HEVC royalty-free via the  
IETF, but that proposition seems so vanishingly unlikely that it won't  
keep me up at night.

> 9. Use signaling to have fine grained enablement of features.

Regardless of any IPR implications, this feels like a technical  
discussion. I.e., there are implications about interoperability,  
profiling, and testing here. You don't want more than 8...10 of these,  
or you add an enormous burden if you want to test all combinations  
exhaustively. I'm not saying this is a bad idea, just that there's a  
lot of details to work through (beyond the obvious "what features  
should the flags affect?"). If someone has an idea of something useful  
we can say in the charter on this subject, I'm all ears.

> 10. Have a clear idea how to get test results to inform WG decisions.

Fortunately, we're in a much better position with video than we were  
with audio. The objective metrics are more useful... everyone knows  
they're still flawed in various ways, but you can still make a lot of  
progress relying on such metrics (see the various ITU/MEPG efforts  
that rely on them exclusively). PSNR measured on a single, short clip  
comparing mostly similar algorithms actually correlate with human  
observer ratings pretty well [1]. Most comparisons between different  
codecs rely on them, too.

By contrast, in audio optimizing for SNR is actively harmful, and even  
more advanced metrics like PEAQ are essentially blind to things like  
transients, which are one of the most important source of artifacts  
for a transform codec. So if we wanted useful results, we didn't have  
a lot of options other than relying on human listening tests.

Humans are still the ultimate gold standard for video, of course. At  
least for the purposes of getting a technique adopted, I think we can  
just say the burden of proof lies on the person proposing the  
technique. If the visual improvement is large enough, even if the  
objective metrics say it looks worse, it shouldn't actually take much  
to convince people it's a good idea.

[1] http://ieeexplore.ieee.org/iel5/2220/4550681/04550695.pdf?arnumber=4550695

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 06 Nov 2012 13:49:49 -0500

On 11/06/2012 11:26 AM, Timothy B. Terriberry wrote:
> We can certainly list some of these examples in the charter, but I think
> it's somewhat premature to say those are the only things we're going to
> do, or even that we're going to do all of them. There was some criticism
> that "optimized for the internet" was ill-defined for Opus, but if you
> look at the actual result, you can see that it informed a lot of
> decisions, resulting in a codec that operates quite a bit differently
> from most audio codecs:

I think "optimized for the internet" is a process more than a feature.
When writing a codec, you make hundreds small (and not so small)
decisions that can have an impact on how well the codec will work in a
particular situation. In this case, what we're saying is that we'll try
to optimize these decisions for use on the Internet. So I think the
charter should only have "optimized for the Internet" and then have the
requirements specify higher-level Internet-related features we're going
to aim for.

> can look at our Opus disclosures to see what they are. Given the strong
> reactions against discussing such issues on the list with Opus, I'm
> hesitant to specify what those terms should be in the charter.

I agree with sorting out the licensing terms as early as possible,
though as Tim points out, this shouldn't be part of the charter itself.

> worthwhile publishing the result. Some people think we should strive for
> much more (i.e., significantly better than HEVC), and I think that's
> great, but if we were merely "competitive" with HEVC, I wouldn't count
> this as a failure. Greg Maxwell has language to this effect in his
> requirements draft. Should there be similar language in the charter as
> well?

The "goal" is certainly to have a codec that's as good as possible,
ideally being better than HEVC. But this codec would be useful even if
it's "only" better than VP8 and Theora. This is similar to Opus, where
having "better than Speex" as a requirement didn't prevent us from
ending up better than even non-real-time encumbered codecs.

> charter, namely that the codec be developed "Under the IPR rules of the
> IETF." I discussed that in a little bit more detail at the BoF.

I really think that "Under the IPR rules of the IETF" is the only thing
that should go in the charter. The rest we should adapt and optimize as
we go along (even though several strategies have already been discussed).

> IETF, but that proposition seems so vanishingly unlikely that it won't

Yeah, I don't really mind leaving the option, but realistically what
we're going to end up doing is creating something new.

> exhaustively. I'm not saying this is a bad idea, just that there's a lot
> of details to work through (beyond the obvious "what features should the
> flags affect?"). If someone has an idea of something useful we can say
> in the charter on this subject, I'm all ears.

Agree with Tim, that's a technical discussion we need to have at some
point. That point is probably late in the process when we can actually
evaluate the impact it would have on the actual codec.

> progress relying on such metrics (see the various ITU/MEPG efforts that
> rely on them exclusively). PSNR measured on a single, short clip

Speaking from experience with Opus, I don't think that was a big issue.
Not having an advantage in getting as much IPR in as possible regardless
of quality meant that we were usually able to come pretty quickly to an
agreement over what was useful. In the end, there is quality, but also
complexity, simplicity, robustness, IPR and so on to keep in mind, so
trying to have fixed rules is more complicated than actually making the
decisions on a rough consensus basis.

Cheers,

	Jean-Marc

---

From: Spencer Dawkins <[email-masked]>
Date: Wed, 07 Nov 2012 14:21:08 -0600

On 11/6/2012 10:26 AM, Timothy B. Terriberry wrote:
> <http://www.ietf.org/proceedings/85/slides/slides-85-videocodec-7.pdf>,
> which he said he wished had been addressed in the charter.

Tim, I have opinions about some of these points, but think it's much 
more important for me to thank you for taking Cullen's list seriously.

I take the comment from the meeting, that we don't need to nail down 
some of these points at charter time, but I think there are some that we 
do need to nail down at charter time.

Thanks for kicking off the discussion.

Spencer

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 07 Nov 2012 15:34:23 -0500

> more important for me to thank you for taking Cullen's list seriously.

I'm still quite interested in hearing what those opinions are.

---

From: Spencer Dawkins <[email-masked]>
Date: Mon, 19 Nov 2012 17:32:14 -0600

> I'm still quite interested in hearing what those opinions are.

Hi, Tim,

Thanks for the interest. I'm looking at Cullen's list, and thinking 
about these points:

2.Be clear about “optimized for internet”

I'm not seeing a problem yet for this proposed working group, but I am 
remembering that "the Internet" covers a LOT of different points in the 
"speed/delay/reliability" space, from interconnected data centers to 
environments that are slightly better-connected than DTNRG networks :-) 
... it would be great to be clear on what environments are being targeted.

This may be tied to 6. below.

3.Better plan for Liaisons with other SDOs - particularly existing Joint *

All of these thoughts are me speaking as an individual participant, but 
in another part of my life, I'm the IAB lead for the Liaison Oversight 
program. I'd love for a chartered videocodec working group to have a 
good plan for liaison relationships, and I say that without reference to 
any other IETF working group having, or not having, a good plan for 
liaison relationships :-)

4.Sort out preferred IPR licensing terms early

So, RTCWeb has been chatting about choosing between two (existing) video 
codecs as "mandatory-to-implement", and it seems to me that what started 
out as a technical discussion has morphed into a discussion about 
preferred IPR licensing terms - and a pretty tenacious discussion, at that.

I'd suggest the proposed working group community discuss whether it's 
worth producing a video codec that is known to not be RF, early in the 
process.

6.Decide if you are doing one codec or many

As the charter suggests, this isn't necessary for a decision to charter 
a working group, and can be discussed in early analysis work, but it's a 
classic IETF post-charter issue, and not just in RAI. I was editor for 
the Softwires problem statement, and the working group went around the 
block a few times on whether "the problem" mapped into a mesh network or 
a hub-and-spoke network ... before converging on "both".

I'm not saying that I have an opinion about "one codec or many", I'm 
saying it would be helpful to understand whether one codec could meet 
the need for a codec that's "optimized for the Internet", before 
deciding whether to produce more than one codec.

7.Have a strategy for achieving RF

and

8.Be clear if WG is creating new technology or selecting existing technology

seemed somewhat related to me. If I followed the early charter 
discussions for codec correctly, there was an oscillation between 
wanting "better than other codecs, so we need to define new technology" 
and "pretty much guaranteed to be unencumbered, so we need to select old 
technology where patents have expired".

If I'm understanding what other people are saying, I'm not hearing "need 
old technology where patents have expired" as a strategy for video 
codecs, but It might be helpful for the proposed working group community 
to be clear on how much new technology people are planning to define, 
early on.

I hope this is helpful. And again, I think the important thing is that 
the proposed working group community take Cullen's suggestions 
seriously, more than what the community decides about them.

Have a great week.

Spencer