Subject: [video-codec] Lessons Learned from Audio Codec Group - Normative Text not Code

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Fri, 5 Oct 2012 02:01:08 +0000

The audio codec WG tried many things that were sort of new for the IETF and some of theses worked and some did not. Having co-chaired that WG, there are some things I would like to comment on if we were to do something like the codec WG again. Note that on almost all of theses, there is someone who can tell me "I told you so", and to all theses people, I owe you a beverage :-) 

The Normative spec should be Text not Code

Code works many places and has many advantages but the IETF process for reviewing does not work well for reviewing large blocks of code.  We should use the text in the draft, not code as the normative spec. It's great if some site happens to have some running code but that is not the spec. It will help find bugs in the spec and in the long run, running, interoperable code is the important thing. But the normative conceptual design should be in the text of the draft with enough detail that someone that was incredibly smart, had lots of time, and with access to all the normative references in the draft, could implement and interoperable version of the codec. 

Send comments - tell me I am right - tell me I am wrong - whatever - the grass is definitely greener on the other side. But lets sort this one out before the BOF.

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 4 Oct 2012 19:41:46 -0700

I find this line of reasoning hard to argue with.  At times, I've been driven crazy by developers asserting "the code is the spec".  Without a spec, it's hard to evaluate whether the code is doing the right thing, or to evaluate why two implementations don't interoperate.  At the same time, we also know that having reference implementations is very important in stimulating implementation and subsequent deployment (see RFC 5218 for examples).  

So having both a spec and code is important -- but for heavens sake, let's not conflate them!

> From: [email-masked]
> To: [email-masked]
> Date: Fri, 5 Oct 2012 02:01:08 +0000
> Subject: [video-codec] Lessons Learned from Audio Codec Group - Normative	Text not Code
> 
> 
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 04 Oct 2012 20:01:34 -0700

> So having both a spec and code is important -- but for heavens sake,

+1. There are a number of successful audio codec standards where "the 
code is the spec", but the video codec culture is very different. I 
think to be taken seriously, we will need a written spec, and that's the 
approach I personally prefer. I knew if I objected too strenuously back 
when the codec WG was getting underway that I would be volunteering to 
be the one writing that spec. I'm explicitly volunteering to do that 
work (with help, hopefully!) for this effort.

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 04 Oct 2012 23:23:36 -0400

I think "code as spec" worked reasonably well for Opus. That being said,
considering my experience with the IETF (e.g. when it comes to review as
Cullen pointed out) and the fact that video codecs tend to be defined
with normative text (which Tim volunteers to write!), I fully support
"text is the spec" for this video effort. Of course, I also support
having a good reference implementation to go with the text.

	Jean-Marc

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 5 Oct 2012 04:23:02 +0000

I also support normative text, covering bitstream syntax and decoder
operation, especially since there is a volunteer for editorship (which can
well be a full-time job).  I prefer a bit-exact decoder operation spec
over the performance based decoder spec we had in Opus.

For those not actively involved in video codec standardization elsewhere,
here a few data points on the ways a standard is being developed in the
ITU and in MPEG.  We do not necessarily have to slavishly follow those
ways, but I will note that they have developed over more than two decades
of video coding standards work...
A) final spec text is the single normative reference, and defines
bitstream syntax and bit-exact decoder operation.  Not the encoder, and no
wiggle-room for performance tunes of the decoder.  (The latter is
different wrt. Opus, where conformance is considered achieved when certain
performance hurdles are passed.)
B) There is a non-normative reference implementation of both encoder and
decoder, called a "test model".  The test model encoder is typically
optimized for coding performance (high quality) only, and not a particular
good starting point for a real-world implementation.  In other words, it
is a standardization tool only.  The test model decoder may or may not
have mechanisms for error control, and is also not optimized.  Both
encoder and decoder are available from the respective SDO as a
non-normative part of the standard.
This is different from Opus in the sense that the Opus implementation of
encoder and decoder try to be reasonably well optimized for real-world
use, at least on PC architectures.  Video codec test models, at least the
encoders, are typically several orders of magnitudes slower than required
for real-time operation (on the machines available at the time the
standard is being developed).  Quite often that translates into several
order of magnitudes slower than possible on a state-of-the-art machine.
C) The test model code is documented in a test model description text,
which focusses on non-normative stuff like encoder design choices and
decoder error handling.  Also published as a non-normative Annex.
D) Finally, there is a normative conformance spec consisting mainly of
bitstreams and reconstructed frames.  Very important for the hardware guys
and lawyers, not so much for anyone else except perhaps as a test tool.
The performance spec is typically published a few meeting cycles after the
standard itself, and only once the standard itself is frozen.
E) In the early phases of standardization, quite often the draft text
lacks somewhat behind certain aspects of the code.  For example, the code
necessarily has to deal with aspects of the high level syntax (above block
level), whereas, at the beginning, the focus of the development in the SDO
is often on the signal processing tools rather than header stuff.
F) As the draft standard matures, quite often certain aspects of the draft
standard documented in the text are not fully implemented in the software;
certainly not the encoder but sometimes also not in the decoder.  For
example, certain exotic coding tools may not make it into the encoder (and
the decoder code exercising this syntax may be relatively untested).  Even
without optional tools, it is unrealistic to assume that all allowed
syntax combinations will be exercised even by a well-designed software
package; and statistical analysis through artificial test vectors (like
done in Opus) is, in video coding, mush less likely to exercise all
options than it is in audio.

Stephan

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Oct 2012 00:46:06 -0400

> over the performance based decoder spec we had in Opus.

Well, considering that video predictors use a unity gain (i.e. they
don't just re-converge like audio predictors), then "bit-exact" and
"matches closely like Opus" are essentially equivalent. If you're not
bit exact for video, you're going to diverge badly after a while. So no
objection to stating "bit exact" explicitly.

> performance hurdles are passed.)

Generally agree, though just like Opus I'd advocate for leaving things
like packet loss concealment out of the normatively specified part.

> order of magnitudes slower than possible on a state-of-the-art machine.

Any particular issue with having a reasonably fast reference
implementation. Seems like it's generally a good thing, no? If only to
make sure the codec isn't too complex.

> options than it is in audio.

There may be exceptions (there were a handful for Opus), but I tend to
think that it's good to be able to test that tools are actually useful
before adding them to a standard. If anything, I suspect we might
sometimes be in a case where it's the text that haven't caught up with
the reference implementation, no?

	Jean-Marc

---

From: Mohammed Raad <[email-masked]>
Date: Fri, 5 Oct 2012 08:49:00 +0200

A couple of comments:
1. If this group gets started and gets into a situation where the reference
implementation is ahead of the text then we can safely assume that the
"royalty-free" requirement will not be met (if the code is the spec).
2. If this group accepts that the process being used by other SDOs for the
development of standards is appropriate for meeting its objectives, why not
conduct this work within those SDOs? I think we need to be explicit in
justifying why this work needs to be conducted within the IETF instead of
those other SDOs.

Mohammed

---

From: Gregory Maxwell <[email-masked]>
Date: Fri, 5 Oct 2012 14:36:08 +0000

Stephan Wenger [[email-masked]] wrote:
>I also support normative text,

Any opinions I may have on that point are overridden by the fact that at least one other person whom I know is good at that sort of thing is volunteering to work on it.

> I prefer a bit-exact decoder operation spec

For video there is no question in my mind for that. Even ignoring the norms and the conformance testing simplifications, the signal characteristics ('low' bit depth and low sampling frequency) in video mostly preclude auto-regressive prediction as we used in Opus and benefit significantly from unity-gain prediction (which requires bit accuracy).

> encoders, are typically several orders of magnitudes slower than required

I do not look favorably on very slow test model encoders: Their overall slowness can hide fundamental performance problems with real-world implications, diminishing their usefulness as a validation of the viability of the standard.  Very slow performance also complicates the development of the format: In Opus we saw tremendous development advantages from being able to run enormous quantities of material through the codec to automatically look for outlying coding performance. This requires a reasonably fast implementation.

But if test implementations are separate from the draft then I don't really see that it merits any real discussion. People can create whatever they find useful. Though I think if there are no fast implementations to prove the point more skepticism is required over the question of a fast implementation even being possible. 

> options than it is in audio.

I expect any decoder implementation I work on to have 100% branch-condition coverage testing, at least of the operative codec part, at a minimum; if not straight from vectors than at least through a test-harness. I think anything less would be irresponsible.  But again, if the code isn't the specification, I'm not sure its something to bother discussing.

I don't agree that it's acceptable to have decoder bitstream syntax specified which isn't tested/testable in any implementation. If someone isn't willing to even construct a contrived test vector to hit some bit of syntax it should probably be omitted.  Codec development is more science and engineering and less art. If something hasn't been run by anyone why should we believe that it works, much less that it's useful?

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 05 Oct 2012 07:44:26 -0700

Mohammed Raad wrote:
> A couple of comments:
> 2. If this group accepts that the process being used by other SDOs for
> the development of standards is appropriate for meeting its objectives,
> why not conduct this work within those SDOs? I think we need to be

There's a lot more to process than the "the code is the spec" question.

> explicit in justifying why this work needs to be conducted within the
> IETF instead of those other SDOs.

There are a number of justifications listed in the BoF request and charter.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 05 Oct 2012 08:18:38 -0700

> For video there is no question in my mind for that. Even ignoring the norms and the conformance testing simplifications, the signal characteristics ('low' bit depth and low sampling frequency) in v…

More to the point, we were using bit-exact decoder definitions in 
VP3/Theora back when MPEG 1, MPEG 2, and MPEG 4 part 2 were still using 
transforms specified only up to a set of tolerances with metrics 
designed to avoid drift (and macroblock refresh requirements in the 
encoder) [1]. I consider the latter approach to be moderately crazy, 
even without the rapid accumulation of error that would be produced in a 
single frame by intra prediction (which is why H.264 was forced to go 
the bit-exact route), but that is just my personal opinion.

> But if test implementations are separate from the draft then I don't really
> see that it merits any real discussion. People can create whatever they

Even if the code is non-normative, that doesn't mean we can't include it 
in the draft. Some in the IETF seemed to strongly support the idea of 
providing reference code in the RFC [2], even if there were objections 
to making it normative.

As for speed, this is a trade-off, like anything else. There are 
definite benefits to having a fast implementation, but this requires 
more work than it does for audio. For Opus, the entire codec can almost 
fit in L1 cache, and the reference includes no SIMD assembly. For video, 
cache considerations are extremely important (and complicate the code), 
and SIMD assembly is a basic necessity for adequate speed on modern 
general purpose CPUs. It's also time-consuming to develop, and has 
compiler/toolchain dependency issues we did not have for Opus. Time 
spent working on those things is time that isn't being spent improving 
basic coding performance, testing reliability, or a host of other 
things. Particular individuals may be much more effective at some of 
those things than others, and slow code may make the other things more 
difficult, but I don't believe that the best approach is to have no 
attempt at optimization, nor is it to invest all the effort needed to 
make a commercial-grade optimized implementation. Though we will do the 
latter, and release it as open-source (because that is what we do), I'm 
not sure that needs to be gating for standardization.

[1] We were, of course, not the first to use bit-exact integer 
transforms. That dates back to at _least_ W.K. Pratt, J. Kane, and H.C. 
Andrews, "Hadamard transform image coding," in Proc. IEEE 57(1):58--68, 
Jan 1969.

[2] 
https://www.ietf.org/ibin/c5i?mid=6&rid=49&gid=0&k1=933&k2=64944&tid=1349448928

---

From: John Koleszar <[email-masked]>
Date: Fri, 5 Oct 2012 08:49:13 -0700

> But if test implementations are separate from the draft then I don't really see that it merits any real discussion. People can create whatever they find useful. Though I think if there are no fast…

Agree. While I think doing a fully optimized implementation is
counterproductive to ongoing research and development, a fundamental
tenet of this codec is that it works well in interactive applications.
This means we need an implementation that at least runs in real time
to adequately evaluate it. Or at least, that makes it a whole lot
easier, since we can use existing applications and test frameworks, vs
creating something synthetic.

> I don't agree that it's acceptable to have decoder bitstream syntax specified which isn't tested/testable in any implementation. If someone isn't willing to even construct a contrived test vector t…

Taking this farther, if these "optional tools" are meant to be non-MTI
decoder features, that should be explicitly disallowed. Avoiding
fragmentation is critical. And if a feature is required, then it
absolutely requires a test implementation.

---

From: Jozsef Vass <[email-masked]>
Date: Mon, 8 Oct 2012 08:32:24 -0700

I completely agree that code should not be specifications. I also do
not like the base64 inclusion of the code into the rfc, it is half of
the document. If turns out to be a bug in the code, what do you do,
create an errata with change in base64 encoded code? If it is
necessary, would have preferred include code as for rfc 3951 (ilbc). I
am pretty sure that most people will simple get and look at the code
in the website.

Jozsef