Subject: Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

From: "codec issue tracker" <[email-masked]>
Date: Tue, 25 Jan 2011 00:45:11 -0000

#12: bit-exact vs. bit-compatible?

Changes (by gmaxwell@…):

  * status:  new => closed
  * resolution:  => worksforme

Comment:

 http://www.ietf.org/proceedings/78/minutes/codec.txt

 "On the topic of bit exact. Consensus was bit exactness is not required."

 I believe this issue is already closed.

-- 
------------------------------------+---------------------------------------
 Reporter:  hoene@…                 |        Owner:            
     Type:  enhancement             |       Status:  closed    
 Priority:  minor                   |    Milestone:            
Component:  requirements            |      Version:            
 Severity:  Active WG Document      |   Resolution:  worksforme
 Keywords:                          |  
------------------------------------+---------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
codec <http://tools.ietf.org/codec/>

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 24 Jan 2011 20:02:12 -0800

Hi all:

Let me speak once more against this decision (if such a decision were
really made; see the p.s.).

There are currently three IPR disclosures against the codec draft and/or
its predecessors.  The Xiph disclosure is at this point a placeholder
(Xiph folks: it's time to fix that!).  However, the two other disclosures
on file provide a patent grant only for necessary patent claims and only
when the standard is practiced in full compliance.  These terms (in
various formulations) are quite common.

In order to ensure one has a license (or can rely on a non-assert
covenant), one has to ensure one meets the conditions set by the
rightholder.  On stuff such as reciprocity clauses this is simple.  On
compliance, it's not always easy.

The traditional compliance test for a media codec is a stimulus-response
test: you feed test vectors into the codec, and you get results.  If the
results match, you are in compliance, if not, you are not.  Simple.

Without bit exactness, the compliance criteria have to be defined
differently.  We can do so, and, indeed, I recall that this has been
mentioned as one plan forward.  However, I have seen zero activity in this
direction, and I have also not seen any language that mentions this in the
requirements draft.  I think that the subject of compliance tests, at
least in its most basic outline, needs to be documented in the
requirements draft.  The details can be taken care of elsewhere and later,
but not too much later.  It should be clear that a codec candidate (if
there were more than one) needs to have compliance criteria defined before
that codec candidate can become an RFC.  Without that, the key goal of the
WG, a reasonably freely practicable codec, is just not achievable in the
current legal environment (which includes, in this case, the IPR
disclosures on file).

Of course, it would be sooooo much simpler if we would mandate a bit exact
decoder...  Is it really that restricting to require that?

Stephan

P.s.: for the IETF procedures newcomers: humms taken at meetings need to
be confirmed on a mailing list, and consensus needs to be declared by the
chairs.  On this subject, I do recall mailing list discussions after
Maastricht, but I do not recall that consensus was reached, yet alone
declared.  (Unfortunately, I currently don't have the time to go through
the mailing list archives to verify my recollection; Sorry.)

On 1.24.2011 16:45 , "codec issue tracker" <[email-masked]> wrote:

>Changes (by gmaxwell@Š):
> Reporter:  hoene@Š                 |        Owner:
>_______________________________________________
>codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 00:03:06 -0500

Hi Stephan,

I understand your concern and I'd be interested if you have alternative 
ways of handling the licensing to avoid any issue. It's not like this is 
a unique situation. As far as I know, most (all?) MPEG codecs have 
similar non-bit exact definitions. I have also heard that they also 
require some IPR licensing...

In general the issue of bit-exactness has been discussed and so far I 
don't recall many arguing in favor of a bit-exact definition. Most of 
the concerns that have been expressed are solved by considering that 
non-bitexact does not mean you cannot be bit-exact with the reference 
encoder. It merely means that you don't *have* to. So regardless of how 
conformance is defined exactly, one always has the option of being 
bit-exact with the reference implementation, which obviously guarantees 
compliance.

As for language mentioning compliance, I believe it belongs more to the 
guidelines (it's not a requirement of the codec itself), which includes 
the following text:

    4.  To reduce the risk of bias towards certain CPU/DSP architectures,
        ideally the decoder specification should not require "bit-exact"
        conformance with the reference implementation.  The output of a
        decoder implementation should only be "close enough" to the
        output of the reference decoder.  A comparison tool should be
        provided along with the codec to verify objectively that the
        output of a decoder is likely to be perceptually
        indistinguishable from that of the reference decoder.  However,
        an implementation may still wish to produce an output that is
        bit-exact with the reference implementation to simplify the
        testing procedure.

Cheers,

	Jean-Marc

On 11-01-24 11:02 PM, Stephan Wenger wrote:
> On 1.24.2011 16:45 , "codec issue tracker"<[email-masked]>  wrote:
>
>> Ticket URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>> codec<http://tools.ietf.org/codec/>
>>

---

From: Roman Shpount <[email-masked]>
Date: Tue, 25 Jan 2011 09:34:30 -0500

One more concern that I have related to bit exactness is codec regression
testing. One can produce a non-bit-exact encoder that produces reasonable
results for a reference decoder, but if paired with a modified,
non-bit-exact decoder will produce significant audio artifacts. Since
neither decoder or encoder are bit exact we will need to have a test
procedure that will validate that both encoder or decoder will not break any
standard compliant and non-bit-exact encoders or decoders. Not really sure
how this can be done.

I might be wrong, but I think MPEG decoders are bit exact and encoders are
not.
_____________
Roman Shpount

On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[email-masked]>wrote:

> ways of handling the licensing to avoid any issue. It's not like this is a
> unique situation. As far as I know, most (all?) MPEG codecs have similar
> non-bit exact definitions. I have also heard that they also require some IPR
> licensing...
>
> In general the issue of bit-exactness has been discussed and so far I don't
> recall many arguing in favor of a bit-exact definition. Most of the concerns
> that have been expressed are solved by considering that non-bitexact does
> not mean you cannot be bit-exact with the reference encoder. It merely means
> that you don't *have* to. So regardless of how conformance is defined
> exactly, one always has the option of being bit-exact with the reference
> implementation, which obviously guarantees compliance.
>
> guidelines (it's not a requirement of the codec itself), which includes the
> following text:
>
>>> Ticket URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1
>>> >

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 25 Jan 2011 10:42:00 -0500

It seems to me that the reference encoder and decoder will be bit exact for
a given floating point format, correct?  So one could specify IEEE 754-2008
when bit exactness is needed in the reference (for instance regression
testing), but not require it for compliance.

Folks who are modifying the reference encoder or decoder algorithms are "on
their own" as far as audio quality is concerned.  Though I agree with
Stephan that we need to address compliance (when exactly does a ported or
otherwise modified encoder/decoder become non-compliant?).

Stephen Botzko

On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount <[email-masked]> wrote:

>> don't recall many arguing in favor of a bit-exact definition. Most of the
>> concerns that have been expressed are solved by considering that
>> conformance is defined exactly, one always has the option of being bit-exact
>> with the reference implementation, which obviously guarantees compliance.
>>
>>>> Ticket URL:<
>>>> http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 10:45:18 -0500

As far as I know, MPEG decoders are specified, but not bit exact. With a 
bit-exact specification, there simply could not be any floating-point 
implementation of an MP3 or AAC decoder.

As long as the decoder is correctly specified with a tolerance on the 
deviation compared to the reference decoder, I do not see how an encoder 
could work with one decoder and not with another. As an example of (very 
simple) tolerance, we could say that a decoder implementation has to be 
within 80 dB SNR of the reference implementation on some test vectors. In 
practice we can use slightly more perceptually relevant metrics.

	Jean-Marc

On 11-01-25 09:34 AM, Roman Shpount wrote:
> procedure that will validate that both encoder or decoder will not break
> any standard compliant and non-bit-exact encoders or decoders. Not really
> sure how this can be done.
>
> I might be wrong, but I think MPEG decoders are bit exact and encoders are not.
> _____________
> Roman Shpount
>
>
> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi Stephan,
>
>     ways of handling the licensing to avoid any issue. It's not like this
>     is a unique situation. As far as I know, most (all?) MPEG codecs have
>         On 1.24.2011 16:45 , "codec issue tracker"<[email-masked]
>         <mailto:[email-masked]>> wrote:
>
>             Changes (by gmaxwell@Å ):
>
>             * status: new => closed
>             Reporter: hoene@Å | Owner:
>             Ticket
>             URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>             [email-masked] <mailto:[email-masked]>
>         [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 10:57:25 -0500

Hi Stephen,

The reference implementation includes both floating-point and fixed-point, 
so I'm not sure it's a good idea to say that one is "better" than the 
other. As for IEEE 754-2008, as far as I know it is not a "bit-exact" 
standard either when it comes to some operations (e.g. transcendental 
function) and even for other operations, most compilers I know of are not 
strict IEEE-754 (at least by default) because of issues such as the x86 
extended precision operations.

Regarding MPEG bit-streams, I'm not sure where to get confirmation, but 
this is what Wikipedia has to say about MPEG-1 Layer 3:

"Decoding, on the other hand, is carefully defined in the standard. Most 
decoders are "bitstream compliant", which means that the decompressed 
output – that they produce from a given MP3 file – will be the same, within 
a specified degree of rounding tolerance, as the output specified 
mathematically in the ISO/IEC standard document (ISO/IEC 11172-3). 
Therefore, comparison of decoders is usually based on how computationally 
efficient they are (i.e., how much memory or CPU time they use in the 
decoding process)."

That implies a standard based on "infinite precision", with a degree of 
tolerance for different implementations.

Cheers,

	Jean-Marc

On 11-01-25 10:42 AM, Stephen Botzko wrote:
> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     One more concern that I have related to bit exactness is codec
>     regression testing. One can produce a non-bit-exact encoder that
>     produces reasonable results for a reference decoder, but if paired with
>     a modified, non-bit-exact decoder will produce significant audio
>     artifacts. Since neither decoder or encoder are bit exact we will need
>     to have a test procedure that will validate that both encoder or
>     decoder will not break any standard compliant and non-bit-exact
>     encoders or decoders. Not really sure how this can be done.
>
>         I understand your concern and I'd be interested if you have
>         alternative ways of handling the licensing to avoid any issue. It's
>         not like this is a unique situation. As far as I know, most (all?)
>         MPEG codecs have similar non-bit exact definitions. I have also
>         heard that they also require some IPR licensing...
>
>         In general the issue of bit-exactness has been discussed and so far
>         I don't recall many arguing in favor of a bit-exact definition.
>         Most of the concerns that have been expressed are solved by
>         considering that non-bitexact does not mean you cannot be bit-exact
>         with the reference encoder. It merely means that you don't *have*
>         to. So regardless of how conformance is defined exactly, one always
>         has the option of being bit-exact with the reference
>         As for language mentioning compliance, I believe it belongs more to
>         the guidelines (it's not a requirement of the codec itself), which
>         includes the following text:
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 11:29:25 -0500

Just to give an idea of what I'm talking about in terms of compliance test, 
here's a tool I just wrote that could be used for testing. There may need 
to be a bit of refining and playing with the thresholds, but the main idea 
is there:

http://jmvalin.ca/misc_stuff/compare.m

It works with Octave, and is likely very easy to get to work with Matlab. 
Any comments?

	Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Tue, 25 Jan 2011 21:37:24 +0100 (CET)

A similar tool is in the code in silk\test\signalCompare.c.  It's based on LPC, and measures the SNR in a partially-whitened domain.  For music the critical-band approach of Jean-Marc's tool may work better, not sure.

One question is about what implementation should be the reference for compliance testing: floating-point or fixed-point?

A fixed-point reference allows for bit-exact implementations, which is nice.  And fixed-point reference implementations are the norm in speech coding.
However, CELT's floating-point implementation has a larger dynamic range than the fixed-point version, and is thus arguable better.  A fixed-point reference could not test this extra dynamic range.

Any suggestions?
koen.

----- Original Message -----
From: "Jean-Marc Valin" <[email-masked]>
To: "Stephen Botzko" <[email-masked]>
Cc: [email-masked]
Sent: Tuesday, January 25, 2011 8:29:25 AM
Subject: Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 15:58:04 -0500

My preference actually goes with using both fixed-point and floating point. 
That way, implementers of floating-point decoder can compare with the float 
version and implementers of the fixed-point decoder can compare with the 
fixed-point version.

This is not unheard of either, if you look at the reference implementation 
of the 3GPP(2?) EVRC codec, the distribution includes two sets of output 
vectors. One is for a fixed-point implementation that uses 31-bit precision 
in the multiplications and the other one is for 32-bit precision. You just 
choose whatever is more efficient for your DSP and use the appropriate 
testvectors for testing.

	Jean-Marc

On 11-01-25 03:37 PM, Koen Vos wrote:
> From: "Jean-Marc Valin"<[email-masked]>
> To: "Stephen Botzko"<[email-masked]>
> Cc: [email-masked]
>>> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount<[email-masked]
>>> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin<[email-masked]
>>> [email-masked]<mailto:[email-masked]>
>>> [email-masked]<mailto:[email-masked]>
>>> [email-masked]<mailto:[email-masked]>
>>> [email-masked]<mailto:[email-masked]>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 25 Jan 2011 18:24:50 -0800

My opinion is that we should not require bit-exactness.  There are many examples of previous codec
standards that did not require bit-exactness, including both royalty-bearing codecs (such as MPEG)
and royalty-free codecs (such as our BroadVoice/BV16/BV32), so I don't see any real reason why we
can't follow that same route.

I can see at least two important advantages for not requiring bit-exactness:
(1) it allows future innovations to improve the performance of the codec without breaking the bit-
stream compatibility, and
(2) it allows the codec implementers the greatest flexibility in implementing the codecs in ways
that are most efficient for the particular hardware or processor architecture they have.
Having a bit-exact standard will kill both.

The main drawback of a non-bit-exact standard is that the verification of the conformance of codec
implementations is not as straightforward as with a bit-exact standard.  However, I don't think
that is a show stopper or even a difficult problem.  There are many possible ways to do software-
based objective measurements to verify that a candidate codec implementation is likely to sound
essentially indistinguishable from the output of the reference codec.  We had previously measured
the SNRs and PESQ scores of a large number of test files using the reference codec and candidate
codec and examined the statistical distribution of the differences in the scores.  Jean-Marc just
posted a nice little Matlab code that built upon psychoacoustic modeling, NMR, averages, outliers
checking, etc.  Koen also mentioned another tool.  Even if these tools and methodologies are not
perfect yet, with some additional tuning of their algorithms and parameters, it is really not that
difficult to come up with objective, software-based verification methods that will give you a high
level of confidence whether a candidate codec will sound at least as good as the reference codec
(or perhaps better if improvements are made).

I like Jean-Marc's idea of having both a floating-point reference codec and a fixed-point
reference codec. For people who don't want to run the risk of degrading audio quality through
their own fixed-point algorithm conversion, they can just implement the fixed-point reference
codec in a bit-exact manner.  For people who want to implement the codec in floating-point (e.g.
on computers), the easiest way is just to use the floating-point reference code as is.  In both
cases, the conformance verification will be quite easy.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jean-Marc Valin
Sent: Tuesday, January 25, 2011 12:58 PM
To: Koen Vos
Cc: [email-masked]; Stephen Botzko
Subject: Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

        Jean-Marc

On 11-01-25 03:37 PM, Koen Vos wrote:
>> output â€“ that they produce from a given MP3 file â€“ will be the same, within
>>> Changes (by gmaxwell@Ã… ):
>>>
>>> * status: new =>  closed
>>> Reporter: hoene@Ã… | Owner:

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 25 Jan 2011 19:59:58 -0800

So I see I'm outgunned on the bit-exact issue.  Fine.

But please add language in the requirement doc to the extent that there
MUST be a conformance spec for each candidate codec that allows to
automatically determine the conformance of a codec implementation.  Absent
this spec, the IPR statements received are meaningless.

My preference would be to have that conformance spec in the same document
as the codec, so to make citations in IPR statements easy and to avoid a
charter-update.  

Stephan

On 1.25.2011 18:24 , "Raymond (Juin-Hwey) Chen" <[email-masked]> wrote:

>> A similar tool is in the code in silk\test\signalCompare.c.  It's based
>>on LPC, and measures the SNR in a partially-whitened domain.  For music
>>the critical-band approach of Jean-Marc's tool may work better, not sure.
>>
>> One question is about what implementation should be the reference for
>>compliance testing: floating-point or fixed-point?
>>
>> A fixed-point reference allows for bit-exact implementations, which is
>>nice.  And fixed-point reference implementations are the norm in speech
>>coding.
>> However, CELT's floating-point implementation has a larger dynamic
>>range than the fixed-point version, and is thus arguable better.  A
>>fixed-point reference could not test this extra dynamic range.
>>
>> Any suggestions?
>> koen.
>>
>>
>>>>------------------------------------+----------------------------------
>>>>---
>>>> --
>>>>------------------------------------+----------------------------------
>>>>---
>>>> --
>>>>
>>>> Ticket

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 25 Jan 2011 23:56:17 -0500

> this spec, the IPR statements received are meaningless.

IIRC, the guidelines already say that we need a conformance test.

> as the codec, so to make citations in IPR statements easy and to avoid a

Agreed. We really want the conformance tool in the same document as the 
codec itself.

Cheers,

	Jean-Marc

> Stephan
>
>
> On 1.25.2011 18:24 , "Raymond (Juin-Hwey) Chen"<[email-masked]>  wrote:
>