Subject: [codec] Summary of test results

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 16 Jun 2011 14:52:54 -0400

Hi,

We've collected all the test results we could find for Opus (and pre-Opus) 
and how they relate to the requirements. We summarized it all in the 
following draft:
http://tools.ietf.org/html/draft-valin-codec-results-00

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 16 Jun 2011 17:07:46 -0400

Hi Christian,

On 11-06-16 04:21 PM, Christian Hoene wrote:
> Thank you very much for your efforts.  Allow me to play again the role of
> the critic. For the readers of Opus characterization document it would be
> more useful to have the measured results in table (concrete figures).

Note that this draft is not meant to include all the details of the tests. 
For that, you should refer to the original results documents that are cited 
for each test. All of them are available on the net except for the most 
recent test: "Voice Quality Characterization of IETF Opus Codec" that will 
be presented by Anssi Ramo (from Nokia) at Interspeech 2011. However, he 
was kind enough to provide me with a copy and gave me permission to 
redistribute it to working group members upon request.

> Also, in order to plan further listening-only tests, an overview of missing
> test cases is missing.
>
> Do you plan to provide anything in this respect in a later update of the
> document? How is going to coordinate future tests? Or do been already enough
> tests been made?

As stated in the conclusion, we believe that the combination of all tests 
demonstrates that Opus is better than the codecs it needs to beat. Feel 
free to conduct any test you deem useful if you disagree with parts of the 
conclusions.

> Do you plan to have an additional informal IETF RFC with the
> characterization results? This would make sense but it requires changes in
> the charter...

At this point, this draft is not meant to ever become an RFC. It is only 
meant for discussion (friendlier than a lengthy post on the mailing list).

Cheers,

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 16 Jun 2011 22:21:58 +0200

Hi Jean-Marc,

 With best regards,

 Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 17 Jun 2011 00:53:19 +0200

Hi Jean-Marc,

> the most

Looks really good! Congratulations! And thanks for sharing

 Christian

---

From: Erik Norvell <[email-masked]>
Date: Tue, 21 Jun 2011 15:04:53 +0200

Hi Jean-Marc,

Thank you for compiling this summary of pre-Opus tests. It should definitely help in designing the listening test on the final Opus.

One comment to section 3:
"While Opus has evolved
   since these tests were conducted, the results should be considered as
   a _lower bound_ on the quality of the final codec."

I would like to think that the sum is always greater than it's parts, but it is definitely possible to make something worse by working on it. Hence, statements about Opus performance must be based on tests made on the final codec.

Best regards,
Erik 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Jean-Marc Valin
> Sent: den 16 juni 2011 20:53
> To: [email-masked]
> Subject: [codec] Summary of test results
> 
> Hi,
> 
> We've collected all the test results we could find for Opus 
> (and pre-Opus) and how they relate to the requirements. We 
> summarized it all in the following draft:
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 21 Jun 2011 15:40:16 -0400

On 11-06-21 09:04 AM, Erik Norvell wrote:
> Thank you for compiling this summary of pre-Opus tests. It should
> definitely help in designing the listening test on the final Opus.

Just to clarify, the Opus bit-stream *is* final and, as far as these
tests (for both speech and music) are concerned, has been since February.
The latest draft also has the final stereo bit-stream for voice, but all
the rest is long frozen.

> One comment to section 3: "While Opus has evolved since these tests were
> conducted, the results should be considered as a _lower bound_ on the
> quality of the final codec."
>
> I would like to think that the sum is always greater than it's parts,
> but it is definitely possible to make something worse by working on it.
> Hence, statements about Opus performance must be based on tests made on
> the final codec.

Of course it's not a guarantee, but there's definitely value in those tests
results in that it's unlikely that everything always worked fine and then 
we just screwed everything up at the end (if that was the case we would 
have realised it in the other tests).

Cheers,

	Jean-Marc

> Best regards, Erik
>
>> -----Original Message----- From: [email-masked]
>> [mailto:[email-masked]] On Behalf Of Jean-Marc Valin Sent: den
>> 16 juni 2011 20:53 To: [email-masked] Subject: [codec] Summary of test
>> results
>>
>> Hi,
>>
>> We've collected all the test results we could find for Opus (and
>> pre-Opus) and how they relate to the requirements. We summarized it
>> all in the following draft:
>> Jean-Marc _______________________________________________ codec
>> mailing list [email-masked]

---

From: Erik Norvell <[email-masked]>
Date: Wed, 22 Jun 2011 11:00:10 +0200

> From: Jean-Marc Valin [mailto:[email-masked]]
> Sent: den 21 juni 2011 21:40
> To: Erik Norvell
> Cc: [email-masked]
> Subject: Re: [codec] Summary of test results
> 
> The latest draft also has the final stereo bit-stream for voice, but 
> all the rest is long frozen.
> 

There are a number of tests which are older than that which are still referenced when making statements about Opus performance.
In addition, a frozen bit-stream is not equal to frozen quality. If the codec itself is still permitted to change its quality may be affected.

> > One comment to section 3: "While Opus has evolved since these tests 
> > were conducted, the results should be considered as a
> _lower bound_ on
> > the quality of the final codec."
> >
> > Hence, statements about Opus performance must be based on
> tests made
> > on the final codec.
> 
> Of course it's not a guarantee, but there's definitely value in those 
> tests results in that it's unlikely that everything always worked fine 
> and then we just screwed everything up at the end (if that was the 
> case we would have realised it in the other tests).
> 

I agree the tests are valuable as quality indicators for the codec by the time they were conducted. However, the only way to make correct statements about the performance of the final Opus codec is to test this final codec. To deduce that performance from tests of previous versions is bound to include some amount of speculation.

Best,
Erik

---

From: Koen Vos <[email-masked]>
Date: Wed, 22 Jun 2011 15:31:33 +0200 (CEST)

Hi Erik,

> However, the only way to make correct statements about the performance 
> of the final Opus codec is to test this final codec.

We took the MPEG approach to standardization (like with MP3/AAC), which means that the codec is defined by its bit-stream rather than its bit-exact behavior.  For this reason, the version tested from February is the final Opus.  Besides, the encoder hasn't actually changed in any meaningful way since February.

best,
koen.

----- Original Message -----
From: "Erik Norvell" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>
Cc: [email-masked]
Sent: Wednesday, June 22, 2011 2:00:10 AM
Subject: Re: [codec] Summary of test results

Best,
Erik

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 22 Jun 2011 06:43:40 -0700

I don't think that anything is final from an IETF's viewpoint until an RFC
is issued.  The WG could still decide to make changes in the bitstream
format if it were so inclined.  The IESG could require the WG to do so.
In the IETF, it's not the group of authors/contributors who can declare a
project finished, it's the community at large.

That said, I'm willing to accept the bitstream format to be considered
frozen for testing purposes.

Stephan

On 6.22.2011 06:31 , "Koen Vos" <[email-masked]> wrote:

>We took the MPEG approach to standardization (like with MP3/AAC), which
>means that the codec is defined by its bit-stream rather than its
>bit-exact behavior.  For this reason, the version tested from February is
>the final Opus.  Besides, the encoder hasn't actually changed in any
>meaningful way since February.
>
>best,
>koen.
>
>
>There are a number of tests which are older than that which are still
>referenced when making statements about Opus performance.
>In addition, a frozen bit-stream is not equal to frozen quality. If the
>codec itself is still permitted to change its quality may be affected.
>
>
>I agree the tests are valuable as quality indicators for the codec by the
>time they were conducted. However, the only way to make correct
>statements about the performance of the final Opus codec is to test this
>final codec. To deduce that performance from tests of previous versions
>is bound to include some amount of speculation.
>
>Best,
>Erik

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 22 Jun 2011 09:56:47 -0400

Hi Stephan,

> frozen for testing purposes.

This is indeed the definition of "final" and "frozen" that we have been 
using from the start. We just got tired of including the small print next 
to these words :-)

	Jean-Marc

> Stephan
>
>
> On 6.22.2011 06:31 , "Koen Vos"<[email-masked]>  wrote:
>
>> Hi Erik,
>>
>> From: "Erik Norvell"<[email-masked]>
>> To: "Jean-Marc Valin"<[email-masked]>
>> Cc: [email-masked]

---

From: "Christian Hoene" <[email-masked]>
Date: Sun, 26 Jun 2011 13:01:16 +0200

Hi,

Erik comments are valid. We should make some tests on the final codec
specification before WGLC. I mean, in the latest listening tests which used
Opus taken from GIT February 16th, 2011, I found the statement that "with
lower bitrates the codec crashes".  

I mean, just checking the implementation against major programming mistakes
is not a bad think. And this should not be made by the codec designers
themselves.

With best regards,

 Christian

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Koen Vos
> Sent: Wednesday, June 22, 2011 3:32 PM
> To: Erik Norvell
> Cc: [email-masked]
bit-exact
> behavior.  For this reason, the version tested from February is the final
Opus.
> Besides, the encoder hasn't actually changed in any meaningful way since
> February.
> 
> best,
> koen.
> 
> 
codec
> itself is still permitted to change its quality may be affected.
> 
> 
time
> they were conducted. However, the only way to make correct statements
> about the performance of the final Opus codec is to test this final codec.
To
> deduce that performance from tests of previous versions is bound to
include
> some amount of speculation.
> 
> Best,
> Erik

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 27 Jun 2011 10:35:18 -0400

Hi,

Indeed, the encoder version that Anssi tested was very prone to  
crashing when operating in ranges it didn't like or when fed options  
combinations it didn't like. That has (AFAICT) been fixed now.  
Fortunately, in the cases that did not crash, the behaviour is still  
the same now as it was for the test, so the results are still valid.  
That being said, we welcome feedback on any remaining code quality  
issues. For example, if anyone can still get the encoder or decoder to  
crash using the latest version from the *draft* (haven't uprated the  
website download yet), please let us know.

Cheers,

     Jean-Marc