Subject: [codec] Audio tests: Further steps

From: "Christian Hoene" <[email-masked]>
Date: Tue, 23 Apr 2013 09:50:16 +0200

Hi,

currently, the codec comparison tests are running. Because of the request of
many codec developers, we plan to extend those tests: We might add audio
tests in which the content is varied to a large extend. For that, we need
sample that cannot be compressed well by Opus or AAC-eLD. For me, it is easy
to get those difficult samples for Opus. It is much challenging to get those
for AAC-eLD. Thus, if somebody had to time to study the weaknesses of
AAC-eLD, please forward me the samples.  I cannot start fair tests if I do
not have challenging samples for both codecs.

The Opus vs. Opus will also start soon. Please send comments to Alfons so
that he can considered them.

In addition, I am happy to announce that the listening tests will be
scientifically followed by the Qualinet group of European researchs (
<http://www.qualinet.eu/> http://www.qualinet.eu/). In addition to the lab
tests, we will conduct crowd sourcing listening tests. Also Qualinet
researchers will to the statistical analysis of both tests.

Thanks a lot to the WG2 of Qualinet team!

With best regards,

Christian Hoene

Symonics GmbH

Sand 13

72076 Tübingen

Tel +49 7071 5681302

Fax +49 7071 5681309

Email:  <mailto:[email-masked]> [email-masked]

Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene, Patrick
Schreiner

Sitz der Gesellschaft/Place of Business: Tübingen

Registereintrag/Commercial Register: Amtsgericht Stuttgart, HRB 739918

---

From: Ron <[email-masked]>
Date: Wed, 24 Apr 2013 05:01:00 +0930

On Tue, Apr 23, 2013 at 09:50:16AM +0200, Christian Hoene wrote:
> AAC-eLD, please forward me the samples.

Uhm, so ...  while I'm certain that the codec developers will be delighted
if you can point out any new killer samples that they aren't yet aware of
(since significant work has already been made to improve the encoder for
the known ones, and that work is still ongoing) -- I'm also pretty certain
that going out of your way to deliberately select such samples immediately
disqualifies this from being characterised as a "comparison test", or at
least claiming that it's even remotely representative of what people will
observe over a general corpus of their own audio, given the degree to which
such samples really are outliers.

> I cannot start fair tests if I do not have challenging samples for both
> codecs.

While such a test might have some novelty value to show "here are some
non-exhaustive results for the worst samples that we could find in a few
days of searching", I'm pretty sure words like "fair" and "scientific
rigour" don't really belong in the same sentence.  Not in the least when
you also say "we have the established list for one codec, but the known
killers for the other is at present entirely unknown to us".

If you want to spend your time doing that, that's fine, and the results
may well be 'interesting'.  But mischaracterising them as a "comparison"
test would just be somewhere on the spectrum from "mildly amusing" to
"a sad day for Modern Science".

It's your reputation though, and I can't tell you how to spend it.
But you might want to think this through a little better if you are
going to paint this with the brush of Being Science.

This isn't the cosmetics industry, other people can measure these things
too, and will continue to for some time to come.

 Cheers,
 Ron

---

From: Paul Coverdale <[email-masked]>
Date: Tue, 23 Apr 2013 17:34:55 -0400

I don't know why you're pouring scorn on this exercise, Ron. It seems to me
that it is a bona-fide attempt to understand the strengths and weaknesses of
the Opus codec in a controlled, unbiased manner, what a characterisation
test should do. It should have been done as part of the IETF codec WG
activity, but better late than never.

Cheers,

...Paul

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Ron
>Sent: Tuesday, April 23, 2013 3:31 PM
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: [codec] Audio tests: Further steps
>
>> currently, the codec comparison tests are running. Because of the
>> request of many codec developers, we plan to extend those tests: We
>> might add audio tests in which the content is varied to a large
>> extend. For that, we need sample that cannot be compressed well by
>> Opus or AAC-eLD. For me, it is easy to get those difficult samples for
>> Opus. It is much challenging to get those for AAC-eLD. Thus, if
>> somebody had to time to study the weaknesses of AAC-eLD, please
>forward me the samples.
>
>Uhm, so ...  while I'm certain that the codec developers will be
>delighted if you can point out any new killer samples that they aren't
>yet aware of (since significant work has already been made to improve
>the encoder for the known ones, and that work is still ongoing) -- I'm
>also pretty certain that going out of your way to deliberately select
>such samples immediately disqualifies this from being characterised as a
>"comparison test", or at least claiming that it's even remotely
>representative of what people will observe over a general corpus of
>their own audio, given the degree to which such samples really are
>outliers.
>
>> I cannot start fair tests if I do not have challenging samples for
>> both codecs.
>
>test would just be somewhere on the spectrum from "mildly amusing" to "a
>sad day for Modern Science".
>
>_______________________________________________
>codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 23 Apr 2013 17:46:55 -0400

On 04/23/2013 05:34 PM, Paul Coverdale wrote:
> I don't know why you're pouring scorn on this exercise, Ron. It
> seems to me that it is a bona-fide attempt to understand the
> strengths and weaknesses of the Opus codec in a controlled,
> unbiased manner, what a characterisation test should do. It should
> have been done as part of the IETF codec WG activity, but better
> late than never.

It's indeed a way to see what the strengths and weaknesses of a codec
are. I think what Ron mostly meant is that any *average* you compute
on such test would not be representative of which codec is better than
the other. Essentially, carefully picking out-liars is the worst form
of sampling you can have. It's useful for developers (knowing what to
focus on assuming you don't already know), but not for making general
quality conclusions.

	Jean-Marc

> Cheers,
> 
> ...Paul
> 
>> -----Original Message----- From: [email-masked]
>> [mailto:[email-masked]] On Behalf Of Ron Sent: Tuesday,
>> April 23, 2013 3:31 PM To: [email-masked] Cc:
>> [email-masked] Subject: Re: [codec] Audio tests:
>> Further steps
>> 
>>> currently, the codec comparison tests are running. Because of
>>> the request of many codec developers, we plan to extend those
>>> tests: We might add audio tests in which the content is varied
>>> to a large extend. For that, we need sample that cannot be
>>> compressed well by Opus or AAC-eLD. For me, it is easy to get
>>> those difficult samples for Opus. It is much challenging to get
>>> those for AAC-eLD. Thus, if somebody had to time to study the
>>> weaknesses of AAC-eLD, please
>> forward me the samples.
>> 
>> delighted if you can point out any new killer samples that they
>> aren't yet aware of (since significant work has already been made
>> to improve the encoder for the known ones, and that work is still
>> ongoing) -- I'm also pretty certain that going out of your way to
>> deliberately select such samples immediately disqualifies this
>> from being characterised as a "comparison test", or at least
>> claiming that it's even remotely representative of what people
>> will observe over a general corpus of their own audio, given the
>> degree to which such samples really are outliers.
>> 
>>> I cannot start fair tests if I do not have challenging samples
>>> for both codecs.
>> 
>> While such a test might have some novelty value to show "here are
>> some non-exhaustive results for the worst samples that we could
>> find in a few days of searching", I'm pretty sure words like
>> "fair" and "scientific rigour" don't really belong in the same
>> sentence.  Not in the least when you also say "we have the
>> established list for one codec, but the known killers for the
>> other is at present entirely unknown to us".
>> 
>> If you want to spend your time doing that, that's fine, and the
>> results may well be 'interesting'.  But mischaracterising them as
>> a "comparison" test would just be somewhere on the spectrum from
>> "mildly amusing" to "a sad day for Modern Science".
>> 
>> It's your reputation though, and I can't tell you how to spend
>> it. But you might want to think this through a little better if
>> you are going to paint this with the brush of Being Science.
>> 
>> This isn't the cosmetics industry, other people can measure these
>> things too, and will continue to for some time to come.
>> 
>> Cheers, Ron
>> 
>> 
>> _______________________________________________ codec mailing
>> list [email-masked] https://www.ietf.org/mailman/listinfo/codec
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 23 Apr 2013 14:59:59 -0700

> activity, but better late than never.

With my chair hat on:

Last I checked, this was still a WG, and we still have a milestone to 
publish test results (though we are admittedly late!). As we depend on 
volunteers to conduct those tests, I think we all share the desire for 
any testing which does get done to be as useful as possible. Let's try 
to focus on that goal.

---

From: Paul Coverdale <[email-masked]>
Date: Tue, 23 Apr 2013 19:12:51 -0400

Well, clearly any conclusions about the significance of outlier input
samples on codec performance must be tempered by the likelihood of these
outlier samples occurring in real life, and simple averaging may not be a
useful metric. I'm assuming that this will be taken into account in the
Qualinet statistical analysis mentioned by Christian.

....Paul

>From: Jean-Marc Valin [mailto:[email-masked]]
>Sent: Tuesday, April 23, 2013 5:47 PM
>To: Paul Coverdale
>Cc: [email-masked]; [email-masked]
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>> I don't know why you're pouring scorn on this exercise, Ron. It seems
>> to me that it is a bona-fide attempt to understand the strengths and
>> weaknesses of the Opus codec in a controlled, unbiased manner, what a
>> characterisation test should do. It should have been done as part of
>> the IETF codec WG activity, but better late than never.
>
>are. I think what Ron mostly meant is that any *average* you compute on
>such test would not be representative of which codec is better than the
>other. Essentially, carefully picking out-liars is the worst form of
>sampling you can have. It's useful for developers (knowing what to focus
>on assuming you don't already know), but not for making general quality
>conclusions.
>
>	Jean-Marc

---

From: Ron <[email-masked]>
Date: Wed, 24 Apr 2013 13:55:09 +0930

On Tue, Apr 23, 2013 at 05:34:55PM -0400, Paul Coverdale wrote:
> test should do.

As I said already, I think we can probably find _something_ interesting in
just about any test that someone wants to run, and personally, given that
the encoder is still undergoing active improvement, I suspect that any
test which does go out of its way to find genuine current weaknesses of
Opus for some edge cases will possibly be the most useful tests of all for
suggesting areas where immediate improvement may be found.

Since openly finding and fixing those things really has been one of the
greatest strengths of this process, and the results compared to a "closed
development, final shootout" methodology have already clearly spoken for
themselves convincingly.  So there's no reason to stop doing that now.
That's fine.  And Good.

Finding nowhere that it might improve would be far less helpful, and would
likewise only show that the given test was blind to already known realities.

But I also think it's very important for any statistical test to be very
clear on what hypothesis it is trying to find relevance for if people are
going to try to attach conclusions to it and claim it to be Science.

Starting with hand-picked samples, deliberately biased to be the hardest
known *for a particular coding method*, pretty much instantly excludes
"fair comparison" from being a meaningful hypothesis.  Putting a lump of
lead on the corner of one die and concluding "hey this one rolls all 6's"
isn't science, it's applied math, or a carny game.  At best.

That doesn't mean you can't make _any_ hypothesis about what the results,
may be testing.  But it's important to clearly make that in advance, if
you want to genuinely analyse the obvious flaws in the proposed method
before you start looking for evidence of pixies in the ink blots.

What does this test aim to demonstrate if it's obviously not a "fair
comparison" as it is currently proposed?  How do you plan to calibrate
and normalise the relative "killerness" of the selected biased samples
in a way that would make any conclusion beyond "Hard sample is Hard"
be even remotely meaningful as a "comparison", unfair or otherwise?

Knowing new hard samples would be great.  Comparing apples and oranges
less usefully so.  The oranges always win.

  Rigorously peer reviewed,
  Ron

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 24 Apr 2013 09:25:50 +0200

We have about 11 experiments:

* Some of those are for the (uninformed) general public (the first part, 6
experiments), 
* Some of those are the guys who want to make Opus work well on the Internet
(the second part, 5 experiments),
* Some of those are for the codec developers only (the difficult item test,
1 experiment).

As for the last one, it is a tit for tat game: "Show me where my codec is
bad and I show you where your codec is bad."  If the codec developers want
to play it, fine with me. I am happy to watch the contest. But if we do not
get input from both sides, we won't start the listening tests on this
subject as we need both teams on the playground.

Christian