Subject: Re: [codec] Jan Skoglund's Opus test results (NB/WB/FB)

From: Anisse Taleb <[email-masked]>
Date: Fri, 01 Apr 2011 07:17:53 +0000

Dear Gregory,

For some reason I couldn't access  https://people.xiph.org/~greg/Opus2011-Jan/OpusTest2011.xls

Is the link correct ? or is something to fix from my side.

Kind regards,
/Anisse

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Gregory Maxwell
> Sent: Thursday, March 31, 2011 6:08 PM
> To: [email-masked]
> Subject: [codec] Jan Skoglund's Opus test results (NB/WB/FB)
> 
> Jan Skoglund  presented listening test results in the WG meeting today:
> 
> http://tools.ietf.org/agenda/80/slides/codec-4.pdf   starting at slide 15.
> 
> I did statistical tests on the results, which are available at:
> 
> http://people.xiph.org/~tterribe/notes/celt/nb-overall-ptests.png
> http://people.xiph.org/~tterribe/notes/celt/wb-overall-ptests.png
> http://people.xiph.org/~tterribe/notes/celt/music-overall-ptests.png
> 
> These are the results of  randomized blocked permutation test with N=100000
> and correction for multiple comparisons.
> Results with a p-value 0.05 of lower are considered significant- and are
> colored red or green based worse/better when read across the rows.
> 
> The complete data from the testing is available at:
> 
> https://people.xiph.org/~greg/Opus2011-Jan/OpusTest2011.xls
> (I'm posting it because I had the data from the statistical tests and easy
> web-hosting available)
> 
> It's pretty self-explanatory- listeners on columns, samples on rows, totals
> driving graphs.
> 
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: Stephen Botzko <[email-masked]>
Date: Fri, 1 Apr 2011 09:19:31 +0200

It is working for me, I will email you a copy separately.

BR
Stephen

On Fri, Apr 1, 2011 at 9:17 AM, Anisse Taleb <[email-masked]>wrote:

> Dear Gregory,
>
> For some reason I couldn't access

---

From: Roman Shpount <[email-masked]>
Date: Fri, 1 Apr 2011 13:01:07 -0400

This might be obvious and I might be just missing something, but one of the
things that I could not see from the presentation was if the audio samples
were normalize for the playback volume. What I run into before, was that
when you filter audio for lower frequencies, you loose the energy in the
higher frequencies and the audio file sounds less loud. When you run the
comparative tests, listeners prefer audio files that are louder skewing the
test. This also can apply to some of the codecs.
_____________
Roman Shpount

On Thu, Mar 31, 2011 at 12:08 PM, Gregory Maxwell <[email-masked]>wrote:

> Results with a p-value 0.05 of lower are considered significant— and are
> It's pretty self-explanatory— listeners on columns, samples on rows, totals
> driving graphs.
>

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 1 Apr 2011 20:54:26 +0200

Absolutely.  Audio level plays a big part in the subjective judgment, both
in presentation to the listeners (by the way, which results used headphones,
loudspeakers, tin-cans.) and input to the codecs. Another reason for having
a detailed test plan.

.Paul

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Roman Shpount
Sent: Friday, April 01, 2011 7:01 PM
To: Gregory Maxwell
Cc: [email-masked]
Subject: Re: [codec] Jan Skoglund's Opus test results (NB/WB/FB)

On Thu, Mar 31, 2011 at 12:08 PM, Gregory Maxwell <[email-masked]>
wrote:

http://people.xiph.org/~tterribe/notes/celt/nb-overall-ptests.png
<http://people.xiph.org/%7Etterribe/notes/celt/nb-overall-ptests.png> 
http://people.xiph.org/~tterribe/notes/celt/wb-overall-ptests.png
<http://people.xiph.org/%7Etterribe/notes/celt/wb-overall-ptests.png> 
http://people.xiph.org/~tterribe/notes/celt/music-overall-ptests.png
<http://people.xiph.org/%7Etterribe/notes/celt/music-overall-ptests.png> 

https://people.xiph.org/~greg/Opus2011-Jan/OpusTest2011.xls
<https://people.xiph.org/%7Egreg/Opus2011-Jan/OpusTest2011.xls> 
(I'm posting it because I had the data from the statistical tests and easy
web-hosting available)

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 01 Apr 2011 15:21:44 -0400

On 04/01/2011 01:01 PM, Roman Shpount wrote:
> This might be obvious and I might be just missing something, but one of
> the things that I could not see from the presentation was if the audio
> samples were normalize for the playback volume.

I can tell you that the Hydrogen Audio test normalized all version of each
sample to equal volume as computed by RMS amplitude.  Whether this is
"right" is a matter of opinion too, as are a dozen other attributes of
this or any comparison.  No bias is visible in the results, so it seems to
have been good enough for this purpose.

I don't know what the other tests did ... and I'm not at all worried about it.

The proper test design depends on the scenario being simulated.  No test
design can predict results in all use cases.  Audio quality is ultimately
an opinion poll, not high precision physics.  In the case of volume
normalization, different listeners have different frequency perception
profiles, so there is no normalization that is exactly right for all
listeners.

--Ben

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 01 Apr 2011 15:33:01 -0400

On 04/01/2011 02:54 PM, Paul Coverdale wrote:
> by the way, which results used
> headphones, loudspeakers, tin-cans…

I can't speak to the other tests, but the Hydrogen Audio test was
performed by listeners on their own hardware, under whatever conditions
they preferred.  The listeners are largely audio-quality and music
aficionados, listening with the same equipment they use for listening to
recorded music, which is typically good quality headphones.  I think the
results are likely a good representation of how the tested codecs would
fare when used by discriminating listeners for listening to music in the home.

In general I prefer these sorts of _in situ_ tests, especially Skype's
large-scale tests of SILK.  Skype arranged for testers to make actual
phone calls using randomly chosen codecs, on whatever hardware they used
to run Skype, and measured performance by phone call duration.  The only
way to know how a codec will fare for your application is to test it in
your application, and measure exactly what you care about.

--Ben