Subject: Re: [codec] getting more listening-test volunteers

From: Anisse Taleb <[email-masked]>
Date: Tue, 27 Jul 2010 16:41:47 +0200

Thanks Koen and JM, for the prompt replies.

After a quick glance at the result, I noted some interesting (and alarming) behavior of the test subjects. 

1.       According to Mushra methodology, one item at least has to be scored at 100, since subjects must be aware that there is a hidden reference.

2.       It is a custom that a post-screening of the subject has to be performed. This is essentially done to eliminate “deaf” subjects (joke). A usual post-screening is to eliminate subjects that :

a.       Fail in identifying the anchors, i.e.  Hidden ref score > LP 7kHz > LP 3.5kHz

b.      Fail in scoring the Hidden reference above 90.

Here is what my “quick and dirty” excel sheet reveals (this has been quickly done, so please cross-check, my conclusions below are based on this):

(1)     Listener 2, 4 and 8 failed in having an item scored at 100.

(2a) Listener 1 fails at keeping the order in two items (item 6 and 8).

(2b)  Listener 1 and 4 fail at scoring the HR above 90.

Also, additional comments:

-          I believe Listener 2 and 6 scoring are extreme. Scoring the LP 3.5 anchor at zero is not something commonly seen in formal mushra tests. 

-          Listener 1 seem to obviously not like CELT at all, since it got a consistent score of zero, which is very peculiar…is this a software glitch ?

At least from my experience I would have eliminated listeners 1, 2, 4, 6, and 8, which does not leave much room for proper statistical analysis with the remaining number of listeners (only 3 left).

I believe, it is very important to maintain high standards when conducting listening tests and to examine listeners results carefully before deriving conclusions. That said, I understand that this is not a formal test, but I would urge you to make the necessary steps of post-screening and proper statistical analysis for future tests. I think this point was raised several times by ITU-T experts on this  mailing list.

Kind regards,

/Anisse

PS: I was planning on taking the test myself, but I am away from the office. 

From: Koen Vos [mailto:[email-masked]] 
Sent: den 27 juli 2010 15:25
To: Anisse Taleb
Cc: Jean-Marc Valin; Yaakov Stein; [email-masked]
Subject: Re: [codec] getting more listening-test volunteers

Attached are the voting results.

As we pointed out during the presentation, this test was not meant to be comprehensive or formal, but rather to provide evidence for the synergy between SILK and CELT in the hybrid codec.

One particular flaw in the test was that Jean-Marc and myself participated in it.  However, we found that our scores did not have a significant impact on the overall results.  (You're welcome to verify: our scores are under "listener3" and "listener6".)

best,
koen.

  _____  

From: "Anisse Taleb" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>, "Yaakov Stein" <[email-masked]>, [email-masked]
Sent: Tuesday, July 27, 2010 2:14:32 PM
Subject: Re: [codec] getting more listening-test volunteers

Thank you Jean Marc.

Is it possible to also upload the raw data results, i.e. scores per user and item to allow for extra statistical analysis?

/Anisse

From: [email-masked] [mailto:[email-masked]] On Behalf Of Jean-Marc Valin
Sent: den 27 juli 2010 12:11
To: Anisse Taleb; Yaakov Stein; [email-masked]
Subject: Re: [codec] getting more listening-test volunteers

Hi,

The results were actually presented yesterday in the codec session. They're on slide 8 of the codec prototype presentation: http://www.ietf.org/proceedings/78/slides/codec-0.pdf
All codecs were tested at 32 kb/s on speech by 8 listeners. All were being used in constant bit-rate (CBR) mode except for SILK-WB, which was using VBR.

Cheers,

     Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 5:59 AM
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Dear Jean Marc,

I understand from you email that you have already some results available?
And if yes, would it possible to share these results with the community ?

Kind regards,

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: den 27 juli 2010 10:10
To: Yaakov Stein; [email-masked]
Subject: Re: [codec] getting more listening-test volunteers

I actually think we should be alright for this time, since the only goal was
to have a rough idea of how well we're doing. We'll keep improving the codec
and bit and then do another listening test.

Cheers,

   Jean-Marc

-----Original Message-----
From: [email-masked] on behalf of Yaakov Stein
Sent: Tue 7/27/2010 2:56 AM
To: [email-masked]
Subject: [codec] getting more listening-test volunteers

Before offering money for listeners on Mechanical Turk,
why not try a request for volunteers on the IETF discussion list?

If it is explained that the aim is to standardize a "non-proprietary"
high-quality audio codec for general Internet use
(strongly implying that this would obviate the need to use codecs with
strong IPR attachments to large companies),
then I bet there would be quite a few volunteers.

If you want even more, then you could raffle off some social-event tickets
or even a grand prize of free admission to IETF79.

Y(J)S

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: "Jean-Marc Valin" <[email-masked]>
Date: Tue, 27 Jul 2010 19:14:14 -0400

Hi,

Just to put things into perspective, we decided to do these listening tests only 2 days before flying to Maastricht, which is why we didn't give listeners highly detailed instructions. That being said, I think the difference between the hybrid codec and the others is so large that it cannot be attributed to methodology. This is pretty much all the information we were interested in at this point because of the very early stage of development of the codec. Once we have the codec in better shape, we'll be doing more controlled tests and we'd be glad to see you getting involved with the methodology (prior to the test this time).

Cheers,

    Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 10:41 AM
To: 'Koen Vos'
Cc: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

2.       It is a custom that a post-screening of the subject has to be performed. This is essentially done to eliminate "deaf" subjects (joke). A usual post-screening is to eliminate subjects that :

Here is what my "quick and dirty" excel sheet reveals (this has been quickly done, so please cross-check, my conclusions below are based on this):

-          Listener 1 seem to obviously not like CELT at all, since it got a consistent score of zero, which is very peculiar.is this a software glitch ?

Kind regards,

/Anisse

best,
koen.

  _____  

Thank you Jean Marc.

/Anisse

Hi,

Cheers,

     Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 5:59 AM
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Dear Jean Marc,

Kind regards,

Cheers,

   Jean-Marc

-----Original Message-----
From: [email-masked] on behalf of Yaakov Stein
Sent: Tue 7/27/2010 2:56 AM
To: [email-masked]
Subject: [codec] getting more listening-test volunteers

Y(J)S

---

From: Michael Knappe <[email-masked]>
Date: Tue, 27 Jul 2010 18:42:22 -0700

Just a quick note to everyone... The meeting yesterday was very productive, we made it through a good number of requirements questions, and set a clear timeline at the end to get the guidelines pulled together over the next few weeks. On the mailing list, let’s start pulling together the overall test methodology of both widescale listening tests as Koen suggested as well as more controlled environment testing as we discussed yesterday, with the goal to get this info into the guidelines doc.

If you were one of the several folks at the meeting yesterday who indicated you’d be interested in being a ‘test house’ for the controlled listening experiments, let me know who you are and we’ll set up a meeting together.

Thanks everyone for participating yesterday!

Cheers,

Mike

On 7/27/10 4:14 PM, "Jean-Marc Valin" <[email-masked]> wrote:

Hi,

Cheers,

    Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 10:41 AM
To: 'Koen Vos'
Cc: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Kind regards,

/Anisse

best,
koen.

  _____

Thank you Jean Marc.

/Anisse

Hi,

Cheers,

     Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 5:59 AM
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Dear Jean Marc,

Kind regards,

Cheers,

   Jean-Marc

-----Original Message-----
From: [email-masked] on behalf of Yaakov Stein
Sent: Tue 7/27/2010 2:56 AM
To: [email-masked]
Subject: [codec] getting more listening-test volunteers

Y(J)S

---

From: Herve Taddei <[email-masked]>
Date: Thu, 29 Jul 2010 17:54:16 +0200

Hi Jean Marc,

A small question as I cant see that on the slides, I would be interested in
knowing what is the delay of the codec under test. Could you please tell me
that?

Best regards

Herve 

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: Wednesday, July 28, 2010 1:14 AM
To: Anisse Taleb; Koen Vos
Cc: [email-masked]
Subject: Re: [codec] getting more listening-test volunteers

Hi,

Just to put things into perspective, we decided to do these listening tests
only 2 days before flying to Maastricht, which is why we didn't give
listeners highly detailed instructions. That being said, I think the
difference between the hybrid codec and the others is so large that it
cannot be attributed to methodology. This is pretty much all the information
we were interested in at this point because of the very early stage of
development of the codec. Once we have the codec in better shape, we'll be
doing more controlled tests and we'd be glad to see you getting involved
with the methodology (prior to the test this time).

Cheers,

    Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 10:41 AM
To: 'Koen Vos'
Cc: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

After a quick glance at the result, I noted some interesting (and alarming)
behavior of the test subjects.

1.       According to Mushra methodology, one item at least has to be scored
at 100, since subjects must be aware that there is a hidden reference.

2.       It is a custom that a post-screening of the subject has to be
performed. This is essentially done to eliminate "deaf" subjects (joke). A
usual post-screening is to eliminate subjects that :

a.       Fail in identifying the anchors, i.e.  Hidden ref score > LP 7kHz >
LP 3.5kHz

Here is what my "quick and dirty" excel sheet reveals (this has been quickly
done, so please cross-check, my conclusions below are based on this):

-          I believe Listener 2 and 6 scoring are extreme. Scoring the LP
3.5 anchor at zero is not something commonly seen in formal mushra tests.

-          Listener 1 seem to obviously not like CELT at all, since it got a
consistent score of zero, which is very peculiar.is this a software glitch ?

At least from my experience I would have eliminated listeners 1, 2, 4, 6,
and 8, which does not leave much room for proper statistical analysis with
the remaining number of listeners (only 3 left).

I believe, it is very important to maintain high standards when conducting
listening tests and to examine listeners results carefully before deriving
conclusions. That said, I understand that this is not a formal test, but I
would urge you to make the necessary steps of post-screening and proper
statistical analysis for future tests. I think this point was raised several
times by ITU-T experts on this  mailing list.

Kind regards,

/Anisse

As we pointed out during the presentation, this test was not meant to be
comprehensive or formal, but rather to provide evidence for the synergy
between SILK and CELT in the hybrid codec.

One particular flaw in the test was that Jean-Marc and myself participated
in it.  However, we found that our scores did not have a significant impact
on the overall results.  (You're welcome to verify: our scores are under
"listener3" and "listener6".)

best,
koen.

  _____ 

From: "Anisse Taleb" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>, "Yaakov Stein"
<[email-masked]>, [email-masked]
Sent: Tuesday, July 27, 2010 2:14:32 PM
Subject: Re: [codec] getting more listening-test volunteers

Thank you Jean Marc.

Is it possible to also upload the raw data results, i.e. scores per user and
item to allow for extra statistical analysis?

/Anisse

Hi,

The results were actually presented yesterday in the codec session. They're
on slide 8 of the codec prototype presentation:
http://www.ietf.org/proceedings/78/slides/codec-0.pdf
All codecs were tested at 32 kb/s on speech by 8 listeners. All were being
used in constant bit-rate (CBR) mode except for SILK-WB, which was using
VBR.

Cheers,

     Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 5:59 AM
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Dear Jean Marc,

Kind regards,

Cheers,

   Jean-Marc

-----Original Message-----
From: [email-masked] on behalf of Yaakov Stein
Sent: Tue 7/27/2010 2:56 AM
To: [email-masked]
Subject: [codec] getting more listening-test volunteers

Y(J)S

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 29 Jul 2010 14:51:57 -0400

Hi Herve,

The codec described in the slides (and included in the listening test) has a
frame size of 20 ms, with a look-ahead of 5.2 ms, for a total algorithmic delay
of 25.2 ms. In general, the delay of the combined codec is adaptive and can
range from 5 ms (2.5+2.5) to 65.2 ms (60+5.2). Hope that answers your question.

Cheers,

   Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Fri, 30 Jul 2010 01:05:26 +0200

Hi JM,
Is there any synthesis delay at the decoder given that CELT (or at least the
CELT part of the hybrid codec) uses an MDCT and that an overlap-add would
need some buffering delay ?

Kind regards,

/Anisse

> Sent: den 29 juli 2010 20:52
> To: Herve Taddei
> Cc: [email-masked]; 'Anisse Taleb'

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 29 Jul 2010 20:11:29 -0400

Hi Anisse,

> need some buffering delay ?

The CELT total algorithmic delay is equal to the frame size, plus the 
amount of MDCT overlap. The frame size is adaptive (2.5 to 20 ms), but 
the overlap is always 2.5 ms. So the algorithmic delay is always 
frame_size+2.5. I tend to view that as a 2.5 ms look-ahead on the 
encoder side, but I guess you can also view it as having no encoder 
look-ahead and a 2.5 ms synthesis delay in the decoder.

If you would like to know more on CELT, I recommend the following:

[1] J.-M. Valin, T. B. Terriberry, C. Montgomery, G. Maxwell, A 
High-Quality Speech and Audio Codec With Less Than 10 ms delay, IEEE 
Transactions on Audio, Speech and Language Processing, Vol. 18, No. 1, 
pp. 58-67, 2010. http://jmvalin.ca/papers/celt_tasl.pdf
[2] J.-M. Valin, T. B. Terriberry, G. Maxwell, A Full-Bandwidth Audio 
Codec with Low Complexity and Very Low Delay, Proc. EUSIPCO, 2009. 
http://jmvalin.ca/papers/celt_eusipco2009.pdf
[3] The video of "CELT: A low-latency, high-quality audio codec", 
presented by Timothy Terriberry at Linux.Conf.Au. 
http://www.celt-codec.org/presentations/

Note that a few things have changed since those were written/presented 
(especially [1]), but the code CELT ideas remain.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 29 Jul 2010 18:53:59 -0700

Hi Jean-Marc,

Thanks for the information on the algorithmic delay of the hybrid codec.  
Congratulations to you and Koen for achieving such excellent audio quality!  
That's pretty impressive.  Do you have a rough estimate of the complexity of 
the hybrid codec at that 32 kb/s mode in terms of MIPS and memory footprint?

According to a Wikipedia article, G.719 has an algorithmic delay of 40 ms and 
a computational complexity of 18 floating-point MIPS.  

BV32 has an algorithmic delay of 5 ms and a computational complexity of 17 
fixed-point MIPS.  

How about the delay and complexity of the other two 32 kb/s codecs, CELT and 
SILKWB, shown in slide 8 of that presentation? 

We know all codecs have four fundamental attributes: bit-rate, quality, 
delay, and complexity.  The figure on slide 8 fixes the bit-rate of 5 codecs 
at 32 kb/s and compares their audio quality with three reference conditions.  
Therefore, it does not give the whole picture of the relative "merit" of each 
codec since it only considers the first two attributes above and ignores the 
delay and complexity.  The lower the delay and complexity, the more difficult 
it is to achieve good quality; there is a clear trade-off.

Also, since SILKWB is a variable-bit-rate (VBR) codec with an average bit-
rate of 32 kb/s.  That's equivalent to a constant-bit-rate (CBR) codec at a 
much higher bit-rate.  There is nothing wrong with VBR, it's just that this 
also needs to be taken into consideration when comparing different codecs on 
slide 8.

Do all 8 conditions on slide 8 use the 48 kHz sampled original signal as the 
reference?

Is it common for MUSHRA tests to compare two conditions with very different 
audio bandwidths? Comparing two different signals with very different audio
bandwidths makes it very easy to tell which signal is reference, so the test
is not "blind" any more.

I have listened to the speech files of each condition in this test. In my 
subjective listening opinion, although I don't necessarily agree with the 
absolute levels of the audio quality of each of these 8 conditions, I do 
generally agree with the relative rank ordering, with just one exception: I 
was very surprised by the relative rank ordering of CELT and BV32.  Does the 
relative order of these two codecs surprise you?  I listened back and forth 
many times and I am still scratching my head about that rank ordering.

Best Regards,

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jean-Marc Valin
Sent: Thursday, July 29, 2010 11:52 AM
To: Herve Taddei
Cc: [email-masked]; 'Anisse Taleb'
Subject: Re: [codec] getting more listening-test volunteers

Hi Herve,

Cheers,

   Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 29 Jul 2010 22:26:39 -0400

Hi Raymond,

> the hybrid codec at that 32 kb/s mode in terms of MIPS and memory footprint?

Not at this point. I'm not too familiar with the complexity of SILK and 
since this is very early in the development (the combined codec has only 
existed for a few weeks), this should improve anyway.

> a computational complexity of 18 floating-point MIPS.

To put things in perspective, last year I tried evaluating the 
complexity of CELT and obtained 17 WMOPS. The methodology was not 
perfect, so there's a margin of error there, but it should give an idea. 
As another data point, the unoptimized (no asm) C floating point code 
runs in real-time (encode+decode) with 0.3% CPU on a 3 GHz Core2 CPU. As 
for the algorithmic delay, it was 25.2 ms in the conditions of the 
listening test.

> SILKWB, shown in slide 8 of that presentation?

CELT was run with 22.5 ms delay and SILK had 25.2 ms delay. You can see 
the quality of CELT when running with much lower delay in these two papers:
http://jmvalin.ca/papers/celt_tasl.pdf
http://jmvalin.ca/papers/celt_eusipco2009.pdf

These are unfortunately a bit old though and the quality of CELT has 
improved a bit since that time.

> it is to achieve good quality; there is a clear trade-off.

Obviously, lowering the delay has implications on quality vs bit-rate. 
CELT is designed to scale from 2.5 ms frames to 20 ms frames (and SILK 
goes to 60 ms when we really want to save bit-rate). At the same time, 
because of the header overhead, it is not as important to maintain a low 
bit-rate at small sampling rates because for 5 ms frames, we are already 
sending 64 kb/s worth of IP/UDP/RTP headers (not to mention Ethernet).

> also needs to be taken into consideration when comparing different codecs on

Just note here that although SILK-WB was operating in VBR, the hybrid 
(even though it has a SILK layer) was using CBR.

> Do all 8 conditions on slide 8 use the 48 kHz sampled original signal as the

The full-band codecs are based on the 48 kHz original. The wideband 
codecs are based on the 7 kHz low-pass after down-sampling, as 
recommended for BV32 (AFAIK).

> bandwidths makes it very easy to tell which signal is reference, so the test

As far as I know, this is common to have material with different 
bandwidth with MUSHRA (not with MOS). The test actually mandates the use 
of a 3.5 kHz low-pass anchor and recommends the use of a 7 kHz LP anchor.

> generally agree with the relative rank ordering,

Agreed.

> with just one exception: I

Yes, it *did* surprise me too, the only explanation I can give is that 
some people really have a strong preference for full-band despite the 
artefacts. That being said, when I took the test myself, I'm pretty sure 
I rated CELT lower than BV32. CELT is simply not designed for running at 
32 kb/s and produces artefacts that I personally find objectionable at 
that rate. I generally recommend using CELT with rates around 48 kb/s 
(though it sounds OK at 40 kb/s) for speech and around 64 kb/s for 
music. The only reasons we ran it at 32 kb/s in this test was to 
demonstrate that the combination of SILK and CELT could produce results 
that were "better than the sum of each part" at the same rate.

Cheers,

	Jean-Marc

> Quoting Herve Taddei<[email-masked]>:
>
>> Hi Jean Marc,
>>
>>
>>
>> a.       Fail in identifying the anchors, i.e.  Hidden ref score>  LP 7kHz>
>> LP 3.5kHz
>>
>> From: "Anisse Taleb"<[email-masked]>
>> To: "Jean-Marc Valin"<[email-masked]>, "Yaakov Stein"

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 30 Jul 2010 10:35:00 -0700

Hi Jean-Marc,

Thank you very much for your detailed answers to my questions.  They are very useful.

Perhaps I wasn't clear enough so you misunderstood my question "Do all 8 conditions on slide 8 use the 48 kHz sampled original signal as the reference?"  I was not asking whether the 48 kHz original was used as the input signal to all codecs (we know we can't due to different sampling rates).  Instead, I was asking whether the 48 kHz original was always used as the reference condition in all the MUSHRA listening pairs.

Thanks.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Thursday, July 29, 2010 7:27 PM
To: Raymond (Juin-Hwey) Chen
Cc: Herve Taddei; [email-masked]; 'Anisse Taleb'
Subject: Re: [codec] getting more listening-test volunteers

Hi Raymond,

Agreed.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 30 Jul 2010 13:53:40 -0400

Raymond (Juin-Hwey) Chen wrote:
> Perhaps I wasn't clear enough so you misunderstood my question "Do all 8
> conditions on slide 8 use the 48 kHz sampled original signal as the
> reference?"  I was not asking whether the 48 kHz original was used as
> the input signal to all codecs (we know we can't due to different
> sampling rates).  Instead, I was asking whether the 48 kHz original was
> always used as the reference condition in all the MUSHRA listening
> pairs.

I had indeed misunderstood your question. I'm still not sure I understand 
it correctly, so let me know if this answers it. In MUSHRA tests, the 
codecs are not compared in pairs, but rather all codecs (along with the 
reference and anchors) are presented together. See this screenshot of a 
MUSHRA application that I wrote (similar to the one we used): 
http://rateit.sourceforge.net/RateIt.png

You can see 9 play buttons: One for the reference ("Ref") and 8 for the 
codecs under test ("A" to "H") presented in random order (different for 
each input sample). Among the randomly ordered audio is the "hidden 
reference" (which is 48 kHz just like the labeled reference), as well as 
the 3.5 kHz low-pass and 7 kHz low-pass anchors. The MUSHRA methodology is 
described in ITU-R BS.1534.

Hope this helps,

	Jean-Marc

> Thanks.
> 
> Raymond
> 
> -----Original Message----- From: Jean-Marc Valin
> [mailto:[email-masked]] Sent: Thursday, July 29, 2010
> 7:27 PM To: Raymond (Juin-Hwey) Chen Cc: Herve Taddei; [email-masked];
> 'Anisse Taleb' Subject: Re: [codec] getting more listening-test
> volunteers
> 
> Hi Raymond,
> 
>> Thanks for the information on the algorithmic delay of the hybrid
>> codec. Congratulations to you and Koen for achieving such excellent
>> audio quality! That's pretty impressive.  Do you have a rough estimate
>> of the complexity of the hybrid codec at that 32 kb/s mode in terms of
>> MIPS and memory footprint?
> 
>> According to a Wikipedia article, G.719 has an algorithmic delay of 40
>> ms and a computational complexity of 18 floating-point MIPS.
> 
>> How about the delay and complexity of the other two 32 kb/s codecs,
>> CELT and SILKWB, shown in slide 8 of that presentation?
> 
> the quality of CELT when running with much lower delay in these two
> papers: http://jmvalin.ca/papers/celt_tasl.pdf 
>> We know all codecs have four fundamental attributes: bit-rate,
>> quality, delay, and complexity.  The figure on slide 8 fixes the
>> bit-rate of 5 codecs at 32 kb/s and compares their audio quality with
>> three reference conditions. Therefore, it does not give the whole
>> picture of the relative "merit" of each codec since it only considers
>> the first two attributes above and ignores the delay and complexity.
>> The lower the delay and complexity, the more difficult it is to
>> achieve good quality; there is a clear trade-off.
> 
>  bit-rate at small sampling rates because for 5 ms frames, we are
> already sending 64 kb/s worth of IP/UDP/RTP headers (not to mention
> Ethernet).
> 
>> Also, since SILKWB is a variable-bit-rate (VBR) codec with an average
>> bit- rate of 32 kb/s.  That's equivalent to a constant-bit-rate (CBR)
>> codec at a much higher bit-rate.  There is nothing wrong with VBR,
>> it's just that this also needs to be taken into consideration when
>> comparing different codecs on slide 8.
> 
>> Do all 8 conditions on slide 8 use the 48 kHz sampled original signal
>> as the reference?
> 
>> Is it common for MUSHRA tests to compare two conditions with very
>> different audio bandwidths? Comparing two different signals with very
>> different audio bandwidths makes it very easy to tell which signal is
>> reference, so the test is not "blind" any more.
> 
>> I have listened to the speech files of each condition in this test. In
>> my subjective listening opinion, although I don't necessarily agree
>> with the absolute levels of the audio quality of each of these 8
>> conditions, I do generally agree with the relative rank ordering,
> 
> Agreed.
> 
>> with just one exception: I was very surprised by the relative rank
>> ordering of CELT and BV32.  Does the relative order of these two
>> codecs surprise you?  I listened back and forth many times and I am
>> still scratching my head about that rank ordering.
> 
>  I rated CELT lower than BV32. CELT is simply not designed for running
> at 32 kb/s and produces artefacts that I personally find objectionable
> at that rate. I generally recommend using CELT with rates around 48 kb/s
>> -----Original Message----- From: [email-masked]
>> [mailto:[email-masked]] On Behalf Of Jean-Marc Valin Sent:
>> Thursday, July 29, 2010 11:52 AM To: Herve Taddei Cc: [email-masked];
>> 'Anisse Taleb' Subject: Re: [codec] getting more listening-test
>> volunteers
>> 
>> Hi Herve,
>> 
>> The codec described in the slides (and included in the listening test)
>> has a frame size of 20 ms, with a look-ahead of 5.2 ms, for a total
>> algorithmic delay of 25.2 ms. In general, the delay of the combined
>> codec is adaptive and can range from 5 ms (2.5+2.5) to 65.2 ms
>> (60+5.2). Hope that answers your question.
>> 
>> Cheers,
>> 
>> Jean-Marc
>> 
>> 
>>> A small question as I cant see that on the slides, I would be
>>> interested in knowing what is the delay of the codec under test.
>>> Could you please tell me that?
>>> 
>>> 
>>> 
>>> Best regards
>>> 
>>> 
>>> 
>>> Herve
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _____
>>> 
>>> From: [email-masked] [mailto:[email-masked]] On
>>> Behalf Of Jean-Marc Valin Sent: Wednesday, July 28, 2010 1:14 AM To:
>>> Anisse Taleb; Koen Vos Cc: [email-masked] Subject: Re: [codec]
>>> getting more listening-test volunteers
>>> 
>>> 
>>> 
>>> Hi,
>>> 
>>> Just to put things into perspective, we decided to do these
>>> listening tests only 2 days before flying to Maastricht, which is
>>> why we didn't give listeners highly detailed instructions. That
>>> being said, I think the difference between the hybrid codec and the
>>> others is so large that it cannot be attributed to methodology. This
>>> is pretty much all the information we were interested in at this
>>> point because of the very early stage of development of the codec.
>>> Once we have the codec in better shape, we'll be doing more
>>> controlled tests and we'd be glad to see you getting involved with
>>> the methodology (prior to the test this time).
>>> 
>>> Cheers,
>>> 
>>> Jean-Marc
>>> 
>>> -----Original Message----- From: Anisse Taleb
>>> [mailto:[email-masked]] Sent: Tue 7/27/2010 10:41 AM To:
>>> 'Koen Vos' Cc: Jean-Marc Valin; 'Yaakov Stein'; [email-masked] 
>>> 1.       According to Mushra methodology, one item at least has to
>>> be scored at 100, since subjects must be aware that there is a
>>> hidden reference.
>>> 
>>> 2.       It is a custom that a post-screening of the subject has to
>>> be performed. This is essentially done to eliminate "deaf" subjects
>>> (joke). A usual post-screening is to eliminate subjects that :
>>> 
>>> a.       Fail in identifying the anchors, i.e.  Hidden ref score>
>>> LP 7kHz> LP 3.5kHz
>>> 
>>> Here is what my "quick and dirty" excel sheet reveals (this has been
>>> quickly done, so please cross-check, my conclusions below are based
>>> on this):
>>> 
>>> 
>>> 
>>> -          I believe Listener 2 and 6 scoring are extreme. Scoring
>>> the LP 3.5 anchor at zero is not something commonly seen in formal
>>> mushra tests.
>>> 
>>> -          Listener 1 seem to obviously not like CELT at all, since
>>> it got a consistent score of zero, which is very peculiar.is this a
>>> software glitch ?
>>> 
>>> 
>>> 
>>> At least from my experience I would have eliminated listeners 1, 2,
>>> 4, 6, and 8, which does not leave much room for proper statistical
>>> analysis with the remaining number of listeners (only 3 left).
>>> 
>>> 
>>> 
>>> I believe, it is very important to maintain high standards when
>>> conducting listening tests and to examine listeners results
>>> carefully before deriving conclusions. That said, I understand that
>>> this is not a formal test, but I would urge you to make the
>>> necessary steps of post-screening and proper statistical analysis
>>> for future tests. I think this point was raised several times by
>>> ITU-T experts on this  mailing list.
>>> 
>>> 
>>> 
>>> Kind regards,
>>> 
>>> /Anisse
>>> 
>>> From: Koen Vos [mailto:[email-masked]] Sent: den 27 juli 2010
>>> 15:25 To: Anisse Taleb Cc: Jean-Marc Valin; Yaakov Stein;
>>> [email-masked] Subject: Re: [codec] getting more listening-test
>>> volunteers
>>> 
>>> 
>>> 
>>> As we pointed out during the presentation, this test was not meant
>>> to be comprehensive or formal, but rather to provide evidence for
>>> the synergy between SILK and CELT in the hybrid codec.
>>> 
>>> One particular flaw in the test was that Jean-Marc and myself
>>> participated in it.  However, we found that our scores did not have
>>> a significant impact on the overall results.  (You're welcome to
>>> verify: our scores are under "listener3" and "listener6".)
>>> 
>>> best, koen.
>>> 
>>> 
>>> 
>>> _____
>>> 
>>> From: "Anisse Taleb"<[email-masked]> To: "Jean-Marc
>>> Valin"<[email-masked]>, "Yaakov Stein" 
>>> <[email-masked]>, [email-masked] Sent: Tuesday, July 27, 2010
>>> 2:14:32 PM Subject: Re: [codec] getting more listening-test
>>> volunteers
>>> 
>>> 
>>> 
>>> 
>>> Thank you Jean Marc.
>>> 
>>> Is it possible to also upload the raw data results, i.e. scores per
>>> user and item to allow for extra statistical analysis?
>>> 
>>> 
>>> 
>>> /Anisse
>>> 
>>> 
>>> 
>>> From: [email-masked] [mailto:[email-masked]] On
>>> Behalf Of Jean-Marc Valin Sent: den 27 juli 2010 12:11 To: Anisse
>>> Taleb; Yaakov Stein; [email-masked] Subject: Re: [codec] getting
>>> more listening-test volunteers
>>> 
>>> 
>>> 
>>> Hi,
>>> 
>>> The results were actually presented yesterday in the codec session.
>>> They're on slide 8 of the codec prototype presentation: 
>>> http://www.ietf.org/proceedings/78/slides/codec-0.pdf All codecs
>>> were tested at 32 kb/s on speech by 8 listeners. All were being used
>>> in constant bit-rate (CBR) mode except for SILK-WB, which was using 
>>> VBR.
>>> 
>>> Cheers,
>>> 
>>> Jean-Marc
>>> 
>>> -----Original Message----- From: Anisse Taleb
>>> [mailto:[email-masked]] Sent: Tue 7/27/2010 5:59 AM To:
>>> Jean-Marc Valin; 'Yaakov Stein'; [email-masked] Subject: RE: [codec]
>>> getting more listening-test volunteers
>>> 
>>> Dear Jean Marc,
>>> 
>>> I understand from you email that you have already some results
>>> available? And if yes, would it possible to share these results with
>>> the community ?
>>> 
>>> 
>>> 
>>> Kind regards,
>>> 
>>> 
>>> 
>>> 
>>> 
>>> From: [email-masked] [mailto:[email-masked]] On
>>> Behalf Of Jean-Marc Valin Sent: den 27 juli 2010 10:10 To: Yaakov
>>> Stein; [email-masked] Subject: Re: [codec] getting more
>>> listening-test volunteers
>>> 
>>> 
>>> 
>>> I actually think we should be alright for this time, since the only
>>> goal was to have a rough idea of how well we're doing. We'll keep
>>> improving the codec and bit and then do another listening test.
>>> 
>>> Cheers,
>>> 
>>> Jean-Marc
>>> 
>>> 
>>> -----Original Message----- From: [email-masked] on behalf of
>>> Yaakov Stein Sent: Tue 7/27/2010 2:56 AM To: [email-masked] Subject:
>>> [codec] getting more listening-test volunteers
>>> 
>>> Before offering money for listeners on Mechanical Turk, why not try
>>> a request for volunteers on the IETF discussion list?
>>> 
>>> If it is explained that the aim is to standardize a
>>> "non-proprietary" high-quality audio codec for general Internet use 
>>> (strongly implying that this would obviate the need to use codecs
>>> with strong IPR attachments to large companies), then I bet there
>>> would be quite a few volunteers.
>>> 
>>> If you want even more, then you could raffle off some social-event
>>> tickets or even a grand prize of free admission to IETF79.
>>> 
>>> Y(J)S
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________ codec mailing list 
>>> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> _______________________________________________ codec mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>> 
>> 
>> 
>> 
> 
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec

---

From: Gregory Maxwell <[email-masked]>
Date: Fri, 30 Jul 2010 11:03:50 -0700

> many times and I am still scratching my head about that rank ordering.

This is unsurprising to me. 

While I'm not currently aware of research which has evaluated this point In detail,  in the past I've conducted testing in order to determine a low-pass selection for a single codec — trading off band-pass in exchange for artifacts.  What I found is that normal unprepared listeners _aggressively_ preferred the wider bandwidth even over an intentionally handicapped alternative with significant artifacts.

In one amusing test I had been frustrated by how severely PEAQ was penalizing the low-pass version so I conducted a human listening test and found that it correlated very well with the PEAQ results, much to my chagrin. 

I believe this is because the low-pass is so obvious that it takes no thought to recognize it and declare the wider bandpass to be the winner.  

I found that if prior to the test the listeners are exposed to constructed examples highlighting common artifacts  (e.g. pre-echo, birdies) that they were more likely to produce ratings similar to the ones which I would produce. I believe that this is also the reason for some of the pre-test procedures in the BS.1534-1 (MUSHRA) document. 

However, I don't know if this constitutes a fair and equitable testing environment. Most users of any particular codec have not had "artifact training"— so the unprepared results might better reflect the real performance.   On the other hand, I could argue that the initial preference for wider bandwidth is an initial impression effect and that even the untrained listeners would eventually prefer the less artifacted version after extended use... but I'm really waving my hands there, I don't have any data to support that position.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 30 Jul 2010 12:20:18 -0700

Hi Jean-Marc,

Thanks for your explanation.  I was just asking whether the 48 kHz
original was used as the only reference for all 8 conditions on slide 8
(as opposed to, say, using "LP 7k" as the reference for SILKWB and BV32).

I admit that I have been too busy and haven't had a chance to do the
MUSHRA test you and Koen prepared, and I haven't read ITU-R BS.1534-1.
I thought I previously saw some audio coding papers where people did
listening tests by presenting the reference condition first, followed by
two conditions in random order, where one of them was the reference and
the other was the codec under test.  I thought this was what you did to
get the figure on slide 8.

The Wikipedia article on MUSHRA has a sentence "This is due to the fact
that all comparisons are paired (i.e. a paired t-test can be used)..."
That's also why I thought your MUSHRA test used paired comparisons.

Thanks.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Friday, July 30, 2010 10:54 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]; 'Anisse Taleb'
Subject: Re: [codec] getting more listening-test volunteers

Hope this helps,

        Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 30 Jul 2010 12:28:59 -0700

Hi Greg,

Thanks a lot for sharing your previous test experience and your thoughts 
and opinions on this subject.  What you shared is very interesting.

I would agree with you that usually the longer a person listens to lossy 
compression of audio by a particular codec, the easier it is for the person 
to detect and perceive the coding distortion produced by that codec.  
At least that's my own personal experience.

Raymond

-----Original Message-----
From: Gregory Maxwell [mailto:[email-masked]] 
Sent: Friday, July 30, 2010 11:04 AM
To: Raymond (Juin-Hwey) Chen; Jean-Marc Valin; Herve Taddei
Cc: [email-masked]; 'Anisse Taleb'
Subject: RE: [codec] getting more listening-test volunteers

While I'm not currently aware of research which has evaluated this point In detail,  in the past I've conducted testing in order to determine a low-pass selection for a single codec - trading off band-pass in exchange for artifacts.  What I found is that normal unprepared listeners _aggressively_ preferred the wider bandwidth even over an intentionally handicapped alternative with significant artifacts.

However, I don't know if this constitutes a fair and equitable testing environment. Most users of any particular codec have not had "artifact training"- so the unprepared results might better reflect the real performance.   On the other hand, I could argue that the initial preference for wider bandwidth is an initial impression effect and that even the untrained listeners would eventually prefer the less artifacted version after extended use... but I'm really waving my hands there, I don't have any data to support that position.

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 30 Jul 2010 15:38:20 -0400

> That's also why I thought your MUSHRA test used paired comparisons.

I think I'm actually the person who wrote that sentence on Wikipedia and 
apparently it was ambiguous :-) What I meant by "paired comparison" in this 
context is not that the listener is presented with pairs, but the fact that 
when doing the statistical analysis of the data later on, one can use 
paired statistical tests to compare different codecs. That's because it's 
the same sample that gets presented to the user with multiple codecs.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 30 Jul 2010 12:41:28 -0700

Hi Jean-Marc,

OK, thanks for the explanation.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Friday, July 30, 2010 12:38 PM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]; 'Anisse Taleb'
Subject: Re: [codec] getting more listening-test volunteers

Cheers,

	Jean-Marc