Subject: Re: [codec] getting more listening-test volunteers

From: Koen Vos <[email-masked]>
Date: Tue, 27 Jul 2010 15:55:01 +0200 (CEST)

The individual Mushra scores are available at 
http://jmvalin.ca/misc_stuff/mushra_results_July_2010.zip 

As we pointed out during the presentation, this test was not meant to be comprehensive or formal, but rather to provide evidence for the synergy between SILK and CELT in the hybrid codec. 

One particular flaw in the test was that Jean-Marc and myself participated in it. However, we found that our scores did not have a significant impact on the overall results. (You're welcome to verify: our scores are under "listener3" and "listener6".) 

best, 
koen. 

From: "Anisse Taleb" <[email-masked]> 
To: "Jean-Marc Valin" <[email-masked]>, "Yaakov Stein" <[email-masked]>, [email-masked] 
Sent: Tuesday, July 27, 2010 2:14:32 PM 
Subject: Re: [codec] getting more listening-test volunteers 

RE: [codec] getting more listening-test volunteers 

Thank you Jean Marc. 

Is it possible to also upload the raw data results, i.e. scores per user and item to allow for extra statistical analysis? 

/Anisse 

From: [email-masked] [mailto:[email-masked]] On Behalf Of Jean-Marc Valin 
Sent: den 27 juli 2010 12:11 
To: Anisse Taleb; Yaakov Stein; [email-masked] 
Subject: Re: [codec] getting more listening-test volunteers 

Hi, 

The results were actually presented yesterday in the codec session. They're on slide 8 of the codec prototype presentation: http://www.ietf.org/proceedings/78/slides/codec-0.pdf 
All codecs were tested at 32 kb/s on speech by 8 listeners. All were being used in constant bit-rate (CBR) mode except for SILK-WB, which was using VBR. 

Cheers, 

Jean-Marc 

-----Original Message----- 
From: Anisse Taleb [ mailto:[email-masked] ] 
Sent: Tue 7/27/2010 5:59 AM 
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked] 
Subject: RE: [codec] getting more listening-test volunteers 

Dear Jean Marc, 

I understand from you email that you have already some results available? 
And if yes, would it possible to share these results with the community ? 

Kind regards, 

From: [email-masked] [ mailto:[email-masked] ] On Behalf Of 
Jean-Marc Valin 
Sent: den 27 juli 2010 10:10 
To: Yaakov Stein; [email-masked] 
Subject: Re: [codec] getting more listening-test volunteers 

I actually think we should be alright for this time, since the only goal was 
to have a rough idea of how well we're doing. We'll keep improving the codec 
and bit and then do another listening test. 

Cheers, 

Jean-Marc 

-----Original Message----- 
From: [email-masked] on behalf of Yaakov Stein 
Sent: Tue 7/27/2010 2:56 AM 
To: [email-masked] 
Subject: [codec] getting more listening-test volunteers 

Before offering money for listeners on Mechanical Turk, 
why not try a request for volunteers on the IETF discussion list? 

If it is explained that the aim is to standardize a "non-proprietary" 
high-quality audio codec for general Internet use 
(strongly implying that this would obviate the need to use codecs with 
strong IPR attachments to large companies), 
then I bet there would be quite a few volunteers. 

If you want even more, then you could raffle off some social-event tickets 
or even a grand prize of free admission to IETF79. 

Y(J)S 

_______________________________________________ 
codec mailing list 
[email-masked] 
https://www.ietf.org/mailman/listinfo/codec

---

From: Koen Vos <[email-masked]>
Date: Tue, 27 Jul 2010 17:30:47 +0200 (CEST)

Thanks Anisse, 

Completely agree with your comments. 
Let's do some real listening tests soon. 

best, 
koen. 

From: "Anisse Taleb" <[email-masked]> 
To: "Koen Vos" <[email-masked]> 
Cc: "Jean-Marc Valin" <[email-masked]>, "Yaakov Stein" <[email-masked]>, [email-masked] 
Sent: Tuesday, July 27, 2010 4:41:47 PM 
Subject: RE: [codec] getting more listening-test volunteers 

Thanks Koen and JM, for the prompt replies. 

After a quick glance at the result, I noted some interesting (and alarming) behavior of the test subjects. 

1. According to Mushra methodology, one item at least has to be scored at 100, since subjects must be aware that there is a hidden reference. 

2. It is a custom that a post-screening of the subject has to be performed. This is essentially done to eliminate “deaf” subjects (joke). A usual post-screening is to eliminate subjects that : 

a. Fail in identifying the anchors, i.e. Hidden ref score > LP 7kHz > LP 3.5kHz 

b. Fail in scoring the Hidden reference above 90. 

Here is what my “quick and dirty” excel sheet reveals (this has been quickly done, so please cross-check, my conclusions below are based on this): 

(1) Listener 2, 4 and 8 failed in having an item scored at 100. 

(2a) Listener 1 fails at keeping the order in two items (item 6 and 8). 

(2b) Listener 1 and 4 fail at scoring the HR above 90. 

Also, additional comments: 

- I believe Listener 2 and 6 scoring are extreme. Scoring the LP 3.5 anchor at zero is not something commonly seen in formal mushra tests. 

- Listener 1 seem to obviously not like CELT at all, since it got a consistent score of zero, which is very peculiar…is this a software glitch ? 

At least from my experience I would have eliminated listeners 1, 2, 4, 6, and 8, which does not leave much room for proper statistical analysis with the remaining number of listeners (only 3 left). 

I believe, it is very important to maintain high standards when conducting listening tests and to examine listeners results carefully before deriving conclusions. That said, I understand that this is not a formal test, but I would urge you to make the necessary steps of post-screening and proper statistical analysis for future tests. I think this point was raised several times by ITU-T experts on this mailing list. 

Kind regards, 

/Anisse 

PS: I was planning on taking the test myself, but I am away from the office. 

From: Koen Vos [mailto:[email-masked]] 
Sent: den 27 juli 2010 15:25 
To: Anisse Taleb 
Cc: Jean-Marc Valin; Yaakov Stein; [email-masked] 
Subject: Re: [codec] getting more listening-test volunteers 

Attached are the voting results. 

best, 
koen. 

Thank you Jean Marc. 

/Anisse 

Hi, 

Cheers, 

Jean-Marc 

-----Original Message----- 
From: Anisse Taleb [ mailto:[email-masked] ] 
Sent: Tue 7/27/2010 5:59 AM 
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked] 
Subject: RE: [codec] getting more listening-test volunteers 

Dear Jean Marc, 

Kind regards, 

Cheers, 

Jean-Marc 

-----Original Message----- 
From: [email-masked] on behalf of Yaakov Stein 
Sent: Tue 7/27/2010 2:56 AM 
To: [email-masked] 
Subject: [codec] getting more listening-test volunteers 

Y(J)S

---

From: Michael Knappe <[email-masked]>
Date: Tue, 27 Jul 2010 19:09:05 -0700

AND... of course the amazing progress by Koen and Jean-Marc on the codec prototype!! Even over the room PA it sounded great.

Cheers,

Mike

----- Original Message -----
From: [email-masked] <[email-masked]>
To: Jean-Marc Valin <[email-masked]>; Anisse Taleb <[email-masked]>; Koen Vos <[email-masked]>
Cc: [email-masked] <[email-masked]>
Sent: Tue Jul 27 21:42:22 2010
Subject: Re: [codec] getting more listening-test volunteers

Just a quick note to everyone... The meeting yesterday was very productive, we made it through a good number of requirements questions, and set a clear timeline at the end to get the guidelines pulled together over the next few weeks. On the mailing list, let’s start pulling together the overall test methodology of both widescale listening tests as Koen suggested as well as more controlled environment testing as we discussed yesterday, with the goal to get this info into the guidelines doc.

If you were one of the several folks at the meeting yesterday who indicated you’d be interested in being a ‘test house’ for the controlled listening experiments, let me know who you are and we’ll set up a meeting together.

Thanks everyone for participating yesterday!

Cheers,

Mike

On 7/27/10 4:14 PM, "Jean-Marc Valin" <[email-masked]> wrote:

Hi,

Just to put things into perspective, we decided to do these listening tests only 2 days before flying to Maastricht, which is why we didn't give listeners highly detailed instructions. That being said, I think the difference between the hybrid codec and the others is so large that it cannot be attributed to methodology. This is pretty much all the information we were interested in at this point because of the very early stage of development of the codec. Once we have the codec in better shape, we'll be doing more controlled tests and we'd be glad to see you getting involved with the methodology (prior to the test this time).

Cheers,

    Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 10:41 AM
To: 'Koen Vos'
Cc: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

2.       It is a custom that a post-screening of the subject has to be performed. This is essentially done to eliminate "deaf" subjects (joke). A usual post-screening is to eliminate subjects that :

Here is what my "quick and dirty" excel sheet reveals (this has been quickly done, so please cross-check, my conclusions below are based on this):

-          Listener 1 seem to obviously not like CELT at all, since it got a consistent score of zero, which is very peculiar.is this a software glitch ?

Kind regards,

/Anisse

best,
koen.

  _____

Thank you Jean Marc.

/Anisse

Hi,

Cheers,

     Jean-Marc

-----Original Message-----
From: Anisse Taleb [mailto:[email-masked]]
Sent: Tue 7/27/2010 5:59 AM
To: Jean-Marc Valin; 'Yaakov Stein'; [email-masked]
Subject: RE: [codec] getting more listening-test volunteers

Dear Jean Marc,

Kind regards,

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: den 27 juli 2010 10:10
To: Yaakov Stein; [email-masked]
Subject: Re: [codec] getting more listening-test volunteers

Cheers,

   Jean-Marc

-----Original Message-----
From: [email-masked] on behalf of Yaakov Stein
Sent: Tue 7/27/2010 2:56 AM
To: [email-masked]
Subject: [codec] getting more listening-test volunteers

Y(J)S