Subject: [codec] Test Results

From: Cullen Jennings <[email-masked]>
Date: Fri, 25 Mar 2011 13:42:54 +0100

Does anyone have opus test results they would like to present in the meeting?

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 25 Mar 2011 11:14:26 -0700

Hi Cullen,

Last December Broadcom conducted a subjective listening test of the CELT-only mode of Opus.  What's tested were an earlier version of CELT (version 0.9.1 I believe) with different configurations, together with MP3 and AAC_LC.  Although it was the latest CELT version at that time, it did not have the improvements and tuning made later in the current version.  

I would like to present the test results.  However, I won't be able to attend the Prague meeting in person.  Although I hope to join the meeting remotely through audio feed and jabber room despite the crazy time for me (1 - 3:30 AM), I am not sure it is technically feasible for me to verbally present our test results at the meeting.  For this reason, I think I may just have to send out some slides summarizing the test results and discuss them by email before the Codec WG meeting next Thursday.  Is that okay?

Thanks.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Cullen Jennings
Sent: Friday, March 25, 2011 5:43 AM
To: [email-masked]
Subject: [codec] Test Results

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Cullen Jennings <[email-masked]>
Date: Sat, 26 Mar 2011 14:51:23 +0100

Please do and we can get your summary slides included with rest of results. 

Thanks, Cullen

> Does anyone have opus test results they would like to present in the meeting?

Cullen Jennings
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 29 Mar 2011 18:48:29 -0700

All,

Please see our CELT listening test result and related description and discussion in the attached PDF file.

Thanks.

Raymond

-----Original Message-----
From: Cullen Jennings [mailto:[email-masked]] 
Sent: Saturday, March 26, 2011 6:51 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: Re: [codec] Test Results

Thanks, Cullen

Cullen Jennings

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 30 Mar 2011 08:51:27 +0000

Dear Raymond,

Thanks for sending these results. One question, which versions of encoders has been used for the reference codecs, AAC-LC and MP3?

Kind regards,
/Anisse

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Raymond (Juin-Hwey) Chen
> Sent: Wednesday, March 30, 2011 3:48 AM
> To: Cullen Jennings
> Cc: [email-masked]
> Subject: [codec] Broadcom's CELT listening test result
> 
> All,
> 
> > Last December Broadcom conducted a subjective listening test of the CELT-
> only mode of Opus.  What's tested were an earlier version of CELT (version
> 0.9.1 I believe) with different configurations, together with MP3 and
> AAC_LC.  Although it was the latest CELT version at that time, it did not
> have the improvements and tuning made later in the current version.
> >
> > I would like to present the test results.  However, I won't be able to
> attend the Prague meeting in person.  Although I hope to join the meeting
> remotely through audio feed and jabber room despite the crazy time for me
> (1 - 3:30 AM), I am not sure it is technically feasible for me to verbally
> present our test results at the meeting.  For this reason, I think I may
> just have to send out some slides summarizing the test results and discuss
> them by email before the Codec WG meeting next Thursday.  Is that okay?
> >
> > Thanks.
> >
> > Raymond
> >

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 30 Mar 2011 18:11:50 +0200

Yes, these are interesting results, Raymond. Do you plan to provide more
details, such as individual results for the different audio stimuli,
instructions to subjects etc?

...Paul

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Anisse Taleb
Sent: Wednesday, March 30, 2011 10:51 AM
To: Raymond (Juin-Hwey) Chen; Cullen Jennings
Cc: [email-masked]
Subject: Re: [codec] Broadcom's CELT listening test result

Dear Raymond,

Thanks for sending these results. One question, which versions of encoders
has been used for the reference codecs, AAC-LC and MP3?

Kind regards,
/Anisse

---

From: "Prasanth Kottantharayil" <[email-masked]>
Date: Wed, 30 Mar 2011 11:45:06 -0700

Hi,

I am Raymond's colleague and I can confirm the encoder versions used in this test.

The AAC-LC encoder was Nero AAC encoder version 1.1.34.2.
The MP3 encoder was the one available in Adobe Audition v3.0.

Thanks
Prasanth

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anisse Taleb
Sent: Wednesday, March 30, 2011 1:51 AM
To: Raymond (Juin-Hwey) Chen; Cullen Jennings
Cc: [email-masked]
Subject: Re: [codec] Broadcom's CELT listening test result

Dear Raymond,

Kind regards,
/Anisse

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 30 Mar 2011 17:57:05 -0700

Hi Paul,

I thought the aggregate scores I presented already give a pretty clear picture of where each codec stand relative to others.  Is your question about individual results for each audio track a matter of curiosity, or do you consider them important for codec evaluation?

If the group thinks it is really necessary to see such detailed break-out scores for each audio track, I can try to do it, but it will take me a while to do it.  Currently I am too busy trying to meet some other urgent project deadlines, so it will have to be at least several days later if the WG wants to see such individual scores.

Thanks.

Raymond

-----Original Message-----
From: Paul Coverdale [mailto:[email-masked]] 
Sent: Wednesday, March 30, 2011 9:12 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] Broadcom's CELT listening test result

...Paul

-----Original Message-----

Dear Raymond,

Kind regards,
/Anisse

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 31 Mar 2011 14:32:37 +0200

Hi Raymond,

The Broadcom results (and others) were discussed during this morning's CODEC
WG session, and my comment generally applies to all results. I think it is
important to see detailed results, since averages can sometimes conceal
problems. Anyway, one conclusion from the meeting is that a number of
organizations indicated that they may be willing to conduct tests, if a
reasonably detailed test plan can be defined and agreed. The chairs asked
for proposals on this topic.

Regards,

...Paul

-----Original Message-----
From: Raymond (Juin-Hwey) Chen [mailto:[email-masked]] 
Sent: Thursday, March 31, 2011 2:57 AM
To: Paul Coverdale
Cc: [email-masked]
Subject: RE: [codec] Broadcom's CELT listening test result

Hi Paul,

I thought the aggregate scores I presented already give a pretty clear
picture of where each codec stand relative to others.  Is your question
about individual results for each audio track a matter of curiosity, or do
you consider them important for codec evaluation?

If the group thinks it is really necessary to see such detailed break-out
scores for each audio track, I can try to do it, but it will take me a while
to do it.  Currently I am too busy trying to meet some other urgent project
deadlines, so it will have to be at least several days later if the WG wants
to see such individual scores.

Thanks.

Raymond

-----Original Message-----
From: Paul Coverdale [mailto:[email-masked]] 
Sent: Wednesday, March 30, 2011 9:12 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] Broadcom's CELT listening test result

...Paul

-----Original Message-----

Dear Raymond,

Kind regards,
/Anisse