Subject: [codec] I-D Action: draft-ietf-codec-results-02.txt

From: [email-masked]
Date: Mon, 05 Nov 2012 07:04:19 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Internet Wideband Audio Codec Working Group of the IETF.

	Title           : Summary of Opus listening test results
	Author(s)       : Christian Hoene
                          Jean-Marc Valin
                          Koen Vos
                          Jan Skoglund
	Filename        : draft-ietf-codec-results-02.txt
	Pages           : 31
	Date            : 2012-11-05

Abstract:
   This document describes and examines listening test results obtained
   for the Opus codec and how they relate to the requirements.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-codec-results

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-codec-results-02

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-codec-results-02

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: "Hoene" <[email-masked]>
Date: Mon, 5 Nov 2012 16:15:30 +0100

Hi all,

no new results so far :-(

But still some good news, in the next weeks we will start some more MUSHRA
and complexity testing. More to come.

With best regards,

 Christian

--
Universität Tübingen, Sand 13, 72076 Tübingen, Germany
Tel +49 7071 2970532, Fax +49 7071 5220
http://kn.inf.uni-tuebingen.de/staff/hoene.html

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
[email-masked]
Gesendet: Montag, 5. November 2012 16:04
An: [email-masked]
Cc: [email-masked]
Betreff: [codec] I-D Action: draft-ietf-codec-results-02.txt

A New Internet-Draft is available from the on-line Internet-Drafts
directories.
 This draft is a work item of the Internet Wideband Audio Codec Working
Group of the IETF.

Abstract:

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 05 Nov 2012 12:04:04 -0500

On 11/05/2012 10:15 AM, Hoene wrote:
> But still some good news, in the next weeks we will start some more
> MUSHRA and complexity testing. More to come.

Just curious, who is "we"?

	Jean-Marc

> With best regards,
> 
> Christian
> 
> 
> -- Universität Tübingen, Sand 13, 72076 Tübingen, Germany Tel +49
> 7071 2970532, Fax +49 7071 5220 
> -----Ursprüngliche Nachricht----- Von: [email-masked]
> [mailto:[email-masked]] Im Auftrag von 
> [email-masked] Gesendet: Montag, 5. November 2012 16:04 
> An: [email-masked] Cc: [email-masked] Betreff: [codec] I-D
> Action: draft-ietf-codec-results-02.txt
> 
> 
> directories. This draft is a work item of the Internet Wideband
> Audio Codec Working Group of the IETF.
> 
> Title           : Summary of Opus listening test results Author(s)
> : Christian Hoene Jean-Marc Valin Koen Vos Jan Skoglund Filename
> : draft-ietf-codec-results-02.txt Pages           : 31 Date
> : 2012-11-05
> 
> Abstract: This document describes and examines listening test
> results obtained for the Opus codec and how they relate to the
> requirements.
> 
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>

---

From: "Hoene" <[email-masked]>
Date: Mon, 5 Nov 2012 13:15:59 -0500

Hi Valin,

we at University of Tübingen/Symonics. I am expecting about 100 participants
in the tests. Give me some more time to write down the testing plan. I will
be happy to read your comments then.

The tests will focus on
1) Comparision with other codecs (similar to the graph on the Opus web page
but based on scientific correctly measured results).
2) Characterisation of Opus and its operational modes (including packet
loss)
Also, I am planning to measure the complexity of the codecs somehow.

As the tests are not intended for the codec developers but for people
outside of the IETF, I will focus on well standardized testing procedures
like MUSHRA and the guidelines given in
"practical procedures of subjective testing"
http://www.itu.int/opb/publications.aspx?media=electronic&parent=T-HDB-QOS.0
2-2011

Jean-Marc, there is no reason to be worried. We will just do some very
transparent MUSHRA testing on use cases that are important for codec users.
But - of course - I cannot promise you that Opus will be best in every
testing condition.

See you today at the Video BOF?

With best regards,

 Christian Hoene

-----Ursprüngliche Nachricht-----
Von: Jean-Marc Valin [mailto:[email-masked]] 
Gesendet: Montag, 5. November 2012 12:04
An: Hoene
Cc: [email-masked]
Betreff: Re: [codec] I-D Action: draft-ietf-codec-results-02.txt

	Jean-Marc

> directories. This draft is a work item of the Internet Wideband Audio 
> Codec Working Group of the IETF.
> 
> Abstract: This document describes and examines listening test results 
> obtained for the Opus codec and how they relate to the requirements.
> 
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 05 Nov 2012 14:09:43 -0500

Hi Christian,

> be happy to read your comments then.

I'll be happy to read your test plan. Note that if you really have 100
willing participants and are going to use MUSHRA, you can actually make
many tests because MUSHRA typically only needs 10-20 listeners to
produce statistically significant results. This would make it possible
to actually help improve the existing implementation, as opposed to only
adding one more data point.

> http://www.itu.int/opb/publications.aspx?media=electronic&parent=T-HDB-QOS.0

About MUSHRA, I suggest you review the exact methodology because your
previous test had issues with that. For example, your slides (
http://wiki.tools.ietf.org/wg/codec/trac/raw-attachment/wiki/TestingResults/summary.pdf
) indicated "Participants were not informed about the presence of hidden
references" which goes against the MUSHRA methodology.

Cheers,

	Jean-Marc