Subject: [codec] Opus comparison test plan version 2

From: Alfons Martin <[email-masked]>
Date: Wed, 17 Apr 2013 15:50:20 +0200

AMR-WB

Based on the feedback we changed the document.

1)No input bandwidth filtering for speech at all

2)"Make sure to tell the Opus encoder what the percentage of loss is so 
it can optimize the encoding for it." -- Ok, but we have to consider 
both cases.

3)"For AMR-WB the 8.85 mode is "intended to be used only temporarily 
during severe radio channel conditions or during network congestion". 
Right, better we use AMR-WB 12.65 with DTX in this case.

We did not consider the following suggestion

1)"Opus should use VBR" -- For direct comparison, we have to choose same 
mode as the other codec. For example, AMR-WB does not support VBR.

2)"If you're going to test packet loss, there should be at least one 
test with FEC -- probably the 23.85 one". As for now, we are not sure 
whether the Opus FEC implementation is bug free (see FEC question email).

AAC-eLD

Based on the feedback we changed the document.

1)We will use SBR

2)"Mono and stereo should not be mixed in one experiment to avoid the 
influence of this factor on the assessment of the coding quality." -- 
agreed.

3)24, 32, 48 kbit/s for mono at 32 kHz sample rate CHECK

4)For stereo 32, 48, 64 and 96 kbit/s could be used. CHECK

We did not consider the following suggestion

1)

General

Based on the feedback we changed the document.

1)We will use loss rates 0, 1, 3, 6%

We did not consider the following suggestion

1)We will keep the MNRU 16 anchor (MOS 2.2) because some of the samples 
will be worse than LP 3.5 but better than MOS 2.2. We skip LP 7 as it is 
optional.

2)We stick to MUSHRA as it is the only tests that covers a wide range of 
degradations.

Thank you very much for your timely feedback.

Alfons, Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 17 Apr 2013 16:26:28 +0200

Hi,

As for 24, 32, 48 kbit/s for mono and 32, 48, 64 and 96 kbit/s for stereo we
can assume that least one out of the two codecs will work. Thus, we keep
those parameter setting in.

With best regards,

Christian

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
Alfons Martin
Gesendet: Mittwoch, 17. April 2013 15:50
An: [email-masked]; [email-masked]
Cc: Christian Hoene; [email-masked]
Betreff: [codec] Opus comparison test plan version 2

AMR-WB

1)      No input bandwidth filtering for speech at all

2)      "Make sure to tell the Opus encoder what the percentage of loss is
so it can optimize the encoding for it." - Ok, but we have to consider both
cases.

3)      "For AMR-WB the 8.85 mode is "intended to be used only temporarily
during severe radio channel conditions or during network congestion". Right,
better we use AMR-WB 12.65 with DTX in this case.

1)      "Opus should use VBR" - For direct comparison, we have to choose
same mode as the other codec. For example, AMR-WB does not support VBR.

2)      "If you're going to test packet loss, there should be at least one
test with FEC -- probably the 23.85 one". As for now, we are not sure
whether the Opus FEC implementation is bug free (see FEC question email).

AAC-eLD

1)      We will use SBR

2)      "Mono and stereo should not be mixed in one experiment to avoid the
influence of this factor on the assessment of the coding quality." - agreed.

3)      24, 32, 48 kbit/s for mono at 32 kHz sample rate CHECK 

4)      For stereo 32, 48, 64 and 96 kbit/s could be used. CHECK

1)       

General

1)      We will use loss rates 0, 1, 3, 6%

1)      We will keep the MNRU 16 anchor (MOS 2.2) because some of the
samples will be worse than LP 3.5 but better than MOS 2.2. We skip LP 7 as
it is optional.

2)      We stick to MUSHRA as it is the only tests that covers a wide range
of degradations.

Alfons, Christian

---

From: Ron <[email-masked]>
Date: Thu, 18 Apr 2013 03:06:34 +0930

On Wed, Apr 17, 2013 at 03:50:20PM +0200, Alfons Martin wrote:
> 1)"Opus should use VBR" -- For direct comparison, we have to choose

Forcing one of the codecs into a non-default mode doesn't seem like
a very useful or 'direct' comparison.  That would mean the only
conclusion that anyone could draw from this test would be either:

 - Opus with one hand tied behind its back still beats AMR-WB.

or

 - AMR-WB narrowly beats, or is no worse than, Opus running in a
   degraded mode that almost nobody will actually ever use.

Neither of which really seems worth spending someone's money to know
instead of finding out how they really compare when used as intended.

Handicap races are for gambling, not performance measurement.

You don't take all but 4 spark plugs out of a Ferrari if you want
to make a "direct comparison" to a Volkswagon.  That won't tell
you anything useful about their relative performance in real use.

  Ron

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 17 Apr 2013 14:03:09 -0400

Well, my original email had 13 points, most of which seem to have been
ignored, including the low-pass filtering of speech and music, the
8.85 kb/s, and the VBR issue. So I guess I'm done with the reviewing.
Good luck with your test.

Cheers,

	Jean-Marc

On 04/17/2013 09:50 AM, Alfons Martin wrote:
> 2)      “Make sure to tell the Opus encoder what the percentage of
> loss is so it can optimize the encoding for it.” – Ok, but we have
> to consider both cases.
> 
> 3)      “For AMR-WB the 8.85 mode is "intended to be used only 
> temporarily during severe radio channel conditions or during
> network congestion". Right, better we use AMR-WB 12.65 with DTX in
> this case.
> 
> 
> 
> 1)      “Opus should use VBR” – For direct comparison, we have to
> choose same mode as the other codec. For example, AMR-WB does not
> support VBR.
> 
> 2)      “If you're going to test packet loss, there should be at
> least one test with FEC -- probably the 23.85 one”. As for now, we
> are not sure whether the Opus FEC implementation is bug free (see
> FEC question email).
> 
> 
> 
> AAC-eLD
> 
> 2)      “Mono and stereo should not be mixed in one experiment to
> avoid the influence of this factor on the assessment of the coding
> quality.” – agreed.
> 
> 1)      We will keep the MNRU 16 anchor (MOS 2.2) because some of
> the samples will be worse than LP 3.5 but better than MOS 2.2. We
> skip LP 7 as it is optional.
> 
> 2)      We stick to MUSHRA as it is the only tests that covers a
> wide range of degradations.
> 
> 
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 17 Apr 2013 20:47:23 +0200

Hello Jean-Marc,

I do not get your arguments.

* the 8.85 issues have been considered. It has been replaced with 12.65 plus
DTX
* low-pass filtering has been removed
* VBR is used in the music modes.
* and, because of the many requests, we will compare AMR-WB with both Opus
CBR und Opus VBR.

Happy? What else?

Christian

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
Jean-Marc Valin
Gesendet: Mittwoch, 17. April 2013 20:03
An: Alfons Martin
Cc: Christian Hoene; [email-masked]; [email-masked]
Betreff: Re: [codec] Opus comparison test plan version 2

Cheers,

	Jean-Marc

On 04/17/2013 09:50 AM, Alfons Martin wrote:
> loss is so it can optimize the encoding for it.” – Ok, but we have to 
> consider both cases.
> 
> temporarily during severe radio channel conditions or during network 
> congestion". Right, better we use AMR-WB 12.65 with DTX in this case.
> 
> 
> 
> least one test with FEC -- probably the 23.85 one”. As for now, we are 
> not sure whether the Opus FEC implementation is bug free (see FEC 
> question email).
> 
> 
> 
> AAC-eLD
> 
> the samples will be worse than LP 3.5 but better than MOS 2.2. We skip 
> LP 7 as it is optional.
> 

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec