Subject: [codec] A concrete proposal for requirements and testing

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 31 Mar 2011 14:53:49 +0000

Hi,

Following the meeting and post-meeting discussions about requirements and testing, we would like to make the following proposal which addresses the opposing views which prevented consensus in the meeting today.

First, we propose to remove the following codecs from the requirements:

- GSM-FR, based on consensus from the list
- G.722, based on being clearly out-performed by G.722.1
- Speex-UWB, based on the fact that the author himself does not recommend it being used :-)

We can keep the other reference codecs as minimum quality requirement and include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice to have", but not hard requirements.

>From there and based on the listening tests presented by Jan Skoglund today, let's see what we can already conclude and what still needs more testing:

1) The narrowband test showed that Opus had higher quality than Speex at 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in narrowband mode?

2) The narrowband test showed that Opus had higher quality at 11 kb/s than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing iLBC.

3) There have been no formal comparison with AMR-NB yet. What do you think would be sufficient to assess the quality of Opus compared to AMR-NB?

4) The wideband test showed that Opus at 19.85 kb/s had higher quality than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?

5) The wideband test showed that Opus at 19.85 kb/s had higher quality than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1?

6) The wideband test showed that Opus at 19.85 kb/s had higher quality than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to concluded that the proposed "nice to have" objective of "no worse than AMR-WB" is met?

7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has already been shown to out-perform G.722.1C

If you disagree with any of the points above -- as may very well be the case -- please do provide a concrete test proposal that would be sufficient to convince you.

Cheers,

    Jean-Marc and Koen

---

From: Roman Shpount <[email-masked]>
Date: Thu, 31 Mar 2011 12:19:39 -0400

I might be missing something, but how is it that G.722 is clearly
out-performed by G.722.1? As far as I know G.722 at 64 KB has higher quality
then wideband G.722.1 at any rate. Obviously, G.722 has higher bandwidth
requirements, but has higher quality and lower bandwidth. Outperforming
G.722 at 16KHz for all languages, and especially for multiple simultaneous
speakers or non-speech sources is not an easy thing and should be tested
against.
_____________
Roman Shpount

On Thu, Mar 31, 2011 at 10:53 AM, Jean-Marc Valin <
[email-masked]> wrote:

>  Hi,
>
> Following the meeting and post-meeting discussions about requirements and
> testing, we would like to make the following proposal which addresses the
> opposing views which prevented consensus in the meeting today.
>
> We can keep the other reference codecs as minimum quality requirement and
> include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
> to have", but not hard requirements.
>
> From there and based on the listening tests presented by Jan Skoglund
> today, let's see what we can already conclude and what still needs more
> testing:
>
> 1) The narrowband test showed that Opus had higher quality than Speex at 11
> kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
> requirement of out-performing Speex in narrowband mode?
>
> 2) The narrowband test showed that Opus had higher quality at 11 kb/s than
> iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing iLBC.
>
> 3) There have been no formal comparison with AMR-NB yet. What do you think
> would be sufficient to assess the quality of Opus compared to AMR-NB?
>
> 4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
> 5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1?
>
> 6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
> concluded that the proposed "nice to have" objective of "no worse than
> AMR-WB" is met?
>
> 7) The fullband test showed that Opus at 32 kb/s had higher quality than
> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
> already been shown to out-perform G.722.1C
>
> If you disagree with any of the points above -- as may very well be the
> case -- please do provide a concrete test proposal that would be sufficient
> to convince you.
>
> Cheers,
>
>     Jean-Marc and Koen
>
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 31 Mar 2011 12:43:29 -0400

Hi,

Sorry, we should have made this clearer that when we mean "better" we  
mean quality vs bitrate coding efficiency. Otherwise nobody can ever  
beat the "16-bit PCM" codec (except maybe the 24-bit PCM codec) ;-)

Or are you suggesting that we compare G.722 to Opus running at 64  
kb/s? In that case should we be running Opus at its optimal point,  
which would probably be fullband stereo?

Cheers,

	Jean-Marc

---

From: Roman Shpount <[email-masked]>
Date: Thu, 31 Mar 2011 13:13:25 -0400

I would like to see a comparison of G.722 to Opus running to fullband mono
at its optimum point. To be honest, my interests are somewhat skewed towards
16 Khz audio, so comparison of G.722 to Opus at the optimal point for 16KHz
mono audio is also of great interest to me and my company.
_____________
Roman Shpount

2011/3/31 Jean-Marc Valin <[email-masked]>

> Hi,
>
> Sorry, we should have made this clearer that when we mean "better" we mean
> quality vs bitrate coding efficiency. Otherwise nobody can ever beat the
> "16-bit PCM" codec (except maybe the 24-bit PCM codec) ;-)
>
> Or are you suggesting that we compare G.722 to Opus running at 64 kb/s? In
> that case should we be running Opus at its optimal point, which would
> probably be fullband stereo?
>
> Cheers,
>
>        Jean-Marc
>
>

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Thu, 31 Mar 2011 13:33:41 -0400

On 03/31/2011 01:13 PM, Roman Shpount wrote:
> at its optimum point. To be honest, my interests are somewhat skewed
> towards 16 Khz audio, so comparison of G.722 to Opus at the optimal point
> for 16KHz mono audio is also of great interest to me and my company.

I would be happy to help you with an informal comparison.  Please select a
sample (or more than one if you prefer) and code it with G.722 as you
normally would.  We can then encode the sample with the latest development
version of Opus, and you can compare the results yourself.

Let's make sure not to let these application-specific tests delay the
standardization process, though.  With the latest test results from
Broadcom, Hydrogen Audio, and Google, we now have strong evidence that the
codec will exceed the working group's compression requirements.  We should
therefore focus our efforts on finalizing the standard without delay.

--Ben