Subject: Re: [codec] Listening tests at Google

From: Paul Beaumont <[email-masked]>
Date: Wed, 19 Oct 2011 00:01:23 +0100

Hi Jan

Thank you for performing and sharing the results of your testing.

Please can you summarise what you see as the key findings wrt ...

1). Opus on Mandarin speech - be good to baseline this against comparable
English speech results and
2). Opus when Transcoded/Tandemed

Regards
Paul

--

On 14/09/2011 06:59, "Jan Skoglund" <[email-masked]> wrote:

>Hey,
>
>The listening tests at Google are now done and here are the results.
>
>Regards,
>Jan
>
>
>On Fri, Aug 19, 2011 at 3:28 PM, Jan Skoglund <[email-masked]> wrote:
>> Hey,
>>
>> We've prepared some listening tests for Opus on Mandarin speech and
>> transcoding (English) speech at Google. We started the tests today and
>> will have results at the end of August. The tests are split in four
>> MUSHRA groups:
>>
>> Test 1a - Mandarin NB
>> --------------------------
>> Opus NB
>> Speex NB
>> iLBC NB
>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>
>> Test 1b - Mandarin "HD Voice" (WB, FB)
>> --------------------------------------------------------
>> Opus FB
>> G.719 FB
>> Opus WB
>> Speex WB
>> G.722.1 WB
>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>
>> Test 2a - Transcoding NB
>> -------------------------------
>> G.711 -> Opus NB
>> G.711 -> AMR-NB
>> Opus NB -> G.711 -> AMR-NB
>> AMR-NB -> G.711 -> Opus NB
>> AMR-NB -> G.711 -> AMR-NB
>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>
>> Test 2b - Transcoding WB
>> --------------------------------
>> Opus WB
>> AMR-WB
>> Opus WB -> AMR-WB
>> AMR-WB -> Opus WB
>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>
>>
>> Cheers,
>> Jan
>>
>_______________________________________________
>codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/codec

---

From: Jan Skoglund <[email-masked]>
Date: Thu, 20 Oct 2011 10:32:48 -0400

Hey,

Sure,

1) For narrowband Mandarin speech Opus at 11 kbps is comparable to
iLBC at 15 kbps and superior to Speex at 11 kbps. In the previous
tests with English speech Opus outperformed them both, so there was a
(small) difference.
For wideband/fullband speech a striking observation is that Opus at 32
kbps is (again) almost transparent, although there is a small and
statistically significant difference from the fullband reference
material. Opus at 32 kbps is also better than G.719 at the same bit
rate.  Opus at 20 kbps is not only better than Speex at 24 kbps and
G.722.1 at 24 kbps, it is also better than the reference signal
low-passed at 7 kHz. This is likely due to its high quality and its
higher signal bandwidth (8 kHz). These results are pretty consistent
with the previous results for English speech, with the difference that
in the previous test Opus at 20 kbps outperformed G.719 at 32 kbps,
which it did not this time.

2) Pre-coding Opus NB with G.711 is comparable to pre-coding AMR NB
with G.711. Adding one stage of transcoding reduces the performance
slightly (although statistically significant). Transcoding AMR NB with
itself through G.711 is worse than transcoding AMR NB to Opus through
G.711 or the other way around.
Opus WB at 19.85 kbps is again better than speech LP-filtered at 7.0
kHz,  as in the Mandarin and English speech tests. Single-coded AMR WB
is significantly worse than single-coded Opus WB, also consistent with
previous tests. Transcoding Opus and AMR WB is only slightly worse
than AMR WB, although the difference is statistically significant.

Jan

---

From: Butrus Damaskus <[email-masked]>
Date: Thu, 20 Oct 2011 17:39:32 +0200

On Thu, Oct 20, 2011 at 4:32 PM, Jan Skoglund <[email-masked]> wrote:
> These results are pretty consistent

Does it mean that there is a regression or is it rather due to the
character of the Mandarine speech?

---

From: Jan Skoglund <[email-masked]>
Date: Fri, 21 Oct 2011 15:59:12 -0400

I think it can be due to the character of Mandarin speech (although
I'm not a linguist), which does not seem to contain as much high
frequency-rich consonants as English. I noticed that during recording
of the Mandarin speech, and discussing afterwards with the talkers,
who said that there are much more, e.g., [s] sounds in English.

---

From: Butrus Damaskus <[email-masked]>
Date: Fri, 21 Oct 2011 22:33:35 +0200

Ok, thanks for the clarification, sounds interesting!