Subject: [codec] Some listening test results

From: "Christian Hoene" <[email-masked]>
Date: Fri, 26 Jul 2013 16:36:36 +0200

Hi,

a first set of results can be downloaded here

 <http://trac.tools.ietf.org/wg/codec/trac/wiki/TestingResults>
http://trac.tools.ietf.org/wg/codec/trac/wiki/TestingResults

More will be added soon.

Thanks to Patrick Schreiner and Alfons Martin, who did the tests.

With best regards,

Christian Hoene

 

Symonics GmbH

Sand 13

72076 Tübingen

Tel +49 7071 5681302

Fax +49 7071 5681309

Email:  <mailto:[email-masked]> [email-masked]

Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene, Patrick
Schreiner

Sitz der Gesellschaft/Place of Business: Tübingen

Registereintrag/Commercial Register: Amtsgericht Stuttgart, HRB 739918

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Fri, 26 Jul 2013 12:07:09 -0400

> Thanks to Patrick Schreiner and Alfons Martin, who did the tests.

Christian,

Thanks for sharing these with us.  Do you have any details on the
specific implementations/versions of the codecs you used?  I'm
particularly interested in Opus because I know that it is under active
development, and quality has improved significantly in recent beta releases.

-- 
Libre Video
http://librevideo.org

---

From: Ron <[email-masked]>
Date: Sat, 27 Jul 2013 06:06:52 +0930

Hi Christian,

Thanks for publishing this.  Do you happen to have some numbers on
what the expected bit error rates are for a "typical AMR-WB usage
scenario"?

I'm curious about that because you note that the Opus bitstream
wasn't designed to be inherently robust against a bit error at
any point in the packet (and some places will obviously effect
it worse than others) - however it's not quite true that it
'cannot tolerate them'.

Opus requires framing for transport, and if the transport itself
doesn't ensure data integrity then it's reasonably easy for the
framing to do so (such as Ogg does with its CRC).

In which case a bit error simply becomes a packet loss, handled
by the normal Opus PLC/FEC mechanisms.  So I'm curious about how
the results you saw for various packet loss rates might map to
expected service quality, even without taking steps to make the
link itself more robust than it already is.

Given the increasing importance of data over cellular links,
I'd expect the current BER targets are already fairly low, though
I don't seem to be able to find any numbers on current expectation.
Just some papers that note "it's much lower than people think",
and "approaching the rates seen on wired networks" - though of
course it's much more likely to be bursty when errors do occur,
and going to vary widely by locale.

  Cheers,
  Ron

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 26 Jul 2013 15:51:06 -0700

> it worse than others) - however it's not quite true that it

It's not quite true to say that Opus was not designed to be robust to 
them, either. We use "raw bits" (RFC 6716 Section 4.1.4) where possible 
precisely because a bit error in one of those will not desync the 
bitstream. We also designed things so that the bits most sensitive to 
errors were at the beginning of the packet (see slide 46 of 
<http://www.celt-codec.org/presentations/misc/lca-celt.pdf> for some 
early measurements on the CELT layer from my 2009 LCA talk). This takes 
advantage of the unequal error protection afforded by Trellis Coded 
Modulation (TCM), and in other modulation schemes a SECDED code on the 
first 64 bits or so of the packet should substantially cure the bulk of 
bit-error impairment.

As Ron notes, the choice of modulation scheme or error protection scheme 
is best dealt with at a layer above the codec itself. We just tried to 
make it easy to do so. I guess you can argue whether or not that's 
"inherent".