Subject: Re: [codec] Updated Opus draft

From: Anisse Taleb <[email-masked]>
Date: Tue, 01 Mar 2011 16:46:40 +0000

Dear Jean-Marc, all.

Thanks for the update. I am not sure if my understanding of the scope of this work is still up-to-date, but wasn't the idea of making the opus codec to actually combine the best and strongest of the technologies of SILK and CELT resulting in a single codec? At least according to the LS to ITU-T SG 16 "The experts got together, took the best ideas from each of their proposals, and combined them to form a new codec."

As it stands out from the code (and the draft), the Opus codec is in fact the SILK codec and CELT codec glued together by the so-called "hybrid" codec. Correct me if I am wrong, but it feels like we started with two codecs and now we are standardizing 3 codecs? Calling them modes will not hide the fact that Opus is a high level merging of two independent codecs. In-band signaling of the codecs as modes to achieve (perhaps?) interoperability while marketing the package as a new codec with a new name is something I find very distressing. Any clarifications would be greatly appreciated. 

I have some additional comments on the code/draft. 

- It would be good to list what functionalities are currently supported. The README file provides very little in that respect, so does the draft, and unless we dig deep in the code, I am not sure what the codec can or cannot do. 

- The code did not compile on my cygwin box (0.9.0 does), I think anyways that this is a minor bug:

src_SigProc_FIX/SKP_Silk_debug.c: In function 'GetHighResolutionTime':
src_SigProc_FIX/SKP_Silk_debug.c:62: error: storage size of 'tv' isn't known
make[1]: *** [libSKP_SILK_SDK_la-SKP_Silk_debug.lo] Error 1

and due to the following:

#ifdef _WIN32

#if (defined(_WIN32) || defined(_WINCE)) 
#include <windows.h>    /* timer */
#else   // Linux or Mac
#include <sys/time.h>
#endif

unsigned long GetHighResolutionTime(void) /* O: time in usec*/ {
    /* Returns a time counter in microsec   */
    /* the resolution is platform dependent */
    /* but is typically 1.62 us resolution  */
    LARGE_INTEGER lpPerformanceCount;
    LARGE_INTEGER lpFrequency;
    QueryPerformanceCounter(&lpPerformanceCount);
    QueryPerformanceFrequency(&lpFrequency);
    return (unsigned long)((1000000*(lpPerformanceCount.QuadPart)) / lpFrequency.QuadPart); }
#else   // Linux or Mac
unsigned long GetHighResolutionTime(void) /* O: time in usec*/ {
    struct timeval tv;
    gettimeofday(&tv, 0);
    return((tv.tv_sec*1000000)+(tv.tv_usec));
}
#endif

A quick fix is to move the conditional includes prior to the #ifdef _WIN32 such that <sys/time.h> is actually included.

- The code does not provide any *working* VS project files, while I understand the desire to stick to open source tools, I think providing the means to compile the code using visual studio or other compilers would 1) engage a wider developer community 2) allow to check how stable the implementation is when ported/compiled in a different platform. From my experience, besides being a good practice, this can also reveal some bugs that are not easily apparent if the code is consistently developed under a single platform. I believe express versions are now free for use, I have tried to make such a project file and after many attempts including removal of debug code etc...the code finally compiled and after many crashes, it could finally be run (the list of changes needed is too long for this email). The waveform output with the VS version deviated by about 16 dB wrt. the cygwin version which is imho quite substantial and would likely result in the codec not passing the conformance tests you proposed. This observation reinforces the need to have the code built by different compilers and to make sure that the code is stable enough.

- Following a particular programming style will greatly help developer to read and understand source code, and help to avoid introducing errors. I am not sure what coding conventions have been used, but the coding style is indeed different between the silk part and celt part. Notably, how come there are two type definitions, one for SILK (SKP_Silk_typedef.h) and one for CELT (celt_types.h), how does this help the developer ? 

Example :
#define SKP_int         int                     /* used for counters etc; at least 16 bits */
#define SKP_int64       long long
#define SKP_int32       int
#define SKP_int16       short
#define SKP_int8        signed char

and 

typedef __int32 celt_int32;
typedef unsigned __int32 celt_uint32;
typedef __int16 celt_int16;
typedef unsigned __int16 celt_uint16;

besides the names being unnecessarily different, one uses a #define while the other uses a typedef. I am not advocating the use of one or the other, for that, please have a look at http://c-faq.com/decl/typedefvsdefine.html but that a convention is kept consistent across the Opus code.

Some other fundamental issue with the code is the separation between floating and fixed point code. I am not sure where it all begins and where it ends and if it works at all. It would be good to have the distribution of Opus clearly distinguish between the two implementations.

- A final comment on the complexity of the codec, ITU-T SG16 maintains a library of fixed point operators as well as tools to estimate the *fixed-point* complexity of floating point operations, this library is known as ITU-T Software Tool Library 2009, is maintained by Q10 and is part of ITU-T Rec. G.191. In my experience, these tools provide a very good estimate of the complexity of codecs in terms of WMOPS and has been instrumental in comparing the complexity of various codecs. I would strongly recommend using it for this exercise, for the simple reason that it would reliably provide the complexity of the Opus codec in relation to well known and widely deployed codecs such as AMR, AMR-WB, G.719, G.729(.1), etc.

Kind regards,
/Anisse

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Jean-Marc Valin
> Sent: Tuesday, February 15, 2011 6:51 AM
> To: [email-masked]
> Subject: [codec] Updated Opus draft
> 
> Hi,
> 
> We have updated the Opus draft. The revised version is available at
> http://www.ietf.org/id/draft-ietf-codec-opus-03.txt and a tarball is
> available at http://downloads.xiph.org/releases/opus/opus-0.9.1.tar.gz
> .
> 
> Because of some minor bugs we found in the previous release, we had to
> change the bit-stream slightly for 0.9.1. The quality should not be
> affected except for small improvements in rare cases. The release also
> adds support for forward error correction in SILK-only and hybrid
> modes.
> The only planned feature that is not in yet is stereo support in
> SILK-only and hybrid mode. We are working on this.
> 
> We have also setup a basic website with some information about Opus:
> http://opus-codec.org/ There isn't much yet, but we'll keep adding to
> it.
> 
> Cheers,
> 
> 	Jean-Marc
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 1 Mar 2011 15:08:16 -0500

Hi Anisse,

Thanks for your feedback. I'll try to answer as many questions as I can 
(Koen should be able to handle the others) below:

On 11-03-01 11:46 AM, Anisse Taleb wrote:
> As it stands out from the code (and the draft), the Opus codec is in
> fact the SILK codec and CELT codec glued together by the so-called
> "hybrid" codec. Correct me if I am wrong, but it feels like we started
> with two codecs and now we are standardizing 3 codecs? Calling them
> modes will not hide the fact that Opus is a high level merging of two
> independent codecs. In-band signaling of the codecs as modes to achieve
> (perhaps?) interoperability while marketing the package as a new codec
> with a new name is something I find very distressing. Any clarifications
> would be greatly appreciated.

Although a lot of the code is split between a silk/ and a celt/ directory, 
the integration is tighter than it may seem. Both SILK and CELT have been 
heavily modified for this to work (shared entropy coding, band layout, 
...), in addition to the visible part of the "glue". When you look at the 
SILK+hybrid modes, it is not unlike G.729.1 codec (which I assume you do 
not find distressing). What we have in addition to this is another mode 
that can handle music much better than the hybrid codec. I think we have 
already demonstrated that the hybrid mode performs pretty well (very high 
full-band speech quality at 32 kb/s) and allowing the other modes 
(SILK-only and CELT-only) is an added benefit with virtually no cost. The 
result is a codec that can scale on a continuous scale from 6 kb/s 
narrowband speech to 256 kb/s (and up) full-band stereo music.

> - It would be good to list what functionalities are currently supported.
> The README file provides very little in that respect, so does the draft,
> and unless we dig deep in the code, I am not sure what the codec can or
> cannot do.

Sorry about this. The README file really needs an update. The main missing 
functionality is stereo support for the SILK-only and hybrid modes. We're 
also working on improving mode switching.

> anyways that this is a minor bug:
<snip>

I'll leave that one to Koen.

> - The code does not provide any *working* VS project files, while I
> understand the desire to stick to open source tools, I think providing
> the means to compile the code using visual studio or other compilers
> would 1) engage a wider developer community 2) allow to check how stable
> the implementation is when ported/compiled in a different platform. From
> my experience, besides being a good practice, this can also reveal some
> bugs that are not easily apparent if the code is consistently developed
> under a single platform.

I know Koen uses VS for development, so it may just be that the files I 
packaged were out-of-date or that I forgot something. I agree that we 
should make sure it works.

> I believe express versions are now free for
> use, I have tried to make such a project file and after many attempts
> including removal of debug code etc...the code finally compiled and
> after many crashes, it could finally be run (the list of changes needed
> is too long for this email). The waveform output with the VS version
> deviated by about 16 dB wrt. the cygwin version which is imho quite
> substantial and would likely result in the codec not passing the
> conformance tests you proposed. This observation reinforces the need to
> have the code built by different compilers and to make sure that the
> code is stable enough.

Did you compare just the decoder or encoder+decoder. Because the encoder is 
not normative (and uses floating-point by default), it's normal that the 
encoder would produce different (but equivalent) output depending on the 
platform and compiler. It could also be a bug that you hit (in which case 
more info would be useful so we can fix it).

> - Following a particular programming style will greatly help developer
> to read and understand source code, and help to avoid introducing
> errors. I am not sure what coding conventions have been used, but the
> coding style is indeed different between the silk part and celt part.
> Notably, how come there are two type definitions, one for SILK
> (SKP_Silk_typedef.h) and one for CELT (celt_types.h), how does this help
> the developer ?

Agreed. This is on the TODO list, but so far we have focused our efforts on 
the bit-stream itself. Once the bit-stream is good, we can focus on a (much 
needed) code cleanup.

> Some other fundamental issue with the code is the separation between
> floating and fixed point code. I am not sure where it all begins and
> where it ends and if it works at all. It would be good to have the
> distribution of Opus clearly distinguish between the two
> implementations.

Currently, there is a *lot* of code that is shared between floating-point 
and fixed-point and I think it makes sense from a development point-of-view 
(to avoid duplicating work). For example, if you look at the CELT module, 
float and fixed-point are based on exactly the same code with just 
different macros for the arithmetic operators. This makes maintenance 
easier. The SILK module has separate files for fixed-point and floating 
point, but even then there is a lot of code in common. Considering that the 
encoder will continue to evolve (because it does not have a bit-exact 
definition), I think it makes sense to continue sharing as much code as 
possible, but that's certainly something we can discuss.

> - A final comment on the complexity of the codec, ITU-T SG16 maintains a
> library of fixed point operators as well as tools to estimate the
> *fixed-point* complexity of floating point operations,

Not sure what you mean here by "fixed-point complexity of floating point 
operations". Do you mean the cost of emulating float using integer 
arithmetic? If so, then I'm not sure how useful that it considering we can 
have a purely fixed-point build for platforms with no (or slow) FPU.

> this library is
> known as ITU-T Software Tool Library 2009, is maintained by Q10 and is
> part of ITU-T Rec. G.191. In my experience, these tools provide a very
> good estimate of the complexity of codecs in terms of WMOPS and has been
> instrumental in comparing the complexity of various codecs. I would
> strongly recommend using it for this exercise, for the simple reason
> that it would reliably provide the complexity of the Opus codec in
> relation to well known and widely deployed codecs such as AMR, AMR-WB,
> G.719, G.729(.1), etc.

This is something we may want to look into. In fact, it may even be 
possible to map our current operators on to the ITU operators, but I'd need 
to take a closer look. Even without that we already have our own similar 
counters. In terms of comparing with codecs like AMR/AMR-WB, correct me if 
I'm wrong, but I was under the impression that the basicops version used in 
those is older and does not define many of the operations we use 
(non-fractional multiplications, non-saturating add, ...). In any case, 
this is another thing we can discuss into more details and given how the 
code is structured, it shouldn't be too hard to define more alternative 
implementations of the current operators.

Cheers,

	Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Tue, 1 Mar 2011 23:56:31 +0100 (CET)

Hi Anisse,

I'll make sure all (platform specific) debug code is properly disabled in the next update.

And yes, I've been using Visual Studio without problems, and I agree with you we should have all the correct project files in the GIT repository.

Switching from floating-point to fixed-point code is as simple as defining "FIXED_POINT" during compilation (at least in VS), but this should be more clearly documented.

best,
koen.

----- Original Message -----
From: "Jean-Marc Valin" <[email-masked]>
To: "Anisse Taleb" <[email-masked]>
Cc: [email-masked]
Sent: Tuesday, March 1, 2011 12:08:16 PM
Subject: Re: [codec] Updated Opus draft

Hi Anisse,

<snip>

Cheers,

	Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 11:29:20 +0000

Dear Jean-Marc,

Thanks for the clarifications

> narrowband speech to 256 kb/s (and up) full-band stereo music.

In G.729.1, things are clear and well separated, A backward interoperability requirement to G.729 and the desire to offer wideband speech coding. What makes me confused is the overlap between the various modes of the Opus codec, there are currently 32 modes (x2 for stereo), and it is not clear to me what range of bitrates these modes operate on. Is the range of 6kb/s to 256kb/s supported by all modes? You have in the past proposed a change of the mode names to distinguish them from the actual codecs, this is something I would support, are you still pursuing that ?

> also working on improving mode switching.

Regarding mode switching, I think this is quite an important functionality to support, we have tried to simulate such aspects and the results were not very good. 

> should make sure it works.

Thanks for that, looking forward to it.
> more info would be useful so we can fix it).

I will wait for the VS project files before escalating this to you. 

On the encoder being non-normative, I am not sure what is the state of consensus on this issue... I see however a great deal in leaving this completely open and in general about the conformance criteria that the WG plans to set for the codecs developed here. 

Conformance criteria are instrumental and an integral part of the standard. An implementation of the Opus codec can only CLAIM to be conformant to the Opus standard if it passes such criteria. My understanding of this activity is that an essential part and reason of this standard is to be royalty-free. Therefore, imho, I believe that conformance should also strive to keep the various conformant implementations *unencumbered*. Without explicitly stating this, the door is basically wide open for Opus implementations being conformant to Opus as specified by the IETF while being encumbered.

Looking at draft-valin-codec-guidelines-08 section 4, I see that there is no mention of this at all and the focus is only on deviation criteria, leaving the encoder totally open is an exacerbation of this. A good illustration of this is the various MDCT implementations some of which (complexity optimized) are encumbered and would pass whatever criteria one can set...

Are we going down the path that would allow Opus implementations to be *encumbered* ? wouldn't this contradict the objectives of this effort? 
 
> possible, but that's certainly something we can discuss.

For a fixed point DSP implementation, it would certainly help to have a quick way of accessing/extracting the fixed point code from the distribution. I however understand that during the development phase, this may not be convenient...Something to discuss when we reach the final release stages.
 
> have a purely fixed-point build for platforms with no (or slow) FPU.

Sorry about the confusion, what I meant is that one could get an estimate of the cost of a fixed point implementation from floating point code. If the fixed point code is available then there may be no need for it.
> counters. 

About the counters, good to know that this exists. Is that part of the 0.9.1 release (sorry been lazy to check this part). It would be good to map these counters to something close enough to the ITU-T counters. As I said, this would allow a fair complexity comparison with other codecs.
 
> In terms of comparing with codecs like AMR/AMR-WB, correct me if
> (non-fractional multiplications, non-saturating add, ...).

In the old days, this was correct, the library has however been updated several times since. For instance, non-saturating add is supported:

L add c(L v1, L v2)
Performs the 32-bit addition with carry. No saturation. Generates carry and overflow values.
The carry and overflow values are binary variables which can be tested and assigned values.

I am not sure however if all operations as used by Opus are supported. I think it would be a good idea that a discussion on the needs of IETF in this respect is initiated with ITU-T SG16/Q10.

Kind regards.
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 11:46:12 +0000

Dear Koen,

Thanks for your efforts, looking forward to the update. I will try the switch FIXED_POINT with our VS homemade project files and will report if we encounter problems.

Kind regards,
/Anisse

> From: Koen Vos [mailto:[email-masked]]
> Sent: Tuesday, March 01, 2011 11:57 PM
> To: [email-masked]
> Cc: Anisse Taleb
> And yes, I've been using Visual Studio without problems, and I agree with
> you we should have all the correct project files in the GIT repository.
> 
> Switching from floating-point to fixed-point code is as simple as defining
> "FIXED_POINT" during compilation (at least in VS), but this should be more
> clearly documented.
> 
> best,
> koen.
> 
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 02 Mar 2011 07:30:37 -0500

Hi Anisse,

On 11-03-02 06:29 AM, Anisse Taleb wrote:
> In G.729.1, things are clear and well separated, A backward
> interoperability requirement to G.729 and the desire to offer
> wideband speech coding. What makes me confused is the overlap between
> the various modes of the Opus codec, there are currently 32 modes (x2
> for stereo), and it is not clear to me what range of bitrates these
> modes operate on. Is the range of 6kb/s to 256kb/s supported by all
> modes?

No, you need all three modes to cover that range. SILK is useful 
(roughly) in the 6-32 kb/s range for mono speech up to wideband. The 
hybrid mode is what makes Opus scale to full-band speech and is useful 
in the 24-64 kb/s range (mono speech). The CELT mode is useful mostly 
for music and for applications that require delay below 10 ms. Its 
useful range is around 32 kb/s (mono) to 256 kb/s (stereo music) and 
beyond. So essentially, we need all three complementary modes to cover 
all that bitrate space.

> You have in the past proposed a change of the mode names to
> distinguish them from the actual codecs, this is something I would
> support, are you still pursuing that ?

Yes. SILK vs hybrid vs CELT are "internal names" and should not be 
"user-visible". We are considering simply using "voice mode" to describe 
both the SILK mode and hybrid mode and "audio mode" to describe the 
CELT-only mode.

> Regarding mode switching, I think this is quite an important
> functionality to support, we have tried to simulate such aspects and
> the results were not very good.

This is something we are working on. We are *not* trying to replicate 
USAC/AMB-WB+ here and allow frequent switching, but we want to support 
occasional switching without noticeable artefacts.

> On the encoder being non-normative, I am not sure what is the state
> of consensus on this issue... I see however a great deal in leaving
> this completely open and in general about the conformance criteria
> that the WG plans to set for the codecs developed here.

The latest draft (-03) includes a tool to check for decoder conformance 
in appendix B. Encoder is much more tricky and I'm not sure how one can 
define encoder conformance without at least a listening test. I'm not 
aware of any conformance test for MPEG codecs like MP3 and AAC, but I 
could be wrong because I'm not that familiar with these standards.

> Conformance criteria are instrumental and an integral part of the
> standard. An implementation of the Opus codec can only CLAIM to be
> conformant to the Opus standard if it passes such criteria.

The current draft defines conformance only based on the decoder. It's 
described in section 6 (which obviously needs more work).

> My
> understanding of this activity is that an essential part and reason
> of this standard is to be royalty-free. Therefore, imho, I believe
> that conformance should also strive to keep the various conformant
> implementations *unencumbered*. Without explicitly stating this, the
> door is basically wide open for Opus implementations being conformant
> to Opus as specified by the IETF while being encumbered.

Well, I'm not sure how one would prevent encumbered implementations. At 
the same time, even though I'd rather see implementations being 
unencumbered, I'm not sure having someone create his own encumbered 
implementation would be that bad. Once the decoder is well-specified and 
the reference encoder is good enough, then I think we have achieved the 
goal of inter-operability with an unencumbeder format.

> Looking at draft-valin-codec-guidelines-08 section 4, I see that
> there is no mention of this at all and the focus is only on deviation
> criteria, leaving the encoder totally open is an exacerbation of
> this. A good illustration of this is the various MDCT implementations
> some of which (complexity optimized) are encumbered and would pass
> whatever criteria one can set...

I don't see how one would prevent that. Even for a bit-exact codec like 
AMB, one could still patent a way to produce the same result with a 
patented technique. At the same time, as I said I don't see this as a 
disaster -- in practice it's not that different from someone writing an 
optimised unencumbered implementation and not releasing the source code 
(yes, I understand the difference between patents and copyright).

In general, do you know what the others are doing about encoder 
conformance (if anything). For example, what does MPEG say about the 
MP3/AAC encoders? Or even what does the ITU say about H.264 encoders?

> Are we going down the path that would allow Opus implementations to
> be *encumbered* ? wouldn't this contradict the objectives of this
> effort?

I don't think it would contradict what we're doing. That is, as long as 
our reference encoder is "the best encoder we have at this time", rather 
than "just a code sample we provide while keeping the good encoder for 
ourselves" as I understand was the case with at least the MP3 reference 
encoder.

> For a fixed point DSP implementation, it would certainly help to have
> a quick way of accessing/extracting the fixed point code from the
> distribution. I however understand that during the development phase,
> this may not be convenient...Something to discuss when we reach the
> final release stages.

I haven't had time to do so, but was planning on providing a simple 
fixed-point Makefile. There could also be a list of the required files 
for float and for fixed-point.

> About the counters, good to know that this exists. Is that part of
> the 0.9.1 release (sorry been lazy to check this part). It would be
> good to map these counters to something close enough to the ITU-T
> counters. As I said, this would allow a fair complexity comparison
> with other codecs.

Yes, they are part of the 0.9.1 release, but we haven't yet "unified 
them" for CELT and SILK. For CELT, they are defined in the fixed_debug.h 
file. When compiling with FIXED_DEBUG (in addition to FIXED_POINT), the 
"counting/debugging" operators are turned on rather than the "optimized" 
ones.

> In the old days, this was correct, the library has however been
> updated several times since. For instance, non-saturating add is
> supported:

Yes, I'm aware of the update. I was just saying that it may not be an 
"Apples to apples" comparison if we were using the latest version and 
AMR was using the old version (or is there backward compatibility?).

Cheers,

	Jean-Marc

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 2 Mar 2011 07:40:07 -0500

I agree that conformance testing is critical, my understanding is that this
would be done with a single reference encoder.  Hopefully that is the
case???

It would also be good if there was a way to qualify encoders against a
baseline quality standard.

...
> In the old days, this was correct, the library has however been updated
> several times since. For instance, non-saturating add is supported:
>
> L add c(L v1, L v2)
> I am not sure however if all operations as used by Opus are supported. I
> think it would be a good idea that a discussion on the needs of IETF in this
> respect is initiated with ITU-T SG16/Q10.
>

Q10 is meeting right before IETF-80.  So if this is desired, I would suggest
that the chairs send the ITU-T a suitable liaison right away.  Then it is
possible that a reply could be received before the Prague meeting.

Otherwise it will be very slow - the next Q10 meeting is in November right *
after* IETF-82.  So the reply wouldn't be available until IETF-83 (unless
there is an intervening Q10 Rapporteur's meeting).

Stephen Botzko

---

From: Erik Norvell <[email-masked]>
Date: Wed, 2 Mar 2011 14:39:24 +0100

Hi all,

Since the guidelines state we should not provide a rubber stamp for existing codecs, perhaps a good solution would be to include the baseline codecs in the listening test and mandate that the standardized modes to be better than the baseline.

Best regards,
Erik 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anisse Taleb
Sent: den 1 mars 2011 17:47
To: Jean-Marc Valin; [email-masked]
Subject: Re: [codec] Updated Opus draft

Dear Jean-Marc, all.

#ifdef _WIN32

- The code does not provide any *working* VS project files, while I understand the desire to stick to open source tools, I think providing the means to compile the code using visual studio or other compilers would 1) engage a wider developer community 2) allow to check how stable the implementation is when ported/compiled in a different platform. From my experience, besides being a good practice, this can also reveal some bugs that are not easily apparent if the code is consistently developed under a single platform. I believe express versions are now free for use, I have tried to make such a project file and after many attempts including removal of debug code etc...the code finally compiled and after many crashes, it could finally be run (the list of changes needed is too long for this email). The waveform output with the VS version deviated by about 16 dB wrt. the cygwin version which is imho quite substantial and would likely result in the codec not passing the conformance
  tests you proposed. This observation reinforces the need to have the code built by different compilers and to make sure that the code is stable enough.

Example :

and 

Kind regards,
/Anisse

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 02 Mar 2011 08:42:11 -0500

On 11-03-02 07:40 AM, Stephen Botzko wrote:
> I agree that conformance testing is critical, my understanding is that
> this would be done with a single reference encoder.  Hopefully that is
> the case???

Is by that you mean using the reference encoder to generate 
"testvectors" and then using those to evaluate conformance of a decoder, 
then that's definitely the case.

> baseline quality standard.

I'm not against, but I'm also unsure how. Any suggestions?

	Jean-Marc

>     I am not sure however if all operations as used by Opus are
>     supported. I think it would be a good idea that a discussion on the
>     needs of IETF in this respect is initiated with ITU-T SG16/Q10.
>
>
> Q10 is meeting right before IETF-80.  So if this is desired, I would
> suggest that the chairs send the ITU-T a suitable liaison right away.
> Then it is possible that a reply could be received before the Prague
> meeting.
>
> Otherwise it will be very slow - the next Q10 meeting is in November
> right _after_ IETF-82.  So the reply wouldn't be available until IETF-83
> (unless there is an intervening Q10 Rapporteur's meeting).
>
> Stephen Botzko
>
>
>

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 2 Mar 2011 10:11:41 -0500

On Wed, Mar 2, 2011 at 8:42 AM, Jean-Marc Valin <[email-masked]> wrote:

> Is by that you mean using the reference encoder to generate "testvectors"
> and then using those to evaluate conformance of a decoder, then that's
> definitely the case.
>
> Yes that was what I meant.  Also, we'd need to archive both the reference
encoder version and the audio sources that were encoded into "testvectors".
This should also allow folks who are simply porting the reference encoder to
validate their implementation.

> I'm not against, but I'm also unsure how. Any suggestions?
I'm unsure also.  I am wondering if an informational RFC that links to a
good variety speech/music samples might be part of the puzzle.  I recall
that someone offered access to their audio sample library (not sure who that
was though).  The RFC might include noisy and reverberant speech, also
samples from different languages / different genres of music.  The idea is
to encourage cross-checks against a wide range of material, including some
stuff that is hard to handle.

Then perhaps some guidance to encoder developers on useful subjective tests
that would hopefully ensure that their improvements did not have negative
side effects.  If suitable objective tests were found that could be also
referenced, they could also be included?  This would all be voluntary of
course, since we are not providing a certification program.

Any other thoughts?

---

From: David Virette <[email-masked]>
Date: Wed, 02 Mar 2011 17:32:59 +0100

Dear all,
If I understand correctly, we need the three modes to cover the full range
of operating modes from low bit rates (starting at 6 kbps) up to high bit
rate (256 kbps for stereo). My concern is that for the intermediate bit rate
and especially at 32 kbps, according to your tests or personal opinion, the
three modes can be used. This does not really look as a unified codec, but
more like two codecs put together with some glue to obtain a third codec.
Moreover, from our tests, the three modes cannot seamlessly switch to each
other without any artifacts. I assume that the three modes will not provide
the same quality depending on the characteristics of the input signal (clean
speech, speech with background noise, music), the real value of merging
several modes should be to always guarantee that the optimal quality can be
achieved by the decoder, the current framework does not seem to reach this
target. 
You mentioned that both SILK and CELT have been adapted to fit in this
single framework, does it mean that they are not compatible with what were
SILK and CELT before this activity? 

I find interesting the idea of testing the OPUS framework against the
baseline codecs (SILK and CELT). This would clearly give an idea of the
progress made in this joint development activity.  To paraphrase Aristotle:
"The whole [should be] greater than the sum of its parts". BTW, is there an
existing document stating the terms of reference for the IETF codec? What
are the codecs used as quality reference in order to demonstrate the value
of the joint effort?
Best regards,
David

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Erik Norvell
Sent: mercredi 2 mars 2011 14:39
To: [email-masked]
Subject: Re: [codec] Updated Opus draft

Hi all,

Since the guidelines state we should not provide a rubber stamp for existing
codecs, perhaps a good solution would be to include the baseline codecs in
the listening test and mandate that the standardized modes to be better than
the baseline.

Best regards,
Erik 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Anisse Taleb
Sent: den 1 mars 2011 17:47
To: Jean-Marc Valin; [email-masked]
Subject: Re: [codec] Updated Opus draft

Dear Jean-Marc, all.

Thanks for the update. I am not sure if my understanding of the scope of
this work is still up-to-date, but wasn't the idea of making the opus codec
to actually combine the best and strongest of the technologies of SILK and
CELT resulting in a single codec? At least according to the LS to ITU-T SG
16 "The experts got together, took the best ideas from each of their
proposals, and combined them to form a new codec."

As it stands out from the code (and the draft), the Opus codec is in fact
the SILK codec and CELT codec glued together by the so-called "hybrid"
codec. Correct me if I am wrong, but it feels like we started with two
codecs and now we are standardizing 3 codecs? Calling them modes will not
hide the fact that Opus is a high level merging of two independent codecs.
In-band signaling of the codecs as modes to achieve (perhaps?)
interoperability while marketing the package as a new codec with a new name
is something I find very distressing. Any clarifications would be greatly
appreciated. 

- It would be good to list what functionalities are currently supported. The
README file provides very little in that respect, so does the draft, and
unless we dig deep in the code, I am not sure what the codec can or cannot
do. 

- The code did not compile on my cygwin box (0.9.0 does), I think anyways
that this is a minor bug:

#ifdef _WIN32

    return (unsigned long)((1000000*(lpPerformanceCount.QuadPart)) /
lpFrequency.QuadPart); }

A quick fix is to move the conditional includes prior to the #ifdef _WIN32
such that <sys/time.h> is actually included.

- The code does not provide any *working* VS project files, while I
understand the desire to stick to open source tools, I think providing the
means to compile the code using visual studio or other compilers would 1)
engage a wider developer community 2) allow to check how stable the
implementation is when ported/compiled in a different platform. From my
experience, besides being a good practice, this can also reveal some bugs
that are not easily apparent if the code is consistently developed under a
single platform. I believe express versions are now free for use, I have
tried to make such a project file and after many attempts including removal
of debug code etc...the code finally compiled and after many crashes, it
could finally be run (the list of changes needed is too long for this
email). The waveform output with the VS version deviated by about 16 dB wrt.
the cygwin version which is imho quite substantial and would likely result
in the codec not passing the conformance
  tests you proposed. This observation reinforces the need to have the code
built by different compilers and to make sure that the code is stable
enough.

- Following a particular programming style will greatly help developer to
read and understand source code, and help to avoid introducing errors. I am
not sure what coding conventions have been used, but the coding style is
indeed different between the silk part and celt part. Notably, how come
there are two type definitions, one for SILK (SKP_Silk_typedef.h) and one
for CELT (celt_types.h), how does this help the developer ? 

Example :
#define SKP_int         int                     /* used for counters etc; at
least 16 bits */
#define SKP_int64       long long
#define SKP_int32       int
#define SKP_int16       short
#define SKP_int8        signed char

and 

besides the names being unnecessarily different, one uses a #define while
the other uses a typedef. I am not advocating the use of one or the other,
for that, please have a look at http://c-faq.com/decl/typedefvsdefine.html
but that a convention is kept consistent across the Opus code.

Some other fundamental issue with the code is the separation between
floating and fixed point code. I am not sure where it all begins and where
it ends and if it works at all. It would be good to have the distribution of
Opus clearly distinguish between the two implementations.

- A final comment on the complexity of the codec, ITU-T SG16 maintains a
library of fixed point operators as well as tools to estimate the
*fixed-point* complexity of floating point operations, this library is known
as ITU-T Software Tool Library 2009, is maintained by Q10 and is part of
ITU-T Rec. G.191. In my experience, these tools provide a very good estimate
of the complexity of codecs in terms of WMOPS and has been instrumental in
comparing the complexity of various codecs. I would strongly recommend using
it for this exercise, for the simple reason that it would reliably provide
the complexity of the Opus codec in relation to well known and widely
deployed codecs such as AMR, AMR-WB, G.719, G.729(.1), etc.

Kind regards,
/Anisse

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 2 Mar 2011 11:33:55 -0500

On 11-03-02 10:11 AM, Stephen Botzko wrote:
> This should also allow folks who are simply porting the reference encoder
> to validate their implementation.

Agreed. Considering the limitations of 72-column ASCII, my suggestion would 
be to host the file somewhere else, but include the md5/sha1 sums of the 
audio/testvectors in the draft.

> course, since we are not providing a certification program.

As long as we don't intend to have a hard compliance test for the encoder, 
then I think objective measurements like PEAQ/PESQ or even a modified 
version of the decoder compliance tool could be useful to establish 
guidelines. In this case, someone merely optimising the encoder by using 
approximation would be able to check that (e.g.) PEAQ does not decrease, 
while someone making perceptual improvements would likely just ignore the 
PEAQ score because it's not reliable for these kinds of changes.

	Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 16:36:50 +0000

Dear Jean Marc,
> beyond. So essentially, we need all three complementary modes to cover

I think the ranges or "sweet spots" of bitrates need to be documented somewhere. I prefer that these ranges should rather be the result of characterization tests than (rough) guesses about where it is believed that a certain mode is exhausted and where the other mode should be used. 

> occasional switching without noticeable artefacts.

I think this needs to be tested and the frequency of switching need to be motivated by real application and would depend on the crossover bitrates of the different modes "sweet spots". Do you have a view on how occasional is the occasional switching? 1Hz? 

> implementation would be that bad. 

I am not sure I would like to see several Opus ports floating around with an even more unclear licensing situation. We have a good opportunity and it is in our guidelines to make sure that the result of this work is as unencumbered as possible. What good is it to have a *certified/conformant* Opus codec that is encumbered? 

> Once the decoder is well-specified and

Is the purpose of the WG to define a CODEC or a FORMAT...any views from the chairs?

> patented technique. 

I guess you meant AMR, if that is the case, AMR never claimed or had as a target to be a royalty free codec. On the other hand, for this activity, the rules of the game are different and we need to be careful into not misleading users by ensuring that the Opus codec in all its forms is and remains as unencumbered as possible. 

> MP3/AAC encoders? Or even what does the ITU say about H.264 encoders?
For encoder conformance, MPEG does nothing except checking that the format is a legal bitstream. MPEG codecs have been traditionally for broadcasting applications and therefore decoders conformance is stricter than that of encoders which are allowed to be of higher complexity (server side encoding). In 3GPP, the situation is different and codecs are defined as bitexact. The only instance of an encoder conformance which is strict, but not bitexact is AMR-WB+, I will get back to you on that.

Kind regards,
/Anisse

---

From: Koen Vos <[email-masked]>
Date: Wed, 2 Mar 2011 17:56:44 +0100 (CET)

> SILK and CELT before this activity? 

Neither is compatible.  Both have undergone substantial changes.

> progress made in this joint development activity.

Adding SILK and CELT as baselines to the listening test is certainly possible.
The largest quality advantage for Opus is in the Hybrid mode, which I'm convinced hits the sweet spot for most VoIP applications.

On the SILK side we've spent the last year mostly on:
1. Integration with CELT (range coder, bitrate distribution, etc)
2. Reducing ROM table sizes, several times
3. Reducing complexity by 10~20%
The last two were done primarily after feedback from the IETF community.

best,
koen.

----- Original Message -----
From: "David Virette" <[email-masked]>
To: "Erik Norvell" <[email-masked]>, [email-masked]
Sent: Wednesday, March 2, 2011 8:32:59 AM
Subject: Re: [codec] Updated Opus draft

Dear all,

-----Original Message-----

Hi all,

Best regards,
Erik 

-----Original Message-----

Dear Jean-Marc, all.

#ifdef _WIN32

Example :

and 

Kind regards,
/Anisse

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 2 Mar 2011 12:08:02 -0500

Hi David,

> more like two codecs put together with some glue to obtain a third codec.

I actually think having a *small* amount of overlap between the modes is a 
good thing because it reduces the need for mode switching. That overlap is 
still relatively small though, but it's important for some use cases. For 
example, if we want a rate of 32 kb/s and the original source is wideband 
(16 kHz), then there is no point in using the hybrid mode, which does 
super-wideband and full-band. In that sense, the three modes are really 
complementary, rather than "competing".

> other without any artifacts.

That is something we're currently working on. The git version already has 
some improvements and we're working on an even higher quality switching 
involving some redundant information (it will still be possible to have 
decent quality even without the extra information). Also, note that the 
only transitions that are "problematic" are the ones between the voice mode 
(SILK or hybrid) and the audio mode (CELT). Transitions between between 
SILK and hybrid mode, as well as changes in audio bandwidth and/or frame 
size are already handled with no artefact. Once we make these changes I 
mentioned, then switching between all modes should work.

Note that in general we still haven't defined the final terminology. We 
have these three "low-level modes" (SILK, CELT, hybrid) that can be thought 
of as two "high-level modes" (voice, audio). Then for each of those, we 
have different combinations of bandwidths and frame sizes, which mean that 
we have a total of 32 configurations that can be used and switched dynamically.

> I assume that the three modes will not provide

In the current state, Opus was not designed with the intent of having 
frequent mode switching like AMR-WB+ or USAC. Rather, it would be mostly up 
to the application to select voice vs music and the encoder could select 
(potentially dynamically) the best configuration based on the bit-rate. 
Note also that CELT is actually pretty efficient on speech (especially with 
Raymond's Chen's prefilter/postfilter), at at any bit-rate, it's speech 
quality will be equal or higher to its music quality. It's speech quality 
is not as high as SILK's, but when there is a mix of speech and music, the 
CELT mode is appropriate and will not make the voice sound worse than the 
music.

> SILK and CELT before this activity?

None of SILK or CELT is even close to being compatible with the original 
versions that existed at the time we started this effort (or the time the 
WG was formed). Just on the CELT side, there have been about 590 commits 
since the working was formed (about half of which impacted the bit-stream), 
including many changes to make frame size/bandwidth switching and the 
hybrid mode possible. On the SILK side, I don't know all the details, but I 
know Koen replaced its entropy coder with CELT's entropy coder (required 
for hybrid mode), removed support for super-wideband (which is now handled 
by the hybrid mode) and significantly reduced the size of many tables (I'm 
sure there's more I forgot). So this was certainly not an effort of 
rubber-stamping existing codecs, but rather a joint effort to create 
something that was better than the original proposals. The fact that a 
SILK/CELT structure still remains is mainly due to convenience in the 
development and the fact that the "signal" models for these two parts are 
different (linear prediction, vs MDCT).

> are the codecs used as quality reference in order to demonstrate the value

That's part of the requirements document: 
http://tools.ietf.org/html/draft-ietf-codec-requirements-02

This is the relevant part:

    In terms of
    quality vs bit-rate, the codec to be developed must be better than
    the currently available codecs that satisfy the IPR requirements in
    the guidelines document, which are:
    o  For narrowband: Speex (NB), GSM-FR, and iLBC(*)
    o  For wideband: Speex (WB), G.722, G.722.1(*)
    o  For super-wideband: Speex (UWB), G.722.1C(*)
    The codecs marked with (*) do not meet all the licensing guidelines,
    but the codecs to be developed should still not perform significantly
    worse.

In terms of the whole being better than the sum of the parts, I think we 
demonstrated this pretty clearly in this presentation we made in 
Maastricht: http://www.ietf.org/proceedings/78/slides/codec-0.pdf (see 
slide 8). Essentially, we found that the hybrid mode at 32 kb/s (full-band) 
significantly out-performed all other codecs, including not only SILK and 
CELT, but also G.719.

Cheers,

	Jean-Marc

> #include<windows.h>     /* timer */
> #else   // Linux or Mac
> #include<sys/time.h>
> #endif
>
> such that<sys/time.h>  is actually included.
>

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 02 Mar 2011 12:09:33 -0500

> I think this needs to be tested and the frequency of switching need to be motivated by real application and would depend on the crossover bitrates of the different modes "sweet spots". Do you have…

I have been developing a patchset for seamless mode switching [1], and
produced a test sample that switches between CELT and Hybrid modes at 10
Hz without a major loss in quality [2].  This particular test sample is
deeply unrealistic in myriad ways, but it demonstrates that the patches
function as expected.

This feature has not yet been accepted into the Opus bitstream.  I hope
that it will be accepted before the final freeze.  If it is accepted, a
future encoder could change modes to suit the signal while maintaining
deterministic decoder behavior.

--Ben Schwartz

[1] http://bemasc.net/~bens/celt/
[2] http://bemasc.net/~bens/celt/cm10.wav

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 2 Mar 2011 12:14:47 -0500

> deterministic decoder behavior.

I would just like to confirm that this is the proposal we will be 
implementing for Opus and what I was referring to earlier when I said that 
we were going to add support for seamless switching.

	Jean-Marc

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Wed, 02 Mar 2011 12:17:49 -0500

> I am not sure I would like to see several Opus ports floating around with an even more unclear licensing situation. We have a good opportunity and it is in our guidelines to make sure that the resu…

I don't believe this is a realistic goal; the only goal that the WG can 
have is produce a specification which, if implemented, does not obligate 
the implementor to deal with any encumbrances just for having 
implemented it.

Implementers are free to license their implementations any way they wish 
to do so (that is compatible with the license of the specification, but 
since that is fairly easy to deal with, won't be an issue). At a 
minimum, anyone who implements Opus by writing their own code has 
copyright interest in that code and distribution of that implementation 
is encumbered by that interest, subject to any licensing terms they 
distribute it under. As Jean-Marc said, it's even possible that an 
implementor might produce an implementation that contains a novel 
technique and is encumbered by patent rights, but as long as that 
implementation is compatible with other implementations, nobody is 
obligated to use it.

The underlying goal of the WG's efforts regarding encumbrance is to 
avoid any user being *forced* to accept encumbrances; if the 
specification is free of them, then there will be least one 
implementation that is also free of them. If there are additional 
implementations that are not free of them, that's fine; users can choose 
based on their requirements and tolerance for dealing with those issue.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
skype: kpfleming | jabber: [email-masked]
Check us out at www.digium.com & www.asterisk.org

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 2 Mar 2011 09:33:50 -0800

> I think the ranges or "sweet spots" of bitrates need to be documented somewhere. I prefer that these ranges should rather be the result of characterization tests than (rough) guesses about where it…

Absolutely.

I was hoping to see people posting test results to the working group after we released test versions of the tools.  These results could be used to further characterize behaviour.  My expectation that people other than the core developers would be performing some testing is why I haven't published any of my informal testing between the modes.

In another post you wrote:
> What makes me confused is the overlap between the various modes of the Opus codec, there are currently 32 modes (x2 for stereo), and it is not clear to me what range of bitrates these modes operate on.

The 32 modes are the permutation of the codec mode (Linear-predictive, hybrid, or Transform),  the frame duration (2.5ms, 5ms, 10ms, 20ms, 40ms, 60ms), and bandpass (NB,MB,WB,SWB,FB). Some combinations are impermissible, so there are 32 rather than 90.  

The purpose of the bandpass is for matching the codec to different channel types/sampling rates (as per the requirements documents) and the frame duration provides an essential latency/efficiency trade-off.  These can currently switched without any  artifacts.

Only the codec mode (Linear-predictive, hybrid, or Transform) results in any significant overlap,  and that is only in the range of ~20-40kbit/sec (I'm being somewhat vague because this should be testing validated rather than just a developer impression). Generally the Hybrid mode wins on speech, the pure-transform mode on music. 

I don't share the concern about the functional overlap.  Many formats provide several ways for an encoder to code an output, leaving the encoder to choose based on it's own rate/distortion/computational-complexity trade-off.

Too much overlap adds implementation complexity and wastes signalling bandwidth. But where the overlap comes purely from signalling that can't be easily avoided and implementation complexity which must exist in any case (e.g. Transform mode is not acceptable for 6kbps speech,  and LP mode is not workable for 5ms 300kbit/sec stereo music)  then the overlap is no harm and can provide benefits.

In particular, overlap means that you are not required to switch modes as frequently.  Even absent the current gap with mode switching, the ability to avoid mode switches is important because changing modes changes the noise character and may produce its own audible problems if forced.  

So, an encoder is free to generate 1.2kbps transform mode packets  which is probably not a reasonable output. However, foreclosing such possibilities  would increase the format complexity rather than decrease it and leaving the option leaves an opportunity for future encoders which are smarter (e.g. by using far more CPU than we've targeted as our main focus) without breaking compatibility or imposing on the receiver. 

> I think this needs to be tested and the frequency of switching need to be motivated by real application and would depend on the crossover bitrates of the different modes "sweet spots". Do you have…

Ben Schwartz has some examples of the work he's been doing on switching.  I think there were ones at 0.5 Hz and 10Hz.  They sounded fine ('fine' in the sense of no audible artifacts). He should post some examples.

> I am not sure I would like to see several Opus ports floating around with an even more unclear licensing situation. We have a good opportunity and it is in our guidelines to make sure that the resu…

I think it's not at all clear to me what you're requesting.   Someone could always add something encumbered to an implementation, and this would be true even if we had a bit exact specification, and that something could be encumbered.

So I think that short of demanding that conforming "implementations" be the reference implementation itself (and perhaps not even then) what I understand you to be asking is just not a possibility.

Even if it were somehow possible to prevent an implementation from containing additional encumbered technology, I don't see why it would be particularly desirable.  Take a look at section "5. Intellectual Property" of the working group guidelines.   By my reading the focus there is  on having the option to have "internet friendly" licensing available, with full quality, to anyone who wants it.  This is true so long as there exists at least one complete (encoder+decoder) implementation meeting the requirements.   I don't believe that anyone in the WG has promoted that the format itself be some kind of copyleft, or the like, preventing it from being used in non-freely licensed contexts. I certainly would not support that.

Perhaps I'm not understanding you?

> Is the purpose of the WG to define a CODEC or a FORMAT...any views from the chairs?

My understanding is that the product of this working group is both a format (which we often call a 'codec') _and_ a reference implementation.

The reference implementation is important both for validation and for making sure that an implementation with quality and performance meeting the working group requirements is available to everyone.

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 18:36:43 +0000

Dear Koen, 
Thanks for the clarification about the compatibility issue, we were under the impression that the SILK and CELT modes were backwards compatible to the original codecs, I guess the name change that Jean Marc proposed is really warranted to avoid confusion.
Kind regards,
/Anisse
________________________________________
From: [email-masked] [[email-masked]] on behalf of Koen Vos [[email-masked]]
Sent: 02 March 2011 17:56
To: David Virette
Cc: [email-masked]
Subject: Re: [codec] Updated Opus draft

best,
koen.

----- Original Message -----
From: "David Virette" <[email-masked]>
To: "Erik Norvell" <[email-masked]>, [email-masked]
Sent: Wednesday, March 2, 2011 8:32:59 AM
Subject: Re: [codec] Updated Opus draft

Dear all,

-----Original Message-----

Hi all,

Best regards,
Erik

-----Original Message-----

Dear Jean-Marc, all.

#ifdef _WIN32

Example :

and

Kind regards,
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 18:42:33 +0000

Dear Benjamin,

Thanks for the update. I am under the impression that development is being done outside this mailing list...looking forward to test this patch, can it be applied directly to 0.9.1?
Kind regards,
/Anisse
________________________________________
From: Benjamin M. Schwartz [[email-masked]]
Sent: 02 March 2011 18:09
To: Anisse Taleb
Cc: Jean-Marc Valin; [email-masked]
Subject: Re: [codec] Updated Opus draft

--Ben Schwartz

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 02 Mar 2011 10:45:24 -0800

> progress made in this joint development activity.  To paraphrase Aristotle:

Even just on the CELT side of things, there has been substantial 
improvement due to this WG's activities, thanks to things like Raymond's 
pitch filter and others.

http://people.xiph.org/~xiphmont/demo/celt/demo.html has samples encoded 
with every major version from the initial 0.1.0 up to 0.10.0 (last 
December, when that page was written). 0.5.2 is the last release before 
the WG activity started, for comparison.

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 02 Mar 2011 13:50:14 -0500

On 03/02/2011 01:42 PM, Anisse Taleb wrote:
> Thanks for the update.

No problem.

> I am under the impression that development is being done outside this mailing list...

Yes, we do not use this mailing list for the high-traffic work of
development.  That work occurs in #celt on irc.freenode.net, which is
public, and all are welcome to discuss code details there.

> looking forward to test this patch, can it be applied directly to 0.9.1?

Unfortunately testing the patches is not too easy, because they apply to
an intermediate development version of the code, which is spread across 3
different git repositories.  However, the functionality from these patches
will be integrated soon into the mainline codebase.

I recommend that you test the different modes separately for now, and wait
until the next release before testing mode switching.

--Ben

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 02 Mar 2011 11:02:51 -0800

> based on their requirements and tolerance for dealing with those issue.

This is basically my opinion as well. Back when the WG was being 
chartered, there was a great deal of opposition to some groups trying to 
impose their "business model" on others. As for me, my "business model" 
(and since moving to Mozilla last year, now also my employer's), is that 
it should be _possible_ to produce and ship an open-source 
implementation of the standard.

Now, someone else's business model may certainly be selling proprietary 
implementations of the public standard, using their own patented 
inventions, and a "GPL for patents" to prevent that is an appealing 
idea. I can argue that there is some harm to me by not being allowed to 
re-implement their invention while they themselves are profiting from 
our efforts in making the codec successful (assuming, for the moment, 
that we are successful), but it is certainly a much-reduced harm 
compared to not being allowed to use the technology I helped develop at 
all without greasing their palms. Mozilla's browser faces proprietary 
competitors who can and do file patents, despite the underlying 
standards being unencumbered, and we still do okay.

The only way I see to try to enforce such a regime is through the 
licensing of what patents there are against Opus (the other alternative, 
imposing requirements on the encoder in the standard that would preclude 
such improvements, is a cure worse than the disease). My impression is 
that such an attempt to "enforce our business model" on others would be 
seen as unwelcome by some here. However, I believe the consensus was 
also that this list is not the place for licensing discussions, so I 
won't go further than that.

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 19:14:37 +0000

Dear Kevin

>the implementor to deal with any encumbrances just for having

I do not disagree on this, but that is just one side of the story.

> based on their requirements and tolerance for dealing with those issue.

The issue is not about if an implementation is free of IPRs or not, but on how much control the IETF wants to exert on the various conforming implementations (especially open source software) to adhere to the guidelines we follow to develop this codec. 

If we want a successful, truly unencumbered codec, one idea is to have the conformance criteria add a clause of non-encumbrance, in the same spirit of open source code licenses. A relaxed version could be that if you introduce an encumbrance, the resulting work must clearly state what these are. A reference encoder of MPEG AAC audio codec is available from ISO, nobody is *forced* to use another implementation, yet everyone does...for the simple reason that any other implementation will deliver far better audio quality.

Not doing anything about this is basically to rubber stamp all conforming implementations (which may or may not contain encumbrances) with the IETF Opus name...I am not sure I like this as an outcome of this WG.

Kind regards,
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 19:20:12 +0000

Dear Gregory,
> 
> I was hoping to see people posting test results to the working group after
> we released test versions of the tools.  These results could be used to
> further characterize behaviour.  My expectation that people other than the
> core developers would be performing some testing is why I haven't published
> any of my informal testing between the modes.

At least from my side, if we are to do test, we need to clearly define what is to be tested. This will in any case be a formal listening test, using standardized methodologies and a controlled testing environment. But that is getting ahead of ourselves...what is needed right now is to clearly define requirements and objectives, formalize the processing procedure and derive a test plan. David and Erik proposed to compare the performance of the Opus codec to the CELT and SILK, which is something I agree with. A potential requirement  would be that the Opus codec shall exceed the best of the CELT and SILK at a given bitrate. We need to also compare Opus to state of the art codecs, each of these *state-of-art* will have an optimal operating range, hence, Opus will be compared to different codecs for a given bitrate.   

> I don't share the concern about the functional overlap.  Many formats
> provide several ways for an encoder to code an output, leaving the encoder
> to choose based on it's own rate/distortion/computational-complexity trade-
> off.

It all boils down to what we leave as open to optimization for a particular implementation and what we knowingly know (after testing) to be not an option. I still think that quality is a prime factor here, giving too much tweaking parameters is good for someone who knows what he is doing, but is very bad for a *naïve* user. It's basically like driving a car with an automatic gear box or a manual one...I like the latter. Providing the user with an interface that would select the best tradeoff between frame length, mode, sampling rate etc....for a given bitrate is something I would like to see in the codec. 

> Ben Schwartz has some examples of the work he's been doing on switching.  I
> think there were ones at 0.5 Hz and 10Hz.  They sounded fine ('fine' in the
> sense of no audible artifacts). He should post some examples.

Ben has posted a patch, not sure when it will be included in the distribution, in any case, we will test this and report, what we tested did not sound good (we had to align the waveforms to get something decent).

> 
> > I am not sure I would like to see several Opus ports floating around with
> an even more unclear licensing situation. We have a good opportunity and it
> is in our guidelines to make sure that the result of this work is as
> unencumbered as possible. What good is it to have a *certified/conformant*
> Opus codec that is encumbered?
> 
> I think it's not at all clear to me what you're requesting.   Someone could
> always add something encumbered to an implementation, and this would be
> true even if we had a bit exact specification, and that something could be
> encumbered.
> 
> So I think that short of demanding that conforming "implementations" be the
> reference implementation itself (and perhaps not even then) what I
> understand you to be asking is just not a possibility.
> 
> Even if it were somehow possible to prevent an implementation from
> containing additional encumbered technology, I don't see why it would be
> particularly desirable.  Take a look at section "5. Intellectual Property"
> of the working group guidelines.   By my reading the focus there is  on
> having the option to have "internet friendly" licensing available, with
> full quality, to anyone who wants it.  This is true so long as there exists
> at least one complete (encoder+decoder) implementation meeting the
> requirements.   I don't believe that anyone in the WG has promoted that the
> format itself be some kind of copyleft, or the like, preventing it from
> being used in non-freely licensed contexts. I certainly would not support
> that.
> 

I think I have not been clear on this, I apologize for this. Please check my reply to Kevin.

Kind regards,
/Anisse

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 02 Mar 2011 11:23:13 -0800

> Did you compare just the decoder or encoder+decoder. Because the encoder
> is not normative (and uses floating-point by default), it's normal that
> the encoder would produce different (but equivalent) output depending on
> the platform and compiler. It could also be a bug that you hit (in which
> case more info would be useful so we can fix it).

This also depends somewhat on the signal level. It is quite normal to 
see differences of 16 dB or more in SNR caused solely by a single 
off-by-one rounding error in a quiet frame. That is part of the reason 
the conformance test does not use simple SNR.

>> have the code built by different compilers and to make sure that the

Besides the platforms already mentioned (i.e., 32- and 64-bit Linux, and 
Windows), we have also done some testing with Open Watcom targeting 
real-mode DOS (as an easily accessible platform with a 16-bit int). This 
indeed helped uncover and fix some real issues. Other people have also 
done ports at various points to DSPs and other platforms (fmod, for 
example, targets at least 20 different compiler/platform combinations), 
and we welcome any feedback about what does and does not work on them.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 02 Mar 2011 11:33:01 -0800

> these are. A reference encoder of MPEG AAC audio codec is available from
 > ISO, nobody is *forced* to use another implementation, yet everyone
 > does...for the simple reason that any other implementation will deliver
 > far better audio quality.

A lot of people use the open-source faac encoder, even though, compared 
to commercial offerings, the quality is terrible (at least that is my 
understanding; I've never compared it myself, but they have a disclaimer 
right on their website). People make decisions based on a lot more than 
audio quality.

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 20:04:11 +0000

Dear Timothy,
> the conformance test does not use simple SNR.

This is not entirely true, I have in the past maintained codec source code for different platforms, where the SNR was kept always above 60 dB. This is an effort that has to be continuously checked...it all depends on the developer's discipline and how much effort you put on that. The best I have obtained between linux and windows was a +/- 1 difference in waveforms. Some tweaking of the compiler floating point operations is needed though, the byproduct of this is that we discovered many bugs some of which are serious.
 
> and we welcome any feedback about what does and does not work on them.

We will to the best of our abilities to feedback any findings.

Kind regards,
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 20:13:27 +0000

> right on their website). People make decisions based on a lot more than

I don't deny that free and open is always a good incentive for people. I personally don't use faac...Not sure this holds true for content creators, it all depends on what side of the market you are and how the licensing of the encoder (code) and the patents are packaged.

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 2 Mar 2011 12:15:49 -0800

> At least from my side, if we are to do test, we need to clearly define what is to be tested. This will in any case be a formal listening test, using standardized methodologies and a controlled test…

There are some broad conditions given in the requirements document about the kind of tests which must be performed.  The exact test plan is something the WG should be working on.  I haven't worked on this because I think it should be the responsibility of people other than the core developers to set the overall plan, though I will gladly comment on specifics.

It's fairly trivial to demonstrate that Opus is superior to either the transform mode or the LP mode alone (or those modes prior to the work done as part of the WG process)— E.g. at low rates the silk/hybrid modes are _obviously_ better (esp for speech) than the transform mode,  and at high rates the transform mode is clearly better (and can achieve lower latencies too).   The presentation of IETF-78 shows the hybrid outperforming SILK and CELT alone (http://tools.ietf.org/agenda/78/slides/codec-0.pdf) at some rate. Though the transform mode's performance at low rates was subsequently improved, the differences at low rate are still pretty stark. 

If you only pick a single given rate the performance might only be equal to one of SILK/CELT alone (well, ignoring the fact that both modes have been improved as part of the process here) if you've picked a rate where one or the other technique is inapplicable.   So rather than "at a given bitrate" if you say "at some relevant bitrates" it would be a more practical requirement.

> It all boils down to what we leave as open to optimization for a particular implementation and what we knowingly know (after testing) to be not an option. I still think that quality is a prime fact…

Yes.

We're on the same page on this point.  Implementations should be as automatic as is reasonably possible, and the format shouldn't do anything to impede automatic operation. 

The testing tool obviously exposes a greater configuration surface than a normal implementation would.   Otherwise you wouldn't be able to use it to make comparisons of different configurations or discover behavioural thresholds.

I'm mostly done with a rewrite of the testing tool, which makes the nature of the options a little more clear.  I'm also intending to allow it to read/write special bitstreams for conformance validation, but I haven't gotten around to implementing that part yet (which is what is currently delaying the release).

Below is the CLI output of my work in progress (use a monospaced font).  Notice that I'm still using "CELT",  it's not referring to the CELT codec but instead the name of the technique (Constrained Energy Lapped Transform).

Hopefully this will make it more clear that the user isn't expected to manually set a zillion options.

[gmaxwell@helmholtz src]$ ./test_opus 
Usage: ./test_opus [options] <input.wav/.bit> <output.wav/.bit>

Decoding settings:
  --rate          <n>  Set decoder sample rate   (default: input)
  --channels      <n>  Set decoder channel count (default: input)
  --loss          <n>  Simulate packet loss per 10^2 frames;  (default: 0)
  --biterror      <n>  Simulate payload errors per 10^4 bits; (default: 0)

Encoding settings:
  --mode          <n>  Force OPUS mode; (default: auto)
                      --mode list to get a list.
  --bitrate       <n>  Encoding bit-rate in kbit/sec; (default: 64*ch)
  --max-packet    <n>  Set the maximum packet size;   (default: 1260)
  --cbr                Use constant bitrate encoding
  --complexity         Set encoder complexity 0-10;   (default: 10)
 Robustness settings
  --inbandfec          Enable LPC inband forward error correction (FEC)
  --dtx                Enable LPC discontinuous transmission      (DTX)
  --independent        Use independant CELT frames
 Frame size settings  (Opus frame sizes: 2.5, 5, 10, 20, 40, 60)
  --max-framesize <n>  Set the maximum frame size in ms; (default: 60)
  --min-framesize <n>  Set the minimum frame size in ms; (default: 2.5)
 Bandwidth settings   (Opus bandwidths: NB, MB, WB, SWB, FB)
  --max-bandpass  <x>  Maximum audio bandwidth; (default: FB)
  --min-bandpass  <x>  Minimum audio bandwidth; (default: NB)
 --quiet              Only report errors

[gmaxwell@helmholtz src]$ ./test_opus --mode list
Allowed modes in OPUS:
auto       Automatically pick a mode
speech     Automatically pick a LPC/hybrid mode
music      Automatically pick a CELT mode
    or any number 0-31:
                   2.5ms  5ms 10ms 20ms 40ms 60ms
LPC     NB  (4kHz)               0    1    2    3
LPC     MB  (6kHz)               4    5    6    7
LPC     WB  (8kHz)               8    9   10   11
Hybrid SWB (16kHz)              12   13
Hybrid  FB (24kHz)              14   15
CELT    NB  (4kHz)    16   17   18   19
CELT    WB  (8kHz)    20   21   22   23
CELT   SWB (16kHz)    24   25   26   27
CELT    FB (24kHz)    28   29   30   31
[gmaxwell@helmholtz src]$

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 02 Mar 2011 12:41:21 -0800

> This is not entirely true, I have in the past maintained codec source
 > code for different platforms, where the SNR was kept always above 60 dB.

This depends a good deal on the frame size, and, as I said, the signal 
level. For example, an example I happened to have open on my desktop 
from my own debugging has two decoded copies of a 960-sample frame made 
up of 16-bit signed samples, with a peak level of 15. There is a total 
of 20 mis-matched samples (about 2%) between the two decoded versions, 
and the largest difference between any two samples is +/- 1. The SNR 
between the two is 28.3 dB. The _best_ you could possibly hope for on 
this frame that is not a perfect match (i.e., one mis-matched sample) is 
about 40 dB. Smaller frame sizes are worse. This is just how ratios work.

You could measure SNR over more than just a single frame or two, but 
that is basically useless for actually identifying errors.

 > This is an effort that has to be continuously checked...it all depends
 > on the developer's discipline and how much effort you put on that.
 > The best I have obtained between linux and windows was a +/- 1
 > difference in waveforms. Some tweaking of the compiler floating point
 > operations is needed though, the byproduct of this is that we discovered
 > many bugs some of which are serious.

I am well aware of the discipline required. See my earlier thread where 
I wrote an entirely separate decoder for the purpose of testing issues 
like these leading up to the (soft) bitstream freeze. I spent a lot of 
time tracking down and fixing places where these implementations 
disagreed, and indeed the bit-stream changes between the 0.9.0 and 0.9.1 
libopus releases were to fix a (rare) situation where the result of 
destructive cancellation was being scaled to unit norm, identified by 
Greg Maxwell during this testing (to say nothing of the many issues that 
we identified before 0.9.0).

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 02 Mar 2011 21:38:33 +0000

> that is basically useless for actually identifying errors.

Correct, it depends on the input signal, I may have overlooked to state that we have specific signals on which to measure the SNR for different implementations based on signals covering the whole dynamic range ~90dB. As I stated I will wait for the updated vs project files to rerun these experiments, hopefully we get something better.

---

From: Erik Norvell <[email-masked]>
Date: Thu, 3 Mar 2011 09:36:52 +0100

Hi Greg,

> 
> It's fairly trivial to demonstrate that Opus is superior to 
> either the transform mode or the LP mode alone (or those 
> modes prior to the work done as part of the WG process)- E.g. 
> at low rates the silk/hybrid modes are _obviously_ better 
> (esp for speech) than the transform mode,  and at high rates 
> the transform mode is clearly better (and can achieve lower 
> latencies too).   The presentation of IETF-78 shows the 
> hybrid outperforming SILK and CELT alone 
> (http://tools.ietf.org/agenda/78/slides/codec-0.pdf) at some 
> rate. Though the transform mode's performance at low rates 
> was subsequently improved, the differences at low rate are 
> still pretty stark. 
> 
> If you only pick a single given rate the performance might 
> only be equal to one of SILK/CELT alone (well, ignoring the 
> fact that both modes have been improved as part of the 
> process here) if you've picked a rate where one or the other 
> technique is inapplicable.   So rather than "at a given 
> bitrate" if you say "at some relevant bitrates" it would be a 
> more practical requirement.
> 

This supports standardization of the hybrid mode, which represents 4 out of the 32 modes. It seems reasonable that the remaining 28 modes are also checked against their corresponding baseline at the same rate/bandwidth.

The listening test from IETF78 has been referenced a few times. We should remember that it was presented with a disclaimer that the results were not that reliable and that the test should be seen more as an example of how it could be conducted.

Best regards,
Erik

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 3 Mar 2011 14:35:49 +0100

Hello,

> > Moreover, from our tests, the three modes cannot seamlessly switch to
> > each other without any artifacts.
> 
decent
> quality even without the extra information). Also, note that the only
> transitions that are "problematic" are the ones between the voice mode

[Christian Hoene] It is good to hear that switching between modes is going
to be supported. I also agree that mode switches will not be as frequent as
switching the bit, frame and/or sampling rate. 

I was just checking some variability requirements written down in
(http://www.ietf.org/id/draft-hoene-codec-quality-00.pdf, Sections 8.4 to
8.6).  Internet transmission paths show a high variability in transmission
delays such as delay spikes. Also, switching frame rate may also change the
algorithmic delay. Thus, it would be nice if the decoder playout could be
fastened or slowed down or if proper audio segments (e.g. pitch periods) may
be inserted or dropped.  Is anybody working on this?

Cheers,
 Christian

What is missing are 
> Note that in general we still haven't defined the final terminology. We
have
> these three "low-level modes" (SILK, CELT, hybrid) that can be thought of
as
> two "high-level modes" (voice, audio). Then for each of those, we have
> different combinations of bandwidths and frame sizes, which mean that we
> have a total of 32 configurations that can be used and switched
dynamically.
> 
> > I assume that the three modes will not provide the same quality
> > depending on the characteristics of the input signal (clean speech,
> > speech with background noise, music), the real value of merging
> > several modes should be to always guarantee that the optimal quality
> > can be achieved by the decoder, the current framework does not seem to
> > reach this target.
> 
> frequent mode switching like AMR-WB+ or USAC. Rather, it would be mostly
> up to the application to select voice vs music and the encoder could
select
> Raymond's Chen's prefilter/postfilter), at at any bit-rate, it's speech
quality
> will be equal or higher to its music quality. It's speech quality is not
as high as
> SILK's, but when there is a mix of speech and music, the CELT mode is
> appropriate and will not make the voice sound worse than the music.
> 
> > single framework, does it mean that they are not compatible with what
> > were SILK and CELT before this activity?
> 
WG
> was formed). Just on the CELT side, there have been about 590 commits
> for hybrid mode), removed support for super-wideband (which is now
> handled by the hybrid mode) and significantly reduced the size of many
> tables (I'm sure there's more I forgot). So this was certainly not an
effort of
> > baseline codecs (SILK and CELT). This would clearly give an idea of
> > the progress made in this joint development activity.  To paraphrase
> Aristotle:
> > "The whole [should be] greater than the sum of its parts". BTW, is
> > there an existing document stating the terms of reference for the IETF
> > codec? What are the codecs used as quality reference in order to
> > demonstrate the value of the joint effort?
> 
CELT,
> but also G.719.
> > Of Erik Norvell
> > Since the guidelines state we should not provide a rubber stamp for
> > existing codecs, perhaps a good solution would be to include the
> > baseline codecs in the listening test and mandate that the
> > standardized modes to be better than the baseline.
> >
> > Best regards,
> > Erik
> >
> > Of Anisse Taleb
> > Thanks for the update. I am not sure if my understanding of the scope
> > of this work is still up-to-date, but wasn't the idea of making the
> > opus codec to actually combine the best and strongest of the
> > technologies of SILK and CELT resulting in a single codec? At least
> > according to the LS to ITU-T SG
"hybrid"
> > codecs and now we are standardizing 3 codecs? Calling them modes will
> > not hide the fact that Opus is a high level merging of two independent
> codecs.
> > interoperability while marketing the package as a new codec with a new
> > name is something I find very distressing. Any clarifications would be
> > greatly appreciated.
> >
> > - It would be good to list what functionalities are currently
> > supported. The README file provides very little in that respect, so
> > does the draft, and unless we dig deep in the code, I am not sure what
> > the codec can or cannot do.
> >
> > A quick fix is to move the conditional includes prior to the #ifdef
> > _WIN32 such that<sys/time.h>  is actually included.
> >
> > would 1) engage a wider developer community 2) allow to check how
> > stable the implementation is when ported/compiled in a different
> > platform. From my experience, besides being a good practice, this can
> > also reveal some bugs that are not easily apparent if the code is
> > consistently developed under a single platform. I believe express
> > versions are now free for use, I have tried to make such a project
> > file and after many attempts including removal of debug code etc...the
> > code finally compiled and after many crashes, it could finally be run
> > (the list of changes needed is too long for this email). The waveform
output
> with the VS version deviated by about 16 dB wrt.
> > the cygwin version which is imho quite substantial and would likely
> > result in the codec not passing the conformance
> >    tests you proposed. This observation reinforces the need to have
> > the code built by different compilers and to make sure that the code
> > is stable enough.
> >
> > besides the names being unnecessarily different, one uses a #define
> > while the other uses a typedef. I am not advocating the use of one or
> > the other, for that, please have a look at
> > http://c-faq.com/decl/typedefvsdefine.html
implementations.
> >
> > - A final comment on the complexity of the codec, ITU-T SG16 maintains
> > a library of fixed point operators as well as tools to estimate the
> > *fixed-point* complexity of floating point operations, this library is
> > good estimate of the complexity of codecs in terms of WMOPS and has
> > been instrumental in comparing the complexity of various codecs. I
> > would strongly recommend using it for this exercise, for the simple
> > reason that it would reliably provide the complexity of the Opus codec
> > in relation to well known and widely deployed codecs such as AMR, AMR-
> WB, G.719, G.729(.1), etc.
> >
> >
> > Kind regards,
> > /Anisse
> >
> >> From: [email-masked] [mailto:[email-masked]] On
> >> Behalf Of Jean-Marc Valin
> >> available at
> >> http://downloads.xiph.org/releases/opus/opus-0.9.1.tar.gz
> >> .
> >>
> >> Because of some minor bugs we found in the previous release, we had
> >> to change the bit-stream slightly for 0.9.1. The quality should not
> >> be affected except for small improvements in rare cases. The release
> >> also adds support for forward error correction in SILK-only and
> >> hybrid modes.

---

From: Koen Vos <[email-masked]>
Date: Thu, 3 Mar 2011 15:56:35 +0100 (CET)

Hi Erik,

> This supports standardization of the hybrid mode, which represents 4 out of the 32 modes. It seems reasonable that the remaining 28 modes are also checked against their corresponding baseline at th…

Are you suggesting that to be added to a standard, each mode has to outperform (or perhaps be now worse than) its corresponding baseline?  I think such requirement is too strong, for multiple reasons.

1. The 28 non-hybrid modes enhance the standard, regardless of how they compare to their baselines:
- The non-hybrid modes greatly expand the useful operating range and flexibility.
- The non-hybrid modes come virtually for free.

2. The seamless switching between the modes, which everyone seems to agree is important, is made possible by having all modes in one codec.

3. It ignores improvements beyond rate/bandwidth/quality, such as the code size and complexity optimizations we did for SILK.

Checking modes against their baselines is of course a good sanity check during development, and we've been doing such tests extensively.  I welcome anyone to do more of them.  However, the sheer amount of listening necessary to compare all 28 non-hybrid modes against their baselines with sufficient statistical significance is large compared to the value it brings.

best,
koen.

----- Original Message -----
From: "Erik Norvell" <[email-masked]>
To: "Gregory Maxwell" <[email-masked]>, "Anisse Taleb" <[email-masked]>, "Jean-Marc Valin" <[email-masked]>
Cc: [email-masked]
Sent: Thursday, March 3, 2011 12:36:52 AM
Subject: Re: [codec] Updated Opus draft

Hi Greg,

Best regards,
Erik

---

From: Erik Norvell <[email-masked]>
Date: Fri, 4 Mar 2011 09:00:16 +0100

Hi Koen,

I appreciate the work being done on extending the codec functionality. However, I am not sure that it comes for free since the more technology is included the higher the risk of including third party IPR. If the codec functionality is already too wide to be covered by a listening test then the IPR analysis might also be lagging behind. 

Seamless switching between a wide range of operating points is a desirable feature, but the primary goal in the charter is still to produce an unencumbered codec. If the additional modes does not produce value over already existing codecs, it would be safer to leave them out.

Best regards,
Erik

> Sent: den 3 mars 2011 15:57
> To: Erik Norvell
> Cc: [email-masked]; Gregory Maxwell; Anisse Taleb; Jean-Marc Valin
> > This supports standardization of the hybrid mode, which 
> represents 4 out of the 32 modes. It seems reasonable that 
> the remaining 28 modes are also checked against their 
> corresponding baseline at the same rate/bandwidth.
> 
> Are you suggesting that to be added to a standard, each mode 
> has to outperform (or perhaps be now worse than) its 
> corresponding baseline?  I think such requirement is too 
> strong, for multiple reasons.
> 
> 2. The seamless switching between the modes, which everyone 
> seems to agree is important, is made possible by having all 
> modes in one codec.
> 
> 3. It ignores improvements beyond rate/bandwidth/quality, 
> such as the code size and complexity optimizations we did for SILK.
> 
> Checking modes against their baselines is of course a good 
> sanity check during development, and we've been doing such 
> tests extensively.  I welcome anyone to do more of them.  
> However, the sheer amount of listening necessary to compare 
> all 28 non-hybrid modes against their baselines with 
> sufficient statistical significance is large compared to the 
> value it brings.
> 
> best,
> koen.
> 
> 
> To: "Gregory Maxwell" <[email-masked]>, "Anisse Taleb" 
> <[email-masked]>, "Jean-Marc Valin" 
> <[email-masked]>
> Cc: [email-masked]
> either the 
> > transform mode or the LP mode alone (or those modes prior 
> to the work 
> > done as part of the WG process)- E.g.
> > at low rates the silk/hybrid modes are _obviously_ better (esp for 
> > speech) than the transform mode,  and at high rates the 
> transform mode 
> > is clearly better (and can achieve lower
> > (http://tools.ietf.org/agenda/78/slides/codec-0.pdf) at some rate. 
> > Though the transform mode's performance at low rates was 
> subsequently 
> > improved, the differences at low rate are still pretty stark.
> > 
> > If you only pick a single given rate the performance might only be 
> > equal to one of SILK/CELT alone (well, ignoring the fact that both 
> > modes have been improved as part of the process here) if 
> you've picked 
> > a rate where one or the other
> > bitrate" if you say "at some relevant bitrates" it would be a more 
> > practical requirement.
> > 
> 
> This supports standardization of the hybrid mode, which 
> represents 4 out of the 32 modes. It seems reasonable that 
> the remaining 28 modes are also checked against their 
> corresponding baseline at the same rate/bandwidth.
> 
> The listening test from IETF78 has been referenced a few 
> times. We should remember that it was presented with a 
> disclaimer that the results were not that reliable and that 
> the test should be seen more as an example of how it could be 
> conducted.
> 
> Best regards,
> Erik

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 04 Mar 2011 07:31:02 -0500

Hi Erik,

Keep in mind that including all three modes is necessary to fulfill the 
requirements we have 
(http://tools.ietf.org/html/draft-ietf-codec-requirements-02). These 
requirements also include baseline codecs to compare with (e.g. Speex, 
iLBC, ...).

Also, I may be misinterpreting here, but you seem to include SILK and 
CELT in the list of "baseline codecs we must beat". If SILK and CELT 
were already standardized codecs then I might agree, but since it's not 
the case I don't think it makes sense. It would essentially by saying 
"your codec must be better than itself to be standardized". While it 
happens that both have improved significantly since this effort started, 
I do not see how this matters here.

Cheers,

	Jean-Marc

On 11-03-04 03:00 AM, Erik Norvell wrote:
> Hi Koen,
>
> I appreciate the work being done on extending the codec
> functionality. However, I am not sure that it comes for free since
> the more technology is included the higher the risk of including
> third party IPR. If the codec functionality is already too wide to be
> covered by a listening test then the IPR analysis might also be
> lagging behind.
>
> Seamless switching between a wide range of operating points is a
> desirable feature, but the primary goal in the charter is still to
> produce an unencumbered codec. If the additional modes does not
> produce value over already existing codecs, it would be safer to
> leave them out.
>
> Best regards, Erik
>
>
>> -----Original Message----- From: Koen Vos
>> [mailto:[email-masked]] Sent: den 3 mars 2011 15:57 To: Erik
>> Norvell Cc: [email-masked]; Gregory Maxwell; Anisse Taleb;
>> Jean-Marc Valin Subject: Re: [codec] Updated Opus draft
>>
>> Hi Erik,
>>
>> represents 4 out of the 32 modes. It seems reasonable that the
>> remaining 28 modes are also checked against their corresponding
>> baseline at the same rate/bandwidth.
>>
>> Are you suggesting that to be added to a standard, each mode has to
>> outperform (or perhaps be now worse than) its corresponding
>> baseline?  I think such requirement is too strong, for multiple
>> reasons.
>>
>> 1. The 28 non-hybrid modes enhance the standard, regardless of how
>> they compare to their baselines: - The non-hybrid modes greatly
>> expand the useful operating range and flexibility. - The non-hybrid
>> modes come virtually for free.
>>
>> 2. The seamless switching between the modes, which everyone seems
>> to agree is important, is made possible by having all modes in one
>> codec.
>>
>> 3. It ignores improvements beyond rate/bandwidth/quality, such as
>> the code size and complexity optimizations we did for SILK.
>>
>> Checking modes against their baselines is of course a good sanity
>> check during development, and we've been doing such tests
>> extensively.  I welcome anyone to do more of them. However, the
>> sheer amount of listening necessary to compare all 28 non-hybrid
>> modes against their baselines with sufficient statistical
>> significance is large compared to the value it brings.
>>
>> best, koen.
>>
>>
>> ----- Original Message ----- From: "Erik
>> Norvell"<[email-masked]> To: "Gregory
>> Maxwell"<[email-masked]>, "Anisse Taleb"
>> <[email-masked]> Cc: [email-masked] Sent: Thursday,
>> March 3, 2011 12:36:52 AM Subject: Re: [codec] Updated Opus draft
>>
>> Hi Greg,
>>
>>>
>>> done as part of the WG process)- E.g. at low rates the
>>> silk/hybrid modes are _obviously_ better (esp for speech) than
>>> the transform mode,  and at high rates the
>> transform mode
>>> is clearly better (and can achieve lower latencies too).   The
>>> presentation of IETF-78 shows the hybrid outperforming SILK and
>>> CELT alone (http://tools.ietf.org/agenda/78/slides/codec-0.pdf)
>>> at some rate. Though the transform mode's performance at low
>>> rates was
>> subsequently
>>> If you only pick a single given rate the performance might only
>>> be equal to one of SILK/CELT alone (well, ignoring the fact that
>>> both modes have been improved as part of the process here) if
>> you've picked
>>> a rate where one or the other technique is inapplicable.   So
>>> rather than "at a given bitrate" if you say "at some relevant
>>> bitrates" it would be a more practical requirement.
>>>
>>
>> This supports standardization of the hybrid mode, which represents
>> 4 out of the 32 modes. It seems reasonable that the remaining 28
>> modes are also checked against their corresponding baseline at the
>> same rate/bandwidth.
>>
>> The listening test from IETF78 has been referenced a few times. We
>> should remember that it was presented with a disclaimer that the
>> results were not that reliable and that the test should be seen
>> more as an example of how it could be conducted.
>>
>> Best regards, Erik _______________________________________________
>> codec mailing list [email-masked]
> _______________________________________________ codec mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>
>

---

From: Erik Norvell <[email-masked]>
Date: Fri, 4 Mar 2011 16:34:07 +0100

Hi Jean-Marc,

I would consider the baselines the codecs that were proposed as starting point for the prototype codecs. If no improvement can be shown above these references it looks like a rubberstamping of these codecs. If, as you say, they have both improved significantly then there is no need for debate. Let's put them in the test just to verify it. 

Best regards,
Erik  

> From: Jean-Marc Valin [mailto:[email-masked]] 
> Sent: den 4 mars 2011 13:31
> To: Erik Norvell
> Cc: Koen Vos; [email-masked]
> Keep in mind that including all three modes is necessary to 
> fulfill the requirements we have 
> (http://tools.ietf.org/html/draft-ietf-codec-requirements-02).
>  These requirements also include baseline codecs to compare 
> with (e.g. Speex, iLBC, ...).
> 
> Also, I may be misinterpreting here, but you seem to include 
> SILK and CELT in the list of "baseline codecs we must beat". 
> If SILK and CELT were already standardized codecs then I 
> might agree, but since it's not the case I don't think it 
> makes sense. It would essentially by saying "your codec must 
> be better than itself to be standardized". While it happens 
> that both have improved significantly since this effort 
> started, I do not see how this matters here.
> 
> Cheers,
> 
> 	Jean-Marc
> 
> functionality. 
> > However, I am not sure that it comes for free since the more 
> > technology is included the higher the risk of including third party 
> > IPR. If the codec functionality is already too wide to be 
> covered by a 
> > listening test then the IPR analysis might also be lagging behind.
> >
> > produce value over already existing codecs, it would be 
> safer to leave 
> > them out.
> >
> > Best regards, Erik
> >
> >
> [mailto:[email-masked]] 
> >> Sent: den 3 mars 2011 15:57 To: Erik Norvell Cc: [email-masked]; 
> >> Gregory Maxwell; Anisse Taleb; Jean-Marc Valin Subject: 
> Re: [codec] 
> >> Updated Opus draft
> >>
> >> Hi Erik,
> >>
> >> outperform (or perhaps be now worse than) its 
> corresponding baseline?  
> >> I think such requirement is too strong, for multiple reasons.
> >>
> >> 2. The seamless switching between the modes, which 
> everyone seems to 
> >> agree is important, is made possible by having all modes in one 
> >> codec.
> >>
> such as the 
> >> code size and complexity optimizations we did for SILK.
> >>
> >> extensively.  I welcome anyone to do more of them. 
> However, the sheer 
> >> amount of listening necessary to compare all 28 non-hybrid modes 
> >> against their baselines with sufficient statistical 
> significance is 
> silk/hybrid 
> >>> modes are _obviously_ better (esp for speech) than the transform 
> >>> mode,  and at high rates the
> >> transform mode
> >>> presentation of IETF-78 shows the hybrid outperforming 
> SILK and CELT 
> >>> alone (http://tools.ietf.org/agenda/78/slides/codec-0.pdf)
> >>> at some rate. Though the transform mode's performance at 
> low rates 
> >>> was
> >> subsequently
> >> results were not that reliable and that the test should be 
> seen more 
> >> as an example of how it could be conducted.
> >>

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 04 Mar 2011 12:26:07 -0500

> Since the guidelines state we should not provide a rubber stamp for existing codecs, perhaps a good solution would be to include the baseline codecs in the listening test and mandate that the stand…

I think you have misread the guidelines.  The phrase "rubber stamp" means
"to process, approve or decide matters routinely rather than through
careful consideration" [1].

The guidelines prohibit a "rubber stamp" process in order to ensure that
we don't standardize a candidate codec without "careful consideration".
They do _not_ prohibit us from standardizing a candidate codec unchanged.
 The charter makes this clear.

You are welcome to test Opus against a baseline of SILK and CELT, although
this is difficult because neither SILK nor CELT is standardized, and both
have produced dozens of different incompatible versions over the course of
development.  However, no such comparison is required by the guidelines.
We are only required to compare Opus to other codecs that have been
standardized by SDOs.

[1] http://en.wiktionary.org/wiki/rubber_stamp#Verb

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 04 Mar 2011 10:31:28 -0700

On 3/4/11 10:26 AM, Benjamin M. Schwartz wrote:
> I think you have misread the guidelines.  The phrase "rubber stamp"
> means "to process, approve or decide matters routinely rather than
> through careful consideration" [1].
> 
> The guidelines prohibit a "rubber stamp" process in order to ensure
> that we don't standardize a candidate codec without "careful
> consideration". They do _not_ prohibit us from standardizing a
> candidate codec unchanged. The charter makes this clear.

Yes, this matter was fully discussed during the first and second BoFs (I
attended the first BoF and co-chaired the second BoF). Feel free to
revisit the textual minutes or audio recordings of those meetings, but
the consensus was incorporated into the charter, which was discussed
within the community and approved by the IESG.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Erik Norvell <[email-masked]>
Date: Mon, 7 Mar 2011 10:16:38 +0100

Hi Benjamin,

Thank you for the clarification.

My main point is to remind about the potential IPR risk of making a feature-rich codec. The more technology is packed together, the more risk of including third party patents. If there are many modes of operation which do not offer any benefit from already existing codecs (standardized or non-standardized), including them should be weighed against their IPR risk. In the beginning of this WG simpler and older technologies were discussed in order to end up with an unencumbered codec. Lately it seems the focus has swung to expanding the functionalities rather than carefully considering their IPR risk.

Best,
Erik 

> From: Benjamin M. Schwartz [mailto:[email-masked]] 
> Sent: den 4 mars 2011 18:26
> To: Erik Norvell
> Cc: [email-masked]
> > Since the guidelines state we should not provide a rubber 
> stamp for existing codecs, perhaps a good solution would be 
> to include the baseline codecs in the listening test and 
> mandate that the standardized modes to be better than the baseline.
> 
> I think you have misread the guidelines.  The phrase "rubber 
> stamp" means "to process, approve or decide matters routinely 
> rather than through careful consideration" [1].
> 
> The guidelines prohibit a "rubber stamp" process in order to 
> ensure that we don't standardize a candidate codec without 
> "careful consideration".
> You are welcome to test Opus against a baseline of SILK and 
> CELT, although this is difficult because neither SILK nor 
> CELT is standardized, and both have produced dozens of 
> different incompatible versions over the course of 
> We are only required to compare Opus to other codecs that 
> have been standardized by SDOs.
>

---

From: "Christian Hoene" <[email-masked]>
Date: Mon, 7 Mar 2011 11:10:24 +0100

Hello Erik,

> Sent: Monday, March 07, 2011 10:17 AM
> To: [email-masked]
> Cc: [email-masked]
> My main point is to remind about the potential IPR risk of making a
feature-
> rich codec. The more technology is packed together, the more risk of
> including third party patents. If there are many modes of operation which
do
> not offer any benefit from already existing codecs (standardized or non-
> standardized), including them should be weighed against their IPR risk. In
the
> beginning of this WG simpler and older technologies were discussed in
order
> to end up with an unencumbered codec. Lately it seems the focus has swung
> to expanding the functionalities rather than carefully considering their
IPR
> risk.

[Christian Hoene] 

On the other side, then upcoming RTC-Web initiative is trying to set up a
complete, unencumbered transmission system for both audio and video.
It shall include - of course - the codecs but also other things that are
required for good multimedia transmissions.

Changing coding modes and playout times are two important building blocks
that are required for good transmission quality. In my impression they are
even more important than minor improvements in coding efficiency. 

Thus, I would recommend to include them into the standardization: Either as
in the original codec definition draft or as a separate document. If they
are described in separate documents, then every codec implementer can decide
whether they want to take the IPR risks.

With best regards,

 Christian Hoene

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 7 Mar 2011 07:32:34 -0800

> My main point is to remind about the potential IPR risk of making a feature-rich codec. The more technology is packed together, the more risk of including third party patents. If there are many mod…

Are you suggesting that if I write some functionality in some development codec sitting on my desk that suddenly the WG codec should not provide that functionality because it would not provide any benefit over an existing codec?

If not then please help me understand what you are suggesting.

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 7 Mar 2011 11:39:27 -0500

Are you suggesting that if I write some functionality in some development
codec sitting on my desk that suddenly the WG codec should not provide that
functionality because it would not provide any benefit over an existing
codec?
I think all he is said is that the more functionality we put in, the greater
the risk of encumbrance from third-party IPR. And that the WG should weigh
the value of the functionality against the IPR risk.

I am thinking we need to include normative implementation of all the modes
we need to meet the requirements.  Modes that aren't needed to meet the
requirements should be considered for removal.  IPR considerations aside,
having too many modes of operation hurts interoperability.

Regards,
Stephen Botzko

On Mon, Mar 7, 2011 at 10:32 AM, Gregory Maxwell <[email-masked]>wrote:

> feature-rich codec. The more technology is packed together, the more risk of
> including third party patents. If there are many modes of operation which do
> not offer any benefit from already existing codecs (standardized or
> non-standardized), including them should be weighed against their IPR risk.
> In the beginning of this WG simpler and older technologies were discussed in
> order to end up with an unencumbered codec. Lately it seems the focus has
> swung to expanding the functionalities rather than carefully considering
> their IPR risk.
>
> Are you suggesting that if I write some functionality in some development
> codec sitting on my desk that suddenly the WG codec should not provide that
> functionality because it would not provide any benefit over an existing
> codec?
>

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 7 Mar 2011 09:38:28 -0800

> Are you suggesting that if I write some functionality in some development codec sitting on my desk that suddenly the WG codec should not provide that functionality because it would not provide any…

Ah— I missed this argument because I haven't noticed anyone directly suggesting that any of the currently included modes were not justified by the requirements and applications in the drafts.

Some of us are obtuse enough that we need things spelled out. If anyone believes that any modes are not justified by the WG targets please say so explicitly.

It's also worth pointing out that the difference between most of the 'modes' is pretty small: For example, the bandwidth variants of the transform mode all just change the frequency at which the codec stops coding (for compatibility with other sampling rates/bandpasses), for the latency variants in transform mode the smaller variants just change the transform size (except for the smallest, which doesn't have short blocks because it's always 'short' so it eliminates that functionality).  These modes take approximately no additional code to implement. 

While I can't say that modes which simply toggle loop ranges pose _no_ potential for IPR complications, they're certainly less concerning than things which add substantive algorithms or signal processing techniques. 

The only 'mode' thing which isn't a small difference in the code size/complexity are the LP and Transform mode settings— in the sense that each deactivates the other. Yet without the LP code Opus isn't an acceptable replacement for Speex (one of our named comparison points in S 4.2), and without the transform mode code the codec doesn't meet the latency requirements for the low latency applications (2.[456]) and can't deliver (near-)transparent music (2.6; as well as G.722.1C parity, most likely).  

On a point I didn't respond to before:
Erik Norvell [[email-masked]]:
> Lately it seems the focus has swung to expanding the functionalities rather than carefully considering their IPR risk.

I don't agree with this claim.  I don't see that there is much reason to argue about it, but I didn't want to leave it silent as though I agreed with it. 

Unless lately is defined broadly enough to include the decision to combine SILK and CELT so many months back then I don't believe we've added any user visibility functionality at all lately. (Though there has been "behind the scenes" functionality added to address quality an integration corner cases, reduce code size, etc.)

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 7 Mar 2011 12:52:15 -0500

Ah— I missed this argument because I haven't noticed anyone directly
suggesting that any of the currently included modes were not justified by
the requirements and applications in the drafts. ...
   If anyone believes that any modes are not justified by the WG targets
please say so explicitly.
I don't think Erik was making it.  To be very clear, I am not saying that we
have unnecessary modes.

I am suggesting that we all should cross-check that all the modes are
justified - that the requirements drive the functionality, not the IPR
concerns.  I agree that if someone thinks there are extra modes that aren't
justified, they should speak up.

Stephen Botzko

On Mon, Mar 7, 2011 at 12:38 PM, Gregory Maxwell <[email-masked]>wrote:

> Ah— I missed this argument because I haven't noticed anyone directly
> suggesting that any of the currently included modes were not justified by
> the requirements and applications in the drafts.
>
> Some of us are obtuse enough that we need things spelled out. If anyone
> believes that any modes are not justified by the WG targets please say so
> explicitly.
>
> It's also worth pointing out that the difference between most of the
> 'modes' is pretty small: For example, the bandwidth variants of the
> transform mode all just change the frequency at which the codec stops coding
> (for compatibility with other sampling rates/bandpasses), for the latency
> variants in transform mode the smaller variants just change the transform
> size (except for the smallest, which doesn't have short blocks because it's
> always 'short' so it eliminates that functionality).  These modes take
> approximately no additional code to implement.
>
> While I can't say that modes which simply toggle loop ranges pose _no_
> potential for IPR complications, they're certainly less concerning than
> things which add substantive algorithms or signal processing techniques.
>
> The only 'mode' thing which isn't a small difference in the code
> size/complexity are the LP and Transform mode settings— in the sense that
> each deactivates the other. Yet without the LP code Opus isn't an acceptable
> replacement for Speex (one of our named comparison points in S 4.2), and
> without the transform mode code the codec doesn't meet the latency
> requirements for the low latency applications (2.[456]) and can't deliver
> (near-)transparent music (2.6; as well as G.722.1C parity, most likely).
>
> > Lately it seems the focus has swung to expanding the functionalities
> rather than carefully considering their IPR risk.
>
> I don't agree with this claim.  I don't see that there is much reason to
> argue about it, but I didn't want to leave it silent as though I agreed with
> it.
>
> Unless lately is defined broadly enough to include the decision to combine
> SILK and CELT so many months back then I don't believe we've added any user
> visibility functionality at all lately. (Though there has been "behind the
> scenes" functionality added to address quality an integration corner cases,
> reduce code size, etc.)
>
>

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 7 Mar 2011 12:55:50 -0500

I agree. Modes should be driven by requirements, and if they are needed to
meet the requirements they should be normative. Modes should also be tested
to verify that they do indeed meet the requirements.

Paul Coverdale

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 07 Mar 2011 14:04:49 -0500

Hi,

All three modes are necessary to meet the requirements. Remove one of 
them and you have a "gap" in the range of useful rate and applications. 
More specifically:

1) We need the hybrid mode for super-wideband/full-band speech (and we 
have already shown that it is much better than SILK or CELT around 32 kb/s).
2) We need the SILK-only mode for narrowband and wideband speech (hybrid 
is for super-wideband and full-band)
3) We need the CELT-only mode for all music applications and for cases 
where we need <10 ms frame sizes.

	Jean-Marc

On 11-03-07 12:55 PM, Paul Coverdale wrote:
>  >>>
> I am thinking we need to include normative implementation of all the
> modes we need to meet the requirements. Modes that aren't needed to meet
> the requirements should be considered for removal. IPR considerations
> aside, having too many modes of operation hurts interoperability.
>
>  >>>
>
> I agree. Modes should be driven by requirements, and if they are needed
> to meet the requirements they should be normative. Modes should also be
> tested to verify that they do indeed meet the requirements.
>
> Paul Coverdale
>
>
>
>
>
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 08 Mar 2011 00:27:15 -0500

Hi Anisse,

I have just finished implementing the mode switching, using a different 
algorithm as originally planned (more details if you're interested). The 
code is in the git repository and works for all tests I did with it, 
though it's still possible that there are cases I missed. Right now, the 
main remaining artefact comes from the fact that the SILK mode uses a 
high-pass filter, while the CELT mode doesn't. If I disable the 
high-pass, this is what I get on a test that switches mode every 5 
frames (100 ms):
http://jmvalin.ca/misc_stuff/switch_100ms.wav
If I leave the high-pass filter on, I get this:
http://jmvalin.ca/misc_stuff/switch_100ms_hp.wav
We can hear some LF artefacts on that one, but I don't think it would be 
that bad if switching is rare (as it should be). Also, note that for 
speech files, the effect of the high-pass is much less noticeable since 
most of the voice content is above the HP cutoff anyway.

Cheers,

	Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 08 Mar 2011 14:23:23 +0000

Dear Jean Marc,

> have already shown that it is much better than SILK or CELT around 32

Is this some new test data or are you referring to the *not broad enough to be conclusive* MUSHRA tests? In addition, this refers to an *old* version of the hybrid mode, I wonder how much the recent changes would affect quality?

Independently of the base codecs being standards or not, testing the Opus codec together with SILK and CELT is necessary to show and validate the collaborative approach in developing the Opus codec as well as the claim of the codec capable of delivering a quality better than SILK and CELT (as a reminder, both statements are part of our PR message to ITU-T).

Kind regards,
/Anisse

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 8 Mar 2011 09:32:38 -0500

Hi,

This is based on the MUSHRA test and the fact that we have actually been 
listening to the output ourselves for a while. So while it does not 
constitute an "absolute proof", it's still a pretty strong indication that 
the hybrid performs better. This is also not really unexpected, since SILK 
simply cannot do full-band at all (i.e. we need the hybrid mode to make it 
scale), and CELT does not scale that well to bit-rates of 32 kb/s and 
below. So while we can possibly do another test to show the hybrid 
performance again, this is not really the first priority in terms of 
testing as far as I'm concerned. In the mean time, I encourage you to do 
your own listening of full-band audio in the 24-40 kb/s range with all 
three modes.

Cheers,

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 8 Mar 2011 15:44:49 +0100

Hello all,

Before everybody does his own listening-only tests in his lab, we shall
coordinate the listening-only efforts because the codec has too many modes
to cover them all with MUSHRA tests. 

We can test them all with PESQ/PEAQ/POLQA but subjective listening test are
important to validate the objective results and to tune the instrumental
assessment algorithms to the given distortion types.

Anisse, Michael, and all: I hope to see you in Prague so that we can discuss
on how to do the characterization tests.

With best regards,

 Christian Hoene

> Sent: Tuesday, March 08, 2011 3:33 PM
> To: Anisse Taleb
> Cc: [email-masked]
constitute
> an "absolute proof", it's still a pretty strong indication that the hybrid
> performs better. This is also not really unexpected, since SILK simply
cannot
> do full-band at all (i.e. we need the hybrid mode to make it scale), and
CELT
> does not scale that well to bit-rates of 32 kb/s and below. So while we
can
> possibly do another test to show the hybrid performance again, this is not
> really the first priority in terms of testing as far as I'm concerned. In
the mean
> time, I encourage you to do your own listening of full-band audio in the
24-40
> kb/s range with all three modes.
> 
> Cheers,
> 
> 	Jean-Marc
> 
> 
> 
> >> 1) We need the hybrid mode for super-wideband/full-band speech (and
> >> we have already shown that it is much better than SILK or CELT around
> >> 32 kb/s).
> >
> > Is this some new test data or are you referring to the *not broad enough
to
> be conclusive* MUSHRA tests? In addition, this refers to an *old* version
of
> the hybrid mode, I wonder how much the recent changes would affect
> quality?
> >
> > Independently of the base codecs being standards or not, testing the
Opus
> codec together with SILK and CELT is necessary to show and validate the
> collaborative approach in developing the Opus codec as well as the claim
of
> the codec capable of delivering a quality better than SILK and CELT (as a
> reminder, both statements are part of our PR message to ITU-T).
> >
> > Kind regards,
> > /Anisse

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 8 Mar 2011 09:46:03 -0500

There were very strong disclaimers on the presented MUSHRA results.  So I
think we really need to stop referring to them.  Or re-present them with
reasons explaining why the disclaimers were wrong.  It feels like a
bait-and-switch maneuver sometimes (though I am sure that is not the
intent).

I appreciate the value of informal expert listening tests, but I don't think
they are a substitute for the controlled testing results.  Is there a
timeline for the latter?

Regards,
Stephen Botzko

On Tue, Mar 8, 2011 at 9:32 AM, Jean-Marc Valin <[email-masked]
> wrote:

> scale), and CELT does not scale that well to bit-rates of 32 kb/s and below.
> So while we can possibly do another test to show the hybrid performance
> again, this is not really the first priority in terms of testing as far as
> I'm concerned. In the mean time, I encourage you to do your own listening of
> full-band audio in the 24-40 kb/s range with all three modes.
>
> Cheers,
>
>        Jean-Marc
>
>
>
>
>> to be conclusive* MUSHRA tests? In addition, this refers to an *old* version
>> of the hybrid mode, I wonder how much the recent changes would affect
>> quality?
>>
>> Independently of the base codecs being standards or not, testing the Opus
>> collaborative approach in developing the Opus codec as well as the claim of

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 08 Mar 2011 14:59:53 +0000

Dear Stephen,

>I am suggesting that we all should cross-check that all the modes are justified - that the >requirements drive the functionality, not the IPR concerns.  I agree that if someone thinks >there are extra modes that aren't justified, they should speak up.

I spoke about the mode issues albeit from a different angle than IPRs or Requirements and that is of functionality overlap.

Basically, I can understand that we need to have different modes to cover all functionalities that the codec strives to fulfill, my concern is that a recommendation of what options/modes need to be used or are needed and for what applications is still needed and this CANNOT be guessed prior to conducting a full test of all the options. And yes, if there are extra modes, either not fulfilling a requirement, or overlapping in functionality, they should be removed to avoid confusing the user. 

A Further note, On the requirements draft, the maximum bit-rate is 128kbps. I wonder why Opus allows the signaling of bitrates up to 512 kbps? Any intended application? 

Kind regards,
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 08 Mar 2011 15:11:28 +0000

Dear Christian,

I am all open for discussions in order to coordinate testing. I would like also to suggest that MUSHRA might not be the best methodology and that we should discuss, based on the requirements and the test conditions the most suitable methodology for what test.  For instance, at very high bitrates and very high quality, MUSHRA might be dropped in favor of BS.1116.

I will try to be there in the Prague meeting.

Kind regards,
/Anisse

> From: Christian Hoene [mailto:[email-masked]]
> Sent: Tuesday, March 08, 2011 3:45 PM
> To: 'Jean-Marc Valin'; Anisse Taleb; Michael Knappe
> Cc: [email-masked]
> Subject: RE: [codec] Updated Opus draft
> 
> Hello all,
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 8 Mar 2011 10:23:07 -0500

On 11-03-08 09:59 AM, Anisse Taleb wrote:
> Basically, I can understand that we need to have different modes to
> cover all functionalities that the codec strives to fulfill, my concern
> is that a recommendation of what options/modes need to be used or are
> needed and for what applications is still needed and this CANNOT be
> guessed prior to conducting a full test of all the options.

Although this is not implemented yet, the idea has always been to have the 
encoder automatically choose the best mode based on the information 
(application, bitrate, frame size, ...) provided by the user. To me this is 
just one more decision that the encoder needs to make and is fundamentally 
no different from all the other decisions like the use of short blocks (for 
transients), bit rate adaptation (VBR), post-filter parameters, and so on. 
This automatic selection is actually next on my TODO list.

> A Further note, On the requirements draft, the maximum bit-rate is
> 128kbps. I wonder why Opus allows the signaling of bitrates up to 512
> kbps? Any intended application?

Well, the requirements say we need to support up to 128 kb/s *per channel* 
(so 256 kb/s for stereo). This isn't meant to say that we must not support 
higher if possible. It turns out that the CELT-only mode (which is the only 
mode that scales to very high bit-rates) does not require any additional 
code to support extra bit-rates (unlike a codec like AMR-NB). When we did 
some testing recently, we observed that the quality (NMR/PEAQ) kept 
increasing up to around 256 kb/s per channel, before reaching a plateau. 
Considering that and the zero cost of supporting up to 256 kb/s per 
channel, it just makes sense not to include arbitrary limitations in the 
codec. In terms of application, I can see this being applied to 
high-fidelity equipment that can afford the extra bandwidth.

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 9 Mar 2011 11:03:40 +0100

> (application, bitrate, frame size, ...) provided by the user. To me this
is just
> one more decision that the encoder needs to make and is fundamentally no
> different from all the other decisions like the use of short blocks (for

[Christian Hoene] 
I am not sure whether automatic coding mode selection is similar to the
other problems as it is closely intervened with rate control and congestion
control. Of course, you can use DCCP which gives you a bit rate estimated
but then again, DCCP is not widely deployed and somewhat too shy (it takes
less bandwidth than modern TCP algorithms)

It would make more sense to add a feedback channel to RTP payload spec so
that the receiver can request a different coding mode. The receiver can then
decide which frame rate, bit rate, coding mode, etc. that the sender shall
use in order to fully that utilized the transmission path. 

Putting this decision on the receiver side has advantages. The receiver
knows about packet losses/ECN marks and it knows how well the PLC works.
Thus, the receiver can decide better the rate vs. loss tradeoff than the
encoder.

I thing that more important that automatic selection would be a proper
support for playout buffers. For example

1. an interface at the decoder to query the VAD state, voicing mode, and
pitch periods.
2. support of extrapolating the audio output with arbitrary lengths (the
current PLC works only with frame size boundaries) 
3. dropping of audio segments to fasten the playout.

I would offer my limited knowledge to help implementing these features - I
have done it for the SBC -  but I would need your codec expertise. Maybe at
the IETF meeting?

With best regards,

 Christian

> mode that scales to very high bit-rates) does not require any additional
code
> to support extra bit-rates (unlike a codec like AMR-NB). When we did some
> testing recently, we observed that the quality (NMR/PEAQ) kept increasing
> up to around 256 kb/s per channel, before reaching a plateau.
channel,
> it just makes sense not to include arbitrary limitations in the codec. In
terms
> of application, I can see this being applied to high-fidelity equipment
that can
> afford the extra bandwidth.
> 
> 	Jean-Marc