Subject: [codec] Discussion around ITU LS

From: Cullen Jennings <[email-masked]>
Date: Tue, 6 Sep 2011 14:54:26 -0600

We received the LS at 

https://datatracker.ietf.org/documents/LIAISON/file1251.pdf

This raises some inter sting points that I think are worth some list discussion.

Section 1. 

Some issues are pointed out with the example code. Can these be fixed? 

WIll the implementation include switching between SILK and CELT modes. 

My understanding is the reference code is not meant to be an optimized version or work on a wide variety of platforms. It is simply supposed to be one implementation that defines the algorithm on a given architecture. Is this correct?

What would people like to see with respect to test vectors? 

Does the WG consider  time shortening / stretching functionality they would want to include in opus as a technique to use for PLC?

Section 2.

The goals of the WG are to develop a RF codec. We have several IPR filings reported. You can find them at 
https://datatracker.ietf.org/ipr/search/?option=document_search&id_document_tag=20667. Not all of them offer RF terms. The IETF does not want to be involved in deterring if the claimed IPR is valid or not - we simple report the claims we have. Many participants of the WG have looked at IPR disclosures and feel this codec meets the goals of the WG. The question we need to ask in LC is do people think this is ready to publish. 

Section 3.

Given the current state of who was willing to step up to do listening tests before the IESG has finalized a codec. I suspect that it will be difficult to get significantly better testing than we have today before moving to a Proposed Standard. Once we have the frozen specification in the form of a Proposed Standard, I expect to see continued work on characterization and testing. Christian has volunteer to be editor on the working group draft to capture these results.

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 06 Sep 2011 18:09:17 -0700

Hi,
One comment inline.
Stephan

On 9.6.2011 13:54 , "Cullen Jennings" <[email-masked]> wrote:

>[Š]
>https://datatracker.ietf.org/ipr/search/?option=document_search&id_documen
>t_tag=20667. Not all of them offer RF terms. The IETF does not want to be
>involved in deterring if the claimed IPR is valid or not - we simple
>report the claims we have. Many participants of the WG have looked at IPR
>disclosures and feel this codec meets the goals of the WG.

Others, including myself, disagree with those "many people" (most of which
are strong supporters of the Opus codec). So does obviously at least one
rightholder (or better: their legal).

>The question we need to ask in LC is do people think this is ready to
>publish.

My reply to this is: the (from the outset unrealistic) goal of a codec
being freely practicable, whether achieved or not, should not interfere
with the publication of what seems to my uninformed mind to be solid
technology.  Adoption may or may not happen based on business situation,
business model, and risk tolerance of a possible adopter, as it is the
case for most (if not all) other standards.

Stephan 

[Š]

---

From: Monty Montgomery <[email-masked]>
Date: Tue, 6 Sep 2011 22:23:03 -0400

> rightholder (or better: their legal).

Simply as a point of information:

I would like to publicly state that after analysis, Xiph.Org does not
believe Opus infringes on any of the patents listed in the Qualcomm
IPR declaration.

Monty

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 07 Sep 2011 00:20:33 -0400

> Some issues are pointed out with the example code. Can these be

We have been working on addressing the comments that were raised during
the WGLC. We should be done within the next few days.

> WIll the implementation include switching between SILK and CELT

Seamless mode switching between all three modes has been part of the
bit-stream (i.e. the decoder) since the initial "freeze". The current
reference encoder includes some simple heuristics that work well in
practice. However, there is no plan right now for optimal encoder-side
mode switching decisions based on automatic speech/music. This can
always be added later in an improved encoder (without any compatibility
issue). Keep in mind that this is much harder to do for a low-delay
real-time codec than it can be for a streaming/archival codec.

> My understanding is the reference code is not meant to be an
> optimized version or work on a wide variety of platforms. It is
> simply supposed to be one implementation that defines the algorithm
> on a given architecture. Is this correct?

The current code actually works reasonably well/fast on a wide variety
of platforms. To make the code simpler, it does not include
architecture-specific optimizations, which are up to the implementers. I
believe the comment about portability was mostly referring to Windows
project files. for the sake of simplicity, those are not included in the
draft, though there is a separate package that includes them. As a note
on portability, Opus has been successfully compiled and run on quite a
few architectures. I think Greg is in a better position to comment on this.

> What would people like to see with respect to test vectors?

Again, Greg's been working on that.

> Does the WG consider  time shortening / stretching functionality they
> would want to include in opus as a technique to use for PLC?

While time stretching/shortening is something that can be useful, I
don't think it's really necessary for a codec *specification*. In fact
even the PLC is provided only as an example.

Cheers,

	Jean-Marc

---

From: Gregory Maxwell <[email-masked]>
Date: Tue, 6 Sep 2011 23:18:20 -0700

IJean-Marc Valin [[email-masked]]:
> draft, though there is a separate package that includes them.

One unfortunate thing about writing highly portable code is that
you end up with a bunch of per-platform configuration which complicates
building the software. The draft includes a basic makefile which should
work on a large cross section of systems, but for everything else some
assembly (in the form of writing a project/makefile) is required.

The autoconf and MSVC projects do make things easier on a broader
set of systems, but they're about half the size of the whole codec so not really
suitable for inclusion in the draft.

But they're in the SCM linked from http://opus-codec.org/

The web interface there will build tarfile snapshots from the repository, 
e.g. http://git.xiph.org/?p=opus.git;a=snapshot;h=HEAD;sf=tgz

> As a note

I do regular testing on the following platforms:
* Linux (x86, x86_64, IA64, PPC, Armv7) (GCC 4.7, 4.5, 3.x depending on the platform)
* Linux with LLVM compiler (x86, x86_64)
* NetBSD (x86)
* FreeBSD (x86)
* Solaris 10 (Ultrasparc)
* Win32 via Mingw
* Win32 via LCC-Win32
* Win32 via OpenWatcom
* Dos32 via OpenWatcom
* IBM S/390
* VAX (MicroVAX 3900, via SIMH, really :) It's quite slow)
(Plus a couple other oddball compilers that I don't test regularly)

None of these require any source modification. I test fixed and floating point builds.
 
I'm also aware of other people who regularly build on OSX (with both llvm-gcc-4.2, and clang 2.1) and with MSVC 

> Again, Greg's been working on that.

For conformance testing my main focus has be on constructing short vectors with complete line-of-code coverage in the decode (save some API and system error handling that can't be reached from the bitstream and non-normative features like the PLC).  I'm pretty close to that now, except for a couple of corner cases in the LPC mode.  I'm planning on testing against bugs we had prior to the bitstream freeze and making sure the comparison tool successfully rejects in those cases.

I'm certainly interested in what else other people are interested in seeing.

I've also been working on "diagnostic" test vectors, which are setup to hit likely failure points in applications (e.g. frame size switches) in a way that will produce audible glitches if the implementation isn't just right.  My thinking is that some people may wish to test embedded implementations which can't be easily put in a full test harness but which you could send a stream to and listen to the output.  Right now the primary impediment on this front is a remaining encoder bug in the reference implementation which cause glitches for my very burdensome test signal at mono/stereo and 10/20ms boundaries. I'm expecting a fix from Koen sometime soon.

Specific to the reference implementation, and separate from the test vectors, I'm also working on a full system test program which can be distributed as part of the source that tests all the API and will give us good coverage of the whole system as part of the build process without having to pack 50 megabytes of vectors in the source directly.   I'll probably be committing this to the SCM later this week.

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 08 Sep 2011 10:38:16 +0000

Dear Jean Marc,

>even the PLC is provided only as an example.

Quoting the requirements document RFC 6366:
"  It is desirable for the reference implementation to provide a time stretching/shortening
   implementation, although it should not be normative."

While I agree that this is something that should be non-normative, such an integrated implementation 
would be an important feature for any codec deployed on the internet and really differentiate Opus wrt. "codecs" which are considered unsuitable for the internet. 

Kind regards,
/Anisse

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 08 Sep 2011 10:30:13 -0400

Hi Anisse,

I agree that time stretching/shortening is a desirable feature. However, 
I think in the case of Opus it is best implemented outside the decoder 
using a good generic algorithm, rather than re-inventing the wheel. Of 
course, if you think an integrated implementation would be better, you 
are welcome submit a patch for it.

Cheers,

     Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Mon, 12 Sep 2011 08:00:36 +0000

Dear Jean Marc,

The issue is really to understand how the Opus codec is optimized for the internet. I was under the impression that a major argument for starting this work in IETF is to supply a codec that is optimized for use on the internet and therefore functionality that facilitates operation in an IP based transport would be defacto part of the codec distribution. 

The question of whether this is integrated in the decoder itself or outside the decoder (i.e. post-processing) depends mainly on the qualify and complexity of the two solution.  You seem to prefer the use of this feature as a post-processing stage to the audio output, would you like to elaborate why is this better than an integrated approach ? 

Regardless of whether this is implemented inside or outside the decoder (your preferred solution), I strongly believe that such a feature needs to be part of the code distribution.

Kind regards,
/Anisse

________________________________________
From: Jean-Marc Valin [[email-masked]]
Sent: 08 September 2011 16:30
To: Anisse Taleb
Cc: Cullen Jennings; Jonathan Rosenberg; [email-masked]
Subject: Re: [codec] Discussion around ITU LS

Hi Anisse,

Cheers,

     Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 13 Sep 2011 18:02:25 -0700

Hi Anisse,

Keep in mind that what we are standardizing here is a format more than
an actual implementation, so there's nothing that prevents anyone from
using any stretching/shortening algorithm. And in fact, thanks to Google
we now have an open-source time stretching/shortening algorithm as part
of the GIPS NetEQ jitter buffer they recently released:
http://webrtc.googlecode.com/svn/trunk/src/modules/audio_coding/NetEQ/main/source/
(see accelerate.c and expand.c more specifically). So I have really no
desire to reinvent the wheel.

In fact, I think this functionality belongs closer to the jitter buffer
than to the codec. Something like the PLC (which is also not going to be
normatively specified) belongs to the Opus decoder because it has to
interact with the codec state (i.e. regenerate not only the missing
data, but its effect on the predictors), which is not the case for the
stretching/shortening algo.

Cheers,

    Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 14 Sep 2011 12:34:24 +0200

Hello,

> (see accelerate.c and expand.c more specifically). 

Yes, the algorithm actually quite similar to the one I have implemented. 
First, you downsample the signal.
Then, you make a crosscorrelation to find the pitch period.
Last, you do an overlap-add operation to smooth the removed pitch.

> So I have really no

No, but it also does not make sense to have an additional function to get
stretch, length, and lag of the pitch period and the energy level of the
signal if this data is already available at the decoder.
Calculating things twice does not make sense.

It would make sense to use the information available at the decoder and then
simply add an pitch drop and replication function together function at the
decoder output. That can be done easily (even by me).

Koen and Jean-Marc: With parameters in the Silk/CELT codec provide this
information?

Thanks

 Christian Hoene

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 14 Sep 2011 12:43:48 +0000

Dear Jean-Marc,
 
> using any stretching/shortening algorithm. 

I think you are confusing format of the specification and objectives of the WG. The charter clearly states that the goal is to ensure the existence of a single high-quality audio codec that is optimized for use over the internet.

>  So I have really no desire to reinvent the wheel.

We are already reinventing new wheels by introducing a new codec :-). Joke aside, as I said in my previous email, it would be good that such functionality is present in the code distribution. I didn't mean that it should be developed from scratch, if what NetEQ offers is suitable, freely available and easily distributed, inclusion of such "example code" in the opus code distribution would be more than welcome, and I would say even necessary according to the target of this group and the requirements on the internet codec.

If that is not agreeable, I would suggest that the codec specification, either clearly and explicitly states that time scaling is not supported by the Opus codec and a link to an external code, NetEQ for example, is provided. Please consider this as a way to avoid confusion (which I strongly believe is the case) on the nature of the functionality offered by the "Internet Codec" developed by IETF.

The purpose of the IETF codec WG is to design a codec which specifically targets Internet applications (easily distributed with all needed functionality) and it was felt that such a codec was not available in other SDOs. There have been many comments on this very mailing list in favor of such functionality, it is even mentioned in the requirements document albeit with a weak wording. I am certainly baffled as to why such an essential functionality for optimal use of the codec on the internet is meeting such resistance. 
 
What functionality does the Opus codec offer which makes it optimized (or more optimized than other codecs) for the use over the internet ?

Kind regards,
/Anisse
Sent: 14 September 2011 03:02

Hi Anisse,

Cheers,

    Jean-Marc

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Wed, 14 Sep 2011 08:23:13 -0500

Jean-Marc,

I believed, as I think a lot of folks on this list did, that a USER of
OPUS need NOT BE A SPEECH/AUDIO CODING EXPERT to use OPUS.

The fact (as you stated on a previous volley) that you could easily add
the time-warping capabilities post OPUS decoding defeats the point - you
want your users NOT to need to be that expert in signal processing.

I find it amazing that Anisse's constructive comments are being met with
such resistance ... as such capabilities were touted as a major reason
why this work needed to be performed in the IETF.

How exactly is OPUS technically differentiated (other than marginal
differences in quality at bit rates within a factor of ~ 1.3x *) from
existing codecs developed in other SDOs?

Although such time warping capabilities were weakly worded in the actual
IETF requirements as "desirable" - the functionality Anisse is asking to
be in the baseline codec was highlighted as a major reason why the IETF
needed to do this work.

So, don't re-invent the wheel ... put an important capability that THE
IETF used as (partial) RATIONALE for this work into the actual delivery
of the base OPUS codec. To infer that a user of OPUS needs to be as
expert as an audio/speech coder to add this capability is ludicrous.

Regards,

Michael Ramalho, Ph.D.

PS - Said another way, if you add these capabilities the CODEC working
group will in fact deliver a codec with native capabilities
differentiated from all other SDOs ... and strengthens the IETF
justification for this work ... and all this debate goes away!

* - The parenthetical is not intended as a derogatory remark or jab, but
rather is just a reflection of the state of the art in speech/audio
coding. You all worked hard for that 1.3x.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Anisse Taleb
Sent: Wednesday, September 14, 2011 8:44 AM
To: Jean-Marc Valin
Cc: Jonathan Rosenberg; [email-masked]
Subject: Re: [codec] Discussion around ITU LS

Dear Jean-Marc,
 
> using any stretching/shortening algorithm. 

I think you are confusing format of the specification and objectives of
the WG. The charter clearly states that the goal is to ensure the
existence of a single high-quality audio codec that is optimized for use
over the internet.

>  So I have really no desire to reinvent the wheel.

We are already reinventing new wheels by introducing a new codec :-).
Joke aside, as I said in my previous email, it would be good that such
functionality is present in the code distribution. I didn't mean that it
should be developed from scratch, if what NetEQ offers is suitable,
freely available and easily distributed, inclusion of such "example
code" in the opus code distribution would be more than welcome, and I
would say even necessary according to the target of this group and the
requirements on the internet codec.

If that is not agreeable, I would suggest that the codec specification,
either clearly and explicitly states that time scaling is not supported
by the Opus codec and a link to an external code, NetEQ for example, is
provided. Please consider this as a way to avoid confusion (which I
strongly believe is the case) on the nature of the functionality offered
by the "Internet Codec" developed by IETF.

The purpose of the IETF codec WG is to design a codec which specifically
targets Internet applications (easily distributed with all needed
functionality) and it was felt that such a codec was not available in
other SDOs. There have been many comments on this very mailing list in
favor of such functionality, it is even mentioned in the requirements
document albeit with a weak wording. I am certainly baffled as to why
such an essential functionality for optimal use of the codec on the
internet is meeting such resistance. 
 
What functionality does the Opus codec offer which makes it optimized
(or more optimized than other codecs) for the use over the internet ?

Kind regards,
/Anisse

Hi Anisse,

http://webrtc.googlecode.com/svn/trunk/src/modules/audio_coding/NetEQ/ma
in/source/

Cheers,

    Jean-Marc

On 12/09/11 01:00 AM, Anisse Taleb wrote:
> Dear Jean Marc,
>
> The issue is really to understand how the Opus codec is optimized for
the internet. I was under the impression that a major argument for
starting this work in IETF is to supply a codec that is optimized for
use on the internet and therefore functionality that facilitates
operation in an IP based transport would be defacto part of the codec
distribution.
>
> The question of whether this is integrated in the decoder itself or
outside the decoder (i.e. post-processing) depends mainly on the qualify
and complexity of the two solution.  You seem to prefer the use of this
feature as a post-processing stage to the audio output, would you like
to elaborate why is this better than an integrated approach ?
>
> Regardless of whether this is implemented inside or outside the
decoder (your preferred solution), I strongly believe that such a
feature needs to be part of the code distribution.
>> would be an important feature for any codec deployed on the internet
and really differentiate Opus wrt. "codecs" which are considered
unsuitable for the internet.
>> /Anisse

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 14 Sep 2011 11:31:07 -0400

> What functionality does the Opus codec offer which makes it optimized (or more optimized than other codecs) for the use over the internet ?

The Opus codec offers coding
- optimized for both speech and non-speech sounds, in stereo and mono,
- coding 5 different bandwidths from Narrow to Full
- with latencies as low as 5ms
- and packet rates as low as 10 Hz,
- over the whole continuum of bitrates from 6kbps to 500kbps,
- with seamless variability in all of these parameters.
- It is robust to packet loss if packet loss is present,
 - but does not sacrifice efficiency when there is low packet loss.
- It can efficiently produce both CBR and VBR streams.
- It is designed to work well on a variety of computing hardware,
including systems with and without floating-point units.
- In each of these aspects its performance is either the best in the world
or not far behind the best, and the specification encourages compatible
improvement.

This functionality makes Opus more optimized than other codecs for use
over the internet because (correspondingly)
- all sorts of audio are transmitted over the internet
- endpoints may be gateways into limited-bandwidth systems
- ultra-low latency is sometimes desired (but not always)
- per-packet overhead on the internet can be significant
- the available bandwidth on the internet spans orders of magnitude
- the performance of a connection over the internet is highly dynamic, as
is its content
- The internet is not a reliable transport
 - but the degree of unreliability varies widely
- Some connections are limited by the available instantaneous bandwidth,
while others are limited by the aggregate cost of data transmission.
- All sorts of different devices are connected to the internet, including
both high-power desktop computers and low-power special-purpose devices.
- Devices connected to the internet often live for a long time and remain
important participants, so standards are typically used for decades once
deployed.

I am not aware of any other codec that is competitive with Opus on all
these fronts.

What makes Opus optimized for the internet is _not_ the non-normative
accessories like clock skew correction.  That kind of (useful)
functionality can go into libraries like opus-tools [1], which already
offers resampling and other similar non-normative additions such as
noise-shaping dither to improve the quality of 16-bit output.

--Ben

[1] http://git.xiph.org/?p=users/jm/opus-tools.git;a=summary

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 14 Sep 2011 08:40:13 -0700

> why this work needed to be performed in the IETF.

I find it unfortunate that Jean-Marc's comments are being interpreted as 
"resistance" as he plainly invited the contribution of such functionality
while saying that he still thought it would best be done another way.

(And, for whatever it's worth,  I think he's right:  It's the jitter buffer
this functionality needs to be integrated with far more so than
the codec itself  I also think that Christian is right and the codec could
export some information which would be useful)

Personally, I don't understand how simply not jumping at the opportunity
to do a bunch of software development and IPR clearance work for
functionality that you aren't interested in, won't use, and think 
could better be done another way can fairly be called resistance.

I also think you should take a careful look at the old discussion
surrounding the requirements like this— you'll see that there was
clear opposition to reinventing the whole of the internet inside the
codec.  There is a careful balancing act.

Generally I think we should avoid adding functionality to the codec
specification which isn't itself important for interoperability or making
the codec functional.

I expect future Opus implementers will add a variety of useful things
to their implementations— better error concealment, more intelligent
encoders, echo cancellation, etc.  We can't hope to match in finite
time the quality and brilliance of the work the whole world can
produce in the indefinite future.

I would probably have let out the loss concealment and the _encoder_
from the draft,  except that having these things was essential for
proving that a codec meeting the community's needs would actually
be available to them and it's needed for the reference implementation
to be functionally complete. But I can't say the same thing about
stretching.

Or perhaps there is a miscommunication here:  Opus _does_ have
rudimentary "stretching" of the at least of the level required
to make the code usable over the internet:  If you invoke the decoder
without input (null data buffer) it will generate filler audio, making
the signal longer. Likewise, if you skip a packet on decode (especially
an inactive one) you'll shorten the signal without anything bad
happening.

This works fine, and it's how all of the pre-final adoption of CELT
in the open source world has worked. As far as I know _none_
of our great many early adopters have requested anything
more.

When JM and I hear complaint that Opus doesn't provide "time
stretching" we're mostly thinking about  high complexity/high fidelity
algorithms for this purpose instead of that basic one.  We think of
Non-basic functionality which isn't needed to interop, which will have
a platform and application specific nature— (since you can usually
afford better stretching on a desktop than on a phone), and which
can be added by anyone at any time in the future...  

One way to address this, if people consider it important, would be
to develop another draft for that purpose.  It's something I would
have instantly proposed in response to the request— except I know
that many people here are concerned about working group
scope creep.  

(Though, if it isn't opposed it as scope creep as part of the
codec draft, why oppose it as a freestanding draft?)

> existing codecs developed in other SDOs?

Can you suggest a comparable codec which supports latency at the 
5ms level? Another codec that can operate at _any_ bitrate congestion
control permits it?  Another codec that supports good fidelity speech at
11kbit/sec? another codec that gives good stereo music at 64kbit/sec?
Transparent stereo music near 100kbit/sec while giving latencies which
are acceptable for communication?  Another with similar licensing
status to Opus?

You can find examples for each of these, though you'll find
that Opus is still best of breed in each considered alone— but
you can not find anything that matches the composition overall or
even comes close.

When I look at the applications and requirements the strongest theme
visible to me is this theme of versatility— it's one that underlies many
of the devices being added to the network today (such as smartphones,
with their 1,00 1 "apps") as well as the the trends in webbrowser 
technology where small collections of flexible tools and development
glue are being used to invent applications that where never dreamed
of by the developers of the infrastructure.

We are already _well_ differentiated, and if you're not willing to
accept that then why would you accept that adding additional
stretching would resolve it? After all, the same code could be copied
to the implementation of any other codec.

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 14 Sep 2011 10:01:06 -0700

> Koen and Jean-Marc: With parameters in the Silk/CELT codec provide this

See commit 25f7f355 I just checked in that implements a OPUS_GET_PITCH()
request that returns the pitch of the latest frame -- if available. Does
this address your issue?

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 14 Sep 2011 10:08:41 -0700

> If that is not agreeable, I would suggest that the codec
> specification, either clearly and explicitly states that time scaling
> is not supported by the Opus codec and a link to an external code,
> NetEQ for example, is provided. Please consider this as a way to
> avoid confusion (which I strongly believe is the case) on the nature
> of the functionality offered by the "Internet Codec" developed by
> IETF.

I think this is a perfectly reasonable suggestion. Opus is not claiming
to solve all the world's problems and I'm fine with explicitly stating
so. So the draft will include a link to the NetEQ code as a suggestion
for both jitter buffer and time stretching. The same code base also does
noise reduction and echo cancellation, so we might want to point to that
as well.

Cheers,

	Jean-Marc

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Wed, 14 Sep 2011 12:58:57 -0500

Gregory,

Thank you for your civil tone, serious debate and tradeoff explanation
of the issues here.

[I would have appreciated you "de-personalizing" your response by not
using phrases like "functionality YOU are not interested in" (perhaps
you meant that the present OPUS developers are not interested in), or
"[functionality] YOU'RE not willing" (perhaps meaning a particular
implementer not willing and not me personally).]

I will address some of your issues in-line (with "MAR:"), but the main
point is that if the time warping capability was added ... a lot of the
present criticism leveled at the CODEC WG process would be evaporate.
Your other points are appreciated.

Regards,

Michael Ramalho

-----Original Message-----
From: Gregory Maxwell [mailto:[email-masked]] 
Sent: Wednesday, September 14, 2011 11:40 AM
To: Michael Ramalho (mramalho); Anisse Taleb; Jean-Marc Valin
Cc: Jonathan Rosenberg; [email-masked]
Subject: RE: [codec] Discussion around ITU LS

"resistance" as he plainly invited the contribution of such
functionality
while saying that he still thought it would best be done another way.

(And, for whatever it's worth,  I think he's right:  It's the jitter
buffer
this functionality needs to be integrated with far more so than
the codec itself  I also think that Christian is right and the codec
could
export some information which would be useful)

MAR (took me a while to absorb the 3x double negative): Assuming that
you intend to say that the OPUS developers don't want to spend
cycles developing a capability they personally intend to do another
way - I understand that. That wasn't the point of my email. Although
I responded to Jean-Marc, my reply was intended for the entire WG.
My apologies to Jean-Marc if my reply was too specific to his prior
post and to him personally.

I also think you should take a careful look at the old discussion
surrounding the requirements like this- you'll see that there was
clear opposition to reinventing the whole of the internet inside the
codec.  There is a careful balancing act.

Generally I think we should avoid adding functionality to the codec
specification which isn't itself important for interoperability or
making
the codec functional.

MAR: Please humor me here. Overlap and add functionality was developed
in the 1970s, PSOLA (pitch synchronous overlap and add) commonplace in
the 1980s and the coder already has a knowledge of pitch (for speech).
How difficult would it be to add a "speed parameter" to the codec
whose value is 1.00 99+% of the time and modulated from 1.00 only
as the jitter buffer nears its high or low water mark? This would
likely be one of the lowest complexity functions to be added to the
decoder (overlap and add and correlation function ONLY invoked when
speed parameter != 1.0).

to their implementations- better error concealment, more intelligent

MAR: Does the decoder know when small time length phonemes are being
produced (e.g., "t") and know NOT to drop those particular frames? I
would be OK with your suggestion if you had "voiced/unvoiced"
determination exposed outside of the decoder (and instructed the
application to preferentially drop only during known voiced or
silence periods) ... or derivatives of this determination (an
indication of how important this phoneme is to intelligibility).

a platform and application specific nature- (since you can usually

MAR: I addressed this above. I (personally) am not asking for
this capability - but incorporating a rudimentary (not high
complexity) version of it is simple (unless you disagree with
what I stated above) AND it deflates the whole "you guys
don't have time warping capabilities" argument.

One way to address this, if people consider it important, would be
to develop another draft for that purpose.  It's something I would
have instantly proposed in response to the request- except I know
that many people here are concerned about working group
scope creep.  

MAR: I put the adjective "technically" on purpose; your last sentence
doesn't quality as a response to my question.

MAR: That being said, you make good points - although many codecs have
one or more of these features (e.g., G.711.0 admittedly narrow-band
only data compression algorithm operates down to 5ms to address your
first point and is royalty-free for softclient use).

You can find examples for each of these, though you'll find
that Opus is still best of breed in each considered alone- but
you can not find anything that matches the composition overall or
even comes close.

visible to me is this theme of versatility- it's one that underlies many

MAR: Hoping your use of "you're not willing" isn't directed at me
personally, as that wasn't the purpose of my email. My point was that
if such capability was added ... I don't see any other major complaints
on the list ... and lack of such functionality was a major rationale
for the work being performed in the IETF. I don't see that the WG
would care if your supposition was acceptable to me personally anyway
(I am only one hum).

MAR: Thanks again for your reasoned reply.

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 14 Sep 2011 15:00:03 -0400

On 09/14/2011 01:01 PM, Jean-Marc Valin wrote:
> request that returns the pitch of the latest frame -- if available.

The attached patch documents this feature in the draft.

--Ben
From 2b3d3eba0ec51181a9636efda577e63157c329d7 Mon Sep 17 00:00:00 2001
From: Benjamin M. Schwartz <[email-masked]>
Date: Wed, 14 Sep 2011 14:53:11 -0400
Subject: [PATCH] Update text regarding PLC and add a new section on clock drift.

The PLC section has been moved from under CELT to the top
level and expanded to cover both SILK and CELT.  The clock drift
text provides some advice and explanation of the new GET_PITCH API.
---
 doc/draft-ietf-codec-opus.xml |   47 ++++++++++++++++++++++++++++++++++++++--
 1 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/doc/draft-ietf-codec-opus.xml b/doc/draft-ietf-codec-opus.xml
index 4bf79b9..3b37ffa 100644
--- a/doc/draft-ietf-codec-opus.xml
+++ b/doc/draft-ietf-codec-opus.xml
@@ -4502,17 +4502,58 @@ where alpha_p=0.8500061035.
 
 </section>
 
+</section>
+
 <section anchor="Packet Loss Concealment" title="Packet Loss Concealment (PLC)">
 <t>
 Packet loss concealment (PLC) is an optional decoder-side feature which
 SHOULD be included when transmitting over an unreliable channel. Because
 PLC is not part of the bitstream, there are several possible ways to
-implement PLC with different complexity/quality trade-offs. The PLC in
-the reference implementation finds a periodicity in the decoded
+implement PLC with different complexity/quality trade-offs.
+</t>
+
+<t>
+The PLC in
+the reference implementation depends on the mode of last packet received.
+In CELT mode, the PLC finds a periodicity in the decoded
 signal and repeats the windowed waveform using the pitch offset. The windowed
 waveform is overlapped in such a way as to preserve the time-domain aliasing
 cancellation with the previous frame and the next frame. This is implemented
-in celt_decode_lost() (mdct.c).
+in celt_decode_lost() (mdct.c).  In SILK mode, the PLC uses LPC extrapolation
+from the previous frame, implemented in silk_PLC() (silk_PLC.c).
+</t>
+
+<section anchor="Clock drift compensation" title="Clock drift compensation">
+<t>
+Clock drift refers to the gradual desynchronization of two endpoints
+whose sample clocks run at different frequencies while they are streaming
+live audio.  Differences in clock frequencies are generally attributable to
+manufacturing variation in the endpoints' clock hardware.  For long-lived
+streams, the time difference between sender and receiver can grow without
+bound.
+</t>
+
+<t>
+When the sender's clock runs slower than the receiver's, the effect is similar
+to packet loss: too few packets are received.  The receiver can distinguish
+between drift and loss if the transport provides packet timestamps.  A receiver
+for live streams SHOULD conceal the effects of drift, and MAY do so by invoking
+the PLC.
+</t>
+
+<t>
+When the sender's clock runs faster than the receiver's, too many packets will
+be received.  The receiver MAY respond by skipping any packet (i.e. not
+submitting the packet for decoding).  This is likely to produce a less severe
+artifact than if the frame were dropped after decoding.
+</t>
+
+<t>
+A decoder MAY employ a more sophisticated drift compensation method, such as
+adding or removing one cycle when the waveform is highly periodic.  The
+reference implementation allows a caller to learn whether the current
+frame's signal is highly periodic, and if so what its period is, using the
+OPUS_GET_PITCH request.
 </t>
 </section>
 
-- 
1.7.0.4

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 14 Sep 2011 16:54:30 -0400

Improved patch to the draft regarding PLC and clock drift compensation.
Clarifies that the webrtc code uses this method.
From 7243d01ad52f8e938a72102ab7d66cab07470986 Mon Sep 17 00:00:00 2001
From: Benjamin M. Schwartz <[email-masked]>
Date: Wed, 14 Sep 2011 14:53:11 -0400
Subject: [PATCH] Update text regarding PLC and add a new section on clock drift.

The PLC section has been moved from under CELT to the top
level and expanded to cover both SILK and CELT.  The clock drift
text provides some advice and explanation of the new GET_PITCH API.
---
 doc/draft-ietf-codec-opus.xml |   60 ++++++++++++++++++++++++++++++++++++----
 1 files changed, 54 insertions(+), 6 deletions(-)

diff --git a/doc/draft-ietf-codec-opus.xml b/doc/draft-ietf-codec-opus.xml
index 4bf79b9..edf6dff 100644
--- a/doc/draft-ietf-codec-opus.xml
+++ b/doc/draft-ietf-codec-opus.xml
@@ -4502,17 +4502,65 @@ where alpha_p=0.8500061035.
 
 </section>
 
+</section>
+
 <section anchor="Packet Loss Concealment" title="Packet Loss Concealment (PLC)">
 <t>
-Packet loss concealment (PLC) is an optional decoder-side feature which
-SHOULD be included when transmitting over an unreliable channel. Because
-PLC is not part of the bitstream, there are several possible ways to
-implement PLC with different complexity/quality trade-offs. The PLC in
-the reference implementation finds a periodicity in the decoded
+Packet loss concealment (PLC) is an optional decoder-side feature that
+SHOULD be included when receiving from an unreliable channel. Because
+PLC is not part of the bitstream, there are many acceptable ways to
+implement PLC with different complexity/quality trade-offs.
+</t>
+
+<t>
+The PLC in
+the reference implementation depends on the mode of last packet received.
+In CELT mode, the PLC finds a periodicity in the decoded
 signal and repeats the windowed waveform using the pitch offset. The windowed
 waveform is overlapped in such a way as to preserve the time-domain aliasing
 cancellation with the previous frame and the next frame. This is implemented
-in celt_decode_lost() (mdct.c).
+in celt_decode_lost() (mdct.c).  In SILK mode, the PLC uses LPC extrapolation
+from the previous frame, implemented in silk_PLC() (silk_PLC.c).
+</t>
+
+<section anchor="Clock drift compensation" title="Clock drift compensation">
+<t>
+Clock drift refers to the gradual desynchronization of two endpoints
+whose sample clocks run at different frequencies while they are streaming
+live audio.  Differences in clock frequencies are generally attributable to
+manufacturing variation in the endpoints' clock hardware.  For long-lived
+streams, the time difference between sender and receiver can grow without
+bound.
+</t>
+
+<t>
+When the sender's clock runs slower than the receiver's, the effect is similar
+to packet loss: too few packets are received.  The receiver can distinguish
+between drift and loss if the transport provides packet timestamps.  A receiver
+for live streams SHOULD conceal the effects of drift, and MAY do so by invoking
+the PLC.
+</t>
+
+<t>
+When the sender's clock runs faster than the receiver's, too many packets will
+be received.  The receiver MAY respond by skipping any packet (i.e. not
+submitting the packet for decoding).  This is likely to produce a less severe
+artifact than if the frame were dropped after decoding.
+</t>
+
+<t>
+A decoder MAY employ a more sophisticated drift compensation method. For
+example, the
+<eref target='http://code.google.com/p/webrtc/source/browse/trunk/src/modules/audio_coding/NetEQ/main/source/webrtc_neteq.c?r=326'>NetEQ component</eref>
+of the
+<eref target='http://code.google.com/p/webrtc/'>WebRTC.org codebase</eref>
+compensates for drift by
+<eref target='http://code.google.com/p/webrtc/source/browse/trunk/modules/audio_coding/NetEQ/main/source/expand.c?r=28'>adding</eref>
+or
+<eref target='http://code.google.com/p/webrtc/source/browse/trunk/modules/audio_coding/NetEQ/main/source/accelerate.c?r=28'>removing</eref>
+one period when the signal is highly periodic. The reference implementation of
+Opus allows a caller to learn whether the current frame's signal is highly
+periodic, and if so what the period is, using the OPUS_GET_PITCH() request.
 </t>
 </section>
 
-- 
1.7.0.4

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 15 Sep 2011 08:25:18 +0000

Dear Jean-Marc,

Thanks for your consideration. I don't think acoustic front-end algorithms such as echo cancellation and noise reduction are necessary to include. This does not mean that this functionality would not be necessary for operation on the internet (or any other access technology for that matter), However, I do not see them as functionality especially included for optimal operation on the internet. 

Kind regards,
/Anisse
Sent: 14 September 2011 19:08

Cheers,

        Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Wed, 21 Sep 2011 20:32:15 +0200 (CEST)

My two cents:

Time scaling is important for using a codec on the Internet, but so is e.g. a jitter buffer.  People who want a full software stack for interactive Internet audio applications should look elsewhere (WebRTC).  By exposing internal pitch information through the API we enable an efficient integration of the codec, and that should be where our responsibility ends.

Furthermore, leaving out superfluous functionality helps to minimize risk of IPR infringement.  

koen.

----- Original Message -----
From: "Anisse Taleb" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>
Cc: "Jonathan Rosenberg" <[email-masked]>, [email-masked]
Sent: Thursday, September 15, 2011 1:25:18 AM
Subject: Re: [codec] Discussion around ITU LS

Dear Jean-Marc,

Kind regards,
/Anisse

Cheers,

        Jean-Marc