Subject: Re: [codec] requirements #8 (new): Sample rates?

From: "codec issue tracker" <[email-masked]>
Date: Tue, 25 Jan 2011 00:10:32 -0000

#8: Sample rates?

Comment(by gmaxwell@…):

 I believe the requirements aspect of this has been addressed.  That there
 is consensus that full-band operation is a requirement, and that other
 rates/behaviors are possibly for consideration but not required.

 Have I misunderstood the situation?

-- 
------------------------------------+---------------------------------------
 Reporter:  hoene@…                 |       Owner:  jean-marc.valin@…             
     Type:  enhancement             |      Status:  new                           
 Priority:  minor                   |   Milestone:                                
Component:  requirements            |     Version:                                
 Severity:  Active WG Document      |    Keywords:                                
------------------------------------+---------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/8#comment:3>
codec <http://tools.ietf.org/codec/>

---

From: Roman Shpount <[email-masked]>
Date: Mon, 24 Jan 2011 19:14:09 -0500

I would like to see 8 and 16 KHz as required rates for the codec to insure
interoperability with existing narrowband and wideband codecs. In other
words we should be able to negotiate 8 or 16 Khz sample rate if audio will
be transcoded for PSTN or wideband codec such as G.722
_____________
Roman Shpount

On Mon, Jan 24, 2011 at 7:10 PM, codec issue tracker <[email-masked]>wrote:

> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 24 Jan 2011 19:20:30 -0500

I believe such requirement for narrowband/wideband is already present, but 
I don't mind making it even more explicit is necessary.

     Jean-Marc

On 11-01-24 07:14 PM, Roman Shpount wrote:
> On Mon, Jan 24, 2011 at 7:10 PM, codec issue tracker <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     #8: Sample rates?
>
>
>     Comment(by gmaxwell@…):
>
>     [email-masked] <mailto:[email-masked]>

---

From: "codec issue tracker" <[email-masked]>
Date: Tue, 25 Jan 2011 00:40:30 -0000

#8: Sample rates?

Comment(by gmaxwell@…):

 > I don't mind making it even more explicit is necessary.

 I believe that we since are close enough to finalizing the draft we should
 try to include proposed language with our issues.  Otherwise there may be
 no clear path forward.

 Some of the requirements specify that compatibility with
 wideband/narrowband is important, but for the avoidance of doubt, I'll
 suggest:

 At the end of 4.1.  Operating space:

 "Because interoperation with existing wideband and narrowband facilities
 is essential at least one method of interoperation must be provided
 regardless of the codec's operating mode, sample rate, or bitrate."

 Of course, I would be perfectly happy to leave this out entirely (as I
 believe the application section 2.1 already implies the requirement) or
 use some other language.

Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/8#comment:4>
codec <http://tools.ietf.org/codec/>

---

From: Roman Shpount <[email-masked]>
Date: Mon, 24 Jan 2011 20:07:54 -0500

I actually do believe that the requirement is already addressed by section
4.1 of the document. I was responding to a comment that only full band is
required, which is clearly not what we agreed upon and not what's in the
document.
_____________
Roman Shpount

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 25 Jan 2011 16:53:38 -0800

I agree that supporting sampling rates of 48, 16, and 8 kHz all makes sense, but I also think it would be highly desirable to support
another common sampling rate used by music CDs: 44.1 kHz.

For music streaming applications, a large percentage of the music sources will have this 44.1 kHz sampling rate.  I know it is
possible to up-sample them to 48 kHz first and then pass them through the 48 kHz version of Opus.  However, doing such up-
sampling requires additional processing power and additional latency, and if the sampling rate conversion is not done with a
high-quality method (which usually requires higher complexity and higher delay), audio quality degradation may be introduced
in the process.  For these reasons, many people prefer not to do the 44.1 kHz to 48 kHz sampling rate conversion and prefer to
encode the 44.1 kHz music directly instead.  Therefore, it is highly desirable for Opus to support this 44.1 kHz sampling rate.

I understand that it may be inconvenient for the SILK mode or the SILK+CELT hybrid mode to support 44.1 kHz.  However, for the
CELT-only mode, my understanding is that the current CELT C code already supports the 44.1 kHz sampling rate.  Since the CELT-
only mode is also the most suitable mode to encode music, it would then make perfect sense for the CELT-only mode to also
support this 44.1 kHz in addition to 48 kHz.

Furthermore, currently the CELT mode at 48 kHz supports frame sizes of 2.5, 5., 10, and 20 ms to be consistent with the frame
sizes of the SILK mode and SILK+CELT hybrid mode, and this results in frame sizes and FFT window sizes that are not powers of 2,
which reduces the implementation efficiency.  If the CELT mode supports the 44.1 kHz sampling rate, then since it is no longer
possible to support exactly 2.5, 5, 10, and 20 ms at this sampling rate anyway, I think we might as well choose the FFT window
sizes to be powers of 2 to allow more efficient implementation.  Again, such power of 2 FFT sizes at 44.1 kHz is already supported
in the current C code for CELT, so no new development is needed.

In summary, I propose that we allow the CELT-only mode of the Opus codec to officially support the 44.1 kHz sampling rate using
power of 2 FFT window sizes to avoid sampling rate conversion and to allow the most efficient implementation.

Raymond

From: [email-masked] [mailto:[email-masked]] On Behalf Of Roman Shpount
Sent: Monday, January 24, 2011 5:08 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [codec] requirements #8 (new): Sample rates?

I actually do believe that the requirement is already addressed by section 4.1 of the document. I was responding to a comment that only full band is required, which is clearly not what we agreed upon and not what's in the document.
_____________
Roman Shpount

On Mon, Jan 24, 2011 at 7:40 PM, codec issue tracker <[email-masked]<mailto:[email-masked]>> wrote:
#8: Sample rates?

Comment(by gmaxwell@...):

 Reporter:  hoene@...                 |       Owner:  jean-marc.valin@...

_______________________________________________
codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/codec

---

From: Koen Vos <[email-masked]>
Date: Wed, 26 Jan 2011 06:59:23 +0100 (CET)

Hi Raymond, 

Streaming of CD music was not an anticipated use case or requirement. That said, 44.1 kHz support at the API is certainly nice to have. 

My understanding of how CELT works is that a *native* 44.1 kHz mode with a power-of-2 frame size would break the flexibility to seamlessly switch API sampling rates or coded audio bandwidth on the fly. (Jean-Marc can shine more light on this.) This feature makes it easy for applications to switch during a call between sources or streams that have different sampling rates. 

I'm also not as negative about resampling as you are: modern hybrid-FIR/IIR designs are very efficient, and I'm convinced a 44.1<->48 kHz resampler with transparent quality can be built with 10~20 MACs per sample. And resampling happens everywhere anyway: most hardware ADCs and DACs resample internally, and even Opus internally resamples in many of its modes already. 

So if desired we could support 44.1 kHz at the API level, without losing the nice flexibility properties we've so painstakingly built into the design. (Agree we have to think about how to handle 2.5 and 5 ms frames at 44.1 kHz, but I'm sure we'll find a solution.) 

best, 
koen. 

----- Original Message -----
From: "Raymond (Juin-Hwey) Chen" <[email-masked]> 
To: [email-masked] 
Cc: "jean-marc valin" <[email-masked]> 
Sent: Tuesday, January 25, 2011 4:53:38 PM 
Subject: Re: [codec] requirements #8 (new): Sample rates? 

Raymond 

On Mon, Jan 24, 2011 at 7:40 PM, codec issue tracker < [email-masked] > wrote: 

#8: Sample rates? 

Comment(by gmaxwell@…): 

-- 

Ticket URL: < http://trac.tools.ietf.org/wg/codec/trac/ticket/8#comment:4 > 

codec < http://tools.ietf.org/codec/ >

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 26 Jan 2011 08:51:16 -0500

Hi,

I think it goes beyond just having native support for 44.1 kHz. There are 
several existing applications that are currently using CELT in modes that 
are do not have a 2.5/5/10/20 ms frame size. There are different reasons 
for this depending on the application. Some of these are:
1) Some implementers choose frames that are powers of two so that they can 
use their platform's highly-optimized FFT
2) For ultra low delay applications, such as "jamming over the network" 
remote music performances, one can reduce latency by choosing a frame size 
that matches the soundcard's buffer size (which is almost always a power of 
two).
3) There are even users that consider 2.5 ms to be too high of a delay for 
their applications and use frames of 1.33 ms (64 samples at 48 kHz).
4) In some cases, transport or protocol issues impose some constraints on 
the frame size. For example, the DAB digital radio standard requires frames 
of 24 ms, which CELT can also handle.

I agree that the use cases I listed above may not be part of our "core use 
cases". However, CELT already handles all these cases with no change 
required in the code. Considering that people will keep using CELT for 
these "special" applications, the only question is how we want it to 
happen. I can see essentially two possible approaches:

1) Defining "custom modes" that handle the use cases above and that operate 
differently from the "main modes" that we have defined (e.g. no bandwidth 
switching on the fly).
2) We can exclude these modes from Opus and have these people using "CELT" 
as a non-standard codec.

I don't have a strong opinion on the subject, but I tend to lean towards 1) 
mainly because we would avoid having Opus and CELT as two "competing" 
standards. Also, it would mean that the IETF would have change control over 
all the operating modes instead of "splitting" change control in two. OTOH, 
I do agree that we would need a way to avoid causing compatibility issues 
(perhaps by using a different labelling for these "custom modes").

Cheers,

	Jean-Marc

On 11-01-26 12:59 AM, Koen Vos wrote:
> Hi Raymond,
>
> Streaming of CD music was not an anticipated use case or requirement. That
> said, 44.1 kHz support at the API is certainly nice to have.
>
> My understanding of how CELT works is that a *native* 44.1 kHz mode with a
> power-of-2 frame size would break the flexibility to seamlessly switch API
> sampling rates or coded audio bandwidth on the fly. (Jean-Marc can shine
> more light on this.) This feature makes it easy for applications to switch
> during a call between sources or streams that have different sampling rates.
>
> I'm also not as negative about resampling as you are: modern hybrid-FIR/IIR
> designs are very efficient, and I'm convinced a 44.1<->48 kHz resampler
> with transparent quality can be built with 10~20 MACs per sample. And
> resampling happens everywhere anyway: most hardware ADCs and DACs resample
> internally, and even Opus internally resamples in many of its modes already.
>
> So if desired we could support 44.1 kHz at the API level, without losing
> the nice flexibility properties we've so painstakingly built into the
> design. (Agree we have to think about how to handle 2.5 and 5 ms frames at
> 44.1 kHz, but I'm sure we'll find a solution.)
>
> best,
> koen.
>
>
> ---------------------------------------------------------------------------
> *From: *"Raymond (Juin-Hwey) Chen" <[email-masked]>
> *To: [email-masked]
> *Cc: *"jean-marc valin" <[email-masked]>
> *Sent: *Tuesday, January 25, 2011 4:53:38 PM
> *Subject: *Re: [codec] requirements #8 (new): Sample rates?
>
> I agree that supporting sampling rates of 48, 16, and 8 kHz all makes
> sense, but I also think it would be highly desirable to support
>
> For music streaming applications, a large percentage of the music sources
> will have this 44.1 kHz sampling rate. I know it is
>
> possible to up-sample them to 48 kHz first and then pass them through the
> 48 kHz version of Opus. However, doing such up-
>
> sampling requires additional processing power and additional latency, and
> if the sampling rate conversion is not done with a
>
> high-quality method (which usually requires higher complexity and higher
> delay), audio quality degradation may be introduced
>
> in the process. For these reasons, many people prefer not to do the 44.1
> kHz to 48 kHz sampling rate conversion and prefer to
>
> encode the 44.1 kHz music directly instead. Therefore, it is highly
> desirable for Opus to support this 44.1 kHz sampling rate.
>
> I understand that it may be inconvenient for the SILK mode or the SILK+CELT
> hybrid mode to support 44.1 kHz. However, for the
>
> CELT-only mode, my understanding is that the current CELT C code already
> supports the 44.1 kHz sampling rate. Since the CELT-
>
> only mode is also the most suitable mode to encode music, it would then
> make perfect sense for the CELT-only mode to also
>
> Furthermore, currently the CELT mode at 48 kHz supports frame sizes of 2.5,
> 5., 10, and 20 ms to be consistent with the frame
>
> sizes of the SILK mode and SILK+CELT hybrid mode, and this results in frame
> sizes and FFT window sizes that are not powers of 2,
>
> which reduces the implementation efficiency. If the CELT mode supports the
> 44.1 kHz sampling rate, then since it is no longer
>
> possible to support exactly 2.5, 5, 10, and 20 ms at this sampling rate
> anyway, I think we might as well choose the FFT window
>
> sizes to be powers of 2 to allow more efficient implementation. Again, such
> power of 2 FFT sizes at 44.1 kHz is already supported
>
> In summary, I propose that we allow the CELT-only mode of the Opus codec to
> officially support the 44.1 kHz sampling rate using
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf Of
> *Roman Shpount
> *Sent:* Monday, January 24, 2011 5:08 PM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [codec] requirements #8 (new): Sample rates?
>

---

From: "Pascal Pochol" <[email-masked]>
Date: Wed, 26 Jan 2011 12:06:20 -0600

Hello,

I just wanted to give in my 2 cents about 44.1Khz native support.

99% of all the audio we use celt and eventually opus for are encoded at
44.1Khz. They are provided to us that way. Which means that without 44.1khz
support we'll have to up-sample 99% of our audio most likely in a
preprocess build making us maintain 2 sets of audibly identical files. We
had to do it before with speex where we converted from 44.1 down to 32khz
to use its native ultrawideband but it really wasn't the easiest. We had
thousands of files duplicated and every now and then a few of these getting
updated from the source, forcing us to redo massive conversions each time
to make sure we didn't miss one somewhere.

Also about upsampling not costing much, I beg to differ. We had to work
with hardware that could decode speex ultrawideband 32Khz just fine but the
decoding alone was eating up all our CPU leaving not much else to do the
real work that we needed to do. We had to use 16khz instead to make it all
work. 48Khz, 44.1khz might not look like a big difference when working on a
desktop but when you're counting bytes to see how you can reduce you memory
consumption it could mean the world.

So strickly from a user of the codec's view, native 44.1Khz would certainly
make working with celt/opus a lot easier. I'm guessing that I'm not the
only one in that case based on this thread. Easier would also lead to
faster adoption.

Sorry I didn't intend to write that much. In short 44.1Khz: great if you
can do it, if not we'll just have to work around it.

-Pascal

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 26 Jan 2011 11:32:01 -0800

I'm very concerned that some people may believe that 44.1k is just another checkbox that can be added
without cost or much consideration. This isn't the case. I think it's essential that we delineate realtime VoIP
style usage from other applications.

I support Jean Marc's option (1), which I think allows us to have our cake and eat it too. It's the closest thing to a 
"cost free" option that I think we're going to get. This option basically separates the codec into two profiles,
one which imposes sampling rate restrictions in exchange for many important advantages, and one which
does not, but misses the advantages.  

JM's option (2) would also work. But I don't like the idea of the market confusion created by a totally
separate codec which has significant overlap with Opus, but isn't Opus.  Basically, I think it's silly
for part of the working group's output to compete with itself.  I'd prefer to just have "Opus" and
"Opus-custom" or whatever.

There are several reasons that supporting 44.1kHz in the primary Opus profile would be bad:

One reason for this is that quite a bit of hardware (even on desktops) can only do 48kHz (or closely related
rates) and even when the hardware can do multiple rates it can only ever do one rate at a time so if 
its even possible that multiple applications may play sound at once then the only way to avoid resampling
is if they are all running at a common rate.

For the 48kHz related rates we can do very computationally cheap handling of different rates purely
inside the codec. If their hardware supports any mode out of the 48kHz family, then they'll need no costly
resampling at all. And if they do  need run at 44.1k they can resample in and out of the codec without imposing on (or
negotiating with) the far end.

Another one is that Opus (as described in the draft, without 44.1kHz) can switch between any of
its supported modes, all on the fly, without creating any surprising impositions on the clients.
This is possible only because of the closely related nature of the supported rates, and 44.1kHz
can't be accommodated in this scheme.

The on the fly switching and lack of requirement to negotiate, also means that two opus devices can
 communicate without transcoding, even if they were spliced long after negotiation. (e.g. as part
of a conference gateway).

On the other end of the spectrum— the current CELT library and bitstream is extremely flexible.
It supports a great many frame sizes and sample rates and I've personally argued against
every limitation we've imposed on it, because I like the idea that CELT can fit into every
niche requirement (like the DAB frame sizes).

But this flexibility has a serious price for interoperable implementations: They must carry substantially
more code (the limited rates/sample sizes means that a simple table can replace several hundred
lines of tricky bit-exact initialization code), cope with increased peak CPU usage (e.g. if some device
only speaks 64 sample frames, 96KHz you might need 10x the CPU power to speak to it compared 
to your preferred mode), and undertake more complicated negotiation and testing (CELT can support
far more unique mixtures of sample rate, frame-size, and channel count then there are RTP payload
types). 

So basically, I support fully supporting oddball configurations as a well specified standardized mode,
but I oppose subjecting the general VoIP/RTP users to the increase complexity and limitations of
the more rate/framesize agile configurations.

I expect that most users which care about the 'custom' modes are doing other specialized things
and won't be expected to ever interop with a random Opus phone except (maybe) via a gateway,
and that most of them won't even speak RTP— so the separation shouldn't even make 
much difference to them at all.

Thoughts?

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Pascal Pochol [[email-masked]]
Sent: Wednesday, January 26, 2011 6:06 PM
To: Jean-Marc Valin
Cc: [email-masked]
Subject: Re: [codec] requirements #8 (new): Sample rates?

Hello,

-Pascal

---

From: "Pascal Pochol" <[email-masked]>
Date: Wed, 26 Jan 2011 14:54:29 -0600

Gregory,

> I'm very concerned that some people may believe that 44.1k is just
> another checkbox that can be added without cost or much consideration.

I don't think that's the case. We all know that if it was easy to handle
every sample rate imaginable that it would have been done already.

Again talking about our personal case which has nothing to do with VOIP we
have a lot of audio data handed to us in 44.1Khz and 22.5Khz. Other popular
rates for us are 16Khz and 8Khz which are already handled if I'm not
mistaken by opus? Easily derived from 48khz. Before using speex last year I
had never encountered anything at 32khz. It'll probably a few more years
before we're allowed to use cpus powerful enough and enough ram to handle
even 32khz. For now we'll most likely still down sample everything to
16khz. The artists don't like programmers massaging their data but they
understand technical limitations so for now we're in luck.

> I support Jean Marc's option (1), which I think allows us to have our
> cake and eat it too. It's the closest thing to a "cost free" option
> that I think we're going to get. This option basically separates the
> codec into two profiles, one which imposes sampling rate restrictions
> in exchange for many important advantages, and one which does not, but
> misses the advantages.

Out of curiosity, what would be the advantages that we miss in such case?

> JM's option (2) would also work. But I don't like the idea of the
> market confusion created by a totally separate codec which has
> significant overlap with Opus, but isn't Opus.  Basically, I think it's
> silly for part of the working group's output to compete with itself.
> I'd prefer to just have "Opus" and "Opus-custom" or whatever.

I totally agree.

> I expect that most users which care about the 'custom' modes are doing
> other specialized things and won't be expected to ever interop with a
> random Opus phone except (maybe) via a gateway, and that most of them
> won't even speak RTP— so the separation shouldn't even make much
> difference to them at all.

Absolutely right In our case we'd be using for things that wouldn't need to
interact with any other devices actually. We use it as a very efficient way
to decode stored pre-encoded music/speech streams.

-Pascal

---

From: Koen Vos <[email-masked]>
Date: Wed, 26 Jan 2011 22:08:51 +0100 (CET)

So what needs to be decided is:

1. How to call these custom modes (for instance in the SDP descriptor).  As long as we agree that it is not "Opus", then we prevent compatibility issues and we keep our on-the-fly switching flexibility within the standard Opus format.  I'm fine with "Opus-custom".

2. How to enable the custom modes in the code.  I think there must be a small barrier so that users don't unintentionally start using them.  Maybe a well-commented #define somewhere, plus an API control flag to put the codec in custom mode?

3. Should there be a set of "official" custom modes, to aid interop even with Opus-custom?  Users could still pick and choose any modes they want, but having the list could make users gravitate towards a few common choices.

koen.

----- Original Message -----
From: "Gregory Maxwell" <[email-masked]>
To: [email-masked], "Jean-Marc Valin" <[email-masked]>
Cc: [email-masked]
Sent: Wednesday, January 26, 2011 11:32:01 AM
Subject: Re: [codec] requirements #8 (new): Sample rates?

Thoughts?

Hello,

-Pascal

---

From: Koen Vos <[email-masked]>
Date: Wed, 26 Jan 2011 22:18:56 +0100 (CET)

And of course: How to signal the custom modes within the packet?

----- Original Message -----
From: "Koen Vos" <[email-masked]>
To: "Gregory Maxwell" <[email-masked]>
Cc: [email-masked], [email-masked]
Sent: Wednesday, January 26, 2011 1:08:51 PM
Subject: Re: [codec] requirements #8 (new): Sample rates?

koen.

----- Original Message -----
From: "Gregory Maxwell" <[email-masked]>
To: [email-masked], "Jean-Marc Valin" <[email-masked]>
Cc: [email-masked]
Sent: Wednesday, January 26, 2011 11:32:01 AM
Subject: Re: [codec] requirements #8 (new): Sample rates?

Thoughts?

Hello,

-Pascal

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 26 Jan 2011 16:22:37 -0500

On 11-01-26 04:08 PM, Koen Vos wrote:
> 1. How to call these custom modes (for instance in the SDP descriptor).
> As long as we agree that it is not "Opus", then we prevent compatibility
> issues and we keep our on-the-fly switching flexibility within the
> standard Opus format.  I'm fine with "Opus-custom".

OK, unless there's objections I think we can go with "Opus-custom".

> 2. How to enable the custom modes in the code.  I think there must be a
> small barrier so that users don't unintentionally start using them.
> Maybe a well-commented #define somewhere, plus an API control flag to
> put the codec in custom mode?

Agreed. The API will *have* to be different because in the normal mode 
there won't even be a need to specify things like frame size. As for a 
#define it also makes sense in terms of saving code size by default.

> 3. Should there be a set of "official" custom modes, to aid interop even
> with Opus-custom?  Users could still pick and choose any modes they
> want, but having the list could make users gravitate towards a few
> common choices.

I think the following modes are main custom modes we want to recommend:
1) 48 kHz, frame sizes 128, 256, 512, 1024 (switchable)
2) 44.1 kHz, frame sizes 128, 256, 512, 1024 (switchable)
3) 48 kHz, frame size 64
4) 44.1 kHz frame size 64

though the other modes will still be available.

Cheers,

	Jean-Marc

> koen.
>
>
>
> ----- Original Message ----- From: "Gregory
> Maxwell"<[email-masked]> To: [email-masked], "Jean-Marc
> Valin"<[email-masked]> Cc: [email-masked] Sent: Wednesday,
> January 26, 2011 11:32:01 AM Subject: Re: [codec] requirements #8 (new):
> Sample rates?
>
>
>
> This isn't the case. I think it's essential that we delineate realtime
> VoIP style usage from other applications.
>
> cake and eat it too. It's the closest thing to a "cost free" option that
> I think we're going to get. This option basically separates the codec
> into two profiles, one which imposes sampling rate restrictions in
> exchange for many important advantages, and one which does not, but
> misses the advantages.
>
> JM's option (2) would also work. But I don't like the idea of the market
> confusion created by a totally separate codec which has significant
> overlap with Opus, but isn't Opus.  Basically, I think it's silly for
> part of the working group's output to compete with itself.  I'd prefer
> to just have "Opus" and "Opus-custom" or whatever.
>
>
> One reason for this is that quite a bit of hardware (even on desktops)
> can only do 48kHz (or closely related rates) and even when the hardware
> can do multiple rates it can only ever do one rate at a time so if its
> even possible that multiple applications may play sound at once then the
> only way to avoid resampling is if they are all running at a common
> rate.
>
> For the 48kHz related rates we can do very computationally cheap
> handling of different rates purely inside the codec. If their hardware
> supports any mode out of the 48kHz family, then they'll need no costly
> resampling at all. And if they do  need run at 44.1k they can resample
> in and out of the codec without imposing on (or negotiating with) the
> far end.
>
> Another one is that Opus (as described in the draft, without 44.1kHz)
> can switch between any of its supported modes, all on the fly, without
> creating any surprising impositions on the clients. This is possible
> only because of the closely related nature of the supported rates, and
> 44.1kHz can't be accommodated in this scheme.
>
> The on the fly switching and lack of requirement to negotiate, also
> means that two opus devices can communicate without transcoding, even if
> they were spliced long after negotiation. (e.g. as part of a conference
> gateway).
>
>
> On the other end of the spectrum— the current CELT library and
> bitstream is extremely flexible. It supports a great many frame sizes
> and sample rates and I've personally argued against every limitation
> we've imposed on it, because I like the idea that CELT can fit into
> every niche requirement (like the DAB frame sizes).
>
> But this flexibility has a serious price for interoperable
> implementations: They must carry substantially more code (the limited
> rates/sample sizes means that a simple table can replace several
> hundred lines of tricky bit-exact initialization code), cope with
> increased peak CPU usage (e.g. if some device only speaks 64 sample
> frames, 96KHz you might need 10x the CPU power to speak to it compared
> to your preferred mode), and undertake more complicated negotiation and
> testing (CELT can support far more unique mixtures of sample rate,
> frame-size, and channel count then there are RTP payload types).
>
> So basically, I support fully supporting oddball configurations as a
> well specified standardized mode, but I oppose subjecting the general
> VoIP/RTP users to the increase complexity and limitations of the more
> rate/framesize agile configurations.
>
> ________________________________________ From: [email-masked]
> [[email-masked]] On Behalf Of Pascal Pochol
> [[email-masked]] Sent: Wednesday, January 26, 2011 6:06 PM To:
> Jean-Marc Valin Cc: [email-masked] Subject: Re: [codec] requirements #8
> (new): Sample rates?
>
> Hello,
>
> 44.1Khz. They are provided to us that way. Which means that without
> 44.1khz support we'll have to up-sample 99% of our audio most likely in
> a preprocess build making us maintain 2 sets of audibly identical files.
> We had to do it before with speex where we converted from 44.1 down to
> 32khz to use its native ultrawideband but it really wasn't the easiest.
> We had thousands of files duplicated and every now and then a few of
> these getting updated from the source, forcing us to redo massive
> conversions each time to make sure we didn't miss one somewhere.
>
> with hardware that could decode speex ultrawideband 32Khz just fine but
> the decoding alone was eating up all our CPU leaving not much else to do
> the real work that we needed to do. We had to use 16khz instead to make
> it all work. 48Khz, 44.1khz might not look like a big difference when
> working on a desktop but when you're counting bytes to see how you can
> reduce you memory consumption it could mean the world.
>
> So strickly from a user of the codec's view, native 44.1Khz would
> certainly make working with celt/opus a lot easier. I'm guessing that
> I'm not the only one in that case based on this thread. Easier would
> also lead to faster adoption.
>
> Sorry I didn't intend to write that much. In short 44.1Khz: great if
> you can do it, if not we'll just have to work around it.
>
> -Pascal
>
> _______________________________________________ codec mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________ codec mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 26 Jan 2011 16:27:01 -0500

> And of course: How to signal the custom modes within the packet?

The custom modes will have to be negotiated at the SDP level. Unlike "Opus" 
packets, you will just not be able to decode an "Opus-custom" packet 
without knowing what was negotiated.

     Jean-Marc

> From: "Koen Vos"<[email-masked]>
> To: "Gregory Maxwell"<[email-masked]>
> From: "Gregory Maxwell"<[email-masked]>
> To: [email-masked], "Jean-Marc Valin"<[email-masked]>
> Cc: [email-masked]

---

From: Koen Vos <[email-masked]>
Date: Thu, 27 Jan 2011 01:07:13 +0100 (CET)

In your previous email you mentioned two custom modes with switchable frame sizes.  For those modes at least the frame size will have to be signaled in each packet, no?

I could see a benefit in making all recommended custom modes "instantly" decodable (meaning without prior knowledge).  Although it does add another layer of "standard custom" vs "custom custom" which may confuse people.

I'm fine either way.
koen.

----- Original Message -----
From: "Jean-Marc Valin" <[email-masked]>
To: "Koen Vos" <[email-masked]>
Cc: "Gregory Maxwell" <[email-masked]>, [email-masked], [email-masked]
Sent: Wednesday, January 26, 2011 1:27:01 PM
Subject: Re: [codec] requirements #8 (new): Sample rates?

     Jean-Marc

> On the other end of the spectrumâ€” the current CELT library and bitstream is extremely flexible.
> and that most of them won't even speak RTPâ€” so the separation shouldn't even make

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 26 Jan 2011 20:14:55 -0500

What I meant in my previous email was that these modes are potentially 
switchable. How exactly (or whether) we want to handle that is still an 
open question. It even something we could handle in the RTP payload 
format. Overall, I don't have a strong opinion on switching of custom modes.

	Jean-Marc

On 11-01-26 07:07 PM, Koen Vos wrote:
> From: "Jean-Marc Valin"<[email-masked]>
> To: "Koen Vos"<[email-masked]>
> Cc: "Gregory Maxwell"<[email-masked]>, [email-masked], [email-masked]

---

From: "Pascal Pochol" <[email-masked]>
Date: Wed, 26 Jan 2011 21:12:08 -0600

Jean-Marc,

> The custom modes will have to be negotiated at the SDP level. Unlike
> "Opus" packets, you will just not be able to decode an "Opus-custom"
> packet without knowing what was negotiated.

in our case I don't even think that we need a per packet signal to tell the
decoder. Unless the Opus-custom encoder does something that varies per
packet we usually set the parameters up front and never change them when
encoding a stream. We could have a little header packet to set things up or
since we're in total control simply set the parameter in the initialization
of the decoder with a few api calls. Not sure if that would work for others
but in our case I think it would without adding extra information to the
packets.

-Pascal

---

From: Butrus Damaskus <[email-masked]>
Date: Thu, 27 Jan 2011 17:04:49 +0100

> but in our case I think it would without adding extra information to the

Well, but you probably need at least one bit to signal header/data packets, no?

---

From: "Pascal Pochol" <[email-masked]>
Date: Thu, 27 Jan 2011 12:18:35 -0600

Brutus,

> Well, but you probably need at least one bit to signal header/data

for what we do, no. We open a stream we can easily assume the header is at
the beginning and everything after that is data packets. We don't have
anything that can join in midstream or change the format midstream or
anything like that. Again this is very specific to what we do it might not
fit others needs. I wouldn't even have a problem if we simply had to call a
few api functions before encoding to set everything up and call a few api
functions before decoding with the same type of parameters to let the
decoder know what it's about to receive and not even use a header. Since we
write both encoder and decoder programs to fit our needs and nobody else
reads our encoded data.

when we used speex we started with the standard speexenc/speexdec format
encapsulated into an ogg. But that was too complex so we dropped the ogg
container and used samplexenc.c and sampledec.c instead. We made our
version of sampleenc.c spit out a tiny header at the beginning that would
give us some basic information that we needed in order to decode
everything. I don't remember off hand what it contained one was the
original number of samples I think. And we loaded that in our decoder and
that was it. the rest of the files were all the smallest speex packets we
could make. We noticed that the first byte was a constant size of the frame
so we dropped that out as well and moved it into the header. That saved a
few hundreds to a few thousands bytes for each file (a few thousands of
files total) for a huge saving. For us a byte is still a full 8 bits which
can be used to store a lot of information, so I perfectly understand and
appreciate this group's concern about adding a few bits to each packets. My
personal vote for opus-custom would be to use api calls and not store any
more data than absolutely necessary in the stream. Since using the custom
mode mean using custom code on both ends it's not a problem for us.

-Pascal

---

From: Gregory Maxwell <[email-masked]>
Date: Thu, 27 Jan 2011 10:21:06 -0800

>Well, but you probably need at least one bit to signal header/data packets, no?

No— the codec itself doesn't carry quite a bit of information which is important
to applications already. E.g. it doesn't carry timing.  These things are left
to upper level protocols to handle.

The same can, and should, be done for configuration negotiation. The reason for
this is that the transport can be well adapted to the environment.   If there were
a 'header packet' that the codec generated only once and sent as part of
the bitstream, if that packet were lost the all the rest of the audio would be
undecodable.   So the only real options are: send it every packet or leave it
for another mechanism to handle.

Even if you wanted to reinvent a whole two way negotiation machine inside the
codec— you couldn't really, because you can't depend on the usage being
duplex.

For modes which can be reasonably switched on the fly sending it in every
packet provides some value.  For ones which can't be— sending it in every
packet is overhead, and worse— a possible opening for a malicious
transmitter to trigger poorly tested corner-cases.

A complete selection of modes that CELT can operate in (when restrictions to
simplify implementation are not imposed) takes about 4 bytes, more if you 
want to add some length coding in order to support packing multiple codec
frames in one packet.   Thats a pretty high overhead for a 64 sample/frame 
mode which is sending 700pps, especially in the case of a lossless
transport for which the configuration would only ever have to be sent once.

So, I think that in packet signaling should generally be limited only
to signalling switches that the codec can actually do on the fly, and
I think for the opus-custom modes it's probably better to not signal in
the packets at all.  If an application needs switching for opus-custom,
it could define a new encapsulation for its transport which provides for it.