Subject: Re: [codec] Seperate list for IPR discutions?

From: Gregory Maxwell <[email-masked]>
Date: Wed, 20 Oct 2010 10:50:00 -0700

There is, has been, and will be plenty of discussion in _private_. Being _willfully_ ignorant of your obligations which you would otherwise know about is just willfully infringing in a pretty dress.

The goal of the developers here is to not infringe on third part patents. Full stop.

The objection is to a public discussion- because patent discussion must be informed by the relevant experts, because it no one wants to be dragged in to testify on random patents, and because exposing your strategy to a possible attacker makes it weaker, etc.

________________________________
From: [email-masked] <[email-masked]>
To: Jean-Marc Valin <[email-masked]>
Cc: [email-masked] <[email-masked]>
Sent: Wed Oct 20 10:42:27 2010
Subject: Re: [codec] Seperate list for IPR discutions?

Stephan's suggestion on the boundary  "citations of patent language, including patent numbers and claim language" is consistent with your boundary.  Detailed analysis of proposed work-arounds, would presumably include patent language.  General discussion of license terms would not.

Since I wouldn't subscribe or otherwise access the new list, it doesn't matter to me if it is publicly archived or not.

Based on past IETF experience, it is quite difficult to convince posters to go off-line for a private discussion.  One practical issue is that the folks who do want to discuss this stuff have no easy way to identify themselves.  I'm thinking that providing the second list might make that easier.  Though perhaps no one but Christian is willing to talk about patent claims?

Stephen Botzko

On Wed, Oct 20, 2010 at 1:27 PM, Jean-Marc Valin <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

Just to make things clear, I think there's two sets of issues regarding IPR. One is discussing the exact terms of IPR disclosure/licensing and the other is discussion about patent themselves. I think the former should remain on this list. OTOH, I think the latter is (unfortunately) best dealt with in private. I don't think there is anything to be gained in discussing patent claims in a publicly archived list. I don't think even having that on a separate list is a good idea and I'm not interested in participating in such a list should it be created.

    Jean-Marc

On 10-10-20 01:03 PM, Stephen Botzko wrote:
+1, I'd like to see it handled precisely this way.

I think it provides a good way forward, and allows those of us who do
not want exposure to patent details to continue to participate on the
main list.

Stephen Botzko

On Wed, Oct 20, 2010 at 12:40 PM, Stephan Wenger <[email-masked]<mailto:[email-masked]>
<mailto:[email-masked]<mailto:[email-masked]>>> wrote:

   Hi,

   Yes, I think a separate mailing list would be an excellent idea.
     There is
   IMO no danger to the IETF when using an IETF-hosted list.  But, as you
   mentioned, there are also other options.  Whatever is the easiest
   for those
   interested in this work.

   My preference would be that this list has a subscription process
   independent
   from the subscription to [email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>.  As

   the minimum, there must be an
   opt-out process.

   I would recommend that the maintainers of the list, periodically, make
   people on [email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>> aware of the

   existence of the list and the ongoing
   discussions, in a summarized form.  I also don't mind if suggested
   design
   choices spill over to codec@ietf; after all, this is the list where
   technical discussions are being held.  The borderline would be
   citations of
   patent language, including patent numbers and claim language.

   Personally, I surely do not want to be involved on this hypothetical new
   list.

   Stephan

   On 10.20.2010 06:42 , "[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>"

   <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>> wrote:

    >
    > Stephan, as our WG technical advisor on topics like this, do you
   have advice
    > on if we should set up a separate list for people that want to
   talk about
    > patent issues related to the standards from this WG? From a
   practical point of
    > view, someone could easily set up a separate list either on the
   ietf.org<http://ietf.org> <http://ietf.org>

    > servers or not on ietf servers.

   _______________________________________________
   codec mailing list
   [email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>

   https://www.ietf.org/mailman/listinfo/codec

_______________________________________________
codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/codec

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 20 Oct 2010 11:43:38 -0700

For the record: Without being interested in any discussion in this venue, I
disagree with your statement ³Being _willfully_ ignorant of your obligations
which you would otherwise know about is just willfully infringing in a
pretty dress.².  

I support the rest of your email.
Stephan

On 10.20.2010 10:50 , "Gregory Maxwell" <[email-masked]> wrote:

> 
> There is, has been, and will be plenty of discussion in _private_.   Being
> _willfully_ ignorant of your obligations which you would otherwise know about
> is just willfully infringing in a pretty dress.
> 
> The objection is to a public discussion- because patent discussion must be
> informed by the relevant experts, because it no one wants to be dragged in to
> testify on random patents, and because exposing your strategy to a possible
> attacker makes it weaker, etc.
> 
> 
> 
> 
> 
> 
> Stephan's suggestion on the boundary  "citations of patent language, including
> patent numbers and claim language" is consistent with your boundary.  Detailed
> analysis of proposed work-arounds, would presumably include patent language. 
> General discussion of license terms would not.
> 
> Based on past IETF experience, it is quite difficult to convince posters to go
> off-line for a private discussion.  One practical issue is that the folks who
> do want to discuss this stuff have no easy way to identify themselves.  I'm
> thinking that providing the second list might make that easier.  Though
> perhaps no one but Christian is willing to talk about patent claims?
> 
> Stephen Botzko
> 
> On Wed, Oct 20, 2010 at 1:27 PM, Jean-Marc Valin <[email-masked]>
> wrote:
>> Hi,
>> 
>> Just to make things clear, I think there's two sets of issues regarding IPR.
>> One is discussing the exact terms of IPR disclosure/licensing and the other
>> is discussion about patent themselves. I think the former should remain on
>> this list. OTOH, I think the latter is (unfortunately) best dealt with in
>> private. I don't think there is anything to be gained in discussing patent
>> claims in a publicly archived list. I don't think even having that on a
>> separate list is a good idea and I'm not interested in participating in such
>> a list should it be created.
>> 
>>      Jean-Marc
>> 
>> 
>>> On Wed, Oct 20, 2010 at 12:40 PM, Stephan Wenger <[email-masked]
>>> <mailto:[email-masked]>> wrote:
>>> 
>>>     Hi,
>>> 
>>>     from the subscription to [email-masked] <mailto:[email-masked]>.  As
>>> 
>>>     people on [email-masked] <mailto:[email-masked]> aware of the
>>> 
>>>     On 10.20.2010 06:42 , "[email-masked] <mailto:[email-masked]>"
>>> 
>>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>>      >
>>>      > Stephan, as our WG technical advisor on topics like this, do you
>>>     have advice
>>>      > on if we should set up a separate list for people that want to
>>>     talk about
>>>      > patent issues related to the standards from this WG? From a
>>>     practical point of
>>>      > view, someone could easily set up a separate list either on the
>>>     ietf.org <http://ietf.org>  <http://ietf.org>
>>> 
>>>      > servers or not on ietf servers.
>>> 
>>> 
>>>     [email-masked] <mailto:[email-masked]>
>>> 
>>> [email-masked]
> [email-masked]

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 20 Oct 2010 15:39:34 -0700

Stephan,

I think that many of us appreciate your cautionary advice and the benefit of your experience. I certainly do.

But I think participants also need to be careful to take it only as cautionary advice and not the gospel truth. I doubt you want your words to create FUD and thereby make participants here afraid to act, but I think your use such confident and firm language may sometimes have that effect. This risk of reasonable caution having the unintended effect of FUD is especially great when your comments are vague.  E.g. I understand why you might advise against discussing licensing terms in some places— due to concerns about creating collusion problems—  but I can not fathom how an antitrust issues could arise when discussing free-to-all licensing. Licensing terms are routinely discussed in the context of free software projects, even with the involvement of billion dollar corporations.  Your comments on the issue of licensing weren't even specific enough for me to ask an attorney if you're right or not.  Responses like that create a cloud of FUD which removes enlightenment rather than creating it.

Some of the advice you've given here directly contradicts the advice I've received from council.  I would encourage people here interested in legal issues consult their own council and obtain an analysis specific to their own circumstances and needs.

In particular, the issue of "willful infringement" seems to frequently suffer from substantial misinformation or, really, out of date information. The information you've provided here is, to the best of my understanding, incorrect.

The effective standard for willful infringement applied by US courts changed a few years ago as a result of In re Seagate[1].   After a brief search I found this reasonable looking overview which reflects the current state of the world: http://www.patentbaristas.com/archives/2007/08/24/is-the-tide-of-willful-patent-infringement-litigation-receding/ (but there are a great many other writeups about this case out there). Things also appear to be stationary on this new precedent (the supreme court declined an appeal of this decision).

The exact nuance of what behaviour is sufficiently reckless to constitute willful infringement in the context of the In re Seagate decision is still an evolving area. But the old 'you must get a pass from a patent attorney every time you see a patent number' is simply no longer true. The various news and journal articles on this case discuss this subject in depth. 

The increasingly common advice now given to software developers interested in creating non-infringing software is "read the patents", even for people without the benefit of the active involvement of a patent attorney. People without a lot of background in this area may find this article to be very informative: http://news.swpat.org/2010/03/transcript-tridgell-patents/

Almost nowhere in law does _intentionally_ blinding yourself because you believe that you will likely find something which creates obligations actually relieve you of those obligations (go look up "constructive knowledge"), so we shouldn't be surprised to find patent law reflecting this greater pattern.

In particular I should draw your attention to SEB S.A. v. Montgomery Ward & Co., Inc.[2]  where the court determined that "deliberate indifference"  was equivalent to specific intent— a requirement for a claim of inducement.

The matter is certainly not clear or settled— which is why I strongly encourage people to consult their own council—  but it appears clear that risk of awareness of patent claims is being significantly overstated and the suggestion that one can get away with intentionally avoiding knowing seems to be both legally risky and socially reprehensible.

Besides— there are plenty of good reasons to not spend time looking at patent claims besides believing that you'll learn about something troublesome:  It's somewhat difficult and time consuming.    If you believe that participation here is so dangerous that you can only do so safely by being wilfully blind to the specifics of risks that you believe you exist, then perhaps you should reconsider your participation.

I want to reiterate that in spite of the above that I'm also not interested in having a public patent discussion here or on another list but I want to make it clear that my lack of interest has nothing to do with an attempt to avoid knowing about patent risk.  I know risk exists and I want to be made aware of existing patents which may pose a risk for the usability or royalty free status of the codec. I have been involved in the review of patents for implications on the involved codec(s), and there will be more review as the work continues. I just don't believe that a public discussion is a productive way to conduct this activity.  And I believe this statement is true for all the technical contributors to our current codec.

Cheers,

[1] http://www.fedcir.gov/opinions/M830.pdf
[2] http://caselaw.findlaw.com/us-federal-circuit/1506829.html / http://www.cafc.uscourts.gov/images/stories/opinions-orders/09-1099.pdf (real link seems to be down)

________________________________________
From: Stephan Wenger [[email-masked]]
Sent: Wednesday, October 20, 2010 7:43 PM
To: Gregory Maxwell; [email-masked]'; [email-masked]'
Cc: [email-masked]'
Subject: Re: [codec] Seperate list for IPR discutions?

For the record: Without being interested in any discussion in this venue, I disagree with your statement “Being _willfully_ ignorant of your obligations which you would otherwise know about is just willfully infringing in a pretty dress.”.

Stephen Botzko

On Wed, Oct 20, 2010 at 1:27 PM, Jean-Marc Valin <[email-masked]> wrote:
Hi,

     Jean-Marc

Stephen Botzko

    Hi,

    Stephan

    have advice
    talk about
    practical point of

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 22 Oct 2010 09:40:57 -0700

Hi all,

Sorry for my late and short reply.  I'm currently a bit time pressed.

I reviewed my recent emails to the list (over the last week or so), and
found nothing that made me jump to post a correction.  Which does not mean
that I may not do so in the future...

One key point of Greg's message I wholeheartedly agree to: when in doubt,
consult your own laywer.

As of the rest of Greg's email, I agree with many of Greg's points below,
but also disagree with a few.  However, I'm not prepared to discuss my
points on this public list.  If Greg is willing, we could probably have a
bit of a dialogue between us two (+ whatever advisors we are pulling in),
and, if we were agreeing so, could post a summary at a later date.  Greg,
please let me know if you are interested.

Regards,
Stephan

On 10.20.2010 15:39 , "Gregory Maxwell" <[email-masked]> wrote:

> But I think participants also need to be careful to take it only as cautionary
> advice and not the gospel truth. I doubt you want your words to create FUD and
> thereby make participants here afraid to act, but I think your use such
> confident and firm language may sometimes have that effect. This risk of
> reasonable caution having the unintended effect of FUD is especially great
> when your comments are vague.  E.g. I understand why you might advise against
> discussing licensing terms in some places‹ due to concerns about creating
> collusion problems‹  but I can not fathom how an antitrust issues could arise
> when discussing free-to-all licensing. Licensing terms are routinely discussed
> in the context of free software projects, even with the involvement of billion
> dollar corporations.  Your comments on the issue of licensing weren't even
> specific enough for me to ask an attorney if you're right or not.  Responses
> like that create a cloud of FUD which removes enlightenment rather than
> creating it.
> 
> Some of the advice you've given here directly contradicts the advice I've
> received from council.  I would encourage people here interested in legal
> issues consult their own council and obtain an analysis specific to their own
> circumstances and needs.
> 
> In particular, the issue of "willful infringement" seems to frequently suffer
> from substantial misinformation or, really, out of date information. The
> information you've provided here is, to the best of my understanding,
> incorrect.
> 
> The effective standard for willful infringement applied by US courts changed a
> few years ago as a result of In re Seagate[1].   After a brief search I found
> this reasonable looking overview which reflects the current state of the
> world: 
> http://www.patentbaristas.com/archives/2007/08/24/is-the-tide-of-willful-paten
> t-infringement-litigation-receding/ (but there are a great many other writeups
> about this case out there). Things also appear to be stationary on this new
> precedent (the supreme court declined an appeal of this decision).
> 
> The exact nuance of what behaviour is sufficiently reckless to constitute
> willful infringement in the context of the In re Seagate decision is still an
> evolving area. But the old 'you must get a pass from a patent attorney every
> time you see a patent number' is simply no longer true. The various news and
> journal articles on this case discuss this subject in depth.
> 
> The increasingly common advice now given to software developers interested in
> creating non-infringing software is "read the patents", even for people
> without the benefit of the active involvement of a patent attorney. People
> without a lot of background in this area may find this article to be very
> informative: http://news.swpat.org/2010/03/transcript-tridgell-patents/
> 
> Almost nowhere in law does _intentionally_ blinding yourself because you
> believe that you will likely find something which creates obligations actually
> relieve you of those obligations (go look up "constructive knowledge"), so we
> shouldn't be surprised to find patent law reflecting this greater pattern.
> 
> In particular I should draw your attention to SEB S.A. v. Montgomery Ward &
> Co., Inc.[2]  where the court determined that "deliberate indifference"  was
> equivalent to specific intent‹ a requirement for a claim of inducement.
> 
> The matter is certainly not clear or settled‹ which is why I strongly
> encourage people to consult their own council‹  but it appears clear that risk
> of awareness of patent claims is being significantly overstated and the
> suggestion that one can get away with intentionally avoiding knowing seems to
> be both legally risky and socially reprehensible.
> 
> Besides‹ there are plenty of good reasons to not spend time looking at patent
> claims besides believing that you'll learn about something troublesome:  It's
> somewhat difficult and time consuming.    If you believe that participation
> here is so dangerous that you can only do so safely by being wilfully blind to
> the specifics of risks that you believe you exist, then perhaps you should
> reconsider your participation.
> 
> I want to reiterate that in spite of the above that I'm also not interested in
> having a public patent discussion here or on another list but I want to make
> it clear that my lack of interest has nothing to do with an attempt to avoid
> knowing about patent risk.  I know risk exists and I want to be made aware of
> existing patents which may pose a risk for the usability or royalty free
> status of the codec. I have been involved in the review of patents for
> implications on the involved codec(s), and there will be more review as the
> work continues. I just don't believe that a public discussion is a productive
> way to conduct this activity.  And I believe this statement is true for all
> the technical contributors to our current codec.
> 
> Cheers,
> 
> 
> [2] http://caselaw.findlaw.com/us-federal-circuit/1506829.html /
> http://www.cafc.uscourts.gov/images/stories/opinions-orders/09-1099.pdf (real
> link seems to be down)
> 
> 
> which you would otherwise know about is just willfully infringing in a pretty
> dress.².
> 
> One is discussing the exact terms of IPR disclosure/licensing and the other is
> discussion about patent themselves. I think the former should remain on this
> list. OTOH, I think the latter is (unfortunately) best dealt with in private.
> I don't think there is anything to be gained in discussing patent claims in a
> publicly archived list. I don't think even having that on a separate list is a
> good idea and I'm not interested in participating in such a list should it be
> created.
> 
>      Jean-Marc
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 22 Oct 2010 22:13:24 -0700

All,

In the last few weeks, I have been working on a pitch prefiltering/postfiltering method to try to 
improve the output audio quality of the IETF mode 1 CELT codec for certain audio signals that CELT 
low-delay modes give more audible coding distortion than usual (typically nearly periodic solo 
music instrument signals).  I am happy to report that this method has achieved fairly significant 
audio quality improvement for many of these kinds of audio signals that I tried.  This is achieved 
without increasing the coding delay and with only a slight increase in the codec complexity and 
encoding bit-rate (a potential increase of around 2 kb/s or so; less if a range coder is used).

I have been in discussion with Jean-Marc about this and have sent him CELT-coded audio signals at 
different bit-rates obtained with or without this method.  After listening, he agreed that the 
method made a significant improvement when compared with the current CELT implementation, and he 
would like to try to incorporate this method into his CELT C code to evaluate and verify the 
performance gain and possibly further develop the method.  

Jean-Marc asked me to write a detailed description of the method in a document and send the 
document to the IETF codec WG email reflector so he can read it, understand it, and then implement 
it himself in his CELT C code.  I am attaching such a technical description of the method.

Note that although this document is 7 pages long, the pitch prefiltering/postfiltering method is 
actually quite simple.  The document is long only because (1) I tried to describe the technical 
details as much as I can so Jean-Marc can implement it exactly as I had it without any ambiguity, 
and (2) I also described/discussed a few alternative approaches for some of the algorithm 
components in case we need to go those routes.

Note also that this is "fresh out of the oven" as I only had time to finish a very rough first cut 
for a proof-of-concept simulation and did not have time yet to refine or tune it.  Thus, I am sure 
there are still many rough edges that need to be smoothed out and many areas that can be improved 
upon.  However, I am very much encouraged that even such a rough first cut showed a lot of promise 
as it made a very large audio quality improvement at several different bit-rates for a trumpet 
test signal, which is a signal that CELT low-delay modes have problems with.  Therefore, after 
this pitch prefiltering/postfiltering method is integrated into CELT and further refined/tuned, I 
think it has the potential to make the IETF mode 1 CELT codec more robust to different audio 
signals.  That is, it can potentially improve CELT performance for many signals that are currently 
difficult for CELT to code well, and thus ensure a more uniform audio quality level across a wider 
variety of different audio signals.  If Jean-Marc's integration, testing, and further development 
of this pitch filtering method confirms this potential, then it is proposed that this pitch 
prefiltering/postfiltering method be incorporated into Mode 1 of the IETF Opus codec to make the 
codec more robust to different input signal types.

Cheers,

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 23 Oct 2010 08:52:14 -0400

Hi Raymond,

Thank you very much for your contribution and the highly detailed
description you just sent. I believe I have enough information to start
implementing it. At this point, I only have one minor question. When
estimating the pitch period/gain, do you recommend working on the input
signal directly or should the input be "whitened" using a filter like
A(z/gamma) (where A(z) is the LPC filter)?

Thanks again,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 23 Oct 2010 12:50:05 -0700

Hi Jean-Marc,

You are very welcome. Glad to contribute and glad to know that the document is good 
enough for you to start implementing the techniques.

Regarding your question, I think it really depends on what your main emphasis is.  
If you just want the best possible pitch period contour and don't mind having a 
little higher complexity, then partially whitening the input signal with the 
A(z/gamma) or A(z)/A(z/gamma) or even A(z/alpha)/A(z/beta) is a common approach used 
in the pitch estimators of many speech codecs (as I am sure you are well aware of), 
and it does help to reduce the chance of detecting the frequency of a strong first 
formant of the input speech signal as the pitch frequency. 

If, on the other hand, your main emphasis is to get a codec complexity as low as 
possible, I think it is possible to get a reasonably good pitch period contour by 
using the input signal directly without partial whitening filtering, thus saving the 
complexity of the associated LPC analysis and filtering (which can be relatively 
high for the 48 kHz sampling rate).  Note that when I said "Any low-complexity pitch 
estimator can be used here..." in the document I sent last night, I used the term 
"low-complexity" because I thought you told me you tried to keep the complexity of 
CELT as low as possible.  Since this pitch prefiltering/postfiltering approach is 
meant to be used with CELT, I thought it is only natural to make the complexity of 
the pitch estimator as low as possible. That's why I used the term "low-complexity 
pitch estimator" in that document.

I think for this pitch prefiltering/postfiltering approach to work well, what's most 
important is that we need to do everything we can to ensure that the pitch period we 
extract is the true pitch period rather than an integer multiple or sub-multiple of 
the pitch period.  Unlike speech coding or packet loss concealment applications 
where double pitch period or triple pitch period still gives you decent pitch 
prediction or replacement waveform, here double or triple pitch period will create 
one or two unwanted frequency response peak(s) in the valley regions between pitch 
harmonic peaks and thus will greatly reduce the noise reduction achievable by the 
noise spectral shaping effect of the pitch prefilter/postfilter approach.  

Hence, ideally we should have some decision logic and smoothing techniques to get a 
smooth contour of the true pitch period.  I think the pitch estimator of the 
BroadVoice codecs (BV16 and BV32) that Broadcom previously submitted can serve as a 
good starting point for further development, because it was optimized to have 
relatively low complexity and it also has decision logic and smoothing built-in to 
try to get a smooth contour of the true pitch period for voiced regions of speech.  
It is optimized for 8 kHz or 16 kHz speech signals, so to use it for 48 kHz speech 
and music signals, we need to adjust the decimation factor and re-optimize it to 
work better for music as well.  We also need to increase the range of the pitch 
period, or equivalently, the pitch frequency.  Typical pitch frequency range for 
adult voice is somewhere around 50 to 400 Hz, but the pitch frequencies of music 
instruments cover a much wider range and can go up to 1600 Hz or higher.  BV16 and 
BV32 detectable pitch ranges are around 59 Hz ~ 800 Hz and 60 Hz ~ 1600 Hz, 
respectively, because we previously already increased their pitch ranges to be able 
to detect the pitch of some of the music instruments.

Of course, if there is another royalty-free and unencumbered pitch estimator 
available that has even lower complexity than the BV16/BV32 pitch estimator and 
performs better, I am all for it.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Saturday, October 23, 2010 5:52 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Thanks again,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 23 Oct 2010 14:56:53 -0700

Hi Jean-Marc,

A few more comments about the technical description document I sent last night:

(1) I should have clarified: when I said "frame size" in my document, I was
referring to the frame size for pitch estimation and pitch filter tap calculation,
not the CELT codec frame size.  For ease of implementation, the boundaries of the
pitch frames and codec frames should obviously be aligned.  Given that the CELT-only
mode of the IETF codec has possible frame sizes of 2.5, 5, 10, and 20 ms, I would
recommend that we update the pitch period and pitch filter tap(s) at a rate of once
every 5 ms if the CELT frame size is 5, 10, or 20 ms, because updating them once
every 10 or 20 ms is probably too infrequent and probably will decrease the system's
effectiveness in the reduction of coding distortion (although it doesn't hurt for
you to try updating them every 10 or 20 ms to be the same as the codec frame size
just to see what happens).  On the other hand, if the CELT frame size is 2.5 ms in
order to achieve an ultra-low delay, then it doesn't make sense for the pitch
prefilter/postfilter to use a 5 ms frame size since it will just increase the system
delay.  In this case, even though 2.5 ms pitch frame size may be more frequent than
necessary and will double the pitch side information bit-rate, it is still a
necessity just to enable CELT to achieve the ultra-low delay.

(2) On a second thought, I think to calculate the normalized correlation c or the
optimal tap weight beta for a single-tap pitch predictor, rather than calculating
the summation over the current frame (n from 1 to L), it probably makes more sense
for the summation length to be the extracted pitch period or the pitch frame size,
whichever is larger. The motivation is this: suppose the pitch frame size is 2.5 ms
or 5 ms and the pitch period is 15 ms, then trying to measure the level of
periodicity using only 1/6 to 1/3 of the pitch cycle waveform seems dangerous or at
least unreliable.  Thus, if the detected pitch period is longer than the frame size,
then we should sum over all of the samples in the current frame plus more samples in
the previous frame(s) until the total number of samples in the summation equals the
pitch period.  If the pitch period is smaller than the frame size and the pitch
frame size is only 2.5 ms or 5 ms, I recommend just summing over the entire current
frame.

(3) If we use multi-tap pitch filters in order to taper off the comb filtering
effect at higher frequencies, we need to be careful about the stability of the IIR
pitch filter, especially if the IIR filter is in the decoder side (as is the case in
my example single-tap all-pole pitch postfilter).  I suppose you probably already
know this, but I would like to mention and discuss it anyway.  Even though the
multi-tap IIR pitch postfilter can be an exact inverse filter of a corresponding
pitch prefilter and supposedly is just "undoing" the effect of the pitch prefilter,
the quantization errors in the CELT codec and even the effects caused by channel
errors can potentially make the output of the multi-tap IIR pitch postfilter "go
wild" (have very large amplitude or even oscillate) if that filter is unstable or
even if it is only close to unstable (i.e. all poles are inside the unit circle but
too close to the unit circle).  If the IIR pitch filter is a one-tap filter,
ensuring the stability is trivial.  If it is a 2-tap or 3-tap filter, Professor
Peter Kabal and his student Ravi Ramachandran has a nice paper on a stability test
and a stabilization technique that can be used to ensure the stability of such a
filter.  Their paper is published in the IEEE Transactions on ASSP, July 1987.  We
can use that stabilization technique to stabilize every single candidate 2-tap or 3-
tap filter coefficient set that we will transmit and use, with enough safety margin
to instability, to ensure that a 2-tap or 3-tap IIR pitch postfilter will not
amplify the output audio signal wildly.

Alternatively, we can use the second form of the pitch prefilter and postfilter that
I mentioned in my document so that the pitch postfilter is a 2-tap or 3-tap FIR
filter in the form of H(z) = 1 + a1 * z^(-p+1) + a2 * z^-p + a3 * z^(-p-1), with a1
or a3 equals to zero in the case of a 2-tap filter.  In this case, we will move the
multi-tap IIR filter from the decoder side to the encoder side where it is applied
directly to the input audio signal and thus we do not have to worry about CELT
quantization errors or channel errors driving a "nearly unstable" filter wild.  Of
course, we do not want the IIR prefilter there to be too close to instability,
either, otherwise the potential wild swing in its output may still cause problems
for the CELT codec.  The main advantage of this alternative arrangement is that the
pitch postfilter is now an FIR filter, which is guaranteed to be stable and no
amount of CELT coding distortion or channel error will ever "drive it crazy and
wild" like they can for a multi-tap IIR pitch postfilter that has not been
stabilized or has not maintained a sufficient safety margin to instability.  The
main disadvantage of this arrangement is that with an FIR pitch postfilter, its
frequency response has downward spikes located at pitch harmonics, and it will tend
to shape the coding noise that way, too.  A noise spectral shape with downward
spikes is probably not as ideal as the noise spectral shape with upward spikes that
the all-pole pitch postfilter provides.  This is more so when the parameter bmax is
closer to 1 but probably won't be too big a deal if bmax is in the neighborhood of
0.4 to 0.6.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Raymond (Juin-Hwey) Chen
Sent: Saturday, October 23, 2010 12:50 PM
To: Jean-Marc Valin
Cc: [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec
Importance: Low

Hi Jean-Marc,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Saturday, October 23, 2010 5:52 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Thanks again,

        Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 23 Oct 2010 20:35:14 -0400

Hi Raymond,

> formant of the input speech signal as the pitch frequency. 

My concern wasn't that much about the period, but the gain. I'll
probably have to try both approaches, but I was mainly concerned that a
non-harmonic signal with one strong sinusoid could cause the comb filter
to be turned on with a high gain.

> try to get a smooth contour of the true pitch period for voiced regions of speech.  

CELT currently has a pitch predictor used for PLC, but I doubt it's very
good for this as it is now. OTOH, it's fairly low complexity too. It's
part of my plans to compare it with the one in BV16/32. I assume that
Broadcom would be willing to relicense it to the simplified BSD license
(it's currently LGPL) if we wanted to use it instead of the current CELT
estimator.

> to detect the pitch of some of the music instruments.

The pitch range is another thing that needs experimenting with, but my
intuition here is that it's probably not necessary to go below 100 Hz
because for these very low pitches, there is enough masking between the
harmonics to make the coding noise inaudible even without a post-filter.
Again worth testing.

> performs better, I am all for it.

There's a few more starting with the SILK one, but it may be too
good/complex for what we need. The Speex one is fairly complex and not
that good, so it's not interesting for this. There's also the codec2
low-bitrate vocoder, but I don't know how well its pitch estimator is.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 23 Oct 2010 21:06:27 -0400

Hi Raymond,

> frame size is only 2.5 ms or 5 ms, I recommend just summing over the entire current

That's indeed an issue that I hadn't thought of. It makes sense that we
want at least one period.

> know this, but I would like to mention and discuss it anyway. 

Indeed, I learned this the hard way with Speex in the past. Fortunately,
the gains we need tend to be small and we probably only want positive
gains. As far as I know, having a sum of the absolute values of the taps
being smaller than 1 is a sufficient condition to ensure stability and
it's probably good enough even if stricter than necessary.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 25 Oct 2010 15:31:40 -0400

Hi,

I have checked in an initial implementation in the CELT exp_comb_filter 
branch. Right now the quality is very bad, but that's mainly an 
implementation problem. For example, the current pitch estimator I'm using 
has atrocious behaviour when it comes to pitch doubling/tripling/... Also, 
I'm still suspecting bugs in the filter's overlap handling.

Cheers,

	Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Mon, 25 Oct 2010 22:53:52 +0200 (CEST)

Hi,

It may be a good idea to look at SILK's pitch estimator.  Having duplicate pitch analysis within one codec is not ideal, and the one in SILK is quite mature/stable.  It may need some changes to work with all of CELT's frame sizes, but it already works for 10 and 20 ms frames - good enough for testing.  As an extra benefit, it efficiently codes the pitch evolution over time, which would benefit CELT too.

koen.

----- Original Message -----
From: "Jean-Marc Valin" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: Monday, October 25, 2010 12:31:40 PM
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi,

I have checked in an initial implementation in the CELT exp_comb_filter
branch. Right now the quality is very bad, but that's mainly an
implementation problem. For example, the current pitch estimator I'm
using has atrocious behaviour when it comes to pitch
doubling/tripling/... Also,
I'm still suspecting bugs in the filter's overlap handling.

Cheers,

Jean-Marc

On 10-10-23 08:35 PM, Jean-Marc Valin wrote:
> probably have to try both approaches, but I was mainly concerned that
> a non-harmonic signal with one strong sinusoid could cause the comb
> filter to be turned on with a high gain.
>
> CELT currently has a pitch predictor used for PLC, but I doubt it's
> very good for this as it is now. OTOH, it's fairly low complexity too.
> It's part of my plans to compare it with the one in BV16/32. I assume
> that Broadcom would be willing to relicense it to the simplified BSD
> license (it's currently LGPL) if we wanted to use it instead of the
> current CELT
> estimator.
>
> because for these very low pitches, there is enough masking between
> the harmonics to make the coding noise inaudible even without a
> post-filter. Again worth testing.
>
>> Of course, if there is another royalty-free and unencumbered pitch
>> estimator available that has even lower complexity than the BV16/BV32
>> pitch estimator and
> _______________________________________________ codec mailing list
> [email-masked]

_______________________________________________ codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 25 Oct 2010 17:29:50 -0400

Hi Koen,

Yes, using SILK's estimator is definitely a possibility. My only concern
was that it may be a bit too complex for what we need. Think you'd be
able to add a "low complexity" option to it? Otherwise, the options are
to improve the existing CELT pitch estimator (it's small, fast, just not
very good), or adapt either the estimator from either Broadvoice or the
codec2. I haven't experimented enough to have any preference over any of
these 4 options.

One other question about SILK's pitch predictor. How hard does it work
to prevent pitch doubling (at the expense of finding a slightly poorer
correlation). What I observed (that Raymond had already warned about) is
that pitch doubling is really a problem for the postfilter.

	Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Tue, 26 Oct 2010 00:28:51 +0200 (CEST)

Hi Jean-Marc,

Yes, SILK has 3 complexity settings for the pitch estimator, you probably ran it in the highest setting.  Also make sure to feed it a signal sampled at 16 or 24 kHz - complexity is obviously much higher with 48 kHz data for which it wasn't designed.

I'd expect SILK's quality to be roughly equally sensitive to pitch doublings as CELT with pre/postfilter.  There's a bias towards shorter lags to avoid these.  And (partial) whitening is also quite important for this.

koen.

----- Original Message -----
From: "Jean-Marc Valin" <[email-masked]>
To: "Koen Vos" <[email-masked]>
Cc: [email-masked]
Sent: Monday, October 25, 2010 2:29:50 PM
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Koen,

Jean-Marc

On 10-10-25 04:53 PM, Koen Vos wrote:
> Hi,
>
> It may be a good idea to look at SILK's pitch estimator. Having
> duplicate pitch analysis within one codec is not ideal, and the one in
> SILK is quite mature/stable. It may need some changes to work with all
> of CELT's frame sizes, but it already works for 10 and 20 ms frames -
> good enough for testing. As an extra benefit, it efficiently codes the
> pitch evolution over time, which would benefit CELT too.
>
> koen.
>
>
> I have checked in an initial implementation in the CELT
> exp_comb_filter branch. Right now the quality is very bad, but that's
> mainly an
>> very good for this as it is now. OTOH, it's fairly low complexity
>> too. It's part of my plans to compare it with the one in BV16/32. I
>> assume that Broadcom would be willing to relicense it to the
>> simplified BSD
>> The pitch range is another thing that needs experimenting with, but
>> my intuition here is that it's probably not necessary to go below 100
>> Hz because for these very low pitches, there is enough masking
>> between the harmonics to make the coding noise inaudible even without
>> a post-filter. Again worth testing.
>>
>>> estimator available that has even lower complexity than the
>>> BV16/BV32 pitch estimator and
>> good/complex for what we need. The Speex one is fairly complex and
>> not that good, so it's not interesting for this. There's also the
>> codec2 low-bitrate vocoder, but I don't know how well its pitch
>> estimator is.
>>
>> Cheers,
>>
>> 	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 25 Oct 2010 22:13:13 -0400

Hi Raymond,

I've just realised that changes in the pitch and/or gain are causing
significant artefacts. While the pitch estimator should definitely
mimimise these changes, I still think they'll be unavoidable. At this
point, I'm not sure whether it's an implementation bug or an issue with
the algorithm.

I've copied below the function I use the the pre-filter and post-filter.
In the case of the pre-filter, the gains are negative, while for the
post-filter, x and y point to the same array. Can you see something
obviously wrong with this code? Otherwise, any idea what the problem may
be? Note that I *do* get perfect reconstruction when the pitch and gain
are constant, or when there's no codec in the middle.

void comb_filter(float *y, float *x, int T0, int T1, int N,
      float g0, float g1, int overlap)
{
   int i;
   for (i=0;i<overlap;i++)
   {
      float f;
      f = i/(float)overlap;
      y[i] = x[i] + (1.-f)*g0*x[i-T0] + f*g1*x[i-T1];
   }
   for (i=overlap;i<N;i++)
      y[i] = x[i] + g1*x[i-T1];
}

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 25 Oct 2010 23:29:27 -0400

Sorry for replying to myself, but before anyone wastes time on this, I'm
fairly sure I found what my error was. It wasn't related to the filter
itself, but the fact that I forgot to compensate for the 2.5 ms delay
introduced by the codec.

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 25 Oct 2010 23:04:31 -0700

Hi Jean-Marc,

Sorry for the late reply.  As I mentioned to you before, I have been and would be in an off-site conference for a few days and I generally don't have Internet access there during the day, so I could not respond earlier.  Sorry.

Glad to hear that you found the error.  To verify the performance of your prefilter/postfilter implementation, I suggest that you try to use the same trumpet signal that I used before and test the prefilter followed by postfilter with and without CELT coding, and then listen to them and compare them with the output signals I sent you previously to see if you get similar audio quality for the same condition.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Monday, October 25, 2010 8:29 PM
To: Jean-Marc Valin
Cc: Raymond (Juin-Hwey) Chen; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

        Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 25 Oct 2010 23:40:39 -0700

Hi Jean-Marc,

See my in-line comments below.

> to be turned on with a high gain.

[Raymond]: I see your concern with introducing artificial harmonics for a single sinusoid.  On the other hand, my concern is that if you derive the gain (or taps) based on partially whitened input signal, the gain or taps may be too small for legitimate periodic music or speech signals.  To address both your and my concerns, I think it is probably best to use a 3-tap filter where the taps are derived directly from the input signal without whitening.  Since a 3-tap filter has the ability to taper off the degree of comb filtering with increasing frequency, a set of 3 filter taps that is chosen to match the input spectrum can probably have a reasonable spectral match for either a single tone or music/speech with a harmonic structure.  The BroadVoice codecs use a 3-tap pitch predictor.  The way those 3 taps are jointly searched and vector quantized might provide a way to quantize the 3 taps and do a spectral match at the same time for our current pitch prefilter/postfilter approach.  We need to experiment to find out.  The BroadVoice codebooks for the 3 taps were designed using mainly speech.  If we adopt this approach, we need to redesign the codebook with sufficient music signals in the training set.  If you are particularly concerned with a single sinusoid, we may want to train one or a few 3-tap codevectors using just sinusoids.  If this doesn't work well, in the worst case, we could potentially design a detector that detect (nearly) periodic time-domain waveform that does not have a spectrum with a harmonic structure (like a single sinusoid) and turn off the prefilter/postfilter accordingly.

> (it's currently LGPL) if we wanted to use it instead of the current CELT

[Raymond]: Broadcom's legal counsel will formally answer that question soon.  My assumption is that if we were previously willing to allow royalty-free use of BroadVoice in the IETF codec, then allowing certain components of it to be used in the IETF codec under IETF's licensing terms is probably not a problem, but this is just my personal thought and not an official Broadcom position yet.  I will talk to our lawyer and let him give the formal answer.

> harmonics to make the coding noise inaudible even without a post-filter.

[Raymond]: I agree that lower pitch frequency generally has better noise masking, so time-domain predictive codecs often sound noisier for female speech than for male speech.  However, my previous work with pitch postfiltering during my Ph.D. study showed that pitch postfiltering alone could still give audible quality improvement to low-bit-rate-coded male speech.  Does it improve quality when the pitch frequency is below 100 Hz? I don't remember the pitch frequencies of my pitch postfilter tests.  I agree it is worth testing.

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 26 Oct 2010 14:22:50 -0400

Hi Raymond,

I've been able to reproduce results that are mostly similar to what you got 
earlier on the trumpet file. Unfortunately, it doesn't work as well on 
other files I'm trying. For example on this one 
(http://jmvalin.ca/misc_stuff/s02a_102.sw) there is simply too much pitch 
doubling and I'm not actually sure it's possible to do a good job. Could 
you try with your setup and see if you can improve things. I think one of 
the tricky thing with the postfilter will be to make it work in all 
circumstances and -- most importantly -- prevent it from causing harm in 
cases where it cannot help.

Regarding the periods we'd need to support, I think the important question 
is not that much "on what range does it help", but "on what range is it 
necessary". From what I observed, CELT copes well with regular male and 
even female speech and even the vega sample tends to sound OK. It's only on 
strongly tonal samples with higher pitches that problems occur. The main 
reason I want to limit the maximum period allowed is that there's a memory 
footprint issue that's directly related. In any case, we'll need to 
experiment as with anything else.

Cheers,

	Jean-Marc

On 10-10-26 02:40 AM, Raymond (Juin-Hwey) Chen wrote:
> [Raymond]: I see your concern with introducing artificial harmonics for
> a single sinusoid.  On the other hand, my concern is that if you derive
> the gain (or taps) based on partially whitened input signal, the gain or
> taps may be too small for legitimate periodic music or speech signals.
> To address both your and my concerns, I think it is probably best to use
> a 3-tap filter where the taps are derived directly from the input signal
> without whitening.  Since a 3-tap filter has the ability to taper off
> the degree of comb filtering with increasing frequency, a set of 3
> filter taps that is chosen to match the input spectrum can probably have
> a reasonable spectral match for either a single tone or music/speech
> with a harmonic structure.  The BroadVoice codecs use a 3-tap pitch
> predictor.  The way those 3 taps are jointly searched and vector
> quantized might provide a way to quantize the 3 taps and do a spectral
> match at the same time for our current pitch prefilter/postfilter
> approa ch.  We need to experiment to find out.  The BroadVoice codebooks
> for the 3 taps were designed using mainly speech.  If we adopt this
> approach, we need to redesign the codebook with sufficient music signals
> in the training set.  If you are particularly concerned with a single
> sinusoid, we may want to train one or a few 3-tap codevectors using just
> sinusoids.  If this doesn't work well, in the worst case, we could
> potentially design a detector that detect (nearly) periodic time-domain
> waveform that does not have a spectrum with a harmonic structure (like a
> single sinusoid) and turn off the prefilter/postfilter accordingly.
>
>> current CELT estimator.
> [Raymond]: Broadcom's legal counsel will formally answer that question
> soon.  My assumption is that if we were previously willing to allow
> royalty-free use of BroadVoice in the IETF codec, then allowing certain
> components of it to be used in the IETF codec under IETF's licensing
> terms is probably not a problem, but this is just my personal thought
> and not an official Broadcom position yet.  I will talk to our lawyer
> and let him give the formal answer.
>
>> Hz because for these very low pitches, there is enough masking between
> [Raymond]: I agree that lower pitch frequency generally has better noise
> masking, so time-domain predictive codecs often sound noisier for female
> speech than for male speech.  However, my previous work with pitch
> postfiltering during my Ph.D. study showed that pitch postfiltering
> alone could still give audible quality improvement to low-bit-rate-coded
> male speech.  Does it improve quality when the pitch frequency is below
> 100 Hz? I don't remember the pitch frequencies of my pitch postfilter
> tests.  I agree it is worth testing.
>
>
> [email-masked] https://www.ietf.org/mailman/listinfo/codec

---

From: Gregory Maxwell <[email-masked]>
Date: Tue, 26 Oct 2010 12:57:37 -0700

> even female speech and even the vega sample tends to sound OK. It's only on

You should be mindful of the low delay modes. At moderate rates in the 5ms mode, for example.

I would expect the need for this to be stronger in these cases, so the bitstream should support them.

I think we're currently at an order of magnitude less memory than anyone has suggested was a requirement on the list— and decoders will typically want a fair amount of memory for PLC purposes— so I think we have room to breath on the design right now and can worry about putting it on a diet after the technique's usefulness has been demonstrated in an implementation.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 26 Oct 2010 23:17:59 -0700

Hi Jean-Marc,

Are you saying the problem is in the pitch estimation for that test file rather than the pitch filtering?  

I will take a look at that file.  Is that a raw file?  I am not familiar with the ".sw" file extension.

I could hear audible improvement when I tried the method on the other solo instrument files that I am working on, although the degree of improvement depends on which file it is, as expected.

I am starting to wonder how many people in the codec WG email list still want to continue to see our low-level technical discussions about the algorithm development in the reflector.  I wonder if some people in the codec WG email list would prefer not to be bombarded with such emails.  Should we consider moving such detailed codec algorithm development discussions off the WG mailing list and just between those of us who are interested or are directly involved?

Or is this how IETF supposed to work (discussing all technical development in emails)?

Thanks.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Tuesday, October 26, 2010 11:23 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

	Jean-Marc

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 27 Oct 2010 06:14:26 -0400

I think it'd be good to have development discussions like this on the list.

Stephen Botzko

On Wed, Oct 27, 2010 at 2:17 AM, Raymond (Juin-Hwey) Chen <
[email-masked]> wrote:

> I could hear audible improvement when I tried the method on the other solo
> instrument files that I am working on, although the degree of improvement
> depends on which file it is, as expected.
>
> I am starting to wonder how many people in the codec WG email list still
> want to continue to see our low-level technical discussions about the
> algorithm development in the reflector.  I wonder if some people in the
> codec WG email list would prefer not to be bombarded with such emails.
>  Should we consider moving such detailed codec algorithm development
> discussions off the WG mailing list and just between those of us who are
> interested or are directly involved?
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 27 Oct 2010 09:13:07 -0400

Hi Raymond,

> rather than the pitch filtering?

The fundamental problem is definitely with the estimation. Reliable and 
consistent pitch estimation is hard. The question is thus whether it's 
possible to either have a reliable enough estimator *or* have a robust 
enough filtering scheme that it can tolerate occasional errors in the 
estimation. I'm still working on both.

> with the ".sw" file extension.

That's just a 48 kHz raw 16-bit PCM file (little endian). sw is just the 
sox program's extension for "signed word".

> I could hear audible improvement when I tried the method on the other
> solo instrument files that I am working on, although the degree of
> improvement depends on which file it is, as expected.

Yes, I'm also observing that. What I mainly want to improve is how it works 
on more complicated signals (like the one I linked to), or non-harmonic 
signals. As long as we're in a situation where it improves on a subset of 
the samples while not degrading the others, I'm happy.

> Or is this how IETF supposed to work (discussing all technical

I think we should keep having this discussion on the list. The IETF lists 
are normally meant for technical discussions so this just helps improve the 
SNR :-)

Cheers,

	Jean-Marc

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 27 Oct 2010 17:13:52 -0400

Absolutely, these discussions should happen here. This working group is 
about producing this codec, and technical discussions about that 
deliverable belong here.

-Jonathan R.
(with WG chair hat on)

On 10/27/2010 6:14 AM, Stephen Botzko wrote:
> On Wed, Oct 27, 2010 at 2:17 AM, Raymond (Juin-Hwey) Chen
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi Jean-Marc,
>
>     I could hear audible improvement when I tried the method on the
>     other solo instrument files that I am working on, although the
>     degree of improvement depends on which file it is, as expected.
>
>     I am starting to wonder how many people in the codec WG email list
>     still want to continue to see our low-level technical discussions
>     about the algorithm development in the reflector.  I wonder if some
>     people in the codec WG email list would prefer not to be bombarded
>     with such emails.  Should we consider moving such detailed codec
>     algorithm development discussions off the WG mailing list and just
>     between those of us who are interested or are directly involved?
>
>     From: Jean-Marc Valin [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     Cc: Jean-Marc Valin; [email-masked] <mailto:[email-masked]>'
>     between

-- 
Jonathan D. Rosenberg, Ph.D.                   SkypeID: jdrosen
Chief Technology Strategist                    Mobile: +1 (732) 766-2496
Skype                                          SkypeIn: +1 (408) 465-0361
[email-masked]                              http://www.skype.com
[email-masked]                            http://www.jdrosen.net

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 27 Oct 2010 18:03:17 -0400

For those curious about what this is all about. This is the frequency 
response of the post-filter for a certain pitch value:

http://jmvalin.ca/misc_stuff/postfilter.png

Note that the roll-off is due to the fact that I've implemented a (fixed) 
3-tap filter to prevent HF artefacts. I'll post some samples a bit later.

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 27 Oct 2010 22:42:51 -0700

Hi Jonathan,

Thanks for the instruction.  Will do.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jonathan Rosenberg
Sent: Wednesday, October 27, 2010 2:14 PM
To: [email-masked]
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

-Jonathan R.
(with WG chair hat on)

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 27 Oct 2010 23:10:16 -0700

Hi Jean-Marc,

Thanks for your comments.  If the problem is with the pitch estimation, the good thing is that it does not affect the bit-stream format if we just use the integer pitch period within a certain range.  Even after the bit-stream format is frozen, the pitch estimation accuracy can still be improved later as it only affects the encoder side. Therefore, even if we can't get a very reliable pitch estimator for music before the IETF 79 meeting, it doesn't necessarily need to hold up the bit-stream definition.

Hence, I think the pitch estimation accuracy is not the most urgent issue.  I think the most urgent issue is to design the pitch filter tap(s) codebook entries so they all correspond to stable pitch filters and there are enough variety of filter tap sets in the codebook so they allow us to shape the envelope of the frequency response of the pitch filters in a wide range from fully periodic (no envelope shaping) to something that effectively turns off the pitch filter.  To finalize the bit-stream, we do need to finalize such a filter tap codebook.

I remember I have seen some papers on pitch estimation techniques for music signals published at IEEE ICASSP conferences, and from the pitch contour figures in the papers, they seem to work well.  I think we can first try to improve the CELT PLC pitch estimator, BroadVoice pitch estimator, or SILK pitch estimator to see if we can get satisfactory pitch estimation performance for both speech and music with low enough complexity.  If not, we can check those music pitch estimation papers out to see what they have there.  We can do such pitch estimator improvement after IETF 79 if we can't finish it before then.

Raymond

P.S. It just so happens that this week I am completely booked solid during the weekdays -- a 3.5-day off-site conference followed by a 1.5-day meeting, so I don't have much time to work on this method this week.  This is very rare for me.  Sorry about that.  Next week I expect to have more time to work on this.

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Wednesday, October 27, 2010 6:13 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 27 Oct 2010 23:26:12 -0700

Jean-Marc,

I forgot to comment: 
As long as the filter tap codebook adequately covers the full possible range from purely periodic (no envelope shaping) to effectively turning off the pitch filter, then even if we can't quite get a perfect pitch estimator or pitch filter tap determination method that we are completely happy with before IETF 79 meeting, we can improve them later even after the possible pitch periods and possible pitch filter tap sets are determined.  The improved pitch estimator and pitch filter tap determination method developed later will improve the audio quality without changing the bit-stream format.  They may increase the complexity of the encoder side, but the decoder side remains low complexity and unchanged. And who knows, maybe with sufficient research, it is possible that the pitch period and pitch filter tap determination can both be improved while reducing the encoder side complexity at the same time.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Raymond (Juin-Hwey) Chen
Sent: Wednesday, October 27, 2010 11:10 PM
To: Jean-Marc Valin
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec
Importance: Low

Hi Jean-Marc,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Wednesday, October 27, 2010 6:13 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 28 Oct 2010 07:09:06 -0400

Hi Raymond and others,

See comments below.

On 10-10-28 02:10 AM, Raymond (Juin-Hwey) Chen wrote:
> Thanks for your comments.  If the problem is with the pitch
> estimation, the good thing is that it does not affect the bit-stream
> format if we just use the integer pitch period within a certain
> range.  Even after the bit-stream format is frozen, the pitch
> estimation accuracy can still be improved later as it only affects
> the encoder side. Therefore, even if we can't get a very reliable
> pitch estimator for music before the IETF 79 meeting, it doesn't
> necessarily need to hold up the bit-stream definition.

Actually, the good news is that I think I now have a pitch estimator
that is good enough to show that the prefilter/postfilter is useful.
It's not perfect, but the error rate no longer causes too many
artefacts. Most importantly, we have run this on many non-harmonic
signals and so far we have not been able to find samples that get
noticeably degraded. That was my main fear with this idea and I think
it's gone now.

> Hence, I think the pitch estimation accuracy is not the most urgent
> issue.  I think the most urgent issue is to design the pitch filter
> tap(s) codebook entries so they all correspond to stable pitch
> filters and there are enough variety of filter tap sets in the
> codebook so they allow us to shape the envelope of the frequency
> response of the pitch filters in a wide range from fully periodic (no
> envelope shaping) to something that effectively turns off the pitch
> filter.  To finalize the bit-stream, we do need to finalize such a
> filter tap codebook.

I'm currently using a fixed 3-tap structure with coefficients [1,
sqrt(3), 1]/(2+sqrt(3) and the frequency response for a gain of 0.5 is
the link i sent earlier: http://jmvalin.ca/misc_stuff/postfilter.png

One issue we found is that using a single-tap filter would create too HF
artefacts -- especially in signals that have a periodic component (i.e
trigger the pitch predictor), but would not otherwise need the filtering.

So unless we can clearly find a case that doesn't work with this single
3-tap predictor, I'm tempted to only have one set of taps. In general, I
think that pitch is less important as frequency goes up for the
following reasons:
1) In-harmonicity as Koen pointed out
2) Critical bands are wide enough that there is more masking (so less
need for a postfilter)
3) Because of the high SNR in higher bands, there is more danger of
changing the total energy of the band, which would be perceptually bad

That being said, I'll revise my opinion if we can find classes of
signals that don't work well with this fixed predictor and give good
results with another predictor that we can't use for "normal" samples.

> I remember I have seen some papers on pitch estimation techniques for
> music signals published at IEEE ICASSP conferences, and from the
> pitch contour figures in the papers, they seem to work well.  I think
> we can first try to improve the CELT PLC pitch estimator, BroadVoice
> pitch estimator, or SILK pitch estimator to see if we can get
> satisfactory pitch estimation performance for both speech and music
> with low enough complexity.  If not, we can check those music pitch
> estimation papers out to see what they have there.  We can do such
> pitch estimator improvement after IETF 79 if we can't finish it
> before then.

As I sad, I'm no longer too worried about the pitch predictor. I'm
currently using the CELT PLC predictor with a "de-doubling" step at the
end, so the good news is that the first stages could probably use any
predictor. Of course, it would be nice to merge code and end up with a
single predictor and we can incorporate ideas from multiple places. And
as you say, since this does not affect the bit-stream, it is less urgent
indeed.

As a note, this is what I'm currently getting with CELT and SILK for the
pitch of the tricky sample i linked to earlier:
http://jmvalin.ca/misc_stuff/pitch_silk_celt.png
Note that this isn't meant to bash the SILK pitch estimator, since
although it makes more error (even after I set the biases Koen
mentioned) it's probably much better than what I have on speech -- which
is what matters most for SILK.

Cheers,

	Jean-Marc

> Raymond
> 
> P.S. It just so happens that this week I am completely booked solid
> during the weekdays -- a 3.5-day off-site conference followed by a
> 1.5-day meeting, so I don't have much time to work on this method
> this week.  This is very rare for me.  Sorry about that.  Next week I
> expect to have more time to work on this.
> 
> -----Original Message----- From: Jean-Marc Valin
> [mailto:[email-masked]] Sent: Wednesday, October 27,
> 2010 6:13 AM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc Valin;
> [email-masked]' Subject: Re: [codec] A pitch filtering method to
> improve the mode 1 CELT codec
> 
> Hi Raymond,
> 
> The fundamental problem is definitely with the estimation. Reliable
> and consistent pitch estimation is hard. The question is thus whether
> it's possible to either have a reliable enough estimator *or* have a
> robust enough filtering scheme that it can tolerate occasional errors
> in the estimation. I'm still working on both.
> 
> That's just a 48 kHz raw 16-bit PCM file (little endian). sw is just
> the sox program's extension for "signed word".
> 
> Yes, I'm also observing that. What I mainly want to improve is how it
> works on more complicated signals (like the one I linked to), or
> non-harmonic signals. As long as we're in a situation where it
> improves on a subset of the samples while not degrading the others,
> I'm happy.
> 
> I think we should keep having this discussion on the list. The IETF
> lists are normally meant for technical discussions so this just helps
> improve the SNR :-)
> 
> Cheers,
> 
> Jean-Marc
> 
> 
> 
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 28 Oct 2010 23:46:10 -0700

Hi Jean-Marc,

Glad to hear that you now have a pitch estimator that's satisfactory to you. That a very good news.

Regarding your comments about single-tap filter, yes, I mentioned in my original email that for some signals it can produce more periodicity in the high frequencies than in the original signal.  However, if you send a mode/flag bit to turn the filter on or off (or make one of the candidate tap weights equal to zero) and if there is a evaluation process in the encoder side to determine whether to turn on the pitch filter, then if we decide that a single-tap pitch filter will degrade quality, we can always turn it off. (The evaluation can even be a closed-loop determination by trying both, although this will increase the encoding complexity significantly.)  I am not advocating using a single-tap pitch filter.  I am just saying that if for some reason a single-tap pitch filter is used for its simplicity, we don't necessarily have to suffer quality degradation because we can always turn it off dynamically for those frames where it can degrade quality.

Regarding your comments about a single fixed set of 3 filter taps, are you planning to send different scaling factor to scale the three taps so as to control the degree of comb filtering?  I agree that generally it makes more sense to use a 3-tap pitch filter than a single-tap pitch filter as you discussed.  Having a single fixed set of 3 filter taps (and thus a single kind of envelope) may be fine, but don't you think we should have different scaling factors to scale these three taps so we can control the degree of comb filtering to fit different kinds of audio signals?

Also, although you can't find any audio signal that this fixed set doesn't work, what if we later find some signals that this fixed set of taps doesn't work well?  In a more conservative design, wouldn't you want at least a few different envelope shapes for the frequency response of the pitch filters?

I am not necessarily proposing some new development direction.  I am just throwing some questions for us to think about.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Thursday, October 28, 2010 4:09 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond and others,

See comments below.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 29 Oct 2010 06:38:15 -0400

Hi Raymond,

On 10-10-29 02:46 AM, Raymond (Juin-Hwey) Chen wrote:
> Regarding your comments about single-tap filter, yes, I mentioned in
> my original email that for some signals it can produce more
> periodicity in the high frequencies than in the original signal.
> However, if you send a mode/flag bit to turn the filter on or off (or
> make one of the candidate tap weights equal to zero) and if there is
> a evaluation process in the encoder side to determine whether to turn
> on the pitch filter, then if we decide that a single-tap pitch filter
> will degrade quality, we can always turn it off. (The evaluation can
> even be a closed-loop determination by trying both, although this
> will increase the encoding complexity significantly.)  I am not
> advocating using a single-tap pitch filter.  I am just saying that if
> for some reason a single-tap pitch filter is used for its simplicity,
> we don't necessarily have to suffer quality degradation because we
> can always turn it off dynamically for those frames where it can
> degrade quality.

At this point, I really don't see what kind of audio signal would
benefit from having a one-tap filter and I don't see how we could
automatically detect this either in the encoder. If anything I think a
5-tap filter would be more useful, though I'm not sure we really want to
go there either.

> Regarding your comments about a single fixed set of 3 filter taps,
> are you planning to send different scaling factor to scale the three
> taps so as to control the degree of comb filtering?  I agree that
> generally it makes more sense to use a 3-tap pitch filter than a
> single-tap pitch filter as you discussed.  Having a single fixed set
> of 3 filter taps (and thus a single kind of envelope) may be fine,
> but don't you think we should have different scaling factors to scale
> these three taps so we can control the degree of comb filtering to
> fit different kinds of audio signals?

Oh, of course. The current code I have has four levels for the "global
gain" of that filter (in increments of 0.125), with 0.5 being the
largest value value encodable. This is mostly probably not optimal and
we would probably want to allow slightly larger gains even if we don't
make use of them yet. Feel free to make suggestions here.

> Also, although you can't find any audio signal that this fixed set
> doesn't work, what if we later find some signals that this fixed set
> of taps doesn't work well?  In a more conservative design, wouldn't
> you want at least a few different envelope shapes for the frequency
> response of the pitch filters?

I think we're more likely to come across audio that requires a different
3-tap filter than audio that requires a single tap. That being said, I
still don't really see what that would look like and there's many
possibilities. One issue here is that for small frame sizes, coding
these parameters can end up being expensive, so before adding anything
we need a reasonable expectation that it will be useful.

> I am not necessarily proposing some new development direction.  I am
> just throwing some questions for us to think about.

Sure. I think in the end we're going to just try a variety of audio
samples and see if anything needs more than what we're currently doing.
I think we will, but I currently have no idea yet in what direction
it'll send us.

	Jean-Marc