Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 01 Nov 2010 13:27:53 -0400

Hi,

I thought I'd give another post-filter status update. The code seems to 
work well now and so far we haven't been able to find cases where it 
breaks. I've also just finished converting the code I wrote to fixed-point.

The main issue that remains now is how to handle the pitch contour for 
frame sizes above 5 ms. Right now, the pitch period is forced constant 
through the whole frame, which is likely suboptimal. Any suggestion on how 
to best estimate and code the pitch contour? The other thing that's still 
undecided is the range of values we want to encode. In the current version, 
the period can range from 30 samples (1500 Hz pitch) to 960 samples (50 Hz 
pitch). As for the gain, non-zero values can range between 0.25 and 0.625 
in increments of 0.125. Any opinion on these?

Cheers,

	Jean-Marc

On 10-10-29 02:46 AM, Raymond (Juin-Hwey) Chen wrote:
> Hi Jean-Marc,
>
> Glad to hear that you now have a pitch estimator that's satisfactory to you. That a very good news.
>
> Regarding your comments about single-tap filter, yes, I mentioned in my original email that for some signals it can produce more periodicity in the high frequencies than in the original signal.  However, if you send a mode/flag bit to turn the filter on or off (or make one of the candidate tap weights equal to zero) and if there is a evaluation process in the encoder side to determine whether to turn on the pitch filter, then if we decide that a single-tap pitch filter will degrade quality, we can always turn it off. (The evaluation can even be a closed-loop determination by trying both, although this will increase the encoding complexity significantly.)  I am not advocating using a single-tap pitch filter.  I am just saying that if for some reason a single-tap pitch filter is used for its simplicity, we don't necessarily have to suffer quality degradation because we can always turn it off dynamically for those frames where it can degrade quality.
>
> Regarding your comments about a single fixed set of 3 filter taps, are you planning to send different scaling factor to scale the three taps so as to control the degree of comb filtering?  I agree that generally it makes more sense to use a 3-tap pitch filter than a single-tap pitch filter as you discussed.  Having a single fixed set of 3 filter taps (and thus a single kind of envelope) may be fine, but don't you think we should have different scaling factors to scale these three taps so we can control the degree of comb filtering to fit different kinds of audio signals?
>
> Also, although you can't find any audio signal that this fixed set doesn't work, what if we later find some signals that this fixed set of taps doesn't work well?  In a more conservative design, wouldn't you want at least a few different envelope shapes for the frequency response of the pitch filters?
>
> I am not necessarily proposing some new development direction.  I am just throwing some questions for us to think about.
>
> Raymond
>
> -----Original Message-----
> From: Jean-Marc Valin [mailto:[email-masked]]
> Sent: Thursday, October 28, 2010 4:09 AM
> To: Raymond (Juin-Hwey) Chen
> Cc: Jean-Marc Valin; [email-masked]'
> Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec
>
> Hi Raymond and others,
>
> See comments below.
>
> On 10-10-28 02:10 AM, Raymond (Juin-Hwey) Chen wrote:
>> Thanks for your comments.  If the problem is with the pitch
>> estimation, the good thing is that it does not affect the bit-stream
>> format if we just use the integer pitch period within a certain
>> range.  Even after the bit-stream format is frozen, the pitch
>> estimation accuracy can still be improved later as it only affects
>> the encoder side. Therefore, even if we can't get a very reliable
>> pitch estimator for music before the IETF 79 meeting, it doesn't
>> necessarily need to hold up the bit-stream definition.
>
> Actually, the good news is that I think I now have a pitch estimator
> that is good enough to show that the prefilter/postfilter is useful.
> It's not perfect, but the error rate no longer causes too many
> artefacts. Most importantly, we have run this on many non-harmonic
> signals and so far we have not been able to find samples that get
> noticeably degraded. That was my main fear with this idea and I think
> it's gone now.
>
>> Hence, I think the pitch estimation accuracy is not the most urgent
>> issue.  I think the most urgent issue is to design the pitch filter
>> tap(s) codebook entries so they all correspond to stable pitch
>> filters and there are enough variety of filter tap sets in the
>> codebook so they allow us to shape the envelope of the frequency
>> response of the pitch filters in a wide range from fully periodic (no
>> envelope shaping) to something that effectively turns off the pitch
>> filter.  To finalize the bit-stream, we do need to finalize such a
>> filter tap codebook.
>
> I'm currently using a fixed 3-tap structure with coefficients [1,
> sqrt(3), 1]/(2+sqrt(3) and the frequency response for a gain of 0.5 is
> the link i sent earlier: http://jmvalin.ca/misc_stuff/postfilter.png
>
> One issue we found is that using a single-tap filter would create too HF
> artefacts -- especially in signals that have a periodic component (i.e
> trigger the pitch predictor), but would not otherwise need the filtering.
>
> So unless we can clearly find a case that doesn't work with this single
> 3-tap predictor, I'm tempted to only have one set of taps. In general, I
> think that pitch is less important as frequency goes up for the
> following reasons:
> 1) In-harmonicity as Koen pointed out
> 2) Critical bands are wide enough that there is more masking (so less
> need for a postfilter)
> 3) Because of the high SNR in higher bands, there is more danger of
> changing the total energy of the band, which would be perceptually bad
>
> That being said, I'll revise my opinion if we can find classes of
> signals that don't work well with this fixed predictor and give good
> results with another predictor that we can't use for "normal" samples.
>
>> I remember I have seen some papers on pitch estimation techniques for
>> music signals published at IEEE ICASSP conferences, and from the
>> pitch contour figures in the papers, they seem to work well.  I think
>> we can first try to improve the CELT PLC pitch estimator, BroadVoice
>> pitch estimator, or SILK pitch estimator to see if we can get
>> satisfactory pitch estimation performance for both speech and music
>> with low enough complexity.  If not, we can check those music pitch
>> estimation papers out to see what they have there.  We can do such
>> pitch estimator improvement after IETF 79 if we can't finish it
>> before then.
>
> As I sad, I'm no longer too worried about the pitch predictor. I'm
> currently using the CELT PLC predictor with a "de-doubling" step at the
> end, so the good news is that the first stages could probably use any
> predictor. Of course, it would be nice to merge code and end up with a
> single predictor and we can incorporate ideas from multiple places. And
> as you say, since this does not affect the bit-stream, it is less urgent
> indeed.
>
> As a note, this is what I'm currently getting with CELT and SILK for the
> pitch of the tricky sample i linked to earlier:
> http://jmvalin.ca/misc_stuff/pitch_silk_celt.png
> Note that this isn't meant to bash the SILK pitch estimator, since
> although it makes more error (even after I set the biases Koen
> mentioned) it's probably much better than what I have on speech -- which
> is what matters most for SILK.
>
> Cheers,
>
> 	Jean-Marc
>
>> Raymond
>>
>> P.S. It just so happens that this week I am completely booked solid
>> during the weekdays -- a 3.5-day off-site conference followed by a
>> 1.5-day meeting, so I don't have much time to work on this method
>> this week.  This is very rare for me.  Sorry about that.  Next week I
>> expect to have more time to work on this.
>>
>> -----Original Message----- From: Jean-Marc Valin
>> [mailto:[email-masked]] Sent: Wednesday, October 27,
>> 2010 6:13 AM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc Valin;
>> [email-masked]' Subject: Re: [codec] A pitch filtering method to
>> improve the mode 1 CELT codec
>>
>> Hi Raymond,
>>
>> On 10-10-27 02:17 AM, Raymond (Juin-Hwey) Chen wrote:
>>> Are you saying the problem is in the pitch estimation for that test
>>> file rather than the pitch filtering?
>>
>> The fundamental problem is definitely with the estimation. Reliable
>> and consistent pitch estimation is hard. The question is thus whether
>> it's possible to either have a reliable enough estimator *or* have a
>> robust enough filtering scheme that it can tolerate occasional errors
>> in the estimation. I'm still working on both.
>>
>>> I will take a look at that file.  Is that a raw file?  I am not
>>> familiar with the ".sw" file extension.
>>
>> That's just a 48 kHz raw 16-bit PCM file (little endian). sw is just
>> the sox program's extension for "signed word".
>>
>>> I could hear audible improvement when I tried the method on the
>>> other solo instrument files that I am working on, although the
>>> degree of improvement depends on which file it is, as expected.
>>
>> Yes, I'm also observing that. What I mainly want to improve is how it
>> works on more complicated signals (like the one I linked to), or
>> non-harmonic signals. As long as we're in a situation where it
>> improves on a subset of the samples while not degrading the others,
>> I'm happy.
>>
>>> I am starting to wonder how many people in the codec WG email list
>>> still want to continue to see our low-level technical discussions
>>> about the algorithm development in the reflector.  I wonder if some
>>> people in the codec WG email list would prefer not to be bombarded
>>> with such emails. Should we consider moving such detailed codec
>>> algorithm development discussions off the WG mailing list and just
>>> between those of us who are interested or are directly involved?
>>>
>>> Or is this how IETF supposed to work (discussing all technical
>>> development in emails)?
>>
>> I think we should keep having this discussion on the list. The IETF
>> lists are normally meant for technical discussions so this just helps
>> improve the SNR :-)
>>
>> Cheers,
>>
>> Jean-Marc
>>
>>
>>
>>
>>
>
>

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 1 Nov 2010 12:47:36 -0700

Jean-Marc Valin [[email-masked]] wrote:
[snip]
> in increments of 0.125. Any opinion on these?

I've been working with Jean-Marc on IRC on this but I thought it might be useful to bring some more of the discussion to the list. :)

I made some measurements of the code on one of the small test sets I used. The test set consists of 10 second segments randomly selected from about 30 hours of audio. It's a bit commercial-recording music heavy and speech light.

http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/pitch-960-2.png
is a graph of the density of the pitch offsets I found with 5ms frames plotted against a 0.33/n curve.

http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/gain1.png
is a graph of the raw gain values our search finds.

In the current encoder these gains are scaled by 0.7 and clamped to the range [0,0.6] before being used as the goal is not to completely whiten the spectrum…

>From listening I think the scaling of 0.7 isn't enough in general and that 0.5 is probably better but there are some samples that its quite helpful for... some more encoder intelligence is probably justified. 

On a more silly note the addition of pitch lets CELT achieve intelligible (but not good) wideband speech at the ludicrous rate of 2.5kbit/sec:  http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/16k_60ms_20-2.wav
(and 8kbit/sec wideband which sounds better than speex, though it still has nothing on SILK for speech low rates. :) )

While I don't expect anyone to intentionally run the codec at such insanely low rates I think is helpful if the codec can maintain intelligibility down to very low rates in order to cope with extreme congestion.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 1 Nov 2010 18:54:18 -0700

Hi Jean-Marc,

Thanks for the update.  That's great news.

Regarding the pitch contour for frame sizes above 5 ms, as I mentioned
before, I recommend that the pitch period still gets updated every 5 ms
even when the frame size is 10 or 20 ms. The simplest way is to run the
pitch estimator at every 5 ms subframe in the same way as the pitch
estimator for a frame size of 5 ms, and code the resulting pitch period
independent of the pitch periods in other 5 ms subframes of the current
frame.  This approach is obviously not as efficient as it can be, but it
is simple, requires essentially no additional code, and has probably the
lowest risk of unintentional IP infringement.

We can potentially also use the future subframes in the current frame as
the "look-ahead" when performing pitch estimation in the earlier subframes
in order to increase the reliability of the estimated pitch period, but
this will require new code and thus additional code space.

We can also independently code the pitch period of the last 5 ms subframe,
and then for the earlier subframes, we interpolate between this pitch
period and the pitch period of the last subframe of the last frame, and
then code only the difference between the estimated pitch period and the
interpolated pitch period.  This approach should significantly reduce the
encoding bit-rate for the pitch period.  However, this approach may have
a higher risk of patent infringement because something like this might
have already been patented (unless the idea is so old that any related
patent has expired).

As to the range of the pitch period, I think lowering the minimum pitch
period from the current 30 samples to something lower, such as 20 samples,
or even to 11 samples (to cover the pitch of the highest piano note at
4186.01 Hz), will cost almost nothing when compared with the current pitch
period range, but it will increase the covered pitch frequency range
substantially. Of course, people may question (1) whether we will get
accurate enough pitch frequency when the pitch period is constrained to be
integer number of samples at such a low pitch period, and (2) whether the
pitch prefilter/postfilter approach still get the desired and expected
perceptual benefit at such a high pitch frequency.  Well, we don't know
the answers until we try it, but at least from the "bang for the buck"
perspective (i.e. additional Hz of pitch frequency coverage per additional
sample of lowered minimum pitch period), we can get a lot more pitch
frequency coverage by increasing only marginally the number of pitch period
candidates that needs to be searched by the pitch estimator, so it appears
to be a no-brainer good deal to me.  In addition, unlike the case of low
pitch frequency where the noise in inter-harmonic valley regions is better
masked, for high pitch frequencies it is the opposite, as the inter-harmonic
noise is not masked well by the pitch harmonics and thus the pitch
prefilter/postfilter approach should give more benefit, theoretically
at least.

I think one thing worth trying is that rather than searching through and
coding every integer between the minimum pitch period and the maximum
pitch period (which takes 10 bits to code independently), perhaps we should
progressively increase the distance between adjacent allowed pitch period
candidates from 1 sample to 2, 3, or even higher number of samples as the
pitch period increases from the minimum pitch period to the maximum pitch
period of 960.  By doing this, we will not only reduce the search
complexity of the pitch estimator, but also reduce the number of bits
required to encode the pitch period from 10 bits to 9 bits, or perhaps even
8 bits or lower.  Of course, as the resolution of the pitch frequency
reduces in the low pitch frequency range, the frequencies of the high pitch
harmonics may be significantly misaligned, and thus the effectiveness of the
pitch prefilter/postfilter approach may diminsh, so some tuning may
be needed to strike a balance.  However, as you said, when the pitch
frequency is low, the benefit of the pitch prefilter/postfilter method
is low in the first place, so it may not be too big of an issue.  As to
the potential bad effect of mismatched higher harmonic frequencies, I
think the pitch tap gain determination step should take care of it
automatically.  That is, if the quantized frequencies at high harmonics
do not match well with the unquantized ones and if the signal energy
at such high harmonics is high, then the pitch prediction gain should be
relatively low, leading to a small pitch tap gain and thus reducing the
effect of the pitch filtering and reducing the potential bad effects.
These are all just theoretical reasoning.  They still need to be verified
by experiments.

Raymond
-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Monday, November 01, 2010 10:28 AM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi,

Cheers,

        Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 1 Nov 2010 19:02:24 -0700

Hi Greg,

Interesting results!

Would you post your 8 kb/s coded speech as well so we can listen.  I am curious what it sounds like.

Greg and Jean-Marc,

Do you have some audio files for "before" and "after" comparisons of CELT codec outputs without or with the pitch prefilter/postfilter, respectively, so we can hear what kind of results you are getting with your latest code version?  I am especially curious to hear the solo trumpet signal processed by your latest version with 3-tap filters.  I would like to compare it with the result I got with single-tap pitch filters.  Thanks.

Raymond

-----Original Message-----
From: Gregory Maxwell [mailto:[email-masked]] 
Sent: Monday, November 01, 2010 12:48 PM
To: Jean-Marc Valin; Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT codec

In the current encoder these gains are scaled by 0.7 and clamped to the range [0,0.6] before being used as the goal is not to completely whiten the spectrum...

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 01 Nov 2010 23:30:47 -0400

Hi Raymond,

> lowest risk of unintentional IP infringement.

I'm afraid that independently coding each subframe might actually
require too many bits (and that those bits would be more useful in
quantization). Because the pitch here is used only as a post-filter, the
accuracy on the period isn't quite as critical as it is when using pitch
for prediction. That being said, I think it's more useful to find a more
efficient way to code the contour, especially considering that we want
that contour to be very smooth, not arbitrary.

> this will require new code and thus additional code space.

I don't think this is worth adding look-ahead for, though if it's
encoder-side, implementers would be free to do so if they like.

> have already been patented (unless the idea is so old that any related

This is actually the approach I had in mind. It's probably worth
investigating what has been done there.

> period range, 

Right now, I first code the octave number, then the pitch within the
octave. Going to 16 would require another octave (7 instead of 6), which
means 0.22 bit.

> but it will increase the covered pitch frequency range
> perceptual benefit at such a high pitch frequency. 

I don't actually think the postfilter would be necessary at that point
as its "resolution" would actually be worse than the resolution of the
CELT bands. The peaks of the response would be too wide to have an
effect within one CELT band and the energy quantization would do a
better job across bands.

> 8 bits or lower.

As I said, I'm currently coding in 6 octaves, so for the highest pitches
(small period), only 7.5 bits are required while for lowest pitches we
need 11.5 bits. I think I prefer this to lowering the resolution of the
period for low pitches

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 01 Nov 2010 23:39:55 -0400

Hi Raymond (and others),

Here's some trumpet samples at 64 kb/s and 96 kb/s stereo for 5 ms
frames (quality with 20 ms frames is obviously better):

without postfilter:
http://jmvalin.ca/misc_stuff/reftpt64_nopf.wav
http://jmvalin.ca/misc_stuff/reftpt96_nopf.wav

with post-filter:
http://jmvalin.ca/misc_stuff/reftpt64_pf.wav
http://jmvalin.ca/misc_stuff/reftpt96_pf.wav

Cheers,

	Jean-Marc

On 10-11-01 10:02 PM, Raymond (Juin-Hwey) Chen wrote:
> Do you have some audio files for "before" and "after" comparisons of
> CELT codec outputs without or with the pitch prefilter/postfilter,
> respectively, so we can hear what kind of results you are getting
> with your latest code version?  I am especially curious to hear the
> solo trumpet signal processed by your latest version with 3-tap
> filters.  I would like to compare it with the result I got with
> single-tap pitch filters.  Thanks.
> 
> Raymond
> 
> -----Original Message----- From: Gregory Maxwell
> [mailto:[email-masked]] Sent: Monday, November 01, 2010 12:48
> PM To: Jean-Marc Valin; Raymond (Juin-Hwey) Chen Cc:
> [email-masked]'; Jean-Marc Valin Subject: RE: [codec] A pitch
> filtering method to improve the mode 1 CELT codec
> 
> Jean-Marc Valin [[email-masked]] wrote: [snip]
>> The main issue that remains now is how to handle the pitch contour
>> for frame sizes above 5 ms. Right now, the pitch period is forced
>> constant through the whole frame, which is likely suboptimal. Any
>> suggestion on how to best estimate and code the pitch contour? The
>> other thing that's still undecided is the range of values we want
>> to encode. In the current version, the period can range from 30
>> samples (1500 Hz pitch) to 960 samples (50 Hz pitch). As for the
>> gain, non-zero values can range between 0.25 and 0.625 in
>> increments of 0.125. Any opinion on these?
> 
> I've been working with Jean-Marc on IRC on this but I thought it
> might be useful to bring some more of the discussion to the list. :)
> 
> I made some measurements of the code on one of the small test sets I
> used. The test set consists of 10 second segments randomly selected
> from about 30 hours of audio. It's a bit commercial-recording music
> heavy and speech light.
> 
> http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/pitch-960-2.png is
> a graph of the density of the pitch offsets I found with 5ms frames
> plotted against a 0.33/n curve.
> 
> http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/gain1.png is a
> graph of the raw gain values our search finds.
> 
> In the current encoder these gains are scaled by 0.7 and clamped to
> the range [0,0.6] before being used as the goal is not to completely
> whiten the spectrum...
> 
>> From listening I think the scaling of 0.7 isn't enough in general
>> and that 0.5 is probably better but there are some samples that its
>> quite helpful for... some more encoder intelligence is probably
>> justified.
> 
> On a more silly note the addition of pitch lets CELT achieve
> intelligible (but not good) wideband speech at the ludicrous rate of
> 2.5kbit/sec:
> http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/16k_60ms_20-2.wav 
> (and 8kbit/sec wideband which sounds better than speex, though it
> still has nothing on SILK for speech low rates. :) )
> 
> While I don't expect anyone to intentionally run the codec at such
> insanely low rates I think is helpful if the codec can maintain
> intelligibility down to very low rates in order to cope with extreme
> congestion.
> 
> 
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 2 Nov 2010 20:19:38 -0700

Hi Jean-Marc,

> encoder-side, implementers would be free to do so if they like.

[Raymond]: You misunderstood what I meant.  I didn't suggest adding look-
ahead or delay to the system. Let me use an example.  Suppose we have a 
20 ms frame size and a 5 ms subframe size, so each frame has 4 subframes.  
Now, when we buffer a 20 ms frame, we already have 4 subframes buffered.  
Thus, except for the last subframe which does not have any "future subframe" 
as look-ahead, the first three subframes have one or more future subframes 
that they can use as look-ahead to improve the reliability of the pitch 
estimate.  This is what I meant by "use the future subframes in the current 
frame as the 'look-ahead' when performing pitch estimation in the earlier 
subframes". Doing this won't increase delay.

> investigating what has been done there.

[Raymond]: Agreed.

> octave. Going to 16 would require another octave (7 instead of 6), which

[Raymond]: It's not clear to me from your description exactly how you code 
the pitch period.  Would you please elaborate?  Thanks.

> effect within one CELT band and the energy quantization would do a

[Raymond]: At low bit-rates, how accurate can you guarantee the energy 
quantization?  Is it possible that at very low bit-rates, the band energy 
quantization can introduce enough coding errors such that the CELT coding 
noise in the inter-harmonic spectral valley regions becomes audible?  
If so, then the pitch prefilter/postfilter approach should still help.

> need 11.5 bits. I think I prefer this to lowering the resolution of the

[Raymond]: Again, would you please describe your method in more detail?  
Thanks.  Again, my point is just that coding very long pitch period at 
1 sample resolution at 48 kHz sampling rate is likely a waste.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 02 Nov 2010 23:42:59 -0400

Hi Raymond,

> subframes". Doing this won't increase delay.

OK, I understand now. In any case, it seems like this is entirely an
encoder issue, so I'm not that concerned for now.

> the pitch period.  Would you please elaborate?  Thanks.

The range of pitch periods is divided into 5 octaves (sorry not 6):
[32,64[
[64,128[
[128,256[
[256,512[
[512,1024[

The octave is coded first using log2(5) bits. After that, the actual
period withing the octave is coded, so if we were in the [32,64[ range,
then we need 5 more bits to code the pitch. On the other hand, if the
pitch is in the [512,1024[ range, then it takes 9 extra bits. For that
reason, the over head of having the pitch go up to 1023 rather than 511
only has a cost of log2(5)-log2(4), rather than a cost of one bit like
it would have had if the pitch was coded uniformly with 10 bits.

> If so, then the pitch prefilter/postfilter approach should still help.

The band energy quantization is not an issue here. The largest
quantization steps are 6 dB (so +/3 dB max error), but in practice it's
almost always 3 dB (1.5 dB max error) or less, even at very low bit-rate.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 2 Nov 2010 20:53:35 -0700

Hi Jean-Marc,

Thanks for sharing the processed versions of the trumpet signal.  I did a listening comparison with the previous processed versions I sent you that were obtained with my preliminary single-tap pitch prefilter/postfilter and with the pre-release version 0.9.0 CELT that you sent me previously.  

I was surprised to find that your latest output files sounded noticeably worse to my ears than my previous files at both 64 and 96 kb/s.  I don't know whether that's due to additional changes you made in the CELT codec itself, in the pitch prefilter/postfilter, or both.  There are many instances of mid- to low-frequency disturbance (I don't know how to describe it) in many places of your latest output files (both pf and nopf versions), the most obvious place being 3.3 to 3.9 seconds from the beginning of the files at both bit-rates.  This distortion is more pronounced in the 64 kb/s version, but it is also quite audible to me even in the 96 kb/s version.  In contrast, the previous processed versions I sent you were mostly free of such distortion.  (The 64 kb/s output without pitch filtering has a slight trace of this distortion, while the pitch-filtered version and both filtered and unfiltered versions of 96 kb/s versions are free of this distortion.)

This got me somewhat concerned. Your 3-tap pitch filters are meant to be an improvement over my original single-tap filters. Could there be some implementation issues that degraded the performance?  Or did some recent tunings you made in the CELT codec significantly degrade the coding performance for this signal? 

Raymond 

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Monday, November 01, 2010 8:40 PM
To: Raymond (Juin-Hwey) Chen
Cc: Gregory Maxwell; Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond (and others),

without postfilter:

with post-filter:

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 00:10:33 -0400

Hi Raymond,

Off the top of my head, here are the differences between the conditions
of your experiment and the ones in the code I have:

1) Different pitch estimator
2) Different choice of "global" gain
3) Unrelated changes to the CELT code (e.g. bit allocation)
4) Quantization of the pitch gain
5) The post-filter is now applied in the pre-emphasized domain
6) 3-tap gain

These are (roughly) ordered from most likely to least likely issue. I
guess we'll have to experiment to find what the cause is. If you like, I
can send you the output for a 1-tap predictor. Maybe you could also send
me the exact pitch period and gain you use for each 5 ms frame so I can
process using that.

Now that I think of it, if you say the nopf file also sounds worse than
it did in the version you had, then it would have to be something
unrelated to the post-filter. Could you tell me the exact version you
had (did I give you the hash value with it)?

Cheers,

	Jean-Marc

On 10-11-02 11:53 PM, Raymond (Juin-Hwey) Chen wrote:
> Hi Jean-Marc,
> 
> Thanks for sharing the processed versions of the trumpet signal.  I
> did a listening comparison with the previous processed versions I
> sent you that were obtained with my preliminary single-tap pitch
> prefilter/postfilter and with the pre-release version 0.9.0 CELT that
> you sent me previously.
> 
> I was surprised to find that your latest output files sounded
> noticeably worse to my ears than my previous files at both 64 and 96
> kb/s.  I don't know whether that's due to additional changes you made
> in the CELT codec itself, in the pitch prefilter/postfilter, or both.
> There are many instances of mid- to low-frequency disturbance (I
> don't know how to describe it) in many places of your latest output
> files (both pf and nopf versions), the most obvious place being 3.3
> to 3.9 seconds from the beginning of the files at both bit-rates.
> This distortion is more pronounced in the 64 kb/s version, but it is
> also quite audible to me even in the 96 kb/s version.  In contrast,
> the previous processed versions I sent you were mostly free of such
> distortion.  (The 64 kb/s output without pitch filtering has a slight
> trace of this distortion, while the pitch-filtered version and both
> filtered and unfiltered versions of 96 kb/s versions are free of this
> distortion.)
> 
> This got me somewhat concerned. Your 3-tap pitch filters are meant to
> be an improvement over my original single-tap filters. Could there be
> some implementation issues that degraded the performance?  Or did
> some recent tunings you made in the CELT codec significantly degrade
> the coding performance for this signal?
> 
> Raymond
> 
> [mailto:[email-masked]] Sent: Monday, November 01,
> 2010 8:40 PM To: Raymond (Juin-Hwey) Chen Cc: Gregory Maxwell;
> Jean-Marc Valin; [email-masked]' Subject: Re: [codec] A pitch
> without postfilter: http://jmvalin.ca/misc_stuff/reftpt64_nopf.wav 
> with post-filter: http://jmvalin.ca/misc_stuff/reftpt64_pf.wav 
>> Do you have some audio files for "before" and "after" comparisons
>> of CELT codec outputs without or with the pitch
>> prefilter/postfilter, respectively, so we can hear what kind of
>> results you are getting with your latest code version?  I am
>> especially curious to hear the solo trumpet signal processed by
>> your latest version with 3-tap filters.  I would like to compare it
>> with the result I got with single-tap pitch filters.  Thanks.
>> 
>> Raymond
>> 
>> [mailto:[email-masked]] Sent: Monday, November 01, 2010
>> 12:48 PM To: Jean-Marc Valin; Raymond (Juin-Hwey) Chen Cc: 
>>> The main issue that remains now is how to handle the pitch
>>> contour for frame sizes above 5 ms. Right now, the pitch period
>>> is forced constant through the whole frame, which is likely
>>> suboptimal. Any suggestion on how to best estimate and code the
>>> pitch contour? The other thing that's still undecided is the
>>> range of values we want to encode. In the current version, the
>>> period can range from 30 samples (1500 Hz pitch) to 960 samples
>>> (50 Hz pitch). As for the gain, non-zero values can range between
>>> 0.25 and 0.625 in increments of 0.125. Any opinion on these?
>> 
>> I made some measurements of the code on one of the small test sets
>> I used. The test set consists of 10 second segments randomly
>> selected from about 30 hours of audio. It's a bit
>> commercial-recording music heavy and speech light.
>> 
>> In the current encoder these gains are scaled by 0.7 and clamped
>> to the range [0,0.6] before being used as the goal is not to
>> completely whiten the spectrum...
>> 
>>> From listening I think the scaling of 0.7 isn't enough in
>>> general and that 0.5 is probably better but there are some
>>> samples that its quite helpful for... some more encoder
>>> intelligence is probably justified.
>> 
>> intelligible (but not good) wideband speech at the ludicrous rate
>> of 2.5kbit/sec: 
>> intelligibility down to very low rates in order to cope with
>> extreme congestion.
>> 
>> 
>> 
>> 
> 
> 
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 2 Nov 2010 21:40:31 -0700

Hi Jean-Marc,

Thanks for the explanation of your pitch coding scheme.  From what you 
described, it seems that every single integer between 512 and 1023 is allowed 
and can be coded by your scheme.  This perhaps makes the code simple, but if 
you are truly concerned about the pitch coding bit-rate, then I think chances 
are we can use a lower time resolution for pitch periods in that range (or 
even the [256,512[ range) and our ears won't hear the difference.  

Regarding high pitch frequency (low pitch period), I am still not sold :-) 
Even if you can code the band energy to +- 1.5 dB, when the overall codec bit-
rate is very low, I think you are still likely to hear substantial coding 
distortion from the inter-harmonic spectral regions, and as long as that's the 
case, I think you can get some reduction of the perceived level of coding 
distortion by using the pitch prefilter/postfilter method.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Tuesday, November 02, 2010 8:43 PM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 01:10:50 -0400

> even the [256,512[ range) and our ears won't hear the difference.  

Well, my reasoning is that when the harmonics are closely spaced, it's
actually very easy to make a small error in the pitch and end up with a
comb filter that doesn't align with the harmonic structure. That's why
I'm not sure that it's worth the 1-2 bit saving of using a lower
resolution. Then again, that's just another thing to check experimentally.

> distortion by using the pitch prefilter/postfilter method.

It's not that much the +/- 1.5 dB but the fact that if there's no energy
in a band between two harmonics, then the decoder will also have no
energy in that band. For the few bands that will have energy, then it's
a case where we can dynamically allocate mode bits.

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 01:25:29 -0400

On 10-11-03 12:23 AM, Raymond (Juin-Hwey) Chen wrote:
> Yes, I believe the codec itself performed significantly worse for
> this file than the previous 0.9.0 you gave me.  The question is
> whether the pitch filters also played a role.
> 
> I am attaching the zip file you sent me previously.  I used the codec
> version in this zip file to do my previous processing.

There's been many changes since the version in that zip file, most of
which not related to the post-filter. These changes have actually
improved audio quality quite a bit, though I guess it's possible that it
degraded this particular sample.

> Do you hear the distortion between 3.3 and 3.9 seconds that I talked
> about?

Will listen tomorrow morning. Can you send the files you have with and
without post-filter (please use [email-masked] for large files like
these)?

> Let's do one thing at a time by first trying to find out whether that
> distortion is entirely due to the codec changes.  If we find that the
> pitch filter differences are also partially responsible, we can then
> go to the next step of comparing pitch parameters.

Sure.

	Jean-Marc

> [mailto:[email-masked]] Sent: Tuesday, November 02, 2010
> 9:11 PM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc Valin; Gregory
> Maxwell; [email-masked]' Subject: Re: [codec] A pitch filtering
> method to improve the mode 1 CELT codec
> 
> Hi Raymond,
> 
> Off the top of my head, here are the differences between the
> conditions of your experiment and the ones in the code I have:
> 
> 1) Different pitch estimator 2) Different choice of "global" gain 3)
> Unrelated changes to the CELT code (e.g. bit allocation) 4)
> Quantization of the pitch gain 5) The post-filter is now applied in
> the pre-emphasized domain 6) 3-tap gain
> 
> These are (roughly) ordered from most likely to least likely issue.
> I guess we'll have to experiment to find what the cause is. If you
> like, I can send you the output for a 1-tap predictor. Maybe you
> could also send me the exact pitch period and gain you use for each 5
> ms frame so I can process using that.
> 
> Now that I think of it, if you say the nopf file also sounds worse
> than it did in the version you had, then it would have to be
> something unrelated to the post-filter. Could you tell me the exact
> version you had (did I give you the hash value with it)?
> 
> Cheers,
> 
> Jean-Marc
> 
> 
>> Thanks for sharing the processed versions of the trumpet signal.
>> I did a listening comparison with the previous processed versions
>> I sent you that were obtained with my preliminary single-tap pitch 
>> prefilter/postfilter and with the pre-release version 0.9.0 CELT
>> that you sent me previously.
>> 
>> noticeably worse to my ears than my previous files at both 64 and
>> 96 kb/s.  I don't know whether that's due to additional changes you
>> made in the CELT codec itself, in the pitch prefilter/postfilter,
>> or both. There are many instances of mid- to low-frequency
>> disturbance (I don't know how to describe it) in many places of
>> your latest output files (both pf and nopf versions), the most
>> obvious place being 3.3 to 3.9 seconds from the beginning of the
>> files at both bit-rates. This distortion is more pronounced in the
>> 64 kb/s version, but it is also quite audible to me even in the 96
>> kb/s version.  In contrast, the previous processed versions I sent
>> you were mostly free of such distortion.  (The 64 kb/s output
>> without pitch filtering has a slight trace of this distortion,
>> while the pitch-filtered version and both filtered and unfiltered
>> versions of 96 kb/s versions are free of this distortion.)
>> 
>> This got me somewhat concerned. Your 3-tap pitch filters are meant
>> to be an improvement over my original single-tap filters. Could
>> there be some implementation issues that degraded the performance?
>> Or did some recent tunings you made in the CELT codec significantly
>> degrade the coding performance for this signal?
>> 
>> Raymond
>> 
>>> Would you post your 8 kb/s coded speech as well so we can
>>> listen. I am curious what it sounds like.
>>> 
>>> Greg and Jean-Marc,
>>> 
>>> Do you have some audio files for "before" and "after"
>>> comparisons of CELT codec outputs without or with the pitch 
>>> your latest version with 3-tap filters.  I would like to compare
>>> it with the result I got with single-tap pitch filters.  Thanks.
>>> 
>>> Raymond
>>> 
>>>> contour for frame sizes above 5 ms. Right now, the pitch
>>>> period is forced constant through the whole frame, which is
>>>> likely suboptimal. Any suggestion on how to best estimate and
>>>> code the pitch contour? The other thing that's still undecided
>>>> is the range of values we want to encode. In the current
>>>> version, the period can range from 30 samples (1500 Hz pitch)
>>>> to 960 samples (50 Hz pitch). As for the gain, non-zero values
>>>> can range between 0.25 and 0.625 in increments of 0.125. Any
>>>> opinion on these?
>>> 
>>> I made some measurements of the code on one of the small test
>>> sets I used. The test set consists of 10 second segments
>>> randomly selected from about 30 hours of audio. It's a bit 
is a graph of the density of the pitch offsets I found with 5ms
>>> frames plotted against a 0.33/n curve.
>>> 
>>> intelligible (but not good) wideband speech at the ludicrous
>>> rate of 2.5kbit/sec: 
>>> While I don't expect anyone to intentionally run the codec at
>>> such insanely low rates I think is helpful if the codec can
>>> maintain intelligibility down to very low rates in order to cope
>>> with extreme congestion.
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 3 Nov 2010 08:27:40 -0700

> Would you post your 8 kb/s coded speech as well so we can listen.  I am curious what it sounds like.

It was mostly a silly test— but sure

This is 16kHz, 8kbit/sec, with 30ms frames:
http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/16k_30ms_30.wav

I think it obviously suffers from the infrequency of pitch updates (once every 30ms).  It will be a little more interesting to try again after subframe pitch is implemented.

For comparison— 
Speex: http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/16k_wb_8kb_spx.wav
and
SILK: http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/16k_wb_8kb_silk.wav

(The silk performance is, as expected, very good)

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 12:04:10 -0400

Hi Raymond,

When I sent you that zip find. I also told you to change the value of 
"trim_index" from 3 to 5. Is it possible that the files you originally sent 
me had trim_index set to 5 (which was never the default). If that's the 
case, then it would explain the difference. Using a higher value means 
allocating more bits to the low frequencies (and thus removing from the 
high frequencies). While this helps for the trumpet file where the tonal 
component is mostly in the low frequencies, it tends to hurt pretty much 
everything else and I recently lowered that value to 2. This value is 
actually signalled in the bit-stream so we can change it every frame, but 
so far I haven't found any reliable way to do so.

Cheers,

	Jean-Marc

On 10-11-03 12:23 AM, Raymond (Juin-Hwey) Chen wrote:
> Hi Jean-Marc,
>
> Yes, I believe the codec itself performed significantly worse for this
> file than the previous 0.9.0 you gave me.  The question is whether the
> pitch filters also played a role.
>
> pitch filter differences are also partially responsible, we can then go
> to the next step of comparing pitch parameters.
>
> Raymond
>
> Maxwell; [email-masked]' Subject: Re: [codec] A pitch filtering method
> to improve the mode 1 CELT codec
>
> Hi Raymond,
>
> Unrelated changes to the CELT code (e.g. bit allocation) 4) Quantization
> of the pitch gain 5) The post-filter is now applied in the
> pre-emphasized domain 6) 3-tap gain
>
> guess we'll have to experiment to find what the cause is. If you like,
> I can send you the output for a 1-tap predictor. Maybe you could also
> send me the exact pitch period and gain you use for each 5 ms frame so I
> can process using that.
>
>> did a listening comparison with the previous processed versions I sent
>> you that were obtained with my preliminary single-tap pitch
>> There are many instances of mid- to low-frequency disturbance (I don't
>> know how to describe it) in many places of your latest output files
>> (both pf and nopf versions), the most obvious place being 3.3 to 3.9
>> seconds from the beginning of the files at both bit-rates. This
>> distortion is more pronounced in the 64 kb/s version, but it is also
>> quite audible to me even in the 96 kb/s version.  In contrast, the
>> previous processed versions I sent you were mostly free of such
>> some implementation issues that degraded the performance?  Or did some
>> recent tunings you made in the CELT codec significantly degrade the
>> coding performance for this signal?
>>
>> Raymond
>>
>>>> and that 0.5 is probably better but there are some samples that
>>>> its quite helpful for... some more encoder intelligence is
>>>> probably justified.
>>>

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 12:17:40 -0400

Just to give you an idea, here's what I get by forcing "trim_index" to 5:

http://jmvalin.ca/misc_stuff/reftpt64_trim5-pf.wav
http://jmvalin.ca/misc_stuff/reftpt96_trim5-pf.wav

	Jean-Marc

On 10-11-03 12:04 PM, Jean-Marc Valin wrote:
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 3 Nov 2010 13:44:14 -0700

Hi Jean-Marc,

> degraded this particular sample.

That's certainly quite possible.

> without post-filter (please use [email-masked] for large files like

They are the same files that I sent you two weeks ago.  I have sent them to your gmail account again as you requested.

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 17:45:33 -0400

Hi Raymond,

I checked the files you used and they are almost certainly generated with a 
version of the encoder I specifically tuned for these files (i.e. 
trim_index=5). If you listen to the files I posted earlier today:

I'm sure you will find that they sound at least as good as what you had. 
Can you confirm?

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 3 Nov 2010 15:05:55 -0700

Hi Jean-Marc,

(I was in the middle of writing this email when you last email came in.)

Since your instructions when sending me the previous code was to change trim_index from 3 to 5, we followed your instructions and changed that line 903 in celt.c.  I just confirmed that again.

I listened to these two new files you sent with trim_index=5 and compared them with the ones you posted Monday night.  The mid- to low-frequency disturbance is somewhat reduced with trim5 files, but they are still there in both 64 and 96 kbps versions.  

For example, the last version of the 64 kbps pf file has three instances of audible "disturbance" at 3.5, 3.6, and 3.7 seconds, while the trim6 version of the 64 kbps pf file only has it at 3.6 and 3.7 seconds.  The one at 3.5 second is gone.  The intensity of the remaining two seems to be reduced slightly as well.  Same with the 96 kbps files, but you can still hear them in both 64 and 96 kbps versions of the trim5 files.

In contrast, there is no audible mid- to low-frequency disturbance in the pitch-filtered 64 and 96 kbps output signals that we processed with your previous pre-release CELT version 0.9.0 with trim_index=5 and with my single-tap pitch prefilter/postfilter and my simple pitch estimator.  They sound completely clean in that regard (mid- to low-frequency disturbance).

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Wednesday, November 03, 2010 2:46 PM
To: Raymond (Juin-Hwey) Chen
Cc: Jean-Marc Valin; Gregory Maxwell; [email-masked]'
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 18:17:12 -0400

Hi Raymond,

If you look at the spectrogram of the section between 3.5 and 3.9 seconds, 
you can see that there's actually two notes at the same time. The previous 
note is fading out, while the new one is beginning. During that time, the 
current pitch detector only finds a small gain, so the the post-filter is 
effectively turned off. I suspect that your pitch detector is still on 
during that time and manages to reduce some of the LF noise. Can you 
confirm that? I think it would be really helpful if you could send me the 
period and gain for each 5 ms frame in that file. That way we can either 
confirm or rule out the pitch estimator as the remaining effect.

	Jean-Marc

On 10-11-03 06:05 PM, Raymond (Juin-Hwey) Chen wrote:
> Since your instructions when sending me the previous code was to change
> trim_index from 3 to 5, we followed your instructions and changed that
> line 903 in celt.c.  I just confirmed that again.
>
> I listened to these two new files you sent with trim_index=5 and
> compared them with the ones you posted Monday night.  The mid- to
> low-frequency disturbance is somewhat reduced with trim5 files, but they
> are still there in both 64 and 96 kbps versions.
>
> For example, the last version of the 64 kbps pf file has three instances
> of audible "disturbance" at 3.5, 3.6, and 3.7 seconds, while the trim6
> version of the 64 kbps pf file only has it at 3.6 and 3.7 seconds.  The
> one at 3.5 second is gone.  The intensity of the remaining two seems to
> be reduced slightly as well.  Same with the 96 kbps files, but you can
> still hear them in both 64 and 96 kbps versions of the trim5 files.
>
> In contrast, there is no audible mid- to low-frequency disturbance in
> the pitch-filtered 64 and 96 kbps output signals that we processed with
> your previous pre-release CELT version 0.9.0 with trim_index=5 and with
> my single-tap pitch prefilter/postfilter and my simple pitch estimator.
> They sound completely clean in that regard (mid- to low-frequency
> disturbance).
>
> Raymond
>
> [mailto:[email-masked]] Sent: Wednesday, November 03, 2010
> 2:46 PM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc Valin; Gregory
> I checked the files you used and they are almost certainly generated
> with a version of the encoder I specifically tuned for these files
> (i.e. trim_index=5). If you listen to the files I posted earlier today:
>
> I'm sure you will find that they sound at least as good as what you
> had. Can you confirm?
>
> Cheers,
>
> Jean-Marc
>
>>> There's been many changes since the version in that zip file, most
>>> of which not related to the post-filter. These changes have
>>> actually improved audio quality quite a bit, though I guess it's
>>> possible that it degraded this particular sample.
>>
>>> Will listen tomorrow morning. Can you send the files you have with
>>> and without post-filter (please use [email-masked] for large
>>> files like these)?
>>
>> They are the same files that I sent you two weeks ago.  I have sent
>> them to your gmail account again as you requested.
>>
>> Jean-Marc
>>
>>> Raymond
>>>
>>> [mailto:[email-masked]] Sent: Tuesday, November 02,
>>> 2010 9:11 PM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc Valin;
>>> Gregory Maxwell; [email-masked]' Subject: Re: [codec] A pitch
>>> 1) Different pitch estimator 2) Different choice of "global" gain
>>> 3) Unrelated changes to the CELT code (e.g. bit allocation) 4)
>>> could also send me the exact pitch period and gain you use for each
>>> 5 ms frame so I can process using that.
>>>
>>>> 96 kb/s.  I don't know whether that's due to additional changes
>>>> you made in the CELT codec itself, in the pitch
>>>> prefilter/postfilter, or both. There are many instances of mid- to
>>>> low-frequency disturbance (I don't know how to describe it) in
>>>> many places of your latest output files (both pf and nopf
>>>> versions), the most obvious place being 3.3 to 3.9 seconds from
>>>> the beginning of the files at both bit-rates. This distortion is
>>>> more pronounced in the 64 kb/s version, but it is also quite
>>>> audible to me even in the 96 kb/s version.  In contrast, the
>>>> distortion.  (The 64 kb/s output without pitch filtering has a
>>>> slight trace of this distortion, while the pitch-filtered version
>>>> and both filtered and unfiltered versions of 96 kb/s versions are
>>>> free of this distortion.)
>>>>
>>>> This got me somewhat concerned. Your 3-tap pitch filters are
>>>> meant to be an improvement over my original single-tap filters.
>>>> Could there be some implementation issues that degraded the
>>>> performance? Or did some recent tunings you made in the CELT codec
>>>> significantly degrade the coding performance for this signal?
>>>>
>>>> Raymond
>>>>
>>>> [mailto:[email-masked]] Sent: Monday, November
>>>> 01, 2010 8:40 PM To: Raymond (Juin-Hwey) Chen Cc: Gregory
>>>> Maxwell; Jean-Marc Valin; [email-masked]' Subject: Re: [codec] A
>>>> pitch filtering method to improve the mode 1 CELT codec
>>>>
>>>> Hi Raymond (and others),
>>>>
>>>> Here's some trumpet samples at 64 kb/s and 96 kb/s stereo for 5
>>>> ms frames (quality with 20 ms frames is obviously better):
>>>>
>>>> without postfilter:
>>>>> your latest version with 3-tap filters.  I would like to
>>>>> compare it with the result I got with single-tap pitch filters.
>>>>> Thanks.
>>>>>
>>>>> Raymond
>>>>>
>>>>>> code the pitch contour? The other thing that's still
>>>>>> undecided is the range of values we want to encode. In the
>>>>>> current version, the period can range from 30 samples (1500 Hz
>>>>>> pitch) to 960 samples (50 Hz pitch). As for the gain, non-zero
>>>>>> values can range between 0.25 and 0.625 in increments of
>>>>>> 0.125. Any opinion on these?
>>>>>
>>>>> I've been working with Jean-Marc on IRC on this but I thought
>>>>> it might be useful to bring some more of the discussion to the
>>>>> list. :)
>>>>>
>>>>> http://myrandomnode.dyndns.org:8080/~gmaxwell/celt/gain1.png is
>>>>> a graph of the raw gain values our search finds.
>>>>>
>>>>> In the current encoder these gains are scaled by 0.7 and
>>>>> clamped to the range [0,0.6] before being used as the goal is
>>>>> not to completely whiten the spectrum...
>>>>>
>>>>> maintain intelligibility down to very low rates in order to
>>>>> cope with extreme congestion.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
>
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 18:40:50 -0400

Hi Raymond,

Here are some updated files:

http://jmvalin.ca/misc_stuff/reftpt64-nopf.wav
http://jmvalin.ca/misc_stuff/reftpt96-nopf.wav
http://jmvalin.ca/misc_stuff/reftpt64-pf.wav
http://jmvalin.ca/misc_stuff/reftpt96-pf.wav

Can you tell me if you can still hear the artefact? Note that these use the 
default (trim 3) allocation.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 3 Nov 2010 17:44:32 -0700

Hi Jean-Marc,

I listened to the new files and compared with the previous files you sent Monday night.  Here are my observations.

(1) For the CELT output without pitch filtering, the new files sounded worse than the old one, with more audible and more frequent instances of mid- to low-frequency disturbance, even in the 0 to 3.3 seconds range.  This is especially true for the 64 kbps version from 2.6 to 3.0 seconds, where there are now several instances of distortion that sounded somewhat like bubbling noise.

(2) Magically, with pitch prefilter and postfilter added, the new pf files sounds much cleaner and significantly better than the old pf files. The mid- to low-frequency disturbance was basically gone and now they sounded similar to the files I sent you previously.  In your separate email to me you indicated that you found some pitch period halving.  Is that all you have fixed to get rid of the MF to LF disturbance in the pitch-filtered version?  I don't understand this: pitch period halving should not affect the nopf files in (1) above, but the MF to LF disturbance is already in the output of CELT without pitch filtering, so it cannot be caused by the pitch period halving; then why would fixing pitch period halving eliminate such disturbance in the output of CELT with pitch filtering?

In any case, from a theoretical perspective, I think detecting a pitch period smaller than the true pitch period is more detrimental than detecting an integer multiple of the true pitch period.  In the latter case, you don't get as much noise attenuation between pitch harmonics compared with using the true pitch period, but at least the pitch harmonics are preserved.  In the former case (detected pitch period << true pitch period), you can attenuate some of the pitch harmonics, so this should be avoided as much as possible.

I was just about ready to send you the pitch periods and pitch taps when you told me you didn't need them because you found the problem (pitch halving).  I can still send them to you if it can help you.  The pitch estimator I used to generate the previous files I sent you was a very simple one and for this trumpet file it produced a lot of integer multiples of the true pitch period, but it hardly produced any sub-multiple of the true pitch period, and despite the frequent detection of integer multiples of the pitch period, the audio quality improvement provided by the resulting pitch prefilter/postfilter is still quite dramatic.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Wednesday, November 03, 2010 3:41 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Cheers,

        Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 21:02:16 -0400

On 10-11-03 08:44 PM, Raymond (Juin-Hwey) Chen wrote:
> (1) For the CELT output without pitch filtering, the new files
> sounded worse than the old one, with more audible and more frequent
> instances of mid- to low-frequency disturbance, even in the 0 to 3.3
> seconds range.  This is especially true for the 64 kbps version from
> 2.6 to 3.0 seconds, where there are now several instances of
> distortion that sounded somewhat like bubbling noise.

This is simply due to the fact that I've recently (earlier today)
increased the HF bit allocation at the expense of LF. On most files (not
this one) this seems to be beneficial.

> (2) Magically, with pitch prefilter and postfilter added, the new pf
> files sounds much cleaner and significantly better than the old pf
> files. The mid- to low-frequency disturbance was basically gone and
> now they sounded similar to the files I sent you previously.  In your
> separate email to me you indicated that you found some pitch period
> halving.  Is that all you have fixed to get rid of the MF to LF
> disturbance in the pitch-filtered version? 

Yes, removing pitch halving is all I did. It means the post-filter can
now overcome the reduced LF bit allocation I mentioned above.

 I don't understand this:
> pitch period halving should not affect the nopf files in (1) above,
> but the MF to LF disturbance is already in the output of CELT without
> pitch filtering, so it cannot be caused by the pitch period halving;
> then why would fixing pitch period halving eliminate such disturbance
> in the output of CELT with pitch filtering?

Not sure what you mean here. Is it still relevant considering the
explanation above? BTW, sorry for changing the "nopf" at the same time
as I'm working on the pf. I've just tweaking all areas of the codec at
the same time.

> In any case, from a theoretical perspective, I think detecting a
> pitch period smaller than the true pitch period is more detrimental
> than detecting an integer multiple of the true pitch period.  In the
> latter case, you don't get as much noise attenuation between pitch
> harmonics compared with using the true pitch period, but at least the
> pitch harmonics are preserved.  In the former case (detected pitch
> period << true pitch period), you can attenuate some of the pitch
> harmonics, so this should be avoided as much as possible.

It was even worse than that. Pitch halving caused the gain to go down
dramatically, so we ended up with no post-filter at all.

> I was just about ready to send you the pitch periods and pitch taps
> when you told me you didn't need them because you found the problem
> (pitch halving).  I can still send them to you if it can help you.
> The pitch estimator I used to generate the previous files I sent you
> was a very simple one and for this trumpet file it produced a lot of
> integer multiples of the true pitch period, but it hardly produced
> any sub-multiple of the true pitch period, and despite the frequent
> detection of integer multiples of the pitch period, the audio quality
> improvement provided by the resulting pitch prefilter/postfilter is
> still quite dramatic.

I'm not sure I need them at this point -- assuming that you're now happy
with the resulting quality. Is that the case?

	Jean-Marc

> [mailto:[email-masked]] Sent: Wednesday, November 03,
> 2010 3:41 PM To: Raymond (Juin-Hwey) Chen Cc: [email-masked]';
> Jean-Marc Valin Subject: Re: [codec] A pitch filtering method to
> Can you tell me if you can still hear the artefact? Note that these
> use the default (trim 3) allocation.
> 
> Cheers,
> 
> Jean-Marc
> 
>> If you look at the spectrogram of the section between 3.5 and 3.9
>> seconds, you can see that there's actually two notes at the same
>> time. The previous note is fading out, while the new one is
>> beginning. During that time, the current pitch detector only finds
>> a small gain, so the the post-filter is effectively turned off. I
>> suspect that your pitch detector is still on during that time and
>> manages to reduce some of the LF noise. Can you confirm that? I
>> think it would be really helpful if you could send me the period
>> and gain for each 5 ms frame in that file. That way we can either 
>>> Since your instructions when sending me the previous code was to
>>> change trim_index from 3 to 5, we followed your instructions and
>>> changed that line 903 in celt.c. I just confirmed that again.
>>> 
>>> low-frequency disturbance is somewhat reduced with trim5 files,
>>> but they are still there in both 64 and 96 kbps versions.
>>> 
>>> For example, the last version of the 64 kbps pf file has three
>>> instances of audible "disturbance" at 3.5, 3.6, and 3.7 seconds,
>>> while the trim6 version of the 64 kbps pf file only has it at 3.6
>>> and 3.7 seconds. The one at 3.5 second is gone. The intensity of
>>> the remaining two seems to be reduced slightly as well. Same with
>>> the 96 kbps files, but you can still hear them in both 64 and 96
>>> kbps versions of the trim5 files.
>>> 
>>> In contrast, there is no audible mid- to low-frequency
>>> disturbance in the pitch-filtered 64 and 96 kbps output signals
>>> that we processed with your previous pre-release CELT version
>>> 0.9.0 with trim_index=5 and with my single-tap pitch
>>> prefilter/postfilter and my simple pitch estimator. They sound
>>> completely clean in that regard (mid- to low-frequency 
>>> disturbance).
>>> 
>>> Raymond
>>> 
>>> [mailto:[email-masked]] Sent: Wednesday, November
>>> 03, 2010 2:46 PM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc
>>> Valin; Gregory Maxwell; [email-masked]' Subject: Re: [codec] A
>>> I checked the files you used and they are almost certainly
>>> generated with a version of the encoder I specifically tuned for
>>> these files (i.e. trim_index=5). If you listen to the files I
>>> posted earlier today:
>>> 
>>> I'm sure you will find that they sound at least as good as what
>>> you had. Can you confirm?
>>> 
>>> Cheers,
>>> 
>>> Jean-Marc
>>> 
>>>>> There's been many changes since the version in that zip file,
>>>>> most of which not related to the post-filter. These changes
>>>>> have actually improved audio quality quite a bit, though I
>>>>> guess it's possible that it degraded this particular sample.
>>>> 
>>>>> Will listen tomorrow morning. Can you send the files you have
>>>>> with and without post-filter (please use [email-masked]
>>>>> for large files like these)?
>>>> 
>>>> They are the same files that I sent you two weeks ago. I have
>>>> sent them to your gmail account again as you requested.
>>>> 
>>>> Jean-Marc
>>>> 
>>>>> Raymond
>>>>> 
>>>>> [mailto:[email-masked]] Sent: Tuesday, November
>>>>> 02, 2010 9:11 PM To: Raymond (Juin-Hwey) Chen Cc: Jean-Marc
>>>>> Valin; Gregory Maxwell; [email-masked]' Subject: Re: [codec]
>>>>> A pitch filtering method to improve the mode 1 CELT codec
>>>>> 
>>>>> Hi Raymond,
>>>>> 
>>>>> 1) Different pitch estimator 2) Different choice of "global"
>>>>> gain 3) Unrelated changes to the CELT code (e.g. bit
>>>>> allocation) 4) Quantization of the pitch gain 5) The
>>>>> post-filter is now applied in the pre-emphasized domain 6)
>>>>> 3-tap gain
>>>>> 
>>>>> These are (roughly) ordered from most likely to least likely
>>>>> issue. I guess we'll have to experiment to find what the
>>>>> cause is. If you like, I can send you the output for a 1-tap
>>>>> predictor. Maybe you could also send me the exact pitch
>>>>> period and gain you use for each 5 ms frame so I can process
>>>>> using that.
>>>>> 
>>>>> Now that I think of it, if you say the nopf file also sounds
>>>>> worse than it did in the version you had, then it would have
>>>>> to be something unrelated to the post-filter. Could you tell
>>>>> me the exact version you had (did I give you the hash value
>>>>> with it)?
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Jean-Marc
>>>>> 
>>>>> 
>>>>>> Thanks for sharing the processed versions of the trumpet
>>>>>> signal. I did a listening comparison with the previous
>>>>>> processed versions I sent you that were obtained with my
>>>>>> preliminary single-tap pitch prefilter/postfilter and with
>>>>>> the pre-release version 0.9.0 CELT that you sent me
>>>>>> previously.
>>>>>> 
>>>>>> I was surprised to find that your latest output files
>>>>>> sounded noticeably worse to my ears than my previous files
>>>>>> at both 64 and 96 kb/s. I don't know whether that's due to
>>>>>> additional changes you made in the CELT codec itself, in
>>>>>> the pitch prefilter/postfilter, or both. There are many
>>>>>> instances of mid- to low-frequency disturbance (I don't
>>>>>> know how to describe it) in many places of your latest
>>>>>> output files (both pf and nopf versions), the most obvious
>>>>>> place being 3.3 to 3.9 seconds from the beginning of the
>>>>>> files at both bit-rates. This distortion is more pronounced
>>>>>> in the 64 kb/s version, but it is also quite audible to me
>>>>>> even in the 96 kb/s version. In contrast, the previous
>>>>>> processed versions I sent you were mostly free of such 
>>>>>> distortion. (The 64 kb/s output without pitch filtering has
>>>>>> a slight trace of this distortion, while the pitch-filtered
>>>>>> version and both filtered and unfiltered versions of 96
>>>>>> kb/s versions are free of this distortion.)
>>>>>> 
>>>>>> This got me somewhat concerned. Your 3-tap pitch filters
>>>>>> are meant to be an improvement over my original single-tap
>>>>>> filters. Could there be some implementation issues that
>>>>>> degraded the performance? Or did some recent tunings you
>>>>>> made in the CELT codec significantly degrade the coding
>>>>>> performance for this signal?
>>>>>> 
>>>>>> Raymond
>>>>>> 
>>>>>> [mailto:[email-masked]] Sent: Monday,
>>>>>> November 01, 2010 8:40 PM To: Raymond (Juin-Hwey) Chen Cc:
>>>>>> Gregory Maxwell; Jean-Marc Valin; [email-masked]' Subject:
>>>>>> Re: [codec] A pitch filtering method to improve the mode 1
>>>>>> CELT codec
>>>>>> 
>>>>>> Hi Raymond (and others),
>>>>>> 
>>>>>> Here's some trumpet samples at 64 kb/s and 96 kb/s stereo
>>>>>> for 5 ms frames (quality with 20 ms frames is obviously
>>>>>> better):
>>>>>> 
>>>>>> without postfilter: 
>>>>>>> Would you post your 8 kb/s coded speech as well so we
>>>>>>> can listen. I am curious what it sounds like.
>>>>>>> 
>>>>>>> Greg and Jean-Marc,
>>>>>>> 
>>>>>>> comparisons of CELT codec outputs without or with the
>>>>>>> pitch prefilter/postfilter, respectively, so we can hear
>>>>>>> what kind of results you are getting with your latest
>>>>>>> code version? I am especially curious to hear the solo
>>>>>>> trumpet signal processed by your latest version with
>>>>>>> 3-tap filters. I would like to compare it with the result
>>>>>>> I got with single-tap pitch filters. Thanks.
>>>>>>> 
>>>>>>> Raymond
>>>>>>> 
>>>>>>> [mailto:[email-masked]] Sent: Monday, November 01,
>>>>>>> 2010 12:48 PM To: Jean-Marc Valin; Raymond (Juin-Hwey)
>>>>>>> Chen Cc: [email-masked]'; Jean-Marc Valin Subject: RE:
>>>>>>> [codec] A pitch filtering method to improve the mode 1
>>>>>>> CELT codec
>>>>>>> 
>>>>>>>> The main issue that remains now is how to handle the
>>>>>>>> pitch contour for frame sizes above 5 ms. Right now,
>>>>>>>> the pitch period is forced constant through the whole
>>>>>>>> frame, which is likely suboptimal. Any suggestion on
>>>>>>>> how to best estimate and code the pitch contour? The
>>>>>>>> other thing that's still undecided is the range of
>>>>>>>> values we want to encode. In the current version, the
>>>>>>>> period can range from 30 samples (1500 Hz pitch) to 960
>>>>>>>> samples (50 Hz pitch). As for the gain, non-zero values
>>>>>>>> can range between 0.25 and 0.625 in increments of 
>>>>>>> I've been working with Jean-Marc on IRC on this but I
>>>>>>> thought it might be useful to bring some more of the
>>>>>>> discussion to the list. :)
>>>>>>> 
>>>>>>> I made some measurements of the code on one of the small
>>>>>>> test sets I used. The test set consists of 10 second
>>>>>>> segments randomly selected from about 30 hours of audio.
>>>>>>> It's a bit commercial-recording music heavy and speech
>>>>>>> light.
>>>>>>> 
>>>>>>> clamped to the range [0,0.6] before being used as the
>>>>>>> goal is not to completely whiten the spectrum...
>>>>>>> 
>>>>>>>> From listening I think the scaling of 0.7 isn't enough
>>>>>>>> in general and that 0.5 is probably better but there
>>>>>>>> are some samples that its quite helpful for... some
>>>>>>>> more encoder intelligence is probably justified.
>>>>>>> 
>>>>>>> On a more silly note the addition of pitch lets CELT
>>>>>>> achieve intelligible (but not good) wideband speech at
>>>>>>> the ludicrous rate of 2.5kbit/sec: 
>>>>>>> While I don't expect anyone to intentionally run the
>>>>>>> codec at such insanely low rates I think is helpful if
>>>>>>> the codec can maintain intelligibility down to very low
>>>>>>> rates in order to cope with extreme congestion.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
>> _______________________________________________ codec mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/codec
> 
> 
> 
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 3 Nov 2010 18:14:51 -0700

Hi Jean-Marc,

> resolution. Then again, that's just another thing to check experimentally.

Agreed.  The only way to find out for sure is by experiment.

> a case where we can dynamically allocate mode bits.

Again, the best way to find out is by experiment.  I tried the flute file I sent you, where the last two notes have pitch values of about 1200 Hz and 1600 Hz, respectively.  These are much larger than the widths of your lower-frequency bands, I suppose.  I tried to code this file using the previous CELT 0.9.0 you sent me with or without pitch prefilter/postfilter.  I then use Adobe Audition to listen to them and examine the spectra.  The output signal with pitch prefilter/postfilter sounded clearly better than CELT without pitch filtering for those two last notes, especially the 1200 Hz note.  Spectral plots also showed that with the pitch prefilter/postfilter, the noise floor between pitch harmonics is much lower than without the pitch filter.  I can send you the spectral plots and the output audio files if you wish, but you can obtain these yourself by doing the same experiment with that flute file.

I think your argument works if the signal intensity in those inter-harmonic frequency bands is lower than the noise masking threshold, and your accurate band energy quantization (to within +- 1.5 dB) keeps the quantized version of the signal below the noise masking threshold, then in that case pitch prefilter/postfilter won't help.  However, with a very high pitch frequency, there is less masking between harmonics (i.e. the masking threshold is lower between harmonics compared with a low pitch frequency), so chances are the signal intensity in many frequency bands between pitch harmonics will be above the noise masking threshold.  In this case, even if you are able to keep the band energy to within +- 1.5 dB of the true value, the shape quantization will still give some audible noise when the bit-rate is low, because that whole band is above the noise masking threshold.  In this case, using pitch prefilter/postfilter to attenuate the noise floor between pitch harmonics can still reduce the perceived level of coding noise.  (Of course, this attenuation of inter-harmonic noise floor comes at the cost of increased noise at pitch harmonic frequencies, which hopefully is masked well by the signal itself at pitch harmonics.)

Raymond

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 3 Nov 2010 18:40:11 -0700

Hi Jean-Marc,

> as I'm working on the pf. I've just tweaking all areas of the codec at

I was saying that if the MF to LF disturbance is not caused by pitch period halving (since it is already in nopf files), then I didn't understand why fixing pitch period halving will help remove the disturbance.  Now with your explanation, I understand that when you previously had pitch period halving, you effectively turned off the pitch prefilter/postfilter, so you let the disturbance in the nopf files "leak through" to the pf files for those frames.  After you fixed the pitch period halving, this "leak through" no longer happens, and thus the pf files sound clean.  It all makes sense now.  Thanks for the explanation.

> with the resulting quality. Is that the case?

Yes, although the goal was not to "make me happy", but rather for me to help you verify performance and hopefully discover areas that can be improved upon before we finalize the bit-stream.

This brings me to the next topic.  You were proposing using a single set of three pitch filter taps and just vary the overall gain of the three taps. That set of filter taps gives decreasing degree of comb filtering with increasing frequency. I expressed some concerns that this fixed set of filter taps may not be optimal for some audio signals.  You said you haven't been able to find any signal that break it (or something like that, I forgot your exact wording).  Although it is true that many audio signals with a harmonic structure have well-defined harmonic structure only for lower frequencies, there are also some audio signals where the harmonic structure is clearly visible all the way to 20 kHz.  The harpsichord signal is one such example.  Have you tried that signal or any other signal with clearly visible harmonic structure to very high frequencies.  I know the critical bandwidth is much wider at high frequencies, but I just wonder for those signals whether using one-tap or three-tap pitch filters with a constant or nearly constant level of comb filtering all the way up to 20 kHz will improve audio quality compared with using your fixed set of three filter taps.

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 23:13:03 -0400

Hi Raymond,

On 10-11-03 09:14 PM, Raymond (Juin-Hwey) Chen wrote:
> Again, the best way to find out is by experiment.  I tried the flute
> file I sent you, where the last two notes have pitch values of about
> 1200 Hz and 1600 Hz, respectively.  These are much larger than the
> widths of your lower-frequency bands, I suppose.  I tried to code
> this file using the previous CELT 0.9.0 you sent me with or without
> pitch prefilter/postfilter.  I then use Adobe Audition to listen to
> them and examine the spectra.  The output signal with pitch
> prefilter/postfilter sounded clearly better than CELT without pitch
> filtering for those two last notes, especially the 1200 Hz note.
> Spectral plots also showed that with the pitch prefilter/postfilter,
> the noise floor between pitch harmonics is much lower than without
> the pitch filter.  I can send you the spectral plots and the output
> audio files if you wish, but you can obtain these yourself by doing
> the same experiment with that flute file.

OK, I'll look into this. I'm still skeptical, but willing to be proven
wrong.

> I think your argument works if the signal intensity in those
> inter-harmonic frequency bands is lower than the noise masking
> threshold, and your accurate band energy quantization (to within +-
> 1.5 dB) keeps the quantized version of the signal below the noise
> masking threshold, then in that case pitch prefilter/postfilter won't
> help.

Well, the idea is that the quantized signal should end up with about the
same energy as the unquantized signal. So if it's above the masking
threshold, it means that the unquantized signal also had unmasked
"noise". At least that's the theory. If it only comes down to the +/-1.5
dB, then it's easy to just bias the quantizer for +0/-3.

> (Of
> course, this attenuation of inter-harmonic noise floor comes at the
> cost of increased noise at pitch harmonic frequencies, which
> hopefully is masked well by the signal itself at pitch harmonics.)

That's also something worth checking. I think we'll need at least about
20 dB SNR in bands that have tones. Fortunately, if there's only a
handful of tones, we can use dynamic allocation to achieve that.

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 03 Nov 2010 23:27:14 -0400

Hi Raymond,

On 10-11-03 09:40 PM, Raymond (Juin-Hwey) Chen wrote:
>> I'm not sure I need them at this point -- assuming that you're now
>> happy with the resulting quality. Is that the case?
> 
> Yes, although the goal was not to "make me happy", but rather for me
> to help you verify performance and hopefully discover areas that can
> be improved upon before we finalize the bit-stream.

Of course. Just saying that if you're happy with the quality, I'm also
likely to be because so far you seem to be better than I am at spotting
defects. I'm sure there's still other areas that could be improved.

> This brings me to the next topic.  You were proposing using a single
> set of three pitch filter taps and just vary the overall gain of the
> three taps. That set of filter taps gives decreasing degree of comb
> filtering with increasing frequency. I expressed some concerns that
> this fixed set of filter taps may not be optimal for some audio
> signals.  You said you haven't been able to find any signal that
> break it (or something like that, I forgot your exact wording).
> Although it is true that many audio signals with a harmonic structure
> have well-defined harmonic structure only for lower frequencies,
> there are also some audio signals where the harmonic structure is
> clearly visible all the way to 20 kHz.  The harpsichord signal is one
> such example.  Have you tried that signal or any other signal with
> clearly visible harmonic structure to very high frequencies. 

Actually, I did have a look at the harpsichord and although it has tones
that go all the way up to 20 kHz (and above), the problem is that these
tones do not follow a purely harmonic pattern. They deviate sufficiently
from k*f_0 that the comb filter becomes completely misaligned. IIRC I
also observed the same pattern with higher harmonics of the flute.

But the main reason I don't think a 1-tap predictor is a good idea is
that we have actually observed it causing problems by "creating"
high-frequency harmonicity where there was none. So far we haven't
observed that with the current 3-tap. Given what I currently know, if I
was going to add another filter option, rather than a 1-tap filter I'd
add a 5-tap filter with even faster roll-off. I'm just not sure the
potential improvement it's worth the extra complexity.

> I know
> the critical bandwidth is much wider at high frequencies, but I just
> wonder for those signals whether using one-tap or three-tap pitch
> filters with a constant or nearly constant level of comb filtering
> all the way up to 20 kHz will improve audio quality compared with
> using your fixed set of three filter taps.

Again worth checking, but I wouldn't be surprised if the 1-tap filter
even hurt the harpsichord by amplifying inter-harmonic noise when the
comb-filter isn't aligned with the (not quite) harmonics.

	Jean-Marc

>> Can you tell me if you can still hear the artefact? Note that
>> these use the default (trim 3) allocation.
>> 
>> Cheers,
>> 
>> Jean-Marc
>> 
>>> If you look at the spectrogram of the section between 3.5 and
>>> 3.9 seconds, you can see that there's actually two notes at the
>>> same time. The previous note is fading out, while the new one is 
>>> beginning. During that time, the current pitch detector only
>>> finds a small gain, so the the post-filter is effectively turned
>>> off. I suspect that your pitch detector is still on during that
>>> time and manages to reduce some of the LF noise. Can you confirm
>>> that? I think it would be really helpful if you could send me the
>>> period and gain for each 5 ms frame in that file. That way we can
>>> either confirm or rule out the pitch estimator as the remaining
>>> effect.
>>> 
>>> Jean-Marc
>>> 
>>> 
>>>> Since your instructions when sending me the previous code was
>>>> to change trim_index from 3 to 5, we followed your instructions
>>>> and changed that line 903 in celt.c. I just confirmed that
>>>> again.
>>>> 
>>>> I listened to these two new files you sent with trim_index=5
>>>> and compared them with the ones you posted Monday night. The
>>>> mid- to low-frequency disturbance is somewhat reduced with
>>>> trim5 files, but they are still there in both 64 and 96 kbps
>>>> versions.
>>>> 
>>>> instances of audible "disturbance" at 3.5, 3.6, and 3.7
>>>> seconds, while the trim6 version of the 64 kbps pf file only
>>>> has it at 3.6 and 3.7 seconds. The one at 3.5 second is gone.
>>>> The intensity of the remaining two seems to be reduced slightly
>>>> as well. Same with the 96 kbps files, but you can still hear
>>>> them in both 64 and 96 kbps versions of the trim5 files.
>>>> 
>>>> disturbance in the pitch-filtered 64 and 96 kbps output
>>>> signals that we processed with your previous pre-release CELT
>>>> version 0.9.0 with trim_index=5 and with my single-tap pitch 
>>>> generated with a version of the encoder I specifically tuned
>>>> for these files (i.e. trim_index=5). If you listen to the files
>>>> I posted earlier today:
>>>> 
>>>> I'm sure you will find that they sound at least as good as
>>>> what you had. Can you confirm?
>>>> 
>>>> Cheers,
>>>> 
>>>> Jean-Marc
>>>> 
>>>>>> There's been many changes since the version in that zip
>>>>>> file, most of which not related to the post-filter. These
>>>>>> changes have actually improved audio quality quite a bit,
>>>>>> though I guess it's possible that it degraded this
>>>>>> particular sample.
>>>>> 
>>>>>> Will listen tomorrow morning. Can you send the files you
>>>>>> have with and without post-filter (please use
>>>>>> [email-masked] for large files like these)?
>>>>> 
>>>>> They are the same files that I sent you two weeks ago. I
>>>>> have sent them to your gmail account again as you requested.
>>>>> 
>>>>> Jean-Marc
>>>>> 
>>>>>> Raymond
>>>>>> 
>>>>>> [mailto:[email-masked]] Sent: Tuesday,
>>>>>> November 02, 2010 9:11 PM To: Raymond (Juin-Hwey) Chen Cc:
>>>>>> Jean-Marc Valin; Gregory Maxwell; [email-masked]' Subject:
>>>>>> Off the top of my head, here are the differences between
>>>>>> the conditions of your experiment and the ones in the code
>>>>>> I have:
>>>>>> 
>>>>>> 1) Different pitch estimator 2) Different choice of
>>>>>> "global" gain 3) Unrelated changes to the CELT code (e.g.
>>>>>> bit allocation) 4) Quantization of the pitch gain 5) The 
>>>>>> These are (roughly) ordered from most likely to least
>>>>>> likely issue. I guess we'll have to experiment to find what
>>>>>> the cause is. If you like, I can send you the output for a
>>>>>> 1-tap predictor. Maybe you could also send me the exact
>>>>>> pitch period and gain you use for each 5 ms frame so I can
>>>>>> process using that.
>>>>>> 
>>>>>> Now that I think of it, if you say the nopf file also
>>>>>> sounds worse than it did in the version you had, then it
>>>>>> would have to be something unrelated to the post-filter.
>>>>>> Could you tell me the exact version you had (did I give you
>>>>>> the hash value with it)?
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Jean-Marc
>>>>>> 
>>>>>> 
>>>>>>> preliminary single-tap pitch prefilter/postfilter and
>>>>>>> with the pre-release version 0.9.0 CELT that you sent me 
>>>>>>> previously.
>>>>>>> 
>>>>>>> sounded noticeably worse to my ears than my previous
>>>>>>> files at both 64 and 96 kb/s. I don't know whether that's
>>>>>>> due to additional changes you made in the CELT codec
>>>>>>> itself, in the pitch prefilter/postfilter, or both. There
>>>>>>> are many instances of mid- to low-frequency disturbance
>>>>>>> (I don't know how to describe it) in many places of your
>>>>>>> latest output files (both pf and nopf versions), the most
>>>>>>> obvious place being 3.3 to 3.9 seconds from the beginning
>>>>>>> of the files at both bit-rates. This distortion is more
>>>>>>> pronounced in the 64 kb/s version, but it is also quite
>>>>>>> audible to me even in the 96 kb/s version. In contrast,
>>>>>>> the previous processed versions I sent you were mostly
>>>>>>> free of such distortion. (The 64 kb/s output without
>>>>>>> pitch filtering has a slight trace of this distortion,
>>>>>>> while the pitch-filtered version and both filtered and
>>>>>>> unfiltered versions of 96 kb/s versions are free of this
>>>>>>> distortion.)
>>>>>>> 
>>>>>>> are meant to be an improvement over my original
>>>>>>> single-tap filters. Could there be some implementation
>>>>>>> issues that degraded the performance? Or did some recent
>>>>>>> tunings you made in the CELT codec significantly degrade
>>>>>>> November 01, 2010 8:40 PM To: Raymond (Juin-Hwey) Chen
>>>>>>> Here's some trumpet samples at 64 kb/s and 96 kb/s
>>>>>>> stereo for 5 ms frames (quality with 20 ms frames is
>>>>>>> obviously better):
>>>>>>> 
>>>>>>> without postfilter: 
>>>>>>>> pitch prefilter/postfilter, respectively, so we can
>>>>>>>> hear what kind of results you are getting with your
>>>>>>>> latest code version? I am especially curious to hear
>>>>>>>> the solo trumpet signal processed by your latest
>>>>>>>> version with 3-tap filters. I would like to compare it
>>>>>>>> [mailto:[email-masked]] Sent: Monday, November
>>>>>>>> 01, 2010 12:48 PM To: Jean-Marc Valin; Raymond
>>>>>>>> (Juin-Hwey) Chen Cc: [email-masked]'; Jean-Marc Valin
>>>>>>>>> the pitch period is forced constant through the
>>>>>>>>> whole frame, which is likely suboptimal. Any
>>>>>>>>> suggestion on how to best estimate and code the pitch
>>>>>>>>> contour? The other thing that's still undecided is
>>>>>>>>> the range of values we want to encode. In the current
>>>>>>>>> version, the period can range from 30 samples (1500
>>>>>>>>> Hz pitch) to 960 samples (50 Hz pitch). As for the
>>>>>>>>> gain, non-zero values can range between 0.25 and
>>>>>>>>> 0.625 in increments of 0.125. Any opinion on these?
>>>>>>>> 
>>>>>>>> I made some measurements of the code on one of the
>>>>>>>> small test sets I used. The test set consists of 10
>>>>>>>> second segments randomly selected from about 30 hours
>>>>>>>> of audio. It's a bit commercial-recording music heavy
>>>>>>>> and speech light.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> In the current encoder these gains are scaled by 0.7
>>>>>>>> and clamped to the range [0,0.6] before being used as
>>>>>>>> the goal is not to completely whiten the spectrum...
>>>>>>>> 
>>>>>>>>> From listening I think the scaling of 0.7 isn't
>>>>>>>>> enough in general and that 0.5 is probably better but
>>>>>>>>> there are some samples that its quite helpful for...
>>>>>>>>> some more encoder intelligence is probably
>>>>>>>>> justified.
>>>>>>>> 
>>>>>>>> the codec can maintain intelligibility down to very
>>>>>>>> low rates in order to cope with extreme congestion.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> _______________________________________________ codec mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/codec
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 4 Nov 2010 17:46:35 -0700

Hi Jean-Marc,

> dB, then it's easy to just bias the quantizer for +0/-3.

Note that I used the term "signal intensity" and not "noise intensity" in the first line of my paragraph above.  You could potentially have some signal components in the inter-harmonic spectral valley regions that are above the noise masking threshold level.  When they are preserved with high fidelity at high bit-rates, you don't perceive them as noise.  However, if the CELT encoding bit-rate is very low, then even if you can accurately preserve the energy level to within +- 1.5 dB, you are not likely to be able to preserve the "shape" accurately, then the quantized version of these signal components will be perceived as noise or distortion, and they are audible because they are above the noise masking threshold.  In this case, applying the pitch prefilter/postfilter will attenuate such noise in the inter-harmonic frequency bands and make the noise less audible.  This will be true even if you have critical bandwidths much smaller than the pitch frequency and you can quantize the band energy accurately.  A direct proof is the experiment I mentioned before with the last two notes of the flute file at 1200 and 1600 Hz pitch frequencies.  I code it using 64 kb/s CELT having a 5 ms frame size, with or without the pitch prefilter/postfilter.  You can try that yourself.  The use of pitch prefilter/postfilter improved the audio quality very significantly, even though the pitch frequencies are much higher than critical bandwidths at lower frequencies and even though you have accurate quantization of the band energy.

Raymond

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 4 Nov 2010 19:06:35 -0700

Hi Jean-Marc,

> also observed the same pattern with higher harmonics of the flute.

How about clarinet?  Have you looked at that signal?  See my attached PDF file.  It shows clear harmonic structure all the way to 20 kHz, and I checked and verified that the peak frequencies are all harmonically related.  Although I didn't analyze any, I suppose some synthetic audio signals from a synthesizer instrument may also have artificially generated harmonic structure all the way to 20 kHz.  My point is just that there exist real-world audio signals with harmonic structure in the entire 20 kHz passband, and my concern is that using a fixed 3-tap pitch filter like the one you are using may unnecessarily limits the achievable performance for these types of signals.

> comb-filter isn't aligned with the (not quite) harmonics.

I agree that always using a single-tap filter is not a good idea, and I have said that previously; that's not what I am advocating.  What I have been advocating since last week is that instead of always using a fixed set of 3 taps and only changing the overall gain for the taps, perhaps we should investigate having a few candidate sets of 3 filter taps having different effective "cut-off" frequencies where the degree of comb filtering reduces to nearly nothing, and one of them may have a nearly flat envelope for nearly constant degree of comb filtering.  Some of them may even have a cut-off frequency lower than the one you are using, and they may be better suited for some audio signals where the harmonic structure is only well-defined at relatively low frequencies (< 5 kHz, for example).

To search though such a codebook of candidate 3-tap filters, we can simply perform the equivalent of pitch prediction error filtering using each of the "full-strength" candidate 3-tap pitch prefilters and pick the one that minimizes the prediction residual energy. This approach should automatically choose the candidate 3-tap filter with a frequency response that best matches the input signal.  Direct implementation of this is costly, but I described an efficient method in my doctoral thesis. (The method is probably well-known by now or may have been independently developed by others.)

I would admit that spending one or two bits on specifying the selected candidate 3-tap filter will increase the bit-rate, the 3-tap codebook search will increase the encoder complexity, and the potential audio quality improvement may not be worth the extra bit-rate and complexity.  However, I think at least we need to try it out and see what the trade off is and then make a decision, rather than just directly go for a fixed "one-size-fit-all" 3-tap filter, which may give too little comb filtering for some audio signals and too much comb filtering for some other audio signals and limit the achievable performance for the pitch prefilter/postfilter approach.  My team member Daniele Giacobello and I can work on this, although we are not sure we can have the result by the IETF 79 codec WG meeting next Tuesday.

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 00:16:03 -0400

Hi Raymond,

FYI, I have extended the pitch to 3 kHz (16 samples), so that should
settle that issue.

On 10-11-04 10:06 PM, Raymond (Juin-Hwey) Chen wrote:
> How about clarinet?  Have you looked at that signal?  See my attached
> PDF file.  It shows clear harmonic structure all the way to 20 kHz,
> and I checked and verified that the peak frequencies are all
> harmonically related.  Although I didn't analyze any, I suppose some
> synthetic audio signals from a synthesizer instrument may also have
> artificially generated harmonic structure all the way to 20 kHz.  My
> point is just that there exist real-world audio signals with harmonic
> structure in the entire 20 kHz passband, and my concern is that using
> a fixed 3-tap pitch filter like the one you are using may
> unnecessarily limits the achievable performance for these types of
> signals.

Did you check that a one-tap predictor actually improves quality? My
concern with 1-tap is that it's likely to be of very limited use
(because the encoder will have to be very careful when using it) and it
costs bits to encode. If we get a reasonable confidence that it can help
in practice that's one thing, but otherwise I'm not sure it's worth
spending X bits in every frame in case it's useful some day. Anyway, I
will test the clarinet tomorrow to see what 1-tap does to it.

>> Again worth checking, but I wouldn't be surprised if the 1-tap
>> filter even hurt the harpsichord by amplifying inter-harmonic noise
>> when the comb-filter isn't aligned with the (not quite) harmonics.
> 
> I agree that always using a single-tap filter is not a good idea, and
> I have said that previously; that's not what I am advocating.  What I
> have been advocating since last week is that instead of always using
> a fixed set of 3 taps and only changing the overall gain for the
> taps, perhaps we should investigate having a few candidate sets of 3
> filter taps having different effective "cut-off" frequencies where
> the degree of comb filtering reduces to nearly nothing, and one of
> them may have a nearly flat envelope for nearly constant degree of
> comb filtering.  Some of them may even have a cut-off frequency lower
> than the one you are using, and they may be better suited for some
> audio signals where the harmonic structure is only well-defined at
> relatively low frequencies (< 5 kHz, for example).

Yes, I understand that you're proposing to have selectable taps. But
before doing that we need to make sure it's worth the bits required to
code the information. Regarding cut-off, the current 3-tap filter is
basically the lowest cutoff I could do with 3 taps. To get lower than
that requires 5 taps and I'm not sure it's worth the trouble.

> To search though such a codebook of candidate 3-tap filters, we can
> simply perform the equivalent of pitch prediction error filtering
> using each of the "full-strength" candidate 3-tap pitch prefilters
> and pick the one that minimizes the prediction residual energy. This
> approach should automatically choose the candidate 3-tap filter with
> a frequency response that best matches the input signal.  Direct
> implementation of this is costly, but I described an efficient method
> in my doctoral thesis. (The method is probably well-known by now or
> may have been independently developed by others.)

I assume that's the algorithm you described in your LC-CELP paper, in
which case I'm indeed familiar with it. However, I'm not sure it can be
applied directly here because we aren't really after the optimal
predictor. Rather, we want to check for high-frequency harmonic contents
that matches the peaks in the comb filter. That's very different because
we need to make sure that low-frequency effects are not taken into account.

> I would admit that spending one or two bits on specifying the
> selected candidate 3-tap filter will increase the bit-rate, the 3-tap
> codebook search will increase the encoder complexity, and the
> potential audio quality improvement may not be worth the extra
> bit-rate and complexity.  However, I think at least we need to try it
> out and see what the trade off is and then make a decision, rather
> than just directly go for a fixed "one-size-fit-all" 3-tap filter,
> which may give too little comb filtering for some audio signals and
> too much comb filtering for some other audio signals and limit the
> achievable performance for the pitch prefilter/postfilter approach.
> My team member Daniele Giacobello and I can work on this, although we
> are not sure we can have the result by the IETF 79 codec WG meeting
> next Tuesday.

Well I'm definitely not against experimenting and if you get results
that show that a codebook is useful then we can still add that. I'm not
really concerned with the complexity because an encoder is free to
always choose the same filter.

In any case, at this point I think that the main place on the CELT side
that needs attention is the bit allocation. I suspect that the current
tuning is still sub-optimal and changes in the allocation can sometimes
have significant effects on the quality. What's also important to know
is that most of the allocation information is actually shared between
the encoder and decoder so it is effectively "part of the bit-stream"
and thus cannot be changed later on.

Oh, and for those who would like to try out the post-filter, it is now
in the default "master" branch in the CELT repository (haven't synced
Opus yet). It is disabled by default for "IPR reasons", but can be
enabled using ENABLE_POSTFILTER.

Cheers,

	Jean-Marc

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Fri, 5 Nov 2010 08:56:39 -0500

Raymond ... Nice to talk with you again.

Jean-Marc ... I don't know you, but feel compelled to type this email
...

Michael Ramalho

<from Jean-Marc>
In any case, from a theoretical perspective, I think detecting a pitch
period smaller than the true pitch period is more detrimental than
detecting an integer multiple of the true pitch period.  In the latter
case, you don't get as much noise attenuation between pitch harmonics
compared with using the true pitch period, but at least the pitch
harmonics are preserved.  In the former case (detected pitch period <<
true pitch period), you can attenuate some of the pitch harmonics, so
this should be avoided as much as possible.

<From Raymond>
My point is just that there exist real-world audio signals with harmonic
structure in the entire 20 kHz passband, and my concern is that using a
fixed 3-tap pitch filter like the one you are using may unnecessarily
limits the achievable performance for these types of signals.

<From Jean-Marc>

<Ramalho>
Come on guys - unless your source is perfectly periodic, you EXPECT that
high-frequency peaks in the Fourier spectrum (note I don't call them
harmonics, per se) WILL deviate from where you EXPECT them to be under a
perfectly periodic spectrum.

Most non-synthesized "harmonic" signals are quasi-harmonic signals.

Quasi-harmonic signals - such as voice (air puffs through your larynx),
reed-based instruments such as clarinets, oboes, ... (reed-based
vibrations), brass instruments (quasi-periodic puffs of air through
mouthpiece) and even string-based instruments such as your harpsichords
will induce slight differences from PERFECT periodicity.

One can view this non-periodic (quasi-harmonic) process as a kind of
phase-modulation FROM a perfectly periodic source .... and thus you can
analytically model it using the following time-based modulation
consideration:

Eq. 1: tau = t + beta(t) .... where beta(t) is the function you need to
"warp your non-periodic source" (in time base t) to a perfectly periodic
source (in time base tau).

The upshot of this is that if you model it this way ... you EXCPECT that
the "harmonic peaks" will widen AND flatten out with increasing
frequency. Indeed, because Eq. 1 induces a NON-LINEAR time modulation
... you don't even expect the peaks to have similar spectral shapes
around their expected frequency (k*f_0). Linear modulation, like AM, has
similarly-shaped harmonic structure. In fact, for some beta(t) the
energy at k*f_0 is ZERO (whenever the Bessel function for that harmonic
and modulation index is zero). Wow, ZERO energy at a "harmonic"! Most
engineers don't expect that!

This is all explained in my Ph.D. available at
http://ramalho.webhop.info .

As a closing (and extreme) example, the attached is a spectrogram of me
holding a "ah" for a relatively long period of time (256 ms ... but
windowed ... shown is effectively ~128 ms of time). I am a singer and
held the pitch as stead as I could (no vibrato). I found the function
beta(t) to convert my quasi-periodic pitch to a perfectly periodic
pitch.

The blue and the red plots in the attached graph represent the SAME
signal energy. Note that the energy from the "high-order harmonics"
completely overlap! ... while the lower-order harmonics are where you
"expect" them to be!

Bottom line is that comb filters are effective for the "low order
harmonics" of a sufficiently periodic signal. However, they simply
CANNOT capture (or eliminate, depending on what you are doing) higher
order "harmonic components" because the energy in them is not where YOU
"expect" them to be. When observing the high-order harmonics some start
"blaming the pitch estimators" ... but that isn't (necessarily) where
the problem is ... the problem is likely in the assumption on where the
"higher harmonic energy" should be!

Please don't use synthetic signals to test the utility of your comb
filter. Please don't blame pitch estimators for them not capturing (or
rejecting) high harmonic energy YOU think should be there. And please
don't blame individual instruments (including humans) for not having "a
perfectly periodic" structure when they are excited with a
not-perfectly-periodic stimuli (or, in the case of the harpsichord,
non-perfectly ridged end-conditions for the strings).

No one said that signal processing should be easy ;-).

Regards,

Michael Ramalho

Raymond: I wish I had a day job to work with you on this ... but I don't
at present.

********************************************

I was just about ready to send you the pitch periods and pitch taps when
you told me you didn't need them because you found the problem (pitch
halving).  I can still send them to you if it can help you.  The pitch
estimator I used to generate the previous files I sent you was a very
simple one and for this trumpet file it produced a lot of integer
multiples of the true pitch period, but it hardly produced any
sub-multiple of the true pitch period, and despite the frequent
detection of integer multiples of the pitch period, the audio quality
improvement provided by the resulting pitch prefilter/postfilter is
still quite dramatic.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Wednesday, November 03, 2010 3:41 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT
codec

Hi Raymond,

Can you tell me if you can still hear the artefact? Note that these use
the
default (trim 3) allocation.

Cheers,

        Jean-Marc

> If you look at the spectrogram of the section between 3.5 and 3.9
seconds,
> period and gain for each 5 ms frame in that file. That way we can
either
>> Since your instructions when sending me the previous code was to
change
>> For example, the last version of the 64 kbps pf file has three
instances
>> [mailto:[email-masked]] Sent: Wednesday, November 03,
2010
>> Maxwell; [email-masked]' Subject: Re: [codec] A pitch filtering
method

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 10:43:54 -0400

Hi Michael,

> perfectly periodic spectrum.

There's actually two types of deviations and I'm not sure we're actually 
talking about the same here. As far as I understand from your email (please 
correct me if I'm wrong), the effect you are talking about is a 
phase/frequency modulation effect that occurs in voice (and possibly some 
instruments -- I don't know) such that the exact pitch changes slightly 
over time.

On the other hand, what Raymond and I were talking about was Inharmonicity 
(http://en.wikipedia.org/wiki/Inharmonicity), where partials are no longer 
integer multiples of the fundamental frequency -- even when looking at the 
signal for a short duration. For example, if you look at the harpsichord 
spectrum I attached, you see that the frequency response of a single-tap 
comb-filter follows the harmonics of the harpsichord very well up to about 
10-12 kHz. From that point, they start to deviate and at 15 kHz, they have 
sufficiently deviated to be in between the peaks of the comb filter. I 
think each instrument behaves differently, but deviations are always 
towards frequencies that are higher than what perfect harmonicity would 
predict.

> will induce slight differences from PERFECT periodicity.

Someone please correct me if I'm wrong, but my understanding was that voice 
was actually purely harmonic because unlike reed-based instruments where 
the frequency of the partials is determined by the vibrating modes of the 
column of air, the partials in voice are simply harmonics of regular 
glottal impulses.

> source (in time base tau).

Now, this is why I think we're talking about two different things. The 
actual variations in time is something we're already modelling when using 5 
ms frames for pitch estimation.

> completely overlap! ... while the lower-order harmonics are where you

 From your example, I see that the harmonics in blue are actually "spread" 
by the modulation, rather than shifted up like they are in the Harpsichord 
spectrogram. I think that clearly demonstrates modulation effects vs 
inharmonicity.

> non-perfectly ridged end-conditions for the strings).

Well, considering I'm better at debugging pitch estimation code than human 
DNA, I think I'll stick to the former :-)

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 14:38:54 -0400

Hi Raymond,

I just uploaded some clarinet files to compare the current 3-tap predictor 
with a 1-tap predictor:

http://jmvalin.ca/misc_stuff/refcla64_1tap.wav
http://jmvalin.ca/misc_stuff/refcla96_1tap.wav
http://jmvalin.ca/misc_stuff/refcla128_1tap.wav
http://jmvalin.ca/misc_stuff/refcla64_3tap.wav
http://jmvalin.ca/misc_stuff/refcla96_3tap.wav
http://jmvalin.ca/misc_stuff/refcla128_3tap.wav

I think the names are self-explanatory. On these clarinet files, I have 
failed to ABX 1-tap vs 3-tap. Maybe you'll have more luck? Also, one thing 
I've just realised is that even the 3-tap does provide some inter-harmonic 
noise attenuation at high frequencies. It just has less than at low 
frequencies. See this frequency response I posted earlier: 
http://jmvalin.ca/misc_stuff/postfilter.png

Cheers,

	Jean-Marc

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Fri, 5 Nov 2010 14:04:27 -0500

Jean-Marc,

In-line below with "MAR:".

Michael

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Friday, November 05, 2010 10:44 AM
To: Michael Ramalho (mramalho)
Cc: Raymond (Juin-Hwey) Chen; [email-masked]; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT
codec

Hi Michael,

On 10-11-05 09:56 AM, Michael Ramalho (mramalho) wrote:
> <Ramalho>
> Come on guys - unless your source is perfectly periodic, you EXPECT 
> that high-frequency peaks in the Fourier spectrum (note I don't call 
> them harmonics, per se) WILL deviate from where you EXPECT them to be 
> under a perfectly periodic spectrum.

There's actually two types of deviations and I'm not sure we're actually
talking about the same here. As far as I understand from your email
(please correct me if I'm wrong), the effect you are talking about is a
phase/frequency modulation effect that occurs in voice (and possibly
some instruments -- I don't know) such that the exact pitch changes
slightly over time.

MAR: What I am referring to is the TIME domain effect of each "periodic
event component" (a singular puff of air in the vocal tract, an
open/close event of a reed) being not being strictly periodic with a
"train of such events" having average period To (i.e., average frequency
Fo) around it.

MAR: Changes from perfect periodicity on the order of 1/2% resulted in
the spectrums I attached in my earlier email. Thus a very small
deviation from perfect periodicity makes the higher-order harmonics
change from where many expect them.

MAR: The effect can be modeled as a time domain modulation (to be
precise, a phase modulation effect).

On the other hand, what Raymond and I were talking about was
Inharmonicity (http://en.wikipedia.org/wiki/Inharmonicity), where
partials are no longer integer multiples of the fundamental frequency --
even when looking at the signal for a short duration. For example, if
you look at the harpsichord spectrum I attached, you see that the
frequency response of a single-tap comb-filter follows the harmonics of
the harpsichord very well up to about
10-12 kHz. From that point, they start to deviate and at 15 kHz, they
have sufficiently deviated to be in between the peaks of the comb
filter. I think each instrument behaves differently, but deviations are
always towards frequencies that are higher than what perfect harmonicity
would predict.

MAR: "... deviations are always towards frequencies that are higher than
what perfect harmonicity would predict" .. perhaps your pitch estimator
is "always" a little on the low side and the error catches up with you
on the high harmonics (I have no clue here).

MAR: Each instrument has "a slightly different aperiodicity" ... thus
the higher order harmonics are all modulated differently. In essence,
one cause of "inharmonicity" is the extent to which the instrument
exhibits non-periodicity in its fundamental.

MAR: Thus, the high-order harmonics DO NOT APPEAR where you expect them
to be owing to the non-linear modulation.

MAR: In LINEAR modulation, you expect the same "spectral width" of each
harmonic to remain the same with increasing harmonic number and that
EACH harmonic would be perfectly centered on integer multiples of the
fundamental. When you have slightly aperiodic events (< 1/2% different
inter-event periods) you lose the precise harmonic relationship with
increasing frequency ... it is only a matter of harmonic number when
this relationship is lost.

> Quasi-harmonic signals - such as voice (air puffs through your 
> larynx), reed-based instruments such as clarinets, oboes, ... 
> (reed-based vibrations), brass instruments (quasi-periodic puffs of 
> air through
> mouthpiece) and even string-based instruments such as your 
> harpsichords will induce slight differences from PERFECT periodicity.

Someone please correct me if I'm wrong, but my understanding was that
voice was actually purely harmonic because unlike reed-based instruments
where the frequency of the partials is determined by the vibrating modes
of the column of air, the partials in voice are simply harmonics of
regular glottal impulses.

MAR: The vocal cords do not exhibit perfect periodicity. In fact, some
vocal cord diagnosis (e.g., vocal polyps) can be inferred by a raspy,
non-periodic opening of the cords.

> phase-modulation FROM a perfectly periodic source .... and thus you 
> can analytically model it using the following time-based modulation
> consideration:
>
> Eq. 1: tau = t + beta(t) .... where beta(t) is the function you need 
> to "warp your non-periodic source" (in time base t) to a perfectly 
> periodic source (in time base tau).

Now, this is why I think we're talking about two different things. The
actual variations in time is something we're already modelling when
using 5 ms frames for pitch estimation.

> As a closing (and extreme) example, the attached is a spectrogram of 
> me holding a "ah" for a relatively long period of time (256 ms ... but

> completely overlap! ... while the lower-order harmonics are where you 

 From your example, I see that the harmonics in blue are actually
"spread" 
by the modulation, rather than shifted up like they are in the
Harpsichord spectrogram. I think that clearly demonstrates modulation
effects vs inharmonicity.

MAR: OK ... you have confused me here. Energy MUST be preserved between
the red and the blue (the sum of the square of the original STFT
points). If a "harmonic component" is spread via modulation ... the
energy in that "harmonic bin" (assuming that harmonic has not been
modulated to as to overlap with adjacent harmonics) MUST be equal to the
"un-modulated harmonic. So harmonics don't "push up" under modulation
... modulation spreads that energy and the energy per unit frequency
(the envelope) goes DOWN.

MAR: How are your harpsichord harmonics "pushed up"? Remember, for a
finite energy signal (such as a recording) ... your energy envelope MUST
decrease with increasing frequency eventually (because it came from a
finite energy signal).

> don't blame individual instruments (including humans) for not having 
> "a perfectly periodic" structure when they are excited with a 

Well, considering I'm better at debugging pitch estimation code than
human DNA, I think I'll stick to the former :-)

MAR: Since I don't know your DNA debugging capabilities ... I have no
reason to doubt you here!

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 15:38:27 -0400

Hi Michael,

I think we're still mixing these two different things. What you seem to 
have in mind is signals whose "instantaneous period" changes slightly over 
time. Applying the right (variable) scaling to the time axis in such signal 
can produce an almost perfectly harmonic signal as you've shown in your 
figure. So that works for speech.

Now in many musical instruments, the vibrating modes are simply not integer 
multiples of a fundamental fundamental frequency. These signals are just 
not periodic, not even over a short time, and no amount of time scaling can 
make them periodic. For example, if you play a 440 Hz A on a guitar, even 
if your fundamental is at 440 Hz, the second vibrating mode will be at a 
frequency that is slightly higher than 880 Hz and they third will be a bit 
higher than 1320 Hz. This is caused by the thickness of the string. An 
another example, my kids have a toy xylophone that has bars that are thick 
and almost as wide as they are long. When playing one note on this 
xylophone, the higher vibrating modes are so far from being multiples of 
the fundamental that it sounds obviously dissonant. I haven't measured it, 
but I wouldn't be surprised if for a fundamental 440 Hz A, the next partial 
were around the 932 Hz A#. Not all instruments sound that bad, but most 
have some amount of deviation. I do not believe you can reproduce such an 
effect with speech.

> precise, a phase modulation effect).

Yes, I understand the phase modulation effect. But it's different from what 
I'm talking about.

> on the high harmonics (I have no clue here).

What I showed you is not related to the pitch predictor. If you do a high 
resolution FFT on a harpsichord note, you can clearly see that the 
sinusoidal partials are not *quite* equally spaced. The look like they are 
at first glance, but when you measure them you can actually see that higher 
"harmonics" are further apart than the lower ones.

> to be owing to the non-linear modulation.

I'm not sure I would even call this non-linear modulation. When you hit a 
string or a metal plate, you generate a large number of vibrating modes 
that have no reason to be equidistant in frequency. The only reason that 
they usually are "almost equidistant" is that instruments are designed to 
avoid having partials that are too dissonant.

> decrease with increasing frequency eventually (because it came from a

Sorry, I meant "up" in frequency, not amplitude. For speech if your pitch 
is modulated as f_0(t), then the kth partial you will observe will be at 
exactly f_k = k*f_0(t). In the harpsichord, you can see that f_k > k*f_0(t).

	Jean-Marc

---

From: Gregory Maxwell <[email-masked]>
Date: Fri, 5 Nov 2010 12:43:18 -0700

Michael Ramalho (mramalho) [[email-masked]] wrote:
> Each instrument has "a slightly different aperiodicity" ... thus
> Thus, the high-order harmonics DO NOT APPEAR where you expect them

I could wax philosophic about some the the really beautiful non-linear behavior that just falls out when you use fluid dynamics to model even the most simple 'flute'  (which is actually quite tricky— since the mass movements and the pressure wave behaviors happen on such different timescales) even before you get into the series truncation that comes from having finite boundaries… and how you might argue them to be to be a violation of the periodicity of the system, continual phase shifting from the delay in the acoustic wave in the cavity, or what have you…

But the fact remains is that most signals don't exhibit perfect harmonic behavior.  That is, — there is no single f_0 such that the peaks of the frequency response computed over any useful window occur precisely at integer multiples of f_0.   As codec developers the cause— and exactly what we call the cause— isn't all that relevant.

The question of pitch precision is a good one— Pitch error is a accumulated, so a pitch which is fairly accurate at f_0 may be fairly inaccurate by the 10th harmonic.  I was able to convince myself that single sample pitch period precision was adequate because the amplitude of the harmonics in most signals is greatly attenuated by the point where a 0.5 sample error would become very significant compared to the comb width, and because the masking width increases towards HF mean that not as much precision is required in the noise shaping.

So I think it's unlikely that increased precision would be justified by the extra signaling overhead and decoder complexity (needing to have multiple filter coefficient sets and more taps for sub-sample periods), but I also don't think that decreasing the pitch resolution is likely to be helpful since it only saves the signaling— and we always have the option of deactivating pitch when the improvement does not justify the cost of signaling the pitch period.   But I'd love to see examples that contradict this thinking.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 5 Nov 2010 13:30:08 -0700

Hi Jean-Marc,

> will test the clarinet tomorrow to see what 1-tap does to it.

Again, let me repeat that I am NOT proposing always using a 1-tap pitch filter.  We all know its inherent limitation of not being able to shape the spectral envelope or the degree of comb filtering with different frequencies.  The only reason I used it in my original simulations was that I was under a lot of time pressure and this was the easiest and quickest way to do a "proof-of-concept" simulation to get an idea of the potential of the pitch prefiltering/postfiltering method. Although it is also the easiest to describe, in my original technical description document submitted two weeks ago that started this email thread, before I even described the single-tap pitch filter in detail, I wrote:

"In its simplest form, a single non-zero filter tap at a bulk delay of the pitch period is used for both the pitch prefilter and postfilter.  Although it is simple, the down side is that the difference between peaks and valleys in the frequency responses of such single-tap pitch filters remain constant across the frequency range.  Thus, for those audio signals where the pitch harmonic peaks are prominent only in low frequencies, such single-tap pitch filters may introduce more periodicity in the high-frequency region than in the original input audio signal.  To avoid this problem, one can use multiple filter taps (for example, 2, 3, 4, or 5 taps) around the bulk delay of the pitch period.  By properly choosing the filter taps, one can reduce the difference between the peaks and valleys of the frequency response as the frequency increases.  It is certainly also possible to use sub-band decomposition and apply pitch prefiltering and postfiltering only to the lower band, although doing so inevitably incurs additional filtering delay and complexity."

> code the information. 

Agreed, but I am not saying we should definitely use multiple candidate sets of 3-tap filters; all I am saying is that we should at least investigate whether doing so makes a difference and is worth the extra bits transmitted.  My concern is that you seem to have already made the decision to go with a single fixed set of 3 taps without investigating whether multiple sets of 3 taps can make meaningful improvement.  I know you said you had not been able to find audio signals where this fixed set of 3 taps didn't work, but without trying multiple candidate sets of 3 taps, how do you know multiple sets can't improve the audio quality even further?  Of course, your intuition may very well be right and using multiple sets of 3 taps can't really improve perceptual audio quality any further than using your selected single fixed set, and even if it does improve, the improvement may not be worth the additional bits transmitted.  However, I don't think we can be 100% sure that this is true without actually trying it out. On the other hand, if we freeze the bit-stream format based on the fixed single set of 3 taps, then later we can't use multiple sets of 3 taps even if it proves to give meaningful improvement that's worth the extra bits transmitted.   

Therefore, my proposal is that we don't freeze the bit-stream format until this issue is investigated.  By the way, that brings me to my next question: Do you currently still plan to freeze the CELT bit-stream format by the IETF 79 meeting next Tuesday?  Is there any compelling reason that we must freeze the bit-stream format by then?  I thought the IETF codec WG charter says the schedule is to "freeze codec structure" (not necessarily bit-stream) by December 2010, and to "Finalize codec parameters" by June 2011.

> Regarding cut-off, the current 3-tap filter is

That may be true if you define "cutoff frequency" as the frequency where the peak-to-valley difference is 0 dB.  However, in practice, comb filtering with a peak-to-valley difference smaller than a certain positive number of X dB won't be detectable to human ears.  Therefore, you could potentially select the 3 taps carefully so that the peak-to-valley difference is less than X dB at all frequencies above Y Hz, where Y is much lower than your current 0 dB point of 20 kHz.  For example, if the three taps are chosen to be [0.33, 0.33, 0.33], and the overall scaling factor for the three taps is 0.5, then this filter will shift the 0 dB point from the 20 kHz of your fixed filter to about 16 kHz, and if the number X is 2 dB (I don't know the real value of X; I am just using this as an example), then the peak-to-valley difference is less than X = 2dB for all frequencies above about 13 kHz.  If you were to make the middle coefficient of the 3 taps smaller than its left and right neighbors, then the 0 dB point can be moved down further to below 15 kHz.

> we need to make sure that low-frequency effects are not taken into account.

Yes, it's the same algorithm as used in LC-CELP.  I know we are not after the optimal pitch predictor, but I believe that if we use that search method, then the candidate "full-strength" 3-tap pitch synthesis filter that wins in that search will have the best overall match between its frequency response and the input signal spectrum (better than the match of any other candidate 3-tap pitch filters in the codebook).  Isn't the best overall match between the input signal spectrum and the frequency response of this "full-strength" pitch synthesis filter (rather than the scaled-down version we eventually use) over the entire frequency range what we should really be after?  If so, then I believe this search method is exactly what we need.  (Disclaimer: I have not verified this with experiments yet, so the above argument is just based on how I understand linear prediction should work.  Theoretically there is no real difference in this spectral match regard between our 3-tap pitch predictor and the common short-term LPC predictor.)  If somehow your intuition is correct and the low-frequency effect should not be taken into account, then that can be accommodated by passing the pitch prediction residual signal through a simple high-pass filter before we calculate the energy and then pick the winning 3-tap filter, but in that case I don't think my efficient method can be used.  On the other hand, if we only have 2 or 4 candidate 3-tap filters, then doing such a brute-force search by doing actual pitch filtering and high-pass filtering may not be that bad (compared with 64 candidate 3-tap filters in LC-CELP).

> always choose the same filter.

My concern is that we may not get sufficient empirical results to conclude one way or the other by the IETF 79 codec WG meeting next Tuesday (Monday night California time).  That's why I ask you whether we must freeze the bit-stream format by that meeting.  If there is no absolute and compelling reason that we must freeze the bit-stream format by then, I would propose that we don't freeze it yet so we can have more time to investigate whether multiple sets of 3-tap pitch filter and whether reduced time resolution for very long pitch period might be better design choices.  You did raise a good point that for those who really want to avoid the search complexity for the 3-tap filter, they can always use your fixed 3-tap filter without any search.

> and thus cannot be changed later on.

Agreed.  Your latest processed versions of the trumpet signal is a good example that the existence of the pitch prefilter/postfilter can affect the optimal bit allocation.  It showed that with the pitch prefilter/postfilter being able to clean up a lot of the lower frequency distortion, you were able to allocate fewer bits to the lower frequencies (thus the CELT without pitch filtering sounded worse than the previous tuning of bit allocation), and yet the prefilter/postfilter method still cleaned up the higher level of lower frequency distortion with the final output audio quality sounding fairly good.  Given that the pitch prefilter/postfilter is a very recent addition (but it improves audio quality a lot for some audio signals), chances are we need to take our time to make sure the bit allocation with this feature on is optimal.

> enabled using ENABLE_POSTFILTER.

Our legal counsel is probably very busy with other things, but I have asked him to submit an IPR declaration regarding our pitch prefilter/postfilter method as soon as he can.  Hopefully we will see that soon.

Cheers,

	Jean-Marc

---

From: Gregory Maxwell <[email-masked]>
Date: Fri, 5 Nov 2010 14:18:15 -0700

> My concern is that you seem to have already made the decision to
> go with a single fixed set of 3 taps without investigating whether
> multiple sets of 3 taps can make meaningful improvement.  I know you
> said you had not been able to find audio signals where this fixed set
> of 3 taps didn't work, but without trying multiple candidate sets of
> 3 taps, how do you know multiple sets can't improve the audio quality
> even further?  Of course, your intuition may very well be right and
> using multiple sets of 3 taps can't really improve perceptual audio
> quality any further than using your selected single fixed set, and even
> if it does improve, the improvement may not be worth the additional
> bits transmitted.  However, I don't think we can be 100% sure that
> this is true without actually trying it out. On the other hand, if we
> freeze the bit-stream format based on the fixed single set of 3 taps,
> then later we can't use multiple sets of 3 taps even if it proves to
> give meaningful improvement that's worth the extra bits transmitted.
> Therefore, my proposal is that we don't freeze the bit-stream format
> until this issue is investigated.  By the way, that brings me to my next
> question: Do you currently still plan to freeze the CELT bit-stream
> format by the IETF 79 meeting next Tuesday?  Is there any compelling
> reason that we must freeze the bit-stream format by then?  I thought the
> IETF codec WG charter says the schedule is to "freeze codec structure"
> (not necessarily bit-stream) by December 2010, and to "Finalize codec
> parameters" by June 2011.
[snip]

I'm pretty sure we've missed the "freeze the celt part of the bitstream
prior to the meeting" goal.  For one— the pitch filtering isn't even
a part of the official CELT bitstream at all currently.   We have code
in the repository but it isn't going into our distribution until the
IPR stuff is ironed out.

As far as the time line goes— what JM has wanted to do is finalize the
CELT part of OPUS ASAP and then we can move our focus to the
combined codec.  (And, assuming that the working group doesn't
remove anything from OPUS that we consider to be a critical CELT 
use case then CELT will likely be abandoned— and the fact that
its bitstream was frozen early won't be especially relevant for anything)

What we're doing right now is a CELT 0.9.0 release in order to get
most of the recent improvements (other than pitch) into people's hands.

In particular we need to get an update out so we can try to solicit example
cases where CELT still performs poorly. We're doing very well on average
but I'm concerned that there are still corner cases where we suffer greatly.

I don't believe that JM intended to sound conclusive about how pitch would
work. We can continue to explore refining this feature.

I did some simple work on different pitch tap coefficients.

Here is what I tried:

I took a single tonal sample and searched the one dimensional space
of  symmetrical unit norm 3-tap predictors for the value with the lowest
noise-to-mask ratio according to the PEAQ basic model. (the gain is
already handled automatically on a frame by frame basis)

The best value was at [0.4195,0.161,0.4195].  In spite of having zero point
around 12kHz rather than at 18.5kHZ I found that it didn't sound all that
different.

Obviously I need to do some more testing with more samples.

I intend to do so, but my initial try produced less significant results than
I was expecting, so I've taken it as a lower priority. I saw much more
significant changes from changing our filter gain selection than any
manipulation of the taps provided.

It may be more interesting now that we've resolved more of the issues
with the pitch search.

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 17:32:37 -0400

On 10-11-05 04:30 PM, Raymond (Juin-Hwey) Chen wrote:
> Agreed, but I am not saying we should definitely use multiple
> candidate sets of 3-tap filters; all I am saying is that we should at
> least investigate whether doing so makes a difference and is worth
> the extra bits transmitted.  My concern is that you seem to have
> already made the decision to go with a single fixed set of 3 taps
> without investigating whether multiple sets of 3 taps can make
> meaningful improvement.

Don't worry, no final decision had been made on this topic and nothing
is really final until the bit-stream is frozen. I have completely
changed my mind dozens of times before on many areas of CELT and I'm
always willing to do so when evidence contradicts earlier evidence (or
in many cases guesswork due to lack of time to do decent experiments).

> I know you said you had not been able to
> find audio signals where this fixed set of 3 taps didn't work, but
> without trying multiple candidate sets of 3 taps, how do you know
> multiple sets can't improve the audio quality even further? 

I don't. However, until we can find cases where a different set of taps
can help, I don't see a reason to support multiple sets of taps. Even if
we *knew* that one set wasn't enough, we would at least need to know
what are the alternate sets of taps are useful.

> until this issue is investigated.  By the way, that brings me to my
> next question: Do you currently still plan to freeze the CELT
> bit-stream format by the IETF 79 meeting next Tuesday?  Is there any
> compelling reason that we must freeze the bit-stream format by then?
> I thought the IETF codec WG charter says the schedule is to "freeze
> codec structure" (not necessarily bit-stream) by December 2010, and
> to "Finalize codec parameters" by June 2011.

The bit-stream will definitely not be frozen by the next IETF meeting --
I'm sorry if I implied otherwise. The goal was to be close to a point
where we consider freezing and although the structure is now pretty
good, there are still *lots* of details to sort out. So don't worry, the
bit-stream will not be frozen before you have a chance to experiment.

>> Regarding cut-off, the current 3-tap filter is basically the lowest
>> cutoff I could do with 3 taps. To get lower than that requires 5
>> taps and I'm not sure it's worth the trouble.
> 
> That may be true if you define "cutoff frequency" as the frequency
> where the peak-to-valley difference is 0 dB.  However, in practice,
> comb filtering with a peak-to-valley difference smaller than a
> certain positive number of X dB won't be detectable to human ears.
> Therefore, you could potentially select the 3 taps carefully so that
> the peak-to-valley difference is less than X dB at all frequencies
> above Y Hz, where Y is much lower than your current 0 dB point of 20
> kHz.  For example, if the three taps are chosen to be [0.33, 0.33,
> 0.33], and the overall scaling factor for the three taps is 0.5, then
> this filter will shift the 0 dB point from the 20 kHz of your fixed
> filter to about 16 kHz, and if the number X is 2 dB (I don't know the
> real value of X; I am just using this as an example), then the
> peak-to-valley difference is less than X = 2dB for all frequencies
> above about 13 kHz.  If you were to make the middle coefficient of
> the 3 taps smaller than its left and right neighbors, then the 0 dB
> point can be moved down further to below 15 kHz.

Indeed, a fixed [1, 1, 1]/3 filter is also something I considered. I
just wasn't able to tell the difference between that and the current
filter, so I just temporarily picked "what made sense to me
intuitively". If you can show that there's another set of tap that
actually sounds better, I'm happy to use it. BTW, one (theoretical)
concern I have with a filter like [1,1,1]/3 is that once it passes the
zero-point, the structure is actually inverted so if the signal really
*was* harmonic past that poing (which I agree is unlikely because of
what we discussed earlier), the the post-filter would actually have a
"noise amplification" behaviour. Again, this is my intuitive reasoning
as I cannot actually hear a difference either way.

> Agreed.  Your latest processed versions of the trumpet signal is a
> good example that the existence of the pitch prefilter/postfilter can
> affect the optimal bit allocation.  It showed that with the pitch
> prefilter/postfilter being able to clean up a lot of the lower
> frequency distortion, you were able to allocate fewer bits to the
> lower frequencies (thus the CELT without pitch filtering sounded
> worse than the previous tuning of bit allocation), and yet the
> prefilter/postfilter method still cleaned up the higher level of
> lower frequency distortion with the final output audio quality
> sounding fairly good.  Given that the pitch prefilter/postfilter is a
> very recent addition (but it improves audio quality a lot for some
> audio signals), chances are we need to take our time to make sure the
> bit allocation with this feature on is optimal.

Actually, the bit allocation change I did recently is one of the very
few that is encoder-only (i.e. explicitly signaled in the bit-stream).
The part that I'm most concerned with is the other part that we cannot
change. That one was a concern even before you proposed this
post-filter. Here's roughly how the allocation works:

1) Each band starts with a "default" bit allocation that is constant in
time (though different bands get a different number of bits) for a
particular bitrate. This "default" allocation is known to both the
encoder and decode and this cannot be changed without changing the
bit-stream.

2) A few weeks ago, I added a "trim" mechanism that allows tilting the
bit allocation either towards the high frequencies or towards the low
frequencies on a frame-by-frame basis. That trim is signaled in the
bit-stream, so an improved encoder can change it without changing the
bit-stream. This is the mechanism I've tuned a few days ago.

3) Also a few weeks ago, I added a mechanism to increase the bit
allocation of a specific band on a frame-by-frame basis. This can
actually help on files like the trumpet because it allows the encoder to
put more bits in the bands that have the tones as opposed to the ones
between the tones. As for 2), an encoder can change the decision for
this without changing the bit-stream.

So until very recently, the only allocation mechanism in CELT was 1).
This is by far the most important and it is hard to tune correctly for
all bit-rates. I believe there are still quality gains to be made by
further tuning there. Those have to be made before freezing the bit-stream.

>> Oh, and for those who would like to try out the post-filter, it is
>> now in the default "master" branch in the CELT repository (haven't
>> synced Opus yet). It is disabled by default for "IPR reasons", but
>> can be enabled using ENABLE_POSTFILTER.
> 
> Our legal counsel is probably very busy with other things, but I have
> asked him to submit an IPR declaration regarding our pitch
> prefilter/postfilter method as soon as he can.  Hopefully we will see
> that soon.

Thanks. In the mean time, it should still be relatively easy to test.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 5 Nov 2010 18:30:58 -0700

Hi Michael,

I am not sure I really understand your arguments.  You said:

> Bottom line is that comb filters are effective for the "low order harmonics" of a
> sufficiently periodic signal. However, they simply CANNOT capture (or eliminate,
> depending on what you are doing) higher order "harmonic components" because the
> energy in them is not where YOU "expect" them to be. When observing the high-order
> harmonics some start "blaming the pitch estimators" ... but that isn't (necessarily)
> where the problem is ... the problem is likely in the assumption on where the "higher
> harmonic energy" should be!

> Please don't use synthetic signals to test the utility of your comb filter. Please
> don't blame pitch estimators for them not capturing (or
> rejecting) high harmonic energy YOU think should be there. And please don't blame
> individual instruments (including humans) for not having "a perfectly periodic"
> structure when they are excited with a not-perfectly-periodic stimuli (or, in the
> case of the harpsichord, non-perfectly ridged end-conditions for the strings).

Are you saying that it is theoretically impossible to find audio signals with harmonically related spectral peaks extending all the way up to 20 kHz, and that as frequency increases such spectral peaks will always "smear out"?  I don't agree with that.  One can easily synthesize a purely periodic signal (within the FFT analysis window at least) that has harmonic peaks throughout the passband at integer multiples of the fundamental frequency f0 without any "smearing" of peaks.

You said "Please don't use synthetic signals to test the utility of your comb filter", but the point is not to use synthetic signals to "test the utility of the comb filter"; instead, the question to ask is: Is it possible for the CELT codec to encounter real-world audio signals that have harmonically related spectral peaks extending up to 20 kHz? I believe the answer is yes.  Many contemporary music pieces use artificially synthesized signals from a synthesizer or keyboard as the main or even the only instrument.  Such synthetic signals are legitimate audio signals that the CELT codec will have to code, and we can't just rule them out as something not worth considering.  It's a given that not all synthetic music signals have harmonically related spectral peaks, but I will be extremely surprised if none of the synthetic music signals from any synthesizer or any keyboard instrument (even the cheap sub-$100 ones) has harmonically related spectral peaks throughout the passband.

Even if we were to rule out such synthetic music signals, it would still be difficult for me to believe that none of the natural (non-synthetic) music instruments can produce harmonically related spectral peaks without smearing throughout the passband.  Did you click open and view the PDF file I previously attached for the spectrum of a clarinet signal?  I will attach it again.  Please take a look at it.  I measured all the spectral peaks, and within my measurement error it seems clear to me that they are all harmonically related, and you don't see any bandwidth widening as the frequency increases; you see sharp spectral peaks throughout.  In fact, in the Wikipedia article that Jean-Marc pointed to about inharmonicity, it states:

"When a string is bowed or tone in a wind instrument initiated by vibrating reed or lips, a phenomenon called mode-locking counteracts the natural inharmonicity of the string or air column and causes the overtones to lock precisely onto integer multiples of the fundamental pitch, even though these are slightly different than the natural resonance points of the instrument. For this reason, a single tone played by a bowed string instrument, brass instrument, or reed instrument does not exhibit inharmonicity."

Later in that article they even mentioned the clarinet as an example.  If we can believe the authority and correctness of this Wikipedia article, then the last sentence in the quoted text above all but settles the dispute.

Raymond

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Wednesday, November 03, 2010 3:41 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Can you tell me if you can still hear the artefact? Note that these use the default (trim 3) allocation.

Cheers,

        Jean-Marc

seconds,
either
change
instances
2010
method
>>>>> to be an improvement over my original single-tap filters.
>>>>>>> period can range from 30 samples (1500 Hz
>>>>>>> values can range between 0.25 and 0.625 in increments of 0.125.
>>>>>>> Any opinion on these?
>>>>>>
>>>>>> sets I used. The test set consists of 10 second segments randomly

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 5 Nov 2010 18:54:09 -0700

Hi Jean-Marc,

> in many cases guesswork due to lack of time to do decent experiments).

OK, thanks.  That's good to know.

> However, until we can find cases where a different set of taps
> can help, I don't see a reason to support multiple sets of taps. 

Agreed.  I thought you were determined to use a single fixed set of 3 pitch taps and will freeze the bit-stream format accordingly at the IETF 79 meeting next Tuesday.  That's why I was concerned and try to argue that we should at least investigate whether having multiple sets of 3-tap filters can improve audio quality meaningfully. Of course, if the experiment shows that having multiple sets doesn't improve or doesn't improve enough to justify the additional bits transmitted, then obviously we should go with your simpler and yet effective fixed 3-tap filter design.

> bit-stream will not be frozen before you have a chance to experiment.

When I first talked to you about the IETF schedule, I got an impression that you planned to freeze the CELT bit-stream format by the IETF 79 meeting.  That's what got me concerned because I thought we still needed to do some more investigation to come up with the best design.  Sorry about the misunderstanding, and thank you for the clarification.  It is good to know that we still have some time to investigate a few loose ends.

> as I cannot actually hear a difference either way.

You may very well be right that at very high frequencies you don't really need comb filtering.  We just haven't had a chance to do enough experiments with it yet. I was just concerned that you may lock up the bit-stream format by next Tuesday and then we won't have a chance to use multiple sets of 3-tap filters even if that later proves to be beneficial. Thus, I was not arguing that using multiple sets of filters will definitely help (I don't know); I was just arguing for a chance to try it out before freezing the bit-stream format.  If using multiple sets of filters turns out not beneficial, we would of course drop the idea. 
Good point about the peak inversion. Hopefully we won't hear anything wrong if the peak-to-valley difference is smaller than a certain threshold.

> Thanks. In the mean time, it should still be relatively easy to test.

Our legal counsel told me he hopes to file our IPR declaration tomorrow.

Raymond

P.S.  Thanks for the explanation of recent changes to the bit allocation scheme.  That was very helpful.

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 05 Nov 2010 22:06:52 -0400

Hi Raymond,

Again sorry about the misunderstanding about freezing the bit-stream.
Regarding the 1-tap vs 3-tap, the you have time to listen to the samples
I posted earlier?

Let me know what you think of those.

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 6 Nov 2010 14:43:42 -0700

Hi Jean-Marc,

> Again sorry about the misunderstanding about freezing the bit-stream.

No problem at all, and I am also partially at fault as I should have asked you last week to confirm your plan to freeze the bit-stream. I am just glad that we still have some time to iron out a few things.

> Let me know what you think of those.

Sorry I was busy yesterday and didn't have time to listen to them yet.  I did careful listening comparisons and spectral plot comparisons today.  I found the following:

(1) I am somewhat surprised that the spectral shapes of the output signals of 1-tap filter and 3-tap filter at high frequencies (say, above 10 kHz) are generally very close (in terms of how clear the harmonic structure can be seen and the distance between harmonic peak values and adjacent spectral valley values).  Given the substantial difference in the frequency responses of the two filters in this high-frequency region, I would have expected a larger difference, with the 1-tap filter output showing stronger harmonic peaks.  Are you sure you have used exactly the same "strengths" of the filters?

(2) Given that the signal spectra in the high-frequency regions are so close, it is not surprising that just like you, I cannot reliably hear any meaningful difference between the two filter outputs in this high-frequency region.  On the other hand, over the years I have found that when compared with other people with "trained ears", my hearing seems to be less sensitive to high-frequency distortion and more sensitive in picking up mid- to low-frequency distortion.  Thus, I may not be a good listener for picking up subtle differences in high-frequency regions.  I will ask one of my team members who has sensitive hearing in high-frequency regions to listen to these files next Monday.

(3) Interestingly, though, I found that the 3-tap filter output signals tend to have more instances of audible mid- to low-frequency disturbances than the corresponding 1-tap filter output signals, or when both have it, the MF to LF distortion in the 3-tap filter output tends to be slightly louder or more audible.  For example, at 4.02 seconds from the beginning of the two 96 kb/s files, both 3-tap and 1-tap versions have an audible MF to LF distortion, but the distortion in the 3-tap version sounds significantly louder to me than the 1-tap version.  As another example, at about 5.6 seconds into the 128 kb/s files, the 3-tap filter output file has a very slight MF to LF distortion, while the 1-tap filter output file is essentially clean there.

I am not sure why the 3-tap and 1-tap filters would make such a difference in the MF to LF distortion.  It is not a big day-and-night difference; the difference is slight, but it is nonetheless a difference that I can reliably hear it repeatedly.

Raymond

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 6 Nov 2010 15:39:34 -0700

Hi Greg,

Thanks for giving a detailed update on the CELT development.  That's very useful. Your experiment also sounds interesting.  What kind of "single tonal sample" did you use?

Regarding your comment that "it isn't going into our distribution until the
IPR stuff is ironed out", I have several emails with our legal counsel today. He is traveling today with intermittent Internet access, but it is still his plan to try to submit an IPR disclosure/statement to the IETF today about Broadcom's pitch prefilter/postfilter method.  He said Broadcom will indeed provide a royalty-free IPR disclosure relating to this method.

Thanks.

Raymond

-----Original Message-----
From: Gregory Maxwell [mailto:[email-masked]] 
Sent: Friday, November 05, 2010 2:18 PM
To: Raymond (Juin-Hwey) Chen; Jean-Marc Valin
Cc: [email-masked]'
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT codec

[snip]

I'm pretty sure we've missed the "freeze the celt part of the bitstream
prior to the meeting" goal.  For one- the pitch filtering isn't even
a part of the official CELT bitstream at all currently.   We have code
in the repository but it isn't going into our distribution until the
IPR stuff is ironed out.

As far as the time line goes- what JM has wanted to do is finalize the
CELT part of OPUS ASAP and then we can move our focus to the
combined codec.  (And, assuming that the working group doesn't
remove anything from OPUS that we consider to be a critical CELT 
use case then CELT will likely be abandoned- and the fact that
its bitstream was frozen early won't be especially relevant for anything)

Here is what I tried:

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 6 Nov 2010 15:47:00 -0700

Hi Greg,

Sorry I was too busy to respond to your email below sent Wednesday.

(1) I thought the CELT-only mode in the IETF codec only goes up to 20 ms frame size.  Did you use a frame size of 30 ms just to do an experiment and try to maximize the coding efficiency?

(2) What happens if you use a frame size of 20 ms for CELT so it is the same frame size as Speex?

(3) I would be interested in also hearing the output of the corresponding CELT codec without the pitch prefilter/postfilter to get a sense of the difference the pitch filters make.

Thanks.

Raymond

-----Original Message-----
From: Gregory Maxwell [mailto:[email-masked]] 
Sent: Wednesday, November 03, 2010 8:28 AM
To: Raymond (Juin-Hwey) Chen; Jean-Marc Valin
Cc: [email-masked]'; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT codec

> Would you post your 8 kb/s coded speech as well so we can listen.  I am curious what it sounds like.

It was mostly a silly test- but sure

For comparison-

---

From: "Jean-Marc Valin" <[email-masked]>
Date: Sun, 7 Nov 2010 07:36:04 -0500

Hi Raymond,

> (1) I am somewhat surprised that the spectral shapes of the output
> signals of 1-tap filter and 3-tap filter at high frequencies (say,
> above 10 kHz) are generally very close (in terms of how clear the
> harmonic structure can be seen and the distance between harmonic peak
> values and adjacent spectral valley values).  Given the substantial
> difference in the frequency responses of the two filters in this
> high-frequency region, I would have expected a larger difference,
> with the 1-tap filter output showing stronger harmonic peaks.  Are
> you sure you have used exactly the same "strengths" of the filters?

Yes, the only thing I changed was the filter taps (their sum before the global gain was one in both cases). I can see two reasons for the lack of difference in the spectra:

1) The 3-tap filter still provides some post-filtering effect even at frequencies above 10 kHz.

2) Past a certain (bitrate-dependent) point, the spectral details are replaced by a "folded" version of the lower frequency spectrum. This means that the harmonics aren't correctly aligned anymore and thus the post-filter does not provide any benefit.

> (2) Given that the signal spectra in the high-frequency regions are
> so close, it is not surprising that just like you, I cannot reliably
> hear any meaningful difference between the two filter outputs in this
> high-frequency region.  On the other hand, over the years I have
> found that when compared with other people with "trained ears", my
> hearing seems to be less sensitive to high-frequency distortion and
> more sensitive in picking up mid- to low-frequency distortion.  Thus,
> I may not be a good listener for picking up subtle differences in
> high-frequency regions.  I will ask one of my team members who has
> sensitive hearing in high-frequency regions to listen to these files
> next Monday.

I'm also more sensitive to the low-end of the spectrum, but even looking at spectrograms, there didn't seem to be that much difference in the high frequencies.

> (3) Interestingly, though, I found that the 3-tap filter output
> signals tend to have more instances of audible mid- to low-frequency
> disturbances than the corresponding 1-tap filter output signals, or
> when both have it, the MF to LF distortion in the 3-tap filter output
> tends to be slightly louder or more audible.  For example, at 4.02
> seconds from the beginning of the two 96 kb/s files, both 3-tap and
> 1-tap versions have an audible MF to LF distortion, but the
> distortion in the 3-tap version sounds significantly louder to me
> than the 1-tap version.  As another example, at about 5.6 seconds
> into the 128 kb/s files, the 3-tap filter output file has a very
> slight MF to LF distortion, while the 1-tap filter output file is
> essentially clean there.

I'll see if I can hear that when I get my headphones back (I'm in Beijing now). I have to admit I'm a little surprised by what you're saying because for LF 3-tap and 1-tap should be equivalent. Could this be the effect of changing the pitch or filter gain at the frame boundaries?

   Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sun, 7 Nov 2010 10:45:08 -0800

Hi Jean-Marc,

> Yes, the only thing I changed was the filter taps (their sum before 
> the global gain was one in both cases). I can see two reasons for 
> the lack of difference in the spectra:
>

Yes, there is "some", but should be much less than the 1-tap filter; 
that's why I was quite surprised. 

> 2) Past a certain (bitrate-dependent) point, the spectral details 
> are replaced by a "folded" version of the lower frequency spectrum. 
> This means that the harmonics aren't correctly aligned anymore and 
> thus the post-filter does not provide any benefit.

OK, that may be the real reason.  I didn't know that you were doing 
The spectral folding.

> I'll see if I can hear that when I get my headphones back (I'm in 
> Beijing now). I have to admit I'm a little surprised by what you're
> saying because for LF 3-tap and 1-tap should be equivalent. Could
> this be the effect of changing the pitch or filter gain at the frame 
> boundaries?

Yes, I am surprised, too.  It could be boundary effect, but even if it
is due to this, I am not sure why the boundary effect could be 
different between 1-tap and 3-tap filters.

Sorry that I cannot go to Beijing to attend the IETF 79 meeting.  I
hope you have a successful meeting there.  I will join the meeting
remotely from California.

By the way, our legal counsel told me that he filed Broadcom's IPR 
disclosure/statement regarding our pitch prefilter/postfilter 
method with the IETF this morning.

Raymond

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Mon, 8 Nov 2010 07:52:47 -0600

Hi Raymond,

Sorry for the lateness of this reply ... I just found your questions in
my Spam folder.

In-line below with "MAR:"

Michael Ramalho

-----Original Message-----
From: Raymond (Juin-Hwey) Chen [mailto:[email-masked]] 
Sent: Friday, November 05, 2010 9:31 PM
To: Michael Ramalho (mramalho); Jean-Marc Valin
Cc: [email-masked]; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT
codec
Importance: Low

Hi Michael,

I am not sure I really understand your arguments.  You said:

> order "harmonic components" because the energy in them is not where 
> YOU "expect" them to be. When observing the high-order harmonics some 
> start "blaming the pitch estimators" ... but that isn't (necessarily) 
> where the problem is ... the problem is likely in the assumption on
where the "higher harmonic energy" should be!

> don't blame individual instruments (including humans) for not having
"a perfectly periodic"
> structure when they are excited with a not-perfectly-periodic stimuli 
> (or, in the case of the harpsichord, non-perfectly ridged
end-conditions for the strings).

Are you saying that it is theoretically impossible to find audio signals
with harmonically related spectral peaks extending all the way up to 20
kHz, and that as frequency increases such spectral peaks will always
"smear out"?  I don't agree with that.

MAR: I never said it was theoretically impossible to find such signals.
However many "instruments", such as the human voice, are just not
"perfectly periodic enough" such that comb filters are effective beyond
3k or so. That is why spectral subtraction (usually implemented with a
comb filter to "subtract" out noise energy "outside the harmonics" has
NEVER worked well for speech. Will they "work" in the energy sense - yes
.... because most of speech energy is in low frequency bands (due to
spectral tilt of speech ... and that tilt exists because speech is a
finite energy signal). Will they "work" in the auditory sense - no ....
because the high frequency speech energy cannot be captured (or rejected
... depending on what you want to do) adequately. This is well known in
the literature.

One can easily synthesize a purely periodic signal (within the FFT
analysis window at least) that has harmonic peaks throughout the
passband at integer multiples of the fundamental frequency f0 without
any "smearing" of peaks.

MAR: Of course one can synthesize perfectly periodic signals.

You said "Please don't use synthetic signals to test the utility of your
comb filter", but the point is not to use synthetic signals to "test the
utility of the comb filter"; instead, the question to ask is: Is it
possible for the CELT codec to encounter real-world audio signals that
have harmonically related spectral peaks extending up to 20 kHz? I
believe the answer is yes.

MAR: I agree. But that doesn't say anything about the LARGER space of
other valid real-world signals.

Many contemporary music pieces use artificially synthesized signals from
a synthesizer or keyboard as the main or even the only instrument.

MAR: The design of such instruments typically includes the
warping/mapping of "pure tones" (i.e., perfectly periodic signals) to
create the timbre of the original instrument. There are many journal
papers on this subject - dating back to the early 1980s. Those papers
usually cite phase, frequency and/or time modulation. If they did not
employ such techniques, they would all sound like "1970-era music
synthesizers". In fact, many modern keyboard instruments use mappings of
real instrument recordings, because no equation-based formulation sounds
"close enough" to the original instrument. Clearly manufacturers of such
instruments would not resort to those techniques if simple summing of
harmonic components was "good enough".

Such synthetic signals are legitimate audio signals that the CELT codec
will have to code, and we can't just rule them out as something not
worth considering.  It's a given that not all synthetic music signals
have harmonically related spectral peaks, but I will be extremely
surprised if none of the synthetic music signals from any synthesizer or
any keyboard instrument (even the cheap sub-$100 ones) has harmonically
related spectral peaks throughout the passband.

MAR: I too would be surprise if "NONE of the synthetic music signals"
[had sufficient periodicity], note the emphasis on the use of your word
"NONE".

Even if we were to rule out such synthetic music signals, it would still
be difficult for me to believe that none of the natural (non-synthetic)
music instruments can produce harmonically related spectral peaks
without smearing throughout the passband.

MAR: I too agree with this statement - but again you used "none of the
natural signals". Did you ever analyze a tuba, or a bassoon, or
something with a fundamental near a human voice (like ~100 Hz)? Your
clarinet sound looks like it has a fundamental near 650 Hz .... and
Jean-Marc's harpsichord (which I think is more harmonic than reed or
brass instruments) had a fundamental in excess of 500 Hz (looks like
~540 Hz). Your 6th harmonic of these signals in frequency is roughly at
the 30th harmonic of a (100 Hz) human voice.

Did you click open and view the PDF file I previously attached for the
spectrum of a clarinet signal?  I will attach it again.  Please take a
look at it.  I measured all the spectral peaks, and within my
measurement error it seems clear to me that they are all harmonically
related, and you don't see any bandwidth widening as the frequency
increases; you see sharp spectral peaks throughout.  In fact, in the
Wikipedia article that Jean-Marc pointed to about inharmonicity, it
states:

"When a string is bowed or tone in a wind instrument initiated by
vibrating reed or lips, a phenomenon called mode-locking counteracts the
natural inharmonicity of the string or air column and causes the
overtones to lock precisely onto integer multiples of the fundamental
pitch, even though these are slightly different than the natural
resonance points of the instrument. For this reason, a single tone
played by a bowed string instrument, brass instrument, or reed
instrument does not exhibit inharmonicity."

MAR: Your 30th harmonic (the rough equivalent of a ~3 KHz harmonic of a
human voice) is in the noise (whereas 3K energy from a voice is well
above it). Nonetheless, it does seem rather harmonic .... but with a
higher F0 (at 600 Hz) you need 1/6 LESS time (in ms) to get a similar
resolution of it ... and therefore (1/6) less variability (modulation)
per analysis time window.

Later in that article they even mentioned the clarinet as an example.
If we can believe the authority and correctness of this Wikipedia
article, then the last sentence in the quoted text above all but settles
the dispute.

MAR: While Wikipedia articles are generally suspect (even I write some
pages), I don't doubt the underlying science too much. But virtually
every talented instrument performer knows how to tweak it's instrument
to do something non-harmonic (saxophone "squeal", violin/bass/cello
vibrato, special horns ... even toilet plungers ... inserted into the
bell of a horned instrument to purposely change it's timbre). And it is
usually the "talented performers" who make it in music (we all know some
exceptions to that statement!).

MAR: As long as the IETF codec group wants to model all sounds as if
they are perfectly periodic ... they will capture most of the energy of
quasi-periodic sources ... but I sincerely doubt that the higher
frequency components of quasi-periodic sounds such as voice will be
accurately modeled. Given the response of the human ear to high
frequency, this may or may not be important. Clearly if Spectral Band
Replication (and other similar techniques) "sounds OK" to the human ear,
a human listener may or may not notice it.

Raymond

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Wednesday, November 03, 2010 3:41 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT
codec

Hi Raymond,

Can you tell me if you can still hear the artefact? Note that these use
the default (trim 3) allocation.

Cheers,

        Jean-Marc

seconds,

either
change
instances

2010
method

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 8 Nov 2010 17:43:55 -0800

Hi Michael,

Thank for your additional comments.

I am not sure which kind of spectral subtraction you were referring to, but there is a very common type of speech enhancement (or noise suppression) technique that applies spectral subtraction to speech signals and gets very good noise suppression.  Some of such spectral subtraction methods don't have sufficient frequency resolution to resolve the pitch harmonics and only attenuate the spectral valleys between speech formants.  However, I have seen spectral subtraction schemes with high frequency resolution (large FFT size) that can resolve the pitch harmonics and indeed attenuate the spectral valleys between pitch harmonics, and these schemes indeed get additional noise suppression compared with the ones that only attenuate the spectral valleys between speech formants.

It seems that you also agree that there are some real-world audio signals, whether natural or synthesized, that are truly periodic or close to truly periodic.  My point is just that as long as this is true (which you seem to agree), even if they are the minority, then such periodic or nearly periodic signals are still "legitimate" input signals that CELT or any other audio codecs sometimes have to deal with in the real world, and therefore I am not sure I can honestly agree with your comment that we should not use such signals to test the codec system.

Thanks for all of your informative comments, though :-)

Raymond

-----Original Message-----
From: Michael Ramalho (mramalho) [mailto:[email-masked]]
Sent: Monday, November 08, 2010 5:53 AM
To: Raymond (Juin-Hwey) Chen; Jean-Marc Valin
Cc: [email-masked]; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT codec

Hi Raymond,

Michael Ramalho

-----Original Message-----
From: Raymond (Juin-Hwey) Chen [mailto:[email-masked]]
Sent: Friday, November 05, 2010 9:31 PM
To: Michael Ramalho (mramalho); Jean-Marc Valin
Cc: [email-masked]; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT
codec
Importance: Low

Hi Michael,

"a perfectly periodic"

Raymond

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Wednesday, November 03, 2010 3:41 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]'; Jean-Marc Valin
Subject: Re: [codec] A pitch filtering method to improve the mode 1 CELT
codec

Hi Raymond,

Cheers,

        Jean-Marc

seconds,

either
change
instances

2010
method

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 8 Nov 2010 18:15:56 -0800

Hi Jean-Marc,

An update:
My team member listened to your 1-tap and 3-tap filter outputs and said "there is no significant difference between 1 and 3 taps" to him.  Thus, I suppose for this particular file the difference is negligible, if any.

However, as I think about this more, perhaps for 48 kHz sampling rate this is the case, but when you go down in sampling rate to 32, 16, or even 8 kHz, then tapering off the comb filtering effect at the folding frequency or 5/6 of the folding frequency as you did for your single fixed set of the 3-tap filter is likely to become more and more suboptimal.  Take 8 kHz for example.  If you were to apply exactly the same set of 3 taps there, then the comb filtering effect will keep decreasing and reach zero at 4*5/6 = 3.33 kHz.  I believe many (nearly) periodic signals still have fairly strong harmonic structure in the 2 to 3.3 kHz range and can benefit from a different set of 3-tap filters where the degree of comb filtering decreases much slower with increasing frequency than your current set of fixed 3-tap filter.  Therefore, perhaps there is a greater need for multiple sets of 3-tap filters as the sampling rate goes down toward 16 kHz and 8 kHz.

Raymond

-----Original Message-----
From: Raymond (Juin-Hwey) Chen 
Sent: Sunday, November 07, 2010 10:45 AM
To: 'Jean-Marc Valin'
Cc: [email-masked]'; 'Jean-Marc Valin'
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT codec
Importance: Low

Hi Jean-Marc,

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 08 Nov 2010 22:07:16 -0500

Hi Raymond,

Regarding the effect of the sampling rate, I totally agree that we  
wouldn't want to keep the same 3-tap filter. What I think is that we  
should still have a single filter, but that filter should actually  
change with the sampling rate. For example, at 8 kHz, a 3-tap filter  
with coefficients [.05,.9,.05] will actually give you a response  
that's very similar to the [1,sqrt(3),1]/(2+sqrt(3)) filter we are  
currently using at 48 kHz.

Cheers,

     Jean-Marc

"Raymond (Juin-Hwey) Chen" <[email-masked]> a écrit :
> Hi Jean-Marc,
>
> An update:
> My team member listened to your 1-tap and 3-tap filter outputs and  
> said "there is no significant difference between 1 and 3 taps" to  
> him.  Thus, I suppose for this particular file the difference is  
> negligible, if any.
>
> However, as I think about this more, perhaps for 48 kHz sampling  
> rate this is the case, but when you go down in sampling rate to 32,  
> 16, or even 8 kHz, then tapering off the comb filtering effect at  
> the folding frequency or 5/6 of the folding frequency as you did for  
> your single fixed set of the 3-tap filter is likely to become more  
> and more suboptimal.  Take 8 kHz for example.  If you were to apply  
> exactly the same set of 3 taps there, then the comb filtering effect  
> will keep decreasing and reach zero at 4*5/6 = 3.33 kHz.  I believe  
> many (nearly) periodic signals still have fairly strong harmonic  
> structure in the 2 to 3.3 kHz range and can benefit from a different  
> set of 3-tap filters where the degree of comb filtering decreases  
> much slower with increasing frequency than your current set of fixed  
> 3-tap filter.  Therefore, perhaps there is a greater need for  
> multiple sets of 3-tap filters as the sampling rate goes down toward  
> 16 kHz and 8 kHz.
>
> Raymond
>

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 9 Nov 2010 08:56:23 -0600

Hi Raymond,

Re: "are still "legitimate" input signals that CELT or any other audio
codecs sometimes have to deal with in the real world"

I agree ... they are legitimate input signals.

My point was not to use that as a PROXY of the effectiveness of any
comb-filter for the much larger space of non-perfectly-harmonic signals.
It was never that the signals themselves were illegitimate.

The ear is very sensitive to coherent time modulation ... that is what
"fuses tones" such that they do not appear to your ear as "individual
tones" but rather as a "sound" (such as an instrument) with a given
timbre. My ears literally hurt while listening to cymbal crashes in
popular music encoded as low-bit-rate MP3s ... because they just add
"high frequency noise" in the given output frame. Well, I'm sorry, a
cymbal crash has a time based attack (the dammed drumstick striking it)
... and my ears can hear the "absence" of the attach (just uniform
amplitude noise smeared in a particular output frame) ... and I know the
music "isn't real". 99+% of casual listeners don't hear a difference ...
but this musician does ... and finds it bothersome. Virtually every
sound will have some time based modulation in it ... either amplitude,
time or both ... and it is precisely that "coherent time modulation"
that differentiates that "sound" from individual tones of a harmonic
series. This too is well documented in the mid 1980 (e.g., Bregman's
"Auditory Scene Analysis" book).

In general I am in favor of most frequency-based approaches (such as
lapped MDCT transforms) not necessarily because they "sound better"
(most of the time they don't for speech at a given bit rate) ... but
rather their linearity in the low frequency band is generally higher and
admit more-effective subsequent additional processing (e.g., echo
control).

I wish I could work with you guys on this stuff in my day job ...

Best to you Raymond (and nice to meet you Jean-Marc),

Michael

-----Original Message-----
From: Raymond (Juin-Hwey) Chen [mailto:[email-masked]] 
Sent: Monday, November 08, 2010 8:44 PM
To: Michael Ramalho (mramalho); Jean-Marc Valin
Cc: [email-masked]; Jean-Marc Valin
Subject: RE: [codec] A pitch filtering method to improve the mode 1 CELT
codec
Importance: Low

Hi Michael,

I am not sure which kind of spectral subtraction you were referring to,
but there is a very common type of speech enhancement (or noise
suppression) technique that applies spectral subtraction to speech
signals and gets very good noise suppression.  Some of such spectral
subtraction methods don't have sufficient frequency resolution to
resolve the pitch harmonics and only attenuate the spectral valleys
between speech formants.  However, I have seen spectral subtraction
schemes with high frequency resolution (large FFT size) that can resolve
the pitch harmonics and indeed attenuate the spectral valleys between
pitch harmonics, and these schemes indeed get additional noise
suppression compared with the ones that only attenuate the spectral
valleys between speech formants.

It seems that you also agree that there are some real-world audio
signals, whether natural or synthesized, that are truly periodic or
close to truly periodic.  My point is just that as long as this is true
(which you seem to agree), even if they are the minority, then such
periodic or nearly periodic signals are still "legitimate" input signals
that CELT or any other audio codecs sometimes have to deal with in the
real world, and therefore I am not sure I can honestly agree with your
comment that we should not use such signals to test the codec system.

Raymond