Subject: [codec] CELT development update (01/2011)

From: Gregory Maxwell <[email-masked]>
Date: Tue, 25 Jan 2011 15:45:34 -0800

I wanted to send a quick update about the recent technical progress
from the CELT development team since the 0.10 software release and
our last public update about a month ago.
(http://people.xiph.org/~xiphmont/demo/celt/demo.html)

Each point is referenced to commits in our public GIT repository.
(http://git.xiph.org/?p=celt.git;a=shortlog)

==Anti-collapse==
(87efe1df, 21af73eb, 5da938b2d, 01fa3389, etc.)

In CELT, transient frames use multiple smaller MDCT transforms rather
than a single large transform in order to provide better temporal
control of quantization noise. These sub-frames are grouped and
coded together.

A previously outstanding issue for transient frames with the larger
(480, 960) frame sizes was that the codec was able to starve some quiet
sub-frames, particularly at low bitrates. This starvation resulted in
short collapse— falling to zero energy— for a whole subframe
per-band, and often for that subframe in all bands. This was not
prevented by our previously existing sparseness prevention mechanisms
which were primarily targeted at whole band starvation.

The effect is most strikingly demonstrated with spectograms on
castanet samples:
  https://people.xiph.org/~tterribe/notes/celt/spec-anti-collapse-off.png
   -vs-
  https://people.xiph.org/~tterribe/notes/celt/spec-anti-collapse-on.png
(It's not as audible as the images suggest, due to forwards/backwards
temporal masking).

The allocation decision logic is correct in so far as the bits were
best allocated to the louder sub-frames, but the short collapse
produces artifacts of its own.

In order to avoid this issue we implemented decoder-side logic which
tracks where collapse is possible and refills it using noise scaled
using the band energy from prior frames. This resolves the issue but
there is still ongoing fine tuning of the specifics.

==Prevent 'busting' at low rates==
(76469c64)

One of the core designs of the CELT bitstream is the lack of a fixed
sized/fixed position structure. The encoder and decoder dynamically
and continuously adapt to the remaining space in a frame without
signaling according to behavior patterns hardcoded into the
bitstream specification.

Code/Bitstream simplifications last year reduced the ability of the
codec to deactivate some signaling elements as the available space
approached zero.  This is reflected in the 'non-simple' block diagram
by signaling elements with a minimum rate greater than zero bits:
https://people.xiph.org/~greg/celt/full_block_diagram.png
(now outdated by these changes)

When running out of bits the encoder would simply stop coding and the
decoder would be left to pick up the pieces. It turns out that sanely
handling these cases while maintaining synchronization between the
encoder and decoder was at least of comparable complexity to not
permitting this situation in the bitstream at all.

With these changes all the signaling elements can reach zero bits
per frame.

As a side effect some nonsensical bit-streams are no longer possible.
For example, it was previous possible for a stupid encoder to spend all
of the remaining bits in a frame signaling that a band should get more
bits, but then not have any bits left to actually give to it.

It's no longer possible to code sequences like this, which should make
implementation QA somewhat simpler and the elimination of redundancy
(unreasonable signaling combinations) has resulted in some small quality
improvements.

The encoder can now happily encode down to one byte per frame. The
ability to completely eliminate signaling parameters also provided
some quality improvement at ridiculously low rates, and we can give
(barely) intelligible speech with a couple bytes per-frame.

Intelligible speech at zero bytes per frame remains elusive.

==Pre/Postfilter tap-sets==
(dfa847a2, 8d367029, 2ce5c63d, d121260f)

One of the recommendations from Raymond Chen and team was that
the bitstream should allow the selection of multiple filter-tap
configurations for the pitch filters in order to better adapt to
signals with different harmonic shapes.

Spot testing early in the implementation of the pre/post filter didn't
show much improvement from additional tapsets, but now that the PF
implementation is basically complete we reevaluated this.

Using an automated noise to masking ratio test we scanned the
complete range of reasonable 3-tap filters, and many 5-tap filters,
for hundreds of sound clips. Some of the results are visible here:
https://people.xiph.org/~greg/celt/tests/20110116/sqam-taps-256/results.html

While the differences were generally small, there were classes of
samples which worked better with configurations on one end of the
spectrum or another.  In many cases the difference between a single
'general' value and a specialized one were easily large enough to
justify some increased signaling overhead.

We ultimately adopted three configurations, a 5-tap filter at
{0.12954,0.21701,0.30690,0.21701,0.12954} with a ~13kHz rolloff which
is now our default, our prior 3-tap filter {0.26795,0.46410,0.26795},
and a much flatter 3-tap filter {0.1,0.8,0.1}. The default takes one
bit to signal, and the other two options require two bits (per frame).

We implemented some very basic automatic selection logic in the
encoder, but it isn't yet adequate. Fortunately the selection logic
doesn't have bitstream impact and can be refined for years to come.

These testing results show the three options and the automatic selection
compared to no filtering for a variety of samples and bitrates:
https://people.xiph.org/~greg/celt/tests/20110118/tap-diff/results.html

==Numerous small improvements==

* The encoder inter/intraframe decision now checks to make sure that
it doesn't prefer a choice which encodes the energy less precisely due
to running out of bits.  (2293e461)

* The encoder could get stuck in an infinite loop while coding an
extremely loud or quiet difference from the predicted energy.  
(68b8d72e)

* Range coder simplifications were made which reduced the number
of calls to the entropy coder, which is good for performance, and
increases the amount of code shared with Silk.
(845dfa19, 1aaa50d1, 30df6cf3)

* Corrected an encoder bug which caused a mismatch with the decoder
for quiet signals, some other encoder signal processing improvements,
and changes to the energy clamping behavior mean that our floating
point implementation can provide consistent performance for signals
attenuated by more than 138dB without harming compatibility with
fixed point implementations.  (8f02c482, 495114b7)

* The reference implementation now provides a simple comfort noise
generator for discontinuous operation.  (eafd8a7f)

* Miscellaneous stereo quality improvements.
(235c64b9, a387ebfc, 051e044d, fa7215fdb, 79d76a2e3, 620e716b)

* Corrections to the design of the Time-frequency tradeoff filters
for various corner cases.  (dc6d69e64, 2b13401f, ecefde3d)

...Plus many other small improvements, as documented in our version
control system.

We've also been maintaining a short term working todo list on the
Xiph.Org wiki: https://wiki.xiph.org/CELT_TODO

As usual, our realtime technical development discussion takes place in
the IRC channel, #celt on irc.freenode.net. Our development collaboration
is almost exclusively public and we welcome more participation.