Subject: [codec] Suggestions for OggOpus draft

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 24 Sep 2012 13:32:37 -0400

Hi,

I suggest adding these two sections to the OggOpus draft, one that
describes how to handle chaining and one that has recommendations for
encoders. I'm sure there's still more things an encoder can do that I
haven't listed.

	Jean-Marc
diff --git a/doc/draft-terriberry-oggopus.xml b/doc/draft-terriberry-oggopus.xml
index 14f962a..6abe788 100644
--- a/doc/draft-terriberry-oggopus.xml
+++ b/doc/draft-terriberry-oggopus.xml
@@ -951,6 +951,47 @@ An implementation could reasonably choose any of these numbers for its internal
 </t>
 </section>
 
+<section anchor="chaining" title="Chaining">
+<t>
+Like all Ogg streams, Opus streams can be "chained", i.e. contatenated
+together. Although it is possible to simply decode the streams
+independently and concatenate the resulting audio, this results in a
+discontinuity in the audio, which may be audible. Such discontinuity can be
+avoided by obtaining more audio from the stream before the boundary, and
+"cross-fading" that audio with the new stream. The extra audio can be
+obtained in two ways:
+<list style="numbers">
+<t>If there are sufficient usable "trimmed" samples past the end of the audio and those trimmed
+samples represent the actual audio past the end of the file, then these samples
+can be used. These trimmed samples may come from the a</t>
+<t>If trimmed samples cannot be used, use linear prediction on the end of the first
+stream to predict about 2.5 ms of audio at the beginning of the second stream.</t>
+</list>
+
+Opus streams do not explicitly contain sufficient information to tell case 1
+from case 2. One way of deciding whether to use the trimmed samples or
+extrapolated samples is to evaluate the mean squared error between the trimmed
+samples and the first samples of the second stream. This error can be compared
+either to a fixed percentage of the signal's mean square, or to the mean squared
+error between the extrapolated signal and the beginning of the second stream.
+</t>
+</section>
+
+<section anchor="encoder" title="Encoder Recommendations">
+<t>
+This section makes a few recommendations for encoders to produce optimal Ogg
+Opus files. To make gapless files possible, it is RECOMMENDED that encoders
+use linear extrapolation for a duration of at least 2.5 ms at the beginning
+of the file (which can be trimmed using the preskip field). For the end of the
+file, the RECOMMENDED extrapolation duration is at
+least equal to the encoder's look-ahead at the end of the file (7 ms for the
+reference encoder), plus 2.5 ms to facilitate chaining.
+To make the extrapolated samples at the end usable to all decoders, it is 
+RECOMMENDED that these samples be part of the same packet as the last 
+non-trimmed samples (possibly by merging multiple frames in a single packet).
+</t>
+</section>
+
 <section anchor="security" title="Security Considerations">
 <t>
 Implementations of the Opus codec need to take appropriate security

---

From: Ralph Giles <[email-masked]>
Date: Mon, 24 Sep 2012 11:03:02 -0700

> encoders. I'm sure there's still more things an encoder can do that I

Thanks for writing this up.  A couple of questions:

By 'linear prediction' do you mean the LPC-based lapping the opusenc
utility already uses on the final packet?

Is having the encoder supply predicted audio only to help decoders which
don't do a cross-fade, or does it also help preserve fidelity
across the boundary even in the case that the decoder does something to
mitigate the discontinuity?

I guess the issue is that if the encoder is processing a section of a
larger stream, with the intention that the output will generally be
concatenated, like tracks within an album with continuous audio, it
should pad with the actual earlier and later samples. Prediction should
therefore be used to smooth the transition only if adjacent audio isn't
available? Or does that decision depend on the quality of the adjacent
audio. E.g. a large transient would disturb the transition when the next
segment *isn't* the expected one, in shuffle play or a custom playlist,
for example?

These don't seem like they're practices decoders can rely on, so they
need to handle dicontinuities regardless. I think it is useful to give
some suggestions in the draft, though if we can reduce the emphasis a bit.

 -r

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 24 Sep 2012 11:33:42 -0700

> encoders. I'm sure there's still more things an encoder can do that I

The recommendations here are quite complex in terms of container handling
— complex enough that alone their implementation would be comparable to
a whole implementation of a basic encoder / decoder (not including the
codec itself).  This complexity is hidden by the fact that it uses DSP
terminology like 'LPC' in place of what would be a dozen step procedure.

It also doesn't address the multitude of corner cases that can arise: like
what should an encoder do if the minimum 2.5ms padding requirement
means two frames must be trimmed, so they need to be repacketized,
but they have different toc's and so can't be repacketized. I expect many
implementations would end up outputting two packets that need to be trimmed,
which may break some decoders.

Opus-tools's encoder does LPC extrapolation for end-padding, it doesn't bother
with the beginning because doing it at the beginning added latency and was
somewhat messy and I wasn't able to find any originally gapless streams that weren't
gapless without it (which surprised me). 

Since even with only a half-padding encoder and no decoder splice handling 
I was unable to find any gaplesness issues for originally gappless audio, I 
suspect the decoder part of this advice is just not needed.  It's also unlikely
to be implemented: it's complicated, requires DSP coding (admittedly modest)
which will take some implementers out of their comfort zone, and can be
quite difficult to implement in some architectures (e.g. if separate links in a
chain are decoded independently— as is common in existing mediaframe
works).

Recommending encoder extrapolation to prevent pre/post leakage of the
discontinuity to silence seems prudent to me. But I think the decoder
advice is unlikely to be followed especially if written as a recommendation.
If it turns out to actually be important it should probably be a requirement
not a recommendation. If it's not important then I think it should be omitted
because reduced implementation behavior diversity is more important than
theoretical improvements.

Chain itself is not widely supported by implementations of Vorbis. Only basic
audio only tools commonly support it.  I fear adding a bunch of complexity to
handle them the recommended way will further encourage implementations
which opt not to work with chaining at all.

(My opinion would be revised with examples which aren't gapless without it,
or by the existence patches for gstreamer / vlc / and or Firefox, all of 
which will, I think, have problems implementing these recommendations.)