Subject: [codec] Multi-mode frame concatenation

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Thu, 28 Jul 2011 20:48:19 -0400

Given the hum in favor of frame concatenation within the codec, I would
like to propose the following addition for multi-mode concatenation in
Code 3 (circa page 14):

"""
If M == 49, p == 0, and v == 1, then the packet is a "multi-mode packet".
 The decoder MUST read the next byte to determine the true values of M, p,
and v.  Decode shall then continue as normal, except that each frame shall
be interpreted as a Code 0 packet.  Decoders MUST NOT continue decoding
the packet if any frame indicates a code other than 0.
"""

This design has zero overhead when not used, and should be as simple as
possible to implement and test.  The implementation lives entirely inside
the codec, so its complexity is no more a problem for the user than is the
complexity of the CELT collapse-prevention system.  The described mode is
currently invalid, so it does not represent a significant change; all
previously valid bitstreams are unaffected.

This design improves security by ensuring that a gateway can always
convert any Opus stream into a CBR stream at a constant packet rate.
Currently, this is not true due to mode switching boundaries, resulting in
information leakage (e.g. whether someone is talking or still listening to
the hold music.)

This design improves interoperability by ensuring that anyone who requires
such functionality is not tempted to implement it in their own layer.

This design improves bitrate efficiency when using Code 1-3 by avoiding an
additional packet's overhead (e.g. RTP+UDP+IP) on every mode switch.

This design allows encoders to switch modes freely, with limited and
predictable overhead.  In the present system, each mode switch is
associated with the unpredictable and potentially large cost of forcing a
packet boundary.  Making this cost more predictable allows an encoder to
switch modes more frequently (i.e. employ RDO) in order to improve
compression, for example by varying the bandpass or framesize based on
signal content.

This design is particularly beneficial in certain advanced applications
such as a multichannel extension where a group of packets must all start
and stop on a shared time boundary.  Without this design, all mode changes
in all channels must occur on that boundary as well.

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Thu, 28 Jul 2011 23:25:15 -0400

On 07/28/2011 08:48 PM, Benjamin M. Schwartz wrote:
> possible to implement and test.

Greg Maxwell pointed out that since implementation complexity is likely to
be the focus of a needless argument, I might do better with some "running
code" to assess the real complexity.  I'm just a volunteer, so I can't
commit much time, but I will try to draft a patch to the reference decoder
that implements this design.

--Ben

P.S. Of course, I would love for someone to beat me to it.

---

From: Ralph Giles <[email-masked]>
Date: Fri, 29 Jul 2011 10:12:09 -0700

> the packet if any frame indicates a code other than 0.

To clarify, the spec limits code 3 packets to 48 frames, so M > 48 is
currently rejected, and we have (2^4 - 2 = 14) extensions to the Opus
framing we can signal by allowing additional values here while
maintaining backward compatibility with draft-ietf-codec-opus-07.

Your proposal for M = 49, p = 0, v = 1, or Opus frame count byte = 177
is to set the multimode extension flag in the decoder state, and then
read the next byte as the Opus frame count byte. Then by "each frame
shall be interpreted as a code 0 packet" you mean the frames are read
according to the second frame count byte, with lengths if v = 1 and
without if v = 0, but instead of reading undecorated frames, the
decoder will read a toc byte at the start of each frame, like so:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |1|1|s| config  |     49    |0|1|     M     |p|1|  Pad length   :
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     : N1 (1-2 bytes): N2 (1-2 bytes):     ...       :     N[M-1]    |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0|0|s| config  |                                               |
     :               Compressed frame 1 (N1 bytes)...                :
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0|0|s| config  |                                               |
     :               Compressed frame 2 (N2 bytes)...                :
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                                                               |
     :                              ...                              :
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0|0|s| config  |                                               |
     :                     Compressed frame M...                     :
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     :                     Padding (Optional)...                     |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      A VBR multimode packet

In any case, multimode packets would still be limited to 48 frames,
it's just that each would have a TOC byte so mode changes could be
signaled without forcing a packet boundary. Is that accurate?

 -r

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 29 Jul 2011 13:31:13 -0400

> maintaining backward compatibility with draft-ietf-codec-opus-07.

For some definitions of backwards compatibility, yes.  The draft's text
indicates that this extensibility is deliberate.  I'm not sure where you
get 14; I get (64-49)*4 = 60, at a minimum.

> decoder will read a toc byte at the start of each frame, like so:

This is all extremely accurate.  I had originally intended (following the
language in your diagram) for Compressed frame 1 to consist of N1-1 bytes,
so that its entirety (including the TOC byte) had size N1.  However, I now
see that your version is superior, as otherwise it would be impossible to
represent the max-size frame (1275 bytes, or 1276 with the TOC byte).

---

From: Ralph Giles <[email-masked]>
Date: Fri, 29 Jul 2011 10:35:12 -0700

Tim pointed out that there are already a config and stereo bits from
TOC byte at the start of the packet, so one of the compressed frames
doesn't need a separate TOC byte. Ben's proposed text should be
amended to show that. I think.

The packet-initial TOC byte could apply the first packet in this case,
which is easier to read in the spec, or the last one, which might be
easier to implement, since the last packet also doesn't have a
signaled length.

It's also possible to just discard the initial stereo flag and config
if the extra bandwidth is judged worth the complexity savings.

 -r

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 29 Jul 2011 13:39:08 -0400

> if the extra bandwidth is judged worth the complexity savings.

This was my intent.  Without code, I think complexity is hard to judge, so
I don't yet have a strong opinion.

--Ben

---

From: Ralph Giles <[email-masked]>
Date: Fri, 29 Jul 2011 10:53:02 -0700

> get 14; I get (64-49)*4 = 60, at a minimum.

I was doing 63 - 48, or using the lower four bits of M when the upper two
bits are set, with b110000 taken for the maximum value of 48, which I
overcounted in the previous email. I now get 15 extension options.

Do you mean using lower values of M when the config field indicates
the frame has a longer duration?

> represent the max-size frame (1275 bytes, or 1276 with the TOC byte).

Ok, good.

>  don't yet have a strong opinion.

The Opus framing tries hard to save bytes elsewhere, so it seems
reasonable to me. I look forward to your implementation experience. :)

Cheers,
 -r

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Fri, 29 Jul 2011 14:13:38 -0400

> the frame has a longer duration?

Nope.  I mean that once M is invalid, the other two bits are also
available for signaling.  I would not regard M=49, p=1, v=0 (byte==113) as
a multi-mode packet indicator, so it's still available for something else.

--Ben

---

From: Ralph Giles <[email-masked]>
Date: Fri, 29 Jul 2011 11:21:07 -0700

> a multi-mode packet indicator, so it's still available for something else.

Ah, I see. (count & 0xCF) > 0, basically. Thanks for explaining.

 -r

---

From: Jonathan Lennox <[email-masked]>
Date: Fri, 29 Jul 2011 14:37:21 -0400

> a multi-mode packet indicator, so it's still available for something else.

It would probably be a good idea to explicitly reserve some extension values for the external container, analogous to the way that H.264 reserves NAL Unit Types 24-31.  This ensures that the RTP payload format can both support a mode where it directly places Opus packets into an RTP payload, and also (if necessary) a mode where it add some additional framing or other transport-related data around them.

--
Jonathan Lennox
[email-masked]

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 29 Jul 2011 15:34:46 -0400

Good idea. We were thinking of reserving the first two bytes "Op", which
is an invalid given the framing restrictions.

	Jean-Marc

On 11-07-29 02:37 PM, Jonathan Lennox wrote:
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
> 
>