Subject: [codec] Frames to Packets in draft-ietf-codec-opus-07?

From: "Christian Hoene" <[email-masked]>
Date: Mon, 25 Jul 2011 10:23:55 -0400

Hello,

I was wondering why the draft-ietf-codec-opus-07 draft has feature to
concatenate multiple codec frames to packets. At the same time,
draft-spittka-payload-rtp-opus-00 does not support to place multiple codec
entities in on RTP packet.

As a consequence, the maximal size of an Opus packet is limited to 120ms.
Especially, on low rate links this might be too low. Also, on links with a
high packet overhead (such as IPsec or IEEE 802.11b WLAN), this limits seems
to be too.

I would suggest to either

1) Remove the concatenation from draft-ietf-codec-opus-07 and add it to
draft-spittka-payload-rtp-opus-00.

Or

2) Add a second concatenation feature to the payload spec.

With best regards,

 Christian

----------------------------------------------------------------------------
-----
Dr.-Ing. Christian Hoene, University of Tübingen, Computer Science, Chair of
Communication Networks, Research Group Interactive Communication Systems
(ICS)
Sand 13, 72076 Tübingen, Germany, Tel +49 7071 2970532,
www.net.uni-tuebingen.de

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Mon, 25 Jul 2011 13:56:55 -0400

> entities in on RTP packet.

I think we are in agreement that for many packet transports, a single
frame per packet (max duration of 20~ms for general audio, minimum of
2.5ms or 400 pps) would be inappropriate, and so a concatenation scheme is
required.  The question is then: where should the concatenation scheme be
specified?

The motivation for specifying concatenation inside the codec is to promote
interoperability and adoption by reducing the total amount of engineering
work required to implement Opus.  This happens in two ways:

1. Avoiding duplicative software work

Given the structure of existing networked audio software, I expect that
software using Opus will typically rely on a pre-existing Opus library,
and there will be only a small number of widely used Opus implementations.
 However, there will likely be many implementations of the Opus-RTP
specification, due to engineering integration requirements.  As such, we
can reduce implementation effort by minimizing the complexity of the RTP
specification.

2. Avoiding duplicative specification work

Opus will be distributed over many transports other than RTP, and a
concatenation scheme may be desirable in each case.  Specifying
concatenation inside the codec avoids the need to specify a new
concatenation scheme for every transport.  Having a single concatenation
scheme should therefore accelerate adoption, minimize confusion when
moving packets between transports, and minimize the amount of software
required to make Opus work.

Personally, I've found H.264 to be an example of how specifying framing
outside the codec can create problems. H.264's packet/framing schemes (the
"network abstraction layer", "access units", bytestream alignment, etc.)
are specified outside the codec, so that they must be handled by the
stream processing layer.  Correctly distinguishing and converting between
the various possibilities is so confusing that, until the latest release
last month, gstreamer's H.264 parser could not correctly depayload H.264
from the popular AVCHD format.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 25 Jul 2011 15:55:51 -0700

> draft-spittka-payload-rtp-opus-00.

We have to this point bent over backwards to keep the bitstream from 
changing since we declared a (soft) freeze in February, because people 
specifically asked for this in order to begin testing. While the working 
group can still agree to change it if a _serious_ issue arises with the 
format, I personally do not think this qualifies.

---

From: Ralph Giles <[email-masked]>
Date: Tue, 26 Jul 2011 11:01:49 -0700

> entities in on RTP packet.

I believe the codec authors have described some of their reasoning for
the internal framing. However, it sounds like your concern is the
presence of framing at the codec level, rather than at the RTP payload
level.

There is precedent for this in previous specifications.
http://tools.ietf.org/html/rfc3551#section-4.4 "Guidelines for
Frame-Based Audio Encodings" says:

  "Frame-based encodings encode a fixed-length block of audio into
   another block of compressed data, typically also of fixed length.
   For frame-based encodings, the sender MAY choose to combine several
   such frames into a single RTP packet.  The receiver can tell the
   number of frames contained in an RTP packet, if all the frames have
   the same length, by dividing the RTP payload length by the audio
   frame size which is defined as part of the encoding.  This does not
   work when carrying frames of different sizes unless the frame sizes
   are relatively prime.  If not, the frames MUST indicate their size.

   [...]

   All frame-oriented audio codecs SHOULD be able to encode and decode
   several consecutive frames within a single packet. [...]"

Opus can use variable length encoded frames, so the clause that
_frames_ must indicate their size, so that the codec can "decode
several consecutive frames within a single packet" applies. The
referent here is frames and codecs, not the payload format, so I don't
think we're breaking new ground here by having some framing at the
codec level.

I would also point out that the opus codec draft is nearly finished.
Continuing to specify framing there, rather than adding complication
to the newer RTP draft, is the quicker path to standardization of both
levels.

FWIW,
 -r

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 27 Jul 2011 15:21:02 -0400

On 07/27/2011 03:10 PM, Jonathan Lennox wrote:
> Based on my reading of the Opus spec, it looks like splitting should be possible, relatively trivially.  Aggregation, however, is only possible if the packets to be aggregated have the same value for the Opus TOC byte.

That is a correct reading of the Opus spec as of Opus-02 and later.  Prior
to Opus-02, the spec also described a packet mode that could contain
frames of different types, allowing arbitrary aggregation.  I believe it
was removed because the authors expected that it would be rarely used, and
removing it allowed them to improve the efficiency of more common cases.

--Ben

---

From: Matthew Kaufman <[email-masked]>
Date: Thu, 28 Jul 2011 19:39:17 -0400

In response to comments at the mic at today's meeting:

Would it be possible to add an appendix or create an additional document that describes how a middlebox could detect mode changes and avoid concatenating in these cases?

Matthew Kaufman

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 28 Jul 2011 16:53:22 -0700

> Would it be possible to add an appendix or create an additional document
> that describes how a middlebox could detect mode changes and avoid
> concatenating in these cases?

This is basically as simple as examining the first byte of the packet. 
But if you have some example text I'd be happy to make an appendix for 
it (and possibly other related issues).

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Fri, 29 Jul 2011 09:57:08 -0400

> it (and possibly other related issues).

If the PAYLOAD group ends up defining a multi-frame packetization 
mechanism for RTP encapsulation of Opus, then a middlebox could just use 
that mechanism and not have to care about whether the encoder changed 
modes between frames.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: [email-masked] | SIP: [email-masked] | Skype: kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at www.digium.com & www.asterisk.org

---

From: Stephen Botzko <[email-masked]>
Date: Fri, 29 Jul 2011 10:14:24 -0400

I agree that this can be handled in PAYLOAD / packetization.

While I see some value in ensuring that OPUS can run over non-RTP
transports, I am thinking that we really don't need to re-create all the
packetization features available in RTP inside of OPUS elementary streams.
At some point the level of return on the investment of time drops off - I am
thinking we are reaching that point.

Stephen Botzko

On Fri, Jul 29, 2011 at 9:57 AM, Kevin P. Fleming <[email-masked]>wrote:

> If the PAYLOAD group ends up defining a multi-frame packetization mechanism
> for RTP encapsulation of Opus, then a middlebox could just use that
> mechanism and not have to care about whether the encoder changed modes
> between frames.
>
> --
> Kevin P. Fleming
> ______________________________**_________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/codec<https://www.ietf.org/mailman/listinfo/codec>
>

---

From: Jonathan Lennox <[email-masked]>
Date: Wed, 27 Jul 2011 15:10:34 -0400

> required.  The question is then: where should the concatenation scheme be

Does the Opus framing allow packets to be split and/or aggregated by a middlebox along frame boundaries, without requiring transcoding?  (The use case for splitting is to forward traffic to a box with a lower maxptime than the stream you're receiving, and the use case for aggregation is to reduce the RTP header overhead at the cost of some additional delay.)

I tried to read the Opus spec to answer my question, and I think I came up with the answer, but I wanted to confirm this with the Opus experts.

I get the impression that the TOC byte is expected to change only relatively infrequently in most encoding modes, in which case this would be fine -- there may be transient packets which couldn't be aggregated, resulting in occasional "short" packets, but most packets could be aggregated.  However, if an encoder might change its TOC frequently, this could be more problematic.

If the (Payload) WG is okay with this limitation (and the limitation of a maximum of 120 ms packets), I think the internal-framing-only is okay; if not, we might need external framing as well. (I appreciate that you don't want to change the Opus bitstream proper, at this point.)

--
Jonathan Lennox
[email-masked]

---

From: Jean-Marc Valin <[email-masked]>
Date: Sun, 31 Jul 2011 21:26:47 -0400

Hi Jonathan, (sorry, just received your post now)

On 11-07-27 03:10 PM, Jonathan Lennox wrote:
> Based on my reading of the Opus spec, it looks like splitting should
> be possible, relatively trivially.  Aggregation, however, is only
> possible if the packets to be aggregated have the same value for the
> Opus TOC byte.

Correct.

> I get the impression that the TOC byte is expected to change only
> relatively infrequently in most encoding modes, in which case this
> would be fine -- there may be transient packets which couldn't be
> aggregated, resulting in occasional "short" packets, but most packets
> could be aggregated.  However, if an encoder might change its TOC
> frequently, this could be more problematic.

I don't really see the TOC byte changing frequently. I would expect that
for many/most uses, the TOC would never change during an RTP session.
For other uses, I'd be surprised to see it changed more than once every
5-10 minutes. The only potentially problematic case (when it comes to
aggregation) would be contents that has music and speech that alternates
frequently.

> If the (Payload) WG is okay with this limitation (and the limitation
> of a maximum of 120 ms packets), I think the internal-framing-only is
> okay; if not, we might need external framing as well. (I appreciate
> that you don't want to change the Opus bitstream proper, at this
> point.)

I don't see these limitations as being a problem, but if they really are
(in practice, not in some theoretical scenario), then I'd rather change
this in the bit-stream than have RTP implementations mess it up on a
regular basis (and having non-RTP implementations do it differently).

Cheers,

	Jean-Marc