Subject: [codec] draft-ietf-codec-oggopus: attached pictures

From: Mark Harris <[email-masked]>
Date: Mon, 25 Aug 2014 00:01:49 -0700

Opus does a fantastic job at its primary goal of compressing audio,
including speech, offering a lower bitrate and latency than other
similar codecs.  This makes it an obvious choice for use in RTP and
many other uses.  However when it comes to storing music, podcasts,
and audiobooks in a file, there are additional requirements beyond
just a great codec, such as good support for attached pictures and
other metadata.

Attached pictures (also known as album art or cover art) are images
(usually in JPEG or PNG format) that are related to the audio, and are
often shown by players while the audio is playing or to help the user
choose the file to be played.  In most audio formats pictures can be
categorized into a number of picture types, with multiple pictures
allowed for most types.

Container formats like Matroska have their own format for storing
attached pictures and other metadata in the container, however Ogg
does not provide this and requires any such data to be encoded within
the packets of a media stream.

Attached pictures are not mentioned at all in the current Ogg Opus
draft (draft-ietf-codec-oggopus-04), which simply defers to the
vorbis-comment specification for the format of metadata, with a few
specific differences such as new gain tags.

When Ogg Vorbis was originally specified, it provided a way of storing
a list of text comment tags for things like the title, artist,
license, and copyright of the work, but no way to attach pictures or
binary data.  Because other formats allow attached pictures and users
wanted to add them to Ogg Vorbis files as well, a method of encoding
pictures in a text comment tag using base64 encoding was devised,
which is documented at https://wiki.xiph.org/VorbisComment#Cover_art .
This allowed pictures to be attached to Ogg Vorbis files without
impacting compatibility, but the downside is that attaching pictures
to an Ogg Vorbis file adds about 134% of the picture size to the size
of the file, compared to just over 100% of the picture size for
competing formats including MP3/ID3, iTunes/Quicktime MPEG-4, FLAC,
and Matroska-based formats.  Also, because comments are always at the
beginning of the file and can appear in any order, it is not easy to
skip the pictures and get to the audio if there are multiple large
pictures, at least when there are some other comments that may be of
interest.

Unfortunately the current Ogg Opus draft has the same issue and has no
way of attaching pictures as binary data, thus requiring the use of
the same base64-encoded comment workaround that was used for Ogg
Vorbis.  This is unbefitting of a compressed file format, which has a
primary purpose of reducing file size, and it would be unfortunate if
Ogg Opus was standardized with the same, easily-addressed limitation.
It is difficult to take a purportedly compressed file format seriously
when the first 400 KB or so of the file is a long base64 string of
text that takes significantly more space than the corresponding data
that was provided to the encoder.

I see a few different ways that this could be addressed:

 (1) Specify that comments may contain either UTF-8 or binary data,
according to some rule.  For example, if the name of the tag begins
with "@" then its value is binary data and not intended to be
displayed as text, and is otherwise UTF-8.  There is no technical
issue with storing binary data since each comment is already preceded
by a length field, but a tag name beginning with "@" would be a signal
to applications that the value should not be assumed to be UTF-8 or
displayed as text.  Pictures could then be attached using a newly
defined tag such as "@PICTURE", with a value that is the same as the
binary format used in FLAC (and the same as Ogg Vorbis before base64
encoding).

     Alternatively, ordinary tag names could be used and the value
itself could be used to indicate that it is non-UTF-8 binary data, for
example by including a null character prefix before the binary data,
or a 0xff byte prefix (which is not valid UTF-8), or a null byte
delimiter in place of "=".  However using distinct tag names seems
cleaner.

 (2) Immediately following the comments, in the same packet, allow a
picture count followed by a length and binary data for each picture,
similar to the way that comments are stored but separated from them.
Separating pictures from comments allows a more appropriate format to
be used to encode them, makes it less likely that tools will treat the
data as text, and also makes it easier to skip all attached pictures
entirely when they are not needed.  Tools based on older versions of
the draft would see anything after the original comment section as
comment padding and ignore any attached pictures.  The content of each
picture would be the same as it is for FLAC, and the same as Ogg
Vorbis without the added base64 encoding or "METADATA_BLOCK_PICTURE="
prefix.

 (3) Specify a way to store attached pictures in the file outside of
the Opus stream.  This is the way that containers such as Quicktime
and Matroska work, but to do that with Ogg would require another
stream that contains the pictures, since the Ogg container itself does
not provide metadata.

 (4) Do not address this.  Attached pictures must be base64-encoded
and written in a comment.  If users complain, recommend use of
Matroska or another container that does not have this issue.

Any thoughts on these or other ideas?

I have created a proof-of-concept implementation of (1) and (2) based
on opus-tools, at:
https://github.com/mark4o/opus-tools/tree/binary_comment (1),
https://github.com/mark4o/opus-tools/tree/pic_section (2).

 - Mark

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 25 Aug 2014 15:45:53 -0700

With my individual hat on...

> specific differences such as new gain tags.

The theory was that it wasn't necessary to re-invent the wheel here. 
Applications which already support Vorbis-style album art can add 
support for Opus album art with as little as one line of code (because 
they can, and do, share comment-parsing routines among 
Vorbis/Theora/Speex/etc.). This is not just a matter of engineering 
simplicity, but also speed of adoption.

> displayed as text, and is otherwise UTF-8.  There is no technical

This has backwards-compatibility problems. Is this unique to Opus or 
would we expect other formats to adopt the same strategy? (see above 
about code-reuse). What would we do about existing tags that might 
already start with an '@'? How would we represent a field name that we 
want to start with an '@' (for whatever reason)?

We could also simply use a more compact form of embedding binary data in 
UTF-8, but those have higher implementation complexity (and two 
encodings to choose from is higher still).

> delimiter in place of "=".  However using distinct tag names seems

Existing, naive tools are likely to mis-handle all of those things 
(e.g., copying the tag as the empty string, rejecting it as invalid and 
removing it, rejecting the whole comment packet as invalid, etc.).

> picture count followed by a length and binary data for each picture,

This is more practical, though we probably want to leave room to define 
additional types of binary data later (e.g., use the FLAC 
METADATA_BLOCK_HEADER and/or BLOCK_TYPE values). Again, I'm interested 
in the issues of code re-use for other formats. Notably, this is hard to 
make compatible with Vorbis because it encodes one non-zero "framing 
bit" after the main comment data (and mandates it be checked).

> stream that contains the pictures, since the Ogg container itself does

This can already be done by specifying a relative URL instead of actual 
picture data (by setting the mime type appropriately). This has the 
advantage that a single file can be used for multiple tracks, which 
provides considerably more space savings than avoiding BASE64 encoding 
as soon as you have more than 1 track from the same album. Support for 
and usage of this approach is almost non-existent. Whether that's 
because the convenience to users of having everything embedded in one 
file is worth the extra space or because of the inconvenience to 
application authors of having to (securely) deal with external files, or 
the combination of both of those effects, I don't know.

> Matroska or another container that does not have this issue.

We could also choose not to address this _here_, and instead continue to 
defer to the existing vorbis-comment documentation. That wouldn't block 
publication of this draft, and give us time to get implementation 
feedback from the authors of media players and tools who would have to 
deal with this change. I think this is the approach that I personally 
prefer.

---

From: Ron <[email-masked]>
Date: Tue, 26 Aug 2014 22:54:01 +0930

On Mon, Aug 25, 2014 at 03:45:53PM -0700, Timothy B. Terriberry wrote:
> Applications which already support Vorbis-style album art can add support
> for Opus album art with as little as one line of code (because they can, and
> do, share comment-parsing routines among Vorbis/Theora/Speex/etc.). This is
> not just a matter of engineering simplicity, but also speed of adoption.
> 
> This has backwards-compatibility problems. Is this unique to Opus or would
> we expect other formats to adopt the same strategy? (see above about
> code-reuse). What would we do about existing tags that might already start
> with an '@'? How would we represent a field name that we want to start with
> an '@' (for whatever reason)?
> 
> UTF-8, but those have higher implementation complexity (and two encodings to
> choose from is higher still).
> 
> Existing, naive tools are likely to mis-handle all of those things (e.g.,
> copying the tag as the empty string, rejecting it as invalid and removing
> it, rejecting the whole comment packet as invalid, etc.).
> 
> METADATA_BLOCK_HEADER and/or BLOCK_TYPE values). Again, I'm interested in
> the issues of code re-use for other formats. Notably, this is hard to make
> compatible with Vorbis because it encodes one non-zero "framing bit" after
> the main comment data (and mandates it be checked).
> 
> advantage that a single file can be used for multiple tracks, which provides
> considerably more space savings than avoiding BASE64 encoding as soon as you
> have more than 1 track from the same album. Support for and usage of this
> approach is almost non-existent. Whether that's because the convenience to
> users of having everything embedded in one file is worth the extra space or
> because of the inconvenience to application authors of having to (securely)
> deal with external files, or the combination of both of those effects, I
> don't know.
> 
> publication of this draft, and give us time to get implementation feedback
> from the authors of media players and tools who would have to deal with this
> change. I think this is the approach that I personally prefer.

I think that sounds like the best approach to me too.

There are real things that can be improved there, but none of them are
actually specific to the Opus Ogg mapping, and any solution to them
that gets buy-in from implementors does seem likely to be something
they'd also want to reuse more widely with other mappings as well.

Whether we'd eventually want to bring that work back here to formalise
a new proposed standard for "Ogg metadata" is a separate question,
but either way it seems like a work that could (and should) occur
independently of OggOpus, and that could apply to many other things
in addition to it.  The important question for this draft is have we
adequately defined the needed codec mapping to encapsulate it in Ogg.

  Ron

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Tue, 26 Aug 2014 19:34:38 -0400

> adequately defined the needed codec mapping to encapsulate it in Ogg.

Some thoughts:

1.  If the last option is preferred by most, then let's get to work on
canvassing the existing players that we think are best suited to this,
rather than letting it languish and becoming forgotten.  In other words,
let's make this our own itch and scratch it.

2.  Isn't there already a specification for Kate that allows usage of
images in Ogg?  Hackish, I know, but that might lend some guidance on a
way (good or bad?) to encode binary image data.

3.  Does Ogg Skeleton have any mechanism or "spot" for this kind of
binary image data?

Personally, I don't think the 33% inflation due to base64 encoding
aspect is that big of a deal.  I'd be interested to know if there are
cases where the image data becomes large in comparison to the audio data
in an OggOpus file, and that might make it more relevant to pursue a
more optimal solution.

-- 
Libre Video
http://librevideo.org

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 26 Aug 2014 16:54:35 -0700

> let's make this our own itch and scratch it.

At a minimum, I planned to bring it up with VLC/libav/FFmpeg/etc. 
developers at VideoLAN Dev Days next month.

---

From: Mark Harris <[email-masked]>
Date: Fri, 29 Aug 2014 22:38:14 -0700

> not just a matter of engineering simplicity, but also speed of adoption.

Use of vorbis-comment is fine for its intended purpose, and to be
clear I do not suggest that something else be used for short,
text comments such as TITLE, ALBUM, or ARTIST.  The referenced
vorbis-comment document explicitly states that it is meant for
short, text comments, not arbitrary metadata.  You don't use a
hammer in place of a screwdriver just because you have a hammer
handy and have found that it can get the screw into the hole.  I
can understand the reasoning behind trying to shoehorn attached
pictures into comments for Ogg Vorbis, as the Ogg Vorbis encapsulation
format was designed before attached pictures were common and it had
already been finalized for a number of years; this was one of the
few options available.  However it is clear that vorbis-comment was
not intended for this kind of data and that the efficiency of this
encoding for attached pictures is significantly worse than what is
available with other common audio file formats.

As for sharing code, native FLAC also uses Vorbis-style comments
for short text tags and a binary format for attached pictures; in
fact it is the same binary format that is being suggested.  I don't
see why an application supporting Ogg Opus and FLAC, or just Ogg
Opus, should need to include code for base64 encoding and decoding
attached pictures just because that was needed by a different older
format.  If an application supports both Ogg Vorbis and Ogg Opus
then it will need to support both the old and new styles of gain
tags as well as the old and new styles of picture tags.  The format
of the new picture data is the same as the old except that the
base64 encoding is no longer needed, so it is easy to support both
just as it is easy to support both gain tags.  It is also easy to
distinguish the old and new styles without having to know which to
expect.  So code sharing should not be an issue.

> code-reuse).

Like the new gain tags, applications could decide to support the
new tags with older formats, although I wouldn't expect the new
tags to be recommended for older formats that already have another
well established standard.  On the other hand, if there are other
reasons to create a new incompatible version of the older standard
(e.g. TransOgg :)) then it would make sense to recommend use of the
new tags in the new version.

>              What would we do about existing tags that might already start

The vorbis-comment specification explicitly allows new unregisterd
"nonstandard" tag names, and does not offer any system of ensuring
unique tag names, so this is an issue with the design of the
vorbis-comment specification in that it is always possible that
someone else has used the same tag name for another purpose.  If
this is considered to be a serious compatibility issue then the
design of vorbis-comment is seriously flawed, because no one would
then be able to introduce new tags since they may already be in use
with a different purpose or syntax.

If there is known existing usage of tag names beginning with @,
with text values, then a different indicator should be chosen for
binary tags.  However even if such tags exist I would not expect
it to be an issue, as this should have little effect other than the
way in which these tags, if unrecognized, are displayed and edited
by default.  I would expect that most applications do not do anything
with unrecognized tags and should be completely unaffected.  Any
ability to copy and delete tags should also also not be impacted.

> choose from is higher still).

Even writing the picture data in binary form, attaching pictures
in Ogg still adds more overhead than any other non-Ogg format that
I tried (results below), primarily due to the Ogg lacing for the
picture data.  However, despite being dead last the binary format
is close enough to be competitive.

  Overhead of attaching 300 KB PNG image:
    MP3/ID3:                 + 0.04% of image size
    Matroska:                + 0.06% of image size
    FLAC:                    + 0.09% of image size
    M4A/iTunes:              + 0.32% of image size
    Ogg Opus base64 comment: +33.95% of image size
    Ogg Opus binary comment: + 0.46% of image size

If the format is not even competitive with other popular encapsulation
formats in common feature areas at the time of standardization,
what that tells me is that it is not ready for standardization yet,
as that will just lock in the bad decisions.  If we were talking
about a difficult problem that required serious tradeoffs in order
to address then it may be worth just acknowledging that other
encapsulation formats are better suited to such uses, however that
would be unfortunate for something that is easily addressed with
no tradeoffs required.

What is really nice about the Opus codec is that it does not require
any compromise; it is freely available and offers the best quality
available for a wide variety of applications from low-latency
low-bitrate speech to transparent quality multi-channel music, which
previously required several non-free codecs and careful codec
selection in order to choose something that will work well for all
use cases.  A standardized encapsulation format that similarly
required no compromise for all of the common use cases would make
an appropriate match.  An encapsulation format with two to three
orders of magnitude more overhead than anything else for a commonly
used feature that can account for a significant portion of actual
files would be an especially poor choice.

> the main comment data (and mandates it be checked).

A block type sounds great.  I was just trying to minimize the changes
by keeping the binary format identical to what Ogg Vorbis passed to the
base64 encoder.

Although something similar could be done for Ogg Vorbis, this is
probably not the place to discuss that.  Anyone who has implemented
the current Ogg Opus draft knew that they were implementing a draft
version of the standard that is subject to change, but that is not
the case for Ogg Vorbis.

> deal with external files, or the combination of both of those effects, I

While that may be useful in some cases, I think there is still a
strong need for the ability to keep everything related to the track
in a single file.  That makes it easier to copy to a phone or other
playback device, easier to make available for download or sharing,
easier to ensure security and privacy, and less likely to break when
files are moved or copied.  To be clear I was referring to some way of
attaching the pictures to the same file.

> change. I think this is the approach that I personally prefer.

If this is not addressed prior to the standard being finalized, it
becomes very difficult to add later.  Ogg Vorbis is still much worse
at this than other formats even after so long because it was not
part of the design, and a kludge was needed in order to add it
afterwards without affecting compatibility.  Now there is an
opportunity to address this in a cleaner manner for Ogg Opus
encapsulation.  This opportunity should not be squandered and the
same mistakes repeated once again.

> way (good or bad?) to encode binary image data.

It looks like Kate can support timed text and timed images, but
not attached pictures that are not associated with particular times.

> 3.  Does Ogg Skeleton have any mechanism or "spot" for this kind of

I do not see anything related in Ogg Skeleton.

> I'd be interested to know if there are

I just checked some audio files that I have locally on this computer.
Considering only commercially sold music, the ratio of attached
picture size (in binary form) to compressed audio size was as high
as 26.28%.  (This was on a 48 second MP3 interlude track that is
part of an album, with a single attached picture.)  The largest
attached picture I found was a 1.9 MB PNG, although that was on a
podcast episode, not commercially sold music.

If it is expected that Ogg Opus will be taken seriously as a
compressed audio format then it would do well to avoid adding 34%
to these sizes for no added benefit.

 - Mark

---

From: Ron <[email-masked]>
Date: Sat, 30 Aug 2014 16:57:45 +0930

On Fri, Aug 29, 2014 at 10:38:14PM -0700, Mark Harris wrote:
> becomes very difficult to add later.

I don't see why you think this is so?  Ogg was designed from the outset
to be easily extensible to carry and multiplex any sort of data.

There are all sorts of things that people might want to append to an
Opus track, and we clearly can't define, let alone imagine, all of
them here and today, since some of them haven't even been invented yet.

If you think something is missing from our ability to generically
extend this to include other 'non-opus' data at a later date, then
perhaps we ought to consider that.  But I think getting bogged down
on finalising the codec mapping to Ogg, because we don't have things
that weren't part of the charter of this group, that nobody has wanted
enough to raise before now, and that there are no published and tested
implementations of, is scope creep that ought to be a separate matter
of study, with a separate proposed standard as a properly generic
extension to RFC 3533 and/or RFC 5334.

Otherwise where does this stop?  We can't give people a published
Ogg mapping for Opus until we have optimal support for embedded
pony holograms?  I'm sure someone somewhere thinks that's a very
important feature.  Like alternate image encodings though, I don't
see why a standard for that can't be created later that will work
in a standard way for anything in an Ogg container, not just Opus.
Which seems like a far better outcome than kludging something in
here at the last minute, doesn't it?

What are we missing that makes that impossible?

  Ron

---

From: Mark Harris <[email-masked]>
Date: Sat, 30 Aug 2014 08:05:56 -0700

> to be easily extensible to carry and multiplex any sort of data.

Yes, arbitrary binary data is allowed by Ogg, and none of my proposals
involved changes to Ogg.  This is about Ogg Opus files as defined by
the draft draft-ietf-codec-oggopus-04.  The current version of the
draft does include some binary metadata within an Ogg Opus stream,
such as channel mappings, but does not provide any mechanism for
adding additional metadata to the Ogg Opus stream in a binary format.
Furthermore the current draft does not permit an Ogg Opus file to
contain any other multiplexed streams (section 10).

> What are we missing that makes that impossible?

If the draft included a way to add arbitrary binary metadata to the
Ogg Opus stream, or allowed an Ogg Opus file to contain attached
pictures that are stored in another multiplexed stream, then the
details of the picture format could be specified outside of this
specification.

 - Mark

---

From: Ron <[email-masked]>
Date: Sun, 31 Aug 2014 01:51:19 +0930

> contain any other multiplexed streams (section 10).

Am I reading a different Section 10 to you?
I don't see anywhere that it prohibits that, and it explicitly defers
to the other RFCs which define how that should be done.

All it says is they are 'not strictly "Ogg Opus files"' - in the same
way that vorbis muxed with theora (or anything else) are "not strictly
Vorbis files".  You use the same mapping as defined here for the Opus
streams, and include whatever else you want along with them.

People have already been doing this for video with Opus.

> details of the picture format could be specified outside of this

The OggOpus draft refers to the RFCs which define how to do that.
Which is why I'm suggesting that one option if you want still
pictures, is to define a generic Ogg mapping for them - which could
then be used with any other format that also has an Ogg mapping,
including this one.

There may be better ways to do that, but any optimal way should not
be Opus specific, it should be reusable for any Ogg stream.  This
standard shows you the mapping for an Opus audio stream.  Ogg lets
you combine that with any other stream that anyone might define in
the future.

Since the number of possible permutation for that is unbounded, it
makes perfect sense to me that each reusable part should be defined
separately, and we already have the general rules for combining them.

If there's something in this draft that would block that somehow,
I'm not seeing it yet.  If the codec mapping itself is ok (and we've
got lots of implementations of it now, with no new issues shaking
out from them about that part of it), then anything extra is really
just a "how do we make this work with RFC 3533" problem, isn't it?

  Ron

---

From: Mark Harris <[email-masked]>
Date: Sat, 30 Aug 2014 16:03:36 -0700

> then be used with any other format that also has an Ogg mapping,

I have no objection to adding the pictures in a separate multiplexed
stream; in fact that is my proposal (3).  That is the only one that I
did not specify with enough detail to create interoperable
implementations, because it is a much larger change to the way that
opus-tools currently attaches pictures, and I therefore did not expect
that approach to be favored.  However if there is consensus that that
is the way to go, I can propose something more concrete, and in that
case the only change that I think is needed in the Ogg Opus draft is
to allow these streams in an "Ogg Opus file".  In particular, when
attached pictures are not specifically mentioned, no requirements or
recommendations (such as recommended file extension) should depend on
whether attached pictures are present.

> be Opus specific, it should be reusable for any Ogg stream.

Oh I agree completely that this should ideally not be codec-specific.
Not only attached pictures but also tags such as TITLE and ARTIST,
channel mapping, output gain, and loudness/gain required for
normalization would ideally all be maintained by the container and not
be specific to any codec.  The reason that I proposed adding attached
pictures in the codec-specific stream is because Ogg does not provide
metadata at the container level, and all of the metadata I mentioned
is currently being stored on a codec-specific page of the
codec-specific stream.  I would love to see all of this moved out of
the codec-specific stream and into an extensible codec-independent
format elsewhere in the file, and the codec-specific stream reduced to
only what is truly codec-specific, but that is not the direction that
was taken for other metadata.

 - Mark

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sat, 30 Aug 2014 22:23:03 -0700

> Ogg Opus stream, or allowed an Ogg Opus file to contain attached

There are two possibilities:

1) A proposal for adding binary metadata breaks already-encoded files, 
in which case I don't think we should do it, considering people have 
been producing those files for over two years now under the expectation 
that they would continue to work, or

2) A proposal for adding binary metadata does not break already-encoded 
files, in which case we have exactly the future extension mechanism you 
want (emphasis on "future").

Your Proposal 3 obviously qualifies. I believe Proposal 4 could be made 
to work (some care is required around the Vorbis issue, and to not paint 
ourselves into a corner later, but those are details).

-As an individual

---

From: Ron <[email-masked]>
Date: Mon, 1 Sep 2014 01:36:24 +0930

On Sat, Aug 30, 2014 at 04:03:36PM -0700, Mark Harris wrote:
> is the way to go, I can propose something more concrete, and

I think it's a bit early, and there hasn't yet been enough thorough
investigation of the available backward compatible options, to declare
this is "the way to go", but I do think there's general recognition
that this is one of the options on table for that which could be
investigated further, and could be made to work if it turns out to
be the best available option.

And which wouldn't require changes to this draft once there was a
real consensus among the various stakeholders about an extension
that met all of their various needs for something like this (not
just still picture data).

> in that

Most of this draft only talks about and defines the "Ogg Opus stream",
ie. the Ogg mapping needed for a single stream of Opus audio.  Which is
the minimal amount of extra data needed to actually play such a stream.

The comment header allows for the inclusion of arbitrary other metadata
but that data is strictly optional.  Tools can freely ignore it, or
strip it away, and the stream still remains valid and playable.  That
data is in a format that existing tools already knew how to handle,
(and to some extent, in some tools, expect for Ogg audio streams).

The only place we talk about an "Ogg Opus file" (with one exception,
which I think is a 'typo' that should instead say stream, and will
address in a separate mail), is where we say you can mux one or more
Ogg Opus streams together to store them in a file.

Which I think is the correct interpretation, since it indicates they
contain *only* Ogg Opus streams.  A minimal tool which recognises only
the Opus stream mapping is all that is required to handle such files.

Obviously you can also create files which mux an Ogg Opus stream with
other streams, for video, or subtitles, or still pictures, or anything
else - and when you do that, they become a more complex media type,
as defined by RFC 5334, and they become some other kind of file, which
requires something more than a minimal Ogg player to render.

Also obviously, we can more specifically define certain combinations
of those things in the future too "Opus with Daala", "Opus with still
pictures (possibly in a slide show)", "Opus with dancing holo-ponies",
whatever people care to create.  But you're going to need a more
specifically capable player to correctly render those, and it does
seem like they should be noted as distinct for the convenience of end
users as well in any case.  That doesn't make them "second class" or
otherwise frowned upon or illegal, they're just different.

The file extension is exactly such a convenience.  No player should
*ever* rely purely on it to know what the content of a file is.
It's just a hint to users that "A minimal Opus-only player should
be able to play this for you" and nothing more.

I think trying to define (and include or exclude) such things here in
detail is putting the cart before the holo-pony.  Once we have an Ogg
mapping for Daala, and for other arbitrary binary data etc. then
a separate document can define the normative and informational rules
for how such things should be combined with Ogg Opus streams and what
they should be called when in the newly defined form.

I think we leave open the most scope for being able to do that well
if this draft just confines itself to the Opus mapping, and leaves
more complex combinations (including with things that don't exist
yet) to future documents to define.  We just need to make sure we
haven't accidentally made something impossible here - which I don't
think we have (or can't yet see if we do).

> was taken for other metadata.

Ogg is purely a generic framing format.  Like TLV, it provides simple
rules that lets a generic tool parse a stream, even if that stream
includes things it doesn't know anything about, to cherry pick the
things that it does out of it.  Unlike TLV, it strikes a balance
between being efficient for small data packets and being able to
contain arbitrarily huge ones in the same format.

And unlike most other media containers, it was designed with the
ability to be streamed right from the outset too.

Which really means that the best format for metadata in any given
use scenario *is* somewhat specific to that scenario.  (ie. it's
totally useless to have all your metadata in container-defined
headers at the beginning of a file if people can join a stream
at any point during its playback, and it's pointless to have
something like "track gain" for a video stream).

For the first few codecs that were mapped to it, the value of
trying to guess and implement a metadata format that would be
properly future-proof was limited.  For the next few, the line
of least resistance was to "do what the other codecs did",
reusing what was good, fixing what turned out to be a mistake
as appropriate.  And much of that was done in an era when the
extra meta-data was always deliberately minimal, since we only
had, like, 100MB disks, if we were the lucky ones!

So yes, the time might well be right to look back on all of the
things that we currently have using it, and devise some format
that can both be used for new mappings that are created, and
retro-fitted to older ones, by and for tools that support it.

Ogg Skeleton was an early attempt along those lines for adding
multi-stream metadata, but it never really seemed to get a lot
of traction, for reasons I don't profess to really know.

But all of that is why I think this draft should confine itself
to just specifying the mapping for Opus streams, and why adding
"enhanced" meta-data should be done as an extension of RFC 3533
in close consultation with the people who'd actually have to
want it and implement it for it to become more generally useful.

I think you've made some good points about that, but I don't
think this group are the ones you need to convince and this
document is the place we need to specify it, in order for that
to really happen now.

Ideally I'd like to see this conversation move from "does this
belong in the Ogg Opus draft", to engagement of the various
interested parties in the merits and needs of the various
options for it (and associated things that haven't been raised
here at all yet), and if there is real momentum for that, to
the question of whether it should become a new milestone for
this group to create a separate document on that (or if not,
where that work should instead best be done).

  Ron

---

From: Ralph Giles <[email-masked]>
Date: Sun, 31 Aug 2014 11:24:36 -0700

On 2014-08-30 9:21 AM, Ron wrote:

> All [sec. 10] says is they are 'not strictly "Ogg Opus files"'

We do use the phrase "Ogg Opus files" in section 5.2.1 as well, so you
could argue either way. I want that flexibility so unless there's an
objection I'll limit use of that term to section 10.

It's nevertheless true that implementations of this draft have so far
been focussed on a single logical stream per chain segment.

 -r

---

From: Mark Harris <[email-masked]>
Date: Sun, 31 Aug 2014 23:29:14 -0700

Timothy B. Terriberry wrote:
> 1) A proposal for adding binary metadata breaks already-encoded files, in
> which case I don't think we should do it, considering people have been
> producing those files for over two years now under the expectation that they
> would continue to work, or
>
> files, in which case we have exactly the future extension mechanism you want
> (emphasis on "future").

None of my proposals should break already-encoded files, so that
should not be an issue.  I think that the only potential issue is that
insufficient guidance in the Ogg Opus draft may lead to newly encoded
files being handled poorly by older tools that are not aware of later
extensions.  It would be best for this document to provide additional
guidance even if details such as anything specific to attached
pictures is specified later in another document.

For proposal (1) (tags named with an initial "@" are binary rather
than UTF-8): even if no tags are specified in the document, the
document should state that values of tags whose name begins with "@"
need not conform to UTF-8, so that binary tags can be defined later
without having conforming tools reject them as invalid or treating
such tags poorly.  For the record, Firefox and Chrome do not seem to
have any issue playing files with such tags.  Only tools that try to
validate the UTF-8 or that assume that all tags are UTF-8 text should
be impacted, and if they do things like display or allow editing of
arbitrary tags they may want to provide a different way of doing that
for binary tags.  Tools are likely to already provide different
display or editing mechanisms for certain tags such as
METADATA_BLOCK_PICTURE, but currently the only way they can do that is
to check for specific tag names.  A naming convention allows tools to
provide a somewhat more appropriate display and editing capability
even for unknown future tags if they so choose.

For proposal (2) (new OpusTags section for binary blocks, separate
from comments): even if the format of pictures and any other binary
blocks are defined later in another document, this document should
provide guidance for tools that wish to modify tags in an existing
file or provide padding for other tools to do so without re-writing
the whole file.  Currently no such guidance is provided, which could
lead to interoperability failure between some tools which write files
and other tools that modify existing files, if they have different
assumptions about the data in the OpusTags packet following the
comment table.  For example without any guidance different tools
modifying tags in the file might assume that additional data in the
packet should be retained, or that it may be overwritten with the
unused portion left as-is, or may be overwritten with the unused
portion zero'd.  This makes it difficult to ensure that any extension
to this packet will be handled reasonably by any conforming tool.

For proposal (3) (use a different multiplexed stream): it might be
nice if the document had at least a SHOULD-level suggestion that
playback of an Ogg Opus stream not be impacted by the presence of
unrecognized multiplexed streams.

 - Mark