Subject: [codec] Ambisonics in an Ogg Opus Container

From: Michael Graczyk <[email-masked]>
Date: Fri, 27 May 2016 01:52:05 -0700

Hello,

I work at Google on spatial audio, 3D audio for virtual reality. We
recently launched spatial audio in 360 videos on Youtube [1] and are
currently streaming ambisonics [2] to Android devices on supported
platforms. We will launch on more platforms in the near future.

We plan to use the Opus codec in an Ogg stream on Youtube. We believe
that the robust quality and relatively low complexity of Opus makes
the codec well suited for ambisonics and VR in general. In order to
ensure consistency in how spatial audio is stored and streamed with
Opus, we have written a draft specification for encapsulating
ambisonics in an Ogg Opus stream:

https://datatracker.ietf.org/doc/draft-graczyk-codec-ambisonics/

We hope that establishing a standard Ogg Opus ambisonics format will
help organizations work together and interoperate. I look forward to
your thoughts and comments on the draft document.

[1] https://support.google.com/youtube/answer/6395969?hl=en
[2] https://en.wikipedia.org/wiki/Ambisonics

-- 

Thanks,
Michael Graczyk

---

From: Jean-Baptiste Kempf <[email-masked]>
Date: Fri, 27 May 2016 11:02:40 +0200

> We plan to use the Opus codec in an Ogg stream on Youtube. We believe

That seems a great idea.

> https://datatracker.ietf.org/doc/draft-graczyk-codec-ambisonics/

How is this mapping related to the mp4 one from google spatial-media:
https://github.com/google/spatial-media/blob/master/docs/spatial-audio-rfc.md
?

With my kindest regards,

-- 
Jean-Baptiste Kempf
http://www.jbkempf.com/ - +33 672 704 734
Sent from my Electronic Device

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 27 May 2016 10:45:07 -0700

> https://datatracker.ietf.org/doc/draft-graczyk-codec-ambisonics/

Thanks for presenting this document to the group.

With my chair hat on: I think it would be helpful for us to gage the 
level of interest if people would comment on the list if they think this 
is something useful for this group to work on.

As an individual, I had a few comments on the draft:

> The Ogg format is a container for transmission and storage of audio.

Ogg is a general purpose container, supporting audio, video, subtitles, 
etc. (though its most common use in current applications is for audio).

I think it may be useful to say that this channel mapping number will 
also likely impact other formats which use the same channel mapping 
families, e.g., Maktroska, MP4, MPEG TS. Just adding a sentence to the 
introduction along the lines of, "This mapping can also be used in other 
contexts which make use of the channel mappings defined by the Opus 
Channel Mapping Families registry."

> Allowed numbers of channels: (1 + n)^2 for n = 0...14.  Explicitly 4,
> 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225.

For n = 0, (1 + n)^2 == 1, but that's not on your explicit list. Was the 
intention here to exclude mono as redundant with other channel mapping 
families? We didn't do that for channel mapping family 1 (since, for 
example, the presence of an explicit channel mapping table allows you to 
extract a single channel from an already-encoded coupled stream, which 
is not possible with channel mapping family 0). But if you did intend to 
exclude mono, you should explicitly say why, and adjust the range of n.

> sqrt((2 - delta(m)) * ((n - m)! / (n + m)!)),

I've read the [ambix] reference before, and it still took me a while to 
figure out what this normalization coefficient is actually referring to. 
It might be useful to say in more detail what "normalization" even 
means: is this a factor you are expecting encoders to apply to the audio 
before encoding? Are you defining what decoders will apply during 
reconstruction? The coefficient in general seems highly dependent on how 
you express the spherical harmonic basis functions. E.g., part of this 
just comes from the Legendre polynomials, which could have simply been 
expressed by saying to use Legendre polynomials with unit norm (under 
some norm).

It may be better to simply refer to [ambix] Section 2.1, equation (1) 
for the definition of the spherical harmonics basis functions (the 
explicit section/equation makes it easy to find), and leave out this 
equation entirely. Conversely, it would be reasonable to spell out the 
full equation for those basis functions in this document (to make it 
more self-contained). But this kind of sits in-between, where it looks 
like it's trying to say something that you can understand on its own, 
but in fact it's impossible to interpret outside of the context of that 
reference (which already says it).

> Figure 1: Stereo Downmixing Matrix

I doubt this will really cause confusion, but you don't define W and Y. 
You might want to say they correspond to the first two ambisonics 
channels, explicitly.

> Updates: 7845 (if approved)

Does this actually update RFC7845? I think the intent was to allow 
updates to the channel mapping registry to be done without an explicit 
update of 7845, since it's possible it might not even be done with an 
IETF RFC (e.g., if there were some broadcast-specific standard done at a 
different standards body).

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 27 May 2016 14:00:19 -0400

On 05/27/2016 01:45 PM, Timothy B. Terriberry wrote:
> With my chair hat on: I think it would be helpful for us to gage
> the level of interest if people would comment on the list if they
> think this is something useful for this group to work on.

I support this work. I believe it is useful and am willing to do
reviews and also contribute directly if necessary.

> For n = 0, (1 + n)^2 == 1, but that's not on your explicit list.
> Was the intention here to exclude mono as redundant with other
> channel mapping families? We didn't do that for channel mapping
> family 1 (since, for example, the presence of an explicit channel
> mapping table allows you to extract a single channel from an
> already-encoded coupled stream, which is not possible with channel
> mapping family 0). But if you did intend to exclude mono, you
> should explicitly say why, and adjust the range of n.

I personally don't think 1-channel ambisonics should be allowed
(unless there's a good reason I missed). We might want to say that
encoders MUST NOT create them, but decoders SHOULD handle them as if
they were regular mono files.

> updates to the channel mapping registry to be done without an
> explicit update of 7845, since it's possible it might not even be
> done with an IETF RFC (e.g., if there were some broadcast-specific
> standard done at a different standards body).

I agree, I'm pretty sure we don't want the "Updates: 7845" line.

Cheers,

	Jean-Marc

---

From: Jean-Baptiste Kempf <[email-masked]>
Date: Fri, 27 May 2016 20:02:31 +0200

On 27 May, Timothy B. Terriberry wrote :
> With my chair hat on: I think it would be helpful for us to gage the level
> of interest if people would comment on the list if they think this is
> something useful for this group to work on.

I believe this work is useful and I'm interested in the outcome.

With my kindest regards,

---

From: Monty Montgomery <[email-masked]>
Date: Fri, 27 May 2016 15:55:59 -0400

I also fully support this Ambisonics work, and am very happy with the
progress progress thusfar.

There'd been interest in Ambisonics in Vorbis for a long time (more
than 15 years at this point), but there hadn't been a group that was
qualified in practical Ambisonics, motivated to do the work, and had
the time resources available for a long-term investment.  It's natural
this interest would move forward to Opus, and I'm glad to see it
finally sink root.

Cheers,
Monty

On Fri, May 27, 2016 at 4:52 AM, Michael Graczyk <[email-masked]> wrote:
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 27 May 2016 22:27:28 +0200

+1

-----Ursprüngliche Nachricht-----
Von: codec [mailto:[email-masked]] Im Auftrag von Jean-Baptiste
Kempf
Gesendet: Freitag, 27. Mai 2016 20:03
An: Timothy B. Terriberry <[email-masked]>
Cc: [email-masked]
Betreff: Re: [codec] Ambisonics in an Ogg Opus Container

On 27 May, Timothy B. Terriberry wrote :
> level of interest if people would comment on the list if they think 
> this is something useful for this group to work on.

With my kindest regards,

--
Jean-Baptiste Kempf
http://www.jbkempf.com/ - +33 672 704 734 Sent from my Electronic Device

---

From: Michael Graczyk <[email-masked]>
Date: Fri, 27 May 2016 17:16:39 -0700

Thanks Jean-Baptise, Tim, and Jean-Marc for your questions and
comments. Also thanks for all the support! I've replied inline.

> https://github.com/google/spatial-media/blob/master/docs/spatial-audio-rfc.md
The RFC that you linked to described a metadata box for spatial audio
(SA3D). This metadata is specific to mp4. We do not currently plan to
encapsulate Opus in mp4, but may be interested in the future if the
encapsulation described here becomes more widely supported:
wiki.xiph.org/Mp4Opus.

Since we have no plans to encapsulate Opus in mp4, I have been
operating under the assumption that all ambisonic metadata should be
contained in the Ogg stream itself (in this case implicitly by
defining only one channel ordering and normalization).

On Fri, May 27, 2016 at 10:45 AM, Timothy B. Terriberry
<[email-masked]> wrote:
> Ogg is a general purpose container, supporting audio, video, subtitles, etc. (though its most common use in current applications is for audio).

I reworded this to say
"Ogg is a general purpose container, supporting audio, video, and other media.
It can be used to encapsulate audio streams coded using the Opus codec"

> I think it may be useful to say that this channel mapping number will also likely impact other formats which use the same channel mapping families, e.g., Maktroska, MP4, MPEG TS. Just adding a sentence to the introduction along the lines of, "This mapping can also be used in other contexts which make use of the channel mappings defined by the Opus Channel Mapping Families registry."

Great, added.
BTW, when either Maktroska or WebM is used to encapsulate opus, the
entire Ogg container is wrapped, including the OpusHead header.

> For n = 0, (1 + n)^2 == 1, but that's not on your explicit list. Was the intention here ...

I would like to include mono to minimize the number of special cases
that decoders have to deal with. For example, if a virtual reality
playback environment expects to always receive ambisonics with mapping
family 2, the encoder could simply send single channel, n=0 ambisonics
when spatial audio is unavailable.

I corrected the explicit list: "Explicitly 1, 4...".

> I've read the [ambix] reference before, and it still took me a while to figure out what this normalization coefficient is actually referring to. ... The coefficient in general seems highly dependent on how you express the spherical harmonic basis functions...

Yes, unfortunately the interpretation of "normalization" depends on
how you define the spherical harmonic functions. I expect that Opus
encoders and decoders would not need to apply this normalization, but
the normalization is part of the semantic meaning of the audio stream.
For example, if the decoder is passing it's decoded stream along to an
ambisonic player which expects PCM with SN3D normalization, then the
decoder does not need to deal with the exact definition of "SN3D". On
the other hand, if the ambisonic player does not support SN3D
playback, an intermediate scaling step would be necessary. I do not
want to require the Opus decoder to perform this scaling, but I want
the Ogg stream to carry enough metadata (implicitly in this case) to
make it clear when and what scaling must be done.

> It may be better to simply refer to [ambix] Section 2.1, equation (1) for the definition of the spherical harmonics basis functions (the explicit section/equation makes it easy to find), and leave out this equation entirely.

I agree, that would be more clear.

> Conversely, it would be reasonable to spell out the full equation for those basis functions in this document (to make it more self-contained). But this kind of sits in-between, where it looks like it's trying to say something that you can understand on its own, but in fact it's impossible to interpret outside of the context of that reference (which already says it).

I agree. I do not think it would be worthwhile to include a
self-contained description of ambisonics, because the interpretation
of the encoded data is fairly complicated and depends on the playback
environment.  I removed this equation and referenced the section in
[ambix] instead.

> I doubt this will really cause confusion, but you don't define W and Y. You might want to say they correspond to the first two ambisonics channels, explicitly.

I added a definition in the paragraph above.

> Does this actually update RFC7845?...

Great, I removed the "updates" attribute from the rfc tag.

> they were regular mono files.

As I described above, I believe there is a good reason to support
1-channel ambisonics. The playback of ambisonics from an Ogg stream
can be simplified if the playback system always receives ambisonics
with channel mapping family 2. With 1-channel ambisonics, this
simplification exists even when the streamed data originally had no
spatial content.

> I agree, I'm pretty sure we don't want the "Updates: 7845" line.
Great, removed.

Question for anyone: Do I submit an -01 draft with the changes
mentioned above, or do I send out an intermediate draft on this list?

-- 

Thanks,
Michael Graczyk

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 27 May 2016 20:19:34 -0400

> mentioned above, or do I send out an intermediate draft on this list?

Just submit a -01 version -- versions are cheap :-)

	Jean-Marc

---

From: Michael Graczyk <[email-masked]>
Date: Fri, 27 May 2016 17:29:15 -0700

Great, I've uploaded a revised draft which can be viewed here.

https://www.ietf.org/id/draft-graczyk-codec-ambisonics-01.txt

---

From: Jean-Baptiste Kempf <[email-masked]>
Date: Sat, 28 May 2016 10:47:43 +0200

> encapsulation described here becomes more widely supported:

Sorry, what I meant is that it would be nice that the info from this RFC
carries the same kind of information than SA3D, so that a player like
VLC only need to have one identical filter to process them, post
decoder.

With my kindest regards,

---

From: Ron <[email-masked]>
Date: Sat, 28 May 2016 19:27:59 +0930

> something useful for this group to work on.

I agree that defining this is in scope for this group and it would be a
valuable thing for us to pursue here.  Anticipation of this sort of work
was exactly why we added the channel mapping to RFC 7845.  It's good to
see this sort of interest in it so soon after that was published, and to
have people with relevant expertise now wanting to contribute that here.

I support proceeding with this under the umbrella of this group.

---

From: Mark Harris <[email-masked]>
Date: Sat, 28 May 2016 22:35:42 -0700

I support this work.

In section 3.2 perhaps it should be clarified that the first ambisonic
channel (W) may be used directly for mono playback, particularly if
streams with channel mapping family 2 might only have one channel.

Also there is a typo "definied" in section 4.

 - Mark

---

From: Marc Lavallée <[email-masked]>
Date: Sun, 29 May 2016 10:42:33 -0400

On Sat, 28 May 2016 22:35:42 -0700,

> I support this work.

I also support this work, with enthusiasm (that's why I registered to
the codec list). 

I tried Opus for "ambisonics on the web" more than a year ago, to
experiment with online binaural decoding. It worked very well, but I
had to modify the encoder to uncouple channels, and I finally chose AAC
for compatibility reasons. It's about time that Opus becomes widely
adopted and supported as a standard, especially for multi-channel
streams. Maybe the situation is better now, as browsers matured (to be
verified).

I would suggest to not include decoding of Ambisonics streams in the
Opus decoders. Exceptions would be decoding/down-mixing to mono
and stereo, as defaults for compatibility reasons, but in a carefully
unobtrusive way; users of Opus for Ambisonics should easily have access
to all channels, even if the Opus decoder (and the system) wants to
"help" them.

Even mono and stereo are special cases in Ambisonics; one of the most
popular usage of Ambisonics is to explore the soundfield with virtual
directional microphones, something that could be used to explore
so-called "360 videos" with desktop speakers (without dedicated "VR"
gears).

Support for mixed-order is important too, mostly for horizontal-only
Ambisonics streams, because horizontal-only surround is still dominant,
and also because humans have more "spatial resolution" on the horizontal
plane; therefore, providing low vertical resolution is acceptable in
many playback situations; for example: 6 channels supports horizontal
at 2nd order and vertical at 1th order.

I think that what's only required is channel uncoupling and metadata
about the Ambisonics stream; everything else should be the
responsibility of the Ambisonics encoders and decoders. 

Normalisation is not the responsibility of a codec. It is used for
Ambisonics format conversion (ex: A to B, FuMa to Ambix). That said,
gain factors could be included to maximize dynamics, but it is
unrelated to Ambisonics.

Opus for Ambisonics, as a standard for the distribution of compressed
"VR audio", would be excellent.

--
Marc

---

From: Michael Graczyk <[email-masked]>
Date: Tue, 31 May 2016 11:38:37 -0700

Thanks Jean-Baptiste, Mark, and Marc for your comments.
Also thank you Ron for your support.

> VLC only need to have one identical filter to process them, post

In lieu of the SA3D box's explicit metadata, this draft requires that
the ambisonic streams use ACN channel ordering and SN3D normalization.
The channel count and channel mapping are contained in the Ogg
identification header. I believe the only thing missing here is the
SA3D box's "ambisonic_type" field. I would like to require periphonic
(full 3D) ambisonics in this draft. I added the word "periphonic" to
section 3.1.

> streams with channel mapping family 2 might only have one channel.

I added a short paragraph to the end of 3.2. What do you think?

> Also there is a typo "definied" in section 4.

Thanks, fixed

> to all channels, even if the Opus decoder (and the system) wants to

Right. I do not want to include ambisonic decoding in the Opus
decoder. I reworded section 3.2 to make it clear that downmixing MAY
be done by Ogg Opus players, not decoders.

> Support for mixed-order is important too, mostly for horizontal-only ...

The draft allows users to send mixed-order ambisonics

> gain factors could be included to maximize dynamics, but it is

Right. Just to be clear, the Opus encoder and decoder do not do the
normalization, nor do they _need_ to know anything about it. The
encoder could perform masking computations that require knowledge of
the normalization, but the main purpose of including it in this draft
is to give Opus ambisonc streams an unambiguous semantic meaning.

When an Ogg Opus ambisonic player receives a stream with channel
mapping 2, that player needs to know the normalization. As written,
this draft promises that the stream would be SN3D normalized.

---

From: Marc Lavallee <[email-masked]>
Date: Tue, 31 May 2016 14:54:55 -0400

On Tue, 31 May 2016 11:38:37 -0700

> (full 3D) ambisonics in this draft. I added the word "periphonic" to

The "ambisonic_type" (as defined in the spatial audio RFC) now only
allows 0 for "periphonic". Other values could be added, like 1 for
"mixed-order", 2 for "horizontal" and 3 for "UHJ".
--
Marc

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 31 May 2016 12:10:17 -0700

> SA3D box's "ambisonic_type" field. I would like to require periphonic

It seems like there's also no way to specify that an audio track is "non 
diegetic" (i.e., should not be changed by a user's head rotation in VR 
settings, e.g., for narration, etc.).

---

From: Mark Harris <[email-masked]>
Date: Tue, 31 May 2016 22:53:51 -0700

> I added a short paragraph to the end of 3.2. What do you think?

Thanks, this looks good in the -02 version.

 - Mark