Subject: Re: [codec] Comments on draft-ietf-codec-ambisonics-01

From: Mark Harris <[email-masked]>
Date: Mon, 13 Mar 2017 10:08:51 -0700

On Fri, Feb 17, 2017 at 1:57 PM, Jean-Marc Valin <[email-masked]> wrote:
> 3.2.  Channel Mapping Family 3
>
> I would suggest removing the "Output Channel Numbering" field because it
> is fully equivalent to simply permuting lines of the matrix. Also, I
> believe that the size of the matrix was meant to be "32*(N+M)*C bits"
> rather than "32*N*C bits".

To expand on this a bit, a mapping family maps M+N decoded channels
(corresponding to the actual order of the coupled and uncoupled
channels in the bitstream) to C output channels (channels with a
specific semantic meaning).  The additional "Output Channel Numbering"
table confuses things by adding an additional mapping from the output
channel numbers to a different set of numbers with actual semantic
meaning, leaving the output channel numbers with no apparent meaning.

This does have a potential benefit as a matrix compression technique,
to reduce the size of the matrix when it would contain rows that are
all zero.  However considering that the matrix occurs only once, and
mapping family 2 already offers a way to compress the matrix, this
alone does not seem worth the complexity of another level of
indirection.  If matrix compression is desired it would probably be
less confusing to describe it in those terms and keep the semantic
meaning tied to the output channels.

The description of the Output Channel Numbering also does not specify
the intended behavior if the same value appears in the table multiple
times.

Additionally, section 4.2 describes how to perform a stereo downmix of
mapping family 3, but makes assumptions about the output channel
numbering.  This seems harmful and likely to promote implementations
that make similar assumptions.  If it is necessary to apply the output
channel numbering described in section 3.2 in order to implement a
correct stereo downmix, then it would be better to simply use the
output channels from section 3 as input to the downmix, consolidating
sections 4.1 and 4.2, rather than specify new formulas that make
assumptions about the mapping.  That would also greatly simplify
section 4.

Eliminating the Output Channel Numbering table as Jean-Marc suggests
should resolve these concerns.

On another note, unless I am missing something, the formula in figure
2 does not appear to be correct.  For example, according to the
formula, k=3 (ACN 2) corresponds to order 0 degree 2, which does not
make sense.  It is also not clear why this channel index begins at 1,
when all others begin at 0.  It might also be a good idea to clarify
that this is an output channel index, as other uses of channel index
in RFC 7845 refer to decoded channel index (and also begin at 0).

 - Mark

---

From: Jan Skoglund <[email-masked]>
Date: Mon, 13 Mar 2017 17:38:54 +0000

Hey,

Thanks for your comments

> should resolve these concerns.

The problem is that once we allow mixed orders there is no unique way for a
receiver/decoder
to resolve the mapping to ACNs from just a number of total output channels.

You're completely right, and there's no reason why the channel index should
start with 1.

 - Mark

---

From: Jan Skoglund <[email-masked]>
Date: Mon, 13 Mar 2017 17:46:17 +0000

Hey Jean-Marc,

Thanks for your comments!

On Fri, Feb 17, 2017 at 1:57 PM Jean-Marc Valin <[email-masked]> wrote:

Hi Jan,

Here's some comments based on my review of the ambisonics draft:

3.  Ambisonics With Ogg Opus

>    Ambisonics MAY be encapsulated in the Ogg format by encoding with the

I would suggest replacing the normative MAY by just "can"

Agree

3.1.  Channel Mapping Family 2

>    This channel mapping uses the same channel mapping table format used
>    by channel mapping families 1 and 255.

I would suggest saying "mapping family 1" rather than "mapping families
1 and 255" since otherwise, you have families 2 referencing itself.
Also, we don't know what families 4 and up will look like.

Agree

I would suggest removing the "Output Channel Numbering" field because it
is fully equivalent to simply permuting lines of the matrix.

Allowing mixed orders means that there's no unique configuration for a
given total number of channels.
Therefore we need to signal to the decoder which ACN numbers each output
channel corresponds to.

About the matrix -- and as previously discussed privately -- I think it
should probably be stored as 16-bit integers in Q15 format. If any
global scaling is ever required, then the "output gain" field can always
be used.

Sure, this will work.

>    Note that [RFC7845] specifies that the identification header cannot
>    exceed one "page", which is 65,025 octets.  This sets a practical
>    maximum ambisonic order of 10, if full order is utilized and the
>    number of coded streams is the same as the ambisonic order plus the
>    two non-diegetic channels.

This text is unclear about whether partial order 11 is allowed. I have
no strong opinion on the subject, but think it should be clarified.

OK, I don't think we need to to specify a maximum allowed, just point out
the headroom's practical aspect

Cheers,

        Jean-Marc

Cheers,
Jan

---

From: Mark Harris <[email-masked]>
Date: Mon, 13 Mar 2017 14:40:55 -0700

> to resolve the mapping to ACNs from just a number of total output channels.

In mapping family 2, the channel count (C) is the number of channels
in the fully periphonic configuration, but it is not necessary to
encode them all.  The channel mapping table can map each ACN to a
specific decoded channel or to silence.  For mixed order, some of the
ACNs will be mapped to silence and will not be encoded.

In mapping family 3, the matrix can do everything that the channel
mapping table can do and more.  Why not treat C in the same manner, as
the number of channels in the fully periphonic configuration, even if
some are silent?

 - Mark

---

From: Drew Allen <[email-masked]>
Date: Mon, 13 Mar 2017 21:44:19 +0000

I think the issue is that the number of total channels rises quadratically
in respect to the ambisonic order (N + 1)^2. If a user wants to use just
the horizontal channels, it is only 2 * N + 1. If they wish to code very
high-order (>10th order) horizontal channels, they would be artifically
limited by all the zero channels being produced, no? Or can this handled
without actually creating all those empty channels?

On Mon, Mar 13, 2017 at 2:41 PM Mark Harris <[email-masked]> wrote:

> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 13 Mar 2017 17:55:07 -0400

On 13/03/17 05:44 PM, Drew Allen wrote:
> I think the issue is that the number of total channels rises
> quadratically in respect to the ambisonic order (N + 1)^2. If a user
> wants to use just the horizontal channels, it is only 2 * N + 1. If they
> wish to code very high-order (>10th order) horizontal channels, they
> would be artifically limited by all the zero channels being produced,
> no? Or can this handled without actually creating all those empty channels?

As far as I understand, the current draft already has all the
limitations you're describing. The channel mapping array is basically
equivalent to a CxC permutation matrix that multiplies the Cx(N+M)
weight matrix. The result is still a Cx(N+M) matrix, so using the
resulting matrix as weights can still do everything without the need for
the channel mapping to do the permutations.

Cheers,

	Jean-Marc

> On Mon, Mar 13, 2017 at 2:41 PM Mark Harris <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On Mon, Mar 13, 2017 at 10:38 AM, Jan Skoglund <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     > Hey,
>     >
>     > Thanks for your comments
>     >
>     > On Mon, Mar 13, 2017 at 10:08 AM Mark Harris <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>
>     >> On Fri, Feb 17, 2017 at 1:57 PM, Jean-Marc Valin
>     <[email-masked] <mailto:[email-masked]>>
>     >> wrote:
>     Also, I
>     [email-masked] <mailto:[email-masked]>

---

From: Drew Allen <[email-masked]>
Date: Mon, 13 Mar 2017 22:00:53 +0000

Got it. In that case, it certainly seems reasonable if I understand
correctly. Thanks for clearing that up!

---

From: Drew Allen <[email-masked]>
Date: Mon, 13 Mar 2017 22:04:12 +0000

so just to be clear, if a user, say, wants to encode some mixed order
ambisonics using ch253, how does the decoder know what ambisonic channels
it has received and know how to render them correctly?

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 13 Mar 2017 18:12:47 -0400

On 13/03/17 06:04 PM, Drew Allen wrote:
> ambisonics using ch253, how does the decoder know what ambisonic
> channels it has received and know how to render them correctly?

Well, each line of the matrix would correspond to a channel in the
ambisonics channel order. If that channel isn't encoded, then the line
would have only zeros.

The only way to avoid that situations would be to encode a separate D
value (D <= C) for the number of non-zero channels among the C
ambisonics channels possible. Then you'd store C values in the channel
mapping array (equivalent to a CxD permutation matrix), followed by a
Dx(M+N) weight matrix that would no longer have entire lines of zeros.
The result would be more compact in the case of sparse representation,
but IMO it'd be pretty ugly and prone to implementation errors. And if
you force D==C and don't code the D (which is what I'm proposing), then
the channel mapping permutation automatically becomes redundant.

Cheers,

	Jean-Marc

> On Mon, Mar 13, 2017 at 3:00 PM Drew Allen <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On Mon, Mar 13, 2017 at 2:55 PM Jean-Marc Valin <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         > On Mon, Mar 13, 2017 at 2:41 PM Mark Harris
>         <[email-masked] <mailto:[email-masked]>
>         > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>         >
>         >     On Mon, Mar 13, 2017 at 10:38 AM, Jan Skoglund
>         <[email-masked] <mailto:[email-masked]>
>         >     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>         >     > Hey,
>         >     >
>         >     > Thanks for your comments
>         >     >
>         >     > On Mon, Mar 13, 2017 at 10:08 AM Mark Harris
>         <[email-masked] <mailto:[email-masked]>
>         >     <mailto:[email-masked] <mailto:[email-masked]>>>
>         wrote:
>         >     >>
>         >     <[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked] <mailto:[email-masked]>>>
>         >     >> wrote:
>         >     >> > I would suggest removing the "Output Channel
>         Numbering" field
>         >     because it
>         >     >> > believe that the size of the matrix was meant to be
>         "32*(N+M)*C
>         >     bits"
>         >     >> that make similar assumptions.  If it is necessary to
>         apply the
>         >     output
>         >     > The problem is that once we allow mixed orders there is
>         no unique
>         >     way for a
>         >     > receiver/decoder
>         >     > to resolve the mapping to ACNs from just a number of
>         total output
>         >     channels.
>         >
>         >
>         <mailto:[email-masked] <mailto:[email-masked]>>

---

From: Jan Skoglund <[email-masked]>
Date: Mon, 13 Mar 2017 22:17:18 +0000

Hey,

Our idea was to avoid a mapping table, potentially sparse, completely for
family 2, and replacing it with a channel numbering list for family 3.

Cheers,
Jan

On Mon, Mar 13, 2017 at 3:12 PM Jean-Marc Valin <[email-masked]> wrote:

> >         >     >> specific semantic meaning).  The additional "Output
> Channel
> >         >     Numbering"
> >         >     >> Additionally, section 4.2 describes how to perform a
> stereo
> >         >     downmix of

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 13 Mar 2017 18:19:48 -0400

On 13/03/17 06:17 PM, Jan Skoglund wrote:
> Our idea was to avoid a mapping table, potentially sparse, completely
> for family 2, and replacing it with a channel numbering list for family 3.

Can you explain what you mean here by "avoid a mapping table" for family
2 and "channel numbering list" for family 3?

	Jean-Marc

> Cheers,
> Jan
> 
> On Mon, Mar 13, 2017 at 3:12 PM Jean-Marc Valin <[email-masked]
>     <mailto:[email-masked]>
>     > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >
>     >     On Mon, Mar 13, 2017 at 2:55 PM Jean-Marc Valin
>     <[email-masked] <mailto:[email-masked]>
>     >     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >
>     >         > quadratically in respect to the ambisonic order (N +
>     1)^2. If
>     >         a user
>     >         > wants to use just the horizontal channels, it is only 2
>     * N +
>     >         1. If they
>     >         resulting matrix as weights can still do everything
>     without the
>     >         need for
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >         > <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >         >
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >         >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >         >     > Hey,
>     >         >     >
>     >         >     > Thanks for your comments
>     >         >     >
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >         >     <mailto:[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>>>
>     >         wrote:
>     >         >     >>
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >         <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>
>     >         >     >> wrote:
>     >         >     >> > is fully equivalent to simply permuting lines
>     of the
>     >         matrix.
>     >         >     Also, I
>     >         >     >> specific semantic meaning).  The additional
>     "Output Channel
>     >         >     Numbering"
>     >         >     >> to reduce the size of the matrix when it would
>     contain
>     >         rows that are
>     >         >     >> indirection.  If matrix compression is desired it
>     would
>     >         probably be
>     >         >     >> the intended behavior if the same value appears
>     in the
>     >         table multiple
>     >         >     >> times.
>     >         >     >>
>     >         >     >> Additionally, section 4.2 describes how to
>     perform a stereo
>     >         >     downmix of
>     >         >     >> channel numbering described in section 3.2 in
>     order to
>     >         implement a
>     >         >     >> output channels from section 3 as input to the
>     downmix,
>     >         consolidating
>     >         >     >> sections 4.1 and 4.2, rather than specify new
>     formulas
>     >         that make
>     >         >     >> assumptions about the mapping.  That would also
>     greatly
>     >         simplify
>     >         >     >> section 4.
>     >         >     >>
>     >         >     specific decoded channel or to silence.  For mixed
>     order,
>     >         some of the
>     >         >     In mapping family 3, the matrix can do everything
>     that the
>     >         channel
>     >         <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>

---

From: Jan Skoglund <[email-masked]>
Date: Mon, 13 Mar 2017 22:25:02 +0000

Ha, sorry, wrong name! I meant avoiding permutation matrices.

Jan

On Mon, Mar 13, 2017 at 3:19 PM Jean-Marc Valin <[email-masked]> wrote:

> >     >         > would be artifically limited by all the zero channels
> being
> >     >         produced,
> >     >         > no? Or can this handled without actually creating all
> those
> >     >         empty channels?
> >     >
> >     >         >     >> (corresponding to the actual order of the coupled
> and
> >     >         uncoupled
> >     >         >     >> table confuses things by adding an additional
> mapping
> >     >         from the output
> >     >         >     >> meaning, leaving the output channel numbers with
> no
> >     >         apparent meaning.
> >     >         >     >>
> >     >         >     >> all zero.  However considering that the matrix
> occurs
> >     >         only once, and
> >     >         >     >> mapping family 2 already offers a way to compress
> the
> >     >         matrix, this
> >     >         >     >> alone does not seem worth the complexity of
> another
> >     >         level of
> >     >         >     >> less confusing to describe it in those terms and
> keep
> >     >         the semantic
> >     >         >     >> The description of the Output Channel Numbering
> also
> >     >         does not specify
> >     >         >     >> numbering.  This seems harmful and likely to
> promote
> >     >         implementations
> >     >         >     In mapping family 2, the channel count (C) is the
> number
> >     >         of channels
> >     >         >     encode them all.  The channel mapping table can map
> each
> >     >         ACN to a
> >     >         >     mapping table can do and more.  Why not treat C in
> the
> >     >         same manner, as

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 13 Mar 2017 18:35:18 -0400

Well, for family 2 is makes sense to have just a mapping table (1-D
array of size C) and no matrix. Note that in previous comments, I was
using the term "permutation matrix" only in the sense that the 1-D array
is mathematically equivalent to a (sparse) permutation matrix.

For family 3, if you already have a Cx(N+M) weight matrix, then the
mapping table becomes completely redundant.

	Jean-Marc

On 13/03/17 06:25 PM, Jan Skoglund wrote:
> On Mon, Mar 13, 2017 at 3:19 PM Jean-Marc Valin <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     > On Mon, Mar 13, 2017 at 3:12 PM Jean-Marc Valin
>     >     > On Mon, Mar 13, 2017 at 3:00 PM Drew Allen
>     <[email-masked] <mailto:[email-masked]>
>     >     <mailto:[email-masked] <mailto:[email-masked]>>
>     >     > <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >     >
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >     >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >     >
>     >     >         > would be artifically limited by all the zero
>     channels being
>     >     >         produced,
>     >     >         > no? Or can this handled without actually creating
>     all those
>     >     >         empty channels?
>     >     >
>     >     >         limitations you're describing. The channel mapping
>     array is
>     >     >         basically
>     >     >         equivalent to a CxC permutation matrix that
>     multiplies the
>     >     Cx(N+M)
>     <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>     >     >         >
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>     >     >         >     > Hey,
>     >     >         >     >
>     >     >         >     > Thanks for your comments
>     >     >         >     >
>     <mailto:[email-masked]>
>     >     <mailto:[email-masked] <mailto:[email-masked]>>>>>
>     >     >         wrote:
>     >     >         >     >>
>     >     >         >     <[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     >     >         <mailto:[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>>
>     >     >         >     >> wrote:
>     >     >         >     >> To expand on this a bit, a mapping family
>     maps M+N
>     >     >         decoded channels
>     >     >         >     >> (corresponding to the actual order of the
>     coupled and
>     >     >         uncoupled
>     >     >         >     >> table confuses things by adding an
>     additional mapping
>     >     >         from the output
>     >     >         >     >> channel numbers to a different set of
>     numbers with
>     >     >         actual semantic
>     >     >         >     >> meaning, leaving the output channel numbers
>     with no
>     >     >         apparent meaning.
>     >     >         >     >>
>     >     >         >     >> all zero.  However considering that the
>     matrix occurs
>     >     >         only once, and
>     >     >         >     >> mapping family 2 already offers a way to
>     compress the
>     >     >         matrix, this
>     >     >         >     >> less confusing to describe it in those
>     terms and keep
>     >     >         the semantic
>     >     >         >     >> The description of the Output Channel
>     Numbering also
>     >     >         does not specify
>     >     >         >     >> mapping family 3, but makes assumptions
>     about the
>     >     >         output channel
>     >     >         >     >> numbering.  This seems harmful and likely
>     to promote
>     >     >         implementations
>     >     >         >     >> correct stereo downmix, then it would be
>     better to
>     >     >         simply use the
>     >     >         >     >> Eliminating the Output Channel Numbering
>     table as
>     >     >         Jean-Marc suggests
>     >     >         >     In mapping family 2, the channel count (C) is
>     the number
>     >     >         of channels
>     >     >         >     in the fully periphonic configuration, but it
>     is not
>     >     >         necessary to
>     >     >         >     encode them all.  The channel mapping table
>     can map each
>     >     >         ACN to a
>     >     >         >     mapping table can do and more.  Why not treat
>     C in the
>     >     >         same manner, as
>     <mailto:[email-masked] <mailto:[email-masked]>>>>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 13 Mar 2017 16:06:55 -0700

> mapping table becomes completely redundant.

I think the issue boils down to whether it's worth it to have a table of 
size C to allow a matrix of size Kx(N+M) instead of Cx(N+M) in the 
headers where D is some number smaller than C (probably D=N+M). If more 
than 80% of your matrix is filled with zeros (100 out of 121 rows for 
10th order with only horizontal channels), that could be seen as a 
little wasteful.

That said, we very intentionally chose the direction of the channel 
mapping table for RFC 7845: one value per output channel that says which 
decoded channel to use. That makes it impossible to send multiple 
decoded channels (mixed channels after multiplying by the weight matrix 
in this draft) to the same speaker (ACN number in this draft). As Mark 
Harris points out, this is a failure case the current draft doesn't 
specify how to deal with. By doing the mapping in the same direction as 
RFC 7845, the worst failure case is that someone specifies a channel 
index larger than what's actually available, but that's easily 
detectable (i.e., you can tell without having to look at any other 
values in the mapping table). You also don't need to handle the diegetic 
channels separately. There is a special case for silent channels, but as 
a benefit it become easy (O(1)) to tell if a channel isn't used.

Having the mapping work in this direction would also be more consistent 
with Channel Mapping Family 2.

---

From: Jan Skoglund <[email-masked]>
Date: Mon, 13 Mar 2017 23:39:07 +0000

Thanks Mark, Jean-Marc, and Tim,

It makes sense to use the table already defined in RFC 7845, which then
automatically allows mixed order also for family 2. We'll remove the
"channel numbering table" for family 3.

Cheers,
Jan