Subject: Re: [codec] #15: Efficiently combine pre-encoded audio

From: "codec issue tracker" <[email-masked]>
Date: Sat, 01 May 2010 14:24:12 -0000

#15: Efficiently combine pre-encoded audio
------------------------------------+---------------------------------------
 Reporter:  hoene@…                 |       Owner:     
     Type:  enhancement             |      Status:  new
 Priority:  minor                   |   Milestone:     
Component:  requirements            |     Version:     
 Severity:  Active WG Document      |    Keywords:     
------------------------------------+---------------------------------------

Comment(by hoene@…):

 [Colin:]
 > [...] conferences implemented using multicast require
 > end system mixing of potentially large numbers of active audio
 > streams, whereas those implemented using conference bridges do the
 > mixing in a single central location, and generally suppress all but
 > one speaker. The differences in mixing and the number of simultaneous
 > active streams that might be received potentially affect the design of
 > the codec.

 [Raymond]: I would like to take this opportunity to express my view that
 although codec complexity isn’t much of an issue for PC-to-PC calls where
 there are GHz of processing power available, the codec complexity is an
 important issue in certain application scenarios.  The following are just
 some examples.
 1) If a conference bridge has to decode a large number of voice channels,
 mix, and re-encode, and if compressed-domain mixing cannot be done (which
 is usually the case), then it is important to keep the decoder complexity
 low.

 [JM]: The decoder complexity is very important. Not only because of mixing
 issue, but also because the decoder is generally not allowed to take
 shortcuts to save on complexity (unlike the encoder). As for compressed-
 domain mixing, as you say it is not always available, but *if* we can do
 it (even if only partially), then that can result in a "free" reduction in
 decoder complexity for mixing.

 [Christian]: Scalable [conferencing]
 -       Receiver side activity detection for music and voice having low
 complexity (for the conference bridge)
 -       Efficient mixing of two to four(?) active flows (is this
 achievable without the complete process of decoding and encoding again?)

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/15#comment:1>
codec <http://tools.ietf.org/codec/>

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 12 May 2010 09:05:46 -0600

For conference bridges, it's probably more important to be able to decide who the active speakers are with low CPU complexity than the actually act of mixing the the selected speakers. Consider a typical call with 7 people who might be speakers and  the 3 most active are selected and mixed. In many systems today, most the MIPS goes to decoding all 7 streams to do speaker detection before the resulting 4 streams are formed and encoded. If there was a cheap way to figure out who the active speakers were without doing a full decode of all 7 streams, that would be sort of nice the for conferences bridges. 

On May 1, 2010, at 8:24 AM, codec issue tracker wrote:

> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

Cullen Jennings
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Brian Rosen <[email-masked]>
Date: Wed, 12 May 2010 11:25:17 -0400

Excellent idea.  Been there, never really did it.  It's complex.
Effectively, you need a distributed adaptive threshold mechanism.

However, if you had it, user experience in multispeaker environments gets a
win.

Brian

On 5/12/10 11:05 AM, "Cullen Jennings" <[email-masked]> wrote:

> 
> For conference bridges, it's probably more important to be able to decide who
> the active speakers are with low CPU complexity than the actually act of
> mixing the the selected speakers. Consider a typical call with 7 people who
> might be speakers and  the 3 most active are selected and mixed. In many
> systems today, most the MIPS goes to decoding all 7 streams to do speaker
> detection before the resulting 4 streams are formed and encoded. If there was
> a cheap way to figure out who the active speakers were without doing a full
> decode of all 7 streams, that would be sort of nice the for conferences
> bridges. 
> 
> 
> 
> 
>> Reporter:  hoene@Š                 |       Owner:
>> Comment(by hoene@Š):
>> although codec complexity isn¹t much of an issue for PC-to-PC calls where

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 12 May 2010 11:26:00 -0400

On 05/12/2010 11:05 AM, Cullen Jennings wrote:
>  If there was a cheap way to figure out who the active speakers were without doing a full decode of all 7 streams, that would be sort of nice the for conferences bridges.

The cheapest solution, of course, is transmit-side activity detection. 
Maybe we need to specify a way for a receiver to request that the 
transmitter employ (or not employ) VAD.

--Ben

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 12 May 2010 11:44:23 -0400

> transmitter employ (or not employ) VAD.

I think you can do better than an encoder VAD. All you need to do is make 
sure that the relevant information you need for a VAD can easily be decoded 
from the bit-stream without having to do a full decoding. For example, if 
you're able to easily extract the gain and spectral envelope, you can do a 
VAD based on that without even having to look at the other parameters in 
the bit-stream.

	Jean-Marc

---

From: stephen botzko <[email-masked]>
Date: Wed, 12 May 2010 11:45:50 -0400

The adaptive threshold doesn't have to be distributed, as the conference
bridge is selecting the highest scores.

You do need a consistent way to compute the scores in the endpoints, ideally
using a method which is not simply energy.

I realize the bridge can alternatively generate scores from bitstream
information; I am thinking that is equivalent to including a metric in the
RTP payload.

Stephen Botzko

On Wed, May 12, 2010 at 11:25 AM, Brian Rosen <[email-masked]> wrote:

> >> Ticket URL: <
> http://trac.tools.ietf.org/wg/codec/trac/ticket/15#comment:1>

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 12 May 2010 11:52:31 -0400

On 05/12/2010 11:44 AM, Jean-Marc Valin wrote:
> I think you can do better than an encoder VAD.

I know that CELT makes decoder VAD very efficient, but how is decoder VAD 
better than encoder VAD?  Encoder VAD saves even more CPU, saves 
bandwidth, and enables easier jitter buffering.

Are you thinking about some sort of adaptive thresholding that requires 
knowing all streams' volume levels?

Anyway, VAD can run on both encode and decode sides at the same time.

--Ben

---

From: stephen botzko <[email-masked]>
Date: Wed, 12 May 2010 12:01:19 -0400

It might be valuable to be able to obtain the VAD without having to decrypt
the bitstream, since that also can become a problem as the number of streams
grows.

There is a security consideration (traffic analysis), however, it still
might be worth thinking about.

Stephen Botzko

On Wed, May 12, 2010 at 11:44 AM, Jean-Marc Valin <
[email-masked]> wrote:

> VAD based on that without even having to look at the other parameters in the
> bit-stream.
>
>        Jean-Marc
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 12 May 2010 12:03:19 -0400

Benjamin M. Schwartz wrote:
> I know that CELT makes decoder VAD very efficient, 

Not only CELT. You can do that with an LPC-based codec too.

> but how is decoder VAD

There's a few reasons why I think decoder-side is better:
- The decision for an encoder-size VAD would take some amount of space in 
the bit-stream
- If we make an encode-size VAD mandatory, then all encoders will have to 
spend the CPU cycles, even when it's not needed. If it's not mandatory, 
then the decoder cannot rely on it, so it still needs to implement a VAD
- A decoder VAD does not need to be specified in an exact way, so 
implementers can choose different implementations depending on that 
information they need.
- You cannot "game" a decode-size VAD.

> knowing all streams' volume levels?

Well, knowing the relative amplitudes of each stream can allow you to take 
more intelligent decisions, e.g. when you have to choose the "most active 
speaker". That's something you can't really get from an encoder VAD.

> Anyway, VAD can run on both encode and decode sides at the same time.

That would just mean nobody would bother implementing the encode side.

	Jean-Marc

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 12 May 2010 12:22:25 -0400

On 05/12/2010 12:03 PM, Jean-Marc Valin wrote:
> - The decision for an encoder-size VAD would take some amount of space
> in the bit-stream

I think I failed to communicate that by VAD I mean _not sending packets_ 
during inactivity.  For the packets that are sent, the overhead should 
average much less than 1 bit per frame.

I'm not suggesting sending 200 packets a second containing a flag 
indicating no voice activity, followed by carefully coded background 
noise.  That would be silly.

> - If we make an encode-size VAD mandatory, then all encoders will have
> to spend the CPU cycles, even when it's not needed. If it's not
> mandatory, then the decoder cannot rely on it, so it still needs to
> implement a VAD

I don't see this as "mandatory".  The encoder can turn off VAD, and 
probably should for full-quality applications.

> implementers can choose different implementations depending on that

The only thing that needs exact specification is the signalling.  The 
encoder may use it or not use it as it pleases.

> - You cannot "game" a decode-size VAD.

I don't know what this means.

> Well, knowing the relative amplitudes of each stream can allow you to
> take more intelligent decisions, e.g. when you have to choose the "most
> active speaker". That's something you can't really get from an encoder VAD.
>

I expect encode-side VAD on a conference call to save more than a factor 
of 2 in bandwidth, which makes it very desirable, especially for large 
deployments.  People will use it to save bandwidth (especially if it's on 
by default in the reference implementation).  The decode-side CPU savings 
are just a minor bonus side-effect.

--Ben

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 12 May 2010 12:37:55 -0400

> average much less than 1 bit per frame.

What you're describing is called DTX (discontinuous transmission). This can 
be useful feature, but it's very different from what we were originally 
talking about in terms of conference mixing.

	Jean-Marc

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 12 May 2010 12:43:57 -0400

On 05/12/2010 12:37 PM, Jean-Marc Valin wrote:
> What you're describing is called DTX (discontinuous transmission).

Oops. Right.  What I'm trying to say is that DTX, based on encoder-side 
VAD, also greatly reduces the (average) computational burden on a 
conference mixer.  Of course, if everyone's really talking at once then 
VAD can't help.

--Ben

---

From: Roman Shpount <[email-masked]>
Date: Wed, 12 May 2010 14:12:04 -0400

There is one more application to efficiently combining pre-encoded
audio: playing announcements or recorded audio. Standard network or
IVR announcements can be encoded once and efficiently inserted or
combined into audio stream. If pre-encoded audio is supported and the
client supports AVT tones, it is trivial to develop a very efficient
IVR server which does not require any CODEC encoding or decoding.

Efficient decoder side VAD is also very helpful in case of speech
recognition, where it allows to save cycles in end-pointer. This way
audio only needs to be decoded and passed to the speech recognition
system only when voice is present.

Bottom line, if we have both efficient decoder side VAD and combining
pre-encoded audio we can develop some very efficient VXML servers,
voice mail and IVR system, not just conferencing servers.
_____________________________
Roman Shpount - www.telurix.com

On Wed, May 12, 2010 at 12:43 PM, Benjamin M. Schwartz
<[email-masked]> wrote:
> Oops. Right.  What I'm trying to say is that DTX, based on encoder-side VAD,
> also greatly reduces the (average) computational burden on a conference
> mixer.  Of course, if everyone's really talking at once then VAD can't help.
>
> --Ben

---

From: "codec issue tracker" <[email-masked]>
Date: Mon, 24 May 2010 14:22:08 -0000

Comment(by hoene@…):

 [Cullen]:
 For conference bridges, it's probably more important to be able to decide
 who the active speakers are with low CPU complexity than the actually act
 of mixing the the selected speakers. Consider a typical call with 7 people
 who might be speakers and  the 3 most active are selected and mixed. In
 many systems today, most the MIPS goes to decoding all 7 streams to do
 speaker detection before the resulting 4 streams are formed and encoded.
 If there was a cheap way to figure out who the active speakers were
 without doing a full decode of all 7 streams, that would be sort of nice
 the for conferences bridges.

 [Brian]: Excellent idea.  Been there, never really did it.  It's complex.
 Effectively, you need a distributed adaptive threshold mechanism.
 However, if you had it, user experience in multispeaker environments gets
 a win.

 [Benjamin]:  The cheapest solution, of course, is transmit-side activity
 detection.
 Maybe we need to specify a way for a receiver to request that the
 transmitter employ (or not employ) VAD.

 [JM]:
 I think you can do better than an encoder VAD. All you need to do is make
 sure that the relevant information you need for a VAD can easily be
 decoded from the bit-stream without having to do a full decoding. For
 example, if you're able to easily extract the gain and spectral envelope,
 you can do a VAD based on that without even having to look at the other
 parameters in the bit-stream.

 [Brian]:
 The adaptive threshold doesn't have to be distributed, as the conference
 bridge is selecting the highest scores.
 You do need a consistent way to compute the scores in the endpoints,
 ideally using a method which is not simply energy.
 I realize the bridge can alternatively generate scores from bitstream
 information; I am thinking that is equivalent to including a metric in the
 RTP payload.

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/15#comment:2>
codec <http://tools.ietf.org/codec/>

---

From: Roman Shpount <[email-masked]>
Date: Mon, 24 May 2010 13:28:05 -0400

I would like to remind that this issue is not about efficient VAD, but
about efficiently combining pre-encoded streams. There are two use
cases that I can think of:

a. Conference servers, where the active speaker was determined by
either receiver or decoder side VAD. As it was mentioned before, it is
possible to implement an efficient decoder side VAD without
implementing a complete decoder. If we can combine pre-encoded audio
we can combine multiple streams on the conference server without going
through decoder/encoder cycle greatly decreasing both CPU requirements
and mixer delay.

b. Announcement and IVR servers, where a small set of pre-encoded
announcements are played to the user. Standard network or IVR
announcements can be encoded once and efficiently inserted or combined
into audio stream. If pre-encoded audio is supported and the client
supports AVT tones, it is trivial to develop a very efficient IVR
server  which does not require any CODEC encoding or decoding.
_____________
Roman Shpount