Subject: [codec] #16: Multicast?

From: "codec issue tracker" <[email-masked]>
Date: Wed, 21 Apr 2010 09:42:28 -0000

#16: Multicast?
------------------------------------+---------------------------------------
 Reporter:  hoene@…                 |       Owner:     
     Type:  enhancement             |      Status:  new
 Priority:  trivial                 |   Milestone:     
Component:  requirements            |     Version:     
 Severity:  Active WG Document      |    Keywords:     
------------------------------------+---------------------------------------
 The question arrose whether the interactive CODEC MUST support multicast
 in addition to teleconferencing.

 On 04/13/2010 11:35 AM, Christian Hoene wrote:
 >>> P.S. On the same note, does anybody here cares about using this
 >>> CODEC with multicast? Is there a single commercial multicast voice
 deployment?
 >>> From what I've seen all multicast does is making IETF voice
 >>> standards harder to understand or implement.
 >>
 >> I think that would be a mistake to ignore multicast - not because of
 >> multicast itself, but because of Xcast (RFC 5058) which is a
 >> promising technology to replace centralized conference bridges.
 >
 > Regarding multicast:
 >
 > I think we shall start at user requirements and scenarios.
 > Teleconference (including mono or spatial audio) might be good
 > starting point. Virtual environments like second live would require
 multicast communication, too. If the requirements of these scenarios are
 well understand, we can start to talk about potential solutions like IP
 multicast, Xcast or conference bridges.

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/16>
codec <http://tools.ietf.org/codec/>

---

From: Colin Perkins <[email-masked]>
Date: Wed, 21 Apr 2010 11:48:24 +0100

On 21 Apr 2010, at 10:42, codec issue tracker wrote:
> #16: Multicast?
> ------------------------------------ 
> +----------------------------------
> ------------------------------------ 
> +----------------------------------
> The question arrose whether the interactive CODEC MUST support  
> multicast in addition to teleconferencing.
>
>>>> CODEC with multicast? Is there a single commercial multicast  
>>>> voice deployment? From what I've seen all multicast does is  
>>>> making IETF voice standards harder to understand or implement.
>>>
>>> I think that would be a mistake to ignore multicast - not because  
>>> of multicast itself, but because of Xcast (RFC 5058) which is a  
>> multicast communication, too. If the requirements of these  
>> scenarios are well understand, we can start to talk about potential  
>> solutions like IP multicast, Xcast or conference bridges.

RTP is inherently a group communication protocol, and any codec  
designed for use with RTP should consider operation in various  
different types of group communication scenario (not just multicast).  
RFC 5117 is a good place to start when considering the different types  
of topology in which RTP is used, and the possible placement of mixing  
and switching functions which the codec will need to work with.

-- 
Colin Perkins
http://csperkins.org/

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 21 Apr 2010 13:14:04 +0200

Hi,

>> Reporter:  hoene@.                 |       Owner:

I have to admit that my thinking is complete different. I first start with user requirements, then look how to achieve them, and
finally consider on how to map them on existing standards - not the other way around.

Users do demand for teleconferencing, preferable at high quality (good spatial audio quality, low acoustic latency and reliable).
Thus, this is a MUST requirement.

I believe that multicast support NEEDS NOT to be particular optimized due to the following two reasons:

a) RFC5117: 4.1.3.  Per Domain Bit-Rate Adaptation

   Participants are most likely to be connected to each other with a
   heterogeneous set of paths.  This makes congestion control in a Point
   to Multipoint set problematic.  For the ASM and "relay" scenario,
   each individual sender has to adapt to the receiver with the least
   capable path.  This is no longer necessary when Media Translators,
   Mixers, or MCUs are involved, as each participant only needs to adapt
   to the slowest path within its own domain.  The Translator, Mixer, or
   MCU topologies all require their respective outgoing streams to
   adjust the bit-rate, packet-rate, etc., to adapt to the least capable
   path in each of the other domains.  That way one can avoid lowering
   the quality to the least-capable participant in all the domains at
   the cost (complexity, delay, equipment) of the Mixer or Translator.

Indeed, the CODEC SHOULD be adapted on each path to achieve best quality.

b) Multicast is used (to my best knowledge) mainly locally and not throughout the Internet, thus if we want to support global
interactive communication with a good CODEC support of MULTICAST cannot be considered as granted.

Yours respectfully,

 Christian

>--
>Colin Perkins
>http://csperkins.org/
>
>
>
>_______________________________________________
>codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/codec

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 21 Apr 2010 07:20:11 -0400

On Apr 21, 2010, at 6:48 AM, Colin Perkins wrote:

>>> starting point. Virtual environments like second live would  
>>> require multicast communication, too. If the requirements of these  
>>> scenarios are well understand, we can start to talk about  
>>> potential solutions like IP multicast, Xcast or conference bridges.
>
>
> different types of group communication scenario (not just  
> multicast). RFC 5117 is a good place to start when considering the  
> different types of topology in which RTP is used, and the possible  
> placement of mixing and switching functions which the codec will  
> need to work with.
>

It is not clear to me what supporting multicast would entail here. If  
this is a codec over RTP, then what
is to stop it from being multicast ?

Regards
Marshall

---

From: stephen botzko <[email-masked]>
Date: Wed, 21 Apr 2010 07:48:21 -0400

Hi Christian

Teleconferencing standards (both in the ITU-T and the IETF) support
multicast..  It would be awkward if the codec decision was somehow coupled
to the transport, and (IMHO) architecturally wrong.

While it is probably true that multicast is more likely to be used on
enterprise networks, I don't see that as terribly relevant.

I see no technical reason to NEED NOT multicast.

Regards,
Stephen Botzko

On Wed, Apr 21, 2010 at 7:14 AM, Christian Hoene <[email-masked]>wrote:

> I have to admit that my thinking is complete different. I first start with
> user requirements, then look how to achieve them, and
> Users do demand for teleconferencing, preferable at high quality (good
> spatial audio quality, low acoustic latency and reliable).
> b) Multicast is used (to my best knowledge) mainly locally and not
> throughout the Internet, thus if we want to support global

---

From: Colin Perkins <[email-masked]>
Date: Wed, 21 Apr 2010 13:13:28 +0100

On 21 Apr 2010, at 12:14, Christian Hoene wrote:
> I have to admit that my thinking is complete different. I first  
> start with user requirements, then look how to achieve them, and  
> Users do demand for teleconferencing, preferable at high quality  
> (good spatial audio quality, low acoustic latency and reliable).  
> I believe that multicast support NEEDS NOT to be particular  
> optimized due to the following two reasons:

You'll notice that I said that RTP is a group communication protocol,  
not explicitly a multicast protocol. Multicast groups are one of many  
ways in which RTP is used for group communication, but not the only one.

> cannot be considered as granted.

No-one said it should be taken for granted. A multicast group is one  
of the different group communication scenarios where the codec can be  
used. That there is a multicast group transport is, I would expect,  
largely irrelevant to the codec design. That such a group results in  
potentially many audio streams that must be mixed by the end systems,  
rather than by middleboxes, is probably more relevant.

The other topologies discussed in RFC 5117 may require different a  
trade-off. This group should probably review them, to consider how  
(if) they affect the codec design.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 21 Apr 2010 13:17:51 +0100

On 21 Apr 2010, at 12:20, Marshall Eubanks wrote:
>>>>> I think that would be a mistake to ignore multicast - not  
>>>>> because of multicast itself, but because of Xcast (RFC 5058)  
>>>>> which is a promising technology to replace centralized  
>>>>> conference bridges.
>>>>
>>>> Regarding multicast:
>>>>
>>>> require multicast communication, too. If the requirements of  
>>>> these scenarios are well understand, we can start to talk about  
> It is not clear to me what supporting multicast would entail here.  
> If this is a codec over RTP, then what is to stop it from being  
> multicast ?

Nothing. However group conferences implemented using multicast require  
end system mixing of potentially large numbers of active audio  
streams, whereas those implemented using conference bridges do the  
mixing in a single central location, and generally suppress all but  
one speaker. The differences in mixing and the number of simultaneous  
active streams that might be received potentially affect the design of  
the codec.

-- 
Colin Perkins
http://csperkins.org/

---

From: stephen botzko <[email-masked]>
Date: Wed, 21 Apr 2010 10:33:44 -0400

in-line

Stephen Botzko

On Wed, Apr 21, 2010 at 8:17 AM, Colin Perkins <[email-masked]> wrote:

>>>> ------------------------------------+----------------------------------
>>>> ------------------------------------+----------------------------------
>>>>> P.S. On the same note, does anybody here cares about using this CODEC
>>>>>>> with multicast? Is there a single commercial multicast voice deployment?
>>>>>>> From what I've seen all multicast does is making IETF voice standards harder
>>>>>>> to understand or implement.
>>>>>>>
>>>>>>
>>>>>> multicast itself, but because of Xcast (RFC 5058) which is a promising
>>>>>> technology to replace centralized conference bridges.
>>>>>>
>>>>>
>>>>> Regarding multicast:
>>>>>
>>>>> Teleconference (including mono or spatial audio) might be good starting
>>>>> point. Virtual environments like second live would require multicast
>>>>> communication, too. If the requirements of these scenarios are well
>>>>> understand, we can start to talk about potential solutions like IP
>>> RTP is inherently a group communication protocol, and any codec designed
>>> for use with RTP should consider operation in various different types of
>>> group communication scenario (not just multicast). RFC 5117 is a good place
>>> to start when considering the different types of topology in which RTP is
>>> used, and the possible placement of mixing and switching functions which the
>>> codec will need to work with.
>>>
>>>
>> It is not clear to me what supporting multicast would entail here. If this
>> is a codec over RTP, then what is to stop it from being multicast ?
>>
>
>
> Nothing. However group conferences implemented using multicast require end
> system mixing of potentially large numbers of active audio streams, whereas
> those implemented using conference bridges do the mixing in a single central
> location, and generally suppress all but one speaker. The differences in
> mixing and the number of simultaneous active streams that might be received
> potentially affect the design of the codec.
>

Conference bridges with central mixing almost always mix multiple speakers.
As you add more streams into the mix, you reduce the chance of missing onset
speech and interruptions, but raise the noise floor. So even if complexity
is not a consideration, there is value in gating the mixer (instead of
always doing a full mix-minus).

More on point, compressed domain mixing and easy detection of VAD have both
been advocated on these lists, and both simplify the large-scale mixing
problem.

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 21 Apr 2010 18:58:19 +0200

Hi, 
 
the teleconferencing issue gets complex. I am trying to compile the different requirements that have been mentioned on this list.
-          low complexity (with just one active speaker) vs. multiple speaker mixing vs. spatial audio/stereo mixing
-          centralized vs. distributed
-          few participants vs. hundreds of listeners and talkers
-          individual distribution of audio streams vs. IP multicast or RTP group communication
-          efficient encoding of multiple streams having the same content (but different quality).
-           I bet I missed some.
 
To make things easier, why not to split the teleconferencing scenario in two: High quality and Scalable?
 
The high quality scenario, intended for a low number of users, could have features like
-          Distributed processing and mixing
-          High computational resources to support spatial audio mixing (at the receiver) and multiple encodings of the same audio
stream at different qualities (at the sender)
-          Enough bandwidth to allow direct N to N transmissions of audio streams (no multicast or group communication). This would
be good for the latency, too.
 
The scalable scenario is the opposite:
-          Central processing and mixing for many participants .
-          N to 1 and 1 to N communication using efficient distribution mechanisms (RTP group communication and IP multicast).
-          Low complexity mixing of many using tricks like VAD, encoding at lowest rate to support many receivers having different
paths, you name it...
 
Then, we need not to compare apples with oranges all the time.
 
Christian
 
---------------------------------------------------------------
Dr.-Ing. Christian Hoene
Interactive Communication Systems (ICS), University of Tübingen 
Sand 13, 72076 Tübingen, Germany, Phone +49 7071 2970532 
 <http://www.net.uni-tuebingen.de/> http://www.net.uni-tuebingen.de/
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of stephen botzko
Sent: Wednesday, April 21, 2010 4:34 PM
To: Colin Perkins
Cc: [email-masked]; [email-masked]
Subject: Re: [codec] #16: Multicast?
 
in-line

Stephen Botzko
The question arrose whether the interactive CODEC MUST support multicast in addition to teleconferencing.

On 04/13/2010 11:35 AM, Christian Hoene wrote:
P.S. On the same note, does anybody here cares about using this CODEC with multicast? Is there a single commercial multicast voice
deployment? From what I've seen all multicast does is making IETF voice standards harder to understand or implement.

I think that would be a mistake to ignore multicast - not because of multicast itself, but because of Xcast (RFC 5058) which is a
promising technology to replace centralized conference bridges.

Regarding multicast:

I think we shall start at user requirements and scenarios. Teleconference (including mono or spatial audio) might be good starting
point. Virtual environments like second live would require multicast communication, too. If the requirements of these scenarios are
well understand, we can start to talk about potential solutions like IP multicast, Xcast or conference bridges.

RTP is inherently a group communication protocol, and any codec designed for use with RTP should consider operation in various
different types of group communication scenario (not just multicast). RFC 5117 is a good place to start when considering the
different types of topology in which RTP is used, and the possible placement of mixing and switching functions which the codec will
need to work with.

It is not clear to me what supporting multicast would entail here. If this is a codec over RTP, then what is to stop it from being
multicast ?
 
Nothing. However group conferences implemented using multicast require end system mixing of potentially large numbers of active
audio streams, whereas those implemented using conference bridges do the mixing in a single central location, and generally suppress
all but one speaker. The differences in mixing and the number of simultaneous active streams that might be received potentially
affect the design of the codec.

Conference bridges with central mixing almost always mix multiple speakers.  As you add more streams into the mix, you reduce the
chance of missing onset speech and interruptions, but raise the noise floor. So even if complexity is not a consideration, there is
value in gating the mixer (instead of always doing a full mix-minus).

More on point, compressed domain mixing and easy detection of VAD have both been advocated on these lists, and both simplify the
large-scale mixing problem.

-- 
Colin Perkins
http://csperkins.org/

---

From: stephen botzko <[email-masked]>
Date: Wed, 21 Apr 2010 13:10:12 -0400

I agree there are lots of use cases.

Though I don't see why high quality has to be given up in order to be
scalable.

Also, I am not sure why you think central mixing is more scalable than
multicast (or why you think it is lower quality either).

Stephen Botzko

On Wed, Apr 21, 2010 at 12:58 PM, Christian Hoene <[email-masked]>wrote:

>  Hi,
>
>
>
> the teleconferencing issue gets complex. I am trying to compile the
> different requirements that have been mentioned on this list.
>
> -          low complexity (with just one active speaker) vs. multiple
> speaker mixing vs. spatial audio/stereo mixing
>
> -          High computational resources to support spatial audio mixing
> (at the receiver) and multiple encodings of the same audio stream at
> different qualities (at the sender)
>
> -          Enough bandwidth to allow direct N to N transmissions of audio
> streams (no multicast or group communication). This would be good for the
> latency, too.
>
>
>
> -          N to 1 and 1 to N communication using efficient distribution
> mechanisms (RTP group communication and IP multicast).
>
> -          Low complexity mixing of many using tricks like VAD, encoding
> at lowest rate to support many receivers having different paths, you name
> it...
>
>
>
> http://www.net.uni-tuebingen.de/
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *stephen botzko
> *Sent:* Wednesday, April 21, 2010 4:34 PM
> *To:* Colin Perkins
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [codec] #16: Multicast?
>
>
>
> in-line
>
> Stephen Botzko
>
> I think we shall start at user requirements and scenarios. Teleconference
> (including mono or spatial audio) might be good starting point. Virtual
> environments like second live would require multicast communication, too. If
> the requirements of these scenarios are well understand, we can start to
> talk about potential solutions like IP multicast, Xcast or conference
> bridges.
>
>
>
> Conference bridges with central mixing almost always mix multiple
> speakers.  As you add more streams into the mix, you reduce the chance of
> missing onset speech and interruptions, but raise the noise floor. So even
> if complexity is not a consideration, there is value in gating the mixer
> (instead of always doing a full mix-minus).
>

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 21 Apr 2010 13:23:51 -0400

On Apr 21, 2010, at 8:17 AM, Colin Perkins wrote:

>>>>>>> P.S. On the same note, does anybody here cares about using  
>>>>>>> this CODEC with multicast? Is there a single commercial  
>>>>>>> multicast voice deployment? From what I've seen all multicast  
>>>>>>> does is making IETF voice standards harder to understand or  
>>>>>>> implement.
>>>>>>
> Nothing. However group conferences implemented using multicast  
> require end system mixing of potentially large numbers of active  
> audio streams, whereas those implemented using conference bridges do  
> the mixing in a single central location, and generally suppress all  
> but one speaker. The differences in mixing and the number of  
> simultaneous active streams that might be received potentially  

Yes, I can see that. However, this seems to be not so much a function  
of the transport protocol as on the desired use - a distributed  
conference or a large scale meeting with many simultaneous active  
streams (such as the old Access Grid) is likely to have the same  
issues whether the transport is unicast or multicast or P2P.

At any rate, I would regard multicast support as a SHOULD.

Regards
Marshall

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 21 Apr 2010 19:27:19 +0200

Hi Stephen,
 
not too bad. You answered faster than the mailing list distributes…
 
Comments inline:

From: stephen botzko [mailto:[email-masked]] 
Sent: Wednesday, April 21, 2010 7:10 PM
To: Christian Hoene
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Though I don't see why high quality has to be given up in order to be scalable.  
CH: These are just experiences from our lab. A spatial audio conference server including the acoustic 3D sound rendering needs a LOT
of processing power. In the end, we have to remain realistic. Processing power is always limited thus if we need a lot then we
cannot serve many clients.
Also, I am not sure why you think central mixing is more scalable than multicast (or why you think it is lower quality either).
CH: With multicast, you need N times 1:N multicast distribution trees (somewhat small tan O(n)=n²).  With central mixing you need N
times 2 transmission paths (O(n)=n). Also, this distributed mixing you need N times the mixing at each client. With centralized, you
can live with one mixing for all (and some tricks for serving the talkers).
Cheers,
 Christian

Stephen Botzko 
On Wed, Apr 21, 2010 at 12:58 PM, Christian Hoene < <mailto:[email-masked]> [email-masked]> wrote:
Hi, 

 

 
Christian

 
in-line

Stephen Botzko

Regarding multicast:

 

-- 
Colin Perkins
http://csperkins.org/

---

From: stephen botzko <[email-masked]>
Date: Wed, 21 Apr 2010 14:19:27 -0400

Inline

> not too bad. You answered faster than the mailing list distributes…
Not sure how that happened!

>
>
> Comments inline:
>
>
>
>
>
> *From:* stephen botzko [mailto:[email-masked]]
> *Sent:* Wednesday, April 21, 2010 7:10 PM
> *To:* Christian Hoene
> *Cc:* [email-masked]
>
> CH: These are just experiences from our lab. A spatial audio conference
> server including the acoustic 3D sound rendering needs a LOT of processing
> power. In the end, we have to remain realistic. Processing power is always
> limited thus if we need a lot then we cannot serve many clients.
>
> CH: With multicast, you need N times 1:N multicast distribution trees
> (somewhat small tan O(n)=n²).  With central mixing you need N times 2
> transmission paths (O(n)=n). Also, this distributed mixing you need N times
> the mixing at each client. With centralized, you can live with one mixing
> for all (and some tricks for serving the talkers).
>
I agree you need more distribution trees for multicast if you allow every
site to talk. There is a corresponding benefit, since there is no central
choke point and also less bandwidth on shared WAN links.

In the distributed case,  you don't need an N-way mixer at each client, and
you also don't need to continuously receive payload on all N streams at each
client either.  In practice you can cap N at a relatively small number (in
the 3-8 range) no matter how large the conference gets.  In a large
conference, you can even choose to drop your comfort noise if you are
receiving two or more streams, and just send enough to keep your firewall
pinhole open.  This is all assuming a suitable voice activity measure in the
RTP packet.  Of course in the worst case, you will receive all N streams.

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 21 Apr 2010 21:27:03 +0200

Hi,
 
if we take those two scenarios (high quality and scalable teleconferencing), what are then the CODEC requirements?
 
High quality:
-          Quite the same requirement as an end-to-end audio transmission: high quality and low latency.
-          Maybe additionally: variable bit rate encoding to achieve a multiplexing gain at the receiver
-          and thus, a fast control loop to cope with variable bitrates on transmission paths.
-          Maybe stereo/multichannel support to send the spatial audio to the headphone or loudspeakers.
 
Scalable:
-          Efficient encoding/transcoding for multiple different qualities (at the conference bridge)
-          The control loop must not react (fast) because (multicast) group communication requires to encode at low quality anyhow.
-          Receiver side activity detection for music and voice having low complexity (for the conference bridge)
-          Efficient mixing of two to four(?) active flows (is this achievable without the complete process of decoding and encoding
again?)
 
Are any teleconferencing requirements missing?
 
 Christian

 
 
From: stephen botzko [mailto:[email-masked]] 
Sent: Wednesday, April 21, 2010 8:19 PM
To: Christian Hoene
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?
 
Inline
On Wed, Apr 21, 2010 at 1:27 PM, Christian Hoene <[email-masked]> wrote:
Hi Stephen,

Comments inline:

 

I agree you need more distribution trees for multicast if you allow every site to talk. There is a corresponding benefit, since
there is no central choke point and also less bandwidth on shared WAN links.

In the distributed case,  you don't need an N-way mixer at each client, and you also don't need to continuously receive payload on
all N streams at each client either.  In practice you can cap N at a relatively small number (in the 3-8 range) no matter how large
the conference gets.  In a large conference, you can even choose to drop your comfort noise if you are receiving two or more
streams, and just send enough to keep your firewall pinhole open.  This is all assuming a suitable voice activity measure in the RTP
packet.  Of course in the worst case, you will receive all N streams.
 
Cheers,
 Christian

Stephen Botzko 

 

 
Christian

 
in-line

Stephen Botzko

Regarding multicast:

 

-- 
Colin Perkins
http://csperkins.org/

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 22 Apr 2010 18:03:37 -0700

Hi Christian,

My comments about your question of CODEC requirements are in-line.

Raymond

From: [email-masked] [mailto:[email-masked]] On Behalf Of Christian Hoene
Sent: Wednesday, April 21, 2010 12:27 PM
To: 'stephen botzko'
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi,

High quality:

-          Quite the same requirement as an end-to-end audio transmission: high quality and low latency.
[Raymond]: High quality is a given, but I would like to emphasize the importance of low latency.
(1) It is well-known that the longer the latency, the lower the perceived quality of the communication link.  The E-model in the ITU-T Recommendation G.107 models such communication quality in MOS_cqe, which among other things depends on the so-called "delay impairment factor" Id.  Basically, MOS_cqe is a monotonically decreasing function of increasing latency, and beyond about 150 ms one-way delay, the perceived quality of the communication link drops rapidly with further delay increase.
(2) The lower the latency, the less audible the echo, and thus the lower the required echo return loss.  Hence, lower latency means easier echo control and simpler echo canceller, and as people already mentioned previously, below a certain delay, an echo is simply perceived as a harmless side-tone and no echo canceller is needed. It seems to me that echo control in conference calls is more difficult than in point-to-point calls.  While I hardly ever heard echoes in domestic point-to-point calls, in my experience with conference calls at work, even with the G.711 codec (which has almost no delay), sometimes I still hear echoes (I just heard another one this afternoon).  If a relatively long-delay IETF codec is used, the echo control will be even more problematic.
(3) In normal phone calls or conference calls, people routinely have a need to interrupt each other, but beyond a certain point, long latency makes it very difficult for people to interrupt each other on the call.  This is because when you try to interrupt another person, that person doesn't hear your interruption until a certain time later, so he keeps talking, but when you hear that he did not stop talking when you interrupted, you stop; then, he hears your interruption, so he stops. When you hear he stops, you start talking again, but then he also hears you stopped (due to the long delay), so he also starts talking again.  The net result is that with a long latency, when you try to interrupt him, you and he end up stopping and starting at roughly the same time for a few cycles, making it difficult to interrupt each other.
(4) We need to keep in mind that the IETF codec may not be the only codec involved in a phone call or a conference call.  We cannot assume that all conference call participants will be using a computer to conduct the call. Not only do people use cell phones for point-to-point phone calls, they also often use cell phones to call in to conference calls.  The one-way delay for a cell phone call through one carriers network is typically around 80 to 110 ms.  A call from a cell phone in a carrier network to another cell phone in a different type of carrier network can easily double this delay to 160 ~ 220 ms and makes the total one-way delay already far exceeding the 150 ms mentioned in (1) above.  Any coding delay added by the IETF codec will be on top of that long delay, and such coding delay will be applied twice when both cell phones call through the IETF codec to a conference bridge.  Even without the IETF codec delay, when I previously called from a Verizon cell phone to an AT&T cell phone, I already experienced the problem mentioned in (3) sometimes.  If the IETF codec has a relatively long delay, adding two times the IETF codec one-way delay to the already long delay of 160 ~ 220 ms will make the situation much worse.  Even if just one cell phone is involved in a conference call, adding twice the one-way delay of a relatively long-delay IETF codec can still easily push the total one-way delay beyond 150 ms.
To summarize, my point is that to help reduce potential echo problems and to ensure a high-quality experience in such a conference call, the IETF codec should have a delay as low as possible while maintaining good enough speech quality and a reasonable bit-rate.

Scalable:

-          Efficient encoding/transcoding for multiple different qualities (at the conference bridge)
[Raymond]: I am not sure whether by "efficient", you meant coding efficiency or computational efficiency.  In any case, I would like to take this opportunity to express my view that although codec complexity isn't much of an issue for PC-to-PC calls where there are GHz of processing power available, the codec complexity is an important issue in certain application scenarios.  The following are just some examples.
1) If a conference bridge has to decode a large number of voice channels, mix, and re-encode, and if compressed-domain mixing cannot be done (which is usually the case), then it is important to keep the decoder complexity low.
2) In topology b) of your other email (IPend-to-transcoding_gateway-to-PSTNend), the transcoding gateway, or VoIP gateway, often has to encode and decode thousands of voice channels in a single box, so not only the computational complexity, but also the per-instance RAM size requirement of the codec become very important for achieving high channel density in the gateway.
3) Many telephone terminal devices at the edge of the Internet use embedded processors with limited processing power, and the processors also have to handle many tasks other than speech coding.  If the IETF codec complexity is too high, some of such devices may not have sufficient processing power to run it.  Even if the codec can fit, some battery-powered mobile devices may prefer to run a lower-complexity codec to reduce power consumption and battery drain.  For example, even if you make a Internet phone call from a computer, you may like the convenience of using a Bluetooth headset that allows you to walk around a bit and have hands-free operation.  Currently most Bluetooth headsets have small form factors with a tiny battery.  This puts a severe constraint on power consumption.  Bluetooth headset chips typically have very limited processing capability, and it has to handle many other tasks such as echo cancellation and noise reduction.  There is just not enough processing power to handle a relatively high-complexity codec.  Most BT headsets today relies on the extremely low-complexity, hardware-based CVSD codec at 64 kb/s to transmit narrowband voice, but CVSD has audible coding noise, so it degrades the overall audio quality.  If the IETF codec has low enough complexity, it would be possible to directly encode and decode the IETF codec bit-stream at the BT headset, thus avoiding the quality degradation of CVSD transcoding.
In summary, my point is that the IETF codec should attempt to achieve a codec complexity as low as possible in both MHz consumption and RAM size requirement while maintaining good enough speech quality.

-          Efficient mixing of two to four(?) active flows (is this achievable without the complete process of decoding and encoding again?)

 Christian

Inline
On Wed, Apr 21, 2010 at 1:27 PM, Christian Hoene <[email-masked]<mailto:[email-masked]>> wrote:
Hi Stephen,

not too bad. You answered faster than the mailing list distributes...
Not sure how that happened!

Comments inline:

From: stephen botzko [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, April 21, 2010 7:10 PM
To: Christian Hoene
Cc: [email-masked]<mailto:[email-masked]>

Though I don't see why high quality has to be given up in order to be scalable.
CH: These are just experiences from our lab. A spatial audio conference server including the acoustic 3D sound rendering needs a LOT of processing power. In the end, we have to remain realistic. Processing power is always limited thus if we need a lot then we cannot serve many clients.
Also, I am not sure why you think central mixing is more scalable than multicast (or why you think it is lower quality either).
CH: With multicast, you need N times 1:N multicast distribution trees (somewhat small tan O(n)=n²).  With central mixing you need N times 2 transmission paths (O(n)=n). Also, this distributed mixing you need N times the mixing at each client. With centralized, you can live with one mixing for all (and some tricks for serving the talkers).
I agree you need more distribution trees for multicast if you allow every site to talk. There is a corresponding benefit, since there is no central choke point and also less bandwidth on shared WAN links.

In the distributed case,  you don't need an N-way mixer at each client, and you also don't need to continuously receive payload on all N streams at each client either.  In practice you can cap N at a relatively small number (in the 3-8 range) no matter how large the conference gets.  In a large conference, you can even choose to drop your comfort noise if you are receiving two or more streams, and just send enough to keep your firewall pinhole open.  This is all assuming a suitable voice activity measure in the RTP packet.  Of course in the worst case, you will receive all N streams.

Cheers,
 Christian

Stephen Botzko
On Wed, Apr 21, 2010 at 12:58 PM, Christian Hoene <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

-           I bet I missed some.

-          High computational resources to support spatial audio mixing (at the receiver) and multiple encodings of the same audio stream at different qualities (at the sender)

-          Enough bandwidth to allow direct N to N transmissions of audio streams (no multicast or group communication). This would be good for the latency, too.

-          Low complexity mixing of many using tricks like VAD, encoding at lowest rate to support many receivers having different paths, you name it...

Christian

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of stephen botzko
Sent: Wednesday, April 21, 2010 4:34 PM
To: Colin Perkins
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [codec] #16: Multicast?

in-line

Stephen Botzko
On Wed, Apr 21, 2010 at 8:17 AM, Colin Perkins <[email-masked]<mailto:[email-masked]>> wrote:
Reporter:  hoene@...                 |       Owner:

On 04/13/2010 11:35 AM, Christian Hoene wrote:
P.S. On the same note, does anybody here cares about using this CODEC with multicast? Is there a single commercial multicast voice deployment? From what I've seen all multicast does is making IETF voice standards harder to understand or implement.

I think that would be a mistake to ignore multicast - not because of multicast itself, but because of Xcast (RFC 5058) which is a promising technology to replace centralized conference bridges.

Regarding multicast:

I think we shall start at user requirements and scenarios. Teleconference (including mono or spatial audio) might be good starting point. Virtual environments like second live would require multicast communication, too. If the requirements of these scenarios are well understand, we can start to talk about potential solutions like IP multicast, Xcast or conference bridges.

RTP is inherently a group communication protocol, and any codec designed for use with RTP should consider operation in various different types of group communication scenario (not just multicast). RFC 5117 is a good place to start when considering the different types of topology in which RTP is used, and the possible placement of mixing and switching functions which the codec will need to work with.

It is not clear to me what supporting multicast would entail here. If this is a codec over RTP, then what is to stop it from being multicast ?

Nothing. However group conferences implemented using multicast require end system mixing of potentially large numbers of active audio streams, whereas those implemented using conference bridges do the mixing in a single central location, and generally suppress all but one speaker. The differences in mixing and the number of simultaneous active streams that might be received potentially affect the design of the codec.

Conference bridges with central mixing almost always mix multiple speakers.  As you add more streams into the mix, you reduce the chance of missing onset speech and interruptions, but raise the noise floor. So even if complexity is not a consideration, there is value in gating the mixer (instead of always doing a full mix-minus).

More on point, compressed domain mixing and easy detection of VAD have both been advocated on these lists, and both simplify the large-scale mixing problem.

--
Colin Perkins
http://csperkins.org/

_______________________________________________
codec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 23 Apr 2010 00:04:32 -0400

Hi,

See me comments below.

> (1) It is well-known that the longer the latency, the lower the 
> perceived quality of the communication link. The E-model in the ITU-T 
> Recommendation G.107 models such communication quality in MOS_cqe, 
> which among other things depends on the so-called “delay impairment 
> factor” /Id/. Basically, MOS_cqe is a monotonically decreasing 
> function of increasing latency, and beyond about 150 ms one-way delay, 
> the perceived quality of the communication link drops rapidly with 
> further delay increase.
>

As the author of CELT, I obviously agree that latency is an important 
aspect for this codec :-) That being said, I tend to say that 20 ms is 
still the most widely used frame size, so we might as well optimise for 
that. This is not really a problem because as the frame size goes down, 
the overhead of the IP/UDP/RTP headers go up, so the codec bit-rate 
becomes a bit less of an issue. For example, with 5 ms frames, we would 
already be sending 64 kb/s worth of headers (excluding the link layer), 
so we might as well spend about as many bits on the actual payload as we 
spend on the headers. And with 64 kb/s of payload, we can actually have 
high-quality full-band audio.

> 1) If a conference bridge has to decode a large number of voice 
> channels, mix, and re-encode, and if compressed-domain mixing cannot 
> be done (which is usually the case), then it is important to keep the 
> decoder complexity low.

Definitely agree here. The decoder complexity is very important. Not 
only because of mixing issue, but also because the decoder is generally 
not allowed to take shortcuts to save on complexity (unlike the 
encoder). As for compressed-domain mixing, as you say it is not always 
available, but *if* we can do it (even if only partially), then that can 
result in a "free" reduction in decoder complexity for mixing.

> 2) In topology b) of your other email 
> (IPend-to-transcoding_gateway-to-PSTNend), the transcoding gateway, or 
> VoIP gateway, often has to encode and decode thousands of voice 
> channels in a single box, so not only the computational complexity, 
> but also the per-instance RAM size requirement of the codec become 
> very important for achieving high channel density in the gateway.
>

Agreed here, although I would say that per-instance RAM -- as long as 
it's reasonable -- is probably a bit less important than complexity.

> 3) Many telephone terminal devices at the edge of the Internet use 
> embedded processors with limited processing power, and the processors 
> also have to handle many tasks other than speech coding. If the IETF 
> codec complexity is too high, some of such devices may not have 
> sufficient processing power to run it. Even if the codec can fit, some 
> battery-powered mobile devices may prefer to run a lower-complexity 
> codec to reduce power consumption and battery drain. For example, even 
> if you make a Internet phone call from a computer, you may like the 
> convenience of using a Bluetooth headset that allows you to walk 
> around a bit and have hands-free operation. Currently most Bluetooth 
> headsets have small form factors with a tiny battery. This puts a 
> severe constraint on power consumption. Bluetooth headset chips 
> typically have very limited processing capability, and it has to 
> handle many other tasks such as echo cancellation and noise reduction. 
> There is just not enough processing power to handle a relatively 
> high-complexity codec. Most BT headsets today relies on the extremely 
> low-complexity, hardware-based CVSD codec at 64 kb/s to transmit 
> narrowband voice, but CVSD has audible coding noise, so it degrades 
> the overall audio quality. If the IETF codec has low enough 
> complexity, it would be possible to directly encode and decode the 
> IETF codec bit-stream at the BT headset, thus avoiding the quality 
> degradation of CVSD transcoding.
>

Any idea what the complexity requirements would be for this use-case to 
be possible?

Cheers,

Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Fri, 23 Apr 2010 01:15:59 -0700

By the time the BlueTooth Special Interest Group will have adopted a  
future IETF codec standard, Moore's law will surely have multiplied  
CPU resources in the BT device by one order of magnitude..?  Not sure  
it makes sense to apply today's BT constraints to tomorrow's codec.

I'm not even convinced BlueTooth is a relevant use case for an  
Internet codec.  BT devices are audio devices more than VoIP end  
points: BT always connects to the Internet through another device.   
You could simply first decode incoming packets and send PCM data to  
the BT device, or use a high-quality/high-bitrate codec like G.722.   
The requirements for BT devices and the Internet are just too  
different.  Similarly, GSM phones use AMR on the network side and a  
different codec towards the BT device.  The required transcoding  
causes no quality problems because BT supports high bitrates.

best,
koen.

Quoting Raymond (Juin-Hwey) Chen:

> if we take those two scenarios (high quality and scalable  
> teleconferencing), what are then the CODEC requirements?
>
> High quality:
>
> -          Quite the same requirement as an end-to-end audio  
> transmission: high quality and low latency.
> perceived quality of the communication link.  The E-model in the  
> ITU-T Recommendation G.107 models such communication quality in  
> MOS_cqe, which among other things depends on the so-called "delay  
> impairment factor" Id.  Basically, MOS_cqe is a monotonically  
> decreasing function of increasing latency, and beyond about 150 ms  
> one-way delay, the perceived quality of the communication link drops  
> rapidly with further delay increase.
> (2) The lower the latency, the less audible the echo, and thus the  
> lower the required echo return loss.  Hence, lower latency means  
> easier echo control and simpler echo canceller, and as people  
> already mentioned previously, below a certain delay, an echo is  
> simply perceived as a harmless side-tone and no echo canceller is  
> needed. It seems to me that echo control in conference calls is more  
> difficult than in point-to-point calls.  While I hardly ever heard  
> echoes in domestic point-to-point calls, in my experience with  
> conference calls at work, even with the G.711 codec (which has  
> almost no delay), sometimes I still hear echoes (I just heard  
> another one this afternoon).  If a relatively long-delay IETF codec  
> is used, the echo control will be even more problematic.
> (3) In normal phone calls or conference calls, people routinely have  
> a need to interrupt each other, but beyond a certain point, long  
> latency makes it very difficult for people to interrupt each other  
> on the call.  This is because when you try to interrupt another  
> person, that person doesn't hear your interruption until a certain  
> time later, so he keeps talking, but when you hear that he did not  
> stop talking when you interrupted, you stop; then, he hears your  
> interruption, so he stops. When you hear he stops, you start talking  
> again, but then he also hears you stopped (due to the long delay),  
> so he also starts talking again.  The net result is that with a long  
> latency, when you try to interrupt him, you and he end up stopping  
> and starting at roughly the same time for a few cycles, making it  
> difficult to interrupt each other.
> (4) We need to keep in mind that the IETF codec may not be the only  
> codec involved in a phone call or a conference call.  We cannot  
> assume that all conference call participants will be using a  
> computer to conduct the call. Not only do people use cell phones for  
> point-to-point phone calls, they also often use cell phones to call  
> in to conference calls.  The one-way delay for a cell phone call  
> through one carriers network is typically around 80 to 110 ms.  A  
> call from a cell phone in a carrier network to another cell phone in  
> a different type of carrier network can easily double this delay to  
> 160 ~ 220 ms and makes the total one-way delay already far exceeding  
> the 150 ms mentioned in (1) above.  Any coding delay added by the  
> IETF codec will be on top of that long delay, and such coding delay  
> will be applied twice when both cell phones call through the IETF  
> codec to a conference bridge.  Even without the IETF codec delay,  
> when I previously called from a Verizon cell phone to an AT&T cell  
> phone, I already experienced the problem mentioned in (3) sometimes.  
>  If the IETF codec has a relatively long delay, adding two times the  
> IETF codec one-way delay to the already long delay of 160 ~ 220 ms  
> will make the situation much worse.  Even if just one cell phone is  
> involved in a conference call, adding twice the one-way delay of a  
> relatively long-delay IETF codec can still easily push the total  
> one-way delay beyond 150 ms.
> To summarize, my point is that to help reduce potential echo  
> problems and to ensure a high-quality experience in such a  
> conference call, the IETF codec should have a delay as low as  
> possible while maintaining good enough speech quality and a  
> reasonable bit-rate.
>
> [Raymond]: I am not sure whether by "efficient", you meant coding  
> efficiency or computational efficiency.  In any case, I would like  
> to take this opportunity to express my view that although codec  
> complexity isn't much of an issue for PC-to-PC calls where there are  
> GHz of processing power available, the codec complexity is an  
> important issue in certain application scenarios.  The following are  
> just some examples.
> be done (which is usually the case), then it is important to keep  
> the decoder complexity low.
> (IPend-to-transcoding_gateway-to-PSTNend), the transcoding gateway,  
> or VoIP gateway, often has to encode and decode thousands of voice  
> embedded processors with limited processing power, and the  
> processors also have to handle many tasks other than speech coding.   
> If the IETF codec complexity is too high, some of such devices may  
> not have sufficient processing power to run it.  Even if the codec  
> can fit, some battery-powered mobile devices may prefer to run a  
> lower-complexity codec to reduce power consumption and battery  
> drain.  For example, even if you make a Internet phone call from a  
> computer, you may like the convenience of using a Bluetooth headset  
> that allows you to walk around a bit and have hands-free operation.   
> Currently most Bluetooth headsets have small form factors with a  
> tiny battery.  This puts a severe constraint on power consumption.   
> Bluetooth headset chips typically have very limited processing  
> capability, and it has to handle many other tasks such as echo  
> cancellation and noise reduction.  There is just not enough  
> processing power to handle a relatively high-complexity codec.  Most  
> BT headsets today relies on the extremely low-complexity,  
> hardware-based CVSD codec at 64 kb/s to transmit narrowband voice,  
> but CVSD has audible coding noise, so it degrades the overall audio  
> quality.  If the IETF codec has low enough complexity, it would be  
> possible to directly encode and decode the IETF codec bit-stream at  
> the BT headset, thus avoiding the quality degradation of CVSD  
> transcoding.
> In summary, my point is that the IETF codec should attempt to  
> achieve a codec complexity as low as possible in both MHz  
> consumption and RAM size requirement while maintaining good enough  
> speech quality.
>
> -          The control loop must not react (fast) because  
> (multicast) group communication requires to encode at low quality  
> anyhow.
>
> -          Receiver side activity detection for music and voice  
> having low complexity (for the conference bridge)
>
> -          Efficient mixing of two to four(?) active flows (is this  
> achievable without the complete process of decoding and encoding  
> again?)
>
> On Wed, Apr 21, 2010 at 1:27 PM, Christian Hoene  
> <[email-masked]<mailto:[email-masked]>> wrote:
> Hi Stephen,
>
> From: stephen botzko  
> [mailto:[email-masked]<mailto:[email-masked]>]
> CH: These are just experiences from our lab. A spatial audio  
> conference server including the acoustic 3D sound rendering needs a  
> LOT of processing power. In the end, we have to remain realistic.  
> Processing power is always limited thus if we need a lot then we  
> Also, I am not sure why you think central mixing is more scalable  
> than multicast (or why you think it is lower quality either).
> CH: With multicast, you need N times 1:N multicast distribution  
> trees (somewhat small tan O(n)=n²).  With central mixing you need N  
> times 2 transmission paths (O(n)=n). Also, this distributed mixing  
> you need N times the mixing at each client. With centralized, you  
> I agree you need more distribution trees for multicast if you allow  
> every site to talk. There is a corresponding benefit, since there is  
> no central choke point and also less bandwidth on shared WAN links.
>
> In the distributed case,  you don't need an N-way mixer at each  
> client, and you also don't need to continuously receive payload on  
> all N streams at each client either.  In practice you can cap N at a  
> relatively small number (in the 3-8 range) no matter how large the  
> conference gets.  In a large conference, you can even choose to drop  
> your comfort noise if you are receiving two or more streams, and  
> just send enough to keep your firewall pinhole open.  This is all  
> assuming a suitable voice activity measure in the RTP packet.  Of  
> course in the worst case, you will receive all N streams.
>
> Cheers,
>  Christian
>
> Stephen Botzko
> On Wed, Apr 21, 2010 at 12:58 PM, Christian Hoene  
> <[email-masked]<mailto:[email-masked]>> wrote:
> Hi,
>
> -          low complexity (with just one active speaker) vs.  
> multiple speaker mixing vs. spatial audio/stereo mixing
>
> To make things easier, why not to split the teleconferencing  
> scenario in two: High quality and Scalable?
>
> -          High computational resources to support spatial audio  
> mixing (at the receiver) and multiple encodings of the same audio  
> -          Enough bandwidth to allow direct N to N transmissions of  
> audio streams (no multicast or group communication). This would be  
> good for the latency, too.
>
> -          N to 1 and 1 to N communication using efficient  
> distribution mechanisms (RTP group communication and IP multicast).
>
> -          Low complexity mixing of many using tricks like VAD,  
> encoding at lowest rate to support many receivers having different  
> paths, you name it...
>
> From: [email-masked]<mailto:[email-masked]>  
> [mailto:[email-masked]<mailto:[email-masked]>] On  
> Behalf Of stephen botzko
> On Wed, Apr 21, 2010 at 8:17 AM, Colin Perkins  
> <[email-masked]<mailto:[email-masked]>> wrote:
> deployment? From what I've seen all multicast does is making IETF  
> voice standards harder to understand or implement.
>
> multicast communication, too. If the requirements of these scenarios  
> are well understand, we can start to talk about potential solutions  
> like IP multicast, Xcast or conference bridges.
>
>
> speakers.  As you add more streams into the mix, you reduce the  
> chance of missing onset speech and interruptions, but raise the  
> noise floor. So even if complexity is not a consideration, there is  
> More on point, compressed domain mixing and easy detection of VAD  
> have both been advocated on these lists, and both simplify the

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 23 Apr 2010 18:43:45 -0700

Hi Jean-Marc,

I agree that the 20 ms frame size or packet size is more efficient in bit-rate.  However, this comment doesn't address my original point on the need to have a low-delay IETF codec for the conferencing bridge scenario, where the voice signal will travel through the codec twice (2 tandems), thus doubling the one-way codec delay.

As you are well aware of, codec design involves many trade-offs between the four major attributes of a codec: delay, complexity, bit-rate, and quality.  For a given codec architecture, improving one attribute normally means sacrificing at least one other attribute.  Nothing comes for free.  Therefore, yes, to get low delay, you need to pay the price of lower bit-rate efficiency, but you can also view it another way: to get higher bit-rate efficiency by using a 20 ms frame size, you pay the price of a higher codec delay.  The question to ask then, is not which frame size is more bit-rate efficient, but whether there are application scenarios where a 20 ms frame size will simply make the one-way delay way too long and greatly degrade the users' communication experience. I believe the answer to the latter question is a definite "yes".

Let's do some math to see why that is so.  Essentially all cellular codecs use a frame size of 20 ms, yet the one-way delay of a cell-to-landline call is typically 80 to 110 ms, or 4 to 5.5 times the codec frame size.  This is because you have not only the codec buffering delay, but also processing delay, transmission delay, and delay due to processor sharing using real-time OS, etc.  An IP phone guru told me that for a typical IP phone application, it is also quite common to see a one-way delay of 5 times the codec frame size.  Let's just take 5X codec frame size as the one-way delay of a typical implementation.  Then, even if all conference participants use their computers to call the conference bridge, if the IETF codec has a frame size of 20 ms, then after the voice signal of a talker goes through the IETF codec to the bridge, it already takes 100 ms one-way delay.  After the bridge decodes all channels, mixes, and re-encodes with the IETF codec and send to every participant, the one-way delay is now already up to 200 ms, way more than the 150 ms limit I mentioned in my last email.  Now if a talker call into the conference bridge through a cell phone call that has 100 ms one-way delay to the edge of the Internet, by the time everyone else hears his voice, it is already 300 ms later.  Anyone trying to interrupt that cell phone caller will experience the talk-stop-talk-stop problem I mentioned before.  Now if another cell phone caller call into the conference bridge, then the one-way delay of his voice to the first cell phone caller will be a whopping 400 ms! That would probably turn it into half-duplex effectively.

When we talk about "high-quality" conference call, it is much more than just the quality or distortion level of the voice signal; the one-way delay is also an important and integral part of the perceived quality of the communication link.  This is clearly documented and well-modeled in the E-model of the ITU-T G.107, and the 150 ms limit, beyond which the perceived quality sort of "falls off the cliff", was also obtained after careful study by telephony experts at the ITU-T.  It would be wise for the IETF codec WG to heed the warning of the ITU-T experts and keep the one-way delay less than 150 ms.

In contrast, if the IETF codec has a codec frame size and packet size of 5 ms, then the on-the-net one-way conferencing delay is only 50 ms. Even if you use a longer jitter buffer, the one-way delay is still unlikely to go above 100 ms, which is still well within the ITU-T's 150 ms guideline.

True, sending 5 ms packets means the packet header overhead would be higher, but that's a small price to pay to enable the conference participants to have a high-quality experience by avoiding the problems associated with a long one-way delay.  The bit-rate penalty is not 64 kb/s as you said, but 3/4 of that, or 48 kb/s, because you don't get zero packet header overhead for a 20 ms frame size, but 16 kb/s, so 64 - 16 = 48.  

Now, with the exception of a small percentage of Internet users who still use dial-up modems, the vast majority of the Internet users today connect to the Internet at a speed of at least several hundred kb/s, and most are in the Mbps range.  A 48 kb/s penalty is really a fairly small price to pay for the majority of Internet users when it can give them a much better high-quality experience with an much lower delay.

Furthermore, it is possible to use header compression technology to shrink that 48 kb/s penalty to almost nothing.

Also, even if a 5 ms packet size is an overkill in some situations, a codec with a 5 ms frame size can easily packs two frames of compressed bit-stream into a 10 ms packet.  Then the packet header overhead bit-rate would be 32 kb/s, so the penalty shrinks by a factor of 3 from 48 kb/s to 32 - 16 = 16 kb/s. With 10 ms packets, the one-way conferencing delay would be 100 ms, still well within the 150 ms guideline. (Actually, since the internal "thread rate" of real-time OS can still run at 5 ms intervals, the one-way delay can be made less than 100 ms, but that's too much detail to go into.) In contrast, a codec with a 20 ms frame size cannot send its bit-stream with 10 ms packets, unless it spreads each frame into two packets, which is what IETF AVT advises against, because it will effectively double the packet loss rate.

The way I see it, for conference bridge applications at least, I think it would be a big mistake for IETF to recommend a codec with a frame size of 20 ms or higher.  From my analysis above, by doing that we will be stuck with too long a delay and the associated problems.   

Best Regards,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Thursday, April 22, 2010 9:05 PM
To: Raymond (Juin-Hwey) Chen
Cc: Christian Hoene; 'stephen botzko'; [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi,

See me comments below.

> which among other things depends on the so-called "delay impairment 
> factor" /Id/. Basically, MOS_cqe is a monotonically decreasing 

Cheers,

Jean-Marc

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 24 Apr 2010 04:05:10 +0200 (CEST)

On Fri, 23 Apr 2010, Raymond (Juin-Hwey) Chen wrote:

> The way I see it, for conference bridge applications at least, I think 
> it would be a big mistake for IETF to recommend a codec with a frame 
> size of 20 ms or higher.  From my analysis above, by doing that we will 
> be stuck with too long a delay and the associated problems.

>From my point of view as a network engineer, it would be a mistake to 
recommend *fixed* frame size at all. The codec needs to be able to handle 
everything from 5ms to 250ms frame size depending on network conditions 
(please see my emails from a few months back regarding the reality of the 
Internet today).

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 23 Apr 2010 22:52:45 -0400

Hi Raymond,

I think I may have been a bit ambiguous in my previous email. I am 
totally in favor of supporting 5 ms frames. TO me, it is becoming clear 
that we want to support both 5 ms *and* 20 ms frames. I don't think it 
would be too hard to support both.

About my comment about the overheard, my point was not to say that 64 
kb/s overhead is necessarily unacceptable. My main point is that the 
codec's "sweet spot" should probably be such that the payload bit-rate 
be in the same order as the header overhead at the selected frame size. 
So when operating with 5 ms frames, since you're already paying 64 kb/s 
in headers, it's probably worth also having 64 kb/s of payload so that 
you can transmit full-band audio. On the other hand, when operating with 
20 ms frames where the overhead is 16 kb/s, then a 16 kb/s payload is 
probably the sweet spot. That way you scale audio quality at the same 
time as you reduce latency.

I hope this clears up the mis-understanding.

Cheers,

	Jean-Marc

On 2010-04-23 21:43, Raymond (Juin-Hwey) Chen wrote:
> Hi Jean-Marc,
>
> I agree that the 20 ms frame size or packet size is more efficient in
> bit-rate.  However, this comment doesn't address my original point on
> the need to have a low-delay IETF codec for the conferencing bridge
> scenario, where the voice signal will travel through the codec twice
> (2 tandems), thus doubling the one-way codec delay.
>
> As you are well aware of, codec design involves many trade-offs
> between the four major attributes of a codec: delay, complexity,
> bit-rate, and quality.  For a given codec architecture, improving one
> attribute normally means sacrificing at least one other attribute.
> Nothing comes for free.  Therefore, yes, to get low delay, you need
> to pay the price of lower bit-rate efficiency, but you can also view
> it another way: to get higher bit-rate efficiency by using a 20 ms
> frame size, you pay the price of a higher codec delay.  The question
> to ask then, is not which frame size is more bit-rate efficient, but
> whether there are application scenarios where a 20 ms frame size will
> simply make the one-way delay way too long and greatly degrade the
> users' communication experience. I believe the answer to the latter
> question is a definite "yes".
>
> Let's do some math to see why that is so.  Essentially all cellular
> codecs use a frame size of 20 ms, yet the one-way delay of a
> cell-to-landline call is typically 80 to 110 ms, or 4 to 5.5 times
> the codec frame size.  This is because you have not only the codec
> buffering delay, but also processing delay, transmission delay, and
> delay due to processor sharing using real-time OS, etc.  An IP phone
> guru told me that for a typical IP phone application, it is also
> quite common to see a one-way delay of 5 times the codec frame size.
> Let's just take 5X codec frame size as the one-way delay of a typical
> implementation.  Then, even if all conference participants use their
> computers to call the conference bridge, if the IETF codec has a
> frame size of 20 ms, then after the voice signal of a talker goes
> through the IETF codec to the bridge, it already takes 100 ms one-way
> delay.  After the bridge decodes all channels, mixes, and re-encodes
> with the IETF codec and send to every participant, the one-way delay
> is now already up to 200 ms, way more than the 150 ms limit I
> mentioned in my last email.  Now if a talker call into the conference
> bridge through a cell phone call that has 100 ms one-way delay to the
> edge of the Internet, by the time everyone else hears his voice, it
> is already 300 ms later.  Anyone trying to interrupt that cell phone
> caller will experience the talk-stop-talk-stop problem I mentioned
> before.  Now if another cell phone caller call into the conference
> bridge, then the one-way delay of his voice to the first cell phone
> caller will be a whopping 400 ms! That would probably turn it into
> half-duplex effectively.
>
> When we talk about "high-quality" conference call, it is much more
> than just the quality or distortion level of the voice signal; the
> one-way delay is also an important and integral part of the perceived
> quality of the communication link.  This is clearly documented and
> well-modeled in the E-model of the ITU-T G.107, and the 150 ms limit,
> beyond which the perceived quality sort of "falls off the cliff", was
> also obtained after careful study by telephony experts at the ITU-T.
> It would be wise for the IETF codec WG to heed the warning of the
> ITU-T experts and keep the one-way delay less than 150 ms.
>
> In contrast, if the IETF codec has a codec frame size and packet size
> of 5 ms, then the on-the-net one-way conferencing delay is only 50
> ms. Even if you use a longer jitter buffer, the one-way delay is
> still unlikely to go above 100 ms, which is still well within the
> ITU-T's 150 ms guideline.
>
> True, sending 5 ms packets means the packet header overhead would be
> higher, but that's a small price to pay to enable the conference
> participants to have a high-quality experience by avoiding the
> problems associated with a long one-way delay.  The bit-rate penalty
> is not 64 kb/s as you said, but 3/4 of that, or 48 kb/s, because you
> don't get zero packet header overhead for a 20 ms frame size, but 16
> kb/s, so 64 - 16 = 48.
>
> Now, with the exception of a small percentage of Internet users who
> still use dial-up modems, the vast majority of the Internet users
> today connect to the Internet at a speed of at least several hundred
> kb/s, and most are in the Mbps range.  A 48 kb/s penalty is really a
> fairly small price to pay for the majority of Internet users when it
> can give them a much better high-quality experience with an much
> lower delay.
>
> Furthermore, it is possible to use header compression technology to
> shrink that 48 kb/s penalty to almost nothing.
>
> Also, even if a 5 ms packet size is an overkill in some situations, a
> codec with a 5 ms frame size can easily packs two frames of
> compressed bit-stream into a 10 ms packet.  Then the packet header
> overhead bit-rate would be 32 kb/s, so the penalty shrinks by a
> factor of 3 from 48 kb/s to 32 - 16 = 16 kb/s. With 10 ms packets,
> the one-way conferencing delay would be 100 ms, still well within the
> 150 ms guideline. (Actually, since the internal "thread rate" of
> real-time OS can still run at 5 ms intervals, the one-way delay can
> be made less than 100 ms, but that's too much detail to go into.) In
> contrast, a codec with a 20 ms frame size cannot send its bit-stream
> with 10 ms packets, unless it spreads each frame into two packets,
> which is what IETF AVT advises against, because it will effectively
> double the packet loss rate.
>
> The way I see it, for conference bridge applications at least, I
> think it would be a big mistake for IETF to recommend a codec with a
> frame size of 20 ms or higher.  From my analysis above, by doing that
> we will be stuck with too long a delay and the associated problems.
>
> Best Regards,
>
> Raymond
>
> -----Original Message----- From: Jean-Marc Valin
> [mailto:[email-masked]] Sent: Thursday, April 22,
> 2010 9:05 PM To: Raymond (Juin-Hwey) Chen Cc: Christian Hoene;
> 'stephen botzko'; [email-masked] Subject: Re: [codec] #16:
> Multicast?
>
> Hi,
>
> See me comments below.
>
>> impairment factor" /Id/. Basically, MOS_cqe is a monotonically
>> one-way delay, the perceived quality of the communication link
>> drops rapidly with further delay increase.
>>
>
> As the author of CELT, I obviously agree that latency is an
> important aspect for this codec :-) That being said, I tend to say
> that 20 ms is still the most widely used frame size, so we might as
> well optimise for that. This is not really a problem because as the
> frame size goes down, the overhead of the IP/UDP/RTP headers go up,
> so the codec bit-rate becomes a bit less of an issue. For example,
> with 5 ms frames, we would already be sending 64 kb/s worth of
> headers (excluding the link layer), so we might as well spend about
> as many bits on the actual payload as we spend on the headers. And
> with 64 kb/s of payload, we can actually have high-quality full-band
> audio.
>
>> channels, mix, and re-encode, and if compressed-domain mixing
>> cannot be done (which is usually the case), then it is important to
>> keep the decoder complexity low.
>
> only because of mixing issue, but also because the decoder is
> generally not allowed to take shortcuts to save on complexity (unlike
> the encoder). As for compressed-domain mixing, as you say it is not
> always available, but *if* we can do it (even if only partially),
> then that can result in a "free" reduction in decoder complexity for
> mixing.
>
>> channels in a single box, so not only the computational
>> complexity, but also the per-instance RAM size requirement of the
>> codec become very important for achieving high channel density in
>> the gateway.
>>
>
> Agreed here, although I would say that per-instance RAM -- as long
> as it's reasonable -- is probably a bit less important than
> complexity.
>
> Any idea what the complexity requirements would be for this use-case
> to be possible?
>
> Cheers,
>
> Jean-Marc
>
>
>
>
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 13:31:06 -0700

Hi Jean-Marc,

Thanks for your helpful clarification.

I totally agree with you and Mikael Abrahamsson that it is unlikely for a single codec frame size to be optimal for all applications.  Different applications can have very different requirements in delay, complexity, bit-rate, and quality, and thus will require different codec trade-offs to get the most optimal solution.  One size just doesn't fit all.  I believe that's why there are so many different speech and audio codecs out there, because no single codec at any single operating point of delay, complexity, bit-rate, and quality can meet the requirements of all possible codec applications.

Given the broad range of different applications that the IETF codec is trying to cover, it is clear to me that the IETF codec needs to have multiple codec "modes" (or "profiles" as some other standard bodies often call them) to cover not only different bit-rates and sampling rates, but also different levels of delay and complexity.  Thus, the IETF codec might need to have a low-bit-rate mode (or high bit-rate efficiency mode), a low-delay mode, a low-complexity mode, etc.

For example, for dial-up modem users, the low-bit-rate mode will be suitable.  For delay-sensitive applications such as the conference bridge applications or any Internet phone call to a cell phone through a cellular network (even point-to-point as oppose to conferencing), the low-delay mode will be suitable.  For devices with very limited processing capabilities such as mono Bluetooth telephone headsets, the low-complexity mode will be suitable.  A conference bridge can even use different IETF codec modes to communicate with different devices in the same conference call at the same time.

Of course, sometimes it may be possible or desirable to combine different modes.  For example, we could have a low-delay-high-quality mode for full-band interactive music performance over the Internet, and we could have a low-delay-low-complexity mode when a Bluetooth headset is used to call a conference bridge.

Regarding your comment that the payload bit-rate should be the same order as the header bit-rate, although it seems to make sense intuitively, I think in reality there are situations where it may not make sense.  For instance, in your example of a 5 ms frame/packet size with a 64 kb/s header bit-rate and 64 kb/s full-band audio bit-rate, what if an IP phone with a 8 kHz sampling rate is used to make a call? Then all the extra bit-rate spent on audio bandwidth above 3.4 kHz (or 4 kHz maximum) is wasted.  Or what if the resulting 128 kb/s is too high for the link because the link still needs to support other data streams?  Similarly, for the 20 ms frame size case, what if you want to transmit full-band audio and 16 kb/s just won't give you the high fidelity level you are looking for?  I understand that you are only talking about a "sweet spot" and are not saying the payload bit-rate and the header bit-rate must be the same, but I am just saying that in certain situations the sweet spot may actually be somewhere else.

Also, I think ideally we should do header compression to minimize the header bit-rate and increase the overall bit-rate efficiency of the system, then there is no "sweet spot" issue as discussed above.  I understand that this may not be possible unless the networking gears support it at both ends, but just from a system perspective, it would seem to me that it doesn't make sense for speech/audio codec developers to try so hard to squeeze the speech/audio signal to as few bits as possible, only to see the packet header having very redundant and repetitive bits sent packet after packet.  There are probably reasons why header compression is not more widely used (which I don't know since it's outside my expertise area), but from a system perspective, it just seems to me to be extremely unbalanced and inefficient to send so many unchanged bits in the header packet after packet, especially in light of the high degree of speech/audio compression done in the payload.

Best Regards,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]]
Sent: Friday, April 23, 2010 7:53 PM
To: Raymond (Juin-Hwey) Chen
Cc: Christian Hoene; 'stephen botzko'; [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi Raymond,

Cheers,

        Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Sat, 24 Apr 2010 13:56:07 -0700

Quoting "Raymond (Juin-Hwey) Chen"

> An IP phone  guru told me that for a typical IP phone application,  
> it is also quite common to see a one-way delay of 5 times the codec  
> frame size.

Sure - for certain frame sizes.  But 1 ms frames won't give you 5 ms  
one-way delay.

For a well-designed system and a typical Internet connection:
- most delay comes from the network and is not codec related, and
- one-way delay grows almost linearly with frame size.

> shrink that 48 kb/s penalty to almost nothing.

Afaik, only RTP headers can be compressed between arbitrary Internet  
end points.  You're still stuck with IP and UDP headers.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen" <[email-masked]>:
> Hi Jean-Marc,
>
> I agree that the 20 ms frame size or packet size is more efficient  
> in bit-rate.  However, this comment doesn't address my original  
> point on the need to have a low-delay IETF codec for the  
> conferencing bridge scenario, where the voice signal will travel  
> through the codec twice (2 tandems), thus doubling the one-way codec  
> delay.
>
> bit-rate, and quality.  For a given codec architecture, improving  
> one attribute normally means sacrificing at least one other  
> attribute.  Nothing comes for free.  Therefore, yes, to get low  
> delay, you need to pay the price of lower bit-rate efficiency, but  
> you can also view it another way: to get higher bit-rate efficiency  
> by using a 20 ms frame size, you pay the price of a higher codec  
> delay.  The question to ask then, is not which frame size is more  
> bit-rate efficient, but whether there are application scenarios  
> where a 20 ms frame size will simply make the one-way delay way too  
> long and greatly degrade the users' communication experience. I  
> believe the answer to the latter question is a definite "yes".
>
>  Let's just take 5X codec frame size as the one-way delay of a  
> typical implementation.  Then, even if all conference participants  
> use their computers to call the conference bridge, if the IETF codec  
> has a frame size of 20 ms, then after the voice signal of a talker  
> goes through the IETF codec to the bridge, it already takes 100 ms  
> one-way delay.  After the bridge decodes all channels, mixes, and  
> re-encodes with the IETF codec and send to every particip
>  ant, the one-way delay is now already up to 200 ms, way more than  
> the 150 ms limit I mentioned in my last email.  Now if a talker call  
> into the conference bridge through a cell phone call that has 100 ms  
> one-way delay to the edge of the Internet, by the time everyone else  
> hears his voice, it is already 300 ms later.  Anyone trying to  
> interrupt that cell phone caller will experience the  
> talk-stop-talk-stop problem I mentioned before.  Now if another cell  
> phone caller call into the conference bridge, then the one-way delay  
> of his voice to the first cell phone caller will be a whopping 400  
> ms! That would probably turn it into half-duplex effectively.
>
> one-way delay is also an important and integral part of the  
> perceived quality of the communication link.  This is clearly  
> documented and well-modeled in the E-model of the ITU-T G.107, and  
> the 150 ms limit, beyond which the perceived quality sort of "falls  
> off the cliff", was also obtained after careful study by telephony  
> experts at the ITU-T.  It would be wise for the IETF codec WG to  
> heed the warning of the ITU-T experts and keep the one-way delay  
> less than 150 ms.
>
> In contrast, if the IETF codec has a codec frame size and packet  
> size of 5 ms, then the on-the-net one-way conferencing delay is only  
> 50 ms. Even if you use a longer jitter buffer, the one-way delay is  
> Also, even if a 5 ms packet size is an overkill in some situations,  
> a codec with a 5 ms frame size can easily packs two frames of  
> the one-way conferencing delay would be 100 ms, still well within  
> the 150 ms guideline. (Actually, since the internal "thread rate" of  
> frame size of 20 ms or higher.  From my analysis above, by doing  
> that we will be stuck with too long a delay and the associated  
> problems.
>
> Best Regards,
>
> Raymond
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 14:40:51 -0700

Hi Jean-Marc,

I would like to respond to a comment and a question in your original email below.

(1) Regarding your comment that "per-instance RAM -- as long as 
it's reasonable -- is probably a bit less important than complexity", I think this may or may not be true depending on what is defined as "reasonable", and it also depends on the VoIP gateway under consideration. However, I think one thing is clear, though: per-instance RAM is definitely not unimportant.  I have worked in a team that designed high-density VoIP gateways, and I know people did all kinds of tricks to reduce the RAM usage as much as possible because of the high cost of the external high-speed SRAM used by high-speed DSPs in such gateways.  

Currently, high-density VoIP gateways can have several tens of thousands of voice channels per box or per equipment rack.  If codec A uses X kbytes more per-instance RAM than codec B, than implementing codec A rather than codec B in all channels will require X * (number of channels) more kbytes of high-speed SRAM, which can be a substantial additional cost.  Similarly, if codec A requires a substantially higher DSP MHz than codec B, then implementing codec A in all channels will mean that more high-speed DSP chips need to be used to maintain the same number of channels.  That's additional cost.  Actually, the more likely scenario is that the number of DSP chips has already maxed out due to board space or heat dissipation constraint, then the gateway can only support a substantially smaller number of voice channels when compared with using codec B.  Consequently, the channel density per box or per rack goes down substantially, and the per-channel deployment cost goes up substantially.  Thus, even if implementing codec A or codec B makes no practical difference in a PC, it can make a very real and substantial cost difference in VoIP gateways.

An important application for the IETF codec is identified as "IPend-to-transcoding_gateway-to-PSTNend".  If the deployment cost of the IETF codec in the VoIP gateway is too high, that will limit the degree of such deployment, thus reducing the availability of such IP-to-PSTN calls and the usefulness of the IETF codec in this application scenario. 

(2) Regarding your question about the complexity requirement for the Bluetooth headset use case to be possible, for the current generation of low-end to mid-end Bluetooth mono headsets where most of the volume is, the 64 kb/s narrowband CVSD codec is universally used but is implemented in chip hardware and consumes no DSP or host processor cycles. (Many of them don't even have a DSP on chip and just have a host processor such as ARM7.)  All the processor cycles are being used by other tasks such as Bluetooth stack, echo control, noise suppression, packet loss concealment, etc.  Therefore, there actually aren't meaningful available processor cycles for a non-hardware-based codec unless we remove some other DSP functions or reduce their MHz consumption to make room for a firmware-implemented codec.

For the low- to mid-end Bluetooth headsets currently under development, the situation is not that much better.  Again we have to remove other functions or reduce their MHz usage to make room for a firmware-based codec.  Given such tight resources constraints, I would say ideally the full-duplex complexity of such a firmware-based codec should be no more than about 5 MHz on a typical 16-bit fixed-point DSP.  In a few years time, the acceptable MHz number may go up to 10 MHz, or at most 15 MHz.

Best Regards,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Thursday, April 22, 2010 9:05 PM
To: Raymond (Juin-Hwey) Chen
Cc: Christian Hoene; 'stephen botzko'; [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi,

See me comments below.

Cheers,

Jean-Marc

---

From: stephen botzko <[email-masked]>
Date: Sat, 24 Apr 2010 19:36:51 -0400

Agreed, this delay model is too simplisti to be useful.

There's algorithmic delay (including framing) + flight time + dejittering.
Flight time depends on the network path, not on the frame size.  And the
amount of jitter is due principally to cross-congestion.

Stephen Botzko

On Sat, Apr 24, 2010 at 4:56 PM, Koen Vos <[email-masked]> wrote:

>  An IP phone  guru told me that for a typical IP phone application, it is
>> also quite common to see a one-way delay of 5 times the codec frame size.
>>
>
> Afaik, only RTP headers can be compressed between arbitrary Internet end
> points.  You're still stuck with IP and UDP headers.
>
> best,
> koen.
>
>
>
>
>> bit-rate.  However, this comment doesn't address my original point on the
>> need to have a low-delay IETF codec for the conferencing bridge scenario,
>> where the voice signal will travel through the codec twice (2 tandems), thus
>> doubling the one-way codec delay.
>>
>> As you are well aware of, codec design involves many trade-offs between
>> the four major attributes of a codec: delay, complexity, bit-rate, and
>> quality.  For a given codec architecture, improving one attribute normally
>> means sacrificing at least one other attribute.  Nothing comes for free.
>>  Therefore, yes, to get low delay, you need to pay the price of lower
>> bit-rate efficiency, but you can also view it another way: to get higher
>> bit-rate efficiency by using a 20 ms frame size, you pay the price of a
>> higher codec delay.  The question to ask then, is not which frame size is
>> more bit-rate efficient, but whether there are application scenarios where a
>> 20 ms frame size will simply make the one-way delay way too long and greatly
>> degrade the users' communication experience. I believe the answer to the
>> latter question is a definite "yes".
>>
>> Let's do some math to see why that is so.  Essentially all cellular codecs
>> use a frame size of 20 ms, yet the one-way delay of a cell-to-landline call
>> is typically 80 to 110 ms, or 4 to 5.5 times the codec frame size.  This is
>> because you have not only the codec buffering delay, but also processing
>> delay, transmission delay, and delay due to processor sharing using
>> real-time OS, etc.  An IP phone guru told me that for a typical IP phone
>> application, it is also quite common to see a one-way delay of 5 times the
>> codec frame size.  Let's just take 5X codec frame size as the one-way delay
>> of a typical implementation.  Then, even if all conference participants use
>> their computers to call the conference bridge, if the IETF codec has a frame
>> size of 20 ms, then after the voice signal of a talker goes through the IETF
>> codec to the bridge, it already takes 100 ms one-way delay.  After the
>> bridge decodes all channels, mixes, and re-encodes with the IETF codec and
>> send to every particip
>>  ant, the one-way delay is now already up to 200 ms, way more than the 150
>> ms limit I mentioned in my last email.  Now if a talker call into the
>> conference bridge through a cell phone call that has 100 ms one-way delay to
>> the edge of the Internet, by the time everyone else hears his voice, it is
>> already 300 ms later.  Anyone trying to interrupt that cell phone caller
>> will experience the talk-stop-talk-stop problem I mentioned before.  Now if
>> another cell phone caller call into the conference bridge, then the one-way
>> delay of his voice to the first cell phone caller will be a whopping 400 ms!
>> That would probably turn it into half-duplex effectively.
>>
>> When we talk about "high-quality" conference call, it is much more than
>> just the quality or distortion level of the voice signal; the one-way delay
>> is also an important and integral part of the perceived quality of the
>> communication link.  This is clearly documented and well-modeled in the
>> E-model of the ITU-T G.107, and the 150 ms limit, beyond which the perceived
>> quality sort of "falls off the cliff", was also obtained after careful study
>> by telephony experts at the ITU-T.  It would be wise for the IETF codec WG
>> to heed the warning of the ITU-T experts and keep the one-way delay less
>> than 150 ms.
>>
>> In contrast, if the IETF codec has a codec frame size and packet size of 5
>> ms, then the on-the-net one-way conferencing delay is only 50 ms. Even if
>> you use a longer jitter buffer, the one-way delay is still unlikely to go
>> above 100 ms, which is still well within the ITU-T's 150 ms guideline.
>>
>> participants to have a high-quality experience by avoiding the problems
>> associated with a long one-way delay.  The bit-rate penalty is not 64 kb/s
>> as you said, but 3/4 of that, or 48 kb/s, because you don't get zero packet
>> header overhead for a 20 ms frame size, but 16 kb/s, so 64 - 16 = 48.
>>
>> Now, with the exception of a small percentage of Internet users who still
>> use dial-up modems, the vast majority of the Internet users today connect to
>> the Internet at a speed of at least several hundred kb/s, and most are in
>> the Mbps range.  A 48 kb/s penalty is really a fairly small price to pay for
>> the majority of Internet users when it can give them a much better
>> high-quality experience with an much lower delay.
>>
>> codec with a 5 ms frame size can easily packs two frames of compressed
>> bit-stream into a 10 ms packet.  Then the packet header overhead bit-rate
>> would be 32 kb/s, so the penalty shrinks by a factor of 3 from 48 kb/s to 32
>> - 16 = 16 kb/s. With 10 ms packets, the one-way conferencing delay would be
>> 100 ms, still well within the 150 ms guideline. (Actually, since the
>> internal "thread rate" of real-time OS can still run at 5 ms intervals, the
>> one-way delay can be made less than 100 ms, but that's too much detail to go
>> into.) In contrast, a codec with a 20 ms frame size cannot send its
>> bit-stream with 10 ms packets, unless it spreads each frame into two
>> packets, which is what IETF AVT advises against, because it will effectively
>> The way I see it, for conference bridge applications at least, I think it
>> would be a big mistake for IETF to recommend a codec with a frame size of 20
>> ms or higher.  From my analysis above, by doing that we will be stuck with
>> too long a delay and the associated problems.
>>
>> Best Regards,
>>
>> Raymond
>>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 17:36:47 -0700

> Sure - for certain frame sizes.  But 1 ms frames won't give you 5 ms  

Of course not.  That's why I added the sentence: "Even if you use a longer jitter buffer, the one-way delay is still unlikely to go above 100 ms, ..." in that email you quoted. I knew that below a certain threshold of the codec frame size, you most likely want to use more frames of jitter buffer delay, thus my added sentence above.  My main point, though, is not in the exact one-way delay value for a codec with a 5 ms frame size, but rather that with a 5 ms frame size you can get a much lower one-way delay than with a 20 ms frame size.  I don't think anyone can credibly argue that this statement is untrue.

> - one-way delay grows almost linearly with frame size.

Doesn't your last line above contradicts with the second last line? I am a bit confused about what you are trying to say.  

> end points.  You're still stuck with IP and UDP headers.

I am aware of a header compression technology for VoIP over Cable applications that can compress the header size to a very small fraction of the original size, but it is probably not widely used.

Raymond

---

From: Koen Vos <[email-masked]>
Date: Sat, 24 Apr 2010 18:16:20 -0700

Quoting "Raymond (Juin-Hwey) Chen":
> My main point, though, is not in the exact one-way delay value for a  
> codec with a 5 ms frame size, but rather that with a 5 ms frame size  
> you can get a much lower one-way delay than with a 20 ms frame size.

It would be about 15 ms lower - don't know if that counts as "much" :)

Also, note that for a given probability of packets arriving too late  
to be played out, the jitter buffer delay is independent of the frame  
size.

> Doesn't your last line above contradicts with the second last line?

I meant that approximately:
    one-way delay = codec-independent delay + frame size

("codec algorithmic delay" would be more accurate than "frame size")

> I am aware of a header compression technology for VoIP over Cable  
> applications that can compress the header size to a very small  
> fraction of the original size, but it is probably not widely used.

Yes, header compression works between end-points on a cable.  That's  
different from "between arbitrary Internet end points".

best,
koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 18:29:42 -0700

On Saturday, April 24, 2010 at 4:37 PM, Stephen Botzko wrote:

   Sure - for certain frame sizes.  But 1 ms frames won't give you 5 ms one-way delay.
> There's algorithmic delay (including framing) + flight time + dejittering.  Flight time depends on the > network path, not on the frame size.  And the amount of jitter is due principally to cross-congestion.
I also agree.  See my last email.   My main point was not the absolute one-way delay value for the 5 ms frame size but the relative delay between 5 ms and 20 ms frame sizes.  I agree that the 5X delay might be too simplistic.  I tried to use a simple formula to make it is easier for people to follow, but I did realize its limitation especially at a small frame size, so I added that "Even if you use a longer jitter buffer, ..." sentence.
Regardless of whether 5X frame size is overly simplistic, the fact remains that cellular codecs have a 20 ms frame size and have a typical one-way delay around 80 to 110 ms or so, and the cellular networks probably don't have the kind of jitter that the Internet has.  What would make us believe that an IETF codec with a 20 ms frame size will get a one-way delay much below 80 ms?  Chances are an Internet call using an IETF codec with a 20 ms frame size will likely have a one-way delay at least as long as the one-way delay of a cell phone call, and more likely to be longer, because PC audio driver software tends to add quite a bit of delay, and an Internet call incurs additional jitter buffer delay when compared with cell phone calls.
Therefore, regardless of the accuracy of the 5X frame size formula, the conclusion remains the same: for the conference bridge application, a 20 ms codec frame size will result in the total one-way delay far exceeding the ITU-T's 150 ms guideline, thus substantially degrading the perceived quality of the communication links, and with one or even two cell phone callers joining the conference, the long latency and the associated problems will just get much worse.  Therefore, it is necessary for the IETF codec to have a low-delay mode using a small codec frame size such as 5 ms to address delay-sensitive applications such as bridge-based conference calls.
Raymond

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 25 Apr 2010 04:45:20 +0200 (CEST)

On Sat, 24 Apr 2010, Raymond (Juin-Hwey) Chen wrote:

> Of course not.  That's why I added the sentence: "Even if you use a 
> longer jitter buffer, the one-way delay is still unlikely to go above 
> 100 ms, ..." in that email you quoted. I knew that below a certain

The light-speed-in-fiber delay RTT is 1ms per 100km. Europe - US West 
coast is ~150ms RTT. I'm in Thailand at the momen. I have 350ms RTT to 
Sweden currently (because the path goes sweden->us->singapore>thailand), 
just to give some datapoints. Add then ADSL2+ 25ms RTT just over the 
access layer, and I'd say that 200ms network RTT (100ms one-way) might be 
a low percentage of the calls, but it's still definitely going to happen. 
Considering the prices of internationall calls out of a place like this, a 
lot of people are going to want to use it to get around it.

I talked to my mother over skype yesterday, because international calls 
are like 0.5USD/minute. To get around the huge jitter and packet loss I 
have on the access, it cranked up the jitter buffers to 2-3seconds. I 
still prefer this to paying 0.5USD/minute.

So these are the diverse network conditions that the solution (where the 
codec is a part) needs to handle, it needs to be able to do interactive 
voice etc over a local network in cd (at least) quality, plus it needs to 
give intelligeble speech communication (walkie talkie quality if needed) 
over very adverse network conditions.

GPRS-EDGE here works fairly well, better than the Wifi I'm on, it might 
very well be that using GPRS-EDGE with 4pps (250ms per packet so as to get 
vey low packet header overhead)) and a very low-bw codec would be better 
than the Wifi.

I absolutely agree that we need quite a few different modes, and these 
need to be decoupled from each other because different conditions need 
different responses from the solution. Low pps for some, low quality but 
high pps for others, and of course all the other combinations of the 
factors involved.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 21:10:29 -0700

Hi Koen,

My comments in-line below.

Best Regards,

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]]
Sent: Saturday, April 24, 2010 6:16 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast?

[Raymond]: I don't agree that it will be only 20 - 5 = 15 ms lower.  That will be true only if your one-way delay formula below is true, but theoretically it cannot be.  See my comment below your formula.

size.

[Raymond]: That may be true theoretically, but in practical implementations, selecting a jitter buffer delay that is not divisible by the packet size would make the adaptive jitter buffer pretty messy to implement.  If we make the it divisible by the packet size, then a smaller packet size gives you more granularity to work with and can result in lower average delay as the codec frames go through the adaptive jitter buffer.

[Raymond]: First, I agree that codec algorithmic buffering delay is more accurate than frame size since it can also include the "look-ahead" delay and filtering delay if sub-band analysis/synthesis is used.  However, your formula implies that for the codec-related delay, the "multiplier" to be used for the codec frame size is only 1.  That's unrealistic and theoretically impossible.  For that to happen, after you wait one frame of time for the current frame of input audio samples to arrive at your input signal buffer (that's one frame of codec-related delay already), you need an infinitely fast processor to finish the encoding operation instantly, then you need an infinitely fast communication link to ship all the bits in the compressed frame to the decoder instantly, and then you need an infinitely fast processor to finish decoding the frame instantly and start playing back the current frame of audio without any delay.  That's just impossible.

In reality, if the processor is just barely fast enough to implement the codec in real time, then you need nearly a full frame of time to finish the encoding and decoding operations. That makes the multiplier to be 2 already.  If your communication link is just barely fast enough to transmit your packets at the same speed they are generated without piling up unsent packets, then it takes another frame of time to finish transmitting the compressed bits in a frame to the decoder.  That makes the multiplier to be 3 already.

Granted, in practice the processor and the communication link are usually faster than just barely enough, so the processing delay and the transmission delay can be less than 1 frame each.  However, there are other miscellaneous uncounted delays that tends to depend on the codec size in various ways.  Thus, a typical IP phone implementation would have

  One-way delay = codec-independent delay + 3*(codec frame size) + (codec look-ahead) + (codec filtering delay if any).

Hence, the one-way delay difference between a 20 ms and a 5 ms codec frame size would be 45 ms + (codec look-ahead difference) + (codec filtering delay difference).

Consequently, for the conference bridge application, the total difference in one-way delay can easily be in the 90 to 100 ms range. When adding this delay difference to all the other codec-independent delay components, it is still a huge difference that the users can easily notice, especially since it will most likely push the total one-way delay significantly beyond the 150 ms limit.

[Raymond]: The cable operators' networks are still IP networks.  If the technology can work there, I don't see why it cannot work elsewhere in the Internet.  I know it is currently not available between arbitrary Internet end points.  I am just saying that technologies exist that can potentially be deployed in the Internet to compress the packet headers to a very small fraction of the uncompressed headers.

best,

koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sat, 24 Apr 2010 23:00:31 -0700

Hi Koen,

Responding to your earlier email about Bluetooth headset application:

(1) Although BT SIG standardization is a preferred route, it is technically feasible to negotiate and use a non-Bluetooth-SIG codec.

(2) Someone familiar with BT SIG told me that it would probably take only 6 months to add an optional codec to the BT SIG spec and 12 to 18 months to add a mandatory codec.

(3) The IETF codec is scheduled to be finalized in 14 months and submitted to IESG in 18 months.  Even if we take the BT SIG route and take 6 to 18 months there.  The total time of 2 to 3 years from now means the Moore's Law would only increase the CPU resources 2X to 3X, and definitely no more than 4X max, not 10X.

(4) Most importantly, guess what, in the last several years the Bluetooth headset chips have been growing its processing power at a MUCH, MUCH slower rate than what the Moore's Law says it should. Sometimes they did not increase the speed at all for years.  The reasons? The ASP (average sale price) of Bluetooth chips plummeted very badly, making it unattractive to invest significant resources to make them significantly faster.  Also, for low-end and mid-end BT headsets, the BT chips were often considered "good enough" and there wasn't a strong drive to increase the computing resources.  In addition, the BT headsets got smaller over the last few years; the corresponding reduction in battery size required a reduction in power consumption, which also limited how fast the processor speed could grow.  In the next several years, it is highly likely that the computing capabilities of Bluetooth headset chips will continue to grow at a rate substantially below what's predicted by the Moore's Law.

(5) Although Bluetooth supports G.711 as an optional codec, basically no one uses it because it is too sensitive to bit errors.  Essentially all the BT mono headsets on the market today are narrowband (8 kHz sampling) headsets using CVSD.  There isn't any real wideband support yet, so your comment about G.722 doesn't apply.  Even after wideband-capable BT headsets come out, for many years to come the majority of the BT headsets (especially mid- to low-end) will still be narrowband only, running only CVSD. Hence, the quality degradation of the CVSD transcoding is real and will be with us for quite a while, so it is desirable for the IETF codec to have a low-complexity mode that can directly run on the BT headsets to avoid the quality degradation of CVSD when using BT headsets to make Internet phone calls.

(6) Even if you could use G.711 or G.722 in the BT headsets, they both operate at 64 kb/s.  A low-complexity mode of the IETF codec can operate at half or one quarter of that bit-rate.  This will help conserve BT headsets' radio power because of the lower transmit duty cycle.  It will also help the Bluetooth + WiFi co-existence technologies.

(7) Already a lot of people are used to using Bluetooth headsets to make phone calls today.  If they have a choice, many of these people will also want to use Bluetooth headsets to make Internet phone calls, not only through computers, but also through smart phones connected to WiFi or cellular networks.  As more and more states and countries pass laws to ban the use of cell phones that are not in hands-free mode while driving, the number of Bluetooth headset users will only increase with time, and many of them will want to make Internet-based phone calls.

Given all the above, I would argue that Bluetooth headset is a very relevant application that the IETF codec should address with a low-complexity mode.

Best Regards,

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Koen Vos
Sent: Friday, April 23, 2010 1:16 AM
To: [email-masked]
Subject: Re: [codec] #16: Multicast?

best,
koen.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 25 Apr 2010 10:06:03 +0200 (CEST)

On Sat, 24 Apr 2010, Raymond (Juin-Hwey) Chen wrote:

> (7) Already a lot of people are used to using Bluetooth headsets to make 
> phone calls today.  If they have a choice, many of these people will 
> also want to use Bluetooth headsets to make Internet phone calls, not 
> only through computers, but also through smart phones connected to WiFi 
> or cellular networks.  As more and more states and countries pass laws 
> to ban the use of cell phones that are not in hands-free mode while 
> driving, the number of Bluetooth headset users will only increase with 
> time, and many of them will want to make Internet-based phone calls.

I purchased a BT headset with the anticipation of using it with my 
computer to make Skype calls. I tried it, but the sound quality when doing 
bidirectional audio (whatever that mode is called) is not good enough, it 
worsens the "skype IP" call quality. I agree that the use case is 
interesting, but as long as BT sound quality is what it is, it's really 
only the "low end" type  sound quality we're talking about.

But yes, I make skype IP calls with my Nokia N900 using BT sometimes, so 
the use case example is definitely valid.

---

From: Koen Vos <[email-masked]>
Date: Sun, 25 Apr 2010 04:00:49 -0700

Hi Raymond,

You seem to suggest that the IETF Internet codec should fit Bluetooth  
requirements in order to enable transcoding-free operation all the way  
from the Internet, through the Internet-connected device, to the BT  
wireless audio device.

A similar argument would hold for ITU-T cellular codecs: AMR-WB and  
G.718 could have been designed with BT as an application.  In reality,  
these codecs have very little in common with BT codecs, because of the  
vastly different requirements in terms of
- complexity
- memory footprint
- bitrate
- scalability
- bit error robustness
- packet loss robustness.

Do you think it's realistic for us to come up with a design that  
fulfills the needs of both worlds?

The alternative is to separately design codecs for Internet  
applications and BT devices, and continue the practice of transcoding  
on the Internet-connected device.  That would have a better chance of  
maximizing quality in all scenarios.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen":

> (1) Although BT SIG standardization is a preferred route, it is  
> technically feasible to negotiate and use a non-Bluetooth-SIG codec.
>
> (2) Someone familiar with BT SIG told me that it would probably take  
> only 6 months to add an optional codec to the BT SIG spec and 12 to  
> 18 months to add a mandatory codec.
>
> (3) The IETF codec is scheduled to be finalized in 14 months and  
> submitted to IESG in 18 months.  Even if we take the BT SIG route  
> and take 6 to 18 months there.  The total time of 2 to 3 years from  
> now means the Moore's Law would only increase the CPU resources 2X  
> to 3X, and definitely no more than 4X max, not 10X.
>
> (4) Most importantly, guess what, in the last several years the  
> Bluetooth headset chips have been growing its processing power at a  
> MUCH, MUCH slower rate than what the Moore's Law says it should.  
> Sometimes they did not increase the speed at all for years.  The  
> reasons? The ASP (average sale price) of Bluetooth chips plummeted  
> very badly, making it unattractive to invest significant resources  
> to make them significantly faster.  Also, for low-end and mid-end BT  
> headsets, the BT chips were often considered "good enough" and there  
> wasn't a strong drive to increase the computing resources.  In  
> addition, the BT headsets got smaller over the last few years; the  
> corresponding reduction in battery size required a reduction in  
> power consumption, which also limited how fast the processor speed  
> could grow.  In the next several years, it is highly likely that the  
> computing capabilities of Bluetooth headset chips will continue to  
> grow at a rate substantially below what's predicted by the Moore's  
> Law.
>
> (5) Although Bluetooth supports G.711 as an optional codec,  
> basically no one uses it because it is too sensitive to bit errors.   
> Essentially all the BT mono headsets on the market today are  
> narrowband (8 kHz sampling) headsets using CVSD.  There isn't any  
> real wideband support yet, so your comment about G.722 doesn't  
> apply.  Even after wideband-capable BT headsets come out, for many  
> years to come the majority of the BT headsets (especially mid- to  
> low-end) will still be narrowband only, running only CVSD. Hence,  
> the quality degradation of the CVSD transcoding is real and will be  
> with us for quite a while, so it is desirable for the IETF codec to  
> have a low-complexity mode that can directly run on the BT headsets  
> to avoid the quality degradation of CVSD when using BT headsets to  
> make Internet phone calls.
>
> (6) Even if you could use G.711 or G.722 in the BT headsets, they  
> both operate at 64 kb/s.  A low-complexity mode of the IETF codec  
> can operate at half or one quarter of that bit-rate.  This will help  
> conserve BT headsets' radio power because of the lower transmit duty  
> cycle.  It will also help the Bluetooth + WiFi co-existence  
> technologies.
>
> (7) Already a lot of people are used to using Bluetooth headsets to  
> make phone calls today.  If they have a choice, many of these people  
> will also want to use Bluetooth headsets to make Internet phone  
> calls, not only through computers, but also through smart phones  
> connected to WiFi or cellular networks.  As more and more states and  
> countries pass laws to ban the use of cell phones that are not in  
> hands-free mode while driving, the number of Bluetooth headset users  
> will only increase with time, and many of them will want to make  
> Internet-based phone calls.
>
> Given all the above, I would argue that Bluetooth headset is a very  
> relevant application that the IETF codec should address with a  
> low-complexity mode.
>
> Best Regards,
>
> Raymond
>

---

From: Koen Vos <[email-masked]>
Date: Sun, 25 Apr 2010 12:24:29 -0700

Hi Raymond,

Jitter buffers have no problem implementing a non-integer-frame delay,  
because packets are queued and read non-synchronously.

Processing time matters on low-end hardware - a small fraction of  
today's VoIP end points.  Even then, the higher coding efficiency of  
longer frames can be translated into lower complexity.

And transmission delay increases (perhaps) linearly with the *packet  
size*, not with the *frame size*.  For a 32 kbps codec with 5 ms  
frames, packets are just 30% smaller than with a 16 kbps codecs with  
20 ms frames.

Let me ask you something: how often is G.729 used with 10 ms packets,  
or Broadvoice with 5 ms packets?

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen":

> [Raymond]: I don't agree that it will be only 20 - 5 = 15 ms lower.   
> That will be true only if your one-way delay formula below is true,  
> but theoretically it cannot be.  See my comment below your formula.
>
>
>
> [Raymond]: That may be true theoretically, but in practical  
> implementations, selecting a jitter buffer delay that is not  
> divisible by the packet size would make the adaptive jitter buffer  
> pretty messy to implement.  If we make the it divisible by the  
> packet size, then a smaller packet size gives you more granularity  
> to work with and can result in lower average delay as the codec  
> frames go through the adaptive jitter buffer.
>
>
>
> [Raymond]: First, I agree that codec algorithmic buffering delay is  
> more accurate than frame size since it can also include the  
> "look-ahead" delay and filtering delay if sub-band  
> analysis/synthesis is used.  However, your formula implies that for  
> the codec-related delay, the "multiplier" to be used for the codec  
> frame size is only 1.  That's unrealistic and theoretically  
> impossible.  For that to happen, after you wait one frame of time  
> for the current frame of input audio samples to arrive at your input  
> signal buffer (that's one frame of codec-related delay already), you  
> need an infinitely fast processor to finish the encoding operation  
> instantly, then you need an infinitely fast communication link to  
> ship all the bits in the compressed frame to the decoder instantly,  
> and then you need an infinitely fast processor to finish decoding  
> the frame instantly and start playing back the current frame of  
> audio without any delay.  That's just impossible.
>
> In reality, if the processor is just barely fast enough to implement  
> the codec in real time, then you need nearly a full frame of time to  
> finish the encoding and decoding operations. That makes the  
> multiplier to be 2 already.  If your communication link is just  
> barely fast enough to transmit your packets at the same speed they  
> are generated without piling up unsent packets, then it takes  
> another frame of time to finish transmitting the compressed bits in  
> a frame to the decoder.  That makes the multiplier to be 3 already.
>
> Granted, in practice the processor and the communication link are  
> usually faster than just barely enough, so the processing delay and  
> the transmission delay can be less than 1 frame each.  However,  
> there are other miscellaneous uncounted delays that tends to depend  
> on the codec size in various ways.  Thus, a typical IP phone  
> implementation would have
>
>   One-way delay = codec-independent delay + 3*(codec frame size) +  
> (codec look-ahead) + (codec filtering delay if any).
>
> Hence, the one-way delay difference between a 20 ms and a 5 ms codec  
> frame size would be 45 ms + (codec look-ahead difference) + (codec  
> filtering delay difference).
>
> Consequently, for the conference bridge application, the total  
> difference in one-way delay can easily be in the 90 to 100 ms range.  
> When adding this delay difference to all the other codec-independent  
> delay components, it is still a huge difference that the users can  
> easily notice, especially since it will most likely push the total  
> one-way delay significantly beyond the 150 ms limit.
>
>
>
> [Raymond]: The cable operators' networks are still IP networks.  If  
> the technology can work there, I don't see why it cannot work  
> elsewhere in the Internet.  I know it is currently not available  
> between arbitrary Internet end points.  I am just saying that  
> technologies exist that can potentially be deployed in the Internet  
> to compress the packet headers to a very small fraction of the  
> uncompressed headers.
>
>
>
> best,
>
> koen.
>
>
>

---

From: stephen botzko <[email-masked]>
Date: Sun, 25 Apr 2010 16:38:12 -0400

And transmission delay increases (perhaps) linearly with the *packet size*,
not with the *frame size*.  For a 32 kbps codec with 5 ms frames, packets
are just 30% smaller than with a 16 kbps codecs with 20 ms frames.
 "Packet size" here has to include layer 2 overhead, not just IP overhead,
making your argument even stronger. In the case of Ethernet, Layer-2
overhead is 38-42 bytes per packet (depending on whether a vlan tag is
present), so it is about the same as the IP/UDP/RTP overhead.  And of course
there's encryption pads, VPN encapsulation, etc. that apply in many cases.

There is a floor transmission delay when you send the minimum size packets
the network path allows.

There is an incremental delay due to serialization when you send larger size
packets than the minimum. (At each hop you wait until you receive last bit
in the packet before you forward the first bit).   I'd agree that a
reasonable model for the incremental delay is that it scales linearly with
the *increase* in packet size.  But the floor delay is usually too large for
this to matter dominate.

And on top of that is the variable delay  (jitter) due to congestion, layer
2 retransmission, and the like.  That also will not scale linearly with
frame size or packet size.

So arguments that increasing the frame size by 10 ms will increase the
overall delay by 50 ms make no sense to me at all.

Stephen Botzko

On Sun, Apr 25, 2010 at 3:24 PM, Koen Vos <[email-masked]> wrote:

> Processing time matters on low-end hardware - a small fraction of today's
> VoIP end points.  Even then, the higher coding efficiency of longer frames
> can be translated into lower complexity.
>
> And transmission delay increases (perhaps) linearly with the *packet size*,
> not with the *frame size*.  For a 32 kbps codec with 5 ms frames, packets
> are just 30% smaller than with a 16 kbps codecs with 20 ms frames.
>
> Let me ask you something: how often is G.729 used with 10 ms packets, or
> Broadvoice with 5 ms packets?
>
> best,
> koen.
>
>
>
>
>> [Raymond]: I don't agree that it will be only 20 - 5 = 15 ms lower.  That
>> will be true only if your one-way delay formula below is true, but
>> theoretically it cannot be.  See my comment below your formula.
>>
>>
>>
>> implementations, selecting a jitter buffer delay that is not divisible by
>> the packet size would make the adaptive jitter buffer pretty messy to
>> implement.  If we make the it divisible by the packet size, then a smaller
>> packet size gives you more granularity to work with and can result in lower
>> average delay as the codec frames go through the adaptive jitter buffer.
>>
>>
>>
>> [Raymond]: First, I agree that codec algorithmic buffering delay is more
>> accurate than frame size since it can also include the "look-ahead" delay
>> and filtering delay if sub-band analysis/synthesis is used.  However, your
>> formula implies that for the codec-related delay, the "multiplier" to be
>> used for the codec frame size is only 1.  That's unrealistic and
>> theoretically impossible.  For that to happen, after you wait one frame of
>> time for the current frame of input audio samples to arrive at your input
>> signal buffer (that's one frame of codec-related delay already), you need an
>> infinitely fast processor to finish the encoding operation instantly, then
>> you need an infinitely fast communication link to ship all the bits in the
>> compressed frame to the decoder instantly, and then you need an infinitely
>> fast processor to finish decoding the frame instantly and start playing back
>> the current frame of audio without any delay.  That's just impossible.
>>
>> In reality, if the processor is just barely fast enough to implement the
>> codec in real time, then you need nearly a full frame of time to finish the
>> encoding and decoding operations. That makes the multiplier to be 2 already.
>>  If your communication link is just barely fast enough to transmit your
>> packets at the same speed they are generated without piling up unsent
>> packets, then it takes another frame of time to finish transmitting the
>> compressed bits in a frame to the decoder.  That makes the multiplier to be
>> 3 already.
>>
>> Granted, in practice the processor and the communication link are usually
>> faster than just barely enough, so the processing delay and the transmission
>> delay can be less than 1 frame each.  However, there are other miscellaneous
>> uncounted delays that tends to depend on the codec size in various ways.
>>  Thus, a typical IP phone implementation would have
>>
>>  One-way delay = codec-independent delay + 3*(codec frame size) + (codec
>> look-ahead) + (codec filtering delay if any).
>>
>> Hence, the one-way delay difference between a 20 ms and a 5 ms codec frame
>> size would be 45 ms + (codec look-ahead difference) + (codec filtering delay
>> difference).
>>
>> Consequently, for the conference bridge application, the total difference
>> in one-way delay can easily be in the 90 to 100 ms range. When adding this
>> delay difference to all the other codec-independent delay components, it is
>> still a huge difference that the users can easily notice, especially since
>> it will most likely push the total one-way delay significantly beyond the
>> 150 ms limit.
>>
>>
>>
>> [Raymond]: The cable operators' networks are still IP networks.  If the
>> technology can work there, I don't see why it cannot work elsewhere in the
>> Internet.  I know it is currently not available between arbitrary Internet
>> end points.  I am just saying that technologies exist that can potentially
>> be deployed in the Internet to compress the packet headers to a very small
>> fraction of the uncompressed headers.
>>
>>
>>
>> best,
>>
>> koen.
>>
>>
>>
>>
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Sun, 25 Apr 2010 23:49:27 -0700

In-line...

-----Original Message-----
From: Koen Vos [mailto:[email-masked]]
Sent: Sunday, April 25, 2010 12:24 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast?

Hi Raymond,

[Raymond]: I am talking about adaptive jitter buffer that tries to minimize the delay through the jitter buffer dynamically depending on the observed network jitter.  If the jitter is small, you decrease that delay, and if it is large, you increase that delay. An engineer who actually implemented such an adaptive jitter buffer in an IP phone told me that the non-integer-frame delay made it pretty messy to implement (I didn't say it was not possible; it's just messy), so for implementation simplicity's sake, the jitter delay was often chosen to be an integer number of frames. He also said that a smaller frame size gives you more frequent observations of the network jitter and thus makes the jitter estimate more responsive and accurate.

today's VoIP end points.

[Raymond]: Processing time certainly matters for IP phones, and there are a lot of enterprise IP phones deployed today. I heard that it is actually significantly cheaper for enterprises to have their entire phone systems IP-phone-based than analog-phone-based. I won't be surprised that before too long the vast majority of enterprises will use only IP phones.  Even consumer phones and cell phones are moving toward IP-based.  Eventually that would be a very large percentage of VoIP end points.

20 ms frames.

[Raymond]: Agreed. My previous comments on transmission delay was based on the TDM rather than packet scenario, but I was just using that simplified TDM example to make a point that transmission delay cannot be zero, as your 1X frame size multiplier would imply.  Even with your statement above, a larger codec frame size still makes a larger packet size, which then increases the transmission delay, so you can't say transmission delay is zero or is independent of the codec size.

In any case, these are really minor details.  My key point is that your 1X multiplier for the codec frame size is simply theoretically impossible.  The rule of thumb used by IP phone engineers is around 3X codec frame size.

[Raymond]: Not very often, but that's because previously network routers/switches didn't like to handle too many packets per second, and the higher packet header overhead associated with a smaller packet size means the overall bit-rate would be higher than desired or allowed, so the time of small packet size for low-delay VoIP hasn't really come yet.  However, with the help of Moore's Law, network routers/switches are becoming much faster now, and I was told that they can handle a 5 ms packet size without problems; furthermore, the speed of backbone networks and access networks keep increasing with time, so the bit-rate concern will also decrease with time.

Unlike processing speed and communication speed that continuously get improved with time for decades, delay is one thing that will NOT get improved with time and Moore's Law cannot do anything about that!

If the IETF codec has a minimum frame size of 20 ms, we will be stuck with the longer overall delay associated with that, and Moore's Law will not help us reduce that delay in the future.  On the other hand, in addition to using a 20 ms frame size for bit-rate-sensitive applications, if the IETF codec also has a low-delay mode that uses a 5 ms frame size, then at least for delay-sensitive applications, people have a choice to achieve a lower delay by paying the price of a higher overall bit-rate (i.e. with packet header counted), and this higher bit-rate will be less and less of a concern as the network speed keep increasing with time.

Therefore, recognizing that delay cannot be helped by Moore's Law but bit-rate can, it would be wise for the IETF codec WG to adopt a low-delay mode for the codec in order to be future-proof.

best,

koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 26 Apr 2010 00:45:35 -0700

Hi Stephen,

Thanks for sharing your broad knowledge in this.  You made some good points, and I basically agree with them.  I just have some comments in-line.

Raymond
From: stephen botzko [mailto:[email-masked]]
Sent: Sunday, April 25, 2010 1:38 PM
To: Koen Vos
Cc: Raymond (Juin-Hwey) Chen; [email-masked]
Subject: Re: [codec] #16: Multicast?

And transmission delay increases (perhaps) linearly with the *packet size*, not with the *frame size*.  For a 32 kbps codec with 5 ms frames, packets are just 30% smaller than with a 16 kbps codecs with 20 ms frames.
 "Packet size" here has to include layer 2 overhead, not just IP overhead, making your argument even stronger. In the case of Ethernet, Layer-2 overhead is 38-42 bytes per packet (depending on whether a vlan tag is present), so it is about the same as the IP/UDP/RTP overhead.  And of course there's encryption pads, VPN encapsulation, etc. that apply in many cases.

There is a floor transmission delay when you send the minimum size packets the network path allows.
[Raymond]: Agreed.

There is an incremental delay due to serialization when you send larger size packets than the minimum. (At each hop you wait until you receive last bit in the packet before you forward the first bit).   I'd agree that a reasonable model for the incremental delay is that it scales linearly with the increase in packet size.  But the floor delay is usually too large for this to matter dominate.
[Raymond]: As I mentioned in my last response to Koen, I was just using a simplified TDM example to make a point that the transmission delay cannot be zero as the 1X frame size multiplier implies.  I agree that for packet systems the delay increase will be smaller than in TDM, but it will not be zero.  This transmission delay component is indeed small compared with what you called the floor delay.  However, using the IP phone engineers' rule of thumb of 3X codec frame size, the total codec-dependent component of the delay can be quite significant when compared with the floor delay, especially for a 20 ms frame size.

And on top of that is the variable delay  (jitter) due to congestion, layer 2 retransmission, and the like.  That also will not scale linearly with frame size or packet size.

So arguments that increasing the frame size by 10 ms will increase the overall delay by 50 ms make no sense to me at all.
[Raymond]: I have abandoned that 5X frame size formula many emails ago.  I was trying to make it simple, but later I realized that this over-simplified approach is not good, so several emails ago I already replaced it with one-way delay = codec-independent delay + 3*(codec frame size) + (codec look-ahead) + (codec filtering delay if any).  The main debate now is centered on whether the multiplier of the codec frame size should be 1 as Koen said or 3 as I was told by experienced IP phone engineers.  I argue that 1X is theoretically impossible.  It is interesting to note that the ITU-T uses a multiplier of 2X.  I think 2X is probably achievable for the idealized situation.  In practice, however, many nitty-gritty details get in the way of getting that idealized situation, and little additional delays just keep getting added, resulting in a real-world realistic 3X multiplier.  With a 3X multiplier, the one-way delay difference between a 20 ms and a 5 ms codec frame size would be 45 ms + (codec look-ahead difference) + (codec filtering delay difference).  For the conference bridge application, the total difference in one-way delay will double to the 90 to 100 ms range.  That's a VERY significant difference that typical users will notice (it's like adding another cell phone call delay), especially if it pushes the total one-way delay significantly beyond the 150 ms guideline.   Therefore, I argue that for the best user experience in conference bridge calls, the IETF codec should have a low-delay mode with a small codec frame size such as 5 ms, and let the continually increasing speed of communication links make the header overhead bit-rate become less and less of an issue in the future.  (Even now, for those people who have high speed connection to their computers, it is already not an issue.  It is better for them to get low delay than to worry about bit-rate or packet header overhead.)

Stephen Botzko
On Sun, Apr 25, 2010 at 3:24 PM, Koen Vos <[email-masked]<mailto:[email-masked]>> wrote:
Hi Raymond,

Jitter buffers have no problem implementing a non-integer-frame delay, because packets are queued and read non-synchronously.

Processing time matters on low-end hardware - a small fraction of today's VoIP end points.  Even then, the higher coding efficiency of longer frames can be translated into lower complexity.

And transmission delay increases (perhaps) linearly with the *packet size*, not with the *frame size*.  For a 32 kbps codec with 5 ms frames, packets are just 30% smaller than with a 16 kbps codecs with 20 ms frames.

Let me ask you something: how often is G.729 used with 10 ms packets, or Broadvoice with 5 ms packets?

best,
koen.

Best Regards,

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Saturday, April 24, 2010 6:16 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [codec] #16: Multicast?

size.

best,

koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 26 Apr 2010 14:02:18 -0700

Hi Mikael,

Thanks for sharing your experience using a BT headset for Skype calls.  I think part of the quality degradation that you experienced was due to the reduction of the audio bandwidth to narrowband (8 kHz sampling, 3.4 kHz bandwidth), and another part of the degradation was due to the audible coding noise of the CVSD codec, a 40-year-old coding technology first proposed in 1970.  

If a high-quality, low-complexity, wider bandwidth IETF codec mode can be implemented in Skype and the Bluetooth headset to avoid the CVSD transcoding (together with wideband upgrade of the transducers and audio path in the BT headset, of course), then not only will you get much better speech quality in your Skype calls than what you have experienced, but also you will get a lower latency.  This is because transcoding between the Skype codec and CVSD not only accumulates the coding distortion of the two codecs, but also accumulates the coding delays.  Although CVSD is a sample-by-sample codec, BT headsets still transmit the CVSD bit-stream in 3.75 ms or 7.5 ms packets, and they can potentially add a one-way delay up to 20 ~ 25 ms through the Bluetooth headset (the exact delay depends on the implementation).

While we were discussing whether a 5 ms packet size can even be considered, for many years Bluetooth headsets have been using an even smaller 3.75 ms packet size.

Best Regards,

Raymond

---

From: Koen Vos <[email-masked]>
Date: Tue, 27 Apr 2010 00:16:01 -0700

Hi Raymond,

Please don't get me wrong: I share your vision that, over time,  
Moore's law will drive adoption of the highest possible audio quality  
in IP end points.  And the creation of the very low-delay, full-band,  
(near-)transparent and even multi-channel codec this requires falls  
indeed within the objectives of this WG, if you ask me.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen":
> [Raymond]: I am talking about adaptive jitter buffer that tries to  
> minimize the delay through the jitter buffer dynamically depending  
> on the observed network jitter.  If the jitter is small, you  
> decrease that delay, and if it is large, you increase that delay. An  
> engineer who actually implemented such an adaptive jitter buffer in  
> an IP phone told me that the non-integer-frame delay made it pretty  
> messy to implement (I didn't say it was not possible; it's just  
> messy), so for implementation simplicity's sake, the jitter delay  
> was often chosen to be an integer number of frames. He also said  
> that a smaller frame size gives you more frequent observations of  
> the network jitter and thus makes the jitter estimate more  
> responsive and accurate.
>
>
>
> [Raymond]: Processing time certainly matters for IP phones, and  
> there are a lot of enterprise IP phones deployed today. I heard that  
> it is actually significantly cheaper for enterprises to have their  
> entire phone systems IP-phone-based than analog-phone-based. I won't  
> be surprised that before too long the vast majority of enterprises  
> will use only IP phones.  Even consumer phones and cell phones are  
> moving toward IP-based.  Eventually that would be a very large  
> percentage of VoIP end points.
>
>
>
> [Raymond]: Agreed. My previous comments on transmission delay was  
> based on the TDM rather than packet scenario, but I was just using  
> that simplified TDM example to make a point that transmission delay  
> cannot be zero, as your 1X frame size multiplier would imply.  Even  
> with your statement above, a larger codec frame size still makes a  
> larger packet size, which then increases the transmission delay, so  
> you can't say transmission delay is zero or is independent of the  
> codec size.
>
> In any case, these are really minor details.  My key point is that  
> your 1X multiplier for the codec frame size is simply theoretically  
> impossible.  The rule of thumb used by IP phone engineers is around  
> 3X codec frame size.
>
>
>
> [Raymond]: Not very often, but that's because previously network  
> routers/switches didn't like to handle too many packets per second,  
> and the higher packet header overhead associated with a smaller  
> packet size means the overall bit-rate would be higher than desired  
> or allowed, so the time of small packet size for low-delay VoIP  
> hasn't really come yet.  However, with the help of Moore's Law,  
> network routers/switches are becoming much faster now, and I was  
> told that they can handle a 5 ms packet size without problems;  
> furthermore, the speed of backbone networks and access networks keep  
> increasing with time, so the bit-rate concern will also decrease  
> with time.
>
> Unlike processing speed and communication speed that continuously  
> get improved with time for decades, delay is one thing that will NOT  
> get improved with time and Moore's Law cannot do anything about that!
>
> If the IETF codec has a minimum frame size of 20 ms, we will be  
> stuck with the longer overall delay associated with that, and  
> Moore's Law will not help us reduce that delay in the future.  On  
> the other hand, in addition to using a 20 ms frame size for  
> bit-rate-sensitive applications, if the IETF codec also has a  
> low-delay mode that uses a 5 ms frame size, then at least for  
> delay-sensitive applications, people have a choice to achieve a  
> lower delay by paying the price of a higher overall bit-rate (i.e.  
> with packet header counted), and this higher bit-rate will be less  
> and less of a concern as the network speed keep increasing with time.
>
> Therefore, recognizing that delay cannot be helped by Moore's Law  
> but bit-rate can, it would be wise for the IETF codec WG to adopt a  
> low-delay mode for the codec in order to be future-proof.
>
>
>
>
>
> best,
>
> koen.
>
>
>
>
>
>
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 29 Apr 2010 15:14:06 -0700

Hi Koen,

For some reason the SPAM filter accidentally routed your email below (sent last Sunday) to my junk email folder and I just saw it. Sorry about the delay of my response.

I agree that there are some fundamental differences in the requirements for cellular codecs and Bluetooth codecs which caused the codecs in these two types of devices to each go their own way.  However, these differences are (or can be) substantially smaller between an Internet codec and Bluetooth codecs, so I think it is easier for Internet devices and Bluetooth devices to use the same codec to avoid the additional delay and coding distortion of transcoding.

(1) Royalty-free requirement:
Cellular codecs are usually royalty-bearing, and that's acceptable in the cellular world.  Not so with Bluetooth.  Bluetooth devices are meant to be simple and low cost.  As such, Bluetooth SIG basically only wants to standardize royalty-free technologies.  That's an important reason why they picked the CVSD codec, a royalty-free old technology of 1970.  We are trying to make the IETF codec royalty-free, so in this regard this goal is consistent with the Bluetooth SIG's royalty-free requirement for codec.

(2) Bit-rate requirement:
Cellular radio spectrum is a limited, fixed resource that doesn't change with time, and cellular operators spent billions of dollars in radio spectrum auctions. Thus, it is extremely important for cellular codecs to have bit-rates as low as possible, with an average bit-rate often going below 1 bit/sample, to maximize the number of cellular subscribers a given amount of radio spectrum can support.  In contrast, the bit-rate is not nearly as big a concern for Bluetooth. Initially Bluetooth SIG picked the relatively high-bit-rate 64 kb/s CVSD narrowband codec (8 bits/sample) for its simplicity and royalty-free nature among other things.  Since the speeds of the Internet back bone and access networks keep growing with time, the bit-rate of an Internet codec is also not nearly as big a concern as in cellular codecs, and an Internet codec around 2 bits/sample can have better trade-offs (e.g. higher quality, lower delay, and lower complexity) for Internet applications than what cellular codecs can provide.  Incidentally, Bluetooth SIG is moving toward 4 bits/sample.  As you can see, in terms of the bit-rate requirement, an Internet codec is much closer to Bluetooth codecs than cellular codecs are.

(3) Complexity requirement:
Bluetooth headsets have much lower processing power and much smaller batteries than cell phones. The complexity of cellular codecs, typically in the range of 20 to 40 MHz on a DSP, is too high to fit most Bluetooth headsets. However, unlike cell phones and Bluetooth headsets where each is a specific type of device with a relatively narrow range of device complexity, Internet voice/audio applications can potentially encompass a large variety of different device types, from desktop computers at the high end with > 3 GHz multi-core CPU to IP phones and possibly even Bluetooth headsets at the low end with a processor of only a few tens of MHz.  It is up to the IETF codec WG how we want the complexity of the IETF codec to be.  We can standardize just one codec mode that works well for computer-to-computer calls but can't fit in low-end devices, or we can keep that mode but also have a low-complexity mode that can be implemented in low-end devices.  Frankly, I think the second approach makes much more sense since it allows many more devices to benefit from the IETF codec and enables the large number of Bluetooth headset users to avoid the additional distortion and delay associated with transcoding when making Internet calls.

(4) Delay requirement: Due to the need for cellular codecs to achieve bit-rates as low as possible, they sacrificed the coding delay and used a 20 ms frame size, because using a 10 or 5 ms frame size would increase the bit-rate for a given level of speech quality.  On the other hand, a Bluetooth headset needs to have a low delay since its delay is added to the already long cell phone delay.  For the IETF codec, again it is up to the codec WG to decide what kind of codec delay we want, and again I think it makes sense to have a higher-delay, higher bit-rate efficiency mode for bit-rate-sensitive applications and another low-delay mode for delay-sensitive applications, since one size doesn't fit all.  If the IETF codec delay is forced to be one size, the resulting codec will be (potentially very) suboptimal for some applications.

> fulfills the needs of both worlds?

With a one-size-fit-all approach, probably not, but with a multi-mode approach, then I think so.

Best Regards,

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]]
Sent: Sunday, April 25, 2010 4:01 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast? (Bluetooth)

Hi Raymond,

best,
koen.

---

From: stephen botzko <[email-masked]>
Date: Fri, 30 Apr 2010 07:34:48 -0400

Though the Bluetooth angle is interesting, it is clearly out-of-scope for
this WG.

Of course Bluetooth SIG could pick up CODEC later on if they think it meets
their requirements.

Stephen Botzko

On Thu, Apr 29, 2010 at 6:14 PM, Raymond (Juin-Hwey) Chen <
[email-masked]> wrote:

> Hi Koen,
>
> For some reason the SPAM filter accidentally routed your email below (sent
> last Sunday) to my junk email folder and I just saw it. Sorry about the
> delay of my response.
>
> I agree that there are some fundamental differences in the requirements for
> cellular codecs and Bluetooth codecs which caused the codecs in these two
> types of devices to each go their own way.  However, these differences are
> (or can be) substantially smaller between an Internet codec and Bluetooth
> codecs, so I think it is easier for Internet devices and Bluetooth devices
> to use the same codec to avoid the additional delay and coding distortion of
> transcoding.
>
> Cellular codecs are usually royalty-bearing, and that's acceptable in the
> cellular world.  Not so with Bluetooth.  Bluetooth devices are meant to be
> simple and low cost.  As such, Bluetooth SIG basically only wants to
> standardize royalty-free technologies.  That's an important reason why they
> picked the CVSD codec, a royalty-free old technology of 1970.  We are trying
> to make the IETF codec royalty-free, so in this regard this goal is
> consistent with the Bluetooth SIG's royalty-free requirement for codec.
>
> Cellular radio spectrum is a limited, fixed resource that doesn't change
> with time, and cellular operators spent billions of dollars in radio
> spectrum auctions. Thus, it is extremely important for cellular codecs to
> have bit-rates as low as possible, with an average bit-rate often going
> below 1 bit/sample, to maximize the number of cellular subscribers a given
> amount of radio spectrum can support.  In contrast, the bit-rate is not
> nearly as big a concern for Bluetooth. Initially Bluetooth SIG picked the
> relatively high-bit-rate 64 kb/s CVSD narrowband codec (8 bits/sample) for
> its simplicity and royalty-free nature among other things.  Since the speeds
> of the Internet back bone and access networks keep growing with time, the
> bit-rate of an Internet codec is also not nearly as big a concern as in
> cellular codecs, and an Internet codec around 2 bits/sample can have better
> trade-offs (e.g. higher quality, lower delay, and lower complexity) for
> Internet applications than what cellular codecs can provide.  Incidentally,
> Bluetooth SIG is moving toward 4 bits/sample.  As you can see, in terms of
> the bit-rate requirement, an Internet codec is much closer to Bluetooth
> codecs than cellular codecs are.
>
> Bluetooth headsets have much lower processing power and much smaller
> batteries than cell phones. The complexity of cellular codecs, typically in
> the range of 20 to 40 MHz on a DSP, is too high to fit most Bluetooth
> headsets. However, unlike cell phones and Bluetooth headsets where each is a
> specific type of device with a relatively narrow range of device complexity,
> Internet voice/audio applications can potentially encompass a large variety
> of different device types, from desktop computers at the high end with > 3
> GHz multi-core CPU to IP phones and possibly even Bluetooth headsets at the
> low end with a processor of only a few tens of MHz.  It is up to the IETF
> codec WG how we want the complexity of the IETF codec to be.  We can
> standardize just one codec mode that works well for computer-to-computer
> calls but can't fit in low-end devices, or we can keep that mode but also
> have a low-complexity mode that can be implemented in low-end devices.
>  Frankly, I think the second approach makes much more sense since it allows
> many more devices to benefit from the IETF codec and enables the large
> number of Bluetooth headset users to avoid the additional distortion and
> delay associated with transcoding when making Internet calls.
>
> (4) Delay requirement: Due to the need for cellular codecs to achieve
> bit-rates as low as possible, they sacrificed the coding delay and used a 20
> ms frame size, because using a 10 or 5 ms frame size would increase the
> bit-rate for a given level of speech quality.  On the other hand, a
> Bluetooth headset needs to have a low delay since its delay is added to the
> already long cell phone delay.  For the IETF codec, again it is up to the
> codec WG to decide what kind of codec delay we want, and again I think it
> makes sense to have a higher-delay, higher bit-rate efficiency mode for
> bit-rate-sensitive applications and another low-delay mode for
> delay-sensitive applications, since one size doesn't fit all.  If the IETF
> codec delay is forced to be one size, the resulting codec will be
> (potentially very) suboptimal for some applications.
>
> You wrote:

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 30 Apr 2010 14:23:07 +0200

Hi,

I just want to share some insights from the recent development of Bluetooth's Hands-Free Profile (HFP) version 1.5, which supports
wideband speech.  One main requirement were on the frame size of 7.5 ms because the Bluetooth MAC protocol support scheduling at
this interval. Actually, to achieve this they modified SBC to work on 15 blocks instead of 4,8,12 or 16 blocks and they decided
against G.722.

The lesson to learn is about the importance of MAC protocol. To get a efficient, low power, and mobile device you have to consider
the impact of packet scheduling. If packets are scheduled regular, the MAC protocol can work more efficient. The more irregular
packet arrive, the more expensive a packet transmission gets.

Actually, you can translate the cost of packet scheduling to bits per packet. Depending on the wireless technology, it might vary
substantially. The worst case is 802.11b at 11 Mbps at long preamble - then, you can add about 1300 bytes to every packet just for
physical headers and MAC scheduling. However, more modern technologies like LTE and IEEE 802.11n are much more efficient in terms of
per packet overhead. 

If you ask me, one important usage scenario is over wireless links supporting low-power mobile devices. If we ignore this scenario
it will be a judge mistake:
a) Battery powered mobile devices must be energy efficient to reduce the size of the batteries. Also, they should not demand to many
computational resources, otherwise they would consume to much energy.
b) Wireless IP access is also in scope because many devices get Internet access LTE, WLAN, Wimax, UMTS, etc. 

Bluetooth headsets are somewhat a special case. Actually, they are two cases:
1) Headphones (A2DP): For me it is not clear whether supporting the Internet CODEC on top of A2DP (which is - by the way - already
possible according to Bluetooth spec A2DP V1.2) or using the Internet CODEC till the Bluetooth AP and transcoding to SBC is more
efficient, cost effect, or energy saving.
2) Mic (HFP): Here is the scheduling of 3.75 or 7.5ms might be an important requirement that the Internet CODEC cannot fulfill
always because it must adapt its parameters to the Internet transmission path not just to the Bluetooth link.

Thus, I would recommend to write a liaison statement to Bluetooth AVT group and ask whether they would have interest to include the
Internet CODEC into a future version of A2DP. Definitely, this must not happen soon because the they can do is only if the Internet
CODEC is finishing. Supporting HFP might be more difficult than A2DP because of the very tough requirements on efficiency.

With best regards,

 Christian 

>From: [email-masked] [mailto:[email-masked]] On Behalf Of Raymond (Juin-Hwey) Chen
>Sent: Monday, April 26, 2010 11:02 PM
>To: Mikael Abrahamsson
>Cc: [email-masked]
>Thanks for sharing your experience using a BT headset for Skype calls.  I think part of the quality
>degradation that you experienced was due to the reduction of the audio bandwidth to narrowband (8 kHz
>sampling, 3.4 kHz bandwidth), and another part of the degradation was due to the audible coding noise
>of the CVSD codec, a 40-year-old coding technology first proposed in 1970.
>
>If a high-quality, low-complexity, wider bandwidth IETF codec mode can be implemented in Skype and the
>Bluetooth headset to avoid the CVSD transcoding (together with wideband upgrade of the transducers and
>audio path in the BT headset, of course), then not only will you get much better speech quality in
>your Skype calls than what you have experienced, but also you will get a lower latency.  This is
>because transcoding between the Skype codec and CVSD not only accumulates the coding distortion of the
>two codecs, but also accumulates the coding delays.  Although CVSD is a sample-by-sample codec, BT
>headsets still transmit the CVSD bit-stream in 3.75 ms or 7.5 ms packets, and they can potentially add
>a one-way delay up to 20 ~ 25 ms through the Bluetooth headset (the exact delay depends on the
>implementation).
>
>While we were discussing whether a 5 ms packet size can even be considered, for many years Bluetooth
>headsets have been using an even smaller 3.75 ms packet size.
>
>Best Regards,
>
>Raymond
>

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 30 Apr 2010 14:42:03 +0200

>version 1.5, which supports

No, not version 1.5 but its not yet published successor.

Sorry,

 Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 30 Apr 2010 15:01:29 +0200

>different from "between arbitrary Internet end points".

BTW: IP header compression using ROHC is covered by patents ;-)
https://datatracker.ietf.org/ipr/search/?option=title_search&title_search=rohc

Cheers,

 Christian

---

From: "Roni Even" <[email-masked]>
Date: Fri, 30 Apr 2010 16:12:32 +0300

Hi,
I think that this is going to a wrong direction.  I already suggested that
since the group will  do one codec we first need to decide on the
applications. The initial request was for a wideband codec for the Internet
and this is the application that should dictate the requirements. We can
look at the other applications in the requirements and charter and continue
with the requirements that are in-line with the original application. 
Other applications are nice to have if they do not add more requirements to
the codec to be defined here. We are talking here on more requirements which
in my understanding are not in the scope of the WG

Roni Even

> Sent: Friday, April 30, 2010 3:23 PM
> To: 'Raymond (Juin-Hwey) Chen'; 'Mikael Abrahamsson'
> Cc: [email-masked]
> I just want to share some insights from the recent development of
> Bluetooth's Hands-Free Profile (HFP) version 1.5, which supports
> wideband speech.  One main requirement were on the frame size of 7.5 ms
> because the Bluetooth MAC protocol support scheduling at
> this interval. Actually, to achieve this they modified SBC to work on
> 15 blocks instead of 4,8,12 or 16 blocks and they decided
> against G.722.
> 
> The lesson to learn is about the importance of MAC protocol. To get a
> efficient, low power, and mobile device you have to consider
> the impact of packet scheduling. If packets are scheduled regular, the
> MAC protocol can work more efficient. The more irregular
> Actually, you can translate the cost of packet scheduling to bits per
> packet. Depending on the wireless technology, it might vary
> substantially. The worst case is 802.11b at 11 Mbps at long preamble -
> then, you can add about 1300 bytes to every packet just for
> physical headers and MAC scheduling. However, more modern technologies
> like LTE and IEEE 802.11n are much more efficient in terms of
> per packet overhead.
> 
> 
> If you ask me, one important usage scenario is over wireless links
> supporting low-power mobile devices. If we ignore this scenario
> a) Battery powered mobile devices must be energy efficient to reduce
> the size of the batteries. Also, they should not demand to many
> b) Wireless IP access is also in scope because many devices get
> Internet access LTE, WLAN, Wimax, UMTS, etc.
> 
> 
> 1) Headphones (A2DP): For me it is not clear whether supporting the
> Internet CODEC on top of A2DP (which is - by the way - already
> possible according to Bluetooth spec A2DP V1.2) or using the Internet
> CODEC till the Bluetooth AP and transcoding to SBC is more
> 2) Mic (HFP): Here is the scheduling of 3.75 or 7.5ms might be an
> important requirement that the Internet CODEC cannot fulfill
> always because it must adapt its parameters to the Internet
> transmission path not just to the Bluetooth link.
> 
> Thus, I would recommend to write a liaison statement to Bluetooth AVT
> group and ask whether they would have interest to include the
> Internet CODEC into a future version of A2DP. Definitely, this must not
> happen soon because the they can do is only if the Internet
> CODEC is finishing. Supporting HFP might be more difficult than A2DP
> because of the very tough requirements on efficiency.
> 
> With best regards,
> 
>  Christian
> 
> make

---

From: "codec issue tracker" <[email-masked]>
Date: Fri, 30 Apr 2010 13:14:16 -0000

#16: Multicast?
------------------------------------+---------------------------------------
 Reporter:  hoene@…                 |        Owner:        
     Type:  enhancement             |       Status:  closed
 Priority:  trivial                 |    Milestone:        
Component:  requirements            |      Version:        
 Severity:  Active WG Document      |   Resolution:  fixed 
 Keywords:                          |  
------------------------------------+---------------------------------------
Changes (by hoene@…):

  * status:  new => closed
  * resolution:  => fixed

Comment:

 [Colin]:
 RTP is inherently a group communication protocol, and any codec designed
 for use with RTP should consider operation in various different types of
 group communication scenario (not just multicast).
 RFC 5117 is a good place to start when considering the different types of
 topology in which RTP is used, and the possible placement of mixing and
 switching functions which the codec will need to work with.

 [Christian]:

 I believe that multicast support NEEDS NOT to be particular optimized due
 to the following two reasons:

 b) Multicast is used (to my best knowledge) mainly locally and not
 throughout the Internet, thus if we want to support global interactive
 communication with a good CODEC support of MULTICAST cannot be considered
 as granted.

 [Marshall]:

 [Stephan]:

 [Colin]:

 [Marshall]:

 [Christian]:
 The scalable [teleconference] scenario is [...]:
 -       Central processing and mixing for many participants .
 -       N to 1 and 1 to N communication using efficient distribution
 mechanisms (RTP group communication and IP multicast).

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/codec/trac/ticket/16#comment:1>
codec <http://tools.ietf.org/codec/>

---

From: stephen botzko <[email-masked]>
Date: Fri, 30 Apr 2010 09:41:01 -0400

Hi Christian

I agree that Wireless IP is in scope, one could also argue that LTE might be
in scope, since it will be using IPv6.

As we all know, the internet runs over multiple physical layers, and usually
an end-to-end connection uses more than one physical layer.  So I am not
sure how we can translate layer 2 constraints into requirements for CODEC
itself.

I can see how it might impact the RTP packetization - allowing frames to be
split across packets would allow media-aware devices to adjust the
packetization to optimize delivery.

BTW, I am a bit puzzled by the G.722 statement, since it can run at a 7.5 ms
packet delivery schedule (using 60 payload bytes per packet).

Stephen Botzko

On Fri, Apr 30, 2010 at 8:23 AM, Christian Hoene <[email-masked]>wrote:

> this interval. Actually, to achieve this they modified SBC to work on 15
> blocks instead of 4,8,12 or 16 blocks and they decided
> against G.722.
>
> the impact of packet scheduling. If packets are scheduled regular, the MAC
> protocol can work more efficient. The more irregular
> physical headers and MAC scheduling. However, more modern technologies like
> LTE and IEEE 802.11n are much more efficient in terms of
> per packet overhead.
>
>
> a) Battery powered mobile devices must be energy efficient to reduce the
> size of the batteries. Also, they should not demand to many
> possible according to Bluetooth spec A2DP V1.2) or using the Internet CODEC
> till the Bluetooth AP and transcoding to SBC is more
> 2) Mic (HFP): Here is the scheduling of 3.75 or 7.5ms might be an important
> requirement that the Internet CODEC cannot fulfill
> Thus, I would recommend to write a liaison statement to Bluetooth AVT group
> and ask whether they would have interest to include the

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 30 Apr 2010 09:01:33 -0600

I don't  agree with this logic. 

You seem to be claiming that network delay scales as a factor of audio frame length. I don't think this is true. If the network takes 100 ms to deliver a 5 ms audio packet from a DSL subscriber in Europe to a conference bridge in the US, I see no reason to believe that a packet with 20 ms of audio is going to have a significantly different network transport time. So if we were  on a conference call with over 300 ms of latency (which is very common BTW), it does not seem that changing from 20 ms to 5 ms is going to result in significant (say over 25%) reduction of latency.

I'm not aging for or against 5 ms packets, I'm just saying the logic of "network latency is 5x the packet audio size" makes no sense to me. 

Cullen

On Apr 23, 2010, at 7:43 PM, Raymond (Juin-Hwey) Chen wrote:

> Let's do some math to see why that is so.  Essentially all cellular codecs use a frame size of 20 ms, yet the one-way delay of a cell-to-landline call is typically 80 to 110 ms, or 4 to 5.5 times the codec frame size.  This is because you have not only the codec buffering delay, but also processing delay, transmission delay, and delay due to processor sharing using real-time OS, etc.  An IP phone guru told me that for a typical IP phone application, it is also quite common to see a one-way delay of 5 times the codec frame size.  Let's just take 5X codec frame size as the one-way delay of a typical implementation.  Then, even if all conference participants use their computers to call the conference bridge, if the IETF codec has a frame size of 20 ms, then after the voice signal of a talker goes through the IETF codec to the bridge, it already takes 100 ms one-way delay.  After the bridge decodes all channels, mixes, and re-encodes with the IETF codec and send to every particip
> ant, the one-way delay is now already up to 200 ms, way more than the 150 ms limit I mentioned in my last email.  Now if a talker call into the conference bridge through a cell phone call that has 100 ms one-way delay to the edge of the Internet, by the time everyone else hears his voice, it is already 300 ms later.  Anyone trying to interrupt that cell phone caller will experience the talk-stop-talk-stop problem I mentioned before.  Now if another cell phone caller call into the conference bridge, then the one-way delay of his voice to the first cell phone caller will be a whopping 400 ms! That would probably turn it into half-duplex effectively.
> 

Cullen Jennings
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 30 Apr 2010 18:40:31 -0700

Hi Cullen,

After my original email below, there were several follow-up emails about this, and in those emails I have replaced this oversimplified 5X formula for total delay with the following more detailed formula for typical IP phone implementations:

  One-way delay = codec-independent delay + 3*(codec frame size) + (codec look-ahead) + (codec filtering delay if any)

This formula was obtained from an experienced engineer who has been working on IP phones related fields for more than a decade, and the formula was based on actual observed one-way delay in real-world IP phone implementations.  Similar 3X multiplier is also observed in VoIP gateways.  Even with a fast processor/system optimized from ground up to be low-delay, the measured "codec-dependent" one-way delay of such a VoIP gateway using the G.711 codec with a 5 ms frame/packet size is between 12 and 17 ms, or around 3X the frame size.

The ITU-T uses 2X codec frame size + look-ahead as the one-way codec delay. In some ideal situations, that 2X multiplier is probably achievable, but in more typical situations, 3X is what you are much more likely to find in real-world implementations.

Raymond  

-----Original Message-----
From: Cullen Jennings [mailto:[email-masked]] 
Sent: Friday, April 30, 2010 8:02 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Cullen

Cullen Jennings

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 1 May 2010 06:04:50 +0200 (CEST)

On Fri, 30 Apr 2010, Raymond (Juin-Hwey) Chen wrote:

I think what the 2X 3X factor is handling is what we in the networking 
world calls "serialisation delay". Most equipment today receives the 
packet, looks at it, then sends it out (store and forward). That means 
that on a 2 megabit/s link, it takes:

1/2000000*100*8
.0004  (0.4ms)
to send out a 100 byte packet.

With multiple such links for the packet to traverse, it might be possible 
to see those kind of amlifications of frame size (even though I can't get 
it to amplify that much, for instance when going from 5ms to 20ms packet 
interval). Could be if there are a lot of slow links for the packet to 
traverse (512 kilobits/s for instance).

We stop worrying about serialisation delays when speeds go over several 
hundred megabit/s, because on a gigabit ethernet link serialisation delay 
for a 1500 byte packet is 0.012 ms.

>> Let's do some math to see why that is so.  Essentially all cellular codecs use a frame size of 20 ms, yet the one-way delay of a cell-to-landline call is typically 80 to 110 ms, or 4 to 5.5 times the codec frame size.  This is because you have not only the codec buffering delay, but also processing delay, transmission delay, and delay due to processor sharing using real-time OS, etc.  An IP phone guru told me that for a typical IP phone application, it is also quite common to see a one-way delay of 5 times the codec frame size.  Let's just take 5X codec frame size as the one-way delay of a typical implementation.  Then, even if all conference participants use their computers to call the conference bridge, if the IETF codec has a frame size of 20 ms, then after the voice signal of a talker goes through the IETF codec to the bridge, it already takes 100 ms one-way delay.  After the bridge decodes all channels, mixes, and re-encodes with the IETF codec and send to every parti
 c
> ip

---

From: Koen Vos <[email-masked]>
Date: Fri, 30 Apr 2010 23:07:56 -0700

Quoting "Raymond (Juin-Hwey) Chen":

> (codec look-ahead) + (codec filtering delay if any)
>
> This formula was obtained from an experienced engineer who has been  
> working on IP phones related fields for more than a decade,

At Skype We have 100+ years of combined VoIP experience, and a focus  
on minimizing delay as part of our goal to maximize quality.  The  
consensus among our engineers is that the multiplier is closer to 1  
than to 2, at least for software VoIP applications over typical  
Internet connections.  Some years ago the situation was slightly worse  
because dial-up was more prevalent.

> Similar 3X multiplier is also observed in VoIP gateways.  Even with  
> a fast processor/system optimized from ground up to be low-delay,  
> the measured "codec-dependent" one-way delay of such a VoIP gateway  
> using the G.711 codec with a 5 ms frame/packet size is between 12  
> and 17 ms, or around 3X the frame size.

As I've pointed out before, that doesn't say much about how the delay  
increases with larger frame sizes.  Perhaps the 12~17 ms includes a  
constant delay of 7 ms, and the marginal growth of delay with frame  
size is 1x.

best,
koen.