Subject: Re: [codec] #16: Multicast? (Bluetooth)

From: Koen Vos <[email-masked]>
Date: Sat, 01 May 2010 01:00:47 -0700

Hi Raymond,

I continue to fail to see the connection between the Internet codec  
and Bluetooth, for the reasons below.

(1) Bluetooth != Internet:
Bluetooth devices are wireless audio devices, not VoIP end points, and  
are indeed used mostly for (mobile) PSTN calls.

(2) Diverging requirements:
A codec/mode that meets the BT requirements for ultra-low complexity  
will have a relatively poor coding efficiency, resulting in lower  
audio quality and/or a higher bitrate.  Both of these negatively  
impact the user experience over the Internet.  Therefore, you do not  
want to run a BT codec over the Internet if you can use a more  
efficient codec instead.

(3) Transcoding:
Even when using a BT audio device, a well-designed VoIP end point will  
always transcode between the Internet codec and the BT codec, because:
   a) the reason given in 2) above
   b) the BT device lacks the CPU power and memory to run the entire VoIP stack
   c) it allows for a packet-loss concealment operation in between two  
lossy lags of the end-to-end connection.
Note that such transcoding is also standard with DECT devices, where  
base stations even transcode between G.722 and G.722 (yes: twice the  
same codec).

In short, there is no benefit from the BT and Internet codecs being  
modes of one and the same codec.  This complete lack of overlap means  
that:
   I) it is better to standardize two separate codecs
  II) Bluetooth is out of scope for the Internet codec.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen":

> Hi Koen,
>
> For some reason the SPAM filter accidentally routed your email below  
> (sent last Sunday) to my junk email folder and I just saw it. Sorry  
> about the delay of my response.
>
> I agree that there are some fundamental differences in the  
> requirements for cellular codecs and Bluetooth codecs which caused  
> the codecs in these two types of devices to each go their own way.   
> However, these differences are (or can be) substantially smaller  
> between an Internet codec and Bluetooth codecs, so I think it is  
> easier for Internet devices and Bluetooth devices to use the same  
> codec to avoid the additional delay and coding distortion of  
> transcoding.
>
> (1) Royalty-free requirement:
> Cellular codecs are usually royalty-bearing, and that's acceptable  
> in the cellular world.  Not so with Bluetooth.  Bluetooth devices  
> are meant to be simple and low cost.  As such, Bluetooth SIG  
> basically only wants to standardize royalty-free technologies.   
> That's an important reason why they picked the CVSD codec, a  
> royalty-free old technology of 1970.  We are trying to make the IETF  
> codec royalty-free, so in this regard this goal is consistent with  
> the Bluetooth SIG's royalty-free requirement for codec.
>
> (2) Bit-rate requirement:
> Cellular radio spectrum is a limited, fixed resource that doesn't  
> change with time, and cellular operators spent billions of dollars  
> in radio spectrum auctions. Thus, it is extremely important for  
> cellular codecs to have bit-rates as low as possible, with an  
> average bit-rate often going below 1 bit/sample, to maximize the  
> number of cellular subscribers a given amount of radio spectrum can  
> support.  In contrast, the bit-rate is not nearly as big a concern  
> for Bluetooth. Initially Bluetooth SIG picked the relatively  
> high-bit-rate 64 kb/s CVSD narrowband codec (8 bits/sample) for its  
> simplicity and royalty-free nature among other things.  Since the  
> speeds of the Internet back bone and access networks keep growing  
> with time, the bit-rate of an Internet codec is also not nearly as  
> big a concern as in cellular codecs, and an Internet codec around 2  
> bits/sample can have better trade-offs (e.g. higher quality, lower  
> delay, and lower complexity) for Internet applications than what  
> cellular codecs can provide.  Incidentally, Bluetooth SIG is moving  
> toward 4 bits/sample.  As you can see, in terms of the bit-rate  
> requirement, an Internet codec is much closer to Bluetooth codecs  
> than cellular codecs are.
>
> (3) Complexity requirement:
> Bluetooth headsets have much lower processing power and much smaller  
> batteries than cell phones. The complexity of cellular codecs,  
> typically in the range of 20 to 40 MHz on a DSP, is too high to fit  
> most Bluetooth headsets. However, unlike cell phones and Bluetooth  
> headsets where each is a specific type of device with a relatively  
> narrow range of device complexity, Internet voice/audio applications  
> can potentially encompass a large variety of different device types,  
> from desktop computers at the high end with > 3 GHz multi-core CPU  
> to IP phones and possibly even Bluetooth headsets at the low end  
> with a processor of only a few tens of MHz.  It is up to the IETF  
> codec WG how we want the complexity of the IETF codec to be.  We can  
> standardize just one codec mode that works well for  
> computer-to-computer calls but can't fit in low-end devices, or we  
> can keep that mode but also have a low-complexity mode that can be  
> implemented in low-end devices.  Frankly, I think the second  
> approach makes much more sense since it allows many more devices to  
> benefit from the IETF codec and enables the large number of  
> Bluetooth headset users to avoid the additional distortion and delay  
> associated with transcoding when making Internet calls.
>
> (4) Delay requirement: Due to the need for cellular codecs to  
> achieve bit-rates as low as possible, they sacrificed the coding  
> delay and used a 20 ms frame size, because using a 10 or 5 ms frame  
> size would increase the bit-rate for a given level of speech  
> quality.  On the other hand, a Bluetooth headset needs to have a low  
> delay since its delay is added to the already long cell phone delay.  
>  For the IETF codec, again it is up to the codec WG to decide what  
> kind of codec delay we want, and again I think it makes sense to  
> have a higher-delay, higher bit-rate efficiency mode for  
> bit-rate-sensitive applications and another low-delay mode for  
> delay-sensitive applications, since one size doesn't fit all.  If  
> the IETF codec delay is forced to be one size, the resulting codec  
> will be (potentially very) suboptimal for some applications.
>
> You wrote:
>> Do you think it's realistic for us to come up with a design that
>> fulfills the needs of both worlds?
>
> With a one-size-fit-all approach, probably not, but with a  
> multi-mode approach, then I think so.
>
> Best Regards,
>
> Raymond
>
> -----Original Message-----
> From: Koen Vos [mailto:[email-masked]]
> Sent: Sunday, April 25, 2010 4:01 AM
> To: Raymond (Juin-Hwey) Chen
> Cc: [email-masked]
> Subject: RE: [codec] #16: Multicast? (Bluetooth)
>
> Hi Raymond,
>
> You seem to suggest that the IETF Internet codec should fit Bluetooth
> requirements in order to enable transcoding-free operation all the way
> from the Internet, through the Internet-connected device, to the BT
> wireless audio device.
>
> A similar argument would hold for ITU-T cellular codecs: AMR-WB and
> G.718 could have been designed with BT as an application.  In reality,
> these codecs have very little in common with BT codecs, because of the
> vastly different requirements in terms of
> - complexity
> - memory footprint
> - bitrate
> - scalability
> - bit error robustness
> - packet loss robustness.
>
> Do you think it's realistic for us to come up with a design that
> fulfills the needs of both worlds?
>
> The alternative is to separately design codecs for Internet
> applications and BT devices, and continue the practice of transcoding
> on the Internet-connected device.  That would have a better chance of
> maximizing quality in all scenarios.
>
> best,
> koen.
>
>
> Quoting "Raymond (Juin-Hwey) Chen":
>
>> Hi Koen,
>>
>> Responding to your earlier email about Bluetooth headset application:
>>
>> (1) Although BT SIG standardization is a preferred route, it is
>> technically feasible to negotiate and use a non-Bluetooth-SIG codec.
>>
>> (2) Someone familiar with BT SIG told me that it would probably take
>> only 6 months to add an optional codec to the BT SIG spec and 12 to
>> 18 months to add a mandatory codec.
>>
>> (3) The IETF codec is scheduled to be finalized in 14 months and
>> submitted to IESG in 18 months.  Even if we take the BT SIG route
>> and take 6 to 18 months there.  The total time of 2 to 3 years from
>> now means the Moore's Law would only increase the CPU resources 2X
>> to 3X, and definitely no more than 4X max, not 10X.
>>
>> (4) Most importantly, guess what, in the last several years the
>> Bluetooth headset chips have been growing its processing power at a
>> MUCH, MUCH slower rate than what the Moore's Law says it should.
>> Sometimes they did not increase the speed at all for years.  The
>> reasons? The ASP (average sale price) of Bluetooth chips plummeted
>> very badly, making it unattractive to invest significant resources
>> to make them significantly faster.  Also, for low-end and mid-end BT
>> headsets, the BT chips were often considered "good enough" and there
>> wasn't a strong drive to increase the computing resources.  In
>> addition, the BT headsets got smaller over the last few years; the
>> corresponding reduction in battery size required a reduction in
>> power consumption, which also limited how fast the processor speed
>> could grow.  In the next several years, it is highly likely that the
>> computing capabilities of Bluetooth headset chips will continue to
>> grow at a rate substantially below what's predicted by the Moore's
>> Law.
>>
>> (5) Although Bluetooth supports G.711 as an optional codec,
>> basically no one uses it because it is too sensitive to bit errors.
>> Essentially all the BT mono headsets on the market today are
>> narrowband (8 kHz sampling) headsets using CVSD.  There isn't any
>> real wideband support yet, so your comment about G.722 doesn't
>> apply.  Even after wideband-capable BT headsets come out, for many
>> years to come the majority of the BT headsets (especially mid- to
>> low-end) will still be narrowband only, running only CVSD. Hence,
>> the quality degradation of the CVSD transcoding is real and will be
>> with us for quite a while, so it is desirable for the IETF codec to
>> have a low-complexity mode that can directly run on the BT headsets
>> to avoid the quality degradation of CVSD when using BT headsets to
>> make Internet phone calls.
>>
>> (6) Even if you could use G.711 or G.722 in the BT headsets, they
>> both operate at 64 kb/s.  A low-complexity mode of the IETF codec
>> can operate at half or one quarter of that bit-rate.  This will help
>> conserve BT headsets' radio power because of the lower transmit duty
>> cycle.  It will also help the Bluetooth + WiFi co-existence
>> technologies.
>>
>> (7) Already a lot of people are used to using Bluetooth headsets to
>> make phone calls today.  If they have a choice, many of these people
>> will also want to use Bluetooth headsets to make Internet phone
>> calls, not only through computers, but also through smart phones
>> connected to WiFi or cellular networks.  As more and more states and
>> countries pass laws to ban the use of cell phones that are not in
>> hands-free mode while driving, the number of Bluetooth headset users
>> will only increase with time, and many of them will want to make
>> Internet-based phone calls.
>>
>> Given all the above, I would argue that Bluetooth headset is a very
>> relevant application that the IETF codec should address with a
>> low-complexity mode.
>>
>> Best Regards,
>>
>> Raymond
>>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On
>> Behalf Of Koen Vos
>> Sent: Friday, April 23, 2010 1:16 AM
>> To: [email-masked]
>> Subject: Re: [codec] #16: Multicast?
>>
>> By the time the BlueTooth Special Interest Group will have adopted a
>> future IETF codec standard, Moore's law will surely have multiplied
>> CPU resources in the BT device by one order of magnitude..?  Not sure
>> it makes sense to apply today's BT constraints to tomorrow's codec.
>>
>> I'm not even convinced BlueTooth is a relevant use case for an
>> Internet codec.  BT devices are audio devices more than VoIP end
>> points: BT always connects to the Internet through another device.
>> You could simply first decode incoming packets and send PCM data to
>> the BT device, or use a high-quality/high-bitrate codec like G.722.
>> The requirements for BT devices and the Internet are just too
>> different.  Similarly, GSM phones use AMR on the network side and a
>> different codec towards the BT device.  The required transcoding
>> causes no quality problems because BT supports high bitrates.
>>
>> best,
>> koen.
>>
>>
>> Quoting Raymond (Juin-Hwey) Chen:
>>
>>> Hi Christian,
>>>
>>> My comments about your question of CODEC requirements are in-line.
>>>
>>> Raymond
>>>
>>> From: [email-masked] [mailto:[email-masked]] On
>>> Behalf Of Christian Hoene
>>> Sent: Wednesday, April 21, 2010 12:27 PM
>>> To: 'stephen botzko'
>>> Cc: [email-masked]
>>> Subject: Re: [codec] #16: Multicast?
>>>
>>> Hi,
>>>
>>> if we take those two scenarios (high quality and scalable
>>> teleconferencing), what are then the CODEC requirements?
>>>
>>> High quality:
>>>
>>> -          Quite the same requirement as an end-to-end audio
>>> transmission: high quality and low latency.
>>> [Raymond]: High quality is a given, but I would like to emphasize
>>> the importance of low latency.
>>> (1) It is well-known that the longer the latency, the lower the
>>> perceived quality of the communication link.  The E-model in the
>>> ITU-T Recommendation G.107 models such communication quality in
>>> MOS_cqe, which among other things depends on the so-called "delay
>>> impairment factor" Id.  Basically, MOS_cqe is a monotonically
>>> decreasing function of increasing latency, and beyond about 150 ms
>>> one-way delay, the perceived quality of the communication link drops
>>> rapidly with further delay increase.
>>> (2) The lower the latency, the less audible the echo, and thus the
>>> lower the required echo return loss.  Hence, lower latency means
>>> easier echo control and simpler echo canceller, and as people
>>> already mentioned previously, below a certain delay, an echo is
>>> simply perceived as a harmless side-tone and no echo canceller is
>>> needed. It seems to me that echo control in conference calls is more
>>> difficult than in point-to-point calls.  While I hardly ever heard
>>> echoes in domestic point-to-point calls, in my experience with
>>> conference calls at work, even with the G.711 codec (which has
>>> almost no delay), sometimes I still hear echoes (I just heard
>>> another one this afternoon).  If a relatively long-delay IETF codec
>>> is used, the echo control will be even more problematic.
>>> (3) In normal phone calls or conference calls, people routinely have
>>> a need to interrupt each other, but beyond a certain point, long
>>> latency makes it very difficult for people to interrupt each other
>>> on the call.  This is because when you try to interrupt another
>>> person, that person doesn't hear your interruption until a certain
>>> time later, so he keeps talking, but when you hear that he did not
>>> stop talking when you interrupted, you stop; then, he hears your
>>> interruption, so he stops. When you hear he stops, you start talking
>>> again, but then he also hears you stopped (due to the long delay),
>>> so he also starts talking again.  The net result is that with a long
>>> latency, when you try to interrupt him, you and he end up stopping
>>> and starting at roughly the same time for a few cycles, making it
>>> difficult to interrupt each other.
>>> (4) We need to keep in mind that the IETF codec may not be the only
>>> codec involved in a phone call or a conference call.  We cannot
>>> assume that all conference call participants will be using a
>>> computer to conduct the call. Not only do people use cell phones for
>>> point-to-point phone calls, they also often use cell phones to call
>>> in to conference calls.  The one-way delay for a cell phone call
>>> through one carriers network is typically around 80 to 110 ms.  A
>>> call from a cell phone in a carrier network to another cell phone in
>>> a different type of carrier network can easily double this delay to
>>> 160 ~ 220 ms and makes the total one-way delay already far exceeding
>>> the 150 ms mentioned in (1) above.  Any coding delay added by the
>>> IETF codec will be on top of that long delay, and such coding delay
>>> will be applied twice when both cell phones call through the IETF
>>> codec to a conference bridge.  Even without the IETF codec delay,
>>> when I previously called from a Verizon cell phone to an AT&T cell
>>> phone, I already experienced the problem mentioned in (3) sometimes.
>>>  If the IETF codec has a relatively long delay, adding two times the
>>> IETF codec one-way delay to the already long delay of 160 ~ 220 ms
>>> will make the situation much worse.  Even if just one cell phone is
>>> involved in a conference call, adding twice the one-way delay of a
>>> relatively long-delay IETF codec can still easily push the total
>>> one-way delay beyond 150 ms.
>>> To summarize, my point is that to help reduce potential echo
>>> problems and to ensure a high-quality experience in such a
>>> conference call, the IETF codec should have a delay as low as
>>> possible while maintaining good enough speech quality and a
>>> reasonable bit-rate.
>>>
>>> -          Maybe additionally: variable bit rate encoding to achieve
>>> a multiplexing gain at the receiver
>>>
>>> -          and thus, a fast control loop to cope with variable
>>> bitrates on transmission paths.
>>>
>>> -          Maybe stereo/multichannel support to send the spatial
>>> audio to the headphone or loudspeakers.
>>>
>>> Scalable:
>>>
>>> -          Efficient encoding/transcoding for multiple different
>>> qualities (at the conference bridge)
>>> [Raymond]: I am not sure whether by "efficient", you meant coding
>>> efficiency or computational efficiency.  In any case, I would like
>>> to take this opportunity to express my view that although codec
>>> complexity isn't much of an issue for PC-to-PC calls where there are
>>> GHz of processing power available, the codec complexity is an
>>> important issue in certain application scenarios.  The following are
>>> just some examples.
>>> 1) If a conference bridge has to decode a large number of voice
>>> channels, mix, and re-encode, and if compressed-domain mixing cannot
>>> be done (which is usually the case), then it is important to keep
>>> the decoder complexity low.
>>> 2) In topology b) of your other email
>>> (IPend-to-transcoding_gateway-to-PSTNend), the transcoding gateway,
>>> or VoIP gateway, often has to encode and decode thousands of voice
>>> channels in a single box, so not only the computational complexity,
>>> but also the per-instance RAM size requirement of the codec become
>>> very important for achieving high channel density in the gateway.
>>> 3) Many telephone terminal devices at the edge of the Internet use
>>> embedded processors with limited processing power, and the
>>> processors also have to handle many tasks other than speech coding.
>>> If the IETF codec complexity is too high, some of such devices may
>>> not have sufficient processing power to run it.  Even if the codec
>>> can fit, some battery-powered mobile devices may prefer to run a
>>> lower-complexity codec to reduce power consumption and battery
>>> drain.  For example, even if you make a Internet phone call from a
>>> computer, you may like the convenience of using a Bluetooth headset
>>> that allows you to walk around a bit and have hands-free operation.
>>> Currently most Bluetooth headsets have small form factors with a
>>> tiny battery.  This puts a severe constraint on power consumption.
>>> Bluetooth headset chips typically have very limited processing
>>> capability, and it has to handle many other tasks such as echo
>>> cancellation and noise reduction.  There is just not enough
>>> processing power to handle a relatively high-complexity codec.  Most
>>> BT headsets today relies on the extremely low-complexity,
>>> hardware-based CVSD codec at 64 kb/s to transmit narrowband voice,
>>> but CVSD has audible coding noise, so it degrades the overall audio
>>> quality.  If the IETF codec has low enough complexity, it would be
>>> possible to directly encode and decode the IETF codec bit-stream at
>>> the BT headset, thus avoiding the quality degradation of CVSD
>>> transcoding.
>>> In summary, my point is that the IETF codec should attempt to
>>> achieve a codec complexity as low as possible in both MHz
>>> consumption and RAM size requirement while maintaining good enough
>>> speech quality.
>>>
>>> -          The control loop must not react (fast) because
>>> (multicast) group communication requires to encode at low quality
>>> anyhow.
>>>
>>> -          Receiver side activity detection for music and voice
>>> having low complexity (for the conference bridge)
>>>
>>> -          Efficient mixing of two to four(?) active flows (is this
>>> achievable without the complete process of decoding and encoding
>>> again?)
>>>
>>> Are any teleconferencing requirements missing?
>>>
>>>  Christian
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------
>>> Dr.-Ing. Christian Hoene
>>> Interactive Communication Systems (ICS), University of Tübingen
>>> Sand 13, 72076 Tübingen, Germany, Phone +49 7071 2970532
>>> http://www.net.uni-tuebingen.de/
>>>
>>> From: stephen botzko [mailto:[email-masked]]
>>> Sent: Wednesday, April 21, 2010 8:19 PM
>>> To: Christian Hoene
>>> Cc: [email-masked]
>>> Subject: Re: [codec] #16: Multicast?
>>>
>>> Inline
>>> On Wed, Apr 21, 2010 at 1:27 PM, Christian Hoene
>>> <[email-masked]<mailto:[email-masked]>> wrote:
>>> Hi Stephen,
>>>
>>> not too bad. You answered faster than the mailing list distributes...
>>> Not sure how that happened!
>>>
>>> Comments inline:
>>>
>>>
>>> From: stephen botzko
>>> [mailto:[email-masked]<mailto:[email-masked]>]
>>> Sent: Wednesday, April 21, 2010 7:10 PM
>>> To: Christian Hoene
>>> Cc: [email-masked]<mailto:[email-masked]>
>>>
>>> Subject: Re: [codec] #16: Multicast?
>>>
>>> I agree there are lots of use cases.
>>>
>>>
>>> Though I don't see why high quality has to be given up in order to
>>> be scalable.
>>> CH: These are just experiences from our lab. A spatial audio
>>> conference server including the acoustic 3D sound rendering needs a
>>> LOT of processing power. In the end, we have to remain realistic.
>>> Processing power is always limited thus if we need a lot then we
>>> cannot serve many clients.
>>> Also, I am not sure why you think central mixing is more scalable
>>> than multicast (or why you think it is lower quality either).
>>> CH: With multicast, you need N times 1:N multicast distribution
>>> trees (somewhat small tan O(n)=n²).  With central mixing you need N
>>> times 2 transmission paths (O(n)=n). Also, this distributed mixing
>>> you need N times the mixing at each client. With centralized, you
>>> can live with one mixing for all (and some tricks for serving the
>>> talkers).
>>> I agree you need more distribution trees for multicast if you allow
>>> every site to talk. There is a corresponding benefit, since there is
>>> no central choke point and also less bandwidth on shared WAN links.
>>>
>>> In the distributed case,  you don't need an N-way mixer at each
>>> client, and you also don't need to continuously receive payload on
>>> all N streams at each client either.  In practice you can cap N at a
>>> relatively small number (in the 3-8 range) no matter how large the
>>> conference gets.  In a large conference, you can even choose to drop
>>> your comfort noise if you are receiving two or more streams, and
>>> just send enough to keep your firewall pinhole open.  This is all
>>> assuming a suitable voice activity measure in the RTP packet.  Of
>>> course in the worst case, you will receive all N streams.
>>>
>>> Cheers,
>>>  Christian
>>>
>>> Stephen Botzko
>>> On Wed, Apr 21, 2010 at 12:58 PM, Christian Hoene
>>> <[email-masked]<mailto:[email-masked]>> wrote:
>>> Hi,
>>>
>>> the teleconferencing issue gets complex. I am trying to compile the
>>> different requirements that have been mentioned on this list.
>>>
>>> -          low complexity (with just one active speaker) vs.
>>> multiple speaker mixing vs. spatial audio/stereo mixing
>>>
>>> -          centralized vs. distributed
>>>
>>> -          few participants vs. hundreds of listeners and talkers
>>>
>>> -          individual distribution of audio streams vs. IP multicast
>>> or RTP group communication
>>>
>>> -          efficient encoding of multiple streams having the same
>>> content (but different quality).
>>>
>>> -           I bet I missed some.
>>>
>>> To make things easier, why not to split the teleconferencing
>>> scenario in two: High quality and Scalable?
>>>
>>> The high quality scenario, intended for a low number of users, could
>>> have features like
>>>
>>> -          Distributed processing and mixing
>>>
>>> -          High computational resources to support spatial audio
>>> mixing (at the receiver) and multiple encodings of the same audio
>>> stream at different qualities (at the sender)
>>>
>>> -          Enough bandwidth to allow direct N to N transmissions of
>>> audio streams (no multicast or group communication). This would be
>>> good for the latency, too.
>>>
>>> The scalable scenario is the opposite:
>>>
>>> -          Central processing and mixing for many participants .
>>>
>>> -          N to 1 and 1 to N communication using efficient
>>> distribution mechanisms (RTP group communication and IP multicast).
>>>
>>> -          Low complexity mixing of many using tricks like VAD,
>>> encoding at lowest rate to support many receivers having different
>>> paths, you name it...
>>>
>>> Then, we need not to compare apples with oranges all the time.
>>>
>>> Christian
>>>
>>> ---------------------------------------------------------------
>>> Dr.-Ing. Christian Hoene
>>> Interactive Communication Systems (ICS), University of Tübingen
>>> Sand 13, 72076 Tübingen, Germany, Phone +49 7071 2970532
>>> http://www.net.uni-tuebingen.de/
>>>
>>> From: [email-masked]<mailto:[email-masked]>
>>> [mailto:[email-masked]<mailto:[email-masked]>] On
>>> Behalf Of stephen botzko
>>> Sent: Wednesday, April 21, 2010 4:34 PM
>>> To: Colin Perkins
>>> Cc: [email-masked]<mailto:[email-masked]>;
>>> [email-masked]<mailto:[email-masked]>
>>> Subject: Re: [codec] #16: Multicast?
>>>
>>> in-line
>>>
>>> Stephen Botzko
>>> On Wed, Apr 21, 2010 at 8:17 AM, Colin Perkins
>>> <[email-masked]<mailto:[email-masked]>> wrote:
>>> On 21 Apr 2010, at 12:20, Marshall Eubanks wrote:
>>> On Apr 21, 2010, at 6:48 AM, Colin Perkins wrote:
>>> On 21 Apr 2010, at 10:42, codec issue tracker wrote:
>>> #16: Multicast?
>>> ------------------------------------+----------------------------------
>>> Reporter:  hoene@...                 |       Owner:
>>>  Type:  enhancement             |      Status:  new
>>> Priority:  trivial                 |   Milestone:
>>> Component:  requirements            |     Version:
>>> Severity:  Active WG Document      |    Keywords:
>>> ------------------------------------+----------------------------------
>>> The question arrose whether the interactive CODEC MUST support
>>> multicast in addition to teleconferencing.
>>>
>>> On 04/13/2010 11:35 AM, Christian Hoene wrote:
>>> P.S. On the same note, does anybody here cares about using this
>>> CODEC with multicast? Is there a single commercial multicast voice
>>> deployment? From what I've seen all multicast does is making IETF
>>> voice standards harder to understand or implement.
>>>
>>> I think that would be a mistake to ignore multicast - not because of
>>> multicast itself, but because of Xcast (RFC 5058) which is a
>>> promising technology to replace centralized conference bridges.
>>>
>>> Regarding multicast:
>>>
>>> I think we shall start at user requirements and scenarios.
>>> Teleconference (including mono or spatial audio) might be good
>>> starting point. Virtual environments like second live would require
>>> multicast communication, too. If the requirements of these scenarios
>>> are well understand, we can start to talk about potential solutions
>>> like IP multicast, Xcast or conference bridges.
>>>
>>>
>>> RTP is inherently a group communication protocol, and any codec
>>> designed for use with RTP should consider operation in various
>>> different types of group communication scenario (not just
>>> multicast). RFC 5117 is a good place to start when considering the
>>> different types of topology in which RTP is used, and the possible
>>> placement of mixing and switching functions which the codec will
>>> need to work with.
>>>
>>> It is not clear to me what supporting multicast would entail here.
>>> If this is a codec over RTP, then what is to stop it from being
>>> multicast ?
>>>
>>> Nothing. However group conferences implemented using multicast
>>> require end system mixing of potentially large numbers of active
>>> audio streams, whereas those implemented using conference bridges do
>>> the mixing in a single central location, and generally suppress all
>>> but one speaker. The differences in mixing and the number of
>>> simultaneous active streams that might be received potentially
>>> affect the design of the codec.
>>>
>>> Conference bridges with central mixing almost always mix multiple
>>> speakers.  As you add more streams into the mix, you reduce the
>>> chance of missing onset speech and interruptions, but raise the
>>> noise floor. So even if complexity is not a consideration, there is
>>> value in gating the mixer (instead of always doing a full mix-minus).
>>>
>>> More on point, compressed domain mixing and easy detection of VAD
>>> have both been advocated on these lists, and both simplify the
>>> large-scale mixing problem.
>>>
>>> --
>>> Colin Perkins
>>> http://csperkins.org/
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [email-masked]<mailto:[email-masked]>
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>>
>>>
>>>
>>
>> _______________________________________________
>> codec mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>
>
>
>
>

---

From: stephen botzko <[email-masked]>
Date: Sat, 1 May 2010 09:31:06 -0400

If the frame-size multiplier is due to serialization, then I agree with
Koen's assessment.  In fact on many connections the multiplier would be less
than 1. Dial-up is of course the worst case here, and on those links the
multiplier ought to be close to 2.  Variations due to congestion (and on
some links, polling) are (IMHO) better modeled as jitter.

Gateways are another matter, with the delays being highly dependent on the
product architecture.  Interupt latencies, context switching, bus
architectures, etc. can dominate, so it is totally possible that reducing
the frame size might actually increase the latency (since it increases the
packets per second load on the gateway).  So I agree with Koen on this as
well.

Anecdotal models based on industry experience can be useful guides - though
if we are going to use these models to drive requirements, I'd prefer
something more analytical.

Stephen Botzko

On Sat, May 1, 2010 at 2:07 AM, Koen Vos <[email-masked]> wrote:

>   One-way delay = codec-independent delay + 3*(codec frame size) + (codec
>> look-ahead) + (codec filtering delay if any)
>>
>> This formula was obtained from an experienced engineer who has been
>> working on IP phones related fields for more than a decade,
>>
>
> At Skype We have 100+ years of combined VoIP experience, and a focus on
> minimizing delay as part of our goal to maximize quality.  The consensus
> among our engineers is that the multiplier is closer to 1 than to 2, at
> least for software VoIP applications over typical Internet connections.
>  Some years ago the situation was slightly worse because dial-up was more
> prevalent.
>
>
>
>  Similar 3X multiplier is also observed in VoIP gateways.  Even with a fast
>> processor/system optimized from ground up to be low-delay, the measured
>> "codec-dependent" one-way delay of such a VoIP gateway using the G.711 codec
>> with a 5 ms frame/packet size is between 12 and 17 ms, or around 3X the
>> frame size.
>>
>
> As I've pointed out before, that doesn't say much about how the delay
> increases with larger frame sizes.  Perhaps the 12~17 ms includes a constant
> delay of 7 ms, and the marginal growth of delay with frame size is 1x.
>
> best,
> koen.
>

---

From: "Christian Hoene" <[email-masked]>
Date: Sat, 1 May 2010 16:19:43 +0200

Hi Raymond,

>I would like to respond to a comment and a question in your original email below.
>
>(1) Regarding your comment that "per-instance RAM -- as long as
>it's reasonable -- is probably a bit less important than complexity", I think this may or may not be
>true depending on what is defined as "reasonable", and it also depends on the VoIP gateway under
>consideration. However, I think one thing is clear, though: per-instance RAM is definitely not
>unimportant.  I have worked in a team that designed high-density VoIP gateways, and I know people did
>all kinds of tricks to reduce the RAM usage as much as possible because of the high cost of the
>external high-speed SRAM used by high-speed DSPs in such gateways.
>
>Currently, high-density VoIP gateways can have several tens of thousands of voice channels per box or
>per equipment rack.  If codec A uses X kbytes more per-instance RAM than codec B, than implementing
>codec A rather than codec B in all channels will require X * (number of channels) more kbytes of high-
>speed SRAM, which can be a substantial additional cost.  Similarly, if codec A requires a
>substantially higher DSP MHz than codec B, then implementing codec A in all channels will mean that
>more high-speed DSP chips need to be used to maintain the same number of channels.  That's additional
>cost.  Actually, the more likely scenario is that the number of DSP chips has already maxed out due to
>board space or heat dissipation constraint, then the gateway can only support a substantially smaller
>number of voice channels when compared with using codec B.  Consequently, the channel density per box
>or per rack goes down substantially, and the per-channel deployment cost goes up substantially.  Thus,
>even if implementing codec A or codec B makes no practical difference in a PC, it can make a very real
>and substantial cost difference in VoIP gateways.

The arguments on costs of the gateways is raised often. Thus, it is worthwhile to consider in-depth.

May I ask you for more information about the design of VoIP Gateways. Particular, I am interested in the partition between audio
processing and protocol stack on DSP and CPU.

Does DSP take over all codec processing? May the CPU do some parts of the computation before, during or after DSP does the signal
processing?

How do you count number of channels? Do all voice channels have the same weight regardless their sampling rate?
Say suppose, if the mixing is done for 48kHz instead of 8kHz, how many resource are we allowed to consume more?

With best regards,

 Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Sat, 1 May 2010 19:11:52 +0200

Hi,
 
I agree that serialization, processing, and implementation delay should be distinguished.
 
Assume a low-cost VoIP phone with its processing power being fully utilized by one call: Then, the DSP/CPU needs an entire frame
duration to encode and decode frames. Thus, the latency is increase by one frame length in addition to the serialization delay,
propagation delay, algorithmic delay, dejittering delay, echo cancelling delay, ...  Running the chips at 100% load is of course
cost saving compared to add some more computational resource. But is this still a relevant issue today? 
 
I am not sure whether it always make sense for mobile device to run at 100% load. Of course, from a energy consumption perceptive it
make sense to run CMOS circuit at the lowest possible frequency as power consumption drops quadratic. But maybe running the CPU/DSP
at higher speed and switching to power save mode if after the a frame has been decoded/encoded is be equally energy efficient…
 
Even if a gateways DSP is utilized fully, the processing delays must not be very large. For example, take a gateway serving 10000
calls and all CPUs/DSPs are at 100%. Then, the time needed for encoding/decoding should be frame duration/10000, if the
encoding/decoding is well scheduled.
 
Also, I do not think we shall consider implementation delay, which occurs due to suboptimal implementation. For example, some years
ago we tested the RTT of two Linux softphones link directly together using G.711. It was 400ms. The implementation delay could be a
good performance metric to differentiate two otherwise equal products. Also, the algorithmic processing delay might be subject to
similar market optimization.
 
Having said this, I would anyhow suggest to include the processing delay into the measurement of the end-to-end (acoustic round)
trip time.  Those measurements should be part of the control loop that optimizes the overall conversation call quality.
 
Christian
 
---------------------------------------------------------------
Dr.-Ing. Christian Hoene
Interactive Communication Systems (ICS), University of Tübingen 
Sand 13, 72076 Tübingen, Germany, Phone +49 7071 2970532 
 <http://www.net.uni-tuebingen.de/> http://www.net.uni-tuebingen.de/
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of stephen botzko
Sent: Saturday, May 01, 2010 3:31 PM
To: Koen Vos
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?
 
If the frame-size multiplier is due to serialization, then I agree with Koen's assessment.  In fact on many connections the
multiplier would be less than 1. Dial-up is of course the worst case here, and on those links the multiplier ought to be close to 2.
Variations due to congestion (and on some links, polling) are (IMHO) better modeled as jitter.  

Gateways are another matter, with the delays being highly dependent on the product architecture.  Interupt latencies, context
switching, bus architectures, etc. can dominate, so it is totally possible that reducing the frame size might actually increase the
latency (since it increases the packets per second load on the gateway).  So I agree with Koen on this as well.

Anecdotal models based on industry experience can be useful guides - though if we are going to use these models to drive
requirements, I'd prefer something more analytical.

Stephen Botzko
On Sat, May 1, 2010 at 2:07 AM, Koen Vos <[email-masked]> wrote:
Quoting "Raymond (Juin-Hwey) Chen":
 One-way delay = codec-independent delay + 3*(codec frame size) + (codec look-ahead) + (codec filtering delay if any)

This formula was obtained from an experienced engineer who has been working on IP phones related fields for more than a decade,
 
At Skype We have 100+ years of combined VoIP experience, and a focus on minimizing delay as part of our goal to maximize quality.
The consensus among our engineers is that the multiplier is closer to 1 than to 2, at least for software VoIP applications over
typical Internet connections.  Some years ago the situation was slightly worse because dial-up was more prevalent.

Similar 3X multiplier is also observed in VoIP gateways.  Even with a fast processor/system optimized from ground up to be
low-delay, the measured "codec-dependent" one-way delay of such a VoIP gateway using the G.711 codec with a 5 ms frame/packet size
is between 12 and 17 ms, or around 3X the frame size.
 
As I've pointed out before, that doesn't say much about how the delay increases with larger frame sizes.  Perhaps the 12~17 ms
includes a constant delay of 7 ms, and the marginal growth of delay with frame size is 1x.

best,
koen.

---

From: stephen botzko <[email-masked]>
Date: Sat, 1 May 2010 13:24:13 -0400

Having said this, I would anyhow suggest to include the processing delay
into the measurement of the end-to-end (acoustic round) trip time.  Those
measurements should be part of the control loop that optimizes the overall
conversation call quality.

Hi Christian

I don't understand what you have in mind on this control loop idea, or what
the control loop has to do with acoustic round trip time.

Maybe you could outline your thoughts on this control loop generally in more
detail?

Stephen Botzko

On Sat, May 1, 2010 at 1:11 PM, Christian Hoene <[email-masked]>wrote:

> Assume a low-cost VoIP phone with its processing power being fully utilized
> by one call: Then, the DSP/CPU needs an entire frame duration to encode and
> decode frames. Thus, the latency is increase by one frame length in addition
> to the serialization delay, propagation delay, algorithmic delay,
> dejittering delay, echo cancelling delay, ...  Running the chips at 100%
> load is of course cost saving compared to add some more computational
> resource. But is this still a relevant issue today?
>
>
>
> I am not sure whether it always make sense for mobile device to run at 100%
> load. Of course, from a energy consumption perceptive it make sense to run
> CMOS circuit at the lowest possible frequency as power consumption drops
> quadratic. But maybe running the CPU/DSP at higher speed and switching to
> power save mode if after the a frame has been decoded/encoded is be equally
> energy efficient…
>
>
>
> Even if a gateways DSP is utilized fully, the processing delays must not be
> very large. For example, take a gateway serving 10000 calls and all
> CPUs/DSPs are at 100%. Then, the time needed for encoding/decoding should be
> frame duration/10000, if the encoding/decoding is well scheduled.
>
>
>
> Also, I do not think we shall consider implementation delay, which occurs
> due to suboptimal implementation. For example, some years ago we tested the
> RTT of two Linux softphones link directly together using G.711. It was
> 400ms. The implementation delay could be a good performance metric to
> differentiate two otherwise equal products. Also, the algorithmic processing
> delay might be subject to similar market optimization.
>
>
>
> Having said this, I would anyhow suggest to include the processing delay
> into the measurement of the end-to-end (acoustic round) trip time.  Those
> measurements should be part of the control loop that optimizes the overall
> conversation call quality.
>
>
>
> Christian
>
>
>
> *From**:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *stephen botzko
> *Sent:* Saturday, May 01, 2010 3:31 PM
> *To:* Koen Vos
> *Cc:* [email-masked]
> *Subject:* Re: [codec] #16: Multicast?
>
>
>
> This formula was obtained from an experienced engineer who has been working
> on IP phones related fields for more than a decade,
>
>
>
>  Similar 3X multiplier is also observed in VoIP gateways.  Even with a
> fast processor/system optimized from ground up to be low-delay, the measured

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 4 May 2010 16:00:42 -0700

Hi Christian,

Sorry for the delay of my response.  I was busy with other things and not available to respond to IETF emails until now.

> May I ask you for more information about the design of VoIP Gateways. Particular, I am > interested in the partition between audio

> Does DSP take over all codec processing? May the CPU do some parts of the computation > before, during or after DSP does the signal
> processing?

[Raymond]: I asked an engineering manager who was deeply involved in the design of high-density VoIP gateways. He said that in such gateways, due to the high number of voice channels (thousands) per box, a large number of DSPs and micro-controllers are used, and they are usually structured in a hierarchical way.  The DSPs typically take care of all speech codec processing, echo cancellation, DMTF tone detection, and fax, etc.  The DSPs are usually divided into groups, with each groups of DSPs controlled by a single micro-controller, which handles things like RTP, jitter buffering, packetization, QoS statistics, and moving the voice traffic to and from the DSPs in the group.  Then, on top of that there may be higher-performance controllers, each connected to many such groups of micro-controller + DSPs.  These higher-performance controllers may handle things like call setup, UDP/IP/RTP, routing to and from internal processor groups, and routing to and from external networks/devices.

> Say suppose, if the mixing is done for 48kHz instead of 8kHz, how many resource are we 

[Raymond]: I am not sure what you meant. The channel count is just counting the actual physical voice channels that the gateway can handle simultaneously; it is not a weighted sum. Are you thinking that a 48 kHz channel should be counted more than an 8 kHz channel because it requires more computational resources? Typical VoIP gateways only support 8 kHz telephone-bandwidth speech, so 48 kHz is out of the picture.  
With that said, the complexity difference between speech codecs can make a big difference in the channel density.  Let's say a VoIP gateway supports X simultaneous voice channels running the G.711 codec.  Since the complexity of G.711 PCM is next to nothing, the complexity of each voice channel is dominated by the echo canceller (EC).  Now if you replace the G.711 codec by the G.729A codec which takes about 10 MIPS of computational complexity for a full-duplex codec, that can easily decrease the channel density to X/2.5 per gateway, depending on the EC and other things.  If you replace the G.711 codec by the G.728 codec that takes 30+ MIPS, the channel density can easily go down to X/4 ~ X/5 or worse.  
Thus, if you choose a high-complexity codec, you would need to buy a lot more VoIP gateways to support the same number of voice channels than if you use a low-complexity codec. The cost difference is very real and can be very big.

Raymond

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 4 May 2010 17:09:53 -0700

Hi Christian,

Not all embedded processors can dynamically adjust their clock rate or adjust it as frequently as once or twice per codec frame.

Regarding the 10000-channel gateway example you gave, the actual processing delay is MUCH larger than (frame size)/10000.  Here are the reasons:

(1) That (frame size)/10000 delay assumes that the gateway uses a single DSP fast enough to handle 10000 channels simultaneously.  No gateway is designed that way since no DSP is that fast.  Instead, typical VoIP gateways use many, many DSPs, with each DSP handling only N voice channels, where N might be from single digit to a few dozens, so the actual time the DSP spent in processing the codec frame is (frame size)/N >> (frame size)/10000.

(2) More importantly, the bulk of the processing delay is actually not the (frame size)/N delay above, but the time that each voice channel spends waiting for their turn to get processed by the DSP, and other various buffering delays and scheduling delays encountered by the DSPs and the micro-controllers, because there are so many voice channels that need to be handled simultaneously, making the timing issue extremely complex. These delays depends on the codec frame size chosen.

The engineering manager I mentioned in my last email (who is a different from the IP phone expert I previously mentioned) told me that "the devil is in the details" and that excluding the jitter buffer delay and other codec-independent delays, a straightforward VoIP gateway implementation without paying attention to minimizing delay may have a codec-dependent one-way delay of 5X to 6X codec frame size because of all of the various delays of (2) above due to complex timing issues that come with supporting so many channels simultaneously.  Even after analyzing all delay components carefully and "optimizing the delay to death" until there is no more room for delay reduction, the worst-case one-way codec-dependent delay is still about 3X codec frame size, excluding jitter and other codec-independent delay.  This is an independent corroboration of what the other IP phone expert said about the codec-independent one-way delay of 3X codec frame size for VoIP gateways. (The two of them worked on different projects in different companies.)

My conclusion: while I am less familiar with VoIP soft client implementations on computers, at least for IP phones and VoIP gateways, the rule of thumb that many engineers found to work well for codec-dependent one-way delay is 3X (codec frame size) + other codec buffering delays (e.g. look-ahead and/or filtering delay).

Raymond

From: [email-masked] [mailto:[email-masked]] On Behalf Of Christian Hoene
Sent: Saturday, May 01, 2010 10:12 AM
To: [email-masked]
Subject: Re: [codec] #19: How large is the frame size depended delay / the serialization delay?

Hi,

Assume a low-cost VoIP phone with its processing power being fully utilized by one call: Then, the DSP/CPU needs an entire frame duration to encode and decode frames. Thus, the latency is increase by one frame length in addition to the serialization delay, propagation delay, algorithmic delay, dejittering delay, echo cancelling delay, ...  Running the chips at 100% load is of course cost saving compared to add some more computational resource. But is this still a relevant issue today?

I am not sure whether it always make sense for mobile device to run at 100% load. Of course, from a energy consumption perceptive it make sense to run CMOS circuit at the lowest possible frequency as power consumption drops quadratic. But maybe running the CPU/DSP at higher speed and switching to power save mode if after the a frame has been decoded/encoded is be equally energy efficient...

Even if a gateways DSP is utilized fully, the processing delays must not be very large. For example, take a gateway serving 10000 calls and all CPUs/DSPs are at 100%. Then, the time needed for encoding/decoding should be frame duration/10000, if the encoding/decoding is well scheduled.

Also, I do not think we shall consider implementation delay, which occurs due to suboptimal implementation. For example, some years ago we tested the RTT of two Linux softphones link directly together using G.711. It was 400ms. The implementation delay could be a good performance metric to differentiate two otherwise equal products. Also, the algorithmic processing delay might be subject to similar market optimization.

Having said this, I would anyhow suggest to include the processing delay into the measurement of the end-to-end (acoustic round) trip time.  Those measurements should be part of the control loop that optimizes the overall conversation call quality.

Christian

If the frame-size multiplier is due to serialization, then I agree with Koen's assessment.  In fact on many connections the multiplier would be less than 1. Dial-up is of course the worst case here, and on those links the multiplier ought to be close to 2.  Variations due to congestion (and on some links, polling) are (IMHO) better modeled as jitter.

Gateways are another matter, with the delays being highly dependent on the product architecture.  Interupt latencies, context switching, bus architectures, etc. can dominate, so it is totally possible that reducing the frame size might actually increase the latency (since it increases the packets per second load on the gateway).  So I agree with Koen on this as well.

Anecdotal models based on industry experience can be useful guides - though if we are going to use these models to drive requirements, I'd prefer something more analytical.

Stephen Botzko
On Sat, May 1, 2010 at 2:07 AM, Koen Vos <[email-masked]<mailto:[email-masked]>> wrote:
Quoting "Raymond (Juin-Hwey) Chen":
 One-way delay = codec-independent delay + 3*(codec frame size) + (codec look-ahead) + (codec filtering delay if any)

At Skype We have 100+ years of combined VoIP experience, and a focus on minimizing delay as part of our goal to maximize quality.  The consensus among our engineers is that the multiplier is closer to 1 than to 2, at least for software VoIP applications over typical Internet connections.  Some years ago the situation was slightly worse because dial-up was more prevalent.

Similar 3X multiplier is also observed in VoIP gateways.  Even with a fast processor/system optimized from ground up to be low-delay, the measured "codec-dependent" one-way delay of such a VoIP gateway using the G.711 codec with a 5 ms frame/packet size is between 12 and 17 ms, or around 3X the frame size.

As I've pointed out before, that doesn't say much about how the delay increases with larger frame sizes.  Perhaps the 12~17 ms includes a constant delay of 7 ms, and the marginal growth of delay with frame size is 1x.

best,
koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 4 May 2010 18:15:51 -0700

Hi Koen,

I respect that Skype engineers have a lot of experience in VoIP soft clients running on computers. Skype is the world leader in that area, no question about that.  However, here I am talking about the codec-dependent one-way delay in typical IP phone and VoIP gateway implementations, which are quite different from VoIP soft clients.  How many of your 100+ years of combined VoIP experience were spent in actually designing IP phones or VoIP gateways?  

Regarding the 3X multiplier for VoIP gateways, I already stated clearly in my original text that the 12 to 17 ms was the codec-dependent one-way delay.  There is no "constant delay of 7 ms" in that (if it were constant, it would not be codec-dependent).  The whole 12 to 17 ms delay was proportional to the codec frame size.

As I said in my last email to Christian, there is another independent corroboration by another person (who was deeply involved in VoIP gateway designs) that this 3*(codec frame size) worst-case codec-dependent one-way delay was about the lowest that can be achieved after they "optimized the delay to death".  What I didn't say is that this was actually for G.711 channels with a 10 ms frame/packet size, where the actual processing time spent on encoding and decoding the 10 ms G.711 codec frame was next to nothing, and yet the complex scheduling and buffering delays throughout the system, which are proportional to the 10 ms processing intervals, still added up to 3X frame size.

Currently, 70% to 80% of the phone shipments to large enterprises are IP phones.  With small enterprises also counted, the overall average is about 60% IP phones.  The current industry projection is that within 5 years, the overall average would be 80% to 90% IP phones. (The large enterprises will probably be close to 100% IP phones by then.)  Hence, there are already a huge number of IP phones deployed, and in the future it would be almost all IP phones in the workplace, especially in medium to large companies.  I think it would be a mistake for the IETF Internet codec to completely ignore such IP phone applications, but if we want to address such a huge installed base of IP phones, the 3*(codec frame size) delay is very real for IP phones and it is desirable to have a low-delay mode for the IETF codec to enhance the user experience when using the IETF codec in such IP phones.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Koen Vos
Sent: Friday, April 30, 2010 11:08 PM
To: [email-masked]
Subject: Re: [codec] #16: Multicast?

Quoting "Raymond (Juin-Hwey) Chen":

>   One-way delay = codec-independent delay + 3*(codec frame size) +  
> (codec look-ahead) + (codec filtering delay if any)
>

At Skype We have 100+ years of combined VoIP experience, and a focus  
on minimizing delay as part of our goal to maximize quality.  The  
consensus among our engineers is that the multiplier is closer to 1  
than to 2, at least for software VoIP applications over typical  
Internet connections.  Some years ago the situation was slightly worse  
because dial-up was more prevalent.

> Similar 3X multiplier is also observed in VoIP gateways.  Even with  
> a fast processor/system optimized from ground up to be low-delay,  
> the measured "codec-dependent" one-way delay of such a VoIP gateway  
> using the G.711 codec with a 5 ms frame/packet size is between 12  
> and 17 ms, or around 3X the frame size.

As I've pointed out before, that doesn't say much about how the delay  
increases with larger frame sizes.  Perhaps the 12~17 ms includes a  
constant delay of 7 ms, and the marginal growth of delay with frame  
size is 1x.

best,
koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 4 May 2010 18:55:11 -0700

Hi Koen,

In the same order as your numbered list below:

(1) True, Bluetooth != Internet for now, but why not look into the future and explore what is possible and will be very good to have for the future?

(2) Your argument here doesn't make sense to me.  For PC-to-PC calls, there is no reason to use an ultra-low-complexity mode, so you don't need to suffer the lower coding efficiency, and therefore your concern is not relevant. For any call that involves a Bluetooth headset, it would be better to use a low-complexity mode of the IETF codec on the Bluetooth headset than to go through CVSD transcoding and suffer the significant quality degradation of CVSD and the additional coding delay due to transcoding.

(3)

a) The reason in 2) is not a reason as I explained above.

b) Didn't you say Moore's Law will take care of that? :-) Furthermore, I am not sure it is necessary for the Bluetooth headset to run the entire VoIP stack.

c) It is not at all clear that doing PLC two times on each of the two lossy links is necessarily better than doing PLC just once after the packets go through the two lossy links.  It may well be that the latter approach is better, considering that the transcoding distortion of the second link may go up substantially when encoding the output of the first PLC for the first lossy link.

I) Standardizing two separate codecs takes more time and effort and requires transcoding which increases total coding distortion and total coding delay.

II) For you and some, it is out of the scope, but for others it is not.  Different people have different views.  My view is that if we can cover this very useful usage scenario without too much trouble, why leave it out?

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]]
Sent: Saturday, May 01, 2010 1:01 AM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast? (Bluetooth)

Hi Raymond,

efficient codec instead.

(3) Transcoding:

same codec).

that:

best,

koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 4 May 2010 19:21:51 -0700

Hi Stephen,

> Koen's assessment.  In fact on many connections the multiplier would be 
> less than 1. Dial-up is of course the worst case here, and on those 
> links the multiplier ought to be close to 2.  Variations due to 
> congestion (and on some links, polling) are (IMHO) better modeled as 
> jitter.  
[Raymond]: The frame-size multiplier has many more components than just serialization as I have discussed in my previous emails.  How can the total multiplier be less than 1 when just buffering the current frame of input speech samples will take one frame?  Perhaps you are only talking about the serialization delay component?  I agree that delay variations due to congestion are better modeled as jitter.  That's always the case, and my previous discussions did not include jitter in the codec-dependent delay.

> Gateways are another matter, with the delays being highly dependent on 
> the product architecture.  Interupt latencies, context switching, bus 
> architectures, etc. can dominate, so it is totally possible that 
> reducing the frame size might actually increase the latency (since it 
> increases the packets per second load on the gateway).  So I agree with 
> Koen on this as well.
[Raymond]: I agree with the first half of your paragraph above but not the second half, because the second half contradicts with the real-world observed behaviors: G.711 with a 5 ms frame/packet size gets 12 to 17 ms of codec-dependent delay, while G.711 with a 10 ms frame/packet size gets 50 to 60 ms of worst-case codec-dependent delay before delay optimization, and 30 ms after delay optimization.  In these two actual real-world VoIP gateway implementations, the codec-dependent delay grow with the codec frame size with about a 3X multiplier.

> Anecdotal models based on industry experience can be useful guides - 
> though if we are going to use these models to drive requirements, I'd 
> prefer something more analytical.
[Raymond]: We broke down the one-way delay into codec-independent delay and codec-dependent delay, and then further broke down the codec-dependent delay into the components of codec buffering delay, processing delay, transmission delay (I guess what you called serialization delay), and scheduling and buffering delays of the micro-controllers and DSPs due to many tasks to many channels competing for the processors, etc. We also analyzed which part doesn't change with improving technology (e.g. codec buffering delay) and how certain delay components may change with increasing processor speed and transmission speed.  Isn't that analytical?  How much more analytical can you get than that?  We didn't just throw a few real-world observed codec-dependent delay values and ask everyone to believe the 3X multiplier without any explanation or analysis. No, we just used these real-world values to support our analysis.

---

From: stephen botzko <[email-masked]>
Date: Wed, 5 May 2010 07:06:27 -0400

In-line

Regards
Stephen Botzko

On Tue, May 4, 2010 at 10:21 PM, Raymond (Juin-Hwey) Chen <
[email-masked]> wrote:

> [Raymond]: The frame-size multiplier has many more components than just
> serialization as I have discussed in my previous emails.  How can the total
> multiplier be less than 1 when just buffering the current frame of input
> speech samples will take one frame?  Perhaps you are only talking about the
> serialization delay component?  I agree that delay variations due to
> congestion are better modeled as jitter.  That's always the case, and my
> previous discussions did not include jitter in the codec-dependent delay.
>

Generally the need to buffer the current frame is treated as part of the
algorithmic delay.  At least I believe that is what the ITU-T does.

So maybe we need a list of what all these components are?  I'd suggest
keeping the gateway out of it for the first pass.

> [Raymond]: I agree with the first half of your paragraph above but not the
> second half, because the second half contradicts with the real-world
> observed behaviors: G.711 with a 5 ms frame/packet size gets 12 to 17 ms of
> codec-dependent delay, while G.711 with a 10 ms frame/packet size gets 50 to
> 60 ms of worst-case codec-dependent delay before delay optimization, and 30
> ms after delay optimization.  In these two actual real-world VoIP gateway
> implementations, the codec-dependent delay grow with the codec frame size
> with about a 3X multiplier.
>
I've worked with Gateways\MCUs where the packet size had to be increased
because packet loading in the product became too high.  Also, if you have
QOS features enabled in many routers, the routers themselves have to start
using a "software path", which creates a similar throughput problem in the
routers.  Too many packets per second can overwhelm these devices, creating
both capacity issues and excessive queuing delays.

> [Raymond]: We broke down the one-way delay into codec-independent delay and
> codec-dependent delay, and then further broke down the codec-dependent delay
> into the components of codec buffering delay, processing delay, transmission
> delay (I guess what you called serialization delay), and scheduling and
> buffering delays of the micro-controllers and DSPs due to many tasks to many
> channels competing for the processors, etc. We also analyzed which part
> doesn't change with improving technology (e.g. codec buffering delay) and
> how certain delay components may change with increasing processor speed and
> transmission speed.  Isn't that analytical?  How much more analytical can
> you get than that?  We didn't just throw a few real-world observed
> codec-dependent delay values and ask everyone to believe the 3X multiplier
> without any explanation or analysis. No, we just used these real-world
> values to support our analysis.
>
Serialization is just one component of transmission delay, and is
independent of distance.  As it turns out, it is the network delay component
that depends on packet size.

I don't think the group has an agreed-upon model which names these
components consistently, and describes are appropriately in-scope and which
are out-of-scope.  Perhaps that is one reason why Koen is saying multiplier
the number is 1x.

Also, there are real-world negative consequences to higher packet rates, and
we have not yet considered them.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 6 May 2010 12:03:08 -0700

Hi Stephen,

Sorry, I was too busy to respond yesterday.

You wrote:
> So maybe we need a list of what all these components are?  

[Raymond]: Sure, my previous analysis was an attempt to do just that, but perhaps my list was not complete enough.

> I'd suggest keeping the gateway out of it for the first pass.

[Raymond]: May I ask why? 

> because packet loading in the product became too high.  Also, if you 
> have QOS features enabled in many routers, the routers themselves have 
> to start using a "software path", which creates a similar throughput 
> problem in the routers.  Too many packets per second can overwhelm these 
> devices, creating both capacity issues and excessive queuing delays. 
 
[Raymond]: OK, now I see what you meant when you said "it is totally possible that reducing the frame size might actually increase the latency". This is probably more likely to happen many years ago but less of a problem now, as I was told by networking guys that nowadays networking gears can handle 5 ms packets without problems.  In fact, the VoIP gateway I talked about, which has a 12 to 17 ms codec-dependent one-way delay for a 5 ms frame/packet size, was done 6 or 7 years ago.  Even back then the gateway can handle it without problems. 

> components consistently, and describes are appropriately in-scope and 
> which are out-of-scope.  Perhaps that is one reason why Koen is saying 
> multiplier the number is 1x.  

> Also, there are real-world negative consequences to higher packet rates, 
> and we have not yet considered them. 

[Raymond]: Yes, higher packet rates means higher packet header overhead bit-rates, more burden on networking gears in I/O bandwidth and throughput, etc.  However, that's the price to pay if we need low latency, just like if we want to avoid all these, the price to pay is higher latency.  It's all a matter of trade-off and the best choice depends on the application at hand.  

In Section 2 of Jean-Marc's Internet Draft draft-ietf-codec-requirements-00, 6 specific applications for the IETF codec were listed.  Fully 5 of these 6 applications list less than 10 ms of codec delay as either a requirement or a desirable feature. (The only exception is point-to-point calls.)  The only way to achieve this less than 10 ms codec delay is with a codec frame size of less than 10 ms, and to get the kind of low latency that these 5 applications desire, each packet had better contain only one codec frame as payload (rather than multiple frames). 

So, yeah, there is negative consequences of the resulting higher packet rates, but hey, if we want to get low latency as desired or required by these 5 applications, that's the price we will need to be prepared to pay.  There is no free lunch.  If we want to use a 20 ms frame/packet size to avoid those consequences, then we need pay the price of not achieving the low latency that these 5 applications desire or require.

Raymond

---

From: stephen botzko <[email-masked]>
Date: Thu, 6 May 2010 15:12:20 -0400

I basically agree with your points below.

There are lots of tradeoffs in codec design, including this one.  Personally
I think there is value in a moderate delay 20 ms frame size, possibly
augmented with a low-delay mode.  20 ms works quite well for video
conferencing, since the video frame rate is no faster than 60 fps (about 15
ms per frame).

Regards
Stephen Botzko

On Thu, May 6, 2010 at 3:03 PM, Raymond (Juin-Hwey) Chen <[email-masked]
> wrote:

> [Raymond]: OK, now I see what you meant when you said "it is totally
> possible that reducing the frame size might actually increase the latency".
> This is probably more likely to happen many years ago but less of a problem
> now, as I was told by networking guys that nowadays networking gears can
> handle 5 ms packets without problems.  In fact, the VoIP gateway I talked
> about, which has a 12 to 17 ms codec-dependent one-way delay for a 5 ms
> frame/packet size, was done 6 or 7 years ago.  Even back then the gateway
> can handle it without problems.
>
> [Raymond]: Yes, higher packet rates means higher packet header overhead
> bit-rates, more burden on networking gears in I/O bandwidth and throughput,
> etc.  However, that's the price to pay if we need low latency, just like if
> we want to avoid all these, the price to pay is higher latency.  It's all a
> matter of trade-off and the best choice depends on the application at hand.
>
> In Section 2 of Jean-Marc's Internet Draft
> draft-ietf-codec-requirements-00, 6 specific applications for the IETF codec
> were listed.  Fully 5 of these 6 applications list less than 10 ms of codec
> delay as either a requirement or a desirable feature. (The only exception is
> point-to-point calls.)  The only way to achieve this less than 10 ms codec
> delay is with a codec frame size of less than 10 ms, and to get the kind of
> low latency that these 5 applications desire, each packet had better contain
> only one codec frame as payload (rather than multiple frames).
>
> So, yeah, there is negative consequences of the resulting higher packet
> rates, but hey, if we want to get low latency as desired or required by
> these 5 applications, that's the price we will need to be prepared to pay.
>  There is no free lunch.  If we want to use a 20 ms frame/packet size to
> avoid those consequences, then we need pay the price of not achieving the
> low latency that these 5 applications desire or require.
>
> Raymond
>
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 6 May 2010 18:40:19 -0700

Hi Stephen,

I agree with your points below. I had never said a 20 ms codec frame size should not be used.  I agree and had previously said that there are applications where that 20 ms frame size makes sense.  All I have been arguing in the last couple of weeks was that there are also application scenarios where a low-delay mode is needed, and there are applications where low codec complexity is desirable or even important.

Even draft-ietf-codec-requirements-00 talks about a low-delay mode.  Although the codec WG charter says that "it is not the goal of working group to produce more than one codec", it does acknowledge that "based on the working group's analysis of the design space, the working
group might determine that it needs to produce more than one codec, or a codec with multiple modes".  Thus, I believe that my proposal to have multiple coding modes in the IETF codec (to address the needs of low bit-rate, low delay, or low complexity in different applications) is completely within the scope of the codec WG's charter.

One more comment about the coding delay issue.  When we compare VoIP with traditional circuit-switched PSTN telephony, VoIP is better in most aspects except one: it has substantially longer one-way delay than PSTN telephony.  In this area of delay, PSTN still beats VoIP by far.  As Moore's Law improves technologies over time, the processing speed and communication speed improves with time, so the codec complexity and encoding bit-rate are going to be less and less of an issue as time goes.  However, delay is one thing that doesn't get improved with Moore's Law once a codec frame size is chosen and fixed.

Therefore, if we take a long-term view and attempt to make VoIP better than or at least not significantly worse than PSTN in all aspects, then I believe that we should address the VoIP's long-delay issue head-on with a low-delay mode in the IETF codec.

Raymond

From: stephen botzko [mailto:[email-masked]]
Sent: Thursday, May 06, 2010 12:12 PM
To: Raymond (Juin-Hwey) Chen
Cc: Koen Vos; [email-masked]
Subject: Re: [codec] #16: Multicast?

There are lots of tradeoffs in codec design, including this one.  Personally I think there is value in a moderate delay 20 ms frame size, possibly augmented with a low-delay mode.  20 ms works quite well for video conferencing, since the video frame rate is no faster than 60 fps (about 15 ms per frame).

Regards
Stephen Botzko

On Thu, May 6, 2010 at 3:03 PM, Raymond (Juin-Hwey) Chen <[email-masked]<mailto:[email-masked]>> wrote:
Hi Stephen,

Raymond

---

From: Koen Vos <[email-masked]>
Date: Thu, 06 May 2010 20:41:11 -0700

Hi Raymond,

> I agree and had previously said that there are applications where  
> that 20 ms frame size makes sense.

That's an interesting way to put it... 20 ms is in fact the norm.

Anyway, I think we're all aligned here: ultra-low delay is important  
and has been part of the requirements from day one.  Has anyone  
disagreed?

Personally I'm convinced that people want super-wideband and probably  
even full-band audio before they want a < 20 ms codec, if they have  
the bitrate to support either.  Yes, even for interactive voice.   
Audio bandwidth just has a bigger impact on user experience.  The  
analysis we've done within our Skype network supports this conclusion.

But maybe it's different with IP phones which apparently have problems  
with delay, dunno..

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen" <[email-masked]>:

> Hi Stephen,
>
> I agree with your points below. I had never said a 20 ms codec frame  
> size should not be used.  I agree and had previously said that there  
> are applications where that 20 ms frame size makes sense.  All I  
> have been arguing in the last couple of weeks was that there are  
> also application scenarios where a low-delay mode is needed, and  
> there are applications where low codec complexity is desirable or  
> even important.
>
> Even draft-ietf-codec-requirements-00 talks about a low-delay mode.   
> Although the codec WG charter says that "it is not the goal of  
> working group to produce more than one codec", it does acknowledge  
> that "based on the working group's analysis of the design space, the  
> working
> group might determine that it needs to produce more than one codec,  
> or a codec with multiple modes".  Thus, I believe that my proposal  
> to have multiple coding modes in the IETF codec (to address the  
> needs of low bit-rate, low delay, or low complexity in different  
> applications) is completely within the scope of the codec WG's  
> charter.
>
> One more comment about the coding delay issue.  When we compare VoIP  
> with traditional circuit-switched PSTN telephony, VoIP is better in  
> most aspects except one: it has substantially longer one-way delay  
> than PSTN telephony.  In this area of delay, PSTN still beats VoIP  
> by far.  As Moore's Law improves technologies over time, the  
> processing speed and communication speed improves with time, so the  
> codec complexity and encoding bit-rate are going to be less and less  
> of an issue as time goes.  However, delay is one thing that doesn't  
> get improved with Moore's Law once a codec frame size is chosen and  
> fixed.
>
> Therefore, if we take a long-term view and attempt to make VoIP  
> better than or at least not significantly worse than PSTN in all  
> aspects, then I believe that we should address the VoIP's long-delay  
> issue head-on with a low-delay mode in the IETF codec.
>
> Raymond
>
> There are lots of tradeoffs in codec design, including this one.   
> Personally I think there is value in a moderate delay 20 ms frame  
> size, possibly augmented with a low-delay mode.  20 ms works quite  
> well for video conferencing, since the video frame rate is no faster  
> than 60 fps (about 15 ms per frame).
>
> Regards
> Stephen Botzko
>
>
>
>
> On Thu, May 6, 2010 at 3:03 PM, Raymond (Juin-Hwey) Chen  
> <[email-masked]<mailto:[email-masked]>> wrote:
> Hi Stephen,
>
> [Raymond]: Sure, my previous analysis was an attempt to do just  
> that, but perhaps my list was not complete enough.
>
> possible that reducing the frame size might actually increase the  
> latency". This is probably more likely to happen many years ago but  
> less of a problem now, as I was told by networking guys that  
> nowadays networking gears can handle 5 ms packets without problems.   
> In fact, the VoIP gateway I talked about, which has a 12 to 17 ms  
> codec-dependent one-way delay for a 5 ms frame/packet size, was done  
> 6 or 7 years ago.  Even back then the gateway can handle it without  
> problems.
>
> [Raymond]: Yes, higher packet rates means higher packet header  
> overhead bit-rates, more burden on networking gears in I/O bandwidth  
> and throughput, etc.  However, that's the price to pay if we need  
> low latency, just like if we want to avoid all these, the price to  
> pay is higher latency.  It's all a matter of trade-off and the best  
> choice depends on the application at hand.
>
> draft-ietf-codec-requirements-00, 6 specific applications for the  
> IETF codec were listed.  Fully 5 of these 6 applications list less  
> than 10 ms of codec delay as either a requirement or a desirable  
> feature. (The only exception is point-to-point calls.)  The only way  
> to achieve this less than 10 ms codec delay is with a codec frame  
> size of less than 10 ms, and to get the kind of low latency that  
> these 5 applications desire, each packet had better contain only one  
> codec frame as payload (rather than multiple frames).
>
> So, yeah, there is negative consequences of the resulting higher  
> packet rates, but hey, if we want to get low latency as desired or  
> required by these 5 applications, that's the price we will need to  
> be prepared to pay.  There is no free lunch.  If we want to use a 20  
> ms frame/packet size to avoid those consequences, then we need pay  
> the price of not achieving the low latency that these 5 applications  
> desire or require.
>
> Raymond
>
>

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 7 May 2010 14:31:42 +0200

Hello Raymond,

thank you for your insights into high-density VoIP gateways. I have been told that similar statement are valid also for other
gateway manufactures and that the design of high-density gateways is much more demanding than of softphones:  Because data and code
memory is limited and code cannot be loaded on demand, costs are already high, power consumption is a problem, execution is highly
paralleled, etc... Thus, it makes sense to have a codec (profile) optimized for this use case. 

What are those specific codec requirements, then?
- narrow-band?
- 5ms or 10ms frame size?
- low complexity
- low memory footprint
- transcoding robust ...

And, are these requirements unique or are they covered by existing codecs like G.711 and G.729 already? Is it likely that gateways,
which operate already on their limits, can support yet another codec?

With best regards,

 Christian Hoene

>From: Raymond (Juin-Hwey) Chen [mailto:[email-masked]]
>Sent: Wednesday, May 05, 2010 1:01 AM
>To: Christian Hoene
>Cc: [email-masked]
>Subject: RE: [codec] #16: Multicast?
>
>Hi Christian,
>
>[Raymond]: I asked an engineering manager who was deeply involved in the design of high-density VoIP
>gateways. He said that in such gateways, due to the high number of voice channels (thousands) per box,
>a large number of DSPs and micro-controllers are used, and they are usually structured in a
>hierarchical way.  The DSPs typically take care of all speech codec processing, echo cancellation,
>DMTF tone detection, and fax, etc.  The DSPs are usually divided into groups, with each groups of DSPs
>controlled by a single micro-controller, which handles things like RTP, jitter buffering,
>packetization, QoS statistics, and moving the voice traffic to and from the DSPs in the group.  Then,
>on top of that there may be higher-performance controllers, each connected to many such groups of
>micro-controller + DSPs.  These higher-performance controllers may handle things like call setup,
>UDP/IP/RTP, routing to and from internal processor groups, and routing to and from external
>networks/devices.
>
>[Raymond]: I am not sure what you meant. The channel count is just counting the actual physical voice
>channels that the gateway can handle simultaneously; it is not a weighted sum. Are you thinking that a
>48 kHz channel should be counted more than an 8 kHz channel because it requires more computational
>resources? Typical VoIP gateways only support 8 kHz telephone-bandwidth speech, so 48 kHz is out of
>the picture.
>With that said, the complexity difference between speech codecs can make a big difference in the
>channel density.  Let's say a VoIP gateway supports X simultaneous voice channels running the G.711
>codec.  Since the complexity of G.711 PCM is next to nothing, the complexity of each voice channel is
>dominated by the echo canceller (EC).  Now if you replace the G.711 codec by the G.729A codec which
>takes about 10 MIPS of computational complexity for a full-duplex codec, that can easily decrease the
>channel density to X/2.5 per gateway, depending on the EC and other things.  If you replace the G.711
>codec by the G.728 codec that takes 30+ MIPS, the channel density can easily go down to X/4 ~ X/5 or
>worse.
>Thus, if you choose a high-complexity codec, you would need to buy a lot more VoIP gateways to support
>the same number of voice channels than if you use a low-complexity codec. The cost difference is very
>real and can be very big.
>
>Raymond
>

---

From: Gregory Maxwell <[email-masked]>
Date: Fri, 7 May 2010 11:31:19 -0700

Christian Hoene [[email-masked]] wrote:
[snip]
> And, are these requirements unique or are they covered by existing
> codecs like G.711 and G.729 already? Is it likely that gateways,

I think this is an essential question.  

There are a number of excellent pre-existing codecs out there— during
the formation of this working group we concluded that there was a 
significant non-addressed application space which a new codec 
could satisfy,  but I've seen a number of  requirements raised here
which may be specific to applications for which existing codecs are
already well suited.  

In particular while computational burden is an essential concern, I don't
think it is reasonable to subject a full-band / super-band / wideband
codec to the same criteria which would be reasonable for a narrow band
codec.

If your gateway can't scale to acceptable size except with very
computationally cheap codecs, then you probably ought to be
using one of the already established narrow-band codecs.

I don't think it's a good idea to design for very high levels of complexity
but we ought to keep in mind that the working group is already targeted
at something high quality (and thus more complex) than narrow 
band.  Together with the normal "moore's law"  progress in  transistor
density, I think these factors may suggest a slight bias towards
additional computational complexity at least  where the increase in
complexity can be effectively used.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 7 May 2010 13:11:58 -0700

Hi Christian,

> - 5ms or 10ms frame size?

[Raymond]: 
- For the foreseeable future, I think most of the VoIP gateway
voice channels will still be narrowband.  We may start to see
some wideband (16 kHz sampling).
- There are different VoIP gateway customers.  Some just want
the lowest possible cost of deployment and don't care too 
much about call quality or latency; they will probably 
use 20 ms packets. However, some want to compete seriously 
with the PSTN telephony offered by incumbent telcos. 
There they have to have good quality and low latency
since PSTN latency is very low. These customers will want to 
use 10 ms packets or even 5 ms packets if their hardware can 
handle it.
- Yes, relatively low complexity and low memory footprint are both important for VoIP gateway implementation of codecs.
- Good transcoding performance is also a plus, although 
generally if a codec's single encoding performance is already 
high, then it transcoding performance is usually good as well.

> And, are these requirements unique or are they covered by 
> existing codecs like G.711 and G.729 already? Is it likely 
> that gateways, which operate already on their limits, can 
> support yet another codec?

[Raymond]:
- G.729 has a 10 ms frame size, but its complexity is around 
20 MIPS or so.  G.729A can be optimized to around 10 MIPS, so 
it's better for gateways, but G.729A voice quality is somewhat 
lower than G.729.  The biggest issue with G.729 and G.729A, 
though, is that they are royalty-bearing.  G.711 has good 
quality, low delay, low complexity, and is royalty-free.  
However, at 8 bits/sample, G.711 is inefficient with the bit-
rate. A 16 kb/s codec will reduce the payload bit-rate by a 
factor of 4 and reduce the overall bit-rate (including packet 
header, without header compression) by a factor of about 2X 
and 1.6X for 10 ms and 5 ms packets, respectively, which are 
still significant saving in transmission bandwidth and I/O 
bandwidth/throughput.
- Supporting yet another codec is not a problem for gateways 
because it just means a little additional storage of the codec 
program and data tables in ROM, which is cheap.  The gateways 
are already at or near their limits in terms of MIPS, RAM, and 
I/O bandwidth, not in terms of ROM. More importantly, you 
don't run the additional codec as additional voice channels; 
you simply replace some of the existing codecs already 
supported by the gateway in some (or all) of the channels. 
Therefore, as long as the new codec added does not require 
significantly more MHz or RAM than the existing supported 
codecs, it shouldn't be an issue.

---

From: Koen Vos <[email-masked]>
Date: Fri, 07 May 2010 14:07:56 -0700

Hi Raymond,

> However, delay is one thing that doesn't get improved with Moore's  
> Law once a codec frame size is chosen and fixed.

You've said this before, and it's not true.  Moore's law has reduced  
the delay that users experience a lot:

1. Faster networks reduce transmission delays.  In Skype we've seen  
the average network roundtrip time during a call gradually go down  
from well over 500 ms in 2005 to below 300 ms today.  Skype calls now  
have lower delay than mobile phone calls.

2. Faster CPUs enable tighter scheduling of audio I/O. As a result,  
the buffering delay in the OS/driver has gone done over the years.

3. Faster CPUs mean less processing time delay.  For example, SILK in  
superwideband mode takes less than 10% of real-time on an iPhone.

best,
koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 7 May 2010 14:28:47 -0700

Hi Gregory,

> There are a number of excellent pre-existing codecs out there- during

[Raymond]: I am not sure which codecs satisfy the three conditions 
listed in the codec WG charter. Would you please share which 
codecs you have in mind?  Thanks.

> codec to the same criteria which would be reasonable for a narrow band

[Raymond]: Agreed. I never propose that we should subject full-
/super-/wide-band codecs to the same complexity constraints as for 
narrowband codecs.  What I am proposing, though, is that for a 
particular sampling-rate, be it 16, 32, 44.1, or 48 kHz, when we 
consider different codec options, we should not ignore the codec 
complexity, because high codec complexity have negative consequences 
in low-end devices and gateways. 

My other point is that although the WG does want a wideband/super-
wideband/full-band audio codec, we shouldn't completely ignore 
narrowband, because that's still how most of the point-to-point 
voice calls and multi-party voice conferencing (the first two of the 
six applications listed in the charter) are conducted today and in 
the foreseeable future.  Just North American cable operators alone 
have tens of millions of VoIP telephony subscribers. If you add 
other countries, telcos, independent VoIP operators like Vonage, and 
enterprise IP phone users, although I don't have the stats, I 
wouldn't be surprised if the total VoIP users worldwide exceed 100M, 
probably significantly.  Most of these are still narrowband.  
Therefore, it makes the most sense for the IETF codec to be able to 
also address narrowband speech coding so it has a chance to be used 
by these VoIP users.  If a call goes through the IETF codec and 
reaches out to a conventional phone through a VoIP gateway, then it 
is better if the call doesn't have to be transcoded to another 
medium- or low-bit-rate codec so the additional codec distortion and 
coding delay can be avoided.  This is where the recent discussion of 
VoIP gateways come in.

> complexity can be effectively used.

[Raymond]: I completely agree with you that we should allow 
additional complexity (compared with narrowband) to enable higher-
bandwidth, higher-quality audio. On the other hand, if we are 
evaluating, say, two wideband (16 kHz) codecs, where one requires 50 
MHz on a DSP while the other requires 20 MHz, then such a 
substantial difference in computational complexity should be taken 
into account when considering all other aspects of the codecs.

---

From: Koen Vos <[email-masked]>
Date: Fri, 07 May 2010 14:48:56 -0700

Quoting "Raymond (Juin-Hwey) Chen":

> I wouldn't be surprised if the total VoIP users worldwide exceed 100M,
> probably significantly.

Correct, Skype alone has > 100M active users...

> Most of these are still narrowband.

No: super-wideband.

Are IP phones that come on the market today still using only  
narrowband?  That must be a quickly reducing segment, no?

I do agree that "we shouldn't completely ignore narrowband."

koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 7 May 2010 17:33:25 -0700

Hi Koen,

Ah, sorry, I should have been more specific and said that the codec buffering delay (frame size + look-ahead) does not get improved with Moore's Law.  Before my original sentence that you quoted below, I wrote "As Moore's Law improves technologies over time, the processing speed and communication speed improves with time,..."  From that context, it should be clear that with increased processing speed and communication speed, the time spent on processing and transmitting a codec frame would decrease with time.

However, your previous argument that the frame size multiplier is closer to 1X than 2X already assumes that the processing delay and transmission delay are essentially negligible (at least for computer IP phone calls according to you).  Therefore, you are not going to get much more help from Moore's Law for these two already small delay components. 

On the other hand, devices like VoIP gateways are already using very fast processors and connected to very fast networks, and yet the codec-dependent one-way delay are still around 3X codec frame size because of complicated timing issues and processor buffering needs due to the large number of voice channels competing for resources.  As Moore's Law makes the processor even faster, chances are each processor will handle even more voice channels, so although the time spent on processing each codec frame size will decrease (it is already fairly small), the scheduling/timing issue and the associated buffering needs probably will get even worse, so I am not convinced that the net result is that the codec-dependent delay will get much smaller than 3X codec frame size in the future.

In the email below, you said the average network roundtrip time is below 300 ms. You didn't say below 250 ms or below 200 ms, so I assume that it is just below 300 ms, which means that the one-way network delay is just below 150 ms. Is that correct?  Doesn't this totally depends on what packet size (or packet rate) you use and how much jitter buffer delay you allow?  

When I used some websites to test the Internet connection between my work location in Orange County, California to Los Angeles, I routinely get as low as 2 or 3 ms delay. Such a low network delay for close-by cities is what makes the live music performance over the Internet (the sixth application identified in the codec WG charter) possible, right?  If all Internet connections has close to 150 ms delay one-way, we might as well forget about all those applications that list low delay as required or highly desirable (which leaves only point-to-point calls as the only application).

BTW, I thought you once told me the one-way delay of a Skype call is about 200 ms (which probably makes sense if your one-way network delay is just below 150 ms).  If so, I am confused by your comment below that "Skype calls now have lower delay than mobile phone calls", since mobile phone calls typically have one-way delays of 80 to 110 ms.  Would you please explain this apparent contradiction?

Thanks.

Best Regards,

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]] 
Sent: Friday, May 07, 2010 2:08 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast?

Hi Raymond,

best,
koen.

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Fri, 7 May 2010 18:07:47 -0700

Hi Koen,

> Correct, Skype alone has > 100M active users...

[Raymond]: I knew Skype has hundreds of millions of subscribers. My 
comment above was specifically referring to the non-computer-based 
VoIP subscribers where a physical telephone set is involved in a 
VoIP phone call.  Sorry for not making it clear.

> No: super-wideband.

[Raymond]: See my comment above.  For those VoIP phone calls that 
involve at least a physical telephone set, most of them are still 
narrowband calls today.  I trust that what you said (super-wideband) 
must be correct for computer-to-computer calls since Skype dominates 
that market there and you know that market better than I do, but I 
am talking about something totally different here.

> narrowband?  That must be a quickly reducing segment, no?

[Raymond]: Many enterprise IP phones shipped today indeed have 
wideband (16 kHz) capabilities. However, a lot of enterprises may 
still configure them to do only narrowband calls.  Even if some 
enterprises do enable wideband calls on these phones, wideband calls 
are typically only available for IP-phone-to-IP-phone calls between 
IP phones in their own corporate network.  The moment an employee 
dial out of the corporate network, it's essentially all narrowband.  

There are some wideband cell phone trials in Europe.  However, when 
I spoke to a veteran in the cell phone industry, his assessment was 
that the up-take on wideband cell phones would be pretty slow, and 
the narrowband cell phones would be here to stay for a VERY, VERY 
long time.  He said even today we still have to support the very 
first digital cellular standard codec, the 13 kb/s GSM Full-Rate 
codec, about 20 years after it came out.  His point was that 
narrowband cell phones would not go away for a long, long time.  I 
guess similar things can be said for narrowband land-line 
telephones.

> I do agree that "we shouldn't completely ignore narrowband."

[Raymond]: Glad that we have an agreement here :o)
Have a good weekend.

---

From: Koen Vos <[email-masked]>
Date: Sat, 08 May 2010 23:08:16 -0700

Hi Raymond,

The discussion seems to be turning a bit tiresome and less relevant.

To help you with the apparent contradiction: I should have said  
"mobile-to-mobile calls," not just "mobile phone calls."  The  
round-trip time between two mobile phones equals 4x the  
mobile-to-core-network delay you mention, plus any additional network  
interconnect delay.  So at least 320~440 ms, and for ILD calls easily  
500+ ms.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen" <[email-masked]>:

> Hi Koen,
>
> Ah, sorry, I should have been more specific and said that the codec  
> buffering delay (frame size + look-ahead) does not get improved with  
> Moore's Law.  Before my original sentence that you quoted below, I  
> wrote "As Moore's Law improves technologies over time, the  
> processing speed and communication speed improves with time,..."   
> From that context, it should be clear that with increased processing  
> speed and communication speed, the time spent on processing and  
> transmitting a codec frame would decrease with time.
>
> However, your previous argument that the frame size multiplier is  
> closer to 1X than 2X already assumes that the processing delay and  
> transmission delay are essentially negligible (at least for computer  
> IP phone calls according to you).  Therefore, you are not going to  
> get much more help from Moore's Law for these two already small  
> delay components.
>
> On the other hand, devices like VoIP gateways are already using very  
> fast processors and connected to very fast networks, and yet the  
> codec-dependent one-way delay are still around 3X codec frame size  
> because of complicated timing issues and processor buffering needs  
> due to the large number of voice channels competing for resources.   
> As Moore's Law makes the processor even faster, chances are each  
> processor will handle even more voice channels, so although the time  
> spent on processing each codec frame size will decrease (it is  
> already fairly small), the scheduling/timing issue and the  
> associated buffering needs probably will get even worse, so I am not  
> convinced that the net result is that the codec-dependent delay will  
> get much smaller than 3X codec frame size in the future.
>
> In the email below, you said the average network roundtrip time is  
> below 300 ms. You didn't say below 250 ms or below 200 ms, so I  
> assume that it is just below 300 ms, which means that the one-way  
> network delay is just below 150 ms. Is that correct?  Doesn't this  
> totally depends on what packet size (or packet rate) you use and how  
> much jitter buffer delay you allow?
>
> When I used some websites to test the Internet connection between my  
> work location in Orange County, California to Los Angeles, I  
> routinely get as low as 2 or 3 ms delay. Such a low network delay  
> for close-by cities is what makes the live music performance over  
> the Internet (the sixth application identified in the codec WG  
> charter) possible, right?  If all Internet connections has close to  
> 150 ms delay one-way, we might as well forget about all those  
> applications that list low delay as required or highly desirable  
> (which leaves only point-to-point calls as the only application).
>
> BTW, I thought you once told me the one-way delay of a Skype call is  
> about 200 ms (which probably makes sense if your one-way network  
> delay is just below 150 ms).  If so, I am confused by your comment  
> below that "Skype calls now have lower delay than mobile phone  
> calls", since mobile phone calls typically have one-way delays of 80  
> to 110 ms.  Would you please explain this apparent contradiction?
>
> Thanks.
>
> Best Regards,
>
> Raymond
>

---

From: Koen Vos <[email-masked]>
Date: Sun, 09 May 2010 14:36:36 -0700

Hi Raymond,

I can still only come to one conclusion:
For typical VoIP applications, Moore's law has lessened the pressure  
to reduce bitrates, delay and complexity, and has shifted the focus to  
fidelity instead.  This trend is also visible in recent ITU-T  
standards like G.718, G.719 and G.729.1.

Some exceptions to the rule may exist, but, like ITU-T, we don't need  
to cover every imaginable use case in one standard.

All in all I'm happy with the current requirements document, and it's  
unclear what specific changes (besides a Bluetooth mode) you have in  
mind...?

best,
koen.

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Sun, 9 May 2010 17:52:19 -0400

Quoting Koen Vos <[email-masked]>:
> fidelity instead.

I think this is a typo, and you mean "lessened the pressure to reduce bitrates
and complexity, and has shifted the focus to fidelity and delay instead".

> unclear what specific changes (besides a Bluetooth mode) you have in

Me too; I'd also like some clarification as to whether we're talking about (a)
existing Bluetooth headsets or (b) future hardware designed to support the
IWAC.  If (b), then this is just a matter of negotiating the acceptable
encode+decode complexity, for eventual implementation by DSP or ASIC.  If (a)
... I'm surprised that such a retrofit is even conceivable.  If it is in fact
possible, I would very much appreciate instructions on how to experiment with
new codecs on existing headsets.  Once we know how to build test rigs, we can
see what the complexity requirement there is.  If we can't test it, I don't
think we can target it.

--Ben

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 11:59:54 -0700

Hi Ben,

> and complexity, and has shifted the focus to fidelity and delay instead".

[Raymond]: I agree with you here.

> Me too; I'd also like some clarification as to whether we're talking about (a) > existing Bluetooth headsets or (b) future hardware designed to support the

[Raymond]: I was definitely talking about (b), not (a).

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 12:11:32 -0700

Hi Koen,

> unclear what specific changes (besides a Bluetooth mode) you have in  

[Raymond]:
Correction: I never argued that we should add a "Bluetooth mode".  My main point in this area is just that there are complexity-sensitive applications such as low-end devices and VoIP gateways where a low codec complexity is important or even necessary.  In other words, if the IETF codec complexity is too high, then either it will become much more expensive for some applications (e.g. gateways), or it may not even be feasible to put the codec in some low-end devices.  Bluetooth headset was merely given as an example.  

Considering that there are a very large number of Bluetooth headset users and that the current Bluetooth headsets can already be used for making VoIP phone calls, it would be great if the IETF codec can be implemented in future Bluetooth headsets to avoid the additional coding distortion and delay associated with transcoding. However, with that said, I didn't mean to push a "Bluetooth mode" for the IETF codec. I merely wanted to use Bluetooth headset as an example to make a point that a low codec complexity is desirable and a high codec complexity can have negative consequences.

Raymond

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Mon, 10 May 2010 14:32:39 -0500

> Considering that there are a very large number of Bluetooth headset users and that the current Bluetooth headsets can already be used for making VoIP phone calls, it would be great if the IETF code…

This is all perfectly reasonable, but given the likely timeframe we are
talking about for this codec to be produced and published as a
standards-track RFC, the definition of 'low complexity' in this
discussion is really talking about the 2012-2013 version of 'low
complexity', not today's. It seems highly likely that the MIPS capacity
of the DSPs designed into Bluetooth headsets in 2012 will be vastly
greater than what is used today, if there is an application to take
advantage of the additional MIPS.

I'd hate to see this codec's development constrained in any significant
way by the requirements of a low-complexity device designed in 2009 or
earlier.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
skype: kpfleming | jabber: [email-masked]
Check us out at www.digium.com & www.asterisk.org

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 13:25:59 -0700

Hi Kevin,

I completely agree with you that the IETF codec development should not 
be constrained by a low-complexity device designed in 2009 or earlier, 
and we should look toward the time frame of 2012 and 2013 instead.  

In my previous emails I have indicated that due to many reasons, over 
the last several years the processing power of Bluetooth headsets has 
been increasing at a rate much slower than what's predicted by Moore's 
Law, and it doesn't look like this will change significantly in the next 
few years.  I also said that for the current-generation Bluetooth 
headset chips, the maximum codec complexity it can support is probably 
somewhere around 5 MIPS on a 16-bit fixed-point DSP, and by the time the 
IETF codec becomes a standard, the number may go up to 10 MIPS, or 15 
MIPS at most.  

Thus, if we want mono Bluetooth headsets in the FUTURE (i.e. in the next 
several years) to be able to run the IETF codec in the narrowband or 
wideband mode at least, a good complexity target to shoot for is 10 to 
20 MIPS on a fixed-point DSP.

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 10 May 2010 16:49:41 -0400

Hi Raymond,

I'm just curious about where you got your MIPS figures for Bluetooth? I'm 
not familiar with the type of DSPs used in those applications, but from a 
quick search of more "general-purpose" DSPs (TI, ADI and similar), the 
lowest speed I was able to find (sold in 2010) was a 50 MIPS DSP. Any idea 
what type of DSPs are currently used in Bluetooth?

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 14:24:02 -0700

I should have clarified:
I am not proposing that we limit the complexity of all IETF codec modes 
to 10 to 20 MIPS.  That would be unreasonable, especially for high-
sampling-rate and high-fidelity applications. 

Just like we seemed to have reached a consensus that a low-delay mode 
is necessary to address delay-sensitive applications (as is specified 
in the codec requirement document), we can also have a low-complexity 
mode to address complexity-sensitive applications such as low-
end/mobile devices and gateways. It is for such a low-complexity mode 
that 10 - 20 MIPS is a good target to shoot for, at least for 
narrowband and wideband. (Super-wideband and full-band can be layered 
coding on top of that and do not need to be subject to this 10 - 20 
MIPS target.) For other coding modes that require more processing 
power, this 10 - 20 MIPS target obviously would not apply.

Also, if we don't like to have too many different coding modes, and if 
some modes can be combined, for example, if the low-delay mode can also 
achieve low complexity, then we can combine the low-delay mode and the 
low-complexity mode into a single mode.  We can have another mode 
that's more efficient in bit-rate but may have higher delay and 
complexity to address those applications that are less sensitive 
to delay and complexity.

Raymond

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Raymond (Juin-Hwey) Chen
Sent: Monday, May 10, 2010 1:26 PM
To: Kevin P. Fleming; [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi Kevin,

Raymond

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 10 May 2010 17:32:54 -0400

Oh, I realise your original message did not state 10-20 MIPS as the target 
for all modes. My only questions was about how you came to the 10-20 MIPS 
figure in the first place. Any DSP vendor(s) roadmap or something? Or what 
would be the relevant existing DSP for which we could extrapolate future 
performance? As I said, I'm not too familiar about the DSPs used in 
Bluetooth because all the regular DSPs I can find are 50 MIPS or above.

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 15:02:59 -0700

Hi Jean-Marc,

I didn't just cook up those MIPS numbers out of my head :o)  I have been working with Bluetooth engineers on voice/audio processing for Bluetooth headsets in the last few years.  

Bluetooth headsets normally don't use general-purpose DSPs from those traditional DSP houses that you mentioned below.  A lot of mid- to low-end Bluetooth headsets (where most of the shipping volume is) don't even have any DSP and instead just rely on an ARM processor to do all voice/audio processing and Bluetooth protocol stack.  The 5 MIPS number for current-generation BT headsets was quoted to me by an experienced Bluetooth audio engineering manager in terms of MIPS on an ARM processor, but since most speech coding engineers are more familiar with the MIPS for a general-purpose 16-bit fixed-point DSP, so I converted the MIPS on an ARM to the equivalent MIPS on a fixed-point DSP, and it came to around 5 MIPS.  The 10 to 20 MIPS number is what we may expect in a several years as Moore's Law helps to increase the processing power of Bluetooth headsets.

High-end Bluetooth headset chips may have a DSP on-chip, but it is usually either a proprietary DSP or a DSP core not from the traditional DSP houses but from companies like ARM.

One thing to keep in mind, though, is that even if you have a DSP on a Bluetooth headset chip, and it is clocked at 50 MHz or higher, it doesn't mean that you can use all or most of that DSP processing power for speech or audio coding.  This is because it has to handle many other signal processing tasks such as acoustic echo canceller, single-mic or multi-mic noise suppression, wind noise suppression, packet loss concealment, voice prompt decoding, and even voice command recognition, to name just a few.  You can't even get half of the DSP processing power solely dedicated to speech coding.  You will be lucky if you can get 20% of the DSP for speech coding.  Remember that currently the speech coding part (CVSD codec) takes 0% of the DSP or ARM because it is done in chip hardware.

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Monday, May 10, 2010 1:50 PM
To: Raymond (Juin-Hwey) Chen
Cc: Kevin P. Fleming; [email-masked]
Subject: Re: [codec] #16: Multicast?

Hi Raymond,

Cheers,

	Jean-Marc

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 10 May 2010 15:06:01 -0700

Hi Jean-Marc,

My email below was meant to be an additional note to my previous email and not as a reply to your first email today.  Sorry for the confusion.  

I hope my last email has answered your questions.

Best Regards,

Raymond

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Monday, May 10, 2010 2:33 PM
To: Raymond (Juin-Hwey) Chen
Cc: Kevin P. Fleming; [email-masked]
Subject: Re: [codec] #16: Multicast?

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 11 May 2010 07:59:32 +0200

Hello,

what do you think about STL2005/9 described in ITU-T G.191. It might be a good metric to measure the codec performance in
low-complexity mode. STL2009 is not yet published officially but the STL 2005 manual is available for free.
http://www.itu.int/rec/T-REC-G.191-200508-I/en
Chapter 13, Page 159 describes a set of basic operator and their complexity weights. STL2009 is similar but has guidelines on data
movement and program ROM estimation tool for fixed-point c code and a complexity evaluation tool for floating-point C Code.

However, STL2005 might not be optimal for soft-phone complexity prediction because many operations have overflow control and
saturation, that typically cannot be translated into a single native assembler instruction. Thus, for soft-phone I would recommend
to use slightly modified measures...

With best regards,

 Christian

>Sent: Tuesday, May 11, 2010 12:03 AM
>To: Jean-Marc Valin
>Cc: [email-masked]
>I didn't just cook up those MIPS numbers out of my head :o)  I have been working with Bluetooth
>engineers on voice/audio processing for Bluetooth headsets in the last few years.
>
>Bluetooth headsets normally don't use general-purpose DSPs from those traditional DSP houses that you
>mentioned below.  A lot of mid- to low-end Bluetooth headsets (where most of the shipping volume is)
>don't even have any DSP and instead just rely on an ARM processor to do all voice/audio processing and
>Bluetooth protocol stack.  The 5 MIPS number for current-generation BT headsets was quoted to me by an
>experienced Bluetooth audio engineering manager in terms of MIPS on an ARM processor, but since most
>speech coding engineers are more familiar with the MIPS for a general-purpose 16-bit fixed-point DSP,
>so I converted the MIPS on an ARM to the equivalent MIPS on a fixed-point DSP, and it came to around 5
>MIPS.  The 10 to 20 MIPS number is what we may expect in a several years as Moore's Law helps to
>increase the processing power of Bluetooth headsets.
>
>High-end Bluetooth headset chips may have a DSP on-chip, but it is usually either a proprietary DSP or
>a DSP core not from the traditional DSP houses but from companies like ARM.
>
>One thing to keep in mind, though, is that even if you have a DSP on a Bluetooth headset chip, and it
>is clocked at 50 MHz or higher, it doesn't mean that you can use all or most of that DSP processing
>power for speech or audio coding.  This is because it has to handle many other signal processing tasks
>such as acoustic echo canceller, single-mic or multi-mic noise suppression, wind noise suppression,
>packet loss concealment, voice prompt decoding, and even voice command recognition, to name just a
>few.  You can't even get half of the DSP processing power solely dedicated to speech coding.  You will
>be lucky if you can get 20% of the DSP for speech coding.  Remember that currently the speech coding
>part (CVSD codec) takes 0% of the DSP or ARM because it is done in chip hardware.
>
>Raymond
>
>>> Considering that there are a very large number of Bluetooth headset users and that the current
>Bluetooth headsets can already be used for making VoIP phone calls, it would be great if the IETF
>codec can be implemented in future Bluetooth headsets to avoid the additional coding distortion and
>delay associated with transcoding. However, with that said, I didn't mean to push a "Bluetooth mode"
>for the IETF codec. I merely wanted to use Bluetooth headset as an example to make a point that a low
>codec complexity is desirable and a high codec complexity can have negative consequences.
>>

---

From: Koen Vos <[email-masked]>
Date: Mon, 10 May 2010 23:22:34 -0700

Quoting Benjamin M. Schwartz:

> I think this is a typo, and you mean "lessened the pressure to  
> reduce bitrates and complexity, and has shifted the focus to  
> fidelity and delay instead".

Not a typo: codecs have become more wasteful with delay, while  
delivering better fidelity.  G.718 evolved out of AMR-WB and has more  
than twice the delay.  Same for G.729.1 versus G.729.  This is not by  
accident.

The main rationale for codec delay being less important today is that  
faster hardware has reduced end-to-end delay in every step along the  
way.  As a result, a typical VoIP connection now operates at a flatter  
part of the "impairment-vs-delay" curve, meaning that reducing delay  
by N ms at a given fidelity gives a smaller improvement to end users  
today than it did some years ago.  Therefore, the weight on minimizing  
delay in the "codec design problem" has gone down, and the optimum  
codec operating point has naturally shifted towards higher delay, in  
favor of fidelity.

I've mentioned before that average delay on Internet connections seems  
to be 40% to 50% lower now than just 5 years ago, which is just one  
contributor to lower end-to-end delay.  That doesn't mean high-delay  
connections don't exist - they do, for instance over dial-up or 3G.   
But in those cases it's still better to use a moderate packet rate  
(and bitrate), to minimize congestion risk.

The confusion may come from the fact that the trade-off between  
fidelity and delay changes towards high quality levels: once fidelity  
saturates, delay gets priority.  Even more so because such high  
fidelity enables new, delay-sensitive applications like distributed  
music performances.  This is reflected in the ultra-low delay  
requirements in the requirements document.

To summarize, the case for using sub-20 ms frame sizes with  
medium-fidelity quality is now weaker than ever, because the relative  
importance of fidelity has gone up.

koen.

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 11 May 2010 16:38:40 +0200

Hello,

may I present some results of the ITU-T SG12 on the perceptual effects of delay?

For many years, it was assumed that 150ms is the boundary for interactive voice conversations (see Nobuhiko Kitawaki, and Kenzo
Itoh: Pure Delay Effects on Speech Quality in Telecommunications, IEEE J. on Selected Areas in Commun., Vol.9, No.4, pp.586-593, May
1991) Until 400ms quality is still acceptable (about toll quality). The ITU-T G.107 quality model reflects this opinion.

However, in the recent years, new results have shown that the impact of delay on conversation quality is NOT as strong as assumed.
At the ITU-T, numerous contributions have been made on this issue:

Contribution of BT “Comparison of E-Model and subjective test data for pure-delay conditions” from 2007-01-08:

Legend:

MOS-CQS are subjective conversational tests

MOS-CQE is the E-Modell (G.107)

MOS-LQO are result from listening-only tests.

Also, LM Ericsson described very interesting results in “Investigation of the influence of pure delay, packet loss and audio-video
synchronization for different conversation tasks” from 2007-09-24. For example:

and

Overall, it seems that the limit of 150ms is greatly overestimated. A much relaxed timing is allowed.

Seeing these figures, I have to assume that the ITU-T G.107 standard was a plot of the telcos to make life of VoIP vendors hard.
Well done...

With best regards,

 Christian Hoene

 

Dr.-Ing. Christian Hoene

 

>Sent: Tuesday, May 11, 2010 8:23 AM

>To: [email-masked]; Benjamin M. Schwartz

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 11 May 2010 12:30:15 -0400

On Tue, 2010-05-11 at 16:38 +0200, Christian Hoene wrote:
>  Until 400ms quality is still acceptable (about toll quality). The
> ITU-T G.107 quality model reflects this opinion.
> 
> However, in the recent years, new results have shown that the impact
> of delay on conversation quality is NOT as strong as assumed.

I have been unable to locate the paper that is the source of these
claims.  However, I will note that

(1) The results conflict with common sense.  A round-trip delay of 800
ms makes normal conversation extremely irritating in practice.  I'm not
surprised these results don't show up in laboratory tests, because fast
conversations with interjections and rapid responses typically require a
social context not available in a lab test.

It's possible that the ITU regards "extremely irritating" as
"acceptable", since effective conversation is still possible.  In that
case, I would say that the working group intends to enable applications
with much better than "acceptable" quality.

(2) Tests may have been done in G.711 narrowband, which introduces its
own intelligibility problems and reduces quality expectation.  Higher
fidelity makes latency more apparent.  Similarly, the equipment used may
have introduced quality impairments that make the delay merely one
problem among many.

(3) I presume the tests were done with careful equipment setup to avoid
echo.  The perceived quality impact of echo at 200 ms one-way delay is
enormous, as shown in 

http://downloads.hindawi.com/journals/asp/2008/185248.pdf

Using an echo-canceller impairs quality significantly.  Imperfect echo
cancellation leaves some residual artifact, which is also irritating at
long delays.

The tests (even in the paper above) were performed using a telephone
handset and earpiece.  High-quality telephony with a freestanding
speaker instead of an earpiece demands especially low delay due to the
difficulties with echo cancellation.

--Ben

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 11 May 2010 12:48:29 -0400

> claims.  However, I will note that

As a point of order, I object to any graphs without an available paper  
behind them.

> social context not available in a lab test.

This depends a lot on what sort of discussion is at issue (and, also  
on the culture of the participants).

For example, in my experience telepresence sessions tend to be  
structured meetings and can typically tolerate even half second delays  
without too much disruption, while for a one-on-one conversation on  
the same equipment the same delay can be pretty objectionable.

Having said that, I myself also find the previously attached graphs a  
little odd, and want to see a written description of just what sort of  
experiments they describe.

> have introduced quality impairments that make the delay merely one

Again, without the papers behind them, we are wasting our time having  
such discussions.

Regards
Marshall

P.S. This thread has wandered far from multicast...

---

From: Brian Rosen <[email-masked]>
Date: Tue, 11 May 2010 13:21:56 -0400

I agree with this.  I was in a group that did some research on this
(unpublished, unfortunately) and we confirmed that there is a cliff, around
500 ms round trip, after which conversation is impaired.  It is remarkably
consistent, is more or less independent of culture (with one interesting
exception), and is really a cliff: under it and further improvement is hard
to notice, over it and conversation is impaired, and the difference between
say 750 and 1500ms isn't all that significant.

Engineers who believe delay is a "less is better" quantity need to be
educated that it is not.  It is a threshold.

The interesting exception has to do where you notice the impairment first.
It is when you argue.  Turn-taking is what is impaired, and arguing requires
rapid turn-taking.

So the exception is in cultures where arguing verbally is discouraged.
Politeness masks the delay, somewhat.

Brian

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 11 May 2010 19:47:31 +0200

Hi,

I will ask the ITU-T SG12 chair and the authors, whether we can make those documents public.

With best regards,

 Christian

>From: Marshall Eubanks [mailto:[email-masked]]
>Sent: Tuesday, May 11, 2010 6:48 PM
>To: Ben Schwartz
>Cc: Christian Hoene; [email-masked]

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 11 May 2010 13:53:27 -0400

> educated that it is not.  It is a threshold.

Considering that the network delay is not a constant, you no longer have an 
absolute cliff. So reducing the delay means you can increase the distance 
without falling off the cliff.

	Jean-Marc

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 11 May 2010 14:06:14 -0400

> As a point of order, I object to any graphs without an available paper  

I have located the first paper mentioned by Christian Hoene at
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=81952
but of course it's paywalled.

One test in that paper told trained subjects to "Take turns reading
random numbers aloud as fast as possible", on a pair of handsets with
narrowband uncompressed audio and no echo.  Subjects were able to detect
round-trip delays down to 90 ms.  Conversational efficiency was impaired
even with round-trip delay of 100 ms.

Let me emphasize again that these delays are round-trip, not one-way,
there is no echo, and the task, while designed to expose latency, is
probably less demanding than musical performance.

In the presence of echo, round-trip delay must be kept below 30 ms to
ensure that the echo is perceived as sidetone, according to the Springer
handbook of speech processing:

(http://books.google.com/books?id=Slg10ekZBkAC&lpg=PA83&ots=wc9yM9WrCs&dq=sidetone%20delay%2030%20ms&lr&pg=PA84#v=onepage&q&f=false)

Such low delays are clearly impossible on many paths, but for Boston to
New York City (or London to Paris), ping times can be less than 18 ms,
making echo->sidetone conversion just barely possible for a codec with
5ms frames.

I accept Brian Rosen's claim that a slow conversation doesn't normally
suffer greatly from round-trip latencies up to 500 ms, but under some
circumstances much lower latencies are valuable.  Let's make sure
they're achievable for those who can use them.

--Ben

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 11 May 2010 11:15:30 -0700

Other than potential echo issues, the biggest problem with a one-way delay longer than a few hundred ms is that such a long delay makes it very difficult to interrupt each other, resulting in the start-stop-start-stop cycles I previously talked about.  Therefore, I agree with Ben that if the lab test did not have echoes and did not involve the test subjects trying to interrupt each other, then the test results may appear more benign than what one would experience in the real world.

Note that the top curve in the first figure below is for "listening-only tests".  Well, in that case there was no interaction/interruption at all, so if there was no echoes, either, then it is no wonder that the curve stayed essentially flat.  I do wonder what made the curve go down at 1300 ms; I guess to understand this we need to know what the lab set up was for this test.  Thus, I echo Marshall's opinion that we need the original paper/contribution.

My personal experience with the delay impairment is much worse than the middle curve (MOS-CQS) would suggest and is close to the bottom curve (MOS-CQE).  Back in early 1980s the phone calls I made from southern California to East Asia were carried through geosynchronous satellites with a one-way delay slightly more than 500 ms (see http://en.wikipedia.org/wiki/Geostationary_orbit).  I absolutely hated it, because turn-taking was severely impaired and the only way to interrupt the person at the other side was to keep talking (rudely, I may say) until the other person finally stopped.  Then, starting in late 1980s undersea cables were used to carry my traditional circuit-switched calls to the same person in East Asia, and all of a sudden the delay was much shorter and interrupting each other felt as easy as face-to-face conversation.  It's a night-and-day difference!  Even in early 2000s when I used my cell phone to call my son's cell phone in another cellular network, I could tell that there was a significant delay that noticeably impaired our turn-taking and our ability to interrupt each other, and I didn't like it at all.  Now you know why I advocate low-delay voice communications, have been working on low-delay speech coding for two decades, and have even published a book chapter on low-delay speech coding :o)

Raymond

From: [email-masked] [mailto:[email-masked]] On Behalf Of Christian Hoene
Sent: Tuesday, May 11, 2010 7:39 AM
To: 'Koen Vos'
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Hello,

may I present some results of the ITU-T SG12 on the perceptual effects of delay?
For many years, it was assumed that 150ms is the boundary for interactive voice conversations (see Nobuhiko Kitawaki, and Kenzo Itoh: Pure Delay Effects on Speech Quality in Telecommunications, IEEE J. on Selected Areas in Commun., Vol.9, No.4, pp.586-593, May 1991) Until 400ms quality is still acceptable (about toll quality). The ITU-T G.107 quality model reflects this opinion.
However, in the recent years, new results have shown that the impact of delay on conversation quality is NOT as strong as assumed. At the ITU-T, numerous contributions have been made on this issue:
Contribution of BT "Comparison of E-Model and subjective test data for pure-delay conditions" from 2007-01-08:
Legend:
MOS-CQS are subjective conversational tests
MOS-CQE is the E-Modell (G.107)
MOS-LQO are result from listening-only tests.

Also, LM Ericsson described very interesting results in "Investigation of the influence of pure delay, packet loss and audio-video synchronization for different conversation tasks" from 2007-09-24. For example:

and
Overall, it seems that the limit of 150ms is greatly overestimated. A much relaxed timing is allowed.
Seeing these figures, I have to assume that the ITU-T G.107 standard was a plot of the telcos to make life of VoIP vendors hard. Well done...

With best regards,

 Christian Hoene

Dr.-Ing. Christian Hoene

---

From: stephen botzko <[email-masked]>
Date: Tue, 11 May 2010 14:19:22 -0400

In the presence of echo, round-trip delay must be kept below 30 ms to
ensure that the echo is perceived as sidetone, according to the Springer
handbook of speech processing:
Though true, I don't think this is a mainstream consideration.

VOIP phones that are capable of speakerphone operation all have acoustic
echo cancelers, and those cancelers are already tuned to deal with internet
delays with other voice codecs.  Certainly our phones and videoconferencing
systems do not have problems with path delays of this order (hundreds of
milliseconds).

>From my own experience (not testing) I agree with Brian's claim that 500 ms
round trip is acceptable for most conversation.

It does depend on what you are doing, and there are certainly tasks where
much lower delays are needed.

Regards,
Stephen Botzko

On Tue, May 11, 2010 at 2:06 PM, Ben Schwartz <[email-masked]>wrote:

> (
> http://books.google.com/books?id=Slg10ekZBkAC&lpg=PA83&ots=wc9yM9WrCs&dq=sidetone%20delay%2030%20ms&lr&pg=PA84#v=onepage&q&f=false
> )
>

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 11 May 2010 20:45:24 +0200

Sorry, for the lacking description. I added some more information into the email.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Christian Hoene
Sent: Tuesday, May 11, 2010 4:39 PM
To: 'Koen Vos'
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Hello,

Link  <http://www.itu.int/md/T05-SG12-C-0030/en> http://www.itu.int/md/T05-SG12-C-0030/en

The conversational tests were done in controlled environments with nine pairs of subjects. Two subjects had the common tasks of
their set of sorting pictures in the same order. Other conditions: No echos, G.711, no frame loss

Legend:

MOS-LQO are result from PESQ.

The delay is a one-way delay. Beside MOS values, they also studied the subjective rating of percentage difficultly (%D). Starting at
about 150ms is goes up at reaches 35% at 900ms. After that it remains constant.

Also, LM Ericsson described very interesting results in “Investigation of the influence of pure delay, packet loss and audio-video
synchronization for different conversation tasks” from 2007-09-24. 

 <http://www.itu.int/md/T05-SG12-C-0119/en> http://www.itu.int/md/T05-SG12-C-0119/en

For example: The done conversational tests similar to ITU-T P.805. The conversation lasted about 3 to 5 minutes. 11 pairs of experts
were taken part.

The tasks at 160ms were done about 50s faster than the same task at 600ms

And in the second tests about 60 naïve subjects and experts were taken part to solve a conversational task.

If they were asked for interactivity the ratings look worse.

Hopefully, I will not get trouble because of providing these information.

With best regards,

 Christian Hoene

 

Dr.-Ing. Christian Hoene

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 11 May 2010 14:46:34 -0400

On Tue, 2010-05-11 at 14:19 -0400, stephen botzko wrote:
> VOIP phones that are capable of speakerphone operation all have
> acoustic echo cancelers, and those cancelers are already tuned to deal
> with internet delays with other voice codecs.

In my experience, the biggest problem is in complex conferences with
many users at different latencies, and endpoints consisting of
multiple-microphone multiple-speaker systems scattered across a room and
frequently repositioned.

Anyway, I don't think it's very contentious that

(a) on high latency links (e.g. long distances), AEC will always be
needed in speakerphones,

(b) the perceived audio quality in the presence of AEC is improved by
very low round-trip acoustic latency, and

(c) AEC may be retuned at very low round-trip acoustic latency for
further improvement in perceived audio quality.

Very high audio quality may not yet be a major concern in this market,
but I think this working group is going to change that.

--Ben

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 11 May 2010 17:17:22 -0700

Hi Christian,

(I changed the subject line to better reflect the nature of the email thread below...)

The ITU-T software tool library has been used to measure the complexity of many fixed-point standard speech codecs, and we have done that, too.  So, yes, I think it is a good and well-accepted way to measure the complexity of fixed-point codecs.

For floating-point implementations, although I don't know of a similar software tool for complexity measurement, it is not that difficult to do something similar and add some complexity counting code to an existing floating-point C code to measure the complexity of a floating-point codec.

Best Regards,

Raymond

-----Original Message-----
From: Christian Hoene [mailto:[email-masked]] 
Sent: Monday, May 10, 2010 11:00 PM
To: Raymond (Juin-Hwey) Chen; 'Jean-Marc Valin'
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast?

Hello,

With best regards,

 Christian

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Tue, 11 May 2010 19:18:25 -0700

Hi Koen,

> than twice the delay.  Same for G.729.1 versus G.729.  This is not by  

[Raymond]: If you read the published technical papers on G.718 and 
G.729.1 carefully, I think you will find that the real reason for the 
increased delay is not because they needed a longer delay to achieve 
better fidelity for speech, but because they wanted to extend speech 
codecs to also get good performance when coding general audio (music, 
etc.).  To get good music coding performance, most audio codecs use 
Modified Discrete Cosine Transform (MDCT) with at least a transform 
window size that is fairly large, so most of the audio codecs have 
longer coding delays than speech codecs.  

To code music well, G.718 and G.729.1 developers naturally had to use 
long MDCT transform windows on top of the codec delay already in AMR-
WB and G.729. Even so, the resulting longer delays of G.718 and 
G.729.1 are still not any longer than typical delays of audio codecs; 
in fact, they are probably somewhat shorter. 

My point is that the increased delays of G.718 and G.729.1 are purely 
a result of changing from "speech-only" to "speech and music". It's 
not because the G.718 and G.729.1 developers knew the network delay 
was getting shorter so they could be more wasteful with delay.  
Furthermore, even after they changed the codecs to handle music as 
well as speech, they still chose to make their codec delays shorter 
than the delays of most audio codecs.  Why?  They wanted to make 
their codec delays as short as they could.  In fact, they even made 
an effort to introduce a "low-delay mode" into both G.718 and 
G.729.1. That shows they were pretty concerned about the higher 
delays they needed to have in order to code music well. 

By the way, G.718 does NOT have "more than twice the delay" of AMR-WB 
as you said.  AMR-WB has a 20 ms frame size, 5 ms look-ahead, and 
1.875 ms of filtering delay, for a total algorithmic buffering delay 
of 26.875 ms.  The "normal mode" of G.718 has a buffering delay of 
42.875 ms for 16 kHz wideband input/output. That's only 59.5% higher 
than AMR-WB.  For Layers 1 and 2 coding of speech, the "low-delay 
mode" shaves 10 ms off to give a delay of 32.875 ms, or only 22.3% 
higher than AMR-WB.

When G.729.1 was first standardized in May 2006, there was already a 
low-delay mode for narrowband speech at 8 and 12 kb/s with a 
algorithmic buffering delay of 25 ms.  Later in August 2007, the 
developers made an effort to add another low-delay mode for wideband 
at 14 kb/s that has a buffering delay of 28.94 ms.  If they wanted to 
sacrifice delay to get higher fidelity as you suggested, then why 
would they bother to go back and add another low-delay mode for 
wideband? 

In fact, only a few months ago in their G.729.1 paper in IEEE 
Communications Magazine, October 2009, Varga, Proust, and Taddei 
still emphasized in multiple instances the importance of achieving a 
low coding delay.  I will quote two of the instances: 

"The low-delay mode... was added to the first wideband layer at 14 
kb/s of G.729.1 (August 2007).  The motivation was to address 
applications such as VoIP in enterprise networks where low end-to-end 
delay is crucial" and 

"Indeed, delay is an important performance parameter, and 
transmitting speech with low end-to-end delay is also required in 
several applications making use of wideband signals".

In summary, I do not see a clear trend where codec developers are 
becoming more wasteful with delay in order to get higher fidelity. If 
anything, in recent years I saw a trend of low-delay audio coding, 
such as low-delay AAC and the CELT codec, and I saw the effort by 
G.718 and G.729.1 developers to introduce low-delay modes.

In any case, I thought a few days ago a consensus was already reached 
in the WG email reflector that the IETF codec needs to have a low-
delay mode with a 5 to 10 ms codec frame size so that it can handle 
delay-sensitive applications (that is 5 out of 6 applications listed 
in the charter and codec requirement document).  Therefore, I think 
the discussion in your last email and my current email is mostly of 
academic interest only and doesn't and shouldn't affect how the IETF 
codec is to be designed.

Best Regards,

Raymond

---

From: <[email-masked]>
Date: Wed, 12 May 2010 10:01:18 +0200

Stephen&co,

Generally I believe that AECs mainly take care of the acoustic echo generated in the phone itself (operating on the microphone signal, acoustic delay up to a few ms). Do you mean that there is additional processing on the receiving side for the echo returning from the B user side?

Best regards,
Jari

From: [email-masked] [mailto:[email-masked]] On Behalf Of ext stephen botzko
Sent: 11 May, 2010 21:19
To: Ben Schwartz
Cc: [email-masked]
Subject: Re: [codec] Thresholds and delay.

In the presence of echo, round-trip delay must be kept below 30 ms to
ensure that the echo is perceived as sidetone, according to the Springer
handbook of speech processing:

VOIP phones that are capable of speakerphone operation all have acoustic echo cancelers, and those cancelers are already tuned to deal with internet delays with other voice codecs.  Certainly our phones and videoconferencing systems do not have problems with path delays of this order (hundreds of milliseconds).

>From my own experience (not testing) I agree with Brian's claim that 500 ms round trip is acceptable for most conversation.

It does depend on what you are doing, and there are certainly tasks where much lower delays are needed.

Regards,
Stephen Botzko

On Tue, May 11, 2010 at 2:06 PM, Ben Schwartz <[email-masked]<mailto:[email-masked]>> wrote:

--Ben

---

From: stephen botzko <[email-masked]>
Date: Wed, 12 May 2010 06:43:08 -0400

Acoustic echo cancelers remove the signal from the local speaker(s) that is
feeding back into the local microphone(s).  So if *my* AEC is turned off,
then *you* will hear the echo (and I will not).

The echo that *you* hear is delayed by the round trip transport time, the
processing delays in the sender and receiver, and the acoustic path delay
between the speaker and the microphone.  However, the echo that the
*AEC*detects is only delayed the acoustic path delay and whatever
processing
delay (buffering) the receiver has on its audio capture and render
circuitry.

So you are correct if you are thinking that the network transport time and
codec delay do not impact the requirements for the AEC.

There can also be path delay between the receiver output and the actual
speaker.  For instance in a video conferencing application, the video
processing in the TV itself can add delay, and the TV will delay the audio
to maintain sync.  So if your phone system can be connected to external
sound systems, then that can effect the delays the AEC has to accommodate.

Also, in larger rooms the acoustic path delay can be substantial since sound
takes about 100 ms to travel 100 feet  (341 meters per second).

Regards
Stephen Botzko

On Wed, May 12, 2010 at 4:01 AM, <[email-masked]> wrote:

>  Stephen&co,
>
>
>
> Generally I believe that AECs mainly take care of the acoustic echo
> generated in the phone itself (operating on the microphone signal, acoustic
> delay up to a few ms). Do you mean that there is additional processing on
> the receiving side for the echo returning from the B user side?
>
>
>
> Best regards,
>
> Jari
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *ext stephen botzko
> *Sent:* 11 May, 2010 21:19
> *To:* Ben Schwartz
> *Cc:* [email-masked]
>
> *Subject:* Re: [codec] Thresholds and delay.
>
>
>
> >>>
>

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 12 May 2010 08:23:02 -0400

On Wed, 2010-05-12 at 10:01 +0200, [email-masked] wrote:
> generated in the phone itself (operating on the microphone signal,
> acoustic delay up to a few ms). Do you mean that there is additional
> processing on the receiving side for the echo returning from the B
> user side?

Only in the user's head.  The psychoacoustics of echo are very different
depending on the echo time.  This means that the perceived echo-canceler
fidelity depends on the acoustic round-trip delay.  It also means that
the AEC should be retuned depending on the round-trip delay.

--Ben

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 12 May 2010 09:06:10 -0400

Dear Stephen;

On May 12, 2010, at 6:43 AM, stephen botzko wrote:

> Acoustic echo cancelers remove the signal from the local speaker(s)  
> that is feeding back into the local microphone(s).  So if my AEC is  
> turned off, then you will hear the echo (and I will not).
>
> The echo that you hear is delayed by the round trip transport time,  
> the processing delays in the sender and receiver, and the acoustic  
> path delay between the speaker and the microphone.  However, the  
> echo that the AEC detects is only delayed the acoustic path delay  
> and whatever processing delay (buffering) the receiver has on its  
> audio capture and render circuitry.
>
> So you are correct if you are thinking that the network transport  
> time and codec delay do not impact the requirements for the AEC.
>
> There can also be path delay between the receiver output and the  
> actual speaker.  For instance in a video conferencing application,  
> the video processing in the TV itself can add delay, and the TV will  
> delay the audio to maintain sync.  So if your phone system can be  
> connected to external sound systems, then that can effect the delays  
> the AEC has to accommodate.
>
> Also, in larger rooms the acoustic path delay can be substantial  
> since sound takes about 100 ms to travel 100 feet  (341 meters per  
> second).

It is not clear to me if the CODEC charter includes stereo (not  
typically used in telephony, but reasonably common in video  
conferencing); stereo AEC is tougher than mono and I believe that  
there is some serious IPR in this area.

Regards
Marshall

> Behalf Of ext stephen botzko
> Sent: 11 May, 2010 21:19
> To: Ben Schwartz
> Cc: [email-masked]
>
>
> acoustic echo cancelers, and those cancelers are already tuned to  
> deal with internet delays with other voice codecs.  Certainly our  
> phones and videoconferencing systems do not have problems with path  
> delays of this order (hundreds of milliseconds).
>
> From my own experience (not testing) I agree with Brian's claim that  
> 500 ms round trip is acceptable for most conversation.
>
> On Tue, May 11, 2010 at 2:06 PM, Ben Schwartz <[email-masked] 
> > wrote:
>

---

From: stephen botzko <[email-masked]>
Date: Wed, 12 May 2010 10:47:54 -0400

It also means that the AEC should be retuned depending on the round-trip
delay.
Jari is (I think) pointing out that this last statement is not true.

I agree with Jari's assessment, though I am not sure exactly what you mean
by "retuning".  The far-end user will perceive the fed-back echo in various
ways, and that does depend on the round-trip time.  However, the job of the
AEC is to remove the echo, so there is no fed-back echo to hear.  And the
algorithms which remove echo do *not* depend on the round-trip delay to the
far end.  At least not the good ones.

Perhaps more substantively, I do not think this AEC discussion actually
matters in this WG context.  We are not working on an AEC, we are working on
an Internet Codec.  Even if (for argumentation purposes) you accept the idea
that somehow the AEC needs to be tuned to the round-trip delay, the
round-trip delay varies enormously depending on the connection, and this
round-trip time in general is not even discoverable (esp. if gateways or
SBCs are used).  Nothing we are doing in this group will change any of the
above.

As far as sidetone goes, I do not understand why that keeps coming up
either.

For the speakerphone use case eliminating AECs from both ends requires two
conditions:
(a) the round trip delay has to be very low (30 ms or less, from all delay
sources)
(b) there has to be sufficient attenuation on the loop.

The first condition has been brought up repeatedly.  For general internet
WAN connections it is clearly not met, and it is in fact difficult to meet
even on more local connections.

The second condition has not been discussed much.  But if you do have low
delay but do not have enough attenuation, what you get is feedback, and not
sidetone.  Since users have control over the speaker volume, the second
condition (in general) also can not be guaranteed.

All speakerphones that I know of are either half-duplex or use AECs.  This
is true even for PSTN phones used in local-loop circuit-switched calls,
which is the lowest delay telephony connection that there is.  So for
telephony at least, it is clear that AECs are needed.

So I do not think continued debate on the value of sidetone is productive.
I agree with Michael's comment "achieving low enough latencies for sidetone
perception should not be a goal of the wg"

Stephen Botzko

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 12 May 2010 09:00:10 -0600

On May 4, 2010, at 7:15 PM, Raymond (Juin-Hwey) Chen wrote:

> the 3*(codec frame size) delay is very real for IP phone

This does not match the measurements I have. And I certainly don't have 100+ year voip experience but I do have two of the #1 selling enterprise phones connected to an oscilloscope. Test with other phones suggest most the major vendors of IP hard phones have fairly comparable performance when it comes to delay. 

Cullen

---

From: Michael Knappe <[email-masked]>
Date: Wed, 12 May 2010 08:43:30 -0700

Yes, let’s close off on the AEC and sidetone discussion,  my comments again from earlier in this thread:

Agreed that achieving low enough latencies for sidetone perception should not be a goal of the wg, but we should be aiming if at all possible for better than 250 ms one-way delay in typical (and non-tandemed) deployments. The knee of the one-way delay impairment factor begins rising non-linearly somewhere between 150 and 250 ms.

Cheers,

Mike

On 5/12/10 7:47 AM, "stephen botzko" <[email-masked]> wrote:

   It also means that the AEC should be retuned depending on the round-trip delay.

I agree with Jari's assessment, though I am not sure exactly what you mean by "retuning".  The far-end user will perceive the fed-back echo in various ways, and that does depend on the round-trip time.  However, the job of the AEC is to remove the echo, so there is no fed-back echo to hear.  And the algorithms which remove echo do not depend on the round-trip delay to the far end.  At least not the good ones.

Perhaps more substantively, I do not think this AEC discussion actually matters in this WG context.  We are not working on an AEC, we are working on an Internet Codec.  Even if (for argumentation purposes) you accept the idea that somehow the AEC needs to be tuned to the round-trip delay, the round-trip delay varies enormously depending on the connection, and this round-trip time in general is not even discoverable (esp. if gateways or SBCs are used).  Nothing we are doing in this group will change any of the above.

As far as sidetone goes, I do not understand why that keeps coming up either.

For the speakerphone use case eliminating AECs from both ends requires two conditions:
(a) the round trip delay has to be very low (30 ms or less, from all delay sources)
(b) there has to be sufficient attenuation on the loop.

The first condition has been brought up repeatedly.  For general internet WAN connections it is clearly not met, and it is in fact difficult to meet even on more local connections.

The second condition has not been discussed much.  But if you do have low delay but do not have enough attenuation, what you get is feedback, and not sidetone.  Since users have control over the speaker volume, the second condition (in general) also can not be guaranteed.

All speakerphones that I know of are either half-duplex or use AECs.  This is true even for PSTN phones used in local-loop circuit-switched calls, which is the lowest delay telephony connection that there is.  So for telephony at least, it is clear that AECs are needed.

So I do not think continued debate on the value of sidetone is productive.  I agree with Michael's comment "achieving low enough latencies for sidetone perception should not be a goal of the wg"

Stephen Botzko

On Wed, May 12, 2010 at 8:23 AM, Ben Schwartz <[email-masked]> wrote:

--Ben

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 12 May 2010 11:28:37 -0700

Hi Cullen,

Hmm... That's interesting.  Would you please share more details of 
your measurement equipment setup, the codec used, the codec frame 
size, the number of codec frames in each packet, the way you 
measured the delay, and the measured delay value, etc.?  

I didn't come up with this 3*(codec frame size) delay number for IP 
phones myself.  A very senior technical lead in Broadcom's IP phone 
chips group told me that, and Broadcom is currently the #1 world-
wide market share leader in IP phone chips, accounting for more than 
half of the world's IP phone chip shipments. Most of the world's 
tier-1 IP phone manufacturers use our IP phone chips at least in 
some of their product lines.

I would be very interested to learn more about your measurements to 
try to reconcile these seemingly contradictory statements from two 
different sources.  Thanks.

Best Regards,

Raymond

-----Original Message-----
From: Cullen Jennings [mailto:[email-masked]] 
Sent: Wednesday, May 12, 2010 8:00 AM
To: Raymond (Juin-Hwey) Chen
Cc: Koen Vos; [email-masked]
Subject: Re: [codec] #16: Multicast?

Cullen

---

From: Koen Vos <[email-masked]>
Date: Sun, 16 May 2010 01:28:30 -0700

Quoting "Raymond (Juin-Hwey) Chen":

> chips group told me that

Still:
- You've presented no satisfactory explanation for your theory
- Nor any verifiable empirical evidence
- It comes from an anonymous source
- It has changed over time (going from 5x to 3x)
- It goes against accepted wisdom

So I'd say the burden of proof is on you..

koen.

---

From: "Christian Hoene" <[email-masked]>
Date: Sun, 16 May 2010 10:59:15 +0200

Hi, 

some recent studies on the speech quality of gateways and IP phones are present in 
http://www.head-acoustics.de/telecomfiles/5th_ETSI_SQTE_Anonymous_Test_Report.doc
Please have a look at the document first...

At the first glance, I did see evidence for the 3x factor for IP gateways. The best in class at a 40.4 ms one-way delay for G.711
(20ms packets) and a 54.3 ms delay for G.729 (20ms packets) referring to Tables 4.1 and 4.2. The algorithmic delay of G.729 is 15ms
(5ms lookahead).  Thus, the algorithmic delay was 20ms for G.711 (plus an unknown delay of PLC) and 25ms for G.729. Thus, a factor
of 3 seems to be valid (5msx2,9=53,3-40,4)

In case of IP phones it is not that clear. Actually, I did not fully understand the document. That is 2N and 8N application force?

I bet the experts from HEAD acoustic could bring some light into this discussion and explain, whether they have seen something like
a 3x factor between the algorithmic delay and the final latency.

For, we could officially send a liaison statement to ETSI asking for more data.

With best regards,

 Christian

>Sent: Sunday, May 16, 2010 10:29 AM

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 16 May 2010 11:33:22 +0200 (CEST)

On Sun, 16 May 2010, Christian Hoene wrote:

> At the first glance, I did see evidence for the 3x factor for IP 
> gateways. The best in class at a 40.4 ms one-way delay for G.711 (20ms 
> packets) and a 54.3 ms delay for G.729 (20ms packets) referring to 
> Tables 4.1 and 4.2. The algorithmic delay of G.729 is 15ms (5ms 
> lookahead).  Thus, the algorithmic delay was 20ms for G.711 (plus an 
> unknown delay of PLC) and 25ms for G.729. Thus, a factor of 3 seems to 
> be valid (5msx2,9=53,3-40,4)

One-way delay of 40.4ms for G.711 sounds like 20ms for packet size and 
20ms for jitter buffer. I wouldn't call that "algorithmic delay", that's 
packet and jitter buffer delay, "algorithmic delay" for G.711 should be 
"zero" or am I getting the concepts wrong (since it doesn't do any 
compression).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Roman Shpount <[email-masked]>
Date: Sun, 16 May 2010 17:21:21 -0400

This is a great discussion, but I don't see how it is relevant. If the
question is should the new CODEC support 5 ms frames, then the answer
is probably yes. If there a compelling reason (like achieving better
quality or having algorithm delay that makes this irrelevant) we might
agree to a bigger minimum frame size. If the question is if the CODEC
should support 20ms and bigger compound frames, then the answer is
absolutely yes. There are use cases for both. I guess, the only
question for me, is if in case of 5 ms frames we care about
compression. The packet header overhead is fairly big that you might
use PCMU in this case.

P.S. From what I know about soft phones the delay component dependent
on frame size is 1x. There are other delay components, but they are
not frame size dependent.
_____________________________
Roman Shpount - www.telurix.com

On Sun, May 16, 2010 at 5:33 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> At the first glance, I did see evidence for the 3x factor for IP gateways.
>> The best in class at a 40.4 ms one-way delay for G.711 (20ms packets) and a
>> 54.3 ms delay for G.729 (20ms packets) referring to Tables 4.1 and 4.2. The
>> algorithmic delay of G.729 is 15ms (5ms lookahead).  Thus, the algorithmic
>> delay was 20ms for G.711 (plus an unknown delay of PLC) and 25ms for G.729.
>> Thus, a factor of 3 seems to be valid (5msx2,9=53,3-40,4)
>
> One-way delay of 40.4ms for G.711 sounds like 20ms for packet size and 20ms
> for jitter buffer. I wouldn't call that "algorithmic delay", that's packet
> and jitter buffer delay, "algorithmic delay" for G.711 should be "zero" or
> am I getting the concepts wrong (since it doesn't do any compression).
>
> --

---

From: stephen botzko <[email-masked]>
Date: Sun, 16 May 2010 19:08:44 -0400

Thus, the algorithmic delay was 20ms for G.711 (plus an unknown delay of
PLC) and 25ms for G.729.
I am not sure what you are calling "algorithmic delay".  Generally it is the
delay inherent to the algorithm (the delay you would get if you had an
infinitely fast processor).

Total algorithmic delay for G.711 should be no more than .125 ms, total
algorithmic delay for G.729 is 20 ms (15 ms per frame and 5 ms look-ahead).
References are here:

http://en.wikipedia.org/wiki/G.711
http://en.wikipedia.org/wiki/G.729

Regards,
Stephen Botzko
On Sun, May 16, 2010 at 4:59 AM, Christian Hoene <[email-masked]>wrote:

> The best in class at a 40.4 ms one-way delay for G.711
> (20ms packets) and a 54.3 ms delay for G.729 (20ms packets) referring to
> Tables 4.1 and 4.2. The algorithmic delay of G.729 is 15ms
> (5ms lookahead).  Thus, the algorithmic delay was 20ms for G.711 (plus an
> unknown delay of PLC) and 25ms for G.729. Thus, a factor
> In case of IP phones it is not that clear. Actually, I did not fully
> understand the document. That is 2N and 8N application force?
>
> I bet the experts from HEAD acoustic could bring some light into this
> discussion and explain, whether they have seen something like

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 17 May 2010 19:32:44 -0700

Hi Koen,

I agree with Roman that this 1X versus 3X debate has reached a 
point that it is almost irrelevant.  What's really relevant now is 
the question of whether the IETF codec should have a low-delay mode 
with a codec frame size of 5 to 10 ms, and for that question you 
previously wrote on May 6:

> Anyway, I think we're all aligned here: ultra-low delay is 
> important and has been part of the requirements from day one. Has 
> anyone disagreed?

After this email of yours, I have not seen anyone explicitly 
disagreed.  In fact, of the 6 intended applications listed in the 
WG charter, draft-ietf-codec-requirements-00 lists 2 of them having 
< 10 ms codec delay as a requirement and 3 of them having < 10 ms 
codec delay as "desirable" or "highly desirable".  It seems to me 
that the consensus in the WG is that we need a low-delay mode with 
a 5 to 10 ms codec frame size to address delay-sensitive 
applications, and we need another higher-delay mode with perhaps a 
20 ms frame size to get higher bit-rate efficiency for applications 
that are less sensitive to delay.  If we agree with that (and I 
thought you already agreed on May 6), then whether the frame size 
multiplier is 1X or 3X doesn't really matter and shouldn't affect 
the codec requirements or the direction of the codec design. 

With that said, I don't want you to think I impolitely ignored your 
comments in your email below, so I will respond to them in-line.  
(For those who are not interested, please ignore the rest of this 
email.)

> - You've presented no satisfactory explanation for your theory

[Raymond]: I already explained and analyzed it in my previous emails.  
The codec-dependent delay = (codec frame size) + (codec look-ahead) + 
(codec filtering delay) + (processing delay to encode and decode one 
frame) + (transmission delay to send one frame of compressed bit-
stream) + (multi-tasking delay due to processor not immediately 
available to start encoding and decoding operations as soon as one 
frame of input audio samples or received bit-stream is ready) + 
(possibly miscellaneous other delays)

Although the processing delay is short (but not zero) for a PC having 
2 ~ 3 GHz CPU, for IP phones the clock speed of the processor is much 
lower, resulting in significantly higher processing delay than in a 
PC.  Also, the multi-tasking delay can be quite significant, and it 
is dependent on the codec frame size in IP phones.  All these delay 
components here and there can easily add up to 3X codec frame size.

If you have a copy of the 1995 Elsevier book "Speech Coding and 
Synthesis" edited by W.B. Kleijn and K.K. Paliwal, please go read 
Section 2 and see Fig. 1 of my chapter 6 on low-delay coding of 
speech.  It should help you understand what I have been saying.  
Unfortunately due to copyright restrictions, I cannot scan those 
pages and attach with this email.

> - Nor any verifiable empirical evidence

[Raymond]: The one-way delay measurement was done at the lab of 
Broadcom's IP phone chip group.  I don't know how I can make that 
"verifiable" to you.  

> - It comes from an anonymous source

[Raymond]: I didn't just say that I heard it from someone who worked 
at some (unnamed) company; I specifically said this info came from 
Broadcom's IP phone group.  Is that not sufficient?

> - It has changed over time (going from 5x to 3x)

[Raymond]: I admit that this was my mistake due to a misunderstanding 
of what that senior technical lead I mentioned above told me. 
Initially he said the real-world one-way delay was about 5X codec 
frame size. I thought he meant codec-dependent delay; that's why I 
initially used 5X.  Later he clarified to me that when he said 5X, he 
included codec-independent delay, and the codec-dependent delay 
component was really 3X, so I corrected myself to 3X.

> - It goes against accepted wisdom

[Raymond]: No, the commonly accepted wisdom back in late 1980s and 
early 1990s (when several dozen low-delay speech coding papers were 
published) was 3X codec frame size, although some might have used 2X 
as the best-case scenario. Since then the CPU of PCs has increased in 
speed by more than 30X (in early 1990s a popular CPU was the 90 MHz 
Pentium) and the network speed also increased a lot, so I can believe 
you that PC-based softphones could potentially reach below 2X as you 
said, but IP phone processors are much slower than the CPUs of PCs, 
making it harder to get below 2X.  In any case, 1X was never the 
commonly accepted wisdom, since it is theoretically impossible.

> So I'd say the burden of proof is on you.

[Raymond]: Broadcom currently ships more than half of the world's IP 
phone chips, and Broadcom's IP phone group has also measured the one-
way codec-dependent delay carefully in the lab with carefully chosen 
signals, equipment, and methodology, so I don't see why Broadcom's 
measurement is any less valid than Cullen's, to the point that the 
burden of proof is on us.  I don't believe Broadcom's IP phone 
implementation is particularly bad in delay compared with others, 
either (otherwise we are unlikely to be the market leader).   In any 
case, I didn't say that I believe Cullen's results must be wrong.  I 
was just surprised at the discrepancy of the measurement results from 
these two different sources and would like to find out why and 
possibly reconcile the seemingly contradictory results.  That's why I 
asked Cullen to share more details about his measurements.

Raymond

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 18 May 2010 11:34:19 -0600

> measured the delay, and the measured delay value, etc.?  

Sure - it's really simple to set up. I use a signal generator that makes a tone burst. Typically I do something like 550 Hz tone that is a 200 ms long burst that occurs once a second. I play this into a speaker near the microphone on one phone and also put it into 1 channel of a scope.  I think put a microphone near the speaker of the other phone and put that on another channel of the scope and measure the mouth to ear delay. It's really easy to see the starts of the two bursts from the speaker and microphone and measure the delay within a few ms. I've done lots of clever things over the years using stats from software in the phones but this technique is easy and pretty fool proof on getting good results. The phones were plugged into Netgear 100Mbps hub as I sometimes look at the timing of the ethernet packets too. I set up phones for G.711, one frame per packet - I choose this as  it is easy to change the length of each frame but results are similar with other codecs. 

I was using two Cisco 7960 phones - no idea what version of the software and may have been a development build but it's pretty unlikely that current production software would have different results from what I tested. I set the phones for G.711 with 20 ms packets, measured delay, then set the phones for 30 ms packets and measured delay. For a given packet length, when I make multiple phone calls or reboot one of the phones, the measurements stay consistent. 

The resulting change in delay between the two experiments was much less than 30ms that a (3x 30 ms - 3 x 20 ms) would have predicted. I feel like a total tool not providing the details of the exact measurements but lots of measurements like this Cisco considers confidential and I'm just not up to arguing with folks about what can and can not be said publicly. I probably should have done a test with 10 ms packets too but I did not. Yes I realize how nuts it is to consider something that anyone can easily measure as confidential. If anyone really cares, I will go do the work to be able to provide the numbers. 

> some of their product lines.

Yah, again, Cisco considers chipsets confidential but if you poke around with google,  such as 

http://lmgtfy.com/?q=teardown+analysis+cisco+7960

Folks claim the 7960 uses a Broadcom chip - of course that looks like it is for the ethernet switch on the phone not much to do with software that impacts audio latency. 

> different sources.  Thanks.

Be glad to talk to whoever it is. I really don't know relevant this all is too figuring out what packets sizes we need to support.

> This does not match the measurements I have. And I certainly don't have 100+ year voip experience but I do have two of the #1 selling enterprise phones connected to an oscilloscope. Test with other…

Cullen Jennings
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Mon, 24 May 2010 18:38:47 -0700

Hi Cullen,

Sorry for the delay of my reply.  I was busy last week and could not respond
earlier.

Thank you for sharing the details of your delay measurements on Cisco 7960 IP
phones.  What you observed does NOT conflict with what I have been saying.
The reason is that the 20 ms and 30 ms you quoted are the "packet sizes", not
the "codec frame sizes".  Codec frame size and packet size have different
impacts on one-way delay.  The G.711 codec that you used is a sample-by-
sample codec. Theoretically its "codec frame size" is only one sample, or
0.125 ms, so the (3 x 30 ms - 3 x 20 ms) formula is not the right target
for comparison.

Furthermore, many telephones have G.711 encoder and decoder directly built
into the chip hardware of A/D and D/A, so they can directly digitize the
input audio signal into 8-bit G.711 codewords and directly playback 8-bit
G.711 codewords as the output audio signal; thus, there is essentially no
processing delay for G.711.  Even if the G.711 encoding/decoding is done in
software or firmware, the G.711 codec complexity is so low that it takes
almost no time to do G.711 processing.  The almost-zero processing delay can
contribute to the extra low delay of G.711-based VoIP systems.

There have been so many discussions about how the codec frame size and packet
size may affect the one-way delay, there has been confusion, and there have
been criticism that there wasn't any rigorous theoretical analysis, so I
thought I would spend some time to give a more rigorous delay analysis below
so we can hopefully settle such disputes. At the end of my analysis, you will
see how the lower bound and upper bound of the one-way delay depend on the
codec frame size AND the packet size under various conditions. Please read on
if you are interested; ignore if you are not; or you can quickly scroll down
to Equations (1) through (3), which are the main results of my delay
analysis, and read the last few paragraphs after Eq. (3).

Before I did the following delay analysis, I consulted extensively with three
Broadcom senior technical leads who have many years of extensive real-time
system architecture and design experiences in IP phones, VoIP gateways, and
video systems (such as cable/satellite set-top boxes), respectively.  What
they told me were consistent with each other and consistent with what I have
been saying.

Before I start the analysis, let me first discuss the multi-tasking, or Real-
Time Scheduling (RTS) delay, because it is a critical component of the total
one-way delay and needs to be clarified first.

In real-time audio or video systems, many tasks have definite completion
deadlines beyond which the real-time operation will be lost and there will be
audible or visible glitches. One way to handle a real-time task is by
interrupting the processor in the hope that the processor will put down
whatever it is doing and service the interrupt first.  If there is only one
real-time task and all other tasks in the system do not have real-time
requirements, then the interrupt will be serviced immediately and there is no
RTS delay.  However, this is rarely the case, since the system typically also
has other real-time tasks. (For example, an IP phone needs to handle the
encoding of the send-path signal, decoding of the receive-path signal, echo
canceller, side-tone, and other real-time tasks at the same time.) Then, the
interrupts generated by different real-time tasks need to be prioritized.
There can be only one highest-priority task.  Any of the other tasks will
have a lower priority and need to wait for its turn if it tries to interrupt
a higher-priority task. That wait time, plus the time it takes the processor
to complete the task, is the RTS delay of that task. The entire audio or
video stream will need to be buffered and delayed by at least the worst-case
wait time in order to have a smooth playback without any gaps or glitch.

If there are a large number of real-time tasks in the system, then a
prioritized interrupt-driven RTS system will become very complex and messy,
and the associated context switching for all the interrupts will reduce the
system efficiency.  Therefore, in IP phones, VoIP gateways, and
cable/satellite set-top boxes, usually a different kind of real-time
scheduling scheme is used, where each real-time task is allowed to run to
completion, but to simplify RT scheduling, all real-time tasks are requested
in a periodic manner, or with similar assumptions such as a minimum interval.
In many of these designs, all real time tasks on any one processor have the
same period (or "thread interval") for maximum real time efficiency.  In the
case of real-time voice communication systems, the most convenient and common
thread interval is the codec frame size.  Thus, the codec frame size
determines how much RTS delay the system has.  I have consulted my Broadcom
colleague Sandy MacInnis, a senior architect who specializes in video and
system design, and who is knowledgeable about real time scheduling.  He was
the chair of the MPEG Systems committee for MPEG-1 and MPEG-2 (i.e. MPEG
Transport, MPEG Programs streams, and MPEG-1 Systems).  I will quote him
below:

"For most efficient scheduling, all tasks should have the same period, and in
the general case, each task may be served any time from immediately after the
request to the last instant before the next request. So, for such efficient,
general and robust systems, the RTS (real time scheduling) latency is up to
one request period, which in this case is a frame duration. When the request
is serviced earlier, the data has to be buffered up because the end-end delay
needs to be constant. While someone might say that they think an RTS scheme
can service requests with consistently less latency than a frame time, I
would challenge them for a theoretical basis that shows they can do so
reliably. What happens when all the requests happen at the same time? That
can certainly happen, in general.  ...  An extremely standard basic
assumption of RTS, and in particular Rate Monotonic Scheduling (RMS), is that
for each task, the deadline equals the period. That means that from the time
a requester makes a request, the RTS system needs to ensure that the request
is completely serviced (finished, not just started) before the period from
that request to the next request expires. Other assumptions are possible, but
longer deadlines don't usually help much and they make the system more
complex, and shorter deadlines make scheduling harder.  If there is a set of
tasks with exactly the same period, i.e. synchronous, then it's possible to
schedule the shared resource to 100% of capacity while ensuring RT
performance. However, in the more typical case, the various tasks do not have
the same period, in which case in general the maximum utilization of the
shared resource that can be scheduled for real time tasks is significantly
less than 100%. Whether the system is real-time schedulable or not can be
determined in various ways, including critical instant analysis.  In either
case, in general the latency of any given request can be anywhere from zero
plus processing time, to exactly the period = deadline."

For a PC with a very powerful processor and a very light real-time load, it
may be reasonable to expect the processor to perform the encoding and
decoding tasks very shortly after they are requested, with the requests being
driven by interrupts, and the processing time of each task may be very short
relative to the interval between requests. The resulting RTS delay may be as
low as a few percent of the frame interval.  This is possible because a
typical PC has much higher processing power than is required by a speech
coder.

The same is not true for VoIP gateways or IP phones, where the processor is
heavily loaded with real-time tasks and is often just barely fast enough to
handle the designated number of voice channels (many for gateways and one for
IP phones).  For example, rather than having a 2 to 3 GHz processor as in a
PC, the processor used to do speech coding in a low-end IP phone may only
have a clock rate of slightly more than 100 MHz.  In this case, it is
reasonable to expect that the time required to service each request,
including processing time, may be as much as the full frame interval.

OK, now that the RTS delay has been discussed, let me proceed with my delay
analysis.  I will break down the delay into many components, with each
component occurring after the components listed earlier.  Let the codec frame
size be F ms and the packet size be P ms.  Let each packet contain N codec
frames, so P = N*F.  For simplicity, we will not consider the codec look-
ahead L ms and codec filtering delay R ms in this analysis and will just add
them at the end because we know their multiplier is 1X.

The one-way mouth-to-ear delay includes the following codec-dependent delay
components:

(1) Encoder buffering delay: d1 = a1*F, where a1 = 1.
This is the time it takes to buffer all input samples of a codec frame.

(2) Encoder RTS delay: d2 = a2*F, where 0 < a2 <= 1.
This includes the encoder processing delay; see the discussion above.

(3) Packetization delay: d3 = a3*F, where a3 = (N-1).
This is the amount of time the first frame in the packet need to wait until
the last frame of encoded bits in the packet is ready.

(4) Packet transmission delay: d4 = a4*F, where 0 < a4 <= N.
This is the time it takes to ship all bits in the packets out of the
transmitter; this can also be considered the decoder bit buffering delay,
since it is the time the decoder needs to wait to get all bits in the packet.
If the speed of the communication channel is very high, then d4 can be a very
small fraction of the packet size P = N*F ms, but it will not be zero.  If
the channel speed is exactly the same as the bit-rate of the packet
(including the packet header), then d4 = P = N*F ms.  Even for the case of
high-speed channel, if we view the bit transmission task as a real-time
scheduling problem for the micro-controller (which may run at a different
thread rate than the DSP), then the scheduling wait time plus the processing
time (i.e. the time to actually transmit bits) may still take up to one
thread interval, which is P = N*F ms in this case.

(5) Decoder RTS delay: d5 = a5*F, where 0 < a5 <= 1.
This includes the decoder processing delay; see the discussion above.

There may be other delay components that may depend on the codec frame size.
For example, in gateways where a few layers of processors are used, each
processor may have its own real-time scheduling delays for all tasks that it
handles.  However, at least the delay components listed above are the major
ones that are commonly encountered.  If we omit the other possible codec-
dependent components for the moment but add back the codec look-ahead L and
codec filtering delay R (if any), the total codec-dependent one-way delay is
then

D = d1 + d2 +... + d5 + L + R = {1 + (0,1] + (N-1) + (0,N] + (0,1]}*F + L + R

Hence, the one-way delay D has a possible range of

N*F + L + R < D <= (2*N + 2)*F + L + R, or

P + L + R < D <= 2*P + 2*F + L + R             Eq. (1)

For heavily loaded real-time systems such as VoIP gateways or IP phones, if
we assume the worst case of one full frame of encoder RTS delay and decoder
RTS delay, then a2 = 1 and a5 = 1, and we get a tighter range for the one-way
delay:

P + 2*F + L + R < D <= 2*P + 2*F + L + R       Eq. (2)

In the special case of N = 1 (each packet contains only one codec frame),
then we get

3*F + L + R < D <= 4*F + L + R                 Eq. (3)

The delay lower bounds in Eq. (1) through Eq. (3) above (under their
individual assumptions) are consistent with what I have been saying.
If the other omitted codec-dependent delay components are significant, or
if the system implementers have not been careful about minimizing the delay,
then the delay upper bounds can be even higher than what are shown in Eq. (1) through Eq. (3).

In your Cisco 7960 IP phone delay measurements, P = 20 ms or 30 ms, L = 0, R
= 0, and theoretically F = 0.125 ms.  If you look at Eq. (2) above, then it
is clear that you won't see 3 times the packet size difference as the delay
difference.  However, here the codec frame size is 0.125 ms, not 20 or 30 ms,
so this result doesn't conflict with what I have been saying (i.e. 3X codec
frame size).

Of course, in reality it is unlikely that an IP phone will use 0.125 ms as
the thread interval.  A more likely thread interval is P.  Then, my delay
analysis above does not apply directly.  However, it is not difficult to
follow the same logic and procedure to see what will happen in this case.  If
G.711 encoding and decoding is built right into the A/D and D/A, then the 8-
bit G.711 codewords directly arrives at the input buffer or leave the output
buffer and the RTS system does not need to schedule G.711 encoding and
decoding tasks, so d2 = d5 = 0. Also, in this case d1 = P, d3 = 0, and 0 < d4
<= P.  Thus, the total one-way delay is P < D <= 2*P.

Even if the G.711 encoding and decoding operations are done in
software/firmware, the G.711 complexity is so low that it takes the processor
almost no time to do encoding and decoding.  In this case, the IP phone is
closer to the case of a PC that has much more processing power than is
required for speech coding, and if the Cisco engineers did a good job of
optimizing RTS to minimize d2 and d5, then d2 and d5 would be closer to 0
than to P.  Then, the total one-way codec-dependent delay would be closer to
P than to 3*P.  This is probably what you have observed.

Best Regards,

Raymond

-----Original Message-----
From: Cullen Jennings [mailto:[email-masked]]
Sent: Tuesday, May 18, 2010 10:34 AM
To: Raymond (Juin-Hwey) Chen
Cc: Koen Vos; [email-masked]
Subject: Re: [codec] #16: Multicast?

Cullen Jennings

---

From: Koen Vos <[email-masked]>
Date: Wed, 26 May 2010 15:13:26 -0700

Hi Raymond,

Thanks for the detailed explanation, this clarifies your earlier  
statements about the 3x multiplier.

The essence, if I understand you correctly, is that there still exist  
low-end platforms with barely enough processing power to run a VoIP  
call.  If such platforms use a naive FIFO scheduler, they'll create up  
to one frame of processing delay for encoder and decoder each, on top  
of the frame of buffering delay.

The good news is that Moore's law will continue to drive down the  
fraction of platforms with such processing delay problems.

I'm a bit surprised by your analysis of "packet transmission delay",  
as it has little bearing on our multiplier (ie the change in delay as  
a function of frame size). See old posts.

best,
koen.

Quoting "Raymond (Juin-Hwey) Chen":

> Sure - it's really simple to set up. I use a signal generator that  
> makes a tone burst. Typically I do something like 550 Hz tone that  
> is a 200 ms long burst that occurs once a second. I play this into a  
> speaker near the microphone on one phone and also put it into 1  
> channel of a scope.  I think put a microphone near the speaker of  
> the other phone and put that on another channel of the scope and  
> measure the mouth to ear delay. It's really easy to see the starts  
> of the two bursts from the speaker and microphone and measure the  
> delay within a few ms. I've done lots of clever things over the  
> years using stats from software in the phones but this technique is  
> easy and pretty fool proof on getting good results. The phones were  
> plugged into Netgear 100Mbps hub as I sometimes look at the timing  
> of the ethernet packets too. I set up phones for G.711, one frame  
> per packet - I choose this as  it is easy to change the length of  
> each frame but results are similar with other codecs.
>
> I was using two Cisco 7960 phones - no idea what version of the  
> software and may have been a development build but it's pretty  
> unlikely that current production software would have different  
> results from what I tested. I set the phones for G.711 with 20 ms  
> packets, measured delay, then set the phones for 30 ms packets and  
> measured delay. For a given packet length, when I make multiple  
> phone calls or reboot one of the phones, the measurements stay  
> consistent.
>
> The resulting change in delay between the two experiments was much  
> less than 30ms that a (3x 30 ms - 3 x 20 ms) would have predicted. I  
> feel like a total tool not providing the details of the exact  
> measurements but lots of measurements like this Cisco considers  
> confidential and I'm just not up to arguing with folks about what  
> can and can not be said publicly. I probably should have done a test  
> with 10 ms packets too but I did not. Yes I realize how nuts it is  
> to consider something that anyone can easily measure as  
> confidential. If anyone really cares, I will go do the work to be  
> able to provide the numbers.
>
>
>>
> Folks claim the 7960 uses a Broadcom chip - of course that looks  
> like it is for the ethernet switch on the phone not much to do with  
> software that impacts audio latency.
>
>>
> Be glad to talk to whoever it is. I really don't know relevant this  
> all is too figuring out what packets sizes we need to support.
>
>>
>> Best Regards,
>>
>> Raymond
>>
>> This does not match the measurements I have. And I certainly don't  
>> have 100+ year voip experience but I do have two of the #1 selling  
>> enterprise phones connected to an oscilloscope. Test with other  
>> phones suggest most the major vendors of IP hard phones have fairly  
>> comparable performance when it comes to delay.
>>
>>
>> Cullen
>>
>
>
> Cullen Jennings

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Wed, 26 May 2010 18:23:38 -0700

Hi Koen,

In-line below...

> of the frame of buffering delay.

[Raymond]: It doesn't have to be low-end platforms.  I wouldn't consider 
high-density VoIP gateways "low-end".  What matters is whether the 
processor is heavily loaded (i.e. busy at a high percentage of time) 
with real-time tasks (and thus is just fast enough). I think this is 
true for typical implementations of IP phones and VoIP gateways.

I also wouldn't use the term "a naïve FIFO scheduler" to describe the 
"run to completion" real-time scheduler that I talked about in my last 
email, because that term seems to imply that it is a very simple-minded 
and inferior approach used by an inexperienced person who doesn't know 
anything better.  My understanding from talking to the three senior 
technical leads of Broadcom is that the reality is when you have many 
real-time tasks that you need to handle concurrently, using a 
prioritized interrupt-driven scheduler is just way too complex and 
messy, and it doesn't even guarantee that you will get a lower delay if 
you do go through the trouble.  In contrast, the kind of "run to 
completion" real-time scheduler that I talked about is a more elegant 
solution as it simplifies the scheduling problem substantially and also 
allows you to have more efficient utilization of the processor. 

Other than these two points, your understanding of my main point is 
correct.

> fraction of platforms with such processing delay problems.

[Raymond]: This may be true for PC but probably not true in general. 
PC is a general-purpose computing device that has to handle numerous 
possible tasks, and a voice phone call takes only a very small fraction 
of the worst-case computational power requirement of a PC.  In contrast, 
for special-purpose dedicated hardware devices such as IP phones or 
VoIP gateways, it would make no sense to use a processor that is many 
times faster than the worst-case computational power requirement.  For 
the sake of cost and power efficiency, the designers of such special-
purpose devices will want to use a processor that's just slightly faster
than required, because then they can use the cheapest and/or lowest 
power-consuming processor that's fast enough to get the job done.  
If they choose to use a processor much faster than is required, then 
competitors using processors just fast enough can have lower costs 
and power consumption and can take market share away from them.

A case in point: after its first appearance several decades ago, 8-bit 
microprocessors are still widely used in many devices today despite the 
several orders of magnitude of speed improvement provided by Moore's 
Law, because those devices just don't need anything faster, so using 
anything faster would be a waste of money and power consumption.

My point is that we should not expect that future IP phones or gateways 
will operate at a very low percentage point of the processor load just 
because Moore's Law can improve processor speed over time. Therefore, 
don't expect the 3X multiplier for codec frame size to go down much 
below where they are now.

In fact, if in addition to a VoIP call, a PC is heavily loaded with a 
lot of other concurrent tasks, many of which may be real-time tasks 
(e.g. video, playing/burning CD/DVD, networking, etc.), then it will be 
difficult for the PC to have small encoding and decoding RTS delays (d2 
and d5 in my delay analysis).  In this case, the codec frame size 
multiplier will be closer to 3X than to 1X, unless you are willing to 
let the voice stream occasionally run out of real time and produce an 
audible glitch (which is not acceptable from the voice quality 
perspective).  If you agree with this and agree that a PC sometimes 
does get very heavily loaded, then if you don't want the voice stream
to run out of real time, the worst-case codec-dependent delay for 
PC can still be around 3X the codec frame size.

> a function of frame size). See old posts.

[Raymond]: I am not sure I understand what you are saying.  You probably 
misunderstood the goal of my analysis. I mentioned in my last email that 
my delay analysis aimed to derive the lower and upper bounds of the 
codec-dependent one-way delay as functions of both the codec frame size 
AND the packet size.  That "packet transmission delay" does depend on 
the packet size, so it should be included.  Also, including it doesn't 
increase the lower bound of the delay (and the codec frame size 
multiplier there); it only affects the upper bound.

Or, are you saying the "packet transmission delay" depends on the packet 
size, not the codec frame size, and therefore is not codec-dependent? 
Well, we know the packet size should be a positive integer multiple of 
the codec frame size.  Once the codec frame size is determined, there 
are only limited choices of packet sizes you can use, so in this sense 
the packet size does depend on the codec frame size.  Therefore, the 
"packet transmission delay" indirectly depends on the choice of the 
codec.

Best Regards,

Raymond

---

From: Koen Vos <[email-masked]>
Date: Wed, 26 May 2010 21:42:55 -0700

Quoting "Raymond (Juin-Hwey) Chen":
> because Moore's Law can improve processor speed over time.

In other words, future manufacturers won't spend a few dimes on  
reducing delay, even though today they're happy to add several dollars  
to the price just to enable wideband?  That's a statement about the  
relative importance of delay.

For the discussion about transmission delay vs. frame size, see e.g.
http://www.ietf.org/mail-archive/web/codec/current/msg01477.html

koen.

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 27 May 2010 12:45:07 +0200

Hello Koen and Raymond,

yesterdays, I had a brief look on ITU-T G.114
http://www1.cs.columbia.edu/~andreaf/new/documents/other/T-REC-G.114-200305.pdf
It might help in your discussion...

Regarding Moore-law and so: We shall keep in mind that cannot continue forever. Already today, due to Quantum tunneling and
subthreshold leakage current very small semiconductor structures consume increasing amounts of energy. Thus, it might not always be
advisable to use the latest technology if power consumption shall be low. 

It is know that "CMOS circuits dissipate power by charging the various load capacitances (mostly gate and wire capacitance, but also
drain and some source capacitances) whenever they are switched. The charge moved is the capacitance multiplied by the voltage
change. Multiply by the switching frequency on the load capacitances to get the current used, and multiply by voltage again to get
the characteristic switching power dissipated by a CMOS device: P = CV²f".

C is the capacity (depending on the size of the structure)
V is Voltage
f is the powering frequency
P is the power

Thus, the power does not decrease if the calculation (e.g., the encoding and decoding) is done faster or slower. 

In order to save power in mobile device, Dynamic frequency scaling and Dynamic voltage scaling change the frequency and/or the
voltage to save power.  If power consumption needs to be reduced, the device reduces voltage and frequency and thus the calculation
takes longer. Thus, even if the CPU can do the encoding/decoding at full speed and in a fraction of the frame duration, it is not
always advisable to do it like that. Instead, if energy supply is limited, then calculations shall be slowed down.

My argumentations above support the position of Raymond that device with low processing power will be used and that this increases
to the transmission delay. However, I am not an expert in system or chip design and thus, I might have missed a few details or
tradeoffs. 

Nevertheless, in the end, my position is simple. The lesser the computational complexity, the smaller the battery. This statement
will remain true for future semiconductor technologies, too. Thus, a low complexity mode and low delay mode is advisable for small,
portable battery-powered devices such as wireless headsets, hearing aids, or wireless sensor nodes. Or other kind of devices, which
have problems with head dissipation.

With best regards,

 Christian Hoene

>Sent: Thursday, May 27, 2010 6:43 AM

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 27 May 2010 07:47:51 -0400

Christian, please leave semiconductor physics out of this. FYI, see below:

> advisable to use the latest technology if power consumption shall be low.

Although power no longer scales linearly with the area of the 
transistors, it still decreases when one uses a smaller technology (with 
the exception being the huge leakage from 65 nm).

> ...

Euh?? what about f? That's the part that takes into account faster vs 
slower. Even if you don't change the clock, the amount of calculations 
you have still has an impact on power. Once you've paid for the idle 
(static) power, the dynamic power is pretty much proportional to how 
fast you do your calculation. That being said, there's also about 100 
other factors involved in the power consumption, including the 
technology used (low/standard/high threshold gates), tons of 
architectural decisions (cache, branch prediction, parallelism, ...), 
power-saving features, and so on.

> always advisable to do it like that. Instead, if energy supply is limited, then calculations shall be slowed down.

Many people have found that for a given CPU, it's very often more 
efficient to do the calculation at full-speed and then enter the 
low-power state as quickly as possible -- as opposed to making the 
calculation slower and being in low-power state for less time. The term 
for that is "race to idle".

> to the transmission delay. However, I am not an expert in system or chip design and thus, I might have missed a few details or

No need to go into chip design to say that "lower complexity is better". 
The key is deciding what's acceptable and what's the tradeoff. If 
complexity was the only issue, we'd all be using G.711. No matter what 
the final complexity will be, there will be chips that aren't powerful 
to execute it and there will be chips that are so powerful that they 
don't wouldn't notice if the complexity doubled.

Cheers,

	Jean-Marc

---

From: Herve Taddei <[email-masked]>
Date: Thu, 27 May 2010 13:53:36 +0200

Hi Christian,

> Nevertheless, in the end, my position is simple. The lesser the
computational
> complexity, the smaller the battery. This statement
> will remain true for future semiconductor technologies, too. Thus, a low
complexity
> mode and low delay mode is advisable for small,
> portable battery-powered devices such as wireless headsets, hearing aids,
or
> wireless sensor nodes. Or other kind of devices, which
Are all those devices being of interest for the codec under development? For
example is it your plan to make use of the IETF codec in hearing aids? In
that case, perhaps requirements for hearing aids should be discussed and
added to the requirements. But I am not sure it is really relevant for an
audio codec designed specifically for use over the Internet.

Best regards

Herve Taddei

> Sent: Thursday, May 27, 2010 12:45 PM
> To: 'Koen Vos'; 'Raymond (Juin-Hwey) Chen'
> Cc: [email-masked]
> Regarding Moore-law and so: We shall keep in mind that cannot continue
forever.
> Already today, due to Quantum tunneling and
> subthreshold leakage current very small semiconductor structures consume
> increasing amounts of energy. Thus, it might not always be
> It is know that "CMOS circuits dissipate power by charging the various
load
> capacitances (mostly gate and wire capacitance, but also
> drain and some source capacitances) whenever they are switched. The charge
> moved is the capacitance multiplied by the voltage
> change. Multiply by the switching frequency on the load capacitances to
get the
> current used, and multiply by voltage again to get
> Thus, the power does not decrease if the calculation (e.g., the encoding
and
> decoding) is done faster or slower.
> 
> In order to save power in mobile device, Dynamic frequency scaling and
Dynamic
> voltage scaling change the frequency and/or the
> voltage to save power.  If power consumption needs to be reduced, the
device
> reduces voltage and frequency and thus the calculation
> takes longer. Thus, even if the CPU can do the encoding/decoding at full
speed
> and in a fraction of the frame duration, it is not
> always advisable to do it like that. Instead, if energy supply is limited,
then
> calculations shall be slowed down.
> 
> My argumentations above support the position of Raymond that device with
low
> processing power will be used and that this increases
> to the transmission delay. However, I am not an expert in system or chip
design
> and thus, I might have missed a few details or
> tradeoffs.
> 
> Nevertheless, in the end, my position is simple. The lesser the
computational
> complexity, the smaller the battery. This statement
> will remain true for future semiconductor technologies, too. Thus, a low
complexity
> mode and low delay mode is advisable for small,
> portable battery-powered devices such as wireless headsets, hearing aids,
or
> wireless sensor nodes. Or other kind of devices, which

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 27 May 2010 14:49:42 +0200

Guys,

all I want to do is to find an end to this discussion. Sometimes it helps to come up with some provocative statements.
The example of hearing aids was mentioned only as an example of a device that might include a phone in future. If you will, I forgot
to mention the tooth-phone http://www.wired.com/science/discoveries/news/2002/06/53302 

Anyhow, shall we consider that

a) devices have plenty of power to do the coding (<2% needed for the codec) 
b) devices are already fully loaded and need all time for encoding and decoding (100% needed for codec and signal processing)
xy) between X% and Y%.

What is the minimal complexity of the codec? Enough to run at 100% load on a 
a) 10 MIPS DSP
b) 50 MIPS DSP
c) 8000 MIPS DSP
d) 60 MHz Pentium
e) 1 GHz Pentium

How to measure the complexity?
a) with gettimeofday on a reference PC setup
b) with ITU-T G.191 library
c) With a reference virtual machine 

Decide and continue...

Christian

>From: Herve Taddei [mailto:[email-masked]]
>Sent: Thursday, May 27, 2010 1:54 PM
>To: 'Christian Hoene'; 'Koen Vos'; 'Raymond (Juin-Hwey) Chen'
>Cc: [email-masked]

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 27 May 2010 09:53:59 -0400

Hi Herve,

> audio codec designed specifically for use over the Internet.

I agree here. We are not chartered to address hearing aids and every device 
on the planet. If what we do turns out to be useful for non-Internet 
applications, that's always a nice bonus. If not, then tough.

	Jean-Marc

---

From: "Sandy (Alexander) MacInnis" <[email-masked]>
Date: Thu, 27 May 2010 07:54:45 -0700

Everyone,

Sorry for stepping in here... full disclosure: I'm not a speech coding expert, and I work at Broadcom, where Raymond works.

I too would like to end this discussion; it seems to have diverged from a discussion of the requirements for the CODEC algorithm to have a mode with low algorithmic delay, which AFAIK is already agreed anyway, to some rather tangential discussions related to, but not really addressing, real time scheduling of the algorithm on a processor.

The point from Raymond that is the head of this particular discussion trail is RTS, i.e. real time scheduling. I know his note about that is long; it might be worth reading it again.

It's not a fair assumption that 100% of a shared resource - in this instance, a processor - can be spent performing real-time-scheduled tasks. If there is a set of RT (real time) tasks that have different periods, and periods = deadlines, all being scheduled on the same processor, the best you can do is less than 100%. How close you can get depends on the details; it might be e.g. 68%, or it could be significantly less; there's a lot of literature on this. If the system is optimally designed for the purposes of RTS, i.e. all other tasks are treated as non-real time and have lower priority than all real time tasks, there are no priority inversions, task switching is very efficient, etc. the RTS performance can come close to theory, but if any of these assumptions are not true, it be significantly worse.

If the total RT demands are only a very small fraction of the total shared resource, i.e. processor cycles, it tends to be easier to perform the scheduling and ensure that it works correctly. Such a scenario may be more important than RTS indicates if the system is not well designed for real time operation, i.e. a PC. And, such systems draw MUCH more power than well-designed embedded products. Conversely, low power and modest clock rates are good design principles for embedded products, if those that are wall (mains) powered. E.g. someone noted leakage power at 65nm - have you looked at 40nm? It just keeps getting worse. Designing for slower max clock rate saves substantial power.

There are good reasons why a common convention of real time scheduling is the assumption that period = deadline. As Raymond noted, other design assumptions are possible, but they have their own problems.

Note also, as Raymond pointed out, that RTS also applies to intermediate points in the end-end system, such as gateways. Such a device may have very powerful processors, and if so, it should be for the specific purpose performing a large number of RT tasks, loading the processors as much as can be guaranteed.

I would hope that this committee is not planning to be in the position of dictating that all implementations of the algorithm require a processor that is so fast that the system can guarantee service that latency is much less than the period of an audio frame. And if not, then a reasonable assumption is that, in general, the deadline of service latency does equal the period of an audio frame. That assumption is part of one of upper-limit calculations from Raymond.

--Sandy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christian Hoene
Sent: Thursday, May 27, 2010 5:50 AM
To: 'Herve Taddei'; 'Koen Vos'; Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: Re: [codec] #16: Multicast?

Guys,

Decide and continue...

Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 27 May 2010 17:48:21 +0200

So, we have consensus on 
1) low delay mode
2) low complexity mode (whatever this means)
3) technical understanding on how latency sums up on different platforms

The remaining discussion seems to be on which kind of platforms to support:
a) fast PC and softphone 
b) embedded, slow CPU on hardphones (+conferences server) 

A bit it is also about how to make money with phone calls in future:
a) Selling services in addition to offering calls via softphones
b) Selling devices that help to conduct calls

I know that there it has a long tradition in standardization to fortify the own market position as early as possible. Thus, it is
well understandable that a) and b) are competing. However, wouldn't it make sense if we support both a) and b) and let future market
forces decide? 

With best regards,

 Christian

>From: Sandy (Alexander) MacInnis [mailto:[email-masked]]
>Sent: Thursday, May 27, 2010 4:55 PM
>To: Christian Hoene; 'Herve Taddei'; 'Koen Vos'; Raymond (Juin-Hwey) Chen
>Cc: [email-masked]
>I too would like to end this discussion; it seems to have diverged from a discussion of the
>requirements for the CODEC algorithm to have a mode with low algorithmic delay, which AFAIK is already
>agreed anyway, to some rather tangential discussions related to, but not really addressing, real time
>scheduling of the algorithm on a processor.
>
>The point from Raymond that is the head of this particular discussion trail is RTS, i.e. real time
>scheduling. I know his note about that is long; it might be worth reading it again.
>
>It's not a fair assumption that 100% of a shared resource - in this instance, a processor - can be
>spent performing real-time-scheduled tasks. If there is a set of RT (real time) tasks that have
>different periods, and periods = deadlines, all being scheduled on the same processor, the best you
>can do is less than 100%. How close you can get depends on the details; it might be e.g. 68%, or it
>could be significantly less; there's a lot of literature on this. If the system is optimally designed
>for the purposes of RTS, i.e. all other tasks are treated as non-real time and have lower priority
>than all real time tasks, there are no priority inversions, task switching is very efficient, etc. the
>RTS performance can come close to theory, but if any of these assumptions are not true, it be
>significantly worse.
>
>If the total RT demands are only a very small fraction of the total shared resource, i.e. processor
>cycles, it tends to be easier to perform the scheduling and ensure that it works correctly. Such a
>scenario may be more important than RTS indicates if the system is not well designed for real time
>operation, i.e. a PC. And, such systems draw MUCH more power than well-designed embedded products.
>Conversely, low power and modest clock rates are good design principles for embedded products, if
>those that are wall (mains) powered. E.g. someone noted leakage power at 65nm - have you looked at
>40nm? It just keeps getting worse. Designing for slower max clock rate saves substantial power.
>
>There are good reasons why a common convention of real time scheduling is the assumption that period =
>deadline. As Raymond noted, other design assumptions are possible, but they have their own problems.
>
>Note also, as Raymond pointed out, that RTS also applies to intermediate points in the end-end system,
>such as gateways. Such a device may have very powerful processors, and if so, it should be for the
>specific purpose performing a large number of RT tasks, loading the processors as much as can be
>guaranteed.
>
>I would hope that this committee is not planning to be in the position of dictating that all
>implementations of the algorithm require a processor that is so fast that the system can guarantee
>service that latency is much less than the period of an audio frame. And if not, then a reasonable
>assumption is that, in general, the deadline of service latency does equal the period of an audio
>frame. That assumption is part of one of upper-limit calculations from Raymond.
>
>--Sandy
>
>

---

From: "Raymond (Juin-Hwey) Chen" <[email-masked]>
Date: Thu, 27 May 2010 12:27:57 -0700

Hi Koen,

I too don't want to see this discussion drags on, but some of your comments 
seem misleading to me, so I would like to respond with some quick comments. 

Wideband is a new feature in some devices and is a check box that a product 
manager needs to check off to remain competitive.  That doesn't mean 
wideband is more important than existing features in a device.  Also, I 
am not sure the cost difference is a few dimes versus several dollars. In 
some devices the extra cost of adding wideband is minimal. Furthermore, it 
is not only a cost issue but also a power consumption issue.  No one in his 
or her right mind will use a processor that's 5X to 10X faster than 
necessary just in order to reduce the encoder and decoder RTS delays to a 
small fraction of the codec frame size; this is just the way it is and has 
nothing to do with the relative importance of delay or anything else.  

You were presenting it as if this were a reasonable choice that device 
designers could easily make but chose not to make, but that's just not 
true.  It has always been the case that the designers will use processors 
just fast enough for the job, perhaps with a little margin for the 
unexpected, but not 5X or 10X.  Given this, the bottom line is that ~ 3X 
codec frame size is the "norm" or a "necessary result" for special-purpose 
hardware devices rather than by a design choice, and you are just lucky to 
get < 2X in PC-based VoIP calls because PCs were not designed for voice 
calls but for other much more computationally demanding tasks.  (Even there 
you can't guarantee that PCs will always give you a multiplier of < 2X.  
What if the PC is heavily loaded with other tasks?  Then you are more likely
to get 3X if you don't want your voice stream to run out of real time.)

Best Regards,

Raymond

-----Original Message-----
From: Koen Vos [mailto:[email-masked]] 
Sent: Wednesday, May 26, 2010 9:43 PM
To: Raymond (Juin-Hwey) Chen
Cc: [email-masked]
Subject: RE: [codec] #16: Multicast?

koen.

---

From: Roman Shpount <[email-masked]>
Date: Thu, 27 May 2010 15:48:27 -0400

Raymond,

What we heard so far was a lot of unsubstantiated claims with some
semi-theoretical arguments attributed to unknown exports supporting them.
First of all, this is all irrelevant. Second, unless someone takes a
reasonable set of different hardware, including high density gateways, soft
and hard phones, and actually measures the delay for different codecs and
packet sizes we will not know what the answer is. So. please. lets can this
discussion for now.
_____________
Roman Shpount

---

From: Herve Taddei <[email-masked]>
Date: Fri, 28 May 2010 11:55:40 +0200

Dear Christian,

Some questions for clarification
> 1) low delay mode
What is the consensus then? Both low delay and longer delay modes will be
supported by the codec? What is the value of the delay of the so called low
delay mode (10ms, 5 ms, lower, higher)? 

> 2) low complexity mode (whatever this means)
It is strange to me to say we have consensus on low delay mode if we are not
sure on what it means. Or perhaps you mean it is still to be discussed?
What do you call low complexity? Which value? I can say my codec is 200
WMOPS and the low complexity mode is 100 WMOPS. Would that fit your
requirements?

> 3) technical understanding on how latency sums up on different platforms
This I must have missed, I saw many discussions, I have not seen large
agreements. The last email I read was rather in the spirit of: please stop
discussing we don’t believe what you wrote. The discussions were very
interesting but obviously coming from two different worlds. 

> b) embedded, slow CPU on hardphones (+conferences server)
Do you think you need to distinguish between slow PC and fast PC? What are
the complexity values you have in mind for a) and b)?

> b) Selling devices that help to conduct calls
Did that ever come in the discussion?

> I know that there it has a long tradition in standardization to fortify
the own market
> position as early as possible. Thus, it is
Could you clarify that?

> well understandable that a) and b) are competing. However, wouldn't it
make sense
How are a) and b) competing? You can sell devices (e.g. iPhone) and make
money on additional services (e.g. Applications).  

> if we support both a) and b) and let future market
Could you prevent to a) and/or b) from happening with this new codec? I am
not sure on where you want to go with this discussion. Is that needed?

Best regards

Hervé

> >Sorry for stepping in here... full disclosure: I'm not a speech coding
expert, and I
> work at Broadcom,
> >where Raymond works.
> >requirements for the CODEC algorithm to have a mode with low algorithmic
delay,
> which AFAIK is already
> >agreed anyway, to some rather tangential discussions related to, but not
really
> addressing, real time
> >The point from Raymond that is the head of this particular discussion
trail is RTS,
> i.e. real time
> >It's not a fair assumption that 100% of a shared resource - in this
instance, a
> processor - can be
> >spent performing real-time-scheduled tasks. If there is a set of RT (real
time) tasks
> that have
> >different periods, and periods = deadlines, all being scheduled on the
same
> processor, the best you
> >can do is less than 100%. How close you can get depends on the details;
it might
> be e.g. 68%, or it
> >could be significantly less; there's a lot of literature on this. If the
system is
> optimally designed
> >for the purposes of RTS, i.e. all other tasks are treated as non-real
time and have
> lower priority
> >than all real time tasks, there are no priority inversions, task
switching is very
> efficient, etc. the
> >RTS performance can come close to theory, but if any of these assumptions
are
> not true, it be
> >significantly worse.
> >
> >If the total RT demands are only a very small fraction of the total
shared resource,
> i.e. processor
> >cycles, it tends to be easier to perform the scheduling and ensure that
it works
> correctly. Such a
> >scenario may be more important than RTS indicates if the system is not
well
> designed for real time
> >Conversely, low power and modest clock rates are good design principles
for
> embedded products, if
> >those that are wall (mains) powered. E.g. someone noted leakage power at
65nm
> - have you looked at
> >40nm? It just keeps getting worse. Designing for slower max clock rate
saves
> substantial power.
> >
> >There are good reasons why a common convention of real time scheduling is
the
> assumption that period =
> >deadline. As Raymond noted, other design assumptions are possible, but
they
> have their own problems.
> >
> >Note also, as Raymond pointed out, that RTS also applies to intermediate
points
> in the end-end system,
> >such as gateways. Such a device may have very powerful processors, and if
so, it
> should be for the
> >specific purpose performing a large number of RT tasks, loading the
processors
> as much as can be
> >guaranteed.
> >
> >I would hope that this committee is not planning to be in the position of
dictating
> that all
> >implementations of the algorithm require a processor that is so fast that
the
> system can guarantee
> >service that latency is much less than the period of an audio frame. And
if not,
> then a reasonable
> >assumption is that, in general, the deadline of service latency does
equal the
> period of an audio
> >all I want to do is to find an end to this discussion. Sometimes it helps
to come up
> with some
> >provocative statements.
> >The example of hearing aids was mentioned only as an example of a device
that
> might include a phone in
> >future. If you will, I forgot
> >to mention the tooth-phone
> http://www.wired.com/science/discoveries/news/2002/06/53302
> >
> >b) devices are already fully loaded and need all time for encoding and
decoding
> (100% needed for codec
> >and signal processing)
> >>> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> >>> have problems with head dissipation.
Behalf
> Of
> >>> Koen Vos
Therefore,