Subject: [codec] proposed charter

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 08 Oct 2009 12:33:00 -0600

In the Codec BoF request that I submitted to [email-masked], I provided
an updated charter proposal. It seems that I neglected to send that to
the list. I do so now, with a request for feedback so that we can work
toward consensus regarding the problem statement, objectives, scope,
deliverables, and milestones of the proposed Codec WG.

/psa

***

PROPOSED CHARTER

Problem Statement

There are no standardized, high-quality audio codecs that are optimized
for use in interactive Internet applications and that can be widely
implemented and easily distributed among  application developers,
service operators, and end users.  This gap in the Internet protocol
stack has hindered protocol designers from being able to specify
mandatory-to-implement codecs in their protocols for the sake of
interoperability, developers from building innovative, interactive
applications, service operators from deploying affordable, high-quality
services, and end users from enjoying an optimal user experience.
Because the Internet Standards Process defines an open, transparent
process for technology standardization, a wide range of codec experts
have committed to actively working on such codecs within the IETF.  The
Codec WG provides a structured forum for these efforts.

Objectives

The goal of this group is to develop one or more high-quality,
widely-distributable audio codecs that is optimized for use over the
Internet. Core considerations include the following:

1. Use in interactive applications (examples include but are not limited
to point-to-point voice calls, multi-party voice conferencing,
telepresence, teleoperation, in-game voice chat, and live music performance)

2. Addressing the real transport conditions of the Internet, especially
packet loss

3. Ensuring interoperability with the Real-time Transport Protocol
(RTP), including secure transport via SRTP

4. Ensuring interoperability with Internet signalling technologies such
as Session Initiation Protocol (SIP), Session Description Protocol
(SDP), and Extensible Messaging and Presence Protocol (XMPP)

Codecs optimized for very low bit rates and codecs for non-interactive
audio are out of scope.

Although the codec(s) produced by the group might be used as
mandatory-to-implement technologies by designers of particular Internet
protocols, it is explicitly not a goal of the group to produce a codec
that will be mandated for use across the entire IETF or Internet community.

The group might produce only one codec or it might produce multiple
codecs; however, to reduce confusion in the marketplace it shall
endeavor not to function as a "codec factory" that produces a large
number of codecs.

Work processes and technical requirements for achieving the foregoing
objectives are outlined in draft-valin-codec-guidelines and
draft-valin-codec-requirements respectively; these will be further
refined by the group in accordance with the usual IETF procedures for
arriving at rough consensus.

Intellectual property rights (IPR) issues will be handled in accordance
with BCP 78 and BCP 79. Many existing codecs with the technical
attributes listed above are encumbered with licensing fees and other IPR
claims that make royalty-free and wide distribution and use across the
Internet community difficult.  The group will try to standardize codecs
that can be relatively freely and easily distributed, and employed as
much as possible. In doing so, it will adhere closely to these BCPs.
More specifically, "In general, IETF working groups prefer technologies
with no known IPR claims or, for technologies with claims against them,
an offer of royalty-free licensing."  Note that in accordance with BCP
79, the group will not explicitly rule out the possibility of adopting
technology that does not meet these IPR requirements; such decisions
will be made in accordance with the rough consensus of the group.

Deliverables

1. Guidelines that define the work process of the group, using
draft-valin-codec-guidelines as the basis for achieving consensus. This
document shall be Informational.

2. Detailed technical requirements, using draft-valin-codec-requirements
as the basis for achieving consensus.  This document shall be Informational.

3. Specification of one or more codecs that meet the agreed-upon
requirements, in the form of Internet-Drafts that define the codec
algorithm and source code for a reference implementation. The text
description of each codec shall indicate which components of the encoder
and decoder are mandatory, recommended, and optional.  It is envisioned
that each such specification shall be a Proposed Standard document.

Milestones

Mar-2010: WGLC on Codec Standardization Guidelines
Mar-2010: WGLC on Requirements
May-2010: Codec Standardization Guidelines to IESG (Informational)
May-2010: Requirements to IESG (Infomational)
Oct-2010: WGLC on codec specification(s)
Jan-2011: Submit codec specification(s) to IESG (Standards Track)

***

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 8 Oct 2009 15:11:57 -0400

This charter is not on the general BOF page for IETF 76

http://trac.tools.ietf.org/bof/trac/wiki/BifIETF76

Can it be added there ? Or is it in transit ? (I would like to send  
the link to some people.)

Regards
Marshall

On Oct 8, 2009, at 2:33 PM, Peter Saint-Andre wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkrOMFwACgkQNL8k5A2w/vyYdgCgkPq/L9sj2x8hx6SE8UGS7aE6
> HN0AoPxQRKcPXPWOdJmnNjiAAS2wTG1F
> =eQ5m
> -----END PGP SIGNATURE-----
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 08 Oct 2009 13:28:03 -0600

> http://trac.tools.ietf.org/bof/trac/wiki/BifIETF76

Updated, with link to:

http://trac.tools.ietf.org/bof/trac/attachment/wiki/BifIETF76/codec-charter.txt

/psa

---

From: Henry Sinnreich <[email-masked]>
Date: Thu, 8 Oct 2009 15:03:09 -0700

The problem statement and objective number 1 are admirably stated.

I would suggest however the codec has no dependency on the signaling protocol, whatever that may be.
It is up to the signaling protocol to inform about the codec and not vice versa.
Last but not least, there are other, non-IETF signaling protocols in wide use on the Internet.

Also the reference to packet loss in the Internet opens a can of worms, since broadband networks don't really suffer from sizable random packet loss, but may suffer from packet burst loss (such as producing voice clipping) due to congestion or route flapping for which no codec can expected to be tolerant or immune.

The can of worms is even bigger for wireless access networks and for mobile devices, but again, for whatever cause, voice clipping is beyond the codecs to fix.

If people still want to mention packet loss, what are the correct metrics (random and burst statistics) and the values for Internet voice codecs to deal with? What is the role of the transport protocols and the role of the codecs in dealing with packet loss? For example, how is packet loss dealt with in streaming HTTP
(Example: <draft-pantos-http-live-streaming>), just to illustrate the point about transport, since streaming HTTP may not be suitable for interactive voice .

In summary, IMO there should be no codec dependency on specific signaling protocols and no mention of packet loss.

Henry

On 10/8/09 1:33 PM, "Peter Saint-Andre" <[email-masked]> wrote:

/psa

***

PROPOSED CHARTER

Problem Statement

Objectives

Deliverables

Milestones

***

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Oct 2009 08:52:01 +0200 (CEST)

On Thu, 8 Oct 2009, Henry Sinnreich wrote:

> If people still want to mention packet loss, what are the correct 
> metrics (random and burst statistics) and the values for Internet voice 
> codecs to deal with? What is the role of the transport protocols and the 
> role of the codecs in dealing with packet loss? For example, how is 
> packet loss dealt with in streaming HTTP (Example: 
> <draft-pantos-http-live-streaming>), just to illustrate the point about 
> transport, since streaming HTTP may not be suitable for interactive 
> voice .
>
> In summary, IMO there should be no codec dependency on specific 
> signaling protocols and no mention of packet loss.

I have earlier proposed that the full solution should be able to cope with 
10% packet loss and 1.5-2 seconds of jitter. Of course there would be 
impact on quality, and continous loss (a full second of loss) is of course 
impossible to handle.

But the two major types I see that needs to be handled are:

Wireless networks with ARQ such as 802.11, 3GPP and alike. These typically 
can induce 1-2 seconds of jitter, but with low or no packet loss.

Routers/switches that do buffering and which use either FIFO/taildrop or 
some kind of more intelligent buffer management such as (W)RED or alike. 
These might drop a few percent of packets (usually never more as TCP backs 
off a lot at these packet drop rates and thus the links are usually not 
loaded more than what causes a few percent packet loss) plus induce jitter 
as other traffic affects this. Routers typically never buffer packets more 
than 600ms, often much lower, thus jitter is less unless there are 
multiple congested links in a row.

I'm a firm believer that for the whole solution to work, this can't be 
seen as a layered problem in which the layers can ignore each other, but 
to work well, they need to be tightly integrated.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Koen Vos <[email-masked]>
Date: Fri, 09 Oct 2009 02:00:49 -0700

Mikael,

My view on integrating jitter buffer and codec:

The integration is best done through an interface that lets the  
decoder indicate to the jitter buffer what state the audio signal is  
in, and lets the jitter buffer tell the decoder what action to take.  
For example, if the jitter buffer wants to accelerate the decoding of  
packets, it should (try to) wait until the decoder signals a silent  
period, or at least a stationary part in the signal, and then tell the  
decoder to compress the audio signal. The interface needs to pass just  
a few parameters in both directions for a tight integration between  
decoder and jitter buffer.

The jitter buffer should however not be part of the decoder, because:
1. The jitter buffer, just like other parts of the voice stack, is not  
essential for interoperability.
2. Voice stacks that contain multiple codecs typically share a common  
jitter buffer.
3. A jitter buffer is tuned towards the network and platform. As you  
point out, a wireless 3G network has rather different packet loss and  
jitter behavior than a DSL modem.
4. Even if we would define a jitter buffer that works great for any  
network today, it may not work well with tomorrow's networks.

I also don't see a use in defining how much jitter a codec can handle.  
It's easy to handle 10 seconds of jitter by having a fixed 10 second  
jitter buffer, but depending on the network characteristics that may  
or may not be a good solution.

In short: for any VoIP application to work well, all of its layers and  
components need to be tightly integrated. But the role of a codec is  
merely to provide the interfaces to allow that to happen - not to  
define the complete VoIP application.

best,
koen.

Quoting Mikael Abrahamsson:

>> metrics (random and burst statistics) and the values for Internet  
>> voice codecs to deal with? What is the role of the transport  
>> protocols and the role of the codecs in dealing with packet loss?  
>> For example, how is packet loss dealt with in streaming HTTP  
>> (Example: <draft-pantos-http-live-streaming>), just to illustrate  
>> the point about transport, since streaming HTTP may not be suitable  
>> for interactive voice .
>>
> I have earlier proposed that the full solution should be able to  
> cope with 10% packet loss and 1.5-2 seconds of jitter. Of course  
> there would be impact on quality, and continous loss (a full second  
> of loss) is of course impossible to handle.
>
> Wireless networks with ARQ such as 802.11, 3GPP and alike. These  
> typically can induce 1-2 seconds of jitter, but with low or no  
> packet loss.
>
> Routers/switches that do buffering and which use either  
> FIFO/taildrop or some kind of more intelligent buffer management  
> such as (W)RED or alike. These might drop a few percent of packets  
> (usually never more as TCP backs off a lot at these packet drop  
> rates and thus the links are usually not loaded more than what  
> causes a few percent packet loss) plus induce jitter as other  
> traffic affects this. Routers typically never buffer packets more  
> I'm a firm believer that for the whole solution to work, this can't  
> be seen as a layered problem in which the layers can ignore each  
> other, but to work well, they need to be tightly integrated.
>
> --

---

From: Ron <[email-masked]>
Date: Fri, 9 Oct 2009 20:18:21 +1030

Hi,

On Fri, Oct 09, 2009 at 02:00:49AM -0700, Koen Vos wrote:
> For example, if the jitter buffer wants to accelerate the decoding
> of packets, it should (try to) wait until the decoder signals a
> silent period, or at least a stationary part in the signal, and then
> tell the decoder to compress the audio signal. The interface needs
> to pass just a few parameters in both directions for a tight
> integration between decoder and jitter buffer.

Why should the jitter buffer presuppose to know when it would be a
good time for the codec to compress?  Shouldn't the jitter buffer
just be giving the codec stats on dataflow that it can parameterise
how it sees best?

I agree both parties need to be talking about what those stats and
parameters need to be though.

Cheers,
Ron

---

From: Lars Eggert <[email-masked]>
Date: Fri, 9 Oct 2009 14:35:29 +0300

Hi,

On 2009-10-9, at 9:52, Mikael Abrahamsson wrote:
> cope with
> 10% packet loss and 1.5-2 seconds of jitter.

I'm not aware of any Internet paths where such conditions would be  
considered normal. As such I question why such cases should be  
supported as a requirement. (Obviously, if a solution happened to work  
under such conditions *while also* having negligible overhead under  
normal conditions, that'd be fine - but I'm simply not sure this can  
be done.)

Lars

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Oct 2009 14:22:47 +0200 (CEST)

On Fri, 9 Oct 2009, Lars Eggert wrote:

> I'm not aware of any Internet paths where such conditions would be considered 
> normal.

During weeks of operation after the tsunami, I saw 7% listed as average 
packet loss for certain networks in the far east. Couple this with some 
wireless network and you have prolonged situations of these conditions. Of 
course this is not good or normal, but it's not very uncommon to see these 
adverse conditions.

> As such I question why such cases should be supported as a requirement. 
> (Obviously, if a solution happened to work under such conditions *while 
> also* having negligible overhead under normal conditions, that'd be fine 
> - but I'm simply not sure this can be done.)

It's my strong belief that if this is not something that is aimed for, 
then we'll end up with network requirements shown to be unrealistic in 
real life. 20ms jitter buffer and 0.1% packet loss as some implementations 
require, ie make the application dictate what the network should do 
instead of trying to adapt the application to real world network 
characteristics.

---

From: Henry Sinnreich <[email-masked]>
Date: Fri, 9 Oct 2009 06:39:36 -0700

It is fortunate this point was discussed.

Designing for 7% packet loss seems to be clearly a non-objective for a high performance voice codec.
Over such networks, IM is much more adequate.

My personal two cents,

Henry

On 10/9/09 7:22 AM, "Mikael Abrahamsson" <[email-masked]> wrote:

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Oct 2009 16:19:10 +0200 (CEST)

> It is fortunate this point was discussed.

Yes.

> Designing for 7% packet loss seems to be clearly a non-objective for a high performance voice codec.

Why? Is there a concrete technical reason why it can't be done? Over these 
kinds of networks you'll see lots of jitter anyhow, is there a concrete 
technical reason why one can't make sound good enough for the other end to 
understand what is being said even at 10% packet loss with 200ms buffert? 
If you send one packet every 20ms, 10% packet loss means you're losing one 
packet every 200ms (on average).

> Over such networks, IM is much more adequate.

I think people want to have the option to speak to each other if they can, 
otherwise they might as well use morse code with long wave radios, it's 
also an "adequate" means of communication, very resiliant.

---

From: Dean Bogdanovic <[email-masked]>
Date: Fri, 9 Oct 2009 10:50:31 -0400

I don't think that IM as always much more adequate. In emergency  
situations, where time is essential, voice is most efficient  
communication method and unfortunately in such situations networks  
have high packetloss and jitter.

Dean

---

From: Lars Eggert <[email-masked]>
Date: Fri, 9 Oct 2009 19:02:23 +0300

> have high packetloss and jitter.

My personal contribution to RFC1925bis, should it ever be published,  
will be Eggert's law: any IETF effort can be derailed by putting it  
into an emergency services context.

:-)

Lars

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 09 Oct 2009 13:43:52 -0400

I think it's important here to distinguish between the charter and the 
precise requirements. The intent of the charter is to define the kind of 
problem we're trying to solve, whereas the requirements I-D should have all 
the details.

Regarding the charter, the intent of objectives 2, 3 and 4 was to specify 
the operating environment we were primarily targeting, which is the 
Internet. Also, the main reason for mentioning RTP/SIP/SDP/SRTP/XMPP was to 
acknowledge that these protocols are important for the IETF and thus that 
the codec designers should ensure that the codec fits well with these 
protocols and not to do anything that would create interoperability issues. 
As Henry pointed out, this can be interpreted as implying that we would not 
consider other protocols on the Internet, but this was not the intent here. 
We should probably make that clearer.

About packet loss, it's obviously not possible to make a codec that 
produces intelligible speech when you have gaps of 1 second in the 
transmission. I personally believe that it's not even necessary to target a 
particular loss rate (as many pointed out, it varies a lot), but that it's 
more about what to prioritize. For a codec that operates on the Internet, I 
believe it is for example more useful to improve the loss robustness than 
to squeeze a few more bits out of the bit-stream, especially considering 
all the header overhead. On other networks (especially the ones where 
bandwidth is very expensive), the trade-offs could be different.

	Jean-Marc

Mikael Abrahamsson wrote:
> Why? Is there a concrete technical reason why it can't be done? Over 
> these kinds of networks you'll see lots of jitter anyhow, is there a 
> concrete technical reason why one can't make sound good enough for the 
> other end to understand what is being said even at 10% packet loss with 
> 200ms buffert? If you send one packet every 20ms, 10% packet loss means 
> you're losing one packet every 200ms (on average).
> 
> I think people want to have the option to speak to each other if they 
> can, otherwise they might as well use morse code with long wave radios, 
> it's also an "adequate" means of communication, very resiliant.
>

---

From: Koen Vos <[email-masked]>
Date: Fri, 09 Oct 2009 15:33:53 -0700

Quoting Ron <[email-masked]>:

> good time for the codec to compress?

State-of-the-art jitter buffers adaptively vary the speed of decoding  
packets to balance delay and packet loss when jitter changes.  
Accelerating or slowing down the decode speed can lead to audible  
artifacts however, and the jitter buffer should therefore be careful  
when to trigger such operations.

To do this, the decoder should pass a measure of "perceptual  
sensitivity to time-scale modification" to the jitter buffer (a voice  
activity measure could be a first approximation). The jitter buffer  
then returns one of these commands to the decoder:
  1. decode
  2. conceal
  3. decode + stretch
  4. decode + compress

All this is relevant and important. For example, 3G networks exhibit  
delay spikes of several seconds. Every few minutes the network just  
stops delivering packets, and then releases them in short order. Same  
for WiFi with delay spikes of up to a second (caused by the radio  
scanning for available access points). Quickly-adapting jitter  
buffering greatly helps with such non-stationary jitter.

> Shouldn't the jitter buffer just be giving the codec stats on  
> dataflow that it can parameterise how it sees best?

What kind of stats are you thinking of? The jitter buffer only needs  
the sensitivity measure from the decoder, and the codec developers are  
probably best at defining this.

koen.

---

From: "Christian Hoene" <[email-masked]>
Date: Sat, 10 Oct 2009 01:11:36 +0200

> then returns one of these commands to the decoder:

Allow me to point out our recent report addressing the interface and those
PLC and strech/compress commands.
http://net.cs.uni-tuebingen.de/html/ristore2net/publications/papers/hoene_sb
c2009.pdf
Chapter 3 contains the speech receiver and dejittering part that might be
relevant for this discussion.

Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Sat, 10 Oct 2009 10:16:02 +0200

Dear Lars Eggert, Dean Bogdanovic and Mikael Abrahamsson,

allow me to raise my voice on the issues of supporting a good voice
transmission even under harsh conditions.

> normal conditions, that'd be fine - but I'm simply not sure this can

Probably there are about a billion phone calls each days world-wide. If the
phone systems works well only on for normal conditions (e.g. 99,9% of them),
this would result in about one million unhappy users each day. In addition,
service centers would kept busy. Thus, if possible one should concentrate on
providing a well working solution even for most of remaining 0,099...%
percentage of conditions---as long as the technical overhead is not too
high.

Let me point out a recent research project by Prof. Dr. Jörg Ott, HKK
Helsinki called DT Talkie. http://www.netlab.tkk.fi/tutkimus/dtn/dttalkie/
It is a phone application for delay tolerant networks and Jörg told me that
it was the job of a student during a seminar. 

Knowing these results, I described the requirements for a reliable system
and sketched a rough technical solution in
http://www.ietf.org/id/draft-hoene-avt-acs-requirements-00.txt 
Please read Section 2.4 and 3.2.

Having such as Push-To-Talk like solution would also help to address other
problems described by Mikael and Koes,

> buffering greatly helps with such non-stationary jitter.

Of course, providing a DT-Talkie or Push-To-Talk application is not the task
of a codec. Rather, it should be standardized in the AVT/transport group.
However, a codec could support those applications with two features:

1. A highly bandwidth-efficient transmission of speech (while taking
advantage of high algorithmic delays and very large frame sizes)
2. Support of discontinues transmission modes to detect voice activity and
compress silence and background noise.

With best regards,

 Christian Hoene

---

From: Ron <[email-masked]>
Date: Sat, 10 Oct 2009 23:55:47 +1030

On Fri, Oct 09, 2009 at 03:33:53PM -0700, Koen Vos wrote:
> State-of-the-art jitter buffers adaptively vary the speed of
> decoding packets to balance delay and packet loss when jitter
> changes.

Fundamentally, elastic buffers are pretty simple and it's probably best
to try to keep them that way.  They need enough tension to stay as small
as possible when they aren't under stress, and enough elasticity to
quickly stretch without ever 'snapping'.  Where 'snapping' in our case
is a 'lost' packet that we've given up on ever receiving.  That's pretty
much all they really need to do to fulfil their role in these systems.
At least for the purpose of examining how they interact with codecs.

> Accelerating or slowing down the decode speed can lead to
> audible artifacts however,

That's the bit where this is a problem in many current codecs, indeed.

> and the jitter buffer should therefore be careful when to trigger
> such operations.

and that's I think where you and I first disagree.  And where Christian
and I see a bit more eye to eye.  But it may just be a choice of words
and polarities :)  I don't think the jitter buffer should be 'trigerring'
these things based on input from the codec -- I think the jitter buffer
should just provide rate information to the codec, and leave it to the
codec to decide when and how to act to meet the longer term goals that
we set for it.  As gradually as it needs to, to avoid abrupt artifacts.

> sensitivity to time-scale modification" to the jitter buffer (a
> voice activity measure could be a first approximation). The jitter
> buffer then returns one of these commands to the decoder:
>  1. decode
>  2. conceal
>  3. decode + stretch
>  4. decode + compress
> 
> All this is relevant and important.

Yes, but I think that saying it is _all_ that is relevant and important,
supposes too much that new codecs will be just like the old ones in all
the ways that this group is forming to solve :)  'conceal' is a rather
rough approximation for many different actions that could be taken in
many different sets of circumstances.

I think I'd rather see the elastic buffer make available to the codec
(and other parts of the system) some simple information along the
lines of:

 1. is it stretching or contracting.  (but codec need not not care)
 2. does it want the codec to stretch or contract (more/faster/slower)?
 3. how far away from 'snapping' are we at present,
    (ie. how long until the first MIA packet, or the end of the buffer)
    and how big will the hole be when we do.

Then the clever codec people can work their best magic armed with some
small foreknowledge of 'the future', that the delay in the buffer can
make available to them.

Trying to tell the codec which frames it should drop samples in, based
on information it gave you as an abstract hint, seems like too many
round trips and too much information loss.  Letting it know when it is
going too fast, or too slow, or when it should just hum to itself for
a while seems like a much more flexible way to achieve the same goals.
It surely knows better than any outside agent where and when it can
fudge a few samples that most people won't notice.  It may be able to
catch up more aggressively on some frames than the elastic buffer
requested, and amortise that on other frames it can't adjust so far,
for the same or better average rate of change to the buffer size.

Regions of "perceptual sensitivity to time-scale modification" don't
seem like something that would always neatly respect packet boundaries,
so that seems like too coarse a sieve to be optimal, yet the packet
buffer can act upon nothing finer.

> For example, 3G networks exhibit
> stops delivering packets, and then releases them in short order.
> Same for WiFi with delay spikes of up to a second (caused by the
> radio scanning for available access points). Quickly-adapting jitter

Ok.  That's reality, so we can't argue with that, and we all agree
that when delays are that long most interactive applications are
doomed anyhow.  So the interesting question is how do we deal with
that and give the user as much of what they want as we can anyway.

I see two sides to it:

 1. The elastic buffer must stretch to fit the expected maximum latency.
    You know you'll 'never' lose packets if you just wait long enough
    for them, but some of them may be very, very, late.  But you also
    know that once the next load arrives, you probably won't need to
    wait that long again for a while -- so you can apply a lot of
    tension to contracting it toward 'realtime' again as quickly as
    possible.

 2. Sometimes you will really lose packets.  And at some point you just
    have to give up on those.  No amount of further stretching is likely
    to help you.  But if you gave the codec early warning about the
    probability of it not arriving, it will have had time to plan its
    best action for if it really doesn't come, and help you to wait for
    it as long as possible in case it still does, by stretching out the
    data you do have for as long as is reasonably can.

> the sensitivity measure from the decoder, and the codec developers
> are probably best at defining this.

If the elastic buffer simply tells the codec when it needs to stretch
or shrink, then that covers the first case above.  As soon as the first
late packets are observed you tell the codec to start slowing down,
then more and more urgently as you realise you're really going to
underflow very soon.  Eventually you fade to noise.  Once the burst
comes in, you tell the codec to hurry things up for a while, because
reality is back and it's not going to wait.

In the case of speech, the user experience is that words  simply   started
slowing       down           [then stopped for a while]    then_came_more_
quickly_for_a_bit before it was all normal again.  But the really important
bit is that you never actually mi   d anything which was said.  Which is
mu h m  e annoy  g, and hard to comprehend when it happens.

In the case of a truly lost individual packet, an intelligent elastic
buffer could warn the codec this was to be expected N frames from now,
and give it time to prepare a synthesis if it thought it could mumble
its way through, or fade to silence gracefully if there will be too
many missing to fake it.  Again early stretching of the buffer and the
codec play rate can buy you a little more time for it to arrive than
would be expected from the "normal rate" of what you currently have
backed up in the buffer.

So to bring it back to what is really relevant here, what does an elastic
buffer need from the codec?  Probably not much.  A way to smoothly stretch
or shrink time by some variable proportion, to balance the tension in the
buffer.  A way to gracefully fade to comfort noise and back again when the
signal is interrupted.  That's about it.

What does the codec need from the elastic buffer?  Forewarning of when the
buffer is about to 'snap' seems like the main information that it could
make some use of.  Anything else?

I'm not really sure the relationship between these two components actually
needs to be much more complicated or formal than that.  What can't we do
with just those simple controls?  We get a "double-ended" elastic buffer
that can stretch in both directions, and require nothing more than that the
codec give us finer control over the average output sample rate than the
traditional quantum leaps from one rate to the next.

If the interactive music use case, or large crowd conferences, become
viable, the idea of only being able to make these sort of adjustments
during 'silence periods' must be steered well clear of, we need to be
able to steal or pad every sample that we can when necessary, since
there is no silence on the scale of a "normal" 1-to-1 phone conversation.

Am I missing something here?  Otherwise I think what we need for this will
mostly be covered by the time-stretching considerations of the codec, no?

 Ron

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 13 Oct 2009 11:13:27 +0200

HI Jean-Marc Valin,

> Had a quick look at the results section of your document (sorry, didn't
> read any of the details). From what I read, I have the following comments:

> 1) The mono results appear a bit low compared to what I usually obtain
> with my own testset and PQevalaudio (-0.5 at 64 kb/s), but that could be
> explained by a different testset and evaluation tool.

I would recommend to use the original PEAQ version from Opticom. 

> 2) The stereo results are mostly consistent with the mono results
> because of a bug in the 0.6.0 and 0.6.1 versions (which is now fixed in
> git). I'm hoping that 0.7.0 will be the first version to have usable
stereo.

Then, I will start the scripts again. 

> 3) The SBC results in stereo look a bit suspicious to me. It seems like
> the bit-rate only increases by ~10 kb/s for the same quality, which is a
> bit surprising. Is it possible that your test samples are almost mono? 

Most of the sample item contain single instruments. Only the “refryc” (jazz
music by Ry Cooder) and “refsop01” (opera) sample items contain more complex
content. I checked for all used items whether left and right channel differ
(they did) but still better sample items could be used for stereo (e.g. old
Beatles songs).

> I
> also heard that PEAQ didn't evaluate the stereo image at all.

Yesterday, I got the PEMO-Q audio assessment software from www.hoertest.de.
I will check whether it outperforms PEAQ. 

Christian

> 
> 	Jean-Marc
> 
> Christian Hoene a écrit :

---

From: Stefan Sayer <[email-masked]>
Date: Tue, 13 Oct 2009 13:50:06 +0200

> relevant for this discussion.

I had a look on that chapter, and I think it gives interesting 
implementation simplification if the whole design is "pull based" (i.e. 
thought completely from the sink side).

Specifically about the method and interface you describe in 3.2 i have a 
few comments:

1. It makes sense that a PLC in the codec would be employed not only for 
extrapolation, but also for interpolation (when packets after a gap are 
already received).

2. In order to not have excessive late loss, you have to start the audio 
sink with a time offset (timestamp offset relative to the receiver). 
Methods like the one you referenced from Liang et al [27] intend to 
minimize exactly this offset (which contributes to total latency, or 
mouth-to-ear delay), while keeping the expected late loss bounded (e.g. 
in [27] using L filter to estimate). Missing in your description is 
exactly this - how do you setup the offset to the sink in the first 
place, and how do you adapt it, e.g. if you notice changes in jitter, or 
notice clock skew ?

3. It appears that, if there is the need to (e.g. next audio not 
available), the time scale modification in your CDCD needs to slow down 
voice immediately, and do all on one audio block. A buffer based 
approach makes it possible, to *slowly* adapt the playout speed (and in 
both directions) this way this operation is less intrusive, because the 
amount of speed modification does not need to be that high; instead, a 
"longer-term goal" of playout buffer length can be set.

Back to the topic on codec PLC and time scale interface, if the codec 
can optimize those two for the case of interpolation (which I don't know 
whether this is true, being not deeply into codec internals), I think it 
makes sense to consider this as requirement for the codec.

About the question where the actual decision happens whether to 
stretch/compress or not, I think what your paper illustrates is that 
there are so many possibilities to design a dejitter/playout/plc buffer, 
and there are many factors that such a buffer might take into account, 
that it will be simpler to signal relevant voice characteristics 
parameters from the codec to the buffer, than the reverse.

Best Regards
Stefan Sayer

[27] Y. J. Liang, N. Fï¿oerber, and B. Girod, “Adaptive playout 
scheduling and loss concealment
for voice communication over IP networks,” IEEE Transactions on 
Multimedia, vol. 5,
no. 4, pp. 532–543, Dec. 2003.

a version of this, optimized for high concurrent call count by 
predicting when to employ TD-WSOLA best, is implemented in SEMS 
(http://iptel.org/sems), see AmAdaptivePlayout in
http://svn.berlios.de/wsvn/sems/trunk/core/AmPlayoutBuffer.cpp .

-- 
Stefan Sayer
VoIP Services

[email-masked]
www.iptego.com

IPTEGO GmbH
Wittenbergplatz 1
10789 Berlin
Germany

Amtsgericht Charlottenburg, HRB 101010
Geschaeftsfuehrer: Alexander Hoffmann

---

From: "Christian Hoene" <[email-masked]>
Date: Thu, 15 Oct 2009 15:52:28 +0200

Dear Peter Saint-Andre and all,

inline you will find my personal comments to the proposed charter.

> Codec WG provides a structured forum for these efforts.

Sounds good!

> 2. Addressing the real transport conditions of the Internet, especially

I do not understand this last consideration. What do you mean with this
statement? Are packet losses the only attribute of transmissions over the
Internet? At least it has been agreed upon that jitter compensation is also
relevant. I would suggest to rephrase this requirement to "Addressing the
real transport conditions of the Internet. Important attributes of packet
transmissions over the Internet are identified in cooperation with other
IETF working groups such as AVT and DCCP."

> Codecs optimized for very low bit rates 

Traditionally, very low bit codecs are operating below 2.4 kbps. It might
not be known to the Internet guys.
 
> and codecs for non-interactive
> audio are out of scope.

Maybe you can add "because they require other kind of optimizations". BTW:
Do you address multicast, too? If not, mention it here.
 
> endeavor not to function as a "codec factory" that produces a large

I would suggest to remove the part after the semicolon. Instead, maybe add
something like "The codec(s) shall be address a large range of typical
transmission conditions on the Internet."
 
> arriving at rough consensus.

I think it is problematic to predefine a draft that must be chosen. A
critical minded person might think that this standardization process is a
put-up affair. Instead, it is in good tradition of the IETF that the best
technical approach should win. Just imagine that you codec experts cannot
agree on a common requirements document or that other codec guys make a
better proposal. We should allow competition because this leads to better
technical solutions. Thus, I suggest to change it in "... outlined in a
codec-guidelines draft such as the draft-valin-codec-guidelines-01 and in a
codec-requirements draft such as draft-valin-codec-requirements-01
respectively; ..."?

> as the basis for achieving consensus.  This document shall be
Informational.

Again, I would suggest to change it to "Guidelines that defined the work
process of the group define in a common codec-guidelines document." and
"Detailed technical requirements defined in a common codec-guidelines
document." due to the same reasons.

> May-2010: Requirements to IESG (Infomational)
+r
> Jan-2011: Submit codec specification(s) to IESG (Standards Track)

This is an ambitious schedule. In my opinion the danger of not keeping this
schedule is high. But then again, who cares...

All other parts are fine for me.
 
With best regards,

 Christian

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 16 Oct 2009 14:46:01 -0600

On 10/9/09 11:43 AM, Jean-Marc Valin wrote:

> Regarding the charter, the intent of objectives 2, 3 and 4 was to
> specify the operating environment we were primarily targeting, which is
> the Internet. Also, the main reason for mentioning RTP/SIP/SDP/SRTP/XMPP
> was to acknowledge that these protocols are important for the IETF and
> thus that the codec designers should ensure that the codec fits well
> with these protocols and not to do anything that would create
> interoperability issues. As Henry pointed out, this can be interpreted
> as implying that we would not consider other protocols on the Internet,
> but this was not the intent here. We should probably make that clearer.

Correct. Ensuring interoperability with those parts of the IETF protocol
stack doesn't imply that the WG will ignore interoperability with any
other protocols. IMHO the text "Core considerations *include the
following*..." and "Ensuring interoperability with Internet signalling
technologies *such as*..." shows that these considerations are by no
means exclusive.

Peter

- --
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 16 Oct 2009 15:12:45 -0600

> inline you will find my personal comments to the proposed charter.

Thank you for your feedback.

> Internet? 

By no means. The word "especially" does not exclude other conditions
that might be important to consider.

> At least it has been agreed upon that jitter compensation is also

I think it might be more productive to say this:

***

2. Addressing the real transport conditions of the Internet as
identified and prioritized by the group

***

It's not really the task of the charter to define the real transport
conditions of the Internet, but if we include this is an objective of
the group then they should certainly keep those conditions in mind as
they complete their work.

> not be known to the Internet guys.

Added.

> Maybe you can add "because they require other kind of optimizations". 

In my working copy I have changed this to:

***

Codecs optimized for very low bit rates (typically below 2.4 kbps) and
codecs for non-interactive audio are out of scope because they might
necessitate specialized optimizations.

***

> BTW:

By "multicast" do you mean many-to-many or one-to-many? We talk a bit
about that in the requirements draft.

> transmission conditions on the Internet."

Your proposed addition does not talk about the point you propose to
remove (i.e., we are not going to produce dozens of codecs) and the
point you add here is discussed elsewhere in the charter. Therefore I'm
not in favor of this change.

> I think it is problematic to predefine a draft that must be chosen. 

I think it is problematic to mention no Internet-Drafts in the charter.
The existing I-Ds will form the starting point for work within the
group. Whether those I-Ds will remain as they are today (I doubt that
they will) is up to the group.

> A

The "such as" here is not especially helpful as guidance to the group or
as part of our contract with the IESG. Given that no one else on this
list has even mentioned the possibility of working on Internet-Drafts
for the procedural guidelines or technical requirements, I think it is
best to show the IESG that the published I-Ds are available for
inspection and will form the starting point of the group's efforts.
Those I-Ds are not final by any means, but at least they are something
that the group can refine over time. Which is exactly what the current
charter text says. However, I'm sure we can improve things slightly.

How about this?

***

Work processes and technical requirements for achieving the foregoing
objectives are outlined in draft-valin-codec-guidelines and
draft-valin-codec-requirements respectively; these documents will form
the starting point for working toward consensus and will be refined by
the group in accordance with the usual IETF procedures.

***

> document." due to the same reasons.

In my working copy I have changed these to:

***

1. Guidelines that define the work process of the group, using
draft-valin-codec-guidelines as the starting point for working
toward consensus. This document shall be Informational.

2. Detailed technical requirements, using
draft-valin-codec-requirementsas the starting point for working
toward consensus.  This document shall be Informational.

***

Does that text address your concerns?

> schedule is high. But then again, who cares...

You're right that it is aggressive. But it seems that we have a core
group of energetic contributors and that we might be able to make fast
progress if the WG is approved. But perhaps giving ourselves a few more
months here and there would be appropriate.

So perhaps this is slightly more realistic yet still ambitious:

***

Mar-2010: WGLC on Codec Standardization Guidelines
May-2010: Codec Standardization Guidelines to IESG (Informational)
May-2010: WGLC on Requirements
Jul-2010: Requirements to IESG (Infomational)
Feb-2011: WGLC on codec specification(s)
Apr-2011: Submit codec specification(s) to IESG (Standards Track)

***

> All other parts are fine for me.

Thanks again for your feedback.

Peter

- --
Peter Saint-Andre
https://stpeter.im/

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 18 Oct 2009 08:51:47 -0500

This is an excellent formulation IMO.

Thanks, Henry

On 10/16/09 4:12 PM, "Peter Saint-Andre" <[email-masked]> wrote:

> iEYEARECAAYFAkrY4c0ACgkQNL8k5A2w/vzmNgCgy4XHU9FgFCdomRzFGftFoKrx
> unMAn0f6r7zkfLVjUCPdWJtazfgy5kAa
> =n0pQ

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 18 Oct 2009 08:53:49 -0500

>IMHO the text "Core considerations *include the following*...
>" and "Ensuring interoperability with Internet signalling

This seems to me a good formulation.
Maybe it is useful to clarify the generic idea that signaling dependent
codecs are out of scope  naturally.

Thanks, Henry

On 10/16/09 3:46 PM, "Peter Saint-Andre" <[email-masked]> wrote:

> iEYEARECAAYFAkrY24kACgkQNL8k5A2w/vxDbACeKMwX7f/JXzda2U4T4uI4uFyi
> yZIAni8GCDJVHJDtipx3Y7oWMJ2qRP2l
> =xl9n

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 19 Oct 2009 12:38:14 -0600

On 10/18/09 7:53 AM, [email-masked] wrote:
> codecs are out of scope – naturally.

Yes, that seems reasonable. Thanks for the suggestion. I shall post a
revised version of the proposed charter soon.

Peter

- --
Peter Saint-Andre
https://stpeter.im/