Subject: Re: [codec] RE : Codec BOF approved, much work needed

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 18 Jun 2009 21:23:32 -0400

Hi,

(I sent a few emails today that didn't go to the list apparently due to
DNS update issues, but the main one was quoted by Christian)

> The ITU, 3GPP and others have standardized PLC algorithms for many codecs.
> PLC is only important for the receiving side and is not required for the
> interoperability with the encoder. For example, an encoder using G.711
> really does not care whether the receiving side implement the G.711 appendix
> 1 PLC. Anyhow, the PLC has been standardized.

I am not opposed to having a *reference implementation* of the PLC for
the specified codec. In fact, this would be the only way to ensure that
the codec is really robust to packet loss. However, I don't think
there's any point in saying "this is how you should do the PLC" because
that is likely to improve. In the case of G.711 appendix I, it turns out
that most vendors are actually their own PLC algorithm to achieve better
quality than G.711 A1.

> More important than the
> algorithm is the interface between protocol stack and decoder. If the
> protocol stack can indicate that a packet has not arrived, any PLC algorithm
> can start working. Similar, if a packet arrives too late or too early, any
> dejittering buffer or time concealment algorithm can try to cope with it.

The interface you're talking about is nothing more than the
implementation's API. This does not need to be specified at all. Again,
you can say "in the reference implementation, you can call do_plc() to
conceal a lost packet", but nobody cares what you call the function in
your implementation or what the arguments are.

> So in good conditions, you can use the prediction,
>> but in bad conditions with high losses, you can turn it off and code all
>> frames independently (at the cost of a slightly higher bitrate). This is
>> the kind of feature that is useful to have in an "Internet codec",
>> without the need to specify exactly how the application has to use it.
> 
> If a feature is not standardized, some will not implement it. If a feature
> is not implemented everywhere, all users will suffer because all users can
> call each other.

The *codec* feature has to be standardised because you have to define
that "this bit turns this feature on", so all decoders will *have* to
correctly handle it. On the other hand, the sender is still free to
decide whether it wants to *use* the feature.

Like other people have said already, standardising a codec means
specifying the meaning of each bit in the packet. There is no need to
standardise things that aren't necessary for inter-operability -- no
matter how important they are. You will notice that none of the ITU
codecs specify how to do the jitter buffering (imaging having to support
10 jitter buffers in your application!). The only exception is the PLC
because it is specific to the codec. But even then, a non-normative
example is fine as far as I'm concerned. If you look at the iLBC RFC,
you'll see that it includes a reference implementation of the PLC but
basically states that you can implement the PLC in any way you like.

Cheers,

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 19 Jun 2009 10:30:37 +0200

Good Morning JM,

> your implementation or what the arguments are.

The IETF recommendation define clearly the services the protocol layer
offer. Also, the separation of functionally is clearly described. I just
want to clarify, how the protocol stack will look like:

Version A:

Acoustics and signal processing
Codec
Dejittering Buffer
RTP
UDP
IP
...

Or Version B:
Acoustics and signal processing
Codec
RTP
UDP
IP

We do not know the exact functionality split between dejittering buffer and
codec, because the times and kinds playout adjustments are depending on the
content of the audio signal, on the PLC, and the decoder. 

> decide whether it wants to *use* the feature.

No, it also concludes the information on which services the codec are
provides and which lower services it uses.

> You will notice that none of the ITU
> codecs specify how to do the jitter buffering.

I follow a clean slate approach in thinking, questioning many decisions that
have been made in the past. At this point I firmly believe that we shall
reconsider the classic thinking on what an audio codec does and what it does
not.

To state it clearly:
1) It is not only about the bits in the packets, it is also about the
interfaces to lower and upper layers.
2) Because the concealment of time adjustments, the loss concealment, and
the decoding are highly intercorrelated, these functionallies shall be
merged into one common block (Version B).
3) If we follow Version A, it should be clear on how the interface between
codec and dejittering buffer has to look like in order to support playout
adjustments.

> If you look at the iLBC RFC,

Yes, I would be perfectly fine having a dejittering buffer in the reference
implementation but stating that everybody is free to implement it
differently.

Cheers,

 Christian

---

From: Koen Vos <[email-masked]>
Date: Fri, 19 Jun 2009 03:28:41 -0700

> implementation but stating that everybody is free to implement it

A jitter buffer is usually a separate module because the mutual  
dependencies with the decoder are simple. And also because a voice  
engine often contains several speech codecs but only one jitter buffer.

That said, it may indeed be useful and fun to develop an open source  
jitter buffer - without having to standardize it. But you may want to  
start by looking at the existing open source jitter buffers (and voice  
engines).

koen.

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 19 Jun 2009 13:42:28 +0200

Hi Koen,

> start by looking at the existing open source jitter buffers (and voice

I think we shall understand which kind of dejittering buffer we mean. I just
copied this section from my PhD thesis

" 2.1.5. Playout scheduler
At the receiver, protocol entities process the packets and deliver them to
the de-jittering buffer (also called playout scheduler or packet voice
receiver), which temporally stores packets so that they can be played out in
a timely manner [11, 13]. They try to playout the speech time-regular manner
to conceal variations in the transmission delay also known as jitter. If
packets are too delayed to be played out on time, they are usually treated
as lost. Consequently, losses as seen by the application are in fact a
superposition of real losses and excessive delays, where “excessive” is used
in terms of playout buffer dimensioning. As the playout buffer contributes
to the end-to-end delay it should not store packets longer than necessary.
Instead, the playout buffer should drop packets that arrive too late to be
played
out at the scheduled time.
The playout scheduling can be static: If packets exceed a given transmission
time they are discarded (we will refer to this scheme as fixed playout
buffer ). Alternatively, adaptive playout buffers redefine the playout time
in accordance to the delay process of the network [146,173]. We refer to
this kind of adaptation as rescheduling. The scheduling of playouts can be
adjusted most easily during silence periods because then it is not
noticeable. Adjustments during voice activity require more sophisticated
concealment algorithms [132, 135]. Some early work on the perception of
playout jitter can be found in [123, 195].

References
[11] G. Barberis, “Buffer sizing of a packet-voice receiver,” IEEE
Transactions on Communications, vol. 29, no. 2, pp. 152–156, Feb. 1981.
[13] G. Barberis and D. Pazzaglia, “Analysis and optimal design of a
packet-voice receiver,” IEEE Transactions on Communications, vol. 28, no. 2,
pp. 217–227, 1980.
[123] N. Kitawaki and K. Itoh, “Pure delay effects on speech quality in
telecommunications,” IEEE Journal on Selected Areas in Communications, vol.
9, no. 4, pp. 586–593, 1991.
[132] Y. J. Liang, N. F¨arber, and B. Girod, “Adaptive playout scheduling
and loss concealment for voice communication over IP networks,” IEEE
Transactions on Multimedia, vol. 5, no. 4, pp. 532–543, Dec. 2003. 
[135] F. Liu, J. Kim, and C.-C. J. Kuo, “Adaptive delay concealment for
internet voice applications with packet-based time-scale modification,” in
IEEE International Conference on Acoustics, Speech, and Signal Processing
(ICASSP ’01), vol. 3, May 2001, pp. 1461–1464.
[146] S. B. Moon, J. Kurose, and D. Towsley, “Packet audio playout delay
adjustments: performance bounds and algorithms,”ACM/Springer Multimedia
Systems, vol. 27, no. 3, pp. 17–28, Jan. 1998.
[173] R. Ramjee, J. F. Kurose, D. F. Towsley, and H. Schulzrinne, “Adaptive
playout mechanisms for packetized audio applications in wide-area networks,”
in 13th Proceedings IEEE INFOCOM ’94, Toronto, Canada, June 1994, pp.
680–688.
[195] R. Steinmetz, “Human perception of jitter and media synchronization,”
IEEE Journal on Selected Areas in Communications, vol. 14, no. 1, pp. 61–72,
1996."

To my understanding, a modern dejittering buffer uses a similar algorithm as
described in [132] and [135] and is
a) medium specific (e.g., can only be used for audio)
b) might be codec-depended if its implementation shall be computational
efficient.

A couple of ETSI's VoIP performance tests have shown that the best VoIP
phones do use time stretching and shrinking algorithms to lower the impact
of playout adjustments.

With best regards,

 Christian

---

From: Randell Jesup <[email-masked]>
Date: Fri, 19 Jun 2009 10:24:02 -0400

...
>b) might be codec-depended if its implementation shall be computational

Agreed: adaptive, media-specific.  I can see knowledge of the codec
being useful, though that could be parameterized -- i.e. tell the jitter
buffer about the characteristics of the codec to let it adjust
tradeoffs, like the tradeoff between delay and loss.  For example, some
codecs might be fairly insensitive to one-packet losses, and the buffer
might crank down delay, while with another codec there may be less
redundant information in surrounding packets, and thus you might
increase delay slightly if it reduces late-packet "losses" due to
jitter.  

Note that much of this is a true matter of heuristic tuning, and the
tradeoffs can be application-dependent as well.  Those are big obstacles
to embedding a jitter buffer model in a codec definition (or
standardizing them at all).

>phones do use time stretching and shrinking algorithms to lower the impact

That would match my understanding and experience.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: Stefan Sayer <[email-masked]>
Date: Sat, 20 Jun 2009 23:34:24 +0200

o Randell Jesup [06/19/09 16:24]:
> increase delay slightly if it reduces late-packet "losses" due tosilence
> jitter.  

What you mention is not the only reason why dejittering and decoding is 
not independent. Like with PLC, it may be possible to stretch and 
compress the audio much more efficiently in the decoder than in an 
independent (even if parametrized) dejitter buffer working on the 
decoded audio: first, if the codec already uses and transmits voice 
activity information, the time point to stretch/compress may be chosen 
much more efficiently, and second, by using and modifying the filter 
coefficients the actual stretch/compress operations may be done more 
efficiently.

So, it is not sufficient to have information about the codec. The 
decoding engine is optimally involved directly in the dejitter process.

Best Regards
Stefan

---

From: Koen Vos <[email-masked]>
Date: Sat, 20 Jun 2009 16:09:02 -0700

Quoting Stefan Sayer:
> What you mention is not the only reason why dejittering and decoding  
> is not independent. Like with PLC, it may be possible to stretch and  
> activity information, the time point to stretch/compress may be  
> chosen much more efficiently,

Absolutely, that's common practice. But easily parametrized between  
the jitter buffer module and the decoder + Packet Loss Concealment.  
The decoder could for example inform the jitter buffer of the voice  
activity in the last decoded payload.

> and second, by using and modifying the filter coefficients the  
> actual stretch/compress operations may be done more efficiently.

Sure, but that's just PLC-internal, not related to the jitter buffer.

Fundamentally, the PLC deals with the signal, and the jitter buffer  
deals with packets.  The PLC is optimized towards the codec, the  
jitter buffer towards the application/platform/protocol/network.

koen.

---

From: Stefan Sayer <[email-masked]>
Date: Tue, 23 Jun 2009 01:44:58 +0200

o Koen Vos [06/21/09 01:09]:
>> much more efficiently,
> Absolutely, that's common practice. But easily parametrized between the 
> jitter buffer module and the decoder + Packet Loss Concealment. The 
> decoder could for example inform the jitter buffer of the voice activity 
> in the last decoded payload.
> 
> 
>> and second, by using and modifying the filter coefficients the actual 
>> stretch/compress operations may be done more efficiently.
> 
> deals with packets.  The PLC is optimized towards the codec, the jitter 
> buffer towards the application/platform/protocol/network.

so you are saying that the same mechanism which is used for PLC is to be 
used for time-scale modification of the voice (stretching/compressing) 
for adaptive playout scheduling like what is used with the method 
described in the mentioned [135]/[132] ?

In my opinion this is only possible if you feed the information about 
desired length of decoded voice into the decoder (so it can e.g. leave 
out or add one or more pitch periods), keep a 'backup' decoder state of 
the previous frame, or add one more frame length of decoding latency - 
otherwise you can not compress the voice of the previous packet and for 
stretching its suboptimal.

Anyway, this requirement would have to be taken into account for the 
decoder, and it is a requirement specific to packet switched networks. I 
f I understood him correctly it is this why Mr. Hoene wants to include 
the dejitter buffer in.

Stefan

> 
> koen.
> 
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

-- 
Stefan Sayer
VoIP Services

[email-masked]
www.iptego.com

IPTEGO GmbH
Wittenbergplatz 1
10789 Berlin
Germany

Amtsgericht Charlottenburg, HRB 101010
Geschaeftsfuehrer: Alexander Hoffmann

---

From: Koen Vos <[email-masked]>
Date: Tue, 23 Jun 2009 01:55:59 -0700

Quoting Stefan Sayer:
> so you are saying that the same mechanism which is used for PLC is  
> to be used for time-scale modification of the voice  
> (stretching/compressing) for adaptive playout scheduling like what  
> is used with the method described in the mentioned [135]/[132] ?

The kind of operations going on in a PLC are similar to those used for  
time stretching/compression. But perhaps Loss and Jitter Concealment  
(LJC) would be a better name for such a module.

In any case, I agree that such time scaling, at least in a basic form,  
should be a requirement for the decoder specification.  It makes the  
codec more suitable for packet networks with unreliable delivery  
times.  It's a shame that, despite the rigorous standardization  
process, ITU standards lack such functionality.

> decoder, and it is a requirement specific to packet switched  
> networks. I f I understood him correctly it is this why Mr. Hoene  
> wants to include the dejitter buffer in.

Right, seems like we're just talking about terminology: besides an LJC  
there is a need for a packet buffer and control mechanism that decides  
when to conceal, stretch or compress.  This functionality does not  
belong in a codec or its specification, in my opinion.

best,
koen.