Subject: Re: [codec] questions to be answered

From: Michael Knappe <[email-masked]>
Date: Tue, 3 Nov 2009 09:46:56 -0800

Agreed, thanks.

One quick note on the 150 ms end-to-end requirement. That is a general rule-of-thumb (and ITU specified) requirement for interactive speech responsiveness and ease of conversation, but one that digging deeper is dependent on language, cultural, and use case factors. There are conference situations which are more forgiving to longer delays, and conversely language types that require tighter delay to avoid emotional misinterpretation of verbal acknowledgements. 

Mike

----- Original Message -----
From: Brian Rosen <[email-masked]>
To: Roni Even <[email-masked]>; Michael Knappe; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Sent: Tue Nov 03 11:23:20 2009
Subject: Re: [codec] questions to be answered

Right, and the codec doesn't have anything special to match the delay:
that's done elsewhere.

Similarly, there is need for stereo and 5.1 acoustic echo cancel, but that's
not part of the codec either, although we sometimes get tempted to put some
cues in the stream to help the echo cancellation.

There is no attempt to match frame sizes or anything else.  You match delay
to get lip sync right. Everything else is pretty independent.

Of course, you have to keep the delay low ("low" in video terms): under 150
ms one way end to end, including any mixers.  That's usually easy for the
audio, hard for the video, so you artificially delay the audio to match the
video.

Brian

On 11/3/09 1:09 PM, "Roni Even" <[email-masked]> wrote:

> Mike,
I am not sure what you mean by tied to video codec. I can think of the
> delay
which can be similar to the video.
The general requirements is support
> for hands free, full band with stereo
support and optional 5.1
> support.
Current codecs used are AAC and G.719.
BTW: there is a different
> between tele-presence with multiple codecs and
hifh quality HD room systems
> with one video codec that provide
tele-presence" experience for small
> rooms.

Roni Even

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf
> Of Michael Knappe
> Sent: Tuesday,
> November 03, 2009 6:40 PM
> To: [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [codec] questions to be answered
> 
> As I've
> been preparing the 'application vs. criteria matrix', it's
> become clear that
> video-centric applications like telepresence will
> have audio codecs that are
> at least partially tied to or driven by
> characteristics of the video coding
> (e.g. frame size). Do we as a group
> want to tackle additional non-voice
> considerations in the WG?
> 
> Agree with MAR on G.711, logarithmic encoding
> with a relatively fixed
> SNR doesn't cut it for monotonically decreasing
> frequency content
> sources like speech beyond 3 kHz. That's why G.722 went
> with a simple
> two-band approach for better representation of the upper
> octave, and
> there are of course many examples today of sub-band
> approaches.
> 
> Mike
> 
> 
> ----- Original Message -----
> From:
> [email-masked] <[email-masked]>
> To: Christian Hoene
> <[email-masked]>; [email-masked]
> <[email-masked]>
> Sent: Tue Nov 03
> 10:33:56 2009
> Subject: Re: [codec] questions to be answered
> 
> Hi
> Christian,
> 
> My replies are in-line below (w/ MAR:).
> 
> Thanks for your
> consideration.
> 
> Michael Ramalho
> 
> -----Original Message-----
> From:
> Christian Hoene [mailto:[email-masked]]
> Sent: Tuesday, November 03,
> 2009 4:30 AM
> To: Michael Ramalho (mramalho); [email-masked]
> Subject: RE:
> [codec] questions to be answered
> 
> Hello Michael,
> 
> > At a minimum, I
> see two legitimate codec classes:
> >
> > CLASS 1) codecs that are bandwidth
> efficient and sound good to humans
> > (and perhaps satisfy tendeming or tone
> pass-through requirements),
> and
> >
> > CLASS 2) codecs whose output may
> later be processed by sophisticated
> > signal processing functions
> (high-quality mixing,
> 
> High-quality mixing? That do you have in mind?
> Stereo or binaural (3D
> sound)
> mixing?
> 
> MAR: I continued in the email
> to say that I could potentially think of
> lots of other examples that require
> some bounds on the linear
> distortion because of the signal processing
> desired at the receiver.
> But yes, one could conceive of mixing to reproduce
> the effect of where
> spatially the sound came from in the sending location
> using only the
> signals rendered at the receiver.
> 
> > many
> >
> classification-based applications such as speaker/sound/intrusion
> >
> identification) that have SNR/waveform/distortion criteria generally
> >
> exceeding CLASS 1.
> 
> You may add distributed speech recognition to the
> second class.
> However, to
> my understanding the current offered codecs have
> not been designed with
> the
> second class in mind.
> 
> MAR: Precisely. I am
> purposely not limiting my thinking to the current
> offered codecs.
> 
> MAR:
> Please note that I am not proposing Class 2 work at this time ...
> my point
> is that some organization in the future could advocate such in
> the future
> and NOT burden the present "Class 1" codec work with (what I
> believe are)
> unnecessary requirements.
> 
> MAR: Personally, I think there will be a need
> for Class 2 codecs soon
> ... but I think it is most important NOT to burden
> the present work
> with "low-runner" or "special-application" requirements.
> The whole
> point of my dissertation was that one codec cannot fit "all
> Internet
> applications". We should focus first on what the WG feels is the
> most
> pressing application - a bandwidth efficient, Internet-friendly,
> human-
> consumable codec. This codec is likely to be model-based. If
> someone
> has a "pet requirement" outside the general Class 1 codec consensus
> -
> let them propose a use case (which would include a market need
>
> statement) and let the WG consider whether that "pet requirement"
> should be
> included in our first work or left for another day.
> 
> Also, class 2 might
> not fall in the interactive
> category (which implies that communication takes
> place between humans).
> By the way, are you training your
> classification-based applications
> while
> taking the distortions caused by
> the transmission system into
> consideration?
> 
> MAR: If the application is
> not real-time interaction ... we would send
> via reliable protocols ... and
> such a codec need not be as robust to
> packet loss.
> 
> MAR: Again I am not
> advocating a Class 2 codec at this time ... but
> apparently you are
> knowledgeable about matched training/testing
> conditions for ASR systems and
> the like.
> 
> You forgot: We need to add CLASS 3 for video and CLASS 4 for
> general
> content, too.
> 
> MAR: No, I didn't. I'll let someone who has
> convinced the WG chairs of
> a market need, a use case, proposed requirements
> for any additional
> classes to be considered. I would hope that a very
> limited number of
> codecs are developed (see next paragraph).
> 
> > Granted,
> we should make each use case as broad as possible so as not
> to
> > have too
> many of them. Once one codec is standardized for a
> particular
> >
> application type - a subsequent candidate codec/application would
> need
> >
> to prove it is "sufficiently different" from the existing codec(s) to
> go
> >
> further. The WG chairs need to exert leadership to ensure the desired
> >
> result (of the smallest number of codecs spanning the greatest
> > application
> space).
> 
> Yes, actually Noll and Jayant have proposed to use µ-Law to
> support
> speech,
> audio
> 
> MAR: This is patently false if you consider
> audio to have content above
> 4 kHz. All natural signals with finite energy (a
> guitar string pluck, a
> glottal speech pulse) have a spectral roll-off with
> increasing
> frequency (e.g., speech has approximately a 6 dB roll-off per
> octave).
> G.711 has a flat distortion characteristic with frequency. This
> means
> if you go out far enough in frequency (depends on the signal) your
> SNR
> turns negative at some point. I have personally interacted with Dr.
>
> Jayant at Bell Labs Murray Hill and I can assure you that he would NOT
>
> advocate G.711 for other than a narrowband speech audio application.
> 
> AND
> video compression.
> 
> MAR: Definitely false ... but would require too long
> an explanation
> (luminance related) ;-). I can't tell if you sarcastic-mode
> is set to
> on.
> 
> But to my understanding it is common consensus
> to
> concentrate on a limited (e.g. class 1) scenario and finished this
> work
>
> first before setting up another construction site.
> 
> MAR: And that is, in
> essence, my proposal - except I have a tweak. If
> you have a "pet
> requirement", I (personally) don't want it to bugger-up
> the process of
> getting to a good a bandwidth efficient, Internet-
> friendly,
> human-consumable codec.
> 
> Christian
> 
> 
>
> _______________________________________________
> codec mailing list
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>
> _______________________________________________
> codec mailing list
>
> [email-masked]
> 
> https://www.ietf.org/mailman/listinfo/codec

_________________________________
> ______________
codec mailing
> list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Brian Rosen <[email-masked]>
Date: Tue, 03 Nov 2009 13:09:24 -0400

Could you cite the research for that?

We did some work in that space and discovered there were no cultural biases;
the timing was a factor in having arguments, interruptions, or fast change
of speaker, where turn taking behavior is not as controlled as it is
usually.  With delay longer than 150, arguments weren't possible naturally,
and behavior suffered significantly.  We only tested a limited range of
cultural differences, but they included US/Canada, Europe and Asia.
Granted, arguments in say, many Asian environments were much less common,
but our work suggested that it still mattered a whole lot.  Of course if the
entire interaction is one way (a lecture with no questions), then delay is
not important as long as lip sync is achieved.

We did not find any advantage of having delay less than around 150.  Of
course, there is a range of human responses, so that 150 is a 95% or so
threshold.  We found the affect of delay is a cliff: once you go over the
edge, you can't converse normally (under some amount of stress), and making
it, say, 250ms vs 400 ms didn't help much.  We noticed no difference under
100ms down to effectively zero.

Lip sync was an issue, but factoring out the delay for the audio, we did not
notice any delay issues for the video itself until it got really big (up
around 1/2 sec).  Then you see some artifacts.

Our research (it was at Marconi, not where I am presently) was not
published, but it was pretty thorough.  We also concluded that delay was
much more important than image quality: in our experiments, a QCIF image
with low delay was preferred overwhelmingly compared to a very high quality
image with lots of delay.

Brian

On 11/3/09 1:46 PM, "Michael Knappe" <[email-masked]> wrote:

> Agreed, thanks.
> 
> One quick note on the 150 ms end-to-end requirement. That is a general
> rule-of-thumb (and ITU specified) requirement for interactive speech
> responsiveness and ease of conversation, but one that digging deeper is
> dependent on language, cultural, and use case factors. There are conference
> situations which are more forgiving to longer delays, and conversely language
> types that require tighter delay to avoid emotional misinterpretation of
> verbal acknowledgements.
> 
> Mike
> 
> 
> 
> To: Roni Even <[email-masked]>; Michael Knappe; [email-masked]
> <[email-masked]>; [email-masked] <[email-masked]>;
> [email-masked] <[email-masked]>

---

From: Gregory Maxwell <[email-masked]>
Date: Tue, 3 Nov 2009 12:52:54 -0800

Michael Knappe wrote:
> One quick note on the 150 ms end-to-end requirement. That is a general rule-of-thumb
> (and ITU specified) requirement for interactive speech responsiveness and ease of
> conversation, but one that digging deeper is dependent on language, cultural, and use
> case factors. There are conference situations which are more forgiving to longer delays,
> and conversely language types that require tighter delay to avoid emotional
> misinterpretation of verbal acknowledgements.

Delay has come up before. A prior post from my cites research which found that 25ms one-way
was the perceptual boundary for interactive music performance: 
http://www.ietf.org/mail-archive/web/codec/current/msg00360.html

One important take away is that while the codec delay is only one cause of delays it is the
*only* cause of delay which the proposed working group would be in a position to influence, 
and some of the other causes of delays are the result of natural law, or other things nearly
as difficult to change (like OS audio APIs).  Since the user is unable to change the speed
of light and may have little control over their network the codec is one of the few places
where delay can be shaved off, for applications that require lower delay.

Some more thoughts on delay…

Minimizing the delay makes it easier to perform echo cancellation and with sufficiently low 
delay (and gain management) echo cancellation can be avoided entirely because the residual 
echo will be masked by room reflections.  This can be important because many echo
cancellation implementations perform somewhat poorly on music, especially in very reverberant
rooms.

There are video codecs which provide sub-frame latency (dirac pro, for example) and although
it is challenging to find low delay video hardware there is interest in subframe latency video
for tele-operating and tele-presence applications.  

Another application where system latency is important is in remote-desktop applications, as
delayed audio can make the system appear less responsive.  (There are other people on this
list who can comment better on the implications to online gaming than I can)

Even if a video system is imposing a particular minimum audio latency,  this does not necessarily
imply that larger audio frames are desirable:  The frame size also implies the minimum loss unit.
Larger frame sizes make random drops look more temporally bursty. With small frame sizes
good loss concealment can provide intelligible speech with random loss rate in excess of 25%,
but this is much harder with larger frames or clumped losses.
(an interesting challenge for internet rate control: if you are seeing loss, do you use larger
packets to reduce rate/overhead, or do you use smaller packets to decrease gap per loss?)

The maximum received codec frame sizes translates directly into codec memory requirements.
While encoders can potentially take advantage of additional lookahead without changing the coding
unit, if larger frames are used the impact on the minimum device size is a potential consideration.

---

From: Alexander Chemeris <[email-masked]>
Date: Wed, 4 Nov 2009 14:21:15 +0300

Hi Gregory,

> where delay can be shaved off, for applications that require lower delay.

Ability to perform well in presence of packet loss can also affect
delay. If you can
efficiently conceal lost packets, you can set lower watermark in your
jitter buffer.

-- 
Regards,
Alexander Chemeris.