Subject: Re: [codec] questions to be answered

From: Michael Knappe <[email-masked]>
Date: Tue, 3 Nov 2009 08:40:18 -0800

As I've been preparing the 'application vs. criteria matrix', it's become clear that video-centric applications like telepresence will have audio codecs that are at least partially tied to or driven by characteristics of the video coding (e.g. frame size). Do we as a group want to tackle additional non-voice considerations in the WG?

Agree with MAR on G.711, logarithmic encoding with a relatively fixed SNR doesn't cut it for monotonically decreasing frequency content sources like speech beyond 3 kHz. That's why G.722 went with a simple two-band approach for better representation of the upper octave, and there are of course many examples today of sub-band approaches.

Mike

----- Original Message -----
From: [email-masked] <[email-masked]>
To: Christian Hoene <[email-masked]>; [email-masked] <[email-masked]>
Sent: Tue Nov 03 10:33:56 2009
Subject: Re: [codec] questions to be answered

Hi Christian,

My replies are in-line below (w/ MAR:).

Thanks for your consideration.

Michael Ramalho

-----Original Message-----
From: Christian Hoene [mailto:[email-masked]] 
Sent: Tuesday, November 03, 2009 4:30 AM
To: Michael Ramalho (mramalho); [email-masked]
Subject: RE: [codec] questions to be answered

Hello Michael,

> At a minimum, I see two legitimate codec classes:
> 
> CLASS 1) codecs that are bandwidth efficient and sound good to humans
> (and perhaps satisfy tendeming or tone pass-through requirements), and
> 
> CLASS 2) codecs whose output may later be processed by sophisticated
> signal processing functions (high-quality mixing, 

High-quality mixing? That do you have in mind? Stereo or binaural (3D sound)
mixing?

MAR: I continued in the email to say that I could potentially think of lots of other examples that require some bounds on the linear distortion because of the signal processing desired at the receiver. But yes, one could conceive of mixing to reproduce the effect of where spatially the sound came from in the sending location using only the signals rendered at the receiver.

> many
> classification-based applications such as speaker/sound/intrusion
> identification) that have SNR/waveform/distortion criteria generally
> exceeding CLASS 1.

You may add distributed speech recognition to the second class. However, to
my understanding the current offered codecs have not been designed with the
second class in mind.

MAR: Precisely. I am purposely not limiting my thinking to the current offered codecs.

MAR: Please note that I am not proposing Class 2 work at this time ... my point is that some organization in the future could advocate such in the future and NOT burden the present "Class 1" codec work with (what I believe are) unnecessary requirements.

MAR: Personally, I think there will be a need for Class 2 codecs soon ... but I think it is most important NOT to burden the present work with "low-runner" or "special-application" requirements. The whole point of my dissertation was that one codec cannot fit "all Internet applications". We should focus first on what the WG feels is the most pressing application - a bandwidth efficient, Internet-friendly, human-consumable codec. This codec is likely to be model-based. If someone has a "pet requirement" outside the general Class 1 codec consensus - let them propose a use case (which would include a market need statement) and let the WG consider whether that "pet requirement" should be included in our first work or left for another day.

Also, class 2 might not fall in the interactive
category (which implies that communication takes place between humans).
By the way, are you training your classification-based applications while
taking the distortions caused by the transmission system into consideration?

MAR: If the application is not real-time interaction ... we would send via reliable protocols ... and such a codec need not be as robust to packet loss.

MAR: Again I am not advocating a Class 2 codec at this time ... but apparently you are knowledgeable about matched training/testing conditions for ASR systems and the like.

You forgot: We need to add CLASS 3 for video and CLASS 4 for general
content, too.

MAR: No, I didn't. I'll let someone who has convinced the WG chairs of a market need, a use case, proposed requirements for any additional classes to be considered. I would hope that a very limited number of codecs are developed (see next paragraph).

> Granted, we should make each use case as broad as possible so as not to
> have too many of them. Once one codec is standardized for a particular
> application type - a subsequent candidate codec/application would need
> to prove it is "sufficiently different" from the existing codec(s) to go
> further. The WG chairs need to exert leadership to ensure the desired
> result (of the smallest number of codecs spanning the greatest
> application space).

Yes, actually Noll and Jayant have proposed to use µ-Law to support speech,
audio

MAR: This is patently false if you consider audio to have content above 4 kHz. All natural signals with finite energy (a guitar string pluck, a glottal speech pulse) have a spectral roll-off with increasing frequency (e.g., speech has approximately a 6 dB roll-off per octave). G.711 has a flat distortion characteristic with frequency. This means if you go out far enough in frequency (depends on the signal) your SNR turns negative at some point. I have personally interacted with Dr. Jayant at Bell Labs Murray Hill and I can assure you that he would NOT advocate G.711 for other than a narrowband speech audio application.

AND video compression.

MAR: Definitely false ... but would require too long an explanation (luminance related) ;-). I can't tell if you sarcastic-mode is set to on.

But to my understanding it is common consensus
to concentrate on a limited (e.g. class 1) scenario and finished this work
first before setting up another construction site.

MAR: And that is, in essence, my proposal - except I have a tweak. If you have a "pet requirement", I (personally) don't want it to bugger-up the process of getting to a good a bandwidth efficient, Internet-friendly, human-consumable codec.

Christian

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: "Roni Even" <[email-masked]>
Date: Tue, 3 Nov 2009 19:09:44 +0200

Mike,
I am not sure what you mean by tied to video codec. I can think of the delay
which can be similar to the video.
The general requirements is support for hands free, full band with stereo
support and optional 5.1 support.
Current codecs used are AAC and G.719.
BTW: there is a different between tele-presence with multiple codecs and
hifh quality HD room systems with one video codec that provide
tele-presence" experience for small rooms.

Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Michael Knappe
> Sent: Tuesday, November 03, 2009 6:40 PM
> To: [email-masked]; [email-masked]; [email-masked]
> As I've been preparing the 'application vs. criteria matrix', it's
> become clear that video-centric applications like telepresence will
> have audio codecs that are at least partially tied to or driven by
> characteristics of the video coding (e.g. frame size). Do we as a group
> want to tackle additional non-voice considerations in the WG?
> 
> Agree with MAR on G.711, logarithmic encoding with a relatively fixed
> SNR doesn't cut it for monotonically decreasing frequency content
> sources like speech beyond 3 kHz. That's why G.722 went with a simple
> two-band approach for better representation of the upper octave, and
> there are of course many examples today of sub-band approaches.
> 
> Mike
> 
> 
> MAR: I continued in the email to say that I could potentially think of
> lots of other examples that require some bounds on the linear
> distortion because of the signal processing desired at the receiver.
> But yes, one could conceive of mixing to reproduce the effect of where
> spatially the sound came from in the sending location using only the
> signals rendered at the receiver.
> 
> > many
> MAR: Please note that I am not proposing Class 2 work at this time ...
> my point is that some organization in the future could advocate such in
> the future and NOT burden the present "Class 1" codec work with (what I
> believe are) unnecessary requirements.
> 
> MAR: Personally, I think there will be a need for Class 2 codecs soon
> ... but I think it is most important NOT to burden the present work
> with "low-runner" or "special-application" requirements. The whole
> point of my dissertation was that one codec cannot fit "all Internet
> applications". We should focus first on what the WG feels is the most
> pressing application - a bandwidth efficient, Internet-friendly, human-
> consumable codec. This codec is likely to be model-based. If someone
> has a "pet requirement" outside the general Class 1 codec consensus -
> let them propose a use case (which would include a market need
> statement) and let the WG consider whether that "pet requirement"
> should be included in our first work or left for another day.
> 
> MAR: If the application is not real-time interaction ... we would send
> via reliable protocols ... and such a codec need not be as robust to
> packet loss.
> 
> MAR: Again I am not advocating a Class 2 codec at this time ... but
> apparently you are knowledgeable about matched training/testing
> conditions for ASR systems and the like.
> 
> MAR: No, I didn't. I'll let someone who has convinced the WG chairs of
> a market need, a use case, proposed requirements for any additional
> classes to be considered. I would hope that a very limited number of
> codecs are developed (see next paragraph).
> 
> MAR: This is patently false if you consider audio to have content above
> 4 kHz. All natural signals with finite energy (a guitar string pluck, a
> glottal speech pulse) have a spectral roll-off with increasing
> frequency (e.g., speech has approximately a 6 dB roll-off per octave).
> G.711 has a flat distortion characteristic with frequency. This means
> if you go out far enough in frequency (depends on the signal) your SNR
> turns negative at some point. I have personally interacted with Dr.
> Jayant at Bell Labs Murray Hill and I can assure you that he would NOT
> advocate G.711 for other than a narrowband speech audio application.
> 
> AND video compression.
> 
> MAR: Definitely false ... but would require too long an explanation
> (luminance related) ;-). I can't tell if you sarcastic-mode is set to
> on.
> 
> MAR: And that is, in essence, my proposal - except I have a tweak. If
> you have a "pet requirement", I (personally) don't want it to bugger-up
> the process of getting to a good a bandwidth efficient, Internet-
> friendly, human-consumable codec.
> 
> Christian
> 
>

---

From: Brian Rosen <[email-masked]>
Date: Tue, 03 Nov 2009 12:23:20 -0400

Right, and the codec doesn't have anything special to match the delay:
that's done elsewhere.

Similarly, there is need for stereo and 5.1 acoustic echo cancel, but that's
not part of the codec either, although we sometimes get tempted to put some
cues in the stream to help the echo cancellation.

There is no attempt to match frame sizes or anything else.  You match delay
to get lip sync right. Everything else is pretty independent.

Of course, you have to keep the delay low ("low" in video terms): under 150
ms one way end to end, including any mixers.  That's usually easy for the
audio, hard for the video, so you artificially delay the audio to match the
video.

Brian

> Mike,
I am not sure what you mean by tied to video codec. I can think of the
> delay
which can be similar to the video.
The general requirements is support
> for hands free, full band with stereo
support and optional 5.1
> support.
Current codecs used are AAC and G.719.
BTW: there is a different
> between tele-presence with multiple codecs and
hifh quality HD room systems
> with one video codec that provide
tele-presence" experience for small
> rooms.

Roni Even

> From: [email-masked]
> [mailto:[email-masked]] On Behalf
> Of Michael Knappe
> Sent: Tuesday,
> As I've
> been preparing the 'application vs. criteria matrix', it's
> become clear that
> video-centric applications like telepresence will
> From:
> [email-masked] <[email-masked]>
> To: Christian Hoene
> <[email-masked]>; [email-masked]
> <[email-masked]>
> Sent: Tue Nov 03
> 10:33:56 2009
> Hi
> Christian,
> From:
> Christian Hoene [mailto:[email-masked]]
> > (and perhaps satisfy tendeming or tone
> pass-through requirements),
> and
> >
> High-quality mixing? That do you have in mind?
> Stereo or binaural (3D
> sound)
> mixing?
> 
> You may add distributed speech recognition to the
> second class.
> However, to
> my understanding the current offered codecs have
> not been designed with
> the
> second class in mind.
> 
> MAR: Precisely. I am
> purposely not limiting my thinking to the current
> offered codecs.
> 
> MAR:
> Please note that I am not proposing Class 2 work at this time ...
> my point
> is that some organization in the future could advocate such in
> the future
> and NOT burden the present "Class 1" codec work with (what I
> believe are)
> should be
> included in our first work or left for another day.
> 
> Also, class 2 might
> By the way, are you training your
> classification-based applications
> while
> taking the distortions caused by
> the transmission system into
> consideration?
> 
> MAR: If the application is
> not real-time interaction ... we would send
> MAR: Again I am not
> advocating a Class 2 codec at this time ... but
> apparently you are
> knowledgeable about matched training/testing
> > Granted,
> we should make each use case as broad as possible so as not
> to
> > have too
> many of them. Once one codec is standardized for a
> particular
> >
> > application
> space).
> Yes, actually Noll and Jayant have proposed to use µ-Law to
> support
> speech,
> audio
> 
> AND
> video compression.
> to
> concentrate on a limited (e.g. class 1) scenario and finished this
> work
>
> MAR: And that is, in
> essence, my proposal - except I have a tweak. If
> you have a "pet
> requirement", I (personally) don't want it to bugger-up
> the process of
> getting to a good a bandwidth efficient, Internet-
> friendly,
> human-consumable codec.
> 
> Christian
> 
> 
>

_________________________________
> ______________
codec mailing
> list
[email-masked]

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 03 Nov 2009 13:12:28 -0500

Michael Knappe wrote:
> become clear that video-centric applications like telepresence will have
> audio codecs that are at least partially tied to or driven by
> characteristics of the video coding (e.g. frame size). 

Actually, why would the frame size of the codec be related to the video 
codec? I don't see any valid reason for doing that and I would have a hard 
time imagining how one would have an audio codec with 29.97 frames per 
second (without using a sampling rate of 59.94 kHz).

 > Do we as a group

(I assume you mean non-audio considerations because we definitely want to 
consider music here) I'm not fundamentally opposed to the idea of including 
non-audio considerations, but there would have to be a good reason for 
that. Right now I cannot see any, but I may have missed something.

	Jean-Marc