Subject: Re: [codec] questions to be answered

From: Michael Knappe <[email-masked]>
Date: Mon, 2 Nov 2009 07:48:56 -0800

Roni,

As per my email last Friday, I will have a draft of that table out today for comment.

Cheers,

Mike

----- Original Message -----
From: [email-masked] <[email-masked]>
To: 'Jean-Marc Valin' <[email-masked]>; 'Michael Ramalho (mramalho)' <[email-masked]>
Cc: [email-masked] <[email-masked]>
Sent: Mon Nov 02 10:11:10 2009
Subject: Re: [codec] questions to be answered

Hi Jean-Marc,
I looked at section 2 and it does not provide enough information why you
need multiple codecs
I would like to start with editorial comment. If you want to show
requirements you should start with the parameters by which you define the
codec like bit rate , sampling rate, hands free support,... and build a
table that will include all the values for each application you think should
be addressed. This will enable to justify the need for different codecs and
to be able to qualify a codec for application. I think that having such
information deserves a separate requirement document for each application.
The current text has some values for some of the examples and some do not
have any values. 

Regards
Roni Even

> -----Original Message-----
> From: Jean-Marc Valin [mailto:[email-masked]]
> Sent: Monday, November 02, 2009 4:24 PM
> To: Michael Ramalho (mramalho)
> Cc: Roni Even; Peter Saint-Andre; [email-masked]
> Subject: Re: [codec] questions to be answered
> 
> Hi Michael,
> 
> Defining the requirements for each set of applications is exactly what
> we've been trying to do in section 2 of the requirements draft. Maybe
> we
> just need to expand this section with more detailed requirements.
> 
> > Example 1 - Let's assume I have an application that requires a
> certain
> > linearity within a certain BW ranges (need > 25 dB of waveform
> linearity
> > in the band 0 - 4k, > 20 db from 4k - 8k, etc). Perhaps this
> application
> > uses adaptive signal processing techniques and linearity is
> important.
> > Perhaps it is a far-end echo cancellation application, or a
> microphone
> > beam array processing application that processes multiple streams at
> the
> > receiver end.
> 
> Well, right now there is no application that requires waveform
> similarity
> in the draft, so if that is needed, we should add a section to the
> requirements draft.
> 
> > I could go on and on ... but I sincerely doubt that ONE MAGICAL codec
> > can satisfy both examples above ... because both APPLICATIONS would
> > result in different REQUIREMENTS.
> >
> > I think the rush to a "single-codec" requirements here is premature
> ...
> > I agree with Roni ... one size does not fit all.
> 
> Indeed, I am now thinking that we really shouldn't set a maximum number
> of
> codecs for the proposed WG until we have a complete list of target
> applications and requirements.
> 
> > Given the simple examples above - isn't it clear that we need to
> define
> > applications (one might call them use cases) - and then requirements
> for
> > them - before we determine how a given candidate satisfies the
> > requirements?
> 
> As I said, this is really the intent of Section 2 of the requirements
> draft. Though I agree it can probably be improved with more
> requirements
> and more details on the existing ones.
> 
> > CLASS 1) codecs that are bandwidth efficient and sound good to humans
> > (and perhaps satisfy tendeming or tone pass-through requirements),
> and
> 
> It seems like all the applications we have so far belong to this class.
> 
> > CLASS 2) codecs whose output may later be processed by sophisticated
> > signal processing functions (high-quality mixing, many
> > classification-based applications such as speaker/sound/intrusion
> > identification) that have SNR/waveform/distortion criteria generally
> > exceeding CLASS 1.
> 
> Can you formulate uses cases for that one?
> 
> My personal opinion is that with some work it should possible to
> satisfy
> both classes you list above with a single codec (possibly with
> different
> encoding options), but of course that remains to be verified in
> practice.
> 
> > Granted, we should make each use case as broad as possible so as not
> to
> > have too many of them. Once one codec is standardized for a
> particular
> > application type - a subsequent candidate codec/application would
> need
> > to prove it is "sufficiently different" from the existing codec(s) to
> go
> > further. The WG chairs need to exert leadership to ensure the desired
> > result (of the smallest number of codecs spanning the greatest
> > application space).
> 
> I don't think mapping applications to different codecs is something
> that
> can be done in advance. In fact, I wouldn't be too surprised if we
> ended up
>   with some classes of applications needing two codecs and/or having
> several applications covered by the same codec.
> 
> > IF
> > a given codec meets the requirements of its use case
> > AND
> > that use case is valid
> > THEN
> > I (for one) don't have an issue with the IETF "rubber stamping" of
> it.
> 
> Wouldn't that lead to more codecs than necessary?
> 
> > PS - I think CLASS 1 codecs should be addressed first by the WG ...
> and
> > later consider other classes as people advocate for their use case.
> 
> Well, your two classes above represents a split along one possible
> dimension, which is waveform vs perceptual-only approach. There are
> many
> more possible splits, for instance: layered vs non-layered, algorithmic
> delay, VBR vs CBR, speech vs music, fullband vs narrowband, ...
> 
> Having one codec for each is neither desirable, nor necessary. We will
> need
> to take a more global approach to split things up. Otherwise, one can
> argue
> that all the already-submitted codecs will fall into a different class
> a
> can be rubberstamped on its own. And if we do that, Roni will not be
> happy
> :-) Nor will I.
> 
> Cheers,
> 
> 	Jean-Marc

_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Michael Knappe <[email-masked]>
Date: Tue, 3 Nov 2009 09:33:37 -0800

Frame size is the first thing that came to mind, but I was unsure as if there may be other considerations that I was unaware of for video and was hoping for some feedback there.

Obviously you could pack and synchronize multiple short audio frames along with a longer duration video frame, but a codec with potential compression quality improvements with longer lookahead audio frames could take better advantage of the allowed video framing time.

This is just an example to explore  Michael Ramalho's discussion of how far we should go to accomodate different application spaces.

Roni, for clarification, can you elaborate further on your 'BTW...' discussion of different telepresence codecs below? Thanks.

Cheers,

Mike

----- Original Message -----
From: Roni Even <[email-masked]>
To: Michael Knappe; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Sent: Tue Nov 03 12:09:44 2009
Subject: RE: [codec] questions to be answered

Mike,
I am not sure what you mean by tied to video codec. I can think of the delay
which can be similar to the video.
The general requirements is support for hands free, full band with stereo
support and optional 5.1 support.
Current codecs used are AAC and G.719.
BTW: there is a different between tele-presence with multiple codecs and
hifh quality HD room systems with one video codec that provide
tele-presence" experience for small rooms.

Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Michael Knappe
> Sent: Tuesday, November 03, 2009 6:40 PM
> To: [email-masked]; [email-masked]; [email-masked]
> As I've been preparing the 'application vs. criteria matrix', it's
> become clear that video-centric applications like telepresence will
> have audio codecs that are at least partially tied to or driven by
> characteristics of the video coding (e.g. frame size). Do we as a group
> want to tackle additional non-voice considerations in the WG?
> 
> Agree with MAR on G.711, logarithmic encoding with a relatively fixed
> SNR doesn't cut it for monotonically decreasing frequency content
> sources like speech beyond 3 kHz. That's why G.722 went with a simple
> two-band approach for better representation of the upper octave, and
> there are of course many examples today of sub-band approaches.
> 
> Mike
> 
> 
> To: Christian Hoene <[email-masked]>; [email-masked]
> <[email-masked]>
> Sent: Tue Nov 03 10:33:56 2009
> Hi Christian,
> My replies are in-line below (w/ MAR:).
> 
> Thanks for your consideration.
> 
> Michael Ramalho
> 
> From: Christian Hoene [mailto:[email-masked]]
> Sent: Tuesday, November 03, 2009 4:30 AM
> To: Michael Ramalho (mramalho); [email-masked]
> Subject: RE: [codec] questions to be answered
> 
> Hello Michael,
> 
> > At a minimum, I see two legitimate codec classes:
> >
> > signal processing functions (high-quality mixing,
> 
> High-quality mixing? That do you have in mind? Stereo or binaural (3D
> sound)
> mixing?
> 
> MAR: I continued in the email to say that I could potentially think of
> lots of other examples that require some bounds on the linear
> distortion because of the signal processing desired at the receiver.
> But yes, one could conceive of mixing to reproduce the effect of where
> spatially the sound came from in the sending location using only the
> signals rendered at the receiver.
> 
> > many
> You may add distributed speech recognition to the second class.
> However, to
> my understanding the current offered codecs have not been designed with
> the
> second class in mind.
> 
> MAR: Precisely. I am purposely not limiting my thinking to the current
> offered codecs.
> 
> MAR: Please note that I am not proposing Class 2 work at this time ...
> my point is that some organization in the future could advocate such in
> the future and NOT burden the present "Class 1" codec work with (what I
> believe are) unnecessary requirements.
> 
> MAR: Personally, I think there will be a need for Class 2 codecs soon
> ... but I think it is most important NOT to burden the present work
> with "low-runner" or "special-application" requirements. The whole
> point of my dissertation was that one codec cannot fit "all Internet
> applications". We should focus first on what the WG feels is the most
> pressing application - a bandwidth efficient, Internet-friendly, human-
> consumable codec. This codec is likely to be model-based. If someone
> has a "pet requirement" outside the general Class 1 codec consensus -
> let them propose a use case (which would include a market need
> statement) and let the WG consider whether that "pet requirement"
> should be included in our first work or left for another day.
> 
> Also, class 2 might not fall in the interactive
> category (which implies that communication takes place between humans).
> By the way, are you training your classification-based applications
> while
> taking the distortions caused by the transmission system into
> consideration?
> 
> MAR: If the application is not real-time interaction ... we would send
> via reliable protocols ... and such a codec need not be as robust to
> packet loss.
> 
> MAR: Again I am not advocating a Class 2 codec at this time ... but
> apparently you are knowledgeable about matched training/testing
> conditions for ASR systems and the like.
> 
> You forgot: We need to add CLASS 3 for video and CLASS 4 for general
> content, too.
> 
> MAR: No, I didn't. I'll let someone who has convinced the WG chairs of
> a market need, a use case, proposed requirements for any additional
> classes to be considered. I would hope that a very limited number of
> codecs are developed (see next paragraph).
> 
> Yes, actually Noll and Jayant have proposed to use µ-Law to support
> speech,
> audio
> 
> MAR: This is patently false if you consider audio to have content above
> 4 kHz. All natural signals with finite energy (a guitar string pluck, a
> glottal speech pulse) have a spectral roll-off with increasing
> frequency (e.g., speech has approximately a 6 dB roll-off per octave).
> G.711 has a flat distortion characteristic with frequency. This means
> if you go out far enough in frequency (depends on the signal) your SNR
> turns negative at some point. I have personally interacted with Dr.
> Jayant at Bell Labs Murray Hill and I can assure you that he would NOT
> advocate G.711 for other than a narrowband speech audio application.
> 
> AND video compression.
> 
> MAR: Definitely false ... but would require too long an explanation
> (luminance related) ;-). I can't tell if you sarcastic-mode is set to
> on.
> 
> But to my understanding it is common consensus
> to concentrate on a limited (e.g. class 1) scenario and finished this
> work
> first before setting up another construction site.
> 
> MAR: And that is, in essence, my proposal - except I have a tweak. If
> you have a "pet requirement", I (personally) don't want it to bugger-up
> the process of getting to a good a bandwidth efficient, Internet-
> friendly, human-consumable codec.
> 
> Christian
> 
>

---

From: Michael Knappe <[email-masked]>
Date: Tue, 3 Nov 2009 10:50:08 -0800

The language sensitivity was empirical from work in early voip deployments I was involved with in international markets. Always made an attempt to stay below 100 ms if at all possible. On large deployments, the 5 percent 'outliers' become very important in terms of customer acceptance of new technology (new at the time). You could counter that since the early days of voip, cell phone quality has changed the delay acceptability bar as well with longer delays becoming more the norm. What I really like about the proposed codec activity is moving internet communications towards being the high-quality choice (frequency range, number of channels, etc) rather than the low-cost quality compromise that voip began its life as.

The 150 ms 'knee' was exactly that, with the impact on conversation grew increasingly until you hit the 'CB radio' half-duplex asymptote past the 400-600 ms range. Agreed that below 150 ms, the quality impact curve quickly for the average listener/speaker levels out quickly. You raise an interesting point about argumentative speech and low delay, involved multi-party conference communications should keep delay as low as possible.

Codec algorithmic delay is of course just a fraction of the end-to-end delay equation, with the relative percent contribution of network quality (both absolute delay and jitter) increasing as you optimize codec delay. Beyond the scope of the proposed WG, but worth keeping in mind in assessing the impact of codec delay on its own.

Mike

----- Original Message -----
From: Brian Rosen <[email-masked]>
To: Michael Knappe; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Sent: Tue Nov 03 12:09:24 2009
Subject: Re: [codec] questions to be answered

Could you cite the research for that?

We did some work in that space and discovered there were no cultural biases;
the timing was a factor in having arguments, interruptions, or fast change
of speaker, where turn taking behavior is not as controlled as it is
usually.  With delay longer than 150, arguments weren't possible naturally,
and behavior suffered significantly.  We only tested a limited range of
cultural differences, but they included US/Canada, Europe and Asia.
Granted, arguments in say, many Asian environments were much less common,
but our work suggested that it still mattered a whole lot.  Of course if the
entire interaction is one way (a lecture with no questions), then delay is
not important as long as lip sync is achieved.

We did not find any advantage of having delay less than around 150.  Of
course, there is a range of human responses, so that 150 is a 95% or so
threshold.  We found the affect of delay is a cliff: once you go over the
edge, you can't converse normally (under some amount of stress), and making
it, say, 250ms vs 400 ms didn't help much.  We noticed no difference under
100ms down to effectively zero.

Lip sync was an issue, but factoring out the delay for the audio, we did not
notice any delay issues for the video itself until it got really big (up
around 1/2 sec).  Then you see some artifacts.

Our research (it was at Marconi, not where I am presently) was not
published, but it was pretty thorough.  We also concluded that delay was
much more important than image quality: in our experiments, a QCIF image
with low delay was preferred overwhelmingly compared to a very high quality
image with lots of delay.

Brian

On 11/3/09 1:46 PM, "Michael Knappe" <[email-masked]> wrote:

> Agreed, thanks.
> 
> One quick note on the 150 ms end-to-end requirement. That is a general
> rule-of-thumb (and ITU specified) requirement for interactive speech
> responsiveness and ease of conversation, but one that digging deeper is
> dependent on language, cultural, and use case factors. There are conference
> situations which are more forgiving to longer delays, and conversely language
> types that require tighter delay to avoid emotional misinterpretation of
> verbal acknowledgements.
> 
> Mike
> 
> 
> 
> From: Brian Rosen <[email-masked]>
> To: Roni Even <[email-masked]>; Michael Knappe; [email-masked]
> <[email-masked]>; [email-masked] <[email-masked]>;
> [email-masked] <[email-masked]>
> Sent: Tue Nov 03 11:23:20 2009
> Right, and the codec doesn't have anything special to match the delay:
> that's done elsewhere.
> 
> Similarly, there is need for stereo and 5.1 acoustic echo cancel, but that's
> not part of the codec either, although we sometimes get tempted to put some
> cues in the stream to help the echo cancellation.
> 
> There is no attempt to match frame sizes or anything else.  You match delay
> to get lip sync right. Everything else is pretty independent.
> 
> Of course, you have to keep the delay low ("low" in video terms): under 150
> ms one way end to end, including any mixers.  That's usually easy for the
> audio, hard for the video, so you artificially delay the audio to match the
> video.
> 
> Brian
> 
> 
> On 11/3/09 1:09 PM, "Roni Even" <[email-masked]> wrote:
> 
>> Mike,
> BTW: there is a different
>> between tele-presence with multiple codecs and
>> From: [email-masked]
>> [mailto:[email-masked]] On Behalf
>> Of Michael Knappe
>> Sent: Tuesday,
>> As I've
>> been preparing the 'application vs. criteria matrix', it's
>> become clear that
>> video-centric applications like telepresence will
>> From:
>> [email-masked] <[email-masked]>
>> To: Christian Hoene
>> Hi
>> Christian,
>> From:
>> Christian Hoene [mailto:[email-masked]]
>> MAR: Precisely. I am
>> purposely not limiting my thinking to the current
>> offered codecs.
>> 
>> MAR:
>> Please note that I am not proposing Class 2 work at this time ...
>> my point
>> is that some organization in the future could advocate such in
>> the future
>> and NOT burden the present "Class 1" codec work with (what I
>> believe are)
>> should be
>> included in our first work or left for another day.
>> 
>> Also, class 2 might
>> MAR: If the application is
>> not real-time interaction ... we would send
>> MAR: Again I am not
>> advocating a Class 2 codec at this time ... but
>> apparently you are
>> knowledgeable about matched training/testing
>>> Granted,
>> we should make each use case as broad as possible so as not
>> to
>>> have too
>> many of them. Once one codec is standardized for a
>> particular
>>> 
>> space).
>> video compression.
>> concentrate on a limited (e.g. class 1) scenario and finished this
>> work
>> 
>> MAR: And that is, in
>> essence, my proposal - except I have a tweak. If
>> you have a "pet
>> requirement", I (personally) don't want it to bugger-up
>> the process of
>> getting to a good a bandwidth efficient, Internet-
>> friendly,
>> human-consumable codec.
>> 
>> Christian
>> 
>> 
>> 
> codec mailing
>> list

---

From: Michael Knappe <[email-masked]>
Date: Tue, 3 Nov 2009 10:59:16 -0800

Of course codecs have not been tied directly to the 15 / 30 fps video requirements from an evenly divisible sample rate perspective, but G.723.1 (30 ms) was closely associated with video conferencing. This was just an example of a codec that structured its processing around longer frame times.

Enough said, I think its clear from discussion that we can keep audio and video requirements separate for the purpose of this effort. Simplifying the problem space is a good thing.

Mike

----- Original Message -----
From: Jean-Marc Valin <[email-masked]>
To: Michael Knappe
Cc: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Sent: Tue Nov 03 13:12:28 2009
Subject: Re: [codec] questions to be answered

Michael Knappe wrote:
> become clear that video-centric applications like telepresence will have
> audio codecs that are at least partially tied to or driven by
> characteristics of the video coding (e.g. frame size). 

Actually, why would the frame size of the codec be related to the video 
codec? I don't see any valid reason for doing that and I would have a hard 
time imagining how one would have an audio codec with 29.97 frames per 
second (without using a sampling rate of 59.94 kHz).

 > Do we as a group

(I assume you mean non-audio considerations because we definitely want to 
consider music here) I'm not fundamentally opposed to the idea of including 
non-audio considerations, but there would have to be a good reason for 
that. Right now I cannot see any, but I may have missed something.

	Jean-Marc