Subject: [codec] Agenda for the Codec Bof

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 27 Jul 2009 10:37:58 +0200

Hi

I am a bit curious regarding the agenda for the codec BoF
Is this the latest ?
http://svn.resiprocate.org/rep/ietf-drafts/codec-bof/agenda.txt
I have probably missed a lot as I have been on vacation but I got the (false?) impression that it was not the intention to present any codecs and instead focus on the question "should IETF do this?"

Regards
/Ingemar

******************************************* 
Ingemar Johansson 
Senior Research Engineer, IETF "nethead" 
EAB/TVK - Multimedia Technologies 
Ericsson Research Ericsson AB 
Box 920 S-971 28 Luleå, Sweden 
Tel: +46 (0)10 7143042 
ECN: 852-43042 
ECC: 852-19042
Mobile: +46 (0)730 783289 
Visit http://labs.ericsson.com !
*******************************************

---

From: Herve Taddei <[email-masked]>
Date: Mon, 27 Jul 2009 10:47:51 +0200

Dear Ingemar, All,

I have not been on vacation yet, and you are right, I don’t think the agenda
reflects the discussions/concerns raised by many people on the email list. I
am also a bit surprised.

Best regards,
Hervé

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Ingemar Johansson S
> Sent: Monday, July 27, 2009 10:38 AM
> To: [email-masked]
> Subject: [codec] Agenda for the Codec Bof
> 
> Hi
> 
> I have probably missed a lot as I have been on vacation but I got the
(false?)
> impression that it was not the intention to present any codecs and instead
focus on
> the question "should IETF do this?"
> 
> 
> Regards
> /Ingemar
> 
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 27 Jul 2009 10:52:32 +0200

Hi,

Indeed, the link has not been updated. The updated agenda was posted here:
http://www.ietf.org/mail-archive/web/codec/current/msg00241.html

Regards,

	Jean-Marc

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 27 Jul 2009 10:52:51 +0200

> I have probably missed a lot as I have been on vacation but I got the (false?) impression that it was not the intention to present any codecs and instead focus on the question "should IETF do this?"

Please do look at the official agenda:

http://www.ietf.org/proceedings/75/agenda/codec.txt

Cheers

Magnus Westerlund

IETF Transport Area Director
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 27 Jul 2009 10:56:15 +0200

OK

Thanks!

/Ingemar 

> From: Magnus Westerlund 
> Sent: den 27 juli 2009 10:53
> To: Ingemar Johansson S
> Cc: [email-masked]
> Subject: Re: [codec] Agenda for the Codec Bof
> 
> Is this the 
> > latest ?
> > I have probably missed a lot as I have been on vacation but 
> I got the (false?) impression that it was not the intention 
> to present any codecs and instead focus on the question 
> "should IETF do this?"
>

---

From: Herve Taddei <[email-masked]>
Date: Mon, 27 Jul 2009 12:12:50 +0200

Dear Jean Marc,

Indeed the agenda was changed, but still the introduction of codecs is
there.

Best regards,
Hervé

> Jean-Marc Valin
> Sent: Monday, July 27, 2009 10:53 AM
> To: Ingemar Johansson S
> Cc: [email-masked]
(false?)
focus on

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 27 Jul 2009 15:43:40 +0200

Hi

I believe that I have expressed earlier that I don't believe that a description of particular codecs gives any additional value, speech/audio coding is a very specialized area and going beyond a descripion of a few "black boxes" in 25 minutes is probably more than the average newbie can digest. This is of course my personal opinion and if you believe that it is important to have his then I respect it.

Regards
Ingemar

> From: Herve Taddei [mailto:[email-masked]] 
> Sent: den 27 juli 2009 12:13
> To: 'Jean-Marc Valin'; Ingemar Johansson S
> Cc: [email-masked]
> Subject: RE: [codec] Agenda for the Codec Bof
> 
> Dear Jean Marc,
> 
> Indeed the agenda was changed, but still the introduction of 
> codecs is there.
> 
> Best regards,
> Hervé
> 
> 
> 
> > From: [email-masked] 
> [mailto:[email-masked]] On Behalf 
> > Of Jean-Marc Valin
> > > I am a bit curious regarding the agenda for the codec BoF Is this 
> > > the latest ?

---

From: Patrick Luthi <[email-masked]>
Date: Mon, 27 Jul 2009 15:57:05 +0200

Hi,

I am surprised to still see codecs listed in item 
4 of the agenda. As I said before I think that it 
would be best to have more time to discuss the 
other items of the agenda. So, please remove the 
codec presentations from the agenda!

Regards,

Patrick

At 15:43 27/07/2009, Ingemar Johansson S wrote:
>Hi
>
>I believe that I have expressed earlier that I 
>don't believe that a description of particular 
>codecs gives any additional value, speech/audio 
>coding is a very specialized area and going 
>beyond a descripion of a few "black boxes" in 25 
>minutes is probably more than the average newbie 
>can digest. This is of course my personal 
>opinion and if you believe that it is important to have his then I respect it.
>
>Regards
>Ingemar
>
>

---

From: "Christian Hoene" <[email-masked]>
Date: Mon, 27 Jul 2009 16:00:50 +0200

> Patrick Luthi
> Sent: Monday, July 27, 2009 3:57 PM
> To: Ingemar Johansson S; Herve Taddei; Jean-Marc Valin
> Cc: [email-masked]
Hi,

> codec presentations from the agenda!

I support this request.

Regards,

 Christian

> >opinion and if you believe that it is important to have his then I
respect
> it.

---

From: Henry Sinnreich <[email-masked]>
Date: Mon, 27 Jul 2009 08:16:37 -0700

The hum item is not fair nor representative IMO, since it favors people who have a travel budget.

>7. HUM on if the IETF should form this WG (All, 15 min)

I would like to understand how/why the hum is representative for the Internet industry,
since it looks humming disadvantages the little guys, exactly those who depend on free codecs.

My personal 2 cents.
Henry

On 7/27/09 10:52 AM, "Jean-Marc Valin" <[email-masked]> wrote:

Hi,

Regards,

        Jean-Marc

---

From: stephen botzko <[email-masked]>
Date: Mon, 27 Jul 2009 11:17:06 -0400

As do I

Regards,
Stephen Botzko
Polycom

---

From: stephen botzko <[email-masked]>
Date: Mon, 27 Jul 2009 11:25:44 -0400

Hi Henry

I believe that hums include chatroom support/dissent.

Regards,
Stephen Botzko
Polycom

On Mon, Jul 27, 2009 at 11:16 AM, Henry Sinnreich <[email-masked]>wrote:

> (false?) impression that it was not the intention to present any codecs and
> instead focus on the question "should IETF do this?"
> >
> >
> > Regards
> > /Ingemar
> >

---

From: Henry Sinnreich <[email-masked]>
Date: Mon, 27 Jul 2009 08:54:46 -0700

The little guys on the Internet who depend on free codecs will still not be represented.
Even if they turn out to be in minority.

I table for this reason to delete the humming item, or find a better way to safeguard their dependence on free codecs.
Take a look for example on the lists of users for free codecs - no names mentioned here.

Henry

On 7/27/09 5:25 PM, "stephen botzko" <[email-masked]> wrote:

Hi Henry

Regards,
Stephen Botzko
Polycom

On Mon, Jul 27, 2009 at 11:16 AM, Henry Sinnreich <[email-masked]> wrote:
The hum item is not fair nor representative IMO, since it favors people who have a travel budget.

My personal 2 cents.
Henry

On 7/27/09 10:52 AM, "Jean-Marc Valin" <[email-masked] <http://[email-masked]> > wrote:

Hi,

Regards,

        Jean-Marc

Ingemar Johansson S a écrit :
> [email-masked] <http://[email-masked]>
_______________________________________________
codec mailing list
[email-masked] <http://[email-masked]>
https://www.ietf.org/mailman/listinfo/codec

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 27 Jul 2009 18:12:06 +0200

Hi
 
I would believe that the hum procedure is quite well established and controlled and I believe that other people with more IETF experience than me will make sure that it is handled in a fair way regarless if people are physically present on the meeting or not. 
 
/Ingemar 

________________________________

	From: Henry Sinnreich [mailto:[email-masked]] 
	Sent: den 27 juli 2009 17:55
	To: stephen botzko
	Cc: Jean-Marc Valin; Ingemar Johansson S; [email-masked]
	Subject: Re: [codec] Agenda for the Codec Bof

	
	Henry

		Hi Henry

		Regards,
		Stephen Botzko
		Polycom

			

			My personal 2 cents.
			Henry

				Hi,

				Regards,
				
				        Jean-Marc

---

From: stephen botzko <[email-masked]>
Date: Mon, 27 Jul 2009 12:22:43 -0400

Folks who participate in person or via the chatroom are represented.  Folks
who choose *not* to participate are *not* represented.  This is standard
practice in the IETF, there is no other way to decide things here.

Regards,
Stephen Botzko
Polycom

On Mon, Jul 27, 2009 at 11:54 AM, Henry Sinnreich <[email-masked]>wrote:

> I table for this reason to delete the humming item, or find a better way to
> safeguard their dependence on free codecs.
> Take a look for example on the lists of users for free codecs – no names
> mentioned here.
>
> Henry
>
>
> On 7/27/09 10:52 AM, "Jean-Marc Valin" <[email-masked] <
> http://[email-masked]> > wrote:
>
> Hi,
>

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 27 Jul 2009 10:10:27 -0700

Hi Henry,
You are really an old hand in the IETF business.  Do you recall a single BOF
where there was no ³humm², or ³show of hands², or similar exercise, to gauge
the readiness of the IETF to take on the work?  My personal recollection is
that a humm is the norm.
Regards,
Stephan

On 7/27/09 8:54 AM, "Henry Sinnreich" <[email-masked]> wrote:

> Take a look for example on the lists of users for free codecs  no names
> mentioned here.
> 
> Henry
> 
> 
>>> On 7/27/09 10:52 AM, "Jean-Marc Valin" <[email-masked]
>>> <http://[email-masked]> > wrote:
>>> 
>>>> Hi,
>>>> 
>>>>> (false?) impression that it was not the intention to present any codecs
>>>>> and instead focus on the question "should IETF do this?"
>>>>> >
>>>>> >
>>>>> > Regards
>>>>> > /Ingemar
>>>>> >

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 27 Jul 2009 10:17:56 -0700

With respect to all this discussion, please note that there is an agenda
item ³agenda bashing² at the very beginning of the BOF.  While I personally
do not consider it to be overly productive to augment an (at this point)
useless presentation of two randomly chosen codec candidates with an even
more useless meta discussion on the agenda, I fear that¹s precisely what¹s
going to happen...
Perhaps we should fix this whole issue by removing both agenda items :-)
Regards,
Stephan

On 7/27/09 9:22 AM, "stephen botzko" <[email-masked]> wrote:

> who choose not to participate are not represented.  This is standard practice
> in the IETF, there is no other way to decide things here.
> 
> Regards,
> Stephen Botzko
> Polycom
> 
> On Mon, Jul 27, 2009 at 11:54 AM, Henry Sinnreich <[email-masked]> wrote:
>> On 7/27/09 5:25 PM, "stephen botzko" <[email-masked]
>> <http://[email-masked]> > wrote:
>> 
>>> Hi Henry
>>> 
>>> On Mon, Jul 27, 2009 at 11:16 AM, Henry Sinnreich <[email-masked]
>>> <http://[email-masked]> > wrote:
>>>> <http://[email-masked]>
>>>>>> > [email-masked] <http://[email-masked]>  <http://[email-masked]>
>>>>> [email-masked] <http://[email-masked]>  <http://[email-masked]>

---

From: Greg Herlein <[email-masked]>
Date: Mon, 27 Jul 2009 10:28:31 -0700

For 'little guys' like me... where is the hum documented such that I could
participate in a chat room if I wanted?  I've lurked (and sometimes actively
participated) on this list for years and only recently learned that there
was such a way to participate.

Greg

On Mon, Jul 27, 2009 at 9:12 AM, Ingemar Johansson S <
[email-masked]> wrote:

>  Hi
>
> I would believe that the hum procedure is quite well established and
> controlled and I believe that other people with more IETF experience than me
> will make sure that it is handled in a fair way regarless if people are
> physically present on the meeting or not.
>
> /Ingemar
>
>  ------------------------------
> *From:* Henry Sinnreich [mailto:[email-masked]]
> *Sent:* den 27 juli 2009 17:55
> *To:* stephen botzko
> *Cc:* Jean-Marc Valin; Ingemar Johansson S; [email-masked]
> *Subject:* Re: [codec] Agenda for the Codec Bof
>

-- 
Greg Herlein
www.herlein.com

---

From: Henry Sinnreich <[email-masked]>
Date: Mon, 27 Jul 2009 10:39:36 -0700

I agree we should raise the "delete hum" in the agenda bashing.
Humming does not represent the little guys who depend on free codecs.

Dear Chairs: Can you accommodate this?

Henry

On 7/27/09 7:17 PM, "Stephan Wenger" <[email-masked]> wrote:

With respect to all this discussion, please note that there is an agenda item "agenda bashing" at the very beginning of the BOF.  While I personally do not consider it to be overly productive to augment an (at this point) useless presentation of two randomly chosen codec candidates with an even more useless meta discussion on the agenda, I fear that's precisely what's going to happen...
Perhaps we should fix this whole issue by removing both agenda items :-)
Regards,
Stephan

Folks who participate in person or via the chatroom are represented.  Folks who choose not to participate are not represented.  This is standard practice in the IETF, there is no other way to decide things here.

Regards,
Stephen Botzko
Polycom

Henry

On 7/27/09 5:25 PM, "stephen botzko" <[email-masked] <http://[email-masked]> > wrote:

Hi Henry

Regards,
Stephen Botzko
Polycom

On Mon, Jul 27, 2009 at 11:16 AM, Henry Sinnreich <[email-masked] <http://[email-masked]> > wrote:
The hum item is not fair nor representative IMO, since it favors people who have a travel budget.

My personal 2 cents.
Henry

On 7/27/09 10:52 AM, "Jean-Marc Valin" <[email-masked] <http://[email-masked]>  <http://[email-masked]> > wrote:

Hi,

Regards,

        Jean-Marc

---

From: Stefan Sayer <[email-masked]>
Date: Mon, 27 Jul 2009 21:36:51 +0200

Hi

o Patrick Luthi [07/27/09 15:57]:
> Hi,
> 
> I am surprised to still see codecs listed in item 4 of the agenda. As I 
> said before I think that it would be best to have more time to discuss 
> the other items of the agenda. So, please remove the codec presentations 
> from the agenda!
in the beginning it was questioned whether
a) there is enough expertise in the IETF to develop an IWAC
b) there is suitable RF codec technology available.

If now there is - thanks to the existence of codec candidates - 
consensus that both a and b are met then it probably does not make sense 
to present the candidates themselves.

Best Regards
Stefan Sayer

---

From: "Roni Even" <[email-masked]>
Date: Tue, 28 Jul 2009 00:48:30 +0300

Hi,
Having two codec suggested does not represent expertise to review codec
work. The IETF "rubber stamped" the iLBC codec. There was a proposed codec,
which you consider expertise, but no capability to do a real review and
quality assessment
Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Stefan Sayer
> Sent: Monday, July 27, 2009 10:37 PM
> To: Patrick Luthi
> Cc: [email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Tue, 28 Jul 2009 08:22:50 +0200

Hi,

> which you consider expertise, but no capability to do a real review  

agree with Roni. Having codec specs available that (apparently) fit  
the requirements is nice, but it's not clear at all that that means  
that the experts who have created these codecs will actively  
participate in an IETF effort that would be based on something other  
than their initial candidate submission.

Lars

---

From: "Gregory M. Lebovitz" <[email-masked]>
Date: Tue, 28 Jul 2009 02:05:55 -0700

Folks,
Let's look again at the charter's text, as posted on the current Agenda at:
<http://tools.ietf.org/agenda/75/codec.html>http://tools.ietf.org/agenda/75/codec.html 

"4. What type of engineering work would this be? ( 25 min )
Have presentations on the two proposed codecs. Focus is to make sure it
looks feasible to do the technical work to meet the goals. This 
should also help clarify what type of work is being considered."

Perhaps the sentence should change as:
  s/presentations on the two proposed codecs/presentations on two 
example codecs
??

So, less than these being discussions of these particular codecs as 
candidates or lead candidates, these two are just serving as 
archetypical models of the type of codec we might see if the work 
progresses, SO THAT THE BoF PARTICIPANTS CAN DISCUSS HOW WE WILL DO 
TECHNICAL work. The goal is NOT to do the technical work on the 
codecs at this BoF. This is not a beauty contest of these codecs. 
It's more like "Here are two specimens of this species. Let's note 
some characteristics about this species. Now, how would we go about 
analyzing this species further?" Versus "I like the left specimen 
because he seems bigger and softer and darker. Can I take it home?" 
We aren't doing the latter.

The idea should be to say, "Okay, we've seen two examples. We aren't 
going to dig into those two codecs in particular, but if we did 
decide do dig into codecs roughly like these, what TYPE of:
  - analysis would we want to do?
  - questions would we ask about such example codecs?
  - metrics would we use for comparing multiple examples?
  - decsisions would be need to make in order to take a codec to RFC or not?

See? It's not about THESE CODECS, they are just examples to jump 
start discussion about the content that must ultimately fill out a charter.

Does that make sense?

As a second order comment, I appreciate seeing a number of different 
codecs being thrown at the mail list by their handlers during this 
bof period because:
  - it helps us gauge the level of interest and contribution in doing 
this work,
  - ensures we have a diversity of options that will be made 
available (i.e. not just one vendor pushing their kool-aide)
  - helps us calibrate our expectations about the types of codecs we 
would see if we proceeded the work.

Hope that helps,
Gregory.

---

From: Koen Vos <[email-masked]>
Date: Tue, 28 Jul 2009 02:20:02 -0700

Clearly, we've already done much more than suggesting two codecs.

For instance, we've had detailed discussions about what it takes to  
make a codec work on the Internet. We seemed to agree that the codec  
should support not only packet loss concealment, but also time  
compression/stretching to handle network jitter. (Incidentally, such  
time modification is not supported by any ITU, 3GPP or ETSI codec,  
which complicates their use for Internet applications.)

Or take the technical discussion between Jean-Marc and myself. We went  
over many details of SILK and CELT, and have been making suggestions  
how to improve each codec.

I will continue to participate actively in any technical review and  
quality assessment.

best,
koen.

---

From: stephen botzko <[email-masked]>
Date: Tue, 28 Jul 2009 07:46:49 -0400

inline comment

Regards,
Stephen Botzko
Polycom

On Tue, Jul 28, 2009 at 5:05 AM, Gregory M. Lebovitz <[email-masked]
> wrote:

>  http://tools.ietf.org/agenda/75/codec.html
>
> looks feasible to do the technical work to meet the goals. This should
> also help clarify what type of work is being considered."
>
> candidates or lead candidates, these two are just serving as archetypical
> models of the type of codec we might see if the work progresses, SO THAT
> THE BoF PARTICIPANTS CAN DISCUSS HOW WE WILL DO TECHNICAL work. The goal
> is NOT to do the technical work on the codecs at this BoF. This is not a
> beauty contest of these codecs. It's more like "Here are two
> specimens of this species. Let's note some characteristics about this
> species. Now, how would we go about analyzing this species further?"
> Versus "I like the left specimen because he seems bigger and softer
> and darker. Can I take it home?" We aren't doing the latter.
>
> going to dig into those two codecs in particular, but if we did decide do
> dig into codecs roughly like these, what TYPE of:
> See? It's not about THESE CODECS, they are just examples to jump start
> discussion about the content that must ultimately fill out a charter.
>
> Does that make sense?
>
> I agree these questions matter, though I think that we can discuss them
just as effective/y w/o the presentations.  My issue with the presentations
is the amount of time they consume (about 1/4 of the total meeting), I just
don't see a sufficient return on the time invested.

> codecs being thrown at the mail list by their handlers during this bof
> period because:
> > From: [email-masked] [ mailto:[email-masked]<[email-masked]>]
> On Behalf
> > Of Stefan Sayer

---

From: "Julian Spittka" <[email-masked]>
Date: Tue, 28 Jul 2009 13:59:37 +0200

Let's also keep in mind that it doesn't take any specific knowledge in
designing a codec to come up with requirements for a new internet codec and
to identify the shortcomings of existing codecs. Anybody who has dealt with
VoIP systems will be able to contribute in that area and participate in the
discussion. Look at it as a "wish list" that will be compared against the
feature set of the codec candidates.

Regards,

Julian.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Koen Vos
Sent: Tuesday, July 28, 2009 11:20 AM
To: [email-masked]
Subject: Re: [codec] Agenda for the Codec Bof

best,
koen.

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 28 Jul 2009 09:06:40 -0400

Re: " We seemed to agree that the codec  
should support not only packet loss concealment, but also time  
compression/stretching to handle network jitter. (Incidentally, such  
time modification is not supported by any ITU, 3GPP or ETSI codec,  
which complicates their use for Internet applications.)"

Two quick comments:

Comment 1: I believe the ITU-T *has* considered the time-modification
ability of some codecs in P.862 (perceptual evaluation of speech
quality) - so dependent on your definition of "supported" your
parenthetical comment may or may not be accurate. While it is true that
no ITU codec supports time modification natively (e.g., the
slow-down/speed-up of model-based playout), overlap and add techniques
can always be applied to such codec outputs to accomplish similar
auditory ends.

Comment 2: Some applications may not desire compression/stretching to be
always enabled - and actually prefer "jumps in time" consistent with
jitter buffer needs. For example any processing that depends on
(piecewise) time-invariance of the send-to-receive path (e.g.,
network-based echo cancellers) can be designed to handle discrete "jumps
in time" better than a some unknowable time-warping function. Also,
musicians can detect extremely small deviations from perfect time (on
the order of 1% change over 1/2 second from 1:1 time warping) - many
music applications (classical music playout) may not desire this
capability.

So if "supported" means an option to enable such time warping - I would
have no issue with your statement (and said attribute should be
negotiated). If "supported" means a mandatory capability that cannot be
turned off, then I think many applications would not desire that
attribute.

As an aside, the ITU-T clearly identifies "use cases" before designing a
codec for reasons such as the above. You need to know if you are
designing a codec for a singular use case (e.g. wideband speech with
little likelihood of uses such as above) before one mandates a
non-optional attribute (e.g., adaptive time playout). These use cases
and resulting desired attributes should be reflected in the requirements
(assuming a WG is formed).

My $0.02,

Michael Ramalho

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Koen Vos
Sent: Tuesday, July 28, 2009 5:20 AM
To: [email-masked]
Subject: Re: [codec] Agenda for the Codec Bof

best,
koen.

---

From: Stefan Sayer <[email-masked]>
Date: Tue, 28 Jul 2009 15:27:01 +0200

Hello,

> can always be applied to such codec outputs to accomplish similar
I expect better performance of time scale modification natively in the 
codec ("supported" in the second sense), especially where extremely low 
playout delay is desired and with short frame sizes (e.g. 5ms) where OLA 
techniques (TD-WSOLA, PICOLA) do not work properly due to fundamental 
frequency of voice.

> jitter buffer needs. For example any processing that depends on
Agreed. This needs to be parametrized between the dejitter buffer and 
the decoder, and thus be possible to disable.

> negotiated). If "supported" means a mandatory capability that cannot be
sorry, I don't understand this: "negotiated" in the codec 
development/selection process?

Thanks
Stefan

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 28 Jul 2009 09:29:45 -0400

Hi Roni,

Depends on your definition of "rubber stamped".

To my knowledge iLBC is NOT a IETF standard.

To be precise, iLBC is not documented in a standards track RFC (which
would result in it being an IETF standard).

RFC 3951 is an experimental RFC. Indeed the first significant paragraph
of RFC 3951 states:

"  This memo defines an Experimental Protocol for the Internet
   community.  It does not specify an Internet standard of any kind.
   Discussion and suggestions for improvement are requested.
   Distribution of this memo is unlimited."

See http://www.ietf.org/tao.html for definitions of experimental RFCs.

So, I believe that this IETF activity may result in the first IETF
standard codec (if WG formed, work progressed, and standard process
agreed).

Michael Ramalho

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Roni Even
Sent: Monday, July 27, 2009 5:49 PM
To: 'Stefan Sayer'; 'Patrick Luthi'
Cc: [email-masked]
Subject: Re: [codec] Agenda for the Codec Bof

Hi,
Having two codec suggested does not represent expertise to review codec
work. The IETF "rubber stamped" the iLBC codec. There was a proposed
codec,
which you consider expertise, but no capability to do a real review and
quality assessment
Roni Even

> > I am surprised to still see codecs listed in item 4 of the agenda.
As
> I

---

From: "Markus Vaalgamaa" <[email-masked]>
Date: Wed, 29 Jul 2009 03:40:58 +0300

Hi Lars, Roni and all,

Roni and Lars wrote:
>> but no capability to do a real review and quality assessment

> than their initial candidate submission.

I'm very willing to contribute to the reviewing and quality assessments.

I have the lead Call/Audio/Speech Quality Assessment role at Skype and I'm
working daily with subjective user feedbacks, stats, objective quality
measurement tools (such as PESQ/P.862), audio measurements, lab, informal &
formal listening tests, user centric and tech. requirements, even codec &
other processing algorithm details etc... I also actively follow and
participate (as an invited expert) to the key standardization forums of
these matters - ITU-T SG 12
(http://www.itu.int/ITU-T/studygroups/com12/index.asp) and ETSI STQ
(http://portal.etsi.org/stq/Summary.asp). So I hope I have something to
bring to the common work. 

I'm also sure there are ways to run a part of formal listening tests at
Skype facilities if that's seen beneficial later. (or we can arrange tests
with some of the 3rd party labs)

Before I moved to Skype a bit more than 3 years ago, I worked for around 8
years in/for Nokia on the similar Speech/Audio Quality Assessment matters +
audio SW algorithms & audio&speech codecs... and the most of that time I've
worked at the same building that Lars appears to be working today ;-)

Best regards with my few Euro cents...

	Markus Vaalgamaa

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Lars Eggert
Sent: Tuesday, July 28, 2009 9:23 AM
To: Roni Even
Cc: [email-masked]
Subject: Re: [codec] Agenda for the Codec Bof

Hi,

Lars

---

From: "Roni Even" <[email-masked]>
Date: Wed, 29 Jul 2009 08:36:55 +0300

Markus,
I am happy to hear it, so here is the challenge for you, I assume you are
following the email discussion and one of the topic is how to define metrics
and test the quality of the audio candidate. I will be interested to see you
position about it. 
Regards 
Roni Even

-----Original Message-----
> From: Markus Vaalgamaa [mailto:[email-masked]]
> Sent: Wednesday, July 29, 2009 3:41 AM
> To: 'Lars Eggert'; 'Roni Even'
> Cc: [email-masked]
> Of

---

From: "Gregory M. Lebovitz" <[email-masked]>
Date: Wed, 29 Jul 2009 01:51:14 -0700

Roni,
Just to clarify how I read the latest charter...

It's not that a WG would "test" the quality of the audio of a 
candidate mechanism or codec. It's not a pass vs fail thing. There 
doesn't need to be a winner and a loser. In fact, the word "test" 
does not appear in the charter. The charter does say:

"define a set of metrics which can be used to evaluate a codec. "

The word "evaluate" is very different. I read it to mean "how will a 
group of people normalize their collective subjective opinion on an 
observed condition". By example, consider the way cars are reviewed 
in magazines. Judges who have been trained on a set of metrics about 
performance under different conditions (say top end  speed, or 
cornering, or acceleration, or storage space) have a shared set of 
definitions of performance they might observe, both good and 
impressive performance as well as poor/faulty performance. They also 
have a standard measurement scale -- in this example scoring by 
points -- by which they record their observations of performance. The 
measurement scale includes different values for different performance 
elements, i.e. certain observations cost 1/2 point deduction, others 
cost 1/4 point, and some gain points because they are more difficult 
or beneficial. When comparing two sports cars one might be better at 
sharp turns at high speeds and acceleration, while another might be 
better at sustained high speeds on straight aways. But both are 
terrible at fitting your family of 6 and all your luggage for a 10 hr 
road trip over dirt and rock terrain, towing a ski boat.

Likewise, this WG will need to come up with a shared set of 
"performance" metrics that they might observe from a codec or its 
sub-mechanisms. And it will need to agree on a system for valuing 
these metrics. Like the vehicle example, the WG may not determine one 
winner, and determine all others losers. But might simply evaluate 
two and agree that X codec/mechanism is better at Alpha and Beta, 
where as Y codec/mechanism just as good at Beta, but not as good in 
Alpha, but much better in Delta, and thus is better applied to 
environment 3, but not as good as X in environment 2.

My point is that the word "evaluate" is intended to have a far 
different meaning than purely "test". We may do many tests in order 
to form a unified evaluation.

This might be a very subtle difference, but I think it important to tease out.

Hope it helps,
Gregory

---

From: "Roni Even" <[email-masked]>
Date: Wed, 29 Jul 2009 15:03:53 +0300

Gregory,
I think that the charter should be clear and should not depend so much on
personal views.
In general I have no problem with your analysis and example.
The thing is that by the end we need to approve each codec and if it is
going through multiple tests will it need to pass all of them, will there be
one that are mandatory to pass and other are optional, this will compose
what you call as the final score. I believe that in order to approve a codec
the wg should see the test results before they accept it in a WGLC. If the
tests results provided to the working group are just for information and are
not relevant for last call a codec than we do not need requirements or tests
and just need to read the codec description and do last call.

Roni Even 

> From: Gregory M. Lebovitz [mailto:[email-masked]]
> Sent: Wednesday, July 29, 2009 11:51 AM
> To: Roni Even
> Cc: [email-masked]
> > > I have the lead Call/Audio/Speech Quality Assessment role at Skype
> and
> > > I'm
> > > formal listening tests, user centric and tech. requirements, even
> codec
> > > &
> > > Skype facilities if that's seen beneficial later. (or we can
> arrange
> > > tests
> > > audio SW algorithms & audio&speech codecs... and the most of that
> time
> > > I've
> > > > which you consider expertise, but no capability to do a real
> review
> > > > and
> > > > quality assessment
> > >

---

From: stephen botzko <[email-masked]>
Date: Wed, 29 Jul 2009 09:53:27 -0400

I suggest the dictionary definition of "evaluate" is sufficient for the
charter.

Evaluate:
1 *:* to determine or fix the value of
2 *:* to determine the significance, worth, or condition of; usually by
careful appraisal and study

I'd say there is a distinction between what you are calling evaluation and
what in other SDOs is called characterization.

Characterization (in my view) is not a tracking document and is not a
weighted scoring system. It is a deliverable that describes the codec
behavior that has lasting value. One example from 3GPP is found here:

http://www.3gpp.org/ftp/Specs/archive/26_series/26.975/26975-800.zip

I would prefer to see a characterization deliverable either as a section of
the codec RFC, or as a separate informational RFC.

Regards
Stephen Botzko
Polycom

On Wed, Jul 29, 2009 at 4:51 AM, Gregory M. Lebovitz <[email-masked]

> It's not that a WG would "test" the quality of the audio of a candidate
> mechanism or codec. It's not a pass vs fail thing. There doesn't need to be
> a winner and a loser. In fact, the word "test" does not appear in the
> charter. The charter does say:
>
> The word "evaluate" is very different. I read it to mean "how will a group
> of people normalize their collective subjective opinion on an observed
> condition". By example, consider the way cars are reviewed in magazines.
> Judges who have been trained on a set of metrics about performance under
> different conditions (say top end  speed, or cornering, or acceleration, or
> storage space) have a shared set of definitions of performance they might
> observe, both good and impressive performance as well as poor/faulty
> performance. They also have a standard measurement scale -- in this example
> scoring by points -- by which they record their observations of performance.
> The measurement scale includes different values for different performance
> elements, i.e. certain observations cost 1/2 point deduction, others cost
> 1/4 point, and some gain points because they are more difficult or
> beneficial. When comparing two sports cars one might be better at sharp
> turns at high speeds and acceleration, while another might be better at
> sustained high speeds on straight aways. But both are terrible at fitting
> your family of 6 and all your luggage for a 10 hr road trip over dirt and
> rock terrain, towing a ski boat.
>
> Likewise, this WG will need to come up with a shared set of "performance"
> metrics that they might observe from a codec or its sub-mechanisms. And it
> will need to agree on a system for valuing these metrics. Like the vehicle
> example, the WG may not determine one winner, and determine all others
> losers. But might simply evaluate two and agree that X codec/mechanism is
> better at Alpha and Beta, where as Y codec/mechanism just as good at Beta,
> but not as good in Alpha, but much better in Delta, and thus is better
> applied to environment 3, but not as good as X in environment 2.
>
> My point is that the word "evaluate" is intended to have a far different
> meaning than purely "test". We may do many tests in order to form a unified
> evaluation.
>

---

From: "Gregory M. Lebovitz" <[email-masked]>
Date: Wed, 29 Jul 2009 16:37:13 -0700

At 06:53 AM 7/29/2009, stephen botzko wrote:
>I suggest the dictionary definition of "evaluate" is sufficient for 
>the charter.
>
>Evaluate:
>1 : to determine or fix the value of
>2 : to determine the significance, worth, or condition of; usually 
>by careful appraisal and study
>
>I'd say there is a distinction between what you are calling 
>evaluation and what in other SDOs is called characterization.
>
>Characterization (in my view) is not a tracking document and is not 
>a weighted scoring system. It is a deliverable that describes the 
>codec behavior that has lasting value. One example from 3GPP is found here:
>
><http://www.3gpp.org/ftp/Specs/archive/26_series/26.975/26975-800.zip>http://www.3gpp.org/ftp/Specs/archive/26_series/26.975/26975-800.zip
>
>I would prefer to see a characterization deliverable either as a 
>section of the codec RFC, or as a separate informational RFC.

I'm not quite sure I yet see the exact difference between what I (a 
non-codec guy just trying to assist in oversight of process issues 
here) called "evaluate" and you called "characterized". They seemed 
pretty similar. Maybe you can explain it to me tomorrow?

However, I feel we are narrowing in on what it ought to be, and that 
is a very good thing. Thanks for your observation,
Gregory.

>Regards
>Stephen Botzko
>Polycom
>
>On Wed, Jul 29, 2009 at 4:51 AM, Gregory M. Lebovitz 
><<mailto:[email-masked]>[email-masked]> wrote:
>Roni,
>points -- by which they record their observations of performance. 
>The measurement scale includes different values for different 
>performance elements, i.e. certain observations cost 1/2 point 
>deduction, others cost 1/4 point, and some gain points because they 
>are more difficult or beneficial. When comparing two sports cars one 
>might be better at sharp turns at high speeds and acceleration, 
>while another might be better at sustained high speeds on straight 
>aways. But both are terrible at fitting your family of 6 and all 
>your luggage for a 10 hr road trip over dirt and rock terrain, 
>towing a ski boat.
>
>these metrics. Like the vehicle example, the WG may not determine 
>one winner, and determine all others losers. But might simply 
>evaluate two and agree that X codec/mechanism is better at Alpha and 
>Beta, where as Y codec/mechanism just as good at Beta, but not as 
>good in Alpha, but much better in Delta, and thus is better applied 
>to environment 3, but not as good as X in environment 2.
>
> > Cc: <mailto:[email-masked]>[email-masked]
> (<http://www.itu.int/ITU-T/studygroups/com12/index.asp>http://www.itu.int/ITU-T/studygroups/com12/index.asp) 
> and ETSI STQ
> > 
> (<http://portal.etsi.org/stq/Summary.asp>http://portal.etsi.org/stq/Summary.asp). 
> So I hope I have something to
> > From: <mailto:[email-masked]>[email-masked] 
> > Cc: <mailto:[email-masked]>[email-masked]
><mailto:[email-masked]>[email-masked]
><mailto:[email-masked]>[email-masked]

---

From: "Gregory M. Lebovitz" <[email-masked]>
Date: Wed, 29 Jul 2009 16:54:44 -0700

>and just need to read the codec description and do last call.

Roni,
We may have an English language translation issue here, or more 
likely I was poorly communicating, because what you just typed above, 
and what I so painstakingly tried to describe below are pretty far apart.

So let me try again:
There is no "test" needed and final score needed. There is need of an 
agreed upon process for evaluation of several different attributes of 
a codec, and a value assigned for each attribute. When the values for 
the various attributes are lined up, we'll be able to see if a codec 
is good in a certain type of environment, and how it compares to others.

Someone else on the list called this "Characterization" and I think 
his definition of that word sounded very similar to what I was trying 
to get across.

Hope that helps clarify.

And thanks for your persistence on the list. It appears you have a 
strong interest in this work, and will be an active voice on the 
list. That's very important for us to measure:  how consistently 
people will be working on list. You've got a thumbs up for sure. 
Thanks again!!

Gregory.

---

From: "Roni Even" <[email-masked]>
Date: Thu, 30 Jul 2009 07:45:09 +0300

Hi,
This is not an issue of English language translation but of audio codec
standardization terminology.
The charter  has in the deliverable the following text "After a codec is
finalized, these guidelines will be used to characterize the codec"

One who is familiar with audio codecs specification terminology will
understand the word characterization as for example  you can see in
http://www.3gpp.org/ftp/Specs/archive/26_series/26.975/26975-800.zip 

I suggest you look at it and see that it is based on test results
Roni Even

> Sent: Thursday, July 30, 2009 2:55 AM
> To: Roni Even
> Cc: [email-masked]
> The
> one
> > > >I am happy to hear it, so here is the challenge for you, I assume
> you
> > > are
> > > >following the email discussion and one of the topic is how to
> define
> > > metrics
> > > >and test the quality of the audio candidate. I will be interested
> to
> > > see you
> > > >position about it.
> > > >Regards
> > > >Roni Even
> > > >
> > > >
> > > >
> > > >
> > > > > > agree with Roni. Having codec specs available that
> (apparently)
> > > fit
> > > > > > participate in an IETF effort that would be based on
> something
> > > other
> > > > > measurement tools (such as PESQ/P.862), audio measurements,
> lab,
> > > > > informal &
> > > > > other processing algorithm details etc... I also actively
> follow
> > > and
> > > > > Before I moved to Skype a bit more than 3 years ago, I worked
> for
> > > > > around 8
> > > > > years in/for Nokia on the similar Speech/Audio Quality
> Assessment
> > > > > matters +
> > > > > worked at the same building that Lars appears to be working
> today
> > > ;-)
> > > > >
> > > > > > Having two codec suggested does not represent expertise to
> review
> > > > > > codec

---

From: "Gregory M. Lebovitz" <[email-masked]>
Date: Wed, 29 Jul 2009 23:35:27 -0700

At 07:40 PM 7/29/2009, stephen botzko wrote:
>I'm not in Stockholm, so we won't be able to connect so easily!
>
> From the examples in your email, "evaluation" seemed to me to be 
> about scoring/assessing the relative performance of proposals in 
> order to guide decision making.   Once the standardization process 
> for the codec is completed, the evaluation results are no longer 
> needed (in the charter draft, it is described being in a tracking document).

Ahh. This illuminates that I did a poor job of expressing my thoughts 
originally. "Evaluation" to me meant more of what you describe as 
"characterization", because once the comparison is done about a 
mechanism within a codec, or a codec as a whole, that comparison info 
needs to be included in its RFC so that everyone going forward knows 
on which attributes that mech/codec is relatively strong (scored 
high) and relatively weak (scored low). So I guess your industry uses 
"characterization" for that? If so, then that's probably a better 
word to use going forward.

>I agree that evaluation in your sense has to be part of the 
>process.  What I was trying to say is that the characterization 
>report has value even after the standard is complete - it provides 
>valuable tested results on how the codec performs under a variety of 
>conditions that really occur.  So it should be viewed as a 
>deliverable in its own right, not as an intermediate result that is 
>discarded.

100% agree.

>The current charter doesn't seem to say that.  It talks about how 
>the metric definitions (and presumably test methods) are 
>deliverables, but does not clearly say that the test results on the 
>final codec are a deliverable.

As I've talked with the BoF operators, I think they intended it to 
say what you and I are refining as "characterization" in addition to 
a scoring for the purpose of preference comparison w/in the work 
group. I assume they will read this, and take action to edit the 
charter appropriately.

Would you like to propose what you feel is better text to capture this?

VERY USEFUL DISCUSSION HERE. Thx,
Gregory.
(PS - this list would be well served by more of this type of 
constructive clarification discussion)

>Regards,
>Stephen Botzko
>Polycom
>
>On Wed, Jul 29, 2009 at 7:37 PM, Gregory M. Lebovitz 
>><<mailto:[email-masked]>[email-masked] > wrote:
>>Roni,
>>The word "evaluate" is very different. I read it to mean "how will 
>>a group of people normalize their collective subjective opinion on 
>>an observed condition". By example, consider the way cars are 
>>reviewed in magazines. Judges who have been trained on a set of 
>>metrics about performance under different conditions (say top 
>>end  speed, or cornering, or acceleration, or storage space) have a 
>>shared set of definitions of performance they might observe, both 
>>good and impressive performance as well as poor/faulty performance. 
>>They also have a standard measurement scale -- in this example 
>>scoring by points -- by which they record their observations of 
>>performance. The measurement scale includes different values for 
>>different performance elements, i.e. certain observations cost 1/2 
>>point deduction, others cost 1/4 point, and some gain points 
>>because they are more difficult or beneficial. When comparing two 
>>sports cars one might be better at sharp turns at high speeds and 
>>acceleration, while another might be better at sustained high 
>>speeds on straight aways. But both are terrible at fitting your 
>>family of 6 and all your luggage for a 10 hr road trip over dirt 
>>and rock terrain, towing a ski boat.
>>
>>evaluate two and agree that X codec/mechanism is better at Alpha 
>>and Beta, where as Y codec/mechanism just as good at Beta, but not 
>>as good in Alpha, but much better in Delta, and thus is better 
>> > From: Markus Vaalgamaa [ mailto:[email-masked]]
>> > ( http://www.itu.int/ITU-T/studygroups/com12/index.asp) and ETSI STQ
>> > ( http://portal.etsi.org/stq/Summary.asp). So I hope I have something to
>> > From: <mailto:[email-masked]>[email-masked] [ 
>> mailto:[email-masked]] On Behalf
>> > Of
>> > Lars Eggert