Subject: [codec] Revised Draft Charter

From: Jason Fischl <[email-masked]>
Date: Fri, 17 Jul 2009 11:42:27 -0700

DRAFT CHARTER:

The goal of this working group is to standardize audio codecs that can  
be implemented and used broadly, and be interoperable between a wide  
set of interested parties. The group is chartered to work on audio  
codecs for interactive communication over the Internet. Codecs  
optimized for very low bit rates or for non-interactive audio are out  
of scope. The codecs should address the following considerations:

* suitability for most fixed-point DSPs;
* medium- to high-quality speech at moderate bit-rate;
* high-quality music encoding at higher bit-rates;
* sampling rate profiles from narrow-band to full audio bandwidth;
* real-time adaptive bit rate;
* source-controlled variable bit-rate (VBR);
* low algorithmic delay;

IPR issues will be handled in accordance with BCP 79.

Proposed Deliverables:
1) Detailed requirements for Internet audio codecs.
2) Algorithm description for wideband Internet audio codecs as PS.

---

From: stephen botzko <[email-masked]>
Date: Fri, 17 Jul 2009 16:21:07 -0400

Can you add "Codec Characterization Test Results" to the Proposed
Deliverables?

Regards,
Stephen Botzko
Polycom

On Fri, Jul 17, 2009 at 2:42 PM, Jason Fischl <[email-masked]>wrote:

> DRAFT CHARTER:
>
> The goal of this working group is to standardize audio codecs that can be
> implemented and used broadly, and be interoperable between a wide set of
> interested parties. The group is chartered to work on audio codecs for
> interactive communication over the Internet. Codecs optimized for very low
> bit rates or for non-interactive audio are out of scope. The codecs should
> address the following considerations:
>
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 17 Jul 2009 14:31:22 -0700

I think there would be a great value in someone explaining what ³codec
characterization² actually means to those poor (majority of?) IETFers who
don¹t know...
Stephan

On 7/17/09 1:21 PM, "stephen botzko" <[email-masked]> wrote:

> On Fri, Jul 17, 2009 at 2:42 PM, Jason Fischl <[email-masked]> wrote:
>> DRAFT CHARTER:
>>

---

From: stephen botzko <[email-masked]>
Date: Fri, 17 Jul 2009 20:49:00 -0400

Basically, its statistically valid testing of the codec in order to prove
that it meets its requirements..

In the ITU-T process, a test plan is developed, generally testing the codec
against a reference codec under stated conditions.  This is worked out
before selection process.  It might include an audio quality requirement
like "Codec x at 16 kbits will sound no worse than G.722 at 48 kibts with
reverberant speech"

A statistically valid listening test is then done by couple of different
test labs, using reverberant speech samples in different languages, wiht a
specified confidence interval.  The results are then reported back for each
candidate codec.

3GPP uses a similar process

I'm not saying that we would need to follow the ITU-T or 3GPP
characterization methods, though we should look at them.  But that we will
need to develop more refined requirements, and work out a statisitically
valid way to prove the standard meets each requirement.

Regards,
Stephen Botzko
Polycom

On Fri, Jul 17, 2009 at 5:31 PM, Stephan Wenger <[email-masked]> wrote:

>  I think there would be a great value in someone explaining what “codec
> characterization” actually means to those poor (majority of?) IETFers who
> don’t know...
> Stephan
>
>
>
>
> ------------------------------

---

From: Steve Underwood <[email-masked]>
Date: Sat, 18 Jul 2009 13:14:48 +0800

Hi Stephen,

stephen botzko wrote:
> Basically, its statistically valid testing of the codec in order to 
> prove that it meets its requirements..
>
> In the ITU-T process, a test plan is developed, generally testing the 
> codec against a reference codec under stated conditions.  This is 
> worked out before selection process.  It might include an audio 
> quality requirement like "Codec x at 16 kbits will sound no worse than 
> G.722 at 48 kibts with reverberant speech"
>
> A statistically valid listening test is then done by couple of 
> different test labs, using reverberant speech samples in different 
> languages, wiht a specified confidence interval.  The results are then 
> reported back for each candidate codec.
>
> characterization methods, though we should look at them.  But that we 
> will need to develop more refined requirements, and work out a 
> statisitically valid way to prove the standard meets each requirement.

Actually, that's just one side of what they consider to be codec 
characterisation. The other is the computational complexity. Nobody 
wants a fantastic sounding codec that takes 100 64x cores to implement. 
:-) The ITU reference code, at least for fixed point, is written so that 
its fairly straightforward to sum up all the saturated multiplies, 
saturated adds, shifts, etc.

Steve

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 06:43:15 -0400

I agree that we should have a complexity requirement that is also tested,
and I am fine with including that as part of characterization.  In addition
to computational load, we could also consider memory requirements (volatile
and non volatile?)

Another aspect is to ensure that the fixed and floating point versions have
the same characteristics  ( that all combinations - fixed->fixed,
fixed->float, float->fixed, float->float - sound identical).

Regards
Stephen Botzko
Polycom

On Sat, Jul 18, 2009 at 1:14 AM, Steve Underwood <[email-masked]> wrote:

>> codec against a reference codec under stated conditions.  This is worked out
> characterisation. The other is the computational complexity. Nobody wants a
> fantastic sounding codec that takes 100 64x cores to implement. :-) The ITU
> reference code, at least for fixed point, is written so that its fairly
> straightforward to sum up all the saturated multiplies, saturated adds,
> shifts, etc.
>
> Steve
>
>

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 08:55:25 -0400

This would be a very worthwhile discussion, and IMHO should be held before
any codec contributions are considered.

Though I fear that we don't have enough time to get consensus on the details
of the codec development/selection/characterization procedures before the
BOF.

Regards,
Stephen Botzko
Polycom

On Fri, Jul 17, 2009 at 9:16 PM, Alan Duric <[email-masked]> wrote:

> Hi all,
>
> agree with Stephen for the need of such deliverable, since that will also
> govern (among other things) what kind of PLC will be enclosed as reference
> PLC etc. Given that goal of the group is to deliver codec of a high quality,
> this is quite essential.
> However, I would be very cautious on copying too much ITU-T and 3GPP and
> using them as reference "a priori", since I find a number of nits in
> those methods as outdated and believe that we can do even better job by
> using it as a benchmark "a posteriori".
>
> regz,
> alan
>
>
> ____________________________________________________________________________
> "The strong do what they have to do and the weak accept what they have to
> accept." -- Thucydides
>
>
> On Jul 18, 2009, at 2:49 AM, stephen botzko wrote:
>

---

From: Koen Vos <[email-masked]>
Date: Sat, 18 Jul 2009 06:13:59 -0700

The validation testing may need some thinking. For example, G.718 used  
7 independent listening labs (which usually cost between $10k and $20k  
each to run one listening batch).  It may be hard to find that kind of  
money for testing a royalty-free codec.  But agree that we will need  
some mechanism to compare a candidate against (1) the requirements and  
(2) other candidates.

Testing the combinations of float and fixed-point versions can perhaps  
be done with objective measurement tools like PESQ or PEAQ. I believe  
this is how ITU also does it(?)

Complexity:
Average WMOPS complexity, measured over a test file?

Memory consumption measures:
- RAM (static/state)
- RAM (scratch)
- ROM (tables)
- Program ROM is a bit trickier, not sure we need it..?

best,
koen.

Quoting stephen botzko <[email-masked]>:

>>> worked out

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 09:20:35 -0400

I'm hearing a consensus that characterization testing should be a charter
deliverable, but that the specific methods need more development.  Does
anyone hold a different view?

Regards,
Stephen Botzko
Polycom

On Sat, Jul 18, 2009 at 9:13 AM, Koen Vos <[email-masked]> wrote:

> The validation testing may need some thinking. For example, G.718 used 7
> independent listening labs (which usually cost between $10k and $20k each to
> run one listening batch).  It may be hard to find that kind of money for
> testing a royalty-free codec.  But agree that we will need some mechanism to
> compare a candidate against (1) the requirements and (2) other candidates.
>
> Testing the combinations of float and fixed-point versions can perhaps be
> done with objective measurement tools like PESQ or PEAQ. I believe this is
> how ITU also does it(?)
>
> Complexity:
>>>> will

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 18 Jul 2009 09:43:38 -0400

Koen Vos a écrit :
> independent listening labs (which usually cost between $10k and $20k

The question is what to use for comparison in the first place. I tend to
think that the only useful reference we have is unencumbered codecs. For
example, SILK might be tested against Speex. I'm not sure what CELT
could be tested against considering the large bandwidth it has and the
very low delay.

Regarding validation, I think we should also have a look at the MPEG
process and see what they do there. I'm not familiar enough with MPEG,
though. Does anyone know more about that process?

> this is how ITU also does it(?)

Sounds reasonable, though that assumes both fixed-point and
floating-point are available. I don't think it's a problem for CELT and
SILK, but do we want to mandate that for everyone.

> Average WMOPS complexity, measured over a test file?

I'm less sure on this one. There are several issues that make WMOPS less
than ideal:
* it forces you to code the fixed-point in a certain way (e.g.
fractional vs interger multiply);
* it's designed for a 16-bit (TI) DSP and is not very respresentative of
modern 32-bit DSPs. For example, a 32-bit add costs more than a MAC;
* it completely ignore issues such as conditional branches, which can be
more important than arithmetic operations on modern architectures;
* some of its operators (e.g. those that use a global overflow flag) are
ugly, to say the least.

I'm not sure what to use instead for complexity, but I'm also wondering
if we want a fixed complexity requirement up-front. For example, why not
use "rough consensus" as a means to determine whether a codec's
complexity is acceptable for most people.

> - Program ROM is a bit trickier, not sure we need it..?

Regarding size, I would think it may also be a good idea to seek rough
consensus.

Cheers,

	Jean-Marc

>>>> different
>>> wants a

---

From: Koen Vos <[email-masked]>
Date: Sat, 18 Jul 2009 07:55:54 -0700

> complexity is acceptable for most people.

Agree with your points on WMOPS, let's forget about that one.

Rough consensus based on MHz on an x86 core is simple to measure and  
gives already a pretty meaningful indication.

> Regarding size, I would think it may also be a good idea to seek rough

Works for me.

koen.

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 15:01:27 -0400

(meant to post this to the list)

---------- Forwarded message ----------
From: stephen botzko <[email-masked]>
Date: Sat, Jul 18, 2009 at 10:33 AM
Subject: Re: [codec] Revised Draft Charter
To: Jean-Marc Valin <[email-masked]>

Some comments in line.

I would suggest though that we should first get agreement on adding the one
line to the charter, and re-issue the draft with that line in it. Iterating
the charter draft is probably the most efficient way to get consensus on the
charter, and I think it is cleanest if the drafts all come through you and
Jason.

Regards,
Stephen Botzko
Polycom

> could be tested against considering the large bandwidth it has and the

I agree that this is an important question.  In other SDOs, this is worked
out on a case by case basis, and depends on the market application that is
being targeted.  Personally, I don't think it matters much if you use RF or
royalty-bearing codecs as references, it is more important that the test
conditions be properly set.  Note that you don't necessarily need to
out-perform the reference - for instance "codec x at 12 kbits is not worse
than G.729 at 8 kbits" might be appropriate for a low-complexity codec.

> though. Does anyone know more about that process?

MPEG should also be looked at of course. I believe its process is similar to
the other SDOs, with one very important caveat - MPEG does not standardize
encoders, so their characterization tests are not easily reproducible.
Given our goals, I think we will be standardizing both the encoder and the
decoder, and should be testing them as a pair.

> SILK, but do we want to mandate that for everyone.

Agreed that if there aren't both versions available, that it would not be
required (how reasonable is THAT!)  Though I suspect we will find that both
fixed and float versions will be necessary.  The ITU-T has recently used
objective measurement tools for this (G.719 was done that way), it is
inexpensive since it does not require any listening tests.

> consensus

.

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 15:03:20 -0400

(taking the liberty of fowarding Jean-Marc's reply)

On Sat, Jul 18, 2009 at 2:00 PM, Jean-Marc Valin <
[email-masked]> wrote:

> stephen botzko a écrit :
> > MPEG should also be looked at of course. I believe its process is
> > similar to the other SDOs, with one very important caveat - MPEG does
> > not standardize encoders, so their characterization tests are not easily
> > reproducible.  Given our goals, I think we will be standardizing both
> > the encoder and the decoder, and should be testing them as a pair.
>
> Actually, there are some significant advantages to the way MPEG
> standardises only the decoders, provided there is a decent reference
> encoder. It means that the codec can keep improving for some time, while
> still retaining compatibility. The best example of that is probably MP3,
> where the current encoders are much, much better than the original
> reference encoder (and other 15 years old encoders). The potential for
> improvement isn't as large for a speech codec, but it's not negligible.
> Speex has improved (in a backward-compatible way) since the bit-stream
> was frozen and CELT certainly has potential for encoder-side
> improvements. I don't know enough about the details of SILK to comment,
> but I'd be surprised if there wasn't room for improvement. Even G.711
> got improved recently (G.711.1 sounds better in compatibility mode even
> without enhancement layers).
>
> In any case, I'm not saying I know how we should standardise the codec
> (decoder vs both), but it's certainly something to consider. I think the
> idea is to have as much flexibility as possible while making sure not to
> create compatibility problems.
>
> Cheers,
>
>        Jean-Marc
>

---

From: stephen botzko <[email-masked]>
Date: Sat, 18 Jul 2009 15:04:31 -0400

(meant to put this on the lost)

---------- Forwarded message ----------
From: stephen botzko <[email-masked]>
Date: Sat, Jul 18, 2009 at 2:34 PM
Subject: Re: [codec] Revised Draft Charter
To: Jean-Marc Valin <[email-masked]>

Though I can't speak authoritatively on the MPEG process, it's my
understanding that the encoder(s) used in the characerization are neither
disclosed nor publicly available.  I can understand how that might work for
broadcast applications, but it seems sub-optimal for speech codecs.

If anyone has a different (or more complete) understanding of the MPEG
approach, I'd be interested in hearing it.

Stephen Botzko
Polycom

---

From: Steve Underwood <[email-masked]>
Date: Sun, 19 Jul 2009 11:07:58 +0800

> complexity is acceptable for most people.
I agree with this concern. The ITU approach was supposed to be kind of 
generic in its counting of instructions, but has ended up quite the 
opposite. It is based around the notion of a machine implementing 
saturating arithmetic as a basic operation. This is true for pretty much 
any DSP, but when you use an ARM, a MIPS or a Intel/AMD machine the 
instruction count is wildly wrong. Also, the ITU approach only tries to 
get an instruction count for the fixed point implementations. The 
floating point computational complexity is also very important these days.
> Regarding size, I would think it may also be a good idea to seek rough
You will probably end up with two sets of reference sizes - one for 
fixed point and one for floating.

I guess the other interesting area of implementational performance is 
bit exactness. For many older codecs (e.g. G.722) bit exactness was a 
vital concern, or the decoder would not properly unwind the actions of 
the coder. For most modern codecs its not that interesting. It is 
generally possible to make fixed point implementations bit exact, and 
its generally impractical to make floating point ones bit exact. 
However, even in the fixed point case, bit exactness may be highly 
undesirable. On an modern x86_64 machine I could make a perfectly good 
fixed point G.729 that runs far faster than a bit exact one.

Steve

---

From: "Roni Even" <[email-masked]>
Date: Mon, 20 Jul 2009 11:56:35 +0300

Hi,
I am not sure I understand what you mean when you say rough consensus here.
Does it mean that after the codec is specified and in WGLC the complexity
will be agreed by rough consensus. I would think that people need of know
ahead of time what is the requirement and how to measure it.

Roni Even 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Koen Vos
> Sent: Saturday, July 18, 2009 5:56 PM
> To: Jean-Marc Valin
> Cc: [email-masked]

---

From: "Roni Even" <[email-masked]>
Date: Mon, 20 Jul 2009 12:02:19 +0300

Hi,
This looks to me like a very wide charter for a long living WG. The charter
says that the WG will standardize audio codecs without giving any limit on
the number of such codecs nor does the charter limit the scope of such
codecs since it uses very broad terms for considerations. 

Is this the real purpose of the WG.

Roni Even

> Of Jason Fischl
> Sent: Friday, July 17, 2009 9:42 PM
> To: [email-masked]
> Subject: [codec] Revised Draft Charter
> 
> DRAFT CHARTER:
>

---

From: "Roni Even" <[email-masked]>
Date: Mon, 20 Jul 2009 12:11:45 +0300

inline

> Sent: Saturday, July 18, 2009 4:14 PM
> To: [email-masked]

The ITU does it for the floating point version after the fixed point was
tested. One thing to take unto account that these methods are not that
suitable for audio codecs like the ones discussed here.

---

From: stephen botzko <[email-masked]>
Date: Mon, 20 Jul 2009 05:55:17 -0400

One thing to take unto account that these methods are not that suitable for
audio codecs like the ones discussed here.

I think the basic techniques in the objective test method would be suitable
for any codec, though perhaps I am missing something?

Regards,
Stephen Botzko
Polycom

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 20 Jul 2009 09:56:17 -0400

> ahead of time what is the requirement and how to measure it.

(contributor hat on)

What I meant was rough consensus *before* approving/freezing the codec. If 
there is rough consensus that the codec has sufficiently low complexity to 
be implemented on most devices where it would be expected to be 
implementable, then it can be approved. On the other hand, there *are* 
complexity concerns, then they would need to be solved. We can decide to 
also characterize the complexity afterwards, but I believe that's less 
useful than making sure it's acceptable in the first place. See my other 
email for concerns I have over the use of WMOPS specifically.

Cheers,

	Jean-Marc

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 20 Jul 2009 07:28:13 -0700

One small thing inline:

> IPR issues will be handled in accordance with BCP 79.

You may wish to add "BCP 78" as well.  IPR is commonly understood as
"Intellectual Property Rights", and those include not only patents, but also
copyright (and other stuff, like trademarks).  BCP 79 handles the IETF
patent (disclosure) policy; BCP 78 is related to the copyright stuff.

Alternatively, you could replace "IPR issues" with "Patent issues".

Although some would argue that the sentence in question is redundant (as any
IETF activity runs under BCP 78 and BCP 79), I'm in favor of keeping the
sentence in the charter, simply to remind everyone of the policy that
applies.

Regards,
Stephan

---

From: Herve Taddei <[email-masked]>
Date: Wed, 22 Jul 2009 14:41:08 +0200

Dear Jean Marc and Steve,

Let me correct some statements made on the WMOPS counting as it seems to me
they are either wrong or you are not looking to the tool we are used to
have. So I am not sure your comments are based on G.191, if that is the
case, I have asked someone heavily involved in the design of ITU-T G.191 to
answer some of your points.

> > fractional vs interger multiply);

In general, in order to "automate" the evaluation of the complexity of an
algorithm on a family of processor architectures, the fixed point code needs
to be written in certain ways. If evaluating the complexity of a codec is an
important factor in the standardisation process, then such coding guidelines
should be agreed by the various parties involved in the standardisation.
W/ regards to fractional vs interger multiply : ITU STL 2000 supports few
integer multiplication (L mult0, L_mac0, L_msu0). It is true that the
variety & numbers of fractional multiplication operators vs integer
multiplication operators is not equal. From historical point of view, the
"fractional" multiplication was priviledged. In 2003 (in preparation of
STL2005), there was a proposal to widen the set of integer multiplication
while taking care of well counting the integer/fractional multiplication
mode switch that may be needed on some processors. This discussion was not
carried on forward. 

> > modern 32-bit DSPs. For example, a 32-bit add costs more than a MAC;

1st of all, I don't believe that the STL2005 is designed for a TI DSP. For
example, during the STL2005 design process STM inputs were factored in,
among others on the reflector.
Then, with regards to 32b add, in STL2005, L_add complexity weight is 1
(same as L_mac).

Now, in a more general statement, is the STL a basic operator library
representing more a 16b or 32b processor ? If I look at the STl2005 list of
operators which have a complexity > 1, I believe we fall in those categories
:
- add / subtract with carry bit.
- shift and round.
- integer multiplication with truncation to 16 lsbs (reflects indeed that
STL is historically fractional based).
- bit rotation operation
- high precision 32x16 and 32x32 multiplication (requires 40b STL headers).

I believe that with the renewed complexity weights made in 2005, the STL is
closer to a 32b processor than a 16b processor to the exception of the
32x32b multiplication which is not having a complexity of 1 ... but of 4
(Mpy 32 32 ss inside the 40b lib).

> > more important than arithmetic operations on modern architectures;

This is incorrect, with STL2005, control operation complexity count is
supported.
Any improvement suggestions welcomed to be discussed.

> Also, the ITU approach only tries to get an instruction count for the
fixed point implementations. 

> The floating point computational complexity is also very important these
days.

I think this is not correct as floating point implementation counters were
proposed in ITU-T (from VoiceAge actually) and have been used in many
standardization exercises for reporting floating point complexity. 

Best regards

Herve

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 22 Jul 2009 10:32:22 -0400

Hi Herve,

(contributor hat on)

My earlier comments were based on the version of the basic ops that I saw 
in many standards (and assuming there was just one WMOPS definition). I was 
not aware that these operators had been "recently" revised. Thanks for 
pointing those out. I agree that the revised version makes more sense and 
is closer to the a 32-bit DSP. I still see a few issues with STL2005 (e.g. 
lack of unsigned add/sub), but they are relatively minor compared to the 
older version.

That being said, I still consider that there are fundamental limitations to 
the WMOPS *approach*, caused by differences in DSP/CPU architectures. For 
example, many (all?) ARM cores do not have saturating operators. When it 
comes to conditional branches, some architectures will pay a fixed price 
for a branch taken, while others (e.g. x86) will have virtually no cost for 
an easily predicted branch and a huge cost for a branch on "random" data. 
In the end, WMOPS will give you a rough idea of the complexity, but I 
believe it is preferable to use rough consensus (possibly backed by some 
benchmarks) to ultimately decide on what's acceptable.

Cheers,

	Jean-Marc

Herve Taddei wrote:
> Let me correct some statements made on the WMOPS counting as it seems to 
> me they are either wrong or you are not looking to the tool we are used 
> to have. So I am not sure your comments are based on G.191, if that is 
> the case, I have asked someone heavily involved in the design of ITU-T 
> G.191 to answer some of your points.
> 
>  
> 
> In general, in order to "automate" the evaluation of the complexity of 
> an algorithm on a family of processor architectures, the fixed point 
> code needs to be written in certain ways. If evaluating the complexity 
> of a codec is an important factor in the standardisation process, then 
> such coding guidelines should be agreed by the various parties involved 
> in the standardisation.
> W/ regards to fractional vs interger multiply : ITU STL 2000 supports 
> few integer multiplication (L mult0, L_mac0, L_msu0). It is true that 
> the variety & numbers of fractional multiplication operators vs integer 
> multiplication operators is not equal. From historical point of view, 
> the "fractional" multiplication was priviledged. In 2003 (in preparation 
> of STL2005), there was a proposal to widen the set of integer 
> multiplication while taking care of well counting the 
> integer/fractional multiplication mode switch that may be needed on some 
> processors. This discussion was not carried on forward.
> 
>  
> 
> 1st of all, I don't believe that the STL2005 is designed for a TI DSP. 
> For example, during the STL2005 design process STM inputs were factored 
> in, among others on the reflector.
> representing more a 16b or 32b processor ? If I look at the STl2005 list 
> of operators which have a complexity > 1, I believe we fall in those 
> categories :
> - integer multiplication with truncation to 16 lsbs (reflects indeed 
> that STL is historically fractional based).
> - bit rotation operation
> I believe that with the renewed complexity weights made in 2005, the STL 
> is closer to a 32b processor than a 16b processor to the exception of 
> the 32x32b multiplication which is not having a complexity of 1 ... but 
> of 4 (Mpy 32 32 ss inside the 40b lib).
> 
>  
> 
>>  The floating point computational complexity is also very important 
> these days.
> 
> I think this is not correct as floating point implementation counters 
> were proposed in ITU-T (from VoiceAge actually) and have been used in 
> many standardization exercises for reporting floating point complexity.
> 
>  
> 
> Best regards
> 
>  
> 
> Herve
> 
> 
> ------------------------------------------------------------------------
>

---

From: stephen botzko <[email-masked]>
Date: Wed, 22 Jul 2009 10:53:44 -0400

The benefit of measured complexity is that it provides a precise definition
of the requirement that can be tested.  Even if it does not exactly map onto
a given processor, this can still be useful.

For instance, if you have two codec candidates, using a "rough consensus"
will make it very hard to reject one of them on the grounds that it doesn't
meet the complexity requirement.  My guess is that the result will either be
that complexity gradually creeps upwards, or that "rough consensus" will not
be achieved, totally blocking progress.

More generally, if you use a process where multiple codec candidates are
considered, "rough consensus" is not efficient - it is likely to be
unachievable if a significant fraction of the WG participants have
candidates in play.  This would be likely true even if the "signficant
fraction" is < 50%.

Regards,
Stephen Botzko
Polycom

On Wed, Jul 22, 2009 at 10:32 AM, Jean-Marc Valin <
[email-masked]> wrote:

> pointing those out. I agree that the revised version makes more sense and is
> closer to the a 32-bit DSP. I still see a few issues with STL2005 (e.g. lack
> of unsigned add/sub), but they are relatively minor compared to the older
> version.
>
> comes to conditional branches, some architectures will pay a fixed price for
> a branch taken, while others (e.g. x86) will have virtually no cost for an
> easily predicted branch and a huge cost for a branch on "random" data. In
> the end, WMOPS will give you a rough idea of the complexity, but I believe
> it is preferable to use rough consensus (possibly backed by some benchmarks)
> to ultimately decide on what's acceptable.
>
> Cheers,
>
>        Jean-Marc
>
> Herve Taddei wrote:
>
>> me they are either wrong or you are not looking to the tool we are used to
>> is closer to a 32b processor than a 16b processor to the exception of the

---

From: stephen botzko <[email-masked]>
Date: Wed, 22 Jul 2009 12:13:42 -0400

Adding requirements during the end-game may not be the best.

Also, I think the "agile" vs "waterfall" comparison does not really apply in
an SDO development, where the various participants do not have aligned
business goals.

Assuming that participants have a stake (e.g. want their own proposals to be
accepted), they will predictably argue for "rough consensus" to relax
requirements that their proposal might not meet, and for "strict assessment"
of requirements that they are sure they meet.  If there is competition for a
specific codec, it will become a mess.  A parallel might be 802.11n in the
IEEE, where failure to achieve consensus on drafts has been a huge issue.

Though perhaps this could wait until after the BOF.

Stephen Botzko
Polycom

On Wed, Jul 22, 2009 at 11:53 AM, Koen Vos <[email-masked]> wrote:

> Agree that rough consensus for complexity evaluation may not work when
> comparing very similar candidates.  However, there is a good chance that
> either there won't be very similar candidates, or the similar candidates
> will all be deemed to have acceptable complexity.  If that happens, the
> rough consensus approach will have saved us a large amount of work.  On the
> other hand, if we do run into a problem, we can -through rough consensus-
> create a detailed WMOPS requirement at that stage.
>
> I see the difference in approach as "waterfall" versus "agile".
>
> koen.
>
>
>>> is
>>> for
>>>> to
>>>> an
>>>> few
>>>> the
>>>>   > * it's designed for a 16-bit (TI) DSP and is not very
>>>>> respresentative
>>>>> of
>>>>>
>>>>>
>>>> For
>>>> of
>>>> the
>>>>   > * it completely ignore issues such as conditional branches, which
>>>>> can
>>>>> be
>>>>>
>>>>>
>>>> were

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 22 Jul 2009 13:44:12 -0400

stephen botzko wrote:
> Assuming that participants have a stake (e.g. want their own proposals 
> to be accepted),

One of the ideas which I (personally) hope we can achieve is a process that 
is more cooperative than competitive. If participants have no direct 
incentive (e.g. patent royalties) to get their technology in, it should be 
easier to achieve consensus on the result.

> they will predictably argue for "rough consensus" to 
> relax requirements that their proposal might not meet, and for "strict 
> assessment" of requirements that they are sure they meet. 

Note also that having strict requirements decided before the 
standardisation process does not necessarily mean a better process. You 
might just end up with participants pushing these requirements in the 
direction of the technology they are planning on proposing later.

> Though perhaps this could wait until after the BOF.

I think it's OK to have some early discussions one the process now, but I 
agree that most of these process issues should be decided after the BoF.

Cheers,

	Jean-Marc

---

From: stephen botzko <[email-masked]>
Date: Wed, 22 Jul 2009 14:00:04 -0400

inline-
Stephen Botzko
Polycom

On Wed, Jul 22, 2009 at 1:44 PM, Jean-Marc Valin <
[email-masked]> wrote:

> stephen botzko wrote:
>
>> Assuming that participants have a stake (e.g. want their own proposals to
>> be accepted),
>>
>

Note the new charter has a different IPR statement from the original. Though
I think indirect incentives can be just as powerful as direct ones.

>
>
>  they will predictably argue for "rough consensus" to relax requirements
>> that their proposal might not meet, and for "strict assessment" of
>> requirements that they are sure they meet.
>>
>

 I agree that participants would do precisely that.  Though it is perhaps
more efficient to have that discussion up front, and not allow it to be
re-visited at the end.

---

From: Koen Vos <[email-masked]>
Date: Wed, 22 Jul 2009 08:53:09 -0700

Agree that rough consensus for complexity evaluation may not work when  
comparing very similar candidates.  However, there is a good chance  
that either there won't be very similar candidates, or the similar  
candidates will all be deemed to have acceptable complexity.  If that  
happens, the rough consensus approach will have saved us a large  
amount of work.  On the other hand, if we do run into a problem, we  
can -through rough consensus- create a detailed WMOPS requirement at  
that stage.

koen.