Subject: Re: [codec] draft test and processing plan for the IETF Codec

From: Paul Coverdale <[email-masked]>
Date: Tue, 19 Apr 2011 02:20:32 +0000

As I mentioned earlier, the situation is not as bad as it may seem, certainly not a "1 in nonillion" chance of passing all requirements. Greg's analysis applies to flipping a coin that has a probability of .90 for "heads" and .10 for "tails." However, listener responses in a MOS test are not random - if they are, we throw that listener out of the results. The "randomness" that forms the basis of a statistical test derives from the distribution of responses ACROSS listeners rather than WITHIN listeners.
 
Regards,
 
...Paul
 
>-----Original Message-----
>From: Anisse Taleb [mailto:[email-masked]]
>Sent: Monday, April 18, 2011 6:37 PM
>To: Jean-Marc Valin; Paul Coverdale
>Cc: [email-masked]
>Subject: RE: [codec] draft test and processing plan for the IETF Codec
> 
>JM, Greg, Paul,
>[taking emails in chronological order was ill advised :-)]
> 
>I do not disagree with the statistical pitfalls you mention. As Paul
>stated and also what I wrote in a direct reply to this, there is no
>single uber-requirement to be passed by the codec, rather a vector of
>requirements that summarize the performance of the codec compared to
>other codecs. These have to be analyzed and discussed one by one.
> 
>Kind regards,
>/Anisse
> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf
>Of
>> Jean-Marc Valin
>> Sent: Thursday, April 14, 2011 3:07 PM
>> To: Paul Coverdale
>> Cc: [email-masked]
>> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> 
>> > I don't think the situation is as dire as you make out. Your
>analysis
>> > assumes that all requirements are completely independent. This is
>not the
>> > case, in many cases if you meet one requirement you are likely to
>meet
>> > others of the same kind (eg performance as a function of bit rate).
>> >
>> > But in any case, the statistical analysis procedure outlined in the
>test
>> > plan doesn't assume that every requirement must be met with absolute
>> > certainty, it allows for a confidence interval.
>> 
>> This is exactly what Greg is considering in his analysis. He's
>starting
>> from the assumption that the codec really meets *all* 162
>requirements.
>> Consider just the NWT requirements: if we were truly no worse than the
>> reference codec, then with 87 tests against a 95% confidence interval,
>we
>> would be expected to fail about 4 tests just by random chance.
>Considering
>> both NWT and BT requirements, the odds of passing Anisse's proposed
>test
>> plan given the assumptions above are 4.1483e-33. See
>http://xkcd.com/882/
>> for a more rigorous analysis.
>> 
>> Cheers,
>> 
>>    Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 19 Apr 2011 03:38:46 +0000

Paul,

Strictly speaking, the probability of failing at least one requirement increases (or stays constant in dependent cases) with increasing the number of requirements. Of course as you mention, the responses of the listeners are not really random and a heads-tail modeling of the pass-fail is way too simplistic. I didn't look at the analysis in depth, neither did I verify the numbers.
I think Greg is mostly concerned with the size of the test and how the analysis of the requirements would be used to derive a conclusion about the codec.

[Off topic]
Statistics are quite fun to play with, a while ago a Conspiracy theorist tried to convince me that man never landed on the moon and that it was impossible that Apollo 11 made it. The millions of components, given the technology of that time, had  a significant probability of failure that in total it was beyond doubt that such a rocket would have gone off course.

Kind regards,
/Anisse

From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Coverdale
Sent: Tuesday, April 19, 2011 4:21 AM
To: [email-masked]
Subject: Re: [codec] draft test and processing plan for the IETF Codec

Regards,

...Paul

>From: Anisse Taleb [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
>Cc: [email-masked]<mailto:[email-masked]>
>> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf
>Of
>> Jean-Marc Valin
>> Cc: [email-masked]<mailto:[email-masked]>

---

From: Ron <[email-masked]>
Date: Tue, 19 Apr 2011 14:48:50 +0930

On Tue, Apr 19, 2011 at 03:38:46AM +0000, Anisse Taleb wrote:
> Paul,
> 
> Strictly speaking, the probability of failing at least one requirement
> increases (or stays constant in dependent cases) with increasing the number
> of requirements. Of course as you mention, the responses of the listeners are
> not really random and a heads-tail modeling of the pass-fail is way too
> simplistic. I didn't look at the analysis in depth, neither did I verify the
> numbers.  I think Greg is mostly concerned with the size of the test and how
> the analysis of the requirements would be used to derive a conclusion about
> the codec.

I think he's worried, as are many others, that it's sufficiently
sophisticated as to be indistinguishable from a rigged demo.

I think we can be equally worried that you haven't done this analysis
and yet still assert that this test is somehow more valid that the ones
which have already been conducted.

> [Off topic]
> Statistics are quite fun to play with, a while ago a Conspiracy theorist
> tried to convince me that man never landed on the moon and that it was
> impossible that Apollo 11 made it. The millions of components, given the
> technology of that time, had  a significant probability of failure that in
> total it was beyond doubt that such a rocket would have gone off course.

I guess you're also unaware of how many rockets they pranged before the
(g)odds did favour them with success, and how unrepeatable that success
subsequently proved to be.  And that they also didn't have a worried
cartel shrilling OMG patent! at them at every step of the way.

So can we please return to the topic at hand?

Do you or do you not have a test plan that you wish to conduct before
we reach the WG milestone of assessing the results of the currently
frozen candidate codec?  The clock is ticking, and you're wasting the
time that you do have remaining to perform that task and present its
results for the group to consider.

Cheers,
Ron

---

From: Monty Montgomery <[email-masked]>
Date: Tue, 19 Apr 2011 02:29:19 -0400

On Mon, Apr 18, 2011 at 10:20 PM, Paul Coverdale <[email-masked]> wrote:
> As I mentioned earlier, the situation is not as bad as it may seem,

I questioned Greg on the statistical assumptions he was making, and
they were ~ sound.  Conservative, in fact, along most axes.

> certainly not a "1 in nonillion" chance of passing all requirements. Greg's
> analysis applies to flipping a coin that has a probability of .90 for
> "heads" and .10 for "tails." However, listener responses in a MOS test are
> not random - if they are, we throw that listener out of the results.

...please be a great deal more descriptive/explicit on what you mean
here, and please don't dumb it down assuming a non-technical audience
without basic statistical analysis training.  Because what you just
said above set off multiple alarm bells.

Probability distribution functions are probability distrbution functions.

Monty
Xiph.Org

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 19 Apr 2011 09:47:02 +0000

Hi Monty,

> they were ~ sound.  Conservative, in fact, along most axes.

I understand the point that Greg was making, If we want to continue discussing this and taking those arguments as granted then please enlighten me, What are the statistical assumptions? Independence was surely assumed otherwise the equation does not hold. What are the axes you mention? 

> Probability distribution functions are probability distrbution functions.

Personally I see no issue with Paul's description. My understanding here, which is also my understanding of Greg's assumptions, is that the outcome of testing a requirement is a coin flip distribution, a.k.a. Bernoulli distribution, the outcome of which is a pass with a probability p=0.9 and fail with a  probability 1-p = 0.1 (I think the actual numbers were an example).  Did I miss something?

Kind regards,
/Anisse