Subject: Re: [codec] Testing: A Novel Proposal

From: Monty Montgomery <[email-masked]>
Date: Tue, 19 Apr 2011 03:47:31 -0400

> The results presented so far are good, the actual tests however are debatable.

So far, the people who want to use the codec have been testing it the
way they'd want to use it.  That's not complete use case testing, but
I wager it covers the most likely uses.

> Testing is not delaying the process, it is part of the process.

That is the form of testing that has been and is being performed; lean
and targeted in-process testing that immediately feeds back into
further perfection of the codec at hand.  Results have been used
progressively and immediately.

I can't agree with halting development for years of distributed
testing across dozens of officially accredited labs.  This form of
testing is welcome as well, but it will not produce a better codec. It
will merely provide bullet points for subsequent marketing materials
down the road (as well as inform development of the next generation of
codecs of course; it is a mistake to be too short-sighted.  But it
won't improve Opus.)

>> 4) perfection is the enemy of good
>>
> So is imperfection.

I have an interesting proposal.   By way of grandiose schoolyard
bravado, in the interests of sparking a more productive direction of
debate:

"If it isn't good enough... Prove it."

The group of proponents actively participating in the development of
Opus have test results showing that so far Opus is doing very well
indeed.

A group of skeptics that happen to be outside the development process
are calling for a halt to give time for further evaluation.

Testing is important OF COURSE, but so far the abstract test plans
suggested, as opposed to the tests actually being performed, are calls
for a complete and indefinite halt to progress (with the side effect
that it will lead to a delay of wider real-world testing as favored by
the proponent faction). On which side is the burden of proof at this
point?  The proponents with good data or the more cautious faction
with an abstract five year plan?  Both sides favor testing.  The
skeptics simply want a different process that the proponents see as
directly harmful to progress... and actual testing.

So here's a Novel Idea: Perhaps what we really need to move things
forward is a 'Devil's Advocate' office, a blessed and fully _parallel_
testing 'faction' that is out to make Opus look as bad as possible (in
the name of progress) and has a responsibility of keeping up with
development.  This will be immensely useful.  It also solves the
problem of "I think it isn't good enough, but it's up to you to prove
it for me."

Any takers?  I have a hat with little horns on it.

Monty
(Individual)

---

From: Koen Vos <[email-masked]>
Date: Tue, 19 Apr 2011 10:01:35 +0200 (CEST)

> "If it isn't good enough... Prove it."

+1 

koen.

----- Original Message -----
From: "Monty Montgomery" <[email-masked]>
To: "Anisse Taleb" <[email-masked]>
Cc: [email-masked]
Sent: Tuesday, April 19, 2011 12:47:31 AM
Subject: Re: [codec] Testing: A Novel Proposal

Monty
(Individual)
_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 19 Apr 2011 10:38:28 +0000

Dear Monty,
 
If you refer to the draft proposal, we are in a presence of a multimode multi-bandwidth codec, the per-bandwidth tests proposed are small in comparison to other tests for other codecs, however putting all these together makes the test quite large. For instance, ITU-T codecs usually address a single bandwidth.

It was never the intention to make this an impossible test/task neither was the intention to delay the publication of this codec. I am disappointed that some of the codec proponents and supporters are defensive and quite dismissive. This is an opportunity for everyone to show and demonstrate the quality of the codec in a hopefully well designed test that could be reproduced at will and hence not be easily dismissed or argued against.

As I repeatedly said this was an initial proposal and I value the technical comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in and outside this mailing list. I am working on an update of that test plan and is hopeful to send a version soon. Please let me know if you have any comments such that I can accommodate your point of view too, even if you have little horns :-).

When it comes to your proposal,

I do not think I have anything to say about that except perhaps to remind you of your own statements 2 years ago, in response to Roni, 

"a lot of characterization, testing, and documentation needs to be done. We're aware of that and we have a lot of work to do."

Kind regards,
/Anisse

---

From: Ron <[email-masked]>
Date: Tue, 19 Apr 2011 22:14:32 +0930

Hi Anisse,

On Tue, Apr 19, 2011 at 10:38:28AM +0000, Anisse Taleb wrote:
> If you refer to the draft proposal, we are in a presence of a multimode
> multi-bandwidth codec, the per-bandwidth tests proposed are small in
> comparison to other tests for other codecs, however putting all these
> together makes the test quite large. For instance, ITU-T codecs usually
> address a single bandwidth.

I don't think there's really all that much confusion as to how you arrived
at the particular initial combination of tests that you did.  The important
part now though is determining what key things you hoped to learn from them
and figuring out how you might distill that into something tractable, which
somebody might actually perform and present to the group.

Perhaps you could pick some representative tests out of that set, and use
those to perform some initial trials, to guide selection of a second round
of tests?

> It was never the intention to make this an impossible test/task neither was
> the intention to delay the publication of this codec. I am disappointed that
> some of the codec proponents and supporters are defensive and quite
> dismissive. This is an opportunity for everyone to show and demonstrate the
> quality of the codec in a hopefully well designed test that could be
> reproduced at will and hence not be easily dismissed or argued against.

I don't think I've seen anyone on either side of any argument here, ever
be dismissive of testing.  And the "proponents" that I've spoken to are
actually, in all good faith not at all feeling defensive about this codec.

We all really, really, do want you to throw the absolute worst you can
conceive of at it.  Because that's exactly how it got to be as good as
it is already.

So please, please, please, please, please.  With a new beach-car on top.
Please just do whatever test you think is the most significant that has
so far been omitted.

That's the one we all really want to see the results of.  But we can't
tell you what it should be, or argue with you over what it shouldn't.
We're all happy to give you what advice we can.  But at the end of the
day, you have to decide, you or your peers need to conduct it, and then
we all need to see what it is that you've really taught us.

That's how engineering works, right?

> As I repeatedly said this was an initial proposal and I value the technical
> comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in and
> outside this mailing list. I am working on an update of that test plan and is
> hopeful to send a version soon. Please let me know if you have any comments
> such that I can accommodate your point of view too, even if you have little
> horns :-).

Some of us may keep ours sharp -- but I assure you we really all are
eagerly awaiting the results of the best test you can perform.  And
have no sane reason to wish you anything but well in that task.

> When it comes to your proposal, "If it isn't good enough... Prove it."
> 
> I do not think I have anything to say about that except perhaps to remind you
> of your own statements 2 years ago, in response to Roni, 
> 
> "a lot of characterization, testing, and documentation needs to be done.
> We're aware of that and we have a lot of work to do."

And I remember somebody saying that G.711 should be good enough for anyone.

There's been two years of characterisation, testing, and documentation done
to show that was utter bunkum.  We really want you to show us what we've
missed.  It's our turn now, not to believe *you* can do it, until you do! :)

But really, I do mean that in the spirit of really hoping you do find some
thing that lets us make it better.  This is why we get this stuff reviewed
by experts - we really do want to see if your best game is better than ours.

Doesn't that sound like fun to you?

Respectfully,
Ron

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 19 Apr 2011 09:51:49 -0400

I don't think this testing process is intended to be adversarial, it
certainly is not for me.

Also, I think the main goal is to benchmark the final codec's performance,
not so much to drive improvements.

Regards,
Stephen Botzko

---

From: Monty Montgomery <[email-masked]>
Date: Wed, 20 Apr 2011 06:27:43 -0400

> It was never the intention to make this an impossible

I didn't think so-- but the inadvertant result was initially a
proposal from an expert that was statistically unwinnable as written.
That caused a profound double-take for several people in the Opus
development camp.   I hadn't yet seen a strong statement on the list
of 'Oops, that was thoroughly unintended' as opposed to several 'oh,
pffft, it's not that bad'.  It was that bad, and when the problems
were downplayed on the list, developers prepared for a fight.

> test/task neither was the intention to delay the publication of this codec.

Thank you for making that clear.  Despite trying to follow the
discussion closely, I was also certain there has been an explicit call
for a halt to development (or publication) while testing was planned
and performed.  I am glad to hear there was no such suggestion.

> I am disappointed that some of the codec proponents and supporters are defensive and quite dismissive.

Several of the codec proponents currently feel the same way about the
skeptics.  More than one has privately expressed frustration that
their testing work and results were being casually dismissed.

> This is an opportunity for everyone to show and demonstrate the quality of the codec in a hopefully well designed test that could be reproduced at will and hence not be easily dismissed or argued against.

More than a few participants believed that this was a call to stop
codec work to run a large, unwinnable test suite to be passed before
proceeding any further.  If this is to be characterization testing,
and it's to be performed in parallel with publication and further
work, that is an entirely different situation.  I am glad you have set
the matter straight.

> "a lot of characterization, testing, and documentation needs to be done. We're aware of that and we have a lot of work to do."

The time to formally characterize is coming but not yet here; it is
not wrong to be planning, especially as this appears to be your
motivated interest. I would not dream of holding you back, but this
has certainly started off on the wrong foot.  We'll get past it.  I
still think the Devil's Advocate suggestion has a place.  For one
thing, in a DA scheme the skeptics would not appear to be so
completely disengaged from development; appearances shape reality and
should be managed.

In-process feedback testing has been continuous for several years
before the input groups came to the WG, and has ramped up considerably
since the WG formed.

Documentation-- ah, well, we can always do a better job of that.  That
said, I had not noticed any detailed discussion of the spec from
outside the core development group-- indeed this is my primary concern
at the current stage of development, and I have been a bit flummoxed
that post-final characterization testing seemed to be a subject of
urgent debate when detailed spec input and review is far more
immediately important.

As a final observation from someone who manages some of the Opus
developers: The implication on the list that the developers have not
been hard at work and producing code in an intelligent fashion ruffled
some feathers to put it mildly.   In addition, we are and have been
concerned about a lack of detailed, concrete review and debate on the
finer technical points of the proposed codec [as opposed to the time
spent reopening benchmarks supposedly settled prior to the WG
charter].

Monty

---

From: Monty Montgomery <[email-masked]>
Date: Wed, 20 Apr 2011 06:37:05 -0400

> I don't think this testing process is intended to be adversarial, it

Also, I what I was proposing with a Devil's Advocate was not so much
authentically adversarial, but more akin to a 'war game'.  Or a
white-hat hacking contest.

> not so much to drive improvements.

I concur, and I hope some substantial confusion has cleared on that issue.

Monty