Subject: [codec] Non-bitstream impacting encoder changes and testing

From: Gregory Maxwell <[email-masked]>
Date: Thu, 14 Apr 2011 00:21:19 -0700

I have some fairly minor patches queued up for which improve some
of the encoder signal analysis and should produce some minor quality
improvements.  I have more development testing to do before I consider
them done, but I've been dragging my feet because I'm not quite sure
how this sort of change will interact with testing.

In Opus the encoder behavior is non-normative— precisely so the
format could continue to enjoy encoder-side improvements (like we've
seen with Speex, Vorbis, Theora, and MPEG codecs). But I'm concerned
that some people might consider the testing which has already been
performed to be invalidated by this sort of improvement.

Rather than fall into a pit over this after the fact, I thought it would
be better to ask if WG participants have any views on this subject?

My view is that changes which do not change the bitstream/decoder should
not be considered to invalidate any testing. Even if a change is later
discovered to degrade the performance of the codec, an implementer is
free to use an older encoder, manually back out that change, etc. So,
basically, every test is at least evidence of what can be achieved with
the format even if the encoder is changed later. Good testing practice
should identify the versions used in any case.

The obvious alternatives I see are to simply not improve the encoder
right now, or to work on a separate encoder implementation (a fork of
the reference encoder). I don't either of these are especially good.

---

From: Anisse Taleb <[email-masked]>
Date: Mon, 18 Apr 2011 22:46:12 +0000

Dear Greg,

I personally have no problems with this, any improvement to the encoder is welcome. When It comes to adopting the codec as a standard, the code has to be frozen at some point such that formal testing can start on an agreed version of the codec. Any other tests performed with intermediate version are of course valuable, but will not reflect the quality of the codec as published by the IETF. 

Kind regards,
/Anisse 

> In Opus the encoder behavior is non-normative- precisely so the
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec