Subject: [codec] Comments on draft-valin-codec-guidelines-05.txt

From: Stephan Wenger <[email-masked]>
Date: Sat, 21 Aug 2010 11:55:07 -0700

Hi all,
In Maastricht we did not accept this draft as a WG item based on lack of
feedback.  So let me provide some feedback here.  These notes have been
taken while going through the document and have not been sorted according to
my priorities.

1. 1. Introduction (editorial): the last sentence of the first paragraph and
the whole second paragraph are redundant and can be removed for conciseness.
2. 2. Development process, first sentence, ³codec²->²wideband speech/audio
codec² or whatever we came up with; it appears more than unclear to me that
the document in question makes sense for anything else but this project, so
the limit of scope should be made clear from the outset.
3. Development process, subsection 2, ³although any contributions made in
accordance with²: the correct reference here would be RFC 2026 / BCP9, not
the 78/79 pair 
4. 2. Development process, subsection 2, ³If there is any IPR related to
[...]².  This sentence both second-guesses and incorrectly interprets the
scope and intention of BCP79.  For example, the term ³IPR² is commonly
understood as including all proprietary rights and not only patents, whereas
BCP79 is solely concerned about patents (copyright issues are dealt with in
BCP78 and elsewhere).  Or, BCP79 does not set a clear timeline for a patent
disclsoure.  Or, the patent disclosure is bound to individuals making the
contribution, not to contributions itself.  As a result, there is no
requirement for making a patent disclosure when submitting a contribution
for a disclosure if the submitter is not aware of IPR.
5. In response to items 3 and 4, I will send to the list a suggested
replacement paragraph.
6. 2. Development process, subsection 3, there is a bit too much definite
future tense for my liking.  Perhaps softening of the language would align
the draft closer to reality.  For example, it is by no means clear to me
that a group ³will² gain a clearer understanding of things based on
reception and discussion of contributions.
7. 2. Development process, subsection 5, suggestion: this ³starting point²
could be expressed in accepting a technology draft as a WG item.
8. 2. Development process, subsection 6, again definite future tense.
9. 2. Development process, subsection 8, I¹m not sure that a WGLC is the
right time to start characterization.  IMHO, characterization should start
way before the WGLC so that the WG has the (independently verified) results
on hand before making a decision in the WGLC.  Characterization results
should ideally be captured in a draft destine for informational RFC and be
presented to the IESG at the same time as the technology draft
10. 3. Evaluation, Testing, and Characterization, characterization and
³final codec²: I¹m a bit nervous about the language here, because it seems
to apply that we are not going back to the drawing board of the
characterization doe snot yield a result acceptable to the WG (or the IESG
for that matter).  Suggest to remove ³final².
11. 3. Evaluation, Testing, Characterization: the draft distinguishes
between three types of tests, objective, subjective, and full-blown
characterization.  That distinction makes sense to me.  What would probably
be helpful is that the draft would make it even clearer what type of
confidence one gets from each test, and what effort is involved.  It may
also be helpful to set minimum bars on the characterization as the highest
level of confidence test we have here (in terms of number/properties of
listeners, languages, ...).  In this regard, I note that the typical
characterization description in codec work outside of the IETF is many pages
long.  While we can certainly set our own rules, the term ³characterization²
implies, to me, a certain level of work.  It¹s not so that we need to define
the characterization process before accepting the subject draft as a WG
item, but there should be a placeholder, expressing the expectation that the
characterization definition is going to grow significantly before
characterization is actually being performed.
12. 4. Requirements, ...: suggest to use arabic numbers and subsections to
distinguish the many unrelated points in tis paragraph
13. 4. Requirements Conformance, second paragraph, ³normative reference
implementation².  While the intention is (hopefully) clear and agreeable to
me, the formulation is misleading.  Suggest to change to ³reference
implementation, documented in an Internet Draft destined for standards track
RFC and that is going to be normatively referenced by the technology RFC².
Or something thereabouts.
14. 4. Requirements..., third paragraph, ³normative².   ³normative
requirements² is wrong here.  What we are talking here are neither
requirements, nor are they normative.  Tightening up the language is left an
an exercise to the reader. :-)
15. 4. Requirements..., fourth paragraph: I don¹t think we have consensus re
bit-exact decoders (or lack thereof).  There is a whole can of licensing
worms attached to non-bit-exact decoders.  Suggest to strike this paragraph,
or, at least, document the lack of WG consensus before accepting this as WG
item (alternatively, force consensus now, but I don¹t think all the facts
are on the table and sufficiently discussed yet).
16. 4. Requirements..., sixth paragraph (starting with ³fourth²): I would
prefer to remove this profiling stuff from the draft for two reasons: first
this is the classic example of stuff where the real world commonly deviates
from what the standard specifies; why add to the list of badly designed
standards?  And, second, at this point, this language unnecessarily limits
the freedom of the WG in the design of the codec.
17. 4. Requirements..., last paragraph: parts of this is redundant with
information previously provided, and other parts do not seem to have WG
consensus 
18. 5. IPR: my previous comments have been addressed, I¹m happy with the
section. 
19. 6. SDO Relationship: language is fine.
20. 7. Security: could use editorial improvement, but the spirit is right.
21. IANA, Acks, references, ..., all fine.

Stephan

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 23 Aug 2010 08:13:49 -0400

Hi,

I've submitted a new version of the draft based on Stephen's comments: 
http://www.ietf.org/id/draft-valin-codec-guidelines-06.txt . I made most 
of the requested changes, with the exception of some points for which I 
don't believe the WG has reached consensus yet. See below for my 
comments on each point.

On 10-08-21 02:55 PM, Stephan Wenger wrote:
> taken while going through the document and have not been sorted
> according to my priorities.
>
>    1. 1. Introduction (editorial): the last sentence of the first
>       paragraph and the whole second paragraph are redundant and can be
>       removed for conciseness.

Agreed. Text removed.

     2. 2. Development process, first sentence, “codec”->”wideband
>       speech/audio codec” or whatever we came up with; it appears more
>       than unclear to me that the document in question makes sense for
>       anything else but this project, so the limit of scope should be
>       made clear from the outset.

Fixed.

>    3. Development process, subsection 2, “although any contributions
>       made in accordance with”: the correct reference here would be RFC
>       2026 / BCP9, not the 78/79 pair
>    4. 2. Development process, subsection 2, “If there is any IPR related
>       to [...]”. This sentence both second-guesses and incorrectly
>       interprets the scope and intention of BCP79. For example, the term
>       “IPR” is commonly understood as including all proprietary rights
>       and not only patents, whereas BCP79 is solely concerned about
>       patents (copyright issues are dealt with in BCP78 and elsewhere).
>       Or, BCP79 does not set a clear timeline for a patent disclsoure.
>       Or, the patent disclosure is bound to individuals making the
>       contribution, not to contributions itself. As a result, there is
>       no requirement for making a patent disclosure when submitting a
>       contribution for a disclosure if the submitter is not aware of IPR.

Suggestion integrated as is.

>    6. 2. Development process, subsection 3, there is a bit too much
>       definite future tense for my liking. Perhaps softening of the
>       language would align the draft closer to reality. For example, it
>       is by no means clear to me that a group “will” gain a clearer
>       understanding of things based on reception and discussion of
>       contributions.

Should be better now.

>    7. 2. Development process, subsection 5, suggestion: this “starting
>       point” could be expressed in accepting a technology draft as a WG
>       item.

Ok. Changed to "accept a baseline codec as a WG item".

>    8. 2. Development process, subsection 6, again definite future tense.

s/will/should/

>    9. 2. Development process, subsection 8, I’m not sure that a WGLC is
>       the right time to start characterization. IMHO, characterization
>       should start way before the WGLC so that the WG has the
>       (independently verified) results on hand before making a decision
>       in the WGLC. Characterization results should ideally be captured
>       in a draft destine for informational RFC and be presented to the
>       IESG at the same time as the technology draft
>       “final codec”: I’m a bit nervous about the language here, because
>       it seems to apply that we are not going back to the drawing board
>       of the characterization doe snot yield a result acceptable to the
>       WG (or the IESG for that matter). Suggest to remove “final”.
>       between three types of tests, objective, subjective, and
>       full-blown characterization. That distinction makes sense to me.
>       What would probably be helpful is that the draft would make it
>       even clearer what type of confidence one gets from each test, and
>       what effort is involved. It may also be helpful to set minimum
>       bars on the characterization as the highest level of confidence
>       test we have here (in terms of number/properties of listeners,
>       languages, ...). In this regard, I note that the typical
>       characterization description in codec work outside of the IETF is
>       many pages long. While we can certainly set our own rules, the
>       term “characterization” implies, to me, a certain level of work.
>       It’s not so that we need to define the characterization process
>       before accepting the subject draft as a WG item, but there should
>       be a placeholder, expressing the expectation that the

These were the subject of some debate at the previous meeting, so I'm 
not addressing it right now.

>   12. 4. Requirements, ...: suggest to use arabic numbers and
>       subsections to distinguish the many unrelated points in tis paragraph

Just put everything in a list. How's that?

>   13. 4. Requirements Conformance, second paragraph, “normative
>       reference implementation”. While the intention is (hopefully)
>       clear and agreeable to me, the formulation is misleading. Suggest
>       to change to “reference implementation, documented in an Internet
>       Draft destined for standards track RFC and that is going to be
>       normatively referenced by the technology RFC”. Or something
>       thereabouts.

Not sure what you mean by "and that is going to be normatively 
referenced by the technology RFC" (what technology RFC?), so I'm leaving 
this out for now.

>   14. 4. Requirements..., third paragraph, “normative”. “normative
>       requirements” is wrong here. What we are talking here are neither
>       requirements, nor are they normative. Tightening up the language
>       is left an an exercise to the reader. :-)

I'm talking about minimizing the number of MUSTs required to be 
compliant with the standard. How would you phrase that?

>   15. 4. Requirements..., fourth paragraph: I don’t think we have
>       consensus re bit-exact decoders (or lack thereof). There is a
>       whole can of licensing worms attached to non-bit-exact decoders.
>       Suggest to strike this paragraph, or, at least, document the lack
>       of WG consensus before accepting this as WG item (alternatively,
>       force consensus now, but I don’t think all the facts are on the
>       table and sufficiently discussed yet).

IIRC most WG participants agreed that we didn't need to "force" 
implementers to be bit-exact as long as they had bit-exactness as an 
option to simplify implementation. After all, forcing bit exactness 
means one can never use floating-point.

>   16. 4. Requirements..., sixth paragraph (starting with “fourth”): I
>       would prefer to remove this profiling stuff from the draft for two
>       reasons: first this is the classic example of stuff where the real
>       world commonly deviates from what the standard specifies; why add
>       to the list of badly designed standards? And, second, at this
>       point, this language unnecessarily limits the freedom of the WG in
>       the design of the codec.

Already saying "Unless a particular profile is defined", not that there 
will be profiles. I don't think we've ever discussed whether profiles 
are acceptable or not, so I'm leaving this hear. That being said, the 
codec we proposed does not have profiles.

>   17. 4. Requirements..., last paragraph: parts of this is redundant
>       with information previously provided, and other parts do not seem
>       to have WG consensus

Not sure what you see as the problem here.

>   18. 5. IPR: my previous comments have been addressed, I’m happy with
>       the section.

Thanks for taking the time to go over this.

	Jean-Marc

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 24 Aug 2010 11:39:17 -0700

Hi Jean-Marc, all,
Thanks for the updated version.  Inline a few comments based on your email,
and not necessarily the updated draft:

On 8.23.2010 05:13 , "Jean-Marc Valin" <[email-masked]>
wrote:

> [...]
> 
>>    9. 2. Development process, subsection 8, I¹m not sure that a WGLC is
>>       ³final codec²: I¹m a bit nervous about the language here, because
>>       WG (or the IESG for that matter). Suggest to remove ³final².
>>       term ³characterization² implies, to me, a certain level of work.
>>       It¹s not so that we need to define the characterization process

That's fine and the right approach.  However, I believe we need to come to a
consensus of some form of a working assumption for all three questions
before it makes sense to accept the draft as a WG document.

Here, and also later, I use the term "technology draft" or "technology RFC"
for the textual description of the final codec; it's called "Specification
of a codec" in the charter.  The charter currently does not reflect directly
the (perceived?) need for a permanent document covering the reference
implementation.  I'm in favor of such a document (std track or informational
RFC), and I'm in favor of putting the code into its own, code-only document
and not into a section of the specification RFC, for size reasons (one never
knows--perhaps there will be a FPO and an integer reference implementation
at some point, or more than one specifictaion covering different operation
points, and so on).

> Just put everything in a list. How's that?

OK

> 
>>   13. 4. Requirements Conformance, second paragraph, ³normative
>>       reference implementation². While the intention is (hopefully)
>>       to change to ³reference implementation, documented in an Internet
>>       normatively referenced by the technology RFC². Or something
>>       thereabouts.
> 

See above.

The issue with "normative reference implementation" is mostly a textual one:
it's too close to a "normative reference", which is something entirely
different. 

> compliant with the standard. How would you phrase that?

Perhaps out of content, but that is what I think we could say: "It is the
intention of the group to allow the greatest possible choice of freedom in
implementing the specification.  Accordingly, the number of binding RFC2119
keywords is going to be the minimum still allowing for interopable
implementations."

You know that you are binding your hands with such a formulation?  In a
highly contested area, all kinds of political compromises end up in deals
such as "if you MUST implement my patent, then I MUST implement yours" and
so on.  (Although the opposite case--removing MUST requirements in favor of
unmotivated SHOULD or MAY is much more the norm.)

> 
>>   15. 4. Requirements..., fourth paragraph: I don¹t think we have
>>       force consensus now, but I don¹t think all the facts are on the

Up to the chairs to make a consensus call.

> 
>>   16. 4. Requirements..., sixth paragraph (starting with ³fourth²): I

I don't care about your proposal, nor should you as the editor of this
draft.  This is one of the problems we have with the composition of this
WG...
That said, on the subject matter, I'm fairly neutral, and I should have
expressed myself.  What I aimed to say was that I don't think we have any
consensus in this area yet.

> Not sure what you see as the problem here.

Let me give it a stab:

I believe the business intention is not to require error concealment at all.
Accordingly, there cannot be a conformance test for error concealment.  It
can be subject to characterization tests, though...

So this is what could go in 4.4:
To ensure freedom of implementation, decoder-side only error concealment
does not need to be specified, although an informal description of the
reference implementation's PLC algorithm, if any, is desirable.  Obviously,
any information signaled in the bitstream intended to aid PLC needs to be
specified.

And move the sentence about minimum requirements of PLC efficiency to an
appropriate place in section 3.

> [...] 

Stephan

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 24 Aug 2010 18:06:48 -0400

Hi Stephen, everyone,

> before it makes sense to accept the draft as a WG document.

We need consensus on everything before going to WGLC, but I don't see why 
we need it for having this as a WG item.

> at some point, or more than one specifictaion covering different operation

We definitely need a place to put the reference implementation. I also 
agree that having the entire code in the appendix of an RFC is a bit 
inconvenient. Now the question is where should that code live? Does the 
IETF have a more convenient way to hold source code than 72-column ASCII? 
Any suggestion on where to put the reference implementation? There's also 
the question of what the reference implementation should look like. Should 
it be just floating point, just fixed-point, both? Should it be reasonably 
optimised (without anything platform-specific) or should the code be 
optimised for readability rather than speed?

> keywords is going to be the minimum still allowing for interopable

Thanks for the suggestion. I've added it as is.

> unmotivated SHOULD or MAY is much more the norm.)

The reasoning for having as little binding keywords is mainly technical. It 
will allow future encoders to produce better quality audio than what the 
reference encoder will do. MP3 is a good example of that. Current encoders 
are *much* better than the original ones (which were already much better 
than the really bad reference implementation).

> Up to the chairs to make a consensus call.

Well, that's in the minutes anyway. I remember people saying that 
bit-exactness can be *useful* and I agree. The current text just says that 
that you don't *have* to be bit-exact to be conformant, it doesn't say you 
can't be if you want to save time on testing.

> consensus in this area yet.

Exactly. Which is why the current wording does not say whether profiles are 
good or bad. And even if/when we decide on having profiles or not having 
them, I don't think that belongs to the guidelines.

> any information signaled in the bitstream intended to aid PLC needs to be

Used your suggestion again, though I changed the first sentence to:

To ensure freedom of implementation, decoder-side only error concealment
does not need to be specified, although the reference implementation should
include the same PLC algorithm as used in the testing phase.

Just like the reference implementation needs to contain a working encoder 
(even though the encoder would not be strictly specified), I also think the
PLC should be part of the reference implementation (not just an informal 
description).

> appropriate place in section 3.

Left it in that section because it's still talking about normative issues, 
not really about testing to decide whether we did a good job with the codec.

Cheers,

	Jean-Marc