Subject: MINOR ISSUE: Putting auditing before criteria

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 12 Jun 2003 22:47:10 +0200

Section 2.2 of -issue- reads:

   Some of the key areas where the IETF's practices appear to need
   tightening up include:

   o  Lack of explicit quality auditing throughout the standards
      development process.

   o  Lack of written guidelines or templates for the content of
      documents (as opposed to the overall layout) and matching lists of
      review criteria.

   o  Poorly defined success criteria for WGs and individual documents.

ISSUE: Quality auditing can only be done by auditing to criteria or 
guidelines. You can't make good guidelines without knowing your success 
criteria.
SUGGESTED RESOLUTION: Swap the first and third bullets in the list, and 
change "quality auditing" to "auditing against criteria for success". Might 
want to reword the sequence so that "quality" still appears.

---

From: John C Klensin <[email-masked]>
Date: Fri, 13 Jun 2003 14:28:34 -0400

--On Thursday, 12 June, 2003 22:47 +0200 Harald Tveit Alvestrand 
<[email-masked]> wrote:

>    Some of the key areas where the IETF's practices appear to
> need    tightening up include:
>
>    o  Lack of explicit quality auditing throughout the
> standards       development process.
>
>    o  Lack of written guidelines or templates for the content
> of       documents (as opposed to the overall layout) and
> matching lists of       review criteria.
>
> ISSUE: Quality auditing can only be done by auditing to
> criteria or guidelines. You can't make good guidelines without
> knowing your success criteria. SUGGESTED RESOLUTION: Swap the
> first and third bullets in the list, and change "quality
> auditing" to "auditing against criteria for success". Might

Harald,

I think this suggestion is correct, but want to caution about 
another aspect of it.  In the "quality improvement" and "quality 
assurance" communities, a distinction is often made between 
those activities and "quality auditing".  The cynics suggest 
that quality auditing is about being able to identify the point 
at which something went wrong, sometimes to more efficiently 
cast blame, but not about either better quality or guarantees of 
quality.

Unless we have a rather specific plan about what to do with the 
audits and how to incorporate their results into an improvement 
process, such audits are likely to rapidly deteriorate into 
meaningless, time-wasting, bureaucracy and improvements in 
blame-casting processes.   We don't, IMO, need either of those.

So, if your change is made, I suggest that a fourth bullet is 
needed, which might read something like:

	o Lack of adequate processes for feeding the results of
	quality audits forward into process improvements and
	success criteria.

In plain English, "quality audits", at their very best, have to 
do with understanding the nature of the mistakes we have made 
and when those mistakes occur in the process; if we are going to 
do them, we need to improve the mechanisms for _learning_ from 
those mistakes.

My personal preference, I think, would be to get rid of the 
notion of "explicit quality auditing" entirely and to focus this 
subsection on the need for processes that iteratively improve 
our criteria for success and for measuring progress and for 
mechanisms for measuring performance (of WGs, WG Chairs, 
Editors, ADs, etc) against those criteria.

As an example (possibly pointing toward solutions, but intended 
only to illustrate what I'm talking about), the IESG often has a 
clear sense of WGs that have gone very well and others that have 
gone badly.  Sometimes the Chairs of those WGs share that 
perspective, sometimes they rate things differently.  But it is, 
I think, rare for an AD to sit down with the leadership of a WG 
that has reached its end (or earlier) for an in-depth 
conversation and analysis of what went well, what went poorly, 
and what can be learned about how to better facilitate WG work 
in the future.  I think that is a problem.  But it isn't about 
"quality audits" as that term is used by professionals in those 
fields.

       regards,
          john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 13 Jun 2003 21:01:53 +0200

--On fredag, juni 13, 2003 14:28:34 -0400 John C Klensin 
<[email-masked]> wrote:

> In plain English, "quality audits", at their very best, have to do with
> understanding the nature of the mistakes we have made and when those
> mistakes occur in the process; if we are going to do them, we need to
> improve the mechanisms for _learning_ from those mistakes.

hmm... actually you point at something that touches on a lot of 
perspective.....

> My personal preference, I think, would be to get rid of the notion of
> "explicit quality auditing" entirely and to focus this subsection on the
> need for processes that iteratively improve our criteria for success and
> for measuring progress and for mechanisms for measuring performance (of
> WGs, WG Chairs, Editors, ADs, etc) against those criteria.
>
> As an example (possibly pointing toward solutions, but intended only to
> illustrate what I'm talking about), the IESG often has a clear sense of
> WGs that have gone very well and others that have gone badly.  Sometimes
> the Chairs of those WGs share that perspective, sometimes they rate
> things differently.  But it is, I think, rare for an AD to sit down with
> the leadership of a WG that has reached its end (or earlier) for an
> in-depth conversation and analysis of what went well, what went poorly,
> and what can be learned about how to better facilitate WG work in the
> future.  I think that is a problem.  But it isn't about "quality audits"
> as that term is used by professionals in those fields.

I left this long quote in because I think it illustrates a point....

there are really multiple purposes at work, at multiple different levels.

One limitation/advantage of our WG process is that it tends to bite off 
reasonable-sized chunks of problem space, chew on them for a while, and 
spit out documents. *Inside* that process, the purpose of auditing, reviews 
or whatever we choose to call it is to make sure the result comes out right 
- and the watchword seems to be "early clue injection" and "no late 
surprises"; the review process needs to be rapidly fed back to chairs, WGs 
and document editors, and written reviews are just so many scraps of 
electronic paper.

However, the WG is always a component in a much *larger* process, dealing 
with the formation, relation, dissolution and retargeting of working groups 
and activities - and it is that landscape of activity that at the moment 
has very little explicit management at any level between the IESG and the 
WG. It is at this level that "learning from the mistakes of the previous 
WG" comes in - not so much as feedback to *this* WG, but as a lesson in 
what to do or not do next time.

It's a level where I believe the IETF has been flailing for a long time - 
the phenomenon of "permanent area-like WGs" is one symptom of one attempt 
to solve the problem without being explicit about it.

At this level, recording history may become important - storing not only 
reviews, but the drafts they were commenting on, as examples of what to 
avoid and why not.... (if anyone ever bothers reading them.....)?
Because the purpose of this review is not to correct *this* action; it is 
to guide the *next* action.

An entirely different kettle of fish.