Subject: RE: Trusting the IESG to manage the reform process (was:Re:Doingthe Right Things?)

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:31:37 -0400

One solution is for the ADs to entrust more authority and empower chairs
to make stronger decisions on what should and should not move forward.
But then the AD would have to give up some of their authority too.

/jim

> -----Original Message-----
> From: Margaret Wasserman [mailto:[email-masked]] 
> Sent: Wednesday, June 04, 2003 10:06 AM
> To: Brian E Carpenter
> Cc: [email-masked]
> Subject: Re: Trusting the IESG to manage the reform process 
> (was:Re:Doingthe Right Things?)
> 
> 
> 
> Hi Brian,
> 
> At 02:49 PM 6/4/2003 +0200, Brian E Carpenter wrote:
> >I think there was something else. The IETF also put in place 
> mechanisms 
> >for renewal and accountability of the decision-taking group. 
> And that, 
> >if I'm not mistaken, was to reduce the incidence of hubris.
> >
> >In other words, there was no attempt to solve a scaling problem.
> >
> >What Harald is referring to is a scaling problem, imho.
> 
> Right.  With the growth of the IETF, the increase in 
> complexity and scope of our work, etc. the job of an AD has 
> become so large, time-consuming and complex that there are 
> very few people who can do it.  And, there are even fewer 
> people who are willing to do it...  We need to figure out 
> some way to solve this problem, and that almost certainly 
> involve some reorganization and re-definition of roles at the 
> top of the IETF.
> 
> This could be driven by the IESG -- they could create new 
> roles under themselves and delegate work.  Or, it could be 
> driven by the community.  I actually would have preferred the 
> former, but the IESG seems to have chosen the latter...
> 
> > > If the latter suffices, then in fact we continue to make 
> decisions 
> > > in the same way. We simply target different types of decisions to 
> > > different groups.
> >
> >...or simply give the existing decision-taking group better input to 
> >work with, such as fully reviewed and nit-free documents.
> 
> I think that we're all in agreement that the quality, 
> timeliness and predictability of WG output is unacceptably low.
> 
> It might be amusing to watch WG chairs come up with ways to 
> blame the IESG for this -- they didn't charter us properly, 
> they didn't manage us properly, they didn't give us early 
> feedback, etc... But, that's all just whining.  It is the 
> responsibility of the WG chair to manage the WG to produce 
> high quality and timely work, to get adequate review for our 
> documents, and to manage our milestones.  And, most of us are 
> falling down on the job.
> 
> As I'm sure you know, high quality, nit-free documents don't 
> happen by accident...  And, it doesn't matter how many times 
> we ask for them, or whine about the fact that our editors 
> don't produce them. We need to actually put some quality 
> processes in place during WG creation of these documents to 
> improve the quality of WG output.
> 
> IMO, though, this won't solve the scaling problems that have 
> lead to the ADs job being so large/complex that there is a 
> limited pool of people that can/will do it...  That needs to 
> be solved separately.
> 
> Margaret
> 
> 
> 
> 
>

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 12:10:28 -0700

> One solution is for the ADs to entrust more authority and empower
> chairs to make stronger decisions on what should and should not
> move forward.

that is not the problem i see day to day.  what i see is chairs not
wanting to take a stand against weak documents, ersatz populism
being easier, and preferring to let it through to ietf last call
and iesg, letting the iesg take the heat.

> But then the AD would have to give up some of their authority

rofl!  all the authority and fifteen cents will get you on the
subway [0].  but it seems not to make a squat difference to get
higher quality documents out from the wgs.

randy

---

[0] showing my age :-)

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 17:09:59 -0700

Randy Bush <[email-masked]> writes:
> subway [0]. 
I'm not sure what you're saying here. The ADs have immense
authority--the authority to block documents. Since companies and WGs
spend lots of time and money trying to get documents passed, the
ability to block 

> but it seems not to make a squat difference to get

As I said previously it's not at all clear to me that  having a
high quality document has much of an impact on how quickly
a document proceeds through the IESG. What incentives do WGs
have to produce high quality documents?

-Ekr

-- 
[Eric Rescorla                                   [email-masked]]
                http://www.rtfm.com/

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 17:11:50 -0700

> authority--the authority to block documents.

nice hyperbole, but the fact is no one blocks documents, we point
out what we think are problems and ask to discuss them to either
find out we have misunderstood or, should the problem be real,
actually get the problem addressed.  this is the same 'authority'
the author, editor, wg chair, or reviewer has.  big whoopie doo.

the question is what can be done to improve the quality of
documents.

> Since companies and WGs spend lots of time and money trying to
> get documents passed, the ability to block

artificial polarization.  as i said above, documents can be called
for discussion by 92.3% of the population.  the problem is that
some authors, editors, wg chairs, ... seem to prefer the lazy but
beauty contest winning path of saying yes to anything and then
hoping the iesg will take the heat for asking the hard questions.

randy

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 17:28:29 -0700

> the author, editor, wg chair, or reviewer has.  big whoopie doo.
Randy, this is disingenuous. A single reviewer cannot make a
document fail to progress, though they can make a big stink.
An AD can, and they frequently do. 

One example: 
The IESG held TLS for nearly a year because they insisted that
their be a mandatory algorithm and that it be DH/3DES/SHA.
No other group of 13 people in the IETF could have done that,
no matter how big a stink they raised. In point of fact,
2 or 3 ADs could have done this.

> the question is what can be done to improve the quality of
I agree that that's the question, but in order to address
that question we must first clearly understand what's 
happening at the moment.

> for discussion by 92.3% of the population. 
Yes, but most of those people can't kill it entirely if they
so choose. The IESG effectively can.

> the problem is that

This neither shocks nor surprises me. People respond to incentives.
I ask again: what incentives do the WGs have to produce documents
that meet the IESG's definition of quality?

-Ekr

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 17:41:16 -0700

> their be a mandatory algorithm and that it be DH/3DES/SHA.  No
> other group of 13 people in the IETF could have done that, no
> matter how big a stink they raised. In point of fact, 2 or 3 ADs
> could have done this.

false.  the authors could have.  the editor(s) if they existed
could have.  the wg chair(s) could have.  respected members of the
security directorate could have.  ...  and my guess, though i am
not a security expert, is that they probably SHOULD HAVE.

> that meet the IESG's definition of quality?

as a multi-decade manager of O(20^(2-3)) engineers, i came to the
conclusion that 82.3% of whether an engineer produces quality is
whether they have pride in their work and their team.  the classic
hiring manager's joke about engineers is that we ask "what is the
project and with whom will i be working.  ....  oh yes, my spouse
told me to ask about salary and benefits."

what keeps the cows in the pasture is the quality of the grass not
the height of the fence.  a fair portion of our culture is weak on
giving credit where it is due, saying "this was maude's idea, not
mine," etc.  when the culture was smaller in number, it was easier
for the work to be known by the person(s) who produced it.  this is
good and bad, depending on whether the work is good or not so good
(i will resist examples:-).  if we spent more of our time praising
and encouraging the good work of our peers as opposed to labeling
them as idiots, black helicopter pilots, etc. perhaps we all would
take the necessary corrective constructive criticism a bit better
and the results would be better.

randy

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 17:52:25 -0700

Eric,

ER> I ask again: what incentives do the WGs have to produce documents
ER> that meet the IESG's definition of quality?

I think you just highlighted a key problem:

The IESG can and must provide a quality control mechanism.  But it is
the community that must define the necessary quality.

And, yes, "the community" is a pretty darn fuzzy construct, which is why
we end up with each AD (and each IAB member) defining their own criteria
and applying them independently.

Still, we need to find reasonable, community based criteria -- beyond
just the criteria of the working group, so that it represents a broader
view -- with the IESG enforcing it, rather than inventing it.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 17:58:39 -0700

> Still, we need to find reasonable, community based criteria --
> beyond just the criteria of the working group, so that it
> represents a broader view -- with the IESG enforcing it, rather
> than inventing it.

does not scale.  this is still the mode where the authors, wgs,
chairs, ... wuss out.

quality management 101: the philosophy of building quality has to
be pushed to the edges so it encompasses the whole organization.

randy

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 18:30:58 -0700

> view -- with the IESG enforcing it, rather than inventing it.

I don't have a problem with this. The point I'm trying to make (and
tried to make in SF) was that it seems to me that a rather small part
of the variation in how long a document takes to clear the IESG is the
quality of the document (whatever that means). Accordingly, I don't
think that WGs have a particularly strong incentive to meet any
particular bar.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 18:36:17 -0700

Randy Bush <[email-masked]> writes:

> could have. 
In this case, the editors wanted something different. But 
you're wrong about what they could have done. Editors have been
removed from documents based on WG consensus before. 

> the wg chair(s) could have.  
The chairs are again subject to WG consensus, and occasionally
do get removed for this kind of thing.

> respected members of the
> security directorate could have.  ...
There was no security directorate at the time, but the security
directorage has no power per se. The only power they have is
to lobby the AD.

The only people in the IETF who can realistically block a 
document semi-indefinitely over the will of the WG are the
ADs. However much we might will it to be so, the IETF is 
not a democracy. At best, it's a republic, and not much
of one at that. 

> told me to ask about salary and benefits."

Hmm... I'm not sure I subscribe to this theory, but in any case...

I think that there are two things going on:

(1) The WGs have a genuinely different definition of quality from
    that of the IESG. As a consequence, many arguments about quality
    are really about whose standard will be enforced. The perennial
    arguments about security come to mind. In cases like this,
    no amount of pride in work will solve the problem since
    the issue is a difference of opinion. The IESG can of course
    force people to comply with their view but that only works
    if people are consistently rewarded for doing so (or
    punished for not doing so).

(2) Protocol quality and document quality are not necessarily
    the same thing. In many cases, people are interested in
    having a protocol they consider good (with the caveat above)
    but don't much care about the technical writing. Engineers
    skimp on tech doc all the time. We don't want that, but again
    we have to force people to do it because it doesn't come
    naturally to them.

Consider the following thought experiment: take all the documents
that come before the IESG in a year. Rank them in terms of quality.
Then plot quality against "time to approve". Ask what the correlation
is. If there's not a high correlation, why would you expect people
to produce high quality work?

-Ekr

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 20:32:16 -0700

>> represents a broader view -- with the IESG enforcing it, rather

1.
RB> does not scale.  this is still the mode where the authors, wgs,
RB> chairs, ... wuss out.

2.
RB> quality management 101: the philosophy of building quality has to
RB> be pushed to the edges so it encompasses the whole organization.

These two two statements appear to be mutually exclusive. The first says you
can't trust all the folks "out there" because they wuss out, while the
second says you have to have quality "at the edges". If the latter does
not mean all those folk "out there" then who are you referring to?

Most interestingly, the first statement fundamentally translates to:

    The entire body of folk who do the core work of the IETF have no
    understanding or integrity about quality.

That doesn't leave much of a base for creating quality, does it?

But perhaps what I originally said was not sufficiently grokked. So
let's try again:

    We need a framework for quality that is embraced by the *general* IETF
    community, so it can be enforced by the IESG, to help individual
    *portions* of the IETF community (ie, individual working groups)
    produce work consistent with that framework.

What we have now is:

    a) no broad base of shared sense about quality,

    b) invention of criteria by individual area directors,

    c) inconsistent application of the criteria, and

    d) application of the criteria far too late in the process.

What we need are specific suggestions, beyond recitation of generic
course teachings. Something we can actually apply, to improve the
production of quality specs in a timely manner.

ER> The only people in the IETF who can realistically block a
ER> document semi-indefinitely over the will of the WG are the
ER> ADs.

It is curious to see some people try to claim that this is not so, or at
least that it never happens, no matter how many of the IETF's worker
bees claim it IS so and it DOES happen.

That kind of disconnect is rather dangerous to an organization.

ER> (1) The WGs have a genuinely different definition of quality from
ER>     that of the IESG. As a consequence, many arguments about quality
ER>     are really about whose standard will be enforced.

Worst of all is that neither 'side' typically has their quality model
documented.  So we wind up arguing nits or intangibles, rather than
substantive issues.

ER> Consider the following thought experiment: take all the documents
ER> that come before the IESG in a year. Rank them in terms of quality.
ER> Then plot quality against "time to approve". Ask what the correlation
ER> is. If there's not a high correlation, why would you expect people
ER> to produce high quality work?

Interesting suggestion.

d/

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 21:38:49 -0700

> The point I'm trying to make (and tried to make in SF) was that
> it seems to me that a rather small part of the variation in how
> long a document takes to clear the IESG is the quality of the
> document (whatever that means).

and what substantiates that perception?

randy

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 21:56:47 -0700

>     understanding or integrity about quality.

no.  it says YOUR STATEMENT ABOUT THEM is incorrect.

randy

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 22:05:09 -0700

Randy,

RB> no.  it says YOUR STATEMENT ABOUT THEM is incorrect.

Thanks.  No doubt that cleared everything up.

d/

ps. exercise for the reader:

    1) which statement and about whom?
    2) extra credit: what about the statement was incorrect and why?

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 22:14:37 -0700

> and what substantiates that perception?
Well, there's people's personal experience, of course, but as we all
know, the plural of anecdote is not data.

As I noted at the time, the variance of time-to-approve is quite
large, even for documents that are approved by the IETF as-is. That's
not completely inconsistent with document quality, but if the
documents that are taking a long time to clear (>100 days) are really
that low quality, it's somewhat surprising that they aren't sent back
for revision.

Do I have any totally convincing evidence? No. It's just my intuition,
combined with the lack of strong evidence to the contrary.  However,
to the extent to which people believe that the quality effect is small
then there's no incentive to quality, regardless of whether it's in
fact true. I of course can't speak for what other people believe,
though I imagine that that would be relatively easy to discover without
too much complicated analyiss.

Really, 
-Ekr

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 22:10:58 -0700

>     1) which statement and about whom?

your quoted at the top of the message.  the part you deleted.

>     2) extra credit: what about the statement was incorrect and why?

the comments i made that you also deleted

randy

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 22:17:25 -0700

> Do I have any totally convincing evidence? No. It's just my

ack

> combined with the lack of strong evidence to the contrary.

uh, i usually avoid the no white crows logic.

maybe we should look at the time to process through iesg and the
reasons we can find.  i do not claim to know any answer, but of
course have silly perceptions based on my narrow little view of the
movie.  so it would be really good to have some actual data.  would
be nice if bill, harald, or someone with time could find a way to
get something real on which we could base a discussion.

> to the extent to which people believe that the quality effect is
> small then there's no incentive to quality, regardless of whether
> it's in fact true.

good point.  most of what i hear is similar to crocker's "the iesg
is imposing it's view of quality," not the iesg is delaying
documents for reasonsa we don't understand.

of course, the data-tracker stuff should help make all this a bit
more visible.

randy

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 22:23:06 -0700

Randy,

>>     1) which statement and about whom?

RB> your quoted at the top of the message.  the part you deleted.

>>     2) extra credit: what about the statement was incorrect and why?

RB> the comments i made that you also deleted

Including brief text segments, for context reference, is rather
different than "deleting" text. Given your own predilection for cryptic
reference, it's odd that you would describe it so pejoratively.

> RB> be pushed to the edges so it encompasses the whole organization.

1) Note that you did not address the contradiction between your two
statements,

2) You did not explain who is left, after you dismiss the worth of
authors, wgs and chairs, and

3) what is it, for example, about "community based criteria" that will
not scale.  And what is it you are suggesting as an alternative that a)
is practical, b) WILL scale, and c) is acceptable to the IETF community
(ie, the folks on whose behalf area directors are supposed to operate)?

d/

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 22:29:25 -0700

Randy,

RB> good point.  most of what i hear is similar to crocker's "the iesg
RB> is imposing it's view of quality," not the iesg is delaying
RB> documents for reasonsa we don't understand.

What is really ironic is that you did not see that I was pointedly being
polite, since the latter explanation you cite would mean that the IESG
is not even being clear about its reasons and hence could be operating
arbitrarily.

And, indeed, that is a common thread of concern that is expressed by all
those worker bees that you have dismissed as wussing out.

But that's ok. Since you believe area directors are not inventing their
quality requirements, please direct folks to the statements of
community-based quality that area directors are working from.

It really would make things quite a bit easier.

d/

---

From: Eric Rescorla <[email-masked]>
Date: 08 Jun 2003 22:37:17 -0700

> uh, i usually avoid the no white crows logic.
I don't see how we can avoid taking one position or the other,
because we have to decide what to do.

> get something real on which we could base a discussion.
Well, Dave, Marshall, and I have all taken a stab at processing
the data we do have, which at least includes the time in and time
out of the IESG, as well as the magnitude of the changes.
I don't know what their conclusions were, but my analysis
didn't exactly sell me that there was a large quality effect.

-Ekr

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 22:32:54 -0700

> 2) You did not explain who is left, after you dismiss the worth
> of authors, wgs and chairs, and

once again, i did not dismiss anyone's worth.  i said your proposal
did.

> 3) what is it, for example, about "community based criteria" that
> will not scale.

nothing.  asking the iesg to be  the only enforcer of them will not
scale.

> And what is it you are suggesting as an alternative that a) is
> practical, b) WILL scale, and c) is acceptable to the IETF
> community (ie, the folks on whose behalf area directors are
> supposed to operate)?

unlike some here, i don't think i have all the answers.  i am still
trying to understand the root problems, hence my participation in
the problem-statement list and my basing most of my comments on my
admittedly small and narrow personal experience and view not
theory.

but as i have previously suggested, i suspect we can not scale and
produce quality without saying "no" to some things, as tempting as
doing everything may be, and without building quality and review
into the whole system, from inception through the wgs, through last
calls and final review, whether that be iesg or whatever the world
becomes.

if we come to agreement that producing quality and scaling are two
core problems, then i think there is some organizational and
management experience from within the computer industry and outside
from which we can draw some ideas on quality management etc.  but i
don't think we've reached consensus that these are the issues yet.

randy

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 22:50:50 -0700

Randy,

>> of authors, wgs and chairs, and
RB> once again, i did not dismiss anyone's worth.  i said your proposal
RB> did.

That is what:

meant???  Wow.

A suggestion that we have community-based criteria dismisses authors wgs
and chairs?

Double wow.

>> 3) what is it, for example, about "community based criteria" that
RB> nothing.  asking the iesg to be  the only enforcer of them will not
RB> scale.

And another "oh, so *that* is what you meant". This is one of those
cases where "cryptic" translates into: It would have helped if you had
not deleted the clarity of your note, before sending it the first time.

The formal statement about the IESG quality function is that it is a
safety mechanism, to help when a working group has essentially running
rogue. It is supposed to be a backup mechanism, not a primary mechanism.

That role is very different from being "the only enforcer" and I don't
recall anyone -- certainly not me -- saying anything like "only" or
"the" or any other such label of exclusivity.

In any event, if you don't want the IESG to perform any quality
functions at all, then what do you propose? Automatically approve
anything a working group approves, rather does not raise a major ruckus
in the IETF Last Call?

RB> but as i have previously suggested, i suspect we can not scale and
RB> produce quality without saying "no" to some things,

and most folks agree with you.  so the concept is not a controversial
one.  as you note, it is the practise that is challenging.

RB> if we come to agreement that producing quality and scaling are two
RB> core problems, then i think there is some organizational and
RB> management experience from within the computer industry and outside
RB> from which we can draw some ideas on quality management etc.  but i
RB> don't think we've reached consensus that these are the issues yet.

really?

time to ask folks whether anyone needs convincing of these two points.

d/

---

From: Randy Bush <[email-masked]>
Date: Sun, 8 Jun 2003 23:07:31 -0700

> A suggestion that we have community-based criteria dismisses authors wgs

i am amazed that you have such a hard time understanding.  but you seem
to have caught it later on in the message at only the third repetition.

your paragraph you keep forgetting:

> than inventing it.           ^^^^^^^^^^^^^^^^^^^^^^^^^^

<whine>
as i am on an 18 hour visit home between a week of us tour and now
a week in tokyo, and i have a <bleep>ing 07:15 conf call, please
excuse that i don't have the free time on my hands to go around
this one point as many times as you seem to.  but do have fun,
dave, do have fun.

randy

---

From: Dave Crocker <[email-masked]>
Date: Sun, 8 Jun 2003 23:27:37 -0700

Randy,

>> A suggestion that we have community-based criteria dismisses authors wgs

RB> i am amazed that you have such a hard time understanding.

You shouldn't be.

>> than inventing it.           ^^^^^^^^^^^^^^^^^^^^^^^^^^

That's really excellent.  Ignore the opening sentence:

DC>The IESG can and must provide a quality control mechanism.

but jump on a later sentence as if there were no context.

Yes, it is difficult to imagine why anyone would have trouble
understanding your meaning.

RB> <whine>
...
RB>excuse that i don't have the free time on my hands to go around

Not too pressed for time to have a bit more constructive fun, it would
seem.

Thanks for your help in developing a constructive exchange. When you say
"i am still trying to understand the root problems," you demonstrate an
inventive approach to the exploration.

Anyone would feel strongly encouraged to participate.

d/

---

From: Pekka Savola <[email-masked]>
Date: Mon, 9 Jun 2003 15:06:50 +0300 (EEST)

> > and what substantiates that perception?
[...]
> that low quality, it's somewhat surprising that they aren't sent back

It might be interesting to note a few more fine-grained metrics for 
determining the document quality, e.g.:

 - the number of DISCUSS votes raised in the IESG
   * and how soon after raising the DISCUSSes a new document is published 
which addresses those issues (ie. the time it's not an IESG issue)
 - documents which were approved as-is, but an RFC editor note was added 
by the IESG (causing some delay)

FWIW, I have personal experience for authoring or commenting a few 
documents which have been through the IESG, a few notes:
 - one document has been pending a revised version from the author for 
over 6 months (the author knows the ball is at his feet)
 - one document for which comments were given took about a month for even 
the authors to reply
 - one document has been pending Expert Review by WG for about a year

So, my perception is that the most time-consuming thing after submission
to the IESG appears to be resolving issues raised by IESG; I hope nobody 
sees this delay as a problem of IESG :-).

To conclude, I guess adding *real-time* transparency ("where does this 
document stand?  where is the ball for the document ie. who's in charge 
at the moment? "do I have any documents I should take action on?") is 
crucial (I-D tracker helps but may not be enough). 

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Scott W Brim <[email-masked]>
Date: Mon, 9 Jun 2003 08:22:37 -0400

On Sun, Jun 08, 2003 06:36:17PM -0700, Eric Rescorla allegedly wrote:
> Consider the following thought experiment: take all the documents that
> come before the IESG in a year. Rank them in terms of quality.  Then
> plot quality against "time to approve". Ask what the correlation is.
> If there's not a high correlation, why would you expect people to
> produce high quality work?

Perhaps, but only if you consider the starting point.  Documents which
take the longest in getting through the IESG can have poor quality to
start with, and still aren't great when they get out but at least
they're tolerable.  What you could look for in your experiment is the
change in quality between when the document is sent to the IESG and when
it emerges, versus time.

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 09 Jun 2003 08:29:20 -0400

Hi Randy,

>don't think we've reached consensus that these are the issues yet.

The problem statement lists both of these concerns, although
it labels them differently:

         - Lack of effective engineering practices (the text
                 of which is all about WGs producing timely,
                 high quality, predictable output).
         - Management structure not matched to size and complexity
                 of the IETF.

Many people have helped us to refine and reword those sections,
but I don't remember anyone indicating a lack of belief that these
two problems exist...

Why don't you think we've reached consensus?  It seems quite clear
to me that we have consensus that both of these problems exist
and need to be fixed.  And, I agree that it might be valid to
draw on experience from the computer industry, academia, other
standards bodies, etc. for ideas on how to fix these problems.

Are you waiting for a particular event -- such as the WG chairs
declaring consensus?  Or document publication?

Margaret

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 09 Jun 2003 15:11:58 +0200

>  - the number of DISCUSS votes raised in the IESG

this could possibly serve as a metric of the number of *different* concerns 
raised by the IESG, but because of the IESG's common practice of saying "no 
further objections" when everyone agrees that the important problems 
they've seen have already have been flagged, it's not a measure of the 
percieved depth of the problems found.

> which addresses those issues (ie. the time it's not an IESG issue)

this metric could be very interesting, if we can measure it.

> by the IESG (causing some delay)

I don't think this adds delay, except for the cases where:
- the IESG takes time to write up the RFC Editor note
- the author or WG objects to the content of the note
The first happens a few times, but is usually (in my recent experience) on 
the order of a week. Have people seen the second happen much?

---

From: Pekka Savola <[email-masked]>
Date: Mon, 9 Jun 2003 17:13:42 +0300 (EEST)

> percieved depth of the problems found.

I agree; it's more important to measure if there is a discuss vote or not
(if not, I guess the doc approval should be a very quick process :-), but
at least in my perception, several very problematic documents have had at
least 4-6 DISCUSS'es from different people.. so I guess it could yield
*some* information, but with some standard disclaimers..  

I think you could say that the more DISCUSSes a document has, the more
potential it has to be a document requiring much more work.  However, this
does not say anything about documents with one or few DISCUSSes: those may
or may not be higher quality documents (depends on what kinds of comments
will surface from different IESG members etc.).

> the order of a week. Have people seen the second happen much?

I was thinking of the first.  But the issue is that I don't know how long
it takes to write these.  It's typically given as an action point, so I
think at least two weeks is a norm.  Tracking this could be useful.

---

From: Eric Rescorla <[email-masked]>
Date: 09 Jun 2003 07:36:29 -0700

Scott W Brim <[email-masked]> writes:

> Perhaps, but only if you consider the starting point. 
That's what I meant.

> Documents which
That depends on what you're trying to measure. For the purposes of
this discussion I'm not trying to measure IESG value-add but rather
the extent to which having a good document makes it clear faster
(since that's the incentive for the WG to do a better doc). For
that question, the starting document quality is what you want.

-Ekr

---

From: "James Kempf" <[email-masked]>
Date: Mon, 9 Jun 2003 08:47:24 -0700

Randy,

> and iesg, letting the iesg take the heat.

Putting on my WG chair hat, it is not so simple. WG chairs are
essentially middle management, and they have all the problems
associated with responsibility without authority that middle
management has. In my experience, a WG chair that tries to make a
technical argument about a document is viewed as being "just another
WG member". WG members with different opinions feel they can endlessly
quibble with the chair, go to the AD and complain that the chair is
being "uncooperative and blocking WG concensus", etc., if the chair
says that they won't release the document to the IESG. If the WG
member couches their opposition in terms of what the AD says or the
IESG may say, the resistence evaporates, because the IESG has the
ultimate authority to block the document. WG chairs only authority is
to hire and fire WG document editors, call concensus, and prune the
mailing list of people who are really disruptive (at the cost of a
long email exchange beforehand to warn the person). They of course
control the agenda at meetings, but they need to be careful to be
evenhanded here, or risk being accused of bias.

If you think that WG chairs should have the authority to block
documents, then it needs to be explicitly written into RFC 2418bis.

            jak

---

From: Randy Bush <[email-masked]>
Date: Mon, 9 Jun 2003 08:57:27 -0700

> documents which were approved as-is, but an RFC editor note was
> added by the IESG (causing some delay)

i am not sure and rfced note causes delay.

> adding *real-time* transparency ("where does this document stand?
> where is the ball for the document ie., who's in charge at the
> moment? "do I have any documents I should take action on?") is

hmmm.  when we started the data-tracker project 2+ years ago, it
was meant as automation of the iesg admin process.  so it is
starting to do a pretty good job of telling me what's on my various
plates.  

but i can guess it does not meet others' needs as well as it might.
this is probably not the appropriate list, but some feedback into
the d-t project so it better meets the needs of the wgs, authors,
etc. would be cool.

randy

---

From: "James Kempf" <[email-masked]>
Date: Mon, 9 Jun 2003 09:01:53 -0700

Dave,

>     produce work consistent with that framework.

I also think we need to explicitly call out and distinguish between:

a) engineering decisions that influence quality and lack thereof,

b) engineering decisions that don't really impact quality but do
determine what ultimately goes into the spec.

In my experience, many disagreements between the WG chairs and WGs,
and IESG and WGs occur because one side views the decision as in a)
and the other in b). In the end, the decisions made in b) don't really
make much difference to the technical quality of the standard, but are
important for other reasons: compatibility with existing base,
relations with outside standards organizations, etc. Or, they may be
completely unrelated to any reason except that someone in the WG
thought of it and fights any attempt to take their idea out of the
spec.

            jak

---

From: Dave Crocker <[email-masked]>
Date: Mon, 9 Jun 2003 09:29:38 -0700

James,

JK> I also think we need to explicitly call out and distinguish between:
...
JK> In my experience, many disagreements between the WG chairs and WGs,
JK> and IESG and WGs occur because one side views the decision as in a)
JK> and the other in b).

Yes, to both the distinction you raise and the conflict you cite.

d/

---

From: Bill Fenner <[email-masked]>
Date: Mon, 9 Jun 2003 11:34:14 -0700

>get something real on which we could base a discussion.

I just wrote a quick hack to analyze the data I have, and put the
results at http://rtg.ietf.org/~fenner/iesg/docstates.txt .  There
are two types of entries in this file - one for the first time
we see a document in my snapshots, e.g.

20020506        draft-ietf-ipngwg-rfc2292bis    entered system in Requested

and then once for each time it changes state:

20030312        draft-ietf-ipngwg-rfc2292bis    IESG Evaluation -> Approved-announcement to be sent after 11 days

There is some noise in the data, like the tildes swapping with dashes
in this example:

20030426        draft-ietf-ipngwg-icmp-name-lookups     IESG Evaluation ~~ Revised ID Needed -> IESG Evaluation  -- Revised ID Needed after 170 days
20030525        draft-ietf-ipngwg-icmp-name-lookups     IESG Evaluation  -- Revised ID Needed -> IESG Evaluation ~~ Revised ID Needed after 29 days

but I'll wait for some evidence that cleaning up the data is desired
or useful before putting more work into it.  (plenty of data, not
clear if there's any information...)

  Bill

---

From: Bill Fenner <[email-masked]>
Date: Mon, 9 Jun 2003 13:14:57 -0700

In the tradition of lies, damn lies, and statistics, I calculated
the mean, median, min and max time that documents spent in any
state and put it at

http://rtg.ietf.org/~fenner/iesg/docstates-stats.html

At a minimum, it points out how noisy the data is and how it's
probably necessary to put it through a few filters before it's
worth drawing conclusions from.

  Bill