Subject: Re: Doing the Right Things?

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 1 Jun 2003 07:58:35 -0500

I agree with Keith's points here - a very valuable contribution to
the discussion. I have one addition.

Spencer

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "Margaret Wasserman" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Saturday, May 31, 2003 10:38 PM
Subject: Re: Doing the Right Things?

[deleted down to]

> 4. We need to reorganize our document series.

This is actually a point where it helps to be a relative newbie.

The RFC series is just about opaque to newcomers to the IETF.
Ask Keith says, we adjusted as we went, but the most reliable way
I have to look at "what's out there" in a field I've not worked in
previously is to find a working group home page that "tags"
relevant RFCs. This is not reliable. "Reviewing the archives" to
find out "what's out there" for Internet Mail is not scalable, for
instance.

The hurdle I see is that there actually is an "editor role" that's missing
- not today's draft editors, and not today's "rfc editor", but an editor
that takes the output of successful Document Actions and merging
them into existing specifications, where this is applicable.

That's how other SDOs I've worked with can maintain a numbering
scheme - because STD 007 really would tell you how TCP works,
as opposed to "STD 007 (RFC 793) plus a bunch of other specifications
that are either proposed standards or experimental-track".

As Keith points out, none of us are getting any younger, and even if we
enforced "decay to Historical", the number of relevant WORDS that
describe the way TCP works are increasing (to use my favorite example),
and the number of relevant documents that describe the way TCP works
are increasing.

Just adding another layer of meat is a great way to make sandwiches and
a less-great way to maintain specifications long-term... I'm not saying we
need to fix this the way other SDOs do, but can others say whether it's
a problem for them?

Spencer

>
> (this is a problem that I don't think is touched on in the current draft)
>
> Lots of people complain that RFCs are too often used to give things the
> appearance of standardization even when they have not been approved for
that.
> That is probably true, but it's not what I mean here.
>
> The problem I see here is that we now have so many RFCs that it's too easy
for
> some important bit of information to get lost.  Sequential numbering was
fine
> when there were only a couple of thousand RFCs (and when you could ignore
> the first several hundred), and many of us had the important RFC numbers
> memorized, but now there are just too many (or maybe I'm getting too old,
> but I'm not the only one getting older..)  I see useful work getting
unnoticed
> because it's buried in some obscure RFC number.
>
> Some SDOs have a numbering scheme that reflects categories, and I think we
> might do well to consider such a scheme for IETF documents.  This probably
> means abandoning the linear RFC numbering scheme entirely, though it
doesn't
> mean that we have to stop calling them RFCs.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 01 Jun 2003 20:26:53 +0200

--On lørdag, mai 31, 2003 23:38:16 -0400 Keith Moore <[email-masked]> 
wrote:

> 3. We need to see if there are specific kinds of areas or activities that
> we need to engage in, for which our WG processes don't work well.
>
> WGs are the IETF version of Maslow's hammer.  Any time we have a problem
> we want to form a working group to look at it.  But working groups (and
> the assumptions about WG operation that go with them) are not always a
> good way to look at a problem.  Sometimes a different mode of
> conversation, or different procedures, or different management
> structures, are  appropriate.
>
> We need to understand the limitations of the WG process and determine
> whether there should be exceptions to that process for activities that
> are not chartered to develop technical protocol standards.

I have thought this too.

There are working groups that function in a number of different modes:

- SIMPLE expects to produce a document set, then shut down - the "classical 
working group" model.

- LDAPBIS is trying to pick up an old document set, and do the work 
required to get it to Draft; it's not trying to invent (much) new 
technology.

- IPv6 is chartered to complete a document set, but has in the past been 
regarded as "the group responsible for everything to do with the IPv6 
technology area"

- MBONED is "a forum for coordinating the deployment, engineering, and 
operation of multicast routing protocols and procedures in the global 
Internet......This is not meant to be a protocol development Working Group."

- IDR seems to be the permanent guardian of the BGP specification

- TSVWG is a forum for developing proposals that do not merit their own 
working groups.

Yet the rules for working groups, and the ideals we sometimes tout for how 
to manage working groups, are strongly biased towards the "classical" model.

We also have directorates. And we have many (probably hundreds) of mailing 
lists that have been IETF WGs, or do IETF things, but with no way anyone 
could find them from the IETF's "official" information.

Would we be better off if we developed a few terms different from "working 
group" that we could use to name classes of entity that do functions in the 
IETF, but do not behave like "classical" working groups?

                      Harald

---

From: Scott W Brim <[email-masked]>
Date: Mon, 2 Jun 2003 12:23:45 -0400

> >are not chartered to develop technical protocol standards.

I like this a lot as well.

> There are working groups that function in a number of different modes:
..
> IETF, but do not behave like "classical" working groups?

In theory "working group" is pretty generic, so maybe you have
"standards working groups", etc.  Or: have one or more special areas for
special *kinds* of WGs.  But first do what Keith said - describe
areas/activities for which WG-classic doesn't work well, and then
explore alternative ways of approaching them.

.swb

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 02 Jun 2003 13:30:03 -0400

>I like this a lot as well.

Me, too.

>explore alternative ways of approaching them.

I have an example...

We've been trying to figure out the *right* way to run a
non-standards-oriented function in the IETF -- specifically
internal education/training.

We want to allow community visibility and input into our
education priorities and programs, however there is no
intention for our internal education/training efforts to
produce RFCs.  This is also an ongoing function, not a
task-oriented function like producing a document or solving
a particular technical problem.

The two existing choices for group activities in the IETF
are:
         - Directorates
         - Working Groups

I suppose that we could have an internal education directorate,
but that wouldn't help to make our efforts more visible and/or
allow for wide community input.

So, maybe we need a WG?  It would be a weird WG, but a WG
seems preferable to continuing to conduct this effort in
a back-room effort...

Anyway, we'll have a BOF in Vienna, and one of the things
that we'll be discussing is how this effort should be
organized and run on an ongoing basis.

Opinions are welcome.

Margaret

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 2 Jun 2003 14:10:12 -0500

Apparently this week, I'll be agreeing with Margaret...

I'm not sure we're really set up to have official meetings that
AREN'T either WGs or BoFs at an IETF. We know how to
have official WG meetings and unofficial directorate meetings,
so if we're shooting for visibility and wide community input,
we either need a BoF or a structural change that allows for
"something else"...

Spencer

----- Original Message ----- 
From: "Margaret Wasserman" <[email-masked]>
Sent: Monday, June 02, 2003 12:30 PM
Subject: Re: New ways to do things (Re: Doing the Right Things?)\

Opinions are welcome.

Margaret

---

From: Dave Crocker <[email-masked]>
Date: Tue, 3 Jun 2003 15:27:25 -0700

Keith,

Excellent points to raise. Each warrants its own thread, so I'm
splitting my response(s).

KM> 1. We need to define the discipline of Internet Protocol Engineering.

KM> By that, I mean we need to enumerate a series of steps that should
KM> normally be followed, more or less in order, when developing or modifying a
KM> protocol for use on the Internet.

Hmmm.  We have "guidelines" for working group process.  Perhaps you are
suggesting "guidelines" for protocol development?  Not a small task, but
certainly a useful one.

KM> The reason I say this is that several groups have demonstrated the inability
KM> to define the problem they are working on,

Unfortunately, i think this problem is deeper than we might wish to
acknowledge.  I tend to be a charter fascist, on the theory that a
crystal clear charter will leave no doubt about the problem being
solved, and/or the benefit to be derived from the result and the
deliverables to be produced.

I view charters as real contracts, making clear what is included and
obligated and what is excluded and prohibited.

However with great regularity, remarkably fuzzy charters are getting
approved. since chartering involves lots of experienced people beyond
the working group, we can't simply assess the problem on the working
group folk.

I don't know how to improve this.  But, yes, we definitely need to keep
trying.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Dave Crocker <[email-masked]>
Date: Tue, 3 Jun 2003 15:40:04 -0700

Keith,

continuing with the separate points you made...

KM> 2. We need to change the methods by which working groups conduct discussions.

I believe you are correctly identifying an issue, but incorrectly
characterizing it. I believe we already have the necessary methods
available. However we do not reliably use them.

The IETF has had an extremely important strength in the flexibility of
its group process. When we have a small, focused, homogeneous groups
with clear deliverables that are clearly understood, it can operate in a
very streamlined way.

Hence, for example, the sub-thread about moderators needs to note that a
separate moderator is required only when they are required. That is,
some groups do better with FEWER people holding positions. Keeping
things simple is a Good Thing when things are, in fact, simple.

Many groups do quite well on their mailing lists and in their meetings.
Let's be careful not to "help" them to add unnecessary structure and
become less efficient.

I believe our repertoire of group management tools is quite rich. I
believe the problem is that we do not USE it properly. In particular, we
often do not have working group management invoke particular tools when
needed.

I would claim, therefore, that our tendency is to fail to notice when
things are not simple, and to fail to implement *and enforce* stronger
structure and management in those cases.

Frankly the reason I originally started working group chair training
roughly 10 years ago was not to teach new chairs about IETF formalities.
Rather, we had too many chairs who were so careful about being fair and
open, that they were not giving enough time to making forward progress.
A working group needs to balance these two -- often conflicting --
requirements. So, the real challenge of chairing a group that has a
difficult goal and/or a heterogeneous group, is shepherding the cats of
diversity.

And, yes, this includes mailing list management. It is quite a bit more
difficult to manage an unruly list than an unruly meeting, partly for
technical reasons and partly for reasons of general experience. But it
is possible and it is done. The key in both cases is to impose highly
active and structured management.

So I am going to claim that the real requirements you are raising are to
make sure that

1) working group management understands the group tools (procedures) it
has available, and

2) that the working groups are encouraged to support the use of those
procedures. ("encouraged" is a euphemism, in case that is not obvious.)

KM> b. It's been repeatedly observed for several years that face-to-face meeting
KM> time isn't used effectively

Oddly, i think the problem is worse than this. Over recent years, we
have not only repeatedly noted the problem, we have repeatedly noted the
solution and it sure looked to me like we had community rough consensus
on the solutions.

But still we fail to heed all this guidance.  I do not understand why.
My best guess is that too few chairs really understand these issues and
ADs do not begin to have the time to enforce things at this level. (And,
yes, some ADs could, but choose not to.)

KM> Powerpoint-style presentations (not that it really matters which
KM> tool is used) tend to lull people into passivity, ...that are just
KM> repeated by the speaker is to make a really ineffective use of
KM> high-bandwidth meeting time.

Sorry, but I view this as an excellent example of entirely missing the
point. The issue is relevance and conciseness of the content, not the
means by which it is presented.

If we start forcing meetings to have carefully constructed agendas that
are designed to use the limited time appropriately, and we *enforce*
that structure, then we will not need to be distracted by the religion
of presentation style.

   My own pet suggestion these days is that meeting agendas need to
   state their deliverables. What is the purpose of the meeting and how
   will the group have made progress by the end of the meeting?

(And, by the way, displaying that purportedly unnecessary text is
*extremely* helpful to non-native english speakers -- as long as the
text is not too cryptic or slang-ful. For that matter, the real-time
blogging helps such folk also.)

KM> I suggest that the WG identify what kinds of things need to take place in
KM> face-to-face meetings;

A casual thought is to beef up the relevant section of the wg guidelines
document.

Another is to get more people to read the damn thing.

KM> face-to-face meetings have to be narrowly focused on the WG; face-to-face
KM> meetings can also be good times to resolve differences with other WGs or other
KM> interests that aren't represented by WGs.

mumble.  I very much like the intent of your suggestion, but seriously
doubt that adding anything other than strict working group content to
working group meetings is a good idea.  WGs are already too easily
distracted.

d/

---

From: Dave Crocker <[email-masked]>
Date: Tue, 3 Jun 2003 15:46:21 -0700

Keith and Harald,

KM> 3. We need to see if there are specific kinds of areas or activities that we
KM> need to engage in, for which our WG processes don't work well.

1. We do have some other mechanisms. IAB retreats and individual
submissions are two examples. And though we often make humorous,
dismissive reference to them, I believe Bar BOFs are a real and very
productive part of our repertoire.

2. Absent specific suggestions, the basic claim that we ought to have
different formal categories of groups sounds entirely reasonable.
However practical alternatives are not obvious, to me at least. That is,
I do not understand what functional differences are needed, to produce
better results. (The labels are not the issue to me; but the structure
and operational style are.)

HTA> I think that for each solution path to be pursued, a core team with
HTA> competence and energy to drive the process is needed, no matter what - and
HTA> that once we have that, it almost doesn't matter how long the formal
HTA> process to start the thing is; they can go to work right away, and do
HTA> course corrections as part of the chartering process, rather than waiting
HTA> for the "formalities" to complete before starting work.

I heartily agree with this.  When a group is motivated enough to start
doing the real work, independent of IESG approval, then the latter is
usually assured.

The danger, of course, is that those "course corrections" might turn out
to be "starting over" and that's not such a good thing.  On the other
hand, my own experience is that a group that has a strong enough sense
of needing to delivery something, so that it does not wait for the IESG,
usually has a compelling goal and effort.

HTA> There are working groups that function in a number of different modes:
...
HTA> Would we be better off if we developed a few terms different from "working
HTA> group"

This is one of those things that sounds completely reasonable, except
for lacking the detail needed to be concrete.

Yes, different projects have different goals. Some are cleanup
exercises. Some are intended to invent whole new systems. Some are
intended to create nicely-modular capabilities. Some involve
well-understood problems and possible solutions. Some involve a great
deal of fuzziness.

But all of them are established to produce one or more documents,
usually specifying a protocol.

Absent concrete details about the operational differences that are
appropriate to these different writing exercises, my own belief
continues to be that any group that is sufficiently clear about the
problem it is solving, and the benefit of that solution, and further
that has a sense of urgency about providing it, will operate just fine
as a working group.

It is easy to create lots of distinguishing labels.  But what will be
the real benefit?

We need to be careful not to make distinctions that lack essential
difference.

MW> We've been trying to figure out the *right* way to run a
MW> non-standards-oriented function in the IETF -- specifically
MW> internal education/training.
...
MW> I suppose that we could have an internal education directorate,
MW> but that wouldn't help to make our efforts more visible and/or
MW> allow for wide community input.

MW> So, maybe we need a WG?  It would be a weird WG, but a WG
MW> seems preferable to continuing to conduct this effort in
MW> a back-room effort...

I believe the core construct that you are citing is one of an *on-going*
effort. Indeed this is something that we have never figured out
comfortably. The history of open-ended working groups is quite poor, so
this is a case in which we well might want to create a different
category. However, we need to be very clear about its nature and
obligations. What will it do that is different from a working group?

Hmmm.  Is it slightly similar to kind of IAB, in the sense of providing a
home for a collection of experts so that they can provide strategic
guidance about one or more topics, with the intent that other groups
then follow through on the work?

d/

---

From: Dave Crocker <[email-masked]>
Date: Tue, 3 Jun 2003 16:12:32 -0700

Keith,

And lastly...

KM> 4. We need to reorganize our document series.

Again, this is something we have lots of agreement about, at this level
of statement.  But then we get bogged down in the details.

So I suggest that people pursue the topic in terms of what stages of a
document are *useful* to the community.  That is, where is community
motivation for changes in a document or a document's status?  If we can
figure that out, then we will have stages the community actually uses.

Here is my own thought on useful stages:

1.  Technical discussion and convergence -- form community consensus
2.  Technical refinement -- lock down the details
3.  Proof of concept and debugging
4.  Stability -- community operational acceptance

The first is pure mayhem, and that's good. People explore, suggest,
debate. Eventually, they should start to form group preferences and
focus on getting the details worked into a specification that provides
detail which readers can understand and see as coherent and sufficient
for achieving a specific set of useful functions. Then they test the
thing, to see whether interoperable implementations can be developed.
Then they actually use the thing, eventually achieving some sort of
critical mass of deployment.

Now, the clever reader will observe that we already have exactly these
stages. (Personal I-D, WG ID -> PS, Draft, Full.) Yet the latter 2 are
virtually unused.

The community does not care about demonstrated ability to operate or
demonstrated deployment and use?  Or is it that they merely do not need
the IETF to tell them (anymore)?

As to whether the barrier to PS is too high, I strongly suspect that the
more serious problem is that we are not getting scope correct.  We are
tending to try to deliver specifications that cover too much, and
thereby any reasonable requirement for completeness, manageability,
security, etc., becomes onerous.

Yes, there well may be a problem with requiring perfection in PS.  The
language "no known defects" can be used as a bludgeon.  But I keep
thinking that that is better fixed by divide-and-conquor than lack of
detail.

We might also want to consider whether the effort to attain Draft or
Full are too high.  Historically, we simply took folks' word that
adequate interoperability had been achieved.  These days we often
required an auditor's accounting that every single feature was tested.

We might also want to consider enforcing time-outs.  If a document is PS
and has not reached Draft after awhile, move it to historic...

Oh.  We already have that rule:

   RFC 2026
   
   6.2  Advancing in the Standards Track
   ...
   When a standards-track specification has not reached the Internet
   Standard level but has remained at the same maturity level for
   twenty-four (24) months, and every twelve (12) months thereafter
   until the status is changed, the IESG shall review the viability of
   the standardization effort responsible for that specification and the
   usefulness of the technology. Following each such review, the IESG
   shall approve termination or continuation of the development effort,

So, once again, it appears the problem is that we are not using the
tools we already have.

Why is that?  Does it really mean that somehow, different rules will
work better?  Or does it mean that we need to organize our
administration of those rules differently?

KM> Some SDOs have a numbering scheme that reflects categories,

Sounds to me like you are describing what STD numbering was intended
for.

My guess is that we do not use it because we do not maintain it. Or
perhaps there is another reason?

KM> and I think we
KM> might do well to consider such a scheme for IETF documents.  This probably
KM> means abandoning the linear RFC numbering scheme entirely, though it doesn't
KM> mean that we have to stop calling them RFCs.

No.  The linear numbering scheme ensures unique reference.  We all like
unique references.  The problem is that we also like stable references.

The idea behind the STD series was to provide both stability of
reference and aggregation of related specifications.

SD> The hurdle I see is that there actually is an "editor role" that's missing
SD> - not today's draft editors, and not today's "rfc editor", but an editor
SD> that takes the output of successful Document Actions and merging
SD> them into existing specifications, where this is applicable.

Localized versions of aggregations are the meta-specifications that are
mostly a vehicle for citing components.  Frankly I like these a lot.

But they do not operate well over multi-working group efforts.  Again,
that's what STDs were intended for.

d/

---

From: Keith Moore <[email-masked]>
Date: Tue, 3 Jun 2003 19:11:16 -0400

> KM> normally be followed, more or less in order, when developing or
> KM> modifying a protocol for use on the Internet.
> 
> suggesting "guidelines" for protocol development?  

yes, this is approximately what I have in mind - at least as I understand
it at present.  I liked the term "roadmap" that someone else suggested also.

> KM> The reason I say this is that several groups have demonstrated the
> KM> inability to define the problem they are working on,
> 
...
> the working group, we can't simply assess the problem on the working

My view is that we often don't have enough understanding of the problem at
the time a WG is chartered to insist that the charter be very precise or
detailed.  And charter refinement is yet another thing that takes up IESG
resources.  Also, asking groups to write "requirements" documents (or as I
would prefer, "design goals" documents) hasn't worked well either as far as I
can tell - groups don't seem to understand the purpose behind this task or how
to do it.  

So we need to find a better way to refine WGs' problem definitions.

---

From: Bob Hinden <[email-masked]>
Date: Tue, 03 Jun 2003 17:03:23 -0700

Dave,

>stages. (Personal I-D, WG ID -> PS, Draft, Full.) Yet the latter 2 are

As you correctly point out the IETF (and as implemented by the IESG) is not 
using the standards process we have defined.  It has been changed where the 
initial barrier has been raised very high at Proposed Standard and hardy 
anything gets to Draft standard, let alone (Full) Standard.

I think that many of the problems folks are complaining about stem from 
this.  It causes the IESG to worry about letting a document get to PS that 
is not perfect.  This results in many detailed reviews and documents being 
blocked for what might seem minor reasons.  This in turn creates a work 
load on the IESG that is close to impossible.  The resulting delays cause 
frustration between authors, chairs, working groups, AD, etc.  I think that 
a lot of this stems from trying to make PS the biggest hurdle.

I think that if we implemented the standard process as written (and as 
intended) that many of these problems would go away.  Perfection would only 
have to be achieved at Standard (and close to perfection at Draft 
standard).  This would mean that problems with specifications could be 
found by people implementation the protocols and running into problems, and 
then bringing the issues back into the working groups.  Interoperability 
problems would be found.  This would spread the load of improving the 
quality of the specifications to a much larger community of people than 
just the IESG members.   Running code and interoperability would be the 
real test for quality.

Enforcing the time outs would create incentives to move the specifications 
through the standards process.  I also suspect that some (perhaps many) 
specifications never really get implemented, so as long as we enforced the 
time outs as specified, that these would just go away.

There is a lot of parallels with the multi-step IETF standards process as 
with building products.  First versions of products always have 
problem.  Most successful product companies learn that trying to make the 
first product perfect is counter productive because it creates long delays 
and the real test is getting the product to the costumers.  Then the 
company really learns how well the product works and meets the market need 
(i.e., requirements).  It through creating new versions of the products 
that problems are fixed and quality is improved.  Companies that delay the 
first release until the product is perfect usually go out of business.  I 
wonder if that is what is happening with the IETF.

Bob

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 03 Jun 2003 20:14:11 -0400

I think we are each making assumptions about the history that led to this, 
and it would not surprise me if we are making different assumptions.
Note that I do not have data to back my understanding / memory, so I may 
well be confused.

My understanding is that the higher bar to PS arose as a consequence of 
things being widely deployed at PS and things not advancing to Draft rather 
than the deployment and non-advancement being a consequence of the high bar.

This is important in the sense that if the lack of advancement came first, 
then simply lowering the bar will not help us get better standards, and in 
fact could result in our ending up with lower quality documents permanently.

Yours,
Joel M. Halpern

At 05:03 PM 6/3/2003 -0700, Bob Hinden wrote (in response to Dave Crocker):
>As you correctly point out the IETF (and as implemented by the IESG) is 
>not using the standards process we have defined.  It has been changed 
>where the initial barrier has been raised very high at Proposed Standard 
>and hardy anything gets to Draft standard, let alone (Full) Standard.
>
>frustration between authors, chairs, working groups, AD, etc.  I think 
>that a lot of this stems from trying to make PS the biggest hurdle.

---

From: Keith Moore <[email-masked]>
Date: Tue, 3 Jun 2003 21:20:52 -0400

O> My understanding is that the higher bar to PS arose as a consequence of 
> than the deployment and non-advancement being a consequence of the high bar.

that's my understanding also.
 
> fact could result in our ending up with lower quality documents permanently.

I'm in strong agreement with this.  What we need to do is find a way for
standards to at least the level that is currently required for PS more
quickly, not to lower the bar for PS.  We might even need to raise the bar
slightly.

---

From: Keith Moore <[email-masked]>
Date: Tue, 3 Jun 2003 21:59:12 -0400

> KM> 2. We need to change the methods by which working groups conduct
> KM> discussions.
> 

I suspect we are mostly in agreement about that.  Working groups already have
a great deal of latitude as to how they conduct discussions, and different
mechanisms are appropriate for different groups with different needs.   
I am not suggesting that we impose new structures on working groups that don't
need them.   I am suggesting that most working groups are not now operating
effectively, and that overall, the operation of most working groups need to
change.  

That doesn't necessarily require imposition of new structures.  However, there
are strong expectations in the community that mailing list discussions and
meetings will be conducted in certain ways.  We need to change these
expectations, especially where they are found to do harm.  I attempted to
list some examples of these expectations in my previous message.  

We may even need to change some of our cherished mechanisms.  Perhaps, for
instance, we should abandon the notion that we can obtain a meaningful
conseneus on a large mailing list whose active participants vary considerably
from one week to the next, and where it's difficult to distinguish silence
that indicates a lack of dissent from silence that indicates exhaustion. 

> procedures. ("encouraged" is a euphemism, in case that is not obvious.)

I don't disagree with either of the above, but I don't think they're quite
equivalent to what I was suggesting.  I don't think this is a change that
can be effected by working group management alone - I think it will require
buyin of the entire community.  Ordinary working group participants need
to understand that the way we participate in working groups has to change,
and we need for the community, not merely the WG leadership, to reinforce this
within itself.  We need to break our bad habits.

> KM> b. It's been repeatedly observed for several years that face-to-face
> KM> meeting time isn't used effectively
> 

My best guess is that in most cases neither chairs nor ADs felt like they had
sufficient actual political power (as opposed to what our procedures say they
have) to make the changes that were necessary, in the face of strong community
expectation that the old, familiar procedures be used.

> means by which it is presented.

The medium really does have an effect.  Yes, you can use PowerPoint to
present a topic in a concise and relevant manner, but the medium encourages
exactly the opposite - it encourages over-use of words and under-use of
pictures (because good pictures are hard to draw in PowerPoint); it encourages
a highly linear and one-way discussion rather than a multi-way exchange of
ideas.   If you had to present the same ideas using a white board or chalk
board, you'd have much more incentive to be concise and focused.

Which is not to say that I want to ban PowerPoint.  But as long as we're
using tools like this, we need to understand how to use the tools in the
best manner, and we need to respect their limitations.

> KM> face-to-face meetings have to be narrowly focused on the WG;
> KM> face-to-face meetings can also be good times to resolve differences with
> KM> other WGs or other interests that aren't represented by WGs.
> 

When planning a meeting, every WG chair needs to ask: "what kind of input or
discussion to we need to be having that we can't easily have over the net?"
If that's purely WG discussion, so be it.  If the WG needs external input,
then by all means the WG should make use of the opportunity to get those
people in the room.  And it would probably be a good idea for each WG chair to
discuss this with his/her AD before meeting planning gets too far along
(assuming, of course, that both the AD and the WG chair can find enough time
to actually plan the meeting - I know that when I was AD meeting planning
didn't get enough of my cycles.)

---

From: Keith Moore <[email-masked]>
Date: Tue, 3 Jun 2003 23:29:35 -0400

> of statement.  But then we get bogged down in the details.

Well, to the extent I was suggesting details, I intended them merely
as examples, not as concrete suggestions.  I don't pretend to have 
worked out enough details to make a credible proposal, and I think that
is the job of the follow-on group rather than the problem statement group.

> document are *useful* to the community. 

I'm thinking that we need to understand what stages of a document will be used
by implementors.  i.e. at what point of apparent stability, maturity, and
buy-in will implementors start prototyping?  At what point will they start
interop testing?  At what point will they start shipping product?  

Once we understand implementors' needs and expectations, we will be in a
better position to decide what the criteria for acceptance of those stages
should be.  Of course we also have to reconcile that understanding
with WG energies and expectations, and with the engineering roadmap.

> security, etc., becomes onerous.

I strongly disagree.  If anything, I'm convinced that we are not covering
enough bases - we are producing too many protocols that work at cross-purposes
with our other efforts.

> thinking that that is better fixed by divide-and-conquor than lack of

I have serious doubts about that.

> required an auditor's accounting that every single feature was tested.

I've never seen an "auditor's accounting" required; only a simple list.

When protocols have large feature sets it really does seem worth examining
which features are implementable and which ones are used.  I'm not sure how
much good it does to try to test the interoperability of each feature
individually, though - I suspect that this about as useful as doing
conformance testing.  So yes, I think our testing criteria are worth
looking at.

But I don't think it's terribly helpful to think in terms of tweaking
the existing PS, DS, and FS levels.  I think we need to start with a
clean white board and figure out what levels we really need, because
currently only one of these levels seems very useful, and there are
arguments that even PS has a poor impedance match with what implementors need
and expect.

> Sounds to me like you are describing what STD numbering was intended

Not really.  STD is just another kind of document serial number, except that
if you revise a document it can keep the old serial number.   But it doesn't
serve as any kind of classification scheme; it doesn't help you group related
documents together.
 
> perhaps there is another reason?

My guess is that STD is not used because STD numbers are only assigned at
FS.  Of course, that hardly ever happens, and the practice of citing
RFC numbers is too well-entrenched to be displaced by a practice of citing
STD numbers that are rarely assigned.

We can't solve this problem by merely creating a new numbering scheme
alongside RFC numbers.  We either need to abandon the "RFC" name entirely
or we need to start numbering them differently.

> The linear numbering scheme ensures unique reference.  We all like

There are other ways to get unique references.  e.g. Dates work well. 

But again, this isn't the place to try to work out the details - I cited these
merely as examples of things we need to look at.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 3 Jun 2003 23:33:03 -0400

> KM> 3. We need to see if there are specific kinds of areas or activities
> KM> that we need to engage in, for which our WG processes don't work well.
> 

agreed.

> better results. 

Nor do I, and I'd have to think about it for several hours before I could
even make a strawman proposal.   For now I merely wanted to suggest this as a
topic for study by the solution WG.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 04 Jun 2003 09:18:51 +0200

--On tirsdag, juni 03, 2003 15:46:21 -0700 Dave Crocker <[email-masked]> 
wrote:

> HTA> There are working groups that function in a number of different
> modes: ...
> HTA> Would we be better off if we developed a few terms different from
> "working HTA> group"
>

Dave,

many of my examples weren't....

<SOLUTIONSPACE>

As for concrete proposals in order to "try this one on for size", I suggest 
that we think about creating an object called a "maintenance team":

- Responsibility for a specific document or document set
  (usually one protocol)
- A chair
- A mailing list, with a public archive
- A web page (probably)
- Milestones strictly set by the IETF standards process:
  - 6 months after publication: "Evaluate suitability for grade change"
  - Every 12 months: "Evaluate whether protocol should be retired"
  (usually these would be filled by an one-liner note saying "no").
- No termination clause

For instance, we could create the "SMTP maintenance team":

- Chair: John Klensin :-)
- Responsibility: RFC 2821 (SMTP) and RFC 1652 (8BITMIME)
- Mailing list: [email-masked]
- Web page:
  "RFC 2821 is ready for Draft if someone has the energy to write the
  interoperability report. Known claimed-conformant implementations
  are: ............."
  "RFC 1652 is at Draft, but refers to MIME, which is still at Draft.
  Cannot be advanced at this time."
  "No RFC Editor Errata for either protocol."
  "The following unclear items are known:......"
  "The following resources exist:....."

The advantage would be that someone who does NOT know who still cares about 
the development of the SMTP protocol or where to go with questions about it 
could relatively easily find the (still-active!) mailing list and a contact 
person for it. (I believe the mailing list has moved around 3 times since 
it was an active IETF WG mailing list. Only its members know where to find 
it, I think.)

And since it's not chartered to produce anything in order to justify its 
existence, and indeed can be specifically barred from promoting new 
specifications, the "noise attraction" effect should be quite a bit smaller 
than the effect we've seen in WGs that "hang around forever".

Once the time came that a new document needs to be written, the maintenance 
team could either decide that a WG needs to be (re)started, or that the 
effort is small enough and uncontroversial enough to let it go forward as 
an individual effort.

</SOLUTIONSPACE>

>From my example list, this might eliminate the need to have LDAPBIS as a 
working group, and might also give IDR a viable path towards splitting off 
its pure maintenance activities, so that the "new developments" can be 
separated from the "maintenance of existing specifications".

I'll repost this separately to [email-masked], if people want to 
comment on the specifics of the idea - its value here is to focus the mind 
on the idea that not everything needs to be a working group.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 04 Jun 2003 11:16:13 +0200

--On tirsdag, juni 03, 2003 15:27:25 -0700 Dave Crocker <[email-masked]> 
wrote:

> inability KM> to define the problem they are working on,
>

Dave,

this is a problem, not only because we have different opinions on what WGs 
should do, but because we have different opinions on what "fuzzy" means.

Take, for instance, the recent LEMONADE charter - which I think is 
reasonably crisp and states what the WG is intended to do.

When it was first floated on the IETF list, it was blasted by at least a 
couple of people (including you, I think) as being fuzzy, unreadable and an 
improper tool for evaluating or constraining the work of the group.
(Yes, there were improvements. But I think the first version floated wasn't 
that bad either.... and the intent of the proposers didn't change one whit, 
as far as I know; it just became expressed more clearly.)

Or take the charter of this working group. I simply did not see a way to 
describe the work to be done by this working group more precisely, without 
unduly constraining the problem area it was "permitted" to explore - a 
constraint I felt at the time that it was quite important for the IESG 
*not* to apply.

What's a charter that you view as a "good model"?

                   Harald

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Jun 2003 14:58:53 +0200

Keith Moore wrote:
> quickly, not to lower the bar for PS.  

Fully agree.

> We might even need to raise the bar slightly.

Do you mean that we are frequently releasing PS documents that contain
significant defects? We certainly can't expect PS documents to be perfect.

Another problem I see is that the barrier to Standard is so high that
nobody is interested, and the barrier to DS is high enough that very
few people are interested. That being so, apart from some academic
ideals, I question the value of having 3 grades at all. Since we hardly
use the top two grades, why have them? 

   Brian

---

From: Ralph Droms <[email-masked]>
Date: Wed, 04 Jun 2003 09:22:45 -0400

My experience with DS and full Standard is that no one is interested
because of the lack of perceived difference between the requirements
between the standards levels and the lack of perceived added value
in progressing a spec to DS or full standard.  That is, I disagree
with Brian's statement about the requirements for DS and full standard
being too high - I think the problem is a lack of differentiation
among the three standards levels.

Here's how I see the standards track:
The required correctness for PS is high enough that a PS spec is
"good enough" for vendor implementation and deployment.  The
reward for moving to DS and full standard is not worth the
investment of effort.

The tension I see is the need for a specification that is
sufficiently baked and stable to be used for prototyping,
interoperability testing and proof-of-correctness for the
spec doc, but that does *not* result in deployment.  It's
the premature deployment that (a) casts goofs in the spec
in stone and (b) eliminates any motivation to move the
specification through the standards track.

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Wed, 4 Jun 2003 14:44:50 -0400

> > What we need to do is find a way for
> > standards to at least the level that is currently required for PS
> > more quickly, not to lower the bar for PS.  
> 
> Fully agree.
> 

of course not.  what I mean is that if we realize that PS in effect is
taken as "ready for deployment" (which seems to be how the industry
interprets it), then maybe we really do need to expect some level of
implementation and interop testing before we approve something as PS.

in general I think earlier implementation would be a good thing -
it would help groups be more honest about the difficulty of implementing
some of these things.

---

From: "C. M. Heard" <[email-masked]>
Date: Wed, 4 Jun 2003 12:45:08 -0700 (PDT)

>>>>> On Wed, 4 Jun 2003, Bob Hinden wrote:
BH As you correctly point out the IETF (and as implemented by the
BH IESG) is not using the standards process we have defined.  It
BH has been changed where the initial barrier has been raised very
BH high at Proposed Standard and hardy anything gets to Draft
BH standard, let alone (Full) Standard.

Agreed that this is a problem.  There seem to be several related
reasons for this.

>>>>> On Wed, 4 Jun 2003, Brian E Carpenter wrote:
BC Keith Moore wrote:
BC >[Joel M. Halpern wrote:]
BC > > My understanding is that the higher bar to PS arose as a
BC > > consequence of things being widely deployed at PS and things
BC > > not advancing to Draft rather than the deployment and
BC > > non-advancement being a consequence of the high bar.
BC >
BC > that's my understanding also.

In other words the high bar imposed on PS is due to the fact that
vendors, operators, and IETF folk fail to treat PS as the standards
process intended.  There are probably many reasons for that.  Here
are a couple that I can think of:

- Most SDOs don't seem to have anything resembling the three-tier
process.  Typically an approved standard is considered ready for
deployment.  It seems that PS has been treated on a par with other
SDO's approved standards.  Yes, there are well-known problems with
deploying standards not backed by implementation experience, but
that seems to be what's happening.

- In some cases the three-tier process is simply ill-suited to the
task.  MIB documents are one particularly problematic case, especially
those whose purpose is to manage technology developed by other SDOs.
The Ethernet MIB modules (currently RFCs 2665 and 2668) have been
recycling at PS for quite a while now, because of new objects being
added to manage new types of Ethernet media.  The one document that
did reach full standard (RFC 1643) is obsolete and the WG has asked
the IESG to formally retire it by moving it to Historic.  (Part of
the problem here is that it's not always possible or reasonable to
split new MIB features off into separate documents that can be
advanced separately.  But let's not get sidetracked on the details.)

There is probably little that the IETF can do to prevent vendors
from deploying stuff that is at PS, short of calling it something
other than a standard.  The IEEE has something called a trial-use
document (http://standards.ieee.org/guides/opman/index.html) that
seems to roughly correspond to the IETF's notion of a PS.  I don't
know how successful that idea has been or how often it is applied,
but it does have a maximum lifetime of two years after which it
must either be affirmed as a full standard, revised, or withdrawn.

BC> > > This is important in the sense that if the lack of
BC> > > advancement came first, then simply lowering the bar will
BC> > > not help us get better standards, and in fact could result
BC> > > in our ending up with lower quality documents permanently.
BC> >
BC> > I'm in strong agreement with this.  What we need to do is find
BC> > a way for standards to at least the level that is currently
BC> > required for PS more quickly, not to lower the bar for PS.
BC> 
BC> Fully agree.

If actual deployment is going to be expected at PS, then it is
hard to argue for lowering the bar of "no known defects".

BC> > We might even need to raise the bar slightly.
BC> 
BC> Do you mean that we are frequently releasing PS documents that
BC> contain significant defects? We certainly can't expect PS
BC> documents to be perfect.

No, not without implementation experience.

BH> I think that if we implemented the standard process as written
BH> (and as intended) that many of these problems would go away.
BH> Perfection would only have to be achieved at Standard (and close
BH> to perfection at Draft standard).

It's not clear to me whether it is really possible for the IETF to
make PS into what RFC 2026 says it's supposed to be.

BH> Enforcing the time outs would create incentives to move the
BH> specifications through the standards process.  I also suspect
BH> that some (perhaps many)  specifications never really get
BH> implemented, so as long as we enforced the time outs as
BH> specified, that these would just go away.

That would help, and it would address a problem that we often fail
to maintain our specs.  However, it would not address something
like the Ethernet MIBs ... the technology moves fast enough that
the spec is likely to recycle at PS forever (even though large
portions of it are stable).

>>>>> On Wed, 4 Jun 2003, Ralph Droms wrote:
RD> Here's how I see the standards track: The required correctness
RD> for PS is high enough that a PS spec is "good enough" for vendor
RD> implementation and deployment.  The reward for moving to DS and
RD> full standard is not worth the investment of effort.
RD> 
RD> The tension I see is the need for a specification that is
RD> sufficiently baked and stable to be used for prototyping,
RD> interoperability testing and proof-of-correctness for the spec
RD> doc, but that does *not* result in deployment.  It's the
RD> premature deployment that (a) casts goofs in the spec in stone
RD> and (b) eliminates any motivation to move the specification
RD> through the standards track.

It seems to me that Experimental documents would fill that bill.

BC> I question the value of having 3 grades at all. Since we
BC> hardly use the top two grades, why have them?

>>>>> On Wed, 4 Jun 2003, Keith Moore wrote:
KM> of course not.  what I mean is that if we realize that PS in
KM> effect is taken as "ready for deployment" (which seems to be how
KM> the industry interprets it), then maybe we really do need to
KM> expect some level of implementation and interop testing before
KM> we approve something as PS.

This suggests to me a two-step process

- specs start out in life as Experimental, recycling there
are necessary until

- there are no known problems and at least two interoperable
implementations, at which point they would be eligible for
elevation to Standard (and could stay there on subsequent
revisions, if the changes were not "too drastic")

but this is intruding on solution space, isn't it?

//cmh

---

From: Pete Resnick <[email-masked]>
Date: Wed, 4 Jun 2003 15:09:46 -0500

On 6/4/03 at 2:44 PM -0400, Keith Moore wrote:

>what I mean is that if we realize that PS in effect is taken as 
>"ready for deployment" (which seems to be how the industry 

So, I read this as saying that industry is interpreting PS as what we 
meant DS to mean. That is, industry thinks:

   A [Proposed] Standard [is] well-understood and known to be quite
   stable, both in its semantics and as a basis for developing an
   implementation. A [Proposed] Standard may still require additional or
   more widespread field experience, since it is possible for
   implementations based on [Proposed] Standard specifications to
   demonstrate unforeseen behavior when subjected to large-scale use in
   production environments.

   A [Proposed] Standard is normally considered to be a final
   specification, and changes are likely to be made only to solve
   specific problems encountered. In most circumstances, it is
   reasonable for vendors to deploy implementations of [Proposed]
   Standards into a disruption sensitive environment.

[2026, section 4.1.2 on Draft Standard, with "industry" edits.]

I don't exactly understand why this should mean that we should (as we 
seem to be on our way to doing unofficially already) raise the bar of 
PS to be identical to the bar of DS unless we really think that there 
is no worth in having Proposed Standard as it was originally intended 
(a vetted, though immature, specification). We could go the route 
(similar to what CMH suggested) of going Experimental->PS->S, or even 
Experimental->DS->S, but I assume that the only thing that this will 
result in is industry thinking that Experimental is ready for prime 
time. (They already assume that "Informational" is an "IETF-Approved 
implementable standard".)

Wouldn't it make more sense to:

- Leave our written processes exactly the way they are;
- *Lower* the bar on PS back to what it was intended to be, get PS's 
out early, and do real interop testing early in the process;
- Start putting a boilerplate on PS RFC's which says "OK, now we 
really mean it; this isn't ready for prime time!"
- Ignore all screams of horror when we significantly change a 
specification that is at PS, publish the changed spec as a new PS and 
change the old PS to Historic, and say "I told you so".

I'm not being facetious. Or at least, I'm not being totally 
facetious. If the problem we are addressing is really, "Industry 
expects more out of PS than we intended", the problem can be 
addressed by either changing our interpretation of PS to meet 
industry expectations, or by changing industry expectations. The 
former seems to be what the IESG has been doing for some time now, 
and the result has been that some industry folks have taken to 
considering Internet Drafts stable enough to implement. This seems 
destined for an arms race. I'm more inclined to attempt the latter if 
we can.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 04 Jun 2003 16:30:06 -0400

[Somewhere on the border between problems and solutions...]

While I like the goal Pete describes, I am afraid that the reality does not 
match.

The last hyphen line suggests ignoring the screams of horror when we change 
PS.  I have currently observed working groups that have been resist to 
(sometimes unable to) changing working group internet drafts due to 
complaints about deployment.  I therefore tend to be doubtful of our 
ability in the current structure to ignore those screams.

While I am not happy with declaring most of our work experimental, the 
suggestion on this list would at least make it clear that things can be 
changed.

Yours,
Joel M. Halpern

At 03:09 PM 6/4/2003 -0500, Pete Resnick wrote:
>- *Lower* the bar on PS back to what it was intended to be, get PS's out 
>early, and do real interop testing early in the process;
>- Start putting a boilerplate on PS RFC's which says "OK, now we really 
>mean it; this isn't ready for prime time!"
>I'm not being facetious. Or at least, I'm not being totally facetious. If 
>the problem we are addressing is really, "Industry expects more out of PS 
>than we intended", the problem can be addressed by either changing our 
>interpretation of PS to meet industry expectations, or by changing 
>industry expectations.

---

From: Keith Moore <[email-masked]>
Date: Wed, 4 Jun 2003 16:33:58 -0400

> change the old PS to Historic, and say "I told you so".

I don't think so.  

Part of the reason I don't think so is that I don't think industry pays
much attention to what we claim about a document, so even an explicit
"not ready for prime time" disclaimer wouldn't be terribly useful.

But basically I think our current maturity levels (regardless of what we
call them or claim about them) are not a good match either with industry
needs or with working group interests/energies/motivations. 

I assume that a revised set of maturity levels would have some stage
more-or-less equivalent to our current PS stage, and that it would
not have stages equivalent to DS or FS - but would replace these with
applicability statements and/or implementation notes.  And I expect that
the stage that was closest to the current PS stage would actually be a
bit higher bar than PS now, and that there would be some earlier stages
as prequesites to that stage (which to discourage deployment would
probably not be published as RFCs)

But of course it doesn't have to work out that way.  For now I'm merely
suggesting that we need to re-examine our document maturity stages to
see how well they serve our needs and the needs of our customers, and if
they don't, to consider changing them.

---

From: Susan Harris <[email-masked]>
Date: Thu, 5 Jun 2003 12:09:46 -0400 (EDT)

> that we think about creating an object called a "maintenance team":

I think this is a *great* idea Harald.