Subject: OPEN ISSUE: Standards Track

From: Margaret Wasserman <[email-masked]>
Date: Thu, 15 May 2003 11:41:27 -0400

The process document current says:

>There is also a more fundamental issue with the IETF's engineering
>practices.  Although our current standards track contains three
>levels of maturity (Proposed Standard, Draft Standard and Full
>Standard), we do not have sufficient differentiation regarding the
>quality and completeness of documents required at each stage.  The
>bar is set very high for publication at Proposed Standard, and very
>few documents advance beyond this stage. [OPEN ISSUE: Do we have
>IETF consensus that this is a problem?]

I believe that this is a real issue, and that we need to make
some changes to our standards-track document processes to
address this.

In particular, I think that we have inadvertently reached a
point where our proposed standards are treated as standards
by most of the industry.  I think that this was caused, in
part, by the high level of scrutiny that we place on documents
before we allowing them to reach this level.  This also leads
to a lack of motivation to move documents to draft standard,
where there interoperability will be demonstrated.

In general, I think that this damages the quality and
integrity of the IETF standards-track documents, and we
should do something to fix it.

Margaret

---

From: "James Kempf" <[email-masked]>
Date: Thu, 15 May 2003 10:02:06 -0700

Margaret,

I've felt for some time that there is a need for a change in this
area, but this analysis leaves out the issue of Working Group drafts,
which I think is a critical component. Right now, they have grey area
status. I've seen drafts move from individual contribution to Working
Group status only because nobody on the mailing list spoke up when the
Working Group chair put out the question about whether a draft should
move to Working Group status or not. Many vendors start implementing
when something becomes a Working Group draft, since realistically,
they view that move as entry into the standardization process,
regardless of what IETF's process RFCs say about it.

Arguments have been made on this list that we should not touch the
current Proposed/Draft distinction, and I agree that the distinction
is useful even if Proposed is treated as "standard" by vendors for all
practical purposes (Full Standard, however, is largely useless except
as a historical distinction). However, I think some attention is
needed to how a draft becomes a Working Group draft. Perhaps that move
is when a preliminary review is made on the design, so that movement
to Working Group draft status does not happen for reasons that have
nothing to do with the technical aspects of the design.

            jak

---

From: Jari Arkko <[email-masked]>
Date: Thu, 15 May 2003 22:50:49 +0300

>> IETF consensus that this is a problem?]

It is a problem.

--Jari

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 15 May 2003 22:49:19 +0200

mumble....

there's another issue here, but I don't know if it is possible to untangle 
it......

*what sort of bugs, if detected, would make a protocol unsuitable for 
Proposed Standard?*

If there are 2 reasonable interpretations of a statement, which could lead 
to 2 noninteroperable interpretations? (Happened with BGP, I believe..... 
common practice now is only one, and there is a new spec being written that 
clarifies the point)

If there is part of the overall function that is not specified, so that you 
can implement it, but it can do nothing useful without further 
specification? (my favourite example is TIP, if anyone remembers that)

If there are issues that you know will arise when people try it, for 
instance "what is an alphabetic character"? (one of the recent specs out of 
PKIX)

It's fairly obvious that we invented the 3-step process so that we could 
fix bugs. And that implies that we think there are bugs that will only be 
flushed out after people try it out in practice.

But - does that mean that we should let known bugs pass into Proposed?
Or will this be forever a judgment call?

.....

A related issue is whether or not the restrictions on going to Draft from 
Proposed are really appropriate; currently, we say that *only* deletion of 
features is appropriate, and that any change or extension, no matter how 
worthwhile the purpose or how sure we are that it makes no problems, 
requires another round through Proposed. Is this really best for the 
Internet?

                     Harald

---

From: Margaret Wasserman <[email-masked]>
Date: Fri, 16 May 2003 05:26:45 -0400

>*what sort of bugs, if detected, would make a protocol unsuitable for 

There are at least two ways to look at the concept of a staged release
process:

         (1) You define the criteria for acceptance at each phase.
                 These criteria are easier to meet at lower phases
                 (i.e. you can have more bugs of certain types, you
                 need implementations for a certain phase, etc.).

         (2) You expend different levels of effort to determine the
                 quality at each phase (i.e. you subject the
                 document to successively wider review and comment
                 at each phase).

We already do (1) to some extent (requiring implementations for DS),
but we don't do (2) at all...  It might be possible, for example for
area management to publish a document to the first phase in the
standards process, without full IESG review. This would also help
with the scaling problems at the top.

I also think that there are two other problems facing us:

         (1) There is no real differentiation in our market between
                 Proposed Standards and Draft Standards.  Our
                 "customers" don't know what documents are at what
                 level, and they don't care.  In fact, most of them
                 can't remember which comes first, PS or DS...  And
                 once something has an RFC number (even experimental
                 or info), it is treated like a standard.  So,
                 there is really no reason to do the work necessary
                 to move to DS, and certainly not to FS.

         (2) The term "Proposed Standard" currently indicates to
                 our market that the document is a standard.  If
                 we create a lower-level category, we should give
                 it a different name, and put something _in the
                 document_ that indicates its provisional status.
                 (No one actually looks at the RFC index to
                 determine the status of a document before
                 specifying it in an RFQ or RFP).

>Or will this be forever a judgment call?

I think it will forever be a judgement call, just as it is in software
and hardware development.

Like all judgement calls, there is a certain amount of effort spent in
gathering the information to make the call, and then the call is made
based on available information.  The best way to create a tiered process
is not to remove the judgement call...  Instead, you could move it to
lower levels of the organization (if we had any) and/or lessen the time and
resources spent gathering information before the judgement call is made.

>requires another round through Proposed. Is this really best for the Internet?

I think that this process is damaging, because it tends to lead to all
of the implementation-related experience being reflected in the PS
version.  This, in turn, offers even less incentive to the group to
produce a DS version.

At this point, there are _many_ WG and non-WG I-Ds in the IETF that
have already been widely implemented and fairly widely deployed.  If
it weren't for the requirement that documents be published at PS for
six months and recycled if there are any changes other than deletions,
many of these documents could go straight to DS...

Sometimes, I feel like a hypocrite about this topic.  On the one
hand, I'm fighting to keep the "integrity" of the standards-track
document set, according to our current rules.  And, on the other
hand, I think that our current rules are more than a bit draconian.

For example, I think that it should be possible for several different
proposals that solve the same problem to be published in an enduring
form by the IETF, and for a subsequent process to determine which
one(s) will advance on the standards-track.  This would solve some
of the problems that we have in the v6ops WG, for example.  But, we
have no tools to do this in a way that makes the status of these
documents visible to our "customers".

As soon as a protocol has an RFC number, our "customers" treat it
as a standard.

Meanwhile, we have individual submission track where people can get
RFC numbers without jumping through any of the IETF hoops (unless
they happen to overlap with a WG).  What is wrong with this picture?

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Fri, 16 May 2003 07:32:10 -0400

In reading over my own response I had a further thought...

>resources spent gathering information before the judgement call is made.

When documents are reviewed, it would be useful if we could
get two types of (clearly delineated) feedback:

         (1) Blocking issues that would prevent the publication
                 of the document at PS.  In the opinion of the
                 reviewer (currently the IESG), it is worth
                 delaying publication of the document to fix
                 this problem.
         (2) Non-blocking issues that should be fixed in the
                 next revision (and certainly before the
                 document goes to the next stage).

Right now, any issue raised by the IESG (through a DISCUSS) is
treated as a blocking issue.

Margaret

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 16 May 2003 15:22:31 +0200

Firstly, yes it's a problem.

Secondly, this is a case where I think a simple first step may
help quite a bit: simply merge Draft Standard and Standard
into a single class, called Standard,  but with the criteria
now used for Draft Standard. 

Arguments: remove a process step that we basically never use,
and make the step up from Proposed Standard worth the trouble.

On James' point about Internet Drafts, maybe we could use a
little clarification in the WG procedures, but the main point
is to require a WG consensus before declaring a draft to be
a WG draft. If that hasn't been happening, it's more of a WG
Chair training issue than anything else.

   Brian

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM 
On assignment at the IBM Zurich Laboratory, Switzerland

---

From: Margaret Wasserman <[email-masked]>
Date: Fri, 16 May 2003 09:30:29 -0400

Hi Brian,

>Chair training issue than anything else.

I've included this in recent WG chair training sessions
(since I started doing them in Atlanta).  I've also
included information about the WG chairs' responsibility
to choose and manage document editors.

It takes a while though, for this type of change to
propagate throughout the organization, and there are
still many WGs where there are no acceptance criteria
applied before documents are given WG draft status.

Margaret

---

From: Ralph Droms <[email-masked]>
Date: Fri, 16 May 2003 09:31:43 -0400

I agree that this issue is a problem.

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 09:47:33 -0400

On Thu, 15 May 2003 11:41:27 -0400

> some changes to our standards-track document processes to

I think there are multiple issues here:
1. insufficient differentiation of quality between the stages
2. we don't consider Proposed "done" (officially it's not even
   deemed ready for wide deployment), yet the industry does
   consider proposed "done" unless serious bugs are found.

> before we allowing them to reach this level. 

I think this is caused by multiple factors:

- Proposed is the first stable publication of a standards-track document,
  in the sense that the RFC won't expire and it won't be changed.
  (Maybe implementors just want a stable, referencable spec before 
  deployment, and they don't care so much about IETF's blessing.)

- The word "standard" appears in the document status.

- By the time a document is published as Proposed, the working group
  is typically burned out, and there's insufficient energy to revise
  the document further.  Getting a document moved to Draft Standard
  is extremely difficult not because the changes are extensive
  (usually they're fairly trivial) but because it requires a lot of
  attention to detail, and coordination between lots of people for 
  interoperability reports, for what is seen as minimal gain.
  (and Draft Standard actually sounds less mature than Proposed Standard)

If I'm reading this right, then this means we have a conflict:  On one hand
we'd like to have a lower bar for initial publication than what is currently
deemed Proposed Standard, so that we provide some incentive, some indication
of externally-recognizable progress, and can get wider review of a document
(perhaps even implementation) before we get to the level that the industry
recognizes as ready for shipping in product.  Oh the other hand if what the
industry is really waiting for is a stable, referencable spec, then publishing
RFCs with less scrutiny than is currently required for Proposed means that
quality of deployed protocols could decrease.  And if the process of moving
from the earlier stage to one which has a similar level of scrutiny to what is
currently Proposed ends up being as difficult as moving a Proposed document to
Draft, we may find that in the future we rarely get as far as (what used to be
called) Proposed.

---

From: "James Kempf" <[email-masked]>
Date: Fri, 16 May 2003 08:05:26 -0700

> documents visible to our "customers".

My understanding is that that is one purpose of the Experimental
designation.

        jak

---

From: Eric Rosen <[email-masked]>
Date: Fri, 16 May 2003 11:12:51 -0400

Brian> Secondly, this is a case where I think a simple first step may
Brian> help quite a bit: simply merge Draft Standard and Standard
Brian> into a single class, called Standard,  but with the criteria
Brian> now used for Draft Standard. 

Brian> Arguments: remove a process step that we basically never use,
Brian> and make the step up from Proposed Standard worth the trouble.

Could you explain how this makes the step up from PS worth the trouble?

---

From: "James Kempf" <[email-masked]>
Date: Fri, 16 May 2003 08:26:08 -0700

> and make the step up from Proposed Standard worth the trouble.

I agree that merging Draft Standard and Standard would simplify the
process. The distinction between the new Standard and Proposed
Standard would be the need for two interoperable implementations and
give the opportunity to debug at Proposed.

> Chair training issue than anything else.

The point I was trying to make, which perhaps wasn't at all clear, was
that this can be viewed in the same category as "earlier feedback is
more helpful than later." It is perfectly possible for a draft to have
WG concensus to become a WG draft and for it to contain fundamental
architectural flaws that don't get worked out in subsequent
discussion, until the draft ends up at the IESG and is rejected. By
recognizing the step from individual contribution to WG draft more
formally, and requiring some formal level of review rather than just
that some group of people in the WG think it is OK and say so might
help to catch those cases. The level of review should not be
burdensome or as extensive as required for the draft to go to RFC, and
could be entirely at the discretion of the shepherding AD and WG
chair. Right now, the step from individual contribution to WG draft is
left to the WG chair's discretion.

            jak

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 16 May 2003 10:43:44 -0500

Err, but you'll notice that we are discussing a "working group
draft" that was initially published as a working group draft
(not accepted as a working group draft from an existing
submission). The draft is a process draft, not a protocol draft, 
but we sure do this a lot if we're doing it wrong!

Having said this - I like Brian's suggestion. I think we would 
STILL have a "three-stage standards track", but it would be:

- working group draft for chartered specification,
- Proposed Standard,
- Standard

and it accomplishes what we had discussed about being able to
publish "Proposed Standard"/"first rung" specifications in a more
timely fashion without devaluing the RFC logo, and giving us 
a clearer conscience when we CHANGE something because
of implementation/deployment experience.

Spencer

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 11:49:24 -0400

I'm in general agreement that it's necessary to consider the role of 
internet-drafts (including working group drafts) when trying to
understand the problems with our current maturity levels.  And I'm also
in agreement that some working groups too easily adopt drafts as work
items - though my concern here is mostly about WG resources being spread
too thin.

I'm not sure it's appropriate to constrain WGs to only adopt drafts that
they believe will end up being products of the WGs.

And while it's fine for people to start implementing drafts (especially
if we'll benefit from the input of those implementing them)  it may be
that we need to find a way to discourage people from assuming that
they'll end up as standards.

> nothing to do with the technical aspects of the design.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 11:57:22 -0400

I'd like to suggest that rather than thinking in terms of lowering the
bar for Proposed, that we instead think in terms of creating a more
explicit preliminary step before getting to Proposed, treating what
is now called Proposed as more mature than we do now, and probably
eliminating Full Standard.  Because what seems to have happened is that
both the industry and IESG are now treating Proposed as "ready for
shipping product".  I don't think it's realistic to expect IESG (or
IETF) to lower the bar for Proposed unless the industry also lowers its
expectations, and I don't know how to make the latter happen.

I assume that we'd probably want to rename the current maturity levels
and change the status of existing documents to match their positions in
the new scheme.

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 12:57:14 -0400

I'd like to suggest that if we're going to change our document maturity
levels and/or their criteria, that we first need to understand what set
of maturity levels would be a good impedance match with what "the
industry" (including non-commercial implementors) needs, while still
serving the needs of the broader Internet community.   

Then we need to figure out what are appropriate criteria to impose at
each stage, given those needs, keeping in mind the level of energy we
can reasonably expect to bring to each transition between stages.  

Then we need to figure out how to label/market each level of maturity. 
This might include deciding what is or is not labeled as "RFC".

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 15:40:07 -0400

>                  document goes to the next stage).

as long as implementors feel free to widely deploy PS protocols, the
two are nearly indistinguishable.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 17 May 2003 00:29:09 +0200

> treated as a blocking issue.

The IESG tries to distinguish between blocking issues (DISCUSS comments) 
and non-blocking issues ("if the document is revised for other reasons, you 
might want to look at this too").
This will be made even more explicit in the upcoming more-automated process 
for IESG opininon-gathering.

But - for instance quite a few of the "I-D Nits" are regarded as blocking 
issues (like "too many authors" or "formatting of code samples is broken" 
or "ABNF does not pass syntax check"). So a DISCUSS doesn't necessarily 
mean that it's an important architectural issue - just that it's something 
that, in the IESG's opinion, has to be fixed before publication.

                      Harald

---

From: John C Klensin <[email-masked]>
Date: Sat, 17 May 2003 08:16:44 -0400

--On Friday, 16 May, 2003 05:26 -0400 Margaret Wasserman 
<[email-masked]> wrote:

> For example, I think that it should be possible for several
> different proposals that solve the same problem to be
> published in an enduring form by the IETF, and for a
> subsequent process to determine which one(s) will advance on
> the standards-track.  This would solve some of the problems
> that we have in the v6ops WG, for example.  But, we have no
> tools to do this in a way that makes the status of these

Sure we do.   Let's separate two issues.

The first is "our customers" (or, more to the point, marketing 
organizations who will do anything they can get away with to 
promote products -- I'd decided over the years that I'm even 
more upset by equating "posted as an individual I-D" with "under 
IETF consideration and review" than I am about "RFC equals 
standard") distorting the status/maturity level of documents. 
That is a problem, of course, but not one that we can easily 
solve.  In particular, I'm not at all convinced that renaming 
this will help -- I'm convinced that most of these distortions 
are deliberate and not honest errors and misunderstandings.

The second is whether we have the mechanisms you describe above. 
We do: the several documents are published as Experimental and, 
once things shake out, someone proposes that one of them be 
reevaluated for Proposed Standard.   That doesn't respond to the 
desire you have expressed elsewhere to be able to move a 
well-tested and deployed specification directly to Draft. 
Perhaps it is worth thinking about criteria for
   Experimental -> Draft
to supplement
   Proposed -> Draft
but that is a separate issue, I think.

    john

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 18 May 2003 16:53:21 +0200

> Could you explain how this makes the step up from PS worth the trouble?

Because you know that the effort of documenting the interoperable
implementations will be rewarded by immediately getting to the final
status, rather than just getting to an intermediate plateau (and one with
a very confusing name, since to outsiders it sounds like a step backwards).

  Brian

---

From: Keith Moore <[email-masked]>
Date: Fri, 16 May 2003 12:01:52 -0400

> *what sort of bugs, if detected, would make a protocol unsuitable for 

the way Proposed is treated by industry now, anything that would cause
problems in wide deployment should make a protocol unsuitable for
Proposed.

certainly failure to interoperate would be in this list,
but so would failure to scale, lack of robustness, failure to provide
adequate security, and a lot of other problems.

> A related issue is whether or not the restrictions on going to Draft
> from Proposed are really appropriate; currently, we say that *only*
> deletion of features is appropriate, and that any change or extension,
> no matter how worthwhile the purpose or how sure we are that it makes
> no problems, requires another round through Proposed. Is this really
> best for the Internet?

basically anything that would break compatibility should cause a reset.
new features must be considered suspect, because they haven't been
subjected to the same level of deployment and/or interop testing as
the old ones.  a new way of providing an old feature might be okay,
if the new way were based on experience with the old feature and we
had high confidence that the new way would work based on that
experience.  but I've seen multiple attempts to solve a problem fail.

Keith

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 20 May 2003 06:47:56 -0500

I haven't seen any followup to Brian's note, and wanted to

(1) agree that moving from PS to full Standard is more
likely interesting than moving from PS to Draft Standard,

(2) agree that Draft Standard does NOT seem like a more
impressive name than PS to anyone outside IETF (and
it's been a while since someone has been confused about
the difference between "draft standard" and "internet draft
on the standards track" in my presence, but it has happened).

Spencer

---

From: Pekka Savola <[email-masked]>
Date: Tue, 20 May 2003 15:04:14 +0300 (EEST)

> likely interesting than moving from PS to Draft Standard,

Moving from PS to Draft typically seems to include a lot of tuning of the 
spec, including removal of features and adding a few new ones (to 
replace/enhance/augment the ones killed for instance), or clarifying ones 
(in such a manner that all the old implementation are not compliant).

If we went from PS to Full, the documents certainly would have to recycled 
at the PS stage far, far longer to ensure there are no glitches left.  In 
addition, requiring more than 2 interop implementations of each feature 
should be required if we're to call something a Standard (unless we 
redefined the way it is, and engrave the "good old standards" in a 
golden plaque).

> on the standards track" in my presence, but it has happened).

True.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 May 2003 15:21:06 +0200

Pekka Savola wrote:
> at the PS stage far, far longer to ensure there are no glitches left. 

Disagree. Good enough is good enough. If you find glitches later, and
anybody cares, recycle at Standard. I do *not* suggest we should raise
the bar for promotion from Proposed Standard by one millimetre. I am
proposing that we simply abolish and forget the second bar.

   Brian

---

From: Keith Moore <[email-masked]>
Date: Tue, 20 May 2003 10:36:53 -0400

> proposing that we simply abolish and forget the second bar.

offhand I don't recall many cases where we recycled at draft.  so I think the
cases where we'd recycle at proposed in order to avoid having the document go
to Standard would also be rare.

one thing I'd like to see us consider is to issue more applicabliity
statements, errata documents, and implementation notes, rather than
revising an entire document to accomodate minor changes.  we could do that
even for Standard documents.

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 20 May 2003 10:42:35 -0400

Hi Keith,

>even for Standard documents.

I would agree, if our existing document set was organized to make
it easier for people to find associated documents.  Maybe that
is something we can change/fix?

Margaret

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 20 May 2003 10:17:52 -0500

Dear Margaret,

Well, now that you mention it, I believe these items would help people find
things:

- actual use of the STD maturity level, so that many questions could be
answered by "look at STD n", no matter what has happened to the underlying
RFCs and errata sheets lately,

- a severe pruning to Historical of protocols that aren't still widely used,

- possibily even a severe pruning to Historical of protocols that aren't
still being deployed (even if they are still in use),

- decent STF organization (note that STD 5 includes SIX RFCs on IP and ICMP.
Conversely, how many Telnet options are there? they're spread across SIX
STDs).

- I really like the Updated By/Obsoleted By information you get when you
SEARCH for a document at http://www.rfc-editor.org/rfcsearch.html, but this
information could appear other places, too (in document bodies would be
nice, but I know we don't change RFCs without changing RFC numbers).

Others?

Spencer

----- Original Message -----
From: "Margaret Wasserman" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>; "Brian E Carpenter"
<[email-masked]>
Sent: Tuesday, May 20, 2003 9:42 AM
Subject: Re: OPEN ISSUE: Standards Track

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 May 2003 17:29:09 +0200

> is something we can change/fix?

http://www.rfc-editor.org/rfcxx00.html goes in this direction, but
could use some more snazzy hyperlinking.

   Brian

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 20 May 2003 10:38:21 -0500

Dear Pekka,

My read of the Draft Standard concept was to say "we now have
two distinct interoperable implementations, so the specs must be
readable, but we haven't seen large-scale interoperation yet, because
we haven't seen large-scale deployment yet. Come back in a short
period of time, when we have large-scale deployment, and we'll
tell you if we killed the Internet".

Your note points up a problem we all know about but we haven't
written down in the Problem Statement draft - reduced genetic
diversity also means that the distinction between Draft Standard and
Standard may mean less than previously. I'm thinking of cases like

- OSPF
- HTTP
- closed-system IM, where the same people provide clients and servers

as examples where there may not be a lot of additional impact on the
Internet
after one or two implementations are deployed  :-{ ... because some large
percentage of the affected systems already run that code.

And if these examples aren't the right ones, I bet you can think of a couple
more!

Spencer

---

From: Pekka Savola <[email-masked]>
Date: Tue, 20 May 2003 22:05:23 +0300 (EEST)

> could use some more snazzy hyperlinking.

http://www.ietf.org/ietf/1rfc_index.txt is your friend.

Btw, I keep wonder why that's so ridiculously difficult to find.
Really.  It should be much more prominent.