Subject: pausable explanation for the Document Series

From: <[email-masked]>
Date: Thu, 5 Jun 2003 06:53:57 +0300

Hi all,

I've always assumed that industry does not use DS or FS simply because
the IETF does not produce them in any great number.  The IETF doesn't
seem to produce them because WGs are, in general, charted to make
PSs; after which, they try to shut down.  There is very little incentive
in the IETF progress documents.  Industry, being industry, takes what
they get & runs with it.

I'm not sure how many of the proposals discussed would impact this 
situation in any meaningful way.

br,
John

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 05 Jun 2003 08:13:39 +0200

> situation in any meaningful way.

at least two try to:

Brian's DS/Full merger

<shameless plug>
Maintenance Teams
</shameless plug>

but industry running on PS is not a core problem, I think.

The more-core problem is industry running on protocols with design flaws 
and protocol bugs, which cannot be fixed because of the installed base.

If PS was perfect, this would not be a serious problem. But it isn't so.

(my favourite example of deployment lock-in is the MIME version number - 
when the first post-RFC revision of the MIME spec was done, we wanted to 
increase the version number from 1.0 to 1.1, to celebrate the fixing of 
many bugs and unclear points in the specification.
One vendor had a product in days-before-release state, which would not 
interoperate with UAs that sent 1.1 in the version number.
We decided to freeze the version number at 1.0 forever......
not a big loss to humanity, but it looks stupid.)

                    Harald

---

From: Bob Hinden <[email-masked]>
Date: Thu, 05 Jun 2003 17:32:30 -0700

Harald,

>If PS was perfect, this would not be a serious problem. But it isn't so.

First versions of anything are never perfect.  This is true for products 
and standards.  As long as we try to solve the problem by trying to make 
the first version perfect we will fail.  It only delays the first version 
and causes it to miss the market need.  The only solution I know of is to 
do new versions.  This seems to work well in industry.

Perfection doesn't work.  Shipping products and getting bug reports works.

Bob

p.s. Bug reports are also a good way to measure real usage.

---

From: Jari Arkko <[email-masked]>
Date: Fri, 06 Jun 2003 08:57:00 +0300

Bob Hinden wrote:

>> The more-core problem is industry running on protocols with design 
>> flaws and protocol bugs, which cannot be fixed because of the 
>> installed base.
>>
> the first version perfect we will fail.  It only delays the first 
> version and causes it to miss the market need.  The only solution I know 
> of is to do new versions.  This seems to work well in industry.
> 

Agree 100%. Waterfall development vs. incremental development.
Serious debates early on in the project how all possible features
"must" be supported. Claims that engineers would not make the
mistakes they usually do, if they had just done more "planning"
or "design". Introduction of reviews, so that more problems
can be caught. Claims that the product we are making is soooo
important that we can not possibly have any bugs or missing
features in it. In the end, key success factors for the project
are missed, because no one bothered to try out how the thing
would work out in real usage. Or competition is on its improvement
cycle 17 when your first buggy release comes out. Or you run out
of money. Finally, the management blames the engineers for
bad quality.

I've been there in the distant past, but I know better now.
But I'm alarmed that the above sounds very much like the
discussions we're having here.

A few observations on what this might mean for
the IETF:

- It really is essential to get real usage, not just
   more reviews or interop tests. So *some* amount of
   industry adoption early on is required.

- Having to "fix" a standard (e.g. deprecate site-locals)
   is not a sign of bad early specifications, its a sign
   of being able to learn and adopt.

- While we normally don't like to compromise on things
   like quality, security, scalability, we have to learn
   to do things in pieces better. And when we are already
   doing things in small pieces, we need to have a better
   view of the full architecture/roadmap early on.

- This doesn't necessarily mean that we have to put out
   more PS RFCs with a lower quality. We could keep
   ahead of the industry's exaggeration of PS status
   value by introducing a new document level lower in
   chain.

--Jari

---

From: Pekka Savola <[email-masked]>
Date: Fri, 6 Jun 2003 09:08:12 +0300 (EEST)

> Perfection doesn't work.  Shipping products and getting bug reports works.

I partially disagree.  It's often too challenging to later change 
fundamental design decisions made early on: just shipping a spec with a 
thought like "we'll fix the issues later" seems like a recipe for 
disaster.

Naturally, when updating the documents some features are ripped off, some 
clarifications added etc., but this *typically* cannot influence the basic 
way specs work.  And therefore, the core things in specs must be in very 
good shape.

In consequence, "security will be described in other documents" or 
"deployment will be considered in other documents" doesn't sound like a 
good idea.  These can often hide some very nasty surprises which affect 
your design decisions on the core spec.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Keith Moore <[email-masked]>
Date: Fri, 6 Jun 2003 09:48:20 -0400

> Agree 100%. 

Disagree 100%.  It costs much more to fix bugs after shipping product than it 
does to do so beforehand.  Forcing customers to accept the costs of poor
protocol design is just irresponsible.

---

From: Charlie Perkins <[email-masked]>
Date: Fri, 06 Jun 2003 07:14:48 -0700

Hello Keith,

>protocol design is just irresponsible.
You seem to assume that the customers are going to sit around waiting.

Most will buy a system that works, and the perfect product will,
after a three year delay, be a footnote in history.

And that's where recent IETF protocol efforts may be heading.

I agree with Bob and Jari.

Regards,
Charlie P.

---

From: Keith Moore <[email-masked]>
Date: Fri, 6 Jun 2003 10:30:08 -0400

> Most will buy a system that works, 

most will buy a system without knowing whether it works or not, and pull their
hair out when they have trouble getting it to work in practice, and then guess
about who is to blame, and make arbitrary demands about who has to fix it that
have nothing to do with the actual cause of the problem.

> And that's where recent IETF protocol efforts may be heading.

NOBODY has said anything about requiring things to be perfect, so stop
accusing people of that.  What has been repeatedly suggested is that the criteria
for maturity levels should be better matched with industry expectation and
IETF energy levels.  What's wrong with that?

---

From: Charlie Perkins <[email-masked]>
Date: Fri, 06 Jun 2003 07:52:17 -0700

Hello Keith,

You're using nasty words like "accuse".

>have nothing to do with the actual cause of the problem.
Life sucks that way.  It helps to buy from reputable vendors.  Industrial
product managers with a habit of success tend to do better than you
portray.

>IETF energy levels.  What's wrong with that?
But when you hear that industry has expectations about timeliness
you seem to go nonlinear.  Furthermore, I suggest that IETF processes
demotivate and drain the energy of otherwise enthusiastic IETF engineers.

Regards,
Charlie P.

---

From: Keith Moore <[email-masked]>
Date: Fri, 6 Jun 2003 11:12:43 -0400

> >most will buy a system without knowing whether it works or not, and pull
> >their hair out when they have trouble getting it to work in practice, and
> >then guess about who is to blame, and make arbitrary demands about who has
> >to fix it that have nothing to do with the actual cause of the problem.
> >
> Life sucks that way.  It helps to buy from reputable vendors. 

gee, it's tempting, but I'm not going to get into an argument about which
vendors are reputable.  suffice it to say that for kinds of the products I
buy, there are none.  YMMV.

> >accusing people of that.  What has been repeatedly suggested is that the
> >criteria for maturity levels should be better matched with industry
> >expectation and IETF energy levels.  What's wrong with that?
> >
> you seem to go nonlinear.  

nope.  it's when I hear suggestions that we should lower the quality of our
output even further because we're supposedly trying to make it "perfect" that
I go nonlinear.

> Furthermore, I suggest that IETF processes

agree entirely.  but I propose that we fix the process to make better use 
of those energies rather than simply relaxing our criteria in the interest
of timeliness.

---

From: Charlie Perkins <[email-masked]>
Date: Fri, 06 Jun 2003 08:38:10 -0700

Hello Keith,

Keith Moore wrote

>buy, there are none.  YMMV.
Indeed, my mileage varies.

>output even further because we're supposedly trying to make it "perfect" that
Let's take site-local as an example.  By your criteria, it seems to me we
still would not have published any IPv6 address architecture document
because of seeming flaw in the site-local design philosophy that was
promulgated.

I suggest that it would have been totally counterproductive to introduce 
such delay.

Furthermore, I believe that other protocol efforts HAVE sustained nearly
equal damage because of similar reluctance to allow wider deployment
experience.  Proposed Standard effectively mean{s,t}, "let's get wider
deployment experience".

>of those energies rather than simply relaxing our criteria in the interest
Your criteria strongly inhibit timeliness.

Regards,
Charlie P.

---

From: Keith Moore <[email-masked]>
Date: Fri, 6 Jun 2003 11:48:53 -0400

> Let's take site-local as an example.  By your criteria, it seems to me
> we still would not have published any IPv6 address architecture
> document because of seeming flaw in the site-local design philosophy
> that was promulgated.

nope.  I would have marked that address range as "reserved" and
published the document without site-local.

> Furthermore, I believe that other protocol efforts HAVE sustained
> nearly equal damage because of similar reluctance to allow wider
> deployment experience.  Proposed Standard effectively mean{s,t},
> "let's get wider deployment experience".

No, it meant "let's get implementation experience".

> >agree entirely.  but I propose that we fix the process to make better
> >use of those energies rather than simply relaxing our criteria in the
> >interest of timeliness.
> >

You don't even know what my criteria are, Charlie, because I haven't
tried to define them in detail.  

This is the problem statement WG, not the problem resolution WG.
Our job is to identify problems, not to argue over the merits of
solutions that haven't even been outlined yet.  We have a identified
a problem that our standards maturity levels do not match with
either industry expectations or IETF energies.  Do you agree with
that problem statement or not?

Keith

---

From: Charlie Perkins <[email-masked]>
Date: Fri, 06 Jun 2003 09:00:16 -0700

Hello Keith

>published the document without site-local.
But the rest of the working group would have disagreed, and the discussion
could have continued for years.

>No, it meant "let's get implementation experience".
Implementation tests whether it can be built.  Deployment tests
whether it works.  I think my formulation is correct.  I also think that
we MUST NOT publish Proposed Standard documents that do not
have implementation experience and interoperability experience.

>tried to define them in detail.
I know that your criteria have the effects that you have described in
your e-mails, enough so that I can stand by my statement.

>that problem statement or not?
Yes.  I agree with the statement.

I want to see a more timely Proposed Standard.  If the "Proposal" isn't
right, I want to see that we can recycle to new a new Proposed Standard.

I do NOT believe we have to issue poorer quality documents to meet
better timeliness, but I _do_ think that some AD demands result in the
need to document entire systems instead of just protocols, thus delaying
protocol publication.  This is another big mistake in my opinion.

I suggest that the criterion has to be "does it work", instead of
"how do I use it"?  This is related to the previous discussion about
the need for an applicability statement.

Regards,
Charlie P.

---

From: Keith Moore <[email-masked]>
Date: Fri, 6 Jun 2003 13:16:00 -0400

> I want to see a more timely Proposed Standard.  If the "Proposal"
> isn't right, I want to see that we can recycle to new a new Proposed
> Standard.

I want to see us completely re-evaluate our maturity levels.  I think if
we try to tweak the ones we have we'll fail to fix most of the problems
we have with the current setup.  Given that assumption, arguments about
how we are using the existing set of maturity levels aren't terribly
enlightening.  

But at any rate the task at hand is to identify the problem, not to
argue about how to fix it.

As for site-local, we need to examine why that got into the IPv6 spec in
the first place without any evidence that it would actually work.  But
that's a separate topic.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 07 Jun 2003 09:49:35 +0200

> Yes.  I agree with the statement.

Charlie, Keith:

Since you have something you agree on, and which I agree on too, and which 
I cannot find in the problem-statement document as currently written, 
perhaps the two of you could try to formulate a proposal for a change to 
the problem-statement document that incorporates this point, and which 
Melinda can track as an "issue to be resolved"?

Unless we capture the agreement, it might become lost again....

                 Harald

---

From: Randy Bush <[email-masked]>
Date: Sat, 7 Jun 2003 14:28:58 -0700

> Disagree 100%.  It costs much more to fix bugs after shipping
> product than it does to do so beforehand.  Forcing customers to
> accept the costs of poor protocol design is just irresponsible.

now now.  we could aspire to being the microsoft of standards.
sell a lot of half-baked crap, laugh our way to the bank, and not
be able to face ourselves in the mirror.

randy

---

ps: sorry to pick on microsoft, it's the image.