Subject: RE: pausable explanation for the Document Series

From: <[email-masked]>
Date: Fri, 6 Jun 2003 08:15:15 +0300

Hi Bob,

> >The more-core problem is industry running on protocols with design flaws 
> >and protocol bugs, which cannot be fixed because of the installed base.
> >
> >If PS was perfect, this would not be a serious problem. But it isn't so.
> 
> First versions of anything are never perfect.  This is true for products 
> and standards.  As long as we try to solve the problem by trying to make 
> the first version perfect we will fail.  It only delays the first version 
> and causes it to miss the market need.  The only solution I know of is to 
> do new versions.  This seems to work well in industry.
> 
> Perfection doesn't work.  Shipping products and getting bug 
> reports works.

I agree with you on this point.  Until someone writes an ascii to 
C (or your favorite coding language) to convert I-Ds to running
code, it'll be impossible to ensure that a PS does not have
bugs.  A more sensible way forward, in my opinion, would be 
to ensure a better way for bug reporting, document updating, etc.
I think Harald's maintainence teams is a good potential solution.

John

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:38:21 -0400

If we want PS to mean do prototype but don't product the outcome.  Then
if a spec cannot get to PS say in 5 IETF meetings then it should just
die or got to experimental.  Then we have to use our partners in
industry like ISOC, Fora, other Standards bodies, and the Press to make
it clear to the market and vendors that PS will no longer or treated by
the IETF as "installed base" to not change it.  We would have to
grandfather all stuff before that for purposes of installed base or it
would be "perceived" as coo to kill something.
Most vendors are very careful with PS.

/jim

> -----Original Message-----
> From: Harald Tveit Alvestrand [mailto:[email-masked]] 
> Sent: Thursday, June 05, 2003 2:14 AM
> To: [email-masked]; [email-masked]
> Subject: Re: pausable explanation for the Document Series
> 
> 
> 
> 
> --On torsdag, juni 05, 2003 06:53:57 +0300 
> [email-masked] wrote:
> 
> > Hi all,
> >
> > I've always assumed that industry does not use DS or FS 
> simply because 
> > the IETF does not produce them in any great number.  The 
> IETF doesn't 
> > seem to produce them because WGs are, in general, charted 
> to make PSs; 
> > after which, they try to shut down.  There is very little 
> incentive in 
> > the IETF progress documents.  Industry, being industry, takes what 
> > they get & runs with it.
> >
> > I'm not sure how many of the proposals discussed would impact this 
> > situation in any meaningful way.
> 
> at least two try to:
> 
> Brian's DS/Full merger
> 
> <shameless plug>
> Maintenance Teams
> </shameless plug>
> 
> but industry running on PS is not a core problem, I think.
> 
> (my favourite example of deployment lock-in is the MIME 
> version number - 
> when the first post-RFC revision of the MIME spec was done, 
> we wanted to 
> increase the version number from 1.0 to 1.1, to celebrate the 
> fixing of 
> many bugs and unclear points in the specification.
> One vendor had a product in days-before-release state, which 
> would not 
> interoperate with UAs that sent 1.1 in the version number.
> We decided to freeze the version number at 1.0 forever...... 
> not a big loss to humanity, but it looks stupid.)
> 
>                     Harald
> 
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:48:20 -0400

> Perfection doesn't work.  Shipping products and getting bug 

It is also called spiral engineering.  And I again fully agree with Bob.

/jim

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:52:43 -0400

And what bugs are permitted and not permitted and why we made those
trade-offs when sending to the AD should be done.  If the AD don't agree
have them relay clearly why they think the WG is wrong and come to the
mail list.

/jim

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Friday, June 06, 2003 1:15 AM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: pausable explanation for the Document Series
> 
> 
> Hi Bob,
> 
> > >If PS was perfect, this would not be a serious problem. 
> But it isn't 
> > >so.
> > 
> C (or your favorite coding language) to convert I-Ds to 
> running code, it'll be impossible to ensure that a PS does 
> not have bugs.  A more sensible way forward, in my opinion, would be 
> to ensure a better way for bug reporting, document updating, 
> etc. I think Harald's maintainence teams is a good potential solution.
> 
> John
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:54:55 -0400

And a lot of them are just sitting there because either someone forgot
or afraid of going to PS to the ADs now they have made it so painful.

Example IPv6 MLDv3 to support exactly what we do for IGMPv3.  Unreal.

/jim 

> From: Jari Arkko [mailto:[email-masked]] 
> Sent: Friday, June 06, 2003 1:57 AM
> To: Bob Hinden
> Bob Hinden wrote:
> >> The more-core problem is industry running on protocols with design
> >> flaws and protocol bugs, which cannot be fixed because of the 
> >> installed base.
> >>
> > the first version perfect we will fail.  It only delays the first 
> > version and causes it to miss the market need.  The only 
> solution I know 
> > of is to do new versions.  This seems to work well in industry.
> > 
> > Perfection doesn't work.  Shipping products and getting bug reports 
> > works.
> 
> Agree 100%. Waterfall development vs. incremental 
> development. Serious debates early on in the project how all 
> possible features "must" be supported. Claims that engineers 
> would not make the mistakes they usually do, if they had just 
> done more "planning" or "design". Introduction of reviews, so 
> that more problems can be caught. Claims that the product we 
> are making is soooo important that we can not possibly have 
> any bugs or missing features in it. In the end, key success 
> factors for the project are missed, because no one bothered 
> to try out how the thing would work out in real usage. Or 
> competition is on its improvement cycle 17 when your first 
> buggy release comes out. Or you run out of money. Finally, 
> the management blames the engineers for bad quality.
> 
> I've been there in the distant past, but I know better now.
> But I'm alarmed that the above sounds very much like the 
> discussions we're having here.
> 
> A few observations on what this might mean for
> the IETF:
> 
> - It really is essential to get real usage, not just
>    more reviews or interop tests. So *some* amount of
>    industry adoption early on is required.
> 
> - Having to "fix" a standard (e.g. deprecate site-locals)
>    is not a sign of bad early specifications, its a sign
>    of being able to learn and adopt.
> 
> - While we normally don't like to compromise on things
>    like quality, security, scalability, we have to learn
>    to do things in pieces better. And when we are already
>    doing things in small pieces, we need to have a better
>    view of the full architecture/roadmap early on.
> 
> - This doesn't necessarily mean that we have to put out
>    more PS RFCs with a lower quality. We could keep
>    ahead of the industry's exaggeration of PS status
>    value by introducing a new document level lower in
>    chain.
> 
> --Jari
> 
> 
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 22:56:48 -0400

This sounds like really good input to our process draft I suggest too
and instead of just having Be conservative what you send but liberal in
what your receive how about adding what Bob Hinden just said about
products and bugs and add that to our "core" values and it would mean a
lot more and get a lot more done.

/jim

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 23:02:25 -0400

> I partially disagree.  It's often too challenging to later change 
> fundamental design decisions made early on: just shipping a 
> spec with a 
> thought like "we'll fix the issues later" seems like a recipe for 
> disaster.

This is a paranoid and low risk way of viewing first implemenation of
products and standards.  The key compromise to this paranoid view is to
make sure that applications do not depend on the change at least for
product.  For a standard that is where the applicability statement comes
in.  The paranoid view will have us doing things like DHCPv6 and MObile
IPv6 for 6 years to just get to PS and this is UACCEPTABLE.

> 
> Naturally, when updating the documents some features are 
> ripped off, some 
> clarifications added etc., but this *typically* cannot 
> influence the basic 
> way specs work.  And therefore, the core things in specs must 
> be in very 
> good shape.

Defining what is core then is a key to success for those who want to do
prototype and for those that are paranoid.

> 
> In consequence, "security will be described in other documents" or 
> "deployment will be considered in other documents" doesn't 
> sound like a 
> good idea.  These can often hide some very nasty surprises 
> which affect 
> your design decisions on the core spec.

I would agree with the above.  But a spec does not have to be secure on
the Internet for PS prototype only in a lab.  That is the core issue for
me.

/jim

> 
> -- 
> Pekka Savola                 "You each name yourselves king, yet the
> Netcore Oy                    kingdom bleeds."
> Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
> 
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 23:07:04 -0400

> In real world, people implement the protocols for pilot/test 
> purposes in 
> any case.  This can be done when the I-D has reached 
> reasonable level of 
> stability.

I am sorry but this is no longer true for most specs except by the
original proponents. The IETF has burned all with the high bar on PS and
most of us "product" engineers won't touch it until it is PS.  Unless we
see consortia out of the IETF to help us move it forward as a group of
vendors.  FYI that is what is happening and will happen with IPv6
Transition mechanisms as a note as one example.

So your thought is old and does not apply.  Not good argument to John
and invalid now.

It used to be true yes. 

> 
> Of course, we could try to add some "experimental" category 
> below PS, to 
> try to document these pilot protocols -- but this is slightly 
> problematic 
> also, as we have to make sure people understand that if protocol is 
> modified for PS, no backward-compatibility must be required..

No this will not work.  Go back to Bob Hinden phrase as priniciple.
Avoiding that principle is just hiding the problem and not facing it
right here and right now.

Someone here will loose and someone here will win.  I hope you loose of
course.

/jim

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 23:15:17 -0400

And I agree with Charlie.
/jim

> From: Charlie Perkins [mailto:[email-masked]] 
> Sent: Friday, June 06, 2003 10:15 AM
> To: Keith Moore
> Hello Keith,
> Keith Moore wrote:
> >>Agree 100%.
> >Disagree 100%.  It costs much more to fix bugs after 
> shipping product 
> >than it
> >does to do so beforehand.  Forcing customers to accept the 
> costs of poor
> >protocol design is just irresponsible.
> >  
> >
> You seem to assume that the customers are going to sit around waiting.
> 
> Most will buy a system that works, and the perfect product 
> will, after a three year delay, be a footnote in history.
> 
> And that's where recent IETF protocol efforts may be heading.
> 
> I agree with Bob and Jari.
> 
> Regards,
> Charlie P.
> 
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 23:18:38 -0400

A favorite phrase of mine and I think my colleague Charlie sums it up
Keith.
Caveat Emptor.

/jim

> Sent: Friday, June 06, 2003 10:52 AM
> To: Keith Moore
> You're using nasty words like "accuse".
> 
> Keith Moore wrote:
> 
> >>Most will buy a system that works,
> >>    
> >>
> >
> >most will buy a system without knowing whether it works or not, and 
> >pull their hair out when they have trouble getting it to work in 
> >practice, and then guess about who is to blame, and make arbitrary 
> >demands about who has to fix it that have nothing to do with 
> the actual 
> >cause of the problem.
> >
> Life sucks that way.  It helps to buy from reputable vendors. 
>  Industrial product managers with a habit of success tend to 
> do better than you portray.
> 
> >NOBODY has said anything about requiring things to be 
> perfect, so stop 
> >accusing people of that.  What has been repeatedly suggested is that 
> >the criteria for maturity levels should be better matched 
> with industry 
> >expectation and IETF energy levels.  What's wrong with that?
> >
> But when you hear that industry has expectations about 
> timeliness you seem to go nonlinear.  Furthermore, I suggest 
> that IETF processes demotivate and drain the energy of 
> otherwise enthusiastic IETF engineers.
> 
> Regards,
> Charlie P.
> 
> 
>

---

From: "Bound, Jim" <[email-masked]>
Date: Sat, 7 Jun 2003 23:40:22 -0400

> now now.  we could aspire to being the microsoft of
> standards. sell a lot of half-baked crap, laugh our way to 
> the bank, and not be able to face ourselves in the mirror.
> 
> randy
> 
> ---
> 
> ps: sorry to pick on microsoft, it's the image.

You should not have done this sir.  I am sorry but you simply cannot do
this anymore. This is very wrong and not right.  Children all over the
world impart have access to the Internet because of Microsoft, Cisco,
Sun, HP, Juniper, Nokia, et al.  To trash them on this very public list
is simply not right.  OK privately but this is not even funny. Realize
there good be person on this list that could influence a procurement and
possibly pause based on a comment here.

You have crossed a line here and I wish you had not done that, I really
do.

In the future can you refrain from calling any vendors parts "crap"?
Especially as an IESG member in this community, and yes that does make
it worse. 

/jim

---

From: "Bound, Jim" <[email-masked]>
Date: Mon, 9 Jun 2003 15:07:47 -0400

I understand.  But keeping bugs out is something I consider pretty easy
to do if we follow a good engineering process and review.  Where it gets
tricky is the classic engineering trade-off of risk vs going forward and
where we live with the warts.  Another is if we have option 1, 2, and 3
and the WG is split on supporting all options.  This gets back to my
issue of we need to put the assumptions we have on the table better.

/jim

> From: Keith Moore [mailto:[email-masked]] 
> Sent: Monday, June 09, 2003 12:13 PM
> To: Bound, Jim
> Cc: [email-masked]; [email-masked]; 
> [email-masked]; [email-masked]
> > You miss the point I think?  And this is exactly how most products 
> > ship it is impossible to find all bugs in any first 
> software release. 
> > What you try to find are the show stoppers based on what that means 
> > for that product function.
> > 
> > So I suggest your disagreeing with reality?
> 
> Not at all.  Protocols aren't quite like products in several 
> ways.  If a vendor makes a buggy product, it screws those 
> that buy that product, but not everyone else who uses the 
> same protocol.  (unless the vendor has a monopoly, in which 
> case I'd argue that the vendor has a higher
> responsibility.)  Also, protocol bugs are often harder to fix 
> than product bugs(depending on the product). Also, protocols 
> aren't specified to the same level of detail that the code to 
> implement those protocols are, so it's reasonable to expect 
> fewer bugs in protocol specs.
> 
> Again, I think the attempt to characterize the desire to have 
> higher quality in our protocol specs before we call them any 
> kind of standard (and before we consider them ready for 
> deployment) as wanting those 
> protocols to be "bug free" or "perfect" is misleading.  
> Clearly it is neither possible nor realistic to expect bug 
> free protocols.  But that does not excuse our publishing a 
> protocol standard with known bugs, nor does it excuse our 
> failures to do due diligence to avoid major problems in our 
> protocol designs.
>