Subject: RE: Discipline of Internet Protocol Engineering

From: <[email-masked]>
Date: Wed, 4 Jun 2003 09:31:07 +0300

Hi Dave & Keith,

> KM> The reason I say this is that several groups have demonstrated the inability
> KM> to define the problem they are working on,
> 
> Unfortunately, i think this problem is deeper than we might wish to
> acknowledge.  I tend to be a charter fascist, on the theory that a
> crystal clear charter will leave no doubt about the problem being
> solved, and/or the benefit to be derived from the result and the
> deliverables to be produced.
> 
> I view charters as real contracts, making clear what is included and
> obligated and what is excluded and prohibited.
> 
> However with great regularity, remarkably fuzzy charters are getting
> approved. since chartering involves lots of experienced people beyond
> the working group, we can't simply assess the problem on the working
> group folk.
> 
> I don't know how to improve this.  But, yes, we definitely 
> need to keep trying.

I would say that there is a breakdown in the chartering process.  I would
think, for example, inviting the proposed WG chairs to the conference call
when the charter is discussed could help, for example.  As a novice chair,
creating a charter is difficult. Better advice & engagement from the IAB/IESG
would be helpful.  For experienced WG chairs, there can be a tendancy that
many of the IAB/IESG seem to be more lenient with their proprosed charters 
because the experienced chair is a known quantity, so the thought is that
they know what they are doing.

John

---

From: Jari Arkko <[email-masked]>
Date: Wed, 04 Jun 2003 11:02:07 +0300

Keith Moore wrote:

> The reason I say this is that several groups have demonstrated the inability
> to define the problem they are working on,

John Loughney wrote:

> I would say that there is a breakdown in the chartering process. 

Hmm... I think that the problem is even deeper than you three
wish to acknowledge ;-) I'm sure there are quality differences
in charters. And that can be improved upon. However, I believe
there's a more fundamental reason why some charters are or may
need to be fuzzy. That reason has to do with the timing when
the IETF gets involved, or the scope of the problem. Are you
doing adding a new algorithm to an existing security protocol?
Pretty specific. Are you going to develop a protocol for a specific
purpose? Still pretty simple. Are you going to develop a whole
set of protocols and extensions to fulfill a function. Starting
to get more complex (example: SIP). Are you going to re-architect
IPn and develop IPn+2? Pretty tough... And the timing -- have
just realized a problem? Or are we at a stage where someone
else has realized the problem, and is bringing to the IETF
solution proposals, or even proposed protocols that we could
standardize? Of course, we could define the IETF as the organization
that only takes on very specific work, and only if there's
advanced protocol work and deployment experience from vendors
with the new work... not sure that would satisfy everyone.

--Jari

---

From: Keith Moore <[email-masked]>
Date: Wed, 4 Jun 2003 10:25:00 -0400

> However, I believe
> the IETF gets involved, or the scope of the problem. 

Part of the problem may be that we often want the initial charter to map
out the entire life cycle of the group.  often this is unrealistic, because
we really don't understand what the group has to do.  I don't think we 
should try to nail down details more than about eight months (say two 
IETF meetings) in advance.  But if we don't have a good idea about what
a WG is going to be doing over the next few months, with fairly concrete,
realizable, measurable goals - something's wrong.

---

From: Jari Arkko <[email-masked]>
Date: Wed, 04 Jun 2003 17:38:44 +0300

> realizable, measurable goals - something's wrong.

Agreed.

We could in addition require that groups produce something concrete
(like an approved document) in that 8 months. If we take in account
that the full protocol (or whatever) can't be done in this time,
this shows a need to produce intermediate deliveries. Often these
are requirements documents, though I'm not quite sure useful they
are. It might be more useful to produce roadmap or architecture
documents.

--Jari

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 04 Jun 2003 16:44:04 +0200

--On onsdag, juni 04, 2003 10:25:00 -0400 Keith Moore <[email-masked]> 
wrote:

> out the entire life cycle of the group.  often this is unrealistic,
> because we really don't understand what the group has to do.  I don't
> think we  should try to nail down details more than about eight months
> (say two  IETF meetings) in advance.  But if we don't have a good idea
> about what a WG is going to be doing over the next few months, with
> fairly concrete, realizable, measurable goals - something's wrong.
>

Might part of the problem be that we're lousy at making plans, *apart* from 
the charter?

I don't think I've ever seen a working group chair come up with a 
document/message/webpage that said something like "OK, in order to achieve 
this milestone in December, here's what we're planning to do in June, July, 
August, September and November.... we didn't do what we were supposed to do 
in May, so that means September's going to have a time crunch....."

again, this is something we have no rule against, it's normal practice in 
other branches of engineering, but we just don't seem to be doing it....

---

From: Keith Moore <[email-masked]>
Date: Wed, 4 Jun 2003 10:45:08 -0400

> Might part of the problem be that we're lousy at making plans, *apart* from 

yup.  or that we think that the charter is all the planning we need.

---

From: "Jeanette Hofmann" <[email-masked]>
Date: Wed, 04 Jun 2003 16:57:39 +0200

> Might part of the problem be that we're lousy at making plans, *apart* from 

I think we are dealing with two more or less unrelated issues here. One 
has to do with the observation of Dorothy L. Sayers that "facts are like 
cows. If you look them in the face hard enough they generally run away." 
Some flexibility regarding both tasks and time is necessary to be prepared 
for unexpected running cows or facts. 

The other issue is general time management. The working groups seem 
not apply what most of their members would do under similar 
circumstances in their day job. 

jeanette

---

From: Dave Crocker <[email-masked]>
Date: Thu, 5 Jun 2003 15:47:02 -0700

Folks,

JA> However, I believe there's a more fundamental reason why some
JA> charters are or may need to be fuzzy. That reason has to do with the
JA> timing when the IETF gets involved, or the scope of the problem.

This has been discussed over the years. We sometimes form working groups
without really knowing what they are supposed to do.

I believe that a working group must formulate its requirements statement
before forming or within a few months. Otherwise it is not ready to do
standards work. The IETF has a very poor track record with fuzzy goals,
and a very good one with very concrete, very near-term and very narrow
goals.

JA> Are you doing adding a new algorithm to an existing security
JA> protocol? Pretty specific. Are you going to develop a protocol for a
JA> specific purpose? Still pretty simple.

agree.

JA> Are you going to develop a whole set of protocols and extensions to
JA> fulfill a function. Starting to get more complex (example: SIP). Are
JA> you going to re-architect IPn and develop IPn+2? Pretty tough

Not just tough but I would say that our track record for such
large-scale project is quite poor, unless we have been able to carve off
pieces that are useful on their own.

That is, if we can do incrementally useful work, then the IETF has a
good chance of doing something that is timely and useful. Otherwise, we
don't. Our work suffers from technical and schedule bloat.

HTA> Take, for instance, the recent LEMONADE charter - which I think is
HTA> reasonably crisp and states what the WG is intended to do.
HTA> When it was first floated on the IETF list, it was blasted by at least a
HTA> couple of people (including you, I think)

Yes. However what I found interesting was that the fundamental concerns
about concreteness that I was raising did not seem to be that
interesting to most folks. Pete Resnick did a stellar job of negotiating
a compromise, but that should not have been necessary.

All I was seeking was a charter that said concretely what
problem/benefit was being attacked, and what the technical scope of work
would be, in language that permitted knowing what was *out* of scope as
much as what was *in* and knowing what would actually be delivered.

HTA> Yes, there were improvements. But I think the first version floated wasn't
HTA> that bad either

My sense is that looking at the history of that charter could be highly
instructive, since my impression is that some iterations of it received
"help" that made it considerably more mushy, in the earlier stages.

KM> Part of the problem may be that we often want the initial charter to map
KM> out the entire life cycle of the group.  often this is unrealistic, because
KM> we really don't understand what the group has to do.  I don't think we
KM> should try to nail down details more than about eight months (say two
KM> IETF meetings) in advance.  But if we don't have a good idea about what
KM> a WG is going to be doing over the next few months, with fairly concrete,
KM> realizable, measurable goals - something's wrong.

Folks -- we need to frame Keith's words and chant them at the beginning
of every meeting -- and it just kills me to say something this nice
about Keith.

JA> We could in addition require that groups produce something concrete
JA> (like an approved document) in that 8 months.

Note that the PACT suggestion was 18 months, yet folks were quite upset
about it.

HTA> Might part of the problem be that we're lousy at making plans, *apart* from
HTA> the charter?
HTA> I don't think I've ever seen a working group chair come up with a
HTA> document/message/webpage that said something like "OK, in order to achieve
HTA> this milestone in December, here's what we're planning to do in June, July,
HTA> August, September and November

Properly designed milestones will, at least, set the stage for exactly
that kind of thinking.  Or perhaps the way to talk about this is that
you are calling for the intermediate milestones that will achieve the
ones in the charter.

Egad. As you note, this starts sounding like classic project management.

What a thought.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Bob Hinden <[email-masked]>
Date: Thu, 05 Jun 2003 17:37:30 -0700

>Egad. As you note, this starts sounding like classic project management.

Worse still, we might even want people in the IESG with some real 
management experience.

Bob

---

From: Jari Arkko <[email-masked]>
Date: Fri, 06 Jun 2003 08:43:35 +0300

> Worse still, we might even want people in the IESG with some real 

And even worse, we might have to start tracking how much
"resources" (that means you and me) we have for our
projects, and figuring out how "big" the projects are.

--Jari

---

From: Margaret Wasserman <[email-masked]>
Date: Sat, 07 Jun 2003 12:36:19 -0400

Hi Dave,

At 03:47 PM 6/5/2003 -0700, Dave Crocker wrote:
>The IETF has a very poor track record with fuzzy goals,

You say this repeatedly, and I agree with it, but we seem
to reach different conclusions based on this fact...

Your reasoning seems to run:  The IETF is bad at resolving
complex issues, so we should only work on small, well-defined
problems.

My reasons runs:  The IETF is bad at resolving complex
problems, so we need to find a way to get better at it.

Some problems just _are_ complex...  Claiming that we
should only form WGs with narrow charters begs the
question:  How do we organize ourselves to solve the
complex problems?

Margaret

---

From: Dave Crocker <[email-masked]>
Date: Sat, 7 Jun 2003 21:52:56 -0700

Margaret,

MW> Your reasoning seems to run:  The IETF is bad at resolving
MW> complex issues, so we should only work on small, well-defined
MW> problems.

Actually, my reasoning runs a bit differently, and that bit of
difference is important:

IETF work has been notably successful only when it has done bite-sized
bits of coherent work. It has attacked complex issues successfully only
when it has attacked the issues piecemeal.

So, we should do what we know we are good at, rather than do what we
know we are bad at.

MW> My reasons runs:  The IETF is bad at resolving complex
MW> problems, so we need to find a way to get better at it.

That would be delightful.  But it does not make sense to place a
requirement into our critical path for something which we have almost 15
years of failure performing.  Better still, we have no basis for yet
believing we know how to perform it.

d/

---

From: Keith Moore <[email-masked]>
Date: Tue, 24 Jun 2003 16:07:13 -0400

> bits of coherent work. It has attacked complex issues successfully
> only when it has attacked the issues piecemeal.

I think we have had far more failures than successes by attacking issues
piecemeal.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Jun 2003 13:41:33 -0700

Keith,

>> only when it has attacked the issues piecemeal.

KM> I think we have had far more failures than successes by attacking issues
KM> piecemeal. 

This sub-thread has really excellent potential for becoming a rat-hole.
However the divide-and-conquer principle that I am citing clearly is
interpreted differently by different people.  So, perhaps it is worth
trying to dodge the rat-holes, while trying to explore the matter
substantively:

Solving a big, complicated problem by trying to develop a single,
coherent specification -- no matter how many documents; the point is
about making all the bits of work proceed in lock-step -- is a
well-known way to ensure failure. One only needs to look at OSI in
general, and X.400 in particular. (And by way of anticipating one of the
rat-holes, I'll note that X.400 achieved the rare success of having too
much be integrated all at once, but still lack core bits of
functionality.)

The usual logic for explaining this problem is that the complexity of
juggling all the issues, across the entire service, simply buries the
development effort. At best, it ensures that the work is produced much,
much later, often after the market has found an alternate solution. (And
that is, most certainly, what happened to OSI. I can elaborate on this
is some detail, if anyone really challenges this point.)

Breaking down a complex problem into smaller pieces that are
individually useful does two things that are quite good:

1) It permits each piece of work to be useful, even if some other piece
of work has a persistent problem; and 2) it permits turning the crank on
the specification engine more quickly and more frequently. This means
that we get operational experience more quickly and can then, quickly,
refine the specifications to match actual field knowledge. With large,
integrated efforts, the inter-dependencies mean that the whole of the
work is not useful until the most problematic part is completed. And it
means that it is quite a few years before there is any feedback from the
user community;. If there was a problem with the major design decisions
that were made, they become essentially impossible to change, because
things took so long to delivery.

A couple of examples of the success stories in the Internet:

(As an aside, I'll note that TCP and IP were originally one protocol,
but the divide-and-conquer approach was adopted prior to fielding the
technology.  This makes exploring and delivering new transport modules
far easier.  Had this not been done, we'd be using TCP even when a UDP
would have been sufficient.)

Is IP a failure?  Besides independent development of TCP and UDP, from
the IP core, ICMP has been able to proceed independently, as has
different address-mapping efforts and, for that matter, address
interpretation efforts.

Is email a failure? Transfer services and content services have
proceeded with considerable independence. This permits multiple transfer
services. (Though folks might think there is only SMTP, there are
others, and there used to be more. So another bonus from good
divide-an-conquer is vastly superior adaptability to changing
conditions, such as adoption by "fringe" or "new" networking stacks.)
And note the several large, integrated efforts to replace the
ridiculously limited capabilities of Internet mail -- notably to achieve
multi-media. By contrast the silly, ugly, inefficient functional module,
called MIME, retrofitted multi-media onto Internet mail with remarkable
success and remarkably quickly.

(One could go down a rathole about nonconformance, noting how badly we
suffer from excessive variance in email system behavior, but I claim
that is due to lack of enforcement, rather than due to any architectural
issues. Given the wide range of support for HTML, and the like, this
might be an applications-level issue. But it is not a big-vs-small
design effort issue.)

In fact, this highlights one of the other, surprising benefits of a
divide-and-conquer approach, namely that it permits post-hoc additions
that were not planned.  When the original philosophy is to do as little
as possible to be useful, knowing that more bits of related work will be
done, then it often is easier to do some of those bits many years later.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Jun 2003 18:43:45 -0400

> individually useful does two things that are quite good:

divide-and-conquer is all well and good, provided the pieces don't 
conflict when we try to use them together.  it's not divide-and-conquer
I object to, it's our too-frequent failure to provide an architecture
that informs the design of each of the pieces in a way that allows
them to work together.

also, there are too many cases where one group tries to take something
that is already in use by others and change it in a way that favors
the group making the change while harming its functionality for
others.  NAT in IPv4, linklocal addressing in IPv4, and content
negotiation over SMTP, are all good examples.

Keith