Subject: Re: Complex Problems (Was: Re: Discipline of Internet Protocol Engineering)

From: John C Klensin <[email-masked]>
Date: Wed, 02 Jul 2003 09:01:00 -0400

Dave,

I've needed to let this set of messages season a bit in the hope 
of saying something coherent and not repeating myself too much.

I think one needs to distinguish between "adequate understanding 
of the problem" and "overly broad attempts at a solution".  For 
the latter, we are, I think, pretty much in agreement: attempts 
to engineer large, integrated, complex solutions have rarely 
been successful in the IETF (or, perhaps, anywhere else).  But, 
when we take a complex _problem_ and say "well, we understand 
the lower-right-corner of it, so let's 'solve' that" often leads 
us into a solution that doesn't integrate well with the other 
pieces when those come along.  That poor integration can 
constrain opportunities and solutions going forward or can force 
us toward incompatible changes, neither of which is good.

Clearly, it is important to find a balance, since the quest for 
comprehensive and perfect understanding of every problem would 
result in our never actually getting anything done (if one were 
cynical, one might make that observation about this WG as well 
as about the IETF's technical/engineering work).  But "this 
piece will work, so let's do it" needs, IMO, to be carefully 
balanced with "and _what_ problem is it solving?".  And we need 
to have clear explanations and a reasonable consensus about the 
latter.   I suggest that failure to take that step has gotten us 
into a lot of trouble in some areas in the past and that may be 
where my views and Keith's intersect.

With that distinction between problem analysis and solutions, 
let's look at some of your examples...

--On Thursday, 26 June, 2003 13:41 -0700 Dave Crocker 
<[email-masked]> wrote:

>...
> Solving a big, complicated problem by trying to develop a
> single, coherent specification -- no matter how many
> documents; the point is about making all the bits of work
> proceed in lock-step -- is a well-known way to ensure failure.
> One only needs to look at OSI in general, and X.400 in
> particular. (And by way of anticipating one of the rat-holes,
> I'll note that X.400 achieved the rare success of having too
> much be integrated all at once, but still lack core bits of
> functionality.)

And I would suggest that the failure in X.400 was that they lost 
track of the problem analysis (which was actually pretty good, 
IMO, in the early stages of that work) in the process of trying 
to accommodate too many different views and "solution" ideas. 
They also left two or three important issues out of that problem 
analysis: a reasonable migration path from earlier systems 
(including, later on, a reasonable migration path from earlier 
versions of X.400) and general usability (aggravated by the fact 
that there were already deployed systems that were easier to 
use).  A possible additional important issue was that we already 
had deployed and successful experience with more or less peer to 
peer email systems, while the analysis that led to X.400 assumed 
a highly-structured "email provider" environment.  I imagine 
that the missing "core bits of functionality" to which you refer 
fall into one of those areas, but might be additional ones in 
which the problem analysis failed to be adequate (and tracked 
into the final results).

By contrast, regardless of the development method, X.400 turned 
out to be fairly modular and composed of separable pieces, as 
evidenced by the number of times people have successfully used 
some pieces of the system and not others.   It is arguably 
better in that regard than SMTP, which assumes --in its handling 
of Received fields if nothing else-- some semblance of 822 
header/body structure in the messages that are being transported.

> The usual logic for explaining this problem is that the
> complexity of juggling all the issues, across the entire
> service, simply buries the development effort. At best, it
> ensures that the work is produced much, much later, often
> after the market has found an alternate solution. (And that
> is, most certainly, what happened to OSI. I can elaborate on
> this is some detail, if anyone really challenges this point.)

No disagreement on this.   But it is possible to do modular 
development against a systems-level problem understanding.  The 
only additional requirement that introduces is that the 
little-piece development proposals must be examined against a 
"does this foul up anything else in the system" criterion.  And 
that is, IMO, one which we have too often managed to bypass.

> Breaking down a complex problem into smaller pieces that are
> individually useful does two things that are quite good:
>
> 1) It permits each piece of work to be useful, even if some
> other piece of work has a persistent problem;

Yes, as long as those individual pieces don't prevent each other 
from working, or overconstrain parts of the problem for which 
pieces are not yet developed.

> and 2) it
> permits turning the crank on the specification engine more
> quickly and more frequently. This means that we get
> operational experience more quickly and can then, quickly,
> refine the specifications to match actual field knowledge.
> With large, integrated efforts, the inter-dependencies mean
> that the whole of the work is not useful until the most
> problematic part is completed.

Again, up to a point.  I would have said "until the boundaries 
of the most problematic parts are understood sufficiently to be 
reasonably confident that they really are isolated".   To draw 
from another recent thread, a _solution_ to the problematic 
parts is not necessary, but some understanding of them and there 
implications often is.

> And it means that it is quite a
> few years before there is any feedback from the user
> community;. If there was a problem with the major design
> decisions that were made, they become essentially impossible
> to change, because things took so long to delivery.

I think one can argue that point into a "lose either way" 
situation, so it is important to strike an appropriate balance. 
>From the other perspective, one could claim that the "smaller 
pieces" approach --without adequate problem analysis-- can lead 
to quick and effective feedback from the user community about 
those specific pieces, but that their deployment may then make 
it essentially impossible to solve the more difficult problems 
by constraining possible solutions to them down to a null set. 
That is reasonable if the more difficult problems are also the 
less important ones, but there are no guarantees that is the 
case.  Sometimes, the most difficult problems are also the most 
important.

>...
> Is IP a failure?  Besides independent development of TCP and
> UDP, from the IP core, ICMP has been able to proceed
> independently, as has different address-mapping efforts and,
> for that matter, address interpretation efforts.

But IP, and the post-split TCP/IP, started from, I think, a 
fairly comprehensive understanding of what problems people were 
trying to solve.   Your example, along with the email 
transfer/content split, has more to do with effective 
modularization of the solution(s) than it does with a particular 
style of development.  Indeed, if one goes back and examines the 
history of the transfer/content split in email, a case can be 
made that, as with IP, the two started out fairly integrated and 
were split up after the advantages of that became clear.   To 
the extent to which that is true, neither one is a good example 
of _design_ or _development_ on a "small pieces" basis: rather, 
they are examples of redefinition of the problem and 
remodularization of the solution after the initial design and 
development were complete.

>...
> (One could go down a rathole about nonconformance, noting how
> badly we suffer from excessive variance in email system
> behavior, but I claim that is due to lack of enforcement,
> rather than due to any architectural issues. Given the wide
> range of support for HTML, and the like, this might be an
> applications-level issue. But it is not a big-vs-small design
> effort issue.)

Agreed.

> In fact, this highlights one of the other, surprising benefits
> of a divide-and-conquer approach, namely that it permits
> post-hoc additions that were not planned.  When the original
> philosophy is to do as little as possible to be useful,
> knowing that more bits of related work will be done, then it
> often is easier to do some of those bits many years later.

Again, I suggest that the advantages you are claiming here --and 
with which I agree-- are due to effective modularization, rather 
than a particular development approach (such as "divide and 
conquer").  And I believe we are more likely to get the 
modularization right if we have a moderately complete 
understanding of the problem --a systems analysis of properties, 
relationships, and interactions-- rather than if we start 
carving off pieces of the problem piecemeal on the assumption 
that we can always fit things back together later.

regards,
    john

---

From: Dave Crocker <[email-masked]>
Date: Fri, 4 Jul 2003 15:20:47 +0200

John,

JCK> I think one needs to distinguish between "adequate understanding
JCK> of the problem" and "overly broad attempts at a solution".

Yes, thank you.  A very useful distinction.

Alas, *each one* needs to be balanced between taking too long and being
too hasty, I think.  (And then, of course, they need to balanced with
each other... it's amazing we ever get it right.)  I believe we are
currently having problems with each of these.

JCK> when we take a complex _problem_ and say "well, we understand
JCK> the lower-right-corner of it, so let's 'solve' that" often leads 
JCK> us into a solution that doesn't integrate well with the other 
JCK> pieces when those come along.

Certainly there is the danger you cite, and certainly there are plenty
of examples of this sin being committed around the world -- very much
including within IETF.

If one takes a view that is too narrow, or too localized, then they are
likely (or certain) to commit this sin.

One could argue that the real, underlying magic of the IETF's success
has been the community's ability to hold the larger "problem
understanding" that is needed, without spending excessive time
developing it. The same has frequently been true for design of the
larger solution space. This probably has been possible because there has
been so much shared culture, so many skilled architects (including skill
with scaling), and a relatively small domain of discourse.

In terms of community proportions, I suspect none of these hold true
anymore. (I said proportions, not actual numbers.)  Hence, we cannot
rely on assumptions and shared culture.  We need to institutionalize
mechanisms that achieve similar results.  (yikes.)

JCK> Clearly, it is important to find a balance, since the quest for
JCK> comprehensive and perfect understanding of every problem would 
JCK> result in our never actually getting anything done

ack.

JCK> (if one were cynical, one might make that observation about this WG
JCK> as well as about the IETF's technical/engineering work).

or, worse, maybe *not* cynical...

JCK>   But "this 
JCK> piece will work, so let's do it" needs, IMO, to be carefully 
JCK> balanced with "and _what_ problem is it solving?".  And we need 
JCK> to have clear explanations and a reasonable consensus about the 
JCK> latter.   I suggest that failure to take that step has gotten us 
JCK> into a lot of trouble in some areas in the past and that may be 
JCK> where my views and Keith's intersect.

and even mine.

(*** Warning.  Folks --

   X.400 rathole section follows.

   Suggestion to the X.400/email technology-challenge: attend to the
   structure and process debate, without worrying whether either John or
   I are precisely correct about the facts. That is, treat it as a bench
   analysis of approaches, rather than an historical review...
***)

>> particular.
JCK> And I would suggest that the failure in X.400 was that they lost
JCK> track of the problem analysis (which was actually pretty good, 
JCK> IMO, in the early stages of that work)

not sure i agree on the assessment of the early stages. my recollection
is that X.400 tried, rather explicitly, to solve a whole array of human
messaging problems for which the constituency was largely theoretical,
and for which the solution(s) were very poorly understood.

JCK> They also left two or three important issues out of that problem 
JCK> analysis: a reasonable migration path from earlier systems

That was explicit. The world was explicitly expected simply to switch
over.

And therein lies and important lesson about installed base.  (How else
could anything as ugly as MIME prove so successful?)

JCK> (including, later on, a reasonable migration path from earlier 
JCK> versions of X.400)

Or, one could argue, that was the problem of making too darn many
changes to existing functionality, and issuing them before vendors could
recover their investment.

And let's not forget just how complicated the specs were/are.

JCK> and general usability (aggravated by the fact 
JCK> that there were already deployed systems that were easier to 
JCK> use).

Another lesson, methinks.

JCK> A possible additional important issue was that we already 
JCK> had deployed and successful experience with more or less peer to 
JCK> peer email systems, while the analysis that led to X.400 assumed 
JCK> a highly-structured "email provider" environment.

A rather striking point about this is that the core of the X.400 work
was done by folks with experience doing Arpanet stuff, though maybe not
much with email.  Hmmmm.

>> service, simply buries the development effort.
JCK> No disagreement on this.   But it is possible to do modular
JCK> development against a systems-level problem understanding.  The 
JCK> only additional requirement that introduces is that the 
JCK> little-piece development proposals must be examined against a 
JCK> "does this foul up anything else in the system" criterion.  And 
JCK> that is, IMO, one which we have too often managed to bypass.

and I agree with that, certainly.

we can, of course, see the great, big, looming trap that this permits us
to fall into, however. If we are not extremely careful.

And one could argue that we have fallen into it quite a few times in
recent years.

>> refine the specifications to match actual field knowledge.
JCK> Again, up to a point.  I would have said "until the boundaries
JCK> of the most problematic parts are understood sufficiently to be 
JCK> reasonably confident that they really are isolated".

My point was that enforcing dependencies between working groups ensures
that none of them get their work done until the slowest is done.

I believe your point pertained to the abstraction of what is needed to
ensure that divide and conquer can proceed *without* the project
management (ie, timing) dependencies.

>> community;.
JCK> I think one can argue that point into a "lose either way"
JCK> situation, so it is important to strike an appropriate balance.

yup.

JCK> From the other perspective, one could claim that the "smaller
JCK> pieces" approach --without adequate problem analysis-- can lead 
JCK> to quick and effective feedback from the user community about 
JCK> those specific pieces, but that their deployment may then make 
JCK> it essentially impossible to solve the more difficult problems 
JCK> by constraining possible solutions to them down to a null set.

the most essential bit of feedback is whether an approach (and its
specification) are useful.  If they are, the feedback means there is an
installed base, and it must be respected henceforth.  If the feedback
says 'not useful', the we get to start over.

Most of the time for any interesting problem, this carries a very real
requirement, , that we must deploy components of a larger solution prior
to fully understanding whether the divide-and-conquer component-ization
has been done well enough.

I believe it is a) the fear of getting this wrong, and/or b) actually
getting it wrong, that cause the massive delays we experience for these
complex problems.

>> Is IP a failure?
JCK> But IP, and the post-split TCP/IP, started from, I think, a 
JCK> fairly comprehensive understanding of what problems people were 
JCK> trying to solve.   Your example, along with the email 
JCK> transfer/content split, has more to do with effective 
JCK> modularization of the solution(s) than it does with a particular 
JCK> style of development.

With TCP/IP, perhaps.  With email, I believe not.

Frankly I would say that email was extremely accidental, in its
architecture. Yes, the FTP mail commands were planned from the start,
but standardizing the headers was done 5 years later, a separate email
transfer protocol done 5 years after that, and the effort to make the
body support multi-media content was done yet another 10 years later.

Email addressing was no better. Ray Tomlinson just stuck the @host onto
an existing structure. The architectural "brilliance" that came from
such incremental growth was that the rest of the net was not allowed to
know anything about the left-hand side (the mailbox) but only the domain
of control (the host). (And let me explain that I put the quotation
marks around brilliance not to diminish the importance, but to highlight
that this kind of excellent work is marked by doing less, not more.)

This left an extremely powerful back door for addressing extensions. By
contrast, X.400 suffered very seriously because it insisted that there
be global semantics to the whole string. (And -- surprise! -- we get
into trouble, today, when relays commit the sin of thinking they know
something about the syntax or semantics of the left-hand side.)

I haven't asked Ray about his approach to addressing, and believe he was
simply doing the minimum necessary to get the thing to work. I do not
believe he had any sort of grand, global scheme for the entire address.
My recollection is that the work on RFC733, in 1977, also did not get
more "sophisticated" with respect to the overall addressing model,
although we did add a feature for source-routing that proved useless.
And when RFC822 added domain name structure to the right-hand side, it
was again a small modification to a localized part of the architecture.

Clearly, MIME worked the same way.

So, none of this came from grand planning, at any point along the way,
in my opinion.

It did, however, benefit from some type of consistency in architectural
thinking at each incremental effort. Maybe. But I am hard-pressed to
claim it was even conscious or "coordinated".

JCK> Indeed, if one goes back and examines the 
JCK> history of the transfer/content split in email, a case can be 
JCK> made that, as with IP, the two started out fairly integrated and 
JCK> were split up after the advantages of that became clear.

I completely disagree.

The entire process was highly fragmented and asynchronous.

JCK> To the extent to which that is true, neither one is a good example
JCK> of _design_ or _development_ on a "small pieces" basis: rather,
JCK> they are examples of redefinition of the problem and
JCK> remodularization of the solution after the initial design and
JCK> development were complete.

Nope.  For email, definitely disagree.

>> post-hoc additions that were not planned.
JCK> Again, I suggest that the advantages you are claiming here --and
JCK> with which I agree-- are due to effective modularization, rather 
JCK> than a particular development approach (such as "divide and 
JCK> conquer").

I think that "effective modularization" and "divide and conquer" are the
same thing...

Hmmmm. I suspect I am using the term divide and conquer more loosely
than you. (That got me in trouble on my Ph.D. qualifying exam, and guess
what degree I did not get...)

JCK> And I believe we are more likely to get the 
JCK> modularization right if we have a moderately complete 
JCK> understanding of the problem --a systems analysis of properties, 
JCK> relationships, and interactions-- rather than if we start 
JCK> carving off pieces of the problem piecemeal on the assumption 
JCK> that we can always fit things back together later.

The bottom line is that I believe there needs to be good consideration
of the larger issues, too.

The question comes back to those matters of 'balance' and 'skill'.

So the real question is what can we do with these larger problem spaces
to facilitate small, rapid development of independently-useful
components that will fit into a larger scheme and support a reasonable
degree of system scaling?

(When we answer that, there are two activities we should pursue next.
One involves the purchase of a certain bridge into New York City and the
other involves world peace.)

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>