Subject: problem statement: Engineering practice vs. management practices

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Sep 2003 11:38:43 -0400

> 2.2 The IETF does not Consistently use Effective Engineering Practices
> 
>    For an organization with 'engineering' in its title and participants
>    who are likely to trot out the statement "Trust me, I'm an engineer!"
>    when confronted with the need to find a solution to a particularly
>    knotty problem, the IETF has, at least in some cases, extremely
>    ineffective Engineering Practices.

> 2.7 Working Group Practices can make Issue Closure Difficult
> 

I have a hard time understanding the difference between 2.2 and 2.7
and parts of 2.6. I _think_ section 2.2 is trying to say something
about how we don't actually use more rigorous and formal internal
processes that would help us avoid the problem of documents taking too
long, etc. But one could argue that a lot of that is related to basic
project management issues (or from Section 2.7 "Working Group
Practices can make Issue Closure Difficult"), and its not always clear
whether a cited issue is the lack of a tool (or lack of its use) or
just lack of use of basic project management techniques, or just bad
WG practice, for which there might be several ways to try and improve
the situation.

Meta issue: Is the "root cause" issue the lack of use of
management/engineering techniques, or would using better techniques be
a _solution_ to existing poor practices (or to _some_ poor practices,
but not others)? This document seems to not make this distinction very
well, because it lists one underlying poor practice (e.g., section
2.7), but calls out a number of others in section 2.2. where they
aren't listed as poor practices, but as consequences of not using
appropriate tools.

Regarding Section 2.7:

>    2.7   Working Group Practices can make Issue Closure Difficult

IMO, Section 2.7 is a bit too narrowly scoped, and there are more
examples that should be added here.  Existing WG practices are also a
root cause issue. In that case, though, the section heading would be
better made more general, e.g.:

>    2.7   Working Group Practices Contribute to Delays

Some additional examples for section 2.7:

 - too many WGs have a adopated a cycle where the bulk of the activity
   is centered around the face-to-face meetings. E.g., Many IDs are
   revised only once per cycle, immediately before a meeting, with too
   little active work going on in the months immediately after a
   meeting.

 - despite IETF mantra that the WG is the mailing list, mailing lists
   are become increasingly ineffective at resolving issues, e.g., too
   many postings for many to keep up with, too many off-topic posts
   (i.e, those not directly related to getting wording changed in a WG
   document), folk being driven away because small number of vocal
   posters, etc.

 - Document authors are often not held accountable enough for
   revising documents in a timely fashion

 - As a WG document is being developed, reviews of WG documents too
   often involve only interested WG members and don't include reviews
   from those outside of the WG, or don't include reviews from subject
   matter experts of topics that are related to the topic in the WG
   document, but are not necessarily the focus of the WG document.
   But when the WG says its done, and those outside reviews come in,
   they are frequently viewed as being "too late" and the cost of
   making changes is "too high". Frustration then results.

If section 2.2 is really aimed at non-management stuff, like tools, I
think it would be useful make 2.2 state that clearly. Or if it is
intended to cover both tools and management techniques, make that more
clear (especially in the opening paragraph). I think it would also be
good to rename/expand section 2.7 to cover more existing bad
practices.

Thomas

---

From: Keith Moore <[email-masked]>
Date: Wed, 17 Sep 2003 13:14:49 -0400

> >    For an organization with 'engineering' in its title and
> >    participants who are likely to trot out the statement "Trust me,
> >    I'm an engineer!" when confronted with the need to find a
> >    solution to a particularly knotty problem, the IETF has, at least
> >    in some cases, extremely ineffective Engineering Practices.
> 
> 

On rereading these parts of the document I've come to the conclusion
that most of 2.2 is really about management practices rather than
engineering practices.  Nevertheless I strongly agree that IETF does not
consistently use effective engineering practices.  (Maybe the problem is
that many people in IETF are not familiar with the discipline of
engineering or do not know how to apply it to protocol design?)

So I think section 2.2 needs to be rewritten a bit, and some of the 
material in 2.2 needs to be moved to 2.7.  In other words we need
to make sure we understand (and the document is clear about) the 
difference between engineering of technical solution and management
of groups that are chartered to produce technical solutions.  
I think it's worth doing because we really need to promote a better
understanding of the discipline of engineering within IETF.  

IMHO, these are about engineering, though to some degree they're stated
in the language of management more than the language of engineering:

>    o  Failure to ensure that there is a uniform view in the WG of the
>       scope of the WG activity, especially the intended purpose of the
>       solution.   

i.e. failure to define the problem to be solved
> 
>    o  Failure to identify at an early stage (before the design is
>       frozen), and/or then to ensure that there is a uniform view in
>       the WG of the issues that need to be resolved to bring the work
>       to a satisfactory conclusion.
> 
>    o  Failure to identify and articulate engineering trade-offs that
>       may be needed to meet the deadlines that the WG has set without
>       inappropriately reducing the 'fitness for purpose' for the
>       intended customers. 

This is another example where the language is expressed in management
rather than engineering terms.  From an engineering perspective it's
appropriate to think about time-to-specify as one of the many costs of
an engineering solution which influence the compromise that must be
made. IMHO it is rarely appropriate to treat a "deadline" as a concern
that overrrides other costs.  And the cost to the net of living with a
poor solution for a long time should also be considered.

>    o  Continued refinement of the solution beyond the point at which
>       it is adequate to meet the requirements placed on it by the
>       intended purpose.

These are mostly management issues rather than engineering issues:

>    o  The charter may not be sufficiently detailed to document the
>       process and timeline to be followed by the WG.  Additional
>       documents may be needed such as a roadmap or detailed plans.

Yes, WGs need to develop their own roadmaps and plans for the work
they expect to do.

>    o  Poorly defined success criteria for WGs and individual
>       documents.

This is a management issue.  Success criteria for the technical
solutions would be an engineering issue.

>    o  Lack of written guidelines or templates for the content of
>       documents (as opposed to the overall layout) and matching lists
>       of review criteria designed to achieve appropriate quality in
>       output.
> 
>    o  Lack of auditing against explicit criteria throughout the
>       standards development process.

Auditing against technical solution criteria would be an engineering
issue, auditing against other criteria (like document quality) would be
a management issue.

>    o  Lack of metrics to measure the achievement of the desired
>       quality and the performance of WGs and the wider IETF..

Again, measurement of WG performance is a management issue;
measurement of technical solutions is an engineering issue.

>    o  Lack of an explicit feed forward path to drive the improvement
>       of the standards development process and success criteria over
>       time and to avoid repetition of failures that may occur.

Anything having to do with the process is not an engineering issue

>    o  Absence of documented debriefing sessions to assess and record
>       the  successes and failures of WGs (and other processes) so that
>       the positive and negative lessons learned can be used to
>       facilitate future work and improve the processes.

see above.

>    o  Lack of criteria for determining when a piece of work is
>       overrunning and/or is unlikely to be concluded successfully
>       either at all or within an acceptable time frame: Lack of
>       process for either extending the time frame, adjusting the scope
>       or terminating the work item or the whole Working Group.

This conflates process/management and engineering issues.  If a piece of
work doesn't satisfy its engineering criteria (including time-to-produce
criteria) this is an engineering issue.  And it would be appropriate to
make those criteria explicit.  But the lack of process for adjusting
time frame or scope of a group's activities is a process or management
issue.

>    o  Tools to support the engineering process are minimal.

This is very vague - it's hard to know what tools are meant here. 

>    o  Despite its commitment to 'running code', the IETF is not
>       proactive in providing ways for developers to verify their
>       implementations of IETF standards.

FWIW, we've never relied on verification, and this was an explicit
decision. We've placed high faith in interoperability testing, which is
not the same thing.  Of course, we don't do much to facilitate
interoperability testing either.  But we should probably amend this
bullet a bit to say something like "...for developers to test the
interoperability of their implementations."

>    little active work going on in the months immediately after a

Saying that WGs have "adopted" this cycle may be giving WG management
more 'credit' than it deserves.   F2F meetings with their imposed I-D
deadlines are naturally going to encourage a tendency toward periodic
releases of drafts, and this is often a good thing.   And for early
phases of operation of most WGs, one draft per cycle can be quite
reasonable.  But when the document is nearing completion it becomes
important to release drafts more quickly, to avoid having to wait many
months to work out small wording changes.  Even more important 
than revising drafts is to identify the devisive issues prior to the f2f
meetings so that there is a chance that they can be worked out in
person.  Sometimes a chair would do better to concentrate on getting
the issues on the table (and using the f2f meeting to discuss those
issues) rather than by merely insisting that the document
be revised before the f2f meeting deadline.

>    document)

These are not equivalent.  maybe you meant e.g. rather than i.e. ?

>   , folk being driven away because small number of vocal posters, etc.

I agree that this is an issue, but it needs more refinement.  There's
an important difference between a list where a small number of people
are exchanging a lot of messages but contributing useful insights while
doing  so, and a list where the level of traffic is similar but the
participants are not making much progress.  (either one might drive some
people away through mere volume, but that's not entirely a bad thing.  a
certain level of discussion is necessary to solve any thorny problem,
and artifically slowing down the discussion to reduce the bandwidth
level is not necessarily helpful.)

Part of this problem may be that we're dealing with more difficult 
issues, and there is more often a need to compromise between very 
disparate conflicting interests, than there was in the past.
A lot of people do get turned off by divisive arguments, but that
doesn't mean that those arguments don't need to take place. 
It might help if the group as a whole could gain a better idea of the
needs of the various interests (both inside and outside of the group)
early in its process - and maybe if these interests were documented 
participants might be able to argue more effectively (less heat and more
light) by having a better idea of their adversaries' points-of-view.  

Keith

---

From: "todd glassey" <[email-masked]>
Date: Wed, 17 Sep 2003 18:10:47 -0700

Thomas
----- Original Message -----
From: "Thomas Narten" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, September 17, 2003 8:38 AM
Subject: problem statement: Engineering practice vs. management practices

> I have a hard time understanding the difference between 2.2 and 2.7

-- SNIP--

ss 2.2. is about a uniform Vetting Model for creating a Standard, and  2.7
is about the management of the WG's; They address related but separate
issues.

BTW - The real problem with this issue is not that the IETF does not use
effective engineering practices, its that the IETF refuses to use uniform
ones so that everyone gets the same bite at the apple.

Todd