Subject: Sampling

From: Dave Crocker <[email-masked]>
Date: Sun, 27 Jul 2003 12:22:03 -0700

Folks,

The latter part of a posting by John cites statistic issues.  I'd like
to explore that issue separately and carefully:

JCK> I think it is interesting, and _very_ important, to try to
JCK> understand why the impression of the sentiment of the Problem WG 
JCK> and the response from the Plenary were so different.

JCK> (1) Regardless of the _authority_ of the plenary, significantly
JCK> more IETF participants were sitting there, raising hands, and 
JCK> generally expressing opinions than have been participating 
JCK> actively and observably in the Problem WG.  Unless there is 
JCK> evidence that the plenary was somehow "packed" --and I saw 

I believe that both the working group participation and the IETF Vienna
plenary presentation represent very, very highly biased samples of the
IETF population.

Sampling bias does not require intentional "packing". It requires
selection methods that make the group unrepresentative of the total
population.

In this case, we had a smaller-than-expected total IETF meeting
attendance, in a venue that prevented significant numbers of regular
participants from attending. And not all of the meeting attendees were
Plenary attendees.

Notably, we have not paid much enough attention to just how small the
participation in the Problems working group has been. And the recent
Plenary had all sorts of confusion to this discussion. Overall, we need
to be very circumspect in making decisions based on either bits of
input.

So it is not in the least surprising that the two, differently biased
samples would produce different results.

JCK> This has created a statistical bias between the composition and
JCK> opinions of the self-selected WG and that of the community.

yes, but...

JCK> In other words, the WG has become an island, having discussions and
JCK> reaching conclusions that are out of touch with the IETF universe.

In all likelihood, so was the Plenary.  At the least, the discussion was
confused and confusing.

JCK> 	Sometimes, because of its composition, a
...

Overall, a nicely assessed and written statement, I think.

However the call for Last Call review can (and does) lead to
re-synchronization far too late, I believe. We need to make sure that
there are ways of doing synchronization with the community early and
often.

Even waiting for a Plenary presentation might be too infrequently.
Perhaps we need tight, summary "presentations" on the Announce-list,
occasionally.  We might think of these as "phase" Last Calls, or
somesuch.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 28 Jul 2003 09:22:04 -0700

> plenary presentation represent very, very highly biased samples of the

Yes. In both cases, they consist of the people that show up.

> to be very circumspect in making decisions based on either bits of

Actually I was pleasantly surprised that about a quarter of the plenary 
actually claimed to have read the problem drafts. This is significantly 
higher than many working groups, and indicates that the part of the 
community that shows up at plenaries outside the US *does* care about the 
issues, enough to do prep work.

We trust WG meetings with less prepwork made with decisions, too.

> occasionally.  We might think of these as "phase" Last Calls, or

actually the participants on ietf-announce (about 2500 of them) are a 
*third* statistically skewed sample of our participants - and the probably 
smaller number who actually read and think about responding to Last Calls 
(of any sort) is even more statistically skewed.

I think of this as a problem, in general.....

                       Harald

---

From: Randy Bush <[email-masked]>
Date: Mon, 28 Jul 2003 09:33:19 -0700

> Yes. In both cases, they consist of the people that show up.

as was yokohama.  as was ...  as will be ...

all samples are biased.  we just don't like the ones that have
results which disagree with our own perceptions.  funny monkeys
we are.

randy

---

From: Dave Crocker <[email-masked]>
Date: Wed, 30 Jul 2003 15:00:20 -0700

Folks,

>> plenary presentation represent very, very highly biased samples of the

HTA> Yes. In both cases, they consist of the people that show up.
and
RB> all samples are biased.  we just don't like the ones that have
RB> results which disagree with our own perceptions.  funny monkeys
RB> we are.

Two points from this brief thread:

1. Two members of the IESG are clear that we need not be concerned with
the basis on which we interpret a meeting's results.

Literally the only thing that matters is who shows up.

In social research methodology discussions about the topic of sampling
bias, there is a careful distinction between "population" and "sample".
There are some useful decades of experience with this distinction.  The
same useful history concerns biasing aspects of the ways questions are
asked.

From the two postings on this thread, it appears that we are simply to
treat sample and population as the same, and we need not worry about the
particular form of questions to the group.

Originally, the IETF worried quite a bit about being inclusive, in its
formulation of meeting logistics, its assessment and use of consensus
statements at meetings, and its general attention to the preferences of
the general community. All of this is difficult and unpleasant.

I have always felt that that careful attention to real inclusiveness was
a primary source of legitimacy for the IETF. It was in this context, of
bending over backwards to be inclusive, that the focus on those who
showed up and participated rang true.

Apparently things have changed. It appears that we no longer have to
worry whether meeting logistics serve to exclude people or whether IETF
process serves to disenfranchise folks. We just tally who shows up -- or
rather, who speaks up -- and that's that.

I'm guessing that some folks might disagree with my assessment.
Clarifications and corrections would be greatly appreciated.

2. No one else has contributed to this thread.

Hence they have not "shown up". By definition this means that the two
IESG members are correct and that I am wrong about my concerns.

Luckily, we do not need to worry whether folks have, in fact, given up
on this forum or this topic. They haven't contributed, so their views
are not relevant.

d/

---

From: Randy Bush <[email-masked]>
Date: Wed, 30 Jul 2003 15:15:06 -0700

> the basis on which we interpret a meeting's results.

no.  two members are clear that we have no good way to interpret the
results other than taking them on face value, as opposed to your
appproach of discounting that and those with whom you disagree.

randy

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 Jul 2003 19:41:37 -0400

Dave,

I don't want to put words into your mouth, or that of any of the 
IESG, so I'm going to try to go up a half-level of abstraction...

(1) It had better be possible to talk about, and question, the 
quality of a claimed consensus.  "Quality" must be able to 
include bias* -- in the narrow statistical sense, not, in 
itself, implying that anything evil is happening -- of whatever 
convenience samples* happen to have been surveyed relative to 
the whole population of participants in some WG, the (usually 
larger) population of people interested in the WG but not 
participating for some reason, the (usually larger) population 
of IETF participants, and even the (larger yet) population of 
(to use ANSI's term) "materially concerned" Internet users and 
suppliers.

If we can't do that, then we have absolutely no protection 
against some individual or group generating so much noise (or 
hostility, or other unpleasantnesses) that they drive everyone 
who disagrees with them away and then claim consensus for their 
position.  Our protection against that sort of attack is, 
primarily, the judgment of WG Chairs, Area Directors, and the 
whole IESG, backed up, if necessary, by appeals processes to 
force additional scrutiny on what occurred.  Those judgments 
are, ultimately, extremely subjective, which is too bad, but 
there is no other way.  In particular, we will not get any 
protection from discussions of sampling theory, since we don't 
have good models or descriptions of either the convenience* or 
self-selection* samples involved or of the underlying presumed 
populations.

In addition, our oft-repeated claim that we rely more on mailing 
list discussions and conclusions than on the group of people who 
happen to show up and a meeting and talk or wave their arms 
becomes completely bogus if we can't evaluate the quality of a 
claimed consensus.  I hope we are not headed in that direction.

(2) Why has there been little or no response other than from a 
few IESG members?    I suggest that every participant in the 
IETF, including the IESG and those few for whom IETF 
participation is a full-time job, must regularly, perhaps 
continually, make decisions about what is important and what 
isn't.   The criteria for importance will differ from one or 
another of us, but might rationally include

	- "will my putting energy into this make any difference
	at the end of the day" and/or
	
	- "will this effort make any real difference when
	examined from the perspective of a few years from now",
	and/or
	
	- "is my following this and contributing to it likely to
	be of enough value to overcome the irritation costs and
	increased blood pressure it will probably involve",

as well as the traditional opportunity cost question of

	"if I don't spend time on this, what else could I be
	doing that would be more useful or productive?".

In my own case, my original use of the term "bias" was, I 
believe, statistically correct.  It didn't take very many 
transactions on this thread for me to conclude that, given the 
personal criteria I'm applying these days, getting drawn into a 
discussion about sampling terminology and theory and its 
relationship to IETF meetings was not an investment I felt like 
making.  So I have been sampling* the thread**, but not 
responding.  I changed that policy/ strategy (and changed the 
subject line) when the thread turned into what appear to be a 
set of assertions about agreements/ conclusions/ consensus that 
do and do not exist based on other responses and where they come 
from.

(3) Once a WG, or set of discussion threads, starts being taken 
over by a relatively small number of passionate people who are 
generating a lot of often-repetitive postings, we need to be, 
IMO, _extremely_ careful about how we interpret whatever polls 
and consensus calls we make and even more careful about traffic 
analysis.  It is very easy to confuse "consensus among those who 
haven't given up on the effort" and "consensus among the group 
who are concerned about the issues".  Those two groups are often 
different and the reasons for the differences may be very 
important.  They are also often different from "the group who 
happens to be gathered together in a given place" or "the group 
who showed up at a meeting (or plenary)" or "the group that is 
actively participating in a WG at a particular time". 
Analysis, whether by statistical or other means, of those 
differences may be quite instructive but, normally, only if it 
moves beyond repeated assertions of personal belief (or about 
beliefs or claims about what others mean by their silence).

(4) In the absence of analysis good enough to project from a 
subset (whether it is strictly a sample or not and whether that 
sample is biased or not) of some population to the opinions of 
that population, just about the only thing one can safely do is 
to accept the opinions (or measurements) as expressed, while 
being cautious about their interpretation based on an 
understanding of threats to validity*.

(5) Let me conclude with a personally-troubling example of how 
some of this works.  Melinda has sent out two messages to try to 
validate the meeting "hums".  That is a legitimate thing for a 
WG Chair to do in the process of trying to determine consensus. 
I've read both messages, thought about responding on the several 
issues in which my personal opinions are different from the 
consensus she has identified, and decided to not do so.  Why? 
Because in my personal opinion, this WG has been extremely 
helpful in discussing and focusing on the issues, but has now 
passed sufficiently far into the range of diminishing returns to 
have outlived its usefulness.   If I were the AD, I'd be trying 
to deliver a "wrap this up before I shut it down; it shouldn't 
drag out much longer" message.

I've said that before in different ways, including at the 
plenary where, independent of how one interprets it, my "lets 
get on with it and do so without setting up more apparatus" 
comments appeared to get more support than most of the specifics 
from this WG or the process of which it is a part.  A corollary 
of that personal conclusion of mine is that it would be useful 
to publish the issues list, as it exists today, as a snapshot of 
some very serious and thoughtful community analysis, but without 
either worrying a great deal about whether it represents 
consensus (and of what) or about getting every statement exactly 
right.  Were we to follow that path, the one thing that would be 
important to get right in the document would be a clear 
statement about what it does, and does not, represent.

So, just as I challenged Harald at the plenary about the 
questions he did (or didn't) ask, I should probably challenge 
Melissa to ask for input on who actually thinks the charge of 
this WG is worth more time and, if so, how much more and what we 
should use as a stopping rule.  The latter is especially 
important given the subject matter of the WG: it would be _lots_ 
better if we reached consensus to shut ourselves down than if we 
wait long enough for it to be obvious that the AD should do it.

Is it beneficial to the WG, or to the IETF, for me to repeat 
those conclusions three or four --or a few dozen-- more times? 
I doubt it.  I think I've been heard and that those who are 
going to agree have agreed already and that those who haven't 
won't change their minds if I repeat myself.   So I'm trying to 
move on, even if the WG is not.   What is the relationship 
between those conclusions and Melissa's "hum closure" calls? 
Well, I don't agree with several of her conclusions and, 
especially since there has mostly been silence, she is on thin 
ice if she concludes that everyone who hasn't expressed 
disagreement agrees with her.    But I don't disagree enough 
--relative to my assessment of importance and what else I could 
be doing with my time-- to want to stand up and say "I disagree, 
and here is my analysis and my reasons".  Bad investment of 
time: I personally believe that the document today is close 
enough for any real, viewed-from-five-years-hence purpose which 
it is likely to leverage and I think we should get on with it.

      john

Footnotes:

* The term "sample" and "sampling" have very precise meanings in 
statistics.  They involve systematic mechanisms for determining 
or describing the relationship between the sample and some 
population or universe that it is supposed to represent, so that 
inferences from the sample can be projected (or translated) into 
inferences about the population.  If it is not possible to 
describe either the population or the relationship of the 
"sampled" subpopulation to it, then "sample" belongs in quotes. 
The term "convenience sample" refers to a subpopulation that is 
selected, not by some formal or systematic means, but on the 
basis of, e.g., whomever happens to be handy or most easily 
accessed.  The term "self-selected sample" refers to what is 
usually considered a special sort of convenience sample in which 
the members of the selected subpopulation select themselves, 
often based on their particular positions on whatever is being 
measured.  Among the people who do sample design for a living, 
both "convenience sample" and "self-selected sample" are terms 
of abuse or, sometimes, derision.

"Bias", technically, refers to the difference, or lack thereof, 
between a sample (selected under some sampling model) and the 
relevant population.  If the population parameters, other than 
size, are different from the equivalent parameters of the 
presumed sample, then the sample is biased and --and this is the 
important issue-- one cannot safely make direct inferences from 
findings or conclusions about the sample to findings or 
inferences of the population.  Convenience samples can be 
unbiased, but that doesn't happen by accident and, in 
statistical sampling circles, the burden of proof is on whomever 
makes that claim and the burden is pretty high.  And, finally, 
"threat to validity" is another technical term, referring to the 
whole collection of things (hidden systematic bias is only one) 
that can make projection of an inference from a sample onto a 
population incorrect, even if "sample statistics", "confidence 
intervals", etc., indicate that things should be ok.

** Yes, I have a sampling model for the list that determines 
what I read and what I don't.  I don't feel like describing it, 
partially because my doing so might possibly introduce 
additional biases.

Finally, an observation on the footnotes:  By most reasonable 
definitions of the term, I'm a card-carrying statistician or at 
least a card-carrying data analyst (but I am not a sampling 
expert and have never had any desire to be one).  It has been a 
while, but I've taught this stuff and made a day-job living 
doing some of it.  And I find it more than mildly interesting 
that the other folks in the IETF whom I'm aware of having 
similar backgrounds have, unlike myself, been smart enough to 
not waste their time speaking up on this issue.

----------

--On Wednesday, 30 July, 2003 15:00 -0700 Dave Crocker 
<[email-masked]> wrote:

> Folks,
>
>>> I believe that both the working group participation and the
>>> IETF Vienna plenary presentation represent very, very highly
>>> biased samples of the IETF population.
>
> HTA> Yes. In both cases, they consist of the people that show
> up. and
> RB> all samples are biased.  we just don't like the ones that
> have RB> results which disagree with our own perceptions.
> funny monkeys RB> we are.
>
> 1. Two members of the IESG are clear that we need not be
> concerned with the basis on which we interpret a meeting's
> results.
>
> In social research methodology discussions about the topic of
> sampling bias, there is a careful distinction between
> "population" and "sample". There are some useful decades of
> experience with this distinction.  The same useful history
> concerns biasing aspects of the ways questions are asked.
>
> From the two postings on this thread, it appears that we are
> simply to treat sample and population as the same, and we need
> not worry about the particular form of questions to the group.
>
> Originally, the IETF worried quite a bit about being
> inclusive, in its formulation of meeting logistics, its
> assessment and use of consensus statements at meetings, and
> its general attention to the preferences of the general
> community. All of this is difficult and unpleasant.
>
> I have always felt that that careful attention to real
> inclusiveness was a primary source of legitimacy for the IETF.
> It was in this context, of bending over backwards to be
> inclusive, that the focus on those who showed up and
> participated rang true.
>
> Apparently things have changed. It appears that we no longer
> have to worry whether meeting logistics serve to exclude
> people or whether IETF process serves to disenfranchise folks.
> We just tally who shows up -- or rather, who speaks up -- and
> that's that.
>
> Hence they have not "shown up". By definition this means that
> the two IESG members are correct and that I am wrong about my
> concerns.
>
> Luckily, we do not need to worry whether folks have, in fact,
> given up on this forum or this topic. They haven't
> contributed, so their views are not relevant.
>
> d/
> --

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 30 Jul 2003 20:10:21 -0500

Dear John,

I wanted to thank you particularly for two points in your note:

- I have been telling myself since January that the problem statement
  editorial team needed to put together the best problem statement
  we could *without* trying to figure out whether the problems
  being reported to us were "real", but I wasn't sure that we
  were doing the right thing until you explained why we were
  doing the right thing in your note. Thanks!

- When the thread title was "sampling" - well, I have some
  formal training in that direction, and I was troubled because
  normally "population" is better-defined than it is in the IETF - 
  Dave was asking about bias between a sample and a population
  when I didn't have a good definition of the "population" (since
  we have no formal membership, etc. - it's almost like our
  population is a sample of a larger population). Your reference
  to those with a "material interest" was helpful to me in thinking 
  about the difference between consensus and voting. Again,
  thanks!

I'm glad we've done the problem statement work that we've
done so far, and I've noted a couple of times that we've only
added one (IIRC) "root problem" since IETF 56, so I agree
that we're on the diminshing returns part of the power curve.
Maybe part of assuming that we're all adults is letting people
read the problem statement and each figure out how to respond
as individuals - there are opportunities for structural change,
but there are an awful lot of opportunities that one or two
people can start a trend to take advantage of.

Spencer

---

From: Dave Crocker <[email-masked]>
Date: Wed, 30 Jul 2003 18:25:41 -0700

Spencer,

SD>   Dave was asking about bias between a sample and a population
SD>   when I didn't have a good definition of the "population" (since

Exactly.

It is the fluidity and informality of the *real* population's boundary
that should make us particularly diligent about whether we are being
adequately inclusive and whether our samples are adequately
representative.

We used to do that.

Now, apparently, we just say "we have no good way to interpret the
results other than taking them on face value".

SD>   we have no formal membership, etc. - it's almost like our
SD>   population is a sample of a larger population).

yup!

d/

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 30 Jul 2003 19:58:36 -0700

Dave,

> Literally the only thing that matters is who shows up.

I never said anything resembling this.

The keywords of what I said in my message of Mon, 28 Jul 2003 09:22:04 

> I think of this as a problem, in general.....

I do not see any logical connection between this message and "we need not 
be concerned with the basis on which we interpret a meeting's results".

                       Harald

---

From: Dave Crocker <[email-masked]>
Date: Wed, 30 Jul 2003 21:43:28 -0700

Harald,

>> Literally the only thing that matters is who shows up.

HTA> I never said anything resembling this.

Let's see:

I post a note calling for careful exploration of an issue and include
the statement:

DC> Notably, we have not paid much enough attention to just how small the
DC> participation in the Problems working group has been. And the recent
DC> Plenary had all sorts of confusion to this discussion. Overall, we need
DC> to be very circumspect in making decisions based on either bits of
DC> input.

You post a note saying we have the people who show up, and Randy
posts a note saying all samples are biased.

Neither of you offers any additional, substantive response to my query,
nor any indication of interest in pursuing it further.

No doubt there is a way of reading your responses as showing that
concern I failed find, and as providing useful insight to the topic, so
I apologize for having missed it entirely.

d/

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 31 Jul 2003 11:05:01 +0200

In technical WGs we generally assume that the people who show
up and hum/raise hands/send email are by definition the set of
people with an active interest in the topic, and (modulo IESG
review and Last Call) that is considered the appropriate
constituency. It's essentially self-selected, and we seem to
be fine with that.

In non-technical topics (such as ipr) we have had to deal with
non-technical people self-selecting themselves for the WG. And
in some political topics (RFC 1984, RFC 2804) we have been happy
to accept a hum from the people who self-select themselves to
attend the Thursday plenary.

For process topics, there's a built in self-reference in any
consensus mechanism. But personally, I think the self-selection
for the Thursday plenary is much less suspect than the self-selection
for this WG. The people (including me of course) who have strong
feelings about process issues are not automatically the people
who should decide.

Self-selection for the Thursday plenary is probably indicative of 
interest in general IETF issues, given that the alternatives are 
going out for a leisurely dinner or leaving early for home. So there 
are good reasons to listen to the Thursday night hum for general issues.
After that, the ball is clearly in the IESG's court.

   Brian

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM 

NEW ADDRESS <[email-masked]> PLEASE UPDATE ADDRESS BOOK

---

From: Melinda Shore <[email-masked]>
Date: Thu, 31 Jul 2003 08:32:32 -0400

> quality of a claimed consensus.  

Sampling is an awful way to determine consensus (indeed,
it's nearly antithetical to consensus process) and while I
think it's clearly appropriate to talk about problems
inherent in having a self-selected group of individual
determine organizational direction I'm going to quibble and
say that the issue here is participation bias, not sampling
bias.

That said, I'm also somewhat concerned to see people saying
that at the plenary there was consensus to ask the IESG to
handle the reform process.  There was a very large minority
who didn't agree, and from where I sat it didn't look very
much like even a very, very rough consensus.  However I
don't think we're going to be able to come to consensus on
how to move forward and that we need to think about
alternatives to consensus to come to a decision.  I think
I'm comfortable with the sort of votes that were taken at
the plenary - they're certainly better than the alternatives
I can think of.

> drag out much longer" message.

Working groups are not asked to go out, kick around a few
ideas, and come to agreement on them.  They're asked to
produce documents.  We've been through the kick-around-a-
few-ideas phase and we're now in the process of trying to
finalize our documents.  It's our intention that the next
version of the problem-statement document will go into last
call, so I should *certainly* hope that we're now producing
diminishing returns.

Melinda