Subject: Re: Doubletree RST 'BOF' meeting notes

From: Bob Braden <[email-masked]>
Date: Sun, 6 Aug 2000 16:48:22 -0700 (PDT)

*> [the SACK people would like this option too - Matt Matthis told me
  *> afterwards however that currently SACK options must be sent with the
  *> SYN as some implementations fall over if the first TCP option is sent
  *> with a non-SYN segment.

Is this actually still true?  Are any of the TCPs with this bug
still in significant use?

Bob Braden

---

From: Alan Cox <[email-masked]>
Date: Mon, 7 Aug 2000 12:44:42 +0100 (BST)

> still in significant use?

If they do they are behind firewalls I suspect. People seem to spray arbitary
IP's with every known tcp attack regularly nowdays

---

From: Richard Wendland <[email-masked]>
Date: Mon, 7 Aug 2000 20:41:52 +0100 (BST)

> still in significant use?

Somewhat related I have noticed that a recent IRIX release sends
SACK-permitted on <SYN,ACK> despite not having received SACK-permitted on
the <SYN>.  I've not heard of this causing any great problems.  (I have
no reason to think it would later send SACK on a non-SYN segment, but
I haven't tested this directly.)

Initially I assumed this behaviour did not comply with RFC 2018, in the
way RFC 1323 explicitly disallows this for the timestamp and window-scale
options.  However on a close study of RFC 2018 I was surprised to find
that this is not explicitly disallowed as in RFC 1323, despite not being
able to see any useful reason to allow this and it having the potential
to break some old implementations.

RFC 2018 simply says "may be sent in a SYN", which must include both
<SYN> and <SYN,ACK> for RFC 2018 to work.  I assume this must be an
oversight in the wording, but it is now the specification, so I suppose
more implementations may start doing this.

Is there any strong reason to try to change the spec to disallow this
at a suitable opportunity?

	Richard
-- 
Richard Wendland				[email-masked]

---

From: Matt Mathis <[email-masked]>
Date: Tue, 8 Aug 2000 19:44:14 -0400 (EDT)

> more implementations may start doing this.

Oops, it was indeed an oversight.  Well, I'm not sure - we were pretty careful
to try to leave a trace of wiggle room in places where the community wanted us
to be more restrictive than we felt absolutely necessary.

However let me clarify:
>> the SACK people would like this option too - Matt Mathis told me
>> afterwards however that currently SACK options must be sent with the
>> SYN as some implementations fall over if the first TCP option is sent
>> with a non-SYN segment.

My conversation was slightly garbled.   The problem is that 1323 timestamps
must be committed (on or off) on the SYN and SYN-ACK.  Once committed you are
stuck if you got it wrong.

Since timestamps so badly waste header compressed slow links, most vendors
ship systems that default to RFC 1323 off.  This adds yet another step for 
ordinary people to do wrong before they can get decent rates out of any well
any connected workstation (e.g. almost any system at a university).

It would be better if all connections could start w/o time stamps and
automaticly turn them on when the data rate reaches some threshold.  I believe
that accepting such a late TS option is a 2 line code change (triggering it is
harder of course).

However, this was expressly forbidden by 1323 due to a couple of (now ancient)
systems known to die horrible deaths on unexpected TCP options.  Nobody has any
reasonable estimates of the residual population of these system.

In my copious spare time I would love to run a semi-production web server that
always sent late experimental (unpublished) options to see if I could detect
any systems with bugs.  I would bet not, and would then consider proposing to
revise 1323 to allow and even encourage late TS options.  (In my spare time!)

I wish that somebody would actually try this experiment or reconstruct more
details of the history.   Perhaps SGI has already done this well enough because
SACK-enabled on the SYN-ACK is guaranteed to be unexpected.

This prohibition on late TCP options is thwarting at least 2 reasonable
proposals for TCP options, including the RST error code.

--MM--

---

From: Alan Cox <[email-masked]>
Date: Wed, 9 Aug 2000 02:04:37 +0100 (BST)

> proposals for TCP options, including the RST error code.

I think the prohibition should just be dropped given the hostile nature of
the net today. Any box with that bug is basically unusable on the internet
anyway

---

From: Matt Mathis <[email-masked]>
Date: Wed, 9 Aug 2000 10:54:31 -0400 (EDT)

On Wed, 9 Aug 2000, Alan Cox wrote:

> the net today. Any box with that bug is basically unusable on the Internet
> anyway

Duh!  Sometime it is real important to rethink old decisions in new contexts.

Ok, lets ask the question the other way:

Can anybody think of *any* compelling argument agaist droping the prohibition
against new TCP options appearing after a SYN?

Thanks,
--MM--

---

From: Rick Jones <[email-masked]>
Date: Wed, 09 Aug 2000 10:17:45 -0700

> against new TCP options appearing after a SYN?

It means I have to be prepared to look for "unexpected" options in my
mainline data path.

I'm not certain it ensures there is enough space for all the options one
might want to carry. True, it means that the options added in the middle
would not have to fight for space with the MSS option or the window
scale option, but it may still have to duke it out with a SACK option
and timestamps.

rick

I cannot help but think about how planes like the B17 kept getting
heavier, more complex, and (iirc) just a little slower as people added
this feature and that feature...

-- 
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, OR post, but please do NOT do BOTH...
my email address is raj in the cup.hp.com domain...

---

From: [email-masked]
Date: Wed, 9 Aug 2000 21:58:14 +0400 (MSK DST)

Hello!

> automaticly turn them on when the data rate reaches some threshold.

Could you elaborate this? RFC1323 is not only PAWS yet, but also RTTM.
PAWS is almost useless paranoia when latency is low, e.g. at
"almost any system at a university" (or an enterprise).
And they really affect performance (negatively, of course)
in this case and can be disabled.

RTTM (and another TS users sort of Eifel) are useful exactly for slow links
with loss, high rtt and, especially, its variance. If those vendors,
who disable it, really observe decrease in performance due to timestamps,
they are _right_. It simply means that their TCP stack does not use timestamps
in any case. That's all.

>								  I believe

Idea is good, criterium is a bit strange.

It seems to be more clever to modify compression protocols
to compress timestamps instead.

Alexey

---

From: Rick Jones <[email-masked]>
Date: Wed, 09 Aug 2000 11:18:49 -0700

> It means I have to be prepared to look for "unexpected" options in my

BTW, I meant that in the context of added path length in my "normal"
data path...

rick jones

---

From: Alan Cox <[email-masked]>
Date: Wed, 9 Aug 2000 19:53:29 +0100 (BST)

> It means I have to be prepared to look for "unexpected" options in my

Only if you support it

> scale option, but it may still have to duke it out with a SACK option

Actually you made me realise a worse problem. It isnt the options and option
space that is a pain, its the sudden change of MSS relative to the MTU due
to additional header overhead

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 9 Aug 2000 12:49:05 -0700 (PDT)

> against new TCP options appearing after a SYN?

I guess this question should also be sent to the ROHC group.  I don't
know if they are prepared to handle this kind of things...

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 9 Aug 2000 12:35:36 -0700 (PDT)

> to additional header overhead

Because of SACK option, TCP needs to deal with that anyway...  And for stacks
which handle the case when one end suddenly stops sending timestamp option so
that TCP also stops sending it, they should handle the change of MSS also...

							K. Poon.
							[email-masked]

---

From: Lloyd Wood <[email-masked]>
Date: Wed, 9 Aug 2000 22:06:41 +0100 (BST)

> against new TCP options appearing after a SYN?

What about:

1. it being an indication that you've just been spoofed and
   that transmissions are now being received by a different host?

2. Interactions with max segment size, jumbo payload computations,
   etc.?

btw, there's no extra SYN negotation procedure for the new RFC2883
D-SACK. That might be some sort of precedent here.

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 09 Aug 2000 14:39:59 -0700

> Actually you made me realise a worse problem. It isnt the options and
> option space that is a pain, its the sudden change of MSS relative to
> the MTU due to additional header overhead

Accounting for an on-the-fly options should be roughly the same for RST
options as it is for SACK options (unless you're reserving MSS space in
every segment for a possible SACK that is).

-- 
Eric A. Hall                                      http://www.ehsco.com/
Internet Core Protocols        http://www.oreilly.com/catalog/coreprot/

---

From: [email-masked]
Date: Thu, 10 Aug 2000 22:30:28 +0400 (MSK DST)

Hello!

> D-SACK. That might be some sort of precedent here.

It is not a precedent. D-SACK is really backward compatible
in all the respects. If you receive it, you use it.

With timestamps the situation is different. Some stacks
(I will not show with finger) are optimized yet.
And if peer starts to send timestamps after some obscure
criteria is satisfied (sort of rate), you will fall to
maximally suboptimal way exactly at high rate.
Besides that connection will be aggressively delacked,
because mss estimation also relies on the fact,
that TS is persistent option.

Nothing will break, but all the things contradicting to original
specs will result in either fall to slow path or to failure
of existing heuristics.

Lots of second order effects.

So, negotiation phase for timestamps is required.
No matter, when it occurs. Stacks, which dislike enabling
timestamps in the middle of connection, will refuse the negotition.

Alexey

---

From: Matt Mathis <[email-masked]>
Date: Thu, 10 Aug 2000 16:10:40 -0400 (EDT)

> specs will result in either fall to slow path or to failure

I think your argument boils down to this:

There are other reasons to do timestamps besides PAWS.  If a stack is using one
of these other algoritms, then it may want to turn on timestamps earlier than
needed by PAWS.

Fine.  Not a problem.  So the choice is timestamps [off, on, or auto] and it is
our mission to make "auto" smart enough where future vendor defaults will
always be auto, and TCP will "just do the right thing".

Except for the ability to change timestamps mid connection, these details are
not protocol specifications and for the most part are allowed under current
standards.

Thanks,
--MM--

---

From: Lloyd Wood <[email-masked]>
Date: Thu, 10 Aug 2000 23:34:59 +0100 (BST)

> in all the respects. If you receive it, you use it.

That's not quite the case. SACK senders may not be D-SACK aware, and
may not use the DSACK information they receive. (I think there's lots
of future scope for algorithms based on D-SACK info, but not on
this list.)

L.

---

From: [email-masked]
Date: Fri, 11 Aug 2000 21:02:54 +0400 (MSK DST)

Hello!

> may not use the DSACK information they receive.

And nothing happens in this case, that's point.

Moreover, _using_ D-SACK info at sender is not so easy task,
it is difficult to expect that vendors will be able to make this soon.

But generating D-SACK is really easy task and sender, receiving
D-SACK and not understanding them does not suffer of them. They
are not more than usual SACKs for it.

> of future scope for algorithms based on D-SACK info, but not on

A bit out of topic, yes. But genarally this list is not so bad place
to discuss tcp yet. 8)

Alexey

---

From: [email-masked]
Date: Fri, 11 Aug 2000 21:32:32 +0400 (MSK DST)

Hello!

> of these other algoritms, then it may want to turn on timestamps earlier than

Not quite.

To use or not to use timestamps and when to use them is choice of OS,
user and/or administator yet.

[ Side note. I just tried to remind you that TS are useful not only
for PAWS, so that you criterium of enabling TS after achieving
some rate is highly dubious. ]

My point is that nobody is allowed to stop or to start sending
some persistent option without explicit negotiation. They will be
considered as "random jitter" sort of SACKs and will move input
process to slow path, which is appropriate for SACK and absolutely
inappropriate with persistent options.

Of course, smart OS would change its prediction algorithm without
special reminder. But it does not this now. And it looks really silly
to add some new flaky heuristics to stateful protocol in the place,
where explicit negotiation can be made. Classic TCP has enough
of such silly places to be bothered about avoiding adding new ones.

> always be auto, and TCP will "just do the right thing".

Provided those vendors, who do not have time machine and cannot
upgrade already sold OSes, will not see "auto" options without
"auto" negotiation, which will fail. Let this negotiation will
be made in the middle of connection, it does not matter.

Alexey

---

From: Richard Wendland <[email-masked]>
Date: Mon, 14 Aug 2000 18:14:53 +0100 (BST)

> details of the history.

Your wish has been partly granted!

A number of HTTP load balancing devices or firewalls do some odd (if
not evil) things with TCP.

One particular device indeed seems particularly broken, in that it
removes all TCP options from SYN-ACK, and then in-effect sends them on
the next ACK!  Thus for example MSS or window-scale options are on a
plain ACK; so although these aren't experimental options, they would be
thoroughly unexpected, potentially breaking TCP stacks.

This particular device is in use on at least something like 30 IP
addresses, running about 50 websites, worldwide.  I don't recognise
any of these sites as especially well-known, but some do seem serious
websites that probably have significant use.

So although this device isn't in widespread use, it probably has
non-trivial use, fairly successfully we must assume.  However the
operators of these sites aren't likely to be very TCP knowledgeable,
so maybe there are problems with clients which they don't recognise
and report.

A point to bear in mind though, as Alan Cox pointed out, is that Internet
connected devices will often see odd TCP effects from hostile actions.
But it may be that off-Internet devices will show significant problems
with unexpected TCP options.  I recall that when FreeBSD enabled T/TCP
by default, there were some reported problems with terminal servers,
and maybe these intranet devices should be the area of most concern.

Some tcpdump output from websites using the above mentioned device are
below.  Note another oddity of the device is a zero window on the SYN-ACK.

This isn't the only such odd device.  Another, in widespread use,
seems in effect to remove all TCP options other than MSS from SYN-ACK.
But later segments are allowed with all TCP options in place, so TCP
stacks will see non-negotiated options, like late TS options; but these
would only go to TCP stacks that knew about them as they would be in
response to those offered on the SYN.

Both these devices seem to me, by removing window-scale on SYN-ACK, to
have the potential to make the two ends think that different window-scale
factors are agreed; but I have not demonstrated this.  So maybe they break
TCP in this regard, even if they get away with the rest of what they do.

Here's some trimmed tcpdump output from the first-mentioned device.

www.allegiancehealth.com:

CLIENT1.6170 > www.allegiance.net.http: S 420261394:420261394(0) win 512 <mss 984>
www.allegiance.net.http > CLIENT1.6170: S 2893365979:2893365979(0) ack 420261395 win 0
CLIENT1.6170 > www.allegiance.net.http: . ack 1 win 32472 (DF)
www.allegiance.net.http > CLIENT1.6170: . ack 1 win 8856 <mss 1460> (DF)
                                        ^^^^^            ^^^^^^^^^^
CLIENT1.6170 > www.allegiance.net.http: P 1:18(17) ack 1 win 32472 (DF)
www.allegiance.net.http > CLIENT1.6170: . ack 18 win 8839 (DF)
...

www.chuden-cs.co.jp:

CLIENT2.3355 > ccsgw1.chuden-cs.co.jp.http: S 2013930150:2013930150(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 5546525 0> (DF)
ccsgw1.chuden-cs.co.jp.http > CLIENT2.3355: S 351095432:351095432(0) ack 2013930151 win 0
CLIENT2.3355 > ccsgw1.chuden-cs.co.jp.http: . ack 1 win 16384 (DF)
ccsgw1.chuden-cs.co.jp.http > CLIENT2.3355: . ack 1 win 10136 <nop,nop,timestamp 591073401 5546525,nop,wscale 0,mss 1460> (DF)
                                            ^^^^^^^           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
CLIENT2.3355 > ccsgw1.chuden-cs.co.jp.http: P 1:18(17) ack 1 win 16384 (DF)
ccsgw1.chuden-cs.co.jp.http > CLIENT2.3355: . ack 18 win 10220 (DF)
...

	Richard