Subject: Re: TCP MSS option value

From: David Borman <[email-masked]>
Date: Fri, 28 Jan 2000 12:53:16 -0600 (CST)

> From [email-masked] Fri Jan 28 11:40:55 2000
> To: Bob Braden <[email-masked]>
> cc: [email-masked], [email-masked]
> Subject: Re: TCP MSS option value 
> Date: Fri, 28 Jan 2000 11:15:17 -0500
> From: Bill Sommerfeld <[email-masked]>
>
> > The MSS option is NOT intended as an MTU discovery mechanism.  It
> > gives the max segment size the TCP receive and reassemble.  It does
> > not, and is not intended to, tell the sender how large a segment to
> > send.
>
> Well, given that it provides an upper bound on segment size, it is
> *pragmatically* the only knob you can use when attempting to
> interoperate with systems not under your control with broken PMTUd
> blackhole detection living in PMTUd black holes (like many web servers
> living behind firewalls which block all ICMP).
>
> Try surfing the web from behind a path with a bottleneck IP MTU of
> 1480 some time..  if you advertise an MSS greater than 1440, you'll
> never see a full page from many servers..

A lot of the time when we've seen this, it has been due to a
misconfiguration in a Cisco router that the local ISP uses to
connect to the backbone.  For example, UUNET defaults to using
a 4K or so MTU across the HDLC connection.  Cisco boxes have a
default 1500 MTU for HDLC connections.  If the local ISP box
gets changed/replaced/reset/whatever, and the configuration for
the connection goes back to the default 1500 MTU, you get the
classic black-hole.  If you connect to a web server that has
a 4K path to the back-bone, and you advertise a 4K MSS, then
it will attempt to send 4K packets.  They make it to the far
side of the ISP connection, but then get dropped by the Cisco
box at the ISP because the packet is too long.  UUNET thinks
4K is just fine and sends the packet, the ISP box drops it in
hardware.  Because it doesn't get dropped by IP, no ICMP
message gets generated, and you have a black hole.

There are various tests you can do from your local side to
verify that this is indeed what is happening, and then you
can call up and complain to your ISP about having their
hardware misconfigured.
	1) Ping the web server, if that works:
	2) ping again with large packets (-s 10000)
	3) Try the test to your ISP's router
	4) try "traceroute www.XXX.com 10000", because
	   the return packets are small, that will
	   succeed where pint -s 10000 won't.

			-David Borman, [email-masked]

---

From: John Hawkinson <[email-masked]>
Date: Fri, 28 Jan 2000 14:24:11 -0500

This sounds kind of confusing.

dab writes:
> default 1500 MTU for HDLC connections. 

Actually, it depends on the interface type. DS3 HDLC connections
have a 4470 MTU, f'rinstance. T1s are 1500.

> If the local ISP box gets changed/replaced/reset/whatever, and the
> configuration for the connection goes back to the default 1500 MTU,
> you get the classic black-hole.

I'm not certain why an ICMP frag failure doesn't get sent,
let me see if I have the scenario correct:

> If you connect to a web server that has a 4K path to the back-bone,
> and you advertise a 4K MSS, then it will attempt to send 4K packets.
> They make it to the far side of the ISP connection, but then get
> dropped by the Cisco box at the ISP because the packet is too long.

Err, you're saying that an the MTU of one side of the interface does
not agree with the MTU of the other side of the interface, and because
IOS assumes it will never get a packet on an interface that is larger
than the MTU for that interface, it has a problem when it receives
such a packet?

> UUNET thinks 4K is just fine and sends the packet, the ISP box drops
> it in hardware.  Because it doesn't get dropped by IP, no ICMP

Hmm...yes, though a counter does get incremented, I think.

I don't know whether it is reasonable to explore solutions
to this problem that involve layering violations or handing
partial packets to IP.

Some routing protocols attempt to look for problems like this,
e.g. OSPF neighbors comparing MTUs and refusing to neighbor
if this happens. Of course, presumably you have an interdomain
case where the routing protocol is either "static" or "BGP".

Perhaps, if it were not considered too ugly, a BGP option could be
added to try to detect this misconfiguration and inform the
administrator; this seems easier than trying to mandate a new protocol
to check for this, or a PPP option (since so many links use "cisco
HDLC"). It seems kind of odd, incidently, that cisco's CDP ("cisco
discovery protocol", which exchanges information about the
name/version/ipaddress of routers on common links) fails to include
MTU information....

> 	   succeed where pint -s 10000 won't.

This is indistinguishable from an ICMP-filtering-induced
PMTU blackhole, though.

--jhawk

---

From: Greg Minshall <[email-masked]>
Date: Fri, 28 Jan 2000 16:23:50 -0800

Dave,

Isn't the problem, as Bill Sommerfeld said, that some firewall is blocking the 
"ICMP dest unreachable" packets, so pmtu isn't working?

Greg (yep, i'm so old that i even remember when there really *was* an 
end-to-end internet...)