Subject: Hung Solaris TCP

From: Wu-chang Feng <[email-masked]>
Date: Fri, 02 Jun 2000 15:01:46 -0700

Has anyone seen problems with TCP hanging on Solaris when accessing a
web site over a lossy link?  I've tried hitting pages from
www.viajeya.com using Solaris 2.6 and Solaris 2.7 (11/99 version). 
When packets arrive ordered correctly, the request immediately
returns.  When the last couple of packets come back out of order, the
client TCP (in this case either a manual telnet or a squid cache
process) sits around indefinitely.....Here's a tcpdump trace of two
identical requests: one fails and the other is OK.

Wu

Bad request
-----------
14:45:12.511537 panamint.35323 > 200.10.104.13.80: S
1806393459:1806393459(0) win 8760 <mss 1460> (DF)
14:45:13.259467 200.10.104.13.80 > panamint.35323: S
2085229208:2085229208(0) ack 1806393460 win 8760 <mss 1460> (DF)
14:45:13.259495 panamint.35323 > 200.10.104.13.80: . ack 1 win 8760
(DF)
14:45:14.198208 panamint.35323 > 200.10.104.13.80: P 1:42(41) ack 1
win 8760 (DF)
14:45:15.144267 200.10.104.13.80 > panamint.35323: . ack 42 win 8719
(DF)
14:45:15.144291 panamint.35323 > 200.10.104.13.80: P 42:44(2) ack 1
win 8760 (DF)
14:45:15.966015 200.10.104.13.80 > panamint.35323: P 1660:1665(5) ack
44 win 8717 (DF)
14:45:15.966058 panamint.35323 > 200.10.104.13.80: P 44:46(2) ack 1
win 8760 (DF)
14:45:15.966069 panamint.35323 > 200.10.104.13.80: . ack 1 win 8760
(DF)
14:45:15.970164 200.10.104.13.80 > panamint.35323: P 1:200(199) ack 44
win 8717 (DF)
14:45:15.970206 panamint.35323 > 200.10.104.13.80: . ack 200 win 8760
(DF)
14:45:16.163957 200.10.104.13.80 > panamint.35323: P 200:1660(1460)
ack 44 win 8717 (DF)
14:45:16.163984 panamint.35323 > 200.10.104.13.80: . ack 1665 win 8760
(DF)
14:45:16.973773 200.10.104.13.80 > panamint.35323: P 3125:4585(1460)
ack 46 win 8715 (DF)
14:45:16.973804 panamint.35323 > 200.10.104.13.80: . ack 1665 win 8760
(DF)
14:45:16.986916 200.10.104.13.80 > panamint.35323: P 1665:3125(1460)
ack 46 win 8715 (DF)
14:45:16.986940 panamint.35323 > 200.10.104.13.80: . ack 4585 win 8760
(DF)
14:45:17.184131 200.10.104.13.80 > panamint.35323: P 6045:7505(1460)
ack 46 win 8715 (DF)
14:45:17.184154 panamint.35323 > 200.10.104.13.80: . ack 4585 win 8760
(DF)
14:45:17.327897 200.10.104.13.80 > panamint.35323: P 4585:6045(1460)
ack 46 win 8715 (DF)
14:45:17.327925 panamint.35323 > 200.10.104.13.80: . ack 7505 win 8760
(DF)
14:45:17.805154 200.10.104.13.80 > panamint.35323: FP 8965:9198(233)
ack 46 win 8715 (DF)
14:45:17.805187 panamint.35323 > 200.10.104.13.80: . ack 7505 win 8760
(DF)
14:45:18.009141 200.10.104.13.80 > panamint.35323: P 7505:8965(1460)
ack 46 win 8715 (DF)
14:45:18.009166 panamint.35323 > 200.10.104.13.80: . ack 9198 win 8760
(DF)
------->Connection killed manually after 5 minutes.
14:50:09.086690 panamint.35323 > 200.10.104.13.80: P 46:51(5) ack 9198
win 8760 (DF)
14:50:09.638441 200.10.104.13.80 > panamint.35323: R
2085238407:2085238407(0) win 0 (DF)

Good request:
-------------
14:43:30.073361 panamint.35321 > 200.10.104.13.80: S
1793430152:1793430152(0) win 8760 <mss 1460> (DF)
14:43:30.578465 200.10.104.13.80 > panamint.35321: S
2085100976:2085100976(0) ack 1793430153 win 8760 <mss 1460> (DF)
14:43:30.578489 panamint.35321 > 200.10.104.13.80: . ack 1 win 8760
(DF)
14:43:37.043337 panamint.35321 > 200.10.104.13.80: P 1:42(41) ack 1
win 8760 (DF)
14:43:37.696615 200.10.104.13.80 > panamint.35321: . ack 42 win 8719
(DF)
14:43:37.696650 panamint.35321 > 200.10.104.13.80: P 42:44(2) ack 1
win 8760 (DF)
14:43:38.198522 200.10.104.13.80 > panamint.35321: P 1:200(199) ack 44
win 8717 (DF)
14:43:38.198673 panamint.35321 > 200.10.104.13.80: . ack 200 win 8760
(DF)
14:43:38.371998 200.10.104.13.80 > panamint.35321: P 200:1660(1460)
ack 44 win 8717 (DF)
14:43:38.372097 panamint.35321 > 200.10.104.13.80: . ack 1660 win 8760
(DF)
14:43:38.372325 200.10.104.13.80 > panamint.35321: P 1660:1665(5) ack
44 win 8717 (DF)
14:43:38.420311 panamint.35321 > 200.10.104.13.80: . ack 1665 win 8760
(DF)
14:43:38.787564 200.10.104.13.80 > panamint.35321: P 1665:3125(1460)
ack 44 win 8717 (DF)
14:43:38.830446 panamint.35321 > 200.10.104.13.80: . ack 3125 win 8760
(DF)
14:43:38.994654 200.10.104.13.80 > panamint.35321: P 3125:4585(1460)
ack 44 win 8717 (DF)
14:43:39.040430 panamint.35321 > 200.10.104.13.80: . ack 4585 win 8760
(DF)
14:43:39.048770 200.10.104.13.80 > panamint.35321: P 4585:6045(1460)
ack 44 win 8717 (DF) 
14:43:39.048834 200.10.104.13.80 > panamint.35321: . 8960:8965(5) ack
44 win 8717 (DF)
14:43:39.048859 panamint.35321 > 200.10.104.13.80: . ack 6045 win 8760
(DF)
14:43:39.109748 200.10.104.13.80 > panamint.35321: P 6045:7500(1455)
ack 44 win 8717 (DF)
14:43:39.109773 panamint.35321 > 200.10.104.13.80: . ack 7500 win 8760
(DF)
14:43:39.133228 200.10.104.13.80 > panamint.35321: P 7500:8960(1460)
ack 44 win 8717 (DF)
14:43:39.133255 panamint.35321 > 200.10.104.13.80: . ack 8965 win 8760
(DF)
14:43:39.340590 200.10.104.13.80 > panamint.35321: FP 8965:9198(233)
ack 44 win 8717 (DF)
14:43:39.340702 panamint.35321 > 200.10.104.13.80: . ack 9199 win 8760
(DF)
14:43:39.341280 panamint.35321 > 200.10.104.13.80: F 44:44(0) ack 9199
win 8760 (DF)
14:43:39.829074 200.10.104.13.80 > panamint.35321: . ack 45 win 8717
(DF)

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 2 Jun 2000 17:02:33 -0700 (PDT)

> identical requests: one fails and the other is OK.

Do you know what platform www.viajeya.com is?

> 14:45:18.009166 panamint.35323 > 200.10.104.13.80: . ack 9198 win 8760

panamint ack's up to 9198.  The version of Solaris you have does not store
out of order FIN.  You can see that this ack does not ack the FIN, otherwise
the ack would be 9199.  But the other side does not retransmit the FIN, and
it should because FIN has not been ack'ed.  This causes the hang.  This is
a bug in www.viajeya.com.

To get around this bug in www.viajeya.com, refer to Sun RFE 4330074.  I think
the latest 2.6 patch should have this enhancement.  The change is to make
Solaris' TCP record the out of order FIN info.

							K. Poon.
							[email-masked]

---

From: Craig Partridge <[email-masked]>
Date: Fri, 02 Jun 2000 20:37:13 -0400

That's an interesting bug.

If I'm reading this right, what happens is the remote server sends a FIN
to panamint, which panamint *fails* to ACK (it acks up to but not
including the FIN) if the FIN arrives out of order.  It looks like
panamint doesn't save the FIN when it saves the out of order data.

The interesting question is why the server doesn't retransmit the FIN but
instead falls into a state where it sends a RESET when panamint tries to
send later...  That is odd, as FIN-WAIT-1 is supposedly a stable
state.

Craig

---

From: "David S. Miller" <[email-masked]>
Date: Fri, 2 Jun 2000 19:59:24 -0700

Date: Fri, 02 Jun 2000 20:37:13 -0400
   From: Craig Partridge <[email-masked]>

   If I'm reading this right, what happens is the remote server sends
   a FIN to panamint, which panamint *fails* to ACK (it acks up to but
   not including the FIN) if the FIN arrives out of order.  It looks
   like panamint doesn't save the FIN when it saves the out of order
   data.

   The interesting question is why the server doesn't retransmit the
   FIN but instead falls into a state where it sends a RESET when
   panamint tries to send later...  That is odd, as FIN-WAIT-1 is
   supposedly a stable state.

This reminds me, we once had a bug under Linux talking to older
Solaris stacks in that if we kept resending the FIN with data
attached, Solaris would not ACK it.  The fix was to drop the data
bytes from the final FIN packet once they were acked to work around
this Solaris TCP bug.

This is probably the same bug being seen here.

Later,
David S. Miller
[email-masked]

---

From: Markus Peuhkuri <[email-masked]>
Date: 05 Jun 2000 11:57:32 +0300

> Do you know what platform www.viajeya.com is?

	Nmap V 2.12 <URL:http://www.insecure.org/nmap/> answers 
	"Remote operating system guess: Windows NT4 / Win95 / Win98"

-- 
Markus Peuhkuri        ! http://www.iki.fi/puhuri/

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 5 Jun 2000 16:21:08 -0700 (PDT)

> This is probably the same bug being seen here.

No, this is not the same.  The problem in discussion is that the other side
does not retransmit the FIN, not failing to accept a FIN.  We have experience
with some version of NT server which does exactly this.  It seems that even
without the FIN acknowledged, the NT server "thinks" that the FIN is ack'ed,
thus closing the connection.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 5 Jun 2000 17:29:59 -0700 (PDT)

> 	"Remote operating system guess: Windows NT4 / Win95 / Win98"

Someone sent me the following bug report in NT.

http://support.microsoft.com/support/kb/articles/Q254/9/30.ASP?LN=EN-US&SD=gn&FR=0

							K. Poon.
							[email-masked]