Subject: RE: Doubletree RST 'BOF' meeting notes

From: Bernie Volz <[email-masked]>
Date: Thu, 3 Aug 2000 09:56:44 -0400

If this situation is critical to your application, I'd recommend the
following flow:

Send the "end-of-connection" message.
Issue a shutdown of the send side.
Issue a read with a short time-out (short can be up to the application).
Note that one option is to use a select, with timeout. If data is read (or
available for the select, read it), restart the read (or select). If there
is an error (timeout) or read returns end of connection, call close.

This is the safest and surest way to close a connection on any stack.

We should have the discussion on whether the RST behavior is correct or not,
but that won't solve the problem for existing stacks that exhibit this
behavior. When working on the "Known Problems" issues, I tested a bunch of
stacks and they did exhibit this behavior (though I can't say whether there
was always data flowing in both directions).

I should also point out that even if one fixes the stacks regarding the RST
behavior, there is still the question as to whether a stack should deliver
all received data when it has received an RST on a connection or drop that
data to notify the application as early as possible that the connection is
gone. If the stack drops any received data, then just changing the sending
of the RST doesn't solve the problem. Consider:

	Client					Server
	------					------
	Sends "large" buffer			Sends "small" shutdown
message
	ACKs "small" shutdown message
							Sends FIN
	ACKs "FIN"
	More of "large" buffer sent
							Sends RST
	Receives RST

Now, if the client side drops all of received data, the application hasn't
read the data and hence never gets the final message anyway.

So, to fix the problem you describe requires TWO changes:
1) Don't send RST on incoming data if there is still outgoing data to
deliver.
2) Don't drop received data when RST received on a connection.

-----Original Message-----
From: Tom Lane [mailto:[email-masked]]
Sent: Wednesday, August 02, 2000 6:42 PM
To: [email-masked]
Subject: Re: Doubletree RST 'BOF' meeting notes

Ian Heavens wrote:
> Tom Lane has highlighted a usage of RSTs that conflicts with RFC 1122
> half duplex close recommendation (and emphasised in RFC 2525 -
> contributed by me).  Bernie Volz and Tom discussed this after so I
> don't know whether the conclusion was that the application should
> avoid sockets close() and use shutdown() - thereby circumventing RFC
> 1122 completely (it only applying to use of APIs that close in both
> directions ).  To be pursued on the list if appropriate, with possible
> clarification of RFC 1122 and RFC 2525.

[ Hi folks.  I just joined the list --- am not a TCP implementor,
but got interested in this particular issue for reasons about to be
stated. ]

The situation that I am interested in is an application where data can
flow asynchronously in both directions.  Actually, it's mostly a client
request/server response protocol, but the server can also send
unsolicited messages at any time.  In particular the server application
can send an error message and immediately close the connection (indeed,
it quits completely) without waiting around for the far-end app to get
that message.  Question for the list is, is this application misusing
the TCP protocol, and if so why?  Seems to me that once the TCP stack
has accepted data to be sent, it ought to make a reasonable effort to
deliver that data, whether the sending app is holding the connection
open or not.

The problem case arises when the far-end app is busy sending a new
request just as the near-end app emits its final message and closes
its socket FD.  RFC 1122 (4.2.2.13) says

            A host MAY implement a "half-duplex" TCP close sequence, so
            that an application that has called CLOSE cannot continue to
            read data from the connection.  If such a host issues a
            CLOSE call while received data is still pending in TCP, or
            if new data is received after CLOSE is called, its TCP
            SHOULD send a RST to show that data was lost.

(See also 2.16 and 2.17 in RFC 2525.)

Some, but not all, stacks appear to issue such an RST when they see
incoming data from the far end, *even though they still have un-acked
outbound data to transmit*.  This results in loss of the outbound data.
A variant is that even if the data has been transmitted and acked but
not actually delivered to the far-end client app, the far-end TCP may
discard the data from its buffer upon receiving the RST.  Still a third
variant (which seems to be what the current Linux stack does, at least
when both ends are on the same machine) is that the far-end client may
receive an ECONNRESET error from its next read() attempt, though not
all the data has been delivered to it.  If the client is loony enough
to think that ECONNRESET is a soft error, it will indeed discover that
it can read more data by trying again!

I claim that all these behaviors are broken, and that the right thing
to do if the TCP has noplace to deliver incoming data to is to ack and
discard the data.  It would be OK to send an RST to discourage the other
end from sending more data *only* if there is no un-ACKed data remaining
to be sent from this end.  Without this, half-duplex connections are
useless.  TCPs should also be dissuaded from dropping buffered inbound
data when they see inbound RST or delivering a report of incoming RST
out-of-sequence from the data.

The discussion at the Doubletree mostly focused on whether the app
should deal with this situation by holding open its socket for some
timeout period after sending its final outgoing message, so as to
read and discard any data that might happen to arrive.  That is
certainly the only available answer if one needs an answer today,
but I still think that it's a workaround for broken TCP behavior.
Essentially, this is asking the app to take on the connection-shutdown
duties that TCP ought to handle for it.  (To take just one example,
how should the app know what an appropriate timeout is?  Seems like
that's a concern the TCP layer should deal with, not the app.  The app
will also be unable to detect when the last outbound data has been
ACKed, and will therefore have to hold the connection open much longer
than would otherwise be necessary in most cases.)

Anyway, that's my story.  Is my app broken, or is the recommendation
for issuing an RST in this scenario misguided?

			regards, tom lane

---

From: Tom Lane <[email-masked]>
Date: Thu, 03 Aug 2000 10:29:48 -0400

Bernie Volz <[email-masked]> writes:
> gone.

I believe that would certainly be a bad way to operate (though I don't
doubt that some stacks may do it, and the Linux kernel's current
behavior is even worse).  To take one example: one of the ideas that was
discussed yesterday was having one side or the other send RST after the
FIN handshake is completed, to allow the other side to drop its TIMEWAIT
state.  That would be completely unsafe if a TCP is allowed to discard
received-but-undelivered data when it receives RST.

> 2) Don't drop received data when RST received on a connection.

Right, I thought I said that ... actually part of my question here is
to what extent these actually are changes, and not existing behavior.

			regards, tom lane