Subject: [quicwg/base-drafts] Resetting Streams (#165)

From: Marten Seemann <[email-masked]>
Date: Mon, 16 Jan 2017 20:04:40 -0800

I'm writing this issue because there was some discussion about a change I made in #145, that seemed to be controversial. I spent a lot of time thinking about this, and realized that my changes in that PR were insufficient to address the problem. I think there are multiple issues in the stream state machinery.

## Two common scenarios

I will describe two scenarios where RST_STREAMs will be used regularly when using HTTP/QUIC.

### Rejected PUSH

The sender (in this case a HTTP server) uses HTTP/2 Push to push a resource that the receiver isn't interested in. The receiver therefore sends a RST_STREAM with offset 0 (since it didn't send any data on that stream) to stop the sender from pushing the resource.

When the sender receives the RST_STREAM, there are two options:

- the transfer hasn't completed yet: The sender then stops sending data and sends a RST_STREAM with a byte offset equal to whatever highest data offset it sent. RST_STREAMs are transmitted reliably, so both sides will *eventually* receive the other side's RST_STREAM and make adjustments to their flow controller. There is no problem here.
- the transfer has already completed (i.e. the sender already sent a STREAM frame with the FIN bit): According to the draft, STREAM frames on reset streams are not retransmitted. If the STREAM frame with the FIN gets lost, and the sender will never be able to make the necessary adjustments to its flow controller. (I'm not sure, but maybe this is what #163 is about).

A fix for this problem is to **always** send a RST_STREAM as a response to a RST_STREAM, even if the stream already transitioned to the *half closed* state by sending a FIN bit.

### POST to a 404 URI

The second example is more involved, since it involves data flowing bidirectionaly (note that for the PUSH case, the receiver never sent any data on the stream):

The sender (in this case a HTTP client) performs a POST request to an URI that the receiver is not reading from, e.g. a non-existing site (HTTP status code 404). The receiver will first send the HTTP response header, and then the (non-empty) 404 body. It will send send a RST_STREAM with a offset equivalent to the body length, to stop the sender, to stop it from sending more data.

On the sender side, we have the same situation as in the PUSH case, but let's assume we already resolved that issue.

There are two more issues here:

- We must ensure reliable delivery of the response body (the 404 page). Thus, although the stream was reset, STREAM frames need to be retransmitted.
- The receiver can only send the RST_STREAM **after** composing the response body (since the final byte offset has to be included). However, it would make sense to send the RST_STREAM as soon as possible, to stop the unwanted upload.

The first issue could be fixed by requiring that the peer that is resetting a stream MUST ensure reliable delivery of stream data it sent, whereas a peer that is receiving the RST_STREAM SHOULD NOT retransmit any stream data.

The second issue could be fixed by defining a RST_STREAM frame that doesn't include a byte offset (or introducing an RST error code that indicates that the byte offset is not set). The receiver in the above example would send this RST_STREAM before sending the 404 body. Since stream data is transmitted reliably, the sender would eventually receive a highest byte offset with a STREAM frame carrying the FIN flag.

## Open points for discussion

Do the three changes capture all possible use cases of resetting streams. Or do we need
* a way to reset a stream such that the peer initiating the reset doesn't ensure the reliable transmission of the data it already sent
* a way to reset a stream such that the peer receiving the RST has to ensure the reliable transmission up to a certain (which?) offset

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165

---

From: mirjak <[email-masked]>
Date: Tue, 17 Jan 2017 02:35:59 -0800

To be honest I still believe you are mixing up transport and application semantics here. Why are you sending a reset in this situation and not a fin? From the transport point of view there is no problem (in both cases) all transmissions as provided by the higher layer have been successful. If you want to make sure that the other end is not sending any more data (but why should it send more data after the 404 was received), you should close the connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273098593

---

From: Mike Bishop <[email-masked]>
Date: Tue, 17 Jan 2017 08:28:02 -0800

@mirjak, I half agree with you.  The issue here is that, in both scenarios, one peer wishes to abruptly terminate the other peer's activity on a stream.  The issue that we've hit with FIN is that it implies a completed message, and in both these scenarios the point is that there *isn't* a completed message.  We could do this at the app layer by introducing an EOM marker plus an app-layer RST command that tells the peer to abruptly stop sending, but.... why?  (Also, that usually hasn't worked well in TCP, where the client isn't reading from the receive buffer while they're dumping the message body into the send buffer.  By the time they see the "stop sending" message, it's already been handed to the transport and can't be revoked without a RST.)  Unfortunately, RST_STREAM is currently defined to abruptly terminate *all* activity on a stream, which isn't what the application needs in this case.

Maybe the simplest solution is to add e.g. a MIC_DROP bit to STREAM that means "FIN my side, RST yours"?  Then RST_STREAM can be reserved for actual errors.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273219695

---

From: mirjak <[email-masked]>
Date: Tue, 17 Jan 2017 08:44:17 -0800

As you said there are two possibilities here:
1) the transport could tell the other end that it is not willing to except any (new?) data anymore on a certain stream, or
2) you tell the other end in the application layer that it should stop sending and you have a local interface to the transport that tell it to drop all data that has not send yet and close the stream.

However, I still don't really understand the use case where 1) is actually fast then 2) because there are any outstanding data that is sitting in the transport buffer but not send yet. 
Regarding the two cases above:
Push: Either you don't want any push message at all anymore. In this case you can tell the other end at any time in the application layer that it should not send new push messages (basically properly unsubscribe) and the other end will probably close the stream. Or you receive a new push message but have to fully receive and understand it to decide that you don't want to have any push message in future; again in this case you tell the other in the application to unsubscribe, after you have received the full message.
404: In this case you also have to receive the full 404 message first and I would not expect to receive any additional data in this case anyway, so you can probably just close the whole connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273224756

---

From: Mike Bishop <[email-masked]>
Date: Tue, 17 Jan 2017 09:02:38 -0800

But the push scenario isn't saying that I don't want *any* push messages in the future.  It's saying that the other side has promised to send:

- resource A on stream X
- resource B on stream Y
- resource C on stream Z

I already have resources A and C, so I want to tell the peer not to waste its time on those.  In HTTP/2, the way to do that is to RST_STREAM streams X and Z.

In the 404 case, you're correct that you won't receive any additional data from the server.  The issue is the reverse, data from the client:

- Client sends message headers
- Client begins sending (large) message body
- Server sees message headers and can already generate a response without parsing the body
    - ...and the response is almost certainly "I don't want the body" if it can be generated at this point.
- Client is still sending the message body

At this point in HTTP/1.1 over TCP, the server is stuck receiving and discarding the entire message body from the client, which could be gigabytes in size.  The client can't continue using the TCP connection without completing the message, but the rest of the message is irrelevant.  The server can't RST the connection because that would discard its response in the client's TCP buffer upon receipt.  (Same problem we're trying to avoid here.)  If the client is smart enough to be reading from the socket even while it's sending, *it* could choose to RST the TCP connection and open a new one rather than sending the rest of the body; more often, it will spew useless bytes into the transport prior to reading.

In HTTP/2, this is at least confined to a single stream.  After sending its response, the server can RST_STREAM with NO_ERROR, telling the client to abort sending data on that stream.  That works because TCP under HTTP/2 is still providing in-order and reliable delivery, so the client is guaranteed to get the response *and* the RST_STREAM in that order.

In QUIC, though, a RST_STREAM can accidentally drop data that was sent before it, such as the server's response.  That leaves the client with a terminated stream, but not having received the response to tell it why.  We need a solution that interrupts the client, but still reliably delivers the server's data.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273230643

---

From: Mike Bishop <[email-masked]>
Date: Tue, 17 Jan 2017 09:05:03 -0800

In other words, both cases arise out of one party's ability to conclude a response without seeing the other party's entire message.  The requirement is to reliably deliver the response, but minimize wasted bandwidth on the remainder of the now-obsolete message.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273231349

---

From: mirjak <[email-masked]>
Date: Tue, 17 Jan 2017 09:29:20 -0800

Okay, still not sure about the push case. But I see your point for 404. Wasn't clear to me that the server could already issue a 404 response before receiving the whole GET request.

I was about to agree that we need a separate "don't send any more data" signal but I still have the feeling that this is the wrong split between transport and application semantics; meaning the application should decide that it does want to send certain data anymore.

Technically there is still another solution. The application/server tells the transport that it does want the data anymore and the transport can announce a receive window of zero for that stream. As soon as the client receives the 404 (or what ever message that indicates it does make sense to send anymore data), it could cancel the sending of any outstanding data on that scream (or completely close the connection).

However, that also seems like a hack and has the same effect, so maybe the unidirectional reset might be better. Let me sleep about this...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273238284

---

From: Mike Bishop <[email-masked]>
Date: Tue, 17 Jan 2017 09:43:24 -0800

To a certain extent it *is* the wrong split.  In an ideal world, the client would be reading from the socket, see that the server responded, then generate a RST_STREAM itself.  But that puts control of server load on well-behaving clients.  The server telling the transport layer not to generate any more WINDOW_UPDATE frames on the stream would partially mitigate that; however, you risk stalling clients that aren't reading in parallel, the clients are still going to retransmit lost frames in what they've already sent, and they're going to fill up their existing window with useless data.

It really depends how much we want to trust peers versus putting more control in the hands of the party getting data thrown at them.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273242495

---

From: mirjak <[email-masked]>
Date: Tue, 17 Jan 2017 12:43:55 -0800

Yes, I managed to convince myself in the mean time that the zero window solution is not a good idea given that you will keep data in the buffer unnecessary that is waiting for a retransmit that is not needed.

However, I think reset is the right terms here; what's about adding two flags to indicate which direction is reset, allowing for resetting one direction but not the other, or both in the error case where one end somehow just lost all state on that stream.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273293772

---

From: Martin Thomson <[email-masked]>
Date: Tue, 17 Jan 2017 14:32:36 -0800

@mirjak, that suggestion smells a lot like unidirectional streams.  That is, breaking the state machine link between the two directions of a stream.  I'm starting to think that it might be feasible to do that, but it does create some issues.  But I think that we already have those issues and we're blithely ignoring them for the moment.  I'll expound further in another message, because a change on that magnitude probably needs better framing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273322018

---

From: Mike Bishop <[email-masked]>
Date: Wed, 18 Jan 2017 10:13:31 -0800

I just took a stab at this, and landed on something that, as @martinthomson says, smells a lot like unidirectional streams.  While I left the description as a joint model, I changed RST_STREAM to close only the sender's direction (though the requirement to respond to RST_STREAM with RST_STREAM that was already in the doc is still there, so you wind up in the same place).

I also added a REQUEST_RST frame, which effectively says "there's no point sending anything else; no one's consuming it."  The recipient SHOULD send a RST_STREAM for their direction upon receipt.  I *think* that covers the above scenarios.

The biggest hole that I see is in the two playing off each other:

- Server sends REQUEST_RST and starts generating the response
    - Due to lost packets, some portion of the response gets lost and is pending retransmission
- Client gets the REQUEST_RST and obediently generates a RST_STREAM
- Server gets the RST_STREAM and responds with RST_STREAM
    - Server aborts retransmission of its response

I'm thinking I might want to remove the "SHOULD respond with RST_STREAM even if you've already sent FIN".  However, in the case of a refused push, you *want* the server to not bother retransmitting anything it hasn't already gotten out successfully.  Maybe that's just the wiggle room in the SHOULD, but I'd like something more deterministic.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-273554901

---

From: Mike Bishop <[email-masked]>
Date: Tue, 24 Jan 2017 21:00:45 -0800

Per discussion in Tokyo, RST_STREAM with QUIC_NO_ERROR already has (mostly) these semantics.  (You have to know the byte offset of the end of your data, which typically means already having sent it.)  Slight advantage to having a separate frame (cleaner handling than a special case, can send earlier than the end of your data).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-275019747

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 Jan 2017 21:09:48 -0800

Discussed in Tokyo; some support for a separate frame type, but need to review PR.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-275020754

---

From: Lucas Clemente <[email-masked]>
Date: Tue, 24 Jan 2017 21:31:37 -0800

I just discussed this topic some more with @janaiyengar and @marten-seemann in the coffee break. I think we found a problem with the REQUEST_RESET frame in its current incarnation: If the peer receiving the REQUEST_RESET has not finished sending his data in the meantime (i.e. no FIN), it has to send some kind of RST_STREAM again. Now, the other peer (possibly still sending data, e.g. its 404 page) would also stop sending, which we don't want in this case.

I now think a clearer solution than the REQUEST_RESET frame would be to modify RST_STREAM. It would gain two bits: One indicating whether a offset is present, and one indicating its direction. Then, the first peer in the above model could indicate that it acknowledges the "REQUEST" (and doesn't want its own reset) with the direction bit. In turn, if it really wants a reset, that would work too.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/165#issuecomment-275023226