Subject: [quicwg/base-drafts] Change RST_STREAM impact on state (#459)

From: Martin Thomson <[email-masked]>
Date: Thu, 20 Apr 2017 18:08:03 -0700

The behaviour of RST_STREAM was fine, but it's description created several problems.  Most of these problems manifested in a convoluted description of the closed state.  The same problem existed in HTTP/2, but it was less of a problem there because of the strict ordering provided by TCP.

This refactors the state machine to recognize that RST_STREAM is really unidirectional.  Both sides need to send it or flow control gets screwy.

So people can understand, here's the old state machine picture:
```
                            +--------+
                            |        |
                            |  idle  |
                            |        |
                            +--------+
                                 |
                                 | send data/
                                 | recv data/
                                 | recv higher stream
                                 |
                                 v
                            +--------+
                recv FIN    |        |    send FIN
                  ,---------|  open  |-----------.
                 /          |        |            \
                v           +--------+             v
         +----------+            |             +----------+
         |   half   |            |             |   half   |
         |  closed  |            | send RST/   |  closed  |
         | (remote) |            | recv RST    |  (local) |
         +----------+            |             +----------+
             |                   |                    |
             | send FIN/         |          recv FIN/ |
             | send RST/         v          send RST/ |
             | recv RST     +--------+      recv RST  |
             `------------->|        |<---------------'
                            | closed |
                            |        |
                            +--------+
```

The new one is a little simpler:
```
                            +--------+
                 recv RST   |        |    send RST
              ,-------------|  idle  |---------------.
             /              |        |                \
            |               +--------+                 |
            |                    |                     |
            |        send STREAM / recv STREAM         |
            |                    |                     |
            |                    v                     |
            |    recv FIN/  +--------+    send FIN/    |
            |    recv RST   |        |    send RST     |
            |     ,---------|  open  |-----------.     |
            |    /          |        |            \    |
            v   v           +--------+             v   v
         +----------+                          +----------+
         |   half   |                          |   half   |
         |  closed  |                          |  closed  |
         | (remote) |                          |  (local) |
         +----------+                          +----------+
             |                                        |
             |   send FIN/  +--------+    recv FIN/   |
              \  send RST   |        |    recv RST   /
               `----------->| closed |<-------------'
                            |        |
                            +--------+
```

I've built this on top of #450 to make my life easier in the case that #450 is accepted (in other words, I'm betting on it being accepted).

Closes #458.
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/459

-- Commit Summary --

  * Split WINDOW_UPDATE into MAX_DATA and MAX_STREAM_DATA
  * The hazards of renaming things automatically
  * Fix IANA table
  * Jana's review comments
  * Fix frame type numbers
  * More feedback from Jana
  * wip
  * Make RST_STREAM affect one direction only

-- File Changes --

    M draft-ietf-quic-transport.md (609)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/459.patch
https://github.com/quicwg/base-drafts/pull/459.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459

---

From: Martin Thomson <[email-masked]>
Date: Thu, 20 Apr 2017 18:22:02 -0700

@martinthomson pushed 1 commit.

83fb6c9  Fixup

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/459/files/eb95bf65e4af280232924f471d6ad5eebcbdaa86..83fb6c93ec2feccd2b7eb9c43ee0cc3959919cbc

---

From: Mike Bishop <[email-masked]>
Date: Fri, 21 Apr 2017 09:45:04 -0700

MikeBishop commented on this pull request.

This is effectively the first layer of unidirectional streams we had already been talking about -- an equivalent way to state this is that streams are pairs of unidirectional bytestreams, each of which moves through the idle->open->closed cycle independently.  "Half-closed" is a shorthand to denote a stream in which one bytestream is closed and the other is not.

When I did my scratch branch of what that would look like, I also divided the frames into "stream-independent," "sender," and "receiver" frames.  For each of those unidirectional byte streams, there are frames that the sender generates (STREAM, BLOCKED, RST_STREAM) and frames that the receiver generates as feedback (WINDOW_UPDATE, REQUEST_RST).  As long as the stream is open in that direction, those frames are valid to flow.  When the stream closes in that direction, those frames are no longer needed/valid.

 ### half-closed (local)
 
 A stream that is in the "half-closed (local)" state MUST NOT be used for sending
-STREAM frames; MAX_STREAM_DATA and RST_STREAM MAY be sent in this state.
-
-A stream transitions from this state to "closed" when a STREAM frame that
-contains a FIN flag is received and all prior data has arrived, or when either
-peer sends a RST_STREAM frame.
+of new STREAM frames.  Retransmission of data that has already been sent on

Nit:  Do we send data "on" STREAM frames?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#pullrequestreview-34070430

---

From: hardie <[email-masked]>
Date: Fri, 21 Apr 2017 10:45:57 -0700

While I agree with Mike that this design could be the fist step toward a unidirectional stream model, it seems to met that it doesn't make sense to switch to that model yet.  Unless/until we get to a point where we stream types that are not always matched by a stream in the "reverse" direction, I think "half-closed" will be more accessible terminology.  The unidirectional stream approach implies capabilities/characteristics that I don't think we're using (yet).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-296257574

---

From: Mike Bishop <[email-masked]>
Date: Fri, 21 Apr 2017 10:55:08 -0700

@hardie, maybe a separate discussion, but there are stages, and we decide how far we want to go.

1. Stream are bidirectional bytestreams with a single state (idle, open, half-closed, closed)
2. Stream are pairs of unidirectional bytestreams, each of which has a state (idle, open, closed)
3. Streams are unidirectional bytestreams, each of which has a state, and any association between streams in opposite directions needs to be defined and communicated by the app protocol.

I agree that (3) would be a huge departure.  Potentially useful -- but we're not there yet, if ever.  (2), I'm thinking it might be time for.  I don't think that's actually a harder mental model -- if anything, it simplifies the state machine.  The current state machine is really just a cross-product of the two.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-296259855

---

From: hardie <[email-masked]>
Date: Fri, 21 Apr 2017 13:17:14 -0700

@MikeBishop  The key question seems to be the extent to which one stream's state can be independent of its peer's.  In model 1, they are tightly coupled.  In model 2, an open stream may be paired with a related stream that is open or idle, but it may not be paired with one that is closed (or you have a unidirectional stream with no way to reset it at the transport layer).  In model 3, a stream may be linked with nothing or to one or more streams and is up to the application to sort it all out.

I think we are still at model 1, even with this description change.  I'm not actually sure I see the advantage of switching to model two unless we believe either that it is a stepping stone to 3 or that there are applications where open streams paired with idle streams makes sense.  I don't have a personal example of that, so it's coming down to "stepping stone to 3" for me.  That's why I'm at "not yet".

I'm hapy to hear if I'm missing something, though.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-296295737

---

From: MikkelFJ <[email-masked]>
Date: Sat, 22 Apr 2017 02:53:43 -0700

I strongly favor uni-directional streams as in model 3. It may be hard to argue for model 3 when considering a classic request reply model, but if implementing an application protocol agnostic async API for QUIC, the API becomes much simpler in model 3 and will consequently be much less error prone and have a smaller attack surface. If sending a request, the stream can be closed immediately and torn down after sending FIN without waiting for the peer - state is then only needed for retransmission.

There are many use cases where there is no direct request/reply pattern, for example the pub/sub pattern where you subscribe on one write stream, handle errors on a read stream, and have a separate read stream for each subscribed topic. Logging and system monitoring also falls into this category.

I also much approve of the new RST_STREAM behavior, but it has the downside that it is not possible to unilaterally force a stream to be fully closed because DISINTEREST is only advisory.

It is possible to implement to implement silent close at the api level if the receiver accepts a peer stream as either R/W or RO and an initiator creates a stream as R/W or WO. Then a RST or FIN is not needed nor expected on the one half of a stream and consequently resources can be freed earlier, and an error can be issued if the expectations are violated. However, this is much more complicated to manage that uni-directional streams.

If a uni-directional approach is chosen, as in model 3, I believe the receiver should have the power to force the stream to be closed and be able to error the connection after a timeout.

The main problem with handling RPC request/reply over unilateral streams is that the request cannot specify which stream the reply should appear on, unless multiplexing replies on known streams. A requester therefore needs to send a request context and look up that context on any new peer initiated reply stream. Since a new peer stream may have several purposes, this further complicates the lookup since both the stream purpose and request context must now be identified. This is an argument for keeping bi-directional streams. I just don't think it is worth the complication.

Model 2 might have some merit in solving the above RPC problem, but I haven't given this a lot of thought. My intuition tells me that model 2 is largely the same as model 1 with the unidiretional RST_STREAM behaviour, and still has a lot of state complexity.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-296362049

---

From: MikkelFJ <[email-masked]>
Date: Sat, 22 Apr 2017 03:35:55 -0700

More thoughts on uni-directional RPC pattern:

There could be a REPLY frame type which is similar to a STREAM frame with offset 0 but instead of storing an offset it stores the STREAM ID of the request stream it is replying to. The request stream is an ordinary stream. The replying endpoint must have observed request stream in OPEN state, but the request may be open or closed when replying, and the requesting endpoint need not maintain any state for the request stream. In other words, the link between the two is very weak and unknown to the request stream.

The same could have been obtained at the application layer by carrying a context id for the request, but having the support at transport level has the benefit of automatic ID assignment and no special framing requirements. A REPLY is trivially identified as such.

It is possible to have many logical replies to the same request stream, and there could be many logical request coming in on the request stream. But these are ordinary stream frames with offsets. (This somewhat ties into partially reliable streams with message boundaries, but that is a separate concern).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-296364431

---

From: Martin Thomson <[email-masked]>
Date: Tue, 25 Apr 2017 15:59:52 -0700

49e8ac1  Nit

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/459/files/c3e42c7f6253d81a24234d8f5474e196633aadad..49e8ac15369e3f4ca8e82617451512281d341126

---

From: Martin Thomson <[email-masked]>
Date: Wed, 26 Apr 2017 19:24:53 -0700

Fair warning: I plan to merge this in about 24 hours.

@RyanAtGoogle, I think that @janaiyengar had you on the hook for double-checking this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-297593537

---

From: Martin Thomson <[email-masked]>
Date: Wed, 26 Apr 2017 19:45:25 -0700

3ccf3d8  Move final offset discussion to new section

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/459/files/dd2344a7960feea31886d618d3f7128133648092..3ccf3d819385416d5f2fba10955b38040f825eff

---

From: Martin Thomson <[email-masked]>
Date: Wed, 26 Apr 2017 19:47:25 -0700

This also closes #179 by virtue of correcting the errors in describing the final offset.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#issuecomment-297596427

---

From: Martin Thomson <[email-masked]>
Date: Thu, 27 Apr 2017 22:48:36 -0700

Merged #459.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/459#event-1061575898