Subject: [quicwg/base-drafts] improve description of flow control (#150)

From: Marten Seemann <[email-masked]>
Date: Fri, 13 Jan 2017 10:35:41 -0800

I made a couple of changes regarding flow control. The most important change is 4.

1. WINDOW_UPDATEs should be sent when the application consumed data sent by the peer, they should not be based on the highest received offset.
2. I specified the error that should be sent if the peer violates the advertised flow control windows.
3. I explicitly allow an optimization both Chrome and quic-go are already doing: WINDOW_UDPATEs are always sent twice. This increases the probability that the peer receives the frame and can continue sending data, even if one of the two packets is lost or delayed.
4. The description of connection-level flow control was incomplete, since not all streams contribute to the connection-level flow control window. Furthermore, the receiver now has to check for connection-level flow control violations.
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/150

-- Commit Summary --

  * window updates should be based on data consumed, not data received
  * specify error code for flow control violations
  * allow sending a WINDOW_UPDATE frame multiple times
  * correctly describe connection-level flow control

-- File Changes --

    M draft-ietf-quic-transport.md (35)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/150.patch
https://github.com/quicwg/base-drafts/pull/150.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/150

---

From: janaiyengar <[email-masked]>
Date: Fri, 13 Jan 2017 16:36:31 -0800

janaiyengar requested changes on this pull request.

Thanks for the changes! Just one question though: WINDOW_UPDATE frames are currently retransmitted when detected as lost (this is noted in the reliability section), which ensures that the sender receives it. Why multiple copies then?

>  A sender MUST send BLOCKED frames to indicate it has data to write but is
 blocked by lack of connection or stream flow control credit.  BLOCKED frames are
 expected to be sent infrequently in common cases, but they are considered useful
 for debugging and monitoring purposes.
 
 A receiver advertises credit for a stream by sending a WINDOW_UPDATE frame with
-the StreamID set appropriately.  A receiver may simply use the current received
-offset to determine the flow control offset to be advertised.
+the StreamID set appropriately. A receiver may use the current offset of data
+consumed to determine the flow control offset to be advertised.
+A receiver MAY send copies of a WINDOW_UPDATE frame in multiple packets in order
+to make sure that the sender receives it even if one of the packets is lost. A
+sender MUST ignore duplicate WINDOW_UPDATE frames.

WINDOW_UPDATE frames are considered "retransmittable", meaning that they need to be reliably delivered, and the sender will retransmit as necessary. Why isn't that enough?

 Connection flow control is a limit to the total bytes of stream data sent in
-STREAM frames.  A receiver advertises credit for a connection by sending a
-WINDOW_UPDATE frame with the StreamID set to zero (0x00).  A receiver may
-maintain a cumulative sum of bytes received cumulatively on all streams to
-determine the value of the connection flow control offset to be advertised in
-WINDOW_UPDATE frames.  A sender may maintain a cumulative sum of stream data
-bytes sent to impose the connection flow control limit.
+STREAM frames on all streams contributing to connection flow control.  A
+receiver advertises credit for a connection by sending a WINDOW_UPDATE frame
+with the StreamID set to zero (0x00).  A receiver maintains a cumulative sum of
+bytes received on all streams contributing to connection-level flow control to
+check for flow control violations. A receiver may maintain a cumulative sum of

add comma: "contributing to connection-level flow control, to check for flow control violations."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/150#pullrequestreview-16678989

---

From: Marten Seemann <[email-masked]>
Date: Fri, 13 Jan 2017 19:29:11 -0800

marten-seemann commented on this pull request.

It's an optimization. The WINDOW_UPDATE will get retransmitted and delivered **eventually** (depending on the details of the loss detection). A lost WINDOW_UPDATE means that the sender might get flow-control limited, either for one stream or even for the whole connection (if it was a connection-level WINDOW_UPDATE).

Clearly, this is an optimization, so I wrote that a sender MAY find a suitable algorithm. I just wanted to make sure that a client is required to ignore duplicate WINDOW_UPDATEs, since the only other statement about multiple WINDOW_UPDATEs is the following
>  A sender may receive WINDOW_UPDATE frames out of order; a sender MUST therefore ignore any reductions in flow control credit.

which is not explicit about duplicates.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 15 Jan 2017 13:41:50 -0800

martinthomson commented on this pull request.

This seems like a reasonable optimization, particularly if you are trying to balance resource allocation and performance needs.  Sending multiple WINDOW_UPDATEs over time, (maybe not with exactly the same content), ensures that they are received.  Consider it a poor-mans FEC that is just for WINDOW_UPDATE.

Rather than insisting that duplicate WINDOW_UPDATE frames be ignored, I would frame this differently.  A WINDOW_UPDATE that doesn't move the window forward can be ignored.  This might happen if you receive an out of order WINDOW_UPDATE, as well as if the other side was aggressively sending the same value.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 15 Jan 2017 13:43:48 -0800

"A receiver *maintains* a cumulative sum..."  The word "may" implies something else.  (Oh, this is original text..., I'll follow up separately then).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/150#pullrequestreview-16724277

---

From: janaiyengar <[email-masked]>
Date: Sun, 15 Jan 2017 16:21:34 -0800

janaiyengar commented on this pull request.

Yeah, I think this is a nice optimization. +1 to Martin's point about ignoring "old" WINDOW_UPDATE frames.

The PR doesn't quite capture the point that a sender must ensure that a WINDOW_UPDATE frame is delivered to the receiver, which the draft currently guarantees by making the frame reliable. What you want is that the most recent value be the one that is transmitted. 

We could define retransmission of control frames as allowing newer control information. This would apply to WINDOW_UPDATE, PING, RST_STREAM, CONNECTION_CLOSE frames, but the only place it would be really useful is in WINDOW_UPDATE. WDYT?

Beyond that, we can allow senders to send multiple WINDOW_UPDATE frames if they want, since, as you suggest in your PR, the receiver MUST ignore "old" ones.

---

From: Marten Seemann <[email-masked]>
Date: Sun, 15 Jan 2017 18:26:00 -0800

I like the sentence that @martinthomson suggested: "A WINDOW_UPDATE that doesn't move the window forward can be ignored".

@janaiyengar This PR doesn't change the part of the draft that says that WINDOW_UPDATEs MUST be retransmitted. In #114 we're having a discussion about relaxing this requirement in cases where the receiver can be sure that the sender already received a WINDOW_UPDATE with a greater offset. I really like that idea, but I think we should keep it out of this PR.

---

From: Marten Seemann <[email-masked]>
Date: Sun, 15 Jan 2017 18:52:59 -0800

@marten-seemann pushed 3 commits.

a369517  add missing comma
c5f1049  improve condition when to ignore WINDOW_UPDATE
446d1bd  clarify the purpose of sending a WINDOW_UPDATE more than once

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/150/files/7ded7534c44b042c3746f8ce44ac2c667a2153dc..446d1bd2e2d1b1bb148b89f06dd449690f41e177

---

From: Marten Seemann <[email-masked]>
Date: Sun, 15 Jan 2017 18:54:06 -0800

I just pushed some changes based on the feedback.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/150#issuecomment-272762711

---

From: Martin Thomson <[email-masked]>
Date: Sun, 15 Jan 2017 19:19:13 -0800

I think that ACK and WINDOW_UPDATE should form a new class of frames, those that are cumulative.  These require that senders generate another frame if they are identified as being lost, but they don't require verbatim retransmission.

Looking at the other frames in the list there:

It doesn't make a lot of sense to retransmit PING as though it were something that needed repair (treat the ping as failed and maybe try again, but don't use automated processing for it for risk of having the overall time measured.  It's possible you might want to send multiple PING frames for redundancy within an RTT.

RST_STREAM clearly needs repair and repeating it verbatim is probably the only rational choice.  CONNECTION_CLOSE is probably the same, though GOAWAY might fall into the cumulative bucket as well.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 31 Jan 2017 19:04:29 -0800

@janaiyengar, can you merge this please?  I'll resolve conflicts for you.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/150#issuecomment-276560842