Subject: [quicwg/base-drafts] Latency Spin Bit (#609)

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 00:35:36 -0700

Adding text to the transport spec to describe the "latency spin bit".
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/609

-- Commit Summary --

  * Merge pull request #1 from quicwg/master
  * Adding description of the latency spin bit

-- File Changes --

    M draft-ietf-quic-transport.md (49)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/609.patch
https://github.com/quicwg/base-drafts/pull/609.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609

---

From: Brian Trammell <[email-masked]>
Date: Thu, 08 Jun 2017 00:40:29 -0700

I like this approach, but would like to dig into how it breaks in corner cases.

It seems to me to be quite related to "coloring" based approaches, see e.g. https://datatracker.ietf.org/doc/draft-ietf-ippm-alt-mark/

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307025068

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 00:40:46 -0700

@huitema pushed 1 commit.

f57f7f4  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/112e1f7aae4dd0b493e4fe55a318c5957fe6efc7..f57f7f4198b8f012b2cc05ca6189ccf3bd273067

---

From: hardie <[email-masked]>
Date: Thu, 08 Jun 2017 00:45:13 -0700

I also like this approach, and I think the one key corner case (out-of-order packet arrival) can be handled by description of how to smooth the base RTT signal.  That would allow out of color packets to be seen as out-of-order indications rather than screwing up the RTT calculation. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307026058

---

From: Martin Thomson <[email-masked]>
Date: Thu, 08 Jun 2017 00:49:40 -0700

martinthomson commented on this pull request.

Is this a per-path signal?  I think that it is and that we need more text about that.

> @@ -198,6 +198,8 @@ strengths of QUIC include:
 
 * Version negotiation
 
+* Support for monitoring

I would rather that we not include this section.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-42813843

---

From: Brian Trammell <[email-masked]>
Date: Thu, 08 Jun 2017 00:51:03 -0700

britram commented on this pull request.

 * Version negotiation

in any case "support for monitoring" is much broader than what this is.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r120820433

---

From: MikkelFJ <[email-masked]>
Date: Thu, 08 Jun 2017 00:52:40 -0700

I would suggest to make it optional. There are some privacy issues associated with this allowing to assocate packets with a connection. Though UDP also associates packets, alternate transport layers can obfuscate routing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307027622

---

From: mirjak <[email-masked]>
Date: Thu, 08 Jun 2017 01:03:01 -0700

Instead of using a 'whole' bit, you can also have another packet type for protected packets. And if you define a well know patter that is used, you can even measure loss with it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307029915

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 01:04:05 -0700

huitema commented on this pull request.

 * Version negotiation

If you completely remove the question, then the bit description just appears out of nowhere. But then, yes, we could be specific about latency monitoring, rather than monitoring in general.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r120822667

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 01:09:42 -0700

18434c7  Rename the section to "support latency monitoring"

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/f57f7f4198b8f012b2cc05ca6189ccf3bd273067..18434c77ca9fda6fc0cd8d454147edb3cb854c44

---

From: mirjak <[email-masked]>
Date: Thu, 08 Jun 2017 01:28:10 -0700

Or just have the flag on short packets...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307035781

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 01:34:39 -0700

ab881a9  Adding reference to IPPM draft

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/18434c77ca9fda6fc0cd8d454147edb3cb854c44..ab881a929395d315fca2c66ec363534efdf27a88

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 01:46:08 -0700

I get Martin's point about per path signal, which it indeed is. But then, we don't really have an end-to-end vs path distinction yet. In the current transport draft, there seems to be just one path at a time. I think the distinction will only become apparent when we start specifying QUIC multipath.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307039989

---

From: ianswett <[email-masked]>
Date: Thu, 08 Jun 2017 01:52:24 -0700

ianswett commented on this pull request.

There's no discussion of what to do when marked packets are lost.  I think there needs to be to ensure the signal doesn't die during the connection.

> @@ -294,6 +296,15 @@ QUIC version negotiation allows for multiple versions of the protocol to be
 deployed and used concurrently. Version negotiation is described in
 {{version-negotiation}}.
 
+## Support for Latency Monitoring
+
+Network administrators often find difficult to monitor the behavior

nit: how about "Network administrators have difficulty monitoring ..."?

> @@ -294,6 +296,15 @@ QUIC version negotiation allows for multiple versions of the protocol to be
 deployed and used concurrently. Version negotiation is described in
 {{version-negotiation}}.
 
+## Support for Latency Monitoring
+
+Network administrators often find difficult to monitor the behavior
+of encrypted protocols. Bandwidth can be monitored by observing the
+flow of encrypted packets, but the usal tools for monitoring

usal/usual

> @@ -294,6 +296,15 @@ QUIC version negotiation allows for multiple versions of the protocol to be
 deployed and used concurrently. Version negotiation is described in
 {{version-negotiation}}.
 
+## Support for Latency Monitoring
+
+Network administrators often find difficult to monitor the behavior
+of encrypted protocols. Bandwidth can be monitored by observing the
+flow of encrypted packets, but the usal tools for monitoring
+latency require observing packet numbers and acknowledgements, which
+in QUIC are encrypted.
+QUIC packets include a clear text latency spin bit that enables

nit: "QUIC packets enable monitoring of latency by including a clear text latency spin bit that is reflected between peers."

> @@ -827,6 +848,24 @@ Implementations MUST assume that an unsupported version uses an unknown packet
 format. All other fields MUST be ignored when processing a packet that contains
 an unsupported version.
 
+## Latency Spin Bit
+
+The latency spin bit enables latency monitoring from observation points on
+the network path. This bit is set as follow:
+
+* The connection responder sets the spin bit value to the value of the
+  spin bit in the last packet received from the connection initiator.

What if two packets were received and the first had the spin bit set and the second did not?  I would think the spin bit would still be set by the responder in that case?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-42825003

---

From: Christian Huitema <[email-masked]>
Date: Thu, 08 Jun 2017 05:53:02 -0700

Mirja, I hesitated between the "packet type" design and the "spin bit". I like having Red/Blue packet types, but I am concerned with the interaction with issue 311, "GREASE the packet type octet". If we do grease the packet type, we will need to leave the spin bit in clear text. This is much easier to do with a reserved bit than with the packet type.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307093952

---

From: mirjak <[email-masked]>
Date: Thu, 08 Jun 2017 06:13:44 -0700

Yes, there is dependency on greasing. We could also just grease the long header because the short header has a bunch of flags that can anyway change all the time. I even thought about having only flags with the short header (and no type anymore) because that's nearly what we have already and that makes sense to me (and also looks slightly clearer to me) because all control packets are long headers.  For me the point of the short header is basically a packet where you want to save some of the header bits and use it for payload instead and the flags simply define which part of the header bits you omit. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307099091

---

From: MikkelFJ <[email-masked]>
Date: Thu, 08 Jun 2017 07:58:35 -0700

There is also the option to grease the type but reserve a single ungreased value for monitoring packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307129893

---

From: ianswett <[email-masked]>
Date: Thu, 08 Jun 2017 15:24:47 -0700

I like the 'spin bit' approach personally. 

+## Latency Spin Bit
+

Also, can you add clarifying text about what to do when exiting quiescence?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-43029633

---

From: Martin Thomson <[email-masked]>
Date: Fri, 09 Jun 2017 00:09:02 -0700

> There is also the option to grease the type but reserve a single ungreased value for monitoring packets.

This is unaffected by my favourite form of greasing (xor with the last octet of the unencrypted frame).  We have a choice as to whether or not this is greased, of course.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307313435

---

From: Martin Thomson <[email-masked]>
Date: Fri, 09 Jun 2017 08:04:38 -0700

Thinking about this some more, this bit will flip lots in ways that aren't immediately obvious.  I think that we can safely not grease this.  (We could also grease it, but that makes it a teensy bit less accessible to the devices who we intend to consume it, so not greasing is probably best.)  Either way, that's not a question we need to resolve with the change as it stands.

I still strongly think that we should avoid adding this to the list of transport features, and I think that you should address @ianswett's comments about dropping an entire flight (the answer being just "oh well" here).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307413865

---

From: Martin Thomson <[email-masked]>
Date: Fri, 09 Jun 2017 08:05:51 -0700

Oh, and I really think that this needs to be crisp about being per-path.  I think that we need to explicitly sasy that we reset the signal when the IP or port at either end changes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-307414224

---

From: Christian Huitema <[email-masked]>
Date: Mon, 12 Jun 2017 20:06:59 -0700

+## Latency Spin Bit
+

Ian, there are two references to quiescence in the current draft. One about the Ping frame, "The default is to send a PING frame after 15 seconds of quiescence." The other about flow control, "it is generally considered best to not let the sender go into quiescence if avoidable." But the concept of quiescence is not well defined. Is it just "has not sent packet for a long time"? 
What would you suggest to do?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r121573754

---

From: Christian Huitema <[email-masked]>
Date: Mon, 12 Jun 2017 20:08:05 -0700

0afcbb9  Editorial fixes

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/ab881a929395d315fca2c66ec363534efdf27a88..0afcbb994812c647ecc7e6ec8606e63eb493757c

---

From: MikkelFJ <[email-masked]>
Date: Tue, 13 Jun 2017 00:43:41 -0700

mikkelfj commented on this pull request.

+## Latency Spin Bit
+

> What if two packets were received and the first had the spin bit set and the second did not? I would think the spin bit would still be set by the responder in that case?

If an old packet race ahead this could add noise to signal. I think it would be good to explain how this would work, or alternatively filter out the noise by only flipping when a new highest packet number is received. Noise could probably be used effectively if understood by analyser.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r121603651

---

From: ianswett <[email-masked]>
Date: Tue, 13 Jun 2017 07:06:47 -0700

+## Latency Spin Bit
+

Scratch my comment, I think if we define this correctly, we don't need to talk about quiescence, it'll just naturally work out.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r121687858

---

From: ianswett <[email-masked]>
Date: Tue, 13 Jun 2017 07:13:23 -0700

+## Latency Spin Bit
+
+* The connection initiator sets the spin bit value to the opposite

I'm confused about why it's the opposite?  My mental model is that the bit would be set to 1 on a packet, and then it would be bounced back and forth, so there would only be a single packet marked with the 1 spin bit at any given point in time.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-43736354

---

From: ianswett <[email-masked]>
Date: Tue, 13 Jun 2017 07:24:06 -0700

+## Latency Spin Bit
+

Ok, ekr explained to me out of band that I misunderstood how the spin bit worked.  I think the design here is more robust to loss than what I was thinking of, so I'm happy with it.

However, an overview sentence explaining what it's doing (ie: mark the entire flight with the spin bit, etc) would have prevented my confusion.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r121693241

---

From: janaiyengar <[email-masked]>
Date: Tue, 13 Jun 2017 07:35:18 -0700

Can we please move discussions to issues, and not have them on PRs?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308136689

---

From: Brian Trammell <[email-masked]>
Date: Wed, 14 Jun 2017 07:25:40 -0700

@janaiyengar I filed #631 for that purpose.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308447926

---

From: Brian Trammell <[email-masked]>
Date: Wed, 14 Jun 2017 07:40:15 -0700

And the loss-detection inference ability of this proposal can be discussed in #632.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308452444

---

From: ianswett <[email-masked]>
Date: Wed, 14 Jun 2017 11:33:10 -0700

+## Latency Spin Bit
+

One more comment: I think we need text on packet reordering here.  Possibly this should be the largest packet received instead of the last?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r122029706

---

From: janaiyengar <[email-masked]>
Date: Wed, 14 Jun 2017 12:56:14 -0700

I've moved discussion to the issue (thanks for filing it, @britram).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308540563

---

From: ianswett <[email-masked]>
Date: Wed, 14 Jun 2017 13:09:25 -0700

@janaiyengar To clarify, I think my comments were editorial, and so those still belong here, not in the issue?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308543882

---

From: janaiyengar <[email-masked]>
Date: Wed, 14 Jun 2017 13:14:45 -0700

Ian -- I wasn't responding to your comments specifically, but I wanted to
move general discussion over to the issue.

On Wed, Jun 14, 2017 at 9:09 PM, ianswett <[email-masked]> wrote:

> @janaiyengar <https://github.com/janaiyengar> To clarify, I think my
> comments were editorial, and so those still belong here, not in the issue?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/pull/609#issuecomment-308543882>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AKjg1AlptAKveShqQ2-_WMy689YPEfzFks5sED30gaJpZM4Nzrwr>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308545243

---

From: ianswett <[email-masked]>
Date: Wed, 14 Jun 2017 13:17:12 -0700

Great, thanks for clarifying.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#issuecomment-308545813

---

From: MikkelFJ <[email-masked]>
Date: Sat, 17 Jun 2017 23:09:24 -0700

+## Latency Spin Bit
+

The terminology elsewhere appears to be Client and Server, not Connecton Initiator and Connection Responder, although I prefer the latter since the protocol is not necessarily used in a client-server setup.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-44713418

---

From: Martin Thomson <[email-masked]>
Date: Sun, 18 Jun 2017 15:50:24 -0700

+## Latency Spin Bit
+

"connection initiator" and "connection responder"  are just pretentious ways of saying client and server.  We happily use client-server protocols in non-client-server arrangements all the time.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r122602784

---

From: Christian Huitema <[email-masked]>
Date: Sun, 18 Jun 2017 18:21:56 -0700

+## Latency Spin Bit
+

Martin, initiator and responder are certainly equivalent to client and server when the application is HTTP. Not necessarily so is other applications, e.g. DNS or Web RTC. But as you seem to feel strongly about it, lets use Client and Server in the PR to make it consistent with the rest of the document.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#discussion_r122607904

---

From: Christian Huitema <[email-masked]>
Date: Sun, 18 Jun 2017 18:31:36 -0700

981d9e2  Changes suggested by Martin and Ian

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/0afcbb994812c647ecc7e6ec8606e63eb493757c..981d9e2c7aa2b4d38710616f44174f552c313116

---

From: Christian Huitema <[email-masked]>
Date: Mon, 26 Jun 2017 16:42:50 -0700

47b685c  Largest sequence number

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/609/files/981d9e2c7aa2b4d38710616f44174f552c313116..47b685c7f071a10170b70a0dc816315353688bdb

---

From: ianswett <[email-masked]>
Date: Mon, 26 Jun 2017 17:53:27 -0700

Thanks Christian, I think this LGTM.

 * The client sets the spin bit value to the opposite
-  of the last value received from the server, or to 0
+  of the value set in the packet received from the server with the
+  largest sequence number, or to 0

nit: I think these two lines may fit on one line?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/609#pullrequestreview-46421343