Subject: [quicwg/base-drafts] Explicit support for on-path calculation of RTT of QUIC flows (#631)

From: Brian Trammell <[email-masked]>
Date: Wed, 14 Jun 2017 07:25:05 -0700

As discussed at the interim, it might be desirable to allow devices on-path observing two sides of a QUIC connection (or the subset of a multipath QUIC connection) to accurately estimate the RTT of the flow, either for in-band operational purposes (AQM) or general monitoring and measurement purposes (cf. an informal survey of IPFIX information elements suggesting multiple vendors shipping devices that perform this measurement for TCP.)

This facility should require bilateral agreement of both endpoints before an RTT signal is available on path; i.e. it should be optional.

This was a large part of the requirement behind #269, #391, and #393, which were expressed in terms of mechanism instead of requirements. #609 contains a proposal addressing this issue which is separate from packet numbering, and is therefore compatible with proposals to make packet numbering inaccessible to on path devices (see #231)

This issue was filed in part to uplevel the discussion on #609 to requirements.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631

---

From: janaiyengar <[email-masked]>
Date: Wed, 14 Jun 2017 12:49:16 -0700

Thanks for filing this issue -- I'd like to move the discussion out of the PR and to this issue. 

On your issue description: I'm not sure that we need to have bilateral agreement. If the mechanism is simple, *and* if we think the information exposed is reasonable, then we could mandate it.

Two issues were raised on PR #609 :

- Maybe use a known/fixed packet type instead of spin bit. I went back and forth on this, and I'm leaning towards using a packet type. Irrespective of greasing, we have packet types, such as Client Initial, that middleboxes will want to identify; this type is no different. We need only specify a single short packet type (with a 4-byte packet number).

- Specify that this signal is per-path. Endpoints will continue receiving and sending packets as usual, so the signal will continuously be rallied between the endpoints despite any path changes. A middlebox that is on one of the two paths will stop or start seeing all packets, and there's nothing about the marked packet that requires it to be the "first" packet. The only place where it *might* matter is with multipath, but we're not there yet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-308538855

---

From: janaiyengar <[email-masked]>
Date: Wed, 14 Jun 2017 12:55:26 -0700

There's another use for this bit/packet type -- it gives a data receiver one RTT sample per round trip, without requiring a PING frame.

BTW, calling it Latency Spin Bit doesn't make sense -- there's positive or negative spin, no latency spin (unless you're thinking that the endpoints could fool the network about path latency with this bit :-))

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-308540372

---

From: Patrick McManus <[email-masked]>
Date: Thu, 15 Jun 2017 06:49:25 -0700

On Wed, Jun 14, 2017 at 3:49 PM, janaiyengar <[email-masked]>
wrote:

> On your issue description: I'm not sure that we need to have bilateral
> agreement. If the mechanism is simple, *and* if we think the information
> exposed is reasonable, then we could mandate it.
>
reasonable is going to be in the eye of the beholder. In this case rather
than needing negotiation it might be as simple as saying "if the endpoint
does not wish to signal the path then always set the bit to 0". Otherwise
you might get less deterministic results :)

-P

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-308734518

---

From: Martin Thomson <[email-masked]>
Date: Thu, 15 Jun 2017 16:33:55 -0700

@janaiyengar,

> Maybe use a known/fixed packet type instead of spin bit. I went back and forth on this, and I'm leaning towards using a packet type. Irrespective of greasing, we have packet types, such as Client Initial, that middleboxes will want to identify; this type is no different. We need only specify a single short packet type (with a 4-byte packet number).

This isn't correct.  Well, unless that you decide that sending shorter packet numbers isn't necessary every other round trip.  Whatever we do here, we're burning a bit on this.

Ian made another point being that this needs to be very clear about how to flip the bit locally.  Packet reordering will cause extra edges if we aren't careful.  Edges need to be driven based on receiving a different bit, but only if the packet number is larger than the last observed bit flip.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-308893184

---

From: Brian Trammell <[email-masked]>
Date: Thu, 15 Jun 2017 23:23:54 -0700

>  In this case rather
than needing negotiation it might be as simple as saying "if the endpoint
does not wish to signal the path then always set the bit to 0".

This works nicely, and doesn't require any coordination to get the property of requiring bilateral cooperation for it to work.

If we decided that we wanted unilateral exposure to work (i.e. that one endpoint could allow an on-path device to calculate its observed RTT and flight size without the cooperation of the other endpoint), then the spin bit could be driven off the transport protocol's control loop, simply flipping the bit once per RTT. This loses the (IMO very nice) property that this proposal has, though, that 

> Edges need to be driven based on receiving a different bit, but only if the packet number is larger than the last observed bit flip.

Yes, although no amount of care will keep signaling from trailing or being otherwise inaccurate on pathologically lossy and reordering prone paths. I need to think about this a bit more, but I think these cases are easy enough to recognize heuristically that we don't need loss and reordering exposure to allow such samples to be detected and used as a "bad path" indication.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-308943852

---

From: ianswett <[email-masked]>
Date: Fri, 16 Jun 2017 07:24:34 -0700

As I mentioned in the PR, I think we can deal with reordering by specifying the spin to be sent is based on the spin of the largest received packet(for a given path) instead of the last received packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309039715

---

From: Brian Trammell <[email-masked]>
Date: Fri, 16 Jun 2017 07:36:31 -0700

largest received packet number?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309042923

---

From: ianswett <[email-masked]>
Date: Fri, 16 Jun 2017 07:43:39 -0700

Yes, the "spin of the packet with the largest received packet number for a given path."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309044908

---

From: janaiyengar <[email-masked]>
Date: Sat, 17 Jun 2017 11:39:08 -0700

@martinthomson,

> Maybe use a known/fixed packet type instead of spin bit. I went back and
> forth on this, and I'm leaning towards using a packet type. Irrespective of
> greasing, we have packet types, such as Client Initial, that middleboxes
> will want to identify; this type is no different. We need only specify a
> single short packet type (with a 4-byte packet number).
>
> This isn't correct. Well, unless that you decide that sending shorter
> packet numbers isn't necessary every other round trip. Whatever we do here,
> we're burning a bit on this.
>

Yes, that's what I said above -- burn bits on having a 4-byte packet
number. This is a 2-byte overhead per RTT, which is fine. We don't have to
burn an entire bit.

Ian made another point being that this needs to be very clear about how to
> flip the bit locally. Packet reordering will cause extra edges if we aren't
> careful. Edges need to be driven based on receiving a different bit, but
> only if the packet number is larger than the last observed bit flip.
>
I don't think this matters (or maybe I'm missing something). I don't think
we need to make this more complicated than it is. A middlebox is basically
recording the time of this signal in one direction, and measuring RTT when
it sees the signal in the opposite direction. I don't see why reordering
matters here... the measured time is clearly a network RTT measurement.

Perhaps you're assuming that the middlebox needs to see the "largest" RTT,
but I think that assumes too much. RTT measurements at endpoints have to be
more careful, since RTT is basically tied entirely to retransmissions, but
I wouldn't presume that a middlebox wants something so specific.

> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/631#issuecomment-308893184>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AKjg1FcijoUilXcXCcxNzPHnH1h4r8VQks5sEb9jgaJpZM4N59cW>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309232548

---

From: janaiyengar <[email-masked]>
Date: Sat, 17 Jun 2017 11:42:51 -0700

>
> As I mentioned in the PR, I think we can deal with reordering by
> specifying the spin to be sent is based on the spin of the largest received
> packet(for a given path) instead of the last received packet.
>

Sorry, I missed this.  As I said in my earlier response, you're assuming
that the middlebox needs to see the "largest" RTT, but I think that assumes
too much. RTT measurements at endpoints have to use the largest RTT, since
RTT is basically tied entirely to retransmissions. I don't think that's
necessarily what a middlebox would want. I would go with the simplest
mechanism. If there's substantial reordering, the middlebox should see two
distinct RTTs over several rounds. The middlebox can deal with anomalies
(by using appropriate filters, min/max/avg/ewma) based on what exactly it
wants to do with the RTT measurement.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309232742

---

From: ianswett <[email-masked]>
Date: Sat, 17 Jun 2017 13:00:22 -0700

Since the algorithm from Christian's PR is not described in this issue, I'll describe it as best I can, because I think there are two different algorithms that people may have in their minds, and I'm worried it's causing confusion.

There are two roles, the initiator, who flips the spin value, and the responder, who reflects the spin value.
1) The initiator sends a value on a flight of packets(ie: 0)
2) The responder reflects back the 0.
3) When the initiator receives the first 0, it starts sending out 1's.
4) The responder reflects back the 1's.
5) The initiator starts sending 0's once it receives a 1, and goto 2.

This means the spin signal may be present on any packet type, though we could decide to make it present only in 1RTT packets if we wanted to conserve packet types.

The reordering issue I was concerned about is if either side sees 1111010000 instead of 111110000 as it was sent, then I was proposing it should treat the packets as though they'd arrived in order, which make the bits sequential again and the responder would reflect 1111000000.  Otherwise any reordering permanently sticks around and at some point, it may look like noise instead of an RTT signal.

The original design I had in mind was that one packet would be marked with the spin bit, initially sent by the connection initiator, and it would ping-pong back and forth.  This design is more friendly to designating a single packet type instead of using a dedicated bit, but it takes more work to deal with packet loss, since one side needs to re-start the signal if the packet with the spin bit set is lost.  On the other hand, reordering doesn't need anything special.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309236816

---

From: Martin Thomson <[email-masked]>
Date: Sun, 18 Jun 2017 15:54:50 -0700

@ianswett, it seems like watching for edges is more robust than having a single-bit signal.

@janaiyengar, if there is any value in having a shorter packet number, then you just proposed halving that value.  Either that or you created incentive not to implement this scheme.  Just spend the bit.  The single packet type only works in Ian's proposed alternative design, but that design isn't anywhere near as robust.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309308373

---

From: Christian Huitema <[email-masked]>
Date: Sun, 18 Jun 2017 20:49:02 -0700

We could certainly work with packet types instead of dedicated bits. The intermediate nodes would need to observe transitions of the first octet instead of looking at just one bit. But then, we also have a proposal to grease the first octet, for example by doing an XOR with a function of the last byte. That type of greasing would randomize the first octet. So we have a choice: if we want to grease the first octet, we need to dedicate a bit that is exempted from the greasing. If we don't then we can use a packet type.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309332099

---

From: Martin Thomson <[email-masked]>
Date: Sun, 18 Jun 2017 21:06:38 -0700

Why do you say that the flip bit needs to be exempt from greasing?  It's not like the intermediary is unable to reverse the greased transform as proposed.  If greasing == encryption then that would be a whole different story of course.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309333858

---

From: Christian Huitema <[email-masked]>
Date: Sun, 18 Jun 2017 21:21:06 -0700

OK, yes we can say that the latency bit should be exempt from "encrypted greasing". If greasing is easily decrypted by middle-boxes, then of course there is no issue. But then, I don't believe that a form of greasing that is easily decrypted is particularly useful. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-309335223

---

From: Christian Huitema <[email-masked]>
Date: Thu, 22 Jun 2017 17:50:48 -0700

By the way, I think I need to change the phrasing in the PR from "latest packet" to "packet with the highest sequence number". Otherwise, a ian pointed out, the mechanism can degenerate. For example, with a Window of 5, the client will send "11111" and the server will echo "11111", after which the client will send "00000", the sender "00000", etc. But suppose that the client sends "11111" and the server receives "111-gap-1". The client sends "00000", but due to the reordering the server receives "101111". Now it is echoing "101111", and the client is sending "010000". And so on. Reordering and gaps create flips in the sequence.

There are natural healing processes such as gaps and ack coalescing, but it is better to avoid the whole problem. If the server only echoes the bit from the highest sequence number received, then the process is much less sensitive to reordering. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/631#issuecomment-310540608