Subject: [quicwg/base-drafts] Public Flags to Aid Troubleshooting (#279)

From: martinduke <[email-masked]>
Date: Thu, 09 Feb 2017 10:35:02 -0800

Something that came up in Tokyo was the need for service providers (i.e. middleboxes) to troubleshoot performance issues in the network. Packet Number Echo (#269) could partially address this problem., particularly for RTT measurement. But it is imperfect for measuring other problems.

As someone who has spent quite a bit of time debugging TCP packet traces from service providers, there are two other things that would help greatly in answering the question "why is my QUIC slow", without significantly compromising privacy. To do both would take two bits in the public flags, which may be available depending on how some other issues turn out.

(1) QUIC endpoints MAY set the BLOCKED bit if they include a BLOCKED frame in the packet. If you want to get fancy, you could have a different bit for "stream blocked" and "connection blocked", but flow control issues are a common cause of traffic pauses with no other discernable symptoms.
(2) QUIC endpoints MUST (SHOULD?) set the RETRANS flag in any packet that contains retransmitted data. This would help operators identify that a connection is slowing down because it is responding to loss, and would allow for very reliable loss statistics for the downstream network.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279

---

From: ianswett <[email-masked]>
Date: Thu, 09 Feb 2017 10:46:45 -0800

The blocked bit is an interesting idea.

The retrans bit has a problem, which is that a QUIC connection may not retransmit the same thing or may not retransmit anything at all.  However, if we make it a LOSS_DETECTION bit, and say it mean "this bit means I detected a packet loss", I believe it conveys the exact signal you're hoping to achieve, and typically it would end up being the retansmit bit.

<Warning, bad idea ahead> If we were feeling clever, we could abuse the packet echo bit for the loss detection case by specifying half the packet number space as valid for echo and the other half for delta to lost packet number.  The only reason this would be useful is to allow us to create time-sequence diagrams without decrypting the payload.

The worst part about abusing the packet echo bit is that consuming extra bytes to indicate a packet number means we can fit less data into packets, and may cause a retransmission to be split into two packets.  I suspect this may be enough hassle implementations wouldn't bother implementing it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278734918

---

From: martinduke <[email-masked]>
Date: Thu, 09 Feb 2017 11:35:19 -0800

LOSS_DETECTION sounds good, though we have to set the bit for each lost packet in a burst loss, or we have the same problem of incomplete data. So in effect, this is the same as the "RETRANS" bit except we have to find a packet to put it on for unretransmittable packets.

The "bad idea" sounds complicated, but I'm open to discussing it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278748261

---

From: mirjak <[email-masked]>
Date: Thu, 09 Feb 2017 11:47:28 -0800

Yes, I agree, blocked it interesting. I didn't think about blocked bit yet but I'm also not sure if I understand the use case fully (see below). 

All in all, my first reaction, however, in-line with ian's comment about naming of the retrans/loss bit, is that this exposes the wrong semantics. Rather than 'just' exposing anything we do in the transport, we should consider what's the information the network actually wants/needs.

In case of loss/retrans, this is probably inline of what ConEx is/was trying to do. We called this whole-path-congestion and it was only based on ECN/CE (not loss) because loss was already exposed in TCP when monitoring retransmissions. I've been also thinking about exposing the ECN counters and potentially a loss counter, however, I wasn't sure if the overhead is worth the win. So the signal the network actually needs is, I think, 'I saw congestion and that's the reason why I'm slowing down (congestion control)'; I guess this could be accounted by using one bit for both together loss or ECN/CE (if the packet rate is high enough to signal multiple losses in a timely manner).

Regarding the blocked bit, I think this is related. My understanding of the use case is that you (the network) sees that a flow is slowing down/sending less data than previously or than possible and the network wants to distinguish the case where it's limited by the endpoint (flow control) or there is something wrong in the network. 
However, there are actually (at least?) three cases how a connection can be limited:
1) limited by the application itself: there is simply no more data to send
2) limited by the flow control in the endpoint's transport
3) limited by a bottleneck in the network (and respectively congestion window)
I guess what you'd like to expose it the difference between 3 and 1/2, right? Or would it be also helpful to expose the difference between 1 and 2 e.g. for wireshark like debugging at the endpoint? Then I guess we would need two bits...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278751540

---

From: ianswett <[email-masked]>
Date: Thu, 09 Feb 2017 12:26:15 -0800

I think the blocked bit is less useful than I originally thought.  I think connections are in two states of the world:
1) Not blocked
2) Blocked, but you'd have to have access to the encrypted context to understand why.
  - ie: A stream is blocked because of a slow server backend?  

TCP flow control is not truly end-to-end, unlike QUIC.  And so I think it means it needs to be debugged end to end.  Also, the path has no control over flow control in QUIC, so I don't think it provides actionable data for a network operator.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278762685

---

From: martinduke <[email-masked]>
Date: Thu, 09 Feb 2017 12:28:42 -0800

> Also, the path has no control over flow control in QUIC, so I don't think it provides actionable data for a network operator.

I agree that there is no direct action that the operator can take, but it is a very clear signal that there is no action to take. It is useful as a "stop worrying" bit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278763434

---

From: martinduke <[email-masked]>
Date: Thu, 09 Feb 2017 15:38:47 -0800

> We called this whole-path-congestion and it was only based on ECN/CE (not loss) because loss was already exposed in TCP when monitoring retransmissions

I think there is a case that the ECE/CWR bits should be in the public flags, rather than the ACK frame, and this discussion is (sort of) happening in the ECN issue #68.

> 1. limited by the application itself: there is simply no more data to send
> 2. limited by the flow control in the endpoint's transport
> 3. limited by a bottleneck in the network (and respectively congestion window)

#1 is hard to detect, even in TCP, and tends not to bring customer complaints. So I think it's really down to 2 or 3, and the BLOCKED bit would handle this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/279#issuecomment-278811624