Subject: Re: [quicwg/base-drafts] Priority in QUIC Transport (#104)

From: Martin Thomson <[email-masked]>
Date: Tue, 03 Jan 2017 21:08:36 -0800

This is worth discussing.  An important lesson from HTTP/2 was that multiplexing is lame without prioritization.

@MikeBishop, I assume that you are thinking one of two things:

1. move the entire prioritization feature to the QUIC layer
2. a more complex arrangement in which stream prioritization is determined by the application layer protocol, but the transport queries the application layer to learn what packets to allocate resources to

I agree that the status quo isn't ideal.  Mostly that's theoretical, because an implementation can add as much cross-talk between layers as it needs to get the job done.  It could mean that retransmissions of packets are made in ignorance of priority.  Speaking of which, we probably need some sort of guidance about how to prioritize those.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270295768

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Jan 2017 10:27:56 -0800

in general moving information down the pipeline as fast as you can without creating a HoL-Blocking problem is a good strategy. The blocking problem is the reason an H2 impl has to hold back data from TCP, and it would seem to correlate somewhat to a congestion controller's desire to not create buffering in the network (via crystal ball).

I think that argues for moving prio to the quic transport layer. mux without pri is indeed lame.. and quic wisely retransmits frames instead of packets, so prio can play a role there if prio is part of the frame otherwise this is all opaque to quic during retransmit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270446893

---

From: janaiyengar <[email-masked]>
Date: Thu, 05 Jan 2017 22:28:00 -0800

Given that QUIC implementations should really not be buffering any/much data, I think an API ought to be adequate. I agree that prioritization is important, but any prio scheme that you bake into QUIC is going to be either a poor fit for the next application. 

QUIC implementations can be (i) encouraged to minimize or not buffer any unsent data, and (ii) strongly encouraged to have an API that allows an application to indicate priorities across all "active streams". This would have the drawback of making priority opaque during QUIC's retransmissions, but I would argue that an application may well consider the priority of a retransmission as higher than that of the original transmission simply because of the amount of time in between the two.   I think that it's reasonable for endpoints to generally prioritize retransmissions over the rest.

In a past life, I implemented priorities in the TCP socket send buffer to prioritize across multiple application "messages". I learned the hard way that simply limiting the send buffer size in TCP and allowing the application to handle priorities above yielded similar benefits. The key was in keeping the TCP send buffer size low.

(When you're worried about prioritizing new data over retransmissions, my intuition tells me that it's either a non-issue or it doesn't matter. It's a non-issue when there are very few retransmissions that it's in the noise, or it doesn't matter since there are too many losses that those dominate latency beyond what priority can help with.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270841122

---

From: Patrick McManus <[email-masked]>
Date: Fri, 06 Jan 2017 07:09:17 -0800

I would argue that keeping the tcp send buffers small is where we currently see a lot of implementation mistakes - I can cite a few open server performance bugs with h2 if you think its worth the time to dig through bugzilla. App developers want to fire and forget when they have data in hand. Small-buffers are apparently not an especially usable interface for application protocols.. so if its plausible to push data down into the transport where you've got a late-binded mux, that would probably yield better results in practice even if in theory there is no advantage.

The argument that quic should not be buffering much data is only obviously true if it isn't doing reordering and cancels as well. It could certainly do all those things and in some senses a protocol like this that already entangles transport, security, and the application is a reasonable place to do it. Embrace the monolith :)

I totally agree that retrans is an unimportant, tho interetsting, corner case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270922727

---

From: janaiyengar <[email-masked]>
Date: Fri, 06 Jan 2017 11:18:45 -0800

I'll take back some of what I said earlier. Since an application's interaction with QUIC uses a stream abstraction, as long as QUIC implementations do a late-binding to the data in the streams, an API that indicates stream priorities ought to be adequate. Data can be buffered in incoming stream buffers from the application to the transport, but we should avoid buffers deeper within the transport (which is natural, I think.) To be clear, I'm not including buffers for reliability in this conversation; those don't interact with priorities much.

On the TCP small buffer point, I think it's difficult to expect high performance if an application simply wants to shoot and forget into a non-prioritized buffer... perhaps your point is that it's still better for TCP (in an alternate Minion world) to provide a priority queue rather than a FIFO queue since that's an easier abstraction for apps to write to. If so, point taken. In this respect, the buffers between the application and QUIC can be seen as a priority queue, and this queue itself could be seen as an API element between the application and QUIC.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270982068

---

From: Patrick McManus <[email-masked]>
Date: Fri, 06 Jan 2017 11:53:34 -0800

conceptually I agree it works best between the app and transport. But then again conceptually the app and transport are intertwined in quic so talking about it doesn't feel radically out of place. If we don't want transport to do anything special (i.e. retransmissions) based on it, then it does become a matter of implementation and api fodder rather than something to worry about in base-drafts I suppose.

but again, seeing huge runs of data of the same resource with maximum frame sizes (obviously buffered in tcp buffers) is a really common quality-of-implementation fail mode for h2 and it has lead to a number of "performance evaluations" claiming h2 performs worse than 1. So if a new protocol can help with that footgun, its only to its own benefit I would think.

One way to do that is to drag priorities down into the transport.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270990219

---

From: ianswett <[email-masked]>
Date: Fri, 06 Jan 2017 12:31:31 -0800

My main concern about putting prioritization into QUIC is that I don't know
what prioritization API would be sufficiently robust for non-HTTP/2
applications as well.  Possibly we could define some pluggable
"prioritization API"?

I want is per-stream send buffers, which allows both optimal prioritization
and a reasonable amount of buffering.  These per-stream send buffers
replace the TCP send buffers for most part.  The number of UDP packets
encrypted and written to the socket at once would be determined by
congestion control and pacing.   Data is retransmitted from these stream
buffers, which allows a stream to be re-prioritized and the retransmissions
to follow the new priority, instead of the original priority.  This also
makes cancellations simpler, because there's nothing to retransmit from the
cancelled stream.  But this could be done with the priorities inside the
transport layer or in the application layer, it's more a matter of what
interface we're designing.

I agree about embracing the monolith.  To make the above implementation
approach work well, a tight coupling makes a lot of sense.

Some pitfalls to be aware we've run into:
 1) Retransmissions in multipath may have smaller MTU, so having a stream
level send buffer is better than just a series of stream frames.
 2) FEC would likely not use retransmission at all.
 3) A zero-copy friendly interface is a good thing.
 4) There are cases when new data is more important than retransmissions,
or one retransmission is more important than another.

Given this seems like it will become clearer after a few more people
implement it, does it make sense to punt this conversation to the future a
bit?

On Fri, Jan 6, 2017 at 11:18 AM, janaiyengar <[email-masked]>
wrote:

> I'll take back some of what I said earlier. Since an application's
> interaction with QUIC uses a stream abstraction, as long as QUIC
> implementations do a late-binding to the data in the streams, an API that
> indicates stream priorities ought to be adequate. Data can be buffered in
> incoming stream buffers from the application to the transport, but we
> should avoid buffers deeper within the transport (which is natural, I
> think.) To be clear, I'm not including buffers for reliability in this
> conversation; those don't interact with priorities much.
>
> On the TCP small buffer point, I think it's difficult to expect high
> performance if an application simply wants to shoot and forget into a
> non-prioritized buffer... perhaps your point is that it's still better for
> TCP (in an alternate Minion world) to provide a priority queue rather than
> a FIFO queue since that's an easier abstraction for apps to write to. If
> so, point taken. In this respect, the buffers between the application and
> QUIC can be seen as a priority queue, and this queue itself could be seen
> as an API element between the application and QUIC.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/104#issuecomment-270982068>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ATJJcWRDEOYgiqcHAMdOQmOzh0sSaKXBks5rPpOVgaJpZM4LWi4b>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-270998565

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 23 Jan 2017 18:39:11 -0800

Discussed in Tokyo; no appetite to make wire changes (e.g., PRIORITY frame), but open question about whether QUIC needs to define how applications can affect priority of sent packets. Different application protocols might have vastly difference prioritisation schemes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-274684894

---

From: Victor Vasiliev <[email-masked]>
Date: Mon, 23 Jan 2017 18:48:06 -0800

I would like to note here that stream scheduling is just like congestion control -- your QUIC implementation *will* have logic to schedule streams, it's just it might be trivial logic.

Unlike congestion control, there are two issues here (from the discussion at the interim): (1) prioritization heavily depends on the interface between the application and transport, and (2) we might not have enough prior experience to draw on in order to provide guidance to the implementations at this point.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-274686151

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 Jan 2017 22:50:52 -0800

As discussed in Tokyo, only advisory text on what implementation might want to do

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/104#issuecomment-275032636