Subject: Re: [quicwg/base-drafts] why require in order delivery? (#252)

From: mirjak <[email-masked]>
Date: Wed, 01 Feb 2017 01:41:39 -0800

Is there even something like out-of-order STREAM delivery? Because my understanding is that if you want out-of-order you need to preserve message boundaries, which is called message-oriented delivery... 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276613685

---

From: Patrick McManus <[email-masked]>
Date: Wed, 01 Feb 2017 05:28:43 -0800

While you can imagine doing 1 stream per video frame if designed especially
for quic, you can also imagine serving legacy video formats over a single
stream (framing inclusive in media type).. sometimes stream offsets are all
the framing you need - raster formats e.g...

this isn't a huge use case - but it seems that the protocol doesn't need to
be prescriptive about in-order delivery to the application modulo anything
we might want to say about flow control. (while at the same time it is
clear that the protocol deals in ordered byte streams).

On Wed, Feb 1, 2017 at 10:41 AM, mirjak <[email-masked]> wrote:

> Is there even something like out-of-order STREAM delivery? Because my
> understanding is that if you want out-of-order you need to preserve message
> boundaries, which is called message-oriented delivery...
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/252#issuecomment-276613685>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAP5s_Ev9pAFO72Bm_ybmnhLpOFuLSynks5rYFNSgaJpZM4LyviU>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276656908

---

From: Mike Bishop <[email-masked]>
Date: Wed, 01 Feb 2017 15:00:12 -0800

Think about it this way:  The bytestream means there are no boundaries internally **that are known to QUIC**.  If you know enough about the contained object to make sense of fragments, you might reasonably skip over missing pieces.  You might even do something really interesting, like early-sending the `moov` box of an MP4 even if it hasn't been web-optimized to place that region at the beginning of the file.  A client might reasonably read enough of the beginning to get the layout, know what offset to seek to, then ask QUIC whether it can read the byte at that offset.

However, doing something like this requires buffer the entire (potentially very large) object at the transport so that the application can seek as much or as little as it needs.

A more transport-friendly style is something like Jana's previous baby, [uTCP](https://www.youtube.com/watch?v=7QUNT5wGUXw), which allowed the undeliverable-due-to-gaps data to be early-indicated to the application if it supported such things.  However, that still requires that there be enough internal context to make sense of it.  Some things, you can reasonably do that with.  An XML-like structure (HTML) could ignore everything until it finds the first open-tag, then parse until that tag closes or the end of the fragment; whatever pre-rendering it does can then be slotted in once it has the full document up to that point and knows where it fits.  There are efforts to do similar things with fragments of JS, images, etc. I believe.

An implementation could certainly choose to make these fragments available, as uTCP did.  Nothing in the protocol stops it, but I question whether there's enough general value to mandate that an implementation expose it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276811111

---

From: ianswett <[email-masked]>
Date: Wed, 01 Feb 2017 15:40:07 -0800

Agreed @MikeBishop .  That's why I think we should have wording that requires exposing an API for an in order of sequence of bytes, but neither require that is the only API, nor require that out of order delivery be made available in all implementations.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276819620

---

From: janaiyengar <[email-masked]>
Date: Wed, 01 Feb 2017 16:44:15 -0800

Two points:
(i) I see the potential use cases, but I can easily see them all solved by
using streams as the framing for messages. Of course you'd need some logic
above in the application to figure out how to order across streams, but
I'll reiterate the point that QUIC streams are supposed to be lightweight,
and they are expected to satisfy the need for message framing.
(ii) I like the clarity of having streams be in-order delivery of bytes.
Not requiring it simply means that some implementations will support an
alternate delivery model making it the app developers' problem to figure
out what to do if the QUIC layer underneath doesn't support it.

uTCP solves a problem, hackishly, that TCP bytestreams are the only thing
we've got. Bear in mind that most of the complexity lies above uTCP (which
was so by design), and this basically duplicated much of the ordering and
de-duplication work that happens in TCP underneath. This requires a complex
data structure above the unordered byte-stream. I would not have done this
hack if we had lightweight streams in TCP.

I'd be much happier with having a clean abstraction that supports
out-of-order delivery of bytes if that's what we want, making it a clean
abstraction that apps can rely on. A USTREAM, if you will. The stream
abstraction is not ossified enough that we need to hack it.

On Wed, Feb 1, 2017 at 3:40 PM, ianswett <[email-masked]> wrote:

> Agreed @MikeBishop <https://github.com/MikeBishop> . That's why I think
> we should have wording that requires exposing an API for an in order of
> sequence of bytes, but neither require that is the only API, nor require
> that out of order delivery be made available in all implementations.
>
> —
> <https://github.com/quicwg/base-drafts/issues/252#issuecomment-276819620>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AKjg1CBC20DSCMYisha8od2FyhmWfgN7ks5rYRfXgaJpZM4LyviU>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276831171

---

From: Patrick McManus <[email-masked]>
Date: Thu, 02 Feb 2017 05:53:14 -0800

The PR #257  is meant to be modest in scope; I hope it reads that way. This is fun to think about, but isn't a big deal.

Based on some offline discussions I've had I think I have a better understanding for the different povs here.. to generalize, some people view a protocol definition as both the interaction between the peers and as an abstract api provided to the application.. a separate view (the one I share) is the definition of how two peers interact to exchange some data and semantics. 

So in the former case you've got a protocol that delivers reliable in order byte streams to applications, and in the latter you've got a protocol that reliably exchanges ordered byte ranges (for which one API would sensibly be a reliable in order byte stream with hol issues).

my thought experiment: if a client is interacting with a normal TCP server, but the client is built from some crazy pile of libpcap for reading and raw_sockets for writing directly to the ethernet driver - is the client still implementing TCP? I would say yes - that's the only way to speak with a TCP server.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-276962571

---

From: janaiyengar <[email-masked]>
Date: Thu, 02 Feb 2017 09:16:41 -0800

I think you're right, and I squarely fit in the first camp.

As experience with TCP has shown, you correctly point out that in-order
delivery is not part of the protocol, which is why Minion was able to do
what it did without violating the spec. The biggest problem with Minion
deployment however was that no OS implemented anything but an sequenced
delivery API. If the abstraction was a sequenced delivery one, the API was
therefore built around it. Historically, the API was attempting to mimic
UNIX file I/O, but the lesson here is that the API ended up defining what
was possible with the protocol. The way that uTCP (the kernel part of
Minion) worked around this was by opening a SOCK_STREAM, but setting a
TCP_UNORDERED socket option. This meant that an application could learn via
the socket option whether uTCP was supported or not. Basically, to get
unordered delivery with TCP, uTCP had to change the API.

We don't standardize APIs, but protocols ultimately define mechanisms for
carrying out those imperatives. The only signal we can provide to
implementers that an API function needs to be supported is by ensuring that
the protocol calls it out explicitly. Otherwise, you have a mix of
implementations that support it and those that don't.

Given that the QUIC protocol isn't ossified yet, I'd yet again strongly
argue for separating these two concepts -- ordered delivery vs. unordered
delivery of bytes -- explicitly. Separating them sends a signal to
implementers that a separate API for unordered delivery is useful, and
applications can then reliably  use it. This is important: effectively
changing a MUST to a MAY in a document technically allows for different
implementations to do different things, but in this case that seems like a
bug, not a feature.

Separately, I'm not convinced that we have an active use case yet. The
assumptions I'm operating on are different from those for TCP evolution
because we are anticipating version revisions and protocol changes to be
possible with QUIC in the future. Without an active use case, we're left
with implementations that are unlikely to support unordered delivery, and
I'd push back against having this in v1.

On Thu, Feb 2, 2017 at 5:53 AM, Patrick McManus <[email-masked]>
wrote:

> The PR #257 <https://github.com/quicwg/base-drafts/pull/257> is meant to
> be modest in scope; I hope it reads that way. This is fun to think about,
> but isn't a big deal.
>
> Based on some offline discussions I've had I think I have a better
> understanding for the different povs here.. to generalize, some people view
> a protocol definition as both the interaction between the peers and as an
> abstract api provided to the application.. a separate view (the one I
> share) is the definition of how two peers interact to exchange some data
> and semantics.
>
> So in the former case you've got a protocol that delivers reliable in
> order byte streams to applications, and in the latter you've got a protocol
> that reliably exchanges ordered byte ranges (for which one API would
> sensibly be a reliable in order byte stream with hol issues).
>
> my thought experiment: if a client is interacting with a normal TCP
> server, but the client is built from some crazy pile of libpcap for reading
> and raw_sockets for writing directly to the ethernet driver - is the client
> still implementing TCP? I would say yes - that's the only way to speak with
> a TCP server.
>
> —
> You are receiving this because you commented.
> <https://github.com/quicwg/base-drafts/issues/252#issuecomment-276962571>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AKjg1O6JwuhEDW1vDVJKCgSgNMgafoaTks5rYd_KgaJpZM4LyviU>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-277021001

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 02 Feb 2017 19:14:50 -0800

I had the same concern when I read that text (and a few other bits of similar text, which need to get changed too if we do this).

Personally -- Jana, I hear your concern, but I think forbidding APIs from exposing out-of-order streams is going too far, and isn't enforceable. Just because we don't immediately see huge use cases doesn't mean they'll never emerge. 

What we want is to encourage convergence of APIs on a minimal set of functionality, and I think the proposed language does that; we needn't ban more exotic uses to achieve it. 

Unless we actually go and create an API (which is outside the scope of our charter), we're going to see some variance in capabilities anyway; consider the discussion on sender priorities and how that went.

Just my .02.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/252#issuecomment-277154066