Subject: [quicwg/base-drafts] Padding ACK-only packets (#837)

From: Martin Thomson <[email-masked]>
Date: Thu, 05 Oct 2017 10:30:01 -0700

PADDING frames count toward bytes-in-flight, so they can't be included in ACK-only packets.  This makes it impossible to pad ACK-only packets.  This could prevent some forms of traffic analysis mitigation strategies.

@ekr observes that you could pad with more ACK frames, but that's clunky.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837

---

From: ekr <[email-masked]>
Date: Thu, 05 Oct 2017 10:36:42 -0700

Note that if you padded at the crypto layer (see issue #197) then this woud not be a problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-334537647

---

From: ianswett <[email-masked]>
Date: Thu, 05 Oct 2017 11:47:52 -0700

@ekr Are you suggesting we remove PADDING from the QUIC layer entirely?  If we needed padding for ACKs, we would have the transport request padding be sent by the crypto layer on stream 0?

That seems really awkward given the simplicity of QUIC padding, and data on stream 0 still counts towards congestion control, so it'd be functionally equivalent.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-334557136

---

From: ekr <[email-masked]>
Date: Thu, 05 Oct 2017 18:52:47 +0000 (UTC)

The way that TLS-style padding works is that it's effectively at the AEAD
layer. You pull it off before you get to the payload.

On Thu, Oct 5, 2017 at 11:47 AM, ianswett <[email-masked]> wrote:

> @ekr <https://github.com/ekr> Are you suggesting we remove PADDING from
> the QUIC layer entirely? If we needed padding for ACKs, we would have the
> transport request padding be sent by the crypto layer on stream 0?
>
> That seems really awkward given the simplicity of QUIC padding, and data
> on stream 0 still counts towards congestion control, so it'd be
> functionally equivalent.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/837#issuecomment-334557136>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABD1oZUZiydOYXEM_mKV-3NjBuJNthwaks5spSRWgaJpZM4PvbtM>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-334558535

---

From: Christian Huitema <[email-masked]>
Date: Fri, 06 Oct 2017 08:44:13 -0700

I do not like the concept that ACKs do not count for bytes in flight. With 255 ranges, ACKs can be almost 800 bytes long, and it is certainly possible to congest a link with just ACK. I understand that we want to be able to send ACK even when blocked by CC, but that does not mean an infinite number of ACK. I would prefer something like extra credits for ACK when blocked by CC, in a controlled way.

I like even less the idea that adding padding to a packet will make acknowledgements mandatory.

And the congestion control argument requires counting padding for CC -- whether included at the framing layer or the AEAD layer. Otherwise, you can send lots of bytes that don't count for CC, and indeed congest the link.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-334792906

---

From: ianswett <[email-masked]>
Date: Fri, 06 Oct 2017 18:38:46 +0000 (UTC)

A suggestion from yesterday's lunch (I believe from @martinduke ) Was to allow a 'small' ack not be congestion controlled(ie: 3 blocks) to ensure forward progress, but require everything else to be.  A different option we discussed was count acks towards congestion control/bytes in flight, but don't prevent them from being sent immediately.

I think Ack congestion control is still a fairly open problem, so I think we'd need to try some of these ideas in the wild if we think they're worth seriously considering for QUIC.  

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-334837065

---

From: Martin Thomson <[email-masked]>
Date: Sun, 08 Oct 2017 16:05:09 -0700

I share Christian's unease, and would prefer if we could take a serious look at the two options that @ianswett identified.  I like the combination: exempt packets from the block, but only if they a) contain ACK frames, and b) are small.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335045428

---

From: Christian Huitema <[email-masked]>
Date: Sun, 08 Oct 2017 21:57:20 -0700

I got my computation slightly wrong. ACK can include 255 blocks. Each block can be up to 9 bytes (mm=3), which means 2295 bytes for 255 blocks. I can't fathom why this would not be included in the bytes-in-flight. And yes, I like the option that @ianswett described. Count the ACK as bytes in flight, but provide extra credits for a couple of small ACK on top of the CWIN.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335066839

---

From: MikkelFJ <[email-masked]>
Date: Sun, 08 Oct 2017 23:57:09 -0700

So what prevents a peer to send lots and lots of small ACK packets? The current form appears to be honor based, and allowing small packets without flow would continue to do so, unless there is some sort of rate limit involved. Small ACK block exception adds some level of complexity.

We don't even know what is supposed to be sent in ACK frames when there is more to be sent than fits in a single frame: https://github.com/quicwg/base-drafts/issues/423 https://github.com/quicwg/base-drafts/issues/764

So what exactly is the problem that ACK flow control is supposed to solve?
(Not saying it is not needed, just that it would be helpful to understand the underlying problem better).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335077987

---

From: Martin Thomson <[email-masked]>
Date: Mon, 09 Oct 2017 01:13:35 -0700

What would prevent abuse is a further restriction on the number of ACK-only packets that are permitted to be exempted from the limit (I forgot that bit).  That limit might be expressed in terms of number of packets per RTT, or something like that.  We probably need someone with more congestion controller insight to tell us what a sensible limit might be, of course.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335090769

---

From: mirjak <[email-masked]>
Date: Mon, 09 Oct 2017 04:03:48 -0700

I guess you could see acks as non congestion control transmissions and apply https://tools.ietf.org/html/rfc8085#section-3.1 which means to have a rate limit and a circuit breaker.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335126229

---

From: ianswett <[email-masked]>
Date: Mon, 09 Oct 2017 13:55:04 +0000 (UTC)

To be clear, an abusive implementation can run without congestion control and send a lot more data than just some extra acks.

I believe the concern needs to be whether correct implementations could inadvertently create pathological behaviors, either for themselves or others.

The sender and receiver already have an incentive to send as few and as small acks as possible, so there's no need to ensure some sort of 'fairness'.

The worst part about including ack only packets in bytes in flight is two competing needs:
1) Do not send an ack frame in response to an ack only packet.
2) We need a way to remove frames from bytes in flight, which is to ack them.

Martin, I would say any less than 4 acks per RTT risks degrading performance.  That being said, it's going to depend on the congestion controller.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335165361

---

From: MikkelFJ <[email-masked]>
Date: Mon, 09 Oct 2017 13:58:43 +0000 (UTC)

The point is that an abused peer has a clear measure of when to close a connection and possibly blacklist in IP. If there are many other places where this isn't controllable, perhaps it should be. I'm also a bit concerned about abuse during connection establishment and when/how to drop packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335166421

---

From: Christian Huitema <[email-masked]>
Date: Mon, 09 Oct 2017 17:07:14 +0000 (UTC)

As pointed out by others, Congestion Control is effectively a sender's choice. What we are writing here is a recommendation, such as "do this and life will be smooth and you will not trip on your own shoelaces." There is no need to get tied up in knots of philosophy, and certainly no need to request unnatural behavior such as "must acknowledge packets if they contain padding". I personally favor three simple rules:

1) Bytes in flights count all bytes in flight, regardless of frame types. 

2) ACKs are not subject to congestion control, but they should be sent parsimoniously during congested periods.

3) A padded pure ACK message is still a pure ACK message. If an ACK of ACK is required, the sender of the ACK should add another frame to the ACK, e.g., PING.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335220121

---

From: Martin Thomson <[email-masked]>
Date: Mon, 09 Oct 2017 17:59:55 -0700

Thinking through the congestion case further: the only reason you want to send ACK-only packets is when you are passive (you are receiving data, but not sending any).  In which case the other side has ample opportunity to send you acknowledgments.  If you are receiving data that requires acknowledgment, then you are receiving acknowledgments also.  I can't conceive of a valid case where you would be congestion limited, but only sending ACKs for an extended period except a case where the endpoint has an extremely low send rate (low 1000s of bits per second might do it).

That suggests that we might be able to define the limit in absolute terms.  For instance, we could say that you get N octets or N packets in total that are exempt from the congestion limit, a quantity that is replenished every time that you send a packet that isn't blocked by the congestion controller.  Wait long enough and you will be able to send again, even with a ridiculously limited upstream bandwidth.

The risk there is that on these tightly-constrained links, ACKs could consume all available capacity. But I believe that's already true; this would make that marginally worse, but it might actually allow the protocol to continue operation in that sort of environment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335329166

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Oct 2017 08:09:58 +0000 (UTC)

Sensor networks are severely power limited, and sattelite/balloon to ground may also make for an expensive return path due to lack of directed antenna. This does not directly translate into congestion control, but is a bandwidth constraint nonetheless. Perhaps this aspect is under appreciated in general congestion control / bandwidth estimation logic?

For ACK's the flow control is different from stream flow because there is no upstream application consumer - so in principle the receiver has infinite capacity to handle ACKs as long as they reach the receiver timely. However, in praxis too frequent ACK's can be cause network interrupts and unnecessary task switching that can severely constrain a receiver that sends a lot of data to many different connections. Such heavy broadcasters might want to receive as few ACK's as possible.

How all this translates into ACK flow control, if at all, I don't know. But making assumptions on how frequent ACK's ought to be received or should be sent can be misleading due to the above concerns.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335395093

---

From: ianswett <[email-masked]>
Date: Tue, 10 Oct 2017 05:20:41 -0700

@huitema I think I'm ok with those three rules, though there are some details to iron out.  I'm still a bit concerned about getting congestion blocked on padding, but I don't expect padding only packets to be very common, and if you didn't have any retransmittable data outstanding, then it doesn't really matter if you're not making forward progress for a bit. 

@martinthomson I don't think we should make specific recommendations on extra bytes we can send/etc for acks.  If we want to do congestion control on acks, we should do so by counting acks towards bytes in flight as Christian suggests, and using the existing congestion control algorithm. 
 That being said, congestion controlling acks correctly is hard.  If I'm the receiver of a large download and I have a small CWND(ie: 2 MSS) then I need to make sure every few acks get acked, or my download stops.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335454338

---

From: martinduke <[email-masked]>
Date: Tue, 10 Oct 2017 10:45:20 -0700

@martinthomson I agree that there probably aren't any true deadlock cases, but I can probably construct bad corner cases for performance here.

For instance, Alice is sending to Bob, there are frequent packet number gaps, and Bob is therefore acking liberally. Alice is acking conservatively. I could probably end up pacing Alice->Bob by the rate at which Alice is sending acks to Bob, which in not super intuitive.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-335553036

---

From: janaiyengar <[email-masked]>
Date: Fri, 13 Oct 2017 20:57:58 +0000 (UTC)

I'll note that this isn't new -- TCP has had acks for a while, and while smaller in max size, they are still exempt from congestion control. Two data points are useful here:

(i) We wrote RFC 5690 a while ago, to go through the exercise of trying to figure out reducing ack rates. Many experiments were run and my personal conclusion at the end of the exercise is that reducing ack rate is non-trivial at best, and a big hit on forward path performance at worst. You are welcome to peruse the list of "Possible Complications" that we composed in that RFC.

(ii) We've been doing expeirments on reducing ack rate in GQUIC. Short summary, it hurt performance (on the forward path) with Cubic, but didn't with BBR. I won't go into the reasons here, but it's important to note that an endpoint should not unilaterally decide to reduce it's ack rate because it has implications on the performance of the congestion controller in the opposite direction.

In general, reducing ack rate is very useful exercise, but it has to be done with care because of its interactions with ack-based congestion controllers. I would NOT want to congestion control acks by default. It's non-trivial.

I agree with @ianswett that we shouldn't make specific recommendations on how many bytes are allowed etc.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/837#issuecomment-336565592