Subject: Re: [tcpPrague] [tcpm] Fwd: I-D Action: draft-bagnulo-tcpm-tcp-low-rtt-00.txt

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Fri, 12 Aug 2016 13:30:38 +0200

Hi Marcello,

I very much appreciate your draft.

Even more so as it addresses one of the things I find to be a bit
"disturbing"  in the l4s drafts - Namely the claim that the finer saw
tooth functionality of l4s/DCTCP alone solves the latency problems (while
DCTCP indeed also is tuning the RTO timeout aggressively).
E.g., the formulation in  section 1.1 of
https://datatracker.ietf.org/doc/draft-briscoe-tsvwg-ecn-l4s-id/?include_t
ext=1

It turns out that a TCP algorithm like DCTCP that solves TCP's
   scalability problem also solves the latency problem, because the
   finer sawteeth cause very little queuing delay.  A supporting paper
   [DCttH15] gives the full explanation of why the design solves both
   the latency and the scaling problems, both in plain English and in
   more precise mathematical form.  The explanation is summarised
   without the maths in [I-D.briscoe-aqm-dualq-coupled].

could benefit from a reference to your draft or the issue that you
addresses here (at least for comprehensiveness).

An additional comment is  that new approaches to retransmissions - like
TCP TLP and TCP RACK (also SCTP TLR which however is not progressing at
the moment) might fundamentally alter the picture. I.e., if
retransmissions are sent pro-actively in tail loss situations then more
conservative RTOs may be kept for situations where it is prudent to wait
longer. Don't know if TCP TLP is so widely deployed that it is something
that you should relate to even if it may be superseded by RACK. Just a
thought.

BR, Karen

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of marcelo bagnulo
> braun
> Sent: 8. juli 2016 23:19
> To: tcpm IETF list <[email-masked]>
> Subject: [tcpm] Fwd: I-D Action: draft-bagnulo-tcpm-tcp-low-rtt-00.txt
>
> Hi,
>
> We just submitted this draft for consideration of the WG. Comments are
> appreciated.
>
> Regards, marcelo
>
>
>
>
> -------- Mensaje reenviado --------
> Asunto: 	I-D Action: draft-bagnulo-tcpm-tcp-low-rtt-00.txt
> Fecha: 	Fri, 08 Jul 2016 14:16:35 -0700
> De: 	[email-masked]
> Responder a: 	[email-masked]
> Para: 	[email-masked]
>
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts
directories.
>
>
>          Title           : Recommendations for increasing TCP
performance in low RTT
> networks.
>          Authors         : Marcelo Bagnulo
>                            Koen De Schepper
>                            Glenn Judd
> 	Filename        : draft-bagnulo-tcpm-tcp-low-rtt-00.txt
> 	Pages           : 7
> 	Date            : 2016-07-08
>
> Abstract:
>     This documents compiles a set of issues that negatively affect TCP
>     performance in low RTT networks as well as the recommendations to
>     overcome them.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-bagnulo-tcpm-tcp-low-rtt/
>
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-bagnulo-tcpm-tcp-low-rtt-00
>
>
> Please note that it may take a couple of minutes from the time of
submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> I-D-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> Internet-Draft directories: http://www.ietf.org/shadow.html or
> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 12 Aug 2016 12:16:50 -0700

> that you should relate to even if it may be superseded by RACK. Just a
TLP and RACK help reduce timeout cases in DC environment for sure. But
still it can not completely avoid timeouts. So I agree we should keep
timeouts conservative, but the current RFCs are way too conservative:
min RTO of 1 second is 4-5 orders of magnitude of DC RTTs.

The issue is, the "right" min-RTO depends on the actual DC and stacks:
they all have different RTTs and timer granularity. The draft cites
Glenn's paper, but Morgan Stanley's DC could be different than others.

Other comments on the draft:

1. min ssthresh or cwnd: with very large incast ((tens of) thousands
of senders into one receiver), cwnd of 0.1 pkt can still be too big. the
only way is to pace the packets over N*RTT intervals.

2. delayed acks:
as mentioned the actual time depends a lot on the DC implementation,
which is really "vender/owner"-specific. instead of perhaps an option
during setup to inform the sender about the max delay in the ack works
better.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 12 Aug 2016 21:40:33 +0200

Karen, Yuchung,
Thanks for the comments.
Replies below.

> Glenn's paper, but Morgan Stanley's DC could be different than others.

Right.

I guess we should could say that the min RTO should be in the order of 
the RTT of the considered network when there is zero queueng delay.
But if the draft is aiming to be a reccomendation for imeplenters of DC 
network TCP stacks, then i guess it should provide a guidance that is 
independent of the DC network itself, since the implementation should 
work for a variety of DC networks, no?

I guess the more general question is whether we need this document i.e. 
a document describing reccoemndation about how to set TCP for low RTT 
networks. If people think it is a good idea to have such a document, we 
then figure how if this is

> only way is to pace the packets over N*RTT intervals.
Right, If i understood correctly Bob's paper (cited in the draft) this 
is exactly what they are proposing.
I dont think this document is the right place to define the behaviour of 
CWND less than 1 MSS. I think this should be done in a separated 
document, which is then referred by the low-rtt draft.

Are you aware of other documents (draft or papers others than the one 
from Bob) describing mechanisms to achieve this?

> during setup to inform the sender about the max delay in the ack works
I am not sure if a follow.
What is the benefit of allowing dynamic negotiation of the delay ack time?
Thanks, marcelo

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 12 Aug 2016 12:50:59 -0700

On Fri, Aug 12, 2016 at 12:40 PM, marcelo bagnulo braun
<[email-masked]> wrote:
> I guess we should could say that the min RTO should be in the order of the
> RTT of the considered network when there is zero queueng delay.
> independent of the DC network itself, since the implementation should work
> for a variety of DC networks, no?
>
>
> I guess the more general question is whether we need this document i.e. a
> document describing reccoemndation about how to set TCP for low RTT
> networks. If people think it is a good idea to have such a document, we then
> figure how if this is
my conundrum is that DC tcp stuff is usually not an inter-op issue for
the Internet so this draft is more of a "how to tune your DC tcp
stack". but I can be convinced either ways.

> Right, If i understood correctly Bob's paper (cited in the draft) this is
> exactly what they are proposing.
> CWND less than 1 MSS. I think this should be done in a separated document,
> which is then referred by the low-rtt draft.
>
> Are you aware of other documents (draft or papers others than the one from
> Bob) describing mechanisms to achieve this?
>
>> 2. delayed acks:
the receiver may decide to delay ack up to certain time. it could be
based on DC implementation or host limit. It'd be good for sender to
learn about that to adapt his RTO.

For example, say it's a cloud VM. VM of OS1 may use a max delay for
d-ack say 2ms. but VM of OS2 may use 40ms. The lack of such
information makes the sender to these VMs use something  conservative
of known max of all, otherwise he risks spurious RTOs.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 12 Aug 2016 20:35:40 +0000

>>2. delayed acks:
as mentioned the actual time depends a lot on the DC implementation, which is really "vender/owner"-specific. instead of perhaps an option during setup to inform the sender about the max delay in the ack works better.

Yeah this is a real problem in IaaS environment today since TCP sender picks minrto and receiver picks delayed ACK timeout independently and can cause spurious RTOs if the minrto is too aggressive. Windows server tries to use the connection handshake RTT (< 10 msec) to auto pick the DC values but this is a heuristic that can fail and doesn't really work in the Linux interop cases. 

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Yuchung Cheng
Sent: Friday, August 12, 2016 12:17 PM
To: Karen Elisabeth Egede Nielsen <[email-masked]>
Cc: tcpm IETF list <[email-masked]>; TCP Prague List <[email-masked]>
Subject: Re: [tcpm] Fwd: I-D Action: draft-bagnulo-tcpm-tcp-low-rtt-00.txt

On Fri, Aug 12, 2016 at 4:30 AM, Karen Elisabeth Egede Nielsen <[email-masked]> wrote:
> tooth functionality of l4s/DCTCP alone solves the latency problems 
> (while DCTCP indeed also is tuning the RTO timeout aggressively).
> An additional comment is  that new approaches to retransmissions - 
> like TCP TLP and TCP RACK (also SCTP TLR which however is not 
> progressing at the moment) might fundamentally alter the picture. 
> I.e., if retransmissions are sent pro-actively in tail loss situations 
> then more conservative RTOs may be kept for situations where it is 
> prudent to wait longer. Don't know if TCP TLP is so widely deployed 
> that it is something that you should relate to even if it may be 
> superseded by RACK. Just a thought.
TLP and RACK help reduce timeout cases in DC environment for sure. But still it can not completely avoid timeouts. So I agree we should keep timeouts conservative, but the current RFCs are way too conservative:
min RTO of 1 second is 4-5 orders of magnitude of DC RTTs.

The issue is, the "right" min-RTO depends on the actual DC and stacks:
they all have different RTTs and timer granularity. The draft cites Glenn's paper, but Morgan Stanley's DC could be different than others.

1. min ssthresh or cwnd: with very large incast ((tens of) thousands of senders into one receiver), cwnd of 0.1 pkt can still be too big. the only way is to pace the packets over N*RTT intervals.

2. delayed acks:
as mentioned the actual time depends a lot on the DC implementation, which is really "vender/owner"-specific. instead of perhaps an option during setup to inform the sender about the max delay in the ack works better.

> > From: tcpm [mailto:[email-masked]] On Behalf Of marcelo 
> > bagnulo braun
> > Sent: 8. juli 2016 23:19
> > We just submitted this draft for consideration of the WG. Comments 
> > are appreciated.
> >
> > Regards, marcelo
> >
> >
> >
> >

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Mon, 15 Aug 2016 11:00:24 +0200

Hi Yuchung,

> still it can not completely avoid timeouts.

[Karen Elisabeth Egede Nielsen] Agreed.

For SCTP TLR we still see RTO-timeouts when also probes/retransmissions are
lost.
I assume that it would be something like this also for Rack/TLP -  though
potentially depending on how many consecutive probes that are allowed.
The role of RTO-timeouts are different when RACK/TLP is enabled and it might
be counterproductive to have RTO-timeouts be too aggressive as the function
indeed with TLP/RACK becomes something much closer to the original intend
(read: how I understand the original intend) - namely to introduce a pause
for the network to recover when things are really (consistently) bad.

If the RTO is lowered down to be comparable in order of magnitude with the
RTT (with some delay_ack considerations) we are narrowing the situation
where TLP/RACK probing is able to kick in before RTO-retransmissions starts
anyway.

I understand that RTOs should be adjusted to the network dynamics, but I do
think that it is important to understand - for the DC environment - if the
need for the low RTOs in DC's is
motivated mainly by having RTO-retransmissions fix the tail loss recovery
deficiencies of TCP, which RACK/TLP addresses, or more generally for a need
for more timely probe and reactivation for/after network recovery.

Or perhaps you see the TLP probe timeout and the RTO timeout eventually
being driven by the same timer, the reactions (e.g. CC reactions)  then
possibly depending on state (probe already sent or not).

BR, Karen

 So I agree we should keep
> bagnulo

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Mon, 15 Aug 2016 11:00:24 +0200

Hi Yuchung,

BR, Karen

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 15 Aug 2016 09:49:12 -0700

> possibly depending on state (probe already sent or not).
I agree with your thought here.

More generally, I'd like to evolve the state of art recovery with
these principles:
1. React with ack events as much as possible

2. Send tiny probes to trigger (1) after 2-3 RTTs

3. Have conservative  RTOs that (exp-backoff) expire at an order of
RTT to repair the head for reliability.

On thing about RTO: RTO current reset cwnd to 1 upon firing no matter
what. We should ONLY reset cwnd to 1 if no news (acks/data) for a long
duration. This is quite important for performance: today cwnd is reset
to 1 even if every packet but the first one has been recently
delivered. This unnecessarily conservative because the ack clock isn't
lost yet. But the collateral damage to performance is huge on large
BDP networks. This results work-arounds to shorten and avoid RTOs.

I'd like to address these points in the upcoming RACK/TLP draft.

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Thu, 18 Aug 2016 10:05:45 +0200

HI Yuchung, Marcello

> From: Yuchung Cheng [mailto:[email-masked]]
> Sent: 15. august 2016 18:49
> Cc: marcelo bagnulo braun <[email-masked]>; TCP Prague List
> <[email-masked]>; tcpm IETF list <[email-masked]>
> >> > I.e., if retransmissions are sent pro-actively in tail loss
> >> > situations then more conservative RTOs may be kept for situations
> >> > where it is prudent to wait longer. Don't know if TCP TLP is so
> >> > widely deployed that it is something that you should relate to even
> >> > if it may be superseded by RACK. Just a thought.
> >> TLP and RACK help reduce timeout cases in DC environment for sure.
> >> But still it can not completely avoid timeouts.
> >
> > For SCTP TLR we still see RTO-timeouts when also
> > probes/retransmissions are lost.
> > I assume that it would be something like this also for Rack/TLP -
> > though potentially depending on how many consecutive probes that are
> allowed.
> > The role of RTO-timeouts are different when RACK/TLP is enabled and it
> > might be counterproductive to have RTO-timeouts be too aggressive as
> > the function indeed with TLP/RACK becomes something much closer to the
> > original intend
> > (read: how I understand the original intend) - namely to introduce a
> > pause for the network to recover when things are really (consistently)
> > bad.
> >
> > If the RTO is lowered down to be comparable in order of magnitude with
> > the RTT (with some delay_ack considerations) we are narrowing the
> > situation where TLP/RACK probing is able to kick in before
> > RTO-retransmissions starts anyway.
> >
> > I understand that RTOs should be adjusted to the network dynamics, but
> > I do think that it is important to understand - for the DC environment
> > - if the need for the low RTOs in DC's is motivated mainly by having
> > RTO-retransmissions fix the tail loss recovery deficiencies of TCP,
> > which RACK/TLP addresses, or more generally for a need for more timely
> > probe and reactivation for/after network recovery.
> >
> > Or perhaps you see the TLP probe timeout and the RTO timeout
> > eventually being driven by the same timer, the reactions (e.g. CC
> > reactions)  then possibly depending on state (probe already sent or
> > not).
> More generally, I'd like to evolve the state of art recovery with these
> principles:
> 3. Have conservative  RTOs that (exp-backoff) expire at an order of RTT to
> repair the head for reliability.
>
>
> what.
> We should ONLY reset cwnd to 1 if no news (acks/data) for a long duration.
> This is quite important for performance: today cwnd is reset to 1 even if
> every packet but the first one has been recently delivered. This
> unnecessarily conservative because the ack clock isn't lost yet. But the
> collateral damage to performance is huge on large BDP networks. This
> results
> work-arounds to shorten and avoid RTOs.
>
>
[Karen Elisabeth Egede Nielsen] I agree completely with your view and
approach.

In respect to the draft draft-bagnulo-tcpm-tcp-low-rtt-00.txt and the given
recommendations for low RTO
is that they do not (likely) take 2. Into account - meaning that they are
looking for the low RTO to solve 2 for tail recovery.
The bad thing is that they then also get the CWND reductions (as you also
say).

I would like for that this aspect is taken into consideration/given thoughts
in the draft draft-bagnulo-tcpm-tcp-low-rtt-00.txt and the general 4LS
effort.
If we end with very low recommendations for RTO in order to handle 2. we
risk undermining the RTO-retransmission function as well as the TLP function
for now and forever.

It might be that one for TCPs, that do not support TLP(/RACK), need to
operate with very low RTOs to handle tail loss's "fast enough", but the
recommendation for
"future standard ?" TCP with TLP always enabled might be different. In this
context lowering the RTO to handle 2. should be considered as a "temporary
hack" ?

BR, Karen

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 19 Aug 2016 10:10:26 -0700

> context lowering the RTO to handle 2. should be considered as a "temporary

One feature worth discussion in the draft is the penalty of cwnd=1 on
spurious timeouts.

For DC, the RTT is so fairly low that we may consider a false reset of
cwnd to 1 is not a big deal. But consider in world of 100Gbps BDP will
still be hundreds of packets so it is going to be a non-trivial
penalty.

Therefore it is important to discuss the timeouts with the remedy
operations: how well can the connection detect spurious timeouts:
F-RTO or Eifel are the two algorithms by IETF, Linux also supports
DSACK-based undo.

However, both F-RTO and Eifel have their limitations: F-RTO requires
the sender to have new data to transmit after the first DUPACK of
timeout. In the DC world, the requests or responses are often short
enough to prevent that. Eifel relies on timestamps which is ticked at
an interval higher than the DC RTT (e.g., Linux == jiffies == 1ms
often), therefore the timestamps fail to distinguish original and
retransmit if RTO was fired before the next tick.

I plan to cover these aspects in next RACK rev.