Subject: Re: [tcpPrague] TCP Prague under-utilising capacity when RTT scaling turned on

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 10 Jun 2020 10:27:42 +0000

Hi Ashutosh,

 > This is about an issue I found with the recently added RTT scaling feature
 > of TCP Prague linux kernel implementation. This was first observed with
 > commit number 7f267bd
 > <https://github.com/L4STeam/linux/commit/7f267bd591c249697f63f81b21a1755202
 > e86cc2#diff-38ce93325583f02d790276f5cafd1c42>  ( Mar 16, 2020 ) where RTT
 > scaling was turned on by default and also with future commits when we
 > explicitly enable RTT scaling.
 > 
 > The plot below shows the throughput of a single TCP Prague flow. Although
 > the available capacity was 1Gbps, TCP Prague is not able to reach that
 > point anytime during the experiment.

Thanks again for this report. The most inexplicable behavior here are the
apparent cwnd reductions that are happening while the connection is
under-utilizing the link (so should not be receiving CE marks).

I am unfortunately unable to reproduce this behavior on a phy or emulated
testbed, so would need additional information to work through this issue.

Could you confirm that this is also happening with the tip of the current
Branch? I.e., 3b63cc0 is the "future commits" you are referring to above?

In addition to the ss plots, could you:
- Confirm that this happens with both the internal TCP pacing and the fq
qdisc as pacer on the data sender
- Confirm that this happens both with and without gro/gso/tso/lro on the
endhosts (data sender and receiver, and also the aqm node as fq does not do
gro splitting prior to enqueue)?
- Confirm that the problem persists if you increase the base RTT--a simple
netem qdisc on the reverse path to add 2-5ms should be sufficient, do not
forget to disable gro/gso as that poorly interacts with netem.
- Log the reported CE marks by the AQM, as well as the
delivered_ce/received_ce counters throughout the experiment on the data
sender/receiver?
- Log eventual drops (dropped counter on AQM, and retransmission
counters on the data sender).

Thanks!

Best,
Olivier

 > 
 > The experiment settings were as follows
 > 
 > *	This experiment was done on the Cloudlab <https://www.cloudlab.us/>
 > testbed with a 3-node topology ( source, router, receiver).
 > 
 > *	The bottleneck between the router and receiver was a 1 Gbps wired
 > link ( 10 Gig interfaces , capacity restricted to 1Gbps using linux traffic
 > shaping tools (tc) ).
 > 
 > *	The flow was sent using iperf3.
 > 
 > *	The AQM at the router was a FQ qdisc with a single bucket and was
 > marking packets with ECN at a marking threshold of 5 ms. You can use the
 > following parameters with the tc-fq qdisc to replicate this setting :  fq
 > limit 5000p flow_limit 5000p orphan_mask 0 ce_threshold 5ms
 > *	The "ECN fallback on detection of classic ECN AQM" feature of TCP
 > Prague were disabled for this set of experiments.
 > *	The propagation / base delay of the setup was very low ( around 0.4
 > ms ) and no delay was added on top.
 > 
 > If interested, you can look at the ss data plots for these experiments at :
 > https://drive.google.com/drive/folders/1O6uEngxrDX5ipY71sjqr36lXCQBoZV--
 > ?usp=sharing
 > <https://drive.google..com/drive/folders/1O6uEngxrDX5ipY71sjqr36lXCQBoZV--
 > ?usp=sharing>
 > 
 > Looking forward to your comments and questions regarding these results.
 > 
 > Thank you,
 > 
 > Ashutosh Srivastava
 > First year PhD student,
 > Department of Electrical and Computer Engineering
 > NYU Tandon School of Engineering

---

From: Ashutosh Srivastava <[email-masked]>
Date: Tue, 16 Jun 2020 23:10:07 -0400

Hi Olivier,

> Branch? I.e., 3b63cc0 is the "future commits" you are referring to above?

Yes, this is also happening with the tip of the current branch.

> qdisc as pacer on the data sender

Yes, this happens with both internal TCP pacing and with the fq qdisc as
pacer at the sender.

> gro splitting prior to enqueue)?

There is still underutilisation after I turn off offloading (gro/gso/tso)
on all interfaces on all hosts but it does not look as bad as earlier.

> forget to disable gro/gso as that poorly interacts with netem.

At higher RTTs, there is some underutilisation at the beginning but TCP
Prague slowly reaches full bandwidth  and operates as expected thereafter.

> delivered_ce/received_ce counters throughout the experiment on the data

I do not observe any CE marks / delivered_ce packets as the link always
remains underutilised.

> counters on the data sender).
I have uploaded additional data for experiments with commit 3b63cc0 on the
google drive I shared  earlier :
https://drive.google.com/drive/folders/1O6uEngxrDX5ipY71sjqr36lXCQBoZV--?usp=sharing

The columns of the AQM csv file are : timestamp, dropped packets, queue
backlog (bytes), queue backlog (packets)
Columns of the ss.csv file at the sender are : timestamp, ID, cwnd, SRTT
(ms), delivered, delivered_ce, pacing rate (Mbps)

I have also added the raw data files. The experiment settings were the same
as in my last mail.

Thank you,
Ashutosh.