Subject: Re: [tcpPrague] TSO burst sizing causing TCP Prague unfairness on high capacity links ?

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 10 Jun 2020 10:20:38 +0000

Hi Ashutosh,

 > We observed unfairness
 > between TCP Prague flows when running over high capacity links ( not just
 > wireless but in general ) and I would like to share some of our findings
 > here.

Thanks for sharing these!

 > As you can observe, the second flow grabs almost all the available
 > bandwidth and the first one is starved. 

That's definitely not the expected behavior, as it seems the first sender
somehow mis-detect a enormous amount of CE marks, making it stay at min-cwnd.

I have unfortunately not been able to reproduce this (neither on a virtual
nor a physical 10G testbed), so would need some additional help/details to
diagnose this.

 > This experiment was done using commit number e741f5a
 > <https://github.com/L4STeam/linux/commit/e741f5ac756503e27be9c183dd107eadbe
 > a40c5c#diff-38ce93325583f02d790276f5cafd1c42>  of the TCP Prague linux
 > kernel implementation ( Apr 8 , 2020 ). After some investigation, we found
 > that there might be something broken with the TSO burst sizing updates
 > dones by TCP Prague. I disabled the TSO burst size updates and ran the
 > experiment with the exact same settings and found that the fairness /
 > convergence this time was much better. ( See next plot ).

Could you confirm that this is also happening with the tip of the current
Branch? I.e., 3b63cc0

 > Also, if interested you can look into the ss data plots ( srtt and cwnd )
 > for these two experiments at this link :
 > https://drive.google.com/drive/folders/1pLC0dcMF0-M1cgtw9IhoFiYOMvFJ-
 > cc7?usp=sharing

In addition to these, could you:
- Confirm that this happens with both the internal TCP pacing and the fq
qdisc as pacer on the data sender
- Confirm that this happens both with and without gro/gso/tso/lro on the
endhosts (data sender and receiver, and also the aqm node as fq does not do
gro splitting prior to enqueue)?
- Confirm that the problem persists if you increase the base RTT--a simple
netem qdisc on the reverse path to add 2-5ms should be sufficient, do not
forget to disable gro/gso as that poorly interacts with netem.
- Log the reported CE marks by the AQM, as well as the
delivered_ce/received_ce counters throughout the experiment on the data
sender/receiver?

All of these would help to pinpoint which component is at fault.

Thanks!

Best,
Olivier

 > 
 > Thank you,
 > 
 > Ashutosh Srivastava
 > First year PhD student
 > Department of Electrical and Computer Engineering
 > NYU Tandon School of Engineering

---

From: Ashutosh Srivastava <[email-masked]>
Date: Tue, 16 Jun 2020 23:24:06 -0400

Hi Olivier,

Please find comments inline.

> Branch? I.e., 3b63cc0

Yes, I observe these unfairness issues with the tip of the current branch (
3b63cc0). This happens both in the case of a flow entering after some gap
or with multiple flows starting together.

> qdisc as pacer on the data sender
Yes , it happens with both  with internal TCP pacing and fq pacing.

> gro splitting prior to enqueue)?

This *does not happen* when I turn off gro/gso/tso on all the nodes (end
hosts, AQM) in my experiment.

> forget to disable gro/gso as that poorly interacts with netem.

The problem persists even with some added delay.

> delivered_ce/received_ce counters throughout the experiment on the data

I have uploaded  additional data for experiments with commit 3b63cc0 on the
google drive I shared earlier :
https://drive.google.com/drive/folders/1pLC0dcMF0-M1cgtw9IhoFiYOMvFJ-cc7?usp=sharing

The columns of the AQM csv file are : timestamp, dropped packets, queue
backlog (bytes), queue backlog (packets) , CE_marks
Columns of the ss data file at the sender are : timestamp, ID, cwnd, SRTT
(ms), delivered, delivered_ce, pacing rate (Mbps)

I have also added the raw data files. The experiment settings were the same
as in my last mail.

Thank you,
Ashutosh.