Subject: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

From: Szilveszter Nadas <[email-masked]>
Date: Tue, 3 Nov 2020 17:30:20 +0000

Hi all,

During the review of our article "A Congestion Control Independent L4S Scheduler" we received comments from one of the reviewers, on why we did not also evaluate TCP Prague. So now we rerun the test cases with replacing DCTCP to TCP Prague.

The results are at the below link. We are somewhat surprised by much increased aggressiveness of TCP Prague. Can you comment on it? The settings and setup we used are described in the pdf (and in the original article).
Results: http://ppv.elte.hu/tcp-prague/
Original article (including YouTube presentation): http://ppv.elte.hu/cc-independent-l4s/

Can you comment on the results and the settings we used?

Cheers,
Szilveszter

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 5 Nov 2020 11:09:48 +0000

Hi Szilveszter,

 > The results are at the below link. We are somewhat surprised by much
 > increased aggressiveness of TCP Prague. Can you comment on it? The settings
 > and setup we used are described in the pdf (and in the original article).

Thanks for sharing these.

* Short answer:
DCTCP actively hurts itself, tripping the step AQM unnecessarily.
This is mostly benign in DC environments due to the almost non
existent RTT, and small cwnd, but causes observable performance hits elsewhere.
Prague implements fixes for these issues.

* Long answer:
Please find below the main changes from dctcp to prague. The intuition
behind these is to avoid triggering the step AQM part of the dual queue
when there is classic traffic, or limit the extent of the received marks.
The first goal ensure that, when there is classic traffic, prague only
receives marks from the coupled PI2 component, which are strong enough on
their own to keep the L4S queue empty most of the time. Additionally
getting marks from the step AQM means that prague is hurting itself, as
it will reduce more than necessary. The second goal ensures that prague
does not unnecessarily drive the AQM (either PI2 or the step) to
high-marking rate; again because it will hurts its throughput and also
because it will create a standing queue.

Note that depending on your experimental setups, not all the below factors
will contribute.

1. Prague is always paced, regardless of the egress qdisc on the data sender,
   and paces itself at 100% of the estimated BDP.
	DCTCP is only paced when used in combination with fq, and paces at 120%
	of the BDP when that is the case.
2. Prague actively limits its gso bursts (defaults to 250us)
	DCTCP uses Linux defaults, 1ms
3. Prague implements a more accurate internal marking estimate (alpha)
	DCTCP, especially when operating with low marking probabilities, will
	tend to push down alpha to 0, delaying its response to marks (and thus
	increasing queue pressure/causing larger reduction down the line)
4. Prague carry over sub-cwnd reduction, such that multiple marks in a row
   occurring at low overall probability will eventually cause a cwnd reduction
	DCTCP's cwnd reduction code has no memory, i.e., as long as alpha is
	low enough compared to cwnd, no cwnd reduction will ever occur. This
	again drives the aqm to larger mark probabilities.

All of these points contribute to prague being more gentle towards the queue,
and more reactive to the received marks (i.e., achieving a lower overall
marking level)--the direct effect being smaller cwnd reduction (thus sawtooth)
than DCTP, which is critical when operating at higher e2e RTTs with a shallow
queue since it lowers the time to recovery.

Additionally, when operating over a long RTT path and controlling by PI2 (i.e.,
random marking, such that it receives marks almost every RTT--as opposed to a
step where it receives nothing for several RTTs and then a RTT with most of the
packets being marked), DCTCP suffers from the inaccurate/memoryless computations
in 3./4. and its interactions with Linux's PRR code (which prague does not use),
which can prevent it from ever reaching its expected rate.

I hope this helps to understand a bit better why prague appears to be more
aggressive--it is actually the opposite: dctcp systematically hurts itself
at longer base RTT, hence progressively give way as the RTT increases.
You can validate this by observing that the expected rate ratios between
classic flows and prague over the dualQ match the theoretical equations
(compounding the queue delay difference) even that higher base RTTs, which
is not the case for DCTCP.

Best,
Olivier

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 12 Nov 2020 09:34:56 +0000

Hi Szilveszter,

Interesting work, thanks for sharing. There is indeed a very big difference between how DCTCP behaves today in recent kernels, than what we used in the original L4S work testing (Linux kernel 3.19). That original version was a "clean" DCTCP version that matched very well the theoretical equations: r=2/(p.RTT) for uniform stable random and r=2/(p².RTT) on/off marking. Due to many interactions, integer range constraints, pacing/GSO interactions and bugs, the performance really degraded in the recent Linux versions. These issues were fixed within the Linux Prague version, so that is why Prague should perform "better" in respect to matching the equations. It would be interesting to get an idea of the deviation from the r=2/(p.RTT) equation in your tests (relevant in DualPI2 as the coupling gives a smooth probability). Collecting the average RTT (base + queuing latency), marking probability, and rate would make it possible to evaluate this.

Main reasons why we saw deviations is when the probability varies a lot in time. As you might know DCTCP's (and Prague's) equation becomes r=2/(p².RTT) when it receives on/off marking episodes (on episodes in the order of 1 RTT, off in the order of multiple RTTs). Any not so stable marking probability results in a rate between those 2 boundaries. This is the theory, looking forward on how your practice matches this.

>From a first quick look at the results it might not deviate that much, I guess. The more aggressive part is probably due to the RTT unfairness: 1 to 5 rate ratio (with a 5ms base RTT, Classic gets a queue of 5ms+20ms target, so about 25ms and 5 times less throughput). Did you try to use the RTT independent version? If you set it to the f=max(15, RTT) mode, the minimum effective RTT becomes 15ms, so the rate ratio is 3 to 5 in that case.

Thanks and Regards,
Koen.

From: tcpPrague <[email-masked]> On Behalf Of Szilveszter Nadas
Sent: Tuesday, November 3, 2020 6:30 PM
To: [email-masked]
Cc: Ferenc Fejes <[email-masked]>; Gombos Gergo <[email-masked]>; LAKI Sandor <[email-masked]>
Subject: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi all,

Cheers,
Szilveszter

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 13 Nov 2020 07:27:09 +0000

Hi

Just some food for thought. 

Hybrid Slowstart (HyStart) in Cubic is known to have the problem that it can
exit to congestion avoidance prematurely. We have in fact seen issues with
HyStart in cellular and the outcome is that it over-reacts to scheduling
jitter. This issue was mentioned in the long since expired
https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-02 (section 4) and
a presentation at ICCRG
https://datatracker.ietf.org/meeting/96/materials/slides-96-iccrg-1 

The Linux TCP stack enables HyStart with both the delay and ack train
algorithms on by default. It could be a good idea to at some stage rerun the
tests with HyStart completely disabled. I am not sure in this is the reason
but believe that it is worth to rule out the problems with HyStart .

/Ingemar

PS HyStart++ developed by Praveen and others at Microsoft  instead exits to
a limited slowstart
https://tools.ietf.org/html/draft-balasubramanian-tcpm-hystartplusplus-03. 

From: De Schepper, Koen (Nokia - BE/Antwerp)
<[email-masked]> 
Sent: den 12 november 2020 10:35
To: Szilveszter Nadas <Szilveszter.Nadas=[email-masked]>;
[email-masked]
Cc: Ferenc Fejes <[email-masked]>; Gombos Gergo <[email-masked]>;
LAKI Sandor <[email-masked]>
Subject: Re: [tcpPrague] A Congestion Control Independent L4S Scheduler -
TCP Prague preliminary results

Hi Szilveszter,

Interesting work, thanks for sharing. There is indeed a very big difference
between how DCTCP behaves today in recent kernels, than what we used in the
original L4S work testing (Linux kernel 3.19). That original version was a
“clean” DCTCP version that matched very well the theoretical equations:
r=2/(p.RTT) for uniform stable random and r=2/(p².RTT) on/off marking. Due
to many interactions, integer range constraints, pacing/GSO interactions and
bugs, the performance really degraded in the recent Linux versions. These
issues were fixed within the Linux Prague version, so that is why Prague
should perform “better” in respect to matching the equations. It would be
interesting to get an idea of the deviation from the r=2/(p.RTT) equation in
your tests (relevant in DualPI2 as the coupling gives a smooth probability).
Collecting the average RTT (base + queuing latency), marking probability,
and rate would make it possible to evaluate this.

Main reasons why we saw deviations is when the probability varies a lot in
time. As you might know DCTCP’s (and Prague’s) equation becomes r=2/(p².RTT)
when it receives on/off marking episodes (on episodes in the order of 1 RTT,
off in the order of multiple RTTs). Any not so stable marking probability
results in a rate between those 2 boundaries. This is the theory, looking
forward on how your practice matches this.

>From a first quick look at the results it might not deviate that much, I
guess. The more aggressive part is probably due to the RTT unfairness: 1 to
5 rate ratio (with a 5ms base RTT, Classic gets a queue of 5ms+20ms target,
so about 25ms and 5 times less throughput). Did you try to use the RTT
independent version? If you set it to the f=max(15, RTT) mode, the minimum
effective RTT becomes 15ms, so the rate ratio is 3 to 5 in that case.

Thanks and Regards,

Koen.

From: tcpPrague <[email-masked]
<mailto:[email-masked]> > On Behalf Of Szilveszter Nadas
Sent: Tuesday, November 3, 2020 6:30 PM
To: [email-masked] <mailto:[email-masked]> 
Cc: Ferenc Fejes <[email-masked] <mailto:[email-masked]> >; Gombos
Gergo <[email-masked] <mailto:[email-masked]> >; LAKI Sandor
<[email-masked] <mailto:[email-masked]> >
Subject: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP
Prague preliminary results

Hi all,

During the review of our article “A Congestion Control Independent L4S
Scheduler” we received comments from one of the reviewers, on why we did not
also evaluate TCP Prague. So now we rerun the test cases with replacing
DCTCP to TCP Prague. 

Original article (including YouTube presentation):
http://ppv.elte.hu/cc-independent-l4s/ 

Cheers,

Szilveszter

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 13 Nov 2020 10:16:06 +0000

Changing the title, to focus on this obserevation...

On 13/11/2020 07:27, Ingemar Johansson S wrote:
> Hybrid Slowstart (HyStart) in Cubic is known to have the problem that 
> it can exit to congestion avoidance prematurely. We have in fact seen 
> issues with HyStart in cellular and the outcome is that it over-reacts 
> to scheduling jitter.
>
> This issue was mentioned in the long since expired 
> https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-02 
> <https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-02> (section 
> 4) and a presentation at ICCRG 
> <https://datatracker.ietf.org/meeting/96/materials/slides-96-iccrg-1>
>
We've also been recently testing QUIC over broadband satellite systems, 
and we also observe the same sort of reaction to the path 
characteristics (likely also due to increased jitter in the 
timeslot-based transmission of ACKs when used on a shared broadband 
satellite return link). The result was a forward path that was 
under-utilised. These experiments used google's (latest) QUIC - and we 
saw a resulting reduction in the throughput when the RTT was larger.

> algorithms on by default. It could be a good idea to at some stage 
> rerun the tests with HyStart completely disabled. I am not sure in 
> this is the reason but believe that it is worth to rule out the 
> problems with HyStart .
>
> /Ingemar
>
> PS HyStart++ developed by Praveen and others at Microsoft  instead 
> exits to a limited slowstart 
> https://tools.ietf.org/html/draft-balasubramanian-tcpm-hystartplusplus-03 
> <https://tools.ietf.org/html/draft-balasubramanian-tcpm-hystartplusplus-03>. 
>
>
This topic isn't new, but I think might need some attention wrt to new 
transport methods, and might be something to explore for a range of 
protocols. If it is still a concern, maybe we should open this up 
perhaps initially in TCPM ?

Gorry
>
> *From:* De Schepper, Koen (Nokia - BE/Antwerp) 
> *Sent:* den 12 november 2020 10:35
> *To:* Szilveszter Nadas 
> <Szilveszter.Nadas=[email-masked]>; [email-masked]
> *Cc:* Ferenc Fejes <[email-masked]>; Gombos Gergo 
> <[email-masked]>; LAKI Sandor <[email-masked]>
> *Subject:* Re: [tcpPrague] A Congestion Control Independent L4S 
> Scheduler - TCP Prague preliminary results
>
> Hi Szilveszter,
>
> Interesting work, thanks for sharing. There is indeed a very big 
> difference between how DCTCP behaves today in recent kernels, than 
> what we used in the original L4S work testing (Linux kernel 3.19). 
> That original version was a “clean” DCTCP version that matched very 
> well the theoretical equations: r=2/(p.RTT) for uniform stable random 
> and r=2/(p².RTT) on/off marking. Due to many interactions, integer 
> range constraints, pacing/GSO interactions and bugs, the performance 
> really degraded in the recent Linux versions. These issues were fixed 
> within the Linux Prague version, so that is why Prague should perform 
> “better” in respect to matching the equations. It would be interesting 
> to get an idea of the deviation from the r=2/(p.RTT) equation in your 
> tests (relevant in DualPI2 as the coupling gives a smooth 
> probability). Collecting the average RTT (base + queuing latency), 
> marking probability, and rate would make it possible to evaluate this.
>
> Main reasons why we saw deviations is when the probability varies a 
> lot in time. As you might know DCTCP’s (and Prague’s) equation becomes 
> r=2/(p².RTT) when it receives on/off marking episodes (on episodes in 
> the order of 1 RTT, off in the order of multiple RTTs). Any not so 
> stable marking probability results in a rate between those 2 
> boundaries. This is the theory, looking forward on how your practice 
> matches this.
>
> From a first quick look at the results it might not deviate that much, 
> I guess. The more aggressive part is probably due to the RTT 
> unfairness: 1 to 5 rate ratio (with a 5ms base RTT, Classic gets a 
> queue of 5ms+20ms target, so about 25ms and 5 times less throughput). 
> Did you try to use the RTT independent version? If you set it to the 
> f=max(15, RTT) mode, the minimum effective RTT becomes 15ms, so the 
> rate ratio is 3 to 5 in that case.
>
> Thanks and Regards,
>
> Koen.
>
> *From:* tcpPrague <[email-masked] 
> <mailto:[email-masked]>> *On Behalf Of *Szilveszter Nadas
> *Sent:* Tuesday, November 3, 2020 6:30 PM
> *To:* [email-masked] <mailto:[email-masked]>
> *Cc:* Ferenc Fejes <[email-masked] <mailto:[email-masked]>>; 
> Gombos Gergo <[email-masked] <mailto:[email-masked]>>; LAKI 
> Sandor <[email-masked] <mailto:[email-masked]>>
> *Subject:* [tcpPrague] A Congestion Control Independent L4S Scheduler 
> - TCP Prague preliminary results
>
> Hi all,
>
> Scheduler” we received comments from one of the reviewers, on why we 
> did not also evaluate TCP Prague. So now we rerun the test cases with 
> replacing DCTCP to TCP Prague.
>
> increased aggressiveness of TCP Prague. Can you comment on it? The 
> settings and setup we used are described in the pdf (and in the 
> original article).
>
> Results: http://ppv.elte.hu/tcp-prague/ <http://ppv.elte.hu/tcp-prague/>
>
> <http://ppv.elte.hu/cc-independent-l4s/>
>
> _______________________________________________
> tcpPrague mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpprague

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 13 Nov 2020 16:37:29 +0000

Minor correction: the most recent HyStart++ draft is draft-ietf-tcpm-hystartplusplus-00 - HyStart++: Modified Slow Start for TCP<https://tools.ietf.org/html/draft-ietf-tcpm-hystartplusplus-00>. Only the delay increase algorithm is used. We have some work planned for measurements on Cellular networks.

From: tcpPrague <[email-masked]> On Behalf Of Ingemar Johansson S
Sent: Thursday, November 12, 2020 11:27 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; Szilveszter Nadas <Szilveszter.Nadas=[email-masked]>; [email-masked]
Cc: Ferenc Fejes <[email-masked]>; Ingemar Johansson S <[email-masked]>; Gombos Gergo <[email-masked]>; LAKI Sandor <[email-masked]>
Subject: [EXTERNAL] Re: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi

Just some food for thought.
Hybrid Slowstart (HyStart) in Cubic is known to have the problem that it can exit to congestion avoidance prematurely. We have in fact seen issues with HyStart in cellular and the outcome is that it over-reacts to scheduling jitter. This issue was mentioned in the long since expired https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-02<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-johansson-cc-for-4g-5g-02&data=04%7C01%7Cpravb%40microsoft.com%7Ca630feb0fdfc4118c97708d887a5922b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637408492508511379%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xQLg%2Bqy%2BlM6l%2FcHAAJTZ6vHEmH5kXdbrEITNuLvOJmU%3D&reserved=0> (section 4) and a presentation at ICCRG https://datatracker.ietf.org/meeting/96/materials/slides-96-iccrg-1<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F96%2Fmaterials%2Fslides-96-iccrg-1&data=04%7C01%7Cpravb%40microsoft.com%7Ca630feb0fdfc4118c97708d887a5922b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637408492508511379%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=G%2B%2Form1EuoO0iooETiJrWCiF4cqnAPC5sLfsDvAI8WQ%3D&reserved=0>
The Linux TCP stack enables HyStart with both the delay and ack train algorithms on by default. It could be a good idea to at some stage rerun the tests with HyStart completely disabled. I am not sure in this is the reason but believe that it is worth to rule out the problems with HyStart .

/Ingemar

PS HyStart++ developed by Praveen and others at Microsoft  instead exits to a limited slowstart https://tools.ietf.org/html/draft-balasubramanian-tcpm-hystartplusplus-03<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-balasubramanian-tcpm-hystartplusplus-03&data=04%7C01%7Cpravb%40microsoft.com%7Ca630feb0fdfc4118c97708d887a5922b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637408492508521375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rXAGBt5p9GiPvbK6d%2BSKuKJLs4yeo1axQo7VOK7i5Ys%3D&reserved=0>.

From: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<mailto:[email-masked]>>
Sent: den 12 november 2020 10:35
To: Szilveszter Nadas <Szilveszter.Nadas=[email-masked]<mailto:Szilveszter.Nadas=[email-masked]>>; [email-masked]<mailto:[email-masked]>
Cc: Ferenc Fejes <[email-masked]<mailto:[email-masked]>>; Gombos Gergo <[email-masked]<mailto:[email-masked]>>; LAKI Sandor <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi Szilveszter,

Thanks and Regards,
Koen.

From: tcpPrague <[email-masked]<mailto:[email-masked]>> On Behalf Of Szilveszter Nadas
Sent: Tuesday, November 3, 2020 6:30 PM
To: [email-masked]<mailto:[email-masked]>
Cc: Ferenc Fejes <[email-masked]<mailto:[email-masked]>>; Gombos Gergo <[email-masked]<mailto:[email-masked]>>; LAKI Sandor <[email-masked]<mailto:[email-masked]>>
Subject: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi all,

The results are at the below link. We are somewhat surprised by much increased aggressiveness of TCP Prague. Can you comment on it? The settings and setup we used are described in the pdf (and in the original article).
Results: http://ppv.elte.hu/tcp-prague/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fppv.elte.hu%2Ftcp-prague%2F&data=04%7C01%7Cpravb%40microsoft.com%7Ca630feb0fdfc4118c97708d887a5922b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637408492508521375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=u6I7lCsVXe107m1tW%2FqgkU7X%2BDwkAS45yLEPLwIGl74%3D&reserved=0>
Original article (including YouTube presentation): http://ppv.elte.hu/cc-independent-l4s/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fppv.elte.hu%2Fcc-independent-l4s%2F&data=04%7C01%7Cpravb%40microsoft.com%7Ca630feb0fdfc4118c97708d887a5922b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637408492508531361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=n3J1Txb%2BvqsuhzmZVxCu0ch5Nr%2B5wwrn2LJNkmyWK%2Fs%3D&reserved=0>

Cheers,
Szilveszter