Subject: [rtcweb] WG Last Call on changes: draft-ietf-avtcore-rtp-circuit-breakers-16

From: Magnus Westerlund <[email-masked]>
Date: Tue, 14 Jun 2016 10:14:41 +0200

AVT, RTCWEB and TSVWG

This announces an additional one week WG last call (in AVT) on the 
changes done to "Multimedia Congestion Control: Circuit Breakers for 
Unicast RTP Sessions" that is intended for Proposed Standard. The last 
call concludes on the 21st of June.

So this last call is deemed necessary due to normative changes in the 
behaviour as result of the conclusions of how to resolve the Discuss 
regarding reaction to ECN markings.

If there are no issues in this additional WG last call, the draft will 
be approved for publication as it has already passed IESG review.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-avtcore-rtp-circuit-breakers/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-avtcore-rtp-circuit-breakers-16

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-avtcore-rtp-circuit-breakers-16

Cheers

Magnus Westerlund
AVTCORE WG chair

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Black, David" <[email-masked]>
Date: Tue, 14 Jun 2016 13:20:39 +0000

In the new ECN text in Section 7, I see:

   Given the above issues, implementations MAY ignore ECN-CE marks when	
   determining if the congestion circuit breaker triggers, since	
   excessive persistent congestion will eventually lead to packet loss	
   that will trigger the circuit breaker.  Doing this will protect the	
   network from congestion collapse, but might result in sub-optimal	
   user experience for competing flows that share the bottleneck queue,	
   since that queue will be driven to overflow, inducing high latency.
   If this is a concern, the only current guidance is for	
   implementations to treat ECN-CE marked packets as equivalent to lost	
   packets, whilst being aware that this might trigger the circuit	
   breaker prematurely in future, depending on how AQM and ECN	
   deployment evolves.  Developers that implement a circuit breaker	
   based on ECN-CE marks will need to track future developments in AQM	
   standards and deployed ECN marking behaviour, and ensure their	
   implementations are updated to match.

I think "MAY" is inappropriate here - this text reads like a "SHOULD NOT"
requirement with an explanation of what happens when something else
is done.   Proposed rewrite:

   Given the above issues, implementations SHOULD NOT ignore ECN-CE marks
   when	determining whether to trigger the congestion circuit breaker, even	
   though excessive persistent congestion will eventually lead to packet loss	
   that triggers the circuit breaker.   Relying on packet loss protects the	
   network from congestion collapse, but might result in sub-optimal	
   user experience for competing flows that share the bottleneck queue,	
   since that queue will be driven to overflow, inducing higher latency.

   Current implementation guidance is for
   implementations to treat ECN-CE marked packets as equivalent to lost	
   packets, whilst being aware that this might trigger the circuit	
   breaker prematurely in the future, depending on how AQM and ECN	
   deployment evolves.  Developers that implement a circuit breaker	
   based on ECN-CE marks will need to track future developments in AQM	
   standards and deployed ECN marking behaviour, and ensure their	
   implementations are updated to match.

The original "MAY" language invites implementers to completely ignore
ECN-CE marks, which I think is very bad advice.

Thanks, --David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Magnus Westerlund
> Sent: Tuesday, June 14, 2016 4:15 AM
> To: IETF AVTCore WG
> Cc: [email-masked]; tsvwg
> Subject: [tsvwg] WG Last Call on changes: draft-ietf-avtcore-rtp-circuit-breakers-
> 16
> 
> AVT, RTCWEB and TSVWG
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 14 Jun 2016 15:43:47 +0200

Hi,

Please see below.

> ECN-CE marks, which I think is very bad advice.

So in the discussion between Mirja, the authors and me in attempting to 
resolve the discuss we ended up in MAY for a reason. We on the call 
agreed after a lengthy discussion that treating a ECN-CE mark the same 
way as a loss, is not appropriate. This despite that current in force 
specification indicate that would be the answer. However, as indicated 
in the updated text, there are a number of ongoing work that indicates 
that is not the expected actual marking behavior.

This resulted in another issue, there are no specified marking behavior 
that we define an appropriate response based on. When such a behavior is 
defined one can update circuit breaker with a well defined response to 
ECN-CE marks. So due to the lack of defined behavior we didn't see a 
possibility for using SHOULD NOT, as we can't point to what should be 
implemented. Thus we chose to propose MAY as it then is open to the 
implementation if they can find a well working response or completely 
ignore this.

So far I lack evidence for any significant deployment of ECN for RTP, 
unfortunately. This lessens the impact of this choice.

Secondly, ignoring the ECN-CE marks for a circuit breaker we believe 
will have a limited impact. If the implementation only have circuit 
breaker, i.e. no full fledged congestion controller and uses ECN, they 
can in worst case drive the buffer into the overload regime where it 
starts dropping packets. So if the flow does cause persistent congestion 
it will drive the AQM into packet drops and thus triggering of the 
circuit breaker. Yes, it will be unfair and push away responsive flows. 
But, that will happen also without ECN, is not something that the 
circuit breaker can protect against.

I hope this makes it clearer why we arrived at MAY and don't see the 
possibility to use SHOULD strength wording regarding ECN-CE marks.

Cheers

Magnus Westerlund
AVTCORE Chair.

---

From: John Leslie <[email-masked]>
Date: Tue, 14 Jun 2016 09:59:27 -0400

Black, David <[email-masked]> wrote:
>...
> is done.

   I'm very sympathetic to your issue: there are implementations of ECN
out there which _will_not_ drop an ECN-capable packet before the sending
queue has overflowed.

   But at the same time, we're headed into changes where ECN-CE marking
will become more frequent than packet-drop.

   We must find a way to avoid ECN-CE triggering the congestion circuit
breaker when the forwarding node has not reached a level of congestion
which could trigger packet loss.

   I'd prefer language (perhaps in an RFC-3168-bis) stating that packet
drop MUST me used, at least occasionally, on packets which would be
ECN-CE marked, when the overall congestion at that node reaches the
drop level for non-ECN-capable packets. (The forwarding node is the only
node with sufficient information to see the issue.)

   (I recommend folks interested in circuit-breaker pay attention to the
L4S BoF being scheduled for Berlin.)

--
John Leslie <[email-masked]>

---

From: "Black, David" <[email-masked]>
Date: Tue, 14 Jun 2016 14:20:19 +0000

My problem is that the new "MAY" sentence is unqualified:

> >    determining if the congestion circuit breaker triggers,

That invites implementers to ignore ECN-CE marks without giving this topic the
level of consideration anticipated in this discussion and elsewhere in Section 7.
If "SHOULD NOT" language is unworkable, please propose alternate text that
qualifies applicability of the "MAY" - e.g., only after careful consideration of
everything that we're now discussing and the rest of the text in Section 7.

I proposed "SHOULD NOT" because it appear that the original RFC 2119 definition
of that term is a good fit to this situation:

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

Thanks, --David

> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Tuesday, June 14, 2016 9:44 AM
> To: Black, David; IETF AVTCore WG
> Subject: Re: [tsvwg] WG Last Call on changes: draft-ietf-avtcore-rtp-circuit-
> breakers-16
> 
> Hi,
> 
> Please see below.
>

---

From: Colin Perkins <[email-masked]>
Date: Wed, 15 Jun 2016 16:04:29 +0100

> On 14 Jun 2016, at 15:20, Black, David <[email-masked]> wrote:
> 

The issue with saying “implementations SHOULD NOT ignore ECN-CE marks”, is that it implies that they should respond to those marks. However, the outcome of the long discussion we had with Mirja was that there’s no consensus on what’s the correct response, and concern that implementations picking the wrong choice will hinder the deployment of ECN. 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Black, David" <[email-masked]>
Date: Wed, 15 Jun 2016 16:06:19 +0000

> The issue with saying “implementations SHOULD NOT ignore ECN-CE marks”, is
> that it implies that they should respond to those marks. However, the outcome
> of the long discussion we had with Mirja was that there’s no consensus on what’s
> the correct response, and concern that implementations picking the wrong
> choice will hinder the deployment of ECN.

Understood, and I'm not asking for that.  I view the current text as providing implementers
with too much latitude to ignore ECN-CE marks (e.g., because an implementer doesn't
want to think about this problem space in the first place).

Could someone propose initial text to qualifies the current "MAY ignore" statement?

I'm looking for some sort of applicability and prerequisite text that describes the
situations and design aspects that ought to be considered before an implementation
decides to ignore ECN-CE marks.  

Thanks, --David

> From: Colin Perkins [mailto:[email-masked]]
> Sent: Wednesday, June 15, 2016 11:04 AM
> To: Black, David
> Cc: Magnus Westerlund; IETF AVTCore WG; [email-masked]; tsvwg
> Subject: Re: [AVTCORE] [tsvwg] WG Last Call on changes: draft-ietf-avtcore-rtp-
> circuit-breakers-16
> 
> The issue with saying “implementations SHOULD NOT ignore ECN-CE marks”, is
> that it implies that they should respond to those marks. However, the outcome
> of the long discussion we had with Mirja was that there’s no consensus on what’s
> the correct response, and concern that implementations picking the wrong
> choice will hinder the deployment of ECN.
> 
> 
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: John Leslie <[email-masked]>
Date: Thu, 16 Jun 2016 18:25:48 -0400

Black, David <[email-masked]> wrote:
> 
> ...  I view the current text as providing implementers with too much
> latitude to ignore ECN-CE marks (e.g., because an implementer doesn't

   Understand, we have at least two proposals to make ECN-CE more frequent
than packet drop would be for non-ECN packets: possibly substantially
more frequent. Unless both are killed off, ECN-CE will show up frequently
enough that closing the flow on ECN-CE would kill too many connections.

   If you want circuit-breaking on such connections, there are two ways:
1. convince the forwarding nodes to drop packets if their queue exceeds
   design capacity; or
2. require the sender to send enough not-ECN-capable packets so that our
   receiver will see enough packet-drops when a circuit-breaker should
   activate.

   (I prefer the first option; but I wouldn't object to the second.)

   There really isn't any way for our circuit-breaker to know _how_much_
more frequent the ECN-CE marks may be. :^( We _will_ be sorry if we
allot the same frequency of CE packets as packet-drops to trigger the
circuit-breaker.

> Could someone propose initial text to qualifies the current "MAY ignore"

   Essentially, for the second option, you might propose text to the
effect of:
] 
] If too many ECN-CE packets are received, the sender SHOULD send some
] not-ECN-capable packets to determine whether enough packets along the
] path are being dropped to justify activating our circuit-breaker.

   I'm not enthusiastic about adding that; but it would resolve the issue.

   BTW, I'm 100% convinced that either of the proposals being considered
in TSVWG could bring _substantial_ benefit to RTCWEB traffic.

---

From: Colin Perkins <[email-masked]>
Date: Fri, 17 Jun 2016 11:14:08 +0100

> On 16 Jun 2016, at 23:25, John Leslie <[email-masked]> wrote:
> 

I agree, but the argument is that doing so is less harmful than deploying a circuit breaker that triggers too often when ECN is used. 

I’m not sure I believe this argument, though, since it seems that any new AQM that applies ECN marks much more often than at present will have to consider backwards compatibility, to work with deployed TCP (e.g., draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem uses ECT(1) as a signal to use the new marking, while existing implementations set ECT(0)). These compatibility mechanisms would seem to prevent the issues with the circuit breaker too.

> more frequent the ECN-CE marks may be. :^(

This is a problem, both for the circuit breaker, and for the algorithms being defined in RMCAT. We do need some understanding what the expected marking rates are likely to be, so congestion control and circuit breakers can be defined. 

> We _will_ be sorry if we
>   I’m not enthusiastic about adding that; but it would resolve the issue.

I’m not convinced this would work. The circuit breaker is looking at long term trends, and in order to have enough not-ECT packets to determine if it should trigger, you’d essentially have to run without ECN for some seconds. 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Black, David" <[email-masked]>
Date: Fri, 17 Jun 2016 14:03:48 +0000

Colin,

> I agree, but the argument is that doing so is less harmful than deploying a circuit
> breaker that triggers too often when ECN is used.
> 
> I’m not sure I believe this argument, though, since it seems that any new AQM
> that applies ECN marks much more often than at present will have to consider
> backwards compatibility, to work with deployed TCP (e.g., draft-briscoe-tsvwg-
> aqm-tcpm-rmcat-l4s-problem uses ECT(1) as a signal to use the new marking,
> while existing implementations set ECT(0)). These compatibility mechanisms
> would seem to prevent the issues with the circuit breaker too.

That roughly matches my line of thinking, and I'll observe that the original DCTCP
protocol design that used more aggressive ECN-CE marking was only safe for
Controlled Environment deployments.   See the TSVWG rfc5405bis draft for the
definition of Controlled Environment, and ignore the fact that the rfc5405bis
draft is a UDP draft - this definition is more broadly applicable.

Going back over Section 7 in this avtcore draft, my views are:

[A] None of these drafts justify a "MAY ignore" response to ECN-CE marks:
	- draft-khademi-tcpm-alternativebackoff-ecn
	- draft-ietf-rmcat-nada
	- draft-ietf-rmcat-scream-cc

[B] In line with Colin's comment on the L4S draft, I think it's incumbent on
the authors of draft-briscoe-aqm-dualq-coupled to figure out how that will
coexist (or avoid) deployed TCP, and this avtcore draft ought not to be
trying to prejudge what will be done there.

So, I don't think the current text in Section 7 has justified the unfettered
"implementations MAY ignore ECN-CE marks" text, as ignoring those marks
is not consistent with any of the four cited drafts.

In more detail, I think making changes to normative requirements here based
on [B] is premature, and I would hope that the rmcat WG could be encouraged
to consider the RTP circuit breaker in its congestion control drafts, as those CC
mechanisms are related to the circuit breaker mechanism, hence likely
to be in related areas of an RTP implementation.

That leaves draft-khademi-tcpm-alternativebackoff-ecn, which TSVWG
will be looking at in Berlin.  If a normative statement about ECN-CE reaction
is going to rest on that draft, then the reference to that draft should be
normative.  Something about doing that strikes me as premature ...

I realize that we're trying to predict and accommodate the future, which
is an imprecise undertaking at best.   As an alternative to the current text,
would it be reasonable to say (without any RFC 2119 keywords) that the
best current guidance is still to treat ECN-CE marks as indicating drops,
with a warning that there is a good possibility of this changing in the
near future due to all of the work in progress cited in Section 7?

Thanks, --David

> Sent: Friday, June 17, 2016 6:14 AM
> To: John Leslie; Black, David
> Cc: [email-masked]; IETF AVTCore WG; tsvwg
> Subject: Re: [rtcweb] [AVTCORE] [tsvwg] WG Last Call on changes: draft-ietf-
> avtcore-rtp-circuit-breakers-16
> 
> 
> I agree, but the argument is that doing so is less harmful than deploying a circuit
> breaker that triggers too often when ECN is used.
> 
> I’m not sure I believe this argument, though, since it seems that any new AQM
> that applies ECN marks much more often than at present will have to consider
> backwards compatibility, to work with deployed TCP (e.g., draft-briscoe-tsvwg-
> aqm-tcpm-rmcat-l4s-problem uses ECT(1) as a signal to use the new marking,
> while existing implementations set ECT(0)). These compatibility mechanisms
> would seem to prevent the issues with the circuit breaker too.
> 
> This is a problem, both for the circuit breaker, and for the algorithms being
> defined in RMCAT. We do need some understanding what the expected marking
> rates are likely to be, so congestion control and circuit breakers can be defined.
> 
> > We _will_ be sorry if we
> I’m not convinced this would work. The circuit breaker is looking at long term
> trends, and in order to have enough not-ECT packets to determine if it should
> trigger, you’d essentially have to run without ECN for some seconds.
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 17 Jun 2016 17:02:37 +0200

+1 to not use normative language here.

However, please note that having a high level of ECN-CE marks (without any losses) means that all packets were received correctly. This situation can even occurs without high delays (depending on the AQM used), which would just mean the services works perfectly. Therefore for me CE marks are a perfect input signal for a congestion control loop (where the AQM tell the sender to take action - whatever that means). But I’m less concerned than David about eventually ignoring it for circuit breaker.

In addition one point on something Magnus wrote earlier:
"If the implementation only have circuit breaker, i.e. no full fledged congestion controller and uses ECN, they can in worst case drive the buffer into the overload regime where it starts dropping packets. „

I’m not sure about this case. ECN is an input signal for congestion control. If you don’t use congestion control but only a circuit breaker, you should probably not enable ECN. At least it not clear to me why you would enable it, and it's definitely not conform to the ECN spec. Probably we should say something about this in the draft...?

Mirja

> Am 17.06.2016 um 16:03 schrieb Black, David <[email-masked]>:
> 
> Colin,
> 
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Gorry (erg)" <[email-masked]>
Date: Sat, 18 Jun 2016 07:22:35 +0100

I think we SHOULD NOT recommend to use ECN marks as inputs to a CB. See below:

> On 17 Jun 2016, at 16:02, Mirja Kühlewind <[email-masked]> wrote:
> 
> However, please note that having a high level of ECN-CE marks (without any losses) means that all packets were received correctly. This situation can even occurs without high delays (depending on the AQM used), which would just mean the services works perfectly. Therefore for me CE marks are a perfect input signal for a congestion control loop (where the AQM tell the sender to take action - whatever that means).

We may in future figure out ways to do this to detect significant failure using a rate adaptive transport and ECN e.g.  Observing 100% CE marks or something, for an RTP flow that is trying to send well below its peak rate decided by CC -- but I think this is speculating at an algorithm and adding details here is not a good idea. Especially as AQM continues to evolve.

> But I’m less concerned than David about eventually ignoring it for circuit breaker.
> 
Agree. Loss is the measurement that a CB MUST respond to.

> I’m not sure about this case. ECN is an input signal for congestion control. If you don’t use congestion control but only a circuit breaker, you should probably not enable ECN. At least it not clea…
Agree, enabling ECN without a responsive CC is going to lead to trouble.

> Mirja
Gorry

---

From: "Black, David" <[email-masked]>
Date: Mon, 20 Jun 2016 13:16:58 +0000

> Agree. Loss is the measurement that a CB MUST respond to.

Mumble.   I would be ok with a clear discouragement for use of ECN-CE marks, accompanied by the sort of design rationale here, or even better, a clear statement that lost packets for the purpose of the RTP circuit breaker have to be actually lost without getting into whether or not ECN-CE marks are involved -i.e., the RTP circuit breaker is specified against actual drops as a network protection backstop.

A related concern is that ECN marks may overstate equivalent loss behavior - a simplistic queue management discipline that marks every packet when the queue is over a threshold (NB: this class of marking behavior is NOT RECOMMENDED - a real AQM SHOULD be used) could yield a run of ECN-CE marks that would not cause a corresponding with a run of packet drops.   This is among the reasons that TCP reacts to ECN-CE marks only once per RTT, and might be a reason to treat multiple ECN-CE marks in an RTT interval as not representing drops of all packets for the RTP circuit breaker's TCP-equivalent throughput calculation.

Thanks, --David

> From: Gorry (erg) [mailto:[email-masked]]
> Sent: Saturday, June 18, 2016 2:23 AM
> To: Mirja Kühlewind
> Cc: Black, David; Magnus Westerlund; Colin Perkins; [email-masked]; IETF
> AVTCore WG; tsvwg
> Subject: Re: [tsvwg] [rtcweb] [AVTCORE] WG Last Call on changes: draft-ietf-
> > However, please note that having a high level of ECN-CE marks (without any
> losses) means that all packets were received correctly. This situation can even
> occurs without high delays (depending on the AQM used), which would just
> mean the services works perfectly. Therefore for me CE marks are a perfect input
> signal for a congestion control loop (where the AQM tell the sender to take action
> - whatever that means).
> 
> We may in future figure out ways to do this to detect significant failure using a
> rate adaptive transport and ECN e.g.  Observing 100% CE marks or something, for
> an RTP flow that is trying to send well below its peak rate decided by CC -- but I
> think this is speculating at an algorithm and adding details here is not a good idea.
> Especially as AQM continues to evolve.
> 
> > "If the implementation only have circuit breaker, i.e. no full fledged congestion
> controller and uses ECN, they can in worst case drive the buffer into the overload
> regime where it starts dropping packets. „
> >
> > I’m not sure about this case. ECN is an input signal for congestion control. If you
> don’t use congestion control but only a circuit breaker, you should probably not
> enable ECN. At least it not clear to me why you would enable it, and it's definitely
> not conform to the ECN spec. Probably we should say something about this in the
> draft...?
> >

---

From: Michael Welzl <[email-masked]>
Date: Mon, 20 Jun 2016 18:35:38 +0200

> On 20. jun. 2016, at 15.16, Black, David <[email-masked]> wrote:
> 

I’m not sure we need such complicated logic to find a case where ECN marks are different from packet drops:

Basically, they simply aren’t - even “real” AQMs marking isn’t exactly the same as a packet drop: the marks themselves inform you that an AQM did its job, and with modern AQMs like CoDel / PIE etc., you’re probably getting this from a shallow queue. Chances are that this is less than a BDP worth of queuing, which is our justification for recommending a different back-off behavior in draft-khademi-tsvwg-ecn-response-00 and draft-khademi-tcpm-alternativebackoff-ecn-00

So the point is not that AQMs would treat ECN marking and dropping differently - it’s that ECN indicates an AQM, and hence probably a shallow queue. With a drop, you just don’t know.

Back to the CB, I think an AQM marking at a shallow queue (like e.g. CoDel) is indeed quite different from a “broken connection”.

Cheers,
Michael

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 27 Jun 2016 15:46:08 +0200

AVT, RTCWEB, TSVWG and ADS.

I have to conclude that there is not consensus on the proposed text. It 
does appear that this subject do require additional discussion and most 
likely drafting of new text.

Please continue to provide input into the matter. I would like to 
conclude this subject no later than by the end of the Berlin meeting.

Cheers

Magnus

-- 

Magnus Westerlund

---

From: "Ben Campbell" <[email-masked]>
Date: Mon, 27 Jun 2016 09:40:17 -0500

Thanks, Magnus!

On 27 Jun 2016, at 8:46, Magnus Westerlund wrote:

> I have to conclude that there is not consensus on the proposed text. 
> It does appear that this subject do require additional discussion and 
> most likely drafting of new text.
>

---

From: "De Schepper, Koen (Nokia - BE)" <[email-masked]>
Date: Mon, 27 Jun 2016 14:53:12 +0000

As far as I understand, this draft is related to circuit breakers in end-systems, right?
It is the end system that determines the use of ECN (currently marking non-ect for drop and ect(0) for Classic ECN). 
In L4S we don't plan to change the behavior of Classic ECN, and ABE's behavior should be close to non-ABE ECN. So I guess there is no problem of describing the behavior of how a Classic ECN based sender would respond today. 

As we only want to significantly change the network behavior of ect(1) marking, can we solve this issue by recommending (or even requiring) senders to mark only ect(0) and describing the classic ECN circuit breaker? When L4S gets defined, also an L4S based circuit breaker extension can be defined for senders that want to use the L4S service (when senders send ect(1) packets).

Regards,
Koen.

> From: tsvwg [mailto:[email-masked]] On Behalf Of Michael Welzl
> Sent: maandag 20 juni 2016 18:36
> To: Black, David
> Cc: <[email-masked]> Fairhurst; Magnus Westerlund; tsvwg; IETF
> AVTCore WG; [email-masked]; Colin Perkins
> Subject: Re: [tsvwg] [rtcweb] [AVTCORE] WG Last Call on changes: draft-
> ietf-avtcore-rtp-circuit-breakers-16
> 
> 
> >>> But I’m less concerned than David about eventually ignoring it for
> circuit
> >> breaker.
> >>>
> > Mumble.   I would be ok with a clear discouragement for use of ECN-CE
> marks, accompanied by the sort of design rationale here, or even better,
> a clear statement that lost packets for the purpose of the RTP circuit
> breaker have to be actually lost without getting into whether or not
> ECN-CE marks are involved -i.e., the RTP circuit breaker is specified
> against actual drops as a network protection backstop.
> >
> > A related concern is that ECN marks may overstate equivalent loss
> behavior - a simplistic queue management discipline that marks every
> packet when the queue is over a threshold (NB: this class of marking
> behavior is NOT RECOMMENDED - a real AQM SHOULD be used) could yield a
> run of ECN-CE marks that would not cause a corresponding with a run of
> packet drops.   This is among the reasons that TCP reacts to ECN-CE
> marks only once per RTT, and might be a reason to treat multiple ECN-CE
> marks in an RTT interval as not representing drops of all packets for
> the RTP circuit breaker's TCP-equivalent throughput calculation.
> 
> Basically, they simply aren’t - even “real” AQMs marking isn’t exactly
> the same as a packet drop: the marks themselves inform you that an AQM
> did its job, and with modern AQMs like CoDel / PIE etc., you’re probably
> getting this from a shallow queue. Chances are that this is less than a
> BDP worth of queuing, which is our justification for recommending a
> different back-off behavior in draft-khademi-tsvwg-ecn-response-00 and
> draft-khademi-tcpm-alternativebackoff-ecn-00
> 
> So the point is not that AQMs would treat ECN marking and dropping
> differently - it’s that ECN indicates an AQM, and hence probably a
> shallow queue. With a drop, you just don’t know.
> 
> Back to the CB, I think an AQM marking at a shallow queue (like e.g.
> CoDel) is indeed quite different from a “broken connection”.
> 
> Cheers,
> Michael
> 
> 
> >
> > Thanks, --David
> >
> >> I think we SHOULD NOT recommend to use ECN marks as inputs to a CB.
> See
> >> below:
> >>
> >>> On 17 Jun 2016, at 16:02, Mirja Kühlewind
> <[email-masked]>
> >> wrote:
> >>>
> >>> But I’m less concerned than David about eventually ignoring it for
> circuit
> >> breaker.
> >>>
> >>>>> I agree, but the argument is that doing so is less harmful than
> deploying a
> >> circuit
> >>>>> I’m not sure I believe this argument, though, since it seems that
> any new
> >> AQM
> >>>>> that applies ECN marks much more often than at present will have
> to
> >> consider
> >>>>> backwards compatibility, to work with deployed TCP (e.g., draft-
> briscoe-
> >> tsvwg-
> >>>> That roughly matches my line of thinking, and I'll observe that the
> original
> >> DCTCP
> >>>> Controlled Environment deployments.   See the TSVWG rfc5405bis
> draft for
> >> the
> >>>>> I agree, but the argument is that doing so is less harmful than
> deploying a
> >> circuit
> >>>>> I’m not sure I believe this argument, though, since it seems that
> any new
> >> AQM
> >>>>> that applies ECN marks much more often than at present will have
> to
> >> consider
> >>>>> backwards compatibility, to work with deployed TCP (e.g., draft-
> briscoe-
> >> tsvwg-
> >>>>> rates are likely to be, so congestion control and circuit breakers
> can be
> >> defined.
> >>>>>
> >>>>>> We _will_ be sorry if we
> >>>>>>> Could someone propose initial text to qualifies the current "MAY
> ignore"
> >>>>>>> statement?
> >>>>>>
> >>>>> trends, and in order to have enough not-ECT packets to determine
> if it
> >> should

---

From: [email-masked]
Date: Mon, 27 Jun 2016 16:38:16 +0100

I think thinking of L4S is maybe off at a tangent. The question really is
about the interpretation of loss and CE-mark as equivalent. I argued that 
each ECN-CE mark should not be counted as equivalent to a lost segment -
in this context we should use ECN to drive a CC algorithm and we should be
cautious to avoid requiring its use within a Circuit Breaker - optional
use, if you understand how to interpret a reaction to many CE-marks as
excessive congestion, are permitted.

Gorry

> It is the end system that determines the use of ECN (currently marking
> non-ect for drop and ect(0) for Classic ECN).
>
> In L4S we don't plan to change the behavior of Classic ECN, and ABE's
> behavior should be close to non-ABE ECN. So I guess there is no problem of
> describing the behavior of how a Classic ECN based sender would respond
> today.
>
> As we only want to significantly change the network behavior of ect(1)
> marking, can we solve this issue by recommending (or even requiring)
> senders to mark only ect(0) and describing the classic ECN circuit
> breaker? When L4S gets defined, also an L4S based circuit breaker
> extension can be defined for senders that want to use the L4S service
> (when senders send ect(1) packets).
>
> Regards,
> Koen.
>
>
>> >> controller and uses ECN, they can in worst case drive the buffer
>> into
>> the overload
>> >>> I’m not sure about this case. ECN is an input signal for
>> congestion
>> control. If you
>> >> don’t use congestion control but only a circuit breaker, you
>> should
>> probably not
>> >>>> with a warning that there is a good possibility of this changing
>> in
>> the
>> >>>>>> allot the same frequency of CE packets as packet-drops to
>> trigger
>> the
>> >>>>>> circuit-breaker.
>> >>>>>>
>> >>>>>> ] If too many ECN-CE packets are received, the sender SHOULD
>> send
>> some
>> >>>>>> I’m not enthusiastic about adding that; but it would resolve
>> the
>> issue.
>> >>>>>
>> >>>>> I’m not convinced this would work. The circuit breaker is
>> looking
>> at long term

---

From: "De Schepper, Koen (Nokia - BE)" <[email-masked]>
Date: Mon, 27 Jun 2016 19:12:50 +0000

> From: [email-masked] [mailto:[email-masked]]
> Sent: maandag 27 juni 2016 17:38
> To: De Schepper, Koen (Nokia - BE)
> Cc: Michael Welzl; Black, David; [email-masked]; Magnus
> Westerlund; tsvwg; IETF AVTCore WG; [email-masked]; Colin Perkins

Why not? As long as an AQM is marking at the same rate as dropping, a 
100% marking means that non-ecn flows are being dropped at a 100%, not?

Koen.

> >> I’m not sure we need such complicated logic to find a case where
> ECN
> >> marks are different from packet drops:
> >>
> >> Basically, they simply aren’t - even “real” AQMs marking
> isn’t
> >> exactly
> >> >> Cc: Black, David; Magnus Westerlund; Colin Perkins;
> [email-masked];
> >> IETF
> >> >> AVTCore WG; tsvwg
> >> >> occurs without high delays (depending on the AQM used), which
> would
> >> just
> >> >> signal for a congestion control loop (where the AQM tell the
> sender
> >> to take action
> >> >> - whatever that means).
> >> >>
> >> >> an RTP flow that is trying to send well below its peak rate
> decided
> >> by CC -- but I
> >> >> think this is speculating at an algorithm and adding details here
> is
> >> not a good idea.
> >> >> enable ECN. At least it not clear to me why you would enable it,
> and
> >> it's definitely
> >> >> not conform to the ECN spec. Probably we should say something
> about
> >> this in the
> >> >> draft...?
> >> >>>
> >> >>>>>>> ...  I view the current text as providing implementers with
> too
> >> much
> >> >>>>> aqm-tcpm-rmcat-l4s-problem uses ECT(1) as a signal to use the
> new
> >> marking,
> >> >>>> protocol design that used more aggressive ECN-CE marking was
> only
> >> safe for
> >> >>>> definition of Controlled Environment, and ignore the fact that
> the
> >> rfc5405bis
> >> >>>> [A] None of these drafts justify a "MAY ignore" response to ECN-
> CE
> >> marks:
> >> >>>> coexist (or avoid) deployed TCP, and this avtcore draft ought
> not
> >> to be
> >> >>>> So, I don't think the current text in Section 7 has justified
> the
> >> unfettered
> >> >>>> "implementations MAY ignore ECN-CE marks" text, as ignoring
> those
> >> marks
> >> >>>> will be looking at in Berlin.  If a normative statement about
> ECN-
> >> CE reaction
> >> >>>> I realize that we're trying to predict and accommodate the
> future,
> >> which
> >> >>>> would it be reasonable to say (without any RFC 2119 keywords)
> that
> >> the
> >> >>>> best current guidance is still to treat ECN-CE marks as
> indicating
> >> drops,
> >> >>>>> Subject: Re: [rtcweb] [AVTCORE] [tsvwg] WG Last Call on
> changes:
> >> draft-ietf-
> >> >>>>>>> ...  I view the current text as providing implementers with
> too
> >> much
> >> >>>>> aqm-tcpm-rmcat-l4s-problem uses ECT(1) as a signal to use the
> new
> >> marking,
> >> >>>>>> If you want circuit-breaking on such connections, there are
> two
> >> ways:
> >> >>>>>> 1. convince the forwarding nodes to drop packets if their
> queue
> >> exceeds
> >> >>>>>> design capacity; or
> >> >>>>>> 2. require the sender to send enough not-ECN-capable packets
> so
> >> that our
> >> >>>>> defined in RMCAT. We do need some understanding what the
> expected
> >> >> marking

---

From: "Black, David" <[email-masked]>
Date: Mon, 27 Jun 2016 20:09:30 +0000

> As long as an AQM is marking at the same rate as dropping

That's an interesting assumption - it should be true for AQMs vetted
here in the past, but there are easy ways for it not to hold (e.g., if dropping
or marking is based on queue occupancy, it is possible that dropping
reduces queue occupancy in a fashion that marking does not).

For ECN "classic" (i.e., see RFC 3168) where ECN-CE markings are treated
as drop-equivalent, that is for congestion control purposes, which is similar
to, (but not the same as) the throughput estimation usage for the RTP circuit
breaker.    I'll note that ECN "classic" was designed congestion control
algorithms for react to ECN-CE marks once per RTT, independent of how
many ECN-CE marks are observed in an RTT.

> > excessive congestion, are permitted.

Something like that may be workable, starting with a clear distinction between
the use of ECN by CC (routine, active at all times) and ECN by a circuit
breaker (monitors for evidence that things have gotten bad, only activated
when things get bad).   This would baseline the RTP circuit breaker on drops
and allow use of ECN as additional evidence of problems, in contrast to
congestion control where ECN-CE is effectively treated as drop-equivalent.

I'm not quite sure how to specify "use of ECN as additional evidence" of
"excessive congestion" as drop-equivalence is about the best we have
for current guidance.

Thanks, --David

> From: De Schepper, Koen (Nokia - BE) [mailto:koen.de_schepper@nokia-bell-
> labs.com]
> Sent: Monday, June 27, 2016 3:13 PM
> To: [email-masked]
> Cc: Michael Welzl; Black, David; Magnus Westerlund; tsvwg; IETF AVTCore WG;
> [email-masked]; Colin Perkins
> Subject: RE: [tsvwg] [rtcweb] [AVTCORE] WG Last Call on changes: draft-ietf-

---

From: Michael Welzl <[email-masked]>
Date: Mon, 27 Jun 2016 22:43:42 +0200

> On 27. jun. 2016, at 21.12, De Schepper, Koen (Nokia - BE) <[email-masked]> wrote:
> 
> 

Yes you can construct an AQM mechanism to act precisely like that.
Do we want rules that make every other behavior “illegal” ?  Should this really be how end systems should interpret all forms of CE-marking?

I think we both would agree that the answer is no: both the L4S and the ABE work are based on the premise that a CE-mark does *NOT* necessarily mean the same as packet loss would.

So I agree with Gorry here - a circuit breaker is a bit of a special beast. It really mustn’t trigger by accident.

Cheers,
Michael

>>>>>>> But Iâ€™m less concerned than David about eventually ignoring it
>>>> for
>>>> circuit
>>>>>> breaker.
>>>>>>> 
>>>> Iâ€™m not sure we need such complicated logic to find a case where
>> ECN
>>>> Basically, they simply arenâ€™t - even â€œrealâ€ AQMs marking
>> isnâ€™t
>>>> exactly
>>>> did its job, and with modern AQMs like CoDel / PIE etc., youâ€™re
>>>> probably
>>>> differently - itâ€™s that ECN indicates an AQM, and hence probably a
>>>> shallow queue. With a drop, you just donâ€™t know.
>>>> 
>>>> CoDel) is indeed quite different from a â€œbroken connectionâ€.
>>>> 
>>>> Cheers,
>>>> Michael
>>>> 
>>>> 
>>>>> 
>>>>> Thanks, --David
>>>>> 
>>>>>> To: Mirja KÃ¼hlewind
>>>>>>> On 17 Jun 2016, at 16:02, Mirja KÃ¼hlewind
>>>>>>> But Iâ€™m less concerned than David about eventually ignoring it
>>>> for
>>>> circuit
>>>>>> breaker.
>>>>>>> 
>>>>>> regime where it starts dropping packets. â€ž
>>>>>>> 
>>>>>>> Iâ€™m not sure about this case. ECN is an input signal for
>>>> congestion
>>>> control. If you
>>>>>> donâ€™t use congestion control but only a circuit breaker, you
>>>> should
>>>> probably not
>>>>>>>>> Iâ€™m not sure I believe this argument, though, since it seems
>>>> that
>>>> any new
>>>>>> AQM
>>>>>>>>> Iâ€™m not sure I believe this argument, though, since it seems
>>>> that
>>>> any new
>>>>>> AQM
>>>>>>>>>> Iâ€™m not enthusiastic about adding that; but it would resolve
>>>> the
>>>> issue.
>>>>>>>>> 
>>>>>>>>> Iâ€™m not convinced this would work. The circuit breaker is
>>>> looking
>>>> at long term
>>>>>>>>> trigger, youâ€™d essentially have to run without ECN for some
>>>> seconds.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Colin Perkins
>>>>>>>>> https://csperkins.org/
>>>>>>>>

---

From: Michael Welzl <[email-masked]>
Date: Mon, 27 Jun 2016 22:52:40 +0200

David,

> On 27. jun. 2016, at 22.09, Black, David <[email-masked]> wrote:
> 

I fail to parse that sentence, so maybe I’m getting you wrong, but anyway I wonder: what’s even the point of this?
Why even bother considering CE-marks as information for a circuit breaker?

CE-marks may *not* indicate *excessive* congestion - and since you say “additional evidence”: I don’t think that a combination of loss and CE-marks makes this any better? CE-marks may be produced by a shallow queue, which can be rather “mild” congestion, at least in the light of what a circuit breaker should consider…

Cheers,
Michael

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 Jun 2016 23:02:17 +0100

> On 27 Jun 2016, at 21:52, Michael Welzl <[email-masked]> wrote:
> 
> David,
> 
> 

Because the alternative is that we only break the circuit once the queue has been driven into overflow, and packets have been lost. We want to avoid that, since it causes latency, and too much latency is very bad for the user experience. 

> CE-marks may *not* indicate *excessive* congestion - and since you say “additional evidence”: I don’t think that a combination of loss and CE-marks makes this any better? CE-marks may be produced b…

Surely this is just arguing for a different threshold for a circuit breaker triggered by ECN-CE marks (using a modern, small queue, AQM) than for one triggered by loss (or ECN marks considered equivalent to loss)? 

If I understand the L4S proposal correctly, that would be treat ECN-CE marks on ECT(0) marked flows as equivalent to loss, but treat ECN-CE marks on ECT(1) marked flows with a (much) higher threshold. 

Assuming, in all cases, that there’s a parallel congestion control algorithm running (and RMCAT has figured out the right congestion response for that; the proposals now treat ECN-CE and loss very similarly).

-- 
Colin Perkins
https://csperkins.org/

---

From: Michael Welzl <[email-masked]>
Date: Tue, 28 Jun 2016 00:29:26 +0200

> On 28. jun. 2016, at 00.02, Colin Perkins <[email-masked]> wrote:
> 
> 

Well - the better way out would be for the application to react. Maybe this is me misunderstanding the circuit breaker, but I did think it’s more like a last resort… you just don’t want to be trigger-happy with such a thing?

> Surely this is just arguing for a different threshold for a circuit breaker triggered by ECN-CE marks (using a modern, small queue, AQM) than for one triggered by loss (or ECN marks considered equi…

If you have room for yet another code point, for the circuit breaker only?  :)    Or maybe I just misunderstand you here?

> If I understand the L4S proposal correctly, that would be treat ECN-CE marks on ECT(0) marked flows as equivalent to loss, but treat ECN-CE marks on ECT(1) marked flows with a (much) higher thresho…

L4S would not change anything about how ECT(0) marked flows are treated, and would CE-mark packets carrying ECT(1) with an instantaneous queue - i.e. a much *lower* threshold. But that’s not the issue - I agree there’s no problem with L4S.

The compatibility problem does exist with the ABE proposal, which works off ECT(0).

The ABE proposal exploits a very simple fact: that CE-marks are, by definition, *not* the same as loss (see David Black’s previous email where he says "if dropping or marking is based on queue occupancy, it is possible that dropping reduces queue occupancy in a fashion that marking does not”). Indeed, queue dynamics play out differently when packets are dropped or marked  ( see Section 7 with Figures 13/14 in https://www.duo.uio.no/bitstream/handle/10852/37381/khademi-AQM_Kids_TR434.pdf ) .

Losses may stem from a DropTail (FIFO) queue somewhere along the path - CE-marks are, however, very likely to only be caused by an AQM algorithm. TCP’s built-in reaction to loss yields full link utilization only when there’s at least a BDP worth of queuing. This is a lot of latency - when the queue is full this doubles the RTT. Modern AQM mechanisms strive to maintain a much smaller average queue size, and this is where they mark packets.

So: if we react to CE-marks the same way as to loss, CoDel and PIE let us underutilize the link.

Thus, it makes more sense to interpret the signal for what it is: an indication that there was congestion, but from a queue that might be much smaller than a BDP.

> Assuming, in all cases, that there’s a parallel congestion control algorithm running

If you assume that there’s a parallel congestion control algorithm running, I understand even less why you want to feed ECN CE-marks into the circuit breaker. The congestion control algorithm should already deal with them.

> (and RMCAT has figured out the right congestion response for that; the proposals now treat ECN-CE and loss very similarly).

I disagree that this is the “right” congestion response. It’s a workable one, sure. Nothing extremely terrible will happen if congestion controllers treat ECN-CE and loss similarly - it just yields unnecessarily poor utilization with ECN, with modern AQMs  (unless one backs off by less than TCP would in response to loss too, which is good if there’s an AQM in place but may be quite bad otherwise).

Bottom line: it really does mean something different, and it seems wrong to me to act as if that wasn’t the case - just because we’ve always done so.

Cheers,
Michael

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Mon, 27 Jun 2016 23:03:03 +0000

> which could trigger packet loss.

I'm not sure I agree.

I tend to think that ECN is relatively cheap - it asks the sender to reduce its effective window but doesn't make him retransmit anything and causes no delivery delay - while packet loss is relatively expensive in that it makes the sender retransmit and introduces at least a one RTT hiccup ( to detect and retransmit the lost packet) in the transmission stream. Neither actually changes the rate at which anyone sends (we talk about the signal asking the sender to slow down, but the rate demonstrably doesn't actually change - we're asking the sender to not pack the pipe so full).

When ECN does ask someone to reduce their effective window, if they are typically increasing it by one per RTT, ECN could very logically reduce it by a small amount, such as two or three. If it reduced it by two, for example, that would cause it to skip the next two transmission opportunities (the next two received acknowledgements, which might actually arrive in the same ack), and would be back to the same level two RTTs later. If ECN is typically keeping the delay relatively shallow, it is oscillating between "a little shallower" and " a little less shallow". So it gets CE every other RTT? So what? If I'm dropping a packet, or a couple of packets, every other RTT, we're going to see notes on NANOG and other lists. "Your network's broken again. It's dropping packets."

I could be argued into setting the CE threshold a little lower than the discard threshold. There are probably some mechanisms we'd do well to explore, but...

---

From: John Leslie <[email-masked]>
Date: Mon, 27 Jun 2016 20:04:00 -0400

Colin Perkins <[email-masked]> wrote:
> Because the alternative is that we only break the circuit once the queue
> has been driven into overflow, and packets have been lost.
> We want to avoid that, since it causes latency, and too much latency is
> very bad for the user experience. 

   If you want a circuit-breaker to control latency, why aren't you measuring
latency?

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 28 Jun 2016 00:10:42 +0000

> On Jun 27, 2016, at 5:04 PM, John Leslie <[email-masked]> wrote:
> 

You mean, like CoDel?

---

From: "Black, David" <[email-masked]>
Date: Tue, 28 Jun 2016 01:04:11 +0000

Trying to shorten up this thread again ...

> > Because the alternative is that we only break the circuit once the queue has
> been driven into overflow, and packets have been lost. We want to avoid that,
> since it causes latency, and too much latency is very bad for the user experience.
> 
> Well - the better way out would be for the application to react. Maybe this is me
> misunderstanding the circuit breaker, but I did think it’s more like a last resort…
> you just don’t want to be trigger-happy with such a thing?

Well, the RTP circuit breaker draft is not trigger happy - for its congestion circuit
breaker to trip, RTP has to be sending at 10x the rate that TCP would send under
those conditions, based on the TCP throughput equation.  See:

https://tools.ietf.org/html/draft-ietf-avtcore-rtp-circuit-breakers-16#section-4.3

The issue here is - when calculating the comparable TCP throughput, how are ECN-CE
marks used to determine the loss rate input to the TCP throughput equation?  Do
ECN-CE marked packets count as having arrived or having been dropped?

When things are relatively stable and the ECN-CE marks are being used to nudge
the sender's rate based on what the network can absorb, whether ECN-CE marks
count as losses or not is probably immaterial - the 10x divergence from the TCP
throughput equation's rate is not going to arise, and the circuit breaker won't trip.
The circuit breaker is only supposed to trip when things are seriously wrong.

(1) If the RTP congestion circuit breaker trips based on ECN-CE marks alone,
something feels intuitively wrong - how'd we get to RTP running at 10x the
comparable TCP sending rate with no losses?  Perhaps the circuit breaker
shouldn't trip on ECN-CE marks alone?

(2) At the other extreme, the congestion circuit breaker clearly has to trip if RTP
gets to 10x the comparable TCP sending rate based on losses alone.  This is the
baseline for the circuit breaker to provide network protection as intended.

So, going back to Gorry's suggestion to use ECN-CE marks as "additional evidence,"
here's a straw proposal to shoot at ... factor in ECN-CE marks as additional losses
*only when* losses are already occurring.   

For example, we could specify that for the RTP congestion circuit breaker to trip, the
RTP sending rate has to be:
	- 10x the equivalent TCP sending rate based on counting ECN-CE marked
		packets as lost; AND
	- 3x the equivalent sending rate based on actual drops (i.e., counting
		ECN-CE marked packets as delivered).
The "3x" above is an off-the-top-of-my-head factor that attempts to roughly
equally weight the inputs (3 is close to the square root of 10) - pick a different
number if that weighting feels wrong.

This would force drops to occur and then consider ECN-CE marks as additional evidence
that something is wrong in the network.

Another possible rationale for this mixing is that if drops start occurring, then many of
the new and proposed uses of ECN that treat ECN-CE marks as less than loss-equivalent
are outside their intended operating envelopes/regions.

Thanks, --David

---

From: Colin Perkins <[email-masked]>
Date: Tue, 28 Jun 2016 22:48:21 +0100

> On 28 Jun 2016, at 01:04, John Leslie <[email-masked]> wrote:

Because the RTP circuit breaker needs to work with unmodified RFC 3550-style RTCP, which doesn’t return RTT samples sufficiently often to allow that, and because we don’t care about the precise latency, we just don’t want to fill a large queue.

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 28 Jun 2016 23:02:48 +0100

> On 28 Jun 2016, at 02:04, Black, David <[email-masked]> wrote:
> 

Right - or do they count somewhere between the two.

> The circuit breaker is only supposed to trip when things are seriously wrong.

Correct.

> shouldn’t trip on ECN-CE marks alone?

Shouldn’t the comparable rate to trigger the circuit breaker be 10x that given to a TCP flow subject to the same ECN-CE marking rate? If the TCP treats ECN-CE as equivalent to loss, for congestion response, then the circuit breaker should do so to, etc.

> are outside their intended operating envelopes/regions.

Clearly if the queue has been driven to overflow, so that packet loss is occurring, then the AQM is outside its intended operating regime. I’m not sure we need to push it so far, though. Is there not a regime where the ECN-CE marking rate indicates excessive congestion, before the queue overflows and drops packets? 

-- 
Colin Perkins
https://csperkins.org/

---

From: <[email-masked]>
Date: Wed, 29 Jun 2016 06:50:08 +0000

Colin, David,

I don't intend to be disturbing. Is there a standards track RFC indicating how a router should be configured to support ECN marking so that there is a network behavior by which an end system is able to relate ECN marking rates unambiguously to drop rates in the network (assuming both to be caused by the same bottleneck)?

I don't need an RFC number, "yes" or "no" will do. The RFC should be clear enough to allow any terminal to interpret the ECN signals correctly, no matter to which ECN supporting network it connects. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg [mailto:[email-masked]] Im Auftrag von Colin Perkins
Gesendet: Mittwoch, 29. Juni 2016 00:03
An: Black, David
Cc: [email-masked]; tsvwg; Michael Welzl; IETF AVTCore WG
Betreff: Re: [tsvwg] [AVTCORE] [rtcweb] WG Last Call on changes: draft-ietf-avtcore-rtp-circuit-breakers-16

>>>>> I'm not quite sure how to specify "use of ECN as additional 
>>>>> evidence" of "excessive congestion" as drop-equivalence is about 
>>>>> the best we have for current guidance.
>>>> 
>>>> I fail to parse that sentence, so maybe I’m getting you wrong, but 
>>>> anyway I
>>>> wonder: what’s even the point of this?
>> been driven into overflow, and packets have been lost. We want to 
>> avoid that, since it causes latency, and too much latency is very bad for the user experience.
>> 
>> Well - the better way out would be for the application to react. 
>> Maybe this is me misunderstanding the circuit breaker, but I did 
>> think it’s more like a last resort… you just don’t want to be trigger-happy with such a thing?
> 
> Well, the RTP circuit breaker draft is not trigger happy - for its 
> congestion circuit breaker to trip, RTP has to be sending at 10x the 
> rate that TCP would send under those conditions, based on the TCP throughput equation.  See:
> 
> #section-4.3
> The issue here is - when calculating the comparable TCP throughput, 
> how are ECN-CE marks used to determine the loss rate input to the TCP 
> throughput equation?  Do ECN-CE marked packets count as having arrived or having been dropped?

> When things are relatively stable and the ECN-CE marks are being used 
> to nudge the sender's rate based on what the network can absorb, 
> whether ECN-CE marks count as losses or not is probably immaterial - 
> the 10x divergence from the TCP throughput equation's rate is not going to arise, and the circuit breaker won't trip.

Correct.

> (1) If the RTP congestion circuit breaker trips based on ECN-CE marks 
> alone, something feels intuitively wrong - how'd we get to RTP running 
> at 10x the comparable TCP sending rate with no losses?  Perhaps the 
> circuit breaker shouldn’t trip on ECN-CE marks alone?

> (2) At the other extreme, the congestion circuit breaker clearly has 
> to trip if RTP gets to 10x the comparable TCP sending rate based on 
> losses alone.  This is the baseline for the circuit breaker to provide network protection as intended.
> 
> For example, we could specify that for the RTP congestion circuit 
> breaker to trip, the RTP sending rate has to be:
> The "3x" above is an off-the-top-of-my-head factor that attempts to 
> roughly equally weight the inputs (3 is close to the square root of 
> 10) - pick a different number if that weighting feels wrong.
> 
> This would force drops to occur and then consider ECN-CE marks as 
> additional evidence that something is wrong in the network.
> 
> Another possible rationale for this mixing is that if drops start 
> occurring, then many of the new and proposed uses of ECN that treat 
> ECN-CE marks as less than loss-equivalent are outside their intended operating envelopes/regions.

--
Colin Perkins
https://csperkins.org/

---

From: Michael Welzl <[email-masked]>
Date: Wed, 29 Jun 2016 10:54:32 +0200

> On 29. jun. 2016, at 00.02, Colin Perkins <[email-masked]> wrote:
> 
> 

Let’s see them clearly for what they are.
They mean: the path is *not* broken (they have arrived!), and a probably an AQM mechanism, potentially using a shallow queue, marked them to indicate congestion. I think “somewhere between the two” really doesn’t capture this well.

> Shouldn’t the comparable rate to trigger the circuit breaker be 10x that given to a TCP flow subject to the same ECN-CE marking rate? If the TCP treats ECN-CE as equivalent to loss, for congestion…

First, TCP shouldn’t (treat ECN-CE as equivalent to loss), and so the circuit breaker shouldn’t.
Second, I guess you’re talking about the equation. Well that goes completely wrong anyway (the derivation assumes packets to be lost, not marked; then again, you’re using loss, not the loss event ratio; then again, you’re close to this with ECN perhaps, using “traditional” ECN receiver behavior).

>> *only when* losses are already occurring.   

I think this is very reasonable.

> Clearly if the queue has been driven to overflow, so that packet loss is occurring, then the AQM is outside its intended operating regime. I’m not sure we need to push it so far, though. Is there n…

Shouldn’t a congestion control mechanism react well before that?

Cheers,
Michael

---

From: "Black, David" <[email-masked]>
Date: Wed, 29 Jun 2016 14:44:09 +0000

Colin responded:

> Clearly if the queue has been driven to overflow, so that packet loss is
> occurring, then the AQM is outside its intended operating regime. I’m not sure
> we need to push it so far, though. Is there not a regime where the ECN-CE
> marking rate indicates excessive congestion, before the queue overflows and
> drops packets?

Yes, but that may not be relevant to this discussion.   My hypothesis for this discussion
is that actual drops will occur well before RTP  is running at 10x the TCP equivalent rate
(based on drops and ECN-CE marks), and that 10x factor is the trip threshold for the
circuit breaker, which is the focus of this discussion. 

I would think that the "regime where the ECN-CE marking rate indicates excessive
congestion, before the queue overflows and drops packets?" would not extend as
far as RTP running at 10x the TCP equivalent rate, which is where the RTP circuit
breaker trips.

This is all a thought exercise - I'm happy to be shown to be wrong based on actual
data and experience with "running code" ...

> Shouldn’t a congestion control mechanism react well before that?

Yes, if there is one ;-).  This is RTP ...

Thanks, --David

> From: Michael Welzl [mailto:[email-masked]]
> Sent: Wednesday, June 29, 2016 4:55 AM
> To: Colin Perkins
> Cc: Black, David; [email-masked]; tsvwg; IETF AVTCore WG
> Subject: Re: [AVTCORE] [rtcweb] [tsvwg] WG Last Call on changes: draft-ietf-
> >>>>> I fail to parse that sentence, so maybe I’m getting you wrong, but anyway
> I
> They mean: the path is *not* broken (they have arrived!), and a probably an
> AQM mechanism, potentially using a shallow queue, marked them to indicate
> congestion. I think “somewhere between the two” really doesn’t capture this
> well.
> 
> 
> > Shouldn’t the comparable rate to trigger the circuit breaker be 10x that given to
> a TCP flow subject to the same ECN-CE marking rate? If the TCP treats ECN-CE as
> equivalent to loss, for congestion response, then the circuit breaker should do so
> to, etc.
> 
> Second, I guess you’re talking about the equation. Well that goes completely
> wrong anyway (the derivation assumes packets to be lost, not marked; then
> again, you’re using loss, not the loss event ratio; then again, you’re close to this
> with ECN perhaps, using “traditional” ECN receiver behavior).
> 
> 
> > Clearly if the queue has been driven to overflow, so that packet loss is
> occurring, then the AQM is outside its intended operating regime. I’m not sure
> we need to push it so far, though. Is there not a regime where the ECN-CE
> marking rate indicates excessive congestion, before the queue overflows and
> drops packets?
>

---

From: Michael Welzl <[email-masked]>
Date: Wed, 29 Jun 2016 16:54:09 +0200

> On 29. jun. 2016, at 16.44, Black, David <[email-masked]> wrote:
> 
> I wrote: 
> 

Seems reasonable to me….

> Yes, if there is one ;-).  This is RTP …

Earlier, in the same conversation, so - I assume - the same context, Colin wrote:
"Assuming, in all cases, that there’s a parallel congestion control algorithm running”

Cheers,
Michael

---

From: Colin Perkins <[email-masked]>
Date: Thu, 30 Jun 2016 15:51:20 +0100

> On 29 Jun 2016, at 15:54, Michael Welzl <[email-masked]> wrote:

Does anyone have data to let us find out either way? 

> "Assuming, in all cases, that there’s a parallel congestion control algorithm running”

If the congestion control is working, then it should adapt the rate so the circuit breaker never triggers. Yes, there should be a parallel congestion control, but for the circuit breaker to be relevant, that congestion control is either not working properly, or not implemented. 

-- 
Colin Perkins
https://csperkins.org/

---

From: "De Schepper, Koen (Nokia - BE)" <[email-masked]>
Date: Thu, 30 Jun 2016 15:58:42 +0000

> From: Black, David [mailto:[email-masked]]
> Sent: maandag 27 juni 2016 22:10
> Cc: tsvwg; IETF AVTCore WG; [email-masked]; Black, David
> Subject: RE: [tsvwg] [rtcweb] [AVTCORE] WG Last Call on changes: draft-
> here in the past,

And I hope for ect(0) the network will keep doing this in the future, at 
least not to deviate too far from this.
L4S proposes to use ect(1) just because it deviates too much, so the network 
can detect the different end-system behavior and can couple it correctly to 
what it is doing for drop and ect(0).

> but there are easy ways for it not to hold (e.g., if
> dropping

Do you mean that this results in different mark and drop probabilities 
for competing ECN and non-ECN flows?

The impact on the dropping/marking probability *value* applied by an AQM can change 
due to presence of ECN flows, but I don't think this results in a different 
drop compared to mark probability. Up to now all classic AQMs smooth their p 
over sufficient long time to remove short fluctuations in queue sizes. 

From an end system I think for classic ECN it is safe to assume that if you 
detect high levels of drop, it means there is a high level of 
congestion AND the network takes care of it. If you detect a high level
of marking, it means lots of congestion and nobody is currently taking care 
of it. Everybody is looking at you as a sender to take care of it. That's why
I think you should not use ECN if you don't use congestion control.

It also means that the competing drop flows are having a high level of matching
drop. That's why I think a classic circuit breaker should also treat drop and 
classic marking similar.

I understood ABE tries to modify the end system behavior slightly for ECN. If 
this can be done safely with benefits that outweighs the disadvantages, 
than that's no problem, but I think nobody is in favor to modify the AQM 
behavior to ect(0).

Based on DualQ compatibility experiments with classic ECN, I also believe 
that, if we want to further promote classic ECN too, we need some extra 
safety measures for classic AQMs using ECN to avoid high CE marking levels, 
where ect(0) flows start to push away too much the non-ect flows. For 
instance PIE has (for good reasons) a max of 10% marking, and then switches 
to drop. As far as I know such measures are not mandatory in other ECN/AQM 
related drafts, is it?

Koen.

> > > I think thinking of L4S is maybe off at a tangent. The question
> really
> > > is
> > > in this context we should use ECN to drive a CC algorithm and we
> should
> > > be
> > > > As far as I understand, this draft is related to circuit breakers
> in
> > > > end-systems, right?
> > > >
> > > >> >>> But I’m less concerned than David about eventually ignoring
> it
> > > >> for
> > > >> circuit
> > > >> >> breaker.
> > > >> >>>
> > > >> > Mumble.   I would be ok with a clear discouragement for use of
> ECN-
> > > CE
> > > >> behavior is NOT RECOMMENDED - a real AQM SHOULD be used) could
> yield
> > > a
> > > >> run of ECN-CE marks that would not cause a corresponding with a
> run
> > > of
> > > >> marks only once per RTT, and might be a reason to treat multiple
> ECN-
> > > CE
> > > >> the same as a packet drop: the marks themselves inform you that
> an
> > > AQM
> > > >> getting this from a shallow queue. Chances are that this is less
> than
> > > a
> > > >> different back-off behavior in draft-khademi-tsvwg-ecn-response-
> 00
> > > and
> > > >> >> Subject: Re: [tsvwg] [rtcweb] [AVTCORE] WG Last Call on
> changes:
> > > >> draft-ietf-
> > > >> >> I think we SHOULD NOT recommend to use ECN marks as inputs to
> a
> > > CB.
> > > >> See
> > > >> >> below:
> > > >> >>
> > > >> >> mean the services works perfectly. Therefore for me CE marks
> are a
> > > >> perfect input
> > > >> >> We may in future figure out ways to do this to detect
> significant
> > > >> failure using a
> > > >> >> rate adaptive transport and ECN e.g.  Observing 100% CE marks
> or
> > > >> something, for
> > > >> >>> But I’m less concerned than David about eventually ignoring
> it
> > > >> for
> > > >> circuit
> > > >> >> breaker.
> > > >> >>>
> > > >> >>> "If the implementation only have circuit breaker, i.e. no
> full
> > > >> fledged congestion
> > > >> >> Agree, enabling ECN without a responsive CC is going to lead
> to
> > > >> trouble.
> > > >> >>
> > > >> >>> Mirja
> > > >> >>>
> > > >> >> Gorry
> > > >> >>
> > > >> >>>>>>> latitude to ignore ECN-CE marks (e.g., because an
> implementer
> > > >> doesn't
> > > >> >>>>> I’m not sure I believe this argument, though, since it
> seems
> > > >> that
> > > >> any new
> > > >> >> AQM
> > > >> >>>>> while existing implementations set ECT(0)). These
> compatibility
> > > >> mechanisms
> > > >> >>>> That roughly matches my line of thinking, and I'll observe
> that
> > > >> the
> > > >> original
> > > >> >> DCTCP
> > > >> >>>> [B] In line with Colin's comment on the L4S draft, I think
> it's
> > > >> incumbent on
> > > >> >>>> the authors of draft-briscoe-aqm-dualq-coupled to figure out
> how
> > > >> that will
> > > >> >>>> In more detail, I think making changes to normative
> requirements
> > > >> here based
> > > >> >>>> on [B] is premature, and I would hope that the rmcat WG
> could be
> > > >> >> encouraged
> > > >> >>>> to consider the RTP circuit breaker in its congestion
> control
> > > >> drafts, as those CC
> > > >> >>>> mechanisms are related to the circuit breaker mechanism,
> hence
> > > >> likely
> > > >> >>>> is going to rest on that draft, then the reference to that
> draft
> > > >> should be
> > > >> >>>> normative.  Something about doing that strikes me as
> premature
> > > ...
> > > >> >>>>
> > > >> >>>> is an imprecise undertaking at best.   As an alternative to
> the
> > > >> current text,
> > > >> >>>> near future due to all of the work in progress cited in
> Section
> > > 7?
> > > >> >>>>
> > > >> >>>> Thanks, --David
> > > >> >>>>
> > > >> >>>>>>> latitude to ignore ECN-CE marks (e.g., because an
> implementer
> > > >> doesn't
> > > >> >>>>> I’m not sure I believe this argument, though, since it
> seems
> > > >> that
> > > >> any new
> > > >> >> AQM
> > > >> >>>>> while existing implementations set ECT(0)). These
> compatibility
> > > >> mechanisms
> > > >> >>>>>> Understand, we have at least two proposals to make ECN-CE
> more
> > > >> >> frequent
> > > >> >>>>>> more frequent. Unless both are killed off, ECN-CE will
> show up
> > > >> frequently
> > > >> >>>>>> receiver will see enough packet-drops when a circuit-
> breaker
> > > >> should
> > > >> >>>>>> activate.
> > > >> >>>>>>
> > > >> >>>>> This is a problem, both for the circuit breaker, and for
> the
> > > >> algorithms being
> > > >> >>>>>>> Could someone propose initial text to qualifies the
> current
> > > >> "MAY
> > > >> ignore"
> > > >> >>>>>>> statement?
> > > >> >>>>>>
> > > >> >>>>>> Essentially, for the second option, you might propose text
> to
> > > >> the
> > > >> >>>>>> effect of:
> > > >> >>>>>> ]
> > > >> >>>>>> ] not-ECN-capable packets to determine whether enough
> packets
> > > >> along the
> > > >> >>>>>> ] path are being dropped to justify activating our
> circuit-
> > > >> breaker.
> > > >> >>>>>>
> > > >> >>>>> trigger, you’d essentially have to run without ECN for
> some
> > > >> seconds.
> > > >> >>>>>
> > > >> >>>>> --
> > > >> >>>>> Colin Perkins
> > > >> >>>>> https://csperkins.org/
> > > >> >>>>

---

From: Michael Welzl <[email-masked]>
Date: Thu, 30 Jun 2016 18:28:15 +0200

Hi,

> On 30. jun. 2016, at 17.58, De Schepper, Koen (Nokia - BE) <[email-masked]> wrote:
> 
> 

I pretty much agree with everything here…

> to drop. As far as I know such measures are not mandatory in other ECN/AQM 

… but here I’d like to put a note of warning: see the appendix here:
http://heim.ifi.uio.no/michawe/research/publications/CAIA-TR-150710A.pdf

here, we found that this behavior is detrimental (the 10% threshold was too small).
=> I agree with the general notion of such a threshold, but the threshold shouldn’t be too low (it can eliminate ECN’s benefits).

Cheers,
Michael