Subject: [TOOLS-DEVELOPMENT] Leadership Update Re: IETF Website Degradation

From: Glen <[email-masked]>
Date: Mon, 3 Aug 2015 16:51:55 -0700

I'm now narrowing the distribution of this discussion a little bit, to the
IETF leadership, as shown in the headers.

Here is where we stand at the moment.

First, as to the all-important question about "reaching a human."  It turns
out that Cloudflare does not provide human phone support.  We are on the
"Business Plan" at $200/month, which, if you look here,
https://www.cloudflare.com/plans , conspicuously does *not* include 24/7
phone support.

And before we run off and determine that we should immediately "upgrade to
the Enterprise plan", I'd like to call out a few other things.

The plan we *are* on includes what they call "Advanced DDoS mitigation".
As outlined here, https://www.cloudflare.com/ddos , Cloudflare is
*supposed* to *protect* us from DDoS attacks of the type we encountered
today.  But if we read the fine print, that turns out to not always be the
case...

--snip--

Layer 7 attacks

A new breed of attacks target Layer 7 of the OSI model, the "application"
layer. These attacks focus on specific characteristics of web applications
that create bottlenecks. For example, the so-called Slow Read attack sends
packets slowly across multiple connections. Because Apache opens a new
thread for each connection, and since connections are maintained as long as
there is traffic being sent, an attacker can overwhelm a web server by
exhausting its thread pool relatively quickly.

CloudFlare has protections in place against many of these attacks, and in
real world experiences we generally reduce HTTP attack traffic by 90%. For
most attacks, and for most of our customers, this is enough to keep them
online. However, the 10% of traffic that does get through traditional
protections can still be overwhelming to customers with limited resources
or in the face of very large attacks. In this case, CloudFlare offers a
security setting called "I'm Under Attack" mode (IUAM).

IUAM is a security level you can set for your site when you're under
attack. When IUAM is turned on, CloudFlare will add an additional layer of
protections to stop malicious HTTP traffic from being passed to your
server. While a number of additional checks are performed in the
background, an interstitial page is presented to your site's visitors for 5
seconds while the checks are completed. Think of it as a challenge where
the tests are automatic and visitors never need to fill in a CAPTCHA.

--snip--

This is the step we had to take today to fix this problem.  And that did
fix the problem.  At a cost, of course, of a javascript-and-cookie-based
interstitial page - hardly a permanent fix.

In other words, Cloudflare did *not* automatically stop this attack.  We
still had to suffer it, detect it, and take client action in order to
mitigate this attack.

And now we have to do a bit more:  Although we were never told about this,
and the sales pages don't mention it, we find via email support that there
is an Apache module we need to install to find out where the attack is
actually coming from, and thereby mitigate it.    So, for your info, our
next steps are:

1. Install the apache2 mod_cloudflare module (we're doing that this evening)
2. Turn off the "IUAM" mode and let the attack continue (assuming it does)
3. Use the logging provided by this module to determine the source of the
attack.
4. Insert that information back into the Cloudflare control panel to block
the IP address(es) from which the attack is originating.

We will be taking these steps as soon as possible.

But I spell this out for you all because I want to make clear that, while
for $200 I might not expect 24/7 phone support, for $200 I certainly expect
some kind of web interface that lets me directly determine where an attack
is coming from (we don't get that with Cloudflare, we have to let the
attack bleed through and figure it out with this add-on Apache module).
And I would certainly prefer (and the site certainly implies this) that
Cloudflare detect *and prevent* this kind of attack before it ever reaches
us, as every other section of their very expansive web page implies they do
automatically.

So I'm not particularly in the "let's jump to Enterprise service" because I
don't see any evidence that it will get us anything more than we already
have that is substantive.

But I do think that they have room for improvement, and so perhaps there is
another way to go:  Perhaps someone in an IETF leadership role might want
to reach out to Cloudflare with these concerns, and see if Cloudflare has
an answer, or perhaps a plan?  Perhaps the IETF could assist them with
these concerns somehow?  It might be to everyone's benefit.

And, on that same note, I've also observed that many companies are willing
to donate services and products to the IETF.  Maybe Cloudflare would like
to be generous as well?  Could be worth asking that question.

At any rate, to conclude, we are following the steps given by Cloudflare,
and we will work to get this resolved as soon as possible given the level
of support (email) and list of steps (webserver modification) we've been
given... but, to my surprise, the token is AMS' at this point, and not
Cloudflare's... which is not what I think any of us would have suspected
given what we thought Cloudflare was offering.

Please let me know if you have any questions, and, again, I apologize for
the disruption, and the learning curve, as we discover more about
Cloudflare's services and options.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Kathleen Moriarty <[email-masked]>
Date: Mon, 3 Aug 2015 20:19:01 -0400

Hi Glen,

Thank you for sharing the details of this, I do have one thought...

On Mon, Aug 3, 2015 at 7:51 PM, Glen <[email-masked]> wrote:

> First, as to the all-important question about "reaching a human."  It
> turns out that Cloudflare does not provide human phone support.  We are on
> the "Business Plan" at $200/month, which, if you look here,
> So I'm not particularly in the "let's jump to Enterprise service" because
> I don't see any evidence that it will get us anything more than we already
> But I do think that they have room for improvement, and so perhaps there
> is another way to go:  Perhaps someone in an IETF leadership role might
> want to reach out to Cloudflare with these concerns, and see if Cloudflare
> has an answer, or perhaps a plan?  Perhaps the IETF could assist them with

We just approved the DDoS Open Threat Signaling (DOTS) working group.  The
WG participants include a bunch of DDoS experts from providers and
vendors.  It might be helpful to get their advice on this attack type.
Additionally, it may also be useful for them to have the real world use
case for their use case document.  CloudFlare does participate in the WG,
so timing might be important to first give them an opportunity to respond
during normal business hours or to see if this attack type could use more
eyes from other experts.  The write up might have to be altered a bit to be
used in a draft as a use case.

Thanks,
Kathleen

-- 

Best regards,
Kathleen

---

From: Glen <[email-masked]>
Date: Mon, 3 Aug 2015 17:50:08 -0700

On Mon, Aug 3, 2015 at 5:19 PM, Kathleen Moriarty <
[email-masked]> wrote:
> We just approved the DDoS Open Threat Signaling (DOTS) working group.
The WG participants include a bunch of DDoS experts from providers and
> Kathleen

Hi Kathleen!

Absolutely agreed.

This particular attack seems to have stopped, so I don't believe we'll have
any more follow-up from Cloudflare this time.

Feel free to use or modify all, some or none of my email if you want to
approach Cloudflare or the WG with this.

If you want me to take any specific action with respect to the WG, or
Cloudflare's interaction thereon, please let me know at any time!

Thanks,
Glen

---

From: Glen <[email-masked]>
Date: Tue, 4 Aug 2015 08:00:14 -0700

Hi Tobias -

It's certainly fine with AMS if you share this information with a larger
audience!  In essence this information "belongs" to the IETF, so we would
not take exception to that!

Best regards,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

On Mon, Aug 3, 2015 at 11:04 PM, Tobias Gondrom <[email-masked]>
wrote:

> Hi Glen and Kathleen,
>
> a good idea.
> This might indeed be a good source of expertise.
>
> As I currently consider the received information "internal", I will wait
> for an "ok" from amsl and IAD before sharing this information with the
> community in DOTS.
>
> A more detailed write-up, e.g. incl. traffic amounts and any specifics
> might also be helpful to minimise question-roundtrips.
>
> Best regards,
>
> Tobias
> (IAOC chair)
> (and coincidentally DOTS co-chair)
>
>
> On 04/08/15 08:19, Kathleen Moriarty wrote:
>
> Hi Glen,
>
>> I'm now narrowing the distribution of this discussion a little bit, to
>> the IETF leadership, as shown in the headers.
>>
>> And before we run off and determine that we should immediately "upgrade
>> to the Enterprise plan", I'd like to call out a few other things.
>>
>> And now we have to do a bit more:  Although we were never told about
>> this, and the sales pages don't mention it, we find via email support that
>> there is an Apache module we need to install to find out where the attack
>> is actually coming from, and thereby mitigate it.    So, for your info, our
>> next steps are:
>>
>> 4. Insert that information back into the Cloudflare control panel to
>> block the IP address(es) from which the attack is originating.
>>
>> And, on that same note, I've also observed that many companies are
>> willing to donate services and products to the IETF.  Maybe Cloudflare
>> would like to be generous as well?  Could be worth asking that question.
>>