Subject: [Tools-implementation] Serving meeting materials from cloudflare

From: Robert Sparks <[email-masked]>
Date: Mon, 22 Jun 2020 10:45:01 -0500

We had issues with heavy load keeping some attendees from retrieving 
slides at the beginning of the plenary at IETF107. As we've discussed 
before, we should be serving all the meeting materials from cloudflare 
to help mitigate that problem.

We have a few things to discuss concerning where the files are on disk 
and where and how they are served.

Slides, for instance are already in a couple of places that are served 
by cloudflare, the best is probably served from 
https://www.ietf.org/proceedings (backed by /a/www/www6s/proceedings).

Links to those look like:

https://www.ietf.org/proceedings/107/slides/slides-107-dispatch-email-security-and-client-id-00.pdf

(Note that on the datatracker we have views that handle not having the 
version # or filetype and take the user to the right place. That 
functionality doesn't exist on www right now, so someone hacking the 
URLs might have a harder time).

Agendas look like:

https://www.ietf.org/proceedings/107/agenda/agenda-107-dispatch-04.txt

(but I note that the experience at 
https://www.ietf.org/proceedings/107/agenda/agenda-107-gendispatch-03.md 
is less pleasant than what happens at the datatracker, where we generate 
some html on the fly).

I think continuing to serve meeting materials from 
https://www.ietf.org/proceedings is probably a reasonable long-term 
plan, as opposed to taking some new chunk of the namespace or moving 
these things into the datatracker's media directory.

Thoughts? Do you see other rough edges?

Do we perhaps want to provide links to _both_ the datatracker served 
pages and the www served pages? (Except for the discrepancies mentioned 
above, I lean towards no).

How do you think we should best address the functional discrepancies? We 
could build redirect rules into apache (will nginx support these?) to 
deal with the filename completion, and we could always automatically 
generate html files when handed md files and ensure they also appear in 
the proceedings directory.

There is a bigger task ahead of us (still) to reconcile the different 
places we're storing files, and the different names we're serving them 
from, but I think we get to avoid the hard part of that while working on 
this particular problem.

RjS

---

From: Glen <[email-masked]>
Date: Mon, 22 Jun 2020 09:06:42 -0700

> to help mitigate that problem.

+1

> https://www.ietf.org/proceedings (backed by /a/www/www6s/proceedings).

The "where on disk" relates to the long-pending issue of disk cleanup,
reorganization, and removal/consolidation of old "www6" (and older) files.

Trimming for clarity...

> Thoughts?

Agree.

> Do you see other rough edges?

> above, I lean towards no).

I agree NO, but I worry about synchronization - how to ensure whatever we
serve is authoritative and updated.

> could build redirect rules into apache (will nginx support these?) to

It had better!  :-)

> the proceedings directory.

I'm mildly opposed to doing extra CPU work.  ISTM that we should have one
onboard source for the data, and we should position it in a way that CF
sees it *and* caches it - I would hope that there would be an easy, direct
way of doing that, but if the pages are dynamically generated by the DT,
then perhaps it's best to just alias them into proceedings so they serve
out from there, and be content?

> from, but I think we get to avoid the hard part of that while working on

Yes, but it's still on the long-term radar.  Controlling the sprawl is
important from an operations POV.

Glen

---

From: Russ Housley <[email-masked]>
Date: Mon, 22 Jun 2020 14:28:46 -0400

I agree with the conclusion.  Serving out of https://www.ietf.org/proceedings to get CDN assistance is a real win.

Russ

> On Jun 22, 2020, at 11:45 AM, Robert Sparks <[email-masked]> wrote:
> 
> We had issues with heavy load keeping some attendees from retrieving slides at the beginning of the plenary at IETF107. As we've discussed before, we should be serving all the meeting materials from cloudflare to help mitigate that problem.
> 
> We have a few things to discuss concerning where the files are on disk and where and how they are served.
> 
> Slides, for instance are already in a couple of places that are served by cloudflare, the best is probably served from https://www.ietf.org/proceedings (backed by /a/www/www6s/proceedings).
> 
> (Note that on the datatracker we have views that handle not having the version # or filetype and take the user to the right place. That functionality doesn't exist on www right now, so someone hacking the URLs might have a harder time).
> 
> Agendas look like:
> 
> (but I note that the experience at https://www.ietf.org/proceedings/107/agenda/agenda-107-gendispatch-03.md is less pleasant than what happens at the datatracker, where we generate some html on the fly).
> 
> I think continuing to serve meeting materials from https://www.ietf.org/proceedings is probably a reasonable long-term plan, as opposed to taking some new chunk of the namespace or moving these things into the datatracker's media directory.
> 
> Do we perhaps want to provide links to _both_ the datatracker served pages and the www served pages? (Except for the discrepancies mentioned above, I lean towards no).
> 
> How do you think we should best address the functional discrepancies? We could build redirect rules into apache (will nginx support these?) to deal with the filename completion, and we could always automatically generate html files when handed md files and ensure they also appear in the proceedings directory.
> 
> There is a bigger task ahead of us (still) to reconcile the different places we're storing files, and the different names we're serving them from, but I think we get to avoid the hard part of that while working on this particular problem.
> 
> RjS
> 
> 
> 
> 
> 
> 
> -- 
> Tools-implementation mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-implementation