Subject: [Tools-implementation] Organization of the mail archives

From: Robert Sparks <[email-masked]>
Date: Tue, 1 Sep 2020 12:30:56 -0500

After consulting with Ryan, I propose this disposition for the following 
email archive related directories:

>>                      Legacy    Mailarchive    /a/ietf    A few very old mail archive scripts
This can simply be removed. The scripts are no longer used.
>>                      Email    Mailarchive    /a/isode    ISODE IMAP server data store
>>                      Email    Mailarchive    /a/mailarch    Mail Archive operations

These should stay where they are.

/a/mailarch/data/archive is the _authoritative_ store.

/a/mailarch/data/mbox_archive/public is an automatically generated set 
of mbox files generated from the authoritative store. That should be 
used to replace several things below.

>>                      Email    Mailman    /a/mailman    Mailman operations
That has to remain (though we should consider what to do with it when we 
bring in Mailman3
>>                      Email    Mailman    /a/mailman/archives Mailman Legacy Archives
This has to remain - it is where mailman's internal archiving takes 
place - any lists using pipermail are kept here, so Legacy isn't really 
accurate above
>>
>>                      Email    Postfix    /a/postfix    Postfix operations
This remains
>>                      FTP    FTP/Website combined /a/www/ietf-ftp/concluded-wg-ietf-mail-archive        ftp, rsync    none    NOT exposed by http
>>
These have all been imported, and are served through the mailarchive. 
These should be stashed. I personally don't think we should serve them 
in any way since they are available through the mailarchive. If they are 
served, they should clearly be in something I'd like to start calling 
the "museum".
>
>>                      FTP    FTP/Website combined /a/www/ietf-ftp/ietf-mail-archive    Contains text versions of mail archive    ftp, rsync    night-runner    All web access is to the /a/www/ietf-mail-archive tree, which this is a copy of
This should go away as soon as ftp stops depending on it (and we had 
another conversation about removing it already).
>>                      Email    Mailarchive /a/www/ietf-mail-archive    Legacy mail archive
>>                      Email    Mailarchive /a/www/ietf-mail-archive/text
>>                      Email    Mailarchive /a/www/ietf-mail-archive/text-secure
>>                      Email    Mailarchive /a/www/ietf-mail-archive/web
>>                      Email    Mailarchive /a/www/ietf-mail-archive/web-old
>>                      Email    Mailarchive /a/www/ietf-mail-archive/web-secure
>>
>> Again, has all of that been imported, and can all of this simply be removed (or stashed offline)?
All of this should be stashed. Everything here, with the possible 
exception of things in web-old have been imported into mailarchive. Ryan 
is looking at making sure those get imported. Once that's done, all of 
these should either be taken offline or put in the museum.

Are there any other puddles of email-archive related things that aren't 
in those directories?

---

From: Glen <[email-masked]>
Date: Tue, 1 Sep 2020 10:44:22 -0700

On Tue, Sep 1, 2020 at 10:31 AM Robert Sparks <[email-masked]> wrote:
> That has to remain

Right.  Any package which can be deployed in a more-or-less isolated
tree should be in/remain in /a/(packagename)

> (though we should consider what to do with it when we bring in Mailman3
At the moment, we are investigating whether the two versions of
Mailman can be run in parallel in our environment.  We do not yet
know, but initial indications are encouraging.

Mailman 3 will be deployed into /a/mailman3.

Glen

---

From: Glen <[email-masked]>
Date: Tue, 1 Sep 2020 10:51:28 -0700

Separate subthread:

> place - any lists using pipermail are kept here, so Legacy isn't really

Point of clarification:
*All lists use pipermail in that Ryan has said that Mailman2's
internal archiver cannot be shut off or replaced.
But
*No lists use pipermail in that /pipermail is not served from anywhere
in the Apache config, and all references to /pipermail are redirected
to the Mailarchive.

So we cannot move or remove, but can safely ignore, everything inside
/a/mailman it seems.

> >>                      FTP    FTP/Website combined /a/www/ietf-ftp/concluded-wg-ietf-mail-archive        ftp, rsync    none    NOT exposed by http

Forgive me, I might be misreading "NOT exposed by http," but:
https://www.ietf.org/ietf-ftp/concluded-wg-ietf-mail-archive/

> in any way since they are available through the mailarchive.

+1 I agree they should be removed from FTP, RSYNC, and HTTP, and not served.

> these should either be taken offline or put in the museum.

+1.

I assume I should do the "taking offline".  How will I know when to
proceed with that?  Robert, I assume you will oversee that?

Glen

---

From: Robert Sparks <[email-masked]>
Date: Tue, 1 Sep 2020 14:03:57 -0500

> So we cannot move or remove, but can safely ignore, everything inside

Thank you!

But I'm still a bit concerned about one thing. I had it in my head that 
there were still some active, or only recently (~5yrs) dormant lists 
that only existed outside what mailarchive was ingesting. If that's not 
true, I'll be relieved. Ryan (or anyone else) - do you remember any such 
thing and can you point to it?

> https://www.ietf.org/ietf-ftp/concluded-wg-ietf-mail-archive/
Old notes - clearly no longer correct. At the time, they said 
"http://www.ietf.org/ietf-ftp goes to ietf-ftp/ietf".
> proceed with that?  Robert, I assume you will oversee that?
Yes. Right now we are still trying to understand, and that understanding 
will build a plan. I talk about what I think _should_ happen above as a 
proposal to that plan, not as directives to take action now.

---

From: Glen <[email-masked]>
Date: Tue, 1 Sep 2020 12:29:05 -0700

> > So we cannot move or remove, but can safely ignore, everything inside

Oh!  Well... you're welcome!

> thing and can you point to it?

Well, there is:

tail -35 /a/postfix/virtual
tail -36 /a/postfix/aliases

which are external feeder points for the mailarchive tool.  All of
them are pointing to new new archive system (only), but I have no idea
which are still active/in use or not.

I'm not aware of anything pointing to MHonarc or any legacy archiving systems.

> proposal to that plan, not as directives to take action now.

Okay, thank you.  I have that September 4 removal directive I'll
proceed with, but everything else I'll just take as conversation
unless you tell me otherwise.

Glen

---

From: Robert Sparks <[email-masked]>
Date: Tue, 15 Sep 2020 11:58:41 -0500

Continuing this in the context of today's conversation.

After the cleanup described below, we are/will-be left with:

/a/mailarch/data/archive as the _authoritative_ store.

/a/mailarch/data/mbox_archive/public as an automatically generated set 
of mbox files generated from the authoritative store.

Should we plan to move the first to be under /doc and the second to be 
under /derived?

RjS

-------- Forwarded Message --------
Subject: 	Organization of the mail archives
Date: 	Tue, 1 Sep 2020 12:30:56 -0500
From: 	Robert Sparks <[email-masked]>
To: 	Ryan Cross <[email-masked]>, [email-masked] 
<[email-masked]>

This remains
>> FTP FTP/Website combined 
>> /a/www/ietf-ftp/concluded-wg-ietf-mail-archive ftp, rsync none NOT 
>> exposed by http
>>
>
>> FTP FTP/Website combined /a/www/ietf-ftp/ietf-mail-archive Contains 
>> text versions of mail archive ftp, rsync night-runner All web access 
>> is to the /a/www/ietf-mail-archive tree, which this is a copy of

---

From: Glen <[email-masked]>
Date: Tue, 15 Sep 2020 10:59:09 -0700

> Should we plan to move the first to be under /doc and the second to be 

No objection.

I have a weak preference for leaving data which is only used by a single 
package in place under /a/packagename, because it will help us identify 
what things can be easily severed.  However, I realize that the mailarch 
data is also accessed by the ISODE IMAP server, so the proposal to move 
them under /data and /derived seems correct.

Glen

---

From: Ryan Cross <[email-masked]>
Date: Tue, 15 Sep 2020 11:17:05 -0700

Inline

> On Sep 15, 2020, at 9:58 AM, Robert Sparks <[email-masked]> wrote:
> 
> /a/mailarch/data/mbox_archive/public as an automatically generated set of mbox files generated from the authoritative store. 
> 
> Should we plan to move the first to be under /doc and the second to be under /derived?
> 
> 
Do you mean

/a/mailarch/data/doc/[list name]

/a/mailarch/data/derived/[list name]

Or something else?
> From:	Robert Sparks <[email-masked]> <mailto:[email-masked]>
> To:	Ryan Cross <[email-masked]> <mailto:[email-masked]>, [email-masked] <mailto:[email-masked]> <[email-masked]> <mailto:[email-masked]>
> 
> After consulting with Ryan, I propose this disposition for the following email archive related directories:
> 
> 
> /a/mailarch/data/mbox_archive/public is an automatically generated set of mbox files generated from the authoritative store. That should be used to replace several things below.
> 
> That has to remain (though we should consider what to do with it when we bring in Mailman3
> This has to remain - it is where mailman's internal archiving takes place - any lists using pipermail are kept here, so Legacy isn't really accurate above
>>> 
> These have all been imported, and are served through the mailarchive. These should be stashed. I personally don't think we should serve them in any way since they are available through the mailarchive. If they are served, they should clearly be in something I'd like to start calling the "museum".
>> 
> This should go away as soon as ftp stops depending on it (and we had another conversation about removing it already).
> All of this should be stashed. Everything here, with the possible exception of things in web-old have been imported into mailarchive. Ryan is looking at making sure those get imported. Once that's done, all of these should either be taken offline or put in the museum.
> 
> Are there any other puddles of email-archive related things that aren't in those directories?
>

---

From: Robert Sparks <[email-masked]>
Date: Tue, 15 Sep 2020 13:39:39 -0500

On 9/15/20 1:17 PM, Ryan Cross wrote:
> Inline
>
>> On Sep 15, 2020, at 9:58 AM, Robert Sparks <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> /a/mailarch/data/mbox_archive/public as an automatically generated 
>> set of mbox files generated from the authoritative store.
>>
No - we've been discussing putting original data in a common place, so 
it would end up being something more like /doc/mailarch/[list name] and 
/derived/mailarch/[listname].
>> After consulting with Ryan, I propose this disposition for the 
>> following email archive related directories:
>>
>>
>> /a/mailarch/data/mbox_archive/public is an automatically generated 
>> set of mbox files generated from the authoritative store. That should 
>> be used to replace several things below.
>>
>> place - any lists using pipermail are kept here, so Legacy isn't 
>> really accurate above
>>>>
>> These should be stashed. I personally don't think we should serve 
>> them in any way since they are available through the mailarchive. If 
>> they are served, they should clearly be in something I'd like to 
>> start calling the "museum".
>>>
>>>> text versions of mail archive ftp, rsync night-runner All web 
>>>> access is to the /a/www/ietf-mail-archive tree, which this is a copy of
>> exception of things in web-old have been imported into mailarchive. 
>> Ryan is looking at making sure those get imported. Once that's done, 
>> all of these should either be taken offline or put in the museum.
>>

---

From: Glen <[email-masked]>
Date: Tue, 15 Sep 2020 11:46:57 -0700

> /derived/mailarch/[listname].

For clarity, Ryan, the full path will be:

/a/doc/mailarch/[listname]
/a/derived/mailarch/[listname]

(with the /a - these will *not* go on the root filesystem.  :-)  )

Glen