Subject: [TOOLS-DEVELOPMENT] Is the mail archive off line

From: Lou Berger <[email-masked]>
Date: Wed, 25 Jan 2017 13:29:00 -0500

I don't see any messages since the 22nd in
https://mailarchive.ietf.org/arch/search/?email_list=teas&so=-date

while https://www.ietf.org/mail-archive/web/teas/current/maillist.html
is just an hour or two behind...

Thanks,

Lou

---

From: Russ Housley <[email-masked]>
Date: Wed, 25 Jan 2017 13:57:45 -0500

I do not know what ought to be on the teas list archive.

The messages on the Last Call thread for draft-ietf-lamps-eai-addresses-05 over the past few days are not in the archive.

Russ

---

From: Lou Berger <[email-masked]>
Date: Wed, 25 Jan 2017 14:07:24 -0500

> I do not know what ought to be on the teas list archive.
That's why I gave both links :-)

Lou

---

From: Glen <[email-masked]>
Date: Wed, 25 Jan 2017 11:08:40 -0800

This is just an indexing problem.  No mail was lost.  Ryan is already
working on this.

He informs me that a quick fix is needed, followed by a re-index,
which should take about 60 minutes.

The archives should be back online once that's done.

Glen

On Wed, Jan 25, 2017 at 10:57 AM, Russ Housley <[email-masked]> wrote:
> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Robert Sparks <[email-masked]>
Date: Wed, 25 Jan 2017 13:16:22 -0600

Thanks Glen (and Ryan).

After the proper indexes are back in place, please let us know if you 
have any insight into _why_ the indexing was disrupted, and what we can 
do to prevent it (or at least automatically detect it) going forward?

RjS

---

From: Lou Berger <[email-masked]>
Date: Wed, 25 Jan 2017 14:20:59 -0500

Thanks. mailarchive looks good -- now
https://www.ietf.org/mail-archive/web/teas/current/maillist.html seems
stuck...

---

From: Matt Larson <[email-masked]>
Date: Wed, 25 Jan 2017 14:28:55 -0500

Hi Lou,

Bear in mind that while the mail archive tool works in real time, the MhonArc tool that builds the html archive pages runs periodically, and so it may lag by up to 30 minutes.

Matt

> On Jan 25, 2017, at 2:20 PM, Lou Berger <[email-masked]> wrote:
> 

-------------------------------------------
Matthew Larson, Software Engineer
Association Management Solutions
Forum Management, Meeting and Event Planning
5177 Brandin Court
Fremont, CA  94538

---

From: Lou Berger <[email-masked]>
Date: Wed, 25 Jan 2017 14:49:18 -0500

understood - it is hours behind at this point...

---

From: Matt Larson <[email-masked]>
Date: Wed, 25 Jan 2017 15:31:00 -0500

Hi!

I just checked the mailarchive tool version and the MhonArc version, as as of the last MhonArc run (at 12:20 Pacific), the archives are in sync.

Part of the lag may have been from the CDN cacheing the old html page.  I've modified our settings to prevent cacheing of pages in the mail-archive/web/ tree to remove another potential source of latency.

Matt

> On Jan 25, 2017, at 2:49 PM, Lou Berger <[email-masked]> wrote:
>

---

From: Ryan Cross <[email-masked]>
Date: Wed, 25 Jan 2017 12:45:36 -0800

Here is a quick summary of the issue.

Mail Archive indexing began failing Jan 22 03:00 PST, when the index task queue started using the wrong version of Django.  This problem has been rectified.  About 1400 messages are queued to be indexed now.  This should be complete by 4pm PST.  We are looking into ways to detect indexing failures to prevent this from happening in the future.

Ryan

> On Jan 25, 2017, at 11:16 AM, Robert Sparks <[email-masked]> wrote:
> 
> Thanks Glen (and Ryan).
> 
> After the proper indexes are back in place, please let us know if you have any insight into _why_ the indexing was disrupted, and what we can do to prevent it (or at least automatically detect it) going forward?
> 
> RjS
> 
>

---

From: Lou Berger <[email-masked]>
Date: Wed, 25 Jan 2017 15:47:21 -0500

Matt,

Thanks for checking.

> I just checked the mailarchive tool version and the MhonArc version, as as of the last MhonArc run (at 12:20 Pacific), the archives are in sync.
great.

> Part of the lag may have been from the CDN cacheing the old html page.  I've modified our settings to prevent cacheing of pages in the mail-archive/web/ tree to remove another potential source of l…
That's believable.

Thanks,
Lou

---

From: "Ben Campbell" <[email-masked]>
Date: Mon, 30 Jan 2017 09:56:45 -0600

I just noticed a number of messages missing from the MMUSIC mailing list 
missing from MailArchive. I thought it was a search problem, but they 
don't show up there, either. (See ticket #2180).

Could this be related to the indexing issue?

Ben.

On 25 Jan 2017, at 14:45, Ryan Cross wrote:

> Mail Archive indexing began failing Jan 22 03:00 PST, when the index 
> task queue started using the wrong version of Django.  This problem 
> has been rectified.  About 1400 messages are queued to be indexed now. 
>  This should be complete by 4pm PST.  We are looking into ways to 
> detect indexing failures to prevent this from happening in the future.
>
> Ryan
>
>> have any insight into _why_ the indexing was disrupted, and what we 
>> can do to prevent it (or at least automatically detect it) going 
>> forward?
>>
>> RjS
>>
>>
>>>> The messages on the Last Call thread for 
>>>> draft-ietf-lamps-eai-addresses-05 over the past few days are not in 
>>>> the archive.
>>>>
>>>> Russ
>>>>
>>>>
>>>>> while 
>>>>> https://www.ietf.org/mail-archive/web/teas/current/maillist.html

---

From: "Ben Campbell" <[email-masked]>
Date: Mon, 30 Jan 2017 09:59:28 -0600

Well, I managed to find the alledgedly missing messages with a different 
search string. So it appears to be a search issue, not a missing message 
issue. Sorry for the confusion.

Thanks!

Ben.

On 30 Jan 2017, at 9:56, Ben Campbell wrote:

> I just noticed a number of messages missing from the MMUSIC mailing 
> list missing from MailArchive. I thought it was a search problem, but 
> they don't show up there, either. (See ticket #2180).
>
>> has been rectified.  About 1400 messages are queued to be indexed 
>> now.  This should be complete by 4pm PST.  We are looking into ways 
>> to detect indexing failures to prevent this from happening in the 
>> future.
>>
>> Ryan
>>
>>> After the proper indexes are back in place, please let us know if 
>>> you have any insight into _why_ the indexing was disrupted, and what 
>>> we can do to prevent it (or at least automatically detect it) going 
>>> forward?
>>>
>>> RjS
>>>
>>>
>>>>> draft-ietf-lamps-eai-addresses-05 over the past few days are not 
>>>>> in the archive.
>>>>>
>>>>> Russ
>>>>>
>>>>>

---

From: "Ben Campbell" <[email-masked]>
Date: Mon, 30 Jan 2017 10:04:06 -0600

Also, the messages were from the avt list, not mmusic. Sorry for the 
further confusion.

On 30 Jan 2017, at 9:59, Ben Campbell wrote:

> Well, I managed to find the alledgedly missing messages with a 
> different search string. So it appears to be a search issue, not a 
> missing message issue. Sorry for the confusion.
>
> Thanks!
>
> Ben.
>
>>>> you have any insight into _why_ the indexing was disrupted, and 
>>>> what we can do to prevent it (or at least automatically detect it) 
>>>> going forward?
>>>>
>>>> RjS
>>>>
>>>>