Subject: [TOOLS-DEVELOPMENT] Slowdown recurring - Potential datatracker issue causing web server failure

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:08:27 -0700

All -

Good morning.

As many of you are aware, we had two more slowdowns following the one I
reported.  This is bad.

In the first case, which occurred at about 0440 Pacific time, the server 
rebooted itself (correctly) before I could get to it and came back online.
This is good, because at least we know the server reboot system appears to
work.

In the second case, which occurred at 0645, I caught it in the act; and,
although the server was heavily impacted, I was able to get control of it
before a reboot.  This is also good, because I was able to catch things in
the act.

As was the case yesterday, the impacted process was Apache.  This time, I
was able to also confirm that a python process being kicked off by apache
was failing.  This process caused the apache footprint to go from a normal
resident memory size of 55M to a size of 12GB.  Swap was almost completely
exhausted, and the server was crawling.

11744 wwwrun    20   0 19.2g  12g 5140 D    6 78.2  22:11.82 httpd2-prefork     
13641 wwwrun    20   0  578m  57m 6700 D    8  0.4   4:51.13 httpd2-prefork     
16574 wwwrun    20   0  569m  55m 6436 S    0  0.3   1:52.44 httpd2-prefork     
17010 wwwrun    20   0  574m  59m 6584 S    0  0.4   1:50.91 httpd2-prefork     

It has been mentioned by Steven that he was trying to enter a position at
the time, and the server choked right when he tried to do so.  Regardless
of whether his action triggered this, the hints I'm seeing make me believe
at this point we have this down to something within the datatracker, and we
need to work further to find out what is happening.

I am copying the Tools Team, and Henrik, and will be asking them to help with
log analysis to see if they can determine why the webserver is crawling.  As
you can guess from the above output, I have a snapshot program running which
takes server information every five minutes and writes it to a file.  Now
that I know this is a webserver/python (and therefore datatracker) issue,
I've added additional data to the file capture.  Still, I hope that we will
be able to resolve this quickly.

Note that since nothing has changed recently in terms of Python or the
datatracker (or the server), it may take some time to localize this.  It
could be some heretofore undetected recursion issue or something similar.
This is why I will be asking the tools team to step in and assist.

Thanks,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:23:23 -0700

Rolf -

THANK YOU!  I am copying the tools team on this, and am preparing additional
data for them now.

Glen

On Mon, Jun 27, 2011 at 02:15:16PM +0000, Rolf Winter wrote:
> Hi Glen,
> 
> another data point for you. When you go to http://tools.ietf.org/html/draft-ietf-ledbat-survey-07 and click on RFC-to-be you also get a python error. I attached the error page for your reference. Hope it helps tracking the bug down.
> 
> Best,
> 
> Rolf
> 
> 
> NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014 
> 
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] On
> > Behalf Of Glen
> > Sent: Montag, 27. Juni 2011 16:08
> > To: [email-masked]; [email-masked]; [email-masked]; [email-masked]
> > Cc: [email-masked]; Romascanu, Dan (Dan); Pete Resnick;
> > [email-masked]
> > Subject: Slowdown recurring - Potential datatracker issue causing web
> > server failure
> > 
> > All -
> > 
> > Good morning.
> > 

Content-Description: rfcmarkup.htm
> <body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
> <body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
> </font> </font> </font> </script> </object> </blockquote> </pre>
> </table> </table> </table> </table> </table> </font> </font> </font><body bgcolor="#f0f0f8">
> <table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
> <tr bgcolor="#6622aa">
> <td valign=bottom>&nbsp;<br>
> <font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>&lt;type 'exceptions.TypeError'&gt;</strong></big></big></font></td
> ><td align=right valign=bottom
> ><font color="#ffffff" face="helvetica, arial">Python 2.6.6: /usr/local/bin/python<br>Mon Jun 27 23:12:56 2011</font></td></tr></table>
>     
> <p>A problem occurred in a Python script.  Here is the sequence of
> function calls leading up to the error, in the order they occurred.</p>
> <table width="100%" cellspacing=0 cellpadding=0 border=0>
> <tr><td bgcolor="#d8bbff"><big>&nbsp;</big><a href="file:///www/tools.ietf.org/tools/rfcmarkup/rfcmarkup">/www/tools.ietf.org/tools/rfcmarkup/rfcmarkup</a> in <strong><module></strong>()</td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;1359</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;"--help"&nbsp;in&nbsp;sys.argv:<br>
> </tt></font></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;1360</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;usage()<br>
> </tt></font></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;1361</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sys.exit()<br>
> </tt></font></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;1362</small>&nbsp;<br>
> </tt></font></td></tr>
> <tr><td bgcolor="#ffccee"><tt><small>&nbsp;1363</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;markup()<br>
> </tt></td></tr>
> <tr><td><small><font color="#909090"><strong>markup</strong>&nbsp;= &lt;function markup&gt;</font></small></td></tr></table>
> <table width="100%" cellspacing=0 cellpadding=0 border=0>
> <tr><td bgcolor="#d8bbff"><big>&nbsp;</big><a href="file:///www/tools.ietf.org/tools/rfcmarkup/rfcmarkup">/www/tools.ietf.org/tools/rfcmarkup/rfcmarkup</a> in <strong>markup</strong>()</td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;&nbsp;899</small>&nbsp;<br>
> </tt></font></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;&nbsp;900</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;elif&nbsp;rfc:<br>
> </tt></font></td></tr>
> <tr><td bgcolor="#ffccee"><tt><small>&nbsp;&nbsp;901</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;attribs.update(stateinfo("rfc%s"&nbsp;%&nbsp;(rfc)))<br>
> </tt></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;&nbsp;902</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;args["favicon"]&nbsp;=&nbsp;"/images/rfc.png"<br>
> </tt></font></td></tr>
> <tr><td><font color="#909090"><tt><small>&nbsp;&nbsp;903</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;filetext("/home/ietf/rfc/meta/rfc%s.errata"&nbsp;%&nbsp;rfc):<br>
> </tt></font></td></tr>
> <tr><td><small><font color="#909090"><strong>attribs</strong>&nbsp;= {'doc': 'rfc6297', 'docreplaces': 'draft-ietf-ledbat-survey-07', 'document': 'draft-ietf-ledbat-survey-07', 'extrastyle': 'body {background-color:#fee;}', 'rfc-repository': 'http://www.rfc-editor.org/authors', 'title': 'RFC-EDITOR AUTH48 REVIEW COPY', 'topmenu': 'true'}, attribs.<strong>update</strong>&nbsp;= &lt;built-in method update of dict object&gt;, <strong>stateinfo</strong>&nbsp;= &lt;function stateinfo&gt;, <strong>rfc</strong>&nbsp;= '6297'</font></small></td></tr></table><p><strong>&lt;type 'exceptions.TypeError'&gt;</strong>: 'NoneType' object is not iterable
> <br><tt><small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small>&nbsp;</tt>args&nbsp;=
> ("'NoneType' object is not iterable",)
> <br><tt><small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small>&nbsp;</tt>message&nbsp;=
> "'NoneType' object is not iterable"
> 
> 
> <!-- The above is a description of an error in a Python program, formatted
>      for a Web browser because the 'cgitb' module was enabled.  In case you
>      are not reading this in a Web browser, here is the original traceback:
> 
> Traceback (most recent call last):
>   File "/www/tools.ietf.org/tools/rfcmarkup/rfcmarkup", line 1363, in &lt;module&gt;
>     markup()
>   File "/www/tools.ietf.org/tools/rfcmarkup/rfcmarkup", line 901, in markup
>     attribs.update(stateinfo("rfc%s" % (rfc)))
> TypeError: 'NoneType' object is not iterable
> 
> -->
>

---

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:23:52 -0700

On Mon, Jun 27, 2011 at 03:17:31PM +0100, Stephen Farrell wrote:
> Just one possibility - I just switched to firefox 5.0.
> Hard to see it being that, but I guess if the tracker
> is doing something with the UA string it could be. If
> you want, I can revert to 4.whatever-it-was and see if
> that works ok. Meanwhile, I'll stay in readonly mode
> for a bit I guess in case it is me causing it;-)

I did too.  No point in reverting unless we could get every user in the world
to revert... which we can't.  :-)

Glen

---

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:44:58 -0700

Excellent, Steve, and good point.  It now does appear that updating the
ballot positions may be the problem... I just sent an update to the I**** .
We need to wait for tools team analysis but I see you cc'ed them so that
will certainly be appreciated by them!

Glen

On Mon, Jun 27, 2011 at 03:41:06PM +0100, Stephen Farrell wrote:
> 
> 
> On 27/06/11 15:23, Glen wrote:
> Indeed. But it could just be that the save-and-send
> form processor is reacting there somehow. As I said, I'll
> stay readonly for a bit but can press buttons for folks
> if asked
> 
> S.
>