mailman crashing in production

This incident is in the very last logs available from forster, so I'm copying the log output here, replacing the names of the email addresses with "XXX".

May 25 20:10:27 2011 (14860) Membership updates for ubuntu-tour: ['<email address hidden> -> <email address hidden>']
May 25 22:15:34 2011 (14860) Cannot talk to Launchpad: [Errno 4] Interrupted system call
May 25 22:15:34 2011 (14860) batch: ['rosalila-studio']
May 25 22:15:35 2011 (14860) Cannot talk to Launchpad: [Errno 4] Interrupted system call
May 25 22:15:35 2011 (14860) batch: ['ubuntu-ba']
May 25 22:15:47 2011 (14860) Cannot talk to Launchpad: [Errno 4] Interrupted system call
May 25 22:15:47 2011 (14860) batch: ['ubuntu-server-ec2-testing-notifications']
May 25 22:15:53 2011 (14860) Membership updates for openstreetmap: ['<email address hidden> -> <email address hidden>']
May 25 22:17:02 2011 (17848) XMLRPC runner starting

I looked through a number of launchpad log files to try and make a connection to the "Cannot talk to Launchpad" error. I didn't see any. FTR, here's where I looked.

gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad135.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad134.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad133.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad132.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad131.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad130.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad129.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad128.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad127.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad126.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad125.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad124.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad123.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/soybean/launchpad122.log-20110521.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/chaenomeles/launchpad55.log-20110525.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/chaenomeles/launchpad54.log-20110525.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/chaenomeles/launchpad53.log-20110525.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/chaenomeles/launchpad52.log-20110525.bz2 | less
gary@carob:~$ bunzip2 -c /srv/launchpad.net-logs/production/chaenomeles/launchpad51.log-20110525.bz2 | less

I could have gone further, and maybe looking at more would show some kind of hang. However, it seems more likely that this is some kind of network hang. Access logs seem to show that we are serving files just fine at the time of the top log entry in comment 1.

In any case, it's likely that we simply need to make mailman have a timeout in its LP connections. The "Cannot talk to Launchpad" string appears repeatedly in lib/lp/services/mailman/monkeypatches/xmlrpcrunner.py . I think a simple fix for this is to add a timeout to the xmlrpclib ServerProxy instance, using the "transport" argument. That's what I plan to do.

Fixed in stable r13541 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13541>.

The code for this has been deployed to production appserver machines, but not to the Mailman machines. I'll make an RT for this.