Person:+mailinglist-moderate timeout

Bug #736012 reported by Robert Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Unassigned

Bug Description

OOPS-1900C665

Branch: launchpad-rev-12589
Revno: 12589
SQL time: 6068 ms
Non-sql time: 6156 ms
Total time: 12224 ms
Statement Count: 174

https://launchpad.net/%7Eayatana/+mailinglist-moderate

3 23 1398 60 1338 SQL-launchpad-main-master
SELECT Person.account,
       Person.creation_comment,
       Person.creation_rationale,
       Person.datecreated,
       Person.defaultmembershipperiod,
       Person.defaultrenewalperiod,
       Person.displayname,
       Person.hide_email_addresses,
       Person.homepage_content,
       Person.icon,
       Person.id,
       Person.logo,
       Person.mailing_list_auto_subscribe_policy,
       Person.merged,
       Person.mugshot,
       Person.name,
       Person.personal_standing,
       Person.personal_standing_reason,
       Person.registrant,
       Person.renewal_policy,
       Person.subscriptionpolicy,
       Person.teamdescription,
       Person.teamowner,
       Person.verbose_bugnotifications,
       Person.visibility
FROM Person
WHERE Person.id = $INTLIMIT $INT

also we're downloading messsages from the librarian twice, for a total of several seconds overhead.

Related branches

Revision history for this message
Francis J. Lacoste (flacoste) wrote :

That's not a Person:+mailinglist-moderate OOPS. And I couldn't find the related OOPS in the OOPS report (even though that page id is supposed to have timed out).

Changed in launchpad:
status: Triaged → Invalid
Revision history for this message
Robert Collins (lifeless) wrote :

In such cases, we need to grep to find the original report.

grep Person:+mailinglist-moderate */2011-03-15/ -r

description: updated
Changed in launchpad:
status: Invalid → Triaged
description: updated
Revision history for this message
Curtis Hovey (sinzui) wrote :

This team is/was one of pathological lists. While the messages have been cleaned up by the list maintainers in recent weeks. it still has more than we expect to be in the queue. I think we should lower the batch rules to 20 since there are only four lists with more than 20 unmoderated messages.

           name | count
---------------------------+-------
 dhis2-kenya | 29
 ecryptfs | 453
 ubuntu-cyclists | 105
 ayatana | 58

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 736012] Re: Person:+mailinglist-moderate timeout

That seems like a cheap workaround. It surprises me that the default
batch is slow, but we can iterate on it later.

Revision history for this message
Robert Collins (lifeless) wrote :

Oh, but this isn't timing out at all now, we don't need to change it
until we see an actual moderation oops.

Revision history for this message
Robert Collins (lifeless) wrote :

Oh bah, confused because I didn't get my own mail back. Yes its timing out. The 23 duplicate lookups for person would help a lot. But it looks like there is a lot of unaccounted time, I guess its talking to mailman or something?

Revision history for this message
Curtis Hovey (sinzui) wrote :

+moderate does not talk to mailman. In this case, 50+ text messages are stored in Lp's db with a reference to the Lp user that sent the email and a status enum. Moderation just changes the status of the message. Mailman polls Lp for changes to status, does the work, then updates the status to indicate the work was done.

I think this issue relates to bug 770329 which was caused of the high number of moderated messages for ecryptfs.

tags: added: mailing-lists
Revision history for this message
Curtis Hovey (sinzui) wrote :

There are now only three lists is large queues. Most of the messages are spam from 2010 when Lp wrongly accepted message claiming to be from the list itself. I think we should just

select team.name, count(*) from person team join mailinglist ml on team.id = ml.team join messageapproval ma on ml.id = ma.mailing_list where ma.status = 0 group by team.name having count(*) > 20;
      name | count
-----------------+-------
 dhis2-kenya | 31
 ubuntu-cyclists | 105
 ayatana | 70

Revision history for this message
Curtis Hovey (sinzui) wrote :

This is a contrived oops to see the what is happening OOPS-cc376cdd298b458c9d1ec3f8dc17a21d. We can see it is going to the librarian for each message.

There are recent oopses for lPerson:+mailinglist-moderate about the librarian like
    http://launchpadlibrarian.net:$INT/$INT/at9XMvRzk21O4razMZE70hu7p7I.msg:

This might indicate that showing a lot of messages that are in the librarian is bad. Well, obviously we do not want to do that. We want to limit the files (text of the email) that we pull from the librarian. I see that HeldMessageDetails.email_message is calling self.message.raw.open(), which is actually is what is talking to the librarian. Every other thing that uses Message calls .text_contents instead.

Curtis Hovey (sinzui)
Changed in launchpad:
assignee: nobody → Curtis Hovey (sinzui)
Curtis Hovey (sinzui)
Changed in launchpad:
status: Triaged → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
tags: added: qa-ok
removed: qa-needstesting
William Grant (wgrant)
Changed in launchpad:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui)
Changed in launchpad:
assignee: Curtis Hovey (sinzui) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.