Merge lp:~sinzui/launchpad/portlet-package-summary-timeout into lp:launchpad

[Curtis Hovey]

[Curtis Hovey (sinzui)]

This is my branch to make getMostRecentlyLinkedPackagings faster.

    lp:~sinzui/launchpad/portlet-package-summary-timeout
    Diff size: 154
    Launchpad bug: https://bugs.launchpad.net/bugs/526583
    Test command: ./bin/test -vv -t reg.*distroseries
    Pre-implementation: Edwin, jtv
    Target release: 10.02

Make getMostRecentlyLinkedPackagings faster
-------------------------------------------

1 in 3 page loads of the /ubuntu/lucid timesout.
getMostRecentlyLinkedPackagings() is expensive call. It needs to be faster, or
we need to consider a cache. We cannot release this portlet if this problem
persists.

Rules
-----

    * Add a filter on bug.heat that will ignore the cold bugs. A filter of
      > 200 reduces the hottest bugs to about from 700 to 200 and is 4 times
      faster on staging. The most viewed pages are unchanged by this filter.
      The query or implementation can be revisited in 10.03 when we get more
      user feedback.
    * bug.hottness was renamed to bug.heat; update the tests to use the same
      term.
    * bug.heat can now be set by admins using setHeat(); remove the SQL
      hacks to set bug heat.
    * Uncomment the distroseries packaging portlet story to verify it
      is how users will discover the needs upstream linking page.

QA
--

    * Visit http://staging.launchpad.net/ubuntu/lucid
    * Verify the page does not timeout.

Lint
----

Linting changed files:
  lib/lp/registry/doc/distroseries.txt
  lib/lp/registry/model/distroseries.py
  lib/lp/registry/stories/distroseries/xx-show-distroseries-packaging.txt
  lib/lp/registry/tests/test_distroseries.py

Test
----

    * lib/lp/registry/doc/distroseries.txt
      * Updated the bug count because the cold bugs are not counted.
    * lib/lp/registry/stories/distroseries/xx-show-distroseries-packaging.txt
      * Added heat to one bug to verify the formatting of a singular bug
        is correct.
      * Updated the story to show that users will get to +needs-packaging
        via the portlet.
    * lib/lp/registry/tests/test_distroseries.py
      * Updated the message count and heat to look like numbers that are
        actually in the database.
      * Removed the sql hack to set bug heat.
      * Renamed hottness to heat.

Implementation
--------------

    * lib/lp/registry/model/distroseries.py
      * Added a filter on bug.heat to ignore the colder bugs.
      * Extracted the filter and weight to variables so that it is easier
        to update the SQL.

[Paul Hummer (rockstar)]

<rockstar> sinzui, what specifically is the portlet you're talking about?
<rockstar> sinzui, ?
<sinzui> https://edge.launchpad.net/ubuntu/lucid <-- Upstream packaging
<rockstar> sinzui, ah, okay, thanks.
<rockstar> sinzui, did you cherry pick to staging or were you running raw queries?
<sinzui> rockstar: I ran raw queries comparing the oops query to the modified one
<rockstar> sinzui, okay.
<rockstar> sinzui, this looks good. r=rockstar

review: Approve (code)

=== modified file 'lib/lp/registry/doc/distroseries.txt'
--- lib/lp/registry/doc/distroseries.txt	2010-02-19 12:03:35 +0000
+++ lib/lp/registry/doc/distroseries.txt	2010-02-25 17:55:29 +0000
@@ -516,7 +516,7 @@
     alsa-utils  0  0
     cnews  0  0
     libstdc++  0  0
-    linux-source-2.6.15  1  0
+    linux-source-2.6.15  0  0
 
 
 The distroseries getPrioritizedlPackagings() method that returns a prioritized

=== modified file 'lib/lp/registry/model/distroseries.py'
--- lib/lp/registry/model/distroseries.py	2010-02-19 20:24:52 +0000
+++ lib/lp/registry/model/distroseries.py	2010-02-25 17:55:29 +0000
@@ -412,6 +412,11 @@
     @property
     def _current_sourcepackage_joins_and_conditions(self):
         """The SQL joins and conditions to prioritize source packages."""
+        # Bugs and PO messages are heuristically scored. These queries
+        # can easily timeout so filters and weights are used to create
+        # an acceptable prioritization of packages that is fast to excecute.
+        bug_heat_filter = 200
+        po_message_weight = .5
         heat_score = ("""
             LEFT JOIN (
                 SELECT
@@ -425,18 +430,21 @@
                     BugTask.sourcepackagename is not NULL
                     AND BugTask.distribution = %(distribution)s
                     AND BugTask.status in %(statuses)s
+                    AND Bug.heat > %(bug_heat_filter)s
                 GROUP BY BugTask.sourcepackagename
                 ) bugs
                 ON SourcePackageName.id = bugs.sourcepackagename
             """ % sqlvalues(
                 distribution=self.distribution,
-                statuses=UNRESOLVED_BUGTASK_STATUSES))
+                statuses=UNRESOLVED_BUGTASK_STATUSES,
+                bug_heat_filter=bug_heat_filter))
         message_score = ("""
             LEFT JOIN (
                 SELECT
                     POTemplate.sourcepackagename,
                     POTemplate.distroseries,
-                    SUM(POTemplate.messagecount) / 2 AS po_messages,
+                    SUM(POTemplate.messagecount) * %(po_message_weight)s
+                        AS po_messages,
                     SUM(POTemplate.messagecount) AS total_messages
                 FROM POTemplate
                 WHERE
@@ -448,7 +456,9 @@
                 ) messages
                 ON SourcePackageName.id = messages.sourcepackagename
                 AND DistroSeries.id = messages.distroseries
-            """ % sqlvalues(distroseries=self))
+            """ % sqlvalues(
+                distroseries=self,
+                po_message_weight=po_message_weight))
         joins = ("""
             SourcePackageName
             JOIN SourcePackageRelease spr

=== modified file 'lib/lp/registry/stories/distroseries/xx-show-distroseries-packaging.txt'
--- lib/lp/registry/stories/distroseries/xx-show-distroseries-packaging.txt	2010-02-17 13:16:21 +0000
+++ lib/lp/registry/stories/distroseries/xx-show-distroseries-packaging.txt	2010-02-25 17:55:29 +0000
@@ -63,23 +63,27 @@
 that need upstream packaging links. The list is prioritized so that the
 packages with the greatest need are listed first.
 
-    >>> # XXX sinzui 2010-01-21 bug=487793: Enable this when the portlet
-    >>> # is added.
-    >>> #anon_browser.open('http://launchpad.dev/ubuntu/hoary')
-    >>> #anon_browser.getLink('Needs upstream links').click()
-    >>> anon_browser.open(
-    ...     'http://launchpad.dev/ubuntu/hoary/+needs-packaging')
+    >>> # Make a very hot bug to verify the text formatting rule.
+    >>> from zope.component import getUtility
+    >>> from lp.bugs.interfaces.bug import IBugSet
+    >>> login('[e-mail]')
+    >>> bug = getUtility(IBugSet).get(10)
+    >>> bug.setHeat(300)
+    >>> logout()
+
+    >>> anon_browser.open('http://launchpad.dev/ubuntu/hoary')
+    >>> anon_browser.getLink('Needs upstream links').click()
     >>> print anon_browser.title
     Needs upstream links : ...
 
     >>> content = find_main_content(anon_browser.contents)
     >>> print extract_text(find_tag_by_id(content, 'packages'))
     Source Package       Bugs     Translations
+    linux-source-2.6.15  1 bug    No strings
     pmount               No bugs  64 strings
     alsa-utils           No bugs  No strings
     cnews                No bugs  No strings
     libstdc++            No bugs  No strings
-    linux-source-2.6.15  1 bug    No strings
 
 The counts in the listing link to their respect bugs and translations
 pages.

=== modified file 'lib/lp/registry/tests/test_distroseries.py'
--- lib/lp/registry/tests/test_distroseries.py	2010-02-19 12:03:35 +0000
+++ lib/lp/registry/tests/test_distroseries.py	2010-02-25 17:55:29 +0000
@@ -12,7 +12,6 @@
 from zope.component import getUtility
 from zope.security.proxy import removeSecurityProxy
 
-from canonical.database.sqlbase import cursor
 from canonical.launchpad.ftests import ANONYMOUS, login
 from lp.soyuz.interfaces.archive import ArchivePurpose, IArchiveSet
 from lp.registry.interfaces.distroseries import (
@@ -210,9 +209,9 @@
         self.main_component = component_set['main']
         self.universe_component = component_set['universe']
         self.makeSeriesPackage('normal')
-        self.makeSeriesPackage('translatable', messages=120)
-        self.makeSeriesPackage('hot', hotness=100)
-        self.makeSeriesPackage('hot-translatable', hotness=80, messages=60)
+        self.makeSeriesPackage('translatable', messages=800)
+        self.makeSeriesPackage('hot', heat=500)
+        self.makeSeriesPackage('hot-translatable', heat=250, messages=1000)
         self.makeSeriesPackage('main', is_main=True)
         self.makeSeriesPackage('linked')
         self.linkPackage('linked')
@@ -220,7 +219,7 @@
         login(ANONYMOUS)
 
     def makeSeriesPackage(self, name,
-                          is_main=False, hotness=None, messages=None):
+                          is_main=False, heat=None, messages=None):
         # Make a published source package.
         if is_main:
             component = self.main_component
@@ -232,14 +231,10 @@
             component=component)
         source_package = self.factory.makeSourcePackage(
             sourcepackagename=sourcepackagename, distroseries=self.series)
-        if hotness is not None:
+        if heat is not None:
             bugtask = self.factory.makeBugTask(
                 target=source_package, owner=self.user)
-            # hotness is not exposed in the model yet.
-            # bugtask.bug.hotness = hotness
-            cur = cursor()
-            cur.execute("UPDATE Bug SET heat = %d WHERE id = %d" % (
-                (hotness, bugtask.bug.id)))
+            bugtask.bug.setHeat(heat)
         if messages is not None:
             template = self.factory.makePOTemplate(
                 distroseries=self.series, sourcepackagename=sourcepackagename,