Merge lp:~cjwatson/launchpad/package-cache-drop-changelog into lp:launchpad

[Colin Watson]

Stop populating DistributionSourcePackageCache.changelog, and make update-pkgcache depopulate it.

Stop populating DistributionSourcePackageCache.changelog, and make update-pkgcache depopulate it.

The changelog was first added here in https://bugs.launchpad.net/launchpad/+bug/48735, and it is at least considered only as a bottom-priority item. But it doesn't make all that much sense, at least nowadays: we now have a pretty decent batched +changelog view, matching on changelog terms will often not be what you expected except in special-purpose circumstances, and general fishing expeditions are probably better handled by a general-purpose search engine. The changelog makes up about half of the table size (166 of 341 MB on dogfood), so I'd like to trim it down so that we can make the package cache quicker to update and easier to use for source package vocabulary pickers.

The next step will be to drop the column from the database, but we need this on production first to rebuild the fti column.

[William Grant (wgrant)]

review: Approve (code)

=== modified file 'lib/lp/soyuz/doc/package-cache-script.txt'
--- lib/lp/soyuz/doc/package-cache-script.txt	2012-06-13 07:35:50 +0000
+++ lib/lp/soyuz/doc/package-cache-script.txt	2016-05-17 13:35:55 +0000
@@ -31,6 +31,11 @@
   >>> warty.searchPackages(u'foobar').count()
   1
 
+The obsolete changelog field is still present in the sampledata.
+
+  >>> ubuntu.searchSourcePackages(u'placeholder').count()
+  2
+
 Normal operation produces INFO level information about the
 distribution and respective distroseriess considered in stderr.
 
@@ -72,6 +77,9 @@
   >>> warty.searchPackages(u'foobar').count()
   0
 
+  >>> ubuntu.searchSourcePackages(u'placeholder').count()
+  0
+
 Explicitly mark the database as dirty so that it is cleaned (see bug 994158).
 
   >>> from lp.testing.layers import DatabaseLayer

=== modified file 'lib/lp/soyuz/interfaces/distributionsourcepackagecache.py'
--- lib/lp/soyuz/interfaces/distributionsourcepackagecache.py	2015-10-21 09:37:08 +0000
+++ lib/lp/soyuz/interfaces/distributionsourcepackagecache.py	2016-05-17 13:35:55 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2016 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Source package in Distribution Cache interfaces."""
@@ -31,7 +31,7 @@
         "distro.")
     changelog = Attribute("A concatenation of the source package release "
         "changelog entries for this source package, where the status is "
-        "not REMOVED.")
+        "not REMOVED.  (Deprecated; due to be removed.)")
 
     distributionsourcepackage = Attribute("The DistributionSourcePackage "
         "for which this is a cache.")

=== modified file 'lib/lp/soyuz/model/distributionsourcepackagecache.py'
--- lib/lp/soyuz/model/distributionsourcepackagecache.py	2015-07-08 16:05:11 +0000
+++ lib/lp/soyuz/model/distributionsourcepackagecache.py	2016-05-17 13:35:55 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2016 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 __metaclass__ = type
@@ -167,13 +167,8 @@
         binpkgnames = set()
         binpkgsummaries = set()
         binpkgdescriptions = set()
-        sprchangelog = set()
         for spr in sprs:
             log.debug("Considering source version %s" % spr.version)
-            # changelog may be empty, in which case we don't want to add it
-            # to the set as the join would fail below.
-            if spr.changelog_entry is not None:
-                sprchangelog.add(spr.changelog_entry)
         binpkgs = IStore(BinaryPackageRelease).find(
             (BinaryPackageName.name, BinaryPackageRelease.summary,
              BinaryPackageRelease.description),
@@ -190,7 +185,8 @@
         cache.binpkgnames = ' '.join(sorted(binpkgnames))
         cache.binpkgsummaries = ' '.join(sorted(binpkgsummaries))
         cache.binpkgdescriptions = ' '.join(sorted(binpkgdescriptions))
-        cache.changelog = ' '.join(sorted(sprchangelog))
+        # Column due for deletion.
+        cache.changelog = None
 
     @classmethod
     def updateAll(cls, distro, archive, log, ztm, commit_chunk=500):

=== modified file 'lib/lp/soyuz/stories/distribution/xx-distribution-packages.txt'
--- lib/lp/soyuz/stories/distribution/xx-distribution-packages.txt	2015-09-04 12:36:43 +0000
+++ lib/lp/soyuz/stories/distribution/xx-distribution-packages.txt	2016-05-17 13:35:55 +0000
@@ -61,47 +61,6 @@
     The Mozilla Firefox web browser
     (Matching binaries: mozilla-firefox, mozilla-firefox-data.)
 
-Now try searching for text that we know to be in a change log entry, to
-prove that FTI works on change logs.  The text we're looking for is
-"placeholder" which is mentioned in the change log entry for pmount and
-libstdc++, so we are looking for two results here as the "placeholder"
-text is not mentioned in anything else that is indexed.
-
-    >>> browser.open("http://localhost/ubuntu/+search")
-    >>> field = browser.getControl(name="text")
-    >>> field.value = 'placeholder'
-    >>> browser.getControl('Search', index=0).click()
-
-Note, by default we only search on binary package names (as the fti is
-not currently so useful), so the initial result is empty, but contains
-a link to the fti/source package search:
-
-    >>> print extract_text(find_tag_by_id(browser.contents, 'no-results'))
-    Your search for “placeholder” did not return any results.
-    ...
-
-Clicking on the provided link to retry the search against source packages
-finds the fti results:
-
-    >>> browser.getLink(id='source-search').click()
-    >>> for tag in find_tags_by_class(
-    ...     browser.contents, 'batch-navigation-index'):
-    ...     print extract_text(tag)
-    All packages with sources matching your query “placeholder”
-    1...2 of 2 results
-
-    >>> soup = find_main_content(browser.contents)
-    >>> results = soup.findAll(attrs={'class': 'pagematch'})
-    >>> len(results)
-    2
-
-    >>> texts = [extract_text(html) for html in results]
-    >>> texts.sort()
-    >>> for text in texts:
-    ...    print text.encode('ascii', 'backslashreplace')
-    libstdc++
-    pmount
-
 
 Distribution package change summary
 -----------------------------------