Merge lp:~mvo/launchpad/maintenance-check-precise into lp:launchpad

[Michael Vogt]

[r=gmb][bug=911175] Make the support-timeframe information that gets added to the 'Packages' file in ubuntu more flexible for the upcomming 12.04 LTS release

This branch makes the support-timeframe information that gets added to the "Packages" file in ubuntu more flexible for the upcomming 12.04 LTS release.

When 10.04 (lucid) was released we added "Supported: {5y,3y,18m}" tags into the Packages file that is stored on archive.ubuntu.com. This was needed because the individual packages get a different support time based on the "seed" they are in. The server packages are longer supported than the ubuntu-desktop packages etc. The additional information makes it trivial people to check the support status of a particular package and also allows scripts like "ubuntu-support-status" to work.

For the upcoming 12.04 the timeframe is changed compared to lucid. The desktop packages are also supported for 5y and there may be architecture changes. The current code in LP is not flexible enough to handle different set of rules for LTS releases, it expected that the rules are all the same. This branch fixes this by adding a UbuntuMaintenance class that can than be used to extend for LTS releases.

Please note that I'm NOT a LP developer, this is a drive-by fix and I did not run ran the testsuite (as outlined in https://dev.launchpad.net/PatchSubmission) yet.

The QA I have done on this to run:
$ python launchpad/trunk/cronscripts/publishing/maintenance-check.py lucid > /tmp/lucid-old-code
$ python launchpad/maintenance-check-precise/cronscripts/publishing/maintenance-check.py lucid > /tmp/lucid-new-code
$ diff -u /tmp/lucid-old-code /tmp/lucid-new-code
(and the same for maverick).

[Graham Binns (gmb)]

Hi Michael,

I'm marking this as Needs-Information because of the point about QA below. I'm happy with the code, barring one or two tweaks.

Thanks for this branch. There were a bunch of cosmetic things that
needed cleaning up, but rather than ask you to do it I've pushed up a
branch with those cleanups here:
lp:~gmb/launchpad/maintenance-check-precise. Please merge that before
landing your branch.

I'll run the test suite on the branch for you presently and report any failures.

A couple of other things:

[1]

I'd love to see some tests for this, but it doesn't seem as if there are any in the codebase (aside from a test to see if maintenance-check.py runs as part of cron.germinate). Do you know how we can QA this properly to ensure that it doesn't break things horribly? (It's okay if you don't; I can ask people who will know).

[2]

131 +
132 +# python-apt compat, the fallback can be removed once the code
133 +# runs on lucid or newer
134 +try:
135 + AcquireProgress = apt.progress.base.AcquireProgress
136 + OpProgress = apt.progress.base.OpProgress
137 +except AttributeError:
138 + # really old (hardy) interface
139 + AcquireProgress = apt.progress.FetchProgress
140 + OpProgress = apt.progress.OpProgress
141 +

Is this really necessary? I thought all our machines ran at least Lucid
now. Or is this necessary until Hardy server gets EoL'd in 2013?

review: Needs Information (code)

[Michael Vogt (mvo)]

Thanks a bunch for your branch! I merged it back into my branch now.

(1) As for the tests, I know this is pretty bad currently, I will think a bit about it - would you accept regular unittest2 tests? A proper LP test setup on a lucid VM is pretty expensive timewise for me.

(2) If all machines that run this code are on lucid, thats really excellent. I put it there because I wanted to be cautious, but if its not needed I will be more than happy to get rid of it.

[Graham Binns (gmb)]

On 16 November 2011 11:26, Michael Vogt <email address hidden> wrote:
> Thanks a bunch for your branch! I merged it back into my branch now.
>
> (1) As for the tests, I know this is pretty bad currently, I will think a bit about it - would you accept regular unittest2 tests? A proper LP test setup on a lucid VM is pretty expensive timewise for me.

That will certainly work for me. If you write the tests I'll do the
work to integrate them into the LP suite.

> (2) If all machines that run this code are on lucid, thats really excellent. I put it there because I wanted to be cautious, but if its not needed I will be more than happy to get rid of it.

Okay, I'll double-check with the LOSAs on this one and get back to you.

[Graham Binns (gmb)]

On 16 November 2011 11:26, Michael Vogt <email address hidden> wrote:
> (2) If all machines that run this code are on lucid, thats really excellent. I put it there because I wanted to be cautious, but if its not needed I will be more than happy to get rid of it.

I've just checked with the LOSAs; mthaddon confirms that all machines
are running Lucid, so let's drop this and rejoice! :)

[Colin Watson (cjwatson)]

If it helps the QA question, I intend to rewrite maintenance-check in terms of python-germinate at some point in the not too distant future and integrate it into generate-extra-overrides. This is part of my work item in https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-image-build-pipeline to get extra overrides generated before the publisher writes indexes so that we stop having hysteresis across publisher runs. As part of that I'm more than happy to write much more comprehensive unit tests, and this will also reduce this script's dependency on external URLs (notably the germinate output on people.canonical.com).

I hope to be able to do this within about the next two months.

[Colin Watson (cjwatson)]

Oh, and to QA this before any of my future work lands, the simplest way to do this right now is as follows, on mawson:

 * Run cronscripts/publishing/cron.germinate.
 * Take a backup of /srv/launchpad.net/ubuntu-archive/ubuntu-misc.
 * Apply Michael's branch.
 * Run cronscripts/publishing/cron.germinate again.
 * Diff /srv/launchpad.net/ubuntu-archive/ubuntu-misc against the backup and sign off any differences.

(You can set the TEST_LAUNCHPADROOT environment variable if at any point you need to run cron.germinate from somewhere other than /srv/launchpad.net/codelines/current.)

This procedure should take well under half an hour. I did it recently for another branch, so I'd be happy to QA this if you want. I'd like to see Michael's branch landed, both because it would be good for appropriate Supported extra overrides to be in place for Precise as soon as possible, and because the longer it remains outstanding the more I'll have to deal with conflicts between this and my work in progress (for instance, I think I'll already have to resolve my new-python-apt branch against this, assuming that this one lands first).

[Graham Binns (gmb)]

Thanks to Colin and Michael for their stirling work. I'll get this landed in the near future.

review: Approve (code)

=== modified file 'cronscripts/publishing/maintenance-check.py'
--- cronscripts/publishing/maintenance-check.py	2011-12-24 13:09:35 +0000
+++ cronscripts/publishing/maintenance-check.py	2012-01-03 11:04:30 +0000
@@ -2,9 +2,6 @@
 #
 # python port of the nice maintainace-check script by  Nick Barcet
 #
-# Taken from:
-#  https://code.edge.launchpad.net/~mvo/ubuntu-maintenance-check/python-port
-# (where it will vanish once taken here)
 
 import logging
 from optparse import OptionParser
@@ -16,6 +13,43 @@
 import apt
 import apt_pkg
 
+
+class UbuntuMaintenance(object):
+    """ Represents the support timeframe for a regular ubuntu release """
+
+    # architectures that are full supported (including LTS time)
+    PRIMARY_ARCHES = [
+        "i386",
+        "amd64",
+        ]
+
+    # architectures we support (but not for LTS time)
+    SUPPORTED_ARCHES = PRIMARY_ARCHES + ["armel"]
+
+    # what defines the seeds is documented in wiki.ubuntu.com/SeedManagement
+    SERVER_SEEDS = [
+        "server-ship",
+        "supported-server",
+        ]
+    DESKTOP_SEEDS = [
+        "ship",
+        "supported-desktop",
+        "supported-desktop-extra",
+        ]
+    SUPPORTED_SEEDS = ["all"]
+
+    # normal support timeframe
+    # time, seeds
+    SUPPORT_TIMEFRAME = [
+        ("18m", SUPPORTED_SEEDS),
+        ]
+
+    # distro names that we check the seeds for
+    DISTRO_NAMES = [
+        "ubuntu",
+        ]
+
+
 # This is fun! We have a bunch of cases for 10.04 LTS
 #
 #  - distro "ubuntu" follows SUPPORT_TIMEFRAME_LTS but only for
@@ -24,41 +58,44 @@
 #    considered *but* only follow SUPPORT_TIMEFRAME
 #  - anything that is in armel follows SUPPORT_TIMEFRAME
 #
-
-# codename of the lts releases
-LTS_RELEASES = ["dapper", "hardy", "lucid"]
-
-# architectures that are full supported (including LTS time)
-PRIMARY_ARCHES = ["i386", "amd64"]
-
-# architectures we support (but not for LTS time)
-SUPPORTED_ARCHES = PRIMARY_ARCHES + ["armel"]
-
-# what defines the seeds is documented in wiki.ubuntu.com/SeedManagement
-SERVER_SEEDS = ["supported-server", "server-ship"]
-DESKTOP_SEEDS = ["ship", "supported-desktop", "supported-desktop-extra"]
-SUPPORTED_SEEDS = ["all"]
-
-# normal support timeframe
-# time, seeds, arches
-SUPPORT_TIMEFRAME = [
-    ("18m", SUPPORTED_SEEDS),
-]
-
-# lts support timeframe
-# time, seeds, arches
-SUPPORT_TIMEFRAME_LTS = [
-    ("5y", SERVER_SEEDS),
-    ("3y", DESKTOP_SEEDS),
-    ("18m", SUPPORTED_SEEDS),
-]
-
-# distro names and if they get LTS support (order is important)
-DISTRO_NAMES_AND_LTS_SUPPORT = [
-    ("ubuntu", True),
-    ("kubuntu", True),
-    ("netbook", False),
-    ]
+class LucidUbuntuMaintenance(UbuntuMaintenance):
+    """ Represents the support timeframe for a 10.04 (lucid) LTS release,
+        the exact rules differ from LTS release to LTS release
+    """
+
+    # lts support timeframe, order is important, least supported must be last
+    # time, seeds
+    SUPPORT_TIMEFRAME = [
+        ("5y",  UbuntuMaintenance.SERVER_SEEDS),
+        ("3y",  UbuntuMaintenance.DESKTOP_SEEDS),
+        ("18m", UbuntuMaintenance.SUPPORTED_SEEDS),
+        ]
+
+    # on a LTS this is significant, it defines what names get LTS support
+    DISTRO_NAMES = [
+        "ubuntu",
+        "kubuntu",
+        ]
+
+
+class PreciseUbuntuMaintenance(UbuntuMaintenance):
+    """ The support timeframe for the 12.04 (precise) LTS release.
+        This changes the timeframe for desktop packages from 3y to 5y
+    """
+
+    # lts support timeframe, order is important, least supported must be last
+    # time, seeds
+    SUPPORT_TIMEFRAME = [
+        ("5y", UbuntuMaintenance.SERVER_SEEDS),
+        ("5y", UbuntuMaintenance.DESKTOP_SEEDS),
+        ("18m", UbuntuMaintenance.SUPPORTED_SEEDS),
+        ]
+
+    # on a LTS this is significant, it defines what names get LTS support
+    DISTRO_NAMES = [
+        "ubuntu",
+        ]
+
 
 # Names of the distribution releases that are not supported by this
 # tool. All later versions are supported.
@@ -312,6 +349,12 @@
     else:
         distro = "lucid"
 
+    # maintenance class to use
+    klass = globals().get("%sUbuntuMaintenance" % distro.capitalize())
+    if klass is None:
+        klass = UbuntuMaintenance
+    ubuntu_maintenance = klass()
+
     # make sure our deb-src information is up-to-date
     create_and_update_deb_src_source_list(distro)
 
@@ -326,7 +369,7 @@
 
     # go over the distros we need to check
     pkg_support_time = {}
-    for (name, lts_supported) in DISTRO_NAMES_AND_LTS_SUPPORT:
+    for name in ubuntu_maintenance.DISTRO_NAMES:
 
         # get basic structure file
         try:
@@ -336,17 +379,13 @@
             continue
 
         # get dicts of pkgname -> support timeframe string
-        support_timeframe = SUPPORT_TIMEFRAME
-        if lts_supported and distro in LTS_RELEASES:
-            support_timeframe = SUPPORT_TIMEFRAME_LTS
-        else:
-            support_timeframe = SUPPORT_TIMEFRAME
+        support_timeframe = ubuntu_maintenance.SUPPORT_TIMEFRAME
         get_packages_support_time(
             structure, name, pkg_support_time, support_timeframe)
 
     # now go over the bits in main that we have not seen (because
     # they are not in any seed and got added manually into "main"
-    for arch in PRIMARY_ARCHES:
+    for arch in ubuntu_maintenance.PRIMARY_ARCHES:
         rootdir = "./aptroot.%s" % distro
         apt_pkg.config.set("APT::Architecture", arch)
         cache = apt.Cache(rootdir=rootdir)
@@ -356,6 +395,9 @@
             logging.exception("cache.update() failed")
         cache.open()
         for pkg in cache:
+            # ignore multiarch package names
+            if ":" in pkg.name:
+                continue
             if not pkg.name in pkg_support_time:
                 pkg_support_time[pkg.name] = support_timeframe[-1][0]
                 logging.warn(
@@ -404,12 +446,12 @@
             # go over the supported arches, they are divided in
             # first-class (PRIMARY) and second-class with different
             # support levels
-            for arch in SUPPORTED_ARCHES:
+            for arch in ubuntu_maintenance.SUPPORTED_ARCHES:
                 # ensure we do not overwrite arch-specific overwrites
                 pkgname_and_arch = "%s/%s" % (pkgname, arch)
                 if pkgname_and_arch in pkg_support_time:
                     break
-                if arch in PRIMARY_ARCHES:
+                if arch in ubuntu_maintenance.PRIMARY_ARCHES:
                     # arch with full LTS support
                     print "%s %s %s" % (
                         pkgname_and_arch, SUPPORT_TAG,
@@ -419,4 +461,4 @@
                     # support_timeframe
                     print "%s %s %s" % (
                         pkgname_and_arch, SUPPORT_TAG,
-                        SUPPORT_TIMEFRAME[0][0])
+                        ubuntu_maintenance.SUPPORT_TIMEFRAME[-1][0])