Merge lp:~jtv/launchpad/bug-526462 into lp:launchpad

[Jeroen T. Vermeulen]

Integrate make-ubuntu-sane.py sample-data cleanup script into Launchpad.

[Jeroen T. Vermeulen (jtv)]

= Bug 526462 =

To run Soyuz locally on a development machine, one needs to run a few scripts that are not being maintained, and are kept outside of the Launchpad source tree. See https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally

This branch takes the make-ubuntu-sane.py script from that page and integrates it into the Launchpad source tree as utilities/soyuz-sampledata-cleanup.py. The original was written by William Grant, but I discussed with him on IRC today and he has no objection to this being included under Canonical's copyright with a "based on code by wgrant" notice.

As you'll see, the script replaces the existing Ubuntu test series in the playground sample data with a series mirroring real-world Ubuntu releases. This is meant for the playground sample data that you would run on your local machine while testing manually. We discussed updating the sample data as included in the source code, but that would involve uploading tarballs to the local librarian which would have complicated the job and bloated the branch.

Since the script is not meant to run on production databases, it attempts to check for that condition. Our procedures should prevent this, however, so it's a pair of suspenders in addition to the belt we already have. There is no guarantee that the check will prevent any attempt at disastrously wrong use, and I don't think it's worth the hassle of automated testing. If deviating from the "happy path" in this way should break the script, then the script is still dealing with the situation properly: die before it deletes any Ubuntu data.

A doctest now verifies that the script will execute properly. The test would have to force a dirty database in order to avoid test isolation problems—if it weren't for the fact that it uses the --dry-run option (not present in the original script) avoids making any changes to the database.

No lint. To Q/A, just use it on your local dev system. To test:
{{{
./bin/test -vv -t sampledata-cleanup
}}}

Jeroen

[Michael Nelson (michael.nelson)]

Hi Jeroen,

This is great! Thanks for getting this in with a test to ensure it'll stay in sync with the code-base. I think it would also be worthwhile to add a test to ensure it won't run as expected in other envs (you never know who will come along and modify your code with unintended effects).

As mentioned on IRC, I realise this is the suspenders and not the belt, but I still think we should *only* allow this to run when LPCONFIG == 'development'. If someone wants to run it in another env. (or for some reason, hasn't set LPCONFIG), they'll know enough to modify the script for their needs. I mean, looking at lp-production-configs, there are *loads*, and so I'm not sure that blacklisting a few is that useful. This would also mean you could get rid of the 'guess' based on the ids?

Thoughts?

review: Needs Fixing (code)

[Michael Nelson (michael.nelson)]

As discussed, we have other damaging scripts intended for local development that do not even do any checks. I do think it would be a simpler (and sufficient) implementation to just check LPCONFIG == 'development', but I'll leave that up to you.

16:42 < jtv> noodles775: thanks for the review!
16:43 < jtv> noodles775: I'm a bit worried though that being this strict would stop people from playing with customized configs.
16:43 < jtv> I realize it's always a tradeoff, but there's also the "do we have a really large production-like table" test.
16:44 < noodles775> jtv: well, if they're playing with customized configs, they're not going to have a problem spending 10 seconds to adjust the script if
                    they want to.
16:45 < jtv> noodles775: true, but this script already has another check to ensure it's not running on production; how many scripts do we have (without ever
             any trouble) that are just as dangerous without anyone ever adding checks like this?
16:45 < jtv> I mean, if I hadn't added the check, would the average reviewer have thought of adding it?
16:45 < noodles775> jtv: which scripts? make schema?
16:46 < jtv> noodles775: I guess, though I've no idea whether that has any checks.
16:46 < noodles775> jtv: yeah, I'm not sure. All the other scripts that I can think of *add* info, not delete.
16:46 < jtv> launchpad-database-setup
16:46 * noodles775 looks
16:47 < jtv> mock-code-import ("warning! run make schema first!")
16:48 < jtv> We also have a script now, apparently, to remove private data. There may be more.
16:49 < noodles775> Yep, you're right.
16:50 < jtv> So I don't want to spend my time guarding against the admin who accidentally goes through the rigmarole for running scripts against a
             production db, with the --force option added.
16:51 < noodles775> yeah, I agree. I was more worried about the situation where a person runs it against a smaller DB with a different config.
16:53 < jtv> noodles775: there's a good chance that the script might fail, and I'd estimate the risk of them running "make schema" by accident considerably
             larger.
16:53 < jtv> (from shell history, f'rinstance)
16:54 < noodles775> jtv: I just thought it would have been a simpler implementation, not more difficult (ie. LPCONFIG == 'development'), but yep, I'm
                    updating the MP now.

review: Approve (code)

[Jeroen T. Vermeulen (jtv)]

Thanks! I did start out thinking that one check should be enough. I ended up with one that might produce false positives, plus one that might produce false negatives. For that situation, a --force against false positives plus a final just-in-case safeguard seemed the right approach. In any case the check is easy; maybe someday we'll build reusable helpers for this.

=== added file 'lib/lp/soyuz/doc/sampledata-cleanup.txt'
--- lib/lp/soyuz/doc/sampledata-cleanup.txt	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/doc/sampledata-cleanup.txt	2010-02-24 12:36:31 +0000
@@ -0,0 +1,22 @@
+= Sample data cleanup =
+
+In order to run Soyuz locally on a development system, the sample data
+must be cleaned up and customized a bit.  This is done by a the script
+utilities/soyuz-sampledata-cleanup.py.
+
+We only need this script for the playground sample data, so there's
+little point in inspecting what it does to the test database in detail.
+
+    >>> from canonical.launchpad.ftests.script import run_script
+
+The --dry-run option makes the script roll back its changes.
+
+    >>> return_code, output, error = run_script(
+    ...     'utilities/soyuz-sampledata-cleanup.py', args=['--dry-run'])
+
+    >>> print return_code
+    0
+
+    >>> print error
+    INFO ...
+    INFO Done.

=== added file 'utilities/soyuz-sampledata-cleanup.py'
--- utilities/soyuz-sampledata-cleanup.py	1970-01-01 00:00:00 +0000
+++ utilities/soyuz-sampledata-cleanup.py	2010-02-24 12:36:31 +0000
@@ -0,0 +1,316 @@
+#!/usr/bin/python2.5
+# pylint: disable-msg=W0403
+
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+#
+# This code is based on William Grant's make-ubuntu-sane.py script, but
+# reorganized to fit Launchpad coding guidelines, and extended.  The
+# code is included under Canonical copyright with his permission
+# (2010-02-24).
+
+"""Clean up sample data so it will allow Soyuz to run locally.
+
+DO NOT RUN ON PRODUCTION SYSTEMS.  This script deletes lots of
+Ubuntu-related data.
+"""
+
+__metaclass__ = type
+
+import _pythonpath
+
+from optparse import OptionParser
+from os import getenv
+import re
+import sys
+
+from zope.component import getUtility
+from zope.event import notify
+from zope.lifecycleevent import ObjectCreatedEvent
+from zope.security.proxy import removeSecurityProxy
+
+from storm.store import Store
+
+from canonical.database.sqlbase import sqlvalues
+
+from canonical.lp import initZopeless
+
+from canonical.launchpad.interfaces.launchpad import (
+    ILaunchpadCelebrities)
+from canonical.launchpad.scripts import execute_zcml_for_scripts
+from canonical.launchpad.scripts.logger import logger, logger_options
+from canonical.launchpad.webapp.interfaces import (
+    IStoreSelector, MAIN_STORE, SLAVE_FLAVOR)
+
+from lp.registry.interfaces.series import SeriesStatus
+from lp.soyuz.interfaces.component import IComponentSet
+from lp.soyuz.interfaces.section import ISectionSet
+from lp.soyuz.interfaces.sourcepackageformat import (
+    ISourcePackageFormatSelectionSet, SourcePackageFormat)
+from lp.soyuz.model.section import SectionSelection
+from lp.soyuz.model.component import ComponentSelection
+
+
+class DoNotRunOnProduction(Exception):
+    """Error: do not run this script on production (-like) systems."""
+
+
+def get_max_id(store, table_name):
+    """Find highest assigned id in given table."""
+    max_id = store.execute("SELECT max(id) FROM %s" % table_name).get_one()
+    if max_id is None:
+        return None
+    else:
+        return max_id[0]
+
+
+def check_preconditions(options):
+    """Try to ensure that it's safe to run.
+
+    This script must not run on a production server, or anything
+    remotely like it.
+    """
+    store = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
+
+    # Just a guess, but dev systems aren't likely to have ids this high
+    # in this table.  Production data does.
+    real_data = (get_max_id(store, "TranslationMessage") >= 1000000)
+    if real_data and not options.force:
+        raise DoNotRunOnProduction(
+            "Refusing to delete Ubuntu data unless you --force me.")
+
+    # For some configs it's just absolutely clear this script shouldn't
+    # run.  Don't even accept --force there.
+    forbidden_configs = re.compile('(edge|lpnet|production)')
+    current_config = getenv('LPCONFIG', 'an unknown config')
+    if forbidden_configs.match(current_config):
+        raise DoNotRunOnProduction(
+            "I won't delete Ubuntu data on %s and you can't --force me."
+            % current_config)
+
+
+def parse_args(arguments):
+    """Parse command-line arguments.
+
+    :return: (options, args, logger)
+    """
+    parser = OptionParser(
+        description="Delete existing Ubuntu releases and set up new ones.")
+    parser.add_option('-f', '--force', action='store_true', dest='force',
+        help="DANGEROUS: run even if the database looks production-like.")
+    parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
+        help="Do not commit changes.")
+    logger_options(parser)
+
+    options, args = parser.parse_args(arguments)
+    return options, args, logger(options)
+
+
+def get_person(name):
+    """Return `IPersonSet` utility."""
+    # Avoid circular import.
+    from lp.registry.interfaces.person import IPersonSet
+    return getUtility(IPersonSet).getByName(name)
+
+
+def retire_series(distribution):
+    """Mark all `DistroSeries` for `distribution` as obsolete."""
+    for series in distribution.series:
+        series.status = SeriesStatus.OBSOLETE
+
+
+def retire_active_publishing_histories(histories, requester):
+    """Retire all active publishing histories in the given collection."""
+    # Avoid circular import.
+    from lp.soyuz.interfaces.publishing import active_publishing_status
+    for history in histories(status=active_publishing_status):
+        history.requestDeletion(
+            requester, "Cleaned up because of missing Librarian files.")
+
+
+def retire_distro_archives(distribution, culprit):
+    """Retire all items in `distribution`'s archives."""
+    for archive in distribution.all_distro_archives:
+        retire_active_publishing_histories(
+            archive.getPublishedSources, culprit)
+        retire_active_publishing_histories(
+            archive.getAllPublishedBinaries, culprit)
+
+
+def retire_ppas(distribution):
+    """Disable all PPAs for `distribution`."""
+    for ppa in distribution.getAllPPAs():
+        removeSecurityProxy(ppa).publish = False
+
+
+def set_lucille_config(distribution):
+    """Set lucilleconfig on all series of `distribution`."""
+    for series in distribution.series:
+        removeSecurityProxy(series).lucilleconfig = '''[publishing]
+components = main restricted universe multiverse'''
+
+
+def create_sections(distroseries):
+    """Set up some sections for `distroseries`."""
+    section_names = (
+        'admin', 'cli-mono', 'comm', 'database', 'devel', 'debug', 'doc',
+        'editors', 'electronics', 'embedded', 'fonts', 'games', 'gnome',
+        'graphics', 'gnu-r', 'gnustep', 'hamradio', 'haskell', 'httpd',
+        'interpreters', 'java', 'kde', 'kernel', 'libs', 'libdevel', 'lisp',
+        'localization', 'mail', 'math', 'misc', 'net', 'news', 'ocaml',
+        'oldlibs', 'otherosfs', 'perl', 'php', 'python', 'ruby', 'science',
+        'shells', 'sound', 'tex', 'text', 'utils', 'vcs', 'video', 'web',
+        'x11', 'xfce', 'zope')
+    store = Store.of(distroseries)
+    for section_name in section_names:
+        section = getUtility(ISectionSet).ensure(section_name)
+        if section not in distroseries.sections:
+            store.add(
+                SectionSelection(distroseries=distroseries, section=section))
+
+
+def create_components(distroseries, uploader):
+    """Set up some components for `distroseries`."""
+    component_names = ('main', 'restricted', 'universe', 'multiverse')
+    store = Store.of(distroseries)
+    main_archive = distroseries.distribution.main_archive
+    for component_name in component_names:
+        component = getUtility(IComponentSet).ensure(component_name)
+        if component not in distroseries.components:
+            store.add(
+                ComponentSelection(
+                    distroseries=distroseries, component=component))
+        main_archive.newComponentUploader(uploader, component)
+        main_archive.newQueueAdmin(uploader, component)
+
+
+def create_series(parent, full_name, version, status):
+    """Set up a `DistroSeries`."""
+    distribution = parent.distribution
+    owner = parent.owner
+    name = full_name.split()[0].lower()
+    title = "The " + full_name
+    displayname = full_name.split()[0]
+    new_series = distribution.newSeries(name=name, title=title,
+        displayname=displayname, summary='Ubuntu %s is good.' % version,
+        description='%s is awesome.' % version, version=version,
+        parent_series=parent, owner=owner)
+    new_series.status = status
+    notify(ObjectCreatedEvent(new_series))
+
+    # This bit copied from scripts/ftpmaster-tools/initialise-from-parent.py.
+    assert new_series.architectures.count() == 0, (
+        "Cannot copy distroarchseries from parent; this series already has "
+        "distroarchseries.")
+
+    store = Store.of(parent)
+    store.execute("""
+        INSERT INTO DistroArchSeries
+          (distroseries, processorfamily, architecturetag, owner, official)
+        SELECT %s, processorfamily, architecturetag, %s, official
+        FROM DistroArchSeries WHERE distroseries = %s
+        """ % sqlvalues(new_series, owner, parent))
+
+    i386 = new_series.getDistroArchSeries('i386')
+    i386.supports_virtualized = True
+    new_series.nominatedarchindep = i386
+
+    new_series.initialiseFromParent()
+    return new_series
+
+
+def create_sample_series(original_series, log):
+    """Set up sample `DistroSeries`.
+
+    :param original_series: The parent for the first new series to be
+        created.  The second new series will have the first as a parent,
+        and so on.
+    """
+    series_descriptions = [
+        ('Dapper Drake', SeriesStatus.SUPPORTED, '6.06'),
+        ('Edgy Eft', SeriesStatus.OBSOLETE, '6.10'),
+        ('Feisty Fawn', SeriesStatus.OBSOLETE, '7.04'),
+        ('Gutsy Gibbon', SeriesStatus.OBSOLETE, '7.10'),
+        ('Hardy Heron', SeriesStatus.SUPPORTED, '8.04'),
+        ('Intrepid Ibex', SeriesStatus.SUPPORTED, '8.10'),
+        ('Jaunty Jackalope', SeriesStatus.SUPPORTED, '9.04'),
+        ('Karmic Koala', SeriesStatus.CURRENT, '9.10'),
+        ('Lucid Lynx', SeriesStatus.DEVELOPMENT, '10.04'),
+        ]
+
+    parent = original_series
+    for full_name, status, version in series_descriptions:
+        log.info('Creating %s...' % full_name)
+        parent = create_series(parent, full_name, version, status)
+
+
+def clean_up(distribution, log):
+    # First we eliminate all active publishings in the Ubuntu main archives.
+    # None of the librarian files exist, so it kills the publisher.
+
+    # Could use IPublishingSet.requestDeletion() on the published sources to
+    # get rid of the binaries too, but I don't trust that there aren't
+    # published binaries without corresponding sources.
+
+    log.info("Deleting all items in official archives...")
+    retire_distro_archives(distribution, get_person('name16'))
+
+    # Disable publishing of all PPAs, as they probably have broken
+    # publishings too.
+    log.info("Disabling all PPAs...")
+    retire_ppas(distribution)
+
+    retire_series(distribution)
+
+
+def set_source_package_format(distroseries):
+    """Register a series' source package format selection."""
+    utility = getUtility(ISourcePackageFormatSelectionSet)
+    format = SourcePackageFormat.FORMAT_1_0
+    if utility.getBySeriesAndFormat(distroseries, format) is None:
+        utility.add(distroseries, format)
+
+
+def populate(distribution, parent_series_name, uploader_name, log):
+    """Set up sample data on `distribution`."""
+    parent_series = distribution.getSeries(parent_series_name)
+
+    # Set up lucilleconfig on all series.  The sample data lacks this.
+    log.info("Setting lucilleconfig...")
+    set_lucille_config(distribution)
+
+    log.info("Configuring sections...")
+    create_sections(parent_series)
+
+    log.info("Configuring components and permissions...")
+    create_components(parent_series, get_person(uploader_name))
+
+    set_source_package_format(parent_series)
+
+    create_sample_series(parent_series, log)
+
+
+def main(argv):
+    options, args, log = parse_args(argv[1:])
+
+    execute_zcml_for_scripts()
+    txn = initZopeless(dbuser='launchpad')
+
+    check_preconditions(options.force)
+
+    ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
+    clean_up(ubuntu, log)
+
+    # Use Hoary as the root, as Breezy and Grumpy are broken.
+    populate(ubuntu, 'hoary', 'ubuntu-team', log)
+
+    if options.dry_run:
+        txn.abort()
+    else:
+        txn.commit()
+
+    log.info("Done.")
+
+
+if __name__ == "__main__":
+    main(sys.argv)