Merge lp:~cjwatson/launchpad/remove-distro-parts into lp:launchpad

[Colin Watson]

Remove cronscripts/publishing/distro-parts/ubuntu and support for in-tree distro-parts scripts. This is now maintained as lp:ubuntu-archive-publishing for Ubuntu, and should similarly be maintained separately for derived distributions.

== Summary ==

The distro-parts tree was apparently always supposed to be maintained by Ubuntu Engineering. I've now ripped it out into lp:ubuntu-archive-publishing, deployed that in /srv/launchpad.net/publisher-parts/ on relevant machines, William Grant has changed production configs to point to that, and I've changed dogfood configs likewise; so it's time to remove this from the Launchpad tree.

== Pre-implementation notes ==

I started out wanting to make changes to 10-sign-releases for bug 1002589; Julian Edwards and William Grant persuaded me that the best way to go about this would be to rip it out of the LP tree first.

== Implementation details ==

The test for relative run_parts_location becomes non-obvious after this removal, since it no longer has an in-tree copy to point to. I opted to just drop support for anything other than absolute paths, since I think relative paths will no longer be necessary. Shout if you disagree.

== Tests ==

bin/test -vvct test_publish_ftpmaster

== Demo and Q/A ==

Dogfood's configs are already updated and I've tested that the publisher still works without this change and uses the new run-parts directories; so I think it'd be sufficient to just give it another quick run there and make sure it doesn't fall over and that it still runs scripts from publish-distro.d and finalize.d.

[Steve Kowalik (stevenk)]

This looks like excellent work, thanks.

review: Approve (code)

=== removed directory 'cronscripts/publishing/distro-parts'
=== removed directory 'cronscripts/publishing/distro-parts/ubuntu'
=== removed directory 'cronscripts/publishing/distro-parts/ubuntu/finalize.d'
=== removed file 'cronscripts/publishing/distro-parts/ubuntu/finalize.d/10-germinate'
--- cronscripts/publishing/distro-parts/ubuntu/finalize.d/10-germinate	2011-03-28 09:18:42 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/finalize.d/10-germinate	1970-01-01 00:00:00 +0000
@@ -1,6 +0,0 @@
-#!/bin/sh
-
-if [ "$SECURITY_UPLOAD_ONLY" != "yes" ]
-then
-    cron.germinate || /bin/true
-fi

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/finalize.d/40-timestamp-trace-file'
--- cronscripts/publishing/distro-parts/ubuntu/finalize.d/40-timestamp-trace-file	2011-04-07 09:51:36 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/finalize.d/40-timestamp-trace-file	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-#! /bin/sh
-#
-# Timestamp the trace file, so we can track when the last archive publisher
-# run took place.
-for DIR in $ARCHIVEROOTS
-do
-    mkdir -p -- "$DIR/project/trace"
-    date -u > "$DIR/project/trace/$(hostname --fqdn)"
-done

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/finalize.d/90-trigger-mirrors'
--- cronscripts/publishing/distro-parts/ubuntu/finalize.d/90-trigger-mirrors	2012-04-26 00:12:52 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/finalize.d/90-trigger-mirrors	1970-01-01 00:00:00 +0000
@@ -1,21 +0,0 @@
-#!/bin/sh
-
-# Prod the master mirrors to propagate the update.
-
-MASTERMIRRORS="archive.syncproxy.ubuntu.com wahoo"
-MASTERMIRRORS_ASYNC="scandium sadalbari kudan"
-echo "$(date -R): Triggering master mirrors..."
-
-for HOST in $MASTERMIRRORS
-do
-    echo "$(date -R): Triggering $HOST:"
-    ssh archvsync@$HOST
-done
-
-for HOST in $MASTERMIRRORS_ASYNC
-do
-    echo "$(date -R): Triggering $HOST in background:"
-    ssh archvsync@$HOST &
-done
-
-echo "$(date -R): Master mirror triggers completed."

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/finalize.d/README.txt'
--- cronscripts/publishing/distro-parts/ubuntu/finalize.d/README.txt	2011-03-31 11:25:53 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/finalize.d/README.txt	1970-01-01 00:00:00 +0000
@@ -1,22 +0,0 @@
-Put scripts here that should be run at completion of the publish-ftpmaster
-script.  They will be executed through the run-parts command, in alphabetical
-order.
-
-The scripts' filenames must consist entirely of ASCII letters (both upper and
-lower case allowed), digits, underscores, and hyphens.  All other files,
-including this text file, are ignored.
-
-Publication happens in two passes: the first, expedited pass processes only
-security updates.  The second pass processes all packages.  The scripts in
-this directory will be run once for each pass, with the variable
-SECURITY_UPLOAD_ONLY set to indicate which pass is in progress; see below.
-
-The following variables will be set for the script:
-
-ARCHIVEROOTS - the list of root directories for the distribution's archives.
-(e.g. "/srv/ubuntu-archive/ubuntu/ /srv/ubuntu-archive/ubuntu-partner/" )
-
-SECURITY_UPLOAD_ONLY - "yes" during the security pass, or "no" otherwise.
-
-The script's PATH will be extended with the Launchpad source tree's
-cronscripts/publishing directory.

=== removed directory 'cronscripts/publishing/distro-parts/ubuntu/publish-distro.d'
=== removed file 'cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/10-sign-releases'
--- cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/10-sign-releases	2011-12-14 14:19:47 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/10-sign-releases	1970-01-01 00:00:00 +0000
@@ -1,30 +0,0 @@
-#!/bin/sh -e
-
-# The calling script may set GNUPGHOME to a value set up by Launchpad's
-# script machinery.  In production, we have a dedicated directory set up
-# for this.
-GNUPGHOME=/srv/launchpad.net/ubuntu-archive/gnupg-home
-
-if ! test -d "$GNUPGHOME"
-then
-	echo "There is no $GNUPGHOME; not signing Release files."
-	exit 0
-fi
-
-RELEASE_FILES=`find $DISTSROOT -maxdepth 2 -name Release`
-DIST_UPGRADER_TARBALLS=`
-	find $DISTSROOT/*/*/dist-upgrader* -name "*.tar.gz" || true`
-INSTALLER_CHECKSUMS=`
-	find $DISTSROOT/*/*/installer-* -name "*SUMS" || true`
-
-for CANDIDATE in $RELEASE_FILES $DIST_UPGRADER_TARBALLS $INSTALLER_CHECKSUMS
-do
-    if [ ! -f "$CANDIDATE.gpg" ] || [ "$CANDIDATE" -nt "$CANDIDATE.gpg" ]
-    then
-        echo "$(date -R): (re-)signing $CANDIDATE"
-        gpg --yes --detach-sign --armor -o "$CANDIDATE.gpg" \
-            --sign "$CANDIDATE"
-    else
-        echo "$(date -R): Not re-signing $CANDIDATE"
-    fi
-done

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/20-remove-uncompressed-listings'
--- cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/20-remove-uncompressed-listings	2011-04-12 07:13:08 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/20-remove-uncompressed-listings	1970-01-01 00:00:00 +0000
@@ -1,11 +0,0 @@
-#!/bin/sh
-#
-# Remove uncompressed Packages and Sources files.
-#
-# The uncompressed versions of these files are very large and could cause
-# problems for our mirrors, so get rid of them.
-#
-# It's safe to do this since the uncompressed MD5 hashes have already been
-# computed for inclusion in the Release files.
-
-find $DISTSROOT \( -name Packages -o -name Sources \) -exec rm -f -- "{}" \;

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/30-copy-indices'
--- cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/30-copy-indices	2011-04-08 18:59:20 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/30-copy-indices	1970-01-01 00:00:00 +0000
@@ -1,12 +0,0 @@
-#!/bin/sh -e
-
-if [ -n "$OVERRIDEROOT" ]
-then
-    echo "$(date -R): Copying the indices into place."
-
-    INDICES=$ARCHIVEROOT/indices
-
-    mkdir -p -- $INDICES
-    rm -f -- $INDICES/override
-    cp -- $OVERRIDEROOT/override.* $INDICES/
-fi

=== removed file 'cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/README.txt'
--- cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/README.txt	2011-04-12 07:13:08 +0000
+++ cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/README.txt	1970-01-01 00:00:00 +0000
@@ -1,25 +0,0 @@
-Put scripts here that should be run after publish-ftpmaster executes
-publish-distro.  They will be executed through the run-parts command, in
-alphabetical order.
-
-The scripts' filenames must consist entirely of ASCII letters (both upper and
-lower case allowed), digits, underscores, and hyphens.  All other files,
-including this text file, are ignored.
-
-Scripts in this directory will be run separately for each distro archive,
-possibly twice because publication happens in two passes: an expedited one for
-just the security uploads and then a second, slower pass for all packages.
-
-Some variables will be set before each script is run:
-
-ARCHIVEROOT - the archive's root directory
-(e.g. /srv/launchpad.net/ubuntu-archive/ubuntu/ )
-
-DISTSROOT - a working copy of the archive's dists root directory
-(e.g. /srv/launchpad.net/ubuntu-archive/ubuntu/dists.new )
-
-OVERRIDEROOT - the archive's overrides root directory (primary archive only)
-(e.g. /srv/launchpad.net/ubuntu-overrides, or the empty string for partner)
-
-The script's PATH will be extended with the Launchpad source tree's
-cronscripts/publishing directory.

=== modified file 'lib/lp/archivepublisher/scripts/publish_ftpmaster.py'
--- lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2012-04-16 15:28:47 +0000
+++ lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2012-05-23 00:36:20 +0000
@@ -131,14 +131,7 @@
     if not run_parts_location:
         return
 
-    if run_parts_location.startswith("/"):
-        # Absolute path.
-        base_dir = run_parts_location
-    else:
-        # Relative path.
-        base_dir = os.path.join(config.root, run_parts_location)
-
-    parts_dir = os.path.join(base_dir, distro.name, parts)
+    parts_dir = os.path.join(run_parts_location, distro.name, parts)
     if file_exists(parts_dir):
         return parts_dir
     else:

=== modified file 'lib/lp/archivepublisher/tests/test_publish_ftpmaster.py'
--- lib/lp/archivepublisher/tests/test_publish_ftpmaster.py	2012-04-16 15:28:47 +0000
+++ lib/lp/archivepublisher/tests/test_publish_ftpmaster.py	2012-05-23 00:36:20 +0000
@@ -88,11 +88,6 @@
     return get_archive_root(pub_config) + "-distscopy"
 
 
-def get_run_parts_path():
-    """Get relative path to run-parts location the Launchpad source."""
-    return os.path.join("cronscripts", "publishing", "distro-parts")
-
-
 def write_marker_file(path, contents):
     """Write a marker file for checking directory movements.
 
@@ -133,12 +128,15 @@
     def enableRunParts(self, parts_directory=None):
         """Set up for run-parts execution.
 
-        :param parts_directory: Base location for the run-parts
-            directories.  If omitted, the run-parts directory from the
-            Launchpad source tree will be used.
+        :param parts_directory: Base location for the run-parts directories.
+            If omitted, a temporary directory will be used.
         """
         if parts_directory is None:
-            parts_directory = get_run_parts_path()
+            parts_directory = self.makeTemporaryDirectory()
+            os.makedirs(os.path.join(
+                parts_directory, "ubuntu", "publish-distro.d"))
+            os.makedirs(os.path.join(parts_directory, "ubuntu", "finalize.d"))
+        self.parts_directory = parts_directory
 
         config.push("run-parts", dedent("""\
             [archivepublisher]
@@ -213,22 +211,14 @@
 class TestFindRunPartsDir(TestCaseWithFactory, HelpersMixin):
     layer = ZopelessDatabaseLayer
 
-    def test_find_run_parts_dir_finds_relative_runparts_directory(self):
+    def test_find_run_parts_dir_finds_runparts_directory(self):
         self.enableRunParts()
         ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
         self.assertEqual(
             os.path.join(
-                config.root, get_run_parts_path(), "ubuntu", "finalize.d"),
+                config.root, self.parts_directory, "ubuntu", "finalize.d"),
             find_run_parts_dir(ubuntu, "finalize.d"))
 
-    def test_find_run_parts_dir_finds_absolute_runparts_directory(self):
-        self.enableRunParts(os.path.join(config.root, get_run_parts_path()))
-        ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
-        self.assertEqual(
-            os.path.join(
-                config.root, get_run_parts_path(), "ubuntu", "finalize.d"),
-                find_run_parts_dir(ubuntu, "finalize.d"))
-
     def test_find_run_parts_dir_ignores_blank_config(self):
         self.enableRunParts("")
         ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
@@ -609,7 +599,7 @@
         command_line, = args
         self.assertIn("run-parts", command_line)
         self.assertIn(
-            "cronscripts/publishing/distro-parts/ubuntu/finalize.d",
+            os.path.join(self.parts_directory, "ubuntu/finalize.d"),
             command_line)
 
     def test_runParts_passes_parameters(self):