Merge lp:~cjwatson/launchpad/634831-germinate-unhardcode into lp:launchpad

[Colin Watson]

[r=jelmer][ui=none][bug=634831] cron.germinate: Avoid relying on implementation details of germinate: use -m option rather than writing to archive.ubuntu.com_* files

Summary

Bug 634831 asks for a change in germinate to make it possible to process input from multiple PPAs. This change is blocked because Launchpad's cron.germinate script relies on an internal implementation detail of germinate which now needs to change (perhaps not in exactly the way suggested in the patch for that bug, but it clearly needs to change somehow).

Proposed fix

The reason for this code in cron.germinate is to take copies of input files in case the archive changes under our feet (e.g. cron.germinate overlaps the next publisher run, or something like that). Currently, it just inserts them into germinate's current cached locations for archive.ubuntu.com URLs. Instead, it would be better to construct a local dists tree and point germinate at that using a file:// URL.

Pre-implementation notes

I haven't discussed this bug with anyone, sorry. This is partly because everyone keeps telling me the platform team owns cron.germinate. :-)

Implementation details

Nothing of note. I think this should be slightly faster than the previous implementation as it only decompresses Packages files once rather than once for each of the six seed collections (ubuntu kubuntu edubuntu xubuntu mythbuntu netbook).

Tests

cron.germinate has no proper tests, but it only generates a couple of output files and this patch shouldn't change them, so it's easy to confirm that. I tested this on mawson by taking a copy of the output directory (/srv/launchpad.net/ubuntu-archive/ubuntu-misc), running the new cron.germinate script (with no arguments), and then running 'diff -ru' on the old and new output directories to confirm that the output was identical.

Demo and Q/A

See the "Tests" section.

[Jelmer Vernooij (jelmer)]

review: Approve (code)

=== modified file 'cronscripts/publishing/cron.germinate'
--- cronscripts/publishing/cron.germinate	2010-09-06 09:46:01 +0000
+++ cronscripts/publishing/cron.germinate	2010-11-02 16:14:46 +0000
@@ -37,10 +37,21 @@
 # Clean up temporary files
 rm -f germinate.output ALL ALL.sources UBUNTU-* KUBUNTU-* EDUBUNTU-* XUBUNTU-* MYTHBUNTU-* NETBOOK-*
 rm -f all_* all.sources_*
+rm -rf dists
 
-# Grab a local copy of Sources files
+# Grab local copies of Sources and Packages files, to avoid problems in case
+# the archive changes under our feet.
 for component in main universe restricted multiverse; do
-  zcat $ARCHIVEROOT/dists/"$suite"/"$component"/source/Sources.gz > archive.ubuntu.com_"$suite"_"$component"_Sources;
+  base="dists/$suite/$component"
+  mkdir -p "$base/source"
+  zcat "$ARCHIVEROOT/$base/source/Sources.gz" > "$base/source/Sources"
+  for arch in i386 amd64 powerpc armel; do
+    mkdir -p "$base/binary-$arch" "$base/debian-installer/binary-$arch"
+    zcat "$ARCHIVEROOT/$base/binary-$arch/Packages.gz" \
+      > "$base/binary-$arch/Packages"
+    zcat "$ARCHIVEROOT/$base/debian-installer/binary-$arch/Packages.gz" \
+      > "$base/debian-installer/binary-$arch/Packages"
+  done
 done
 
 > "$MISCROOT/more-extra.override.$suite.main.new"
@@ -50,15 +61,9 @@
   DISTRO="$(echo $distro | tr a-z A-Z)"
   germinate_suite="$distro.$suite"
   for arch in i386 amd64 powerpc armel; do
-    # Grab local copy of Packages and InstallerPackages files
-    for component in main universe restricted multiverse; do
-      zcat $ARCHIVEROOT/dists/"$suite"/"$component"/binary-$arch/Packages.gz > archive.ubuntu.com_"$suite"_"$component"_Packages
-      zcat $ARCHIVEROOT/dists/"$suite"/"$component"/debian-installer/binary-$arch/Packages.gz > archive.ubuntu.com_"$suite"_"$component"_InstallerPackages
-    done
-
     # Run germinate
     echo " **************** $distro/$suite/$arch ********************* " >> germinate.output
-    germinate --no-rdepends -s "$germinate_suite" -d "$suite" -c "$germinate_components" -a $arch >> germinate.output 2>&1
+    germinate --no-rdepends -m "file://$(pwd)/" -s "$germinate_suite" -d "$suite" -c "$germinate_components" -a $arch >> germinate.output 2>&1
 
     # The structure file is generally useful; keep per distro/suite/arch
     # copies for convenience