Merge lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-cron-job into lp:launchpad

[Julian Edwards]

Cron job bash script glue to publish 'copy' archives.

[Julian Edwards (julian-edwards)]

= Summary =
New bash cron script to publish COPY archives

== Proposed fix ==
This is some glue to pull in the various bits necessary to run the publisher.
It's largely a copy of the existing cron.publish-ftpmaster but with some un-
needed stuff removed.

Yeah, it's bash. It blows. No, there's no tests.

[Michael Nelson (michael.nelson)]

Hi Julian,

When I first looked at this, I was wondering why you don't define DISTRONAME earlier and then do:

ARCHIVEROOT=/srv/launchpad.net/DISTRONAME-archive/DISTRONAME

etc., but I see that it's the same in cron.publish-ftpmaster.

Also, another thing that's been inherited from the ftpmaster - the comment at the top about LPCONFIG coming from the environment so we can run it unaltered on dogfood, but then the check before copying indices that "$LPCONFIG" = "$PRODUCTION_CONFIG"... wouldn't we want to copy the indices on df?

I'm assuming this is what you've been testing like crazy on df yesterday...

15:00 < noodles> bigjools: this bash script - is it what you've been testing on df yesterday?
15:01 < bigjools> and today, yes

Great.

review: Approve (code)

=== added file 'cronscripts/publishing/cron.publish-copy-archives'
--- cronscripts/publishing/cron.publish-copy-archives	1970-01-01 00:00:00 +0000
+++ cronscripts/publishing/cron.publish-copy-archives	2010-02-25 12:10:37 +0000
@@ -0,0 +1,84 @@
+#!/bin/sh
+#
+# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+# LPCONFIG will come from the environment so this script can run unaltered
+# on dogfood.
+if [ -z $LPCONFIG ]; then
+    echo LPCONFIG must be set to run this script.
+    exit 1
+fi
+
+set -x
+set -e
+set -u
+
+#
+# This script publishes the COPY (rebuild) archvies *only*.
+#
+
+
+# Informational -- this *MUST* match the database.
+ARCHIVEROOT=/srv/launchpad.net/ubuntu-archive/ubuntu
+DISTSROOT=$ARCHIVEROOT/dists
+OVERRIDEROOT=$ARCHIVEROOT/../ubuntu-overrides
+INDICES=$ARCHIVEROOT/indices
+PRODUCTION_CONFIG=ftpmaster
+
+if [ "$LPCONFIG" = "$PRODUCTION_CONFIG" ]; then
+    GNUPGHOME=/srv/launchpad.net/ubuntu-archive/gnupg-home
+else
+    echo GPG keys will come from ~/.gnupg
+    # GNUPGHOME does not need to be set, keys can come from ~/.gnupg.
+fi
+
+# Configuration options.
+LAUNCHPADROOT=/srv/launchpad.net/codelines/current
+LOCKFILE=/srv/launchpad.net/ubuntu-archive/cron.daily.lock
+DISTRONAME=ubuntu
+TOUCHLIST=$ARCHIVEROOT/project/trace/$(hostname --fqdn)
+
+# Manipulate the environment.
+export GNUPGHOME
+PATH=$PATH:$LAUNCHPADROOT/scripts:$LAUNCHPADROOT/cronscripts:$LAUNCHPADROOT/cronscripts/publishing:$LAUNCHPADROOT/scripts/ftpmaster-tools
+
+# Claim the lock.
+if ! lockfile -r1 $LOCKFILE; then
+  echo "Could not claim lock file."
+  exit 1
+fi
+
+# Lock claimed.
+
+cleanup () {
+  echo "Cleaning up lockfile."
+  rm -f $LOCKFILE
+}
+
+trap cleanup EXIT
+
+# Process the accepted queue into the publishing records.
+process-accepted.py --copy-archive -v -v -v $DISTRONAME
+
+# Publish the packages to disk.
+publish-distro.py -v -v --copy-archive -d $DISTRONAME
+
+set +x
+
+echo Removing uncompressed Packages and Sources files
+find ${DISTSROOT} \( -name "Packages" -o -name "Sources" \) -exec rm "{}" \;
+
+# Copy in the indices.
+if [ "$LPCONFIG" = "$PRODUCTION_CONFIG" ]; then
+    echo Copying the indices into place.
+    rm -f $INDICES/override.*
+    cp $OVERRIDEROOT/override.* $INDICES
+fi
+
+# Touch everything you asked us to do.
+if [ "$LPCONFIG" = "$PRODUCTION_CONFIG" ]; then
+    for FILE in $TOUCHLIST; do
+      touch "$FILE"
+    done
+fi