Merge lp:~free.ekanayaka/landscape-client/jaunty.fix-347983 into lp:ubuntu/jaunty/landscape-client

[Free Ekanayaka]

[Free Ekanayaka (free.ekanayaka)]

SRU for landscape-client version 1.3.2.3-0ubuntu0.9.04.0

[Mathias Gug (mathiaz)]

Looks good to me.

review: Approve

=== modified file 'Makefile'
--- Makefile	2008-09-08 16:35:57 +0000
+++ Makefile	2009-09-21 16:05:18 +0000
@@ -15,8 +15,13 @@
 	-rm docs/api -rf;
 	-rm man/\*.1 -rf
 
-doc:
-	${PYDOCTOR} --make-html --html-output docs/api --add-package landscape
+doc: docs/api/twisted/pickle
+	mkdir -p docs/api
+	${PYDOCTOR} --make-html --html-output docs/api --add-package landscape --extra-system=docs/api/twisted/pickle:twisted/
+
+docs/api/twisted/pickle:
+	mkdir -p docs/api/twisted
+	-${PYDOCTOR} --make-html --html-output docs/api/twisted --add-package /usr/share/pyshared/twisted -o docs/api/twisted/pickle
 
 manpages:
 	${TXT2MAN} -P Landscape -t landscape-client < man/landscape-client.txt > man/landscape-client.1

=== modified file 'debian/README.source'
--- debian/README.source	2009-04-09 17:09:50 +0000
+++ debian/README.source	2009-09-21 16:12:27 +0000
@@ -7,6 +7,9 @@
 
   1.0.29-0ubuntu0.9.04.1
 
-In addition, when you build a new package, it would be appreciated if you also
-updated landscape/__init__.py to include the entire new version number. This
-helps us keep track of exact version of clients in use.
+In addition, when you build a package for a new client version, it would be
+appreciated if you also updated the UPSTREAM_VERSION variable in
+landscape/__init__.py to include the entire new client version number. This
+helps us keep track of exact version of clients in use. There's no need to
+update the DEBIAN_REVISION variable, as it gets automatically set at build
+time. 

=== modified file 'debian/changelog'
--- debian/changelog	2009-04-13 14:33:31 +0000
+++ debian/changelog	2009-09-21 16:12:27 +0000
@@ -1,3 +1,32 @@
+landscape-client (1.3.2.3-0ubuntu0.9.04.0) jaunty-proposed; urgency=low
+
+  * New upstream release (LP: #347983):
+    - Don't clear the hash_id_requests table upon resynchronize (LP #417122)
+    - Include the README file in landscape-client (LP: #396260)
+    - Fix client capturing stderr from run_command when constructing
+      hash-id-databases url (LP: #397480)
+    - Use substvars to conditionally depend on update-motd or
+      libpam-modules (LP: #393454)
+    - Fix reporting wrong version to the server (LP: #391225)
+    - The init script does not wait for the network to be available
+      before checking for EC2 user data (LP: #383336)
+    - When the broker is restarted by the watchdog, the state of the client
+      is inconsistent (LP: #380633)
+    - Package stays unknown forever in the client with hash-id-databases
+      support (LP: #381356)
+    - Standard error not captured when calling smart-update (LP: #387441)
+    - Changer calls reporter without switching groups, just user (LP: #388092)
+    - Run smart update in the package-reporter instead of having a cronjob (LP: #362355)
+    - Package changer does not inherit proxy settings (LP: #381241)
+    - The ./test script doesn't work in landscape-client (LP: #381613)
+    - The source package should build on all supported releases (LP: #385098)
+    - Strip smart update's output (LP: #387331)
+    - The fetch() timeout isn't based on activity (#389224)
+    - Client can use a UUID of "None" when fetching the hash-id-database (LP: #381291)
+    - Registration should use the fqdn rather than just the hostname (LP: #385730)
+
+ -- Free Ekanayaka <[e-mail]>  Mon, 21 Sep 2009 17:59:31 +0200
+
 landscape-client (1.0.29.1-0ubuntu0.9.04.0) jaunty; urgency=low
 
   * Apply a fix for segfault bug involving curl timeouts. (LP: #360510)

=== added file 'debian/cloud-default.conf'
--- debian/cloud-default.conf	1970-01-01 00:00:00 +0000
+++ debian/cloud-default.conf	2009-09-21 16:12:27 +0000
@@ -0,0 +1,7 @@
+[client]
+cloud = True
+url = https://landscape.canonical.com/message-system
+data_path = /var/lib/landscape/client
+ping_url = http://landscape.canonical.com/ping
+include_manager_plugins = ScriptExecution
+script_users = ALL

=== modified file 'debian/compat'
--- debian/compat	2008-09-08 16:35:57 +0000
+++ debian/compat	2009-09-21 16:12:27 +0000
@@ -1,1 +1,1 @@
-6
+5

=== modified file 'debian/control'
--- debian/control	2008-10-17 12:42:23 +0000
+++ debian/control	2009-09-21 16:12:27 +0000
@@ -1,34 +1,46 @@
 Source: landscape-client
 Section: admin
 Priority: optional
-Maintainer: Ubuntu Core Developers <[e-mail]>
-Build-Depends: debhelper (>= 6.0.7~), po-debconf
-Build-Depends-Indep: python-dev, python-central, lsb-release, gawk
+Maintainer: Ubuntu Developers <[e-mail]>
+XSBC-Original-Maintainer: Landscape Team <[e-mail]>
+Build-Depends: debhelper (>= 5), po-debconf, python-dev, python-central | python-support, lsb-release, gawk
 Standards-Version: 3.8.0
-Vcs-Bzr: http://bazaar.launchpad.net/~ubuntu-core-dev/landscape-client/ubuntu
-XS-Python-Version: >= 2.4
+XS-Python-Version: >= 2.4, << 2.7
 
 Package: landscape-common
 Architecture: all
 Pre-Depends: python, python-gobject
-Depends: ${python:Depends}, ${misc:Depends}, python-twisted-core, python-dbus, lsb-release, lsb-base, perl-modules,  python-gdbm,  ca-certificates, adduser, update-motd
-XB-Python-Version: ${python:Versions}
-Replaces: landscape-client (< 1.0.23-0ubuntu0.8.10)
+Depends: ${python:Depends}, ${misc:Depends}, ${extra:Depends},
+         python-smartpm (>= 1.2-0ubuntu1),
+         python-dbus | python2.4-dbus,
+         python-twisted-core,
+         ca-certificates,
+         perl-modules,
+         python-gdbm,
+         lsb-release,
+         lsb-base,
+         adduser
+Replaces: landscape-client (<= 1.0.23-0ubuntu0.8.10)
 Description: The Landscape administration system client
  Landscape is a web-based tool for managing Ubuntu systems. This
  package is necessary if you want your machine to be managed in a
  Landscape account.
  .
  This package provides the core libraries.
- 
+XB-Python-Version: ${python:Versions}
+
 Package: landscape-client
-Architecture: all
+Architecture: any
 Pre-Depends: python
-Depends: ${python:Depends}, ${misc:Depends}, landscape-common, hal, python-twisted-web, python-smartpm, python-pycurl, cron
-XB-Python-Version: ${python:Versions}
+Depends: ${python:Depends}, ${misc:Depends},
+         python-pycurl | python2.4-pycurl,
+         python-twisted-web,
+         landscape-common (>= ${Source-Version}),
+         hal
 Description: The Landscape administration system client
  Landscape is a web-based tool for managing Ubuntu systems. This
  package is necessary if you want your machine to be managed in a
  Landscape account.
  .
  This package provides the Landscape client and requires a Landscape account.
+XB-Python-Version: ${python:Versions}

=== removed file 'debian/landscape-client.cron.hourly'
--- debian/landscape-client.cron.hourly	2008-09-23 18:19:26 +0000
+++ debian/landscape-client.cron.hourly	1970-01-01 00:00:00 +0000
@@ -1,12 +0,0 @@
-#!/bin/sh
-
-[ -f "/etc/default/landscape-client" ] || exit 0
-
-. /etc/default/landscape-client
-
-if [ "$RUN" -eq "1" ] 
-then
-	[ -x /usr/share/smart/smart ] && /usr/share/smart/smart update > /dev/null 2>&1
-fi
-
-exit 0

=== added file 'debian/landscape-client.docs'
--- debian/landscape-client.docs	1970-01-01 00:00:00 +0000
+++ debian/landscape-client.docs	2009-09-21 16:12:27 +0000
@@ -0,0 +1,1 @@
+README

=== modified file 'debian/landscape-client.init'
--- debian/landscape-client.init	2009-03-18 20:42:05 +0000
+++ debian/landscape-client.init	2009-09-21 16:12:27 +0000
@@ -17,15 +17,32 @@
 PIDDIR=/var/run/landscape
 PIDFILE=$PIDDIR/$NAME.pid
 RUN=0 # overridden in /etc/default/landscape-client
+CLOUD=0 # overridden in /etc/default/landscape-client
+DAEMON_GROUP=landscape
 
 [ -f $DAEMON ] || exit 0
 
 . /lib/lsb/init-functions
 [ -f $LANDSCAPE_DEFAULTS ] && . $LANDSCAPE_DEFAULTS
 
+# This $RUN check should match the semantics of
+# l.sysvconfig.SysVConfig.is_configured_to_run.
 if [ $RUN -eq 0 ]; then
-	echo "$NAME is not configured, please run landscape-config."
-	exit 0
+	if [ $CLOUD -eq 1 ]; then
+		if landscape-is-cloud-managed; then
+			# Install the cloud default configuration file
+			cp /usr/share/landscape/cloud-default.conf /etc/landscape/client.conf
+			# Override default file for not going in this conditional again at
+			# next startup
+			echo "RUN=1" > $LANDSCAPE_DEFAULTS
+		else
+			echo "$NAME is not configured, please run landscape-config."
+			exit 0
+		fi
+	else
+		echo "$NAME is not configured, please run landscape-config."
+		exit 0
+	fi
 fi
 
 case "$1" in
@@ -35,11 +52,11 @@
 			mkdir $PIDDIR
 			chown landscape $PIDDIR
 		fi
-		FULL_COMMAND="start-stop-daemon --start --quiet --oknodo --startas $DAEMON --pidfile $PIDFILE -- --daemon --pid-file $PIDFILE"
+		FULL_COMMAND="start-stop-daemon --start --quiet --oknodo --startas $DAEMON --pidfile $PIDFILE -g $DAEMON_GROUP -- --daemon --pid-file $PIDFILE"
 		if [ x"$DAEMON_USER" != x ]; then
-		    sudo -u $DAEMON_USER $FULL_COMMAND
+			sudo -u $DAEMON_USER $FULL_COMMAND
 		else
-		    $FULL_COMMAND
+			$FULL_COMMAND
 		fi
 		log_end_msg $?
 		;;

=== modified file 'debian/landscape-client.install'
--- debian/landscape-client.install	2008-09-15 17:21:53 +0000
+++ debian/landscape-client.install	2009-09-21 16:12:27 +0000
@@ -6,4 +6,7 @@
 usr/bin/landscape-monitor
 usr/bin/landscape-package-changer
 usr/bin/landscape-package-reporter
+usr/bin/landscape-is-cloud-managed
 etc/dbus-1/system.d/landscape.conf
+usr/share/landscape/cloud-default.conf
+usr/lib/landscape

=== modified file 'debian/landscape-client.postinst'
--- debian/landscape-client.postinst	2008-09-25 17:54:00 +0000
+++ debian/landscape-client.postinst	2009-09-21 16:12:27 +0000
@@ -76,6 +76,23 @@
             echo "Old monolithic client data file migrated to new format."
         fi
 
+        # Add the setuid flag to smart-update and it be executable by users in
+        # the landscape group (that normally means landscape itself)
+        smart_update=/usr/lib/landscape/smart-update
+        if ! dpkg-statoverride --list $smart_update >/dev/null 2>&1; then
+            dpkg-statoverride --update --add root landscape 4754 $smart_update
+        fi
+
+        # Remove old cron jobs
+        old_cron_job=/etc/cron.hourly/landscape-client
+        if [ -e $old_cron_job ]; then
+            rm $old_cron_job
+        fi
+        very_old_cron_job=/etc/cron.hourly/smartpm-core
+        if [ -e $very_old_cron_job ]; then
+            rm $very_old_cron_job
+        fi
+
     ;;
 
     abort-upgrade|abort-remove|abort-deconfigure)

=== modified file 'debian/landscape-client.prerm'
--- debian/landscape-client.prerm	2008-09-15 17:21:53 +0000
+++ debian/landscape-client.prerm	2009-09-21 16:12:27 +0000
@@ -19,6 +19,13 @@
 
 case "$1" in
     remove|upgrade|deconfigure)
+
+        # Remove statoverride for smart-update
+        smart_update=/usr/lib/landscape/smart-update
+        if dpkg-statoverride --list $smart_update >/dev/null 2>&1; then
+            dpkg-statoverride --remove $smart_update
+        fi
+
     ;;
 
     failed-upgrade)

=== added file 'debian/landscape-common.dirs'
--- debian/landscape-common.dirs	1970-01-01 00:00:00 +0000
+++ debian/landscape-common.dirs	2009-09-21 16:12:27 +0000
@@ -0,0 +1,1 @@
+etc/update-motd.d

=== modified file 'debian/landscape-common.install'
--- debian/landscape-common.install	2009-02-25 12:03:23 +0000
+++ debian/landscape-common.install	2009-09-21 16:12:27 +0000
@@ -1,3 +1,3 @@
-usr/lib/python*/*-packages/*
+usr/lib/python*
 usr/bin/landscape-sysinfo
 usr/share/landscape/landscape-sysinfo.wrapper

=== modified file 'debian/landscape-common.postinst'
--- debian/landscape-common.postinst	2008-12-11 17:11:08 +0000
+++ debian/landscape-common.postinst	2009-09-21 16:12:27 +0000
@@ -26,28 +26,28 @@
 case "$1" in
     configure)
 
-        db_get $PACKAGE/sysinfo
-        # Choices:
-        #  * Do not display sysinfo on login
-        #  * Cache sysinfo in /etc/motd
-        #  * Run sysinfo on every login
-        SYSINFO="${RET:-Cache sysinfo in /etc/motd}"
-        WRAPPER=/usr/share/landscape/landscape-sysinfo.wrapper
-        PROFILE_LOCATION=/etc/profile.d/50-landscape-sysinfo.sh
-        UPDATE_MOTD_LOCATION=/etc/update-motd.d/50-landscape-sysinfo
-        if [ "$RET" = "Cache sysinfo in /etc/motd" ]; then
-            rm -f $PROFILE_LOCATION 2>/dev/null || true
-            ln -sf $WRAPPER $UPDATE_MOTD_LOCATION
-            /usr/sbin/update-motd 2>/dev/null || true
-        elif [ "$RET" = "Run sysinfo on every login" ]; then
-            rm -f $UPDATE_MOTD_LOCATION 2>/dev/null || true
-            /usr/sbin/update-motd 2>/dev/null || true
-            ln -sf $WRAPPER $PROFILE_LOCATION
-        else
-            rm -f $UPDATE_MOTD_LOCATION 2>/dev/null || true
-            /usr/sbin/update-motd 2>/dev/null || true
-            rm -f $PROFILE_LOCATION || true
-        fi
+       db_get $PACKAGE/sysinfo
+       # Choices:
+       #  * Do not display sysinfo on login
+       #  * Cache sysinfo in /etc/motd
+       #  * Run sysinfo on every login
+       SYSINFO="${RET:-Cache sysinfo in /etc/motd}"
+       WRAPPER=/usr/share/landscape/landscape-sysinfo.wrapper
+       PROFILE_LOCATION=/etc/profile.d/50-landscape-sysinfo.sh
+       UPDATE_MOTD_LOCATION=/etc/update-motd.d/50-landscape-sysinfo
+       if [ "$RET" = "Cache sysinfo in /etc/motd" ]; then
+           rm -f $PROFILE_LOCATION 2>/dev/null || true
+           ln -sf $WRAPPER $UPDATE_MOTD_LOCATION
+           /usr/sbin/update-motd 2>/dev/null || true
+       elif [ "$RET" = "Run sysinfo on every login" ]; then
+           rm -f $UPDATE_MOTD_LOCATION 2>/dev/null || true
+           /usr/sbin/update-motd 2>/dev/null || true
+           ln -sf $WRAPPER $PROFILE_LOCATION
+       else
+           rm -f $UPDATE_MOTD_LOCATION 2>/dev/null || true
+           /usr/sbin/update-motd 2>/dev/null || true
+           rm -f $PROFILE_LOCATION || true
+       fi
 
         # 0.9.1 introduces non-backwards compatible changes.  This detects
         # whether or not the data is in the current format.  If not, all
@@ -65,10 +65,18 @@
 
         # Create landscape system user
         if ! getent passwd landscape >/dev/null; then
-            adduser --quiet --system --disabled-password \
+            adduser --quiet --system --group --disabled-password \
                 --home /var/lib/landscape --no-create-home landscape
         fi
 
+        # Create landscape system group (for <= 1.0.29.1-0ubuntu0.9.04.0)
+        if ! getent group landscape >/dev/null; then
+            addgroup --quiet --system landscape
+        fi
+
+        # Ensure primary group is landscape (for <= 1.0.29.1-0ubuntu0.9.04.0)
+        usermod -g landscape landscape
+
         # Fix prior ownerships
         find /var/lib/landscape/ -wholename /var/lib/landscape/client/custom-graph-scripts -prune -or -exec chown landscape {} \;
         chown -R landscape /var/log/landscape

=== modified file 'debian/landscape-common.prerm'
--- debian/landscape-common.prerm	2008-09-18 16:47:08 +0000
+++ debian/landscape-common.prerm	2009-09-21 16:12:27 +0000
@@ -19,9 +19,9 @@
 
 case "$1" in
     remove|upgrade|deconfigure)
-        rm -f /etc/update-motd.d/50-landscape-sysinfo 2>/dev/null || true
-	/usr/sbin/update-motd 2>/dev/null || true
-        rm -f /etc/profile.d/landscape-sysinfo.sh 2>/dev/null || true
+       rm -f /etc/update-motd.d/50-landscape-sysinfo 2>/dev/null || true
+       /usr/sbin/update-motd 2>/dev/null || true
+       rm -f /etc/profile.d/landscape-sysinfo.sh 2>/dev/null || true
     ;;
 
     failed-upgrade)

=== modified file 'debian/rules'
--- debian/rules	2009-02-25 12:03:23 +0000
+++ debian/rules	2009-09-21 16:12:27 +0000
@@ -16,10 +16,16 @@
 package = landscape-client
 root_dir = debian/tmp/
 
+revision = $(shell dpkg-parsechangelog | grep ^Version | cut -f 2 -d " "| cut -f 2 -d "-")
+
+landscape_substvars = debian/landscape-common.substvars
+
 build: build-stamp
 build-stamp:
 	dh_testdir
+	sed -i -e "s/^DEBIAN_REVISION = \"\"/DEBIAN_REVISION = \"-$(revision)\"/g" landscape/__init__.py
 	python setup.py build
+	make -C smart-update
 	touch build-stamp
 
 clean:
@@ -27,8 +33,10 @@
 	dh_testroot
 	rm -f build-stamp
 	rm -rf build
+	make -C smart-update clean
 	dh_clean
 	debconf-updatepo
+	sed -i -e "s/^DEBIAN_REVISION = .*/DEBIAN_REVISION = \"\"/g" landscape/__init__.py
 
 install: build
 	dh_testdir
@@ -44,37 +52,48 @@
 
 	install -D -o root -g root -m 644 dbus/landscape.conf $(root_dir)/etc/dbus-1/system.d/landscape.conf
 	install -D -o root -g root -m 755 debian/landscape-sysinfo.wrapper $(root_dir)/usr/share/landscape/landscape-sysinfo.wrapper
-
-
-	dh_installcron
-
-binary-arch:
+	install -D -o root -g root -m 644 debian/cloud-default.conf $(root_dir)/usr/share/landscape/cloud-default.conf
+	install -D -o root -g root -m 755 smart-update/smart-update $(root_dir)/usr/lib/landscape/smart-update
+
+binary-indep:
 # do nothing
 #
-binary-indep: build install
+binary-arch: build install
 	# That's not present in Ubuntu releases we still support, so
 	# we're just installing the lintian overrides file by hand
 	# for now.
-	dh_lintian
+	#dh_lintian
 	dh_testdir
 	dh_testroot
 	dh_installdocs
 	dh_installman -p landscape-client man/landscape-client.1 man/landscape-config.1 man/landscape-message.1
 	dh_installchangelogs
 	dh_install --sourcedir debian/tmp/
-	dh_installinit -- start 25 2 3 4 5 . stop 15 1 .
+	dh_installinit -- start 45 2 3 4 5 . stop 15 1 .
 	dh_installlogrotate
 	dh_installdebconf
 	dh_compress
 	dh_fixperms
 
+ifneq (,$(findstring $(dist_release),"dapper"))
+	# We need python2.4-pysqlite2 and a non-buggy libcurl3-gnutls on dapper
+	echo "extra:Depends=python2.4-pysqlite2, libcurl3-gnutls (>= 7.15.1-1ubuntu3)" >> $(landscape_substvars)
+endif
+ifneq (,$(findstring $(dist_release),"intrepid jaunty"))
+	# We want update-motd in intrepid and jaunty
+	echo "extra:Depends=update-motd" >> $(landscape_substvars)
+endif
+ifneq (,$(findstring $(dist_release),"karmic"))
+	# We want libpam-modules in karmic
+	echo "extra:Depends=libpam-modules (>= 1.0.1-9ubuntu3)" >> $(landscape_substvars)
+endif
 
 ifeq ($(use_pycentral),yes)
-  ifneq (,$(py_setup_install_args))
+   ifneq (,$(py_setup_install_args))
 	DH_PYCENTRAL=include-links dh_pycentral
-  else
+   else
 	DH_PYCENTRAL=nomove dh_pycentral
-  endif
+   endif
 else
 	dh_python
 endif

=== modified file 'landscape/__init__.py'
--- landscape/__init__.py	2009-04-13 14:33:31 +0000
+++ landscape/__init__.py	2009-09-21 16:07:58 +0000
@@ -1,2 +1,4 @@
-VERSION = "AUTOPPA_VERSION(1.0.29.1-0ubuntu0.9.04.0)"[len("AUTOPPA_VERSION("):-1]
+DEBIAN_REVISION = ""
+UPSTREAM_VERSION = "1.3.2.1"
+VERSION = "%s%s" % (UPSTREAM_VERSION, DEBIAN_REVISION)
 API = "3.2"

=== modified file 'landscape/broker/broker.py'
--- landscape/broker/broker.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/broker.py	2009-09-21 16:05:18 +0000
@@ -19,7 +19,23 @@
 
 
 class BrokerDBusObject(Object):
-    """A DBus-published object which allows adding messages to the queue."""
+    """A DBus-published object exposing broker's services and emitting signals.
+
+    Each public method decorated with C{@method(IFACE_NAME)} exposes a
+    broker's service to the other Landscape client processes, which can
+    remote call it through the D-Bus interface.
+
+    Each public method decorated with C{@signal(IFACE_NAME)} turns the
+    corresponding broker's L{TwistedReactor} event into a D-Bus signal,
+    which gets broadcasted over the bus and eventually received by the
+    other Landscape client processes.
+
+    In order to receive D-Bus signals from the broker, the Landscape
+    client processes have to register themselves as L{BrokerPlugin}s
+    using the L{register_plugin} method. The L{BrokerPlugin.message}
+    method of each plugin will be called on every C{"message"} event
+    generated by the L{TwistedReactor}.
+    """
 
     bus_name = BUS_NAME
     object_path = OBJECT_PATH
@@ -27,9 +43,11 @@
     def __init__(self, config, reactor, exchange, registration,
                  message_store, bus):
         """
-        @param exchange: The
-            L{MessageExchange<landscape.exchange.MessageExchange>} to send
-            messages with.
+        @param config: The L{BrokerConfiguration} used by the broker.
+        @param reactor: The L{TwistedReactor} driving the broker's events.
+        @param exchange: The L{MessageExchange} to send messages with.
+        @param registration: The {RegistrationHandler}.
+        @param message_store: The broker's L{MessageStore}.
         @param bus: The L{Bus} that represents where we're listening.
         """
         super(BrokerDBusObject, self).__init__(bus)
@@ -53,6 +71,7 @@
             self.server_uuid_changed(old_uuid or "", new_uuid or "")
         reactor.call_on("server-uuid-changed", server_uuid_changed)
         reactor.call_on("resynchronize-clients", self.resynchronize)
+        self.broker_started()
 
     @signal(IFACE_NAME)
     def resynchronize(self):
@@ -67,6 +86,10 @@
         pass
 
     def _broadcast_message(self, message):
+        """Call the C{message} method of all the registered plugins.
+
+        @see: L{register_plugin}.
+        """
         blob = byte_array(dumps(message))
         results = []
         for plugin in self.get_plugin_objects():
@@ -148,6 +171,10 @@
     def registration_failed(self):
         pass
 
+    @signal(IFACE_NAME)
+    def broker_started(self):
+        pass
+
     @method(IFACE_NAME, out_signature="as")
     def get_accepted_message_types(self):
         return self.message_store.get_accepted_types()
@@ -172,6 +199,14 @@
 
     @method(IFACE_NAME)
     def register_plugin(self, bus_name, object_path):
+        """Register a new plugin.
+
+        Plugins are DBus-published objects identified by C{bus_name}
+        and C{object_path}.
+
+        Examples of plugins are L{MonitorDBusObject} and L{ManagerDBusObject}
+        from the monitor and manager Landscape services respectively.
+        """
         self._registered_plugins.add((bus_name, object_path))
 
     @method(IFACE_NAME)

=== modified file 'landscape/broker/deployment.py'
--- landscape/broker/deployment.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/deployment.py	2009-09-21 16:05:18 +0000
@@ -14,7 +14,10 @@
 
 
 class BrokerConfiguration(Configuration):
-    """Specialized configuration for the Landscape Broker."""
+    """Specialized configuration for the Landscape Broker.
+
+    @cvar required_options: C{["url"]}
+    """
 
     required_options = ["url"]
 
@@ -24,9 +27,21 @@
         self._original_https_proxy = os.environ.get("https_proxy")
 
     def make_parser(self):
-        """
-        Specialize L{Configuration.make_parser}, adding many
-        broker-specific options.
+        """Parser factory for broker-specific options.
+
+        @return: An L{OptionParser} preset for all the options
+            from L{Configuration.make_parser} plus:
+              - C{account_name}
+              - C{registration_password}
+              - C{computer_title}
+              - C{url}
+              - C{ssl_public_key}
+              - C{exchange_interval} (C{15*60})
+              - C{urgent_exchange_interval} (C{1*60})
+              - C{ping_url}
+              - C{http_proxy}
+              - C{https_proxy}
+              - C{cloud}
         """
         parser = super(BrokerConfiguration, self).make_parser()
 
@@ -63,13 +78,16 @@
 
     @property
     def message_store_path(self):
+        """Get the path to the message store."""
         return os.path.join(self.data_path, "messages")
 
     def load(self, args, accept_nonexistent_config=False):
         """
+        Load options from command line arguments and a config file.
+
         Load the configuration with L{Configuration.load}, and then set
-        http_proxy and https_proxy environment variables based on that config
-        data.
+        C{http_proxy} and C{https_proxy} environment variables based on
+        that config data.
         """
         super(BrokerConfiguration, self).load(
             args, accept_nonexistent_config=accept_nonexistent_config)
@@ -85,15 +103,34 @@
 
 
 class BrokerService(LandscapeService):
-    """
-    The core Twisted Service which creates and runs all necessary
-    components when started.
+    """The core C{Service} of the Landscape Broker C{Application}.
+
+    The Landscape broker service handles all the communication between the
+    client and server. When started it creates and runs all necessary components
+    to exchange messages with the Landscape server.
+
+    @ivar persist_filename: Path to broker-specific persisted data.
+    @ivar persist: A L{Persist} object saving and loading from
+        C{self.persist_filename}.
+    @ivar message_store: A L{MessageStore} used by the C{exchanger} to
+        queue outgoing messages.
+    @ivar transport: A L{HTTPTransport} used by the C{exchanger} to deliver messages.
+    @ivar identity: The L{Identity} of the Landscape client the broker runs on.
+    @ivar exchanger: The L{MessageExchange} exchanges messages with the server.
+    @ivar pinger: The L{Pinger} checks if the server has new messages for us.
+    @ivar registration: The L{RegistrationHandler} performs the initial
+        registration.
+
+    @cvar service_name: C{"broker"}
     """
 
     transport_factory = HTTPTransport
     service_name = "broker"
 
     def __init__(self, config):
+        """
+        @param config: a L{BrokerConfiguration}.
+        """
         self.persist_filename = os.path.join(
             config.data_path, "%s.bpickle" % (self.service_name,))
         super(BrokerService, self).__init__(config)
@@ -128,9 +165,10 @@
         reactor.stop()
 
     def startService(self):
-        """
-        Set up the persist, message store, transport, reactor, and
-        dbus message exchange service.
+        """Start the broker.
+
+        Create the DBus-published L{BrokerDBusObject}, and start
+        the L{MessageExchange} and L{Pinger} services.
 
         If the configuration specifies the bus as 'session', the DBUS
         message exchange service will use the DBUS Session Bus.

=== modified file 'landscape/broker/exchange.py'
--- landscape/broker/exchange.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/exchange.py	2009-09-21 16:05:18 +0000
@@ -27,6 +27,16 @@
                  monitor_interval=None,
                  max_messages=100,
                  create_time=time.time):
+        """
+        @param reactor: A L{TwistedReactor} used to fire events in response
+            to messages received by the server.
+        @param store: A L{MessageStore} used to queue outgoing messages.
+        @param transport: A L{HTTPTransport} used to deliver messages.
+        @param exchange_interval: time interval between subsequent
+            exchanges of non-urgent messages.
+        @param urgent_exachange_interval: time interval between subsequent
+            exchanges of urgent messages.
+        """
         self._reactor = reactor
         self._message_store = store
         self._create_time = create_time
@@ -50,6 +60,7 @@
         reactor.call_on("pre-exit", self.stop)
 
     def get_exchange_intervals(self):
+        """Return a binary tuple with urgent and normal exchange intervals."""
         return (self._urgent_exchange_interval, self._exchange_interval)
 
     def send(self, message, urgent=False):
@@ -57,6 +68,8 @@
 
         If urgent is True, an exchange with the server will be
         scheduled urgently.
+
+        @param message: Same as in L{MessageStore.add}.
         """
         if "timestamp" not in message:
             message["timestamp"] = int(self._reactor.time())
@@ -85,9 +98,9 @@
         If this makes existing held messages available for sending,
         urgently exchange messages.
 
-        If new types are made available, a
-        C{("message-type-accepted", type_name)} reactor event will
-        be fired.
+        If new types are made available or old types are dropped a
+        C{("message-type-acceptance-changed", type, bool)} reactor
+        event will be fired.
         """
         old_types = set(self._message_store.get_accepted_types())
         new_types = set(message["types"])
@@ -123,7 +136,13 @@
     def exchange(self):
         """Send pending messages to the server and process responses.
 
-        @return: A deferred that is fired when exchange has completed.
+        An C{pre-exchange} reactor event will be emitted just before the
+        actual exchange takes place.
+
+        An C{exchange-done} or C{exchange-failed} reactor event will be
+        emitted after a successful or failed exchange.
+
+        @return: A L{Deferred} that is fired when exchange has completed.
 
         XXX Actually that is a lie right now. It returns before exchange is
         actually complete.
@@ -182,11 +201,10 @@
         approximately 10 seconds before the exchange is started.
 
         @param urgent: If true, ensure an exchange happens within the
-                       urgent interval.  This will reschedule the exchange
-                       if necessary.  If another urgent exchange is already
-                       scheduled, nothing happens.
+            urgent interval.  This will reschedule the exchange if necessary.
+            If another urgent exchange is already scheduled, nothing happens.
         @param force: If true, an exchange will necessarily be scheduled,
-                      even if it was already scheduled before.
+            even if it was already scheduled before.
         """
         # The 'not self._exchanging' check below is currently untested.
         # It's a bit tricky to test as it is preventing rehooking 'exchange'
@@ -217,7 +235,12 @@
         self._reactor.fire("impending-exchange")
 
     def make_payload(self):
-        """Return a dict representing the complete payload."""
+        """Return a dict representing the complete exchange payload.
+
+        The payload will contain all pending messages eligible for
+        delivery, up to a maximum of C{max_messages} as passed to
+        the L{__init__} method.
+        """
         store = self._message_store
         accepted_types_digest = self._hash_types(store.get_accepted_types())
         messages = store.get_pending_messages(self._max_messages)
@@ -263,6 +286,19 @@
         return md5.new(accepted_types_str).digest()
 
     def _handle_result(self, payload, result):
+        """Handle a response from the server.
+
+        Called by L{exchange} after a batch of messages has been
+        successfully delivered to the server.
+
+        If the C{server_uuid} changed, a C{"server-uuid-changed"} event
+        will be fired.
+
+        Call L{handle_message} for each message in C{result}.
+
+        @param payload: The payload that was sent to the server.
+        @param result: The response got in reply to the C{payload}.
+        """
         message_store = self._message_store
         self._client_accepted_types_hash = result.get("client-accepted-types-hash")
         next_expected = result.get("next-expected-sequence")
@@ -309,9 +345,10 @@
                 self.schedule_exchange(urgent=True)
 
     def register_message(self, type, handler):
-        """
-        Register a handler to be called when a message of the given
-        type has been received from the server.
+        """Register a handler for the given message type.
+
+        The C{handler} callable will to be executed when a message of
+        type C{type} has been received from the server.
 
         Multiple handlers for the same type will be called in the
         order they were registered.

=== modified file 'landscape/broker/registration.py'
--- landscape/broker/registration.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/registration.py	2009-09-21 16:05:18 +0000
@@ -1,4 +1,5 @@
 
+import time
 import logging
 import socket
 
@@ -7,9 +8,11 @@
 from landscape.lib.twisted_util import gather_results
 from landscape.lib.bpickle import loads
 from landscape.lib.log import log_failure
-
-
-EC2_API = "http://169.254.169.254/latest"
+from landscape.lib.fetch import fetch, FetchError
+
+
+EC2_HOST = "169.254.169.254"
+EC2_API = "http://%s/latest" % (EC2_HOST,)
 
 
 class InvalidCredentialsError(Exception):
@@ -33,7 +36,15 @@
 
 
 class Identity(object):
-    """Maintains details about the identity of this Landscape client."""
+    """Maintains details about the identity of this Landscape client.
+
+    @ivar secure_id: A server-provided ID for secure message exchange.
+    @ivar insecure_id: Non-secure server-provided ID, mainly used with
+        the ping server.
+    @ivar computer_title: See L{BrokerConfiguration}.
+    @ivar account_name: See L{BrokerConfiguration}.
+    @ivar registration_password: See L{BrokerConfiguration}.
+    """
 
     secure_id = persist_property("secure-id")
     insecure_id = persist_property("insecure-id")
@@ -42,6 +53,11 @@
     registration_password = config_property("registration_password")
 
     def __init__(self, config, persist):
+        """
+        @param config: A L{BrokerConfiguration} object, used to set the
+            C{computer_title}, C{account_name} and C{registration_password}
+            instance variables.
+        """
         self._config = config
         self._persist = persist.root_at("registration")
 
@@ -98,36 +114,6 @@
         self._exchange.exchange()
         return result
 
-    def _extract_ec2_instance_data(self, raw_user_data, launch_index):
-        """
-        Given the raw string of EC2 User Data, parse it and return the dict of
-        instance data for this particular instance.
-
-        If the data can't be parsed, a debug message will be logged and None
-        will be returned.
-        """
-        try:
-            user_data = loads(raw_user_data)
-        except ValueError:
-            logging.debug("Got invalid user-data %r" % (raw_user_data,))
-            return
-
-        if not isinstance(user_data, dict):
-            logging.debug("user-data %r is not a dict" % (user_data,))
-            return
-        for key in "otps", "exchange-url", "ping-url":
-            if key not in user_data:
-                logging.debug("user-data %r doesn't have key %r."
-                              % (user_data, key))
-                return
-        if len(user_data["otps"]) <= launch_index:
-            logging.debug("user-data %r doesn't have OTP for launch index %d"
-                          % (user_data, launch_index))
-            return
-        return {"otp": user_data["otps"][launch_index],
-                "exchange-url": user_data["exchange-url"],
-                "ping-url": user_data["ping-url"]}
-
     def _fetch_ec2_data(self):
         id = self._identity
         if self._cloud and not id.secure_id:
@@ -170,7 +156,7 @@
                 self._ec2_data["launch_index"] = int(
                     self._ec2_data["launch_index"])
 
-                instance_data = self._extract_ec2_instance_data(
+                instance_data = _extract_ec2_instance_data(
                     raw_user_data, int(launch_index))
                 if instance_data is not None:
                     self._otp = instance_data["otp"]
@@ -191,9 +177,15 @@
             registration_data.addErrback(log_error)
 
     def _handle_exchange_done(self):
+        """Registered handler for the C{"exchange-done"} event.
+
+        If we are not registered yet, schedule another message exchange.
+
+        The first exchange made us accept the message type "register", so
+        the next "pre-exchange" event will make L{_handle_pre_exchange}
+        queue a registration message for delivery.
+        """
         if self.should_register() and not self._should_register:
-            # We received accepted-types (first exchange), so we now trigger
-            # the second exchange for registration
             self._exchange.exchange()
 
     def _handle_pre_exchange(self):
@@ -218,7 +210,7 @@
                     logging.info("Queueing message to register with OTP")
                     message = {"type": "register-cloud-vm",
                                "otp": self._otp,
-                               "hostname": socket.gethostname(),
+                               "hostname": socket.getfqdn(),
                                "account_name": None,
                                "registration_password": None,
                                }
@@ -229,7 +221,7 @@
                                  "%r as an EC2 instance." % (id.account_name,))
                     message = {"type": "register-cloud-vm",
                                "otp": None,
-                               "hostname": socket.gethostname(),
+                               "hostname": socket.getfqdn(),
                                "account_name": id.account_name,
                                "registration_password": \
                                    id.registration_password}
@@ -246,15 +238,18 @@
                            "computer_title": id.computer_title,
                            "account_name": id.account_name,
                            "registration_password": id.registration_password,
-                           "hostname": socket.gethostname()}
+                           "hostname": socket.getfqdn()}
                 self._exchange.send(message)
             else:
                 self._reactor.fire("registration-failed")
 
     def _handle_set_id(self, message):
-        """
+        """Registered handler for the C{"set-id"} event.
+        
         Record and start using the secure and insecure IDs from the given
         message.
+
+        Fire C{"registration-done"} and C{"resynchronize-clients"}.
         """
         id = self._identity
         id.secure_id = message.get("id")
@@ -301,3 +296,76 @@
     def _failed(self):
         self.deferred.errback(InvalidCredentialsError())
         self._cancel_calls()
+
+
+def _extract_ec2_instance_data(raw_user_data, launch_index):
+    """
+    Given the raw string of EC2 User Data, parse it and return the dict of
+    instance data for this particular instance.
+
+    If the data can't be parsed, a debug message will be logged and None
+    will be returned.
+    """
+    try:
+        user_data = loads(raw_user_data)
+    except ValueError:
+        logging.debug("Got invalid user-data %r" % (raw_user_data,))
+        return
+
+    if not isinstance(user_data, dict):
+        logging.debug("user-data %r is not a dict" % (user_data,))
+        return
+    for key in "otps", "exchange-url", "ping-url":
+        if key not in user_data:
+            logging.debug("user-data %r doesn't have key %r."
+                          % (user_data, key))
+            return
+    if len(user_data["otps"]) <= launch_index:
+        logging.debug("user-data %r doesn't have OTP for launch index %d"
+                      % (user_data, launch_index))
+        return
+    return {"otp": user_data["otps"][launch_index],
+            "exchange-url": user_data["exchange-url"],
+            "ping-url": user_data["ping-url"]}
+
+
+def _wait_for_network():
+    """
+    Keep trying to connect to the EC2 metadata server until it becomes
+    accessible or until five minutes pass.
+
+    This is necessary because the networking init script on Ubuntu is
+    asynchronous; the network may not actually be up by the time the
+    landscape-client init script is invoked.
+    """
+    timeout = 5*60
+    port = 80
+
+    start = time.time()
+    while True:
+        s = socket.socket()
+        try:
+            s.connect((EC2_HOST, port))
+            s.close()
+            return
+        except socket.error, e:
+            time.sleep(1)
+            if time.time() - start > timeout:
+                break
+                
+
+def is_cloud_managed(fetch=fetch):
+    """
+    Return C{True} if the machine has been started by Landscape, i.e. if we can
+    find the expected data inside the EC2 user-data field.
+    """
+    _wait_for_network()
+    try:
+        raw_user_data = fetch(EC2_API + "/user-data",
+                              connect_timeout=5)
+        launch_index = fetch(EC2_API + "/meta-data/ami-launch-index",
+                             connect_timeout=5)
+    except FetchError:
+        return False
+    instance_data = _extract_ec2_instance_data(raw_user_data, int(launch_index))
+    return instance_data is not None

=== modified file 'landscape/broker/store.py'
--- landscape/broker/store.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/store.py	2009-09-21 16:05:18 +0000
@@ -6,6 +6,7 @@
 import os
 
 from landscape.lib import bpickle
+from landscape.lib.persist import Persist
 from landscape.lib.monitor import Monitor
 from landscape import API
 
@@ -22,7 +23,7 @@
     that reason, let's review the terminology here.
 
     Assume we have 10 messages in the store, which we label by
-    the following uppercase letters:
+    the following uppercase letters::
 
         A, B, C, D, E, F, G, H, I, J
                  ^
@@ -41,6 +42,11 @@
 
     def __init__(self, persist, directory, directory_size=1000,
                  monitor_interval=60*60, get_time=time.time):
+        """
+        @param persist: a L{Persist} used to save state parameters like
+            the accepted message types, sequence, server uuid etc.
+        @param directory: base of the file system hierarchy
+        """
         self._get_time = get_time
         self._directory = directory
         self._directory_size = directory_size
@@ -52,7 +58,7 @@
             os.makedirs(message_dir)
 
     def commit(self):
-        """Save metadata to disk."""
+        """Persist metadata to disk."""
         self._original_persist.save()
 
     def set_accepted_types(self, types):
@@ -67,34 +73,40 @@
         self._reprocess_holding()
 
     def get_accepted_types(self):
+        """Get a list of all accepted message types."""
         return self._persist.get("accepted-types", ())
 
     def accepts(self, type):
+        """Return bool indicating if C{type} is an accepted message type."""
         return type in self.get_accepted_types()
 
     def get_sequence(self):
-        """
-        Get the sequence number of the message that the server expects us to
-        send on the next exchange.
+        """Get the current sequence.
+
+        @return: The sequence number of the message that the server expects us to
+           send on the next exchange.
         """
         return self._persist.get("sequence", 0)
 
     def set_sequence(self, number):
-        """
+        """Set the current sequence.
+
         Set the sequence number of the message that the server expects us to
         send on the next exchange.
         """
         self._persist.set("sequence", number)
 
     def get_server_sequence(self):
-        """
-        Get the sequence number of the message that we will ask the server to
-        send to us on the next exchange.
+        """Get the current server sequence.
+
+        @return: the sequence number of the message that we will ask the server to
+            send to us on the next exchange.
         """
         return self._persist.get("server_sequence", 0)
 
     def set_server_sequence(self, number):
-        """
+        """Set the current server sequence.
+
         Set the sequence number of the message that we will ask the server to
         send to us on the next exchange.
         """
@@ -109,16 +121,19 @@
         self._persist.set("server_uuid", uuid)
 
     def get_pending_offset(self):
+        """Get the current pending offset."""
         return self._persist.get("pending_offset", 0)
 
     def set_pending_offset(self, val):
-        """
+        """Set the current pending offset.
+
         Set the offset into the message pool to consider assigned to the
         current sequence number as returned by l{get_sequence}.
         """
         self._persist.set("pending_offset", val)
 
     def add_pending_offset(self, val):
+        """Increment the current pending offset by C{val}."""
         self.set_pending_offset(self.get_pending_offset() + val)
 
     def count_pending_messages(self):
@@ -186,7 +201,9 @@
 
     def add(self, message):
         """Queue a message for delivery.
-        
+
+        @param message: a C{dict} with a C{type} key and other keys conforming
+            to the L{Message} schema for that specifc message type.
         @return: message_id, which is an identifier for the added message.
         """
         assert "type" in message

=== modified file 'landscape/broker/tests/test_registration.py'
--- landscape/broker/tests/test_registration.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/tests/test_registration.py	2009-09-21 16:05:18 +0000
@@ -1,3 +1,4 @@
+import unittest
 import logging
 import pycurl
 import socket
@@ -5,12 +6,13 @@
 from twisted.internet.defer import succeed, fail
 
 from landscape.broker.registration import (
-    InvalidCredentialsError, RegistrationHandler)
+    InvalidCredentialsError, RegistrationHandler, is_cloud_managed, EC2_HOST,
+    EC2_API)
 
 from landscape.broker.deployment import BrokerConfiguration
 from landscape.tests.helpers import LandscapeTest, ExchangeHelper
 from landscape.lib.bpickle import dumps
-from landscape.lib.fetch import HTTPCodeError
+from landscape.lib.fetch import HTTPCodeError, FetchError
 
 
 class RegistrationTest(LandscapeTest):
@@ -23,9 +25,9 @@
         self.identity = self.broker_service.identity
         self.handler = self.broker_service.registration
         logging.getLogger().setLevel(logging.INFO)
-        self.hostname = "ooga"
-        self.addCleanup(setattr, socket, "gethostname", socket.gethostname)
-        socket.gethostname = lambda: self.hostname
+        self.hostname = "ooga.local"
+        self.addCleanup(setattr, socket, "getfqdn", socket.getfqdn)
+        socket.getfqdn = lambda: self.hostname
 
     def check_persist_property(self, attr, persist_name):
         value = "VALUE"
@@ -141,7 +143,7 @@
                               "computer_title": "Computer Title",
                               "account_name": "account_name",
                               "registration_password": None,
-                              "hostname": "ooga"}
+                              "hostname": "ooga.local"}
                             ])
         self.assertEquals(self.logfile.getvalue().strip(),
                           "INFO: Queueing message to register with account "
@@ -159,7 +161,7 @@
                               "computer_title": "Computer Title",
                               "account_name": "account_name",
                               "registration_password": "SEKRET",
-                              "hostname": "ooga"}
+                              "hostname": "ooga.local"}
                             ])
         self.assertEquals(self.logfile.getvalue().strip(),
                           "INFO: Queueing message to register with account "
@@ -341,7 +343,7 @@
                               "computer_title": "Computer Title",
                               "account_name": "account_name",
                               "registration_password": "SEKRET",
-                              "hostname": socket.gethostname()}
+                              "hostname": socket.getfqdn()}
                              ])
 
     def get_registration_handler_for_cloud(
@@ -406,7 +408,7 @@
         """
         message = dict(type="register-cloud-vm",
                        otp="otp1",
-                       hostname="ooga",
+                       hostname="ooga.local",
                        local_hostname="ooga.local",
                        public_hostname="ooga.amazon.com",
                        instance_key=u"key1",
@@ -641,7 +643,7 @@
                               "computer_title": u"whatever",
                               "account_name": u"onward",
                               "registration_password": u"password",
-                              "hostname": socket.gethostname()}])
+                              "hostname": socket.getfqdn()}])
 
     def test_should_register_in_cloud(self):
         """
@@ -729,3 +731,151 @@
         config.computer_title = None
         self.broker_service.identity.secure_id = None
         self.assertFalse(handler.should_register())
+
+
+class IsCloudManagedTests(LandscapeTest):
+
+    def setUp(self):
+        super(IsCloudManagedTests, self).setUp()
+        self.urls = []
+        self.responses = []
+
+    def fake_fetch(self, url, connect_timeout=None):
+        self.urls.append((url, connect_timeout))
+        return self.responses.pop(0)
+
+    def mock_socket(self):
+        """
+        Mock out socket usage by is_cloud_managed to wait for the network.
+        """
+        # Mock the socket.connect call that it also does
+        socket_class = self.mocker.replace("socket.socket", passthrough=False)
+        socket = socket_class()
+        socket.connect((EC2_HOST, 80))
+        socket.close()
+        
+
+    def test_is_managed(self):
+        """
+        L{is_cloud_managed} returns True if the EC2 user-data contains Landscape
+        instance information.  It fetches the EC2 data with low timeouts.
+        """
+        user_data = {"otps": ["otp1"], "exchange-url": "http://exchange",
+                     "ping-url": "http://ping"}
+        self.responses = [dumps(user_data), "0"]
+
+        self.mock_socket()
+        self.mocker.replay()
+
+        self.assertTrue(is_cloud_managed(self.fake_fetch))
+        self.assertEquals(
+            self.urls,
+            [(EC2_API + "/user-data", 5),
+             (EC2_API + "/meta-data/ami-launch-index", 5)])
+
+    def test_is_managed_index(self):
+        user_data = {"otps": ["otp1", "otp2"],
+                     "exchange-url": "http://exchange",
+                     "ping-url": "http://ping"}
+        self.responses = [dumps(user_data), "1"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertTrue(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_wrong_index(self):
+        user_data = {"otps": ["otp1"], "exchange-url": "http://exchange",
+                     "ping-url": "http://ping"}
+        self.responses = [dumps(user_data), "1"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_exchange_url(self):
+        user_data = {"otps": ["otp1"], "ping-url": "http://ping"}
+        self.responses = [dumps(user_data), "0"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_ping_url(self):
+        user_data = {"otps": ["otp1"], "exchange-url": "http://exchange"}
+        self.responses = [dumps(user_data), "0"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_bpickle(self):
+        self.responses = ["some other user data", "0"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_no_data(self):
+        self.responses = ["", "0"]
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(self.fake_fetch))
+
+    def test_is_managed_fetch_not_found(self):
+        def fake_fetch(url, connect_timeout=None):
+            raise HTTPCodeError(404, "ohnoes")
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(fake_fetch))
+
+    def test_is_managed_fetch_error(self):
+        def fake_fetch(url, connect_timeout=None):
+            raise FetchError(7, "couldn't connect to host")
+        self.mock_socket()
+        self.mocker.replay()
+        self.assertFalse(is_cloud_managed(fake_fetch))
+
+    def test_waits_for_network(self):
+        """
+        is_cloud_managed will wait until the network before trying to fetch
+        the EC2 user data.
+        """
+        user_data = {"otps": ["otp1"], "exchange-url": "http://exchange",
+                     "ping-url": "http://ping"}
+        self.responses = [dumps(user_data), "0"]
+
+        self.mocker.order()
+        time_sleep = self.mocker.replace("time.sleep", passthrough=False)
+        socket_class = self.mocker.replace("socket.socket", passthrough=False)
+        socket_obj = socket_class()
+        socket_obj.connect((EC2_HOST, 80))
+        self.mocker.throw(socket.error("woops"))
+        time_sleep(1)
+        socket_obj = socket_class()
+        socket_obj.connect((EC2_HOST, 80))
+        self.mocker.result(None)
+        socket_obj.close()
+        self.mocker.replay()
+        self.assertTrue(is_cloud_managed(self.fake_fetch))
+
+    def test_waiting_times_out(self):
+        """
+        We'll only wait five minutes for the network to come up.
+        """
+        def fake_fetch(url, connect_timeout=None):
+            raise FetchError(7, "couldn't connect to host")
+
+        self.mocker.order()
+        time_sleep = self.mocker.replace("time.sleep", passthrough=False)
+        time_time = self.mocker.replace("time.time", passthrough=False)
+        time_time()
+        self.mocker.result(100)
+        socket_class = self.mocker.replace("socket.socket", passthrough=False)
+        socket_obj = socket_class()
+        socket_obj.connect((EC2_HOST, 80))
+        self.mocker.throw(socket.error("woops"))
+        time_sleep(1)
+        time_time()
+        self.mocker.result(401)
+        self.mocker.replay()
+        # Mocking time.time is dangerous, because the test harness calls it. So
+        # we explicitly reset mocker before returning from the test.
+        try:
+            self.assertFalse(is_cloud_managed(fake_fetch))
+        finally:
+            self.mocker.reset()

=== modified file 'landscape/broker/tests/test_transport.py'
--- landscape/broker/tests/test_transport.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/tests/test_transport.py	2009-09-21 16:05:18 +0000
@@ -1,9 +1,8 @@
 import os
 
-from pycurl import error as PyCurlError
-
 from landscape import VERSION
 from landscape.broker.transport import HTTPTransport
+from landscape.lib.fetch import PyCurlError
 from landscape.lib import bpickle
 
 from landscape.tests.helpers import LandscapeTest, LogKeeperHelper

=== modified file 'landscape/broker/transport.py'
--- landscape/broker/transport.py	2009-03-18 20:42:05 +0000
+++ landscape/broker/transport.py	2009-09-21 16:05:18 +0000
@@ -15,13 +15,19 @@
     """Transport makes a request to exchange message data over HTTP."""
 
     def __init__(self, url, pubkey=None):
+        """
+        @param url: URL of the remote Landscape server message system.
+        @param pubkey: SSH pubblic key used for secure communication.
+        """
         self._url = url
         self._pubkey = pubkey
 
     def get_url(self):
+        """Get the URL of the remote message system."""
         return self._url
 
     def set_url(self, url):
+        """Set the URL of the remote message system."""
         self._url = url
 
     def _curl(self, payload, computer_id, message_api):
@@ -37,7 +43,16 @@
     def exchange(self, payload, computer_id=None, message_api=API):
         """Exchange message data with the server.
 
-        THREAD SAFE (HOPEFULLY)
+        @param payload: The object to send, it must be L{bpickle}-compatible.
+        @param computer_id: The computer ID to send the message as (see
+            also L{Identity}).
+
+        @type: C{dict}
+        @return: The server's response to sent message or C{None} in case
+            of error.
+
+        @note: This code is thread safe (HOPEFULLY).
+
         """
         spayload = bpickle.dumps(payload)
         try:

=== modified file 'landscape/configuration.py'
--- landscape/configuration.py	2009-03-18 20:42:05 +0000
+++ landscape/configuration.py	2009-09-21 16:05:18 +0000
@@ -12,15 +12,13 @@
 from ConfigParser import ConfigParser, Error as ConfigParserError
 from StringIO import StringIO
 
-import pycurl
-
 from dbus.exceptions import DBusException
 
 from landscape.sysvconfig import SysVConfig, ProcessError
 from landscape.lib.dbus_util import (
     get_bus, NoReplyError, ServiceUnknownError, SecurityError)
 from landscape.lib.twisted_util import gather_results
-from landscape.lib.fetch import fetch, HTTPCodeError
+from landscape.lib.fetch import fetch, FetchError
 
 from landscape.broker.registration import InvalidCredentialsError
 from landscape.broker.deployment import BrokerConfiguration
@@ -499,9 +497,7 @@
     error_message = None
     try:
         content = fetch(url)
-    except pycurl.error, error:
-        error_message = error.args[1]
-    except HTTPCodeError, error:
+    except FetchError, error:
         error_message = str(error)
     if error_message is not None:
         raise ImportOptionError(

=== modified file 'landscape/deployment.py'
--- landscape/deployment.py	2009-03-18 20:42:05 +0000
+++ landscape/deployment.py	2009-09-21 16:05:18 +0000
@@ -44,12 +44,12 @@
 class BaseConfiguration(object):
     """Base class for configuration implementations.
 
-    @var required_options: Optionally, a sequence of key names to require when
+    @cvar required_options: Optionally, a sequence of key names to require when
         reading or writing a configuration.
-    @var unsaved_options: Optionally, a sequence of key names to never write
+    @cvar unsaved_options: Optionally, a sequence of key names to never write
         to the configuration file.  This is useful when you want to provide
         command-line options that should never end up in a configuration file.
-    @var default_config_filenames: A sequence of filenames to check when
+    @cvar default_config_filenames: A sequence of filenames to check when
         reading or writing a configuration.
     """
 
@@ -63,6 +63,10 @@
     config_section = "client"
 
     def __init__(self):
+        """Default configuration.
+
+        Default values for supported options are set as in L{make_parser}.
+        """
         self._set_options = {}
         self._command_line_args = []
         self._command_line_options = {}
@@ -78,10 +82,10 @@
         """Find and return the value of the given configuration parameter.
 
         The following sources will be searched:
-          * The attributes that were explicitly set on this object,
-          * The parameters specified on the command line,
-          * The parameters specified in the configuration file, and
-          * The defaults.
+          - The attributes that were explicitly set on this object,
+          - The parameters specified on the command line,
+          - The parameters specified in the configuration file, and
+          - The defaults.
 
         If no values are found and the parameter does exist as a possible
         parameter, C{None} is returned.
@@ -107,6 +111,7 @@
         return value
 
     def get(self, name, default=None):
+        """Return the value of the C{name} option or C{default}."""
         try:
             return self.__getattr__(name)
         except AttributeError:
@@ -124,6 +129,10 @@
             self._set_options[name] = value
 
     def reload(self):
+        """Reload options using the configured command line arguments.
+
+        @see: L{load_command_line}
+        """
         self.load(self._command_line_args)
 
     def load(self, args, accept_nonexistent_config=False):
@@ -189,15 +198,12 @@
 
         Options are considered in the following precedence:
 
-        1. Manually set options (config.option = value)
-        2. Options passed in the command line
-        3. Previously existent options in the configuration file
-
-        The filename picked for saving configuration options is:
-
-        1. self.config, if defined
-        2. The last loaded configuration file, if any
-        3. The first filename in self.default_config_filenames
+          1. Manually set options (C{config.option = value})
+          2. Options passed in the command line
+          3. Previously existent options in the configuration file
+
+        The filename picked for saving configuration options is the one
+        returned by L{get_config_filename}.
         """
         # The filename we'll write to
         filename = self.get_config_filename()
@@ -223,9 +229,12 @@
         config_file.close()
 
     def make_parser(self):
-        """
-        Return an L{OptionParser} preset with options that all
-        landscape-related programs accept.
+        """Parser factory for supported options
+
+        @return: An L{OptionParser} preset with options that all
+            landscape-related programs accept. These include
+              - C{config} (C{None})
+              - C{bus} (C{system})
         """
         parser = OptionParser(version=VERSION)
         parser.add_option("-c", "--config", metavar="FILE",
@@ -237,6 +246,13 @@
         return parser
 
     def get_config_filename(self):
+        """Pick the proper configuration file.
+
+        The picked filename is:
+          1. C{self.config}, if defined
+          2. The last loaded configuration file, if any
+          3. The first filename in C{self.default_config_filenames}
+        """
         if self.config:
             return self.config
         if self._config_filename:
@@ -249,6 +265,10 @@
         return None
 
     def get_command_line_options(self):
+        """Get currently loaded command line options.
+
+        @see: L{load_command_line}
+        """
         return self._command_line_options
 
 
@@ -263,9 +283,15 @@
         return os.path.join(self.data_path, "hash.db")
 
     def make_parser(self):
-        """
-        Return an L{OptionParser} preset with options that all
-        landscape-related programs accept.
+        """Parser factory for supported options.
+
+        @return: An L{OptionParser} preset for all options
+            from L{BaseConfiguration.make_parser} plus:
+              - C{data_path} (C{"/var/lib/landscape/client/"})
+              - C{quiet} (C{False})
+              - C{log_dir} (C{"/var/log/landscape"})
+              - C{log_level} (C{"info"})
+              - C{ignore_sigint} (C{False})
         """
         parser = super(Configuration, self).make_parser()
         parser.add_option("-d", "--data-path", metavar="PATH",
@@ -286,9 +312,10 @@
 
 
 def get_versioned_persist(service):
-    """
-    Load a Persist database for the given service and upgrade or mark
-    as current, as necessary.
+    """Get a L{Persist} database with upgrade rules applied.
+
+    Load a L{Persist} database for the given C{service} and upgrade or
+    mark as current, as necessary.
     """
     persist = Persist(filename=service.persist_filename)
     upgrade_manager = UPGRADE_MANAGERS[service.service_name]
@@ -301,12 +328,12 @@
 
 
 class LandscapeService(Service, object):
-    """
-    A utility superclass for defining Landscape services.
+    """Utility superclass for defining Landscape services.
 
     This sets up the reactor, bpickle/dbus integration, a Persist object, and
     connects to the bus when started.
 
+    @ivar reactor: a L{TwistedReactor} object.
     @cvar service_name: The lower-case name of the service. This is used to
         generate the bpickle filename.
     """
@@ -322,6 +349,11 @@
         signal.signal(signal.SIGUSR1, lambda signal, frame: rotate_logs())
 
     def startService(self):
+        """Extend L{twisted.application.service.IService.startService}.
+
+        Create a a new DBus connection (normally using a C{SystemBus}) and
+        save it in the public L{self.bus} instance variable.
+        """
         Service.startService(self)
         self.bus = get_bus(self.config.bus)
         info("%s started on '%s' bus with config %s" % (
@@ -353,9 +385,15 @@
 def run_landscape_service(configuration_class, service_class, args, bus_name):
     """Run a Landscape service.
 
-    @param configuration_class: A subclass of L{Configuration} for processing
-        the C{args} and config file.
-    @param service_class: A subclass of L{LandscapeService} to create and start.
+    The function will instantiate the given L{LandscapeService} subclass
+    and attach the resulting service object to a Twisted C{Application}.
+
+    After that it will start the Twisted L{Application} and call the
+    L{TwistedReactor.run} method of the L{LandscapeService}'s reactor.
+
+    @param configuration_class: The service-specific subclass of L{Configuration} used
+        to parse C{args} and build the C{service_class} object.
+    @param service_class: The L{LandscapeService} subclass to create and start.
     @param args: Command line arguments.
     @param bus_name: A bus name used to verify if the service is already
         running.

=== added file 'landscape/lib/command.py'
--- landscape/lib/command.py	1970-01-01 00:00:00 +0000
+++ landscape/lib/command.py	2009-09-21 16:05:18 +0000
@@ -0,0 +1,38 @@
+"""Shell commands execution."""
+import commands
+
+class CommandError(Exception):
+    """
+    Raised by L{run_command} in case of non-0 exit code.
+
+    @cvar command: The shell command that failed.
+    @cvar exit_status: Its non-zero exit status.
+    @cvar output: The command's output.
+    """
+    def __init__(self, command, exit_status, output):
+        self.command = command
+        self.exit_status = exit_status
+        self.output = output
+
+    def __str__(self):
+        return "'%s' exited with status %d (%s)" % (
+            self.command, self.exit_status, self.output)
+
+    def __repr__(self):
+        return "<CommandError command=<%s> exit_status=%d output=<%s>>" % (
+            self.command, self.exit_status, self.output)
+
+
+def run_command(command):
+    """
+    Execute a command in a shell and return the command's output.
+
+    If the command's exit status is not 0 a L{CommandError} exception
+    is raised.
+    """
+    exit_status, output = commands.getstatusoutput(command)
+    # shift down 8 bits to get shell-like exit codes
+    exit_status = exit_status >> 8
+    if exit_status != 0:
+        raise CommandError(command, exit_status, output)
+    return output

=== modified file 'landscape/lib/fetch.py'
--- landscape/lib/fetch.py	2009-04-13 14:33:31 +0000
+++ landscape/lib/fetch.py	2009-09-21 16:05:18 +0000
@@ -6,8 +6,10 @@
 
 from twisted.internet.threads import deferToThread
 
+class FetchError(Exception):
+    pass
 
-class HTTPCodeError(Exception):
+class HTTPCodeError(FetchError):
 
     def __init__(self, http_code, body):
         self.http_code = http_code
@@ -20,7 +22,20 @@
         return "<HTTPCodeError http_code=%d>" % self.http_code
 
 
-def fetch(url, post=False, data="", headers={}, cainfo=None, curl=None):
+class PyCurlError(FetchError):
+    def __init__(self, error_code, message):
+        self.error_code = error_code
+        self.message = message
+
+    def __str__(self):
+        return "Error %d: %s" % (self.error_code, self.message)
+
+    def __repr__(self):
+        return "<PyCurlError args=(%d, '%s')>" % (self.error_code,
+                                                  self.message)
+
+def fetch(url, post=False, data="", headers={}, cainfo=None, curl=None,
+          connect_timeout=30, total_timeout=600):
     """Retrieve a URL and return the content.
 
     @param url: The url to be fetched.
@@ -53,12 +68,16 @@
     curl.setopt(pycurl.URL, url)
     curl.setopt(pycurl.FOLLOWLOCATION, True)
     curl.setopt(pycurl.MAXREDIRS, 5)
-    curl.setopt(pycurl.CONNECTTIMEOUT, 30)
-    curl.setopt(pycurl.TIMEOUT, 600)
+    curl.setopt(pycurl.CONNECTTIMEOUT, connect_timeout)
+    curl.setopt(pycurl.LOW_SPEED_LIMIT, 1)
+    curl.setopt(pycurl.LOW_SPEED_TIME, total_timeout)
     curl.setopt(pycurl.NOSIGNAL, 1)
     curl.setopt(pycurl.WRITEFUNCTION, input.write)
 
-    curl.perform()
+    try:
+        curl.perform()
+    except pycurl.error, e:
+        raise PyCurlError(e.args[0], e.args[1])
 
     body = input.getvalue()
 

=== modified file 'landscape/lib/persist.py'
--- landscape/lib/persist.py	2008-09-08 16:35:57 +0000
+++ landscape/lib/persist.py	2009-09-21 16:05:18 +0000
@@ -40,22 +40,23 @@
 
 
 class Persist(object):
-    """Persistence handler.
+
+    """Persist a hierarchical database of key=>value pairs.
 
     There are three different kinds of opition maps, regarding the
     persistence and priority that maps are queried.
 
-    hard - Options are persistent.
-    soft - Options are not persistent, and have a higher priority
+      - hard - Options are persistent.
+      - soft - Options are not persistent, and have a higher priority
            than persistent options.
-    weak - Options are not persistent, and have a lower priority
+      - weak - Options are not persistent, and have a lower priority
            than persistent options.
 
     @ivar filename: The name of the file where persist data is saved
-        or None if not filename is available.
+        or None if no filename is available.
+
     """
 
-
     def __init__(self, backend=None, filename=None):
         """
         @param backend: The backend to use. If none is specified,
@@ -91,13 +92,19 @@
     modified = property(_get_modified)
 
     def reset_modified(self):
+        """Set the database status as non-modified."""
         self._modified = False
 
     def assert_writable(self):
+        """Assert if the object is writable
+
+        @raise: L{PersistReadOnlyError}
+        """
         if self._readonly:
             raise PersistReadOnlyError("Configuration is in readonly mode.")
 
     def load(self, filepath):
+        """Load a persisted database."""
         filepath = os.path.expanduser(filepath)
         if not os.path.isfile(filepath):
             raise PersistError("File not found: %s" % filepath)
@@ -313,12 +320,31 @@
         return result
 
     def root_at(self, path):
+        """
+        Rebase the database hierarchy.
+
+        @return: A L{RootedPersist} using this L{Persist} as parent.
+        """
         return RootedPersist(self, path)
 
 
 class RootedPersist(object):
+    """Root a L{Persist}'s tree at a particular branch.
+
+    This class shares the same interface of L{Persist} and provides a shortcut
+    to access the nodes of a particular branch in a L{Persist}'s tree.
+
+    The chosen branch will be viewed as the root of the tree of the
+    L{RootedPersist} and all operations will be forwarded to the parent
+    L{Persist} as appropriate.
+    """
 
     def __init__(self, parent, root):
+        """
+        @param parent: the parent L{Persist}.
+        @param root: a branch of the parent L{Persist}'s tree, that
+            will be used as root of this L{RootedPersist}.
+        """
         self.parent = parent
         if type(root) is str:
             self.root = path_string_to_tuple(root)

=== added file 'landscape/lib/tests/test_command.py'
--- landscape/lib/tests/test_command.py	1970-01-01 00:00:00 +0000
+++ landscape/lib/tests/test_command.py	2009-09-21 16:05:18 +0000
@@ -0,0 +1,32 @@
+from landscape.tests.helpers import LandscapeTest
+
+from landscape.lib.command import run_command, CommandError
+
+
+class CommandTest(LandscapeTest):
+
+    def setUp(self):
+        super(CommandTest, self).setUp()
+
+    def test_basic(self):
+        self.assertEquals(run_command("echo test"), "test")
+
+    def test_non_0_exit_status(self):
+        try:
+            run_command("false")
+        except CommandError, error:
+            self.assertEquals(error.command, "false")
+            self.assertEquals(error.output, "")
+            self.assertEquals(error.exit_status, 1)
+        else:
+            self.fail("CommandError not raised")
+
+    def test_error_str(self):
+        self.assertEquals(str(CommandError("test_command", 1, "test output")),
+                          "'test_command' exited with status 1 "
+                          "(test output)")
+
+    def test_error_repr(self):
+        self.assertEquals(repr(CommandError("test_command", 1, "test output")),
+                          "<CommandError command=<test_command> "
+                          "exit_status=1 output=<test output>>")

=== modified file 'landscape/lib/tests/test_fetch.py'
--- landscape/lib/tests/test_fetch.py	2009-04-13 14:33:31 +0000
+++ landscape/lib/tests/test_fetch.py	2009-09-21 16:05:18 +0000
@@ -1,16 +1,17 @@
 import pycurl
 
-from landscape.lib.fetch import fetch, fetch_async, HTTPCodeError
+from landscape.lib.fetch import fetch, fetch_async, HTTPCodeError, PyCurlError
 from landscape.tests.helpers import LandscapeTest
 
 
 class CurlStub(object):
 
-    def __init__(self, result=None, http_code=200):
+    def __init__(self, result=None, http_code=200, error=None):
         self.result = result
         self._http_code = http_code
         self.options = {}
         self.performed = False
+        self.error = error
 
     def getinfo(self, what):
         if what == pycurl.HTTP_CODE:
@@ -23,6 +24,8 @@
         self.options[option] = value
 
     def perform(self):
+        if self.error:
+            raise self.error
         if self.performed:
             raise AssertionError("Can't perform twice")
         self.options[pycurl.WRITEFUNCTION](self.result)
@@ -45,7 +48,8 @@
                            pycurl.FOLLOWLOCATION: True,
                            pycurl.MAXREDIRS: 5,
                            pycurl.CONNECTTIMEOUT: 30,
-                           pycurl.TIMEOUT: 600,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 600,
                            pycurl.NOSIGNAL: 1,
                            pycurl.WRITEFUNCTION: Any()})
 
@@ -58,7 +62,8 @@
                            pycurl.FOLLOWLOCATION: True,
                            pycurl.MAXREDIRS: 5,
                            pycurl.CONNECTTIMEOUT: 30,
-                           pycurl.TIMEOUT: 600,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 600,
                            pycurl.NOSIGNAL: 1,
                            pycurl.WRITEFUNCTION: Any(),
                            pycurl.POST: True})
@@ -73,7 +78,8 @@
                            pycurl.FOLLOWLOCATION: True,
                            pycurl.MAXREDIRS: 5,
                            pycurl.CONNECTTIMEOUT: 30,
-                           pycurl.TIMEOUT: 600,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 600,
                            pycurl.NOSIGNAL: 1,
                            pycurl.WRITEFUNCTION: Any(),
                            pycurl.POST: True,
@@ -89,7 +95,8 @@
                            pycurl.FOLLOWLOCATION: True,
                            pycurl.MAXREDIRS: 5,
                            pycurl.CONNECTTIMEOUT: 30,
-                           pycurl.TIMEOUT: 600,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 600,
                            pycurl.NOSIGNAL: 1,
                            pycurl.WRITEFUNCTION: Any(),
                            pycurl.CAINFO: "cainfo"})
@@ -110,11 +117,27 @@
                            pycurl.FOLLOWLOCATION: True,
                            pycurl.MAXREDIRS: 5,
                            pycurl.CONNECTTIMEOUT: 30,
-                           pycurl.TIMEOUT: 600,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 600,
                            pycurl.NOSIGNAL: 1,
                            pycurl.WRITEFUNCTION: Any(),
                            pycurl.HTTPHEADER: ["a: 1", "b: 2"]})
 
+    def test_timeouts(self):
+        curl = CurlStub("result")
+        result = fetch("http://example.com", connect_timeout=5, total_timeout=30,
+                       curl=curl)
+        self.assertEquals(result, "result")
+        self.assertEquals(curl.options,
+                          {pycurl.URL: "http://example.com",
+                           pycurl.FOLLOWLOCATION: True,
+                           pycurl.MAXREDIRS: 5,
+                           pycurl.CONNECTTIMEOUT: 5,
+                           pycurl.LOW_SPEED_LIMIT: 1,
+                           pycurl.LOW_SPEED_TIME: 30,
+                           pycurl.NOSIGNAL: 1,
+                           pycurl.WRITEFUNCTION: Any()})
+
     def test_non_200_result(self):
         curl = CurlStub("result", http_code=404)
         try:
@@ -125,14 +148,32 @@
         else:
             self.fail("HTTPCodeError not raised")
 
-    def test_error_str(self):
+    def test_http_error_str(self):
         self.assertEquals(str(HTTPCodeError(501, "")),
                           "Server returned HTTP code 501")
 
-    def test_error_repr(self):
+    def test_http_error_repr(self):
         self.assertEquals(repr(HTTPCodeError(501, "")),
                           "<HTTPCodeError http_code=501>")
 
+    def test_pycurl_error(self):
+        curl = CurlStub(result=None, http_code=None,
+                        error=pycurl.error(60, "pycurl error"))
+        try:
+            fetch("http://example.com", curl=curl)
+        except PyCurlError, error:
+            self.assertEquals(error.error_code, 60)
+            self.assertEquals(error.message, "pycurl error")
+        else:
+            self.fail("PyCurlError not raised")
+
+    def test_pycurl_error_str(self):
+        self.assertEquals(str(PyCurlError(60, "pycurl error")),
+                          "Error 60: pycurl error")
+
+    def test_pycurl_error_repr(self):
+        self.assertEquals(repr(PyCurlError(60, "pycurl error")),
+                          "<PyCurlError args=(60, 'pycurl error')>")
 
     def test_create_curl(self):
         curls = []
@@ -151,7 +192,8 @@
                                pycurl.FOLLOWLOCATION: True,
                                pycurl.MAXREDIRS: 5,
                                pycurl.CONNECTTIMEOUT: 30,
-                               pycurl.TIMEOUT: 600,
+                               pycurl.LOW_SPEED_LIMIT: 1,
+                               pycurl.LOW_SPEED_TIME: 600,
                                pycurl.NOSIGNAL: 1,
                                pycurl.WRITEFUNCTION: Any()})
         finally:

=== modified file 'landscape/manager/deployment.py'
--- landscape/manager/deployment.py	2008-12-11 17:11:08 +0000
+++ landscape/manager/deployment.py	2009-09-21 16:05:18 +0000
@@ -84,11 +84,18 @@
                                               self.config, self.bus, store_name)
         self.dbus_service = ManagerDBusObject(self.bus, self.registry)
         DBusSignalToReactorTransmitter(self.bus, self.reactor)
-        self.remote_broker.register_plugin(self.dbus_service.bus_name,
-                                           self.dbus_service.object_path)
+
         for plugin in self.plugins:
             self.registry.add(plugin)
 
+        def broker_started():
+            self.remote_broker.register_plugin(self.dbus_service.bus_name,
+                                               self.dbus_service.object_path)
+            self.registry.broker_started()
+
+        broker_started()
+        self.bus.add_signal_receiver(broker_started, "broker_started")
+
 
 def run(args):
     run_landscape_service(ManagerConfiguration, ManagerService, args,

=== modified file 'landscape/manager/scriptexecution.py'
--- landscape/manager/scriptexecution.py	2008-12-11 17:11:08 +0000
+++ landscape/manager/scriptexecution.py	2009-09-21 16:05:18 +0000
@@ -137,7 +137,9 @@
     def _respond(self, status, data, opid, result_code=None):
         message =  {"type": "operation-result",
                     "status": status,
-                    "result-text": data,
+                    # Let's decode result-text, replacing non-printable
+                    # characters
+                    "result-text": data.decode("utf-8", "replace"),
                     "operation-id": opid}
         if result_code:
             message["result-code"] = result_code

=== modified file 'landscape/manager/tests/test_deployment.py'
--- landscape/manager/tests/test_deployment.py	2008-12-11 17:11:08 +0000
+++ landscape/manager/tests/test_deployment.py	2009-09-21 16:05:18 +0000
@@ -1,5 +1,7 @@
 import os
 
+from twisted.internet.defer import Deferred
+
 from landscape.tests.helpers import (
     LandscapeTest, LandscapeIsolatedTest, RemoteBrokerHelper)
 from landscape.manager.deployment import ManagerService, ManagerConfiguration
@@ -70,32 +72,63 @@
 
     helpers = [RemoteBrokerHelper]
 
+    def setUp(self):
+        super(DeploymentBusTests, self).setUp()
+        configuration = ManagerConfiguration()
+        self.path = self.make_dir()
+        configuration.load(["-d", self.path, "--bus", "session",
+                            "--manager-plugins", "ProcessKiller"])
+        self.manager_service = ManagerService(configuration)
+        self.manager_service.startService()
+
     def test_dbus_reactor_transmitter_installed(self):
-        configuration = ManagerConfiguration()
-        configuration.load(["-d", self.make_dir(), "--bus", "session",
-                            "--manager-plugins", "ProcessKiller"])
-        manager_service = ManagerService(configuration)
-        manager_service.startService()
         return assertTransmitterActive(self, self.broker_service,
-                                       manager_service.reactor)
+                                       self.manager_service.reactor)
 
     def test_receives_messages(self):
-        configuration = ManagerConfiguration()
-        configuration.load(["-d", self.make_dir(), "--bus", "session",
-                            "--manager-plugins", "ProcessKiller"])
-        manager_service = ManagerService(configuration)
-        manager_service.startService()
-        return assertReceivesMessages(self, manager_service.dbus_service,
+        return assertReceivesMessages(self, self.manager_service.dbus_service,
                                       self.broker_service, self.remote)
 
     def test_manager_store(self):
-        configuration = ManagerConfiguration()
-        path = self.make_dir()
-        configuration.load(["-d", path, "--bus", "session",
-                            "--manager-plugins", "ProcessKiller"])
-        manager_service = ManagerService(configuration)
-        manager_service.startService()
-        self.assertNotIdentical(manager_service.registry.store, None)
-        self.assertTrue(
-            isinstance(manager_service.registry.store, ManagerStore))
-        self.assertTrue(os.path.isfile(os.path.join(path, "manager.database")))
+        self.assertNotIdentical(self.manager_service.registry.store, None)
+        self.assertTrue(
+            isinstance(self.manager_service.registry.store, ManagerStore))
+        self.assertTrue(
+            os.path.isfile(os.path.join(self.path, "manager.database")))
+
+    def test_register_plugin_on_broker_started(self):
+        """
+        When the broker is restarted, it fires a "broker-started" signal which
+        makes the Manager plugin register itself again.
+        """
+        d = Deferred()
+        def register_plugin(bus_name, object_path):
+            d.callback((bus_name, object_path))
+        def patch(ignore):
+            self.manager_service.remote_broker.register_plugin = register_plugin
+            self.broker_service.dbus_object.broker_started()
+            return d
+        return self.remote.get_registered_plugins(
+            ).addCallback(patch
+            ).addCallback(self.assertEquals,
+                ("com.canonical.landscape.Manager",
+                 "/com/canonical/landscape/Manager"))
+
+    def test_register_message_on_broker_started(self):
+        """
+        When the broker is restarted, it fires a "broker-started" signal which
+        makes the Manager plugin register all registered messages again.
+        """
+        self.manager_service.registry.register_message("foo", lambda x: None)
+        d = Deferred()
+        def register_client_accepted_message_type(type):
+            if type == "foo":
+                d.callback(type)
+        def patch(ignore):
+            self.manager_service.remote_broker.register_client_accepted_message_type = \
+                register_client_accepted_message_type
+            self.broker_service.dbus_object.broker_started()
+            return d
+        return self.remote.get_registered_plugins(
+            ).addCallback(patch
+            ).addCallback(self.assertEquals, "foo")

=== modified file 'landscape/manager/tests/test_scriptexecution.py'
--- landscape/manager/tests/test_scriptexecution.py	2008-12-11 17:11:08 +0000
+++ landscape/manager/tests/test_scriptexecution.py	2009-09-21 16:05:18 +0000
@@ -3,6 +3,7 @@
 import sys
 import tempfile
 import stat
+import pwd
 
 from twisted.internet.defer import gatherResults
 from twisted.internet.error import ProcessDone
@@ -19,7 +20,7 @@
 
 
 def get_default_environment():
-    username = os.getlogin()
+    username = pwd.getpwuid(os.getuid())[0]
     uid, gid, home = get_user_info(username)
     return  {
         "PATH": UBUNTU_PATH,
@@ -334,11 +335,9 @@
         protocol.childDataReceived(1, "hi")
         protocol.processEnded(Failure(ProcessDone(0)))
         self.manager.reactor.advance(501)
-        def got_error(f):
-            print f
-            self.assertTrue(f.check(ProcessTimeLimitReachedError))
-            self.assertEquals(f.value.data, "hi\n")
-        result.addErrback(got_error)
+        def got_result(output):
+            self.assertEquals(output, "hi")
+        result.addCallback(got_result)
         return result
 
     def test_script_is_owned_by_user(self):
@@ -348,7 +347,7 @@
         correct permissions. Therefore os.chmod and os.chown must be called
         before data is written.
         """
-        username = os.getlogin()
+        username = pwd.getpwuid(os.getuid())[0]
         uid, gid, home = get_user_info(username)
 
         mock_chown = self.mocker.replace("os.chown", passthrough=False)
@@ -474,7 +473,7 @@
 
     def test_user(self):
         """A user can be specified in the message."""
-        username = os.getlogin()
+        username = pwd.getpwuid(os.getuid())[0]
         uid, gid, home = get_user_info(username)
 
         # ignore the call to chown!
@@ -551,7 +550,7 @@
 
     def test_urgent_response(self):
         """Responses to script execution messages are urgent."""
-        username = os.getlogin()
+        username = pwd.getpwuid(os.getuid())[0]
         uid, gid, home = get_user_info(username)
 
         # ignore the call to chown!
@@ -585,6 +584,43 @@
         result.addCallback(got_result)
         return result
 
+    def test_binary_output(self):
+        """
+        If a script outputs non-printable characters not handled by utf-8, they
+        are replaced during the encoding phase but the script succeeds.
+        """
+        username = pwd.getpwuid(os.getuid())[0]
+        uid, gid, home = get_user_info(username)
+
+        mock_chown = self.mocker.replace("os.chown", passthrough=False)
+        mock_chown(ARGS)
+
+        def spawn_called(protocol, filename, uid, gid, path, env):
+            protocol.childDataReceived(1,
+            "\x7fELF\x01\x01\x01\x00\x00\x00\x95\x01")
+            protocol.processEnded(Failure(ProcessDone(0)))
+            self._verify_script(filename, sys.executable, "print 'hi'")
+        process_factory = self.mocker.mock()
+        process_factory.spawnProcess(
+            ANY, ANY, uid=uid, gid=gid, path=ANY,
+            env=get_default_environment())
+        self.mocker.call(spawn_called)
+
+        self.mocker.replay()
+
+        self.manager.add(ScriptExecutionPlugin(process_factory=process_factory))
+
+        def got_result(r):
+            self.assertTrue(self.broker_service.exchanger.is_urgent())
+            [message] = self.broker_service.message_store.get_pending_messages()
+            self.assertEquals(
+                message["result-text"],
+                 u"\x7fELF\x01\x01\x01\x00\x00\x00\ufffd\x01")
+
+        result = self._send_script(sys.executable, "print 'hi'")
+        result.addCallback(got_result)
+        return result
+
     def test_parse_error_causes_operation_failure(self):
         """
         If there is an error parsing the message, an operation-result will be
@@ -596,12 +632,20 @@
         self.manager.dispatch_message(
             {"type": "execute-script", "operation-id": 444})
 
+        if sys.version_info[:2] < (2, 6):
+            expected_message = [{"type": "operation-result",
+                                 "operation-id": 444,
+                                 "result-text": u"KeyError: 'username'",
+                                 "status": FAILED}]
+        else:
+            expected_message = [{"type": "operation-result",
+                                 "operation-id": 444,
+                                 "result-text": u"KeyError: username",
+                                 "status": FAILED}]
+
         self.assertMessages(
             self.broker_service.message_store.get_pending_messages(),
-            [{"type": "operation-result",
-              "operation-id": 444,
-              "result-text": u"KeyError: 'username'",
-              "status": FAILED}])
+            expected_message)
 
         self.assertTrue("KeyError: 'username'" in self.logfile.getvalue())
 

=== modified file 'landscape/message_schemas.py'
--- landscape/message_schemas.py	2009-03-18 20:42:05 +0000
+++ landscape/message_schemas.py	2009-09-21 16:05:18 +0000
@@ -324,6 +324,10 @@
 CUSTOM_GRAPH = Message("custom-graph", {
     "data": Dict(Int(), GRAPH_DATA)})
 
+REBOOT_REQUIRED = Message(
+    "reboot-required",
+    {"flag": Bool()})
+
 message_schemas = {}
 for schema in [ACTIVE_PROCESS_INFO, COMPUTER_UPTIME, CLIENT_UPTIME,
                OPERATION_RESULT, COMPUTER_INFO, DISTRIBUTION_INFO,
@@ -332,5 +336,6 @@
                REGISTER, REGISTER_CLOUD_VM, TEMPERATURE, PROCESSOR_INFO,
                USERS, PACKAGES,
                CHANGE_PACKAGES_RESULT, UNKNOWN_PACKAGE_HASHES,
-               ADD_PACKAGES, TEXT_MESSAGE, TEST, CUSTOM_GRAPH]:
+               ADD_PACKAGES, TEXT_MESSAGE, TEST, CUSTOM_GRAPH,
+               REBOOT_REQUIRED]:
     message_schemas[schema.type] = schema

=== modified file 'landscape/monitor/computerinfo.py'
--- landscape/monitor/computerinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/computerinfo.py	2009-09-21 16:05:18 +0000
@@ -13,10 +13,10 @@
 
     persist_name = "computer-info"
 
-    def __init__(self, get_hostname=socket.gethostname,
+    def __init__(self, get_fqdn=socket.getfqdn,
                  meminfo_file="/proc/meminfo",
                  lsb_release_filename="/etc/lsb-release"):
-        self._get_hostname = get_hostname
+        self._get_fqdn = get_fqdn
         self._meminfo_file = meminfo_file
         self._lsb_release_filename = lsb_release_filename
 
@@ -55,7 +55,7 @@
     def _create_computer_info_message(self):
         message = {}
         self._add_if_new(message, "hostname",
-                         self._get_hostname())
+                         self._get_fqdn())
         total_memory, total_swap = self._get_memory_info()
         self._add_if_new(message, "total-memory",
                          total_memory)

=== modified file 'landscape/monitor/deployment.py'
--- landscape/monitor/deployment.py	2008-12-11 17:11:08 +0000
+++ landscape/monitor/deployment.py	2009-09-21 16:05:18 +0000
@@ -15,7 +15,7 @@
 ALL_PLUGINS = ["ActiveProcessInfo", "ComputerInfo", "HardwareInventory",
                "LoadAverage", "MemoryInfo", "MountInfo", "ProcessorInfo",
                "Temperature", "PackageMonitor",
-               "UserMonitor"]
+               "UserMonitor", "RebootRequired"]
 
 
 class MonitorConfiguration(Configuration):
@@ -68,14 +68,13 @@
 
         # If this raises ServiceUnknownError, we should do something nice.
         self.remote_broker = RemoteBroker(self.bus)
+
         self.registry = MonitorPluginRegistry(self.remote_broker, self.reactor,
                                               self.config, self.bus,
                                               self.persist,
                                               self.persist_filename)
         self.dbus_service = MonitorDBusObject(self.bus, self.registry)
         DBusSignalToReactorTransmitter(self.bus, self.reactor)
-        self.remote_broker.register_plugin(self.dbus_service.bus_name,
-                                           self.dbus_service.object_path)
 
         for plugin in self.plugins:
             self.registry.add(plugin)
@@ -83,6 +82,14 @@
         self.flush_call_id = self.reactor.call_every(
             self.config.flush_interval, self.registry.flush)
 
+        def broker_started():
+            self.remote_broker.register_plugin(self.dbus_service.bus_name,
+                                               self.dbus_service.object_path)
+            self.registry.broker_started()
+
+        broker_started()
+        self.bus.add_signal_receiver(broker_started, "broker_started")
+
     def stopService(self):
         """Stop the monitor.
 

=== modified file 'landscape/monitor/monitor.py'
--- landscape/monitor/monitor.py	2008-12-11 17:11:08 +0000
+++ landscape/monitor/monitor.py	2009-09-21 16:05:18 +0000
@@ -73,11 +73,13 @@
             self._persist = registry.persist.root_at(self.persist_name)
 
     def call_on_accepted(self, type, callable, *args, **kwargs):
+
         def acceptance_changed(acceptance):
             if acceptance:
                 return callable(*args, **kwargs)
-        self.registry.reactor.call_on(("message-type-acceptance-changed", type),
-                                      acceptance_changed)
+
+        self.registry.reactor.call_on(("message-type-acceptance-changed",
+                                       type), acceptance_changed)
 
 
 class DataWatcher(MonitorPlugin):

=== modified file 'landscape/monitor/mountinfo.py'
--- landscape/monitor/mountinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/mountinfo.py	2009-09-21 16:05:18 +0000
@@ -26,6 +26,7 @@
         self._create_time = create_time
         self._free_space = []
         self._mount_info = []
+        self._mount_info_to_persist = None
         self._mount_activity = []
         self._prev_mount_activity = {}
         self._hal_manager = hal_manager or HALManager()
@@ -61,6 +62,7 @@
     def create_mount_info_message(self):
         if self._mount_info:
             message = {"type": "mount-info", "mount-info": self._mount_info}
+            self._mount_info_to_persist = self._mount_info[:]
             self._mount_info = []
             return message
         return None
@@ -74,12 +76,20 @@
 
     def send_messages(self, urgent=False):
         for message in self.create_messages():
-            self.registry.broker.send_message(message, urgent=urgent)
+            d = self.registry.broker.send_message(message, urgent=urgent)
+            if message["type"] == "mount-info":
+                d.addCallback(lambda x: self.persist_mount_info())
 
     def exchange(self):
         self.registry.broker.call_if_accepted("mount-info",
                                               self.send_messages)
 
+    def persist_mount_info(self):
+        for timestamp, mount_info in self._mount_info_to_persist:
+            mount_point = mount_info["mount-point"]
+            self._persist.set(("mount-info", mount_point), mount_info)
+        self._mount_info_to_persist = None
+
     def run(self):
         self._monitor.ping()
         now = int(self._create_time())
@@ -97,8 +107,8 @@
 
             prev_mount_info = self._persist.get(("mount-info", mount_point))
             if not prev_mount_info or prev_mount_info != mount_info:
-                self._persist.set(("mount-info", mount_point), mount_info)
-                self._mount_info.append((now, mount_info))
+                if mount_info not in [m for t, m in self._mount_info]:
+                    self._mount_info.append((now, mount_info))
 
             if not self._prev_mount_activity.get(mount_point, False):
                 self._mount_activity.append((now, mount_point, True))

=== modified file 'landscape/monitor/processorinfo.py'
--- landscape/monitor/processorinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/processorinfo.py	2009-09-21 16:05:18 +0000
@@ -148,6 +148,43 @@
         return processors
 
 
+class ARMMessageFactory:
+    """Factory for arm-based processors provides processor information."""
+
+    def __init__(self, source_filename):
+        """Initialize reader with filename of data source."""
+        self._source_filename = source_filename
+
+    def create_message(self):
+        """Returns a list containing information about each processor."""
+        processors = []
+        file = open(self._source_filename)
+
+        try:
+            regexp = re.compile("(?P<key>.*?)\s*:\s*(?P<value>.*)")
+            current = {}
+
+            for line in file:
+                match = regexp.match(line.strip())
+                if match:
+                    key = match.group("key")
+                    value = match.group("value")
+
+                    if key == "Processor":
+                        # ARM doesn't support SMP, thus no processor-id in
+                        # the cpuinfo
+                        current["processor-id"] = 0
+                        current["model"] =  value
+                    elif key == "Cache size":
+                        current["cache-size"] = int(value)
+
+            if current:
+                processors.append(current)
+        finally:
+            file.close()
+
+        return processors
+
 class SparcMessageFactory:
     """Factory for sparc-based processors provides processor information."""
 
@@ -216,6 +253,7 @@
         return processors
 
 
-message_factories = [("ppc(64)?", PowerPCMessageFactory),
+message_factories = [("arm*", ARMMessageFactory),
+                     ("ppc(64)?", PowerPCMessageFactory),
                      ("sparc[64]", SparcMessageFactory),
                      ("i[3-7]86|x86_64", X86MessageFactory)]

=== added file 'landscape/monitor/rebootrequired.py'
--- landscape/monitor/rebootrequired.py	1970-01-01 00:00:00 +0000
+++ landscape/monitor/rebootrequired.py	2009-09-21 16:05:18 +0000
@@ -0,0 +1,48 @@
+import os
+import logging
+
+from landscape.monitor.monitor import MonitorPlugin
+
+
+class RebootRequired(MonitorPlugin):
+    """
+    Report whether the system requires a reboot.
+    """
+
+    persist_name = "reboot-required"
+    run_interval = 900 # 15 minutes
+
+    def __init__(self, reboot_required_filename="/var/run/reboot-required"):
+        self._reboot_required_filename = reboot_required_filename
+
+    def _check_reboot_required(self):
+        """Return a boolean indicating whether the computer needs a reboot."""
+        return os.path.exists(self._reboot_required_filename)
+
+    def _create_message(self):
+        """Return the body of the reboot-required message to be sent."""
+        message = {}
+        key = "flag"
+        value = self._check_reboot_required()
+        if value != self._persist.get(key):
+            self._persist.set(key, value)
+            message[key] = value
+        return message
+
+    def send_message(self):
+        """Send a reboot-required message if needed.
+
+        A message will be sent only if the reboot-required status of the
+        system has changed.
+        """
+        message = self._create_message()
+        if message:
+            message["type"] = "reboot-required"
+            logging.info("Queueing message with updated "
+                         "reboot-required status.")
+            self.registry.broker.send_message(message)
+
+    def run(self):
+        """Send reboot-required messages if the server accepts them."""
+        return self.registry.broker.call_if_accepted(
+            "reboot-required", self.send_message)

=== modified file 'landscape/monitor/tests/test_computerinfo.py'
--- landscape/monitor/tests/test_computerinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/tests/test_computerinfo.py	2009-09-21 16:05:18 +0000
@@ -5,8 +5,8 @@
 from landscape.tests.mocker import ANY
 
 
-def get_hostname():
-    return "ooga"
+def get_fqdn():
+    return "ooga.local"
 
 
 class ComputerInfoTest(LandscapeTest):
@@ -48,16 +48,16 @@
 DISTRIB_DESCRIPTION="Ubuntu 6.06.1 LTS"
 """)
 
-    def test_get_hostname(self):
+    def test_get_fqdn(self):
         self.mstore.set_accepted_types(["computer-info"])
-        plugin = ComputerInfo(get_hostname=get_hostname,
+        plugin = ComputerInfo(get_fqdn=get_fqdn,
                               lsb_release_filename=self.lsb_release_filename)
         self.monitor.add(plugin)
         plugin.exchange()
         messages = self.mstore.get_pending_messages()
         self.assertEquals(len(messages), 1)
         self.assertEquals(messages[0]["type"], "computer-info")
-        self.assertEquals(messages[0]["hostname"], "ooga")
+        self.assertEquals(messages[0]["hostname"], "ooga.local")
 
     def test_get_real_hostname(self):
         self.mstore.set_accepted_types(["computer-info"])
@@ -72,7 +72,7 @@
 
     def test_only_report_changed_hostnames(self):
         self.mstore.set_accepted_types(["computer-info"])
-        plugin = ComputerInfo(get_hostname=get_hostname)
+        plugin = ComputerInfo(get_fqdn=get_fqdn)
         self.monitor.add(plugin)
         plugin.exchange()
         messages = self.mstore.get_pending_messages()
@@ -89,7 +89,7 @@
                 i = i + 1
 
         self.mstore.set_accepted_types(["computer-info"])
-        plugin = ComputerInfo(get_hostname=hostname_factory().next)
+        plugin = ComputerInfo(get_fqdn=hostname_factory().next)
         self.monitor.add(plugin)
 
         plugin.exchange()
@@ -270,14 +270,14 @@
         """
         self.mstore.set_accepted_types(["distribution-info", "computer-info"])
         meminfo_filename = self.make_path(self.sample_memory_info)
-        plugin = ComputerInfo(get_hostname=get_hostname,
+        plugin = ComputerInfo(get_fqdn=get_fqdn,
                               meminfo_file=meminfo_filename,
                               lsb_release_filename=self.lsb_release_filename)
         self.monitor.add(plugin)
         plugin.exchange()
         self.reactor.fire("resynchronize")
         plugin.exchange()
-        computer_info = {"type": "computer-info", "hostname": "ooga",
+        computer_info = {"type": "computer-info", "hostname": "ooga.local",
                          "timestamp": 0, "total-memory": 1510,
                          "total-swap": 1584}
         dist_info = {"type": "distribution-info",

=== modified file 'landscape/monitor/tests/test_deployment.py'
--- landscape/monitor/tests/test_deployment.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/tests/test_deployment.py	2009-09-21 16:05:18 +0000
@@ -8,6 +8,8 @@
 from landscape.broker.tests.test_remote import assertTransmitterActive
 from landscape.tests.test_plugin import assertReceivesMessages
 
+from twisted.internet.defer import Deferred
+
 
 class DeploymentTest(LandscapeTest):
 
@@ -27,7 +29,7 @@
                             "-d", self.make_path()])
         monitor = MonitorService(configuration)
         plugins = monitor.plugins
-        self.assertEquals(len(plugins), 10)
+        self.assertEquals(len(plugins), 11)
 
 
 class MonitorServiceTest(LandscapeIsolatedTest):
@@ -58,6 +60,43 @@
         return assertReceivesMessages(self, self.monitor.dbus_service,
                                       self.broker_service, self.remote)
 
+    def test_register_plugin_on_broker_started(self):
+        """
+        When the broker is restarted, it fires a "broker-started" signal which
+        makes the Monitor plugin register itself again.
+        """
+        d = Deferred()
+        def register_plugin(bus_name, object_path):
+            d.callback((bus_name, object_path))
+        def patch(ignore):
+            self.monitor.remote_broker.register_plugin = register_plugin
+            self.broker_service.dbus_object.broker_started()
+            return d
+        return self.remote.get_registered_plugins(
+            ).addCallback(patch
+            ).addCallback(self.assertEquals,
+                ("com.canonical.landscape.Monitor",
+                 "/com/canonical/landscape/Monitor"))
+
+    def test_register_message_on_broker_started(self):
+        """
+        When the broker is restarted, it fires a "broker-started" signal which
+        makes the Monitor plugin register all registered messages again.
+        """
+        self.monitor.registry.register_message("foo", lambda x: None)
+        d = Deferred()
+        def register_client_accepted_message_type(type):
+            if type == "foo":
+                d.callback(type)
+        def patch(ignore):
+            self.monitor.remote_broker.register_client_accepted_message_type = \
+                register_client_accepted_message_type
+            self.broker_service.dbus_object.broker_started()
+            return d
+        return self.remote.get_registered_plugins(
+            ).addCallback(patch
+            ).addCallback(self.assertEquals, "foo")
+
 
 class MonitorTest(MonitorServiceTest):
 

=== modified file 'landscape/monitor/tests/test_mountinfo.py'
--- landscape/monitor/tests/test_mountinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/tests/test_mountinfo.py	2009-09-21 16:05:18 +0000
@@ -1,5 +1,7 @@
 import tempfile
 
+from twisted.internet.defer import succeed
+
 from landscape.monitor.mountinfo import MountInfo
 from landscape.tests.test_hal import MockHALManager, MockRealHALDevice
 from landscape.tests.helpers import (LandscapeTest, MakePathHelper,
@@ -38,7 +40,7 @@
         """
         plugin = self.get_mount_info(create_time=self.reactor.time)
         self.monitor.add(plugin)
-        
+
         self.reactor.advance(self.monitor.step_size)
 
         message = plugin.create_mount_info_message()
@@ -675,8 +677,49 @@
 
         remote_broker_mock = self.mocker.replace(self.remote)
         remote_broker_mock.send_message(ANY, urgent=True)
+        self.mocker.result(succeed(None))
         self.mocker.count(3)
         self.mocker.replay()
 
         self.reactor.fire(("message-type-acceptance-changed", "mount-info"),
                           True)
+
+    def test_persist_timing(self):
+        """Mount info are only persisted when exchange happens.
+
+        Previously mount info were persisted as soon as they were gathered: if
+        an event happened between the persist and the exchange, the server
+        didn't get the mount info at all. This test ensures that mount info are
+        only saved when exchange happens.
+        """
+        def statvfs(path):
+            return (4096, 0, mb(1000), mb(100), 0, 0, 0, 0, 0)
+
+        filename = self.make_path("""\
+/dev/hda1 / ext3 rw 0 0
+""")
+        plugin = MountInfo(mounts_file=filename, create_time=self.reactor.time,
+                           statvfs=statvfs, mtab_file=filename)
+        self.monitor.add(plugin)
+        plugin.run()
+        message1 = plugin.create_mount_info_message()
+        self.assertEquals(
+            message1.get("mount-info"),
+            [(0, {"device": "/dev/hda1",
+                  "filesystem": "ext3",
+                  "mount-point": "/",
+                  "total-space": 4096000L})])
+        plugin.run()
+        message2 = plugin.create_mount_info_message()
+        self.assertEquals(
+            message2.get("mount-info"),
+            [(0, {"device": "/dev/hda1",
+                  "filesystem": "ext3",
+                  "mount-point": "/",
+                  "total-space": 4096000L})])
+        # Run again, calling create_mount_info_message purge the information
+        plugin.run()
+        plugin.exchange()
+        plugin.run()
+        message3 = plugin.create_mount_info_message()
+        self.assertIdentical(message3, None)

=== modified file 'landscape/monitor/tests/test_processorinfo.py'
--- landscape/monitor/tests/test_processorinfo.py	2008-09-08 16:35:57 +0000
+++ landscape/monitor/tests/test_processorinfo.py	2009-09-21 16:05:18 +0000
@@ -1,6 +1,7 @@
 from landscape.plugin import PluginConfigError
 from landscape.monitor.processorinfo import ProcessorInfo
-from landscape.tests.helpers import LandscapeTest, MakePathHelper, MonitorHelper
+from landscape.tests.helpers import (
+    LandscapeTest, MakePathHelper, MonitorHelper)
 from landscape.tests.mocker import ANY
 
 
@@ -58,7 +59,6 @@
         self.assertEquals(len(messages), 2)
 
 
-
 class PowerPCMessageTest(LandscapeTest):
     """Tests for powerpc-specific message builder."""
 
@@ -167,6 +167,131 @@
         self.assertEquals(processor["model"], "7447A, altivec supported")
 
 
+class ARMMessageTest(LandscapeTest):
+    """Tests for ARM-specific message builder."""
+
+    helpers = [MonitorHelper, MakePathHelper]
+
+    ARM_NOKIA = """
+Processor       : ARMv6-compatible processor rev 2 (v6l)
+BogoMIPS        : 164.36
+Features        : swp half thumb fastmult vfp edsp java
+CPU implementer : 0x41
+CPU architecture: 6TEJ
+CPU variant     : 0x0
+CPU part        : 0xb36
+CPU revision    : 2
+Cache type      : write-back
+Cache clean     : cp15 c7 ops
+Cache lockdown  : format C
+Cache format    : Harvard
+I size          : 32768
+I assoc         : 4
+I line length   : 32
+I sets          : 256
+D size          : 32768
+D assoc         : 4
+D line length   : 32
+D sets          : 256
+
+Hardware        : Nokia RX-44
+Revision        : 24202524
+Serial          : 0000000000000000
+"""
+
+    ARMv7 = """
+Processor       : ARMv7 Processor rev 1 (v7l)
+BogoMIPS        : 663.55
+Features        : swp half thumb fastmult vfp edsp
+CPU implementer : 0x41
+CPU architecture: 7
+CPU variant     : 0x2
+CPU part        : 0xc08
+CPU revision    : 1
+Cache type      : write-back
+Cache clean     : read-block
+Cache lockdown  : not supported
+Cache format    : Unified
+Cache size              : 768
+Cache assoc             : 1
+Cache line length       : 8
+Cache sets              : 64
+
+Hardware        : Sample Board
+Revision        : 81029
+Serial          : 0000000000000000
+"""
+
+    ARMv7_reverse = """
+Serial          : 0000000000000000
+Revision        : 81029
+Hardware        : Sample Board
+
+Cache sets              : 64
+Cache line length       : 8
+Cache assoc             : 1
+Cache size              : 768
+Cache format    : Unified
+Cache lockdown  : not supported
+Cache clean     : read-block
+Cache type      : write-back
+CPU revision    : 1
+CPU part        : 0xc08
+CPU variant     : 0x2
+CPU architecture: 7
+CPU implementer : 0x41
+Features        : swp half thumb fastmult vfp edsp
+BogoMIPS        : 663.55
+Processor       : ARMv7 Processor rev 1 (v7l)
+"""
+
+    def test_read_sample_nokia_data(self):
+        """Ensure the plugin can parse /proc/cpuinfo from a Nokia N810."""
+        filename = self.make_path(self.ARM_NOKIA)
+        plugin = ProcessorInfo(machine_name="armv6l",
+                               source_filename=filename)
+        message = plugin.create_message()
+        self.assertEquals(message["type"], "processor-info")
+        self.assertTrue(len(message["processors"]) == 1)
+
+        processor_0 = message["processors"][0]
+        self.assertEquals(len(processor_0), 2)
+        self.assertEquals(processor_0["model"],
+                          "ARMv6-compatible processor rev 2 (v6l)")
+        self.assertEquals(processor_0["processor-id"], 0)
+
+    def test_read_sample_armv7_data(self):
+        """Ensure the plugin can parse /proc/cpuinfo from a sample ARMv7."""
+        filename = self.make_path(self.ARMv7)
+        plugin = ProcessorInfo(machine_name="armv7l",
+                               source_filename=filename)
+        message = plugin.create_message()
+        self.assertEquals(message["type"], "processor-info")
+        self.assertTrue(len(message["processors"]) == 1)
+
+        processor_0 = message["processors"][0]
+        self.assertEquals(len(processor_0), 3)
+        self.assertEquals(processor_0["model"],
+                          "ARMv7 Processor rev 1 (v7l)")
+        self.assertEquals(processor_0["processor-id"], 0)
+        self.assertEquals(processor_0["cache-size"], 768)
+
+    def test_read_sample_armv7_reverse_data(self):
+        """Ensure the plugin can parse a reversed sample ARMv7 /proc/cpuinfo"""
+        filename = self.make_path(self.ARMv7_reverse)
+        plugin = ProcessorInfo(machine_name="armv7l",
+                               source_filename=filename)
+        message = plugin.create_message()
+        self.assertEquals(message["type"], "processor-info")
+        self.assertTrue(len(message["processors"]) == 1)
+
+        processor_0 = message["processors"][0]
+        self.assertEquals(len(processor_0), 3)
+        self.assertEquals(processor_0["model"],
+                          "ARMv7 Processor rev 1 (v7l)")
+        self.assertEquals(processor_0["processor-id"], 0)
+        self.assertEquals(processor_0["cache-size"], 768)
+
 class SparcMessageTest(LandscapeTest):
     """Tests for sparc-specific message builder."""
 
@@ -407,4 +532,3 @@
 
         self.mstore.set_accepted_types(["processor-info"])
         self.assertMessages(list(self.mstore.get_pending_messages()), [])
-

=== added file 'landscape/monitor/tests/test_rebootrequired.py'
--- landscape/monitor/tests/test_rebootrequired.py	1970-01-01 00:00:00 +0000
+++ landscape/monitor/tests/test_rebootrequired.py	2009-09-21 16:05:18 +0000
@@ -0,0 +1,62 @@
+import os
+
+from landscape.monitor.rebootrequired import RebootRequired
+from landscape.tests.helpers import LandscapeIsolatedTest
+from landscape.tests.helpers import (
+    MakePathHelper, MonitorHelper, LogKeeperHelper)
+
+
+class RebootRequiredTest(LandscapeIsolatedTest):
+
+    helpers = [MakePathHelper, MonitorHelper, LogKeeperHelper]
+
+    def setUp(self):
+        super(RebootRequiredTest, self).setUp()
+        self.reboot_required_filename = self.make_path("")
+        self.plugin = RebootRequired(self.reboot_required_filename)
+        self.monitor.add(self.plugin)
+        self.mstore.set_accepted_types(["reboot-required"])
+
+    def test_wb_check_reboot_required(self):
+        """
+        L{RebootRequired.check_reboot_required} should return C{True} if the
+        reboot-required flag file is present, C{False} otherwise.
+        """
+        self.assertTrue(self.plugin._check_reboot_required())
+        os.remove(self.reboot_required_filename)
+        self.assertFalse(self.plugin._check_reboot_required())
+
+    def test_wb_create_message(self):
+        """
+        A message should be created if and only if the reboot-required status
+        of the system has changed.
+        """
+        self.assertEquals(self.plugin._create_message(), {"flag": True})
+        self.assertEquals(self.plugin._create_message(), {})
+
+    def test_send_message(self):
+        """
+        A new C{"reboot-required"} message should be enqueued if and only
+        if the reboot-required status of the system has changed.
+        """
+        self.plugin.send_message()
+        self.assertIn("Queueing message with updated reboot-required status.",
+                      self.logfile.getvalue())
+        self.assertMessages(self.mstore.get_pending_messages(),
+                            [{"type": "reboot-required", "flag": True}])
+        self.mstore.delete_all_messages()
+        self.plugin.send_message()
+        self.assertMessages(self.mstore.get_pending_messages(), [])
+
+    def test_run(self):
+        """
+        If the server can accept them, the plugin should send
+        C{reboot-required} messages.
+        """
+        mock_plugin = self.mocker.patch(self.plugin)
+        mock_plugin.send_message()
+        self.mocker.count(1)
+        self.mocker.replay()
+        self.plugin.run()
+        self.mstore.set_accepted_types([])
+        self.plugin.run()

=== modified file 'landscape/package/changer.py'
--- landscape/package/changer.py	2009-04-09 17:09:50 +0000
+++ landscape/package/changer.py	2009-09-21 16:05:18 +0000
@@ -3,8 +3,9 @@
 import sys
 import os
 import pwd
+import grp
 
-from twisted.internet.defer import succeed, fail
+from twisted.internet.defer import fail
 
 from landscape.package.reporter import find_reporter_command
 from landscape.package.taskhandler import PackageTaskHandler, run_task_handler
@@ -42,14 +43,24 @@
 
     def run(self):
         task1 = self._store.get_next_task(self.queue_name)
-        result = super(PackageChanger, self).run()
+
         def finished(result):
             task2 = self._store.get_next_task(self.queue_name)
             if task1 and task1.id != (task2 and task2.id):
+                # In order to let the reporter run smart-update cleanly,
+                # we have to deinitialize Smart, so that the write lock
+                # gets released
+                self._facade.deinit()
                 if os.getuid() == 0:
+                    os.setgid(grp.getgrnam("landscape").gr_gid)
                     os.setuid(pwd.getpwnam("landscape").pw_uid)
                 os.system(find_reporter_command())
-        return result.addCallback(finished)
+
+        result = self.use_hash_id_db()
+        result.addCallback(lambda x: self.handle_tasks())
+        result.addCallback(finished)
+
+        return result
 
     def handle_tasks(self):
         result = super(PackageChanger, self).handle_tasks()

=== modified file 'landscape/package/facade.py'
--- landscape/package/facade.py	2009-04-09 17:09:50 +0000
+++ landscape/package/facade.py	2009-09-21 16:05:18 +0000
@@ -1,5 +1,5 @@
 from smart.transaction import Transaction, PolicyInstall, PolicyUpgrade, Failed
-from smart.const import INSTALL, REMOVE, UPGRADE
+from smart.const import INSTALL, REMOVE, UPGRADE, ALWAYS, NEVER
 
 import smart
 
@@ -23,6 +23,10 @@
     """Raised when Smart fails in an undefined way."""
 
 
+class ChannelError(Exception):
+    """Raised when channels fail to load."""
+
+
 class SmartFacade(object):
     """Wrapper for tasks using Smart.
 
@@ -33,6 +37,10 @@
     _deb_package_type = None
 
     def __init__(self, smart_init_kwargs={}):
+        """
+        @param smart_init_kwargs: A dictionary that can be used to pass
+            specific keyword parameters to to L{smart.init}.
+        """
         self._smart_init_kwargs = smart_init_kwargs.copy()
         self._smart_init_kwargs.setdefault("interface", "landscape")
         self._reset()
@@ -46,6 +54,8 @@
 
         self._marks = {}
 
+        self._caching = ALWAYS
+
     def deinit(self):
         """Deinitialize the Facade and the Smart library."""
         if self._ctrl:
@@ -59,6 +69,8 @@
                     install_landscape_interface)
                 install_landscape_interface()
             self._ctrl = smart.init(**self._smart_init_kwargs)
+            smart.initDistro(self._ctrl)
+            smart.initPlugins()
             smart.sysconf.set("pm-iface-output", True, soft=True)
             smart.sysconf.set("deb-non-interactive", True, soft=True)
 
@@ -74,10 +86,18 @@
         """Hook called when the Smart library is initialized."""
 
     def reload_channels(self):
-        """Reload Smart channels, getting all the cache (packages) in memory.
+        """
+        Reload Smart channels, getting all the cache (packages) in memory.
+
+        @raise: L{ChannelError} if Smart fails to reload the channels.
         """
         ctrl = self._get_ctrl()
-        ctrl.reloadChannels()
+
+        reload_result = ctrl.reloadChannels(caching=self._caching)
+        if not reload_result and self._caching == NEVER:
+            # Raise an error only if we are trying to download remote lists
+            raise ChannelError("Smart failed to reload channels (%s)"
+                               % smart.sysconf.get("channels"))
 
         self._hash2pkg.clear()
         self._pkg2hash.clear()
@@ -101,23 +121,46 @@
         @param with_info: If True, the skeleton will include information
             useful for sending data to the server.  Such information isn't
             necessary if the skeleton will be used to build a hash.
+
+        @return: a L{PackageSkeleton} object.
         """
         return build_skeleton(pkg, with_info)
 
     def get_package_hash(self, pkg):
-        """Return a hash from the given package."""
+        """Return a hash from the given package.
+
+        @param pkg: a L{smart.backends.deb.base.DebPackage} objects
+        """
         return self._pkg2hash.get(pkg)
 
+    def get_package_hashes(self):
+        """Get the hashes of all the packages available in the channels."""
+        return self._pkg2hash.values()
+
     def get_packages(self):
+        """
+        Get all the packages available in the channels.
+
+        @return: a C{list} of L{smart.backends.deb.base.DebPackage} objects
+        """
         return [pkg for pkg in self._get_ctrl().getCache().getPackages()
                 if isinstance(pkg, self._deb_package_type)]
 
     def get_packages_by_name(self, name):
-        """Return a list with all known (available) packages."""
+        """
+        Get all available packages matching the provided name.
+
+        @return: a C{list} of L{smart.backends.deb.base.DebPackage} objects
+        """
         return [pkg for pkg in self._get_ctrl().getCache().getPackages(name)
                 if isinstance(pkg, self._deb_package_type)]
 
     def get_package_by_hash(self, hash):
+        """
+        Get all available packages matching the provided hash.
+
+        @return: a C{list} of L{smart.backends.deb.base.DebPackage} objects
+        """
         return self._hash2pkg.get(hash)
 
     def mark_install(self, pkg):
@@ -181,3 +224,77 @@
         cache = self._get_ctrl().getCache()
         cache.reset()
         cache.load()
+
+    def get_arch(self):
+        """
+        Get the host dpkg architecture.
+        """
+        self._get_ctrl()
+        from smart.backends.deb.loader import DEBARCH
+        return DEBARCH
+
+    def set_arch(self, arch):
+        """
+        Set the host dpkg architecture.
+
+        To take effect it must be called before L{reload_channels}.
+
+        @param arch: the dpkg architecture to use (e.g. C{"i386"})
+        """
+        self._get_ctrl()
+        smart.sysconf.set("deb-arch", arch)
+
+        # XXX workaround Smart setting DEBARCH statically in the
+        # smart.backends.deb.base module
+        import smart.backends.deb.loader as loader
+        loader.DEBARCH = arch
+
+    def set_caching(self, mode):
+        """
+        Set Smart's caching mode.
+
+        @param mode: The caching mode to pass to Smart's C{reloadChannels}
+            when calling L{reload_channels} (e.g C{smart.const.NEVER} or
+            C{smart.const.ALWAYS}).
+        """
+        self._caching = mode
+
+    def reset_channels(self):
+        """Remove all configured Smart channels."""
+        self._get_ctrl()
+        smart.sysconf.set("channels", {}, soft=True)
+
+    def add_channel(self, alias, channel):
+        """
+        Add a Smart channel.
+
+        This method can be called more than once to set multiple channels.
+        To take effect it must be called before L{reload_channels}.
+
+        @param alias: A string identifying the channel to be added.
+        @param channel: A C{dict} holding information about the channel to
+            add (see the Smart API for details about valid keys and values).
+        """
+        channels = self.get_channels()
+        channels.update({alias: channel})
+        smart.sysconf.set("channels", channels, soft=True)
+
+    def get_channels(self):
+        """
+        @return: A C{dict} of all configured channels.
+        """
+        self._get_ctrl()
+        return smart.sysconf.get("channels")
+
+
+def make_apt_deb_channel(baseurl, distribution, components):
+    """Convenience to create Smart channels of type C{"apt-deb"}."""
+    return {"baseurl": baseurl,
+            "distribution": distribution,
+            "components": components,
+            "type": "apt-deb"}
+
+def make_deb_dir_channel(path):
+    """Convenience to create Smart channels of type C{"deb-dir"}."""
+    return {"path": path,
+            "type": "deb-dir"}

=== modified file 'landscape/package/reporter.py'
--- landscape/package/reporter.py	2008-09-08 16:35:57 +0000
+++ landscape/package/reporter.py	2009-09-21 16:05:18 +0000
@@ -1,12 +1,15 @@
+import urlparse
 import logging
 import time
 import sys
 import os
 
 from twisted.internet.defer import Deferred, succeed
+from twisted.internet.utils import getProcessOutputAndValue
 
 from landscape.lib.sequenceranges import sequence_to_ranges
 from landscape.lib.twisted_util import gather_results
+from landscape.lib.fetch import fetch_async
 
 from landscape.package.taskhandler import PackageTaskHandler, run_task_handler
 from landscape.package.store import UnknownHashIDRequest
@@ -17,13 +20,30 @@
 
 
 class PackageReporter(PackageTaskHandler):
+    """Report information about the system packages.
 
+    @cvar queue_name: Name of the task queue to pick tasks from.
+    @cvar smart_update_interval: Time interval in minutes to pass to
+        the C{--after} command line option of C{smart-update}.
+    """
     queue_name = "reporter"
+    smart_update_interval = 60
+    smart_update_filename = "/usr/lib/landscape/smart-update"
 
     def run(self):
         result = Deferred()
 
-        # First, handle any queued tasks.
+        # Run smart-update before anything else, to make sure that
+        # the SmartFacade will load freshly updated channels
+        result.addCallback(lambda x: self.run_smart_update())
+
+        # If the appropriate hash=>id db is not there, fetch it
+        result.addCallback(lambda x: self.fetch_hash_id_db())
+
+        # Attach the hash=>id database if available
+        result.addCallback(lambda x: self.use_hash_id_db())
+
+        # Now, handle any queued tasks.
         result.addCallback(lambda x: self.handle_tasks())
 
         # Then, remove any expired hash=>id translation requests.
@@ -38,6 +58,105 @@
         result.callback(None)
         return result
 
+    def fetch_hash_id_db(self):
+        """
+        Fetch the appropriate pre-canned database of hash=>id mappings
+        from the server. If the database is already present, it won't
+        be downloaded twice.
+
+        The format of the database filename is <uuid>_<codename>_<arch>,
+        and it will be downloaded from the HTTP directory set in
+        config.package_hash_id_url, or config.url/hash-id-databases if
+        the former is not set.
+
+        Fetch failures are handled gracefully and logged as appropriate.
+        """
+
+        def fetch_it(hash_id_db_filename):
+
+            if hash_id_db_filename is None:
+                # Couldn't determine which hash=>id database to fetch,
+                # just ignore the failure and go on
+                return
+
+            if os.path.exists(hash_id_db_filename):
+                # We don't download twice
+                return
+
+            base_url = self._get_hash_id_db_base_url()
+            if not base_url:
+                logging.warning("Can't determine the hash=>id database url")
+                return
+
+            # Cast to str as pycurl doesn't like unicode
+            url = str(base_url + os.path.basename(hash_id_db_filename))
+
+            def fetch_ok(data):
+                hash_id_db_fd = open(hash_id_db_filename, "w")
+                hash_id_db_fd.write(data)
+                hash_id_db_fd.close()
+                logging.info("Downloaded hash=>id database from %s" % url)
+
+            def fetch_error(failure):
+                exception = failure.value
+                logging.warning("Couldn't download hash=>id database: %s" %
+                                str(exception))
+
+            result = fetch_async(url)
+            result.addCallback(fetch_ok)
+            result.addErrback(fetch_error)
+
+            return result
+
+        result = self._determine_hash_id_db_filename()
+        result.addCallback(fetch_it)
+        return result
+
+    def _get_hash_id_db_base_url(self):
+
+        base_url = self._config.get("package_hash_id_url")
+
+        if not base_url:
+
+            if not self._config.get("url"):
+                # We really have no idea where to download from
+                return None
+
+            # If config.url is http://host:123/path/to/message-system
+            # then we'll use http://host:123/path/to/hash-id-databases
+            base_url = urlparse.urljoin(self._config.url.rstrip("/"),
+                                        "hash-id-databases")
+
+        return base_url.rstrip("/") + "/"
+
+    def run_smart_update(self):
+        """Run smart-update and log a warning in case of non-zero exit code.
+
+        @return: a deferred returning (out, err, code)
+        """
+        result = getProcessOutputAndValue(self.smart_update_filename,
+                                          args=("--after", "%d" %
+                                                self.smart_update_interval))
+
+        def callback((out, err, code)):
+            # smart-update --after N will exit with error code 1 when it
+            # doesn't actually run the update code because to enough time
+            # has passed yet, but we don't actually consider it a failure.
+            smart_failed = False
+            if code != 0 and code != 1:
+                smart_failed = True
+            if code == 1 and err.strip() != "":
+                smart_failed = True
+            if smart_failed:
+                logging.warning("'%s' exited with status %d (%s)" % (
+                    self.smart_update_filename, code, err))
+            logging.debug("'%s' exited with status %d (out='%s', err='%s'" % (
+                self.smart_update_filename, code, out, err))
+            return (out, err, code)
+
+        result.addCallback(callback)
+        return result
+
     def handle_task(self, task):
         message = task.data
         if message["type"] == "package-ids":
@@ -82,7 +201,23 @@
         self._store.clear_available()
         self._store.clear_available_upgrades()
         self._store.clear_installed()
-        self._store.clear_hash_id_requests()
+
+        # Don't clear the hash_id_requests table because the messages
+        # associated with the existing requests might still have to be
+        # delivered, and if we clear the table and later create a new request,
+        # that new request could get the same id of one of the deleted ones,
+        # and when the pending message eventually gets delivered the reporter
+        # would think that the message is associated to the newly created
+        # request, as it have the same id has the deleted request the message
+        # actually refers to. This would cause the ids in the message to be
+        # possibly mapped to the wrong hashes.
+        #
+        # This problem would happen for example when switching the client from
+        # one Landscape server to another, because the uuid-changed event would
+        # cause a resynchronize task to be created by the monitor. See #417122.
+        #
+        #self._store.clear_hash_id_requests()
+
         return succeed(None)
 
     def _handle_unknown_packages(self, hashes):

=== modified file 'landscape/package/skeleton.py'
--- landscape/package/skeleton.py	2008-09-08 16:35:57 +0000
+++ landscape/package/skeleton.py	2009-09-21 16:05:18 +0000
@@ -45,7 +45,7 @@
         return digest.digest()
 
 
-def build_skeleton(pkg, with_info=False):
+def build_skeleton(pkg, with_info=False, with_unicode=False):
     if not build_skeleton.inited:
         build_skeleton.inited = True
         global DebPackage, DebNameProvides, DebOrDepends
@@ -57,7 +57,11 @@
     if not isinstance(pkg, DebPackage):
         raise PackageTypeError()
 
-    skeleton = PackageSkeleton(DEB_PACKAGE, pkg.name, pkg.version)
+    if with_unicode:
+        skeleton = PackageSkeleton(DEB_PACKAGE, unicode(pkg.name),
+                                   unicode(pkg.version))
+    else:
+        skeleton = PackageSkeleton(DEB_PACKAGE, pkg.name, pkg.version)
     relations = set()
     for relation in pkg.provides:
         if isinstance(relation, DebNameProvides):

=== modified file 'landscape/package/store.py'
--- landscape/package/store.py	2009-03-18 20:42:05 +0000
+++ landscape/package/store.py	2009-09-21 16:05:18 +0000
@@ -1,5 +1,5 @@
+"""Provide access to the persistent data used by L{PackageTaskHandler}s."""
 import time
-import os
 
 try:
     import sqlite3
@@ -13,6 +13,10 @@
     """Raised for unknown hash id requests."""
 
 
+class InvalidHashIdDb(Exception):
+    """Raised when trying to add an invalid hash=>id lookaside database."""
+
+
 def with_cursor(method):
     """Decorator that encloses the method in a database transaction.
 
@@ -22,6 +26,7 @@
     the autocommit mode, we explicitly terminate transactions and enforce
     cursor closing with this decorator.
     """
+
     def inner(self, *args, **kwargs):
         try:
             cursor = self._db.cursor()
@@ -37,21 +42,34 @@
     return inner
 
 
-class PackageStore(object):
+class HashIdStore(object):
+    """Persist package hash=>id mappings in a file.
+
+    The implementation uses a SQLite database as backend, with a single
+    table called "hash", whose schema is defined in L{ensure_hash_id_schema}.
+    """
 
     def __init__(self, filename):
-        self._db = sqlite3.connect(filename)
-        ensure_schema(self._db)
+        """
+        @param filename: The file where the mappings are persisted to.
+        """
+        self._filename = filename
+        self._db = sqlite3.connect(self._filename)
+        ensure_hash_id_schema(self._db)
 
     @with_cursor
     def set_hash_ids(self, cursor, hash_ids):
+        """Set the ids of a set of hashes.
+
+        @param hash_ids: a C{dict} of hash=>id mappings.
+        """
         for hash, id in hash_ids.iteritems():
             cursor.execute("REPLACE INTO hash VALUES (?, ?)",
                            (id, buffer(hash)))
 
     @with_cursor
     def get_hash_id(self, cursor, hash):
-        assert isinstance(hash, basestring)
+        """Return the id associated to C{hash}, or C{None} if not available."""
         cursor.execute("SELECT id FROM hash WHERE hash=?", (buffer(hash),))
         value = cursor.fetchone()
         if value:
@@ -59,7 +77,14 @@
         return None
 
     @with_cursor
+    def get_hash_ids(self, cursor):
+        """Return a C{dict} holding all the available hash=>id mappings."""
+        cursor.execute("SELECT hash, id FROM hash")
+        return dict([(str(row[0]), row[1]) for row in cursor.fetchall()])
+
+    @with_cursor
     def get_id_hash(self, cursor, id):
+        """Return the hash associated to C{id}, or C{None} if not available."""
         assert isinstance(id, (int, long))
         cursor.execute("SELECT hash FROM hash WHERE id=?", (id,))
         value = cursor.fetchone()
@@ -69,9 +94,102 @@
 
     @with_cursor
     def clear_hash_ids(self, cursor):
+        """Delete all hash=>id mappings."""
         cursor.execute("DELETE FROM hash")
 
     @with_cursor
+    def check_sanity(self, cursor):
+        """Check database integrity.
+
+        @raise: L{InvalidHashIdDb} if the filenme passed to the constructor is
+            not a SQLite database or does not have a table called "hash" with
+            a compatible schema.
+        """
+        try:
+            cursor.execute("SELECT id FROM hash WHERE hash=?", ("",))
+        except sqlite3.DatabaseError, e:
+            raise InvalidHashIdDb(self._filename)
+
+
+class PackageStore(HashIdStore):
+    """Persist data about system packages and L{PackageTaskHandler}'s tasks.
+
+    This class extends L{HashIdStore} by adding tables to the SQLite database
+    backend for storing information about the status of the system packages and
+    about the tasks to be performed by L{PackageTaskHandler}s.
+
+    The additional tables and schemas are defined in L{ensure_package_schema}.
+    """
+
+    def __init__(self, filename):
+        """
+        @param filename: The file where data is persisted to.
+        """
+        super(PackageStore, self).__init__(filename)
+        self._hash_id_stores = []
+        ensure_package_schema(self._db)
+
+    def add_hash_id_db(self, filename):
+        """
+        Attach an additional "lookaside" hash=>id database.
+
+        This method can be called more than once to attach several
+        hash=>id databases, which will be queried *before* the main
+        database, in the same the order they were added.
+
+        If C{filename} is not a SQLite database or does not have a
+        table called "hash" with a compatible schema, L{InvalidHashIdDb}
+        is raised.
+
+        @param filename: a secondary SQLite databases to look for pre-canned
+                         hash=>id mappings.
+        """
+        hash_id_store = HashIdStore(filename)
+
+        try:
+            hash_id_store.check_sanity()
+        except InvalidHashIdDb, e:
+            # propagate the error
+            raise e
+
+        self._hash_id_stores.append(hash_id_store)
+
+    def has_hash_id_db(self):
+        """Return C{True} if one or more lookaside databases are attached."""
+        return len(self._hash_id_stores) > 0
+
+    def get_hash_id(self, hash):
+        """Return the id associated to C{hash}, or C{None} if not available.
+
+        This method composes the L{HashIdStore.get_hash_id} methods of all
+        the attached lookaside databases, falling back to the main one, as
+        described in L{add_hash_id_db}.
+        """
+        assert isinstance(hash, basestring)
+
+        # Check if we can find the hash=>id mapping in the lookaside stores
+        for store in self._hash_id_stores:
+            id = store.get_hash_id(hash)
+            if id:
+                return id
+
+        # Fall back to the locally-populated db
+        return HashIdStore.get_hash_id(self, hash)
+
+    def get_id_hash(self, id):
+        """Return the hash associated to C{id}, or C{None} if not available.
+
+        This method composes the L{HashIdStore.get_id_hash} methods of all
+        the attached lookaside databases, falling back to the main one in
+        case the hash associated to C{id} is not found in any of them.
+        """
+        for store in self._hash_id_stores:
+            hash = store.get_id_hash(id)
+            if hash is not None:
+                return hash
+        return HashIdStore.get_id_hash(self, id)
+
+    @with_cursor
     def add_available(self, cursor, ids):
         for id in ids:
             cursor.execute("REPLACE INTO available VALUES (?)", (id,))
@@ -244,15 +362,34 @@
         cursor.execute("DELETE FROM task WHERE id=?", (self.id,))
 
 
-def ensure_schema(db):
+def ensure_hash_id_schema(db):
+    """Create all tables needed by a L{HashIdStore}.
+
+    @param db: A connection to a SQLite database.
+    """
+    cursor = db.cursor()
+    try:
+        cursor.execute("CREATE TABLE hash"
+                       " (id INTEGER PRIMARY KEY, hash BLOB UNIQUE)")
+    except (sqlite3.OperationalError, sqlite3.DatabaseError):
+        cursor.close()
+        db.rollback()
+    else:
+        cursor.close()
+        db.commit()
+
+
+def ensure_package_schema(db):
+    """Create all tables needed by a L{PackageStore}.
+
+    @param db: A connection to a SQLite database.
+    """
     # FIXME This needs a "patch" table with a "version" column which will
     #       help with upgrades.  It should also be used to decide when to
     #       create the schema from the ground up, rather than that using
     #       try block.
     cursor = db.cursor()
     try:
-        cursor.execute("CREATE TABLE hash"
-                       " (id INTEGER PRIMARY KEY, hash BLOB UNIQUE)")
         cursor.execute("CREATE TABLE available"
                        " (id INTEGER PRIMARY KEY)")
         cursor.execute("CREATE TABLE available_upgrade"

=== modified file 'landscape/package/taskhandler.py'
--- landscape/package/taskhandler.py	2008-09-08 16:35:57 +0000
+++ landscape/package/taskhandler.py	2009-09-21 16:05:18 +0000
@@ -1,12 +1,14 @@
 import os
+import logging
 
 from twisted.internet.defer import Deferred, succeed
 
 from landscape.lib.dbus_util import get_bus
 from landscape.lib.lock import lock_path, LockError
 from landscape.lib.log import log_failure
+from landscape.lib.command import run_command, CommandError
 from landscape.deployment import Configuration, init_logging
-from landscape.package.store import PackageStore
+from landscape.package.store import PackageStore, InvalidHashIdDb
 from landscape.broker.remote import RemoteBroker
 
 
@@ -14,11 +16,13 @@
 
     queue_name = "default"
 
-    def __init__(self, package_store, package_facade, remote_broker):
+    def __init__(self, package_store, package_facade, remote_broker, config):
         self._store = package_store
         self._facade = package_facade
         self._broker = remote_broker
+        self._config = config
         self._channels_reloaded = False
+        self._server_uuid = None
 
     def ensure_channels_reloaded(self):
         if not self._channels_reloaded:
@@ -53,6 +57,79 @@
     def handle_task(self, task):
         return succeed(None)
 
+    def use_hash_id_db(self):
+        """
+        Attach the appropriate pre-canned hash=>id database to our store.
+        """
+
+        def use_it(hash_id_db_filename):
+
+            if hash_id_db_filename is None:
+                # Couldn't determine which hash=>id database to use,
+                # just ignore the failure and go on
+                return
+
+            if not os.path.exists(hash_id_db_filename):
+                # The appropriate database isn't there, but nevermind
+                # and just go on
+                return
+
+            try:
+                self._store.add_hash_id_db(hash_id_db_filename)
+            except InvalidHashIdDb:
+                # The appropriate database is there but broken,
+                # let's remove it and go on
+                logging.warning("Invalid hash=>id database %s" %
+                                hash_id_db_filename)
+                os.remove(hash_id_db_filename)
+                return
+
+        result = self._determine_hash_id_db_filename()
+        result.addCallback(use_it)
+        return result
+
+    def _determine_hash_id_db_filename(self):
+        """Build up the filename of the hash=>id database to use.
+
+        @return: a deferred resulting in the filename to use or C{None}
+            in case of errors.
+        """
+
+        def got_server_uuid(server_uuid):
+
+            warning = "Couldn't determine which hash=>id database to use: %s"
+
+            if server_uuid is None:
+                logging.warning(warning % "server UUID not available")
+                return None
+
+            try:
+                # XXX replace this with a L{SmartFacade} method
+                codename = run_command("lsb_release -cs")
+            except CommandError, error:
+                logging.warning(warning % str(error))
+                return None
+
+            arch = self._facade.get_arch()
+            if arch is None:
+                # The Smart code should always return a proper string, so this
+                # branch shouldn't get executed at all. However this check is
+                # kept as an extra paranoia sanity check.
+                logging.warning(warning % "unknown dpkg architecture")
+                return None
+
+            package_directory = os.path.join(self._config.data_path, "package")
+            hash_id_db_directory = os.path.join(package_directory, "hash-id")
+
+            return os.path.join(hash_id_db_directory,
+                                "%s_%s_%s" % (server_uuid,
+                                              codename,
+                                              arch))
+
+        result = self._broker.get_server_uuid()
+        result.addCallback(got_server_uuid)
+        return result
+
 
 def run_task_handler(cls, args, reactor=None):
     from twisted.internet.glib2reactor import install
@@ -69,8 +146,10 @@
     config.load(args)
 
     package_directory = os.path.join(config.data_path, "package")
-    if not os.path.isdir(package_directory):
-        os.mkdir(package_directory)
+    hash_id_directory = os.path.join(package_directory, "hash-id")
+    for directory in [package_directory, hash_id_directory]:
+        if not os.path.isdir(directory):
+            os.mkdir(directory)
 
     lock_filename = os.path.join(package_directory, program_name + ".lock")
     try:
@@ -98,7 +177,7 @@
     package_facade = SmartFacade()
     remote = RemoteBroker(get_bus(config.bus))
 
-    handler = cls(package_store, package_facade, remote)
+    handler = cls(package_store, package_facade, remote, config)
 
     def got_err(failure):
         log_failure(failure)

=== modified file 'landscape/package/tests/helpers.py'
--- landscape/package/tests/helpers.py	2008-09-08 16:35:57 +0000
+++ landscape/package/tests/helpers.py	2009-09-21 16:05:18 +0000
@@ -21,15 +21,15 @@
     def set_up(self, test_case):
         super(SmartFacadeHelper, self).set_up(test_case)
 
-        from landscape.package.facade import SmartFacade
+        from landscape.package.facade import SmartFacade, make_deb_dir_channel
 
         class Facade(SmartFacade):
             repository_dir = test_case.repository_dir
 
             def smart_initialized(self):
-                smart.sysconf.set("channels",
-                                  {"alias": {"type": "deb-dir",
-                                             "path": test_case.repository_dir}})
+                self.reset_channels()
+                self.add_channel("alias",
+                                 make_deb_dir_channel(test_case.repository_dir))
 
         test_case.Facade = Facade
         test_case.facade = Facade({"datadir": test_case.smart_dir})

=== added directory 'landscape/package/tests/repository'
=== added directory 'landscape/package/tests/repository/dists'
=== added directory 'landscape/package/tests/repository/dists/hardy'
=== added file 'landscape/package/tests/repository/dists/hardy/Release'
--- landscape/package/tests/repository/dists/hardy/Release	1970-01-01 00:00:00 +0000
+++ landscape/package/tests/repository/dists/hardy/Release	2009-09-21 16:05:18 +0000
@@ -0,0 +1,22 @@
+Origin: Ubuntu
+Label: Ubuntu
+Codename: hardy
+Version: 8.04
+Date: Mon, 30 Mar 2009 19:08:02 +0000
+Architectures: i386 amd64
+Components: main
+Description: Test Repository
+MD5Sum:
+ 356312bc1c0ab2b8dbe5c67f09879497 827 main/binary-i386/Packages
+ ad2d9b94381264ce25cda7cfa1b2da03 555 main/binary-i386/Packages.gz
+ 2335af6bede2d86bdf542595d75e0144 107 main/binary-i386/Release
+ f0fd5c1bb18584cf07f9bf4a9f2e6d92 605 main/binary-amd64/Packages
+ 98860034ca03a73a9face10af8238a81 407 main/binary-amd64/Packages.gz
+ 771950159a4e9c8bfca599f89f77c0e2 108 main/binary-amd64/Release
+SHA1:
+ 1f39494284f8da4a1cdd788a3d91a048c5edf7f5 827 main/binary-i386/Packages
+ e79a66d7543f24f77a9ffe1409431ae717781375 555 main/binary-i386/Packages.gz
+ 0d02cbecfee1760b855ec60ce8fc7f6a910fc196 107 main/binary-i386/Release
+ 37ba69be70f4a79506038c0124293187bc879014 605 main/binary-amd64/Packages
+ 65dca66c72b18d59cdcf671775104e86cbe2123a 407 main/binary-amd64/Packages.gz
+ cc42af4b0ec4b3e35cd3e1562fef6ef14b20d712 108 main/binary-amd64/Release

=== added directory 'landscape/package/tests/repository/dists/hardy/main'
=== added directory 'landscape/package/tests/repository/dists/hardy/main/binary-amd64'
=== added file 'landscape/package/tests/repository/dists/hardy/main/binary-amd64/Packages'
--- landscape/package/tests/repository/dists/hardy/main/binary-amd64/Packages	1970-01-01 00:00:00 +0000
+++ landscape/package/tests/repository/dists/hardy/main/binary-amd64/Packages	2009-09-21 16:05:18 +0000
@@ -0,0 +1,17 @@
+Package: libclthreads2
+Source: clthreads
+Version: 2.4.0-1
+Architecture: amd64
+Maintainer: Debian Multimedia Maintainers <[e-mail]>
+Installed-Size: 80
+Depends: libc6 (>= 2.3.2), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.1.1)
+Priority: extra
+Section: libs
+Filename: pool/main/c/clthreads/libclthreads2_2.4.0-1_amd64.deb
+Size: 12938
+SHA1: dc6cb78896642dd436851888b8bd4454ab8f421b
+MD5sum: 19960adb88e178fb7eb4997b47eee05b
+Description: POSIX threads C++ access library
+ C++ wrapper library around the POSIX threads API. This package includes
+ the shared library object.
+

=== added file 'landscape/package/tests/repository/dists/hardy/main/binary-amd64/Release'
--- landscape/package/tests/repository/dists/hardy/main/binary-amd64/Release	1970-01-01 00:00:00 +0000
+++ landscape/package/tests/repository/dists/hardy/main/binary-amd64/Release	2009-09-21 16:05:18 +0000
@@ -0,0 +1,6 @@
+Version: 8.04
+Component: main
+Origin: Ubuntu
+Label: Ubuntu
+Architecture: amd64
+Description: Test Repository

=== added directory 'landscape/package/tests/repository/dists/hardy/main/binary-i386'
=== added file 'landscape/package/tests/repository/dists/hardy/main/binary-i386/Packages'
--- landscape/package/tests/repository/dists/hardy/main/binary-i386/Packages	1970-01-01 00:00:00 +0000
+++ landscape/package/tests/repository/dists/hardy/main/binary-i386/Packages	2009-09-21 16:05:18 +0000
@@ -0,0 +1,20 @@
+Package: syslinux
+Version: 2:3.73+dfsg-2
+Architecture: i386
+Maintainer: Daniel Baumann <[e-mail]>
+Installed-Size: 140
+Depends: libc6 (>= 2.7-1), syslinux-common (= 2:3.73+dfsg-2), dosfstools, mtools
+Homepage: http://syslinux.zytor.com/
+Priority: optional
+Section: utils
+Filename: pool/main/s/syslinux/syslinux_3.73+dfsg-2_i386.deb
+Size: 70384
+SHA1: 6edf6a7e81a5e9759270872e45c782394dfa85e5
+MD5sum: ae8baa9f6c6a172a3b127af1e6675046
+Description: utilities for the syslinux bootloaders
+ SYSLINUX is a suite of lightweight bootloaders, currently supporting DOS FAT
+ filesystems (SYSLINUX), Linux ext2/ext3 filesystems (EXTLINUX), PXE network
+ booting (PXELINUX), or bootable "El Torito" ISO 9660 CD-ROMs (ISOLINUX). It
+ also includes a tool, MEMDISK, which loads legacy operating systems (such as
+ DOS) from these media.
+

=== added file 'landscape/package/tests/repository/dists/hardy/main/binary-i386/Release'
--- landscape/package/tests/repository/dists/hardy/main/binary-i386/Release	1970-01-01 00:00:00 +0000
+++ landscape/package/tests/repository/dists/hardy/main/binary-i386/Release	2009-09-21 16:05:18 +0000
@@ -0,0 +1,6 @@
+Version: 8.04
+Component: main
+Origin: Ubuntu
+Label: Ubuntu
+Architecture: i386
+Description: Test Repository

=== modified file 'landscape/package/tests/test_changer.py'
--- landscape/package/tests/test_changer.py	2008-09-08 16:35:57 +0000
+++ landscape/package/tests/test_changer.py	2009-09-21 16:05:18 +0000
@@ -7,14 +7,12 @@
 
 from smart.cache import Provides
 
-from landscape.lib.lock import lock_path
-
 from landscape.package.changer import (
     PackageChanger, main, find_changer_command, UNKNOWN_PACKAGE_DATA_TIMEOUT)
 from landscape.package.store import PackageStore
 from landscape.package.facade import (
-    SmartFacade, DependencyError, TransactionError, SmartError)
-from landscape.broker.remote import RemoteBroker
+    DependencyError, TransactionError, SmartError)
+from landscape.deployment import Configuration
 
 from landscape.tests.mocker import ANY
 from landscape.tests.helpers import (
@@ -31,7 +29,8 @@
         super(PackageChangerTest, self).setUp()
 
         self.store = PackageStore(self.makeFile())
-        self.changer = PackageChanger(self.store, self.facade, self.remote)
+        self.config = Configuration()
+        self.changer = PackageChanger(self.store, self.facade, self.remote, self.config)
 
         service = self.broker_service
         service.message_store.set_accepted_types(["change-packages-result"])
@@ -420,18 +419,40 @@
                               "REPORTER RUN")
         return result.addCallback(got_result)
 
-    def test_set_effective_uid_when_running_as_root(self):
+    def test_set_effective_uid_and_gid_when_running_as_root(self):
         """
         After the package changer has run, we want the package-reporter to run
         to report the recent changes.  If we're running as root, we want to
-        change to the "landscape" user.
+        change to the "landscape" user and "landscape" group. We also want to
+        deinitialize Smart to let the reporter run smart-update cleanly.
         """
+
         # We are running as root
         getuid_mock = self.mocker.replace("os.getuid")
         getuid_mock()
         self.mocker.result(0)
 
-        # We want to return a known uid
+        # The order matters (first smart then gid and finally uid)
+        self.mocker.order()
+
+        # Deinitialize smart
+        facade_mock = self.mocker.patch(self.facade)
+        facade_mock.deinit()
+
+        # We want to return a known gid
+        grnam_mock = self.mocker.replace("grp.getgrnam")
+        grnam_mock("landscape")
+
+        class FakeGroup(object):
+            gr_gid = 199
+
+        self.mocker.result(FakeGroup())
+
+        # First the changer should change the group
+        setgid_mock = self.mocker.replace("os.setgid")
+        setgid_mock(199)
+
+        # And a known uid as well
         pwnam_mock = self.mocker.replace("pwd.getpwnam")
         pwnam_mock("landscape")
 
@@ -440,11 +461,11 @@
 
         self.mocker.result(FakeUser())
 
-        # And now, the changer should change the user
+        # And now the user as well
         setuid_mock = self.mocker.replace("os.setuid")
         setuid_mock(199)
 
-        # Finally, we don't really want the package changer to run.
+        # Finally, we don't really want the package reporter to run.
         system_mock = self.mocker.replace("os.system")
         system_mock(ANY)
 
@@ -457,6 +478,29 @@
         return self.changer.run()
 
 
+    def test_run(self):
+        changer_mock = self.mocker.patch(self.changer)
+
+        self.mocker.order()
+
+        results = [Deferred() for i in range(2)]
+
+        changer_mock.use_hash_id_db()
+        self.mocker.result(results[0])
+
+        changer_mock.handle_tasks()
+        self.mocker.result(results[1])
+
+        self.mocker.replay()
+
+        self.changer.run()
+
+        # It must raise an error because deferreds weren't yet fired.
+        self.assertRaises(AssertionError, self.mocker.verify)
+
+        for deferred in reversed(results):
+            deferred.callback(None)
+
     def test_dont_spawn_reporter_after_running_if_nothing_done(self):
         output_filename = self.makeFile("REPORTER NOT RUN")
         reporter_filename = self.makeFile("#!/bin/sh\necho REPORTER RUN > %s" %
@@ -478,7 +522,6 @@
         return result.addCallback(got_result)
 
     def test_main(self):
-        data_path = self.makeDir()
         self.mocker.order()
         
         run_task_handler = self.mocker.replace("landscape.package.taskhandler"

=== modified file 'landscape/package/tests/test_facade.py'
--- landscape/package/tests/test_facade.py	2009-04-09 17:09:50 +0000
+++ landscape/package/tests/test_facade.py	2009-09-21 16:05:18 +0000
@@ -1,14 +1,21 @@
-import gdbm
 import time
 import os
+import re
+import sys
 
 from smart.control import Control
 from smart.cache import Provides
+from smart.const import NEVER
+
+from twisted.internet import reactor
+from twisted.internet.defer import Deferred
+from twisted.internet.utils import getProcessOutputAndValue
 
 import smart
 
 from landscape.package.facade import (
-    SmartFacade, TransactionError, DependencyError, SmartError)
+    TransactionError, DependencyError, ChannelError, SmartError,
+    make_apt_deb_channel, make_deb_dir_channel)
 
 from landscape.tests.mocker import ANY
 from landscape.tests.helpers import LandscapeTest
@@ -79,6 +86,11 @@
         pkg = self.facade.get_packages_by_name("name2")[0]
         self.assertEquals(self.facade.get_package_hash(pkg), HASH2)
 
+    def test_get_package_hashes(self):
+        self.facade.reload_channels()
+        hashes = self.facade.get_package_hashes()
+        self.assertEquals(sorted(hashes), sorted([HASH1, HASH2, HASH3]))
+
     def test_get_package_by_hash(self):
         self.facade.reload_channels()
         pkg = self.facade.get_package_by_hash(HASH1)
@@ -91,7 +103,6 @@
     def test_reload_channels_clears_hash_cache(self):
         # Load hashes.
         self.facade.reload_channels()
-        start = time.time()
 
         # Hold a reference to packages.
         pkg1 = self.facade.get_packages_by_name("name1")[0]
@@ -393,3 +404,104 @@
 
         self.facade.reload_channels()
         self.assertEquals(self.facade.get_package_hash(pkg), None)
+
+    def test_reset_add_get_channels(self):
+
+        channels = [("alias0", {"type": "test"}),
+                    ("alias1", {"type": "test"})]
+
+        self.facade.reset_channels()
+
+        self.assertEquals(self.facade.get_channels(), {})
+
+        self.facade.add_channel(*channels[0])
+        self.facade.add_channel(*channels[1])
+
+        self.assertEquals(self.facade.get_channels(), dict(channels))
+
+    def test_make_channels(self):
+
+        channel0 = make_apt_deb_channel("http://my.url/dir", "hardy", "main")
+        channel1 = make_deb_dir_channel("/my/repo")
+
+        self.assertEquals(channel0, {"baseurl": "http://my.url/dir",
+                                     "distribution": "hardy",
+                                     "components": "main",
+                                     "type": "apt-deb"})
+
+        self.assertEquals(channel1, {"path": "/my/repo",
+                                     "type": "deb-dir"})
+
+    def test_get_arch(self):
+        """
+        The L{SmartFacade.get_arch} should return the system dpkg
+        architecture.
+        """
+        deferred = Deferred()
+
+        def do_test():
+            result = getProcessOutputAndValue("/usr/bin/dpkg",
+                                              ("--print-architecture",))
+            def callback((out, err, code)):
+                self.assertEquals(self.facade.get_arch(), out.strip())
+            result.addCallback(callback)
+            result.chainDeferred(deferred)
+
+        reactor.callWhenRunning(do_test)
+        return deferred
+
+    def test_set_arch_multiple_times(self):
+
+        repository_dir = os.path.join(os.path.dirname(__file__), "repository")
+
+        alias = "alias"
+        channel = {"baseurl": "file://%s" % repository_dir,
+                   "distribution": "hardy",
+                   "components": "main",
+                   "type": "apt-deb"}
+
+        self.facade.set_arch("i386")
+        self.facade.reset_channels()
+        self.facade.add_channel(alias, channel)
+        self.facade.reload_channels()
+
+        pkgs = self.facade.get_packages()
+        self.assertEquals(len(pkgs), 1)
+        self.assertEquals(pkgs[0].name, "syslinux")
+
+        self.facade.deinit()
+        self.facade.set_arch("amd64")
+        self.facade.reset_channels()
+        self.facade.add_channel(alias, channel)
+        self.facade.reload_channels()
+
+        pkgs = self.facade.get_packages()
+        self.assertEquals(len(pkgs), 1)
+        self.assertEquals(pkgs[0].name, "libclthreads2")
+
+    def test_set_caching_with_reload_error(self):
+
+        alias = "alias"
+        channel = {"type": "deb-dir",
+                   "path": "/does/not/exist"}
+
+        self.facade.reset_channels()
+        self.facade.add_channel(alias, channel)
+        self.facade.set_caching(NEVER)
+
+        self.assertRaises(ChannelError, self.facade.reload_channels)
+        self.facade._channels = {}
+
+        ignore_re = re.compile("\[Smart\].*'alias'.*/does/not/exist")
+
+        self.log_helper.ignored_exception_regexes = [ignore_re]
+
+    def test_init_landscape_plugins(self):
+        """
+        The landscape plugin which helps managing proxies is loaded when smart
+        is initialized: this sets a smart configuration variable and load the
+        module.
+        """
+        self.facade.reload_channels()
+        self.assertTrue(smart.sysconf.get("use-landscape-proxies"))
+        self.assertIn("smart.plugins.landscape", sys.modules)

=== modified file 'landscape/package/tests/test_reporter.py'
--- landscape/package/tests/test_reporter.py	2008-09-08 16:35:57 +0000
+++ landscape/package/tests/test_reporter.py	2009-09-21 16:05:18 +0000
@@ -1,10 +1,13 @@
 import glob
 import sys
 import os
+import logging
 
 from twisted.internet.defer import Deferred
+from twisted.internet import reactor
 
-from landscape.lib.lock import lock_path
+from landscape.lib.fetch import fetch_async, FetchError
+from landscape.lib.command import CommandError
 
 from landscape.package.store import PackageStore, UnknownHashIDRequest
 from landscape.package.reporter import (
@@ -12,6 +15,7 @@
 from landscape.package import reporter
 from landscape.package.facade import SmartFacade
 
+from landscape.deployment import Configuration
 from landscape.broker.remote import RemoteBroker
 
 from landscape.package.tests.helpers import (
@@ -30,7 +34,8 @@
         super(PackageReporterTest, self).setUp()
 
         self.store = PackageStore(self.makeFile())
-        self.reporter = PackageReporter(self.store, self.facade, self.remote)
+        self.config = Configuration()
+        self.reporter = PackageReporter(self.store, self.facade, self.remote, self.config)
 
     def set_pkg2_upgrades_pkg1(self):
         previous = self.Facade.channels_reloaded
@@ -219,6 +224,375 @@
         deferred = self.reporter.handle_tasks()
         return deferred.addCallback(got_result)
 
+    def test_fetch_hash_id_db(self):
+
+        # Assume package_hash_id_url is set
+        self.config.data_path = self.makeDir()
+        self.config.package_hash_id_url = "http://fake.url/path/"
+        os.makedirs(os.path.join(self.config.data_path, "package", "hash-id"))
+        hash_id_db_filename = os.path.join(self.config.data_path, "package",
+                                           "hash-id", "uuid_codename_arch")
+
+        # Fake uuid, codename and arch
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+        self.facade.set_arch("arch")
+
+        # Let's say fetch_async is successful
+        hash_id_db_url = self.config.package_hash_id_url + "uuid_codename_arch"
+        fetch_async_mock = self.mocker.replace("landscape.lib.fetch.fetch_async")
+        fetch_async_mock(hash_id_db_url)
+        fetch_async_result = Deferred()
+        fetch_async_result.callback("hash-ids")
+        self.mocker.result(fetch_async_result)
+
+        # The download should be properly logged
+        logging_mock = self.mocker.replace("logging.info")
+        logging_mock("Downloaded hash=>id database from %s" % hash_id_db_url)
+        self.mocker.result(None)
+
+        # We don't have our hash=>id database yet
+        self.assertFalse(os.path.exists(hash_id_db_filename))
+
+        # Now go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        # Check the database
+        def callback(ignored):
+            self.assertTrue(os.path.exists(hash_id_db_filename))
+            self.assertEquals(open(hash_id_db_filename).read(), "hash-ids")
+        result.addCallback(callback)
+
+        return result
+
+    def test_fetch_hash_id_db_does_not_download_twice(self):
+
+        # Let's say that the hash=>id database is already there
+        self.config.package_hash_id_url = "http://fake.url/path/"
+        self.config.data_path = self.makeDir()
+        os.makedirs(os.path.join(self.config.data_path, "package", "hash-id"))
+        hash_id_db_filename = os.path.join(self.config.data_path, "package",
+                                           "hash-id", "uuid_codename_arch")
+        open(hash_id_db_filename, "w").write("test")
+
+        # Fake uuid, codename and arch
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+        self.facade.set_arch("arch")
+
+        # Intercept any call to fetch_async
+        fetch_async_mock = self.mocker.replace("landscape.lib.fetch.fetch_async")
+        fetch_async_mock(ANY)
+
+        # Go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        def callback(ignored):
+            # Check that fetch_async hasn't been called
+            self.assertRaises(AssertionError, self.mocker.verify)
+            fetch_async(None)
+
+            # The hash=>id database is still there
+            self.assertEquals(open(hash_id_db_filename).read(), "test")
+
+        result.addCallback(callback)
+
+        return result
+
+    def test_fetch_hash_id_db_undetermined_server_uuid(self):
+        """
+        If the server-uuid can't be determined for some reason, no download
+        should be attempted and the failure should be properly logged.
+        """
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid(None)
+
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("Couldn't determine which hash=>id database to use: "
+                     "server UUID not available")
+        self.mocker.result(None)
+        self.mocker.replay()
+
+        result = self.reporter.fetch_hash_id_db()
+        return result
+
+    def test_fetch_hash_id_db_undetermined_codename(self):
+
+        # Fake uuid
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+
+        # Undetermined codename
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        command_error = CommandError("lsb_release -cs", 1, "error")
+        self.mocker.throw(command_error)
+
+        # The failure should be properly logged
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("Couldn't determine which hash=>id database to use: %s" %
+                     str(command_error))
+        self.mocker.result(None)
+
+        # Go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        return result
+
+    def test_fetch_hash_id_db_undetermined_arch(self):
+
+        # Fake uuid and codename
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+
+        # Undetermined arch
+        self.facade.set_arch(None)
+
+        # The failure should be properly logged
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("Couldn't determine which hash=>id database to use: "\
+                     "unknown dpkg architecture")
+        self.mocker.result(None)
+
+        # Go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        return result
+
+    def test_fetch_hash_id_db_with_default_url(self):
+
+        # Let's say package_hash_id_url is not set but url is
+        self.config.data_path = self.makeDir()
+        self.config.package_hash_id_url = None
+        self.config.url = "http://fake.url/path/message-system/"
+        os.makedirs(os.path.join(self.config.data_path, "package", "hash-id"))
+        hash_id_db_filename = os.path.join(self.config.data_path, "package",
+                                           "hash-id", "uuid_codename_arch")
+
+        # Fake uuid, codename and arch
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+        self.facade.set_arch("arch")
+
+        # Check fetch_async is called with the default url
+        hash_id_db_url = "http://fake.url/path/hash-id-databases/" \
+                         "uuid_codename_arch"
+        fetch_async_mock = self.mocker.replace("landscape.lib.fetch.fetch_async")
+        fetch_async_mock(hash_id_db_url)
+        fetch_async_result = Deferred()
+        fetch_async_result.callback("hash-ids")
+        self.mocker.result(fetch_async_result)
+
+        # Now go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        # Check the database
+        def callback(ignored):
+            self.assertTrue(os.path.exists(hash_id_db_filename))
+            self.assertEquals(open(hash_id_db_filename).read(), "hash-ids")
+        result.addCallback(callback)
+        return result
+
+    def test_fetch_hash_id_db_with_download_error(self):
+
+        # Assume package_hash_id_url is set
+        self.config.data_path = self.makeDir()
+        self.config.package_hash_id_url = "http://fake.url/path/"
+
+        # Fake uuid, codename and arch
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+        self.facade.set_arch("arch")
+
+        # Let's say fetch_async fails
+        hash_id_db_url = self.config.package_hash_id_url + "uuid_codename_arch"
+        fetch_async_mock = self.mocker.replace("landscape.lib.fetch.fetch_async")
+        fetch_async_mock(hash_id_db_url)
+        fetch_async_result = Deferred()
+        fetch_async_result.errback(FetchError("fetch error"))
+        self.mocker.result(fetch_async_result)
+
+        # The failure should be properly logged
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("Couldn't download hash=>id database: fetch error")
+        self.mocker.result(None)
+
+        # Now go!
+        self.mocker.replay()
+        result = self.reporter.fetch_hash_id_db()
+
+        # We shouldn't have any hash=>id database
+        def callback(ignored):
+            hash_id_db_filename = os.path.join(self.config.data_path, "package",
+                                               "hash-id", "uuid_codename_arch")
+            self.assertEquals(os.path.exists(hash_id_db_filename), False)
+        result.addCallback(callback)
+
+        return result
+
+    def test_fetch_hash_id_db_with_undetermined_url(self):
+
+        # We don't know where to fetch the hash=>id database from
+        self.config.url = None
+        self.config.package_hash_id_url = None
+
+        # Fake uuid, codename and arch
+        message_store = self.broker_service.message_store
+        message_store.set_server_uuid("uuid")
+        command_mock = self.mocker.replace("landscape.lib.command.run_command")
+        command_mock("lsb_release -cs")
+        self.mocker.result("codename")
+        self.facade.set_arch("arch")
+
+        # The failure should be properly logged
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("Can't determine the hash=>id database url")
+        self.mocker.result(None)
+
+        # Let's go
+        self.mocker.replay()
+
+        result = self.reporter.fetch_hash_id_db()
+
+        # We shouldn't have any hash=>id database
+        def callback(ignored):
+            hash_id_db_filename = os.path.join(self.config.data_path, "package",
+                                               "hash-id", "uuid_codename_arch")
+            self.assertEquals(os.path.exists(hash_id_db_filename), False)
+        result.addCallback(callback)
+
+        return result
+
+    def test_run_smart_update(self):
+        """
+        The L{PackageReporter.run_smart_update} method should run smart-update
+        with the proper arguments.
+        """
+        self.reporter.smart_update_filename = self.makeFile(
+            "#!/bin/sh\necho -n $@")
+        os.chmod(self.reporter.smart_update_filename, 0755)
+        logging_mock = self.mocker.replace("logging.debug")
+        logging_mock("'%s' exited with status 0 (out='--after %d', err=''" % (
+            self.reporter.smart_update_filename,
+            self.reporter.smart_update_interval))
+        self.mocker.replay()
+        deferred = Deferred()
+
+        def do_test():
+            def raiseme(x):
+                raise Exception
+            logging.warning = raiseme
+            result = self.reporter.run_smart_update()
+            def callback((out, err, code)):
+                interval = self.reporter.smart_update_interval
+                self.assertEquals(out, "--after %d" % interval)
+                self.assertEquals(err, "")
+                self.assertEquals(code, 0)
+            result.addCallback(callback)
+            result.chainDeferred(deferred)
+
+        reactor.callWhenRunning(do_test)
+        return deferred
+
+    def test_run_smart_update_warns_about_failures(self):
+        """
+        The L{PackageReporter.run_smart_update} method should log a warning
+        in case smart-update terminates with a non-zero exit code other than 1.
+        """
+        self.reporter.smart_update_filename = self.makeFile(
+            "#!/bin/sh\necho -n error >&2\necho -n output\nexit 2")
+        os.chmod(self.reporter.smart_update_filename, 0755)
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("'%s' exited with status 2"
+                     " (error)" % self.reporter.smart_update_filename)
+        self.mocker.replay()
+        deferred = Deferred()
+
+        def do_test():
+            result = self.reporter.run_smart_update()
+            def callback((out, err, code)):
+                interval = self.reporter.smart_update_interval
+                self.assertEquals(out, "output")
+                self.assertEquals(err, "error")
+                self.assertEquals(code, 2)
+            result.addCallback(callback)
+            result.chainDeferred(deferred)
+
+        reactor.callWhenRunning(do_test)
+        return deferred
+
+    def test_run_smart_update_warns_exit_code_1_and_non_empty_stderr(self):
+        """
+        The L{PackageReporter.run_smart_update} method should log a warning
+        in case smart-update terminates with exit code 1 and non empty stderr.
+        """
+        self.reporter.smart_update_filename = self.makeFile(
+            "#!/bin/sh\necho -n \"error  \" >&2\nexit 1")
+        os.chmod(self.reporter.smart_update_filename, 0755)
+        logging_mock = self.mocker.replace("logging.warning")
+        logging_mock("'%s' exited with status 1"
+                     " (error  )" % self.reporter.smart_update_filename)
+        self.mocker.replay()
+        deferred = Deferred()
+        def do_test():
+            result = self.reporter.run_smart_update()
+            def callback((out, err, code)):
+                interval = self.reporter.smart_update_interval
+                self.assertEquals(out, "")
+                self.assertEquals(err, "error  ")
+                self.assertEquals(code, 1)
+            result.addCallback(callback)
+            result.chainDeferred(deferred)
+
+        reactor.callWhenRunning(do_test)
+        return deferred
+
+    def test_run_smart_update_ignores_exit_code_1_and_empty_output(self):
+        """
+        The L{PackageReporter.run_smart_update} method should not log anything
+        in case smart-update terminates with exit code 1 and output containing
+        only a newline character.
+        """
+        self.reporter.smart_update_filename = self.makeFile(
+            "#!/bin/sh\necho\nexit 1")
+        os.chmod(self.reporter.smart_update_filename, 0755)
+        deferred = Deferred()
+        def do_test():
+            def raiseme(x):
+                raise Exception
+            logging.warning = raiseme
+            result = self.reporter.run_smart_update()
+            def callback((out, err, code)):
+                interval = self.reporter.smart_update_interval
+                self.assertEquals(out, "\n")
+                self.assertEquals(err, "")
+                self.assertEquals(code, 1)
+            result.addCallback(callback)
+            result.chainDeferred(deferred)
+
+        reactor.callWhenRunning(do_test)
+        return deferred
+
     def test_remove_expired_hash_id_request(self):
         request = self.store.add_hash_id_request(["hash1"])
         request.message_id = 9999
@@ -569,19 +943,28 @@
 
         self.mocker.order()
 
-        results = [Deferred() for i in range(4)]
+        results = [Deferred() for i in range(7)]
+
+        reporter_mock.run_smart_update()
+        self.mocker.result(results[0])
+
+        reporter_mock.fetch_hash_id_db()
+        self.mocker.result(results[1])
+
+        reporter_mock.use_hash_id_db()
+        self.mocker.result(results[2])
 
         reporter_mock.handle_tasks()
-        self.mocker.result(results[0])
+        self.mocker.result(results[3])
 
         reporter_mock.remove_expired_hash_id_requests()
-        self.mocker.result(results[1])
+        self.mocker.result(results[4])
 
         reporter_mock.request_unknown_hashes()
-        self.mocker.result(results[2])
+        self.mocker.result(results[5])
 
         reporter_mock.detect_changes()
-        self.mocker.result(results[3])
+        self.mocker.result(results[6])
 
         self.mocker.replay()
 
@@ -625,7 +1008,8 @@
     def test_resynchronize(self):
         """
         When a resynchronize task arrives, the reporter should clear
-        out all the data in the package store, except the hash ids.
+        out all the data in the package store, except the hash ids and
+        the hash ids requests.
         This is done in the reporter so that we know it happens when
         no other reporter is possibly running at the same time.
         """
@@ -636,6 +1020,11 @@
         request1 = self.store.add_hash_id_request(["hash3"])
         request2 = self.store.add_hash_id_request(["hash4"])
 
+        # Set the message id to avoid the requests being deleted by the
+        # L{PackageReporter.remove_expired_hash_id_requests} method.
+        request1.message_id = 1
+        request2.message_id = 2
+
         # Let's make sure the data is there.
         self.assertEquals(self.store.get_available_upgrades(), [2])
         self.assertEquals(self.store.get_available(), [1])
@@ -643,7 +1032,7 @@
         self.assertEquals(self.store.get_hash_id_request(request1.id).id, request1.id)
 
         self.store.add_task("reporter", {"type": "resynchronize"})
-        
+
         deferred = self.reporter.run()
 
         def check_result(result):
@@ -659,13 +1048,22 @@
             self.assertEquals(self.store.get_available(), [3, 4])
             self.assertEquals(self.store.get_installed(), [])
 
-            # A New hash id request should also be detected for HASH3,
-            # but there should be no other hash id requests.
-            request = self.store.get_hash_id_request(request1.id)
-            self.assertEquals(request.id, request1.id)
-            self.assertEquals(request.hashes, [HASH3])
-            self.assertRaises(UnknownHashIDRequest,
-                              self.store.get_hash_id_request, request2.id)
+            # The two original hash id requests should be still there, and
+            # a new hash id request should also be detected for HASH3.
+            requests_count = 0
+            new_request_found = False
+            for request in self.store.iter_hash_id_requests():
+                requests_count += 1
+                if request.id == request1.id:
+                    self.assertEquals(request.hashes, ["hash3"])
+                elif request.id == request2.id:
+                    self.assertEquals(request.hashes, ["hash4"])
+                elif not new_request_found:
+                    self.assertEquals(request.hashes, [HASH3])
+                else:
+                    self.fail("Unexpected hash-id request!")
+            self.assertEquals(requests_count, 3)
+
         deferred.addCallback(check_result)
         return deferred
 

=== modified file 'landscape/package/tests/test_store.py'
--- landscape/package/tests/test_store.py	2009-03-18 20:42:05 +0000
+++ landscape/package/tests/test_store.py	2009-09-21 16:05:18 +0000
@@ -1,20 +1,35 @@
 import threading
 import time
-import os
+
+try:
+    import sqlite3
+except ImportError:
+    from pysqlite2 import dbapi2 as sqlite3
 
 from landscape.tests.helpers import LandscapeTest
 
-from landscape.package.store import PackageStore, UnknownHashIDRequest
-
-
-class PackageStoreTest(LandscapeTest):
+from landscape.package.store import (HashIdStore, PackageStore,
+                                     UnknownHashIDRequest, InvalidHashIdDb)
+
+
+class HashIdStoreTest(LandscapeTest):
 
     def setUp(self):
-        super(PackageStoreTest, self).setUp()
+        super(HashIdStoreTest, self).setUp()
 
         self.filename = self.makeFile()
-        self.store1 = PackageStore(self.filename)
-        self.store2 = PackageStore(self.filename)
+        self.store1 = HashIdStore(self.filename)
+        self.store2 = HashIdStore(self.filename)
+
+    def test_set_and_get_hash_id(self):
+        self.store1.set_hash_ids({"ha\x00sh1": 123, "ha\x00sh2": 456})
+        self.assertEquals(self.store1.get_hash_id("ha\x00sh1"), 123)
+        self.assertEquals(self.store1.get_hash_id("ha\x00sh2"), 456)
+
+    def test_get_hash_ids(self):
+        hash_ids = {"hash1": 123, "hash2": 456}
+        self.store1.set_hash_ids(hash_ids)
+        self.assertEquals(self.store1.get_hash_ids(), hash_ids)
 
     def test_wb_transactional_commits(self):
         mock_db = self.mocker.replace(self.store1._db)
@@ -28,11 +43,6 @@
         self.mocker.replay()
         self.assertRaises(Exception, self.store1.set_hash_ids, None)
 
-    def test_set_and_get_hash_id(self):
-        self.store1.set_hash_ids({"ha\x00sh1": 123, "ha\x00sh2": 456})
-        self.assertEquals(self.store2.get_hash_id("ha\x00sh1"), 123)
-        self.assertEquals(self.store2.get_hash_id("ha\x00sh2"), 456)
-
     def test_get_id_hash(self):
         self.store1.set_hash_ids({"hash1": 123, "hash2": 456})
         self.assertEquals(self.store2.get_id_hash(123), "hash1")
@@ -70,6 +80,115 @@
         self.assertTrue(time.time()-started < 5,
                         "Setting 20k hashes took more than 5 seconds.")
 
+    def test_check_sanity(self):
+
+        store_filename = self.makeFile()
+        db = sqlite3.connect(store_filename)
+        cursor = db.cursor()
+        cursor.execute("CREATE TABLE hash"
+                       " (junk INTEGER PRIMARY KEY, hash BLOB UNIQUE)")
+        cursor.close()
+        db.commit()
+
+        store = HashIdStore(store_filename)
+        self.assertRaises(InvalidHashIdDb, store.check_sanity)
+
+
+class PackageStoreTest(LandscapeTest):
+
+    def setUp(self):
+        super(PackageStoreTest, self).setUp()
+
+        self.filename = self.makeFile()
+        self.store1 = PackageStore(self.filename)
+        self.store2 = PackageStore(self.filename)
+
+    def test_has_hash_id_db(self):
+
+        self.assertFalse(self.store1.has_hash_id_db())
+
+        hash_id_db_filename = self.makeFile()
+        HashIdStore(hash_id_db_filename)
+        self.store1.add_hash_id_db(hash_id_db_filename)
+
+        self.assertTrue(self.store1.has_hash_id_db())
+
+    def test_add_hash_id_db_with_non_sqlite_file(self):
+
+        def junk_db_factory():
+            filename = self.makeFile()
+            open(filename, "w").write("junk")
+            return filename
+
+        def raiseme():
+            store_filename = junk_db_factory()
+            try:
+                self.store1.add_hash_id_db(store_filename)
+            except InvalidHashIdDb, e:
+                self.assertEqual(str(e), store_filename)
+            else:
+                self.fail()
+
+        raiseme()
+        self.assertFalse(self.store1.has_hash_id_db())
+
+    def test_add_hash_id_db_with_wrong_schema(self):
+
+        def non_compliant_db_factory():
+            filename = self.makeFile()
+            db = sqlite3.connect(filename)
+            cursor = db.cursor()
+            cursor.execute("CREATE TABLE hash"
+                           " (junk INTEGER PRIMARY KEY, hash BLOB UNIQUE)")
+            cursor.close()
+            db.commit()
+            return filename
+
+        self.assertRaises(InvalidHashIdDb, self.store1.add_hash_id_db,
+                          non_compliant_db_factory())
+        self.assertFalse(self.store1.has_hash_id_db())
+
+    def hash_id_db_factory(self, hash_ids):
+        filename = self.makeFile()
+        store = HashIdStore(filename)
+        store.set_hash_ids(hash_ids)
+        return filename
+
+    def test_get_hash_id_using_hash_id_dbs(self):
+
+
+        # Without hash=>id dbs
+        self.assertEquals(self.store1.get_hash_id("hash1"), None)
+        self.assertEquals(self.store1.get_hash_id("hash2"), None)
+
+        # This hash=>id will be overriden
+        self.store1.set_hash_ids({"hash1": 1})
+
+        # Add a couple of hash=>id dbs
+        self.store1.add_hash_id_db(self.hash_id_db_factory({"hash1": 2,
+                                                            "hash2": 3}))
+        self.store1.add_hash_id_db(self.hash_id_db_factory({"hash2": 4,
+                                                            "ha\x00sh1": 5}))
+
+        # Check look-up priorities and binary hashes
+        self.assertEquals(self.store1.get_hash_id("hash1"), 2)
+        self.assertEquals(self.store1.get_hash_id("hash2"), 3)
+        self.assertEquals(self.store1.get_hash_id("ha\x00sh1"), 5)
+
+    def test_get_id_hash_using_hash_id_db(self):
+        """
+        When lookaside hash->id dbs are used, L{get_id_hash} has
+        to query them first, falling back to the regular db in case
+        the desired mapping is not found.
+        """
+        self.store1.add_hash_id_db(self.hash_id_db_factory({"hash1": 123}))
+        self.store1.add_hash_id_db(self.hash_id_db_factory({"hash1": 999,
+                                                            "hash2": 456}))
+        self.store1.set_hash_ids({"hash3": 789})
+        self.assertEquals(self.store1.get_id_hash(123), "hash1")
+        self.assertEquals(self.store1.get_id_hash(456), "hash2")
+        self.assertEquals(self.store1.get_id_hash(789), "hash3")
+
     def test_add_and_get_available_packages(self):
         self.store1.add_available([1, 2])
         self.assertEquals(self.store2.get_available(), [1, 2])
@@ -326,7 +445,7 @@
                           self.store1.get_hash_id_request, request1.id)
         self.assertRaises(UnknownHashIDRequest,
                           self.store1.get_hash_id_request, request2.id)
-    
+
     def test_clear_tasks(self):
         data = {"answer": 42}
         task = self.store1.add_task("reporter", data)
@@ -388,4 +507,3 @@
             thread.join()
 
         self.assertEquals(error, [])
-

=== modified file 'landscape/package/tests/test_taskhandler.py'
--- landscape/package/tests/test_taskhandler.py	2008-09-08 16:35:57 +0000
+++ landscape/package/tests/test_taskhandler.py	2009-09-21 16:05:18 +0000
@@ -1,23 +1,21 @@
 import os
-import sys
-
-from cStringIO import StringIO
 
 from twisted.internet import reactor
 from twisted.internet.defer import Deferred, fail
 
 from landscape.lib.lock import lock_path
+from landscape.lib.command import CommandError
 
 from landscape.deployment import Configuration
 from landscape.broker.remote import RemoteBroker