Merge lp:~mbp/launchpad/800295-buildd-split into lp:launchpad

[Martin Pool]

[r=mbp][bug=800295] [no-qa] prune dependencies between buildd and the rest of Launchpad

This is a partial fix for bug 800295, splitting all the files shared between buildd and lp proper into one or the other.

Beyond this we might like to split them in to separate trees, but I thought this would be easier to review and test by itself.

[Robert Collins (lifeless)]

This will fail tests. the daemon is used and tested from the lp tree,
which is what readyservice is for.

[Martin Pool (mbp)]

I should probably move 'wait for the port' up to the base class.

[Julian Edwards (julian-edwards)]

I can't see where _hasDaemonStarted() is getting called from here. It's called in the whole tree from only one place in lib/canonical/launchpad/daemons/tachandler.py but I can't see how it's replacing the readyService in the buildd code.

Am I missing something or are you? :)

review: Needs Information

[Martin Pool (mbp)]

_hasDaemonStarted is called from the TestTacServer base class; I'm overriding it here so that the fixture decides the server is ready not when it writes to the log, but rather when the daemon starts listening on its port. That general concept could usefully go into the base class, with the subclass just defining the port.

This patch as it is passes all tests, and it removes the need to actually copy files from the lp tree into the buildd directory. It doesn't make it possible to run the buildd tests without the Launchpad tree, which would be useful to do to make it totally split out.

[Martin Pool (mbp)]

The current version of this does everything short of cutting things in to separate packages. The buildd code no longer (I think) depends on anything from the rest of lp except TacTestSetup.

The next steps are to split out TacTestSetup into one package, and then to split the buildd itself.

[Martin Pool (mbp)]

... I'd like to do those splits as separate mps because they'll be harder to reviews (since they will mostly just look like deletions.)

=== renamed file 'daemons/buildd-slave.tac' => 'lib/canonical/buildd/buildd-slave.tac'
--- daemons/buildd-slave.tac	2011-06-21 17:07:05 +0000
+++ lib/canonical/buildd/buildd-slave.tac	2011-11-04 01:08:26 +0000
@@ -1,12 +1,9 @@
 # Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-# XXX: JonathanLange 2011-06-21 bug=800295: The only modules in the Launchpad
-# tree that this is permitted to depend on are canonical.buildd and
-# canonical.launchpad.daemons.readyservice, since canonical.buildd is deployed
-# by copying lib/canonical/buildd,
-# lib/canonical/launchpad/daemons/readyservice.py and daemons/buildd-slave.tac
-# only.
+# CAUTION: The only modules in the Launchpad tree that this is permitted to
+# depend on are canonical.buildd, since buildds are deployed by copying that
+# directory only. (See also bug=800295.)
 
 # Buildd Slave implementation
 # XXX: dsilvers: 2005/01/21: Currently everything logged in the slave gets
@@ -19,7 +16,6 @@
     SourcePackageRecipeBuildManager)
 from canonical.buildd.translationtemplates import (
     TranslationTemplatesBuildManager)
-from canonical.launchpad.daemons import readyservice
 
 from twisted.web import server, resource, static
 from ConfigParser import SafeConfigParser
@@ -42,9 +38,6 @@
 application = service.Application('BuildDSlave')
 builddslaveService = service.IServiceCollection(application)
 
-# Service that announces when the daemon is ready
-readyservice.ReadyService().setServiceParent(builddslaveService)
-
 root = resource.Resource()
 root.putChild('rpc', slave)
 root.putChild('filecache', static.File(conf.get('slave', 'filecache')))

=== modified file 'lib/canonical/buildd/debian/changelog'
--- lib/canonical/buildd/debian/changelog	2011-10-17 13:12:47 +0000
+++ lib/canonical/buildd/debian/changelog	2011-11-04 01:08:26 +0000
@@ -1,3 +1,11 @@
+launchpad-buildd (82) hardy-cat; urgency=low
+
+  * Cut out readyservice from the buildds.  LP: #800295
+  * buildrecipe shows the bzr and bzr-builder versions.  LP: #884092
+  * buildrecipe shows bzr rusage.  LP: 884997
+
+ -- Martin Pool <[e-mail]>  Thu, 03 Nov 2011 17:11:25 +1100
+
 launchpad-buildd (81) hardy-cat; urgency=low
 
   * generate-translation-templates: switch to Python 2.7.

=== modified file 'lib/canonical/buildd/debian/rules'
--- lib/canonical/buildd/debian/rules	2010-10-20 18:43:29 +0000
+++ lib/canonical/buildd/debian/rules	2011-11-04 01:08:26 +0000
@@ -14,7 +14,6 @@
 target = debian/launchpad-buildd
 topdir = ../../..
 
-daemons = $(topdir)/daemons
 buildd = $(topdir)/lib/canonical/buildd
 
 targetshare = $(target)/usr/share/launchpad-buildd
@@ -45,8 +44,7 @@
 	# Do installs here
 	touch $(pytarget)/../launchpad/__init__.py
 	touch $(pytarget)/../launchpad/daemons/__init__.py
-	cp launchpad-files/readyservice.py $(pytarget)/../launchpad/daemons/
-	install -m644 launchpad-files/buildd-slave.tac $(targetshare)/buildd-slave.tac
+	install -m644 buildd-slave.tac $(targetshare)/buildd-slave.tac
 	cp -r pottery $(pytarget)
 	for pyfile in $(pyfiles); do \
 	  install -m644 ./$$pyfile $(pytarget)/$$pyfile; \
@@ -90,8 +88,6 @@
 	dh_clean
 
 prepare:
-	install -m644 $(daemons)/buildd-slave.tac launchpad-files/buildd-slave.tac
-	cp ../launchpad/daemons/readyservice.py launchpad-files/readyservice.py
 
 package: prepare
 	debuild -uc -us -S

=== modified file 'lib/canonical/buildd/tests/harness.py'
--- lib/canonical/buildd/tests/harness.py	2010-09-28 11:05:14 +0000
+++ lib/canonical/buildd/tests/harness.py	2011-11-04 01:08:26 +0000
@@ -1,9 +1,8 @@
-# Copyright 2009-2010 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 __metaclass__ = type
 __all__ = [
-    'BuildlogSecurityTests',
     'BuilddTestCase',
     ]
 
@@ -13,6 +12,7 @@
 from ConfigParser import SafeConfigParser
 
 import canonical
+import canonical.buildd
 
 from canonical.buildd.slave import BuildDSlave
 from canonical.launchpad.daemons.tachandler import TacTestSetup
@@ -119,8 +119,8 @@
     @property
     def tacfile(self):
         return os.path.abspath(os.path.join(
-            os.path.dirname(canonical.__file__), os.pardir, os.pardir,
-            'daemons/buildd-slave.tac'
+            os.path.dirname(canonical.buildd.__file__),
+            'buildd-slave.tac'
             ))
 
     @property
@@ -130,3 +130,11 @@
     @property
     def logfile(self):
         return '/var/tmp/build-slave.log'
+
+    def _hasDaemonStarted(self):
+        """Called by the superclass to check if the daemon is listening.
+
+        The slave is ready when it's accepting connections.
+        """
+        # This must match buildd-slave-test.conf.
+        return self._isPortListening('localhost', 8221)

=== modified file 'lib/canonical/launchpad/daemons/tachandler.py'
--- lib/canonical/launchpad/daemons/tachandler.py	2011-08-19 15:36:31 +0000
+++ lib/canonical/launchpad/daemons/tachandler.py	2011-11-04 01:08:26 +0000
@@ -11,7 +11,9 @@
     ]
 
 
+import errno
 import os
+import socket
 import subprocess
 import sys
 import time
@@ -115,6 +117,25 @@
         else:
             return False
 
+    def _isPortListening(self, host, port):
+        """True if a tcp port is accepting connections.
+
+        This can be used by subclasses overriding _hasDaemonStarted, if they
+        want to check the port is up rather than for the contents of the log
+        file.
+        """
+        try:
+            s = socket.socket()
+            s.settimeout(2.0)
+            s.connect((host, port))
+            s.close()
+            return True
+        except socket.error, e:
+            if e.errno == errno.ECONNREFUSED:
+                return False
+            else:
+                raise
+
     def _waitForDaemonStartup(self):
         """ Wait for the daemon to fully start.
 

=== renamed file 'lib/canonical/buildd/tests/test_generate_translation_templates.py' => 'lib/lp/translations/tests/test_generate_translation_templates.py'