Merge lp:~salgado/launchpad/kill-rogue-ec2-instances into lp:launchpad

[Guilherme Salgado]

[Guilherme Salgado (salgado)]

= Summary =

Today, for the second time in less than 6 months, I started working and
realized an 'ec2 land' I ran (last Friday) on behalf of a community
member was hung and thus failed to bring the ec2 instance down.

This is pretty bad because in this case the instance was left up for
more than 48 hours, so I thought of adding scheduling a shutdown for 8h
after we start the preparations to run the test suite. This is more than
twice the time our test suite takes to complete, and it won't happen for
demo instances.

[Jonathan Lange (jml)]

You're right, we need some sort of timeout, given that normal termination is unreliable. Ideally, the timeout would be based on inactivity, rather than total time, but I think this patch is a net win. Please land.

review: Approve

[Jonathan Lange (jml)]

Looks good to me.

review: Approve

=== modified file 'lib/devscripts/ec2test/builtins.py'
--- lib/devscripts/ec2test/builtins.py	2009-11-30 16:26:24 +0000
+++ lib/devscripts/ec2test/builtins.py	2010-02-08 13:41:20 +0000
@@ -301,7 +301,8 @@
             pqm_submit_location=pqm_submit_location,
             open_browser=open_browser, pqm_email=pqm_email,
             include_download_cache_changes=include_download_cache_changes,
-            instance=instance, launchpad_login=instance._launchpad_login)
+            instance=instance, launchpad_login=instance._launchpad_login,
+            timeout=480)
 
         instance.set_up_and_run(postmortem, not headless, runner.run_tests)
 

=== modified file 'lib/devscripts/ec2test/instance.py'
--- lib/devscripts/ec2test/instance.py	2009-11-27 07:24:49 +0000
+++ lib/devscripts/ec2test/instance.py	2010-02-08 13:41:20 +0000
@@ -129,7 +129,7 @@
 
 apt-get -y install launchpad-developer-dependencies apache2 apache2-mpm-worker
 
-# Creat the ec2test user, give them passwordless sudo.
+# Create the ec2test user, give them passwordless sudo.
 adduser --gecos "" --disabled-password ec2test
 echo 'ec2test\tALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
 

=== modified file 'lib/devscripts/ec2test/testrunner.py'
--- lib/devscripts/ec2test/testrunner.py	2009-12-01 22:53:47 +0000
+++ lib/devscripts/ec2test/testrunner.py	2010-02-08 13:41:20 +0000
@@ -115,10 +115,14 @@
                  pqm_submit_location=None,
                  open_browser=False, pqm_email=None,
                  include_download_cache_changes=None, instance=None,
-                 launchpad_login=None):
+                 launchpad_login=None,
+                 timeout=None):
         """Create a new EC2TestRunner.
 
-        This sets the following attributes:
+        :param timeout: Number of minutes before we force a shutdown. This is
+            useful because sometimes the normal instance termination might
+            fail.
+
           - original_branch
           - test_options
           - headless
@@ -129,6 +133,7 @@
           - email (after validating email capabilities)
           - image (after connecting to ec2)
           - file
+          - timeout
         """
         self.original_branch = branch
         self.test_options = test_options
@@ -137,6 +142,7 @@
         self.open_browser = open_browser
         self.file = file
         self._launchpad_login = launchpad_login
+        self.timeout = timeout
 
         trunk_specified = False
         trunk_branch = TRUNK_BRANCH
@@ -314,6 +320,10 @@
 
     def configure_system(self):
         user_connection = self._instance.connect()
+        if self.timeout is not None:
+            user_connection.perform(
+                "echo sudo shutdown -h now | at today + %d minutes"
+                % self.timeout)
         as_user = user_connection.perform
         # Set up bazaar.conf with smtp information if necessary
         if self.email or self.message: