Merge lp:~smoser/ubuntu-server-ec2-testing/fixes into lp:ubuntu-server-ec2-testing

[Scott Moser]

try harder to get terminated logs, actually terminate with boto 2.0

This adds the terminate_and_wait method to TestCaseExecutor. As seen at
[1], TidyUp would raise an exception and prevent the final retrieval of
logs. Without those final logs, there was no way to debug what went
wrong. We make tidyup use this and basically try harder to get logs
of the terminated instance.

In the "all is well" path, there should be no additional wait, as
terminate_and_wait will recognize that the instance is already terminated.
Also, collect-console will recognize that there already exists console
output, so we wont' overwrite it.

This also fixes a bug in the test suite when using boto 2.0. Natty's
version of boto a boto.ec2.instance.stop() would terminate an instance,
but that is obviously wrong. Both 2.0 and natty's version have a
'terminate' method, so we use that.

--
[1] https://jenkins.qa.ubuntu.com/job/oneiric-server-ec2/10/ARCH=i386,REGION=ap-southeast-1,STORAGE=instance-store,TEST=cloud-config,label=ubuntu-server-ec2-testing/console

[lp:~smoser/ubuntu-server-ec2-testing/fixes]

21. By Scott Moser on 2011-10-13

fix DEFAULT_SLEEP missing name

22. By Scott Moser on 2011-10-13

make uuid contain a notable prefix and timestamp

since uuid is used in security groups and keypairs and
other places, it would be nice to make it identifyable.

This way, if i'm looking at keypairs and security groups, I
can clean up ones from the testing suite without any harm.

23. By Scott Moser on 2011-10-13

fix incorrect variable name in terminate_and_wait

24. By Scott Moser on 2011-10-13

terminate_and_wait: fix signature to have 'self'

25. By Scott Moser on 2011-10-13

collect all consoles in tidyup

This also adds a check in collect_console to avoid re-collecting
a console that was already collected. That has been added
because tidyup and the normal path will get call collect_console
with 'terminated'.

26. By Scott Moser on 2011-10-13

put terminate retries settings into settings

27. By Scott Moser on 2011-10-13

improve log message for 'already collected'

28. By Scott Moser on 2011-10-13

terminate instance with boto.ec2.instance.terminate()

apparently, with the version of boto that was in natty (1.9b-1ubuntu5)
boto.ec2.instance.stop() would terminate an instance.
In oneiric (2.0-0ubuntu1) that does a 'stop'.

Both versions have a 'boto.ec2.instance.terminate()' method, so
this change just uses that.

[James Page (james-page)]

Scott

All looks good to me - I've pulled this and the documentation MP in and added one other minor update - when ebs instances where being stopped they where doing weird internal calls on the instance to make that happen - probably related to the fact that stop actually terminated the instance.

I updated that call to use stop() instead.

I'll merge this in and push a new version to the PPA for oneiric.

Cheers

James

review: Approve

=== modified file 'src/ubuntu/ec2/settings.py'
--- src/ubuntu/ec2/settings.py	2011-04-13 07:18:38 +0000
+++ src/ubuntu/ec2/settings.py	2011-10-13 16:25:20 +0000
@@ -32,6 +32,11 @@
 # Time to sleep before monitoring instances
 START_SLEEP = 30
 
+# Time to pause after call to TerminateInstances before checking
+# to see if instance has terminated.
+TERMINATE_SLEEP = 10
+TERMINATE_SLEEP_TRIES = 12
+
 # Valid combinations of regions and availability zones
 REGION_PLACEMENT_LOOKUP = {
     'eu-west-1' : [ 'eu-west-1a' , 'eu-west-1b' , 'eu-west-1c', None],

=== modified file 'src/ubuntu/ec2/testing.py'
--- src/ubuntu/ec2/testing.py	2011-04-26 08:57:28 +0000
+++ src/ubuntu/ec2/testing.py	2011-10-13 16:25:20 +0000
@@ -61,6 +61,7 @@
 UEC_IMAGES_ALL_URL = 'http://uec-images.ubuntu.com/query/%s/%s/daily.txt'
 LATEST_RELEASE = 'current'
 DEFAULT_TEST_DIR = os.path.expanduser('~/tests')
+UUID_PREFIX = "uec2"
 
 CI_SCRIPT = "text/x-shellscript"
 CI_SCRIPT_START = "#!"
@@ -230,7 +231,9 @@
         self.test_case = test_case  
         self.access_key_id = aws_access_key_id
         self.secret_access_key = aws_secret_access_key
-        self.uuid = str(uuid.uuid4())
+        self.uuid = "%s-%s-%s" % ( UUID_PREFIX,
+            time.strftime("%Y%m%d-%H%M"),
+            str(uuid.uuid4()).replace("-","")[0:14])
         self.timeout = settings.DEFAULT_TIMEOUT
         # To start with check placement against region
         if (self.test_case.placement not in 
@@ -310,7 +313,7 @@
             self.tidyup()
             if (l_timed_out):
                 # Collect consoles from all instances 
-                self.collect_all_consoles()
+                self.collect_all_consoles('timeout')
             
         return (success and not self.any_errors())
 
@@ -441,11 +444,11 @@
         logging.debug("Testing completed %s",test_error)
         return test_error  
      
-    def collect_all_consoles(self):
+    def collect_all_consoles(self, suffix):
         ''' Collects consoles from all Test Case Instances marked with
             timeout '''
         for l_instance in self.test_instances:
-            l_instance.collect_console('timeout')
+            l_instance.collect_console(suffix)
             
     def collect_timeout_metadata(self):
         ''' Collect instance metadata via SSH if not complete and still
@@ -454,6 +457,35 @@
             if ((not l_instance.complete) and l_instance.sshable):
                 l_instance.collect_metadata('timeout')
 
+    def terminate_and_wait(self, tries=settings.TERMINATE_SLEEP_TRIES,
+        naplen=settings.TERMINATE_SLEEP):
+        if not self.reservation:
+            return
+        # turn a result list into an instance list
+        def _resl2instl(result_set):
+            ret = []
+            for res in result_set:
+                for inst in res.instances:
+                    ret.append(inst)
+            return(ret)
+
+        iid_l = [l.instance.id for l in self.test_instances]
+        # check to see if there is work to do, if not, return
+        for tnum in range(1,tries+1):
+            res_l = self.conn.get_all_instances(instance_ids=iid_l)
+            instances = _resl2instl(res_l)
+            to_term = [ i.id for i in instances if i.state != "terminated" ]
+            if len(to_term) == 0:
+                return
+            self.conn.terminate_instances(to_term)
+            iid_l = to_term
+            logging.debug("Waiting on terminate-instances [%i/%i]: %s" %
+                          (tnum, tries, to_term))
+            time.sleep(naplen)
+
+        logging.warning("These instances did not terminate: %s" % to_term)
+        return
+
     def tidyup(self):
         ''' Tidyup any residual stuff left over ''' 
         # Remove the private key!!!!
@@ -470,8 +502,10 @@
         # terminated
         if self.reservation:
             logging.debug('Terminating all instances associated with test case')
-            self.reservation.stop_all()
+            self.terminate_and_wait()
             
+        self.collect_all_consoles('terminated')
+
         # Tidyup the keypair
         if self.keypair:
             logging.debug('Removing keypair associated with test case')
@@ -709,13 +743,19 @@
                              ktype, self.instance.id, host)
                 self.keyscan_ok = True      
                 
-    def collect_console(self,suffix):
+    def collect_console(self,suffix,update=False):
         ''' Grabs the console for the instance in its current state '''
+        output_path = os.path.join(self.working_dir,
+                                   '%s-%s.console.txt'
+                                   % (self.tce.uuid, suffix))
+        if os.path.isfile(output_path):
+            logging.debug("console %s already collected for %s" % 
+                          (self.instance.id, suffix))
+            return
+
         l_output = self.instance.get_console_output()
         logging.debug(l_output.output)
-        console_file = open(os.path.join(self.working_dir,
-                                         '%s-%s.console.txt')
-                            % (self.tce.uuid, suffix), 'w')
+        console_file = open(output_path, 'w')
         console_file.write(l_output.output)
         console_file.close()        
 
@@ -813,7 +853,7 @@
                 if (self.is_running()):
                     self.collect_metadata('restarted')
                     logging.debug('Terminating Instance %s', self.instance.id)
-                    self.instance.stop()
+                    self.instance.terminate()
                     # Replace with JUnit Test
                     self.validate_host_keys()
                     self.sshable = False