Merge lp:~darkmuggle-deactivatedaccount/cloud-utils/syncimgs into lp:cloud-utils

[Ben Howard]

Further code clean up and improvement per previous MP.

syncimgs/syncmessage.py and syncimgs/syncmsg-reader.py:
    Removed as unneeded

bin/cloudimg-sync:
    - Added example help to argparse commands
    - Fixed bug where --config was required, preventing --how or
          --test-config cli arguments from working
    - URL for file downloads is now coming from build_json.py
    - Minor whitespace cleanup

bin/ubuntu-cloudimg-query2:
    - Added support for version number queries
    - Removed hard-coded version names; replaced with python-distro-info
    - Fixed minor bug where all results for --build were being displayed
    - Minor white space changes for readability
    - Added more example for invokation

syncimgs/build_json.py:
    - Added support for Query1 string formating
    - Added attributes to BuildFile and Registration objects when
        iterating to better support ubuntu-cloudimg-query2 and
        cloud-syncimgs programs.
    - Selection of host for download is now done by random selection
        in build_json.py

syncimgs/execute.py:
    - Fixed bug where stdin and stderr tuple where being inproperly
         split.

syncimgs/easyrep.py:
    - Added set function; used by build_json.py to set query2 object
     attributes

ubuntu-cloudimg-keyring.gpg.b64:
    - Replaced corrupted version with a valid keyring

syncimgs/yaml_config.py:
    - Better documented program functionality about how rules work for
     cloudimg-sync

[Ben Howard (darkmuggle-deactivatedaccount)]

I'm going to work on getting execute.py to not shell tomorrow. I'll submit another MP for that.

[Scott Moser (smoser)]

- ubuntu-cloudimg-keyring.gpg.b64
  the file you have here is zero length. also, I've tested that the
  version in trunk when installed does successfully queries (no complaints
  about bad keyring)
- please add debian/control dependensy on python-distro-info, remove dependency on 'distro-info'

- confusing in build_json:
  + xarch = self.arch
  + if 'amd64' in xarch:
  + xarch = 'x86_64'

  is there a reason you used 'in' there? why not ==?

- * The command can use the following as parametes:
  ^ spelling

- for list_cmd, I'm just curious what value today's date could bring:
  - %(date)s: todays date in YYYY-MM-DD format

- for list_cmd, you say it must return '<tag> <build_serial> <arch>

review: Needs Fixing

[Ben Howard (darkmuggle-deactivatedaccount)]

- ubuntu-cloudimg-keyring.gpg.b64
Fixed...it looks like my pull was corrupted.

- please add debian/control dependensy on python-distro-info, remove dependency on
done

- confusing in build_json:
replaced with "==" over "in"

- * The command can use the following as parametes:
Fixed

- for list_cmd, I'm just curious what value today's date could bring:
there isn't much value, other than that is a valid replacement elase where

- for list_cmd, you say it must return '<tag> <build_serial> <arch>
clarified its usage

[lp:~darkmuggle-deactivatedaccount/cloud-utils/syncimgs]

185. By Scott Moser on 2012-07-12

Further code clean up and improvement per previous MP.

syncimgs/syncmessage.py and syncimgs/syncmsg-reader.py:
    Removed as unneeded

bin/cloudimg-sync:
    - Added example help to argparse commands
    - Fixed bug where --config was required, preventing --how or
          --test-config cli arguments from working
    - URL for file downloads is now coming from build_json.py
    - Minor whitespace cleanup

bin/ubuntu-cloudimg-query2:
  - Added support for version number queries
    - Removed hard-coded version names; replaced with python-distro-info
    - Fixed minor bug where all results for --build were being displayed
    - Minor white space changes for readability
    - Added more example for invokation

syncimgs/build_json.py:
    - Added support for Query1 string formating
    - Added attributes to BuildFile and Registration objects when
        iterating to better support ubuntu-cloudimg-query2 and
        cloud-syncimgs programs.
    - Selection of host for download is now done by random selection
        in build_json.py

syncimgs/execute.py:
    - Fixed bug where stdin and stderr tuple where being inproperly
         split.

syncimgs/easyrep.py:
    - Added set function; used by build_json.py to set query2 object
     attributes

syncimgs/yaml_config.py:
    - Better documented program functionality about how rules work for
     cloudimg-sync

=== modified file 'bin/cloudimg-sync'
--- bin/cloudimg-sync	2012-07-11 16:52:42 +0000
+++ bin/cloudimg-sync	2012-07-12 16:31:27 +0000
@@ -23,9 +23,24 @@
 logger.setLevel(logging.DEBUG)
 
 if __name__ == '__main__':
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument('--config', action='store', required=True,
+    parser = argparse.ArgumentParser(prog='Ubuntu Cloud Image Sync Tool',
+                        description="""
+Rule based program for syncronizing and customizing Ubuntu Cloud Images
+""",
+                        epilog="""
+Example usage:
+    cloudimg-sync --config <configuration>
+        Use <configuration> as the rule file
+
+    cloudimg-sync --how
+        Show how to write rule file
+
+    cloudimg-sync --test-config
+        Show an example configuration file
+""",
+                        formatter_class=argparse.RawDescriptionHelpFormatter)
+
+    parser.add_argument('--config', action='store',
                         help='Configuration file')
     parser.add_argument('--force', action='store_true', default=False,
                         help='Force option')
@@ -56,12 +71,16 @@
         logger.info('Displaying configuration help information')
         yaml_config.config_help()
         sys.exit(0)
+
     elif opts.test_config:
-
         logger.info('Displaying a test configuration')
         print yaml_config.get_default_config()
         sys.exit(0)
 
+    if not opts.config:
+        logger.info('Must define --config, --how or --test_config')
+        sys.exit(1)
+
     logger.info('Setting up')
 
     configuration = yaml_config.read_yaml_file(opts.config)

=== modified file 'bin/ubuntu-cloudimg-query2'
--- bin/ubuntu-cloudimg-query2	2012-07-11 16:52:42 +0000
+++ bin/ubuntu-cloudimg-query2	2012-07-12 16:31:27 +0000
@@ -12,6 +12,7 @@
 from datetime import datetime, time, date
 import syncimgs.build_json
 import argparse
+import distro_info
 import logging
 import sys
 import json
@@ -22,47 +23,7 @@
 logger.setLevel(logging.CRITICAL)
 
 
-def formatting_help():
-    print """
-The following are valid format values for displaying Query2 data for clouds
-
-    %(region)s - region name
-    %(id)s - the cloud vendor ID
-    %(instance_type)s - instance type
-    %(cloud)s - cloud vendor name, i.e "ec2"
-    %(ramdisk_id)s - ramdisk ID, largely obsolete
-    %(registered_name)s - registered name in cloud system
-    %(build_serial)s - the build serial ID
-    %(path)s - Download URL for corresponding instance
-    %(sha1)s - SHA1 of the file
-    %(arch)s - architecture of the image
-
-The following are valid format values when running with "--builds":
-
-    %(path)s - the path of the file
-    %(sha1)s - the SHA1 checksum of the file
-    %(sha512)s - the SHA512 checksum of the file
-    %(description) - description of the file
-
-Example:
-    ubuntu-cloudimg-query2 --format "%(build_serials) %(id)s" us-west-2 ec2
-        Show the build serial and the ids of the images for us-west-2
-        region of EC2
-
-    ubuntu-cloudimg-query2 --builds --format "%(path)s" precise armhf
-        Show the files for Precise
-
-"""
-
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser(prog='Ubuntu Cloud Image Query2',
-                        description="""
-This program uses the Query2 format for describing the Ubuntu Cloud Images.
-""",
-                        epilog="""
-
+epilog_text = """
 Arguments:
     All arguments after the arguments flags are parsed as parameters. After
     arguments, the position does not matter.
@@ -80,13 +41,16 @@
     File Types: root.tar.gz, tar.gz, qcow2, manifest, vmdk, kernel, ovf
     Release Tags: release, daily
     Cloud: ec2 (other public clouds may be supported later)
-    Ubuntu Releases: hardy, lucid, maverick, natty, oneiric, precise, quantal
+    Ubuntu Release Code Names: %s
+    Ubuntu Release Version Numbers: %s
 
 Defaults:
     EC2 is the default cloud.
     us-east-1 is the default region.
     Ubuntu Server 12.04 (Precise) is the default image for query.
+"""
 
+epilog_footer="""
 Example Usage:
     ubuntu-cloudimg-query2 --include qcow2 --builds armhf
         Display the URL for the qcow2 armhf images
@@ -98,7 +62,72 @@
     ubuntu-cloudimg-query2 quantal amd64 hvm
         Display the AMI ID of the AMD64 HVM image for quantal in us-east-1
 
+    ubuntu-cloudimg-query hvm eu-west-1 12.10 --format "%(ami)s %(serial)s"
+        Display the AMI ID and the build serial for 12.10
+"""
+
+def formatting_help():
+    print """
+The following are common format values for both cloud and build queries
+
+    %(path)s - Download URL for corresponding instance
+    %(sha1)s - SHA1 of the file
+    %(arch)s - architecture of the image
+    %(url)s - download url of image
+    %(host)s - host URL
+    %(release)s - release tag, (i.e. release, daily)
+    %(release_tag)s - release tag, (i.e. release, daily)
+    %(stream)s - build type (i.e. server, desktop)
+    %(bname)s - build type (i.e. server, desktop)
+    %(xarch)s - machine architecture
+    %(build_serial)s - the build serial ID
+    %(serial)s - build serial number
+    %(suggested name)s - name suggested for downloaded files
+
+The following are specific format values for querying Cloud data:
+
+    %(region)s - region name
+    %(id)s - the cloud vendor ID
+    %(instance_type)s - instance type
+    %(cloud)s - cloud vendor name, i.e "ec2"
+    %(ramdisk_id)s - ramdisk ID, largely obsolete
+    %(registered_name)s - registered name in cloud system
+    %(build_serial)s - the build serial ID
+    %(summary)s - summary of the image
+    %(pname)s - virtualization type
+    %(dlpath)s - download path
+
+The following are specific format values to "--builds":
+
+    %(description) - description of the file
+    %(file_type) - file type
+
+Example:
+    ubuntu-cloudimg-query2 --format "%(build_serials) %(id)s" us-west-2 ec2
+        Show the build serial and the ids of the images for us-west-2
+        region of EC2
+
+    ubuntu-cloudimg-query2 --builds --format "%(path)s" precise armhf
+        Show the files for Precise
+
+"""
+
+
+if __name__ == '__main__':
+
+    udistro = distro_info.UbuntuDistroInfo()
+    code_names = udistro.supported()
+    ver_codes = map(lambda ver: ver.replace(" LTS",''),
+                    udistro.supported(result='release'))
+
+    epilog = epilog_text % (", ".join(code_names), ", ".join(ver_codes)) + \
+             epilog_footer
+
+    parser = argparse.ArgumentParser(prog='Ubuntu Cloud Image Query2',
+                        description="""
+This program uses the Query2 format for describing the Ubuntu Cloud Images.
 """,
+                        epilog=epilog,
                         formatter_class=argparse.RawDescriptionHelpFormatter)
 
     parser.add_argument('--url', action='store',
@@ -148,7 +177,10 @@
     if not opts.excluded:
         opts.excluded = []
 
-    if not opts.format:
+    if not opts.format and opts.builds:
+        opts.format = '%(serial)s %(url)s'
+
+    elif not opts.format:
         opts.format = '%(id)s'
 
     if opts.show_format:
@@ -166,9 +198,8 @@
     daily_re = re.compile('(daily)')
     cloud_re = re.compile('(ec2)|(azure)')
     base_re = re.compile('(server)|(desktop)')
-    distro_re = \
-        re.compile('(hardy)|(lucid)|(maverick)|(natty)|(oneiric)|(precise)|(quantal)'
-                   )
+    distro_re = re.compile('(' + ")|(".join(code_names) + ')')
+    ver_re = re.compile('(' + ")|(".join(ver_codes) + ')')
 
     for val in opts.args:
         val = val.lower()
@@ -177,30 +208,33 @@
         if region_re.match(val):
             key = 'region'
         elif type_re.match(val):
-
             key = 'instance_type'
+
         elif daily_re.match(val):
-
             key = 'release_tag'
             val = 'daily'
+
         elif release_re.match(val):
-
             key = 'release_tag'
             val = 'release'
+
         elif base_re.match(val):
-
             key = 'stream'
+
         elif cloud_re.match(val):
-
             key = 'cloud'
+
         elif arch_re.match(val):
-
             key = 'arch'
+
         elif distro_re.match(val):
-
-            key = 'distro'
+            key = 'distro'
+
+        elif ver_re.match(val):
+            key = 'distro'
+            val = code_names[ver_codes.index(val)]
+
         elif file_re.match(val):
-
             key = 'file_type'
 
         if key not in v.keys():
@@ -295,6 +329,7 @@
             opts.included,
             opts.excluded,
             serial=opts.serial,
+            output=opts.format,
             all=opts.all,
             )
 

=== modified file 'debian/control'
--- debian/control	2012-07-11 15:51:37 +0000
+++ debian/control	2012-07-12 16:31:27 +0000
@@ -15,6 +15,7 @@
          python,
          python-paramiko,
          python-yaml,
+         python-distro-info,
          util-linux (>= 2.17.2),
          ${misc:Depends}
 Description: cloud image management utilities

=== modified file 'syncimgs/build_json.py'
--- syncimgs/build_json.py	2012-07-11 16:52:42 +0000
+++ syncimgs/build_json.py	2012-07-12 16:31:27 +0000
@@ -66,10 +66,12 @@
     __name__ = 'Regsitration'
 
     def dump(self,
-             output='%(build_serial)s %(id)s %(arch)s %(instance_type)s %(region)s '
-             ):
+            output='%(build_serial)s %(id)s %(arch)s %(instance_type)s %(region)s',
+            url_base=None,
+            stream=None,
+            ):
 
-        supported_keys = [
+        required_keys = [
             'region',
             'id',
             'ramdisk_id',
@@ -80,23 +82,41 @@
             'cloud',
             'instance_type',
             'arch',
+            'virt_type',
             ]
 
-        for key in supported_keys:
+        for key in required_keys:
             if not hasattr(self, key):
                 setattr(self, key, None)
 
+        xarch = self.arch
+        if 'amd64' == xarch:
+            xarch = 'x86_64'
+
         output = output % {
             'region': self.region_name,
             'id': self.published_id,
+            'ami': self.published_id,
             'instance_type': self.instance_type,
             'cloud': self.cloud,
             'ramdisk_id': self.ramdisk_id,
+            'summary': self.registered_name,
+            'bname': stream,
+            'pname': self.virt_type,
             'registered_name': self.registered_name,
             'build_serial': self.build_serial,
+            'dlpath': self.path,
+            'url': self.url,
+            'host': url_base,
             'path': self.path,
             'sha1': self.sha1,
             'arch': self.arch,
+            'xarch': xarch,
+            'release': self.release_tag,
+            'release_tag': self.release_tag,
+            'serial': self.build_serial,
+            'stream': self.stream,
+            'suggested name': self.suggested_name,
             }
 
         return output
@@ -222,13 +242,14 @@
 
             keep = latest.lower()
             if keep == 'latest':
-                index = sorted(b_list)[-1]
-                yield Build(unpack=b_list[index])
-                skip = True
+
+                if len(b_list) > 0:
+                    index = sorted(b_list)[-1]
+                    yield Build(unpack=b_list[index])
+                    skip = True
+
             elif not re.search(r'\w-\d', keep):
-
                 start_index = total_builds - int(keep.replace('n-', ''))
-
                 if start_index < 0:
                     start_index = 0
 
@@ -312,13 +333,29 @@
         if url_base:
             url = "%s/%s" % (url_base, self.path)
 
+        xarch = self.arch
+        if 'amd64' == xarch:
+            xarch = 'x86_64'
+
         output = output % {
             'description': self.description,
             'sha1': self.sha1,
             'sha512': self.sha512,
             'buildid': self.buildid,
-            'path': url,
+            'path': self.path,
+            'url': url,
+            'host': url_base,
             'file_type': self.file_type,
+            'bname': self.stream,
+            'distro': self.distro,
+            'buildid': self.buildid,
+            'serial': self.serial,
+            'arch': self.arch,
+            'xarch': xarch,
+            'release': self.release,
+            'release_tag': self.release,
+            'stream': self.stream,
+            'suggested name': self.suggested_name,
             }
 
         return output
@@ -363,8 +400,8 @@
                                 self.distros[i['distro_code_name']] = []
 
                             if bt \
-                                not in self.distros[i['distro_code_name'
-                                    ]]:
+                                not in self.distros[i[
+                                        'distro_code_name']]:
                                 self.distros[i['distro_code_name'
                                         ]].append(bt)
 
@@ -493,8 +530,9 @@
         base = self.__get__('%s_%s' % (distro, stream))
 
         for sub in base:
-            if sub['release_tag'] == release_tag or release_tag \
-                == 'all':
+            if sub['release_tag'] == release_tag or \
+                 release_tag == 'all':
+
                 build_serial = sub['build_serial']
                 ret[build_serial] = {}
 
@@ -503,8 +541,19 @@
                     ret[build_serial][arch] = []
 
                     for fl in sub['arches'][arch]['file_list']:
-                        anon = BuildFiles(buildid=sub['arches'
-                                ][arch]['build_id'], unpack=fl)
+                        anon = BuildFiles(
+                                buildid=sub['arches'][arch]['build_id'],
+                                unpack=fl,
+                                )
+
+                        anon.set('suggested_name', anon.path.split('/')[-1])
+                        anon.set('serial', build_serial)
+                        anon.set('distro', distro)
+                        anon.set('stream', stream)
+                        anon.set('arch', arch)
+                        anon.set('build_id', sub['arches'][arch]['build_id'])
+                        anon.set('release', sub['release_tag'])
+
                         ret[build_serial][arch].append(anon)
 
         return ret
@@ -550,14 +599,19 @@
                     included,
                     excluded,
                     serial,
-                    all,
+                    all=all,
                     ):
 
             url_base = random.choice(self.mirrors_transfer)
+            d = f.dump(output=output,
+                    url_base=url_base,
+                    )
 
-            d = f.dump(output=output, url_base=url_base)
             if d:
-               data += "%s\n" % d
+                if data:
+                    data += "\n%s" % d
+                else:
+                    data = d
 
         return data
 
@@ -594,11 +648,12 @@
 
         for bf in results:
 
-            if serial and bf != serial:
-                continue
+            if serial:
+                if bf != serial:
+                    continue
 
             elif not all and bf != serials[-1]:
-                next
+                continue
 
             for b_arch in results[bf]:
 
@@ -640,7 +695,9 @@
 
         builds = self.distro_builds(distro, stream,
                                     release_tag=release_tag)
+
         for bf in builds.iter_builds(latest=latest):
+
             if release_tag != 'all' and release_tag != bf.release_tag:
                 print '%s %s' % (release_tag, bf.release_tag)
                 continue
@@ -660,15 +717,20 @@
                             continue
 
                         for reg in itypes.iter_registrations():
-                            reg.set('path', '%s/%s'
-                                    % (str(self.mirrors_transfer[0]),
-                                    path))
+                            reg.set('url', '%s/%s'
+                                    % (random.choice(self.mirrors_transfer),
+                                      path)
+                                   )
+                            reg.set('path', path)
+                            reg.set('suggested_name', path.split('/')[-1])
                             reg.set('sha1', sha1)
                             reg.set('build_serial', bf.build_serial)
                             reg.set('instance_type', itypes.name)
                             reg.set('cloud', _cloud)
                             reg.set('arch', _arch)
                             reg.set('release_tag', bf.release_tag)
+                            reg.set('stream', stream)
+                            reg.set('virt_type', itypes.name)
                             yield reg
 
     def get_reg(
@@ -695,11 +757,13 @@
             cloud=cloud,
             ):
 
+            url_base = random.choice(self.mirrors_transfer)
+
             if all_regions:
-                reg_data += "%s\n" % r.dump(output=output)
+                reg_data += "%s\n" % r.dump(output=output, url_base=url_base)
 
             elif r.region_name == region:
-                return r.dump(output=output)
+                return r.dump(output=output, url_base=url_base)
 
         return reg_data
 
@@ -840,15 +904,18 @@
             request = urllib2.Request(url)
             request.add_header('Accept-encoding', 'gzip')
             response = urllib2.urlopen(request)
+
             fetched_json = None
 
             if response.info().get('Content-Encoding') == 'gzip':
                 buf = StringIO(response.read())
                 f = gzip.GzipFile(fileobj=buf)
                 fetched_json = f.read()
+
             elif response.info().get('Content-Type') \
                 == 'application/x-bzip2':
                 fetched_json = bz2.decompress(response.read())
+
             else:
                 fetched_json = response.read()
 
@@ -877,6 +944,7 @@
 
             setattr(self, 'build_catalog',
                     BuildCatalog(json=json.loads(fetched_json)))
+
         except IOError as e:
 
             raise Exception('Unable to fetch JSON from remote source.\n%s'
@@ -884,7 +952,6 @@
 
         return False
 
-
 # Example of how to parse out some details...still needs work
 #logger.setLevel(logging.DEBUG)
 #cloudjson = CloudJSON(gpg_verify=True, url="http://cloud-images.ubuntu.com/query2/server/quantal/ec2.json.bz2")

=== modified file 'syncimgs/cloudimg.py'
--- syncimgs/cloudimg.py	2012-07-11 16:52:42 +0000
+++ syncimgs/cloudimg.py	2012-07-12 16:31:27 +0000
@@ -178,9 +178,10 @@
         stream,
         ):
         """Basic replacements of:
-                    - %(distro)s:        code name of the distro
+                    - %(distro)s:       code name of the distro
                     - %(version)s:      suite version name, i.e 12.04
                     - %(date)s:         todays date in YYYY-MM-DD format
+                    - %(stream)s:       stream of build, i.e. server, desktop
         """
 
         return string % {
@@ -778,20 +779,20 @@
                         logger.warn('       Already registered, skipping processing'
                                     )
                         continue
+
                     elif reg and not rec:
-
                         logger.warn('       Image was registered _outside_ of this program!'
                                     )
                         continue
+
                     elif not reg and rec:
-
                         logger.warn('       Image is recorded in logs as registered, but is missing.'
                                     )
                         logger.warn('       This is likely due to a problem with your registration script!'
                                     )
                         logger.warn('       Re-processing image!')
+
                     else:
-
                         logger.info('      Arch %s is cleared for processing'
                                      % arch)
 
@@ -819,8 +820,6 @@
                             continue
 
                         # Do the work
-
-                        url = '%s/%s' % (d_conf.transfer, fd.path)
                         local_file = '%s/%s' % (d_conf.sync_dir,
                                 fd.path)
                         pristine_file = '%s/%s' % (d_conf.pristine,
@@ -833,7 +832,7 @@
                                     % fd.file_type)
                         logger.info('         Expected SHA1: %s'
                                     % fd.sha1)
-                        logger.info('         Fetch URL:     %s' % url)
+                        logger.info('         Fetch URL:     %s' % fd.url)
                         logger.info('         Local Path:    %s'
                                     % local_file)
                         logger.info('         Log Path:      %s'
@@ -843,7 +842,7 @@
                             pristine_file = None
 
                         furl = URLFetcher(
-                            url,
+                            fd.url,
                             fd.sha1,
                             local_file,
                             conf.download_log,
@@ -852,10 +851,11 @@
                             pristine=pristine_file,
                             spaces='         ',
                             )
+
                         if not furl.get():
                             print furl
                             raise Exception('Failed to download %s'
-                                    % url)
+                                    % fd.url)
 
                         replacements = self.__replacements__(
                             fd,
@@ -909,13 +909,16 @@
                                 for line in out.splitlines():
                                     if emitted_re.match(line):
                                         local_file = \
-    line.replace('::EMITTED-FILE::', '')
+                                            line.replace('::EMITTED-FILE::', '')
+
                                         replacements['local'] = \
-    local_file
+                                            local_file
+
                                         logger.info('%sCustom command emitted: %s'
-         % (spaces, local_file))
+                                             % (spaces, local_file))
+
                                         logger.info('%sUsing emitted file for publishing'
-         % spaces)
+                                             % spaces)
                             else:
 
                                 logger.critical('       FAILED TO CUSTOMIZE. Processing of this build\n       IS ABORTED! Other builds will be processed'

=== modified file 'syncimgs/easyrep.py'
--- syncimgs/easyrep.py	2012-07-11 16:52:42 +0000
+++ syncimgs/easyrep.py	2012-07-12 16:31:27 +0000
@@ -77,5 +77,8 @@
     def get(self, name):
         return self.__get__(name)
 
+    def set(self, name, value):
+        setattr(self, name, value)
+
 
 # vi: ts=4 expandtab

=== modified file 'syncimgs/execute.py'
--- syncimgs/execute.py	2012-07-11 16:52:42 +0000
+++ syncimgs/execute.py	2012-07-12 16:31:27 +0000
@@ -57,18 +57,18 @@
                                     '\n'.join(cmds)))
 
             run_cmd = Popen([temp_f], shell=True, stdout=PIPE, stderr=PIPE)
-            self.output = run_cmd.communicate()
+            (self.output, self.error) = run_cmd.communicate()
 
             if 'output' in kargs:
                 if kargs['output'] == 'screen':
                     print self.output
                 elif kargs['output'] == 'log':
 
-                    for line in self.output[0].splitlines():
+                    for line in self.output.splitlines():
                         logger.info('%sCMD stdout: %s ' % (self.spaces,
                                     line))
 
-                    for line in self.output[1].splitlines():
+                    for line in self.error.splitlines():
                         logger.info('%sCMD stderr: %s ' % (self.spaces,
                                     line))
 

=== removed file 'syncimgs/syncmessage.py'
--- syncimgs/syncmessage.py	2012-07-11 16:52:42 +0000
+++ syncimgs/syncmessage.py	1970-01-01 00:00:00 +0000
@@ -1,206 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-## Copyright (C) 2011 Ben Howard <[e-mail]>
-## Date: 25 February 2012
-##
-## This comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
-## This is free software, and you are welcome to redistribute it
-## under certain conditions; see copying for details.
-
-# Simple messaging class
-
-from boto.sqs.connection import SQSConnection
-from boto.sqs.message import Message
-import argparse
-import json
-import re
-import uuid
-import pickle
-import platform
-import sys
-
-
-class SyncMessage:
-
-    __name__ = 'SyncMessage'
-
-    def __init__(self, **kargs):
-        self.uuid = str(uuid.uuid1())
-        self.__python_version = platform.python_version()
-        (
-            self.__system,
-            self.__node,
-            self.__release,
-            self.__version,
-            self.__machine,
-            self.__process,
-            ) = platform.uname()
-        self.json_key = '%s_' % self.uuid
-
-        for key in kargs:
-            if key == 'json':
-                try:
-                    j = None
-                    with open(kargs[key], 'rb') as f:
-                        j = json.load(f)
-                    f.close()
-
-                    if j:
-                        for item in j:
-                            setattr(self, '%s_%s' % (self.json_key,
-                                    item), j[item])
-                except IOError, e:
-
-                    print e
-                    sys.exit(1)
-            else:
-                setattr(self, key, kargs[key])
-
-    def iter_unpack(self, key=None):
-        if not key:
-            key = self.json_key
-
-        key_re = re.compile('%s.*' % key)
-        for val in self.__dict__.keys():
-            if key_re.match(val):
-                nkey = val.replace('%s_' % key, '')
-                yield (nkey, getattr(self, val))
-
-    def unpack(self, key=None, pretty=False):
-        pack = {}
-        for (key, val) in self.iter_unpack(key=key):
-            pack[key] = val
-
-        if pretty:
-            return json.dumps(pack)
-
-        return pack
-
-    def passed(self):
-        try:
-            return self.success
-        except KeyError:
-            return None
-
-    def failed(self):
-        try:
-            return self.success
-        except KeyError:
-            return None
-
-    def get(self, key):
-        try:
-            return getattr(self, key)
-        except KeyError:
-            return None
-
-    def set(self, key, value):
-        setattr(self, key, value)
-
-    def ack(self, uuid):
-        try:
-            if getattr(self, 'ack') == uuid:
-                return True
-            else:
-                return False
-        except KeyError:
-            return None
-
-    def __repr__(self):
-        ret_string = ''
-        for i in self.__dict__.keys():
-            if not ret_string:
-                ret_string = '%s=%s' % (i, getattr(self, i))
-            else:
-
-                ret_string = ret_string + ', %s=%s' % (i, getattr(self,
-                        i))
-
-        ret_string = str('%s(%s)' % (self.__class__.__name__,
-                         ret_string))
-        return ret_string
-
-
-class SQSConn:
-
-    def __init__(
-        self,
-        access_key,
-        secret_key,
-        queue_name,
-        ):
-        self.conn = SQSConnection(access_key, secret_key)
-        self.queue = self.conn.create_queue(queue_name)
-
-    def connected(self):
-        c = False
-        q = False
-        if self.conn:
-            c = True
-        if self.queue:
-            q = True
-
-        return (c, q)
-
-    def send(self, syncmessage):
-
-        if not isinstance(syncmessage, SyncMessage):
-            raise Exception('Invalid message type')
-
-        if self.queue:
-            m = Message()
-            m.set_body(pickle.dumps(syncmessage))
-            self.queue.write(m)
-        else:
-            raise Exception('No Queue Connection!')
-
-    def get(self, delete=False):
-        if self.queue:
-            while self.queue.count > 0:
-                try:
-                    m = self.queue.read()
-                    if m:
-                        message = pickle.loads(m.get_body())
-                        yield message
-
-                        if delete:
-                            m.delete()
-                    else:
-                        break
-                except pickle.UnpicklingError as e:
-                    raise 'Error de-pickling!\n%s' % e
-
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--secret_key', action='store', required=True,
-                        help='AWS Secret Key')
-    parser.add_argument('--access_key', action='store', required=True,
-                        help='AWS Access Key/ID')
-    parser.add_argument('--queue', action='store', required=True,
-                        help='Queue to use for storing message')
-    parser.add_argument('--success', dest='success', action='store_true'
-                        , help='Record command as successfull')
-    parser.add_argument('--fail', dest='success', action='store_false',
-                        help='Record command as a failure')
-    parser.add_argument('--msg', action='store', required=True,
-                        help='Message component')
-    parser.add_argument('--json', action='store',
-                        help='Name of JSON file containing futher key/value pairs'
-                        )
-
-    opts = parser.parse_args()
-
-    # Prepare the message
-
-    message = SyncMessage(msg=opts.msg, json=opts.json,
-                          success=opts.success)
-
-    # Write the message
-
-    sqsconn = SQSConn(opts.access_key, opts.secret_key, opts.queue)
-    sqsconn.send(message)
-
-# vi: ts=4 expandtab

=== removed file 'syncimgs/syncmsg-reader.py'
--- syncimgs/syncmsg-reader.py	2012-07-11 16:52:42 +0000
+++ syncimgs/syncmsg-reader.py	1970-01-01 00:00:00 +0000
@@ -1,42 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-## Copyright (C) 2011 Ben Howard <[e-mail]>
-## Date: 25 February 2012
-##
-## This comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
-## This is free software, and you are welcome to redistribute it
-## under certain conditions; see copying for details.
-
-# Simple Utility for reading SQS Message Queue
-
-import argparse
-from syncimgs.syncmessage import SQSConn
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--secret_key', action='store', required=True,
-                        help='AWS Secret Key')
-    parser.add_argument('--access_key', action='store', required=True,
-                        help='AWS Access Key/ID')
-    parser.add_argument('--queue', action='store', required=True,
-                        help='Queue to use for storing message')
-    parser.add_argument('--all', action='store_true',
-                        help='Read all queue items')
-    parser.add_argument('--max', action='store', type=int,
-                        help='Max number of elements to read')
-
-    opts = parser.parse_args()
-    sqsconn = SQSConn(opts.access_key, opts.secret_key, opts.queue)
-
-    counted = 0
-    for item in sqsconn.get():
-        print item.unpack(pretty=True)
-        print '\n'
-
-        if opts.max and not opts.all:
-            if opts.max >= counted:
-                break
-
-# vi: ts=4 expandtab

=== modified file 'syncimgs/yaml_config.py'
--- syncimgs/yaml_config.py	2012-07-11 16:52:42 +0000
+++ syncimgs/yaml_config.py	2012-07-12 16:31:27 +0000
@@ -139,59 +139,14 @@
 
 def config_help():
     print """
-The following rules apply:
-    1. There must be a single section "--- !Control" defined; that is the control
-        section that defines the basis for downloading and mirroring the images.
-    2. There must be a single section "--- !GeneralRules" defined; that section
-        defines the default behavior of the mirroring.
-    3. There may be any number of "--- !SuiteRuleOverride" defined, which allows
-        you to override general rules.
-    4. Each SuiteRuleOverride will be treated a separate and distinct task on a per
-        suite basis.
-
-#cloudimg-sync-config:
-sync-dir: /srv/cloudimgs/data
-pristine: /srv/cloudimgs/pristine
-history_log: /srv/cloudimgs/history.log
-download_log: /srv/cloudimgs/download.log
-host_url: http://cloud-images.ubuntu.com/query2/server/release/build.json
-gpg_validate: True
-suites: { available, hardy, lucid, maverick, natty, oneiric, precise }
-stream: [ desktop, server ]
-publish: [ qcow2 ]
-mirror: [ tar.gz, manifest ]
-name_convention: %(name)s-%(build_date)s
-copy_pristine: {True, False}
-max_dailies: {all,#,latest}
-max_milestones: {all,#,latest}
-keep_pre_release: {True, False}
-arches: [ i386, amd64 ]
-mirror_arches: [ armel, armhf ]
-check_types: [ manifest ]
-customize_types: [ qcow2 ]
-custom_cmd: |
-     /bin/true
-list_cmd: |
-    /bin/true
-check_cmd: |
-    /bin/false
-publish_cmd: |
-    /bin/true
-unpublish_cmd: |
-    /bin/true
-overrides:
- -precise:
-  publish: [ qcow2 ]
-  mirror: [ tar.gz ]
-  copy_pristine: False
- -oneiric:
-  publish: [ tar.gz ]
-  mirror: [ qcow2 ]
-  mirror_arches: [ armel ]
-  check_cmd: |
-      /bin/false
-
-
+cloudimg-sync is a tool for mirroring, customizing and registering the
+Ubuntu Cloud Images on arbitrary clouds using a rule-engine.
+
+Example configuration:
+---------------------------
+"""
+    get_default_config()
+    print """
 ---------------------------
 
 Overrides:
@@ -269,7 +224,7 @@
 
 Custom Commands:
 
-    Custom commands are indented lists of arbitrary CLI commands or language commands.
+    Custom commands are lists of arbitrary CLI commands or language commands.
     If your command(s) requires an interpreter, use "#!/..." syntax. For
     example, to run a Perl command, you would use:
 
@@ -285,8 +240,7 @@
 
 
     If using multiple "#!/..." files, it is recommended to use files instead of storing
-    these commands in the configuration file. Each "#!/.." encountered is interpreted to
-    be a different set of commands.
+    these commands in the configuration file.
 
     To pass arguments to custom commands, you can use any of the string replacements
     explained above. The example above passes the value JSON value of the file's SHA512
@@ -303,27 +257,45 @@
             This command is called with substitutions. For example:
                 /srv/cloudimgs/bin/publish-ec2 %(directory)s %(file)s %(distro)s %(arch)s %(build_serial)s
 
+            This command is run on _each_ file that is listed in the check file list.
+
+            If command exists successfully, then image processing for that specific build serial
+                continues normally.
+
         custom_cmd: Commands for customizing the image, such as installing keys or packages
             These commands _are not_ run in a chroot.
 
             The command is called with the directory of the downloaded files and the file
                 to be customized.
 
-            If the command exits succesfully, then the last line should be the name of the
+            If the command exits successfully, then the last line should be the name of the
                 file to be published. If there is no output from the command, but it
                 successfully exits, then the filename is assumed to be unchanged.
 
-        list_cmd: Indented list of command(s) to run a query to get a listing of names of
+        list_cmd: Command(s) to run a query to get a listing of names of
             published and/or mirrored images.
 
-            The command _MUST_ return registratrations with the following rules:
-                * One registration per line
-                * Each line formated as: <tag> <build_serial> <arch>
-                * The command _must_ accept "%(distro)s" or "%(version)s" as the only parameter
+            The command _MUST_ return registrations with the following rules:
+                * One registration per line, formated as:
+                     <tag> <build_serial> <arch>
+                * The command can use the following as parameters:
+                    - %(distro)s:       code name of the distro
+                    - %(version)s:      suite version name, i.e 12.04
+                    - %(date)s:         todays date in YYYY-MM-DD format
+                    - %(stream)s:       stream of build, i.e. server, desktop
+
+            This option should be used to _check_ if an image should be processed or not.
+
+            A non-zero exit status is treated as a failure. In the event of a failure the program
+                will skip the build serial.
+
+            State tracking is done both locally and by cloud registration if list_cmd is defined.
+                The output of list_cmd is authoritative, while the local state tracking is used
+                purely for logging and to reduce bandwidth usage.
 
         publish_cmd: This command published an image.
 
-            The command will be passed subsitutions defined above. For example:
+            The command will be passed substitutions defined above. For example:
                 /srv/cloudimgs/bin/publish-ec2 %(directory)s %(file)s %(distro)s %(arch)s %(build_serial)s
 
                 * You must define the inputs.