Merge lp:~abentley/juju-release-tools/agents-stanzas-2 into lp:juju-release-tools

[Aaron Bentley]

Support all kinds of stanzas.

This branch adds support for alternate agent generation.

It supports Windows and Centos, but also supports using a single agent for all "living" ubuntus.

Because the number of parameters was getting unweildy, it creates a class to hold them. This class has various factory methods to support different generation approaches.

Since make_stanza can now make multiple stanzas, it is renamed to make_agent_json.

[Curtis Hovey (sinzui)]

I have a few comments inline.

review: Needs Information (code)

[lp:~abentley/juju-release-tools/agents-stanzas-2]

237. By Aaron Bentley on 2015-11-10

Updates from review.

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've pushed a new version with the changes you asked for.

>> + self.version_name = date.today().strftime('%Y%m%d')
>
> Could this get weird if we build on machines in different
> timezones? maybe datetime.utcnow().strftime('%Y%m%d') is safer

The odds are low, because versions are only relevant if there are
competing item_names for a given content_id, and we don't do that.

Even if we did, items with the a given item_name will typically be built
on the same machine every time, so they will have the same timezone.

Still, there's no harm in using UTC time. Done.

>> + self.filename = filename + + @classmethod + def
>> for_ubuntu(cls, release, series, arch, version, revision_build,
>
> As an aside, I am constantly annoyed that debian calls the
> series/codename release, so when I work with lxc templates or
> scripts that make changelogs, I pass the series as the release.
> What we call a release is closer to debian version. While we don't
> see it often, I see that precise and trusty in the clouds are
> actually 12.04.5 and 14.04.2. There is no right ways to think about
> this.

The vagueness about release/series/codename bugs me, too. (And I can't
help wondering whether we really need both :-) Still, I tried to use
the terminology of the job. That meant that I had to do to do
"stanza['release'] = series" in make_stanzas(). Ugh.

> We need a single definition of this list that can be imported. Can
> you make this list a module level constant? I want formal
> definition of the series we support.

Done.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWQgD5AAoJEK84cMOcf+9hAQQH/0+zqHNBoQ6+J4EwYMWcUln4
M0JaHGfe2XJXlrmlwqGRpYBtTrEcH9DWiCih1/2I7QyY4dXcpVoUtqkPqA8ROBTD
gZNjJRVQapWSStxs+4snfDVatGxmaM+/A+ayq4Vzp8XGSiteSxOQZVQRs0b3d+cE
3HooPlNx2m2pdqiY4s+6fDFkhqUSjJmyxq4un03LAmH/gjOhuZcyNzpt9+PjBfjI
6iI/xtC2U/hBW3gcZW8Qk7HyVCHPOW6aGKomMGNN9HX/E0vkZAfOihEdBc7xZYp2
H4hdA+s6UjzO9QjNDF37t7STmzLSXTYQeMmUF92rr9+yQed1leuU7BL6naPBkfQ=
=8nzG
-----END PGP SIGNATURE-----

[Curtis Hovey (sinzui)]

Thank you.

review: Approve (code)

=== modified file 'make-ubuntu-agent.bash'
--- make-ubuntu-agent.bash	2015-11-06 21:00:26 +0000
+++ make-ubuntu-agent.bash	2015-11-10 14:34:57 +0000
@@ -10,5 +10,5 @@
 (.*)\/bin\/jujud/\1/")
 tarfile=juju-$version-$SERIES-$ARCH.tgz
 tar -czvf $tarfile  -C $(dirname $jujud) jujud
-$RELEASE_TOOLS/make_stanza.py ubuntu $RELEASE $SERIES $ARCH $version \
-  $revision_build $tarfile
+$RELEASE_TOOLS/make_agent_json.py ubuntu $tarfile $revision_build $version\
+  $ARCH $RELEASE $SERIES

=== renamed file 'make_stanza.py' => 'make_agent_json.py'
--- make_stanza.py	2015-11-06 21:00:26 +0000
+++ make_agent_json.py	2015-11-10 14:34:57 +0000
@@ -1,61 +1,131 @@
 #!/usr/bin/env python
 from argparse import ArgumentParser
-from datetime import date
+from copy import deepcopy
+from datetime import datetime
 import hashlib
 import os
 import sys
 
+from build_package import juju_series
 from generate_simplestreams import json_dump
 
 
+supported_windows_releases = (
+    'win2012', 'win2012hv', 'win2012hvr2', 'win2012r2', 'win7', 'win81',
+    'win8',
+    )
+
+
 def parse_args():
     parser = ArgumentParser()
     parsers = parser.add_subparsers(dest='command')
     ubuntu = parsers.add_parser('ubuntu')
+    living_ubuntu = parsers.add_parser('living-ubuntu')
+    windows = parsers.add_parser('windows')
+    centos = parsers.add_parser('centos')
+    for subparser in [ubuntu, living_ubuntu, windows, centos]:
+        subparser.add_argument('tarfile')
+        subparser.add_argument('revision_build')
+        subparser.add_argument('version')
+    for subparser in [ubuntu, living_ubuntu]:
+        subparser.add_argument('arch')
     ubuntu.add_argument('release')
     ubuntu.add_argument('series')
-    ubuntu.add_argument('arch')
-    ubuntu.add_argument('version')
-    ubuntu.add_argument('revision_build')
-    ubuntu.add_argument('tarfile')
     return parser.parse_args()
 
 
-def write_stanzas(release, series, arch, version, revision_build, tarfile):
-    path = 'agent/revision-build-{}/{}'.format(revision_build,
-                                               os.path.basename(tarfile))
-    hash_obj = hashlib.sha256()
-    with open(tarfile) as tarfile_fp:
-        content = tarfile_fp.read()
-    stanza = {
-        'content_id': 'com.ubuntu.juju:revision-build-{}:tools'.format(
-            revision_build),
-        'version_name': date.today().strftime('%Y%m%d'),
-        'item_name': '{}-{}-{}'.format(version, series, arch),
-        'product_name': 'com.ubuntu.juju:{}:{}'.format(release, arch),
-        'path': path,
-        'arch': arch,
-        'version': version,
-        'format': 'products:1.0',
-        'release': series,
-        'ftype': 'tar.gz',
-        'size': len(content),
-        }
-    for hash_algorithm in ['sha256', 'md5', 'sha1']:
-        hash_obj = hashlib.new(hash_algorithm)
-        hash_obj.update(content)
-        stanza[hash_algorithm] = hash_obj.hexdigest()
-    filename = 'revision-build-{}-{}-{}.json'.format(revision_build, series,
-                                                     arch)
-    json_dump([stanza], filename)
+class StanzaWriter:
+
+    def __init__(self, releases, arch, version, revision_build,
+                 tarfile, filename):
+        self.releases = releases
+        self.arch = arch
+        self.version = version
+        self.revision_build = revision_build
+        self.tarfile = tarfile
+        self.version_name = datetime.utcnow().strftime('%Y%m%d')
+        self.filename = filename
+
+    @classmethod
+    def for_ubuntu(cls, release, series, arch, version, revision_build,
+                   tarfile):
+        filename = 'revision-build-{}-{}-{}.json'.format(
+            revision_build, series, arch)
+        return cls(
+            [(release, series)], arch, version, revision_build, tarfile,
+            filename)
+
+    @classmethod
+    def for_living_ubuntu(cls, arch, version, revision_build, tarfile):
+        filename = 'revision-build-ubuntu-{}.json'.format(arch)
+        releases = [
+            (juju_series.get_version(name), name) for name
+            in juju_series.get_living_names()]
+        return cls(
+            releases, arch, version, revision_build, tarfile, filename)
+
+    @classmethod
+    def for_windows(cls, version, revision_build, tarfile):
+        filename = 'revision-build-{}-windows.json'.format(
+            revision_build)
+        releases = [(r, r) for r in supported_windows_releases]
+        return cls(releases, 'amd64', version, revision_build, tarfile,
+                   filename)
+
+    @classmethod
+    def for_centos(cls, version, revision_build, tarfile):
+        filename = 'revision-build-{}-centos.json'.format(revision_build)
+        return cls([('centos7', 'centos7')], 'amd64', version, revision_build,
+                   tarfile, filename)
+
+    def write_stanzas(self):
+        path = 'agent/revision-build-{}/{}'.format(
+            self.revision_build, os.path.basename(self.tarfile))
+        with open(self.tarfile) as tarfile_fp:
+            content = tarfile_fp.read()
+        hashes = {}
+        for hash_algorithm in ['sha256', 'md5', 'sha1']:
+            hash_obj = hashlib.new(hash_algorithm)
+            hash_obj.update(content)
+            hashes[hash_algorithm] = hash_obj.hexdigest()
+        stanzas = list(self.make_stanzas(path, hashes, len(content)))
+        json_dump(stanzas, self.filename)
+
+    def make_stanzas(self, path, hashes, size):
+        for release, series in self.releases:
+            stanza = {
+                'content_id': 'com.ubuntu.juju:revision-build-{}:tools'.format(
+                    self.revision_build),
+                'version_name': self.version_name,
+                'item_name': '{}-{}-{}'.format(self.version, series,
+                                               self.arch),
+                'product_name': 'com.ubuntu.juju:{}:{}'.format(release,
+                                                               self.arch),
+                'path': path,
+                'arch': self.arch,
+                'version': self.version,
+                'format': 'products:1.0',
+                'release': series,
+                'ftype': 'tar.gz',
+                'size': size,
+                }
+            stanza.update(deepcopy(hashes))
+            yield stanza
 
 
 def main():
     args = parse_args()
     kwargs = dict(args.__dict__)
     del kwargs['command']
-    write_stanzas(**kwargs)
-
+    if args.command == 'ubuntu':
+        writer = StanzaWriter.for_ubuntu(**kwargs)
+    if args.command == 'living-ubuntu':
+        writer = StanzaWriter.for_living_ubuntu(**kwargs)
+    elif args.command == 'windows':
+        writer = StanzaWriter.for_windows(**kwargs)
+    elif args.command == 'centos':
+        writer = StanzaWriter.for_centos(**kwargs)
+    writer.write_stanzas()
 
 if __name__ == '__main__':
     sys.exit(main())