Created by Robert C Jennings and last modified
Get this branch:
bzr branch lp:~rcj/simplestreams/lxd
Only Robert C Jennings can upload to this branch. If you are Robert C Jennings please log in for upload directions.

Branch merges

Related bugs

Related blueprints

Branch information

Robert C Jennings

Recent revisions

437. By Robert C Jennings

Create chksum for LXD metadata+root for squashfs (LP: #1577922)

This adds a 'combined_sqashfs_sha256' element to items that have
both a LXD metadata tar and a squashfs root. Additionally, we
have 'combined_rootxz_sha256' which wil have the same value as
'combined_sha256' and allow for sunsetting that ambiguous name.

436. By Scott Moser

GlanceMirror: fix a couple of race-related problems

First, what can happen is that .sync() is interrupted for external reasons
(service restarts, network issues,...) after first image has been uploaded
to Glance when syncing multiple images. On re-run, the uploaded image is
considered "synced" already, but since simplestreams index is written out
only at the end of the sync, all the metadata is lost. We solve this by
adding "simplestreams_metadata" as one of the properties of image in
Glance where we put a JSON representation of all the metadata fields from
the original simplestreams index entry. We then load them in
load_products() if they are defined. This results in a slightly different
auto.sync.json entry: "endpoint" and "region" are on the product entry,
instead of on the disk1.img entry.

Secondly, if sync is indeed interrupted after one or several images are
completely uploaded (but not all of them), there's no index file at all,
so any attempt to make use of them with juju would fail, even though it
should be fully functional. We solve this by calling "insert_products()"
at the end of the insert_item(). This means that the index is regenerated
after every single image is uploaded.

Along the way, I slightly improve insert_item() tests from the pre-req
branch to add a test for minimal data required, rename the existing one
using real world data to test_insert_item_full, and re-use that data for a
test for a newly added call to insert_products().

435. By Scott Moser

GlanceMirror: refactor insert_item for easier testing

This change refactors GlanceMirror.insert_item() to allow for easier and
more contained testing. I needed to do this to understand everything that
was going on inside insert_item and other bits of code. There are now four
distinct things happening in it:

 1. Download image to a local file from a ContentSource
 2. Construct extra properties to store in Glance along with image
 3. Prepare arguments for GlanceClient.images.create() call
 4. Adapt source simplestreams entry for an image for use in the target
    simplestreams index

It should be fully backwards compatible, and test coverage for all the
individual steps should be much better (I admit to it not being perfect,
but it's a step in the right direction, imho at least).

434. By Scott Moser

tools/toolutil.py: early yakkety images did not have ova files.

this code will soon be obsolete but for now just ignore the ova files
on older builds of yakkety.

433. By Scott Moser

glance: ignore inactive images

If connection to glance is broken during GlanceMirror.sync(), and image
would be left over in "status": "saving".

Glance itself tries to protect against that, however when it is restarted
(eg. "service glance-api restart"), it does not clean the image up.

This change makes load_product() ignore any images that do not have
"status": "active". Images can only have deleted, active or saving

Note that this isn't perfect. The image in "saving" state is never cleaned
up, but at least the resulting cloud works. If we want to clean up broken
images, it'd be hard to do on the simplestreams side (glance is better
placed to do that: it knows if anything is going on with any of them, eg.
is it being currently updated or not).

432. By Scott Moser

tools/make-test-data: sort keys when writing json, sort product names

This uses the simplestreams/util 'dump_data' to dump json data in
each place that was writing json in tools/make-test-data.

Also, it sorts the 'products' data in the index stream.

431. By Scott Moser

support stream output for yakkety builds

For yakkety builds:
 - yakkety build output renames the -disk1.img file to .img
   but stream data should maintain -disk1.img as the ftype and item key.
 - -root.tar.gz and -root.tar.xz files are not produced
 - squashfs images are available (and also were available partially through

As a cleanup, we a helper to ubuntu_versions to compare versions more
directly. String compare was less useful in some cases and also would end
up breaking after 'z' because whatever release comes after z would string
sort to < z.

This also adds a simple tool for comparing two trees output of 'make exdata'.
    ./tools/compare-test-data ../trunk.dist ./
will output a 'diff' from old to new list of files indexed.

430. By Robie Basak

Optimise read_signed

Use join instead of iteratively "appending" to a string. In my use case,
this reduces signature check time from over 400 seconds to almost a
tenth of a second.

429. By Scott Moser

set a default user-agent

Follow on to the custom user agent merge by adding a default user agent.

428. By Scott Moser

glance mirror: do not require that hypervisor_config be present

This just allows for hypervisor_config to not be present.

Branch metadata

Branch format:
Branch format 7
Repository format:
Bazaar repository format 2a (needs bzr 1.16 or later)
Stacked on:
This branch contains Public information 
Everyone can see this information.