lp:~danilo/simplestreams/trusty-pkg-patch

Branch merges

Propose for merging

No branches dependent on this one.

Related source package recipes

1 recipe using this branch. (?)

Create packaging recipe

Related snap packages

Related bugs

Related blueprints

440. By Данило Шеган on 2016-06-20

Drop test3 from the test target for trusty.

439. By Данило Шеган on 2016-06-20

Re-add python2 build deps.

438. By Данило Шеган on 2016-06-20

Re-add SS_REQUIRE_DISTRO_INFO=0.

437. By Данило Шеган on 2016-06-20

Drop missing deps for trusty build.

436. By Scott Moser on 2016-06-14

GlanceMirror: fix a couple of race-related problems

First, what can happen is that .sync() is interrupted for external reasons
(service restarts, network issues,...) after first image has been uploaded
to Glance when syncing multiple images. On re-run, the uploaded image is
considered "synced" already, but since simplestreams index is written out
only at the end of the sync, all the metadata is lost. We solve this by
adding "simplestreams_metadata" as one of the properties of image in
Glance where we put a JSON representation of all the metadata fields from
the original simplestreams index entry. We then load them in
load_products() if they are defined. This results in a slightly different
auto.sync.json entry: "endpoint" and "region" are on the product entry,
instead of on the disk1.img entry.

Secondly, if sync is indeed interrupted after one or several images are
completely uploaded (but not all of them), there's no index file at all,
so any attempt to make use of them with juju would fail, even though it
should be fully functional. We solve this by calling "insert_products()"
at the end of the insert_item(). This means that the index is regenerated
after every single image is uploaded.

Along the way, I slightly improve insert_item() tests from the pre-req
branch to add a test for minimal data required, rename the existing one
using real world data to test_insert_item_full, and re-use that data for a
test for a newly added call to insert_products().

435. By Scott Moser on 2016-06-14

GlanceMirror: refactor insert_item for easier testing

This change refactors GlanceMirror.insert_item() to allow for easier and
more contained testing. I needed to do this to understand everything that
was going on inside insert_item and other bits of code. There are now four
distinct things happening in it:

 1. Download image to a local file from a ContentSource
 2. Construct extra properties to store in Glance along with image
 3. Prepare arguments for GlanceClient.images.create() call
 4. Adapt source simplestreams entry for an image for use in the target
    simplestreams index

It should be fully backwards compatible, and test coverage for all the
individual steps should be much better (I admit to it not being perfect,
but it's a step in the right direction, imho at least).

434. By Scott Moser on 2016-05-20

tools/toolutil.py: early yakkety images did not have ova files.

this code will soon be obsolete but for now just ignore the ova files
on older builds of yakkety.

433. By Scott Moser on 2016-05-20

glance: ignore inactive images

If connection to glance is broken during GlanceMirror.sync(), and image
would be left over in "status": "saving".

Glance itself tries to protect against that, however when it is restarted
(eg. "service glance-api restart"), it does not clean the image up.

This change makes load_product() ignore any images that do not have
"status": "active". Images can only have deleted, active or saving
status.

Note that this isn't perfect. The image in "saving" state is never cleaned
up, but at least the resulting cloud works. If we want to clean up broken
images, it'd be hard to do on the simplestreams side (glance is better
placed to do that: it knows if anything is going on with any of them, eg.
is it being currently updated or not).

432. By Scott Moser on 2016-05-19

tools/make-test-data: sort keys when writing json, sort product names

This uses the simplestreams/util 'dump_data' to dump json data in
each place that was writing json in tools/make-test-data.

Also, it sorts the 'products' data in the index stream.

431. By Scott Moser on 2016-05-17

support stream output for yakkety builds

For yakkety builds:
 - yakkety build output renames the -disk1.img file to .img
   but stream data should maintain -disk1.img as the ftype and item key.
 - -root.tar.gz and -root.tar.xz files are not produced
 - squashfs images are available (and also were available partially through
   xenial)

As a cleanup, we a helper to ubuntu_versions to compare versions more
directly. String compare was less useful in some cases and also would end
up breaking after 'z' because whatever release comes after z would string
sort to < z.

This also adds a simple tool for comparing two trees output of 'make exdata'.
    ./tools/compare-test-data ../trunk.dist ./
will output a 'diff' from old to new list of files indexed.