Merge lp:~blake-rouse/maas/bootresource-simplestreams-endpoint into lp:~maas-committers/maas/trunk

[Blake Rouse]

Added simplestreams endpoint on the region controller at path images-stream. Cluster controllers will connect to this endpoint to download the boot resource image data.

[Gavin Panella (allenap)]

Looks good... but I don't understand a lot of it.

The branch is a bit too big. I don't mean just line count, I mean that there's a lot going on. Things like TransactionWrapper and its tests could probably have been landed separately, and SimpleStreamsHandler could have been landed/reviewed in stages.

Pretty much all tests could do with some comments to explain what's going on. Remember that I'm you in about a month, when you've lost all the context you've currently got in-brain.

Basically it _looks_ fine, but add some comments, please :) Also, I have a suggestion for a simpler TransactionWrapper.

review: Needs Information

[Blake Rouse (blake-rouse)]

I will add some comments to the tests, fix the TransactionWrapper, and try to use django url routing.

Thanks for the review.

[Gavin Panella (allenap)]

Tip top, thanks for the replies. I'm around tomorrow so ping me whenever you need a follow-up.

[Blake Rouse (blake-rouse)]

Having an issue with the TransactionWrapper implementation you suggested. See inline comment.

[Gavin Panella (allenap)]

Changes look good.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~blake-rouse/maas/bootresource-simplestreams-endpoint into lp:maas failed. Below is the output from the failed tests.

Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [59.7 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:5 http://security.ubuntu.com trusty-security/main Sources [39.6 kB]
Get:6 http://security.ubuntu.com trusty-security/universe Sources [11.3 kB]
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [129 kB]
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [46.2 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [108 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [74.9 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [293 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [181 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Fetched 1,005 kB in 0s (1,920 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi p...

=== added file 'src/maasserver/bootresources.py'
--- src/maasserver/bootresources.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/bootresources.py	2014-08-15 01:28:48 +0000
@@ -0,0 +1,265 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Boot Resources."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = [
+    "get_simplestream_endpoint",
+    "simplestreams_file_handler",
+    "simplestreams_stream_handler",
+    "SIMPLESTREAMS_URL_REGEXP",
+]
+
+from django.db import (
+    close_old_connections,
+    transaction,
+    )
+from django.http import (
+    Http404,
+    HttpResponse,
+    StreamingHttpResponse,
+    )
+from django.shortcuts import get_object_or_404
+from maasserver.enum import BOOT_RESOURCE_TYPE
+from maasserver.models import (
+    BootResource,
+    BootResourceFile,
+    BootResourceSet,
+    )
+from maasserver.utils import absolute_reverse
+from simplestreams import util as sutil
+
+# Used by maasserver.middleware.AccessMiddleware to allow
+# anonymous access to the simplestreams endpoint.
+SIMPLESTREAMS_URL_REGEXP = '^/images-stream/'
+
+
+def get_simplestream_endpoint():
+    """Returns the simplestreams endpoint for the Region."""
+    return {
+        'url': absolute_reverse(
+            'simplestreams_stream_handler', kwargs={'filename': 'index.json'}),
+        'selections': [],
+        }
+
+
+class TransactionWrapper:
+    """Wraps `LargeObjectFile` in transaction, so `StreamingHttpResponse`
+    can be used. Once the stream is done, then the transaction is
+    closed.
+    """
+
+    def __init__(self, largeobject):
+        self.largeobject = largeobject
+        self._atomic = None
+        self._stream = None
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        if self._atomic is None:
+            self._atomic = transaction.atomic()
+            self._atomic.__enter__()
+        if self._stream is None:
+            self._stream = self.largeobject.open('rb')
+        data = self._stream.read(self.largeobject.block_size)
+        if len(data) == 0:
+            raise StopIteration
+        return data
+
+    def close(self):
+        if self._stream is not None:
+            self._stream.close()
+            self._stream = None
+        if self._atomic is not None:
+            self._atomic.__exit__()
+            self._atomic = None
+        close_old_connections()
+
+
+class SimpleStreamsHandler:
+    """Simplestreams endpoint, that the clusters talk to.
+
+    This is not called from piston, as piston uses emitters which
+    breaks the ability to return streaming content.
+
+    Anyone can access this endpoint. No credentials are required.
+    """
+
+    def get_json_response(self, content):
+        """Return `HttpResponse` for JSON content."""
+        response = HttpResponse(content.encode('utf-8'))
+        response['Content-Type'] = "application/json"
+        return response
+
+    def get_boot_resource_identifiers(self, resource):
+        """Return tuple (os, arch, subarch, series) for the given resource."""
+        arch, subarch = resource.split_arch()
+        if resource.rtype == BOOT_RESOURCE_TYPE.UPLOADED:
+            os = 'custom'
+            series = resource.name
+        else:
+            os, series = resource.name.split('/')
+        return (os, arch, subarch, series)
+
+    def get_product_name(self, resource):
+        """Return product name for the given resource."""
+        return 'maas:boot:%s:%s:%s:%s' % self.get_boot_resource_identifiers(
+            resource)
+
+    def gen_complete_boot_resources(self):
+        """Return generator of `BootResource` that contains a complete set."""
+        resources = BootResource.objects.all()
+        for resource in resources:
+            # Only add resources that have a complete set.
+            if resource.get_latest_complete_set() is None:
+                continue
+            yield resource
+
+    def gen_products_names(self):
+        """Return generator of avaliable products on the endpoint."""
+        for resource in self.gen_complete_boot_resources():
+            yield self.get_product_name(resource)
+
+    def get_product_index(self):
+        """Returns the streams product index `index.json`."""
+        products = list(self.gen_products_names())
+        updated = sutil.timestamp()
+        index = {
+            'index': {
+                'maas:v2:download': {
+                    'datatype': "image-downloads",
+                    'path': "streams/v1/maas:v2:download.json",
+                    'updated': updated,
+                    'products': products,
+                    'format': "products:1.0",
+                    },
+                },
+            'updated': updated,
+            'format': "index:1.0"
+            }
+        data = sutil.dump_data(index) + "\n"
+        return self.get_json_response(data)
+
+    def get_product_item(self, resource, resource_set, rfile):
+        """Returns the item description for the `rfile`."""
+        os, arch, subarch, series = self.get_boot_resource_identifiers(
+            resource)
+        path = '%s/%s/%s/%s/%s/%s' % (
+            os, arch, subarch, series, resource_set.version, rfile.filename)
+        item = {
+            'path': path,
+            'ftype': rfile.filetype,
+            'sha256': rfile.largefile.sha256,
+            'size': rfile.largefile.total_size,
+            }
+        item.update(rfile.extra)
+        return item
+
+    def get_product_data(self, resource):
+        """Returns the product data for this resource."""
+        os, arch, subarch, series = self.get_boot_resource_identifiers(
+            resource)
+        versions = {}
+        label = None
+        for resource_set in resource.sets.order_by('id').reverse():
+            if not resource_set.complete:
+                continue
+            # Set the label to the latest complete set label. In most cases the
+            # label will be the same for all sets. Only time it will differ is
+            # when daily has been enabled for a resource, that was previously
+            # only release. Only the latest version of the resource will be
+            # downloaded.
+            if label is None:
+                label = resource_set.label
+            items = {
+                rfile.filename: self.get_product_item(
+                    resource, resource_set, rfile)
+                for rfile in resource_set.files.all()
+                }
+            versions[resource_set.version] = {
+                'items': items
+                }
+        product = {
+            'versions': versions,
+            'subarch': subarch,
+            'label': label,
+            'version': series,
+            'arch': arch,
+            'release': series,
+            'krel': series,
+            'os': os,
+            }
+        product.update(resource.extra)
+        return product
+
+    def get_product_download(self):
+        """Returns the streams download index `download.json`."""
+        products = {}
+        for resource in self.gen_complete_boot_resources():
+            name = self.get_product_name(resource)
+            products[name] = self.get_product_data(resource)
+        updated = sutil.timestamp()
+        index = {
+            'datatype': "image-downloads",
+            'updated': updated,
+            'content_id': "maas:v2:download",
+            'products': products,
+            'format': "products:1.0"
+            }
+        data = sutil.dump_data(index) + "\n"
+        return self.get_json_response(data)
+
+    def streams_handler(self, request, filename):
+        """Handles requests into the "streams/" content."""
+        if filename == 'index.json':
+            return self.get_product_index()
+        elif filename == 'maas:v2:download.json':
+            return self.get_product_download()
+        raise Http404()
+
+    def files_handler(
+            self, request, os, arch, subarch, series, version, filename):
+        """Handles requests for getting the boot resource data."""
+        if os == "custom":
+            name = series
+        else:
+            name = '%s/%s' % (os, series)
+        arch = '%s/%s' % (arch, subarch)
+        resource = get_object_or_404(
+            BootResource, name=name, architecture=arch)
+        try:
+            resource_set = resource.sets.get(version=version)
+        except BootResourceSet.DoesNotExist:
+            raise Http404()
+        try:
+            rfile = resource_set.files.get(filename=filename)
+        except BootResourceFile.DoesNotExist:
+            raise Http404()
+        response = StreamingHttpResponse(
+            TransactionWrapper(rfile.largefile.content),
+            content_type='application/octet-stream')
+        response['Content-Length'] = rfile.largefile.total_size
+        return response
+
+
+def simplestreams_stream_handler(request, filename):
+    handler = SimpleStreamsHandler()
+    return handler.streams_handler(request, filename)
+
+
+def simplestreams_file_handler(
+        request, os, arch, subarch, series, version, filename):
+    handler = SimpleStreamsHandler()
+    return handler.files_handler(
+        request, os, arch, subarch, series, version, filename)

=== modified file 'src/maasserver/middleware.py'
--- src/maasserver/middleware.py	2014-02-11 12:19:25 +0000
+++ src/maasserver/middleware.py	2014-08-15 01:28:48 +0000
@@ -52,6 +52,7 @@
     build_request_repr = repr
 from django.utils.http import urlquote_plus
 from maasserver import logger
+from maasserver.bootresources import SIMPLESTREAMS_URL_REGEXP
 from maasserver.exceptions import (
     ExternalComponentException,
     MAASAPIException,
@@ -97,6 +98,8 @@
             reverse('metadata'),
             # RPC information is for use by clusters; no login.
             reverse('rpc-info'),
+            # Boot resources simple streams endpoint; no login.
+            SIMPLESTREAMS_URL_REGEXP,
             # API calls are protected by piston.
             settings.API_URL_REGEXP,
             ]

=== added file 'src/maasserver/tests/test_bootresources.py'
--- src/maasserver/tests/test_bootresources.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/tests/test_bootresources.py	2014-08-15 01:28:48 +0000
@@ -0,0 +1,421 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Test maasserver.bootresources."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = []
+
+import httplib
+import json
+from random import randint
+
+from django.core.urlresolvers import reverse
+from django.db import transaction
+from django.http import StreamingHttpResponse
+from django.test.client import Client
+from maasserver.bootresources import get_simplestream_endpoint
+from maasserver.enum import (
+    BOOT_RESOURCE_FILE_TYPE,
+    BOOT_RESOURCE_TYPE,
+    )
+from maasserver.testing.factory import factory
+from maasserver.testing.testcase import MAASServerTestCase
+from maasserver.utils import absolute_reverse
+from maastesting.testcase import MAASTestCase
+from testtools.matchers import ContainsAll
+
+
+class TestHelpers(MAASServerTestCase):
+    """Tests for `maasserver.bootresources` helpers."""
+
+    def test_get_simplestreams_endpoint(self):
+        endpoint = get_simplestream_endpoint()
+        self.assertEqual(
+            absolute_reverse(
+                'simplestreams_stream_handler',
+                kwargs={'filename': 'index.json'}),
+            endpoint['url'])
+        self.assertEqual([], endpoint['selections'])
+
+
+class TestSimpleStreamsHandler(MAASServerTestCase):
+    """Tests for `maasserver.bootresources.SimpleStreamsHandler`."""
+
+    def reverse_stream_handler(self, filename):
+        return reverse(
+            'simplestreams_stream_handler', kwargs={'filename': filename})
+
+    def reverse_file_handler(
+            self, os, arch, subarch, series, version, filename):
+        return reverse(
+            'simplestreams_file_handler', kwargs={
+                'os': os,
+                'arch': arch,
+                'subarch': subarch,
+                'series': series,
+                'version': version,
+                'filename': filename,
+                })
+
+    def get_stream_client(self, filename):
+        return self.client.get(self.reverse_stream_handler(filename))
+
+    def get_file_client(self, os, arch, subarch, series, version, filename):
+        return self.client.get(
+            self.reverse_file_handler(
+                os, arch, subarch, series, version, filename))
+
+    def get_product_name_for_resource(self, resource):
+        arch, subarch = resource.architecture.split('/')
+        if resource.rtype == BOOT_RESOURCE_TYPE.UPLOADED:
+            os = 'custom'
+            series = resource.name
+        else:
+            os, series = resource.name.split('/')
+        return 'maas:boot:%s:%s:%s:%s' % (os, arch, subarch, series)
+
+    def make_usable_product_boot_resource(self):
+        resource = factory.make_usable_boot_resource()
+        return self.get_product_name_for_resource(resource), resource
+
+    def test_streams_other_than_allowed_returns_404(self):
+        allowed_paths = [
+            'index.json',
+            'maas:v2:download.json',
+            ]
+        invalid_paths = [
+            '%s.json' % factory.make_name('path')
+            for _ in range(3)
+            ]
+        for path in allowed_paths:
+            response = self.get_stream_client(path)
+            self.assertEqual(httplib.OK, response.status_code)
+        for path in invalid_paths:
+            response = self.get_stream_client(path)
+            self.assertEqual(httplib.NOT_FOUND, response.status_code)
+
+    def test_streams_product_index_contains_keys(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertThat(output, ContainsAll(['index', 'updated', 'format']))
+
+    def test_streams_product_index_format_is_index_1(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertEqual('index:1.0', output['format'])
+
+    def test_streams_product_index_index_has_maas_v2_download(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertThat(output['index'], ContainsAll(['maas:v2:download']))
+
+    def test_streams_product_index_maas_v2_download_contains_keys(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertThat(
+            output['index']['maas:v2:download'],
+            ContainsAll([
+                'datatype', 'path', 'updated', 'products', 'format']))
+
+    def test_streams_product_index_maas_v2_download_has_valid_values(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertEqual(
+            'image-downloads',
+            output['index']['maas:v2:download']['datatype'])
+        self.assertEqual(
+            'streams/v1/maas:v2:download.json',
+            output['index']['maas:v2:download']['path'])
+        self.assertEqual(
+            'products:1.0',
+            output['index']['maas:v2:download']['format'])
+
+    def test_streams_product_index_empty_products(self):
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertEqual(
+            [],
+            output['index']['maas:v2:download']['products'])
+
+    def test_streams_product_index_empty_with_incomplete_resource(self):
+        resource = factory.make_boot_resource()
+        factory.make_boot_resource_set(resource)
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        self.assertEqual(
+            [],
+            output['index']['maas:v2:download']['products'])
+
+    def test_streams_product_index_with_resources(self):
+        products = []
+        for _ in range(3):
+            product, _ = self.make_usable_product_boot_resource()
+            products.append(product)
+        response = self.get_stream_client('index.json')
+        output = json.loads(response.content)
+        # Product listing should be the same as all of the completed
+        # boot resources in the database.
+        self.assertItemsEqual(
+            products,
+            output['index']['maas:v2:download']['products'])
+
+    def test_streams_product_download_contains_keys(self):
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertThat(output, ContainsAll([
+            'datatype', 'updated', 'content_id', 'products', 'format']))
+
+    def test_streams_product_download_has_valid_values(self):
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertEqual('image-downloads', output['datatype'])
+        self.assertEqual('maas:v2:download', output['content_id'])
+        self.assertEqual('products:1.0', output['format'])
+
+    def test_streams_product_download_empty_products(self):
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertEqual(
+            {},
+            output['products'])
+
+    def test_streams_product_download_empty_with_incomplete_resource(self):
+        resource = factory.make_boot_resource()
+        factory.make_boot_resource_set(resource)
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertEqual(
+            {},
+            output['products'])
+
+    def test_streams_product_download_has_valid_product_keys(self):
+        products = []
+        for _ in range(3):
+            product, _ = self.make_usable_product_boot_resource()
+            products.append(product)
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        # Product listing should be the same as all of the completed
+        # boot resources in the database.
+        self.assertThat(
+            output['products'],
+            ContainsAll(products))
+
+    def test_streams_product_download_product_contains_keys(self):
+        product, _ = self.make_usable_product_boot_resource()
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertThat(
+            output['products'][product],
+            ContainsAll([
+                'versions', 'subarch', 'label', 'version',
+                'arch', 'release', 'krel', 'os']))
+
+    def test_streams_product_download_product_has_valid_values(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        label = resource.get_latest_complete_set().label
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        output_product = output['products'][product]
+        self.assertEqual(subarch, output_product['subarch'])
+        self.assertEqual(label, output_product['label'])
+        self.assertEqual(series, output_product['version'])
+        self.assertEqual(arch, output_product['arch'])
+        self.assertEqual(series, output_product['release'])
+        self.assertEqual(series, output_product['krel'])
+        self.assertEqual(os, output_product['os'])
+        for key, value in resource.extra.items():
+            self.assertEqual(value, output_product[key])
+
+    def test_streams_product_download_product_uses_latest_complete_label(self):
+        product, resource = self.make_usable_product_boot_resource()
+        # Incomplete resource_set
+        factory.make_boot_resource_set(resource)
+        newest_set = factory.make_boot_resource_set(resource)
+        factory.make_boot_resource_file_with_content(newest_set)
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        output_product = output['products'][product]
+        self.assertEqual(newest_set.label, output_product['label'])
+
+    def test_streams_product_download_product_contains_multiple_versions(self):
+        resource = factory.make_boot_resource()
+        resource_sets = [
+            factory.make_boot_resource_set(resource)
+            for _ in range(3)
+            ]
+        versions = []
+        for resource_set in resource_sets:
+            factory.make_boot_resource_file_with_content(resource_set)
+            versions.append(resource_set.version)
+        product = self.get_product_name_for_resource(resource)
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        self.assertThat(
+            output['products'][product]['versions'],
+            ContainsAll(versions))
+
+    def test_streams_product_download_product_version_contains_items(self):
+        product, resource = self.make_usable_product_boot_resource()
+        resource_set = resource.get_latest_complete_set()
+        items = [
+            rfile.filename
+            for rfile in resource_set.files.all()
+            ]
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        version = output['products'][product]['versions'][resource_set.version]
+        self.assertThat(
+            version['items'],
+            ContainsAll(items))
+
+    def test_streams_product_download_product_item_contains_keys(self):
+        product, resource = self.make_usable_product_boot_resource()
+        resource_set = resource.get_latest_complete_set()
+        resource_file = resource_set.files.order_by('?')[0]
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        version = output['products'][product]['versions'][resource_set.version]
+        self.assertThat(
+            version['items'][resource_file.filename],
+            ContainsAll(['path', 'ftype', 'sha256', 'size']))
+
+    def test_streams_product_download_product_item_has_valid_values(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        resource_set = resource.get_latest_complete_set()
+        resource_file = resource_set.files.order_by('?')[0]
+        path = '%s/%s/%s/%s/%s/%s' % (
+            os, arch, subarch, series, resource_set.version,
+            resource_file.filename)
+        response = self.get_stream_client('maas:v2:download.json')
+        output = json.loads(response.content)
+        version = output['products'][product]['versions'][resource_set.version]
+        item = version['items'][resource_file.filename]
+        self.assertEqual(path, item['path'])
+        self.assertEqual(resource_file.filetype, item['ftype'])
+        self.assertEqual(resource_file.largefile.sha256, item['sha256'])
+        self.assertEqual(resource_file.largefile.total_size, item['size'])
+        for key, value in resource_file.extra.items():
+            self.assertEqual(value, item[key])
+
+    def test_download_invalid_boot_resource_returns_404(self):
+        os = factory.make_name('os')
+        series = factory.make_name('series')
+        arch = factory.make_name('arch')
+        subarch = factory.make_name('subarch')
+        version = factory.make_name('version')
+        filename = factory.make_name('filename')
+        response = self.get_file_client(
+            os, arch, subarch, series, version, filename)
+        self.assertEqual(httplib.NOT_FOUND, response.status_code)
+
+    def test_download_invalid_version_returns_404(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        version = factory.make_name('version')
+        filename = factory.make_name('filename')
+        response = self.get_file_client(
+            os, arch, subarch, series, version, filename)
+        self.assertEqual(httplib.NOT_FOUND, response.status_code)
+
+    def test_download_invalid_filename_returns_404(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        resource_set = resource.get_latest_complete_set()
+        version = resource_set.version
+        filename = factory.make_name('filename')
+        response = self.get_file_client(
+            os, arch, subarch, series, version, filename)
+        self.assertEqual(httplib.NOT_FOUND, response.status_code)
+
+    def test_download_valid_path_returns_200(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        resource_set = resource.get_latest_complete_set()
+        version = resource_set.version
+        resource_file = resource_set.files.order_by('?')[0]
+        filename = resource_file.filename
+        response = self.get_file_client(
+            os, arch, subarch, series, version, filename)
+        self.assertEqual(httplib.OK, response.status_code)
+
+    def test_download_returns_streaming_response(self):
+        product, resource = self.make_usable_product_boot_resource()
+        _, _, os, arch, subarch, series = product.split(':')
+        resource_set = resource.get_latest_complete_set()
+        version = resource_set.version
+        resource_file = resource_set.files.order_by('?')[0]
+        filename = resource_file.filename
+        with resource_file.largefile.content.open('rb') as stream:
+            content = stream.read()
+        response = self.get_file_client(
+            os, arch, subarch, series, version, filename)
+        self.assertIsInstance(response, StreamingHttpResponse)
+        self.assertEqual(content, b''.join(response.streaming_content))
+
+
+class TestTransactionWrapper(MAASTestCase):
+    """Tests the use of StreamingHttpResponse(TransactionWrapper(stream)).
+
+    We do not run this inside of `MAASServerTestCase` as that wraps a
+    transaction around each test. This removes that behavior so we can
+    test that the transaction is remaining open for all of the content.
+    """
+
+    def test_download(self):
+        # Do the setup inside of a transaction, as we are running in a test
+        # that doesn't enable transactions per test.
+        with transaction.atomic():
+            os = factory.make_name('os')
+            series = factory.make_name('series')
+            arch = factory.make_name('arch')
+            subarch = factory.make_name('subarch')
+            name = '%s/%s' % (os, series)
+            architecture = '%s/%s' % (arch, subarch)
+            version = factory.make_name('version')
+            filetype = factory.pick_enum(BOOT_RESOURCE_FILE_TYPE)
+            # We set the filename to the same value as filetype, as in most
+            # cases this will always be true. The simplestreams content from
+            # maas.ubuntu.com, is formatted this way.
+            filename = filetype
+            size = randint(1024, 2048)
+            content = factory.make_string(size=size)
+            resource = factory.make_boot_resource(
+                rtype=BOOT_RESOURCE_TYPE.SYNCED, name=name,
+                architecture=architecture)
+            resource_set = factory.make_boot_resource_set(
+                resource, version=version)
+            largefile = factory.make_large_file(content=content, size=size)
+            factory.make_boot_resource_file(
+                resource_set, largefile, filename=filename, filetype=filetype)
+
+        # Outside of the transaction, we run the actual test. The client will
+        # run inside of its own transaction, but once the streaming response
+        # is returned that transaction will be closed.
+        client = Client()
+        response = client.get(
+            reverse(
+                'simplestreams_file_handler', kwargs={
+                    'os': os,
+                    'arch': arch,
+                    'subarch': subarch,
+                    'series': series,
+                    'version': version,
+                    'filename': filename,
+                    }))
+
+        # If TransactionWrapper does not work, then a ProgramError will be
+        # thrown. If it works then content will match.
+        self.assertEqual(content, b''.join(response.streaming_content))
+        self.assertTrue(largefile.content.closed)

=== modified file 'src/maasserver/urls.py'
--- src/maasserver/urls.py	2014-08-13 21:49:35 +0000
+++ src/maasserver/urls.py	2014-08-15 01:28:48 +0000
@@ -21,6 +21,10 @@
     url,
     )
 from django.contrib.auth.decorators import user_passes_test
+from maasserver.bootresources import (
+    simplestreams_file_handler,
+    simplestreams_stream_handler,
+    )
 from maasserver.enum import NODEGROUP_STATUS
 from maasserver.models import Node
 from maasserver.views import TextTemplateView
@@ -104,6 +108,13 @@
     'maasserver.views',
     url(r'^accounts/login/$', login, name='login'),
     url(
+        r'^images-stream/streams/v1/(?P<filename>.*)$',
+        simplestreams_stream_handler, name='simplestreams_stream_handler'),
+    url(
+        r'^images-stream/(?P<os>.*)/(?P<arch>.*)/(?P<subarch>.*)/'
+        '(?P<series>.*)/(?P<version>.*)/(?P<filename>.*)$',
+        simplestreams_file_handler, name='simplestreams_file_handler'),
+    url(
         r'^robots\.txt$', TextTemplateView.as_view(
             template_name='maasserver/robots.txt'),
         name='robots'),