Merge lp:~blake-rouse/maas/maascli-multipart-upload into lp:~maas-committers/maas/trunk

[Blake Rouse]

Modified the maascli to allow for custom Action classes based on the handler and action. Added the BootResourceCreateAction, that performs the upload of the content parameter using multiple part uploading, that the MAAS API now supports.

Also contains a drive by fix for LargeFile not being deleted correctly when the reference object is the last.

[Jason Hobbs (jason-hobbs)]

I have comments and questions inline but I don't think any of them are blockers.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~blake-rouse/maas/maascli-multipart-upload into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [59.7 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [43.8 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security/universe Sources [10.8 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [139 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [47.2 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [118 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [83.7 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [316 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [201 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Fetched 1,080 kB in 0s (1,770 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi pyt...

=== added directory 'src/maascli/actions'
=== added file 'src/maascli/actions/__init__.py'
=== added file 'src/maascli/actions/boot_resources_create.py'
--- src/maascli/actions/boot_resources_create.py	1970-01-01 00:00:00 +0000
+++ src/maascli/actions/boot_resources_create.py	2014-09-12 03:32:27 +0000
@@ -0,0 +1,186 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""MAAS Boot Resources Action."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = [
+    'action_class'
+    ]
+
+from cStringIO import StringIO
+import hashlib
+import json
+from urlparse import urljoin
+
+from apiclient.multipart import (
+    build_multipart_message,
+    encode_multipart_message,
+    )
+from maascli.api import (
+    Action,
+    http_request,
+    )
+from maascli.command import CommandError
+
+# Send 4MB of data per request.
+CHUNK_SIZE = 1 << 22
+
+
+class BootResourcesCreateAction(Action):
+    """Provides custom logic to the boot-resources create action.
+
+    Command: maas username boot-resources create
+
+    The create command has the ability to upload the content in pieces, using
+    the upload_uri that is returned in the response. This class provides the
+    logic to upload over that API.
+    """
+
+    def __call__(self, options):
+        # TODO: this is el-cheapo URI Template
+        # <http://tools.ietf.org/html/rfc6570> support; use uritemplate-py
+        # <https://github.com/uri-templates/uritemplate-py> here?
+        uri = self.uri.format(**vars(options))
+        content = self.initial_request(uri, options)
+
+        # Get the created resource file for the boot resource
+        rfile = self.get_resource_file(content)
+        if rfile is None:
+            print("Failed to identify created resource.")
+            raise CommandError(2)
+        if rfile['complete']:
+            # File already existed in the database, so no
+            # reason to upload it.
+            return
+
+        # Upload content
+        data = dict(options.data)
+        upload_uri = urljoin(uri, rfile['upload_uri'])
+        self.upload_content(
+            upload_uri, data['content'], insecure=options.insecure)
+
+    def initial_request(self, uri, options):
+        """Performs the initial POST request, to create the boot resource."""
+        # Bundle things up ready to throw over the wire.
+        body, headers = self.prepare_initial_payload(options.data)
+
+        # Headers are returned as a list, but they must be a dict for
+        # the signing machinery.
+        headers = dict(headers)
+
+        # Sign request if credentials have been provided.
+        if self.credentials is not None:
+            self.sign(uri, headers, self.credentials)
+
+        # Use httplib2 instead of urllib2 (or MAASDispatcher, which is based
+        # on urllib2) so that we get full control over HTTP method. TODO:
+        # create custom MAASDispatcher to use httplib2 so that MAASClient can
+        # be used.
+        response, content = http_request(
+            uri, self.method, body=body, headers=headers,
+            insecure=options.insecure)
+
+        # 2xx status codes are all okay.
+        if response.status // 100 != 2:
+            if options.debug:
+                self.print_debug(response)
+            self.print_response(response, content)
+            raise CommandError(2)
+        return content
+
+    def prepare_initial_payload(self, data):
+        """Return the body and headers for the initial request.
+
+        This is method is only used for the first request to MAAS. It
+        removes the passed content, and replaces it with the sha256 and size
+        of that content.
+
+        :param data: An iterable of ``name, value`` or ``name, opener``
+            tuples (see `name_value_pair`) to pack into the body or
+            query, depending on the type of request.
+        """
+        data = dict(data)
+        if 'content' not in data:
+            print('Missing content.')
+            raise CommandError(2)
+
+        content = data.pop('content')
+        size, sha256 = self.calculate_size_and_sha256(content)
+        data['size'] = '%s' % size
+        data['sha256'] = sha256
+
+        data = [(key, value) for key, value in data.items()]
+        message = build_multipart_message(data)
+        headers, body = encode_multipart_message(message)
+        return body, headers
+
+    def calculate_size_and_sha256(self, content):
+        """Return the size and sha256 of the content."""
+        size = 0
+        sha256 = hashlib.sha256()
+        with content() as fd:
+            while True:
+                buf = fd.read(CHUNK_SIZE)
+                length = len(buf)
+                size += length
+                sha256.update(buf)
+                if length != CHUNK_SIZE:
+                    break
+        return size, sha256.hexdigest()
+
+    def get_resource_file(self, content):
+        """Return the boot resource file for the created file."""
+        data = json.loads(content)
+        if len(data['sets']) == 0:
+            # No sets returned, no way to get the resource file.
+            return None
+        newest_set = sorted(data['sets'].keys(), reverse=True)[0]
+        resource_set = data['sets'][newest_set]
+        if len(resource_set['files']) != 1:
+            # This api only supports uploading one file. If the set doesn't
+            # have just one file, then there is no way of knowing which file.
+            return None
+        _, rfile = resource_set['files'].popitem()
+        return rfile
+
+    def put_upload(self, upload_uri, data, insecure=False):
+        """Send PUT method to upload data."""
+        headers = {
+            'Content-Type': 'application/octet-stream',
+            'Content-Length': '%s' % len(data),
+        }
+        if self.credentials is not None:
+            self.sign(upload_uri, headers, self.credentials)
+        # httplib2 expects the body to be file-like if its binary data
+        data = StringIO(data)
+        response, content = http_request(
+            upload_uri, 'PUT', body=data, headers=headers,
+            insecure=insecure)
+        if response.status != 200:
+            self.print_response(response, content)
+            raise CommandError(2)
+
+    def upload_content(self, upload_uri, content, insecure=False):
+        """Upload the content in chunks."""
+        with content() as fd:
+            while True:
+                buf = fd.read(CHUNK_SIZE)
+                length = len(buf)
+                if length > 0:
+                    self.put_upload(upload_uri, buf, insecure=insecure)
+                if length != CHUNK_SIZE:
+                    break
+
+
+# Each action sets this variable so the class can be picked up
+# by get_action_class.
+action_class = BootResourcesCreateAction
 
=== added directory 'src/maascli/actions/tests'
=== added file 'src/maascli/actions/tests/__init__.py'
=== added file 'src/maascli/actions/tests/test_boot_resources_create.py'
--- src/maascli/actions/tests/test_boot_resources_create.py	1970-01-01 00:00:00 +0000
+++ src/maascli/actions/tests/test_boot_resources_create.py	2014-09-12 03:32:27 +0000
@@ -0,0 +1,207 @@
+# Copyright 2012-2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Test for boot-resources create action."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = []
+
+from functools import partial
+import hashlib
+import json
+import os
+from random import randint
+
+from apiclient.testing.credentials import make_api_credentials
+import httplib2
+from maascli.actions import boot_resources_create
+from maascli.actions.boot_resources_create import (
+    BootResourcesCreateAction,
+    CHUNK_SIZE,
+    )
+from maascli.command import CommandError
+from maastesting.factory import factory
+from maastesting.fixtures import TempDirectory
+from maastesting.matchers import MockCalledOnceWith
+from maastesting.testcase import MAASTestCase
+from mock import (
+    ANY,
+    Mock,
+    sentinel,
+    )
+
+
+class TestBootResourcesCreateAction(MAASTestCase):
+    """Tests for `BootResourcesCreateAction`."""
+
+    def configure_http_request(self, status, content):
+        response = httplib2.Response({'status': status})
+        self.patch(
+            boot_resources_create,
+            'http_request').return_value = (response, content)
+
+    def make_boot_resources_create_action(self):
+        action_name = "create".encode("ascii")
+        action_bases = (BootResourcesCreateAction,)
+        action_ns = {
+            "action": {'method': 'POST'},
+            "handler": {'uri': '/api/1.0/boot-resources/', 'params': []},
+            "profile": {'credentials': make_api_credentials()}
+            }
+        action_class = type(action_name, action_bases, action_ns)
+        action = action_class(Mock())
+        self.patch(action, 'print_debug')
+        self.patch(action, 'print_response')
+        return action
+
+    def make_content(self, size=None):
+        tmpdir = self.useFixture(TempDirectory()).path
+        if size is None:
+            size = randint(1024, 2048)
+        data = factory.make_bytes(size)
+        content_path = os.path.join(tmpdir, 'content')
+        with open(content_path, 'wb') as stream:
+            stream.write(data)
+        sha256 = hashlib.sha256()
+        sha256.update(data)
+        return size, sha256.hexdigest(), partial(open, content_path, "rb")
+
+    def test_initial_request_returns_content(self):
+        content = factory.make_name('content')
+        self.configure_http_request(200, content)
+        action = self.make_boot_resources_create_action()
+        self.patch(action, 'prepare_initial_payload').return_value = ("", {})
+        self.assertEqual(
+            content, action.initial_request(sentinel.uri, Mock()))
+
+    def test_initial_request_raises_CommandError_on_error(self):
+        self.configure_http_request(500, factory.make_name('content'))
+        action = self.make_boot_resources_create_action()
+        self.patch(action, 'prepare_initial_payload').return_value = ("", {})
+        self.assertRaises(
+            CommandError,
+            action.initial_request, sentinel.uri, Mock())
+
+    def test_prepare_initial_payload_raises_CommandError_missing_content(self):
+        action = self.make_boot_resources_create_action()
+        self.patch(boot_resources_create, 'print')
+        self.assertRaises(
+            CommandError,
+            action.prepare_initial_payload, [('invalid', '')])
+
+    def test_prepare_initial_payload_adds_size_and_sha256(self):
+        size, sha256, stream = self.make_content()
+        action = self.make_boot_resources_create_action()
+        mock_build_message = self.patch(
+            boot_resources_create,
+            'build_multipart_message')
+        self.patch(
+            boot_resources_create,
+            'encode_multipart_message').return_value = (None, None)
+        action.prepare_initial_payload([('content', stream)])
+        self.assertThat(
+            mock_build_message,
+            MockCalledOnceWith([('sha256', sha256), ('size', '%s' % size)]))
+
+    def test_get_resource_file_returns_None_when_no_sets(self):
+        content = {
+            'sets': {}
+            }
+        action = self.make_boot_resources_create_action()
+        self.assertIsNone(action.get_resource_file(json.dumps(content)))
+
+    def test_get_resource_file_returns_None_when_no_files(self):
+        content = {
+            'sets': {
+                '20140910': {
+                    'files': {}
+                    },
+                },
+            }
+        action = self.make_boot_resources_create_action()
+        self.assertIsNone(action.get_resource_file(json.dumps(content)))
+
+    def test_get_resource_file_returns_None_when_more_than_one_file(self):
+        content = {
+            'sets': {
+                '20140910': {
+                    'files': {
+                        'root-image.gz': {},
+                        'root-tgz': {},
+                        }
+                    },
+                },
+            }
+        action = self.make_boot_resources_create_action()
+        self.assertIsNone(action.get_resource_file(json.dumps(content)))
+
+    def test_get_resource_file_returns_file_from_newest_set(self):
+        filename = factory.make_name('file')
+        content = {
+            'sets': {
+                '20140910': {
+                    'files': {
+                        filename: {
+                            'name': filename,
+                            },
+                        },
+                    },
+                '20140909': {
+                    'files': {
+                        'other': {
+                            'name': 'other',
+                            },
+                        },
+                    },
+                },
+            }
+        action = self.make_boot_resources_create_action()
+        self.assertEqual(
+            {'name': filename},
+            action.get_resource_file(json.dumps(content)))
+
+    def test_put_upload_raise_CommandError_if_status_not_200(self):
+        self.configure_http_request(500, '')
+        action = self.make_boot_resources_create_action()
+        self.assertRaises(
+            CommandError,
+            action.put_upload, sentinel.upload_uri, '')
+
+    def test_put_upload_sends_content_type_and_length_headers(self):
+        response = httplib2.Response({'status': 200})
+        mock_request = self.patch(boot_resources_create, 'http_request')
+        mock_request.return_value = (response, '')
+        action = self.make_boot_resources_create_action()
+        self.patch(action, 'sign')
+        data = factory.make_bytes()
+        action.put_upload(sentinel.upload_uri, data)
+        headers = {
+            'Content-Type': 'application/octet-stream',
+            'Content-Length': '%s' % len(data),
+            }
+        self.assertThat(
+            mock_request,
+            MockCalledOnceWith(
+                sentinel.upload_uri, 'PUT', body=ANY,
+                headers=headers, insecure=False))
+
+    def test_upload_content_calls_put_upload_with_sizeof_CHUNK_SIZE(self):
+        size = CHUNK_SIZE * 2
+        size, sha256, stream = self.make_content(size=size)
+        action = self.make_boot_resources_create_action()
+        mock_upload = self.patch(action, 'put_upload')
+        action.upload_content(sentinel.upload_uri, stream)
+
+        call_data_sizes = [
+            len(call[0][1])
+            for call in mock_upload.call_args_list
+            ]
+        self.assertEqual([CHUNK_SIZE, CHUNK_SIZE], call_data_sizes)
 
=== modified file 'src/maascli/api.py'
--- src/maascli/api.py	2014-08-13 21:49:35 +0000
+++ src/maascli/api.py	2014-09-12 03:32:27 +0000
@@ -53,6 +53,7 @@
     handler_command_name,
     parse_docstring,
     safe_name,
+    try_import_module,
     )
 
 
@@ -402,12 +403,32 @@
         sys.exit(0)
 
 
+def get_action_class(handler, action):
+    """Return custom action handler class."""
+    handler_name = handler_command_name(handler["name"]).replace('-', '_')
+    action_name = '%s_%s' % (
+        handler_name,
+        safe_name(action["name"]).replace('-', '_'))
+    action_module = try_import_module('maascli.actions.%s' % action_name)
+    if action_module is not None:
+        return action_module.action_class
+    return None
+
+
+def get_action_class_bases(handler, action):
+    """Return the base classes for the dynamic class."""
+    action_class = get_action_class(handler, action)
+    if action_class is not None:
+        return (action_class,)
+    return (Action,)
+
+
 def register_actions(profile, handler, parser):
     """Register a handler's actions."""
     for action in handler["actions"]:
         help_title, help_body = parse_docstring(action["doc"])
         action_name = safe_name(action["name"]).encode("ascii")
-        action_bases = (Action,)
+        action_bases = get_action_class_bases(handler, action)
         action_ns = {
             "action": action,
             "handler": handler,
 
=== modified file 'src/maascli/tests/test_api.py'
--- src/maascli/tests/test_api.py	2014-08-13 21:49:35 +0000
+++ src/maascli/tests/test_api.py	2014-09-12 03:32:27 +0000
@@ -24,6 +24,7 @@
 
 import httplib2
 from maascli import api
+from maascli.actions.boot_resources_create import BootResourcesCreateAction
 from maascli.command import CommandError
 from maascli.config import ProfileConfig
 from maascli.parser import ArgumentParser
@@ -179,6 +180,34 @@
             "disable the certificate check.")
         self.assertEqual(error_expected, "%s" % error)
 
+    def test_get_action_class_returns_None_for_unknown_handler(self):
+        handler = {'name': factory.make_name('handler')}
+        action = {'name': 'create'}
+        self.assertIsNone(api.get_action_class(handler, action))
+
+    def test_get_action_class_returns_BootResourcesCreateAction_class(self):
+        # Test uses BootResourcesCreateAction as its know to exist.
+        handler = {'name': 'BootResourcesHandler'}
+        action = {'name': 'create'}
+        self.assertEqual(
+            BootResourcesCreateAction,
+            api.get_action_class(handler, action))
+
+    def test_get_action_class_bases_returns_Action(self):
+        handler = {'name': factory.make_name('handler')}
+        action = {'name': 'create'}
+        self.assertEqual(
+            (api.Action,),
+            api.get_action_class_bases(handler, action))
+
+    def test_get_action_class_bases_returns_BootResourcesCreateAction(self):
+        # Test uses BootResourcesCreateAction as its know to exist.
+        handler = {'name': 'BootResourcesHandler'}
+        action = {'name': 'create'}
+        self.assertEqual(
+            (BootResourcesCreateAction,),
+            api.get_action_class_bases(handler, action))
+
 
 class TestIsResponseTextual(MAASTestCase):
     """Tests for `is_response_textual`."""
 
=== modified file 'src/maascli/utils.py'
--- src/maascli/utils.py	2013-10-07 09:31:09 +0000
+++ src/maascli/utils.py	2014-09-12 03:32:27 +0000
@@ -25,6 +25,7 @@
     getdoc,
     )
 import re
+import sys
 from urlparse import urlparse
 
 
@@ -107,3 +108,17 @@
     if re.search("/api/[0-9.]+/?$", url.path) is None:
         url = url._replace(path=url.path + "api/1.0/")
     return url.geturl()
+
+
+def import_module(import_str):
+    """Import a module."""
+    __import__(import_str)
+    return sys.modules[import_str]
+
+
+def try_import_module(import_str, default=None):
+    """Try to import a module."""
+    try:
+        return import_module(import_str)
+    except ImportError:
+        return default
 
=== modified file 'src/maasserver/api/boot_resources.py'
--- src/maasserver/api/boot_resources.py	2014-09-03 01:05:32 +0000
+++ src/maasserver/api/boot_resources.py	2014-09-12 03:32:27 +0000
@@ -273,12 +273,14 @@
 
     read = create = delete = None
 
+    hidden = True
+
     @admin_method
     def update(self, request, id, file_id):
         """Upload piece of boot resource file."""
         resource = get_object_or_404(BootResource, id=id)
         rfile = get_object_or_404(BootResourceFile, id=file_id)
-        size = request.META.get('CONTENT_LENGTH', 0)
+        size = int(request.META.get('CONTENT_LENGTH', '0'))
         data = request.body
         if size == 0:
             raise MAASAPIBadRequest("Missing data.")
 
=== modified file 'src/maasserver/api/boot_source_selections.py'
--- src/maasserver/api/boot_source_selections.py	2014-08-22 15:21:48 +0000
+++ src/maasserver/api/boot_source_selections.py	2014-09-12 03:32:27 +0000
@@ -101,7 +101,7 @@
     be set globally for the whole region and clusters. This api is now
     deprecated, and only exists for backwards compatibility.
     """
-    deprecated = True
+    hidden = True
 
     def read(self, request, uuid, boot_source_id, id):
         """Read a boot source selection."""
@@ -175,7 +175,7 @@
     be set globally for the whole region and clusters. This api is now
     deprecated, and only exists for backwards compatibility.
     """
-    deprecated = True
+    hidden = True
 
     def read(self, request, uuid, boot_source_id):
         """List boot source selections.
 
=== modified file 'src/maasserver/api/boot_sources.py'
--- src/maasserver/api/boot_sources.py	2014-08-22 15:21:48 +0000
+++ src/maasserver/api/boot_sources.py	2014-09-12 03:32:27 +0000
@@ -119,7 +119,7 @@
     be set globally for the whole region and clusters. This api is now
     deprecated, and only exists for backwards compatibility.
     """
-    deprecated = True
+    hidden = True
 
     def read(self, request, uuid, id):
         """Read a boot source."""
@@ -186,7 +186,7 @@
     be set globally for the whole region and clusters. This api is now
     deprecated, and only exists for backwards compatibility.
     """
-    deprecated = True
+    hidden = True
 
     def read(self, request, uuid):
         """List boot sources.
 
=== modified file 'src/maasserver/api/doc.py'
--- src/maasserver/api/doc.py	2014-08-22 15:21:48 +0000
+++ src/maasserver/api/doc.py	2014-09-12 03:32:27 +0000
@@ -43,12 +43,14 @@
     Handlers are of type :class:`HandlerMetaClass`, and must define a
     `resource_uri` method.
 
+    Handlers that have the attribute hidden set to True, will not be returned.
+
     :rtype: Generator, yielding handlers.
     """
     p_has_resource_uri = lambda resource: (
         getattr(resource.handler, "resource_uri", None) is not None)
-    p_is_not_deprecated = lambda resource: (
-        getattr(resource.handler, "deprecated", False))
+    p_is_not_hidden = lambda resource: (
+        getattr(resource.handler, "hidden", False))
     for pattern in resolver.url_patterns:
         if isinstance(pattern, RegexURLResolver):
             accumulate_api_resources(pattern, accumulator)
@@ -56,7 +58,7 @@
             if isinstance(pattern.callback, Resource):
                 resource = pattern.callback
                 if p_has_resource_uri(resource) and \
-                        not p_is_not_deprecated(resource):
+                        not p_is_not_hidden(resource):
                     accumulator.add(resource)
         else:
             raise AssertionError(
 
=== modified file 'src/maasserver/api/tests/test_doc.py'
--- src/maasserver/api/tests/test_doc.py	2014-08-20 23:54:33 +0000
+++ src/maasserver/api/tests/test_doc.py	2014-09-12 03:32:27 +0000
@@ -88,6 +88,18 @@
         module.urlpatterns = patterns("", url("^metal", resource))
         self.assertSetEqual({resource}, find_api_resources(module))
 
+    def test_urlpatterns_with_resource_hidden(self):
+        # Resources for handlers with resource_uri attributes are discovered
+        # in a urlconf module and returned, unless hidden = True.
+        handler = type(b"\m/", (BaseHandler,), {
+            "resource_uri": True,
+            "hidden": True,
+            })
+        resource = Resource(handler)
+        module = self.make_module()
+        module.urlpatterns = patterns("", url("^metal", resource))
+        self.assertSetEqual(set(), find_api_resources(module))
+
     def test_nested_urlpatterns_with_handler(self):
         # Resources are found in nested urlconfs.
         handler = type(b"\m/", (BaseHandler,), {"resource_uri": True})
 
=== modified file 'src/maasserver/models/bootresourcefile.py'
--- src/maasserver/models/bootresourcefile.py	2014-08-31 02:47:48 +0000
+++ src/maasserver/models/bootresourcefile.py	2014-09-12 03:32:27 +0000
@@ -20,6 +20,8 @@
     CharField,
     ForeignKey,
     )
+from django.db.models.signals import post_delete
+from django.dispatch import receiver
 from maasserver import DefaultMeta
 from maasserver.enum import (
     BOOT_RESOURCE_FILE_TYPE,
@@ -70,3 +72,16 @@
 
     def __repr__(self):
         return "<BootResourceFile %s/%s>" % (self.filename, self.filetype)
+
+
+@receiver(post_delete)
+def delete_large_file(sender, instance, **kwargs):
+    """Call delete on the LargeFile, now that the relation has been removed.
+    If this was the only resource file referencing this LargeFile then it will
+    be delete.
+
+    This is done using the `post_delete` signal because only then has the
+    relation been removed.
+    """
+    if sender == BootResourceFile:
+        instance.largefile.delete()