Merge lp:~blake-rouse/maas/fix-interfaces-read into lp:~maas-committers/maas/trunk

Proposed by Blake Rouse
Status: Merged
Approved by: Blake Rouse
Approved revision: no longer in the source branch.
Merged at revision: 6002
Proposed branch: lp:~blake-rouse/maas/fix-interfaces-read
Merge into: lp:~maas-committers/maas/trunk
Diff against target: 121 lines (+66/-3)
3 files modified
src/maasserver/api/interfaces.py (+20/-1)
src/maasserver/api/nodes.py (+8/-2)
src/maasserver/api/tests/test_interfaces.py (+38/-0)
To merge this branch: bzr merge lp:~blake-rouse/maas/fix-interfaces-read
Reviewer Review Type Date Requested Status
Mike Pontillo (community) Approve
Review via email: mp+322887@code.launchpad.net

Commit message

Prefetch interface data for GET interfaces.

This make the number of queries for interfaces read to always be the same. Doesn't matter the number of interfaces on the machine the number of queries will be the same.

Description of the change

make_complex_interfaces with read was 100+ queries, second time it was 200+ queries, now its around 50 always (no matter the number of interfaces).

To post a comment you must log in.
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Very nice!

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.9 MiB)

The attempt to merge lp:~blake-rouse/maas/fix-interfaces-read into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (605 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libnss-wrapper libpq-dev make nodejs-legacy npm postgresql psmisc pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
psmisc is already the newest version (22.21-2.1build1).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-net...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/maasserver/api/interfaces.py'
--- src/maasserver/api/interfaces.py 2017-03-29 13:34:01 +0000
+++ src/maasserver/api/interfaces.py 2017-04-20 22:42:24 +0000
@@ -124,7 +124,26 @@
124 """124 """
125 node = Node.objects.get_node_or_404(125 node = Node.objects.get_node_or_404(
126 system_id, request.user, NODE_PERMISSION.VIEW)126 system_id, request.user, NODE_PERMISSION.VIEW)
127 return node.interface_set.all()127 queryset = node.interface_set.all()
128 prefetch_related_fields = [
129 'node',
130 'vlan__primary_rack',
131 'vlan__secondary_rack',
132 'vlan__fabric__vlan_set',
133 'vlan__space',
134 'parents',
135 'ip_addresses__subnet',
136 # Prefetch 3 levels deep, anything more will require extra queries.
137 'children_relationships__child__vlan',
138 ('children_relationships__child__'
139 'children_relationships__child__vlan'),
140 ('children_relationships__child__'
141 'children_relationships__child__'
142 'children_relationships__child__vlan'),
143 ]
144 for prefetch in prefetch_related_fields:
145 queryset = queryset.prefetch_related(prefetch)
146 return queryset
128147
129 @operation(idempotent=False)148 @operation(idempotent=False)
130 def create_physical(self, request, system_id):149 def create_physical(self, request, system_id):
131150
=== modified file 'src/maasserver/api/nodes.py'
--- src/maasserver/api/nodes.py 2017-04-17 17:25:47 +0000
+++ src/maasserver/api/nodes.py 2017-04-20 22:42:24 +0000
@@ -352,14 +352,20 @@
352 'boot_interface__parents',352 'boot_interface__parents',
353 ('boot_interface__children_relationships__child__'353 ('boot_interface__children_relationships__child__'
354 'children_relationships__child'),354 'children_relationships__child'),
355 'interface_set__node',
355 'interface_set__vlan__primary_rack',356 'interface_set__vlan__primary_rack',
356 'interface_set__vlan__secondary_rack',357 'interface_set__vlan__secondary_rack',
357 'interface_set__vlan__fabric__vlan_set',358 'interface_set__vlan__fabric__vlan_set',
358 'interface_set__vlan__space',359 'interface_set__vlan__space',
359 'interface_set__parents',360 'interface_set__parents',
360 'interface_set__ip_addresses__subnet',361 'interface_set__ip_addresses__subnet',
361 ('interface_set__children_relationships__child__'362 # Prefetch 3 levels deep, anything more will require extra queries.
362 'children_relationships__child'),363 'interface_set__children_relationships__child__vlan',
364 ('interface_set__children_relationships__child__'
365 'children_relationships__child__vlan'),
366 ('interface_set__children_relationships__child__'
367 'children_relationships__child__'
368 'children_relationships__child__vlan'),
363 'tags',369 'tags',
364 ]370 ]
365 for prefetch in prefetch_related_fields:371 for prefetch in prefetch_related_fields:
366372
=== modified file 'src/maasserver/api/tests/test_interfaces.py'
--- src/maasserver/api/tests/test_interfaces.py 2017-01-28 00:51:47 +0000
+++ src/maasserver/api/tests/test_interfaces.py 2017-04-20 22:42:24 +0000
@@ -9,6 +9,7 @@
9import random9import random
1010
11from django.core.urlresolvers import reverse11from django.core.urlresolvers import reverse
12from maasserver import middleware
12from maasserver.enum import (13from maasserver.enum import (
13 INTERFACE_LINK_TYPE,14 INTERFACE_LINK_TYPE,
14 INTERFACE_TYPE,15 INTERFACE_TYPE,
@@ -24,6 +25,7 @@
24from maasserver.testing.factory import factory25from maasserver.testing.factory import factory
25from maasserver.utils.converters import json_load_bytes26from maasserver.utils.converters import json_load_bytes
26from maasserver.utils.orm import reload_object27from maasserver.utils.orm import reload_object
28from maastesting.djangotestcase import count_queries
27from testtools.matchers import (29from testtools.matchers import (
28 Contains,30 Contains,
29 ContainsDict,31 ContainsDict,
@@ -135,6 +137,42 @@
135 self.assertEqual(137 self.assertEqual(
136 interface.id, json_load_bytes(response.content)[0]['id'])138 interface.id, json_load_bytes(response.content)[0]['id'])
137139
140 def test_read_uses_constant_number_of_queries(self):
141 # Patch middleware so it does not affect query counting.
142 self.patch(
143 middleware.ExternalComponentsMiddleware,
144 '_check_rack_controller_connectivity')
145
146 node = factory.make_Node()
147 bond1, parents1, children1 = make_complex_interface(node)
148 uri = get_interfaces_uri(node)
149
150 num_queries1, response1 = count_queries(self.client.get, uri)
151
152 bond2, parents2, children2 = make_complex_interface(node)
153 num_queries2, response2 = count_queries(self.client.get, uri)
154
155 # Make sure the responses are ok as it's not useful to compare the
156 # number of queries if they are not.
157 parsed_result_1 = json_load_bytes(response1.content)
158 parsed_result_2 = json_load_bytes(response2.content)
159 self.assertEqual(
160 [
161 http.client.OK, http.client.OK,
162 len([bond1] + parents1 + children1),
163 len(
164 [bond1, bond2] +
165 parents1 + parents2 +
166 children1 + children2),
167 ],
168 [
169 response1.status_code,
170 response2.status_code,
171 len(parsed_result_1),
172 len(parsed_result_2),
173 ])
174 self.assertEqual(num_queries1, num_queries2)
175
138 def test_create_physical(self):176 def test_create_physical(self):
139 self.become_admin()177 self.become_admin()
140 for status in (NODE_STATUS.READY, NODE_STATUS.BROKEN):178 for status in (NODE_STATUS.READY, NODE_STATUS.BROKEN):