Merge lp:~mpontillo/maas/networking-constraints-refactor-part1 into lp:~maas-committers/maas/trunk

Proposed by Mike Pontillo
Status: Merged
Approved by: Mike Pontillo
Approved revision: no longer in the source branch.
Merged at revision: 4445
Proposed branch: lp:~mpontillo/maas/networking-constraints-refactor-part1
Merge into: lp:~maas-committers/maas/trunk
Prerequisite: lp:~mpontillo/maas/networking-constraint-filters
Diff against target: 1545 lines (+734/-303)
12 files modified
.idea/inspectionProfiles/Project_Default.xml (+16/-0)
.idea/inspectionProfiles/profiles_settings.xml (+7/-0)
src/maasserver/models/node.py (+32/-39)
src/maasserver/models/subnet.py (+197/-18)
src/maasserver/models/tests/test_node.py (+7/-7)
src/maasserver/models/tests/test_subnet.py (+158/-1)
src/maasserver/node_constraint_filter_forms.py (+94/-201)
src/maasserver/testing/factory.py (+8/-7)
src/maasserver/tests/test_node_constraint_filter_forms.py (+146/-30)
src/maasserver/utils/orm.py (+24/-0)
src/provisioningserver/utils/network.py (+22/-0)
src/provisioningserver/utils/tests/test_network.py (+23/-0)
To merge this branch: bzr merge lp:~mpontillo/maas/networking-constraints-refactor-part1
Reviewer Review Type Date Requested Status
Gavin Panella (community) Approve
Review via email: mp+276242@code.launchpad.net

Commit message

Networking constraints refactoring (part 1).
  * Add fabrics, not_fabrics, fabric_classes, and not_fabric_classes constraints.
    * Covers nodes only, not with the ability to relate back to interfaces yet.
    * Fabric name handling only works for user-specified names currently, and must be fixed in a follow-on commit.
  * Rename methods in NodeQueriesMixin to better match Django conventions.
  * Refactor Subnets to use a new SubnetQueriesMixin class.
  * Refactor common query mixin logic to a common superclass.
  * Rename get_subnets_with_ip() to raw_subnets_containing_ip() to make it more clear that it returns a raw query.
  * Add specifier-based Subnet queries (now used for networking constraint matching).
  * Remove legacy subnet specifier handling code
    * Disable legacy subnet specifier tests dealing with multiple constraint matching
       (this logic will be moved to the interfaces= constraint, since it is specific to individual interfaces)
  * Remove legacy 'networks' form fields
    * Add backward-compatibility handling to use 'subnets' for these fields
       (we may want to remove this completely, since it behaves slightly differently)
    * No validators have been implemented yet for fabric and fabric class constraints; again, this will happen in a follow-on commit.
  * Add 'space' specifier to subnet constraints.
  * Add 'cidr' specifier to subnet constraints.
  * Change default behavior of subnet constraints to first match on CIDR, then try matching on name if no match is found.
  * Remove Fabric matching from custom query, since subnet CIDRs are "globally" unique. (within MAAS)
  * Move generic hex/decimal integer parsing code into its own utility function.
  * Add PyCharm inspection profiles for Node and Subnet query mixins.

To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote :

I have reformatted the commit message; it looked like there was a missing line-break.

Revision history for this message
Gavin Panella (allenap) wrote :

I have made loads of comments but it's all basically good. It's also too long for me to really take it all in at once, so I skipped over some bits. In other words, I could have made even MORE comments :)

Needs Fixing because parse_integer() needs tests, but also because I might have a read through the bits I skimmed before it lands.

review: Needs Fixing
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Thanks for the review.

Revision history for this message
Gavin Panella (allenap) :
Revision history for this message
Gavin Panella (allenap) wrote :

Nice!

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory '.idea/inspectionProfiles'
2=== added file '.idea/inspectionProfiles/Project_Default.xml'
3--- .idea/inspectionProfiles/Project_Default.xml 1970-01-01 00:00:00 +0000
4+++ .idea/inspectionProfiles/Project_Default.xml 2015-10-30 19:51:38 +0000
5@@ -0,0 +1,16 @@
6+<component name="InspectionProjectProfileManager">
7+ <profile version="1.0">
8+ <option name="myName" value="Project Default" />
9+ <option name="myLocal" value="true" />
10+ <inspection_tool class="PyUnresolvedReferencesInspection" enabled="true" level="WARNING" enabled_by_default="true">
11+ <option name="ignoredIdentifiers">
12+ <list>
13+ <option value="maasserver.models.node.NodeQueriesMixin.filter" />
14+ <option value="maasserver.models.node.NodeQueriesMixin.exclude" />
15+ <option value="maasserver.models.subnet.SubnetQueriesMixin.filter" />
16+ <option value="maasserver.models.subnet.SubnetQueriesMixin.exclude" />
17+ </list>
18+ </option>
19+ </inspection_tool>
20+ </profile>
21+</component>
22\ No newline at end of file
23
24=== added file '.idea/inspectionProfiles/profiles_settings.xml'
25--- .idea/inspectionProfiles/profiles_settings.xml 1970-01-01 00:00:00 +0000
26+++ .idea/inspectionProfiles/profiles_settings.xml 2015-10-30 19:51:38 +0000
27@@ -0,0 +1,7 @@
28+<component name="InspectionProjectProfileManager">
29+ <settings>
30+ <option name="PROJECT_PROFILE" value="Project Default" />
31+ <option name="USE_PROJECT_PROFILE" value="true" />
32+ <version value="1.0" />
33+ </settings>
34+</component>
35\ No newline at end of file
36
37=== modified file 'src/maasserver/models/node.py'
38--- src/maasserver/models/node.py 2015-10-30 12:27:37 +0000
39+++ src/maasserver/models/node.py 2015-10-30 19:51:38 +0000
40@@ -116,6 +116,7 @@
41 from maasserver.utils.mac import get_vendor_for_mac
42 from maasserver.utils.orm import (
43 get_one,
44+ MAASQueriesMixin,
45 post_commit,
46 post_commit_do,
47 transactional,
48@@ -168,97 +169,89 @@
49 "power_parameters"))
50
51
52-class NodeQueriesMixin(object):
53+class NodeQueriesMixin(MAASQueriesMixin):
54
55 def filter_by_spaces(self, spaces):
56- """Return the set of nodes (optionally based on the specified
57- self) with at least one interface in the specified spaces.
58+ """Return the set of nodes with at least one interface in the specified
59+ spaces.
60 """
61 return self.filter(
62 interface__ip_addresses__subnet__space__in=spaces)
63
64- def filter_by_not_spaces(self, spaces):
65- """Return the set of nodes (optionally based on the specified
66- self) without any interfaces in the specified spaces.
67+ def exclude_spaces(self, spaces):
68+ """Return the set of nodes without any interfaces in the specified
69+ spaces.
70 """
71 return self.exclude(
72 interface__ip_addresses__subnet__space__in=spaces)
73
74 def filter_by_fabrics(self, fabrics):
75- """Return the set of nodes (optionally based on the specified
76- self) with at least one interface in the specified fabrics.
77+ """Return the set of nodes with at least one interface in the specified
78+ fabrics.
79 """
80 return self.filter(
81 interface__vlan__fabric__in=fabrics)
82
83- def filter_by_not_fabrics(self, fabrics):
84- """Return the set of nodes (optionally based on the specified
85- self) without any interfaces in the specified fabrics.
86+ def exclude_fabrics(self, fabrics):
87+ """Return the set of nodes without any interfaces in the specified
88+ fabrics.
89 """
90 return self.exclude(
91 interface__vlan__fabric__in=fabrics)
92
93 def filter_by_fabric_classes(self, fabric_classes):
94- """Return the set of nodes (optionally based on the specified
95- self) with at least one interface in the specified fabric
96- classes.
97+ """Return the set of nodes with at least one interface in the specified
98+ fabric classes.
99 """
100 return self.filter(
101 interface__vlan__fabric__class_type__in=fabric_classes)
102
103- def filter_by_not_fabric_classes(
104+ def exclude_fabric_classes(
105 self, fabric_classes):
106- """Return the set of nodes (optionally based on the specified
107- self) without any interfaces in the specified fabric
108- classes.
109+ """Return the set of nodes without any interfaces in the specified
110+ fabric classes.
111 """
112 return self.exclude(
113 interface__vlan__fabric__class_type__in=fabric_classes)
114
115 def filter_by_vids(self, vids):
116- """Return the set of nodes (optionally based on the specified
117- self) with at least one interface whose VLAN has one of the
118- specified VIDs.
119+ """Return the set of nodes with at least one interface whose VLAN has
120+ one of the specified VIDs.
121 """
122 return self.filter(
123 interface__vlan__vid__in=vids)
124
125- def filter_by_not_vids(self, vids):
126- """Return the set of nodes (optionally based on the specified
127- self) without any interfaces whose VLAN has one of the
128- specified VIDs.
129+ def exclude_vids(self, vids):
130+ """Return the set of nodes without any interfaces whose VLAN has one of
131+ the specified VIDs.
132 """
133 return self.exclude(
134 interface__vlan__vid__in=vids)
135
136 def filter_by_subnets(self, subnets):
137- """Return the set of nodes (optionally based on the specified
138- self) with at least one interface configured on one of the
139- specified subnets.
140+ """Return the set of nodes with at least one interface configured on
141+ one of the specified subnets.
142 """
143 return self.filter(
144 interface__ip_addresses__subnet__in=subnets)
145
146- def filter_by_not_subnets(self, subnets):
147- """Return the set of nodes (optionally based on the specified
148- self) without any interfaces configured on one of the
149- specified subnets.
150+ def exclude_subnets(self, subnets):
151+ """Return the set of nodes without any interfaces configured on one of
152+ the specified subnets.
153 """
154 return self.exclude(
155 interface__ip_addresses__subnet__in=subnets)
156
157 def filter_by_subnet_cidrs(self, subnet_cidrs):
158- """Return the set of nodes (optionally based on the specified
159- self) with at least one interface configured on one of the
160- specified subnet with the given CIDRs.
161+ """Return the set of nodes with at least one interface configured on
162+ one of the specified subnet with the given CIDRs.
163 """
164 return self.filter(
165 interface__ip_addresses__subnet__cidr__in=subnet_cidrs)
166
167- def filter_by_not_subnet_cidrs(self, subnet_cidrs):
168- """Return the set of nodes (optionally based on the specified
169- self) without any interfaces configured on one of the
170- specified subnet with the given CIDRs.
171+ def exclude_subnet_cidrs(self, subnet_cidrs):
172+ """Return the set of nodes without any interfaces configured on one of
173+ the specified subnet with the given CIDRs.
174 """
175 return self.exclude(
176 interface__ip_addresses__subnet__cidr__in=subnet_cidrs)
177
178=== modified file 'src/maasserver/models/subnet.py'
179--- src/maasserver/models/subnet.py 2015-09-24 16:22:12 +0000
180+++ src/maasserver/models/subnet.py 2015-10-30 19:51:38 +0000
181@@ -28,7 +28,9 @@
182 ForeignKey,
183 Manager,
184 PROTECT,
185+ Q,
186 )
187+from django.db.models.query import QuerySet
188 from django.shortcuts import get_object_or_404
189 from djorm_pgarray.fields import ArrayField
190 from maasserver import DefaultMeta
191@@ -42,14 +44,18 @@
192 )
193 from maasserver.models.cleansave import CleanSave
194 from maasserver.models.timestampedmodel import TimestampedModel
195+from maasserver.utils.orm import MAASQueriesMixin
196 from netaddr import (
197+ AddrFormatError,
198 IPAddress,
199 IPNetwork,
200 )
201+from provisioningserver.utils import flatten
202 from provisioningserver.utils.network import (
203 MAASIPSet,
204 make_ipaddress,
205 make_iprange,
206+ parse_integer,
207 )
208
209
210@@ -92,13 +98,65 @@
211 return unicode(cidr)
212
213
214-class SubnetManager(Manager):
215- """Manager for :class:`Subnet` model."""
216-
217- def create_from_cidr(self, cidr, vlan, space):
218- """Create a subnet from the given CIDR."""
219- name = "subnet-" + unicode(cidr)
220- return self.create(name=name, cidr=cidr, vlan=vlan, space=space)
221+def parse_item_operation(specifier):
222+ """
223+ Returns a tuple indicating the specifier string, and its related
224+ operation (if one was found).
225+
226+ If the first character in the specifier is '|', the operator will be OR.
227+
228+ If the first character in the specifier is '&', the operator will be AND.
229+
230+ If unspecified, the default operator is OR.
231+
232+ :param specifier: a string containing the specifier.
233+ :return: tuple
234+ """
235+ specifier = specifier.strip()
236+
237+ from operator import (
238+ and_ as AND,
239+ or_ as OR,
240+ )
241+
242+ if specifier.startswith('|'):
243+ op = OR
244+ specifier = specifier[1:]
245+ elif specifier.startswith('&'):
246+ op = AND
247+ specifier = specifier[1:]
248+ else:
249+ # Default to OR.
250+ op = OR
251+ return specifier, op
252+
253+
254+def parse_item_specifier_type(specifier, types=frozenset(), separator=':'):
255+ """
256+ Returns a tuple that splits the string int a specifier, and its specifier
257+ type.
258+
259+ Retruns a tuple of (specifier, specifier_type). If no specifier type could
260+ be found in the set, returns None in place of the specifier_type.
261+
262+ :param specifier: The specifier string, such as "ip:10.0.0.1".
263+ :param types: A set of strings that will be recognized as specifier types.
264+ :param separator: Optional specifier. Defaults to ':'.
265+ :return: tuple
266+ """
267+ if separator in specifier:
268+ tokens = specifier.split(separator, 1)
269+ if tokens[0] in types:
270+ specifier_type = tokens[0]
271+ specifier = tokens[1].strip()
272+ else:
273+ specifier_type = None
274+ else:
275+ specifier_type = None
276+ return specifier, specifier_type
277+
278+
279+class SubnetQueriesMixin(MAASQueriesMixin):
280
281 find_subnets_with_ip_query = """
282 SELECT DISTINCT subnet.*, masklen(subnet.cidr) "prefixlen"
283@@ -109,7 +167,7 @@
284 ORDER BY prefixlen DESC
285 """
286
287- def get_subnets_with_ip(self, ip):
288+ def raw_subnets_containing_ip(self, ip):
289 """Find the most specific Subnet the specified IP address belongs in.
290 """
291 return self.raw(
292@@ -135,8 +193,7 @@
293 LEFT OUTER JOIN maasserver_nodegroup AS nodegroup
294 ON ngi.nodegroup_id = nodegroup.id
295 WHERE
296- %s << subnet.cidr AND /* Specified IP is inside range */
297- vlan.fabric_id = %s
298+ %s << subnet.cidr /* Specified IP is inside range */
299 ORDER BY
300 /* For nodegroup_status, 1=ENABLED, 2=DISABLED, and NULL
301 means the outer join didn't find a related NodeGroup. */
302@@ -151,27 +208,149 @@
303 LIMIT 1
304 """
305
306- def get_best_subnet_for_ip(self, ip, fabric=None):
307+ def get_best_subnet_for_ip(self, ip):
308 """Find the most-specific managed Subnet the specified IP address
309 belongs to.
310
311 The most-specific Subnet is a Subnet that is both referred to by
312- a managed, active NodeGroupInterface, and on the specified Fabric.
313-
314- If no Fabric is specified, uses the default Fabric.
315+ a managed, active NodeGroupInterface.
316 """
317- # Circular imports
318- fabric = self._find_fabric(fabric)
319-
320 subnets = self.raw(
321 self.find_best_subnet_for_ip_query,
322- params=[unicode(ip), fabric])
323+ params=[unicode(ip)])
324
325 for subnet in subnets:
326 return subnet # This is stable because the query is ordered.
327 else:
328 return None
329
330+ SUBNET_SPECIFIER_TYPES = frozenset({
331+ 'ip',
332+ 'cidr',
333+ 'name',
334+ 'vlan',
335+ 'vid',
336+ 'space',
337+ })
338+
339+ def validate_filter_specifiers(self, specifiers):
340+ """Validate the given filter string."""
341+ try:
342+ self.filter_by_specifiers(specifiers)
343+ except (ValueError, AddrFormatError) as e:
344+ raise ValidationError(e.message)
345+
346+ def _get_specifiers_query(self, specifiers):
347+ """Returns a Q object for objects matching the given specifiers.
348+
349+ See documentation for `filter_by_specifiers()`.
350+
351+ :return:django.db.models.Q
352+ """
353+ current_q = Q()
354+ # Handle a single item, or a list.
355+ specifiers = list(flatten(specifiers))
356+ for item in specifiers:
357+ item, op = parse_item_operation(item)
358+ item, specifier_type = parse_item_specifier_type(
359+ item, types=self.SUBNET_SPECIFIER_TYPES)
360+
361+ if specifier_type == 'ip':
362+ # Try to validate this before it hits the database, since this
363+ # is going to be a raw query.
364+ item = unicode(IPAddress(item))
365+ # This is a special case. If a specific IP filter is included,
366+ # a custom query is needed to get the result. We can't chain
367+ # a raw query using Q without grabbing the IDs first.
368+ ids = self.get_id_list(self.raw_subnets_containing_ip(item))
369+ current_q = op(current_q, Q(id__in=ids))
370+ elif specifier_type == 'name':
371+ current_q = op(current_q, Q(name=item))
372+ elif specifier_type == 'vlan' or specifier_type == 'vid':
373+ if item.lower() == 'untagged':
374+ vid = 0
375+ else:
376+ vid = parse_integer(item)
377+ if vid < 0 or vid >= 0xfff:
378+ raise ValidationError(
379+ "VLAN tag (VID) out of range "
380+ "(0-4094; 0 for untagged.)")
381+ current_q = op(current_q, Q(vlan__vid=vid))
382+ elif specifier_type == 'cidr':
383+ ip = IPNetwork(item)
384+ cidr = unicode(ip.cidr)
385+ current_q = op(current_q, Q(cidr=cidr))
386+ elif specifier_type == 'space':
387+ current_q = op(current_q, Q(space__name=item))
388+ else:
389+ # By default, search for a specific CIDR first, then
390+ # fall back to the name.
391+ try:
392+ ip = IPNetwork(item)
393+ except AddrFormatError:
394+ current_q = op(current_q, Q(name=item))
395+ else:
396+ cidr = unicode(ip.cidr)
397+ current_q = op(current_q, Q(cidr=cidr))
398+ return current_q
399+
400+ def filter_by_specifiers(self, specifiers):
401+ """Filters subnets by the given list of specifiers (or single
402+ specifier).
403+
404+ Allows a number of types to be prefixed in front of each specifier:
405+ * 'ip:' Matches the subnet that best matches the given IP address.
406+ * 'cidr:' Matches a subnet with the exact given CIDR.
407+ * 'name': Matches a subnet with the given name.
408+ * 'vid:' Matches a subnet whose VLAN has the given VID.
409+ Can be used with a hexadecimal or binary string by prefixing
410+ it with '0x' or '0b'.
411+ ' 'vlan:' Synonym for 'vid' for compatibility with older MAAS
412+ versions.
413+
414+ If no specifier is given, the input will be treated as a CIDR. If
415+ the input is not a valid CIDR, it will be treated as subnet name.
416+
417+ :raise:AddrFormatError:If a specific IP address or CIDR is requested,
418+ but the address could not be parsed.
419+ :return:QuerySet
420+ """
421+ query = self._get_specifiers_query(specifiers)
422+ return self.filter(query)
423+
424+ def exclude_by_specifiers(self, specifiers):
425+ """Excludes subnets by the given list of specifiers (or single
426+ specifier).
427+
428+ See documentation for `filter_by_specifiers()`.
429+
430+ :raise:AddrFormatError:If a specific IP address or CIDR is requested,
431+ but the address could not be parsed.
432+ :return:QuerySet
433+ """
434+ query = self._get_specifiers_query(specifiers)
435+ return self.exclude(query)
436+
437+
438+class SubnetQuerySet(QuerySet, SubnetQueriesMixin):
439+ """Custom QuerySet which mixes in some additional queries specific to
440+ subnets. This needs to be a mixin because an identical method is needed on
441+ both the Manager and all QuerySets which result from calling the manager.
442+ """
443+
444+
445+class SubnetManager(Manager, SubnetQueriesMixin):
446+ """Manager for :class:`Subnet` model."""
447+
448+ def get_queryset(self):
449+ queryset = SubnetQuerySet(self.model, using=self._db)
450+ return queryset
451+
452+ def create_from_cidr(self, cidr, vlan, space):
453+ """Create a subnet from the given CIDR."""
454+ name = "subnet-" + unicode(cidr)
455+ return self.create(name=name, cidr=cidr, vlan=vlan, space=space)
456+
457 def _find_fabric(self, fabric):
458 from maasserver.models import Fabric
459
460
461=== modified file 'src/maasserver/models/tests/test_node.py'
462--- src/maasserver/models/tests/test_node.py 2015-10-30 12:27:37 +0000
463+++ src/maasserver/models/tests/test_node.py 2015-10-30 19:51:38 +0000
464@@ -2703,7 +2703,7 @@
465 ip = iface.ip_addresses.first()
466 space = ip.subnet.space
467 self.assertItemsEqual(
468- [extra_node], Node.objects.filter_by_not_spaces([space]))
469+ [extra_node], Node.objects.exclude_spaces([space]))
470
471 def test_filter_nodes_by_fabrics(self):
472 fabric = factory.make_Fabric()
473@@ -2722,7 +2722,7 @@
474 iface = node.get_boot_interface()
475 fabric = iface.vlan.fabric
476 self.assertItemsEqual(
477- [extra_node], Node.objects.filter_by_not_fabrics([fabric]))
478+ [extra_node], Node.objects.exclude_fabrics([fabric]))
479
480 def test_filter_nodes_by_fabric_classes(self):
481 fabric1 = factory.make_Fabric(class_type="10g")
482@@ -2738,7 +2738,7 @@
483 factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
484 node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
485 self.assertItemsEqual(
486- [node2], Node.objects.filter_by_not_fabric_classes(["10g"]))
487+ [node2], Node.objects.exclude_fabric_classes(["10g"]))
488
489 def test_filter_nodes_by_vids(self):
490 vlan1 = factory.make_VLAN(vid=1)
491@@ -2754,7 +2754,7 @@
492 factory.make_Node_with_Interface_on_Subnet(vlan=vlan1)
493 node2 = factory.make_Node_with_Interface_on_Subnet(vlan=vlan2)
494 self.assertItemsEqual(
495- [node2], Node.objects.filter_by_not_vids([1]))
496+ [node2], Node.objects.exclude_vids([1]))
497
498 def test_filter_nodes_by_subnet(self):
499 subnet1 = factory.make_Subnet()
500@@ -2770,7 +2770,7 @@
501 factory.make_Node_with_Interface_on_Subnet(subnet=subnet1)
502 node2 = factory.make_Node_with_Interface_on_Subnet(subnet=subnet2)
503 self.assertItemsEqual(
504- [node2], Node.objects.filter_by_not_subnets([subnet1]))
505+ [node2], Node.objects.exclude_subnets([subnet1]))
506
507 def test_filter_nodes_by_subnet_cidr(self):
508 subnet1 = factory.make_Subnet(cidr='192.168.1.0/24')
509@@ -2786,7 +2786,7 @@
510 factory.make_Node_with_Interface_on_Subnet(subnet=subnet1)
511 node2 = factory.make_Node_with_Interface_on_Subnet(subnet=subnet2)
512 self.assertItemsEqual(
513- [node2], Node.objects.filter_by_not_subnet_cidrs(
514+ [node2], Node.objects.exclude_subnet_cidrs(
515 ['192.168.1.0/24']))
516
517 def test_filter_fabric_subnet_filter_chain(self):
518@@ -2800,7 +2800,7 @@
519 self.assertItemsEqual(
520 [node2], Node.objects
521 .filter_by_fabrics([fabric1])
522- .filter_by_not_subnet_cidrs(['192.168.1.0/24']))
523+ .exclude_subnet_cidrs(['192.168.1.0/24']))
524
525 def test_get_visible_node_or_404_ok(self):
526 """get_node_or_404 fetches nodes by system_id."""
527
528=== modified file 'src/maasserver/models/tests/test_subnet.py'
529--- src/maasserver/models/tests/test_subnet.py 2015-09-24 16:22:12 +0000
530+++ src/maasserver/models/tests/test_subnet.py 2015-10-30 19:51:38 +0000
531@@ -98,6 +98,163 @@
532 self.assertEqual("169.254.0.0/16", cidr)
533
534
535+class TestSubnetManager(MAASServerTestCase):
536+
537+ def test__filter_by_specifiers_takes_single_item(self):
538+ subnet1 = factory.make_Subnet(name="subnet1")
539+ factory.make_Subnet(name="subnet2")
540+ self.assertItemsEqual(
541+ Subnet.objects.filter_by_specifiers("subnet1"),
542+ [subnet1])
543+
544+ def test__filter_by_specifiers_takes_multiple_items(self):
545+ subnet1 = factory.make_Subnet(name="subnet1")
546+ subnet2 = factory.make_Subnet(name="subnet2")
547+ self.assertItemsEqual(
548+ Subnet.objects.filter_by_specifiers(["subnet1", "subnet2"]),
549+ [subnet1, subnet2])
550+
551+ def test__filter_by_specifiers_takes_multiple_cidr_or_name(self):
552+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
553+ subnet2 = factory.make_Subnet(name="subnet2")
554+ self.assertItemsEqual(
555+ Subnet.objects.filter_by_specifiers(["8.8.8.8/24", "subnet2"]),
556+ [subnet1, subnet2])
557+
558+ def test__filter_by_specifiers_empty_filter_matches_all(self):
559+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
560+ subnet2 = factory.make_Subnet(name="subnet2")
561+ self.assertItemsEqual(
562+ Subnet.objects.filter_by_specifiers([]),
563+ [subnet1, subnet2])
564+
565+ def test__filter_by_specifiers_matches_name_if_requested(self):
566+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
567+ subnet2 = factory.make_Subnet(name="subnet2")
568+ factory.make_Subnet(name="subnet3")
569+ self.assertItemsEqual(
570+ Subnet.objects.filter_by_specifiers(
571+ ["name:subnet1", "name:subnet2"]),
572+ [subnet1, subnet2])
573+
574+ def test__filter_by_specifiers_matches_space_name_if_requested(self):
575+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
576+ subnet2 = factory.make_Subnet(name="subnet2")
577+ factory.make_Subnet(name="subnet3")
578+ self.assertItemsEqual(
579+ Subnet.objects.filter_by_specifiers(
580+ ["space:%s" % subnet1.space.name,
581+ "space:%s" % subnet2.space.name]),
582+ [subnet1, subnet2])
583+
584+ def test__filter_by_specifiers_matches_vid_if_requested(self):
585+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24", vid=1)
586+ subnet2 = factory.make_Subnet(name="subnet2", vid=2)
587+ subnet3 = factory.make_Subnet(name="subnet3", vid=3)
588+ factory.make_Subnet(name="subnet4", vid=4)
589+ self.assertItemsEqual(
590+ Subnet.objects.filter_by_specifiers(
591+ ["vlan:0b1", "vlan:0x2", "vlan:3"]),
592+ [subnet1, subnet2, subnet3])
593+
594+ def test__filter_by_specifiers_matches_untagged_vlan_if_requested(self):
595+ fabric = factory.make_Fabric()
596+ vlan = fabric.get_default_vlan()
597+ subnet1 = factory.make_Subnet(
598+ name="subnet1", cidr="8.8.8.0/24", vlan=vlan)
599+ subnet2 = factory.make_Subnet(name="subnet2", vid=2)
600+ subnet3 = factory.make_Subnet(name="subnet3", vid=3)
601+ factory.make_Subnet(name="subnet4", vid=4)
602+ self.assertItemsEqual(
603+ Subnet.objects.filter_by_specifiers(
604+ ["vid:UNTAGGED", "vid:0x2", "vid:3"]),
605+ [subnet1, subnet2, subnet3])
606+
607+ def test__filter_by_specifiers_raises_for_invalid_vid(self):
608+ fabric = factory.make_Fabric()
609+ vlan = fabric.get_default_vlan()
610+ factory.make_Subnet(
611+ name="subnet1", cidr="8.8.8.0/24", vlan=vlan)
612+ factory.make_Subnet(name="subnet2", vid=2)
613+ factory.make_Subnet(name="subnet3", vid=3)
614+ factory.make_Subnet(name="subnet4", vid=4)
615+ with ExpectedException(ValidationError):
616+ Subnet.objects.filter_by_specifiers(["vid:4095"])
617+
618+ def test__filter_by_specifiers_works_with_chained_filter(self):
619+ factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
620+ subnet2 = factory.make_Subnet(name="subnet2")
621+ self.assertItemsEqual(
622+ Subnet.objects
623+ .exclude(name="subnet1")
624+ .filter_by_specifiers(["8.8.8.8/24", "subnet2"]),
625+ [subnet2])
626+
627+ def test__filter_by_specifiers_ip_filter_matches_specific_ip(self):
628+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
629+ subnet2 = factory.make_Subnet(name="subnet2", cidr="7.7.7.0/24")
630+ self.assertItemsEqual(
631+ Subnet.objects.filter_by_specifiers("ip:8.8.8.8"),
632+ [subnet1])
633+ self.assertItemsEqual(
634+ Subnet.objects.filter_by_specifiers("ip:7.7.7.7"),
635+ [subnet2])
636+ self.assertItemsEqual(
637+ Subnet.objects.filter_by_specifiers("ip:1.1.1.1"),
638+ [])
639+
640+ def test__filter_by_specifiers_ip_filter_raises_for_invalid_ip(self):
641+ factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
642+ factory.make_Subnet(name="subnet2", cidr="2001:db8::/64")
643+ with ExpectedException(AddrFormatError):
644+ Subnet.objects.filter_by_specifiers("ip:x8.8.8.0"),
645+ with ExpectedException(AddrFormatError):
646+ Subnet.objects.filter_by_specifiers("ip:x2001:db8::"),
647+
648+ def test__filter_by_specifiers_ip_filter_matches_specific_cidr(self):
649+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
650+ subnet2 = factory.make_Subnet(name="subnet2", cidr="2001:db8::/64")
651+ self.assertItemsEqual(
652+ Subnet.objects.filter_by_specifiers("cidr:8.8.8.0/24"),
653+ [subnet1])
654+ self.assertItemsEqual(
655+ Subnet.objects.filter_by_specifiers("cidr:2001:db8::/64"),
656+ [subnet2])
657+
658+ def test__filter_by_specifiers_ip_filter_raises_for_invalid_cidr(self):
659+ factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
660+ factory.make_Subnet(name="subnet2", cidr="2001:db8::/64")
661+ with ExpectedException(ValueError):
662+ # netaddr.IPNetwork should probably raise AddrFormatError here,
663+ # but it actually raises a ValueError when it tries to parse "x8".
664+ Subnet.objects.filter_by_specifiers("cidr:x8.8.8.0/24"),
665+ with ExpectedException(AddrFormatError):
666+ Subnet.objects.filter_by_specifiers("cidr:x2001:db8::/64"),
667+
668+ def test__filter_by_specifiers_ip_chained_filter_matches_specific_ip(self):
669+ subnet1 = factory.make_Subnet(name="subnet1", cidr="8.8.8.0/24")
670+ factory.make_Subnet(name="subnet2", cidr="7.7.7.0/24")
671+ subnet3 = factory.make_Subnet(name="subnet3", cidr="6.6.6.0/24")
672+ self.assertItemsEqual(
673+ Subnet.objects.filter_by_specifiers(
674+ ["ip:8.8.8.8", "name:subnet3"]), [subnet1, subnet3])
675+
676+ def test__filter_by_specifiers_ip_filter_matches_specific_ipv6(self):
677+ subnet1 = factory.make_Subnet(
678+ name="subnet1", cidr="2001:db8::/64")
679+ subnet2 = factory.make_Subnet(
680+ name="subnet2", cidr="2001:db8:1::/64")
681+ self.assertItemsEqual(
682+ Subnet.objects.filter_by_specifiers("ip:2001:db8::5"),
683+ [subnet1])
684+ self.assertItemsEqual(
685+ Subnet.objects.filter_by_specifiers("ip:2001:db8:1::5"),
686+ [subnet2])
687+ self.assertItemsEqual(
688+ Subnet.objects.filter_by_specifiers("ip:1.1.1.1"),
689+ [])
690+
691+
692 class TestSubnetManagerGetSubnetOr404(MAASServerTestCase):
693
694 def test__user_view_returns_subnet(self):
695@@ -152,7 +309,7 @@
696 class SubnetTest(MAASServerTestCase):
697
698 def assertIPBestMatchesSubnet(self, ip, expected):
699- subnets = Subnet.objects.get_subnets_with_ip(IPAddress(ip))
700+ subnets = Subnet.objects.raw_subnets_containing_ip(IPAddress(ip))
701 for tmp in subnets:
702 subnet = tmp
703 break
704
705=== modified file 'src/maasserver/node_constraint_filter_forms.py'
706--- src/maasserver/node_constraint_filter_forms.py 2015-10-28 21:43:50 +0000
707+++ src/maasserver/node_constraint_filter_forms.py 2015-10-30 19:51:38 +0000
708@@ -15,10 +15,6 @@
709 ]
710
711
712-from abc import (
713- ABCMeta,
714- abstractmethod,
715-)
716 from collections import defaultdict
717 import itertools
718 from itertools import chain
719@@ -39,17 +35,13 @@
720 PhysicalBlockDevice,
721 Subnet,
722 Tag,
723- VLAN,
724 Zone,
725 )
726-from maasserver.models.subnet import SUBNET_NAME_VALIDATOR
727 from maasserver.models.zone import ZONE_NAME_VALIDATOR
728 from maasserver.utils.orm import (
729 macs_contain,
730 macs_do_not_contain,
731 )
732-from netaddr import IPAddress
733-from netaddr.core import AddrFormatError
734 from provisioningserver.utils.constraints import LabeledConstraintMap
735
736 # Matches the storage constraint from Juju. Format is an optional label,
737@@ -100,7 +92,7 @@
738 if ','.join(rendered_groups) != value:
739 raise ValidationError('Malformed storage constraint, "%s".' % value)
740
741-NETWORKING_CONSTRAINT_KEYS = frozenset({
742+NETWORKING_CONSTRAINT_NAMES = {
743 'space',
744 'not_space',
745 'fabric_class',
746@@ -114,21 +106,21 @@
747 'subnet',
748 'not_subnet',
749 'mode',
750-})
751-
752-
753-def networking_validator(constraint_map):
754+}
755+
756+
757+def interfaces_validator(constraint_map):
758 """Validate the given LabeledConstraintMap object."""
759 # At this point, the basic syntax of a labeled constraint map will have
760 # already been validated by the underlying form field. However, we also
761 # need to validate the specified things we're looking for in the
762- # networking domain.
763+ # interfaces domain.
764 for label in constraint_map:
765 constraints = constraint_map[label]
766 for constraint_name in constraints.iterkeys():
767- if constraint_name not in NETWORKING_CONSTRAINT_KEYS:
768+ if constraint_name not in NETWORKING_CONSTRAINT_NAMES:
769 raise ValidationError(
770- "Unknown networking constraint: '%s" % constraint_name)
771+ "Unknown interfaces constraint: '%s" % constraint_name)
772
773
774 def generate_architecture_wildcards(arches):
775@@ -385,169 +377,6 @@
776 return nodes
777
778
779-def strip_type_tag(type_tag, specifier):
780- """Return a network specifier minus its type tag."""
781- prefix = type_tag + ':'
782- assert specifier.startswith(prefix)
783- return specifier[len(prefix):]
784-
785-
786-class SubnetSpecifier:
787- """A :class:`SubnetSpecifier` identifies a :class:`Subnet`.
788-
789- For example, in placement constraints, a user may specify that a node
790- must be attached to a certain subnet. They identify the subnet through
791- a subnet specifier, which may be its name (`dmz`), an IP address
792- (`ip:10.12.0.0`), or a VLAN tag (`vlan:15` or `vlan:0xf`).
793-
794- Each type of subnet specifier has its own `SubnetSpecifier`
795- implementation class. The class constructor validates and parses a
796- subnet specifier of its type, and the object knows how to retrieve
797- whatever subnet it identifies from the database.
798- """
799- __metaclass__ = ABCMeta
800-
801- # Most subnet specifiers start with a type tag followed by a colon, e.g.
802- # "ip:10.1.0.0".
803- type_tag = None
804-
805- @abstractmethod
806- def find_subnet(self):
807- """Load the identified :class:`Subnet` from the database.
808-
809- :raise Subnet.DoesNotExist: If no subnet matched the specifier.
810- :return: The :class:`Subnet`.
811- """
812-
813-
814-class NameSpecifier(SubnetSpecifier):
815- """Identify a subnet by its name.
816-
817- This type of subnet specifier has no type tag; it's just the name. A
818- subnet name cannot contain colon (:) characters.
819- """
820-
821- def __init__(self, spec):
822- SUBNET_NAME_VALIDATOR(spec)
823- self.name = spec
824-
825- def find_subnet(self):
826- return Subnet.objects.get(name=self.name)
827-
828-
829-class IPSpecifier(SubnetSpecifier):
830- """Identify a subnet by any IP address it contains.
831-
832- The IP address is prefixed with a type tag `ip:`, e.g. `ip:10.1.1.0`.
833- It can name any IP address within the subnet, including its base address,
834- its broadcast address, or any host address that falls in its IP range.
835- """
836- type_tag = 'ip'
837-
838- def __init__(self, spec):
839- ip_string = strip_type_tag(self.type_tag, spec)
840- try:
841- self.ip = IPAddress(ip_string)
842- except AddrFormatError as e:
843- raise ValidationError("Invalid IP address: %s." % e)
844-
845- def find_subnet(self):
846- subnets = list(Subnet.objects.get_subnets_with_ip(self.ip))
847- if len(subnets) > 0:
848- return subnets[0]
849- raise Subnet.DoesNotExist()
850-
851-
852-class VLANSpecifier(SubnetSpecifier):
853- """Identify a subnet by its (nonzero) VLAN tag.
854-
855- This only applies to VLANs. The VLAN tag is a numeric value prefixed with
856- a type tag of `vlan:`, e.g. `vlan:12`. Tags may also be given in
857- hexadecimal form: `vlan:0x1a`. This is case-insensitive.
858- """
859- type_tag = 'vlan'
860-
861- def __init__(self, spec):
862- vlan_string = strip_type_tag(self.type_tag, spec)
863- if vlan_string.lower().startswith('0x'):
864- # Hexadecimal.
865- base = 16
866- else:
867- # Decimal.
868- base = 10
869- try:
870- self.vlan_tag = int(vlan_string, base)
871- except ValueError:
872- raise ValidationError("Invalid VLAN tag: '%s'." % vlan_string)
873- if self.vlan_tag <= 0 or self.vlan_tag >= 0xfff:
874- raise ValidationError("VLAN tag out of range (1-4094).")
875-
876- def find_subnet(self):
877- # The best we can do since we now support a more complex model is
878- # get the first VLAN with the VID and the first subnet in that VLAN.
879- vlans = VLAN.objects.filter(vid=self.vlan_tag)
880- if len(vlans) > 0:
881- subnet = vlans[0].subnet_set.first()
882- if subnet is not None:
883- return subnet
884- raise Subnet.DoesNotExist()
885-
886-
887-SPECIFIER_CLASSES = [NameSpecifier, IPSpecifier, VLANSpecifier]
888-
889-SPECIFIER_TAGS = {
890- spec_class.type_tag: spec_class
891- for spec_class in SPECIFIER_CLASSES
892-}
893-
894-
895-def get_specifier_type(specifier):
896- """Obtain the specifier class that knows how to parse `specifier`.
897-
898- :raise ValidationError: If `specifier` does not match any accepted type of
899- network specifier.
900- :return: A concrete `NetworkSpecifier` subclass that knows how to parse
901- `specifier`.
902- """
903- if ':' in specifier:
904- type_tag, _ = specifier.split(':', 1)
905- else:
906- type_tag = None
907- specifier_class = SPECIFIER_TAGS.get(type_tag)
908- if specifier_class is None:
909- raise ValidationError(
910- "Invalid network specifier type: '%s'." % type_tag)
911- return specifier_class
912-
913-
914-def parse_subnet_spec(spec):
915- """Parse a network specifier; return it as a `NetworkSpecifier` object.
916-
917- :raise ValidationError: If `spec` is malformed.
918- """
919- specifier_class = get_specifier_type(spec)
920- return specifier_class(spec)
921-
922-
923-def get_subnet_from_spec(spec):
924- """Find a single `Subnet` from a given network specifier.
925-
926- Note: This exists for a backward compatability layer for how MAAS used to
927- do networking. It might not always be correct since the model is now more
928- complex, but it will atleast still work.
929-
930- :raise ValidationError: If `spec` is malformed.
931- :raise Subnet.DoesNotExist: If the subnet specifier does not match
932- any known subnet.
933- :return: The one `Subnet` matching `spec`.
934- """
935- specifier = parse_subnet_spec(spec)
936- try:
937- return specifier.find_subnet()
938- except Subnet.DoesNotExist:
939- raise Subnet.DoesNotExist("No subnet matching '%s'." % spec)
940-
941-
942 class LabeledConstraintMapField(Field):
943
944 def __init__(self, *args, **kwargs):
945@@ -584,16 +413,49 @@
946 not_tags = UnconstrainedMultipleChoiceField(
947 label="Not having tags", required=False)
948
949+ # XXX mpontillo 2015-10-30 need validators for fabric constraints
950+ fabrics = ValidatorMultipleChoiceField(
951+ validator=lambda x: True, label="Attached to fabrics",
952+ required=False, error_messages={
953+ 'invalid_list': "Invalid parameter: list of fabrics required.",
954+ })
955+
956+ not_fabrics = ValidatorMultipleChoiceField(
957+ validator=lambda x: True, label="Not attached to fabrics",
958+ required=False, error_messages={
959+ 'invalid_list': "Invalid parameter: list of fabrics required.",
960+ })
961+
962+ fabric_classes = ValidatorMultipleChoiceField(
963+ validator=lambda x: True,
964+ label="Attached to fabric with specified classes",
965+ required=False, error_messages={
966+ 'invalid_list':
967+ "Invalid parameter: list of fabric classes required.",
968+ })
969+
970+ not_fabric_classes = ValidatorMultipleChoiceField(
971+ validator=lambda x: True,
972+ label="Not attached to fabric with specified classes",
973+ required=False, error_messages={
974+ 'invalid_list':
975+ "Invalid parameter: list of fabric classes required."
976+ })
977+
978 networks = ValidatorMultipleChoiceField(
979- validator=parse_subnet_spec, label="Attached to networks",
980+ validator=Subnet.objects.validate_filter_specifiers,
981+ label="Attached to subnets",
982 required=False, error_messages={
983- 'invalid_list': "Invalid parameter: list of networks required.",
984+ 'invalid_list':
985+ "Invalid parameter: list of subnet specifiers required.",
986 })
987
988 not_networks = ValidatorMultipleChoiceField(
989- validator=parse_subnet_spec, label="Not attached to networks",
990+ validator=Subnet.objects.validate_filter_specifiers,
991+ label="Not attached to subnets",
992 required=False, error_messages={
993- 'invalid_list': "Invalid parameter: list of networks required.",
994+ 'invalid_list':
995+ "Invalid parameter: list of subnet specifiers required.",
996 })
997
998 connected_to = ValidatorMultipleChoiceField(
999@@ -619,8 +481,8 @@
1000 storage = forms.CharField(
1001 validators=[storage_validator], label="Storage", required=False)
1002
1003- networking = LabeledConstraintMapField(
1004- validators=[networking_validator], label="Networking", required=False)
1005+ interfaces = LabeledConstraintMapField(
1006+ validators=[interfaces_validator], label="Interfaces", required=False)
1007
1008 ignore_unknown_constraints = True
1009
1010@@ -696,23 +558,24 @@
1011 {self.get_field_name('not_in_zone'): [error_msg]})
1012 return value
1013
1014+ def _clean_subnet_specifiers(self, specifiers):
1015+ if not specifiers:
1016+ return None
1017+ subnets = set(Subnet.objects.filter_by_specifiers(specifiers))
1018+ if len(subnets) == 0:
1019+ raise ValidationError("No matching subnets found.")
1020+ return subnets
1021+
1022 def clean_networks(self):
1023 value = self.cleaned_data[self.get_field_name('networks')]
1024- if value is None:
1025- return None
1026- try:
1027- return [get_subnet_from_spec(spec) for spec in value]
1028- except Subnet.DoesNotExist as e:
1029- raise ValidationError(e.message)
1030+ return self._clean_subnet_specifiers(value)
1031
1032 def clean_not_networks(self):
1033 value = self.cleaned_data[self.get_field_name('not_networks')]
1034- if value is None:
1035- return None
1036- try:
1037- return [get_subnet_from_spec(spec) for spec in value]
1038- except Subnet.DoesNotExist as e:
1039- raise ValidationError(e.message)
1040+ return self._clean_subnet_specifiers(value)
1041+
1042+ def __init__(self, *args, **kwargs):
1043+ super(AcquireNodeForm, self).__init__(*args, **kwargs)
1044
1045 def clean(self):
1046 if not self.ignore_unknown_constraints:
1047@@ -829,21 +692,51 @@
1048 not_in_zones = Zone.objects.filter(name__in=not_in_zone)
1049 filtered_nodes = filtered_nodes.exclude(zone__in=not_in_zones)
1050
1051- # Filter by networks.
1052+ # Filter by subnet.
1053 subnets = self.cleaned_data.get(self.get_field_name('networks'))
1054- if subnets is not None:
1055+ if subnets is not None and len(subnets) > 0:
1056 for subnet in set(subnets):
1057 filtered_nodes = filtered_nodes.filter(
1058 interface__ip_addresses__subnet=subnet)
1059
1060- # Filter by not_networks.
1061+ # Filter by not_subnets.
1062 not_subnets = self.cleaned_data.get(
1063 self.get_field_name('not_networks'))
1064- if not_subnets is not None:
1065+ if not_subnets is not None and len(not_subnets) > 0:
1066 for not_subnet in set(not_subnets):
1067 filtered_nodes = filtered_nodes.exclude(
1068 interface__ip_addresses__subnet=not_subnet)
1069
1070+ # Filter by fabrics.
1071+ fabrics = self.cleaned_data.get(self.get_field_name('fabrics'))
1072+ if fabrics is not None and len(fabrics) > 0:
1073+ # XXX mpontillo 2015-10-30 need to also handle fabrics whose name
1074+ # is null (fabric-<id>).
1075+ filtered_nodes = filtered_nodes.filter(
1076+ interface__vlan__fabric__name__in=fabrics)
1077+
1078+ # Filter by not_fabrics.
1079+ not_fabrics = self.cleaned_data.get(self.get_field_name('not_fabrics'))
1080+ if not_fabrics is not None and len(not_fabrics) > 0:
1081+ # XXX mpontillo 2015-10-30 need to also handle fabrics whose name
1082+ # is null (fabric-<id>).
1083+ filtered_nodes = filtered_nodes.exclude(
1084+ interface__vlan__fabric__name__in=not_fabrics)
1085+
1086+ # Filter by fabric classes.
1087+ fabric_classes = self.cleaned_data.get(self.get_field_name(
1088+ 'fabric_classes'))
1089+ if fabric_classes is not None and len(fabric_classes) > 0:
1090+ filtered_nodes = filtered_nodes.filter(
1091+ interface__vlan__fabric__class_type__in=fabric_classes)
1092+
1093+ # Filter by not_fabric_classes.
1094+ not_fabric_classes = self.cleaned_data.get(self.get_field_name(
1095+ 'not_fabric_classes'))
1096+ if not_fabric_classes is not None and len(not_fabric_classes) > 0:
1097+ filtered_nodes = filtered_nodes.exclude(
1098+ interface__vlan__fabric__class_type__in=not_fabric_classes)
1099+
1100 # Filter by connected_to.
1101 connected_to = self.cleaned_data.get(
1102 self.get_field_name('connected_to'))
1103
1104=== modified file 'src/maasserver/testing/factory.py'
1105--- src/maasserver/testing/factory.py 2015-10-30 12:27:37 +0000
1106+++ src/maasserver/testing/factory.py 2015-10-30 19:51:38 +0000
1107@@ -277,7 +277,7 @@
1108 routers=None, zone=None, networks=None, boot_type=None,
1109 sortable_name=False, power_type=None, power_parameters=None,
1110 power_state=None, power_state_updated=undefined, disable_ipv4=None,
1111- with_boot_disk=True, vlan=None, **kwargs):
1112+ with_boot_disk=True, vlan=None, fabric=None, **kwargs):
1113 """Make a :class:`Node`.
1114
1115 :param sortable_name: If `True`, use a that will sort consistently
1116@@ -327,7 +327,8 @@
1117 if networks is not None:
1118 node.networks.add(*networks)
1119 if interface:
1120- self.make_Interface(INTERFACE_TYPE.PHYSICAL, node=node, vlan=vlan)
1121+ self.make_Interface(
1122+ INTERFACE_TYPE.PHYSICAL, node=node, vlan=vlan, fabric=fabric)
1123 if installable and with_boot_disk:
1124 self.make_PhysicalBlockDevice(node=node)
1125
1126@@ -578,7 +579,7 @@
1127 iftype = kwargs['iftype']
1128 del kwargs['iftype']
1129 node = self.make_Node(
1130- nodegroup=nodegroup, **kwargs)
1131+ nodegroup=nodegroup, fabric=fabric, **kwargs)
1132 if vlan is None:
1133 vlan = self.make_VLAN(fabric=fabric)
1134 if subnet is None:
1135@@ -694,11 +695,11 @@
1136
1137 def make_Subnet(self, name=None, vlan=None, space=None, cidr=None,
1138 gateway_ip=None, dns_servers=None, host_bits=None,
1139- fabric=None):
1140+ fabric=None, vid=None):
1141 if name is None:
1142 name = factory.make_name('name')
1143 if vlan is None:
1144- vlan = factory.make_VLAN(fabric=fabric)
1145+ vlan = factory.make_VLAN(fabric=fabric, vid=vid)
1146 if space is None:
1147 space = factory.make_Space()
1148 if cidr is None:
1149@@ -759,13 +760,13 @@
1150 def make_Interface(
1151 self, iftype=INTERFACE_TYPE.PHYSICAL, node=None, mac_address=None,
1152 vlan=None, parents=None, name=None, cluster_interface=None,
1153- ip=None, enabled=True):
1154+ ip=None, enabled=True, fabric=None):
1155 if name is None and iftype != INTERFACE_TYPE.VLAN:
1156 name = self.make_name('name')
1157 if iftype is None:
1158 iftype = INTERFACE_TYPE.PHYSICAL
1159 if vlan is None:
1160- vlan = self.make_VLAN()
1161+ vlan = self.make_VLAN(fabric=fabric)
1162 if (mac_address is None and
1163 iftype in [
1164 INTERFACE_TYPE.PHYSICAL,
1165
1166=== modified file 'src/maasserver/tests/test_node_constraint_filter_forms.py'
1167--- src/maasserver/tests/test_node_constraint_filter_forms.py 2015-10-28 17:09:26 +0000
1168+++ src/maasserver/tests/test_node_constraint_filter_forms.py 2015-10-30 19:51:38 +0000
1169@@ -233,7 +233,7 @@
1170
1171 def assertConstrainedNodes(self, nodes, data):
1172 form = AcquireNodeForm(data=data)
1173- self.assertTrue(form.is_valid(), form.errors)
1174+ self.assertTrue(form.is_valid(), dict(form.errors))
1175 filtered_nodes, _ = form.filter_nodes(Node.objects.all())
1176 self.assertItemsEqual(nodes, filtered_nodes)
1177
1178@@ -299,6 +299,22 @@
1179 (False, {'mem': ["Invalid memory: number of MiB required."]}),
1180 (form.is_valid(), form.errors))
1181
1182+ def test_legacy_networks_field_falls_back_to_subnets_query(self):
1183+ subnets = [
1184+ factory.make_Subnet()
1185+ for _ in range(3)
1186+ ]
1187+ nodes = [
1188+ factory.make_Node_with_Interface_on_Subnet(subnet=subnet)
1189+ for subnet in subnets
1190+ ]
1191+ # Filter for this subnet. Take one in the middle to avoid
1192+ # coincidental success based on ordering.
1193+ pick = 1
1194+ self.assertConstrainedNodes(
1195+ {nodes[pick]},
1196+ {'networks': [subnets[pick].name]})
1197+
1198 def test_networks_filters_by_name(self):
1199 subnets = [
1200 factory.make_Subnet()
1201@@ -310,11 +326,27 @@
1202 ]
1203 # Filter for this subnet. Take one in the middle to avoid
1204 # coincidental success based on ordering.
1205- pick = 2
1206+ pick = 1
1207 self.assertConstrainedNodes(
1208 {nodes[pick]},
1209 {'networks': [subnets[pick].name]})
1210
1211+ def test_networks_filters_by_space(self):
1212+ subnets = [
1213+ factory.make_Subnet()
1214+ for _ in range(3)
1215+ ]
1216+ nodes = [
1217+ factory.make_Node_with_Interface_on_Subnet(subnet=subnet)
1218+ for subnet in subnets
1219+ ]
1220+ # Filter for this subnet. Take one in the middle to avoid
1221+ # coincidental success based on ordering.
1222+ pick = 1
1223+ self.assertConstrainedNodes(
1224+ {nodes[pick]},
1225+ {'networks': ["space:%s" % subnets[pick].space.name]})
1226+
1227 def test_networks_filters_by_ip(self):
1228 subnets = [
1229 factory.make_Subnet()
1230@@ -326,7 +358,7 @@
1231 ]
1232 # Filter for this subnet. Take one in the middle to avoid
1233 # coincidental success based on ordering.
1234- pick = 2
1235+ pick = 1
1236 self.assertConstrainedNodes(
1237 {nodes[pick]},
1238 {'networks': [
1239@@ -345,7 +377,7 @@
1240 ]
1241 # Filter for this network. Take one in the middle to avoid
1242 # coincidental success based on ordering.
1243- pick = 2
1244+ pick = 1
1245 self.assertConstrainedNodes(
1246 {nodes[pick]},
1247 {'networks': ['vlan:%d' % vlan_tags[pick]]})
1248@@ -371,14 +403,15 @@
1249 {node},
1250 {'networks': [subnets[1].name]})
1251
1252- def test_invalid_networks(self):
1253+ def test_invalid_subnets(self):
1254 form = AcquireNodeForm(data={'networks': 'ip:10.0.0.0'})
1255 self.assertEquals(
1256 (
1257 False,
1258 {
1259 'networks': [
1260- "Invalid parameter: list of networks required.",
1261+ "Invalid parameter: "
1262+ "list of subnet specifiers required.",
1263 ],
1264 },
1265 ),
1266@@ -388,7 +421,8 @@
1267 # is being consulted.
1268 form = AcquireNodeForm(data={'networks': ['vlan:-1']})
1269 self.assertEquals(
1270- (False, {'networks': ["VLAN tag out of range (1-4094)."]}),
1271+ (False, {'networks': [
1272+ "VLAN tag (VID) out of range (0-4094; 0 for untagged.)"]}),
1273 (form.is_valid(), form.errors))
1274
1275 def test_networks_combines_filters(self):
1276@@ -430,6 +464,17 @@
1277 [node],
1278 {'networks': [this_subnet.name]})
1279
1280+ def test_legacy_not_networks_falls_back_to_not_networks_query(self):
1281+ [not_subnet, subnet] = [
1282+ factory.make_Subnet()
1283+ for _ in range(2)
1284+ ]
1285+ factory.make_Node_with_Interface_on_Subnet(subnet=not_subnet)
1286+ node = factory.make_Node_with_Interface_on_Subnet(subnet=subnet)
1287+ self.assertConstrainedNodes(
1288+ {node},
1289+ {'not_networks': [not_subnet.name]})
1290+
1291 def test_not_networks_filters_by_name(self):
1292 [subnet, not_subnet] = [
1293 factory.make_Subnet()
1294@@ -474,7 +519,7 @@
1295 {interfaceless_node, unconnected_node},
1296 {'not_networks': [factory.make_Subnet().name]})
1297
1298- def test_not_networks_exclude_node_with_any_interface_on_not_subnets(self):
1299+ def test_not_networks_exclude_node_with_any_interface(self):
1300 subnet = factory.make_Subnet()
1301 node = factory.make_Node_with_Interface_on_Subnet(subnet=subnet)
1302 other_subnet = factory.make_Subnet()
1303@@ -497,7 +542,8 @@
1304 False,
1305 {
1306 'not_networks': [
1307- "Invalid parameter: list of networks required.",
1308+ "Invalid parameter: "
1309+ "list of subnet specifiers required.",
1310 ],
1311 },
1312 ),
1313@@ -507,7 +553,8 @@
1314 # is being consulted.
1315 form = AcquireNodeForm(data={'not_networks': ['vlan:-1']})
1316 self.assertEquals(
1317- (False, {'not_networks': ["VLAN tag out of range (1-4094)."]}),
1318+ (False, {'not_networks': [
1319+ "VLAN tag (VID) out of range (0-4094; 0 for untagged.)"]}),
1320 (form.is_valid(), form.errors))
1321
1322 def test_not_networks_combines_filters(self):
1323@@ -866,25 +913,88 @@
1324 id=constraint_map[node.id].keys()[1]) # 2nd constraint with name
1325 self.assertGreaterEqual(disk1.size, 5 * 1000 ** 3)
1326
1327- def test_networking_constraint_rejected_if_syntax_is_invalid(self):
1328- factory.make_Node_with_Interface_on_Subnet()
1329- form = AcquireNodeForm({
1330- u'networking': u'label:x'})
1331- self.assertFalse(form.is_valid(), dict(form.errors))
1332- self.assertThat(form.errors, Contains('networking'))
1333-
1334- def test_networking_constraint_rejected_if_key_is_invalid(self):
1335- factory.make_Node_with_Interface_on_Subnet()
1336- form = AcquireNodeForm({
1337- u'networking': u'label:chirp_chirp_thing=silenced'})
1338- self.assertFalse(form.is_valid(), dict(form.errors))
1339- self.assertThat(form.errors, Contains('networking'))
1340-
1341- def test_networking_constraint_validated(self):
1342- factory.make_Node_with_Interface_on_Subnet()
1343- form = AcquireNodeForm({
1344- u'networking': u'label:fabric=fabric-0'})
1345- self.assertTrue(form.is_valid(), dict(form.errors))
1346+ def test_fabrics_constraint(self):
1347+ fabric1 = factory.make_Fabric(name="fabric1")
1348+ fabric2 = factory.make_Fabric(name="fabric2")
1349+ factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
1350+ node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
1351+ form = AcquireNodeForm({
1352+ u'fabrics': [u'fabric2']})
1353+ self.assertTrue(form.is_valid(), dict(form.errors))
1354+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1355+ self.assertItemsEqual([node2], filtered_nodes)
1356+
1357+ def test_not_fabrics_constraint(self):
1358+ fabric1 = factory.make_Fabric(name="fabric1")
1359+ fabric2 = factory.make_Fabric(name="fabric2")
1360+ factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
1361+ node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
1362+ form = AcquireNodeForm({
1363+ u'not_fabrics': [u'fabric1']})
1364+ self.assertTrue(form.is_valid(), dict(form.errors))
1365+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1366+ self.assertItemsEqual([node2], filtered_nodes)
1367+
1368+ def test_fabric_classes_constraint(self):
1369+ fabric1 = factory.make_Fabric(class_type="10g")
1370+ fabric2 = factory.make_Fabric(class_type="1g")
1371+ factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
1372+ node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
1373+ form = AcquireNodeForm({
1374+ u'fabric_classes': [u'1g']})
1375+ self.assertTrue(form.is_valid(), dict(form.errors))
1376+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1377+ self.assertItemsEqual([node2], filtered_nodes)
1378+
1379+ def test_not_fabric_classes_constraint(self):
1380+ fabric1 = factory.make_Fabric(class_type="10g")
1381+ fabric2 = factory.make_Fabric(class_type="1g")
1382+ factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
1383+ node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
1384+ form = AcquireNodeForm({
1385+ u'not_fabric_classes': [u'10g']})
1386+ self.assertTrue(form.is_valid(), dict(form.errors))
1387+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1388+ self.assertItemsEqual([node2], filtered_nodes)
1389+
1390+ def test_interfaces_constraint_rejected_if_syntax_is_invalid(self):
1391+ factory.make_Node_with_Interface_on_Subnet()
1392+ form = AcquireNodeForm({
1393+ u'interfaces': u'label:x'})
1394+ self.assertFalse(form.is_valid(), dict(form.errors))
1395+ self.assertThat(form.errors, Contains('interfaces'))
1396+
1397+ def test_interfaces_constraint_rejected_if_key_is_invalid(self):
1398+ factory.make_Node_with_Interface_on_Subnet()
1399+ form = AcquireNodeForm({
1400+ u'interfaces': u'label:chirp_chirp_thing=silenced'})
1401+ self.assertFalse(form.is_valid(), dict(form.errors))
1402+ self.assertThat(form.errors, Contains('interfaces'))
1403+
1404+ def test_interfaces_constraint_validated(self):
1405+ factory.make_Node_with_Interface_on_Subnet()
1406+ form = AcquireNodeForm({
1407+ u'interfaces': u'label:fabric=fabric-0'})
1408+ self.assertTrue(form.is_valid(), dict(form.errors))
1409+
1410+ @skip("XXX mpontillo 2015-10-30 need to use upcoming interfaces filter")
1411+ def test_interfaces_filters_by_fabric_class(self):
1412+ fabric1 = factory.make_Fabric(class_type="1g")
1413+ fabric2 = factory.make_Fabric(class_type="10g")
1414+ node1 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric1)
1415+ node2 = factory.make_Node_with_Interface_on_Subnet(fabric=fabric2)
1416+
1417+ form = AcquireNodeForm({
1418+ u'interfaces': u'label:fabric_class=10g'})
1419+ self.assertTrue(form.is_valid(), dict(form.errors))
1420+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1421+ self.assertItemsEqual([node2], filtered_nodes)
1422+
1423+ form = AcquireNodeForm({
1424+ u'interfaces': u'label:fabric_class=1g'})
1425+ self.assertTrue(form.is_valid(), dict(form.errors))
1426+ filtered_nodes, _ = form.filter_nodes(Node.nodes)
1427+ self.assertItemsEqual([node1], filtered_nodes)
1428
1429 def test_combined_constraints(self):
1430 tag_big = factory.make_Tag(name='big')
1431@@ -985,7 +1095,13 @@
1432 'zone': factory.make_Zone(),
1433 'not_in_zone': [factory.make_Zone().name],
1434 'storage': '0(ssd),10(ssd)',
1435- 'networking': 'label:fabric=fabric-0',
1436+ 'interfaces': 'label:fabric=fabric-0',
1437+ 'fabrics': [factory.make_Fabric().name],
1438+ 'not_fabrics': [factory.make_Fabric().name],
1439+ 'fabric_classes': [
1440+ factory.make_Fabric(class_type="10g").class_type],
1441+ 'not_fabric_classes': [
1442+ factory.make_Fabric(class_type="1g").class_type],
1443 }
1444 form = AcquireNodeForm(data=constraints)
1445 self.assertTrue(form.is_valid(), form.errors)
1446
1447=== modified file 'src/maasserver/utils/orm.py'
1448--- src/maasserver/utils/orm.py 2015-10-06 23:55:25 +0000
1449+++ src/maasserver/utils/orm.py 2015-10-30 19:51:38 +0000
1450@@ -666,3 +666,27 @@
1451 """Close database connections in the current thread."""
1452 for alias in connections:
1453 connections[alias].close()
1454+
1455+
1456+class MAASQueriesMixin(object):
1457+ """Contains utility functions that any mixin for model object manager
1458+ queries may need to make use of."""
1459+
1460+ def get_id_list(self, raw_query):
1461+ """Returns a list of IDs for each row in the specified raw query.
1462+
1463+ This can be used to create additional filters to chain from a raw
1464+ query, which would not otherwise be possible.
1465+
1466+ Note that using this method risks a race condition, since a row
1467+ could be inserted after the raw query runs.
1468+ """
1469+ ids = [row.id for row in raw_query]
1470+ return self.filter(id__in=ids)
1471+
1472+ def get_id_filter(self, raw_query):
1473+ """Returns a `QuerySet` for the specified raw query, by executing it
1474+ and adding an 'in' filter with the ID of each object in the raw query.
1475+ """
1476+ ids = self.get_id_list(raw_query)
1477+ return self.filter(id__in=ids)
1478
1479=== modified file 'src/provisioningserver/utils/network.py'
1480--- src/provisioningserver/utils/network.py 2015-10-05 00:12:27 +0000
1481+++ src/provisioningserver/utils/network.py 2015-10-30 19:51:38 +0000
1482@@ -505,3 +505,25 @@
1483 """Convert IPv4 and IPv6 addresses from binary to text form.
1484 (See also inet_ntop(3), the C function with the same name and function.)"""
1485 return unicode(IPAddress(value))
1486+
1487+
1488+def parse_integer(value_string):
1489+ """Convert the specified `value_string` into a decimal integer.
1490+
1491+ Strips whitespace, and handles hexadecimal or binary format strings,
1492+ if the string is prefixed with '0x' or '0b', respectively.
1493+
1494+ :raise:ValueError if the conversion to int fails
1495+ :return:int
1496+ """
1497+ value_string = value_string.strip()
1498+ if value_string.lower().startswith('0x'):
1499+ # Hexadecimal.
1500+ base = 16
1501+ elif value_string.lower().startswith('0b'):
1502+ # Binary
1503+ base = 2
1504+ else:
1505+ # When all else fails, assume decimal.
1506+ base = 10
1507+ return int(value_string, base)
1508
1509=== modified file 'src/provisioningserver/utils/tests/test_network.py'
1510--- src/provisioningserver/utils/tests/test_network.py 2015-10-05 00:12:27 +0000
1511+++ src/provisioningserver/utils/tests/test_network.py 2015-10-30 19:51:38 +0000
1512@@ -52,6 +52,7 @@
1513 MAASIPSet,
1514 make_iprange,
1515 make_network,
1516+ parse_integer,
1517 resolve_hostname,
1518 )
1519 from testtools.matchers import (
1520@@ -733,3 +734,25 @@
1521 self.assertThat(json['inefficiency_string'], Equals("TBD"))
1522 self.assertThat(json, Contains('ranges'))
1523 self.assertThat(json['ranges'], Equals(stats.ranges.render_json()))
1524+
1525+
1526+class TestParseInteger(MAASTestCase):
1527+
1528+ def test__parses_decimal_integer(self):
1529+ self.assertThat(parse_integer("0"), Equals(0))
1530+ self.assertThat(parse_integer("1"), Equals(1))
1531+ self.assertThat(parse_integer("-1"), Equals(-1))
1532+ self.assertThat(parse_integer("1000"), Equals(1000))
1533+ self.assertThat(parse_integer("10000000"), Equals(10000000))
1534+
1535+ def test__parses_hexadecimal_integer(self):
1536+ self.assertThat(parse_integer("0x0"), Equals(0))
1537+ self.assertThat(parse_integer("0x1"), Equals(1))
1538+ self.assertThat(parse_integer("0x1000"), Equals(0x1000))
1539+ self.assertThat(parse_integer("0x10000000"), Equals(0x10000000))
1540+
1541+ def test__parses_binary_integer(self):
1542+ self.assertThat(parse_integer("0b0"), Equals(0))
1543+ self.assertThat(parse_integer("0b1"), Equals(1))
1544+ self.assertThat(parse_integer("0b1000"), Equals(0b1000))
1545+ self.assertThat(parse_integer("0b10000000"), Equals(0b10000000))