Merge lp:~jason-hobbs/maas/ucs-xml-api into lp:~maas-committers/maas/trunk

Proposed by Jason Hobbs
Status: Merged
Approved by: Jason Hobbs
Approved revision: no longer in the source branch.
Merged at revision: 2303
Proposed branch: lp:~jason-hobbs/maas/ucs-xml-api
Merge into: lp:~maas-committers/maas/trunk
Diff against target: 1349 lines (+1215/-0)
12 files modified
etc/maas/templates/power/ucsm.template (+15/-0)
src/maasserver/api.py (+23/-0)
src/maasserver/models/nodegroup.py (+11/-0)
src/maasserver/tests/test_api_nodegroup.py (+27/-0)
src/provisioningserver/custom_hardware/tests/test_ucsm.py (+638/-0)
src/provisioningserver/custom_hardware/ucsm.py (+435/-0)
src/provisioningserver/power/tests/test_poweraction.py (+11/-0)
src/provisioningserver/power_schema.py (+10/-0)
src/provisioningserver/tasks.py (+8/-0)
src/provisioningserver/tests/test_tasks.py (+12/-0)
src/provisioningserver/utils/__init__.py (+5/-0)
src/provisioningserver/utils/tests/test_utils.py (+20/-0)
To merge this branch: bzr merge lp:~jason-hobbs/maas/ucs-xml-api
Reviewer Review Type Date Requested Status
Gavin Panella (community) Abstain
Jeroen T. Vermeulen (community) Approve
Review via email: mp+216643@code.launchpad.net

Commit message

Add support for managing nodes via Cisco UCS Manager's HTTP-XML API.

This adds a new probe and enlist API method and a new power control template for UCS managed nodes.

Description of the change

This adds probe/enlist and power control support for Cisco's UCS HTTP-XML API. I'm still in the final stages of testing it, but it's ready for review now. It's big commit; it wasn't clear to me how to make it smaller but still make it useful.

The power template here is kind of pointless, since the power control is pure python, but it wasn't trivial to work around it. I think we need to consider moving away from always using templates for power control, and make it an option to use a python method directly.

I also think there is room for some common code in handling probe and enlist. The basic pattern is build a list of new nodes and then add them to MAAS. How we deal with existing nodes could be common across all types of enlistments, and having a common interface for the methods could maybe make it easier to build a web UI interface at some point too.

To post a comment you must log in.
Revision history for this message
Julian Edwards (julian-edwards) wrote :

1 === added file 'etc/maas/templates/power/ucsm.template'
2 --- etc/maas/templates/power/ucsm.template 1970-01-01 00:00:00 +0000
3 +++ etc/maas/templates/power/ucsm.template 2014-04-29 13:55:35 +0000
4 @@ -0,0 +1,8 @@
5 +# -*- mode: shell-script -*-
6 +#
7 +# Control a system via Cisco UCS Manager XML API.
8 +
9 +python - << END
10 +from provisioningserver.custom_hardware.ucsm import power_control_ucsm
11 +power_control_ucsm('{{power_address}}', '{{power_user}}', '{{power_pass}}', '{{uuid}}', '{{power_change}}')
12 +END

This is "hope quoting" (as Gavin calls it) and ripe for abuse with unchecked inputs. See the conversation in https://code.launchpad.net/~blake-rouse/maas/virsh-probe-and-enlist/+merge/216632

Revision history for this message
Gavin Panella (allenap) wrote :

This is half a review. I have to go out, and I'll finish it later.

As for splitting it up, for one you could have added an empty add_ucsm()
function to NodeGroup, building the API calls around it, and writing
tests in which you mock the behaviour of add_ucsm().

[1]

Hope-quoting, as Julian has mentioned.

[2]

+    def add_ucsm(self, url, username, password):

This isn't a great name for this method. However, I see that it's
consistent with what's gone before.

[3]

+        children = map(Element, children_tags)
+        request_data = make_request_data('foo', fields, children)

Bear in mind that map() is lazy in Python 3. Will make_request_data()
deal with that? If not, make it explicit with a list comprehension.

[4]

+        self.assertEqual(set(children_tags), set([e.tag for e in root[:]]))

The list-comprehension is not needed:

        self.assertEqual(set(children_tags), set(e.tag for e in root[:]))

Not sure the slice is either?

        self.assertEqual(set(children_tags), set(e.tag for e in root))

Having said that, assertItemsEqual() is useful here:

        self.assertItemsEqual(children_tags, (e.tag for e in root))

[5]

+        self.assertRaises(UCSM_XML_API_Error, lambda: parse_response(xml))

assertRaises() allows arguments, so you can write this as:

        self.assertRaises(UCSM_XML_API_Error, parse_response, xml)

[6]

+    def make_test_api(self, url='http://url', user='u', password='p',
+                      cookie='foo', mock_call=True):
+
+        api = make_test_api(url, user, password, cookie)
+
+        if mock_call:
+            mock = self.patch(api, '_call')
+        else:
+            mock = None
+
+        return api, mock

Boolean arguments are a bit smelly, as is returning irrelevant
things. Can you split this into two methods?

    def make_test_api(
            self, url='http://url', user='u', password='p', cookie='foo'):
        return make_test_api(url, user, password, cookie)

    def make_test_api(
            self, url='http://url', user='u', password='p', cookie='foo'):
        api = make_test_api(url, user, password, cookie)
        return api, self.patch(api, '_call')

To be continued...

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

There's lots of good stuff here, well-factored, and with a very nice breakdown into coherent, low-complexity functions. But there's just so much of it that I can't complete this review all in one go!

I'll give you the notes I have now. I reviewed up to UCSM_XML_API, moving from the top of the diff downwards, except I skipped its tests for now.

.

The docstring for probe_and_enlist_hardware needs updating, and probe_and_enlist_ucsm should probably be a bit more specific about the hardware types it supports.

.

Typo: Cicso.

.

The test case for make_request_data is called TestGetRequestData — uncompleted change of name?

.

In TestUCSMXMLAPIError, I appreciate the sentiment of raising the exception and catching it before testing its properties, but I don't see it buying us anything: it basically tests that “raise” and assertRaises pass on the exception they received.

(Also, “raise” is a keyword, not a function — no parentheses needed when using it.)

.

In TestParseResponse.test_error:

        self.assertRaises(UCSM_XML_API_Error, lambda: parse_response(xml))

No need for the lambda. You can just pass additional arguments for the callable (as well as keyword arguments) to assertRaises:

        self.assertRaises(UCSM_XML_API_Error, parse_response, xml)

.

Unless you're in the process of adding more methods to APITestCase, I'd just drop it as a base class. Using inheritance to add features seems like a small thing, but consider how many test-case classes are suddenly tied together in a vague relationship! And once in place, these relationships tend to grow.

What to do with APITestCase.make_test_api? You could just pass the test-case object to the global make_test_api, and maybe skip the patching if testcase is None.

Actually, I'd probably just leave out the option to suppress the patching, and accept a slightly special case in the one call site where you use that option. This kind of boolean argument is almost always a constant, which means you've really got two similar-but-different functions, and the caller already knows exactly which one they want. You're almost always better off writing the two forms as separate functions. Often, one can call the other, or you can extract meaningful parts that will be called by both.

.

UCSM_XML_API is quite a test for naming conventions! The underscores are unusual in a class name for us, but these are unusual circumstances so you won't hear me complain. :)

I do wish I had a clearer understanding of the “children” that are being mentioned throughout. I understand that it's a domain term, so I'd probably need to read up on the UCSM API in order to understand. A very brief note might make it easier to deal with though. Setting the right expectations often helps prevents disasters.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Looks like you're getting multiple reviews. One of the risks of large branches! Reviewers' ability to spot problems drops off sharply after the first few hundred lines, so it's probably for the best. (The “sweet spot” for reviews seems to lie around 200 lines).

Some more notes:

.

Could you document what RO_KEYS and strip_ro_keys mean? I see how they fit together, but I lack context about the wider problem it solves.

.

In get_power_command, there's an if/elif without a final ‘else.’ So an unknown power state will just result in a None, which could produce failures that are hard to diagnose. You could avoid that hazard by adding an “else: raise AssertionError(...)” at the end.

.

Some identifiers have abbreviations whose meanings are a complete mystery to me, such as dn and bp. Long, verbose names can be unpleasant, but names with unknown meanings can be a positive menace!

.

In power_control_ucsm, you do:

        servers = get_servers(api, uuid)
        server = servers[0]

Why server[0]? Are you taking an arbitrary one? The latest? Something else? Would be worth a comment.

Or, if you are absolutely 100% totally completely certainly positive that there is exactly one item in that list, pattern-matching lets you write the assignment as:

        [server] = get_servers(api, uuid)

.

You document power_control_ucsm as “Power control template handler.” I think I sort of understand all of that, except the “template.” How does that word fit in? It may help to summarise what the function does in the imperative.

.

Similarly, probe_and_enlist_ucsm is documented as “Probe and enlist request handler.” I *think* this fails to state what the function does in the imperative, but I'm not sure. It's ambiguous: there is a request handler, and this function probes and enlists it? Or is the function a “Probe-and-enlist request handler”? If so, where does the request come in?

.

For the addition to src/provisioningserver/power_schema.py, might I suggest "double quotes" around free-form text? I'm not saying that “URL for XML API” should be changed to “XML API's URL” for example, but if anyone tried, the apostrophe would break the strings quoting!

.

In TestAddUCSM, a disadvantage of passing highly predictable fixed strings like “username” and “password” is that the test doesn't rule out silly bugs like “always passes ‘username’ instead of the actual user name.” Not saying that that is terribly likely in any given instance, but if we build the right habits, we won't need to make the mistake at all. These mistakes can happen in TDD when you try to write the simplest stub that will satisfy a test.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

By the way, we deviate from PEP-8 in formatting. When linebreaking (except in def/class definitions) we only indent by 4 columns. We don't align to the opening brace/bracket/parenthesis.

So, where you wrote

        self.assertThat(mock, MockCalledOnceWith('configResolveClass', fields,
                                                 ANY))

...we might write:

        self.assertThat(
            mock, MockCalledOnceWith('configResolveClass', fields, ANY))

When we need to line-break an arguments list, we do make it a rule to start the first argument on a fresh line.

.

In further bikeshedding, method names “login” and “logout” look a bit strange for me — we generally prefer our function names to start with verbs, but these are the nouns.

.

At this point I'm a bit stuck. I'd like to review the tests in more detail, but it might not be much use because of my limited understanding of the API code's higher-level meaning. It would be nice to have that little bit more explanation!

Revision history for this message
Gavin Panella (allenap) wrote :
Download full text (4.1 KiB)

Neat, nice stuff. My comments are almost all about style, not approach.
I think my only real problems with this branch are [1] and the lack of
commentary about how things work. If it's biting me now, it'll bite you
in 3 months when you've long moved on from this.

[7]

+        self.assertEqual([filter_element], in_filters[0][:])

What's the [:] slice for?

[8]

+    def test_parameters(self):
...
+        in_configs = mock.call_args[0][2]
+        self.assertEqual(config_items, in_configs[0][:])

I don't know what this is demonstrating. Can you add a comment, or
perhaps make it more intrinsically clear. There's a mock.ANY object
which you can use when you don't care what an argument is (e.g. so you
can use one of the Mock* matchers but only test one argument of
several), which may or may not be useful here.

[9]

+        self.assertThat(mock_make_request_data,
+                        MockCalledOnceWith(name, fields, children))

Fwiw, and it's a minor thing, the MAAS core team (former Red Squad) long
ago chose to not indent after braces like this. For one, it gets silly
when the function name is long and all the arguments get smashed against
the right margin. Prefer starting a new line:

        self.assertThat(
            mock_make_request_data,
            MockCalledOnceWith(name, fields, children))

For functions, it might be worth indenting twice:

    def this_is_a_function(
            with_a_long, argument_list, foo, bar):
        do_something(...)

or using a docstring or comment to delineate arguments from body:

    def this_is_a_function(
        with_a_long, argument_list, foo, bar):
        """Docstring, oh docstring, where art thou o docstring?"""
        do_something(...)

I'm definitely not going to block on this though.

[10]

+    def test_uses_uuid(self):
+        uuid = factory.getRandomUUID()
+        api = make_test_api()
+        mock = self.patch(api, 'config_resolve_class')
+        get_servers(api, uuid)
+        filters = mock.call_args[0][1]
+        attrib = {'class': 'computeItem', 'property': 'uuid', 'value': uuid}
+        self.assertEqual(attrib, filters[0].attrib)

There could have been multiple calls to config_resolve_class. Consider
using MockCalledOnceWith() and mock.ANY here, and elsewhere, to tighten
up the test, and make the test clearer and more concise.

[11]

+    def login(self):
...
+        UCS Manager allows a limited number of active cookies at any
+        point in time, so it's important to free the cookie up when
+        finished by logging out via the ``logout`` method.

Consider making UCSM_XML_API a context manager, where __enter__ logs-in,
and __exit__ logs-out.

[12]

+    def config_resolve_class(self, class_id, filters=None):
+        """Issue a configResolveClass request."""

What does configResolveClass do? Either comment here, or provide a
(stable, preferably) link to the UCS API documentation. Extra beer for
both. The same goes for most of the other methods, if not all.

[13]

+def get_servers(api, uuid=None):
+    """Retrieve a list of servers from the UCS Manager."""
+    if uuid:
+        attrs = {'class': 'computeItem', 'property': 'uuid', 'value': uuid}
+        fil...

Read more...

review: Needs Fixing
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

I've addressed what I think are the critical feedback items, along with some others. There is time sensitivity in getting this in, so I want to repost for review again before all of the feedback items are addressed. I will post a summary of what is left to address, after I get my laptop on a charger!

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

Here's the review comments I haven't addressed yet.

From Jeroen:
In TestUCSMXMLAPIError, I appreciate the sentiment of raising the exception and catching it before testing its properties, but I don't see it buying us anything: it basically tests that “raise” and assertRaises pass on the exception they received.

In TestAddUCSM, a disadvantage of passing highly predictable fixed strings like “username” and “password” is that the test doesn't rule out silly bugs like “always passes ‘username’ instead of the actual user name.” Not saying that that is terribly likely in any given instance, but if we build the right habits, we won't need to make the mistake at all. These mistakes can happen in TDD when you try to write the simplest stub that will satisfy a test.

By the way, we deviate from PEP-8 in formatting. When linebreaking (except in def/class definitions) we only indent by 4 columns. We don't align to the opening brace/bracket/parenthesis.

In further bikeshedding, method names “login” and “logout” look a bit strange for me — we generally prefer our function names to start with verbs, but these are the nouns.

From Gavin:
[8], [9], [10], [11], [13], [14]

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Great, thanks. I see no reason why this shouldn't be allowed in at this point. I think the remaining things are pretty much all non-essential cleanups that will just happen organically if they bother us again.

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (46.7 KiB)

The attempt to merge lp:~jason-hobbs/maas/ucs-xml-api into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Get:2 http://security.ubuntu.com trusty-security Release [58.5 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [58.5 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [11.3 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security/universe Sources [1,785 B]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [36.5 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [9,525 B]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:9 http://security.ubuntu.com trusty-security/main Translation-en [16.2 kB]
Get:10 http://security.ubuntu.com trusty-security/universe Translation-en [5,005 B]
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://security.ubuntu.com trusty-security/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Ign http://security.ubuntu.com trusty-security/universe Translation-en_US
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [19.4 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [9,282 B]
Get:13 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [55.4 kB]
Get:14 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [26.2 kB]
Get:15 http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en [25.3 kB]
Get:16 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en [13.1 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en_US
Fetched 348 kB in 1s (285 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make postgresql python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-...

Revision history for this message
Gavin Panella (allenap) wrote :

I haven't gone through the changes but Jeroen has, and I doubt I can add much.

review: Abstain

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'etc/maas/templates/power/ucsm.template'
2--- etc/maas/templates/power/ucsm.template 1970-01-01 00:00:00 +0000
3+++ etc/maas/templates/power/ucsm.template 2014-05-06 13:49:30 +0000
4@@ -0,0 +1,15 @@
5+# -*- mode: shell-script -*-
6+#
7+# Control a system via Cisco UCS Manager XML API.
8+
9+{{py: from provisioningserver.utils import escape_py_literal}}
10+python - << END
11+from provisioningserver.custom_hardware.ucsm import power_control_ucsm
12+power_control_ucsm(
13+ {{escape_py_literal(power_address) | safe}},
14+ {{escape_py_literal(power_user) | safe}},
15+ {{escape_py_literal(power_pass) | safe}},
16+ {{escape_py_literal(uuid) | safe}},
17+ {{escape_py_literal(power_change) | safe}},
18+)
19+END
20
21=== modified file 'src/maasserver/api.py'
22--- src/maasserver/api.py 2014-05-01 05:38:03 +0000
23+++ src/maasserver/api.py 2014-05-06 13:49:30 +0000
24@@ -1728,6 +1728,29 @@
25
26 return HttpResponse(status=httplib.OK)
27
28+ @admin_method
29+ @operation(idempotent=False)
30+ def probe_and_enlist_ucsm(self, request, uuid):
31+ """Add the nodes from a Cisco UCS Manager.
32+
33+ :param : The URL of the UCS Manager API.
34+ :type url: unicode
35+ :param username: The username for the API.
36+ :type username: unicode
37+ :param password: The password for the API.
38+ :type password: unicode
39+
40+ """
41+ nodegroup = get_object_or_404(NodeGroup, uuid=uuid)
42+
43+ url = get_mandatory_param(request.data, 'url')
44+ username = get_mandatory_param(request.data, 'username')
45+ password = get_mandatory_param(request.data, 'password')
46+
47+ nodegroup.enlist_nodes_from_ucsm(url, username, password)
48+
49+ return HttpResponse(status=httplib.OK)
50+
51 DISPLAYED_NODEGROUPINTERFACE_FIELDS = (
52 'ip', 'management', 'interface', 'subnet_mask',
53 'broadcast_ip', 'ip_range_low', 'ip_range_high')
54
55=== modified file 'src/maasserver/models/nodegroup.py'
56--- src/maasserver/models/nodegroup.py 2014-04-01 06:50:01 +0000
57+++ src/maasserver/models/nodegroup.py 2014-05-06 13:49:30 +0000
58@@ -41,6 +41,7 @@
59 from provisioningserver.tasks import (
60 add_new_dhcp_host_map,
61 add_seamicro15k,
62+ enlist_nodes_from_ucsm,
63 import_boot_images,
64 report_boot_images,
65 )
66@@ -286,6 +287,16 @@
67 args = (mac, username, password, power_control)
68 add_seamicro15k.apply_async(queue=self.uuid, args=args)
69
70+ def enlist_nodes_from_ucsm(self, url, username, password):
71+ """ Add the servers from a Cicso UCS Manager.
72+
73+ :param URL: URL of the Cisco UCS Manager HTTP-XML API.
74+ :param username: username for UCS Manager.
75+ :param password: password for UCS Manager.
76+ """
77+ args = (url, username, password)
78+ enlist_nodes_from_ucsm.apply_async(queue=self.uuid, args=args)
79+
80 def add_dhcp_host_maps(self, new_leases):
81 if len(new_leases) > 0 and len(self.get_managed_interfaces()) > 0:
82 # XXX JeroenVermeulen 2012-08-21, bug=1039362: the DHCP
83
84=== modified file 'src/maasserver/tests/test_api_nodegroup.py'
85--- src/maasserver/tests/test_api_nodegroup.py 2014-04-22 15:10:32 +0000
86+++ src/maasserver/tests/test_api_nodegroup.py 2014-05-06 13:49:30 +0000
87@@ -52,6 +52,7 @@
88 from maasserver.testing.testcase import MAASServerTestCase
89 from maastesting.celery import CeleryFixture
90 from maastesting.fakemethod import FakeMethod
91+from maastesting.matchers import MockCalledOnceWith
92 from metadataserver.fields import Bin
93 from metadataserver.models import (
94 commissioningscript,
95@@ -431,6 +432,32 @@
96 httplib.BAD_REQUEST, response.status_code,
97 explain_unexpected_response(httplib.BAD_REQUEST, response))
98
99+ def test_probe_and_enlist_ucsm_adds_ucsm(self):
100+ nodegroup = factory.make_node_group()
101+ url = 'http://url'
102+ username = factory.make_name('user')
103+ password = factory.make_name('password')
104+ self.become_admin()
105+
106+ mock = self.patch(nodegroup_module, 'enlist_nodes_from_ucsm')
107+
108+ response = self.client.post(
109+ reverse('nodegroup_handler', args=[nodegroup.uuid]),
110+ {
111+ 'op': 'probe_and_enlist_ucsm',
112+ 'url': url,
113+ 'username': username,
114+ 'password': password,
115+ })
116+
117+ self.assertEqual(
118+ httplib.OK, response.status_code,
119+ explain_unexpected_response(httplib.OK, response))
120+
121+ args = (url, username, password)
122+ matcher = MockCalledOnceWith(queue=nodegroup.uuid, args=args)
123+ self.assertThat(mock.apply_async, matcher)
124+
125
126 class TestNodeGroupAPIAuth(MAASServerTestCase):
127 """Authorization tests for nodegroup API."""
128
129=== added file 'src/provisioningserver/custom_hardware/tests/test_ucsm.py'
130--- src/provisioningserver/custom_hardware/tests/test_ucsm.py 1970-01-01 00:00:00 +0000
131+++ src/provisioningserver/custom_hardware/tests/test_ucsm.py 2014-05-06 13:49:30 +0000
132@@ -0,0 +1,638 @@
133+# Copyright 2014 Canonical Ltd. This software is licensed under the
134+# GNU Affero General Public License version 3 (see the file LICENSE).
135+
136+"""Tests for ``provisioningserver.custom_hardware.ucsm``."""
137+
138+from __future__ import (
139+ absolute_import,
140+ print_function,
141+ unicode_literals,
142+ )
143+
144+str = None
145+
146+__metaclass__ = type
147+__all__ = []
148+
149+from itertools import permutations
150+import random
151+from StringIO import StringIO
152+import urllib2
153+
154+from lxml.etree import (
155+ Element,
156+ SubElement,
157+ XML,
158+ )
159+from maastesting.factory import factory
160+from maastesting.matchers import (
161+ MockCalledOnceWith,
162+ MockCallsMatch,
163+ MockNotCalled,
164+ )
165+from maastesting.testcase import MAASTestCase
166+from mock import (
167+ ANY,
168+ call,
169+ Mock,
170+ )
171+from provisioningserver.custom_hardware import (
172+ ucsm,
173+ utils,
174+ )
175+from provisioningserver.custom_hardware.ucsm import (
176+ get_children,
177+ get_first_booter,
178+ get_macs,
179+ get_power_command,
180+ get_server_power_control,
181+ get_servers,
182+ get_service_profile,
183+ logged_in,
184+ make_policy_change,
185+ make_request_data,
186+ parse_response,
187+ power_control_ucsm,
188+ probe_and_enlist_ucsm,
189+ probe_servers,
190+ RO_KEYS,
191+ set_lan_boot_default,
192+ set_server_power_control,
193+ strip_ro_keys,
194+ UCSM_XML_API,
195+ UCSM_XML_API_Error,
196+ )
197+
198+
199+def make_api(url='http://url', user='u', password='p',
200+ cookie='foo', mock_call=True):
201+ api = UCSM_XML_API(url, user, password)
202+ api.cookie = cookie
203+ return api
204+
205+
206+def make_api_patch_call(testcase, *args, **kwargs):
207+ api = make_api(*args, **kwargs)
208+ mock = testcase.patch(api, '_call')
209+ return api, mock
210+
211+
212+def make_fake_result(root_class, child_tag, container='outConfigs'):
213+ fake_result = Element(root_class)
214+ outConfigs = SubElement(fake_result, container)
215+ outConfigs.append(Element(child_tag))
216+ return outConfigs
217+
218+
219+def make_class():
220+ return factory.make_name('class')
221+
222+
223+def make_dn():
224+ return factory.make_name('dn')
225+
226+
227+def make_server():
228+ return factory.make_name('server')
229+
230+
231+class TestUCSMXMLAPIError(MAASTestCase):
232+ """Tests for ``UCSM_XML_API_Error``."""
233+
234+ def test_includes_code_and_msg(self):
235+ def raise_error():
236+ raise UCSM_XML_API_Error('bad', 4224)
237+
238+ error = self.assertRaises(UCSM_XML_API_Error, raise_error)
239+
240+ self.assertEqual('bad', error.args[0])
241+ self.assertEqual(4224, error.code)
242+
243+
244+class TestMakeRequestData(MAASTestCase):
245+ """Tests for ``make_request_data``."""
246+
247+ def test_no_children(self):
248+ fields = {'hello': 'there'}
249+ request_data = make_request_data('foo', fields)
250+ root = XML(request_data)
251+ self.assertEqual('foo', root.tag)
252+ self.assertEqual('there', root.get('hello'))
253+
254+ def test_with_children(self):
255+ fields = {'hello': 'there'}
256+ children_tags = ['bar', 'baz']
257+ children = [Element(child_tag) for child_tag in children_tags]
258+ request_data = make_request_data('foo', fields, children)
259+ root = XML(request_data)
260+ self.assertEqual('foo', root.tag)
261+ self.assertItemsEqual(children_tags, (e.tag for e in root))
262+
263+ def test_no_fields(self):
264+ request_data = make_request_data('foo')
265+ root = XML(request_data)
266+ self.assertEqual('foo', root.tag)
267+
268+
269+class TestParseResonse(MAASTestCase):
270+ """Tests for ``parse_response``."""
271+
272+ def test_no_error(self):
273+ xml = '<foo/>'
274+ response = parse_response(xml)
275+ self.assertEqual('foo', response.tag)
276+
277+ def test_error(self):
278+ xml = '<foo errorCode="123" errorDescr="mayday!"/>'
279+ self.assertRaises(UCSM_XML_API_Error, parse_response, xml)
280+
281+
282+class TestLogin(MAASTestCase):
283+ """"Tests for ``UCSM_XML_API.login``."""
284+
285+ def test_login_assigns_cookie(self):
286+ cookie = 'chocolate chip'
287+ api, mock = make_api_patch_call(self)
288+ mock.return_value = Element('aaaLogin', {'outCookie': cookie})
289+ api.login()
290+ self.assertEqual(cookie, api.cookie)
291+
292+ def test_login_call_parameters(self):
293+ user = 'user'
294+ password = 'pass'
295+ api, mock = make_api_patch_call(self, user=user, password=password)
296+ api.login()
297+ fields = {'inName': user, 'inPassword': password}
298+ self.assertThat(mock, MockCalledOnceWith('aaaLogin', fields))
299+
300+
301+class TestLogout(MAASTestCase):
302+ """"Tests for ``UCSM_XML_API.logout``."""
303+
304+ def test_logout_clears_cookie(self):
305+ api = make_api()
306+ self.patch(api, '_call')
307+ api.logout()
308+ self.assertIsNone(api.cookie)
309+
310+ def test_logout_uses_cookie(self):
311+ api, mock = make_api_patch_call(self)
312+ cookie = api.cookie
313+ api.logout()
314+ fields = {'inCookie': cookie}
315+ self.assertThat(mock, MockCalledOnceWith('aaaLogout', fields))
316+
317+
318+class TestConfigResolveClass(MAASTestCase):
319+ """"Tests for ``UCSM_XML_API.config_resolve_class``."""
320+
321+ def test_no_filters(self):
322+ class_id = make_class()
323+ api, mock = make_api_patch_call(self)
324+ api.config_resolve_class(class_id)
325+ fields = {'cookie': api.cookie, 'classId': class_id}
326+ self.assertThat(mock, MockCalledOnceWith('configResolveClass', fields,
327+ ANY))
328+
329+ def test_with_filters(self):
330+ class_id = make_class()
331+ filter_element = Element('hi')
332+ api, mock = make_api_patch_call(self)
333+ api.config_resolve_class(class_id, [filter_element])
334+ in_filters = mock.call_args[0][2]
335+ self.assertEqual([filter_element], in_filters[0][:])
336+
337+ def test_return_response(self):
338+ api, mock = make_api_patch_call(self)
339+ mock.return_value = Element('test')
340+ result = api.config_resolve_class('c')
341+ self.assertEqual(mock.return_value, result)
342+
343+
344+class TestConfigResolveChildren(MAASTestCase):
345+ """"Tests for ``UCSM_XML_API.config_resolve_children``."""
346+
347+ def test_parameters(self):
348+ dn = make_dn()
349+ class_id = make_class()
350+ api, mock = make_api_patch_call(self)
351+ api.config_resolve_children(dn, class_id)
352+ fields = {'inDn': dn, 'classId': class_id, 'cookie': api.cookie}
353+ self.assertThat(mock,
354+ MockCalledOnceWith('configResolveChildren', fields))
355+
356+ def test_no_class_id(self):
357+ dn = make_dn()
358+ api, mock = make_api_patch_call(self)
359+ api.config_resolve_children(dn)
360+ fields = {'inDn': dn, 'cookie': api.cookie}
361+ self.assertThat(mock,
362+ MockCalledOnceWith('configResolveChildren', fields))
363+
364+ def test_return_response(self):
365+ api, mock = make_api_patch_call(self)
366+ mock.return_value = Element('test')
367+ result = api.config_resolve_children('d', 'c')
368+ self.assertEqual(mock.return_value, result)
369+
370+
371+class TestConfigConfMo(MAASTestCase):
372+ """"Tests for ``UCSM_XML_API.config_conf_mo``."""
373+
374+ def test_parameters(self):
375+ dn = make_dn()
376+ config_items = [Element('hi')]
377+ api, mock = make_api_patch_call(self)
378+ api.config_conf_mo(dn, config_items)
379+ fields = {'dn': dn, 'cookie': api.cookie}
380+ self.assertThat(mock, MockCalledOnceWith('configConfMo', fields, ANY))
381+ in_configs = mock.call_args[0][2]
382+ self.assertEqual(config_items, in_configs[0][:])
383+
384+
385+class TestCall(MAASTestCase):
386+ """"Tests for ``UCSM_XML_API._call``."""
387+
388+ def test_call(self):
389+ name = 'method'
390+ fields = {1: 2}
391+ children = [3, 4]
392+ request = '<yes/>'
393+ response = Element('good')
394+ api = make_api()
395+
396+ mock_make_request_data = self.patch(ucsm, 'make_request_data')
397+ mock_make_request_data.return_value = request
398+
399+ mock_send_request = self.patch(api, '_send_request')
400+ mock_send_request.return_value = response
401+
402+ api._call(name, fields, children)
403+ self.assertThat(mock_make_request_data,
404+ MockCalledOnceWith(name, fields, children))
405+ self.assertThat(mock_send_request, MockCalledOnceWith(request))
406+
407+
408+class TestSendRequest(MAASTestCase):
409+ """"Tests for ``UCSM_XML_API._send_request``."""
410+
411+ def test_send_request(self):
412+ request_data = 'foo'
413+ api = make_api()
414+ self.patch(api, '_call')
415+ stream = StringIO('<hi/>')
416+ mock = self.patch(urllib2, 'urlopen')
417+ mock.return_value = stream
418+ response = api._send_request(request_data)
419+ self.assertEqual('hi', response.tag)
420+ urllib_request = mock.call_args[0][0]
421+ self.assertEqual(request_data, urllib_request.data)
422+
423+
424+class TestConfigResolveDn(MAASTestCase):
425+ """Tests for ``UCSM_XML_API.config_resolve_dn``."""
426+
427+ def test_parameters(self):
428+ api, mock = make_api_patch_call(self)
429+ test_dn = make_dn()
430+ fields = {'cookie': api.cookie, 'dn': test_dn}
431+ api.config_resolve_dn(test_dn)
432+ self.assertThat(mock,
433+ MockCalledOnceWith('configResolveDn', fields))
434+
435+
436+class TestGetServers(MAASTestCase):
437+ """Tests for ``get_servers``."""
438+
439+ def test_uses_uuid(self):
440+ uuid = factory.getRandomUUID()
441+ api = make_api()
442+ mock = self.patch(api, 'config_resolve_class')
443+ get_servers(api, uuid)
444+ filters = mock.call_args[0][1]
445+ attrib = {'class': 'computeItem', 'property': 'uuid', 'value': uuid}
446+ self.assertEqual(attrib, filters[0].attrib)
447+
448+ def test_returns_result(self):
449+ uuid = factory.getRandomUUID()
450+ api = make_api()
451+ fake_result = make_fake_result('configResolveClass', 'found')
452+ self.patch(api, 'config_resolve_class').return_value = fake_result
453+ result = get_servers(api, uuid)
454+ self.assertEqual('found', result[0].tag)
455+
456+ def test_class_id(self):
457+ uuid = factory.getRandomUUID()
458+ api = make_api()
459+ mock = self.patch(api, 'config_resolve_class')
460+ get_servers(api, uuid)
461+ self.assertThat(mock, MockCalledOnceWith('computeItem', ANY))
462+
463+
464+class TestGetChildren(MAASTestCase):
465+ """Tests for ``get_children``."""
466+
467+ def test_returns_result(self):
468+ search_class = make_class()
469+ api = make_api()
470+ fake_result = make_fake_result('configResolveChildren', search_class)
471+ self.patch(api, 'config_resolve_children').return_value = fake_result
472+ in_element = Element('test', {'dn': make_dn()})
473+ class_id = search_class
474+ result = get_children(api, in_element, class_id)
475+ self.assertEqual(search_class, result[0].tag)
476+
477+ def test_parameters(self):
478+ search_class = make_class()
479+ parent_dn = make_dn()
480+ api = make_api()
481+ mock = self.patch(api, 'config_resolve_children')
482+ in_element = Element('test', {'dn': parent_dn})
483+ class_id = search_class
484+ get_children(api, in_element, class_id)
485+ self.assertThat(mock, MockCalledOnceWith(parent_dn, search_class))
486+
487+
488+class TestGetMacs(MAASTestCase):
489+ """Tests for ``get_macs``."""
490+
491+ def test_gets_adaptors(self):
492+ adaptor = 'adaptor'
493+ server = make_server()
494+ mac = 'xx'
495+ api = make_api()
496+ mock = self.patch(ucsm, 'get_children')
497+
498+ def fake_get_children(api, element, class_id):
499+ if class_id == 'adaptorUnit':
500+ return [adaptor]
501+ elif class_id == 'adaptorHostEthIf':
502+ return [Element('ethif', {'mac': mac})]
503+
504+ mock.side_effect = fake_get_children
505+ macs = get_macs(api, server)
506+ self.assertThat(mock, MockCallsMatch(
507+ call(api, server, 'adaptorUnit'),
508+ call(api, adaptor, 'adaptorHostEthIf')))
509+ self.assertEqual([mac], macs)
510+
511+
512+class TestProbeServers(MAASTestCase):
513+ """Tests for ``probe_servers``."""
514+
515+ def test_uses_api(self):
516+ api = make_api()
517+ mock = self.patch(ucsm, 'get_servers')
518+ probe_servers(api)
519+ self.assertThat(mock, MockCalledOnceWith(api))
520+
521+ def test_returns_results(self):
522+ servers = [{'uuid': factory.getRandomUUID()}]
523+ mac = 'mac'
524+ api = make_api()
525+ self.patch(ucsm, 'get_servers').return_value = servers
526+ self.patch(ucsm, 'get_macs').return_value = [mac]
527+ server_list = probe_servers(api)
528+ self.assertEqual([(servers[0], [mac])], server_list)
529+
530+
531+class TestGetServerPowerControl(MAASTestCase):
532+ """Tests for ``get_server_power_control``."""
533+
534+ def test_get_server_power_control(self):
535+ api = make_api()
536+ mock = self.patch(api, 'config_resolve_children')
537+ fake_result = make_fake_result('configResolveChildren', 'lsPower')
538+ mock.return_value = fake_result
539+ dn = make_dn()
540+ server = Element('computeItem', {'assignedToDn': dn})
541+ power_control = get_server_power_control(api, server)
542+ self.assertThat(mock, MockCalledOnceWith(dn, 'lsPower'))
543+ self.assertEqual('lsPower', power_control.tag)
544+
545+
546+class TestSetServerPowerControl(MAASTestCase):
547+ """Tests for ``set_server_power_control``."""
548+
549+ def test_set_server_power_control(self):
550+ api = make_api()
551+ power_dn = make_dn()
552+ power_control = Element('lsPower', {'dn': power_dn})
553+ config_conf_mo_mock = self.patch(api, 'config_conf_mo')
554+ state = 'state'
555+ set_server_power_control(api, power_control, state)
556+ self.assertThat(config_conf_mo_mock, MockCalledOnceWith(power_dn, ANY))
557+ power_change = config_conf_mo_mock.call_args[0][1][0]
558+ self.assertEqual(power_change.tag, 'lsPower')
559+ self.assertEqual({'state': state, 'dn': power_dn}, power_change.attrib)
560+
561+
562+class TestLoggedIn(MAASTestCase):
563+ """Tests for ``logged_in``."""
564+
565+ def test_logged_in(self):
566+ mock = self.patch(ucsm, 'UCSM_XML_API')
567+ url = 'url'
568+ username = 'username'
569+ password = 'password'
570+ mock.return_value = Mock()
571+
572+ with logged_in(url, username, password) as api:
573+ self.assertEqual(mock.return_value, api)
574+ self.assertThat(api.login, MockCalledOnceWith())
575+
576+ self.assertThat(mock.return_value.logout, MockCalledOnceWith())
577+
578+
579+class TestValidGetPowerCommand(MAASTestCase):
580+ scenarios = [
581+ ('Power On', dict(
582+ power_mode='on', current_state='down', command='admin-up')),
583+ ('Power On', dict(
584+ power_mode='on', current_state='up', command='cycle-immediate')),
585+ ('Power Off', dict(
586+ power_mode='off', current_state='up', command='admin-down')),
587+ ]
588+
589+ def test_get_power_command(self):
590+ command = get_power_command(self.power_mode, self.current_state)
591+ self.assertEqual(self.command, command)
592+
593+
594+class TestInvalidGetPowerCommand(MAASTestCase):
595+
596+ def test_get_power_command_raises_assertion_error_on_bad_power_mode(self):
597+ bad_power_mode = factory.make_name('unlikely')
598+ error = self.assertRaises(AssertionError, get_power_command,
599+ bad_power_mode, None)
600+ self.assertIn(bad_power_mode, error.args[0])
601+
602+
603+class TestPowerControlUCSM(MAASTestCase):
604+ """Tests for ``power_control_ucsm``."""
605+
606+ def test_power_control_ucsm(self):
607+ uuid = factory.getRandomUUID()
608+ api = Mock()
609+ self.patch(ucsm, 'UCSM_XML_API').return_value = api
610+ get_servers_mock = self.patch(ucsm, 'get_servers')
611+ server = make_server()
612+ state = 'admin-down'
613+ power_control = Element('lsPower', {'state': state})
614+ get_servers_mock.return_value = [server]
615+ get_server_power_control_mock = self.patch(ucsm,
616+ 'get_server_power_control')
617+ get_server_power_control_mock.return_value = power_control
618+ set_server_power_control_mock = self.patch(ucsm,
619+ 'set_server_power_control')
620+ power_control_ucsm('url', 'username', 'password', uuid,
621+ 'off')
622+ self.assertThat(get_servers_mock, MockCalledOnceWith(api, uuid))
623+ self.assertThat(set_server_power_control_mock,
624+ MockCalledOnceWith(api, power_control, state))
625+
626+
627+class TestProbeAndEnlistUCSM(MAASTestCase):
628+ """Tests for ``probe_and_enlist_ucsm``."""
629+
630+ def test_probe_and_enlist(self):
631+ url = 'url'
632+ username = 'username'
633+ password = 'password'
634+ api = Mock()
635+ self.patch(ucsm, 'UCSM_XML_API').return_value = api
636+ server_element = {'uuid': 'uuid'}
637+ server = (server_element, ['mac'],)
638+ probe_servers_mock = self.patch(ucsm, 'probe_servers')
639+ probe_servers_mock.return_value = [server]
640+ set_lan_boot_default_mock = self.patch(ucsm, 'set_lan_boot_default')
641+ create_node_mock = self.patch(utils, 'create_node')
642+ probe_and_enlist_ucsm(url, username, password)
643+ self.assertThat(set_lan_boot_default_mock,
644+ MockCalledOnceWith(api, server_element))
645+ self.assertThat(probe_servers_mock, MockCalledOnceWith(api))
646+ params = {
647+ 'power_address': url,
648+ 'power_user': username,
649+ 'power_pass': password,
650+ 'uuid': server[0]['uuid']
651+ }
652+ self.assertThat(create_node_mock,
653+ MockCalledOnceWith(server[1], 'amd64', 'ucsm', params))
654+
655+
656+class TestGetServiceProfile(MAASTestCase):
657+ """Tests for ``get_service_profile.``"""
658+
659+ def test_get_service_profile(self):
660+ test_dn = make_dn()
661+ server = Element('computeBlade', {'assignedToDn': test_dn})
662+ api = make_api()
663+ mock = self.patch(api, 'config_resolve_dn')
664+ mock.return_value = make_fake_result('configResolveDn', 'lsServer',
665+ 'outConfig')
666+ service_profile = get_service_profile(api, server)
667+ self.assertThat(mock, MockCalledOnceWith(test_dn))
668+ self.assertEqual(mock.return_value[0], service_profile)
669+
670+
671+def make_boot_order_scenarios(size):
672+ minimum = random.randint(0, 500)
673+ ordinals = xrange(minimum, minimum + size)
674+
675+ elements = [
676+ Element('Entry%d' % i, {'order': '%d' % i})
677+ for i in ordinals
678+ ]
679+
680+ orders = permutations(elements)
681+ orders = [{'order': order} for order in orders]
682+
683+ scenarios = [('%d' % i, order) for i, order in enumerate(orders)]
684+ return scenarios, minimum
685+
686+
687+class TestGetFirstBooter(MAASTestCase):
688+ """Tests for ``get_first_booter.``"""
689+
690+ scenarios, minimum = make_boot_order_scenarios(3)
691+
692+ def test_first_booter(self):
693+ root = Element('outConfigs')
694+ root.extend(self.order)
695+ picked = get_first_booter(root)
696+ self.assertEqual(picked.tag, 'Entry%d' % self.minimum)
697+
698+
699+class TestsForStripRoKeys(MAASTestCase):
700+ """Tests for ``strip_ro_keys.``"""
701+
702+ def test_strip_ro_keys(self):
703+ attributes = {key: 'DC' for key in RO_KEYS}
704+
705+ elements = [
706+ Element('Element%d' % i, attributes)
707+ for i in xrange(random.randint(0, 10))
708+ ]
709+
710+ strip_ro_keys(elements)
711+
712+ for key in RO_KEYS:
713+ values = [element.get(key) for element in elements]
714+ for value in values:
715+ self.assertIsNone(value)
716+
717+
718+class TestMakePolicyChange(MAASTestCase):
719+ """Tests for ``make_policy_change``."""
720+
721+ def test_lan_already_top_priority(self):
722+ boot_profile_response = make_fake_result('configResolveChildren',
723+ 'lsbootLan')
724+ mock = self.patch(ucsm, 'get_first_booter')
725+ mock.return_value = boot_profile_response[0]
726+ change = make_policy_change(boot_profile_response)
727+ self.assertIsNone(change)
728+ self.assertThat(mock, MockCalledOnceWith(boot_profile_response))
729+
730+ def test_change_lan_to_top_priority(self):
731+ boot_profile_response = Element('outConfigs')
732+ lan_boot = Element('lsbootLan', {'order': 'second'})
733+ storage_boot = Element('lsbootStorage', {'order': 'first'})
734+ boot_profile_response.extend([lan_boot, storage_boot])
735+ self.patch(ucsm, 'get_first_booter').return_value = storage_boot
736+ self.patch(ucsm, 'strip_ro_keys')
737+ change = make_policy_change(boot_profile_response)
738+ lan_boot_order = change.xpath('//lsbootPolicy/lsbootLan/@order')
739+ storage_boot_order = \
740+ change.xpath('//lsbootPolicy/lsbootStorage/@order')
741+ self.assertEqual(['first'], lan_boot_order)
742+ self.assertEqual(['second'], storage_boot_order)
743+
744+
745+class TestSetLanBootDefault(MAASTestCase):
746+ """Tets for ``set_lan_boot_default.``"""
747+
748+ def test_no_change(self):
749+ api = make_api()
750+ server = make_server()
751+ self.patch(ucsm, 'get_service_profile')
752+ self.patch(api, 'config_resolve_children')
753+ self.patch(ucsm, 'make_policy_change').return_value = None
754+ config_conf_mo = self.patch(api, 'config_conf_mo')
755+ set_lan_boot_default(api, server)
756+ self.assertThat(config_conf_mo, MockNotCalled())
757+
758+ def test_with_change(self):
759+ api = make_api()
760+ server = make_server()
761+ test_dn = make_dn()
762+ test_change = 'change'
763+ service_profile = Element('test', {'operBootPolicyName': test_dn})
764+ self.patch(ucsm, 'get_service_profile').return_value = service_profile
765+ self.patch(api, 'config_resolve_children')
766+ self.patch(ucsm, 'make_policy_change').return_value = test_change
767+ config_conf_mo = self.patch(api, 'config_conf_mo')
768+ set_lan_boot_default(api, server)
769+ self.assertThat(config_conf_mo,
770+ MockCalledOnceWith(test_dn, [test_change]))
771
772=== added file 'src/provisioningserver/custom_hardware/ucsm.py'
773--- src/provisioningserver/custom_hardware/ucsm.py 1970-01-01 00:00:00 +0000
774+++ src/provisioningserver/custom_hardware/ucsm.py 2014-05-06 13:49:30 +0000
775@@ -0,0 +1,435 @@
776+# Copyright 2014 Canonical Ltd. This software is licensed under the
777+# GNU Affero General Public License version 3 (see the file LICENSE).
778+
779+"""Support for managing nodes via Cisco UCS Manager's HTTP-XML API.
780+
781+It's useful to have a cursory understanding of how UCS Manager XML API
782+works. Cisco has a proprietary document that describes all of this in
783+more detail, and I would suggest you get a copy of that if you want more
784+information than is provided here.
785+
786+The Cisco DevNet website for UCS Manager has a link to the document,
787+which is behind a login wall, and links to example UCS queries:
788+
789+https://developer.cisco.com/web/unifiedcomputing/home
790+
791+UCS Manager is a tool for managing servers. It provides an XML API for
792+external applications to use to interact with UCS Manager to manage
793+servers. The API is available via HTTP, and requests and responses are
794+made of XML strings. MAAS's code for interacting with a UCS Manager is
795+concerned with building these requests, sending them to UCS Manager, and
796+processing the responses.
797+
798+UCS Manager stores information in a hierarchical structure known as the
799+management information tree. This structure is exposed via the XML API,
800+where we can manipulate objects in the tree by finding them, reading
801+them, and writing them.
802+
803+Some definitions for terms that are used in this code:
804+
805+Boot Policy - Controls the boot order for a server. Each service profile
806+is associated with a boot policy.
807+
808+Distinguished Name (DN) - Each object in UCS has a unique DN, which
809+describes its position in the tree. This is like a fully qualified path,
810+and provides a way for objects to reference other objects at other
811+places in the tree, or for API users to look up specific objects in the
812+tree.
813+
814+Class - Classes define the properties and states of objects. An object's
815+class is given in its tag name.
816+
817+Managed Object (MO) - An object in the management information tree.
818+Objects are recursive, and may have children of multiple types. With the
819+exception of the root object, all objects have parents. In the XML API,
820+objects are represented as XML elements.
821+
822+Method - Actions performed by the API on managed objects. These can
823+change state, or read the current state, or both.
824+
825+Server - A physical server managed by UCS Manager. Servers must be
826+associated with service profiles in order to be used.
827+
828+Service Profile - A set of configuration options for a server. Service
829+profiles define the server's personality, and can be migrated from
830+server to server. Service profiles describe boot policy, MAC addresses,
831+network connectivity, IPMI configuration, and more. MAAS requires
832+servers to be associated with service profiles.
833+
834+UUID - The UUID for a server. MAAS persists the UUID of each UCS managed
835+server it enlists, and uses it as a key for looking the server up later.
836+"""
837+
838+from __future__ import (
839+ absolute_import,
840+ print_function,
841+ unicode_literals,
842+ )
843+
844+import contextlib
845+import urllib2
846+import urlparse
847+
848+from lxml.etree import (
849+ Element,
850+ tostring,
851+ XML,
852+ )
853+import provisioningserver.custom_hardware.utils as utils
854+
855+
856+str = None
857+
858+__metaclass__ = type
859+__all__ = [
860+ 'power_control_ucsm',
861+ 'probe_and_enlist_ucsm',
862+]
863+
864+
865+class UCSM_XML_API_Error(Exception):
866+ """Failure talking to a Cisco UCS Manager."""
867+
868+ def __init__(self, msg, code):
869+ super(UCSM_XML_API_Error, self).__init__(msg)
870+ self.code = code
871+
872+
873+def make_request_data(name, fields=None, children=None):
874+ """Build a request string for an API method."""
875+ root = Element(name, fields)
876+ if children is not None:
877+ root.extend(children)
878+ return tostring(root)
879+
880+
881+def parse_response(response_string):
882+ """Parse the response from an API method."""
883+ doc = XML(response_string)
884+
885+ error_code = doc.get('errorCode')
886+ if error_code is not None:
887+ raise UCSM_XML_API_Error(doc.get('errorDescr'), error_code)
888+
889+ return doc
890+
891+
892+class UCSM_XML_API(object):
893+ """Provides access to a Cisco UCS Manager's XML API. Public methods
894+ on this class correspond to UCS Manager XML API methods.
895+
896+ Each request uses a new connection. The server supports keep-alive,
897+ so this client could be optimized to use it too.
898+ """
899+
900+ def __init__(self, url, username, password):
901+ self.url = url
902+ self.api_url = urlparse.urljoin(self.url, 'nuova')
903+ self.username = username
904+ self.password = password
905+ self.cookie = None
906+
907+ def _send_request(self, request_data):
908+ """Issue a request via HTTP and parse the response."""
909+ request = urllib2.Request(self.api_url, request_data)
910+ response = urllib2.urlopen(request)
911+ response_text = response.read()
912+ response_doc = parse_response(response_text)
913+ return response_doc
914+
915+ def _call(self, name, fields=None, children=None):
916+ request_data = make_request_data(name, fields, children)
917+ response = self._send_request(request_data)
918+ return response
919+
920+ def login(self):
921+ """Login to the API and get a cookie.
922+
923+ Logging into the API gives a new cookie in response. The cookie
924+ will become inactive after it has been inactive for some amount
925+ of time (10 minutes is the default.)
926+
927+ UCS Manager allows a limited number of active cookies at any
928+ point in time, so it's important to free the cookie up when
929+ finished by logging out via the ``logout`` method.
930+ """
931+ fields = {'inName': self.username, 'inPassword': self.password}
932+ response = self._call('aaaLogin', fields)
933+ self.cookie = response.get('outCookie')
934+
935+ def logout(self):
936+ """Logout from the API and free the cookie."""
937+ fields = {'inCookie': self.cookie}
938+ self._call('aaaLogout', fields)
939+ self.cookie = None
940+
941+ def config_resolve_class(self, class_id, filters=None):
942+ """Issue a configResolveClass request.
943+
944+ This returns all of the objects of class ``class_id`` from the
945+ UCS Manager.
946+
947+ Filters provide a way of limiting the classes returned according
948+ to their attributes. There are a number of filters available -
949+ Cisco's XML API documentation has a full chapter on filters.
950+ All we care about here is that filters are described with XML
951+ elements.
952+ """
953+ fields = {'cookie': self.cookie, 'classId': class_id}
954+
955+ in_filters = Element('inFilter')
956+ if filters:
957+ in_filters.extend(filters)
958+
959+ return self._call('configResolveClass', fields, [in_filters])
960+
961+ def config_resolve_children(self, dn, class_id=None):
962+ """Issue a configResolveChildren request.
963+
964+ This returns all of the children of the object named by ``dn``,
965+ or if ``class_id`` is not None, all of the children of type
966+ ``class_id``.
967+ """
968+ fields = {'cookie': self.cookie, 'inDn': dn}
969+ if class_id is not None:
970+ fields['classId'] = class_id
971+ return self._call('configResolveChildren', fields)
972+
973+ def config_resolve_dn(self, dn):
974+ """Retrieve a single object by name.
975+
976+ This returns the object named by ``dn``, but not its children.
977+ """
978+ fields = {'cookie': self.cookie, 'dn': dn}
979+ return self._call('configResolveDn', fields)
980+
981+ def config_conf_mo(self, dn, config_items):
982+ """Issue a configConfMo request.
983+
984+ This makes a configuration change on an object (MO).
985+ """
986+ fields = {'cookie': self.cookie, 'dn': dn}
987+
988+ in_configs = Element('inConfig')
989+ in_configs.extend(config_items)
990+
991+ self._call('configConfMo', fields, [in_configs])
992+
993+
994+def get_servers(api, uuid=None):
995+ """Retrieve a list of servers from the UCS Manager."""
996+ if uuid:
997+ attrs = {'class': 'computeItem', 'property': 'uuid', 'value': uuid}
998+ filters = [Element('eq', attrs)]
999+ else:
1000+ filters = None
1001+
1002+ resolved = api.config_resolve_class('computeItem', filters)
1003+ return resolved.xpath('//outConfigs/*')
1004+
1005+
1006+def get_children(api, element, class_id):
1007+ """Retrieve a list of child elements from the UCS Manager."""
1008+ resolved = api.config_resolve_children(element.get('dn'), class_id)
1009+ return resolved.xpath('//outConfigs/%s' % class_id)
1010+
1011+
1012+def get_macs(api, server):
1013+ """Retrieve the list of MAC addresses assigned to a server.
1014+
1015+ Network interfaces are represented by 'adaptorUnit' objects, and
1016+ are stored as children of servers.
1017+ """
1018+ adaptors = get_children(api, server, 'adaptorUnit')
1019+
1020+ macs = []
1021+ for adaptor in adaptors:
1022+ host_eth_ifs = get_children(api, adaptor, 'adaptorHostEthIf')
1023+ macs.extend([h.get('mac') for h in host_eth_ifs])
1024+
1025+ return macs
1026+
1027+
1028+def probe_servers(api):
1029+ """Retrieve the UUID and MAC addresses for servers from the UCS Manager."""
1030+ servers = get_servers(api)
1031+ server_list = [(s, get_macs(api, s)) for s in servers]
1032+ return server_list
1033+
1034+
1035+def get_server_power_control(api, server):
1036+ """Retrieve the power control object for a server."""
1037+ service_profile_dn = server.get('assignedToDn')
1038+ resolved = api.config_resolve_children(service_profile_dn, 'lsPower')
1039+ power_controls = resolved.xpath('//outConfigs/lsPower')
1040+ return power_controls[0]
1041+
1042+
1043+def set_server_power_control(api, power_control, command):
1044+ """Issue a power command to a server's power control."""
1045+ attrs = {'state': command, 'dn': power_control.get('dn')}
1046+ power_change = Element('lsPower', attrs)
1047+ api.config_conf_mo(power_control.get('dn'), [power_change])
1048+
1049+
1050+def get_service_profile(api, server):
1051+ """Get the server's assigned service profile."""
1052+ service_profile_dn = server.get('assignedToDn')
1053+ result = api.config_resolve_dn(service_profile_dn)
1054+ service_profile = result.xpath('//outConfig/lsServer')[0]
1055+ return service_profile
1056+
1057+
1058+def get_first_booter(boot_profile_response):
1059+ """Find the device currently set to boot by default."""
1060+ ordinals = boot_profile_response.xpath('//outConfigs/*/@order')
1061+ top_boot_order = min(ordinals)
1062+ first_query = '//outConfigs/*[@order=%s]' % top_boot_order
1063+ current_first = boot_profile_response.xpath(first_query)[0]
1064+ return current_first
1065+
1066+
1067+RO_KEYS = ['access', 'type']
1068+
1069+
1070+def strip_ro_keys(elements):
1071+ """Remove read-only keys from configuration elements.
1072+
1073+ These are keys for attributes that aren't allowed to be changed via
1074+ configConfMo request. They are included in MO's that we read from the
1075+ API; stripping these attributes lets us reuse the elements for those
1076+ MO's rather than building new ones from scratch.
1077+ """
1078+ for ro_key in RO_KEYS:
1079+ for element in elements:
1080+ del(element.attrib[ro_key])
1081+
1082+
1083+def make_policy_change(boot_profile_response):
1084+ """Build the policy change tree required to make LAN boot first
1085+ priority.
1086+
1087+ The original top priority will be swapped with LAN boot's original
1088+ priority.
1089+ """
1090+ current_first = get_first_booter(boot_profile_response)
1091+ lan_boot = boot_profile_response.xpath('//outConfigs/lsbootLan')[0]
1092+
1093+ if current_first == lan_boot:
1094+ return
1095+
1096+ top_boot_order = current_first.get('order')
1097+ current_first.set('order', lan_boot.get('order'))
1098+ lan_boot.set('order', top_boot_order)
1099+
1100+ elements = [current_first, lan_boot]
1101+ strip_ro_keys(elements)
1102+ policy_change = Element('lsbootPolicy')
1103+ policy_change.extend(elements)
1104+ return policy_change
1105+
1106+
1107+def set_lan_boot_default(api, server):
1108+ """Set a server to boot via LAN by default.
1109+
1110+ If LAN boot is already the top priority, no change will
1111+ be made.
1112+
1113+ This command changes the server's boot profile, which will affect
1114+ any other servers also using that boot profile. This is ok, because
1115+ probe and enlist enlists all the servers in the chassis.
1116+ """
1117+ service_profile = get_service_profile(api, server)
1118+ boot_profile_dn = service_profile.get('operBootPolicyName')
1119+ response = api.config_resolve_children(boot_profile_dn)
1120+ policy_change = make_policy_change(response)
1121+ if policy_change is None:
1122+ return
1123+ api.config_conf_mo(boot_profile_dn, [policy_change])
1124+
1125+
1126+@contextlib.contextmanager
1127+def logged_in(url, username, password):
1128+ """Context manager that ensures the logout from the API occurs."""
1129+ api = UCSM_XML_API(url, username, password)
1130+ api.login()
1131+ try:
1132+ yield api
1133+ finally:
1134+ api.logout()
1135+
1136+
1137+def get_power_command(maas_power_mode, current_state):
1138+ """Translate a MAAS on/off state into a UCSM power command.
1139+
1140+ If the node is up already and receives a request to power on, power
1141+ cycle the node.
1142+ """
1143+ if maas_power_mode == 'on':
1144+ if current_state == 'up':
1145+ return 'cycle-immediate'
1146+ return 'admin-up'
1147+ elif maas_power_mode == 'off':
1148+ return 'admin-down'
1149+ else:
1150+ message = 'Unexpected maas power mode: %s' % (maas_power_mode)
1151+ raise AssertionError(message)
1152+
1153+
1154+def power_control_ucsm(url, username, password, uuid, maas_power_mode):
1155+ """Handle calls from the power template for nodes with a power type
1156+ of 'ucsm'.
1157+ """
1158+ with logged_in(url, username, password) as api:
1159+ # UUIDs are unique per server, so we get either one or zero
1160+ # servers for a given UUID.
1161+ [server] = get_servers(api, uuid)
1162+ power_control = get_server_power_control(api, server)
1163+ command = get_power_command(maas_power_mode,
1164+ power_control.get('state'))
1165+ set_server_power_control(api, power_control, command)
1166+
1167+
1168+def probe_and_enlist_ucsm(url, username, password):
1169+ """Probe a UCS Manager and enlist all its servers.
1170+
1171+ Here's what happens here: 1. Get a list of servers from the UCS
1172+ Manager, along with their MAC addresses.
1173+
1174+ 2. Configure each server to boot from LAN first.
1175+
1176+ 3. Add each server to MAAS as a new node, with a power control
1177+ method of 'ucsm'. The URL and credentials supplied are persisted
1178+ with each node so MAAS knows how to access UCSM to manage the node
1179+ in the future.
1180+
1181+ This code expects each server in the system to have already been
1182+ associated with a service profile. The servers must have networking
1183+ configured, and their boot profiles must include a boot from LAN
1184+ option. During enlistment, the boot profile for each service profile
1185+ used by a server will be modified to move LAN boot to the highest
1186+ priority boot option.
1187+
1188+ Also, if any node fails to enlist, this enlistment process will
1189+ stop and won't attempt to enlist any additional nodes. If a node is
1190+ already known to MAAS, it will fail to enlist, so all nodes must be
1191+ added at once.
1192+
1193+ There is also room for optimization during enlistment. While our
1194+ client deals with a single server at a time, the API is capable
1195+ of reading/writing the settings of multiple servers in the same
1196+ request.
1197+ """
1198+ with logged_in(url, username, password) as api:
1199+ servers = probe_servers(api)
1200+ for server, _ in servers:
1201+ set_lan_boot_default(api, server)
1202+
1203+ for server, macs in servers:
1204+ params = {
1205+ 'power_address': url,
1206+ 'power_user': username,
1207+ 'power_pass': password,
1208+ 'uuid': server.get('uuid'),
1209+ }
1210+ utils.create_node(macs, 'amd64', 'ucsm', params)
1211
1212=== modified file 'src/provisioningserver/power/tests/test_poweraction.py'
1213--- src/provisioningserver/power/tests/test_poweraction.py 2014-02-27 02:33:54 +0000
1214+++ src/provisioningserver/power/tests/test_poweraction.py 2014-05-06 13:49:30 +0000
1215@@ -223,3 +223,14 @@
1216 power_pass='me', power_hwaddress='me', ipmitool='echo')
1217 output = action.run_shell(script)
1218 self.assertIn("Got unknown power state from ipmipower", output)
1219+
1220+ def test_ucsm_renders_template(self):
1221+ # I'd like to assert that escape_py_literal is being used here,
1222+ # but it's not obvious how to mock things in the template
1223+ # rendering namespace so I passed on that.
1224+ action = PowerAction('ucsm')
1225+ script = action.render_template(
1226+ action.get_template(), power_address='foo',
1227+ power_user='bar', power_pass='baz',
1228+ uuid=factory.getRandomUUID(), power_change='on')
1229+ self.assertIn('power_control_ucsm', script)
1230
1231=== modified file 'src/provisioningserver/power_schema.py'
1232--- src/provisioningserver/power_schema.py 2014-04-23 14:21:03 +0000
1233+++ src/provisioningserver/power_schema.py 2014-05-06 13:49:30 +0000
1234@@ -242,4 +242,14 @@
1235 make_json_field('power_pass', "Power password"),
1236 ],
1237 },
1238+ {
1239+ 'name': 'ucsm',
1240+ 'description': "Cisco UCS Manager",
1241+ 'fields': [
1242+ make_json_field('uuid', "Server UUID"),
1243+ make_json_field('power_address', "URL for XML API"),
1244+ make_json_field('power_user', "API user"),
1245+ make_json_field('power_pass', "API password"),
1246+ ],
1247+ },
1248 ]
1249
1250=== modified file 'src/provisioningserver/tasks.py'
1251--- src/provisioningserver/tasks.py 2014-04-22 15:10:32 +0000
1252+++ src/provisioningserver/tasks.py 2014-05-06 13:49:30 +0000
1253@@ -45,6 +45,7 @@
1254 from provisioningserver.custom_hardware.seamicro import (
1255 probe_seamicro15k_and_enlist,
1256 )
1257+from provisioningserver.custom_hardware.ucsm import probe_and_enlist_ucsm
1258 from provisioningserver.dhcp import (
1259 config,
1260 detect,
1261@@ -477,3 +478,10 @@
1262 power_control=power_control)
1263 else:
1264 logger.warning("Couldn't find IP address for MAC %s" % mac)
1265+
1266+
1267+@task
1268+@log_exception_text
1269+def enlist_nodes_from_ucsm(url, username, password):
1270+ """ See `maasserver.api.NodeGroupsHandler.enlist_nodes_from_ucsm`. """
1271+ probe_and_enlist_ucsm(url, username, password)
1272
1273=== modified file 'src/provisioningserver/tests/test_tasks.py'
1274--- src/provisioningserver/tests/test_tasks.py 2014-04-22 15:10:32 +0000
1275+++ src/provisioningserver/tests/test_tasks.py 2014-05-06 13:49:30 +0000
1276@@ -72,6 +72,7 @@
1277 from provisioningserver.tags import MissingCredentials
1278 from provisioningserver.tasks import (
1279 add_new_dhcp_host_map,
1280+ enlist_nodes_from_ucsm,
1281 import_boot_images,
1282 Omshell,
1283 power_off,
1284@@ -676,3 +677,14 @@
1285 mock_callback = Mock()
1286 import_boot_images(callback=mock_callback)
1287 self.assertThat(mock_callback.delay, MockCalledOnceWith())
1288+
1289+
1290+class TestAddUCSM(PservTestCase):
1291+
1292+ def test_enlist_nodes_from_ucsm(self):
1293+ url = 'url'
1294+ username = 'username'
1295+ password = 'password'
1296+ mock = self.patch(tasks, 'probe_and_enlist_ucsm')
1297+ enlist_nodes_from_ucsm(url, username, password)
1298+ self.assertThat(mock, MockCalledOnceWith(url, username, password))
1299
1300=== modified file 'src/provisioningserver/utils/__init__.py'
1301--- src/provisioningserver/utils/__init__.py 2014-04-21 23:20:49 +0000
1302+++ src/provisioningserver/utils/__init__.py 2014-05-06 13:49:30 +0000
1303@@ -512,6 +512,11 @@
1304 self.__class__.__name__, self.value)
1305
1306
1307+def escape_py_literal(string):
1308+ """Escape and quote a string for use as a python literal."""
1309+ return repr(string).decode('ascii')
1310+
1311+
1312 class ShellTemplate(tempita.Template):
1313 """A Tempita template specialised for writing shell scripts.
1314
1315
1316=== modified file 'src/provisioningserver/utils/tests/test_utils.py'
1317--- src/provisioningserver/utils/tests/test_utils.py 2014-04-24 08:51:41 +0000
1318+++ src/provisioningserver/utils/tests/test_utils.py 2014-05-06 13:49:30 +0000
1319@@ -82,6 +82,7 @@
1320 MainScript,
1321 parse_key_value_file,
1322 pick_new_mtime,
1323+ escape_py_literal,
1324 read_text_file,
1325 Safe,
1326 ShellTemplate,
1327@@ -1393,3 +1394,22 @@
1328 # modification. The arguments passed back match those passed in
1329 # from do_stuff_in_thread().
1330 self.assertEqual(((3, 4), {"five": 5}), result)
1331+
1332+
1333+class TestQuotePyLiteral(MAASTestCase):
1334+ def test_uses_repr(self):
1335+ string = factory.make_name('string')
1336+ repr_mock = self.patch(provisioningserver.utils, 'repr')
1337+ escape_py_literal(string)
1338+ self.assertThat(repr_mock, MockCalledOnceWith(string))
1339+
1340+ def test_decodes_ascii(self):
1341+ string = factory.make_name('string')
1342+ output = factory.make_name('output')
1343+ repr_mock = self.patch(provisioningserver.utils, 'repr')
1344+ ascii_value = Mock()
1345+ ascii_value.decode = Mock(return_value=output)
1346+ repr_mock.return_value = ascii_value
1347+ value = escape_py_literal(string)
1348+ self.assertThat(ascii_value.decode, MockCalledOnceWith('ascii'))
1349+ self.assertEqual(value, output)