Merge lp:~freyes/maas-deployer/virtual-nodes into lp:maas-deployer

[Felipe Reyes]

Maintainers,

I would like to add the option to create virtual machines (virtual-nodes) and get them registered into the freshly installed maas.

This MP includes a exmple yaml file to understand how this is intended to be used.

Rationale
---------

maas-deployer already has all the plumbing needed to create (through virsh) and register VMs/bare-metal (into MAAS), but some scenarios (specially labs and bug reproducers) require to create extra VMs for the workload that it's intended to be deployed with juju (typically OpenStack), so I pretend to use this for that, as a tool to build disposable labs easily.

Best,

[Billy Olsen (billy-olsen)]

In general, I'm +1 on this. I feel that the virtual-nodes set is something that may be confusing rather than just nodes, however I do understand why its there (so there's a distinction of nodes to create and nodes which are expected to be present). I really waffled back and forth about whether to use virtual-nodes or to use a tag inside of a node, but ultimately it doesn't matter that much so I'll approve as is.

review: Approve

[Nobuto Murata (nobuto)]

Wait, the branch seems not based on /next branch. So some changes in /next have been lost. For example, 'architecture'='amd64/generic' is hard-coded again (I'm using maas-deployer for ppc64el as well). The branch should be re-based and re-submitted.

------------------------------------------------------------
revno: 69 [merge]
committer: Billy Olsen <email address hidden>
branch nick: next
timestamp: Fri 2016-04-15 17:03:06 -0700
message:
  [freyes, r=billy-olsen]

  Add 'virtual-nodes' config key to create extra VMs during deployment.
    ------------------------------------------------------------
    revno: 63.1.2
    committer: Felipe Reyes <email address hidden>
    branch nick: maas-deployer.bzr
    timestamp: Tue 2016-03-15 13:11:10 -0300
    message:
      Add example on how to create virtual nodes during deployment
    ------------------------------------------------------------
    revno: 63.1.1
    committer: Felipe Reyes <email address hidden>
    branch nick: maas-deployer.bzr
    timestamp: Mon 2016-03-14 18:55:46 -0300
    message:
      Add 'virtual-nodes' config key to create extra VMs

=== modified file 'maas_deployer/vmaas/engine.py'
--- maas_deployer/vmaas/engine.py 2016-04-15 16:18:34 +0000
+++ maas_deployer/vmaas/engine.py 2016-04-17 17:07:02 +0000
@@ -77,20 +80,21 @@ class DeploymentEngine(object):
         self.wait_for_import_boot_images(client, maas_config)
         self.configure_maas(client, maas_config)

- def _get_juju_node_params(self, juju_domain, juju_config, maas_config):
+ def _get_node_params(self, node_domain, node_config, maas_config,
+ tags=None):
         """
- Determines the mac address of the juju machine specified.
+ Determines the mac address of the node machine specified.

- :param juju_domain: the juju bootstrap image domain
+ :param node_domain: the juju bootstrap image domain
         :param include_power: a boolean value of whether to include
                               power parameters or not for virsh power
                               control.
         """
         node = {
- 'name': juju_domain.name,
- 'architecture': juju_config.get('arch'),
- 'mac_addresses': [x for x in juju_domain.mac_addresses],
- 'tags': 'bootstrap'
+ 'name': node_domain.name,
+ 'architecture': 'amd64/generic',
+ 'mac_addresses': [x for x in node_domain.mac_addresses],
+ 'tags': tags if tags else node_config['tags'],
         }

         virsh_info = maas_config.get('virsh')

review: Needs Fixing

=== added file 'examples/virtual-nodes.yaml'
--- examples/virtual-nodes.yaml	1970-01-01 00:00:00 +0000
+++ examples/virtual-nodes.yaml	2016-03-15 21:55:48 +0000
@@ -0,0 +1,166 @@
+# This file defines the deployment for the MAAS environment which is to be
+# deployed and automated.
+demo-maas:
+    # Contains the virtual machine parameters for creating the Juju bootstrap
+    # node virtual machine
+    juju-bootstrap:
+        name: juju-boot-vm-dc1
+        interfaces: ['bridge=virbr0,model=virtio']
+        memory: 2048
+        vcpus: 2
+        arch: amd64
+        pool: default
+        disk_size: 20G
+        sticky_ip_address:
+            requested_address: 192.168.122.5
+
+    # Contains a list of nodes to be created and registered
+    virtual-nodes:
+        - name: keystone
+          interfaces: ['bridge=virbr0,model=virtio']
+          vcpus: 1
+          memory: 2048
+          arch: amd64
+          pool: default
+          disk_size: 10G
+          sticky_ip_address:
+              requested_address: 192.168.122.50
+          tags: keystone
+        - name: glance
+          interfaces: ['bridge=virbr0,model=virtio']
+          vcpus: 1
+          memory: 2048
+          arch: amd64
+          pool: default
+          disk_size: 50G
+          sticky_ip_address:
+              requested_address: 192.168.122.51
+          tags: glance
+
+    maas:
+        # Defines the general setup for the MAAS environment, including the
+        # username and password for the host as well as the MAAS server.
+        user: ubuntu
+        password: ubuntu
+
+        # Contains the virtual machine parameters for creating the MAAS virtual
+        # server. Here you can configure the name of the virsh domain, the
+        # parameters for how the network is attached.
+        name: maas-boot-vm-dc1
+        interfaces: ['bridge=virbr0,model=virtio']
+        memory: 4096
+        vcpus: 2
+        arch: amd64
+        pool: default
+        disk_size: 60G
+
+        # Apt http proxy setting(s)
+        #apt_http_proxy:
+
+        # Package sources. These will be used on the MAAS controller.
+        apt_sources:
+          - ppa:maas/stable
+          - ppa:juju/stable
+
+        # Virsh power settings
+        # Specifies the uri and keys to use for virsh power control of the 
+        # juju virtual machine. If the uri is omitted, the value for the
+        # --remote is used. If no power settings are desired, then do not
+        # supply the virsh block.
+        #virsh:
+        #    rsa_priv_key: /path/to/id_rsa
+        #    rsa_pub_key: /path/to/id_rsa.pub
+        #    uri: qemu+ssh://user@10.0.3.1/system
+
+        # Defines the IP Address that the configuration script will use to
+        # to access the MAAS controller via SSH.
+        ip_address: 192.168.122.2
+
+        # This section allows the user to set a series of options on the
+        # MAAS server itself. The list of config options can be found in
+        # the upstream MAAS documentation:
+        #  - http://maas.ubuntu.com/docs/api.html#maas-server
+        settings:
+            main_archive: http://archive.ubuntu.com/ubuntu
+            upstream_dns: 192.168.122.1
+            maas_name: automaas
+            # kernel_opts: "console=tty0 console=ttyS1,115200n8"
+            # ntp_server: ntp.ubuntu.com
+
+        # This section is used to define the networking parameters for when
+        # the node first comes up. It is fed into the meta-data cloud-init
+        # configuration and is used to configure the networking piece of the
+        # service. The contents of this section are written directly to the
+        # /etc/network/interfaces file.
+        #
+        # Please note, this is slightly different than the
+        # node-group-interfaces section below. This will configure the
+        # machine's networking params, and the node-group-interfaces will
+        # configure the maas node-group interfaces which is used for
+        # controlling the dhcp, dns, etc.
+        network_config: |
+            auto lo
+            iface lo inet loopback
+
+            auto eth0
+            iface eth0 inet static
+              address 192.168.122.2
+              netmask 255.255.248.0
+              network 192.168.122.0
+              broadcast 192.168.122.255
+              gateway 192.168.122.1
+              dns-nameservers 192.168.122.1 127.0.0.1
+
+        # See https://maas.ubuntu.com/docs/maascli.html#node-groups for
+        # description and full list of supported options.
+        # NOTE: interfaces are added using the node_group_interfaces section
+        #       and only one node_group can be created by this bundle.
+        #       Additional node groups can be added post deployment.
+        #node_group:
+        #    # This is the cluster DNS name.
+        #    name: maas
+
+        # The node-group-interfaces section is used to configure the MAAS
+        # network interfaces. Basic configuration is supported, such as which
+        # device should be bound, the range of IP addresses, etc.
+        # Note: this may contain the special identifiers:
+        #   ${maas_net} - the first 3 octets of the ipv4 address
+        #   ${maas_ip} - the ip address of the MAAS controller 
+        node_group_ifaces:
+            - device: eth0
+              ip: 192.168.122.2
+              subnet_mask: 255.255.255.0
+              broadcast_ip: 192.168.122.255
+              router_ip: 192.168.122.1
+              static_range:
+                  low: 192.168.122.51
+                  high: 192.168.122.60
+              dynamic_range:
+                  low: 192.168.122.5
+                  high: 192.168.122.50
+
+        # Physical nodes to be added to the MAAS cluster. Nodes will be
+        # configured, commissioned and put into the Ready state so
+        # they are ready to be deployed to.
+        #nodes:
+        #    - name: node1
+        #      tags: api
+        #      architecture: amd64/generic
+        #      mac_addresses:
+        #          - "38:63:bb:43:b8:9c"
+        #      # MAAS power settings. Supported key/values are described here:
+        #      # https://maas.ubuntu.com/docs/api.html#power-types. For keys
+        #      # that start with 'power_' this part can optionally be omitted
+        #      # and will be automatically prepended by maas-deployer. The
+        #      # 'type' key should be set to the power type you want to
+        #      # configure.
+        #      power:
+        #          type: ipmi
+        #          address: 10.0.1.1
+        #          user: maas
+        #          pass: passw0rd
+        #          driver: LAN_2_0
+        #      sticky_ip_address:
+        #          mac_address: "38:63:bb:43:b8:9c"
+        #          requested_address: 192.168.122.6
+

=== modified file 'maas_deployer/vmaas/engine.py'
--- maas_deployer/vmaas/engine.py	2016-02-11 09:57:23 +0000
+++ maas_deployer/vmaas/engine.py	2016-03-15 21:55:48 +0000
@@ -60,6 +60,11 @@
                                                maas_config)
         nodes.append(juju_node)
 
+        # create extra VMs
+        for node in config.get('virtual-nodes', {}):
+            n = self.deploy_virtual_node(node, maas_config)
+            nodes.append(n)
+
         self.deploy_maas_node(maas_config)
 
         self.wait_for_maas_installation(maas_config)
@@ -74,20 +79,21 @@
         self.wait_for_import_boot_images(client, maas_config)
         self.configure_maas(client, maas_config)
 
-    def _get_juju_node_params(self, juju_domain, juju_config, maas_config):
+    def _get_node_params(self, node_domain, node_config, maas_config,
+                         tags=None):
         """
-        Determines the mac address of the juju machine specified.
+        Determines the mac address of the node machine specified.
 
-        :param juju_domain: the juju bootstrap image domain
+        :param node_domain: the juju bootstrap image domain
         :param include_power: a boolean value of whether to include
                               power parameters or not for virsh power
                               control.
         """
         node = {
-            'name': juju_domain.name,
+            'name': node_domain.name,
             'architecture': 'amd64/generic',
-            'mac_addresses': [x for x in juju_domain.mac_addresses],
-            'tags': 'bootstrap'
+            'mac_addresses': [x for x in node_domain.mac_addresses],
+            'tags': tags if tags else node_config['tags'],
         }
 
         virsh_info = maas_config.get('virsh')
@@ -96,10 +102,10 @@
             node.update({
                 'power_type': 'virsh',
                 'power_parameters_power_address': uri,
-                'power_parameters_power_id': juju_domain.name,
+                'power_parameters_power_id': node_domain.name,
             })
 
-        sticky_cfg = juju_config.get('sticky_ip_address')
+        sticky_cfg = node_config.get('sticky_ip_address')
         if sticky_cfg:
             node['sticky_ip_address'] = sticky_cfg
             node['sticky_ip_address']['mac_address'] = node['mac_addresses'][0]
@@ -117,7 +123,16 @@
             juju_node.define()
             # Insert juju node information into the maas nodes list.
             # This allows us to define it in maas.
-            return self._get_juju_node_params(juju_node, params, maas_config)
+            return self._get_node_params(juju_node, params, maas_config,
+                                              tags='bootstrap')
+
+    def deploy_virtual_node(self, params, maas_config):
+        log.debug('Creating VM: %s' % params['name'])
+        with vm.Instance(params) as node:
+            node.netboot = True
+            node.define()
+
+            return self._get_node_params(node, params, maas_config)
 
     def deploy_maas_node(self, params):
         """
@@ -538,12 +553,13 @@
             """
             util.exec_script_remote(maas_config['user'], self.ip_addr, script)
 
-        # Start juju domain
+        # Start nodes
         virsh_info = maas_config.get('virsh')
-        juju_node = self._get_juju_nodename(nodes)
-        if juju_node is not None and not virsh_info:
+        for n in nodes:
+            name = n['name']
             try:
-                util.virsh(['start', juju_node])
+                log.info('Starting: %s' % name)
+                util.virsh(['start', name])
             except CalledProcessError as exc:
                 # Ignore already started error
                 msg = 'Domain is already active'