Merge lp:~heber013/qakit/adding-juju-bootstrapping into lp:qakit

[Heber Parrucci]

Adding juju bootstrapping

Automate bootstrapping for cloud, controller and model in the host machine using juju.

To test it, you need the file that contains the credentials for the cloud provider.
Example:

if your credentials file is: $HOME/.canonistack/novarc

Then run:

./juju_bootstrap --cloud_name ^THE_NAME_YOU_LIKE^ --nova_file "$HOME/.canonistack/novarc"

Too see instructions in details refer to README.rst

The script was implemented in part using subprocess because python juju clients have several bugs and limitations. I tested these clients:
- jujuclient
- juju

[lp:~heber013/qakit/adding-juju-bootstrapping]

220. By Heber Parrucci on 2017-03-23

Fixing relative paths in scripts

221. By Heber Parrucci on 2017-03-23

updating README.rst

222. By Heber Parrucci on 2017-03-27

Fixing relative paths when executing the script

223. By Heber Parrucci on 2017-03-27

Ensuring the nova key is always created

224. By Heber Parrucci on 2017-03-27

Adding check if bootstrap command failed

225. By Heber Parrucci on 2017-03-27

Adding proper ip range for scaling stack hosts

[Max Brustkern (nuclearbob)]

I've got a variety of diff comments and questions, many of which are concerned with the potential to duplicate config options or interfere with existing setups.

[Heber Parrucci (heber013)]

Thanks for the review!!! All comments addressed. Check replies inline for details.

[lp:~heber013/qakit/adding-juju-bootstrapping]

226. By Heber Parrucci on 2017-03-27

Addressing review feedback

227. By Heber Parrucci on 2017-03-27

minor indentation change

[Max Brustkern (nuclearbob)]

Awesome, thanks for all the changes! One question about the placement of the new --replace argument, but other than that, I think it looks good.

[Heber Parrucci (heber013)]

Fixing:
* review comment when parsing replace parameter
* logging
* added logic so if replace parameter is set and the controller is bootstrapped it will delete it first before proceeding.

[lp:~heber013/qakit/adding-juju-bootstrapping]

228. By Heber Parrucci on 2017-03-28

Fixing:
* review comment when parsing replace parameter
* logging
* added logic so if replace parameter is set and the controller is bootstrapped it will delete it first before proceeding.

[Max Brustkern (nuclearbob)]

Looks good! Let's land it!

review: Approve

=== added directory 'qakit/juju'
=== added file 'qakit/juju/README.rst'
--- qakit/juju/README.rst	1970-01-01 00:00:00 +0000
+++ qakit/juju/README.rst	2017-03-28 17:46:02 +0000
@@ -0,0 +1,19 @@
+#############################
+Bootstrapping cloud with juju
+#############################
+
+
+You need the file that contains the credentials for the cloud provider.
+
+Follow this steps:
+
+Bootstrap the cloud, controller and model in the machine:
+$ ./qakit/juju/juju_bootstrap --cloud_name ^THE_NAME_YOU_LIKE^ --nova_file ^NOVA_CREDENTIALS_FILE^
+For example if you have canonistack credentials in $HOME/.canonistack/novarc:
+$ ./qakit/juju/juju_bootstrap --cloud_name canonistack --nova_file "$HOME/.canonistack/novarc"
+Enter the sudo password if asked (Some commands are executed as root and others do not).
+Note: You need to provide the full path for the nova file. You can use $HOME but not ~.
+It will install the needed packages and bootstrap the cloud.
+The controller name will be the same of the cloud.
+Use --replace parameter to override any existing cloud/credentials with the same name.
+By default it will not override anything.

=== added file 'qakit/juju/__init__.py'
--- qakit/juju/__init__.py	1970-01-01 00:00:00 +0000
+++ qakit/juju/__init__.py	2017-03-28 17:46:02 +0000
@@ -0,0 +1,18 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+#
+# Juju Utils
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#

=== added file 'qakit/juju/hosts'
--- qakit/juju/hosts	1970-01-01 00:00:00 +0000
+++ qakit/juju/hosts	2017-03-28 17:46:02 +0000
@@ -0,0 +1,5 @@
+Host 10.55.32.* 10.55.33.* 10.55.34.* 10.55.35.* 10.55.36.* 10.55.37.* 10.55.38.* 10.55.39.* 10.55.40.* 10.55.41.* 10.55.42.* 10.55.43.* 10.55.44.* 10.55.45.* 10.55.46.* 10.55.47.* 10.55.58.* 10.55.60.* 10.55.61.* 10.55.62.* 10.55.63.* *.canonistack
+  User ubuntu
+
+Host 10.42.184.* 10.42.185.* 10.42.186.* 10.42.187.* 10.42.188.* 10.42.189.* 10.42.190.* 10.42.191.* *.scalingstack
+  User ubuntu

=== added file 'qakit/juju/juju_bootstrap'
--- qakit/juju/juju_bootstrap	1970-01-01 00:00:00 +0000
+++ qakit/juju/juju_bootstrap	2017-03-28 17:46:02 +0000
@@ -0,0 +1,105 @@
+#!/bin/bash
+
+#
+# Juju Utils
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+sudo apt-get install cloud-utils python-novaclient -y
+
+
+while [[ $# -gt 0 ]]
+    do
+        key="$1"
+        case ${key} in
+            --nova_file)
+            nova_file="$2"
+            shift # past argument
+            ;;
+            --cloud_name)
+            cloud_name="$2"
+            shift # past argument
+            ;;
+            --proxy)
+            proxy="$2"
+            shift # past argument
+            ;;
+            --replace)
+            replace=YES
+            ;;
+        esac
+        shift # past argument or value
+    done
+
+
+function add_nova_key {
+    nova keypair-show ${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}
+    if [ $? != 0 ]; then
+        install -m 600 /dev/null ${NOVA_KEY_DIR}/${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}.key
+        nova keypair-add ${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name} > ${NOVA_KEY_DIR}/${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}.key
+    fi
+    }
+
+source ${nova_file}
+
+# Adding hosts if they are not present
+hosts=$(cat ${DIR}/hosts)
+config=$(cat ~/.ssh/config)
+
+if [[ ! ${config} == *"$hosts"* ]]; then
+    cat ${DIR}/hosts  >> ~/.ssh/config
+else
+    echo "SSH config already contains hosts"
+fi
+
+# Create nova key
+add_nova_key
+
+# Create python virtual env
+. ${DIR}/mk-venv -r ${DIR}/requirements.txt --name juju_ve --proxy ${proxy}
+
+# Bootstrap juju cloud/controller/model
+parameters="--cloud_name ${cloud_name} --username ${OS_USERNAME} --password ${OS_PASSWORD} --tenant ${OS_TENANT_NAME} --region ${OS_REGION_NAME} --url ${OS_AUTH_URL}"
+if [ ! -z ${replace} ]; then
+    parameters="$parameters --replace"
+fi
+${DIR}/juju_ve/bin/python3 ${DIR}/juju_bootstrap.py ${parameters}
+bootstrap_result=$?
+if [ ${bootstrap_result} != 0 ]; then
+    echo "Juju Bootstrap failed. Exiting..."
+    exit 1
+fi
+
+# Enable ssh
+my_temp=$(mktemp)
+juju add-ssh-key "$(cat ~/.ssh/id_rsa.pub)" &> ${my_temp}
+if [ -s my_temp ]; then
+    grep -q "duplicate ssh key" ${my_temp}
+    if [ $? -eq 0 ]; then
+        echo "ssh key already exists in the host"
+    else
+        echo "Adding public ssh key failed. Exiting..."
+        exit 1
+    fi
+fi
+
+rm -f my_temp
+
+deactivate
+
+echo "Juju Bootstrap successful!!!"

=== added file 'qakit/juju/juju_bootstrap.py'
--- qakit/juju/juju_bootstrap.py	1970-01-01 00:00:00 +0000
+++ qakit/juju/juju_bootstrap.py	2017-03-28 17:46:02 +0000
@@ -0,0 +1,213 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Juju Utils
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+"""Bootstrap a cloud using juju"""
+
+import argparse
+import subprocess
+import tempfile
+
+import logging
+import yaml
+
+PACKAGES = ['juju', 'lxd', 'zfsutils-linux']
+PPAS = ['ppa:juju/stable']
+
+logging.basicConfig()
+LOGGER = logging.getLogger('juju_bootstrap')
+LOGGER.setLevel(logging.DEBUG)
+LOGGER.propagate = True
+
+
+def is_installed(pgk_name):
+    """
+    Check if package is installed
+    :param pgk_name: package name
+    :return: True if package is installed, False otherwise
+    """
+    try:
+        subprocess.check_call(['dpkg', '-s', pgk_name])
+        return True
+    except subprocess.CalledProcessError:
+        return False
+
+
+def install_packages(packages):
+    """Install the given packages"""
+    subprocess.check_call(['sudo', 'apt', 'update'])
+    for package in packages:
+        if not is_installed(package):
+            subprocess.check_call(['sudo', 'apt', 'install', package, '-y'])
+
+
+def add_ppas(ppas):
+    """Add the given ppas"""
+    for ppa in ppas:
+        subprocess.check_call(['sudo', 'add-apt-repository', ppa, '-y'])
+    subprocess.check_call(['sudo', 'apt', 'update'])
+
+
+def build_cloud_yaml(name, region, url):
+    """Build cloud into yaml file
+
+    :param name: the cloud name
+    :param region: the region name
+    :param url: the auth url
+    :return: the cloud yaml file
+    """
+    cloud = {"auth-types": ["access-key", "userpass"],
+             'regions': {region: {'endpoint': url}}, "type": "openstack"}
+    clouds_dict = {'clouds': {name: cloud}}
+    cloud_yaml = tempfile.NamedTemporaryFile()
+    with open(cloud_yaml.name, 'w') as temp:
+        temp.write(yaml.dump(clouds_dict))
+    return cloud_yaml
+
+
+def build_credentials(cloud, username, password, tenant, auth_type='userpass'):
+    """Build credentials into yaml file
+
+    :param cloud: the cloud name
+    :param username: the user name
+    :param password: the password
+    :param tenant: the tenant name
+    :param auth_type: the auth-type
+    :return: the credentials yaml file
+    """
+    credentials_dict = {
+        'credentials': {
+            cloud: {
+                cloud:
+                {'auth-type': auth_type,
+                 'username': username,
+                 'password': password,
+                 'tenant-name': tenant,
+                }
+            }
+        }
+    }
+    credentials_yaml = tempfile.NamedTemporaryFile()
+    with open(credentials_yaml.name, 'w') as temp:
+        temp.write(yaml.dump(credentials_dict))
+    return credentials_yaml
+
+
+def add_cloud(name, region, url, replace):
+    """Add cloud with juju
+
+    :param name: the cloud name
+    :param region: the region name
+    :param url: the auth url
+    :param replace: whether to replace the cloud if exists
+    """
+    LOGGER.info('Trying to add cloud %s', name)
+    cloud_yaml = build_cloud_yaml(name, region, url)
+    command = ['juju', 'add-cloud', name, cloud_yaml.name]
+    if replace:
+        command.append('--replace')
+    subprocess.check_call(command)
+    LOGGER.info('Cloud %s added successfully', name)
+
+
+def add_credentials(cloud_name, username, password, tenant, replace):
+    """Add credentials for the cloud
+
+    :param cloud_name: the cloud name
+    :param username: the user name
+    :param password: the password
+    :param tenant: the tenant name
+    :param replace: whether to replace the credentials if exists
+    """
+    LOGGER.info('Trying to add credentials for cloud %s', cloud_name)
+    credentials_yaml = build_credentials(cloud_name, username, password, tenant)
+    command = ['juju', 'add-credential', cloud_name, '-f', credentials_yaml.name]
+    if replace:
+        command.append('--replace')
+    subprocess.check_call(command)
+    LOGGER.info('Credentials added successfully for cloud %s', cloud_name)
+
+
+def is_controller_bootstrapped(name):
+    """Check if controller is already bootstrapped
+
+    :param name: the controller name to check
+    :return: True if controller is bootstrapped, False otherwise
+    """
+    try:
+        LOGGER.info('Checking if controller %s is bootstrapped ', name)
+        subprocess.check_call(['juju', 'show-controller', name])
+        LOGGER.info('Controller %s is already bootstrapped ', name)
+        return True
+    except subprocess.CalledProcessError:
+        LOGGER.info('Controller %s is not bootstrapped ', name)
+        return False
+
+
+def bootstrap(cloud, username, password, tenant, region, url, replace):
+    """Bootstrap a cloud using juju
+
+    :param cloud: the cloud name
+    :param username: the username
+    :param password: the password
+    :param tenant: the tenant name
+    :param region: the region name
+    :param url: the auth url
+    :param replace: whether to replace the cloud and
+    credentials if exists
+    """
+    bootstrapped = is_controller_bootstrapped(cloud)
+    if replace and bootstrapped:
+        LOGGER.info('Controller %s already bootstrapped. '
+                    'Destroying it before proceeding...', cloud)
+        subprocess.check_call(['juju', 'destroy-controller', cloud,
+                               '--destroy-all-models', '-y'])
+    elif not replace and bootstrapped:
+        LOGGER.info('Controller %s already bootstrapped and '
+                    'replace parameter is not set...Exiting', cloud)
+        return
+    LOGGER.info('About to bootstrap controller %s...', cloud)
+    add_cloud(cloud, region, url, replace)
+    add_credentials(cloud, username, password, tenant, replace)
+    subprocess.check_call(['juju',
+                           'bootstrap',
+                           cloud,
+                           cloud,
+                           '--credential',
+                           cloud])
+
+
+if __name__ == '__main__':
+    PARSER = argparse.ArgumentParser(description=
+                                     'Bootstrap a juju cloud')
+    PARSER.add_argument('--cloud_name',
+                        help='Cloud name that will be also use '
+                             'for controller name')
+    PARSER.add_argument('--username')
+    PARSER.add_argument('--password')
+    PARSER.add_argument('--tenant')
+    PARSER.add_argument('--region')
+    PARSER.add_argument('--url')
+    PARSER.add_argument('--replace', action='store_true')
+    ARGS = PARSER.parse_args()
+    add_ppas(PPAS)
+    install_packages(PACKAGES)
+    bootstrap(ARGS.cloud_name, ARGS.username, ARGS.password, ARGS.tenant,
+              ARGS.region, ARGS.url, ARGS.replace)

=== added file 'qakit/juju/mk-venv'
--- qakit/juju/mk-venv	1970-01-01 00:00:00 +0000
+++ qakit/juju/mk-venv	2017-03-28 17:46:02 +0000
@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+#
+# Juju Utils
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+while [[ $# -gt 1 ]]
+    do
+        key="$1"
+        case ${key} in
+            --proxy)
+            proxy="$2"
+            shift # past argument
+            ;;
+            -r|--requirements)
+            requirements="$2"
+            shift # past argument
+            ;;
+            --name)
+            name="$2"
+            shift # past argument
+            ;;
+        esac
+        shift # past argument or value
+    done
+
+if [ -z "$name" ]; then
+    name=ve
+fi
+
+if [ -z "$requirements" ]; then
+    requirements=requirements.txt
+fi
+
+virtualenv --python=python3 ${DIR}/${name}
+. ${DIR}/${name}/bin/activate
+
+if [ ! -z "$proxy" ]; then
+    echo "Using proxy: ${proxy} to install dependencies"
+    pip3 install --proxy ${proxy} -r ${requirements}
+else
+    pip3 install -r ${requirements}
+fi

=== added file 'qakit/juju/requirements.txt'
--- qakit/juju/requirements.txt	1970-01-01 00:00:00 +0000
+++ qakit/juju/requirements.txt	2017-03-28 17:46:02 +0000
@@ -0,0 +1,1 @@
+pyyaml