Merge lp:~heber013/qakit/adding-juju-bootstrapping into lp:qakit

Proposed by Heber Parrucci
Status: Merged
Approved by: Max Brustkern
Approved revision: 228
Merged at revision: 228
Proposed branch: lp:~heber013/qakit/adding-juju-bootstrapping
Merge into: lp:qakit
Diff against target: 456 lines (+421/-0)
7 files modified
qakit/juju/README.rst (+19/-0)
qakit/juju/__init__.py (+18/-0)
qakit/juju/hosts (+5/-0)
qakit/juju/juju_bootstrap (+105/-0)
qakit/juju/juju_bootstrap.py (+213/-0)
qakit/juju/mk-venv (+60/-0)
qakit/juju/requirements.txt (+1/-0)
To merge this branch: bzr merge lp:~heber013/qakit/adding-juju-bootstrapping
Reviewer Review Type Date Requested Status
Max Brustkern (community) Approve
Review via email: mp+320869@code.launchpad.net

Commit message

Adding juju bootstrapping

Description of the change

Automate bootstrapping for cloud, controller and model in the host machine using juju.

To test it, you need the file that contains the credentials for the cloud provider.
Example:

if your credentials file is: $HOME/.canonistack/novarc

Then run:

./juju_bootstrap --cloud_name ^THE_NAME_YOU_LIKE^ --nova_file "$HOME/.canonistack/novarc"

Too see instructions in details refer to README.rst

The script was implemented in part using subprocess because python juju clients have several bugs and limitations. I tested these clients:
- jujuclient
- juju

To post a comment you must log in.
220. By Heber Parrucci

Fixing relative paths in scripts

221. By Heber Parrucci

updating README.rst

222. By Heber Parrucci

Fixing relative paths when executing the script

223. By Heber Parrucci

Ensuring the nova key is always created

224. By Heber Parrucci

Adding check if bootstrap command failed

225. By Heber Parrucci

Adding proper ip range for scaling stack hosts

Revision history for this message
Max Brustkern (nuclearbob) wrote :

I've got a variety of diff comments and questions, many of which are concerned with the potential to duplicate config options or interfere with existing setups.

Revision history for this message
Heber Parrucci (heber013) wrote :

Thanks for the review!!! All comments addressed. Check replies inline for details.

226. By Heber Parrucci

Addressing review feedback

227. By Heber Parrucci

minor indentation change

Revision history for this message
Max Brustkern (nuclearbob) wrote :

Awesome, thanks for all the changes! One question about the placement of the new --replace argument, but other than that, I think it looks good.

Revision history for this message
Heber Parrucci (heber013) wrote :

Fixing:
* review comment when parsing replace parameter
* logging
* added logic so if replace parameter is set and the controller is bootstrapped it will delete it first before proceeding.

228. By Heber Parrucci

Fixing:
* review comment when parsing replace parameter
* logging
* added logic so if replace parameter is set and the controller is bootstrapped it will delete it first before proceeding.

Revision history for this message
Max Brustkern (nuclearbob) wrote :

Looks good! Let's land it!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added directory 'qakit/juju'
=== added file 'qakit/juju/README.rst'
--- qakit/juju/README.rst 1970-01-01 00:00:00 +0000
+++ qakit/juju/README.rst 2017-03-28 17:46:02 +0000
@@ -0,0 +1,19 @@
1#############################
2Bootstrapping cloud with juju
3#############################
4
5
6You need the file that contains the credentials for the cloud provider.
7
8Follow this steps:
9
10Bootstrap the cloud, controller and model in the machine:
11$ ./qakit/juju/juju_bootstrap --cloud_name ^THE_NAME_YOU_LIKE^ --nova_file ^NOVA_CREDENTIALS_FILE^
12For example if you have canonistack credentials in $HOME/.canonistack/novarc:
13$ ./qakit/juju/juju_bootstrap --cloud_name canonistack --nova_file "$HOME/.canonistack/novarc"
14Enter the sudo password if asked (Some commands are executed as root and others do not).
15Note: You need to provide the full path for the nova file. You can use $HOME but not ~.
16It will install the needed packages and bootstrap the cloud.
17The controller name will be the same of the cloud.
18Use --replace parameter to override any existing cloud/credentials with the same name.
19By default it will not override anything.
020
=== added file 'qakit/juju/__init__.py'
--- qakit/juju/__init__.py 1970-01-01 00:00:00 +0000
+++ qakit/juju/__init__.py 2017-03-28 17:46:02 +0000
@@ -0,0 +1,18 @@
1# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
2#
3# Juju Utils
4# Copyright (C) 2017 Canonical
5#
6# This program is free software: you can redistribute it and/or modify
7# it under the terms of the GNU General Public License as published by
8# the Free Software Foundation, either version 3 of the License, or
9# (at your option) any later version.
10#
11# This program is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14# GNU General Public License for more details.
15#
16# You should have received a copy of the GNU General Public License
17# along with this program. If not, see <http://www.gnu.org/licenses/>.
18#
019
=== added file 'qakit/juju/hosts'
--- qakit/juju/hosts 1970-01-01 00:00:00 +0000
+++ qakit/juju/hosts 2017-03-28 17:46:02 +0000
@@ -0,0 +1,5 @@
1Host 10.55.32.* 10.55.33.* 10.55.34.* 10.55.35.* 10.55.36.* 10.55.37.* 10.55.38.* 10.55.39.* 10.55.40.* 10.55.41.* 10.55.42.* 10.55.43.* 10.55.44.* 10.55.45.* 10.55.46.* 10.55.47.* 10.55.58.* 10.55.60.* 10.55.61.* 10.55.62.* 10.55.63.* *.canonistack
2 User ubuntu
3
4Host 10.42.184.* 10.42.185.* 10.42.186.* 10.42.187.* 10.42.188.* 10.42.189.* 10.42.190.* 10.42.191.* *.scalingstack
5 User ubuntu
06
=== added file 'qakit/juju/juju_bootstrap'
--- qakit/juju/juju_bootstrap 1970-01-01 00:00:00 +0000
+++ qakit/juju/juju_bootstrap 2017-03-28 17:46:02 +0000
@@ -0,0 +1,105 @@
1#!/bin/bash
2
3#
4# Juju Utils
5# Copyright (C) 2017 Canonical
6#
7# This program is free software: you can redistribute it and/or modify
8# it under the terms of the GNU General Public License as published by
9# the Free Software Foundation, either version 3 of the License, or
10# (at your option) any later version.
11#
12# This program is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU General Public License for more details.
16#
17# You should have received a copy of the GNU General Public License
18# along with this program. If not, see <http://www.gnu.org/licenses/>.
19#
20
21DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
22
23sudo apt-get install cloud-utils python-novaclient -y
24
25
26while [[ $# -gt 0 ]]
27 do
28 key="$1"
29 case ${key} in
30 --nova_file)
31 nova_file="$2"
32 shift # past argument
33 ;;
34 --cloud_name)
35 cloud_name="$2"
36 shift # past argument
37 ;;
38 --proxy)
39 proxy="$2"
40 shift # past argument
41 ;;
42 --replace)
43 replace=YES
44 ;;
45 esac
46 shift # past argument or value
47 done
48
49
50function add_nova_key {
51 nova keypair-show ${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}
52 if [ $? != 0 ]; then
53 install -m 600 /dev/null ${NOVA_KEY_DIR}/${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}.key
54 nova keypair-add ${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name} > ${NOVA_KEY_DIR}/${NOVA_USERNAME}_${OS_REGION_NAME}_${cloud_name}.key
55 fi
56 }
57
58source ${nova_file}
59
60# Adding hosts if they are not present
61hosts=$(cat ${DIR}/hosts)
62config=$(cat ~/.ssh/config)
63
64if [[ ! ${config} == *"$hosts"* ]]; then
65 cat ${DIR}/hosts >> ~/.ssh/config
66else
67 echo "SSH config already contains hosts"
68fi
69
70# Create nova key
71add_nova_key
72
73# Create python virtual env
74. ${DIR}/mk-venv -r ${DIR}/requirements.txt --name juju_ve --proxy ${proxy}
75
76# Bootstrap juju cloud/controller/model
77parameters="--cloud_name ${cloud_name} --username ${OS_USERNAME} --password ${OS_PASSWORD} --tenant ${OS_TENANT_NAME} --region ${OS_REGION_NAME} --url ${OS_AUTH_URL}"
78if [ ! -z ${replace} ]; then
79 parameters="$parameters --replace"
80fi
81${DIR}/juju_ve/bin/python3 ${DIR}/juju_bootstrap.py ${parameters}
82bootstrap_result=$?
83if [ ${bootstrap_result} != 0 ]; then
84 echo "Juju Bootstrap failed. Exiting..."
85 exit 1
86fi
87
88# Enable ssh
89my_temp=$(mktemp)
90juju add-ssh-key "$(cat ~/.ssh/id_rsa.pub)" &> ${my_temp}
91if [ -s my_temp ]; then
92 grep -q "duplicate ssh key" ${my_temp}
93 if [ $? -eq 0 ]; then
94 echo "ssh key already exists in the host"
95 else
96 echo "Adding public ssh key failed. Exiting..."
97 exit 1
98 fi
99fi
100
101rm -f my_temp
102
103deactivate
104
105echo "Juju Bootstrap successful!!!"
0106
=== added file 'qakit/juju/juju_bootstrap.py'
--- qakit/juju/juju_bootstrap.py 1970-01-01 00:00:00 +0000
+++ qakit/juju/juju_bootstrap.py 2017-03-28 17:46:02 +0000
@@ -0,0 +1,213 @@
1# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
2
3#
4# Juju Utils
5# Copyright (C) 2017 Canonical
6#
7# This program is free software: you can redistribute it and/or modify
8# it under the terms of the GNU General Public License as published by
9# the Free Software Foundation, either version 3 of the License, or
10# (at your option) any later version.
11#
12# This program is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU General Public License for more details.
16#
17# You should have received a copy of the GNU General Public License
18# along with this program. If not, see <http://www.gnu.org/licenses/>.
19#
20
21
22"""Bootstrap a cloud using juju"""
23
24import argparse
25import subprocess
26import tempfile
27
28import logging
29import yaml
30
31PACKAGES = ['juju', 'lxd', 'zfsutils-linux']
32PPAS = ['ppa:juju/stable']
33
34logging.basicConfig()
35LOGGER = logging.getLogger('juju_bootstrap')
36LOGGER.setLevel(logging.DEBUG)
37LOGGER.propagate = True
38
39
40def is_installed(pgk_name):
41 """
42 Check if package is installed
43 :param pgk_name: package name
44 :return: True if package is installed, False otherwise
45 """
46 try:
47 subprocess.check_call(['dpkg', '-s', pgk_name])
48 return True
49 except subprocess.CalledProcessError:
50 return False
51
52
53def install_packages(packages):
54 """Install the given packages"""
55 subprocess.check_call(['sudo', 'apt', 'update'])
56 for package in packages:
57 if not is_installed(package):
58 subprocess.check_call(['sudo', 'apt', 'install', package, '-y'])
59
60
61def add_ppas(ppas):
62 """Add the given ppas"""
63 for ppa in ppas:
64 subprocess.check_call(['sudo', 'add-apt-repository', ppa, '-y'])
65 subprocess.check_call(['sudo', 'apt', 'update'])
66
67
68def build_cloud_yaml(name, region, url):
69 """Build cloud into yaml file
70
71 :param name: the cloud name
72 :param region: the region name
73 :param url: the auth url
74 :return: the cloud yaml file
75 """
76 cloud = {"auth-types": ["access-key", "userpass"],
77 'regions': {region: {'endpoint': url}}, "type": "openstack"}
78 clouds_dict = {'clouds': {name: cloud}}
79 cloud_yaml = tempfile.NamedTemporaryFile()
80 with open(cloud_yaml.name, 'w') as temp:
81 temp.write(yaml.dump(clouds_dict))
82 return cloud_yaml
83
84
85def build_credentials(cloud, username, password, tenant, auth_type='userpass'):
86 """Build credentials into yaml file
87
88 :param cloud: the cloud name
89 :param username: the user name
90 :param password: the password
91 :param tenant: the tenant name
92 :param auth_type: the auth-type
93 :return: the credentials yaml file
94 """
95 credentials_dict = {
96 'credentials': {
97 cloud: {
98 cloud:
99 {'auth-type': auth_type,
100 'username': username,
101 'password': password,
102 'tenant-name': tenant,
103 }
104 }
105 }
106 }
107 credentials_yaml = tempfile.NamedTemporaryFile()
108 with open(credentials_yaml.name, 'w') as temp:
109 temp.write(yaml.dump(credentials_dict))
110 return credentials_yaml
111
112
113def add_cloud(name, region, url, replace):
114 """Add cloud with juju
115
116 :param name: the cloud name
117 :param region: the region name
118 :param url: the auth url
119 :param replace: whether to replace the cloud if exists
120 """
121 LOGGER.info('Trying to add cloud %s', name)
122 cloud_yaml = build_cloud_yaml(name, region, url)
123 command = ['juju', 'add-cloud', name, cloud_yaml.name]
124 if replace:
125 command.append('--replace')
126 subprocess.check_call(command)
127 LOGGER.info('Cloud %s added successfully', name)
128
129
130def add_credentials(cloud_name, username, password, tenant, replace):
131 """Add credentials for the cloud
132
133 :param cloud_name: the cloud name
134 :param username: the user name
135 :param password: the password
136 :param tenant: the tenant name
137 :param replace: whether to replace the credentials if exists
138 """
139 LOGGER.info('Trying to add credentials for cloud %s', cloud_name)
140 credentials_yaml = build_credentials(cloud_name, username, password, tenant)
141 command = ['juju', 'add-credential', cloud_name, '-f', credentials_yaml.name]
142 if replace:
143 command.append('--replace')
144 subprocess.check_call(command)
145 LOGGER.info('Credentials added successfully for cloud %s', cloud_name)
146
147
148def is_controller_bootstrapped(name):
149 """Check if controller is already bootstrapped
150
151 :param name: the controller name to check
152 :return: True if controller is bootstrapped, False otherwise
153 """
154 try:
155 LOGGER.info('Checking if controller %s is bootstrapped ', name)
156 subprocess.check_call(['juju', 'show-controller', name])
157 LOGGER.info('Controller %s is already bootstrapped ', name)
158 return True
159 except subprocess.CalledProcessError:
160 LOGGER.info('Controller %s is not bootstrapped ', name)
161 return False
162
163
164def bootstrap(cloud, username, password, tenant, region, url, replace):
165 """Bootstrap a cloud using juju
166
167 :param cloud: the cloud name
168 :param username: the username
169 :param password: the password
170 :param tenant: the tenant name
171 :param region: the region name
172 :param url: the auth url
173 :param replace: whether to replace the cloud and
174 credentials if exists
175 """
176 bootstrapped = is_controller_bootstrapped(cloud)
177 if replace and bootstrapped:
178 LOGGER.info('Controller %s already bootstrapped. '
179 'Destroying it before proceeding...', cloud)
180 subprocess.check_call(['juju', 'destroy-controller', cloud,
181 '--destroy-all-models', '-y'])
182 elif not replace and bootstrapped:
183 LOGGER.info('Controller %s already bootstrapped and '
184 'replace parameter is not set...Exiting', cloud)
185 return
186 LOGGER.info('About to bootstrap controller %s...', cloud)
187 add_cloud(cloud, region, url, replace)
188 add_credentials(cloud, username, password, tenant, replace)
189 subprocess.check_call(['juju',
190 'bootstrap',
191 cloud,
192 cloud,
193 '--credential',
194 cloud])
195
196
197if __name__ == '__main__':
198 PARSER = argparse.ArgumentParser(description=
199 'Bootstrap a juju cloud')
200 PARSER.add_argument('--cloud_name',
201 help='Cloud name that will be also use '
202 'for controller name')
203 PARSER.add_argument('--username')
204 PARSER.add_argument('--password')
205 PARSER.add_argument('--tenant')
206 PARSER.add_argument('--region')
207 PARSER.add_argument('--url')
208 PARSER.add_argument('--replace', action='store_true')
209 ARGS = PARSER.parse_args()
210 add_ppas(PPAS)
211 install_packages(PACKAGES)
212 bootstrap(ARGS.cloud_name, ARGS.username, ARGS.password, ARGS.tenant,
213 ARGS.region, ARGS.url, ARGS.replace)
0214
=== added file 'qakit/juju/mk-venv'
--- qakit/juju/mk-venv 1970-01-01 00:00:00 +0000
+++ qakit/juju/mk-venv 2017-03-28 17:46:02 +0000
@@ -0,0 +1,60 @@
1#!/bin/bash
2
3# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
4#
5# Juju Utils
6# Copyright (C) 2017 Canonical
7#
8# This program is free software: you can redistribute it and/or modify
9# it under the terms of the GNU General Public License as published by
10# the Free Software Foundation, either version 3 of the License, or
11# (at your option) any later version.
12#
13# This program is distributed in the hope that it will be useful,
14# but WITHOUT ANY WARRANTY; without even the implied warranty of
15# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16# GNU General Public License for more details.
17#
18# You should have received a copy of the GNU General Public License
19# along with this program. If not, see <http://www.gnu.org/licenses/>.
20#
21
22DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
23
24while [[ $# -gt 1 ]]
25 do
26 key="$1"
27 case ${key} in
28 --proxy)
29 proxy="$2"
30 shift # past argument
31 ;;
32 -r|--requirements)
33 requirements="$2"
34 shift # past argument
35 ;;
36 --name)
37 name="$2"
38 shift # past argument
39 ;;
40 esac
41 shift # past argument or value
42 done
43
44if [ -z "$name" ]; then
45 name=ve
46fi
47
48if [ -z "$requirements" ]; then
49 requirements=requirements.txt
50fi
51
52virtualenv --python=python3 ${DIR}/${name}
53. ${DIR}/${name}/bin/activate
54
55if [ ! -z "$proxy" ]; then
56 echo "Using proxy: ${proxy} to install dependencies"
57 pip3 install --proxy ${proxy} -r ${requirements}
58else
59 pip3 install -r ${requirements}
60fi
061
=== added file 'qakit/juju/requirements.txt'
--- qakit/juju/requirements.txt 1970-01-01 00:00:00 +0000
+++ qakit/juju/requirements.txt 2017-03-28 17:46:02 +0000
@@ -0,0 +1,1 @@
1pyyaml

Subscribers

People subscribed via source and target branches