UTAH

Merge lp:~nuclearbob/utah/reorg into lp:utah

  1. reorg
  2. Merge into dev
Proposed by Max Brustkern
Status: Merged
Approved by: Javier Collado
Approved revision: 816
Merged at revision: 790
Proposed branch: lp:~nuclearbob/utah/reorg
Merge into: lp:utah
Diff against target: 2747 lines (+1107/-1358)
20 files modified
conf/config (+1/-1)
conf/utah/uqt-vm-tools.conf (+0/-60)
debian/control (+25/-1)
debian/rules (+35/-8)
debian/utah.postinst (+0/-1)
docs/source/reference.rst (+9/-3)
examples/run_install_test.py (+1/-2)
examples/run_test_bamboo_feeder.py (+2/-1)
examples/run_test_cobbler.py (+5/-2)
examples/run_test_vm.py (+1/-1)
examples/utah-user-setup.sh (+0/-7)
examples/vmtools-user-setup.sh (+0/-59)
utah/provisioning/baremetal/bamboofeeder.py (+3/-3)
utah/provisioning/baremetal/cobbler.py (+3/-3)
utah/provisioning/baremetal/inventory.py (+153/-0)
utah/provisioning/inventory/sqlite.py (+0/-183)
utah/provisioning/provisioning.py (+3/-243)
utah/provisioning/ssh.py (+261/-0)
utah/provisioning/vm/libvirtvm.py (+0/-777)
utah/provisioning/vm/vm.py (+605/-3)
To merge this branch: bzr merge lp:~nuclearbob/utah/reorg
Reviewer Review Type Date Requested Status
Javier Collado (community) Approve
Review via email: mp+139596@code.launchpad.net

Description of the change

This branch reorganizes the provisioning code.

SSHMixin now lives in ssh.py

TinySQLiteInventory was moved into vm.py since it's only useful for VMs.

vmtools support was removed. We can add uvt support later if we need it.

Everything from libvirtvm.py was moved into vm.py since we don't support any other VM types.

ManualBaremetalSQLiteInventory was moved into an inventory.py file under baremetal.

The baremetal directory is now several packages: utah-baremetal, utah-cobbler, and utah-bamboofeeder. utah-cobbler and utah-bamboofeeder depend on utah-baremetal, and utah-all installs every utah package (including the parser.)

I've built and installed the packages and tested that VM provisioning still works as expected. I'm still working out a way to test a new package on physical installs without disrupting the ongoing tests in magners, but we can test those packages separately later if we need to. The code in them hasn't changed for the most part, it's just moved around a bit.

To post a comment you must log in.
Revision history for this message
Javier Collado (javier.collado) wrote :

Looks good. I've checked that the packages and the documentation build
correctly and that the pass run list succeeds.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'conf/config'
2--- conf/config 2012-07-11 13:56:31 +0000
3+++ conf/config 2012-12-13 00:38:21 +0000
4@@ -1,1 +1,1 @@
5-IdentityFile %d/.ssh/%u
6+IdentityFile %d/.ssh/utah
7
8=== removed file 'conf/utah/uqt-vm-tools.conf'
9--- conf/utah/uqt-vm-tools.conf 2012-11-07 19:52:52 +0000
10+++ conf/utah/uqt-vm-tools.conf 1970-01-01 00:00:00 +0000
11@@ -1,60 +0,0 @@
12-# list of all active releases (included devel)
13-vm_release_list="hardy lucid oneiric precise quantal raring"
14-
15-# used by vm-repo (ie 'umt repo' puts stuff in /var/www/debs/testing/..., so
16-# vm_repo_url should be the URL to those files. The IP of the host is by
17-# default 192.168.122.1, and guests are 192.168.122.2-254.
18-vm_repo_url="http://192.168.122.1/debs/testing"
19-
20-# vm-tools specific settings
21-vm_path="/var/lib/utah/vm" # where to store the VM images
22-vm_aptproxy="" # set if you want to use a local proxy (like, say, apt-cacher-ng)
23-vm_mirror="http://us.archive.ubuntu.com/ubuntu"
24-vm_security_mirror="" # set if want to use a local mirror for security
25-vm_mirror_host="us.archive.ubuntu.com" # Used with the mini iso
26-vm_mirror_dir="/ubuntu" # Used with the mini iso
27-vm_dir_iso="/var/cache/utah/iso" # set to directory containing .iso images
28-vm_dir_iso_cache="/var/cache/utah/iso/cache" # set to directory for preseeded iso cache
29-vm_image_size="8" # size in GB of vm-new disk images
30-vm_memory="512" # 384 is needed for desktops, 256 for servers
31-vm_ssh_key=~/.ssh/utah.pub # defaults to $HOME/.ssh/id_rsa.pub
32-vm_connect="qemu:///system"
33-vm_flavor="" # blank for default, set to override (eg 'rt')
34-vm_archs="i386 amd64" # architectures to use when using '-p PREFIX'
35- # with some commands
36-vm_extra_packages="python-yaml bzr git" # list of packages to also
37- # install via postinstall.sh
38-vm_username="utah" # defaults to your username (`whoami`)
39-vm_password="ubuntu" # defaults to "ubuntu"
40-vm_latecmd="" # allows specifying an additional late command
41-
42-vm_root_size="4096" # Used by deprecated vm-new-vmbuilder tool
43-vm_swap_size="1024" # Used by deprecated vm-new-vmbuilder tool
44-
45-# vm-iso specific settings (also uses vm-tools settings)
46-vm_iso_ndisks="1" # number of disks
47-vm_iso_vcpus="1" # number of virtual CPUs
48-vm_iso_ostype="linux"
49-vm_iso_osvariant="ubuntuLucid"
50-# see 'man virt-install' for details on 'sparse', 'format' and 'cache'
51-vm_iso_fully_allocate="yes" # fully allocate the disk image (usu. faster)
52-vm_iso_disk_format="qcow2" # raw, qcow2, vmdk, etc
53-vm_iso_disk_cache="none" # none, writethrough, writeback
54-
55-# vm-new locale
56-vm_locale="en_US.UTF-8"
57-
58-# vm-new keyboard layout
59-vm_setkeyboard="false" # set to "true" to enable the custom settings below
60-vm_xkbmodel="pc105"
61-vm_xkblayout="ca"
62-vm_xkbvariant=""
63-vm_xkboptions="lv3:ralt_switch"
64-
65-# Use an alternate viewer such as gvncviewer or xtightvncviewer. Defaults to
66-# virt-viewer if unset. You can specify arguments to the viewer here.
67-#vm_viewer="xvnc4viewer"
68-#vm_viewer_args=""
69-
70-# Set to 'no' to disable '.local' mDNS (avahi) lookups for VMs
71-vm_host_use_avahi="yes"
72
73=== modified file 'debian/control'
74--- debian/control 2012-12-11 14:28:43 +0000
75+++ debian/control 2012-12-13 00:38:21 +0000
76@@ -18,10 +18,34 @@
77 python-netifaces, python-paramiko, python-psutil,
78 utah-client (=${binary:Version})
79 Recommends: dl-ubuntu-test-iso, kvm
80-Suggests: cobbler, u-boot-tools, vm-tools
81 Description: Ubuntu Test Automation Harness
82 Automation framework for testing in Ubuntu
83
84+Package: utah-all
85+Architecture: all
86+Depends: ${misc:Depends}, ${python:Depends}, utah-bamboofeeder, utah-cobbler,
87+ utah-parser
88+Description: Ubuntu Test Automation Harness Complete Package
89+ Automation framework for testing in Ubuntu, all sections
90+
91+Package: utah-bamboofeeder
92+Architecture: all
93+Depends: ${misc:Depends}, ${python:Depends}, u-boot-tools, utah-baremetal
94+Description: Ubuntu Test Automation Harness Bamboo Feeder Support
95+ Automation framework for testing in Ubuntu, bamboo feeder portion
96+
97+Package: utah-baremetal
98+Architecture: all
99+Depends: ${misc:Depends}, ${python:Depends}, utah
100+Description: Ubuntu Test Automation Harness Bare Metal Support
101+ Automation framework for testing in Ubuntu, bare metal portion
102+
103+Package: utah-cobbler
104+Architecture: all
105+Depends: ${misc:Depends}, ${python:Depends}, cobbler, utah-baremetal
106+Description: Ubuntu Test Automation Harness Cobbler Support
107+ Automation framework for testing in Ubuntu, cobbler portion
108+
109 Package: utah-client
110 Architecture: all
111 Depends: ${misc:Depends}, ${python:Depends},
112
113=== modified file 'debian/rules'
114--- debian/rules 2012-11-30 14:03:47 +0000
115+++ debian/rules 2012-12-13 00:38:21 +0000
116@@ -22,6 +22,7 @@
117 # utah should only install the provisioning subdirectory, and utah-client should install everything else
118 # except parser.py, which is now its own package
119 # and __init__.py, which is now in the common package
120+ # The baremetal subdirectory in the provisioning directory is now 3 packages
121 # If we just use dh_auto_install, all packages will get everything
122 # We start by building the whole thing
123 set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --install-layout=deb --root=$(CURDIR); done
124@@ -37,6 +38,8 @@
125 rm -r build/*/utah/*
126 # And we put provisioning back
127 mv provisioning build/*/utah
128+ # But remove baremetal
129+ mv build/*/utah/provisioning/baremetal .
130 # Now we install just the provisioning directory, again using --skip-build into the utah package tree
131 set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --skip-build --install-layout=deb --root=$(CURDIR)/debian/utah; done
132 # And now we put back the parser and install that
133@@ -47,24 +50,48 @@
134 rm -r build/*/utah/*
135 mv __init__.py build/*/utah
136 set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --skip-build --install-layout=deb --root=$(CURDIR)/debian/utah-common; done
137+ # Now we install just baremetal
138+ rm -r build/*/utah/*
139+ mkdir provisioning
140+ mv baremetal provisioning
141+ mv provisioning build/*/utah
142+ # But without cobbler
143+ mv build/*/utah/provisioning/baremetal/cobbler.py .
144+ # And without bamboofeeder
145+ mv build/*/utah/provisioning/baremetal/bamboofeeder.py .
146+ set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --skip-build --install-layout=deb --root=$(CURDIR)/debian/utah-baremetal; done
147+ # Now just cobbler
148+ rm -r build/*/utah/provisioning/baremetal/*
149+ mv cobbler.py build/*/utah/provisioning/baremetal
150+ set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --skip-build --install-layout=deb --root=$(CURDIR)/debian/utah-cobbler; done
151+ # Now just bamboofeeder
152+ rm -r build/*/utah/provisioning/baremetal/*
153+ mv bamboofeeder.py build/*/utah/provisioning/baremetal
154+ set -e && for pyvers in $(PYVERS); do python$$pyvers setup.py install --skip-build --install-layout=deb --root=$(CURDIR)/debian/utah-bamboofeeder; done
155 # Since the client changes names from client.py to utah, we can't use utah-client.install for that
156 # We also need to make our directory since we do this before dh_auto_install
157 mkdir -p $(CURDIR)/debian/utah-client/usr/bin
158 cp -aL client.py $(CURDIR)/debian/utah-client/usr/bin/utah
159 # phoenix.py needs to be in here to lose the .py
160 cp -aL utah/client/phoenix.py $(CURDIR)/debian/utah-client/usr/bin/phoenix
161- # Since utah and utah-client both got installed using python distutils, they both get an egg info directory
162- # This will conflict if we leave it in both of them, so we remove it from the client
163- rm -r $(CURDIR)/debian/utah-client/usr/lib/python*/dist-packages/utah-*.egg-info
164- # Let's remove it from the parser as well
165- rm -r $(CURDIR)/debian/utah-parser/usr/lib/python*/dist-packages/utah-*.egg-info
166- # We'll remove it from the server and leave it in the client package
167- rm -r $(CURDIR)/debian/utah/usr/lib/python*/dist-packages/utah-*.egg-info
168+ # Since all packages get installed using python distutils, they all get an egg info directory
169+ # This will conflict if we leave it in all of them, so we remove it from everything but common
170+ for egg in $$(ls -d $(CURDIR)/debian/utah*/usr/lib/python*/dist-packages/utah-*.egg-info | grep -v "utah-common");\
171+ do rm -r $$egg;\
172+ done
173 # We want to symlink the utah example scripts into /usr/bin, and we need a directory for that
174 mkdir -p $(CURDIR)/debian/utah/usr/bin
175+ mkdir -p $(CURDIR)/debian/utah-cobbler/usr/bin
176+ mkdir -p $(CURDIR)/debian/utah-bamboofeeder/usr/bin
177 for script in $$(find examples -type f -name "*.py" -printf "%f\n"); do\
178 if [ ! -h "$(CURDIR)/debian/utah/usr/bin/$$script" ]; then\
179- ln -s ../share/utah/examples/$$script $(CURDIR)/debian/utah/usr/bin/$$script;\
180+ if [ "$$script" = "run_test_cobbler.py" ]; then\
181+ ln -s ../share/utah/examples/$$script $(CURDIR)/debian/utah-cobbler/usr/bin/$$script;\
182+ elif [ "$$script" = "run_test_bamboo_feeder.py" ]; then\
183+ ln -s ../share/utah/examples/$$script $(CURDIR)/debian/utah-bamboofeeder/usr/bin/$$script;\
184+ else\
185+ ln -s ../share/utah/examples/$$script $(CURDIR)/debian/utah/usr/bin/$$script;\
186+ fi;\
187 fi;\
188 done
189 dh_auto_install
190
191=== modified file 'debian/utah.postinst'
192--- debian/utah.postinst 2012-10-31 15:57:11 +0000
193+++ debian/utah.postinst 2012-12-13 00:38:21 +0000
194@@ -77,7 +77,6 @@
195
196 usersetup
197
198- ln -sf /etc/utah/uqt-vm-tools.conf ~utah/.uqt-vm-tools.conf
199 ln -sf /etc/utah/shell-profile ~utah/.profile
200
201 if ! ([ -f ~utah/.ssh/utah ] && [ -f ~utah/.ssh/utah.pub ])
202
203=== modified file 'docs/source/reference.rst'
204--- docs/source/reference.rst 2012-11-30 09:36:56 +0000
205+++ docs/source/reference.rst 2012-12-13 00:38:21 +0000
206@@ -88,6 +88,9 @@
207 .. automodule:: utah.provisioning.provisioning
208 :members:
209
210+.. automodule:: utah.provisioning.ssh
211+ :members:
212+
213 .. automodule:: utah.provisioning.exceptions
214 :members:
215
216@@ -96,9 +99,15 @@
217
218 .. automodule:: utah.provisioning.baremetal
219
220+.. automodule:: utah.provisioning.baremetal.inventory
221+ :members:
222+
223 .. automodule:: utah.provisioning.baremetal.cobbler
224 :members:
225
226+.. automodule:: utah.provisioning.baremetal.bamboofeeder
227+ :members:
228+
229 .. automodule:: utah.provisioning.baremetal.exceptions
230 :members:
231
232@@ -124,9 +133,6 @@
233 .. automodule:: utah.provisioning.vm.exceptions
234 :members:
235
236-.. automodule:: utah.provisioning.vm.libvirtvm
237- :members:
238-
239 .. automodule:: utah.provisioning.vm.vm
240 :members:
241
242
243=== modified file 'examples/run_install_test.py'
244--- examples/run_install_test.py 2012-12-08 02:10:12 +0000
245+++ examples/run_install_test.py 2012-12-13 00:38:21 +0000
246@@ -22,8 +22,7 @@
247 from utah.exceptions import UTAHException
248 from utah.url import url_argument
249 from utah.group import check_user_group, print_group_error_message
250-from utah.provisioning.inventory.sqlite import TinySQLiteInventory
251-from utah.provisioning.vm.libvirtvm import CustomVM
252+from utah.provisioning.vm.vm import CustomVM, TinySQLiteInventory
253 from utah.run import run_tests
254
255
256
257=== modified file 'examples/run_test_bamboo_feeder.py'
258--- examples/run_test_bamboo_feeder.py 2012-12-11 08:41:24 +0000
259+++ examples/run_test_bamboo_feeder.py 2012-12-13 00:38:21 +0000
260@@ -23,7 +23,8 @@
261 from utah.exceptions import UTAHException
262 from utah.group import check_user_group, print_group_error_message
263 from utah.provisioning.baremetal.bamboofeeder import BambooFeederMachine
264-from utah.provisioning.inventory.sqlite import ManualBaremetalSQLiteInventory
265+from utah.provisioning.baremetal.inventory import \
266+ ManualBaremetalSQLiteInventory
267 from utah.run import run_tests
268 from utah.url import url_argument
269
270
271=== modified file 'examples/run_test_cobbler.py'
272--- examples/run_test_cobbler.py 2012-12-11 18:56:17 +0000
273+++ examples/run_test_cobbler.py 2012-12-13 00:38:21 +0000
274@@ -21,7 +21,9 @@
275 from utah import config
276 from utah.exceptions import UTAHException
277 from utah.group import check_user_group, print_group_error_message
278-from utah.provisioning.inventory.sqlite import ManualBaremetalSQLiteInventory
279+from utah.provisioning.baremetal.cobbler import CobblerMachine
280+from utah.provisioning.baremetal.inventory import \
281+ ManualBaremetalSQLiteInventory
282 from utah.run import run_tests
283 from utah.url import url_argument
284
285@@ -105,7 +107,8 @@
286 kw[arg] = value
287 if getattr(args, 'type') is not None:
288 kw['installtype'] = args.type
289- machine = inventory.request(clean=(not args.no_destroy),
290+ machine = inventory.request(CobblerMachine,
291+ clean=(not args.no_destroy),
292 debug=args.debug, dlpercentincrement=10,
293 name=args.name, new=True, **kw)
294 exitstatus, locallogs = run_tests(args, machine)
295
296=== modified file 'examples/run_test_vm.py'
297--- examples/run_test_vm.py 2012-12-10 19:18:18 +0000
298+++ examples/run_test_vm.py 2012-12-13 00:38:21 +0000
299@@ -21,7 +21,7 @@
300 from utah import config
301 from utah.exceptions import UTAHException
302 from utah.group import check_user_group, print_group_error_message
303-from utah.provisioning.inventory.sqlite import TinySQLiteInventory
304+from utah.provisioning.vm.vm import TinySQLiteInventory
305 from utah.run import run_tests
306
307
308
309=== modified file 'examples/utah-user-setup.sh'
310--- examples/utah-user-setup.sh 2012-07-19 20:00:02 +0000
311+++ examples/utah-user-setup.sh 2012-12-13 00:38:21 +0000
312@@ -63,13 +63,6 @@
313
314 set +e
315
316-CMD=/usr/share/utah/examples/vmtools-user-setup.sh
317-if [ -f "$CMD" ]
318-then
319- echo "Setting up vm-tools config"
320- keepgoing $CMD $AUTO
321-fi
322-
323 if [ "$LOGOUT" ]
324 then
325 echo "User groups were updated"
326
327=== removed file 'examples/vmtools-user-setup.sh'
328--- examples/vmtools-user-setup.sh 2012-07-19 20:00:02 +0000
329+++ examples/vmtools-user-setup.sh 1970-01-01 00:00:00 +0000
330@@ -1,59 +0,0 @@
331-#!/bin/bash
332-
333-set -e
334-
335-function keepgoing
336-{
337- echo "Going to run:"
338- echo "$@"
339- if [ "$AUTO" ]
340- then
341- $@
342- else
343- echo "Hit enter to continue, enter any text to stop"
344- read ENTRY
345- if [ -n "$ENTRY" ]
346- then
347- exit 1
348- else
349- echo $@ | bash
350-# echo "$@"
351-# $@
352- fi
353- fi
354-}
355-
356-if (echo "$@" | grep -iq "a")
357-then
358- export AUTO="auto"
359-else
360- export AUTO=""
361-fi
362-
363-if ! [ -f ~/.uqt-vm-tools.conf ]
364-then
365- echo "Running vm-new to create ~/.uqv-vm-tools.conf"
366- if [ "$AUTO" ]
367- then
368- CMD="echo \"y\" | vm-new"
369- else
370- CMD=vm-new
371- fi
372- keepgoing $CMD
373-fi
374-
375-SSHKEY=$(echo -e "import utah.config\nprint(utah.config.sshpublickey)" | python)
376-if ! (grep -q "^vm_ssh_key=$SSHKEY" ~/.uqt-vm-tools.conf)
377-then
378- CMD="sed 's@^vm_ssh_key=[^#]*#@vm_ssh_key=$SSHKEY #@' -i ~/.uqt-vm-tools.conf"
379- echo "Adding SSH key to config file"
380- keepgoing $CMD
381-fi
382-
383-if ! (grep "^vm_extra_packages=" ~/.uqt-vm-tools.conf | grep "python-yaml" | grep "bzr" | grep -q "git")
384-then
385- CMD="sed 's/^vm_extra_packages=\"/vm_extra_packages=\"python-yaml bzr git /' -i ~/.uqt-vm-tools.conf"
386- echo "Adding client dependencies to config file"
387- keepgoing $CMD
388-fi
389-
390
391=== modified file 'utah/provisioning/baremetal/bamboofeeder.py'
392--- utah/provisioning/baremetal/bamboofeeder.py 2012-12-11 09:37:12 +0000
393+++ utah/provisioning/baremetal/bamboofeeder.py 2012-12-13 00:38:21 +0000
394@@ -25,13 +25,13 @@
395 import tempfile
396
397 from utah import config
398+from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
399+from utah.provisioning.baremetal.power import PowerMixin
400 from utah.provisioning.provisioning import (
401 CustomInstallMixin,
402 Machine,
403- SSHMixin,
404 )
405-from utah.provisioning.baremetal.power import PowerMixin
406-from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
407+from utah.provisioning.ssh import SSHMixin
408 from utah.retry import retry
409
410
411
412=== modified file 'utah/provisioning/baremetal/cobbler.py'
413--- utah/provisioning/baremetal/cobbler.py 2012-12-11 18:38:12 +0000
414+++ utah/provisioning/baremetal/cobbler.py 2012-12-13 00:38:21 +0000
415@@ -26,13 +26,13 @@
416 import time
417
418 from utah import config
419+from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
420+from utah.provisioning.baremetal.power import PowerMixin
421 from utah.provisioning.provisioning import (
422 CustomInstallMixin,
423 Machine,
424- SSHMixin,
425 )
426-from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
427-from utah.provisioning.baremetal.power import PowerMixin
428+from utah.provisioning.ssh import SSHMixin
429 from utah.retry import retry
430
431
432
433=== added file 'utah/provisioning/baremetal/inventory.py'
434--- utah/provisioning/baremetal/inventory.py 1970-01-01 00:00:00 +0000
435+++ utah/provisioning/baremetal/inventory.py 2012-12-13 00:38:21 +0000
436@@ -0,0 +1,153 @@
437+# Ubuntu Testing Automation Harness
438+# Copyright 2012 Canonical Ltd.
439+
440+# This program is free software: you can redistribute it and/or modify it
441+# under the terms of the GNU General Public License version 3, as published
442+# by the Free Software Foundation.
443+
444+# This program is distributed in the hope that it will be useful, but
445+# WITHOUT ANY WARRANTY; without even the implied warranties of
446+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
447+# PURPOSE. See the GNU General Public License for more details.
448+
449+# You should have received a copy of the GNU General Public License along
450+# with this program. If not, see <http://www.gnu.org/licenses/>.
451+
452+"""
453+Provide inventory functions specific to bare metal deployments.
454+"""
455+
456+import os
457+import psutil
458+from utah.provisioning.inventory.exceptions import \
459+ UTAHProvisioningInventoryException
460+from utah.provisioning.inventory.sqlite import SQLiteInventory
461+
462+
463+class ManualBaremetalSQLiteInventory(SQLiteInventory):
464+ """
465+ Keep an inventory of manually entered machines.
466+ All columns other than machineid, name, and state are assumed to be
467+ arguments for system creation (i.e., with cobbler).
468+ """
469+ # TODO: rename this
470+ def __init__(self, db='~/.utah-baremetal-inventory',
471+ lockfile='~/.utah-baremetal-lock', *args, **kw):
472+ db = os.path.expanduser(db)
473+ lockfile = os.path.expanduser(lockfile)
474+ if not os.path.isfile(db):
475+ raise UTAHProvisioningInventoryException(
476+ 'No machine database found at ' + db)
477+ super(ManualBaremetalSQLiteInventory, self).__init__(*args, db=db,
478+ lockfile=lockfile,
479+ **kw)
480+ machines_count = (self.connection
481+ .execute('SELECT COUNT(*) FROM machines')
482+ .fetchall()[0][0])
483+ if machines_count == 0:
484+ raise UTAHProvisioningInventoryException('No machines in database')
485+ self.machines = []
486+
487+ def request(self, machinetype, name=None, *args, **kw):
488+ query = 'SELECT * FROM machines'
489+ queryvars = []
490+ if name is not None:
491+ query += ' WHERE name=?'
492+ queryvars.append(name)
493+ result = self.connection.execute(query, queryvars).fetchall()
494+ if result is None:
495+ raise UTAHProvisioningInventoryException(
496+ 'No machines meet criteria')
497+ else:
498+ for minfo in result:
499+ machineinfo = dict(minfo)
500+ if machineinfo['state'] == 'available':
501+ return self._take(machineinfo, machinetype, *args, **kw)
502+ for minfo in result:
503+ machineinfo = dict(minfo)
504+ pid = machineinfo['pid']
505+ try:
506+ if not (psutil.pid_exists(pid) and ('utah' in
507+ ' '.join(psutil.Process(pid).cmdline)
508+ or 'run_test' in
509+ ' '.join(psutil.Process(pid).cmdline))):
510+ return self._take(machineinfo, machinetype,
511+ *args, **kw)
512+ except ValueError:
513+ continue
514+
515+ raise UTAHProvisioningInventoryException(
516+ 'All machines meeting criteria are currently unavailable')
517+
518+ def _take(self, machineinfo, machinetype, *args, **kw):
519+ machineid = machineinfo.pop('machineid')
520+ name = machineinfo.pop('name')
521+ state = machineinfo.pop('state')
522+ machineinfo.pop('pid')
523+ update = self.connection.execute(
524+ "UPDATE machines SET pid=?, state='provisioned' WHERE machineid=?"
525+ "AND state=?",
526+ [os.getpid(), machineid, state]).rowcount
527+ if update == 1:
528+ machine = machinetype(*args, inventory=self,
529+ machineinfo=machineinfo, name=name, **kw)
530+ self.machines.append(machine)
531+ return machine
532+ elif update == 0:
533+ raise UTAHProvisioningInventoryException(
534+ 'Machine was requested by another process '
535+ 'before we could request it')
536+ elif update > 1:
537+ raise UTAHProvisioningInventoryException(
538+ 'Multiple machines exist '
539+ 'matching those criteria; '
540+ 'database ' + self.db + ' may be corrupt')
541+ else:
542+ raise UTAHProvisioningInventoryException(
543+ 'Negative rowcount returned '
544+ 'when attempting to request machine')
545+
546+ def release(self, machine=None, name=None):
547+ if machine is not None:
548+ name = machine.name
549+ if name is None:
550+ raise UTAHProvisioningInventoryException(
551+ 'name required to release a machine')
552+ query = "UPDATE machines SET state='available' WHERE name=?"
553+ queryvars = [name]
554+ update = self.connection.execute(query, queryvars).rowcount
555+ if update == 1:
556+ if machine is not None:
557+ if machine in self.machines:
558+ self.machines.remove(machine)
559+ elif update == 0:
560+ raise UTAHProvisioningInventoryException(
561+ 'SERIOUS ERROR: Another process released this machine '
562+ 'before we could, which means two processes provisioned '
563+ 'the same machine simultaneously')
564+ elif update > 1:
565+ raise UTAHProvisioningInventoryException(
566+ 'Multiple machines exist matching those criteria; '
567+ 'database ' + self.db + ' may be corrupt')
568+ else:
569+ raise UTAHProvisioningInventoryException(
570+ 'Negative rowcount returned '
571+ 'when attempting to release machine')
572+
573+ # Here is how I currently create the database:
574+ # CREATE TABLE machines (machineid INTEGER PRIMARY KEY,
575+ # name TEXT NOT NULL UNIQUE,
576+ # state TEXT default 'available',
577+ # pid INT,
578+ # [mac-address] TEXT NOT NULL UNIQUE,
579+ # [power-address] TEXT DEFAULT '10.97.0.13',
580+ # [power-id] TEXT,
581+ # [power-user] TEXT DEFAULT 'ubuntu',
582+ # [power-pass] TEXT DEFAULT 'ubuntu',
583+ # [power-type] TEXT DEFAULT 'sentryswitch_cdu');
584+
585+ # Here is how I currently populate the database:
586+ # INSERT INTO machines (name, [mac-address], [power-id])
587+ # VALUES ('acer-veriton-01-Pete', 'd0:27:88:9f:73:ce', 'Veriton_1');
588+ # INSERT INTO machines (name, [mac-address], [power-id])
589+ # VALUES ('acer-veriton-02-Pete', 'd0:27:88:9b:84:5b', 'Veriton_2');
590
591=== modified file 'utah/provisioning/inventory/sqlite.py'
592--- utah/provisioning/inventory/sqlite.py 2012-12-11 18:45:40 +0000
593+++ utah/provisioning/inventory/sqlite.py 2012-12-13 00:38:21 +0000
594@@ -17,12 +17,7 @@
595
596 import sqlite3
597 import os
598-import psutil
599-from utah.provisioning.inventory.exceptions import \
600- UTAHProvisioningInventoryException
601 from utah.provisioning.inventory.inventory import Inventory
602-from utah.provisioning.vm.libvirtvm import CustomVM
603-from utah.provisioning.baremetal.cobbler import CobblerMachine
604
605
606 class SQLiteInventory(Inventory):
607@@ -41,181 +36,3 @@
608 def delete(self):
609 os.unlink(self.db)
610 super(SQLiteInventory, self).delete()
611-
612-
613-class TinySQLiteInventory(SQLiteInventory):
614- """
615- Tiny SQLite inventory that implements request, release, and destroy.
616-
617- No authentication or conflict checking currently exists.
618- """
619- def __init__(self, *args, **kw):
620- """
621- Initialize simple database.
622- """
623- super(TinySQLiteInventory, self).__init__(*args, **kw)
624- self.connection.execute(
625- 'CREATE TABLE IF NOT EXISTS '
626- 'machines(machineid INTEGER PRIMARY KEY, state TEXT)')
627-
628- def request(self, machinetype=CustomVM, *args, **kw):
629- """
630- Takes a Machine class as machinetype, and passes the newly generated
631- machineid along with all other arguments to that class's constructor,
632- returning the resulting object.
633- """
634- cursor = self.connection.cursor()
635- cursor.execute("INSERT INTO machines (state) VALUES ('provisioned')")
636- machineid = cursor.lastrowid
637- return machinetype(machineid=machineid, *args, **kw)
638-
639- def release(self, machineid):
640- """
641- Updates the database to indicate the machine is available.
642- """
643- if self.connection.execute(
644- "UPDATE machines SET state='available' WHERE machineid=?",
645- [machineid]):
646- return True
647- else:
648- return False
649-
650- def destroy(self, machineid):
651- """
652- Updates the database to indicate the machine is destroyed, but does not
653- destroy the machine.
654- """
655- if self.connection.execute(
656- "UPDATE machines SET state='destroyed' ""WHERE machineid=?",
657- [machineid]):
658- return True
659- else:
660- return False
661-
662-
663-class ManualBaremetalSQLiteInventory(SQLiteInventory):
664- """
665- Keep an inventory of manually entered machines.
666- All columns other than machineid, name, and state are assumed to be
667- arguments for system creation (i.e., with cobbler).
668- """
669- # TODO: rename this
670- def __init__(self, db='~/.utah-baremetal-inventory',
671- lockfile='~/.utah-baremetal-lock', *args, **kw):
672- db = os.path.expanduser(db)
673- lockfile = os.path.expanduser(lockfile)
674- if not os.path.isfile(db):
675- raise UTAHProvisioningInventoryException(
676- 'No machine database found at ' + db)
677- super(ManualBaremetalSQLiteInventory, self).__init__(*args, db=db,
678- lockfile=lockfile,
679- **kw)
680- machines_count = (self.connection
681- .execute('SELECT COUNT(*) FROM machines')
682- .fetchall()[0][0])
683- if machines_count == 0:
684- raise UTAHProvisioningInventoryException('No machines in database')
685- self.machines = []
686-
687- def request(self, machinetype=CobblerMachine, name=None, *args, **kw):
688- query = 'SELECT * FROM machines'
689- queryvars = []
690- if name is not None:
691- query += ' WHERE name=?'
692- queryvars.append(name)
693- result = self.connection.execute(query, queryvars).fetchall()
694- if result is None:
695- raise UTAHProvisioningInventoryException(
696- 'No machines meet criteria')
697- else:
698- for minfo in result:
699- machineinfo = dict(minfo)
700- if machineinfo['state'] == 'available':
701- return self._take(machineinfo, *args, **kw)
702- for minfo in result:
703- machineinfo = dict(minfo)
704- pid = machineinfo['pid']
705- try:
706- if not (psutil.pid_exists(pid) and ('utah' in
707- ' '.join(psutil.Process(pid).cmdline)
708- or 'run_test' in
709- ' '.join(psutil.Process(pid).cmdline))):
710- return self._take(machineinfo, *args, **kw)
711- except ValueError:
712- continue
713-
714- raise UTAHProvisioningInventoryException(
715- 'All machines meeting criteria are currently unavailable')
716-
717- def _take(self, machineinfo, *args, **kw):
718- machineid = machineinfo.pop('machineid')
719- name = machineinfo.pop('name')
720- state = machineinfo.pop('state')
721- machineinfo.pop('pid')
722- update = self.connection.execute(
723- "UPDATE machines SET pid=?, state='provisioned' WHERE machineid=?"
724- "AND state=?",
725- [os.getpid(), machineid, state]).rowcount
726- if update == 1:
727- machine = CobblerMachine(*args, inventory=self,
728- machineinfo=machineinfo, name=name, **kw)
729- self.machines.append(machine)
730- return machine
731- elif update == 0:
732- raise UTAHProvisioningInventoryException(
733- 'Machine was requested by another process '
734- 'before we could request it')
735- elif update > 1:
736- raise UTAHProvisioningInventoryException(
737- 'Multiple machines exist '
738- 'matching those criteria; '
739- 'database ' + self.db + ' may be corrupt')
740- else:
741- raise UTAHProvisioningInventoryException(
742- 'Negative rowcount returned '
743- 'when attempting to request machine')
744-
745- def release(self, machine=None, name=None):
746- if machine is not None:
747- name = machine.name
748- if name is None:
749- raise UTAHProvisioningInventoryException(
750- 'name required to release a machine')
751- query = "UPDATE machines SET state='available' WHERE name=?"
752- queryvars = [name]
753- update = self.connection.execute(query, queryvars).rowcount
754- if update == 1:
755- if machine is not None:
756- if machine in self.machines:
757- self.machines.remove(machine)
758- elif update == 0:
759- raise UTAHProvisioningInventoryException(
760- 'SERIOUS ERROR: Another process released this machine '
761- 'before we could, which means two processes provisioned '
762- 'the same machine simultaneously')
763- elif update > 1:
764- raise UTAHProvisioningInventoryException(
765- 'Multiple machines exist matching those criteria; '
766- 'database ' + self.db + ' may be corrupt')
767- else:
768- raise UTAHProvisioningInventoryException(
769- 'Negative rowcount returned '
770- 'when attempting to release machine')
771-
772- # Here is how I currently create the database:
773- # CREATE TABLE machines (machineid INTEGER PRIMARY KEY,
774- # name TEXT NOT NULL UNIQUE,
775- # state TEXT default 'available',
776- # pid TEXT,
777- # [mac-address] TEXT NOT NULL UNIQUE,
778- # [power-address] TEXT DEFAULT '10.97.0.13',
779- # [power-id] TEXT,
780- # [power-user] TEXT DEFAULT 'ubuntu',
781- # [power-pass] TEXT DEFAULT 'ubuntu',
782- # [power-type] TEXT DEFAULT 'sentryswitch_cdu');
783-
784- # Here is how I currently populate the database:
785- # INSERT INTO machines (name, [mac-address], [power-id])
786- # VALUES ('acer-veriton-01-Pete', 'd0:27:88:9f:73:ce', 'Veriton_1');
787- # INSERT INTO machines (name, [mac-address], [power-id])
788- # VALUES ('acer-veriton-02-Pete', 'd0:27:88:9b:84:5b', 'Veriton_2');
789
790=== modified file 'utah/provisioning/provisioning.py'
791--- utah/provisioning/provisioning.py 2012-12-11 09:50:08 +0000
792+++ utah/provisioning/provisioning.py 2012-12-13 00:38:21 +0000
793@@ -18,26 +18,24 @@
794 Functions here should apply to multiple machine types (VM, bare metal, etc.)
795 """
796
797+import apt.cache
798 import logging
799 import logging.handlers
800 import os
801 import pipes
802+import re
803 import shutil
804-import socket
805 import subprocess
806 import sys
807 import time
808 import urllib
809 import uuid
810-import paramiko
811-import re
812-import apt.cache
813
814 from glob import glob
815-from stat import S_ISDIR
816
817 import utah.timeout
818
819+from utah import config
820 from utah.commandstr import commandstr
821 from utah.iso import ISO
822 from utah.orderedcollections import (
823@@ -47,7 +45,6 @@
824 from utah.preseed import Preseed
825 from utah.provisioning.exceptions import UTAHProvisioningException
826 from utah.retry import retry
827-from utah import config
828
829
830 class Machine(object):
831@@ -679,243 +676,6 @@
832 'Try rerunning install.')
833
834
835-class SSHMixin(object):
836- """
837- Provide common commands for machines accessed via ssh.
838- """
839- def __init__(self, *args, **kwargs):
840- # Note: Since this is a mixin it doesn't expect any argument
841- # However, it calls super to initialize any other mixins in the mro
842- super(SSHMixin, self).__init__(*args, **kwargs)
843- ssh_client = paramiko.SSHClient()
844- ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
845- self.ssh_client = ssh_client
846-
847- def run(self, command, _quiet=None, root=False, timeout=None):
848- """
849- Run a command using ssh.
850- """
851- if isinstance(command, basestring):
852- commandstring = command
853- else:
854- commandstring = ' '.join(command)
855- if root:
856- user = 'root'
857- else:
858- user = config.user
859-
860- self.activecheck()
861- # Some commands expect run to return the output status of the command
862- # We're going to try the method described here:
863- # http://stackoverflow.com/questions/3562403/
864- # With additions from here:
865- # http://od-eon.com/blogs/
866- # stefan/automating-remote-commands-over-ssh-paramiko/
867- self.logger.debug('Connecting SSH')
868- self.ssh_client.connect(self.name,
869- username=user,
870- key_filename=config.sshprivatekey)
871-
872- self.logger.debug('Opening SSH session')
873- channel = self.ssh_client.get_transport().open_session()
874-
875- self.logger.info('Running command through SSH: ' + commandstring)
876- stdout = channel.makefile('rb')
877- stderr = channel.makefile_stderr('rb')
878- if timeout is None:
879- channel.exec_command(commandstring)
880- else:
881- utah.timeout.timeout(timeout, channel.exec_command, commandstring)
882- retval = channel.recv_exit_status()
883-
884- self.logger.debug('Closing SSH connection')
885- self.ssh_client.close()
886-
887- log_level = logging.DEBUG if retval == 0 else logging.WARNING
888- log_message = 'Return code: {}'.format(retval)
889- self.logger.log(log_level, log_message)
890-
891- self.logger.debug('Standard output follows:')
892- stdout_lines = stdout.readlines()
893- for line in stdout_lines:
894- self.logger.debug(line.strip())
895-
896- self.logger.debug('Standard error follows:')
897- stderr_lines = stderr.readlines()
898- for line in stderr_lines:
899- self.logger.debug(line.strip())
900-
901- return retval, ''.join(stdout_lines), ''.join(stderr_lines)
902-
903- def uploadfiles(self, files, target=os.path.normpath('/tmp/')):
904- """
905- Copy a file or list of files to a target directory on the machine.
906- """
907- if isinstance(files, basestring):
908- files = [files]
909-
910- self.activecheck()
911- self.ssh_client.connect(self.name,
912- username=config.user,
913- key_filename=config.sshprivatekey)
914- sftp_client = self.ssh_client.open_sftp()
915- failed = []
916- try:
917- for localpath in files:
918- if os.path.isfile(localpath):
919- self.logger.info('Uploading ' + localpath
920- + ' from the host to ' + target
921- + ' on the machine')
922- remotepath = os.path.join(target,
923- os.path.basename(localpath))
924- sftp_client.put(localpath, remotepath)
925- else:
926- failed.append(localpath)
927- finally:
928- sftp_client.close()
929- if len(failed) > 0:
930- err = UTAHProvisioningException('Files do not exist: '
931- + ' '.join(failed))
932- err.files = failed
933- raise err
934-
935- def downloadfiles(self, files, target=os.path.normpath('/tmp/')):
936- """
937- Copy a file or list of files from the machine to a target directory on
938- the local system.
939- """
940- # TODO: check for directories and recurse into them
941- if isinstance(files, basestring):
942- files = [files]
943-
944- self.activecheck()
945- self.ssh_client.connect(self.name,
946- username=config.user,
947- key_filename=config.sshprivatekey)
948- sftp_client = self.ssh_client.open_sftp()
949- if os.path.isdir(target):
950- get_localpath = lambda remotepath: \
951- os.path.join(target, os.path.basename(remotepath))
952- else:
953- get_localpath = lambda remotepath: target
954-
955- try:
956- for remotepath in files:
957- localpath = get_localpath(remotepath)
958- self.logger.info('Downloading ' + remotepath
959- + ' from the machine to ' + target
960- + ' on the host')
961- sftp_client.get(remotepath, localpath)
962- finally:
963- sftp_client.close()
964-
965- def downloadfilesrecursive(self, files, target=os.path.normpath('/tmp/')):
966- """
967- Recursively copy all files in files to the target directory target.
968- """
969- self.activecheck()
970- self.ssh_client.connect(self.name,
971- username=config.user,
972- key_filename=config.sshprivatekey)
973- sftp_client = self.ssh_client.open_sftp()
974- myfiles = []
975-
976- if isinstance(files, basestring):
977- files = [files]
978-
979- for myfile in files:
980- newtarget = os.path.join(target, os.path.basename(myfile))
981- if S_ISDIR(sftp_client.stat(myfile).st_mode):
982- self.logger.debug(myfile + ' is a directory, recursing')
983- if not os.path.isdir(newtarget):
984- self.logger.debug('Attempting to create ' + newtarget)
985- os.makedirs(newtarget)
986- myfiles = [os.path.join(myfile, x)
987- for x in sftp_client.listdir(myfile)]
988-# for basename in sftp_client.listdir(dirname):
989-# myfile = os.path.join(dirname, basename)
990-# if S_ISDIR(sftp_client.stat(myfile).st_mode):
991-# if not os.path.isdir(newtarget):
992-# os.makedirs(newtarget)
993-# self.downloadfilesrecursive(myfile, newtarget)
994-# else:
995-# myfiles.append(myfile)
996- self.downloadfilesrecursive(myfiles, newtarget)
997- else:
998- self.downloadfiles(myfile, newtarget)
999-
1000- def destroy(self, *args, **kw):
1001- """
1002- Clean up known hosts for machine.
1003- """
1004- # TODO: evaluate value of known_hosts with paramiko
1005- self.logger.info('Removing machine addresses '
1006- 'from ssh known_hosts file')
1007- addresses = [self.name]
1008- try:
1009- addresses.append(socket.gethostbyname(self.name))
1010- except socket.gaierror as err:
1011- if err.errno in [-2, -5]:
1012- self.logger.debug(self.name
1013- + ' is not resolvable, '
1014- + 'so not removing from known_hosts')
1015- else:
1016- raise err
1017-
1018- old_host_keys = self.ssh_client.get_host_keys()
1019- new_host_keys = paramiko.HostKeys()
1020- addresses = set(addresses)
1021- for address, key in old_host_keys.iteritems():
1022- # Skip keys so that they don't get added
1023- # into the new keys (i.e. they're removed)
1024- if address in addresses:
1025- continue
1026- new_host_keys[address] = key
1027- new_host_keys.save(config.sshknownhosts)
1028- self.ssh_client.close()
1029-
1030- super(SSHMixin, self).destroy(*args, **kw)
1031-
1032- def sshcheck(self, timeout=config.checktimeout):
1033- """
1034- Sleep for a while and check if the machine is available via ssh.
1035- Return a retryable exception if it is not.
1036- Intended for use with retry.
1037- """
1038- self.logger.info('Sleeping {timeout} seconds'
1039- .format(timeout=timeout))
1040- time.sleep(timeout)
1041- self.logger.info('Checking for ssh availability')
1042- try:
1043- self.ssh_client.connect(self.name,
1044- username=config.user,
1045- key_filename=config.sshprivatekey)
1046- except socket.error as err:
1047- raise UTAHProvisioningException(str(err), retry=True)
1048-
1049- def sshpoll(self, timeout=None,
1050- checktimeout=config.checktimeout, logmethod=None):
1051- """
1052- Run sshcheck over and over until timeout expires.
1053- """
1054- if timeout is None:
1055- timeout = self.boottimeout
1056- if logmethod is None:
1057- logmethod = self.logger.debug
1058- utah.timeout.timeout(timeout, retry, self.sshcheck, checktimeout,
1059- logmethod=logmethod)
1060-
1061- def activecheck(self):
1062- """
1063- Start the machine if needed, and check for SSH login.
1064- """
1065- self.logger.debug('Checking if machine is active')
1066- self.provisioncheck()
1067- if not self.active:
1068- self._start()
1069- self.sshcheck()
1070-
1071-
1072 class CustomInstallMixin(object):
1073 """
1074 Provide routines for unpacking necessary boot files from images,
1075
1076=== added file 'utah/provisioning/ssh.py'
1077--- utah/provisioning/ssh.py 1970-01-01 00:00:00 +0000
1078+++ utah/provisioning/ssh.py 2012-12-13 00:38:21 +0000
1079@@ -0,0 +1,261 @@
1080+# Ubuntu Testing Automation Harness
1081+# Copyright 2012 Canonical Ltd.
1082+
1083+# This program is free software: you can redistribute it and/or modify it
1084+# under the terms of the GNU General Public License version 3, as published
1085+# by the Free Software Foundation.
1086+
1087+# This program is distributed in the hope that it will be useful, but
1088+# WITHOUT ANY WARRANTY; without even the implied warranties of
1089+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
1090+# PURPOSE. See the GNU General Public License for more details.
1091+
1092+# You should have received a copy of the GNU General Public License along
1093+# with this program. If not, see <http://www.gnu.org/licenses/>.
1094+
1095+"""
1096+Provide a mixin class for machines with SSH support.
1097+"""
1098+
1099+import logging
1100+import os
1101+import paramiko
1102+import socket
1103+import time
1104+
1105+from stat import S_ISDIR
1106+
1107+import utah.timeout
1108+
1109+from utah import config
1110+from utah.provisioning.exceptions import UTAHProvisioningException
1111+from utah.retry import retry
1112+
1113+
1114+class SSHMixin(object):
1115+ """
1116+ Provide common commands for machines accessed via ssh.
1117+ """
1118+ def __init__(self, *args, **kwargs):
1119+ # Note: Since this is a mixin it doesn't expect any argument
1120+ # However, it calls super to initialize any other mixins in the mro
1121+ super(SSHMixin, self).__init__(*args, **kwargs)
1122+ ssh_client = paramiko.SSHClient()
1123+ ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
1124+ self.ssh_client = ssh_client
1125+
1126+ def run(self, command, _quiet=None, root=False, timeout=None):
1127+ """
1128+ Run a command using ssh.
1129+ """
1130+ if isinstance(command, basestring):
1131+ commandstring = command
1132+ else:
1133+ commandstring = ' '.join(command)
1134+ if root:
1135+ user = 'root'
1136+ else:
1137+ user = config.user
1138+
1139+ self.activecheck()
1140+ # Some commands expect run to return the output status of the command
1141+ # We're going to try the method described here:
1142+ # http://stackoverflow.com/questions/3562403/
1143+ # With additions from here:
1144+ # http://od-eon.com/blogs/
1145+ # stefan/automating-remote-commands-over-ssh-paramiko/
1146+ self.logger.debug('Connecting SSH')
1147+ self.ssh_client.connect(self.name,
1148+ username=user,
1149+ key_filename=config.sshprivatekey)
1150+
1151+ self.logger.debug('Opening SSH session')
1152+ channel = self.ssh_client.get_transport().open_session()
1153+
1154+ self.logger.info('Running command through SSH: ' + commandstring)
1155+ stdout = channel.makefile('rb')
1156+ stderr = channel.makefile_stderr('rb')
1157+ if timeout is None:
1158+ channel.exec_command(commandstring)
1159+ else:
1160+ utah.timeout.timeout(timeout, channel.exec_command, commandstring)
1161+ retval = channel.recv_exit_status()
1162+
1163+ self.logger.debug('Closing SSH connection')
1164+ self.ssh_client.close()
1165+
1166+ log_level = logging.DEBUG if retval == 0 else logging.WARNING
1167+ log_message = 'Return code: {}'.format(retval)
1168+ self.logger.log(log_level, log_message)
1169+
1170+ self.logger.debug('Standard output follows:')
1171+ stdout_lines = stdout.readlines()
1172+ for line in stdout_lines:
1173+ self.logger.debug(line.strip())
1174+
1175+ self.logger.debug('Standard error follows:')
1176+ stderr_lines = stderr.readlines()
1177+ for line in stderr_lines:
1178+ self.logger.debug(line.strip())
1179+
1180+ return retval, ''.join(stdout_lines), ''.join(stderr_lines)
1181+
1182+ def uploadfiles(self, files, target=os.path.normpath('/tmp/')):
1183+ """
1184+ Copy a file or list of files to a target directory on the machine.
1185+ """
1186+ if isinstance(files, basestring):
1187+ files = [files]
1188+
1189+ self.activecheck()
1190+ self.ssh_client.connect(self.name,
1191+ username=config.user,
1192+ key_filename=config.sshprivatekey)
1193+ sftp_client = self.ssh_client.open_sftp()
1194+ failed = []
1195+ try:
1196+ for localpath in files:
1197+ if os.path.isfile(localpath):
1198+ self.logger.info('Uploading ' + localpath
1199+ + ' from the host to ' + target
1200+ + ' on the machine')
1201+ remotepath = os.path.join(target,
1202+ os.path.basename(localpath))
1203+ sftp_client.put(localpath, remotepath)
1204+ else:
1205+ failed.append(localpath)
1206+ finally:
1207+ sftp_client.close()
1208+ if len(failed) > 0:
1209+ err = UTAHProvisioningException('Files do not exist: '
1210+ + ' '.join(failed))
1211+ err.files = failed
1212+ raise err
1213+
1214+ def downloadfiles(self, files, target=os.path.normpath('/tmp/')):
1215+ """
1216+ Copy a file or list of files from the machine to a target directory on
1217+ the local system.
1218+ """
1219+ # TODO: check for directories and recurse into them
1220+ if isinstance(files, basestring):
1221+ files = [files]
1222+
1223+ self.activecheck()
1224+ self.ssh_client.connect(self.name,
1225+ username=config.user,
1226+ key_filename=config.sshprivatekey)
1227+ sftp_client = self.ssh_client.open_sftp()
1228+ if os.path.isdir(target):
1229+ get_localpath = lambda remotepath: \
1230+ os.path.join(target, os.path.basename(remotepath))
1231+ else:
1232+ get_localpath = lambda remotepath: target
1233+
1234+ try:
1235+ for remotepath in files:
1236+ localpath = get_localpath(remotepath)
1237+ self.logger.info('Downloading ' + remotepath
1238+ + ' from the machine to ' + target
1239+ + ' on the host')
1240+ sftp_client.get(remotepath, localpath)
1241+ finally:
1242+ sftp_client.close()
1243+
1244+ def downloadfilesrecursive(self, files, target=os.path.normpath('/tmp/')):
1245+ """
1246+ Recursively copy all files in files to the target directory target.
1247+ """
1248+ self.activecheck()
1249+ self.ssh_client.connect(self.name,
1250+ username=config.user,
1251+ key_filename=config.sshprivatekey)
1252+ sftp_client = self.ssh_client.open_sftp()
1253+ myfiles = []
1254+
1255+ if isinstance(files, basestring):
1256+ files = [files]
1257+
1258+ for myfile in files:
1259+ newtarget = os.path.join(target, os.path.basename(myfile))
1260+ if S_ISDIR(sftp_client.stat(myfile).st_mode):
1261+ self.logger.debug(myfile + ' is a directory, recursing')
1262+ if not os.path.isdir(newtarget):
1263+ self.logger.debug('Attempting to create ' + newtarget)
1264+ os.makedirs(newtarget)
1265+ myfiles = [os.path.join(myfile, x)
1266+ for x in sftp_client.listdir(myfile)]
1267+ self.downloadfilesrecursive(myfiles, newtarget)
1268+ else:
1269+ self.downloadfiles(myfile, newtarget)
1270+
1271+ def destroy(self, *args, **kw):
1272+ """
1273+ Clean up known hosts for machine.
1274+ """
1275+ # TODO: evaluate value of known_hosts with paramiko
1276+ self.logger.info('Removing machine addresses '
1277+ 'from ssh known_hosts file')
1278+ addresses = [self.name]
1279+ try:
1280+ addresses.append(socket.gethostbyname(self.name))
1281+ except socket.gaierror as err:
1282+ if err.errno in [-2, -5]:
1283+ self.logger.debug(self.name
1284+ + ' is not resolvable, '
1285+ + 'so not removing from known_hosts')
1286+ else:
1287+ raise err
1288+
1289+ old_host_keys = self.ssh_client.get_host_keys()
1290+ new_host_keys = paramiko.HostKeys()
1291+ addresses = set(addresses)
1292+ for address, key in old_host_keys.iteritems():
1293+ # Skip keys so that they don't get added
1294+ # into the new keys (i.e. they're removed)
1295+ if address in addresses:
1296+ continue
1297+ new_host_keys[address] = key
1298+ new_host_keys.save(config.sshknownhosts)
1299+ self.ssh_client.close()
1300+
1301+ super(SSHMixin, self).destroy(*args, **kw)
1302+
1303+ def sshcheck(self, timeout=config.checktimeout):
1304+ """
1305+ Sleep for a while and check if the machine is available via ssh.
1306+ Return a retryable exception if it is not.
1307+ Intended for use with retry.
1308+ """
1309+ self.logger.info('Sleeping {timeout} seconds'
1310+ .format(timeout=timeout))
1311+ time.sleep(timeout)
1312+ self.logger.info('Checking for ssh availability')
1313+ try:
1314+ self.ssh_client.connect(self.name,
1315+ username=config.user,
1316+ key_filename=config.sshprivatekey)
1317+ except socket.error as err:
1318+ raise UTAHProvisioningException(str(err), retry=True)
1319+
1320+ def sshpoll(self, timeout=None,
1321+ checktimeout=config.checktimeout, logmethod=None):
1322+ """
1323+ Run sshcheck over and over until timeout expires.
1324+ """
1325+ if timeout is None:
1326+ timeout = self.boottimeout
1327+ if logmethod is None:
1328+ logmethod = self.logger.debug
1329+ utah.timeout.timeout(timeout, retry, self.sshcheck, checktimeout,
1330+ logmethod=logmethod)
1331+
1332+ def activecheck(self):
1333+ """
1334+ Start the machine if needed, and check for SSH login.
1335+ """
1336+ self.logger.debug('Checking if machine is active')
1337+ self.provisioncheck()
1338+ if not self.active:
1339+ self._start()
1340+ self.sshcheck()
1341
1342=== removed file 'utah/provisioning/vm/libvirtvm.py'
1343--- utah/provisioning/vm/libvirtvm.py 2012-12-03 14:02:18 +0000
1344+++ utah/provisioning/vm/libvirtvm.py 1970-01-01 00:00:00 +0000
1345@@ -1,777 +0,0 @@
1346-# Ubuntu Testing Automation Harness
1347-# Copyright 2012 Canonical Ltd.
1348-
1349-# This program is free software: you can redistribute it and/or modify it
1350-# under the terms of the GNU General Public License version 3, as published
1351-# by the Free Software Foundation.
1352-
1353-# This program is distributed in the hope that it will be useful, but
1354-# WITHOUT ANY WARRANTY; without even the implied warranties of
1355-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
1356-# PURPOSE. See the GNU General Public License for more details.
1357-
1358-# You should have received a copy of the GNU General Public License along
1359-# with this program. If not, see <http://www.gnu.org/licenses/>.
1360-
1361-"""
1362-Provide classes to provision libvirt-based virtual machines.
1363-"""
1364-
1365-import os
1366-import random
1367-import shutil
1368-import string
1369-import subprocess
1370-import tempfile
1371-
1372-from xml.etree import ElementTree
1373-
1374-import apt.cache
1375-import libvirt
1376-
1377-from utah.provisioning.provisioning import (
1378- SSHMixin,
1379- CustomInstallMixin,
1380-)
1381-from utah.provisioning.vm.vm import VM
1382-from utah.provisioning.vm.exceptions import UTAHVMProvisioningException
1383-from utah import config
1384-from utah.process import ProcessChecker
1385-from utah.timeout import UTAHTimeout
1386-
1387-
1388-class LibvirtVM(VM):
1389- """
1390- Provide a class to utilize VMs using libvirt.
1391-
1392- Capable of utilizing existing VMs.
1393- Creation currently handled by sublcasses.
1394- """
1395- def __init__(self, *args, **kw):
1396- super(LibvirtVM, self).__init__(*args, **kw)
1397- libvirt.registerErrorHandler(self.libvirterrorhandler, None)
1398- self.lv = libvirt.open(config.qemupath)
1399- if self.lv is None:
1400- raise UTAHVMProvisioningException('Cannot connect to libvirt')
1401- self.logger.debug('LibvirtVM init finished')
1402-
1403- def _load(self):
1404- """
1405- Load an existing VM.
1406- """
1407- self.logger.info('Loading VM')
1408- self.vm = self.lv.lookupByName(self.name)
1409- self.logger.info('VM loaded')
1410- return True
1411-
1412- def _provision(self):
1413- """
1414- Make an existing VM available using libvirt to look up the VM by name.
1415- """
1416- self.logger.info('Provisioning VM')
1417- if self.new:
1418- self.logger.debug('New VM requested')
1419- try:
1420- self._load()
1421- self.logger.error('VM already exists')
1422- raise UTAHVMProvisioningException('Request new VM, but '
1423- + self.name
1424- + ' already exists')
1425- except libvirt.libvirtError as err:
1426- if err.get_error_code() == 42:
1427- self._create()
1428- else:
1429- raise err
1430-
1431- try:
1432- self._load()
1433- except libvirt.libvirtError as err:
1434- if err.get_error_code() == 42:
1435- self.logger.debug('Lookup failed')
1436- try:
1437- self._create()
1438- self._load()
1439- except UTAHVMProvisioningException as error:
1440- self.logger.error('VM lookup failed')
1441- raise UTAHVMProvisioningException('Cannot find VM named '
1442- + self.name +
1443- ' and ' + str(error))
1444- else:
1445- raise err
1446- self.provisioned = True
1447- self.logger.info('VM provisioned')
1448-
1449- def activecheck(self):
1450- """
1451- Verify the machine is provisioned, then start it if it is not started.
1452- """
1453- self.logger.debug('Checking if VM is active')
1454- self.provisioncheck()
1455- if self.vm is not None:
1456- if self.vm.isActive() == 0:
1457- self._start()
1458- else:
1459- self.active = True
1460- else:
1461- raise UTAHVMProvisioningException('Failed to provision VM')
1462-
1463- def _start(self):
1464- """
1465- Start the VM.
1466- """
1467- self.logger.info('Starting VM')
1468- if self.vm is not None:
1469- if self.vm.isActive() == 0:
1470- self.vm.create()
1471- else:
1472- raise UTAHVMProvisioningException('Failed to provision VM')
1473- self.active = True
1474-
1475- def stop(self, force=False):
1476- """
1477- Stop the machine.
1478- Setting force to true will do a hard shutdown instead of a graceful
1479- one.
1480- """
1481- self.logger.info('Stopping VM')
1482- if self.vm is not None:
1483- if self.vm.isActive() == 0:
1484- self.logger.info('VM is already stopped')
1485- else:
1486- if force:
1487- self.logger.info('Forced shutdown requested')
1488- self.vm.destroy()
1489- else:
1490- self.vm.shutdown()
1491- else:
1492- self.logger.info('VM not yet created')
1493- self.active = False
1494-
1495- def libvirterrorhandler(self, _context, err):
1496- """
1497- Log libvirt errors instead of sending them directly to the console.
1498- """
1499- errorcode = err.get_error_code()
1500- if errorcode in [9, 42]:
1501- # We see these as part of normal operations,
1502- # so we send them to debug
1503- # 9 is trying to create a VM that already exists
1504- # 42 is trying to load a VM that doesn't exist
1505- logmethod = self.logger.debug
1506- else:
1507- logmethod = self.logger.error
1508- logmethod('libvirt error: ' + err['message'])
1509- logmethod('libvirt error number is: ' + str(errorcode))
1510-
1511-
1512-class VMToolsVM(SSHMixin, LibvirtVM):
1513- """
1514- Provide a class to provision a VM using the ubuntu-qa-tools vm-tools.
1515- """
1516- def __init__(self, machineid=None, prefix='utah', *args, **kw):
1517- if not apt.cache.Cache()['vm-tools'].is_installed:
1518- raise UTAHVMProvisioningException(
1519- 'vm-tools is not installed. '
1520- 'Try: sudo apt-get install vm-tools')
1521- super(VMToolsVM, self).__init__(*args, machineid=machineid, name=None,
1522- **kw)
1523- self.logger.debug('VMToolsVM init finished')
1524-
1525- def _start(self):
1526- """
1527- Start the VM using vm-start, which will wait until SSH is up.
1528- """
1529- self.logger.info('Starting the vm using vm-start')
1530- args = ['vm-start', '-v', '-w', self.name]
1531- self.active = (self._runargs(args) == 0)
1532- return self.active
1533-
1534- def activecheck(self):
1535- """
1536- Verify the machine is provisioned, then start it if it is not started.
1537- Use vm-wait to make sure it's up.
1538- """
1539- self.provisioncheck()
1540- self.logger.debug('Checking if VMToolsVM is active')
1541- self._start()
1542- self.logger.info('Using vm-wait to ensure VM is active')
1543- if (self._runargs(['vm-wait', self.name, '300']) != 0):
1544- self.active = False
1545- raise UTAHVMProvisioningException('Timed out waiting for VM '
1546- 'to be reachable')
1547- else:
1548- self.active = True
1549-
1550- def _getcreationargs(self):
1551- """
1552- Return the vm-new syntax used to create the VM.
1553- Can be used for debugging or instructional purposes.
1554- """
1555- args = ['vm-new', '-f', '-r', '-v', '-b', config.bridge,
1556- '-t', self.installtype, self.series, self.arch, self.prefix]
1557- self.logger.debug('VM Creation args: ' + str(args))
1558- return args
1559-
1560- def _create(self):
1561- """
1562- Run the command generated by getcreationargs
1563- and either return a good status or raise an exception.
1564- """
1565- self.logger.info('Creating vm using vm-new')
1566- self.logger.info('This may take up to two hours, '
1567- 'excluding download times, and may go over an hour '
1568- 'without output during virt-install')
1569- args = self._getcreationargs()
1570- percent = 0
1571- p = subprocess.Popen(args,
1572- stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
1573- while p.poll() is None:
1574- line = p.stdout.readline().strip()
1575- self.logger.debug(line)
1576- if 'Fetching release iso' in line:
1577- self.logger.info('Downloading ISO')
1578- if len(line.split()) >= 3 and '%' in line.split()[-3]:
1579- dlpercent = int(line.split()[-3].strip('%'))
1580- if dlpercent >= percent:
1581- self.logger.info('ISO ' + str(dlpercent) + '% downloaded, '
1582- + line.split(' ')[-1] + ' remaining')
1583- percent += self.dlpercentincrement
1584- if ' saved ' in line:
1585- self.logger.info('ISO download complete')
1586- if 'Creating preseeded iso' in line:
1587- self.logger.info('Preseeding ISO')
1588- if 'virt-install' in line:
1589- self.logger.info('Installing system on VM '
1590- '(may take over an hour)')
1591- self.logger.info('You can watch the progress with:')
1592- self.logger.info("\tvm-view " + self.name)
1593- self.logger.info('Take care not to interrupt the install')
1594- if 'Verifying lsb_release' in line:
1595- self.logger.info('Waiting for post-install '
1596- 'and verifying system '
1597- '(may take over a half hour)')
1598- if "No domains available for virt type 'hvm'" in line:
1599- self.logger.error('No hardware virtual machine '
1600- 'support available')
1601- self.logger.info('Please ensure the following:')
1602- self.logger.info("\tYour processor supports hardware "
1603- "virtualization extensions")
1604- self.logger.info("\tHardware virtualization "
1605- "is not disabled in the BIOS")
1606- self.logger.info("\tkvm is installed")
1607- self.logger.info("\tThe kvm and processor-specific "
1608- "kvm kernel modules are installed and loaded")
1609- self.logger.error('Software virtual machine support '
1610- 'is implemented in the CustomVM class')
1611- raise UTAHVMProvisioningException(
1612- 'No hardware virtual machine support available')
1613- if ('Could not find' in line
1614- and ".uqt-vm-tools.conf' configuration file!" in line):
1615- self.logger.error('No .uqt-vm-tools.conf configuration file '
1616- 'found in home directory')
1617- self.logger.info('If you are running as the utah user, '
1618- 'you can use the packaged config file:')
1619- self.logger.info("\tln -s /etc/utah/uqt-vm-tools.conf "
1620- "~utah/.uqt-vm-tools.conf")
1621- self.logger.info('As a different user, '
1622- 'run vm-new with no arguments, '
1623- 'and answer y when prompted '
1624- 'to create an initial config file')
1625- self.logger.info('We recommend adding python-yaml '
1626- 'to the vm_extra_packages line '
1627- 'of this file, i.e.:')
1628- self.logger.info("\tsed "
1629- "'s/^vm_extra_packages=\""
1630- "/vm_extra_packages=\"python-yaml /' "
1631- "-i ~/.uqt_vm_tools.conf")
1632- raise UTAHVMProvisioningException(
1633- 'No vm-tools config file available; '
1634- 'more info available in '
1635- + self.filehandler.baseFilename)
1636- if p.returncode == 0:
1637- return True
1638- else:
1639- raise UTAHVMProvisioningException(
1640- 'Failed to create VM: vm-new exit status: '
1641- + str(p.returncode))
1642-
1643- def destroy(self):
1644- """
1645- Use vm-remove to destroy the vm.
1646- """
1647- self.logger.info('Destroying vm using vm-remove')
1648- self.provisioned = False
1649- args = ['vm-remove', '-f', self.name]
1650- return (self._runargs(args) == 0)
1651-
1652- def _getcommandargs(self, command, quiet=None, root=False, timeout=300):
1653- """
1654- Get the arguments to send to vm-cmd.
1655- Used by multiple other functions.
1656- """
1657- if quiet is None:
1658- quiet = not self.debug
1659- args = ['vm-cmd', '-f']
1660- if timeout is not None:
1661- args.extend(['-s', '-t', str(timeout)])
1662- if quiet:
1663- args.append('-q')
1664- if root:
1665- args.append('-r')
1666- args.append(self.name)
1667- if isinstance(command, basestring):
1668- args.append(command)
1669- else:
1670- args.extend(command)
1671- self.logger.debug('vm-cmd command args: ' + str(args))
1672- return args
1673-
1674- def run(self, command, quiet=None, root=False, timeout=300):
1675- """
1676- Run a command using vm-cmd.
1677- Timeout defaults to 300 seconds, but can be set to None to not send a
1678- timeout argument.
1679- """
1680- if quiet is None:
1681- quiet = not self.debug
1682- self.activecheck()
1683- args = self._getcommandargs(command=command, quiet=quiet,
1684- root=root, timeout=timeout)
1685- self.logger.info('Running command on VM: ' + ' '.join(args))
1686- p = subprocess.Popen(args,
1687- stdout=subprocess.PIPE, stderr=subprocess.PIPE)
1688- while p.poll() is None:
1689- pass
1690- return p.returncode, p.communicate()[0], p.communicate()[1]
1691-
1692- def uploadfiles(self, files, target=os.path.normpath('/tmp/')):
1693- """
1694- Copy a file or list of files to a target directory on the machine.
1695- """
1696- if isinstance(files, basestring):
1697- files = [files]
1698- if not target.endswith('/'):
1699- target += '/'
1700- self.activecheck()
1701- success = True
1702- for myfile in files:
1703- self.logger.info('Uploading ' + myfile
1704- + ' from the host to ' + target + ' on the VM')
1705- returncode = self._runargs(['vm-scp', '-f', '-p',
1706- self.name, myfile, target])
1707- if returncode != 0:
1708- success = False
1709- return success
1710-
1711-
1712-class CustomVM(CustomInstallMixin, SSHMixin, LibvirtVM):
1713- """
1714- Install a VM from an image using libvirt direct kernel booting.
1715- """
1716- def __init__(self, diskbus=None, disksizes=None, emulator=None,
1717- machineid=None, macs=None, name=None, prefix='utah', *args,
1718- **kw):
1719- # Make sure that no other virtualization solutions are running
1720- # TODO: see if this is needed for qemu or just kvm
1721- process_checker = ProcessChecker()
1722- for cmdline, app in [('/usr/lib/virtualbox/VirtualBox', 'VirtualBox'),
1723- ('/usr/lib/vmware/bin', 'VMware')]:
1724- if process_checker.check_cmdline(cmdline):
1725- message = process_checker.get_error_message(app)
1726- raise UTAHVMProvisioningException(message)
1727-
1728- if diskbus is None:
1729- self.diskbus = config.diskbus
1730- else:
1731- self.diskbus = diskbus
1732- if disksizes is None:
1733- disksizes = config.disksizes
1734- if disksizes is None:
1735- self.disksizes = [8]
1736- else:
1737- self.disksizes = disksizes
1738- self.disks = []
1739- if name is None:
1740- autoname = True
1741- name = '-'.join([str(prefix), str(machineid)])
1742- else:
1743- autoname = False
1744- super(CustomVM, self).__init__(machineid=machineid, name=name, *args,
1745- **kw)
1746- # TODO: do a better job of separating installation
1747- # into _create rather than __init__
1748- if self.image is None:
1749- raise UTAHVMProvisioningException('Image file required '
1750- 'for custom VM installation')
1751- self._custominit()
1752- if autoname:
1753- self._namesetup()
1754- self._loggerunsetup()
1755- self._loggersetup()
1756- self._dirsetup()
1757- if emulator is None:
1758- emulator = config.emulator
1759- if emulator is None:
1760- if self._supportsdomaintype('kvm'):
1761- self.logger.info('Setting type to kvm '
1762- 'since it is present in libvirt capabilities')
1763- self.domaintype = 'kvm'
1764- elif self._supportsdomaintype('qemu'):
1765- self.logger.info('Setting type to qemu '
1766- 'since it is present in libvirt capabilities')
1767- self.domaintype = 'qemu'
1768- else:
1769- raise UTAHVMProvisioningException(
1770- 'kvm and qemu not supported in libvirt capabilities; '
1771- 'please make sure qemu and/or kvm are installed '
1772- 'and libvirt is configured correctly')
1773- else:
1774- self.domaintype = emulator
1775- if self.domaintype == 'qemu':
1776- self.logger.debug('Raising boot timeout for qemu domain')
1777- self.boottimeout *= 4
1778- if macs is None:
1779- macs = []
1780- self.macs = macs
1781- self.dircheck()
1782- self.logger.debug('CustomVM init finished')
1783-
1784- def _createdisks(self, disksizes=None):
1785- """
1786- Create disk files if needed and build a list of them.
1787- """
1788- self.logger.info('Creating disks')
1789- if disksizes is None:
1790- disksizes = self.disksizes
1791- for index, size in enumerate(disksizes):
1792- disksize = '{}G'.format(size)
1793- basename = 'disk{}.qcow2'.format(index)
1794- diskfile = os.path.join(self.directory, basename)
1795- if not os.path.isfile(diskfile):
1796- cmd = ['qemu-img', 'create', '-f', 'qcow2', diskfile, disksize]
1797- self.logger.debug('Creating ' + disksize + ' disk using:')
1798- self.logger.debug(' '.join(cmd))
1799- if self._runargs(cmd) != 0:
1800- raise UTAHVMProvisioningException(
1801- 'Could not create disk image at ' + diskfile)
1802- disk = {'bus': self.diskbus,
1803- 'file': diskfile,
1804- 'size': disksize,
1805- 'type': 'qcow2'}
1806- self.disks.append(disk)
1807- self.logger.debug('Adding disk to list')
1808-
1809- def _supportsdomaintype(self, domaintype):
1810- """
1811- Check emulator support in libvirt capabilities.
1812- """
1813- capabilities = ElementTree.fromstring(self.lv.getCapabilities())
1814- for guest in capabilities.iterfind('guest'):
1815- for arch in guest.iterfind('arch'):
1816- for domain in arch.iterfind('domain'):
1817- if domaintype in domain.get('type'):
1818- return True
1819- return False
1820-
1821- def _installxml(self, cmdline=None, image=None, initrd=None,
1822- kernel=None, tmpdir=None, xml=None):
1823- """
1824- Return the XML tree to be passed to libvirt for VM installation.
1825- """
1826- self.logger.info('Creating installation XML')
1827- if cmdline is None:
1828- cmdline = self.cmdline
1829- if image is None:
1830- image = self.image.image
1831- if initrd is None:
1832- initrd = self.initrd
1833- if kernel is None:
1834- kernel = self.kernel
1835- if xml is None:
1836- xml = self.xml
1837- if tmpdir is None:
1838- tmpdir = self.tmpdir
1839- xmlt = ElementTree.ElementTree(file=xml)
1840- if self.rewrite in ['all', 'minimal']:
1841- self.logger.debug('Setting VM to shutdown on reboot')
1842- xmlt.find('on_reboot').text = 'destroy'
1843- if self.rewrite == 'all':
1844- self._installxml_rewrite_all(cmdline, image, initrd, kernel,
1845- xmlt)
1846- else:
1847- self.logger.info('Not rewriting XML because rewrite is ' +
1848- self.rewrite)
1849- if self.debug:
1850- xmlt.write(os.path.join(tmpdir, 'install.xml'))
1851- self.logger.info('Installation XML ready')
1852- return xmlt
1853-
1854- def _installxml_rewrite_all(self, cmdline_txt, image, initrd_txt,
1855- kernel_txt, xmlt):
1856- """
1857- Rewrite the whole configuration file for the VM
1858- """
1859- self.logger.debug('Rewriting basic info')
1860- xmlt.find('name').text = self.name
1861- xmlt.find('uuid').text = self.uuid
1862- self.logger.debug('Setting type to qemu in case no '
1863- 'hardware virtualization present')
1864- xmlt.getroot().set('type', self.domaintype)
1865- ose = xmlt.find('os')
1866- if self.arch == ('i386'):
1867- ose.find('type').set('arch', 'i686')
1868- elif self.arch == ('amd64'):
1869- ose.find('type').set('arch', 'x86_64')
1870- else:
1871- ose.find('type').set('arch', self.arch)
1872- self.logger.debug('Setting up boot info')
1873- for kernele in list(ose.iterfind('kernel')):
1874- ose.remove(kernele)
1875- kernele = ElementTree.Element('kernel')
1876- kernele.text = kernel_txt
1877- ose.append(kernele)
1878- for initrde in list(ose.iterfind('initrd')):
1879- ose.remove(initrde)
1880- initrde = ElementTree.Element('initrd')
1881- initrde.text = initrd_txt
1882- ose.append(initrde)
1883- for cmdlinee in list(ose.iterfind('cmdline')):
1884- ose.remove(cmdlinee)
1885- cmdlinee = ElementTree.Element('cmdline')
1886- cmdlinee.text = cmdline_txt
1887- ose.append(cmdlinee)
1888- self.logger.debug('Setting up devices')
1889- devices = xmlt.find('devices')
1890- self.logger.debug('Setting up disks')
1891- for disk in list(devices.iterfind('disk')):
1892- if disk.get('device') == 'disk':
1893- devices.remove(disk)
1894- self.logger.debug('Removed existing disk')
1895- #TODO: Add a cdrom if none exists
1896- if disk.get('device') == 'cdrom':
1897- if disk.find('source') is not None:
1898- disk.find('source').set('file', image)
1899- self.logger.debug('Rewrote existing CD-ROM')
1900- else:
1901- source = ElementTree.Element('source')
1902- source.set('file', image)
1903- disk.append(source)
1904- self.logger.debug('Added source to existing '
1905- 'CD-ROM')
1906- for disk in self.disks:
1907- diske = ElementTree.Element('disk')
1908- diske.set('type', 'file')
1909- diske.set('device', 'disk')
1910- driver = ElementTree.Element('driver')
1911- driver.set('name', 'qemu')
1912- driver.set('type', disk['type'])
1913- diske.append(driver)
1914- source = ElementTree.Element('source')
1915- source.set('file', disk['file'])
1916- diske.append(source)
1917- target = ElementTree.Element('target')
1918- dev = "vd%s" % (string.ascii_lowercase[self.disks.index(disk)])
1919- target.set('dev', dev)
1920- target.set('bus', disk['bus'])
1921- diske.append(target)
1922- devices.append(diske)
1923- self.logger.debug('Added ' + str(disk['size']) + ' disk')
1924- macs = list(self.macs)
1925- for interface in devices.iterfind('interface'):
1926- if interface.get('type') in ['network', 'bridge']:
1927- if len(macs) > 0:
1928- mac = macs.pop(0)
1929- interface.find('mac').set('address', mac)
1930- self.logger.debug('Rewrote interface '
1931- 'to use specified mac address ' + mac)
1932- else:
1933- mac = random_mac_address()
1934- interface.find('mac').set('address', mac)
1935- self.macs.append(mac)
1936- self.logger.debug('Rewrote interface '
1937- 'to use random mac address ' + mac)
1938- if interface.get('type') == 'bridge':
1939- interface.find('source').set('bridge', config.bridge)
1940- serial = ElementTree.Element('serial')
1941- serial.set('type', 'file')
1942- source = ElementTree.Element('source')
1943- log_filename = os.path.join(config.logpath, self.name + '.syslog.log')
1944- source.set('path', log_filename)
1945- serial.append(source)
1946- target = ElementTree.Element('target')
1947- target.set('port', '0')
1948- serial.append(target)
1949- devices.append(serial)
1950-
1951- def _installvm(self, lv=None, tmpdir=None, xml=None):
1952- """
1953- Install a VM, then undefine it in libvirt.
1954- The final installation will recreate the VM using the existing disks.
1955- """
1956- self.logger.info('Creating VM')
1957- if lv is None:
1958- lv = self.lv
1959- if xml is None:
1960- xml = self.xml
1961- if tmpdir is None:
1962- tmpdir = self.tmpdir
1963- vm = lv.defineXML(ElementTree.tostring(xml.getroot()))
1964- os.chmod(tmpdir, 0755)
1965- vm.create()
1966- self.logger.info('Installing system on VM (may take over an hour)')
1967- self.logger.info('You can watch the progress with virt-viewer')
1968- log_filename = os.path.join(config.logpath, self.name + '.syslog.log')
1969- self.logger.info('Logs will be written to ' + log_filename)
1970-
1971- while vm.isActive() is not 0:
1972- pass
1973-
1974- vm.undefine()
1975- self.logger.info('Installation complete')
1976-
1977- def _finalxml(self, tmpdir=None, xml=None):
1978- """
1979- Create the XML to be used for the post-installation VM.
1980- This may be a transformation of the installation XML.
1981- """
1982- self.logger.info('Creating final VM XML')
1983- if xml is None:
1984- xml = ElementTree.ElementTree(file=self.xml)
1985- if tmpdir is None:
1986- tmpdir = self.tmpdir
1987- if self.rewrite in ['all', 'minimal']:
1988- self.logger.debug('Setting VM to reboot normally on reboot')
1989- xml.find('on_reboot').text = 'restart'
1990- if self.rewrite == 'all':
1991- self.logger.debug('Removing VM install parameters')
1992- ose = xml.find('os')
1993- for kernel in ose.iterfind('kernel'):
1994- ose.remove(kernel)
1995- for initrd in ose.iterfind('initrd'):
1996- ose.remove(initrd)
1997- for cmdline in ose.iterfind('cmdline'):
1998- ose.remove(cmdline)
1999- devices = xml.find('devices')
2000- devices.remove(devices.find('serial'))
2001- for disk in list(devices.iterfind('disk')):
2002- if disk.get('device') == 'cdrom':
2003- disk.remove(disk.find('source'))
2004- else:
2005- self.logger.info('Not rewriting XML because rewrite is ' +
2006- self.rewrite)
2007- if self.debug:
2008- xml.write(os.path.join(tmpdir, 'final.xml'))
2009- return xml
2010-
2011- def _tmpimage(self, image=None, tmpdir=None):
2012- """
2013- Create a temporary copy of the image so libvirt will lock that copy.
2014- This allows other simultaneous processes to update the cached image.
2015- """
2016- if image is None:
2017- image = self.image.image
2018- if tmpdir is None:
2019- tmpdir = self.tmpdir
2020- self.logger.info('Making temp copy of install image')
2021- tmpimage = os.path.join(tmpdir, os.path.basename(image))
2022- self.logger.debug('Copying ' + image + ' to ' + tmpimage)
2023- shutil.copyfile(image, tmpimage)
2024- return tmpimage
2025-
2026- def _create(self):
2027- """
2028- Create the VM, install the system, and prepare it to boot.
2029- This primarily calls functions from CustomInstallMixin and CustomVM.
2030- """
2031- self.logger.info('Creating custom virtual machine')
2032-
2033- tmpdir = tempfile.mkdtemp(prefix='/tmp/' + self.name + '_')
2034- self.logger.debug('Working dir: ' + tmpdir)
2035- os.chdir(tmpdir)
2036-
2037- kernel = self._preparekernel(kernel=self.kernel, tmpdir=tmpdir)
2038-
2039- initrd = self._prepareinitrd(initrd=self.initrd, tmpdir=tmpdir)
2040-
2041- self._unpackinitrd(initrd=initrd, tmpdir=tmpdir)
2042-
2043- self._setuplatecommand(tmpdir=tmpdir)
2044-
2045- self._setuppreseed(tmpdir=tmpdir)
2046-
2047- if self.rewrite == 'all':
2048- self._setuplogging(tmpdir=tmpdir)
2049- else:
2050- self.logger.debug('Skipping logging setup because rewrite is' +
2051- self.rewrite)
2052-
2053- initrd = self._repackinitrd(tmpdir=tmpdir)
2054-
2055- self._createdisks()
2056-
2057- image = self._tmpimage(image=self.image.image, tmpdir=tmpdir)
2058-
2059- xml = self._installxml(cmdline=self.cmdline, image=image,
2060- initrd=initrd, kernel=kernel,
2061- tmpdir=tmpdir, xml=self.xml)
2062-
2063- self._installvm(lv=self.lv, tmpdir=tmpdir, xml=xml)
2064-
2065- xml = self._finalxml(tmpdir=tmpdir, xml=xml)
2066-
2067- self.logger.info('Setting up final VM')
2068- self.vm = self.lv.defineXML(ElementTree.tostring(xml.getroot()))
2069-
2070- if self.debug:
2071- self.logger.info('Leaving temp directory '
2072- 'because debug is enabled: ' + tmpdir)
2073- else:
2074- self.logger.info('Cleaning up temp directory')
2075- shutil.rmtree(tmpdir)
2076- return True
2077-
2078- def _start(self):
2079- """
2080- Start the VM.
2081- """
2082- self.logger.info('Starting CustomVM')
2083- if self.vm is not None:
2084- if self.vm.isActive() == 0:
2085- self.vm.create()
2086- else:
2087- raise UTAHVMProvisioningException('Failed to provision VM')
2088- self.logger.info('Waiting ' + str(self.boottimeout) +
2089- ' seconds to allow machine to boot')
2090- try:
2091- self.pingpoll(timeout=self.boottimeout)
2092- except UTAHTimeout:
2093- # Ignore timeout for ping, since depending on the network
2094- # configuration ssh might still work despite of the ping failure.
2095- self.logger.warning('Network connectivity (ping) failure')
2096- self.sshpoll(timeout=self.boottimeout)
2097- self.active = True
2098-
2099- def destroy(self, *args, **kw):
2100- """
2101- Remove the machine from libvirt and remove all the disk files.
2102- """
2103- # TODO: make this use standard cleanup
2104- super(CustomVM, self).destroy(*args, **kw)
2105- self.stop(force=True)
2106- if self.vm is not None:
2107- self.vm.undefine()
2108- else:
2109- self.logger.info('VM not created')
2110- for disk in self.disks:
2111- os.unlink(disk['file'])
2112- shutil.rmtree(self.directory)
2113-
2114-
2115-# See http://kennethreitz.com/blog/generate-a-random-mac-address-in-python/
2116-def random_mac_address():
2117- """Returns a completely random Mac Address"""
2118- mac = [0x52, 0x54, 0x00,
2119- random.randint(0x00, 0xff),
2120- random.randint(0x00, 0xff),
2121- random.randint(0x00, 0xff)]
2122- return ':'.join(map(lambda x: "%02x" % x, mac))
2123
2124=== modified file 'utah/provisioning/vm/vm.py'
2125--- utah/provisioning/vm/vm.py 2012-12-03 14:02:18 +0000
2126+++ utah/provisioning/vm/vm.py 2012-12-13 00:38:21 +0000
2127@@ -14,11 +14,26 @@
2128 # with this program. If not, see <http://www.gnu.org/licenses/>.
2129
2130 """
2131-Consolidate functions specific to virtual machine provisioning,
2132-but not specific to any type of virtual machine.
2133+Consolidate functions for virtual machine provisioning.
2134+Non-libvirt VMs can be supported elsewhere if support for them is needed.
2135 """
2136
2137-from utah.provisioning.provisioning import Machine
2138+import libvirt
2139+import os
2140+import random
2141+import shutil
2142+import string
2143+import tempfile
2144+
2145+from xml.etree import ElementTree
2146+
2147+from utah import config
2148+from utah.process import ProcessChecker
2149+from utah.provisioning.inventory.sqlite import SQLiteInventory
2150+from utah.provisioning.provisioning import CustomInstallMixin, Machine
2151+from utah.provisioning.ssh import SSHMixin
2152+from utah.provisioning.vm.exceptions import UTAHVMProvisioningException
2153+from utah.timeout import UTAHTimeout
2154
2155
2156 class VM(Machine):
2157@@ -29,3 +44,590 @@
2158 super(VM, self).__init__(*args, **kw)
2159 self.vm = None
2160 self.logger.debug('VM init finished')
2161+
2162+
2163+class LibvirtVM(VM):
2164+ """
2165+ Provide a class to utilize VMs using libvirt.
2166+
2167+ Capable of utilizing existing VMs.
2168+ Creation currently handled by sublcasses.
2169+ """
2170+ def __init__(self, *args, **kw):
2171+ super(LibvirtVM, self).__init__(*args, **kw)
2172+ libvirt.registerErrorHandler(self.libvirterrorhandler, None)
2173+ self.lv = libvirt.open(config.qemupath)
2174+ if self.lv is None:
2175+ raise UTAHVMProvisioningException('Cannot connect to libvirt')
2176+ self.logger.debug('LibvirtVM init finished')
2177+
2178+ def _load(self):
2179+ """
2180+ Load an existing VM.
2181+ """
2182+ self.logger.info('Loading VM')
2183+ self.vm = self.lv.lookupByName(self.name)
2184+ self.logger.info('VM loaded')
2185+ return True
2186+
2187+ def _provision(self):
2188+ """
2189+ Make an existing VM available using libvirt to look up the VM by name.
2190+ """
2191+ self.logger.info('Provisioning VM')
2192+ if self.new:
2193+ self.logger.debug('New VM requested')
2194+ try:
2195+ self._load()
2196+ self.logger.error('VM already exists')
2197+ raise UTAHVMProvisioningException('Request new VM, but '
2198+ + self.name
2199+ + ' already exists')
2200+ except libvirt.libvirtError as err:
2201+ if err.get_error_code() == 42:
2202+ self._create()
2203+ else:
2204+ raise err
2205+
2206+ try:
2207+ self._load()
2208+ except libvirt.libvirtError as err:
2209+ if err.get_error_code() == 42:
2210+ self.logger.debug('Lookup failed')
2211+ try:
2212+ self._create()
2213+ self._load()
2214+ except UTAHVMProvisioningException as error:
2215+ self.logger.error('VM lookup failed')
2216+ raise UTAHVMProvisioningException('Cannot find VM named '
2217+ + self.name +
2218+ ' and ' + str(error))
2219+ else:
2220+ raise err
2221+ self.provisioned = True
2222+ self.logger.info('VM provisioned')
2223+
2224+ def activecheck(self):
2225+ """
2226+ Verify the machine is provisioned, then start it if it is not started.
2227+ """
2228+ self.logger.debug('Checking if VM is active')
2229+ self.provisioncheck()
2230+ if self.vm is not None:
2231+ if self.vm.isActive() == 0:
2232+ self._start()
2233+ else:
2234+ self.active = True
2235+ else:
2236+ raise UTAHVMProvisioningException('Failed to provision VM')
2237+
2238+ def _start(self):
2239+ """
2240+ Start the VM.
2241+ """
2242+ self.logger.info('Starting VM')
2243+ if self.vm is not None:
2244+ if self.vm.isActive() == 0:
2245+ self.vm.create()
2246+ else:
2247+ raise UTAHVMProvisioningException('Failed to provision VM')
2248+ self.active = True
2249+
2250+ def stop(self, force=False):
2251+ """
2252+ Stop the machine.
2253+ Setting force to true will do a hard shutdown instead of a graceful
2254+ one.
2255+ """
2256+ self.logger.info('Stopping VM')
2257+ if self.vm is not None:
2258+ if self.vm.isActive() == 0:
2259+ self.logger.info('VM is already stopped')
2260+ else:
2261+ if force:
2262+ self.logger.info('Forced shutdown requested')
2263+ self.vm.destroy()
2264+ else:
2265+ self.vm.shutdown()
2266+ else:
2267+ self.logger.info('VM not yet created')
2268+ self.active = False
2269+
2270+ def libvirterrorhandler(self, _context, err):
2271+ """
2272+ Log libvirt errors instead of sending them directly to the console.
2273+ """
2274+ errorcode = err.get_error_code()
2275+ if errorcode in [9, 42]:
2276+ # We see these as part of normal operations,
2277+ # so we send them to debug
2278+ # 9 is trying to create a VM that already exists
2279+ # 42 is trying to load a VM that doesn't exist
2280+ logmethod = self.logger.debug
2281+ else:
2282+ logmethod = self.logger.error
2283+ logmethod('libvirt error: ' + err['message'])
2284+ logmethod('libvirt error number is: ' + str(errorcode))
2285+
2286+
2287+class CustomVM(CustomInstallMixin, SSHMixin, LibvirtVM):
2288+ """
2289+ Install a VM from an image using libvirt direct kernel booting.
2290+ """
2291+ def __init__(self, diskbus=None, disksizes=None, emulator=None,
2292+ machineid=None, macs=None, name=None, prefix='utah', *args,
2293+ **kw):
2294+ # Make sure that no other virtualization solutions are running
2295+ # TODO: see if this is needed for qemu or just kvm
2296+ process_checker = ProcessChecker()
2297+ for cmdline, app in [('/usr/lib/virtualbox/VirtualBox', 'VirtualBox'),
2298+ ('/usr/lib/vmware/bin', 'VMware')]:
2299+ if process_checker.check_cmdline(cmdline):
2300+ message = process_checker.get_error_message(app)
2301+ raise UTAHVMProvisioningException(message)
2302+
2303+ if diskbus is None:
2304+ self.diskbus = config.diskbus
2305+ else:
2306+ self.diskbus = diskbus
2307+ if disksizes is None:
2308+ disksizes = config.disksizes
2309+ if disksizes is None:
2310+ self.disksizes = [8]
2311+ else:
2312+ self.disksizes = disksizes
2313+ self.disks = []
2314+ if name is None:
2315+ autoname = True
2316+ name = '-'.join([str(prefix), str(machineid)])
2317+ else:
2318+ autoname = False
2319+ super(CustomVM, self).__init__(machineid=machineid, name=name, *args,
2320+ **kw)
2321+ # TODO: do a better job of separating installation
2322+ # into _create rather than __init__
2323+ if self.image is None:
2324+ raise UTAHVMProvisioningException('Image file required '
2325+ 'for custom VM installation')
2326+ self._custominit()
2327+ if autoname:
2328+ self._namesetup()
2329+ self._loggerunsetup()
2330+ self._loggersetup()
2331+ self._dirsetup()
2332+ if emulator is None:
2333+ emulator = config.emulator
2334+ if emulator is None:
2335+ if self._supportsdomaintype('kvm'):
2336+ self.logger.info('Setting type to kvm '
2337+ 'since it is present in libvirt capabilities')
2338+ self.domaintype = 'kvm'
2339+ elif self._supportsdomaintype('qemu'):
2340+ self.logger.info('Setting type to qemu '
2341+ 'since it is present in libvirt capabilities')
2342+ self.domaintype = 'qemu'
2343+ else:
2344+ raise UTAHVMProvisioningException(
2345+ 'kvm and qemu not supported in libvirt capabilities; '
2346+ 'please make sure qemu and/or kvm are installed '
2347+ 'and libvirt is configured correctly')
2348+ else:
2349+ self.domaintype = emulator
2350+ if self.domaintype == 'qemu':
2351+ self.logger.debug('Raising boot timeout for qemu domain')
2352+ self.boottimeout *= 4
2353+ if macs is None:
2354+ macs = []
2355+ self.macs = macs
2356+ self.dircheck()
2357+ self.logger.debug('CustomVM init finished')
2358+
2359+ def _createdisks(self, disksizes=None):
2360+ """
2361+ Create disk files if needed and build a list of them.
2362+ """
2363+ self.logger.info('Creating disks')
2364+ if disksizes is None:
2365+ disksizes = self.disksizes
2366+ for index, size in enumerate(disksizes):
2367+ disksize = '{}G'.format(size)
2368+ basename = 'disk{}.qcow2'.format(index)
2369+ diskfile = os.path.join(self.directory, basename)
2370+ if not os.path.isfile(diskfile):
2371+ cmd = ['qemu-img', 'create', '-f', 'qcow2', diskfile, disksize]
2372+ self.logger.debug('Creating ' + disksize + ' disk using:')
2373+ self.logger.debug(' '.join(cmd))
2374+ if self._runargs(cmd) != 0:
2375+ raise UTAHVMProvisioningException(
2376+ 'Could not create disk image at ' + diskfile)
2377+ disk = {'bus': self.diskbus,
2378+ 'file': diskfile,
2379+ 'size': disksize,
2380+ 'type': 'qcow2'}
2381+ self.disks.append(disk)
2382+ self.logger.debug('Adding disk to list')
2383+
2384+ def _supportsdomaintype(self, domaintype):
2385+ """
2386+ Check emulator support in libvirt capabilities.
2387+ """
2388+ capabilities = ElementTree.fromstring(self.lv.getCapabilities())
2389+ for guest in capabilities.iterfind('guest'):
2390+ for arch in guest.iterfind('arch'):
2391+ for domain in arch.iterfind('domain'):
2392+ if domaintype in domain.get('type'):
2393+ return True
2394+ return False
2395+
2396+ def _installxml(self, cmdline=None, image=None, initrd=None,
2397+ kernel=None, tmpdir=None, xml=None):
2398+ """
2399+ Return the XML tree to be passed to libvirt for VM installation.
2400+ """
2401+ self.logger.info('Creating installation XML')
2402+ if cmdline is None:
2403+ cmdline = self.cmdline
2404+ if image is None:
2405+ image = self.image.image
2406+ if initrd is None:
2407+ initrd = self.initrd
2408+ if kernel is None:
2409+ kernel = self.kernel
2410+ if xml is None:
2411+ xml = self.xml
2412+ if tmpdir is None:
2413+ tmpdir = self.tmpdir
2414+ xmlt = ElementTree.ElementTree(file=xml)
2415+ if self.rewrite in ['all', 'minimal']:
2416+ self.logger.debug('Setting VM to shutdown on reboot')
2417+ xmlt.find('on_reboot').text = 'destroy'
2418+ if self.rewrite == 'all':
2419+ self._installxml_rewrite_all(cmdline, image, initrd, kernel,
2420+ xmlt)
2421+ else:
2422+ self.logger.info('Not rewriting XML because rewrite is ' +
2423+ self.rewrite)
2424+ if self.debug:
2425+ xmlt.write(os.path.join(tmpdir, 'install.xml'))
2426+ self.logger.info('Installation XML ready')
2427+ return xmlt
2428+
2429+ def _installxml_rewrite_all(self, cmdline_txt, image, initrd_txt,
2430+ kernel_txt, xmlt):
2431+ """
2432+ Rewrite the whole configuration file for the VM
2433+ """
2434+ self.logger.debug('Rewriting basic info')
2435+ xmlt.find('name').text = self.name
2436+ xmlt.find('uuid').text = self.uuid
2437+ self.logger.debug('Setting type to qemu in case no '
2438+ 'hardware virtualization present')
2439+ xmlt.getroot().set('type', self.domaintype)
2440+ ose = xmlt.find('os')
2441+ if self.arch == ('i386'):
2442+ ose.find('type').set('arch', 'i686')
2443+ elif self.arch == ('amd64'):
2444+ ose.find('type').set('arch', 'x86_64')
2445+ else:
2446+ ose.find('type').set('arch', self.arch)
2447+ self.logger.debug('Setting up boot info')
2448+ for kernele in list(ose.iterfind('kernel')):
2449+ ose.remove(kernele)
2450+ kernele = ElementTree.Element('kernel')
2451+ kernele.text = kernel_txt
2452+ ose.append(kernele)
2453+ for initrde in list(ose.iterfind('initrd')):
2454+ ose.remove(initrde)
2455+ initrde = ElementTree.Element('initrd')
2456+ initrde.text = initrd_txt
2457+ ose.append(initrde)
2458+ for cmdlinee in list(ose.iterfind('cmdline')):
2459+ ose.remove(cmdlinee)
2460+ cmdlinee = ElementTree.Element('cmdline')
2461+ cmdlinee.text = cmdline_txt
2462+ ose.append(cmdlinee)
2463+ self.logger.debug('Setting up devices')
2464+ devices = xmlt.find('devices')
2465+ self.logger.debug('Setting up disks')
2466+ for disk in list(devices.iterfind('disk')):
2467+ if disk.get('device') == 'disk':
2468+ devices.remove(disk)
2469+ self.logger.debug('Removed existing disk')
2470+ #TODO: Add a cdrom if none exists
2471+ if disk.get('device') == 'cdrom':
2472+ if disk.find('source') is not None:
2473+ disk.find('source').set('file', image)
2474+ self.logger.debug('Rewrote existing CD-ROM')
2475+ else:
2476+ source = ElementTree.Element('source')
2477+ source.set('file', image)
2478+ disk.append(source)
2479+ self.logger.debug('Added source to existing '
2480+ 'CD-ROM')
2481+ for disk in self.disks:
2482+ diske = ElementTree.Element('disk')
2483+ diske.set('type', 'file')
2484+ diske.set('device', 'disk')
2485+ driver = ElementTree.Element('driver')
2486+ driver.set('name', 'qemu')
2487+ driver.set('type', disk['type'])
2488+ diske.append(driver)
2489+ source = ElementTree.Element('source')
2490+ source.set('file', disk['file'])
2491+ diske.append(source)
2492+ target = ElementTree.Element('target')
2493+ dev = "vd%s" % (string.ascii_lowercase[self.disks.index(disk)])
2494+ target.set('dev', dev)
2495+ target.set('bus', disk['bus'])
2496+ diske.append(target)
2497+ devices.append(diske)
2498+ self.logger.debug('Added ' + str(disk['size']) + ' disk')
2499+ macs = list(self.macs)
2500+ for interface in devices.iterfind('interface'):
2501+ if interface.get('type') in ['network', 'bridge']:
2502+ if len(macs) > 0:
2503+ mac = macs.pop(0)
2504+ interface.find('mac').set('address', mac)
2505+ self.logger.debug('Rewrote interface '
2506+ 'to use specified mac address ' + mac)
2507+ else:
2508+ mac = random_mac_address()
2509+ interface.find('mac').set('address', mac)
2510+ self.macs.append(mac)
2511+ self.logger.debug('Rewrote interface '
2512+ 'to use random mac address ' + mac)
2513+ if interface.get('type') == 'bridge':
2514+ interface.find('source').set('bridge', config.bridge)
2515+ serial = ElementTree.Element('serial')
2516+ serial.set('type', 'file')
2517+ source = ElementTree.Element('source')
2518+ log_filename = os.path.join(config.logpath, self.name + '.syslog.log')
2519+ source.set('path', log_filename)
2520+ serial.append(source)
2521+ target = ElementTree.Element('target')
2522+ target.set('port', '0')
2523+ serial.append(target)
2524+ devices.append(serial)
2525+
2526+ def _installvm(self, lv=None, tmpdir=None, xml=None):
2527+ """
2528+ Install a VM, then undefine it in libvirt.
2529+ The final installation will recreate the VM using the existing disks.
2530+ """
2531+ self.logger.info('Creating VM')
2532+ if lv is None:
2533+ lv = self.lv
2534+ if xml is None:
2535+ xml = self.xml
2536+ if tmpdir is None:
2537+ tmpdir = self.tmpdir
2538+ vm = lv.defineXML(ElementTree.tostring(xml.getroot()))
2539+ os.chmod(tmpdir, 0755)
2540+ vm.create()
2541+ self.logger.info('Installing system on VM (may take over an hour)')
2542+ self.logger.info('You can watch the progress with virt-viewer')
2543+ log_filename = os.path.join(config.logpath, self.name + '.syslog.log')
2544+ self.logger.info('Logs will be written to ' + log_filename)
2545+
2546+ while vm.isActive() is not 0:
2547+ pass
2548+
2549+ vm.undefine()
2550+ self.logger.info('Installation complete')
2551+
2552+ def _finalxml(self, tmpdir=None, xml=None):
2553+ """
2554+ Create the XML to be used for the post-installation VM.
2555+ This may be a transformation of the installation XML.
2556+ """
2557+ self.logger.info('Creating final VM XML')
2558+ if xml is None:
2559+ xml = ElementTree.ElementTree(file=self.xml)
2560+ if tmpdir is None:
2561+ tmpdir = self.tmpdir
2562+ if self.rewrite in ['all', 'minimal']:
2563+ self.logger.debug('Setting VM to reboot normally on reboot')
2564+ xml.find('on_reboot').text = 'restart'
2565+ if self.rewrite == 'all':
2566+ self.logger.debug('Removing VM install parameters')
2567+ ose = xml.find('os')
2568+ for kernel in ose.iterfind('kernel'):
2569+ ose.remove(kernel)
2570+ for initrd in ose.iterfind('initrd'):
2571+ ose.remove(initrd)
2572+ for cmdline in ose.iterfind('cmdline'):
2573+ ose.remove(cmdline)
2574+ devices = xml.find('devices')
2575+ devices.remove(devices.find('serial'))
2576+ for disk in list(devices.iterfind('disk')):
2577+ if disk.get('device') == 'cdrom':
2578+ disk.remove(disk.find('source'))
2579+ else:
2580+ self.logger.info('Not rewriting XML because rewrite is ' +
2581+ self.rewrite)
2582+ if self.debug:
2583+ xml.write(os.path.join(tmpdir, 'final.xml'))
2584+ return xml
2585+
2586+ def _tmpimage(self, image=None, tmpdir=None):
2587+ """
2588+ Create a temporary copy of the image so libvirt will lock that copy.
2589+ This allows other simultaneous processes to update the cached image.
2590+ """
2591+ if image is None:
2592+ image = self.image.image
2593+ if tmpdir is None:
2594+ tmpdir = self.tmpdir
2595+ self.logger.info('Making temp copy of install image')
2596+ tmpimage = os.path.join(tmpdir, os.path.basename(image))
2597+ self.logger.debug('Copying ' + image + ' to ' + tmpimage)
2598+ shutil.copyfile(image, tmpimage)
2599+ return tmpimage
2600+
2601+ def _create(self):
2602+ """
2603+ Create the VM, install the system, and prepare it to boot.
2604+ This primarily calls functions from CustomInstallMixin and CustomVM.
2605+ """
2606+ self.logger.info('Creating custom virtual machine')
2607+
2608+ tmpdir = tempfile.mkdtemp(prefix='/tmp/' + self.name + '_')
2609+ self.logger.debug('Working dir: ' + tmpdir)
2610+ os.chdir(tmpdir)
2611+
2612+ kernel = self._preparekernel(kernel=self.kernel, tmpdir=tmpdir)
2613+
2614+ initrd = self._prepareinitrd(initrd=self.initrd, tmpdir=tmpdir)
2615+
2616+ self._unpackinitrd(initrd=initrd, tmpdir=tmpdir)
2617+
2618+ self._setuplatecommand(tmpdir=tmpdir)
2619+
2620+ self._setuppreseed(tmpdir=tmpdir)
2621+
2622+ if self.rewrite == 'all':
2623+ self._setuplogging(tmpdir=tmpdir)
2624+ else:
2625+ self.logger.debug('Skipping logging setup because rewrite is' +
2626+ self.rewrite)
2627+
2628+ initrd = self._repackinitrd(tmpdir=tmpdir)
2629+
2630+ self._createdisks()
2631+
2632+ image = self._tmpimage(image=self.image.image, tmpdir=tmpdir)
2633+
2634+ xml = self._installxml(cmdline=self.cmdline, image=image,
2635+ initrd=initrd, kernel=kernel,
2636+ tmpdir=tmpdir, xml=self.xml)
2637+
2638+ self._installvm(lv=self.lv, tmpdir=tmpdir, xml=xml)
2639+
2640+ xml = self._finalxml(tmpdir=tmpdir, xml=xml)
2641+
2642+ self.logger.info('Setting up final VM')
2643+ self.vm = self.lv.defineXML(ElementTree.tostring(xml.getroot()))
2644+
2645+ if self.debug:
2646+ self.logger.info('Leaving temp directory '
2647+ 'because debug is enabled: ' + tmpdir)
2648+ else:
2649+ self.logger.info('Cleaning up temp directory')
2650+ shutil.rmtree(tmpdir)
2651+ return True
2652+
2653+ def _start(self):
2654+ """
2655+ Start the VM.
2656+ """
2657+ self.logger.info('Starting CustomVM')
2658+ if self.vm is not None:
2659+ if self.vm.isActive() == 0:
2660+ self.vm.create()
2661+ else:
2662+ raise UTAHVMProvisioningException('Failed to provision VM')
2663+ self.logger.info('Waiting ' + str(self.boottimeout) +
2664+ ' seconds to allow machine to boot')
2665+ try:
2666+ self.pingpoll(timeout=self.boottimeout)
2667+ except UTAHTimeout:
2668+ # Ignore timeout for ping, since depending on the network
2669+ # configuration ssh might still work despite of the ping failure.
2670+ self.logger.warning('Network connectivity (ping) failure')
2671+ self.sshpoll(timeout=self.boottimeout)
2672+ self.active = True
2673+
2674+ def destroy(self, *args, **kw):
2675+ """
2676+ Remove the machine from libvirt and remove all the disk files.
2677+ """
2678+ # TODO: make this use standard cleanup
2679+ super(CustomVM, self).destroy(*args, **kw)
2680+ self.stop(force=True)
2681+ if self.vm is not None:
2682+ self.vm.undefine()
2683+ else:
2684+ self.logger.info('VM not created')
2685+ for disk in self.disks:
2686+ os.unlink(disk['file'])
2687+ shutil.rmtree(self.directory)
2688+
2689+
2690+# See http://kennethreitz.com/blog/generate-a-random-mac-address-in-python/
2691+def random_mac_address():
2692+ """Returns a completely random Mac Address"""
2693+ mac = [0x52, 0x54, 0x00,
2694+ random.randint(0x00, 0xff),
2695+ random.randint(0x00, 0xff),
2696+ random.randint(0x00, 0xff)]
2697+ return ':'.join(map(lambda x: "%02x" % x, mac))
2698+
2699+
2700+class TinySQLiteInventory(SQLiteInventory):
2701+ """
2702+ Tiny SQLite inventory that implements request, release, and destroy.
2703+ No authentication or conflict checking currently exists.
2704+ Only suitable for VMs at present.
2705+ """
2706+ def __init__(self, *args, **kw):
2707+ """
2708+ Initialize simple database.
2709+ """
2710+ super(TinySQLiteInventory, self).__init__(*args, **kw)
2711+ self.connection.execute(
2712+ 'CREATE TABLE IF NOT EXISTS '
2713+ 'machines(machineid INTEGER PRIMARY KEY, state TEXT)')
2714+
2715+ def request(self, machinetype=CustomVM, *args, **kw):
2716+ """
2717+ Takes a Machine class as machinetype, and passes the newly generated
2718+ machineid along with all other arguments to that class's constructor,
2719+ returning the resulting object.
2720+ """
2721+ cursor = self.connection.cursor()
2722+ cursor.execute("INSERT INTO machines (state) VALUES ('provisioned')")
2723+ machineid = cursor.lastrowid
2724+ return machinetype(machineid=machineid, *args, **kw)
2725+
2726+ def release(self, machineid):
2727+ """
2728+ Updates the database to indicate the machine is available.
2729+ """
2730+ if self.connection.execute(
2731+ "UPDATE machines SET state='available' WHERE machineid=?",
2732+ [machineid]):
2733+ return True
2734+ else:
2735+ return False
2736+
2737+ def destroy(self, machineid):
2738+ """
2739+ Updates the database to indicate the machine is destroyed, but does not
2740+ destroy the machine.
2741+ """
2742+ if self.connection.execute(
2743+ "UPDATE machines SET state='destroyed' ""WHERE machineid=?",
2744+ [machineid]):
2745+ return True
2746+ else:
2747+ return False

Subscribers

People subscribed via source and target branches