Auto Upgrade Testing

Merge lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option into lp:auto-upgrade-testing

  1. adding-keep-overlay-option
  2. Merge into autopkgtest
Proposed by Heber Parrucci
Status: Merged
Approved by: Jean-Baptiste Lallement
Approved revision: 96
Merged at revision: 89
Proposed branch: lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option
Merge into: lp:auto-upgrade-testing
Diff against target: 860 lines (+714/-8)
8 files modified
.flake8 (+5/-0)
.project-flake8.sh (+19/-1)
debian/control (+6/-0)
upgrade_testing/command_line.py (+14/-4)
upgrade_testing/provisioning/_provisionconfig.py (+3/-0)
upgrade_testing/provisioning/backends/_qemu.py (+201/-3)
upgrade_testing/provisioning/backends/_ssh.py (+260/-0)
upgrade_testing/provisioning/executors.py (+206/-0)
To merge this branch: bzr merge lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option
Reviewer Review Type Date Requested Status
Jean-Baptiste Lallement Approve
platform-qa-bot continuous-integration Approve
Review via email: mp+327450@code.launchpad.net

Commit message

Adding keep-overlay option to persist the resulting image file in order to be able to run system tests after the upgrade is done.

Description of the change

Adding keep-overlay option to persist the resulting image file in order to be able to run system tests after the upgrade is done.

To post a comment you must log in.
Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:89
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/39/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/23/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/23/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2359/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/39/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
90. By Heber Parrucci

Fixing flake8 issues

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:90
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/40/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/24/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/24/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2360/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/40/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:90
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/41/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/25/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/25/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2361/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/41/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
91. By Heber Parrucci

fixing virtualenv issue

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:91
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/42/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/26/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/26/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2362/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/42/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
92. By Heber Parrucci

running flake8 without virtualenv

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:92
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/43/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/27/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/27/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2363/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/43/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
93. By Heber Parrucci

removing deactivate from script

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:93
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/44/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/28/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/28/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2364/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/44/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
94. By Heber Parrucci

fixing import error when running selftests

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:94
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/45/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/29/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/29/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2365/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/45/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
95. By Heber Parrucci

Adding missing dependency

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

FAILED: Continuous integration, rev:95
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/46/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/30/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/30/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2366/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/46/rebuild

review: Needs Fixing (continuous-integration)
lp:~heber013/auto-upgrade-testing/adding-keep-overlay-option updated
96. By Heber Parrucci

Adding missing dependecies

Revision history for this message
platform-qa-bot (platform-qa-bot) wrote :

PASSED: Continuous integration, rev:96
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/47/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/31
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-amd64-package/31/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/31
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-zesty-i386-package/31/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2367/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/auto-upgrade-testing-ci/47/rebuild

review: Approve (continuous-integration)
Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

Thanks. Approved!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file '.flake8'
2--- .flake8 1970-01-01 00:00:00 +0000
3+++ .flake8 2017-07-14 19:31:31 +0000
4@@ -0,0 +1,5 @@
5+[flake8]
6+ignore = E999, W503
7+exclude = venv_tests/
8+max-complexity = 7
9+max-line-length = 80
10
11=== modified file '.project-flake8.sh'
12--- .project-flake8.sh 2016-03-03 04:29:09 +0000
13+++ .project-flake8.sh 2017-07-14 19:31:31 +0000
14@@ -1,2 +1,20 @@
15 # Helpful doc: http://pep8.readthedocs.org/en/latest/intro.html#error-codes
16-python3 -m flake8.run --ignore=W503 .
17+#!/bin/bash
18+RED='\033[0;31m'
19+GREEN='\033[0;32m'
20+
21+OUTPUT=flake8_output.txt
22+if [ -f $OUTPUT ]; then
23+ rm $OUTPUT
24+fi
25+
26+python3 -m flake8 --output-file=$OUTPUT .
27+
28+result=$?
29+if [ $result != 0 ]; then
30+ echo -e -n "${RED}Flake8 errors. Check flake8 output for more information\n"
31+ cat $OUTPUT
32+ exit 1;
33+else
34+ echo -e -n "${GREEN}Congratulations!!! Static check PASSED\n"
35+fi
36
37=== modified file 'debian/control'
38--- debian/control 2017-02-03 20:16:09 +0000
39+++ debian/control 2017-07-14 19:31:31 +0000
40@@ -7,6 +7,9 @@
41 python3-all-dev (>= 3.4),
42 python3-flake8,
43 python3-lxc,
44+ python3-paramiko,
45+ python3-pexpect,
46+ python3-retrying,
47 python3-setuptools,
48 python3-yaml,
49 Standards-Version: 3.9.3
50@@ -22,7 +25,10 @@
51 lxc-templates,
52 python3-yaml,
53 python3-lxc,
54+ python3-paramiko,
55+ python3-pexpect,
56 python3-pkg-resources,
57+ python3-retrying,
58 android-tools-adb,
59 phablet-tools,
60 Description: Test release upgrades in a virtual environment
61
62=== modified file 'upgrade_testing/command_line.py'
63--- upgrade_testing/command_line.py 2017-05-23 09:54:22 +0000
64+++ upgrade_testing/command_line.py 2017-07-14 19:31:31 +0000
65@@ -69,6 +69,11 @@
66 parser.add_argument(
67 '--adt-args', '-a', default='',
68 help='Arguments to pass through to the autopkgtest runner.')
69+ parser.add_argument(
70+ '--keep-overlay', '-k',
71+ default=False,
72+ action='store_true',
73+ help='Whether to keep the resulting overlay image')
74 return parser.parse_args()
75
76
77@@ -137,7 +142,8 @@
78 print('\n'.join(output))
79
80
81-def execute_adt_run(testsuite, testrun_files, output_dir, adt_args=''):
82+def execute_adt_run(testsuite, testrun_files, output_dir, adt_args='',
83+ keep_overlay=False):
84 """Prepare the autopkgtest to execute.
85
86 Copy all the files into the expected place etc.
87@@ -153,13 +159,14 @@
88 output_dir,
89 testsuite.backend_args,
90 adt_args,
91+ keep_overlay
92 )
93 subprocess.check_call(adt_run_command)
94
95
96 def get_adt_run_command(
97 provisioning, testrun_files, results_dir, backend_args=[],
98- adt_args=''):
99+ adt_args='', keep_overlay=False):
100 """Construct the adt command to run.
101
102 :param provisioning: upgrade_testing.provisioning.ProvisionSpecification
103@@ -210,7 +217,8 @@
104 )
105
106 backend_args = provisioning.get_adt_run_args(
107- tmp_dir=testrun_files.testrun_tmp_dir
108+ tmp_dir=testrun_files.testrun_tmp_dir,
109+ keep_overlay=keep_overlay
110 ) + backend_args
111
112 return adt_cmd + ['---'] + backend_args
113@@ -263,7 +271,9 @@
114 output_dir = get_output_dir(args)
115
116 execute_adt_run(testsuite, created_files, output_dir,
117- args.adt_args)
118+ args.adt_args, args.keep_overlay)
119+
120+ testsuite.provisioning.close()
121
122 display_results(output_dir)
123
124
125=== modified file 'upgrade_testing/provisioning/_provisionconfig.py'
126--- upgrade_testing/provisioning/_provisionconfig.py 2016-03-18 06:15:14 +0000
127+++ upgrade_testing/provisioning/_provisionconfig.py 2017-07-14 19:31:31 +0000
128@@ -58,6 +58,9 @@
129 """Provision the stored backend."""
130 return self.backend.create(adt_base_path)
131
132+ def close(self):
133+ return self.backend.close() if hasattr(self.backend, 'close') else None
134+
135 def get_adt_run_args(self, **kwargs):
136 """Return list with the adt args for this provisioning backend."""
137 raise NotImplementedError()
138
139=== modified file 'upgrade_testing/provisioning/backends/_qemu.py'
140--- upgrade_testing/provisioning/backends/_qemu.py 2016-05-24 18:32:17 +0000
141+++ upgrade_testing/provisioning/backends/_qemu.py 2017-07-14 19:31:31 +0000
142@@ -18,16 +18,60 @@
143
144 import logging
145 import os
146+import shlex
147+import shutil
148+import signal
149+import subprocess
150+import tempfile
151+import threading
152+
153+from paramiko.ssh_exception import SSHException
154
155 from upgrade_testing.provisioning._util import run_command_with_logged_output
156-from upgrade_testing.provisioning.backends._base import ProviderBackend
157+from upgrade_testing.provisioning.backends._ssh import SshBackend
158
159 CACHE_DIR = '/var/cache/auto-upgrade-testing'
160+OVERLAY_DIR = os.path.join(CACHE_DIR, 'overlay')
161+QEMU_LAUNCH_OPTS = (
162+ '{qemu} -m {ram} -smp {cpu} -pidfile {workdir}/qemu.pid -localtime '
163+ '-cpu core2duo -enable-kvm '
164+)
165+QEMU_SYSTEM_AMD64 = 'qemu-system-x86_64'
166+QEMU_SYSTEM_I386 = 'qemu-system-i386'
167+ARCH_AMD64 = 'amd64'
168+ARCH_I386 = 'i386'
169+QEMU_DISPLAY_OPTS = (
170+ '-display sdl '
171+)
172+QEMU_DISPLAY_VGA_OPTS = (
173+ '-vga qxl '
174+)
175+QEMU_SOUND_OPTS = (
176+ '-soundhw all '
177+)
178+QEMU_DISPLAY_HEADLESS = (
179+ '-display none '
180+)
181+QEMU_NET_OPTS = (
182+ '-net nic,model=virtio -net user'
183+)
184+QEMU_PORT_OPTS = (
185+ ',hostfwd=tcp::{port}-:22 '
186+)
187+QEMU_DISK_IMAGE_OPTS = (
188+ '-drive file={disk_img},if=virtio '
189+)
190+QEMU_DISK_IMAGE_OVERLAY_OPTS = (
191+ '-drive file={overlay_img},cache=unsafe,if=virtio,index=0 '
192+)
193+DEFAULT_RAM = '2G'
194+DEFAULT_CPU = '2'
195+HEADLESS = True
196
197 logger = logging.getLogger(__name__)
198
199
200-class QemuBackend(ProviderBackend):
201+class QemuBackend(SshBackend):
202
203 # We can change the Backends to require just what they need. In this case
204 # it would be distribution, release name (, arch)
205@@ -38,10 +82,14 @@
206 details.
207
208 """
209+ super().__init__(release, arch, image_name, build_args)
210 self.release = release
211 self.arch = arch
212 self.image_name = image_name
213 self.build_args = build_args
214+ self.working_dir = None
215+ self.qemu_runner = None
216+ self.find_free_port()
217
218 def available(self):
219 """Return true if a qemu exists that matches the provided args.
220@@ -73,9 +121,60 @@
221 os.rename(initial_image_path, final_image_path)
222 logger.info('Image created.')
223
224- def get_adt_run_args(self, **kwargs):
225+ def close(self):
226+ if self.qemu_runner:
227+ try:
228+ self.shutdown()
229+ except PermissionError:
230+ print('Shutdown sudo command failed. '
231+ 'Check password: "{}".'.format(self.password))
232+ self.stop_qemu()
233+ except SSHException:
234+ self.stop_qemu()
235+ finally:
236+ self.qemu_runner.join(timeout=5)
237+ shutil.rmtree(self.working_dir)
238+ self.working_dir = None
239+ self.qemu_runner = None
240+ super().close()
241+
242+ def reboot(self):
243+ self.close()
244+ self.connect()
245+
246+ def stop_qemu(self):
247+ pid_file = os.path.join(self.working_dir, 'qemu.pid')
248+ with open(pid_file) as f:
249+ pid = int(f.read().strip())
250+ os.kill(pid, signal.SIGTERM)
251+
252+ def get_adt_run_args(self, keep_overlay=False, **kwargs):
253+ if keep_overlay:
254+ self.qemu_runner = self.launch_qemu(
255+ self.image_name,
256+ kwargs.get('ram', DEFAULT_RAM),
257+ kwargs.get('cpu', DEFAULT_CPU),
258+ kwargs.get('headless', HEADLESS),
259+ port=self.port,
260+ overlay=os.path.join(OVERLAY_DIR,
261+ self.image_name))
262+ super().connect()
263+ return super().get_adt_run_args()
264 return ['qemu', os.path.join(CACHE_DIR, self.image_name)]
265
266+ def create_overlay_image(self, overlay_img):
267+ """Create an overlay image for specified base image."""
268+ overlay_dir = os.path.dirname(overlay_img)
269+ if os.path.isfile(overlay_img):
270+ os.remove(overlay_img)
271+ elif not os.path.isdir(overlay_dir):
272+ os.makedirs(overlay_dir)
273+ subprocess.check_call(
274+ ['qemu-img', 'create', '-f', 'qcow2', '-b',
275+ os.path.join(CACHE_DIR, self.image_name),
276+ overlay_img])
277+ subprocess.check_call(['sudo', 'chmod', '777', overlay_img])
278+
279 @property
280 def name(self):
281 return 'qemu'
282@@ -85,3 +184,102 @@
283 classname=self.__class__.__name__,
284 release=self.release
285 )
286+
287+ @staticmethod
288+ def get_architecture():
289+ """Return architecture string for system."""
290+ return subprocess.check_output(
291+ ['dpkg', '--print-architecture']).decode().strip()
292+
293+ def get_qemu_path(self):
294+ """Return path of qemu-system executable for system."""
295+ if self.get_architecture() == ARCH_AMD64:
296+ target = QEMU_SYSTEM_AMD64
297+ else:
298+ target = QEMU_SYSTEM_I386
299+ return subprocess.check_output(['which', target]).decode().strip()
300+
301+ def get_disk_args(self, overlay):
302+ """Return qemu-system disk args. If overlay is specified then an overlay
303+ image at that path will be created and specified in returned arguments.
304+ If no overlay is none then the base image will be returned in
305+ the arguments.
306+ :param overlay: Path of overlay image to use, otherwise None
307+ if not needed.
308+ :return: Disk image arguments as string.
309+ """
310+ if overlay:
311+ self.create_overlay_image(overlay)
312+ return QEMU_DISK_IMAGE_OVERLAY_OPTS.format(overlay_img=overlay)
313+ else:
314+ return QEMU_DISK_IMAGE_OPTS.format(disk_img=self.image_name)
315+
316+ @staticmethod
317+ def get_display_args(headless):
318+ """Return qemu-system display arguments based on headless parameter.
319+ :param headless: Whether qemu-system should run in headless mode or not.
320+ :return: Display parameters for required display state.
321+ """
322+ if headless:
323+ return QEMU_DISPLAY_HEADLESS + QEMU_DISPLAY_VGA_OPTS
324+ else:
325+ return QEMU_DISPLAY_OPTS + QEMU_DISPLAY_VGA_OPTS + QEMU_SOUND_OPTS
326+
327+ def launch_qemu(self, img, ram, cpu, headless, port, overlay):
328+ """Boot the qemu from a different thread to stop this thread from being
329+ blocked whilst the qemu is running.
330+ """
331+ self.working_dir = tempfile.mkdtemp()
332+ runner = threading.Thread(
333+ target=self._launch_qemu,
334+ args=(self.working_dir, img, ram, cpu, headless, port, overlay))
335+ runner.start()
336+ return runner
337+
338+ def _launch_qemu(self, working_dir, disk_image_path, ram, cpu, headless,
339+ port=None, overlay=None):
340+ """Launch qemu-system to install the iso file into the disk image.
341+ :param working_dir: Working directory to use.
342+ :param disk_image_path: Path of the disk image file used for
343+ installation.
344+ :param ram: Amount of ram allocated to qemu.
345+ :param cpu: Number of cpus allocated to qemu.
346+ :param headless: Whether to run installer in headless mode or not.
347+ :param port: Host port number to enable port forwarding to qemu port 22.
348+
349+ """
350+ cmd = self.get_qemu_launch_command(
351+ working_dir, disk_image_path, ram, cpu, headless,
352+ port, overlay)
353+ print(' '.join(cmd))
354+ subprocess.check_call(cmd)
355+
356+ def get_qemu_launch_command(self, work_dir, disk_img, ram, cpu,
357+ headless, port=None, overlay=None):
358+ """Return command to launch qemu process using optional install parameters.
359+ :param work_dir: Working directory to use.
360+ :param disk_img: Path of the disk image file used for installation.
361+ :param ram: Amount of ram allocated to qemu.
362+ :param cpu: Number of cpus allocated to qemu.
363+ :param headless: Whether to run installer in headless mode or not.
364+ :return: Qemu launch command string.
365+ :param port: Host port number to enable port forwarding to qemu port 22.
366+ :param overlay: path to the overlay image to be created
367+ """
368+ # Create command base with resource parameters
369+ cmd = QEMU_LAUNCH_OPTS.format(
370+ qemu=self.get_qemu_path(), ram=ram, cpu=cpu, disk_img=disk_img,
371+ workdir=work_dir)
372+ # Get disk args including overlay image if specified
373+ cmd += self.get_disk_args(overlay)
374+ # Add display parameters
375+ cmd += self.get_display_args(headless)
376+ # Add network. This must preceed the port forwarding option.
377+ cmd += QEMU_NET_OPTS
378+ # Add port forwarding if specified
379+ if port:
380+ cmd += QEMU_PORT_OPTS.format(port=port)
381+ else:
382+ # Add space to separate options
383+ cmd += ' '
384+ return shlex.split(cmd)
385
386=== added file 'upgrade_testing/provisioning/backends/_ssh.py'
387--- upgrade_testing/provisioning/backends/_ssh.py 1970-01-01 00:00:00 +0000
388+++ upgrade_testing/provisioning/backends/_ssh.py 2017-07-14 19:31:31 +0000
389@@ -0,0 +1,260 @@
390+#
391+# Ubuntu Upgrade Testing
392+# Copyright (C) 2015 Canonical
393+#
394+# This program is free software: you can redistribute it and/or modify
395+# it under the terms of the GNU General Public License as published by
396+# the Free Software Foundation, either version 3 of the License, or
397+# (at your option) any later version.
398+#
399+# This program is distributed in the hope that it will be useful,
400+# but WITHOUT ANY WARRANTY; without even the implied warranty of
401+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
402+# GNU General Public License for more details.
403+#
404+# You should have received a copy of the GNU General Public License
405+# along with this program. If not, see <http://www.gnu.org/licenses/>.
406+#
407+
408+import logging
409+import os
410+import socket
411+import subprocess
412+
413+import time
414+
415+import errno
416+import pexpect
417+from retrying import retry
418+
419+from upgrade_testing.provisioning.backends._base import ProviderBackend
420+from upgrade_testing.provisioning.executors import SSHExecutor
421+
422+CACHE_DIR = '/var/cache/auto-upgrade-testing'
423+
424+logger = logging.getLogger(__name__)
425+TIMEOUT_CMD = 60
426+TIMEOUT_CONNECT = 120
427+TIMEOUT_WAIT_FOR_DEVICE = 120
428+
429+
430+class SshBackend(ProviderBackend):
431+
432+ # We can change the Backends to require just what they need. In this case
433+ # it would be distribution, release name (, arch)
434+ def __init__(self, release, arch, image_name, build_args=[],
435+ username=None, password=None, device_ip=None):
436+ """Provide backend capabilities as requested in the provision spec.
437+
438+ :param provision_spec: ProvisionSpecification object containing backend
439+ details.
440+
441+ """
442+ self.release = release
443+ self.arch = arch
444+ self.image_name = image_name
445+ self.build_args = build_args
446+ self.executor = SSHExecutor()
447+ self.username = username or 'ubuntu'
448+ self.password = password or 'ubuntu'
449+ self.connected = False
450+ self.key_file = None
451+ self.device_ip = device_ip or 'localhost'
452+ self.port = -1
453+
454+ def available(self):
455+ """Return true if a qemu exists that matches the provided args.
456+
457+ """
458+ image_name = self.image_name
459+ logger.info('Checking for {}'.format(image_name))
460+ return image_name in os.listdir(CACHE_DIR)
461+
462+ def create(self, adt_base_path):
463+ raise NotImplementedError('Cannot create ssh backend directly')
464+
465+ def get_adt_run_args(self, **kwargs):
466+ return ['ssh', '--port', str(self.port),
467+ '--login', self.username,
468+ '--password', self.password,
469+ '--identity', self.key_file,
470+ '--hostname',
471+ self.device_ip,
472+ '--reboot']
473+
474+ @property
475+ def name(self):
476+ return 'ssh'
477+
478+ def __repr__(self):
479+ return '{classname}(release={release})'.format(
480+ classname=self.__class__.__name__,
481+ release=self.release
482+ )
483+
484+ port_from = 22220
485+ port_to = 23000
486+
487+ def connect(self, timeout=TIMEOUT_CONNECT):
488+ if not self.connected:
489+ self.enable_ssh()
490+ self.executor.connect(
491+ self.username, self.password, self.port, self.device_ip,
492+ timeout)
493+ self.connected = True
494+
495+ def close(self):
496+ if self.connected:
497+ self.executor.close()
498+ self.connected = False
499+
500+ def reboot(self):
501+ self.executor.reboot()
502+
503+ def shutdown(self):
504+ self.executor.shutdown()
505+
506+ def put(self, src, dst):
507+ self.executor.put(src, dst)
508+
509+ def run(self, command, timeout=TIMEOUT_CMD, log_stdout=True):
510+ return self.executor.run(command, timeout, log_stdout)
511+
512+ def run_sudo(self, command, timeout=TIMEOUT_CMD, log_stdout=True):
513+ return self.executor.run_sudo(command, timeout, log_stdout)
514+
515+ def find_free_port(self):
516+ for port in range(self.port_from, self.port_to):
517+ try:
518+ s = socket.create_connection(('127.0.0.1', port))
519+ except socket.error as e:
520+ if e.errno == errno.ECONNREFUSED:
521+ # This means port is not currently used, so use this one
522+ self.port = port
523+ return
524+ else:
525+ pass
526+ else:
527+ # port is already taken
528+ s.close()
529+ raise RuntimeError('Could not find free port for SSH connection.')
530+
531+ def enable_ssh(self):
532+ """Enable ssh using public key."""
533+ self._wait_for_device()
534+ self._get_ssh_id_path()
535+ if not self._try_public_key_login():
536+ self._update_device_host_key()
537+ self._copy_ssh_id_to_device()
538+ self._verify_ssh_connect()
539+
540+ def _wait_for_device(self, timeout=TIMEOUT_CONNECT):
541+ end = time.time() + timeout
542+ while time.time() < end:
543+ try:
544+ s = socket.create_connection(
545+ (self.device_ip, str(self.port)), timeout)
546+ except ConnectionRefusedError:
547+ time.sleep(1)
548+ else:
549+ s.close()
550+ return
551+ raise TimeoutError(
552+ 'Could not connect to {} '
553+ 'port {}.'.format(self.device_ip, self.port))
554+
555+ def _update_device_host_key(self):
556+ hosts_path = os.path.expanduser('~/.ssh/known_hosts')
557+ subprocess.call([
558+ 'ssh-keygen', '-f', hosts_path, '-R',
559+ '[{}]:{}'.format(self.device_ip, self.port)])
560+
561+ @retry(stop_max_attempt_number=20, wait_fixed=2000,
562+ retry_on_exception=lambda exception: isinstance(
563+ exception, RuntimeError))
564+ def _copy_ssh_id_to_device(self):
565+ pub_path = '{}.pub'.format(self.key_file)
566+ home_ssh = '/home/{u}/.ssh'.format(u=self.username)
567+ authorized_keys = os.path.join(home_ssh, 'authorized_keys')
568+ self._run(['mkdir', '-p', home_ssh])
569+ self._put(pub_path, authorized_keys)
570+ self._run(
571+ ['chown', '{u}:{u}'.format(u=self.username), '-R', home_ssh])
572+ self._run(['chmod', '700', home_ssh])
573+ self._run(['chmod', '600', authorized_keys])
574+
575+ def _get_ssh_id_path(self):
576+ match = False
577+ for id in ['~/.ssh/id_rsa', '~/.ssh/id_autopkgtest']:
578+ path = os.path.expanduser(id)
579+ if os.path.exists(path):
580+ match = True
581+ break
582+ if not match:
583+ subprocess.check_call([
584+ 'ssh-keygen', '-q', '-t', 'rsa', '-f', path, '-N', ''])
585+ self.key_file = path
586+
587+ def _try_public_key_login(self):
588+ """Try and log in using public key. If this succeeds then there is no
589+ need to do any further ssh setup.
590+ :return: True if login was successful, False otherwise
591+ """
592+ cmd = ' '.join(['ssh', '-p', str(self.port),
593+ '-o', 'UserKnownHostsFile=/dev/null',
594+ '-o', 'StrictHostKeyChecking=no',
595+ '-i', self.key_file, '-l', self.username,
596+ self.device_ip])
597+ child = pexpect.spawn(cmd)
598+ try:
599+ index = child.expect(
600+ ['\$', 'password', 'denied'], timeout=TIMEOUT_CONNECT)
601+ except (pexpect.exceptions.TIMEOUT, pexpect.exceptions.EOF):
602+ index = -1
603+ finally:
604+ child.close()
605+ return index == 0
606+
607+ def _verify_ssh_connect(self):
608+ """Verify that an ssh connection can be established without using
609+ password.
610+ """
611+ for count in range(20):
612+ if self._try_public_key_login():
613+ return
614+ else:
615+ time.sleep(1)
616+ raise RuntimeError('Could not create ssh connection')
617+
618+ def _run(self, commands, timeout=TIMEOUT_CMD):
619+ """Run a command setting up an ssh connection using password."""
620+ ssh_cmd = [
621+ 'ssh', '-o', 'StrictHostKeyChecking=no', '-p', str(self.port),
622+ '{}@{}'.format(self.username, self.device_ip)]
623+ self._run_with_password(ssh_cmd + commands, self.password, timeout)
624+
625+ def _put(self, src, dst, timeout=TIMEOUT_CMD):
626+ """Put file onto device setting up an ssh connection using password."""
627+ scp_cmd = [
628+ 'scp', '-o', 'StrictHostKeyChecking=no', '-P', str(self.port),
629+ src, '{}@{}:{}'.format(self.username, self.device_ip, dst)]
630+ self._run_with_password(scp_cmd, self.password, timeout)
631+
632+ def _run_with_password(self, commands, password, timeout=TIMEOUT_CMD):
633+ """Run command expecting a password prompt to be displayed."""
634+ command = ' '.join(commands)
635+ child = pexpect.spawn(command)
636+ try:
637+ child.expect('password', timeout=timeout)
638+ except pexpect.exceptions.EOF:
639+ # No password prompt is displayed, so just continue
640+ pass
641+ else:
642+ child.sendline(password)
643+ if child.expect([pexpect.EOF, 'denied'], timeout=timeout):
644+ raise PermissionError(
645+ 'Check password is correct: "{}"'.format(password))
646+ finally:
647+ child.close()
648+ if child.exitstatus:
649+ raise RuntimeError('Error running {}'.format(command))
650
651=== added file 'upgrade_testing/provisioning/executors.py'
652--- upgrade_testing/provisioning/executors.py 1970-01-01 00:00:00 +0000
653+++ upgrade_testing/provisioning/executors.py 2017-07-14 19:31:31 +0000
654@@ -0,0 +1,206 @@
655+#
656+# Ubuntu Upgrade Testing
657+# Copyright (C) 2017 Canonical
658+#
659+# This program is free software: you can redistribute it and/or modify
660+# it under the terms of the GNU General Public License as published by
661+# the Free Software Foundation, either version 3 of the License, or
662+# (at your option) any later version.
663+#
664+# This program is distributed in the hope that it will be useful,
665+# but WITHOUT ANY WARRANTY; without even the implied warranty of
666+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
667+# GNU General Public License for more details.
668+#
669+# You should have received a copy of the GNU General Public License
670+# along with this program. If not, see <http://www.gnu.org/licenses/>.
671+#
672+
673+
674+import os
675+import sys
676+import time
677+from abc import ABCMeta
678+from abc import abstractmethod
679+
680+import paramiko
681+
682+TIMEOUT_CMD = 60
683+TIMEOUT_CONNECT = 120
684+TIMEOUT_WAIT_FOR_DEVICE = 120
685+
686+
687+class Result:
688+ """Result of command with status and output properties."""
689+
690+ def __init__(self):
691+ self.status = None
692+ self.output = ''
693+
694+
695+class SSHClient:
696+ """This class manages the paramiko ssh client"""
697+
698+ def __init__(self):
699+ """ The ssh can be initialized either through a password or with a
700+ private key file and the passphrase
701+ :param hostname: The hostname to connect
702+ :param user: The remote user in the host
703+ :param keyfile: The private key used to connect to the remote host
704+ :param password: The password used to connect to the remote host
705+ :param timeout: An optional timeout (in seconds) for the TCP connect
706+ """
707+ self.client = paramiko.SSHClient()
708+ self.client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
709+
710+ def connect(self, hostname, user, password, port, timeout=60):
711+ """Connect to remote host."""
712+ timeout = float(timeout)
713+ self.client.connect(
714+ hostname, username=user, password=password, port=port,
715+ timeout=timeout, banner_timeout=timeout)
716+
717+ def close(self):
718+ """Close the connection"""
719+ self.client.close()
720+
721+ def run(self, cmd, timeout=TIMEOUT_CMD, log_stdout=True):
722+ """Run a command in the remote host.
723+ :param cmd: Command to run.
724+ :param timeout: Period to wait before raising TimeoutError.
725+ :param log_stdout: Whether to log output to stdout.
726+ :return: Result object containing command output and status code.
727+ """
728+ channel = self.client.get_transport().open_session()
729+ channel.set_combine_stderr(True)
730+ end = time.time() + timeout
731+ result = Result()
732+ channel.exec_command(cmd)
733+ while (not channel.exit_status_ready() or (
734+ channel.exit_status_ready() and channel.recv_ready())):
735+ if channel.recv_ready():
736+ self._process_output(
737+ result, log_stdout, channel.recv(1024).decode())
738+ elif time.time() > end:
739+ print('Timeout waiting {} seconds for command to '
740+ 'complete: {}'.format(timeout, cmd))
741+ raise TimeoutError
742+ time.sleep(0.2)
743+ # Add a new line after command has completed to ensure output
744+ # is separated.
745+ self._process_output(result, log_stdout, '\n')
746+ result.status = channel.recv_exit_status()
747+ return result
748+
749+ def _process_output(self, result, log_stdout, content):
750+ """Save output to result and print to stdout if required."""
751+ result.output += content
752+ if log_stdout:
753+ sys.stdout.write(content)
754+ sys.stdout.flush()
755+
756+ def put(self, local_path, remote_path):
757+ """Copy a file through sftp from the local_path to the remote_path"""
758+ if not os.path.isfile(local_path):
759+ raise RuntimeError('File to copy does not exist')
760+
761+ with self.client.open_sftp() as sftp:
762+ sftp.put(local_path, remote_path)
763+
764+ def get(self, remote_path, local_path):
765+ """Copy a file through sftp from the remote_path to the local_path"""
766+ with self.client.open_sftp() as sftp:
767+ sftp.get(remote_path, local_path)
768+
769+ if not os.path.isfile(local_path):
770+ raise RuntimeError('File couldn\'t be copied')
771+
772+
773+class Executor:
774+ __metaclass__ = ABCMeta
775+ """Base class for all target executors."""
776+
777+ @abstractmethod
778+ def connect(self, username, password, port, host=None, timeout=None):
779+ pass
780+
781+ @abstractmethod
782+ def close(self):
783+ pass
784+
785+ @abstractmethod
786+ def run(self, cmd, timeout=None, log_stdout=True):
787+ pass
788+
789+ @abstractmethod
790+ def run_sudo(self, cmd, timeout=None, log_stdout=True):
791+ pass
792+
793+ def reboot(self):
794+ result = self.run_sudo('shutdown -r now')
795+ if result.status > 0:
796+ raise PermissionError('Reboot failed, check password.')
797+
798+ def shutdown(self):
799+ result = self.run_sudo('shutdown now')
800+ if result.status > 0:
801+ raise PermissionError('Shutdown failed, check password.')
802+
803+ @abstractmethod
804+ def wait_for_device(self, timeout=None):
805+ pass
806+
807+ @abstractmethod
808+ def put(self, localpath, remotepath):
809+ pass
810+
811+ @abstractmethod
812+ def get(self, remotepath, localpath):
813+ pass
814+
815+ def _get_sudo_command(self, cmd):
816+ command = 'sudo {}'.format(cmd)
817+ if self.password:
818+ command = 'echo {} | sudo -S {}'.format(self.password, cmd)
819+ return command
820+
821+
822+class SSHExecutor(Executor):
823+
824+ def __init__(self):
825+ self.ssh_client = SSHClient()
826+
827+ def connect(self, username, password, port, host='localhost',
828+ timeout=TIMEOUT_CONNECT):
829+ self.password = password
830+ count = max(1, timeout)
831+ for attempt in range(count):
832+ try:
833+ self.ssh_client.connect(
834+ host, username, password, port, timeout)
835+ except TypeError:
836+ # This can happen when target not yet running so just try again
837+ time.sleep(1)
838+ except paramiko.ssh_exception.AuthenticationException:
839+ raise
840+ else:
841+ return
842+ raise RuntimeError('Could not connect to target.')
843+
844+ def close(self):
845+ self.ssh_client.close()
846+
847+ def _run(self, cmd, timeout=TIMEOUT_CMD, log_stdout=True):
848+ return self.ssh_client.run(cmd, timeout, log_stdout)
849+
850+ def run(self, cmd, timeout=TIMEOUT_CMD, log_stdout=True):
851+ return self._run(cmd, timeout, log_stdout)
852+
853+ def run_sudo(self, cmd, timeout=TIMEOUT_CMD, log_stdout=True):
854+ return self._run(self._get_sudo_command(cmd), timeout, log_stdout)
855+
856+ def put(self, localpath, remotepath):
857+ self.ssh_client.put(localpath, remotepath)
858+
859+ def get(self, remotepath, localpath):
860+ self.ssh_client.get(remotepath, localpath)

Subscribers

People subscribed via source and target branches