Merge lp:~terceiro/lava-dispatcher/nexus into lp:lava-dispatcher

[Antonio Terceiro]

This is work in progress; you will note the number of FIXME's and TODO's in lava_dispatcher/device/nexus.py. ;-)

With the current state, I can already run a full boot test job, like the one in the commit message of rev 475:

http://bazaar.launchpad.net/~terceiro/lava-dispatcher/nexus/revision/475

It would be nice to receive comments about the changes that were needed to other files outside of the Nexus implementation itself.

[Michael Hudson-Doyle (mwhudson)]

Antonio Terceiro <email address hidden> writes:

> Antonio Terceiro has proposed merging lp:~terceiro/lava-dispatcher/nexus into lp:lava-dispatcher.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~terceiro/lava-dispatcher/nexus/+merge/136807
>
> This is work in progress; you will note the number of FIXME's and TODO's in lava_dispatcher/device/nexus.py. ;-)

This looks pretty good on the whole!

> With the current state, I can already run a full boot test job, like the one in the commit message of rev 475:
>
> http://bazaar.launchpad.net/~terceiro/lava-dispatcher/nexus/revision/475
>
> It would be nice to receive comments about the changes that were needed to other files outside of the Nexus implementation itself.

I think they mostly seem reasonable. I'd be happy to see them merged
without the nexus bits if that would make your life easier.

> === added file 'lava_dispatcher/device/nexus.py'
> --- lava_dispatcher/device/nexus.py 1970-01-01 00:00:00 +0000
> +++ lava_dispatcher/device/nexus.py 2012-11-28 22:45:26 +0000
> @@ -0,0 +1,106 @@
> +# Copyright (C) 2012 Linaro Limited
> +#
> +# Author: Antonio Terceiro <email address hidden>
> +#
> +# This file is part of LAVA Dispatcher.
> +#
> +# LAVA Dispatcher is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# LAVA Dispatcher is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along
> +# with this program; if not, see <http://www.gnu.org/licenses>.
> +
> +import subprocess
> +import pexpect
> +from time import sleep
> +from lava_dispatcher.device.target import (
> + Target
> +)
> +from lava_dispatcher.downloader import (
> + download_image
> +)
> +from lava_dispatcher.utils import (
> + logging_spawn
> +)
> +
> +class NexusTarget(Target):
> +
> + def __init__(self, context, config):
> + super(NexusTarget, self).__init__(context, config)
> +
> + def deploy_android(self, boot, system, userdata):
> + sdir = self.scratch_dir
> +
> + # TODO it seems boot, system and userdata are usually .tar.gz files,
> + # and I am currently assuming they are .img and flashing them directly

Would be good to talk to the android team about what to do here I guess.

> + boot = download_image(boot, self.context, sdir, decompress=False)
> + # FIXME uncomment these two - skipping them makes testing faster
> + #system = download_image(system, self.context, sdir, decompress=False)
> + #userdata = download_image(userdata, self.context, sdir, decompress=False)
> +
> + self.reboot()
> + sleep(10)
> +
> + self.fastboot(['erase', 'boot'])
> + # FIXME uncomment these two - skipp...

[Antonio Terceiro (terceiro)]

On Sun, Dec 02, 2012 at 10:16:20PM -0000, Michael Hudson-Doyle wrote:
> Antonio Terceiro <email address hidden> writes:
>
> > Antonio Terceiro has proposed merging lp:~terceiro/lava-dispatcher/nexus into lp:lava-dispatcher.
> >
> > Requested reviews:
> > Linaro Validation Team (linaro-validation)
> >
> > For more details, see:
> > https://code.launchpad.net/~terceiro/lava-dispatcher/nexus/+merge/136807
> >
> > This is work in progress; you will note the number of FIXME's and TODO's in lava_dispatcher/device/nexus.py. ;-)
>
> This looks pretty good on the whole!

Thanks! :)

> > With the current state, I can already run a full boot test job, like the one in the commit message of rev 475:
> >
> > http://bazaar.launchpad.net/~terceiro/lava-dispatcher/nexus/revision/475
> >
> > It would be nice to receive comments about the changes that were needed to other files outside of the Nexus implementation itself.
>
> I think they mostly seem reasonable. I'd be happy to see them merged
> without the nexus bits if that would make your life easier.

I will merge them, in that case. A cleaner diff will make my life a
little easier, for sure.

> > === added file 'lava_dispatcher/device/nexus.py'
> > --- lava_dispatcher/device/nexus.py 1970-01-01 00:00:00 +0000
> > +++ lava_dispatcher/device/nexus.py 2012-11-28 22:45:26 +0000
> > @@ -0,0 +1,106 @@
> > +# Copyright (C) 2012 Linaro Limited
> > +#
> > +# Author: Antonio Terceiro <email address hidden>
> > +#
> > +# This file is part of LAVA Dispatcher.
> > +#
> > +# LAVA Dispatcher is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# LAVA Dispatcher is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along
> > +# with this program; if not, see <http://www.gnu.org/licenses>.
> > +
> > +import subprocess
> > +import pexpect
> > +from time import sleep
> > +from lava_dispatcher.device.target import (
> > + Target
> > +)
> > +from lava_dispatcher.downloader import (
> > + download_image
> > +)
> > +from lava_dispatcher.utils import (
> > + logging_spawn
> > +)
> > +
> > +class NexusTarget(Target):
> > +
> > + def __init__(self, context, config):
> > + super(NexusTarget, self).__init__(context, config)
> > +
> > + def deploy_android(self, boot, system, userdata):
> > + sdir = self.scratch_dir
> > +
> > + # TODO it seems boot, system and userdata are usually .tar.gz files,
> > + # and I am currently assuming they are .img and flashing them directly
>
> Would be good to talk to the android team about what to do here I guess.

OK, will do it.

> > + boot = download_image(boot, self.context, sdir, decompress=False)
> > + # FIXME uncomment these two - skipping t...

[lp:~terceiro/lava-dispatcher/nexus]

479. By Antonio Terceiro on 2012-11-30

merge from trunk

480. By Antonio Terceiro on 2012-12-03

merge recent trunk updates

481. By Antonio Terceiro on 2012-12-05

recent trunk changes

482. By Antonio Terceiro on 2012-12-05

no need to erase partitions just before they are reflashed

483. By Antonio Terceiro on 2012-12-05

only accepts .img files. that's all the android team need.

484. By Antonio Terceiro on 2012-12-06

trunk updates

485. By Antonio Terceiro on 2012-12-07

trunk updates

486. By Antonio Terceiro on 2012-12-10

Implement get_device_version

487. By Antonio Terceiro on 2012-12-11

implement file_system() API

With this I got lava-test-shell pretty much working. I still need to do more
tests, though.

488. By Antonio Terceiro on 2012-12-11

oops - previously commented code had a syntax error

489. By Antonio Terceiro on 2012-12-11

oops, another embarassing bug

490. By Antonio Terceiro on 2012-12-11

missing import statement

491. By Antonio Terceiro on 2012-12-11

fix example config

492. By Antonio Terceiro on 2012-12-12

sanitize terminfo hack

493. By Antonio Terceiro on 2012-12-12

Fixes trying to make lava-test-shell work on the Nexus

exec > /dev/console in special does not really help because we only have `adb
shell` and not access to an actual console.

494. By Antonio Terceiro on 2012-12-12

Need to mark make_filesystem_readwrite as a context manager

495. By Antonio Terceiro on 2012-12-12

AOSP does not have `tee` :-/

496. By Antonio Terceiro on 2012-12-12

turn the device on if it's not already.

This makes it possible to use lava_test_shell the way it is supposed to (i.e. not requiring to explicitly run boot_linaro*_image before).

497. By Antonio Terceiro on 2012-12-12

Chenges to make lava-test-runner run on the nexus

498. By Antonio Terceiro on 2012-12-13

oops, forgot to actually echo the line to stdout

499. By Antonio Terceiro on 2012-12-13

trunk updates

500. By Antonio Terceiro on 2012-12-13

Also remove `sleep 15` from lava-test-runner-ubuntu

501. By Antonio Terceiro on 2012-12-14

mark pending work

502. By Antonio Terceiro on 2012-12-16

use shell builtin type instead of which

which is not present in AOSP

503. By Antonio Terceiro on 2012-12-16

Install custom-made mkfifo binary

504. By Antonio Terceiro on 2012-12-17

Fix lava_test_shell bin directory location

505. By Antonio Terceiro on 2012-12-18

Log commands run on host when in debug mode

506. By Antonio Terceiro on 2012-12-18

Do not send output to /dev/console

507. By Antonio Terceiro on 2012-12-18

file_system already implemented

508. By Antonio Terceiro on 2012-12-18

Remove `-t N` from read calls.

BTW I've just found out that `read -t N < FIFO` does not work at all.

509. By Antonio Terceiro on 2012-12-18

Do not expect that `read -t` will ever work

510. By Antonio Terceiro on 2012-12-18

trunk updates

511. By Antonio Terceiro on 2012-12-19

trunk updates

512. By Antonio Terceiro on 2012-12-19

Run lava-test-runner in the background; drop mkfifo binary

513. By Antonio Terceiro on 2012-12-19

sync with trunk

514. By Antonio Terceiro on 2012-12-19

remove /etc/ cleanup hack and do not ignore unexisting directory when pulling (create it before pulling instead)

515. By Antonio Terceiro on 2012-12-19

more lava-test-runner stdin usage fixes

516. By Antonio Terceiro on 2012-12-19

grrr - fix reference to undefined variable

517. By Antonio Terceiro on 2012-12-19

fix argument handling in lava-test-*-attach

518. By Antonio Terceiro on 2012-12-19

revert earlier change that broke things.

519. By Antonio Terceiro on 2012-12-19

finish fixes for running lava-test-runner in the foreground

520. By Antonio Terceiro on 2012-12-20

Extract method

521. By Antonio Terceiro on 2012-12-20

Allow overriding the image directory.

This is useful when running the dispatcher inside the VM, but can also support
the use case where the dispatcher is on a machine and the phone is physically
plugged to another.

522. By Antonio Terceiro on 2012-12-20

sync from trunk

523. By Antonio Terceiro on 2012-12-20

using `lava dispatch --target DEVICE` here

524. By Antonio Terceiro on 2012-12-20

Get version number from configured adb_command

525. By Antonio Terceiro on 2012-12-20

Use the working dir (renamed from image dir) also for pulling data out of the device

526. By Antonio Terceiro on 2012-12-20

oops - missed changes to default config

527. By Antonio Terceiro on 2012-12-21

merge changes required to make lava-android-test work

528. By Antonio Terceiro on 2013-01-02

sync from trunk

529. By Antonio Terceiro on 2013-01-02

merge latest lava-android-test changes

530. By Antonio Terceiro on 2013-01-03

sync from trunk

531. By Antonio Terceiro on 2013-01-03

I don't believe that extract_tarball and get_test_data_attachment will be useful

532. By Antonio Terceiro on 2013-01-04

sync from trunk

533. By Antonio Terceiro on 2013-01-04

Replay fix done in the lava-test-shell-nexus-aosp branch

534. By Antonio Terceiro on 2013-01-04

sync from better command line handling branch

535. By Antonio Terceiro on 2013-01-14

merge wait-for-prompt branch for testing

536. By Antonio Terceiro on 2013-01-15

sync from trunk

537. By Antonio Terceiro on 2013-01-15

power_off is not viable

538. By Antonio Terceiro on 2013-01-15

Extract Methods

539. By Antonio Terceiro on 2013-01-15

Missed reference to reboot method

540. By Antonio Terceiro on 2013-01-15

No crazy stuff, I guess

All of the required fixes were merged into trunk before

541. By Antonio Terceiro on 2013-01-18

Separate all required API methods from private implementation methods

private methods marked with _ and moved to below the public API

542. By Antonio Terceiro on 2013-01-18

Require deploy before power_on()

543. By Antonio Terceiro on 2013-01-21

sync from trunk

544. By Antonio Terceiro on 2013-01-21

remove spaces around = in keyword argument

545. By Antonio Terceiro on 2013-01-21

minor changes to doc

546. By Antonio Terceiro on 2013-01-21

Clarify intent of some methods

547. By Antonio Terceiro on 2013-01-21

address more review comments

- removed sudo from config file. I assume the user running the dispatcher has
  the proper permissions
- removed _make_filesystem_readwrite - not needed anymore
- implemented fastboot detection
- added timeout to all calls. Default: 10 minutes

548. By Antonio Terceiro on 2013-01-21

Name revealling intention: s/_powered_on/_booted/

549. By Antonio Terceiro on 2013-01-22

Add some debug logging

550. By Antonio Terceiro on 2013-01-22

Better explanation for nexus_working_directory

Unmerged revisions

=== modified file 'lava_dispatcher/config.py'
--- lava_dispatcher/config.py	2013-01-10 21:37:58 +0000
+++ lava_dispatcher/config.py	2013-01-15 18:36:22 +0000
@@ -91,6 +91,9 @@
     arm_probe_config = schema.StringOption(default='/usr/local/etc/arm-probe-config')
     arm_probe_channels = schema.ListOption(default=['VDD_VCORE1'])
 
+    adb_command = schema.StringOption()
+    fastboot_command = schema.StringOption()
+    nexus_working_directory = schema.StringOption(default = None)
 
 class OptionDescriptor(object):
     def __init__(self, name):

=== added file 'lava_dispatcher/default-config/lava-dispatcher/device-types/nexus.conf'
--- lava_dispatcher/default-config/lava-dispatcher/device-types/nexus.conf	1970-01-01 00:00:00 +0000
+++ lava_dispatcher/default-config/lava-dispatcher/device-types/nexus.conf	2013-01-15 18:36:22 +0000
@@ -0,0 +1,40 @@
+client_type = nexus
+
+# The ADB command line.
+#
+# In the case where there are more than one android devices plugged into a
+# single host, this connection command must be overriden on each device to
+# include the serial number of the device, e.g.
+#
+#   serial_number = XXXXXXXXXXXXXXXX
+#   adb_command = adb -s %(serial_number)s
+adb_command = adb
+
+# The fastboot command.
+#
+# The same as above: if you have more than one device, you will want to
+# override this in your device config to add a serial number, e.g.
+#
+#   serial_number = XXXXXXXXXXXXXXXX
+#   fastboot_command = sudo fastboot -s %(serial_number)s
+#
+# Of course, in the case you override both adb_command *and* fastboot_command,
+# you don't need to specify `serial_number` twice.
+fastboot_command = sudo fastboot
+
+# Working directory for temporary. By default, the usual place for LAVA images
+# will be used.
+#
+# This is useful when the lava dispatcher is controlling Nexus phones that are
+# physically plugged to other machines. In this case, you should set
+# nexus_work_directory to a shared directory between the machine running the
+# dispatcher and the machine where the phone is plugged.
+#
+# OBS: this shared directory must have the same path in both machines.
+nexus_working_directory =
+
+connection_command = %(adb_command)s shell
+
+enable_network_after_boot_android = false
+android_adb_over_usb = true
+android_adb_over_tcp = false

=== added file 'lava_dispatcher/device/nexus.py'
--- lava_dispatcher/device/nexus.py	1970-01-01 00:00:00 +0000
+++ lava_dispatcher/device/nexus.py	2013-01-15 18:36:22 +0000
@@ -0,0 +1,168 @@
+# Copyright (C) 2012 Linaro Limited
+#
+# Author: Antonio Terceiro <[e-mail]>
+#
+# This file is part of LAVA Dispatcher.
+#
+# LAVA Dispatcher is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# LAVA Dispatcher is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.    See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along
+# with this program; if not, see <http://www.gnu.org/licenses>.
+
+import subprocess
+import pexpect
+from time import sleep
+import logging
+import contextlib
+
+from lava_dispatcher.device.target import (
+    Target
+)
+from lava_dispatcher.downloader import (
+    download_image
+)
+from lava_dispatcher.utils import (
+    logging_spawn,
+    mkdtemp
+)
+
+class NexusTarget(Target):
+
+    def __init__(self, context, config):
+        super(NexusTarget, self).__init__(context, config)
+        self._powered_on = False
+        self._working_dir = None
+
+    def deploy_android(self, boot, system, userdata):
+
+        boot = self._get_image(boot)
+        system = self._get_image(system)
+        userdata = self._get_image(userdata)
+
+        self.reboot_os()
+
+        self.fastboot('erase boot')
+
+        self.fastboot('flash system %s' % system)
+        self.fastboot('flash userdata %s' % userdata)
+
+        self.deployment_data = Target.android_deployment_data
+        self.deployment_data['boot_image'] = boot
+
+    def power_on(self):
+        self.reboot_os()
+        self.reboot_bootloader()
+        self.boot_test_image()
+
+        self._powered_on = True
+        proc = self.adb('shell', spawn = True)
+        proc.sendline("") # required to put the adb shell in a reasonable state
+        proc.sendline("export PS1='%s'" % self.deployment_data['TESTER_PS1'])
+        self._runner = self._get_runner(proc)
+
+        return proc
+
+    def reboot_os(self):
+        # tell android to reboot. A failure probably means that the device is not
+        # booted on android, and we ignore that.
+        self.adb('reboot', ignore_failure = True)
+        sleep(10)
+
+    def reboot_bootloader(self):
+        self.fastboot('reboot')
+        sleep(10)
+
+    def boot_test_image(self):
+        self.fastboot('boot %s' % self.deployment_data['boot_image'])
+        self.adb('wait-for-device')
+
+    def power_off(self, proc):
+        # there is no way to power off the Nexus while USB is plugged on; even
+        # if you remove power, it will stay on.
+        pass
+
+    @contextlib.contextmanager
+    def file_system(self, partition, directory):
+
+        if not self._powered_on:
+            self.power_on()
+
+        mount_point = self.get_partition_mount_point(partition)
+
+        with self.make_filesystem_readwrite(mount_point):
+            host_dir = '%s/mnt/%s' % (self.working_dir, directory)
+            target_dir = '%s/%s' % (mount_point, directory)
+
+            subprocess.check_call(['mkdir', '-p', host_dir])
+            self.adb('pull %s %s' % (target_dir, host_dir), ignore_failure = True)
+
+            yield host_dir
+
+            self.adb('push %s %s' % (host_dir, target_dir))
+
+    def get_partition_mount_point(self, partition):
+        lookup = {
+            self.config.data_part_android_org: '/data',
+            self.config.sys_part_android_org: '/system',
+        }
+        return lookup[partition]
+
+    @contextlib.contextmanager
+    def make_filesystem_readwrite(self, mount_point):
+        if mount_point  == '/system':
+            self._runner.run("mount -o remount,rw %s" % mount_point)
+        yield
+        if mount_point  == '/system':
+            self._runner.run("mount -o remount,ro %s" % mount_point)
+
+    def get_device_version(self):
+        # this is tricky, because fastboot does not have a visible version
+        # number. For now let's use just the adb version number.
+        return subprocess.check_output(
+            "%s version | sed 's/.* version //'" % self.config.adb_command,
+            shell = True
+        ).strip()
+
+    def adb(self, args, ignore_failure = False, spawn = False):
+        cmd = self.config.adb_command + ' ' + args
+        if spawn:
+            return logging_spawn(cmd, timeout = 60)
+        else:
+            self._call(cmd, ignore_failure)
+
+    def fastboot(self, args, ignore_failure = False):
+        self._call(self.config.fastboot_command + ' ' + args, ignore_failure)
+
+    def _call(self, cmd, ignore_failure):
+        logging.debug("Running on the host: %s" % cmd)
+        if ignore_failure:
+            subprocess.call(cmd, shell = True)
+        else:
+            subprocess.check_call(cmd, shell = True)
+
+    def _get_image(self, url):
+        sdir = self.working_dir
+        image = download_image(url, self.context, sdir, decompress=False)
+        return image
+
+    @property
+    def working_dir(self):
+        if (self.config.nexus_working_directory is None or
+            self.config.nexus_working_directory.strip() == ''):
+            return self.scratch_dir
+
+        if self._working_dir is None:
+            self._working_dir = mkdtemp(self.config.nexus_working_directory)
+        return self._working_dir
+
+
+target_class = NexusTarget