Merge lp:~qzhang/lava-dispatcher/test-abrek into lp:lava-dispatcher

[Spring Zhang]

Implement cmd_test_abrek a simple version.
1. r15: Add return value in run_shell_command, I think we need to check the result after we send the command via serial line sometimes.
2. Split cmd_test_abrek to two slices: install and run.
3. Add timeout parameter in cmd_test_abrek
3. For tester_str(="root@localhost:") and master_str, I think they are used universally in LAVA, so I suggest adding them to constant variables in a module, just like BOARD_TYPE.
4. Some concern about abrek test suite installation, how long can we wait for it? I gave 600s.

[Paul Larson (pwlars)]

We talked about this a bit on irc, and in previous email threads, but I wanted to capture it here too. Abrek should be installed from bzr, and done while booted into the master image. To accomplish this, we previously talked about having a small extension to the job format that would, as part of deployment, tell it to install "extra" things. one of the supported "extras" to install could be abrek, and one of the parameters it could take is the name of the test you would like to install under abrek. This way, after deploying the image, we can have it chroot into the newly-installed image, pull the latest abrek branch, install dependencies, and run abrek install [testname]

25 + rc = self.client.run_shell_command(
26 + 'abrek list-installed | grep %s' % test_name,
27 + response = test_name, timeout=10)
We don't need to check for this. We should only be installing tests once on a fresh image, so there shouldn't be any tests installed already.

58 - self.proc.expect(response, timeout=timeout)
59 + id = self.proc.expect(response, timeout=timeout)
60 + if id == 0:
61 + return True
62 + else:
63 + return False
64 + return True
I don't think this is going to do what you think. Either it get's the only response it knows about, or, it will timeout and raise an exception. If you try to use run_shell_command and gave it a response to look for, and didn't get an exception, then you know it worked.

[lp:~qzhang/lava-dispatcher/test-abrek]

17. By Spring Zhang on 2011-03-15

Modification refer to comments
1. abrek will install in master image
2. pexpect will issue timeout exception, no need to check return value

18. By Spring Zhang on 2011-03-15

merge from mainline r15

19. By Spring Zhang on 2011-03-16

Initialise action cmd_deploy_abrek for deploy abrek to test image by chroot

[Spring Zhang (qzhang)]

libc6 will install and update when installing python-distutils-extra, and when cron is running, it will pop out a dialog, need to stop cron service at first. See belowing:

-----------------------
Package configuration

 ���������������������������Ĵ Configuring libc6 ���������������������������Ŀ
 � Running services and programs that are using NSS need to be restarted, �
 � otherwise they might not be able to do lookup or authentication any more �
 � (for services such as ssh, this can affect your ability to login). �
 � Please review the following space-separated list of init.d scripts for �
 � services to be restarted now, and correct it if needed. �
 � �
 � Note: restarting sshd/telnetd should not affect any existing �
 � connections. �
 � �
 � Services to restart for GNU libc library upgrade: �
 � �
 � cron_____________________________________________________________________ �
 � �
 � <Ok> �
 � �
 ����������������������������������������������������������������������������

[lp:~qzhang/lava-dispatcher/test-abrek]

20. By Spring Zhang on 2011-03-17

finish command sequences of deploying abrek to test image

[Spring Zhang (qzhang)]

Finish the main command sequences of deploying abrek to test image in a new action cmd_deploy_abrek, and have a test, it can work basically. It's the first part of the whole test_abrek.

In my mind, test_abrek splits to three parts, first is cmd_deploy_abrek, the second is cmd_install_abrek for abrek installing test suites like ltp, the last is cmd_test_abrek to execute real test suite command.

Now I put the cmd_deploy_abrek in abrek.py, do you think I need to move it to deploy.py for it is deployment.

Improvements:
1. stop cron service to avoid libc dialog when installing python-distutils-extra
2. Do I need to verify every command to ensure it succeed? like install bzr
3. does it need to change to a temp path to install abrek? I checkout abrek to / now.
4. does it need to restore old resolv.conf in test image?

Some issues:
1. The commands send to serial get lost some part sometimes, so the command will fail.
2. Can we repeat the action if abrek can't install first time if something wrong.

[lp:~qzhang/lava-dispatcher/test-abrek]

21. By Spring Zhang on 2011-03-17

Move cmd_deploy_abrek to cmd_install_abrek
1. stop cron service to avoid dialog when installing python-distutils-extra
2. install test suite after abrek deployment

[Spring Zhang (qzhang)]

> Finish the main command sequences of deploying abrek to test image in a new
> action cmd_deploy_abrek, and have a test, it can work basically. It's the
> first part of the whole test_abrek.
>
> In my mind, test_abrek splits to three parts, first is cmd_deploy_abrek, the
> second is cmd_install_abrek for abrek installing test suites like ltp, the
> last is cmd_test_abrek to execute real test suite command.

Now rename cmd_deploy_abrek to cmd_install_abrek

>
> Now I put the cmd_deploy_abrek in abrek.py, do you think I need to move it to
> deploy.py for it is deployment.
>
> Improvements:
> 1. stop cron service to avoid libc dialog when installing python-distutils-
> extra
Done.

[lp:~qzhang/lava-dispatcher/test-abrek]

22. By Spring Zhang on 2011-03-17

clean code in cmd_test_abrek

[Spring Zhang (qzhang)]

A simple implementation of cmd_test_abrek, please comment if something miss.

review: Needs Resubmitting

[Paul Larson (pwlars)]

Looking good, just a few comments. Also, did you test this? Does it seem to work ok for you?

7 +from lava.client import OperationFailed
Imported but never used

12 + tester_str = "root@localhost:"
As you mentioned previously, we should probably define this elsewhere and just import it for consistency. OK for now, but let's do a patch later to clean this up everywhere, rather than doing it piecemeal.

13 print "abrek run %s" % test_name
Was this for debug, or information to the user? If it's for information about what's going on, I'd recommend being a bit more descriptive, rather than just outputting the command.

21 + self.client.run_shell_command(
22 + 'abrek run %s -o /lava/results/%s' % (test_name, test_name),
Let's give it a .json, or .bundle extension for the testoutput, or something like that. Just in case we ever decide to put anything else here, it would be nice to have an easy way to distinguish between files.

25 +class cmd_install_abrek(BaseAction):
26 + """
27 + abrek test tool deployment to test image rootfs by chroot
28 + may be placed in deploy.py, it can move later
I think abrek.py is a perfectly sensible place to put it.

30 + def run(self, tests):
31 + #Make sure in master image
32 + #, or exception can be caught and do boot_master_image()
This sounds like you were intending to catch the exception from in_master_shell() and correct for it if we are not in the master image, but you don't seem to do that.

42 + self.client.run_shell_command(
43 + 'cp -L /etc/resolv.conf /mnt/root/etc',
Please backup the original and restore it before unmounting.

45 + self.client.run_shell_command(
46 + 'cp -L /etc/apt/apt.conf.d/70debconf /mnt/root/etc/apt/apt.conf.d',
This shouldn't be needed, isn't that already in the image? It's there on the root.tgz I'm looking at.

48 + self.client.run_shell_command(
49 + 'chroot /mnt/root mount -t proc proc /proc',
is this needed? Possibly, just curious if you are doing it because you saw an error without it, or if you are just being extra careful.

56 + #ensure no libc6 config dialog popout when apt-get
57 + self.client.run_shell_command(
58 + 'chroot /mnt/root stop cron',
I can't see how this would be needed, or how it would help. Cron shouldn't be running at all, the image isn't even running! Also, we should be able to get around things like this another way, such as telling apt to be extra quiet. And the libc6 thing is just temporary, I don't expect we'll run into that frequently. This probably has more to do with the image you were testing, and the updates that are available right now.

64 + 'chroot /mnt/root apt-get -y install bzr',
Don't do this in separate steps, go ahead and install all the dependencies we need

79 + #Test if abrek installed
80 + self.client.run_shell_command(
81 + 'chroot /mnt/root abrek help',
82 + response = "list-tests")
Good idea to check this, but you may want to give it a better error, and shorten the timeout. We don't want to wait the default timeout, because this should happen easily within something shorter, say 10 seconds to be generous. Catch the exception if you get one, do any cleanup needed, and then raise a new exception with an error saying that ab...

[Paul Larson (pwlars)]

review: Needs Fixing

[Spring Zhang (qzhang)]

Thanks for the comments.

On Thu, 2011-03-17 at 20:51 +0000, Paul Larson wrote:
> Looking good, just a few comments. Also, did you test this? Does it seem to work ok for you?

Yes, I tested the sequence on mx51evk, it's ok except the char lost
issue I mentioned via USB-to-serial. Also I find the cron service issue
in the process.

>
> 7 +from lava.client import OperationFailed
> Imported but never used
I'll use it in abrek detection.

>
>
> 12 + tester_str = "root@localhost:"
> As you mentioned previously, we should probably define this elsewhere and just import it for consistency. OK for now, but let's do a patch later to clean this up everywhere, rather than doing it piecemeal.
Ok. Here I have a question, if I send a mp to mainline and it is
accepted, can I go on commit on the branch and send mp again?

>
> 13 print "abrek run %s" % test_name
> Was this for debug, or information to the user? If it's for information about what's going on, I'd recommend being a bit more descriptive, rather than just outputting the command.
Delete it now.

>
> 21 + self.client.run_shell_command(
> 22 + 'abrek run %s -o /lava/results/%s' % (test_name, test_name),
> Let's give it a .json, or .bundle extension for the testoutput, or something like that. Just in case we ever decide to put anything else here, it would be nice to have an easy way to distinguish between files.
hmm, I'll look into it.

>
> 25 +class cmd_install_abrek(BaseAction):
> 26 + """
> 27 + abrek test tool deployment to test image rootfs by chroot
> 28 + may be placed in deploy.py, it can move later
> I think abrek.py is a perfectly sensible place to put it.
>
> 30 + def run(self, tests):
> 31 + #Make sure in master image
> 32 + #, or exception can be caught and do boot_master_image()
> This sounds like you were intending to catch the exception from in_master_shell() and correct for it if we are not in the master image, but you don't seem to do that.
Change soon.

>
> 42 + self.client.run_shell_command(
> 43 + 'cp -L /etc/resolv.conf /mnt/root/etc',
> Please backup the original and restore it before unmounting.
Done.

>
> 45 + self.client.run_shell_command(
> 46 + 'cp -L /etc/apt/apt.conf.d/70debconf /mnt/root/etc/apt/apt.conf.d',
> This shouldn't be needed, isn't that already in the image? It's there on the root.tgz I'm looking at.
Yes, it makes sure apt-get working in the same network environment like
master image besides /etc/resolv.conf, the configuration may contain the
proxy information, however, we use direct Internet connection or bzr
can't work either, so I will remove it here.

>
> 48 + self.client.run_shell_command(
> 49 + 'chroot /mnt/root mount -t proc proc /proc',
> is this needed? Possibly, just curious if you are doing it because you saw an error without it, or if you are just being extra careful.
No error message, I can remove it.

>
> 56 + #ensure no libc6 config dialog popout when apt-get
> 57 + self.client.run_shell_command(
> 58 + 'chroot /mnt/root stop cron',
> I can't see how this would be needed, or how it would help. Cron shouldn't be running at all, the image isn't even running! Also, we should be able to get around things like this another way, such ...

[lp:~qzhang/lava-dispatcher/test-abrek]

23. By Spring Zhang on 2011-03-18

1. Backup test image config and restore
2. Do a clean up on test image after abrek deployment
3. Enhance based on comments

[Spring Zhang (qzhang)]

Finish on most comments.

But I can't umount /mnt/root/dev after I use "mount --rbind /dev /mnt/root/dev", however, the mount occurs in master image, so I don't think it does matter, or who know how to umount rebind device?

root@master:~# umount -f /mnt/root/dev
umount2: Device or resource busy
umount: /mnt/root/dev: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

review: Needs Resubmitting

[lp:~qzhang/lava-dispatcher/test-abrek]

24. By Spring Zhang on 2011-03-18

Use right /dev mount method

[Paul Larson (pwlars)]

> > 12 + tester_str = "root@localhost:"
> > As you mentioned previously, we should probably define this elsewhere and
> just import it for consistency. OK for now, but let's do a patch later to
> clean this up everywhere, rather than doing it piecemeal.
> Ok. Here I have a question, if I send a mp to mainline and it is
> accepted, can I go on commit on the branch and send mp again?
Right, just do a new merge proposal for that. It should be a pretty quick one.

> I use an alpha version for master image to test, actually, cron is
> running in master image, and no service run from test image, but after
> chroot, it is still running and test image can see it, so I met the
> error. If cron won't run in our master image finally, I'll remove it
> now.
The reason you see cron running is because you mounted proc. :) That should go away when you remove that extra mount.

> > 87 + self.client.run_shell_command(
> > 88 + 'chroot /mnt/root umount /proc',
> > 89 + response = master_str)
> > There's more cleanup you need to do here. You have a bind mount for /dev,
> and would probably be good to remove /abrek since it was just needed for the
> install, replace resolv.conf original file,
> For /dev bind mount, it doesn't help to eliminate the warning message
> "Can not write log, openpty() failed (/dev/pts not mounted?), does not
> work" after I tried. Do you have some other way? Or we can ignore it for
> all seems working.
The reason is because you used -rbind. You have additional mountpoints, probably for shm and pts, under your remounted /dev. If the error doesn't cause any problem, I think it's safe to ignore. Otherwise, you'll need to either manually mount /dev/pts, and just unmount that when done. Or, you can do something like:
cat /proc/mounts | awk '{print $2}' | grep "^/mnt/root/dev" | sort -r | xargs umount

[Spring Zhang (qzhang)]

> > > 87 + self.client.run_shell_command(
> > > 88 + 'chroot /mnt/root umount /proc',
> > > 89 + response = master_str)
> > > There's more cleanup you need to do here. You have a bind mount for /dev,
> > and would probably be good to remove /abrek since it was just needed for the
> > install, replace resolv.conf original file,
> > For /dev bind mount, it doesn't help to eliminate the warning message
> > "Can not write log, openpty() failed (/dev/pts not mounted?), does not
> > work" after I tried. Do you have some other way? Or we can ignore it for
> > all seems working.
> The reason is because you used -rbind. You have additional mountpoints,
> probably for shm and pts, under your remounted /dev. If the error doesn't
> cause any problem, I think it's safe to ignore. Otherwise, you'll need to
> either manually mount /dev/pts, and just unmount that when done. Or, you can
> do something like:
> cat /proc/mounts | awk '{print $2}' | grep "^/mnt/root/dev" | sort -r | xargs
> umount

I have fixed "Can not write log, openpty() failed (/dev/pts not mounted?), does not..." warning message by r24, using "mount --rbind /dev /mnt/root/dev", not "chroot /mnt/root mount --rbind /dev /mnt/root/dev".

For the mount occurs in master image, I think we can ignore the umount for /mnt/root/dev.

All the comments are finished.

[Paul Larson (pwlars)]

> I have fixed "Can not write log, openpty() failed (/dev/pts not mounted?),
> does not..." warning message by r24, using "mount --rbind /dev /mnt/root/dev",
> not "chroot /mnt/root mount --rbind /dev /mnt/root/dev".

But the umount /mnt/root at the end should fail for the same reason. You still have things mounted under /mnt/root, so it will show up as busy. See my previous comment for ideas on how to fix this.

[lp:~qzhang/lava-dispatcher/test-abrek]

25. By Spring Zhang on 2011-03-22

fix umount /mnt/root/dev error

[Spring Zhang (qzhang)]

Thanks Paul, I have fixed the error by r25.
The "mount" output is different with "cat /proc/mounts", which makes me stop on the command.

[Paul Larson (pwlars)]

Ok, I'll go ahead and merge this so we can push forward to the next thing - dashboard submission. Will try to get some testing of what we have so far in this week.

review: Approve

=== modified file 'lava/actions/abrek.py'
--- lava/actions/abrek.py	2011-02-24 19:18:57 +0000
+++ lava/actions/abrek.py	2011-03-22 03:31:29 +0000
@@ -1,6 +1,87 @@
 #!/usr/bin/python
 from lava.actions import BaseAction
+from lava.client import OperationFailed
 
 class cmd_test_abrek(BaseAction):
-    def run(self, test_name):
-        print "abrek run %s" % test_name
+    def run(self, test_name, timeout=-1):
+        tester_str = "root@localhost:"
+        #Make sure in test image now
+        self.in_test_shell()
+
+        self.client.run_shell_command('mkdir -p /lava/results',
+            response = tester_str)
+
+        self.client.run_shell_command(
+            'abrek run %s -o /lava/results/%s.bundle' % (test_name, test_name),
+            response = tester_str, timeout = timeout)
+
+class cmd_install_abrek(BaseAction):
+    """
+    abrek test tool deployment to test image rootfs by chroot
+    """
+    def run(self, tests):
+        #Make sure in master image
+        #, or exception can be caught and do boot_master_image()
+        master_str = "root@master:"
+        try:
+            self.client.in_master_shell()
+        except:
+            self.client.boot_master_image()
+
+        #install bazaar in tester image
+        self.client.run_shell_command(
+            'mkdir -p /mnt/root',
+            response = master_str)
+        self.client.run_shell_command(
+            'mount /dev/disk/by-label/testrootfs /mnt/root',
+            response = master_str)
+        self.client.run_shell_command(
+            'cp -f /mnt/root/etc/resolv.conf /mnt/root/etc/resolv.conf.bak',
+            response = master_str)
+        self.client.run_shell_command(
+            'cp -L /etc/resolv.conf /mnt/root/etc',
+            response = master_str)
+        #eliminate warning: Can not write log, openpty() failed 
+        #                   (/dev/pts not mounted?), does not work
+        self.client.run_shell_command(
+            'mount --rbind /dev /mnt/root/dev',
+            response = master_str)
+        self.client.run_shell_command(
+            'chroot /mnt/root apt-get update',
+            response = master_str)
+        #Install necessary packages for build abrek
+        self.client.run_shell_command(
+            'chroot /mnt/root apt-get -y install bzr python-apt python-distutils-extra',
+            response = master_str)
+        self.client.run_shell_command(
+            'chroot /mnt/root bzr branch lp:abrek',
+            response = master_str)
+        self.client.run_shell_command(
+            'chroot /mnt/root sh -c "cd abrek && python setup.py install"',
+            response = master_str)
+
+        #Test if abrek installed
+        try:
+            self.client.run_shell_command(
+                'chroot /mnt/root abrek help',
+                response = "list-tests", timeout = 10)
+        except:
+            raise OperationFailed("abrek deployment failed")
+
+        for test in tests:
+            self.client.run_shell_command(
+                'chroot /mnt/root abrek install %s' % test,
+                response = master_str)
+        #clean up
+        self.client.run_shell_command(
+            'cp -f /mnt/root/etc/resolv.conf.bak /mnt/root/etc/resolv.conf',
+            response = master_str)
+        self.client.run_shell_command(
+            'rm -rf /mnt/root/abrek',
+            response = master_str)
+        self.client.run_shell_command(
+            'cat /proc/mounts | awk \'{print $2}\' | grep "^/mnt/root/dev" | sort -r | xargs umount',
+            response = master_str)
+        self.client.run_shell_command(
+            'umount /mnt/root',
+            response = master_str)