Merge lp:~mabac/linaro-image-tools/scripts-hooks into lp:linaro-image-tools/11.11

[Mattias Backman]

Hi,

This branch adds two command line options to linaro-media-create to enable users to customize the behaviour of l-m-c.

  --preinstall-script PREINSTALL_SCRIPT, --preinstall_script PREINSTALL_SCRIPT
                        Script to execute in the chroot before installing
                        hwpacks.
  --postinstall-script POSTINSTALL_SCRIPT, --postinstall_script POSTINSTALL_SCRIPT
                        Script to execute in the chroot after installing
                        hwpacks.

The scripts will be executed in the chroot before and after installing hwpacks.

Thanks,

Mattias

[lp:~mabac/linaro-image-tools/scripts-hooks]

487. By Mattias Backman on 2012-01-24

Update call signature in tests.

[Ricardo Salveti (rsalveti)]

Tested with 2 simple scripts and it worked well.

A few comments:
1. At the preinstall script there's no way to access the hwpack content, because it wasn't yet extracted. This may be useful for people that want to read from hwpack packages or even customize in some way.
2. Would be useful if we could also copy both scripts at the rootfs, as it might help debugging any issues in the future.
3. Would also be useful if we could stamp that the image was customized by a script, like by adding a message via /etc/update-motd.d, so it'd show that info to the user.

[Fathi Boudra (fboudra)]

Hi,

Do we need the extra command line options? I'm thinking about a hook dir where you drop your hooks, in the same way it's done on live-build [1] or pbuilder[2]. A prefix will determine at which stage the scripts will be used. Until the proper solution is implemented, you can hardcode the scripts name.

[1] /usr/share/live/build/examples/hooks
[2] /usr/share/doc/pbuilder/examples

> 2. Would be useful if we could also copy both scripts at the rootfs
> 3. Would also be useful if we could stamp that the image was customized by a script

for both, I'll leave these outside of l-m-c. It's up to the developer and could be done through the hooks.

[Mattias Backman (mabac)]

> Tested with 2 simple scripts and it worked well.
>
> A few comments:
> 1. At the preinstall script there's no way to access the hwpack content,
> because it wasn't yet extracted. This may be useful for people that want to
> read from hwpack packages or even customize in some way.

Would you like the pre-hooks to be run in linaro-hwpack-install instead? That way the script can be run for every hwpack after unpacking it, meaning the pre-hook is run once per hwpack. If you mean we should unpack all the hwpacks and then run the pre-hook once it will require some more work but it is possible of course.

> 2. Would be useful if we could also copy both scripts at the rootfs, as it
> might help debugging any issues in the future.
> 3. Would also be useful if we could stamp that the image was customized by a
> script, like by adding a message via /etc/update-motd.d, so it'd show that
> info to the user.

[Mattias Backman (mabac)]

> Hi,
>
> Do we need the extra command line options? I'm thinking about a hook dir where

It's all down to how you'd like this to behave, we can do that instead. I think the discussions on irc kept mentioning command line options which is why I did that. Any other opinions about cli options vs a hook dir?

> you drop your hooks, in the same way it's done on live-build [1] or
> pbuilder[2]. A prefix will determine at which stage the scripts will be used.
> Until the proper solution is implemented, you can hardcode the scripts name.
>
> [1] /usr/share/live/build/examples/hooks
> [2] /usr/share/doc/pbuilder/examples
>
> > 2. Would be useful if we could also copy both scripts at the rootfs
> > 3. Would also be useful if we could stamp that the image was customized by a
> script
>
> for both, I'll leave these outside of l-m-c. It's up to the developer and
> could be done through the hooks.

I think it can be settled by sorting out if this always will be needed or if there may be cases where the hook author would want no trace of the hooks on the target system. If we feel that we need this for helping people with messed up images we need to do it in l-m-c. If it's up to the developer, I'm happy to leave this bit out.

[Ricardo Salveti (rsalveti)]

On Wed, Jan 25, 2012 at 6:35 AM, Mattias Backman
<email address hidden> wrote:
>> Tested with 2 simple scripts and it worked well.
>>
>> A few comments:
>> 1. At the preinstall script there's no way to access the hwpack content,
>> because it wasn't yet extracted. This may be useful for people that want to
>> read from hwpack packages or even customize in some way.
>
> Would you like the pre-hooks to be run in linaro-hwpack-install instead? That way the script can be run for every hwpack after unpacking it, meaning the pre-hook is run once per hwpack. If you mean we should unpack all the hwpacks and then run the pre-hook once it will require some more work but it is possible of course.

Can you flash the image using more than one hwpack? I thought it'd use
just one hwpack :-)

Guess to simplify things having the pre-hook the way you implemented
should be fine for now.

[Ricardo Salveti (rsalveti)]

On Wed, Jan 25, 2012 at 6:42 AM, Mattias Backman
<email address hidden> wrote:
>> Hi,
>>
>> Do we need the extra command line options? I'm thinking about a hook dir where
>
> It's all down to how you'd like this to behave, we can do that instead. I think the discussions on irc kept mentioning command line options which is why I did that. Any other opinions about cli options vs a hook dir?

While I like the pbuilder/livebuild way of hooking the script, I think
having them as a parameter (for pre/post) is more than enough, besides
easier to implement. If the user really want to customize the image
this much, I'd prefer him to go and create his own image by running
live-build by hand.

>> > 2. Would be useful if we could also copy both scripts at the rootfs
>> > 3. Would also be useful if we could stamp that the image was customized by a
>> script
>>
>> for both, I'll leave these outside of l-m-c. It's up to the developer and
>> could be done through the hooks.
>
> I think it can be settled by sorting out if this always will be needed or if there may be cases where the hook author would want no trace of the hooks on the target system. If we feel that we need this for helping people with messed up images we need to do it in l-m-c. If it's up to the developer, I'm happy to leave this bit out.

Do we have the logs from the l-m-c available at the image already?
Guess that this would be enough already.

[Fathi Boudra (fboudra)]

> On Wed, Jan 25, 2012 at 6:42 AM, Mattias Backman
> <email address hidden> wrote:
> >> Hi,
> >>
> >> Do we need the extra command line options? I'm thinking about a hook dir
> where
> >
> > It's all down to how you'd like this to behave, we can do that instead. I
> think the discussions on irc kept mentioning command line options which is why
> I did that. Any other opinions about cli options vs a hook dir?
>
> While I like the pbuilder/livebuild way of hooking the script, I think
> having them as a parameter (for pre/post) is more than enough, besides
> easier to implement. If the user really want to customize the image
> this much, I'd prefer him to go and create his own image by running
> live-build by hand.

I see at least one argument to prefer linaro-media-create instead of live-build for the job: it will work on other distributions. afaik, live-build isn't available on something else than Debian/Ubuntu, more over we carry a modified version available in our PPA.

> >> > 2. Would be useful if we could also copy both scripts at the rootfs
> >> > 3. Would also be useful if we could stamp that the image was customized
> by a
> >> script
> >>
> >> for both, I'll leave these outside of l-m-c. It's up to the developer and
> >> could be done through the hooks.
> >
> > I think it can be settled by sorting out if this always will be needed or if
> there may be cases where the hook author would want no trace of the hooks on
> the target system. If we feel that we need this for helping people with messed
> up images we need to do it in l-m-c. If it's up to the developer, I'm happy to
> leave this bit out.

As a hook author, I don't want trace of the hooks as I want to keep it small :)

> Do we have the logs from the l-m-c available at the image already?
> Guess that this would be enough already.

Do you mean a build log shipped inside the image? We don't have that.
We can add traces on l-m-c build log (though, it isn't shipped in the generated rootfs).

[Ricardo Salveti (rsalveti)]

On Wed, Jan 25, 2012 at 4:08 PM, Fathi Boudra <email address hidden> wrote:
>> On Wed, Jan 25, 2012 at 6:42 AM, Mattias Backman
>> <email address hidden> wrote:
>> >> Hi,
>> >>
>> >> Do we need the extra command line options? I'm thinking about a hook dir
>> where
>> >
>> > It's all down to how you'd like this to behave, we can do that instead. I
>> think the discussions on irc kept mentioning command line options which is why
>> I did that. Any other opinions about cli options vs a hook dir?
>>
>> While I like the pbuilder/livebuild way of hooking the script, I think
>> having them as a parameter (for pre/post) is more than enough, besides
>> easier to implement. If the user really want to customize the image
>> this much, I'd prefer him to go and create his own image by running
>> live-build by hand.
>
> I see at least one argument to prefer linaro-media-create instead of live-build for the job: it will work on other distributions. afaik, live-build isn't available on something else than Debian/Ubuntu, more over we carry a modified version available in our PPA.

Yeah, having linaro-media-create working on any other distro is
already a pain, requesting further customization with live-build is
even worse, +1 for fathi's solution then.

>> >> > 2. Would be useful if we could also copy both scripts at the rootfs
>> >> > 3. Would also be useful if we could stamp that the image was customized
>> by a
>> >> script
>> >>
>> >> for both, I'll leave these outside of l-m-c. It's up to the developer and
>> >> could be done through the hooks.
>> >
>> > I think it can be settled by sorting out if this always will be needed or if
>> there may be cases where the hook author would want no trace of the hooks on
>> the target system. If we feel that we need this for helping people with messed
>> up images we need to do it in l-m-c. If it's up to the developer, I'm happy to
>> leave this bit out.
>
> As a hook author, I don't want trace of the hooks as I want to keep it small :)

Fair enough.

>> Do we have the logs from the l-m-c available at the image already?
>> Guess that this would be enough already.
>
> Do you mean a build log shipped inside the image? We don't have that.
> We can add traces on l-m-c build log (though, it isn't shipped in the generated rootfs).

Yeah, I believe having the build logs is useful, and could be
available at /var/something. But guess this can be tracked by another
bug/merge request.

[Mattias Backman (mabac)]

On Wed, Jan 25, 2012 at 6:31 PM, Ricardo Salveti <email address hidden> wrote:
> On Wed, Jan 25, 2012 at 6:35 AM, Mattias Backman
> <email address hidden> wrote:
>>> Tested with 2 simple scripts and it worked well.
>>>
>>> A few comments:
>>> 1. At the preinstall script there's no way to access the hwpack content,
>>> because it wasn't yet extracted. This may be useful for people that want to
>>> read from hwpack packages or even customize in some way.
>>
>> Would you like the pre-hooks to be run in linaro-hwpack-install instead? That way the script can be run for every hwpack after unpacking it, meaning the pre-hook is run once per hwpack. If you mean we should unpack all the hwpacks and then run the pre-hook once it will require some more work but it is possible of course.
>
> Can you flash the image using more than one hwpack? I thought it'd use
> just one hwpack :-)

You can give as many hwpacks to l-m-c as you like, but most hwpacks
won't make any sense to combine of course. It's intended for having
base and addon hwpacks I think, but I'm not sure it's the most used
feature of l-m-c.

>
> Guess to simplify things having the pre-hook the way you implemented
> should be fine for now.
>
> --
> https://code.launchpad.net/~mabac/linaro-image-tools/scripts-hooks/+merge/89889
> You are the owner of lp:~mabac/linaro-image-tools/scripts-hooks.

[Mattias Backman (mabac)]

On Thu, Jan 26, 2012 at 3:40 AM, Ricardo Salveti <email address hidden> wrote:
> On Wed, Jan 25, 2012 at 4:08 PM, Fathi Boudra <email address hidden> wrote:
>>> On Wed, Jan 25, 2012 at 6:42 AM, Mattias Backman
>>> <email address hidden> wrote:
>>> >> Hi,
>>> >>
>>> >> Do we need the extra command line options? I'm thinking about a hook dir
>>> where
>>> >
>>> > It's all down to how you'd like this to behave, we can do that instead. I
>>> think the discussions on irc kept mentioning command line options which is why
>>> I did that. Any other opinions about cli options vs a hook dir?
>>>
>>> While I like the pbuilder/livebuild way of hooking the script, I think
>>> having them as a parameter (for pre/post) is more than enough, besides
>>> easier to implement. If the user really want to customize the image
>>> this much, I'd prefer him to go and create his own image by running
>>> live-build by hand.
>>
>> I see at least one argument to prefer linaro-media-create instead of live-build for the job: it will work on other distributions. afaik, live-build isn't available on something else than Debian/Ubuntu, more over we carry a modified version available in our PPA.
>
> Yeah, having linaro-media-create working on any other distro is
> already a pain, requesting further customization with live-build is
> even worse, +1 for fathi's solution then.

So we change this to the hooks drop-in folder then.

>
>>> >> > 2. Would be useful if we could also copy both scripts at the rootfs
>>> >> > 3. Would also be useful if we could stamp that the image was customized
>>> by a
>>> >> script
>>> >>
>>> >> for both, I'll leave these outside of l-m-c. It's up to the developer and
>>> >> could be done through the hooks.
>>> >
>>> > I think it can be settled by sorting out if this always will be needed or if
>>> there may be cases where the hook author would want no trace of the hooks on
>>> the target system. If we feel that we need this for helping people with messed
>>> up images we need to do it in l-m-c. If it's up to the developer, I'm happy to
>>> leave this bit out.
>>
>> As a hook author, I don't want trace of the hooks as I want to keep it small :)
>
> Fair enough.
>
>>> Do we have the logs from the l-m-c available at the image already?
>>> Guess that this would be enough already.
>>
>> Do you mean a build log shipped inside the image? We don't have that.
>> We can add traces on l-m-c build log (though, it isn't shipped in the generated rootfs).
>
> Yeah, I believe having the build logs is useful, and could be
> available at /var/something. But guess this can be tracked by another
> bug/merge request.

We brough this up as part of the image metadata discussion. It is a
separate feature but we can probably sneak it in while doing some
other work.

>
> --
> https://code.launchpad.net/~mabac/linaro-image-tools/scripts-hooks/+merge/89889
> You are the owner of lp:~mabac/linaro-image-tools/scripts-hooks.

[Данило Шеган (danilo)]

Follow-up MP using directory paths as arguments to l-m-c: https://code.launchpad.net/~danilo/linaro-image-tools/bug-751531-script-hooks/+merge/98679

Unmerged revisions

487. By Mattias Backman on 2012-01-24

Update call signature in tests.

486. By Mattias Backman on 2012-01-24

Add pre and post hwpack install script hooks.

=== modified file 'linaro-media-create'
--- linaro-media-create	2012-01-17 14:50:47 +0000
+++ linaro-media-create	2012-01-24 14:06:24 +0000
@@ -160,7 +160,8 @@
     if lmc_dir == '':
         lmc_dir = None
     install_hwpacks(ROOTFS_DIR, TMP_DIR, lmc_dir, args.hwpack_force_yes,
-                    verified_files, *hwpacks)
+                    verified_files, args.preinstall_script, args.postinstall_script,
+                    *hwpacks)
 
     if args.rootfs == 'btrfs':
         install_packages(ROOTFS_DIR, TMP_DIR, "btrfs-tools")

=== modified file 'linaro_image_tools/media_create/__init__.py'
--- linaro_image_tools/media_create/__init__.py	2012-01-19 06:31:50 +0000
+++ linaro_image_tools/media_create/__init__.py	2012-01-24 14:06:24 +0000
@@ -152,6 +152,12 @@
         '--nocheck-mmc', dest='nocheck_mmc',
         action='store_true',
         help='Assume yes to the question "Are you 100%% sure, on selecting [mmc]"')
+    parser.add_argument(
+        '--preinstall-script', '--preinstall_script',
+        help='Script to execute in the chroot before installing hwpacks.')
+    parser.add_argument(
+        '--postinstall-script', '--postinstall_script',
+        help='Script to execute in the chroot after installing hwpacks.')
     
     add_common_options(parser)
     return parser

=== modified file 'linaro_image_tools/media_create/chroot_utils.py'
--- linaro_image_tools/media_create/chroot_utils.py	2011-11-16 21:09:18 +0000
+++ linaro_image_tools/media_create/chroot_utils.py	2012-01-24 14:06:24 +0000
@@ -19,6 +19,7 @@
 
 import os
 import sys
+import logging
 
 from linaro_image_tools import cmd_runner
 from linaro_image_tools.utils import (
@@ -42,8 +43,38 @@
         copy_file('/usr/bin/qemu-arm-static',
                   os.path.join(chroot_dir, 'usr', 'bin'))
 
+
+def run_script_in_chroot(chroot_dir, script):
+    logger = logging.getLogger("linaro_image_tools")
+    logger.info(" -- Executing user script %s in %s. --" % (script, chroot_dir))
+
+    script_in_chroot = os.path.join('usr', 'bin', 'lmc-user-script')
+    copy_file(script, os.path.join(chroot_dir, script_in_chroot))
+
+    cmd_runner.run([script_in_chroot], as_root=True, chroot=chroot_dir).wait()
+    logger.info(" -- User script %s finished. --" % script)
+
+
+def try_qemu_static(chroot_dir):
+    try:
+        # Sometimes the host will have qemu-user-static installed but
+        # another package (i.e. scratchbox) will have mangled its config
+        # and thus we won't be able to chroot and install the hwpack, so
+        # we fail here and tell the user to ensure qemu-arm-static is
+        # setup before trying again.
+        cmd_runner.run(['true'], as_root=True, chroot=chroot_dir).wait()
+    except:
+        print ("Cannot proceed with hwpack installation because "
+               "there doesn't seem to be a binfmt interpreter registered "
+               "to execute armel binaries in the chroot. Please check "
+               "that qemu-user-static is installed and properly "
+               "configured before trying again.")
+        raise
+
+
 def install_hwpacks(
-    chroot_dir, tmp_dir, tools_dir, hwpack_force_yes, verified_files, *hwpack_files):
+    chroot_dir, tmp_dir, tools_dir, hwpack_force_yes, verified_files, 
+    preinstall_script, postinstall_script, *hwpack_files):
     """Install the given hwpacks onto the given chroot."""
     prepare_chroot(chroot_dir, tmp_dir)
 
@@ -57,26 +88,16 @@
 
     try:
         mount_chroot_proc(chroot_dir)
-        try:
-            # Sometimes the host will have qemu-user-static installed but
-            # another package (i.e. scratchbox) will have mangled its config
-            # and thus we won't be able to chroot and install the hwpack, so
-            # we fail here and tell the user to ensure qemu-arm-static is
-            # setup before trying again.
-            cmd_runner.run(['true'], as_root=True, chroot=chroot_dir).wait()
-        except:
-            print ("Cannot proceed with hwpack installation because "
-                   "there doesn't seem to be a binfmt interpreter registered "
-                   "to execute armel binaries in the chroot. Please check "
-                   "that qemu-user-static is installed and properly "
-                   "configured before trying again.")
-            raise
-
+        try_qemu_static(chroot_dir)
+        if preinstall_script:
+            run_script_in_chroot(chroot_dir, preinstall_script)
         for hwpack_file in hwpack_files:
             hwpack_verified = False
             if os.path.basename(hwpack_file) in verified_files:
                 hwpack_verified = True
             install_hwpack(chroot_dir, hwpack_file, hwpack_force_yes or hwpack_verified)
+        if postinstall_script:
+            run_script_in_chroot(chroot_dir, postinstall_script)
     finally:
         run_local_atexit_funcs()
 

=== modified file 'linaro_image_tools/media_create/tests/test_media_create.py'
--- linaro_image_tools/media_create/tests/test_media_create.py	2012-01-17 14:50:47 +0000
+++ linaro_image_tools/media_create/tests/test_media_create.py	2012-01-24 14:06:24 +0000
@@ -2956,8 +2956,8 @@
         prefer_dir = preferred_tools_dir()
 
         install_hwpacks(
-            chroot_dir, tmp_dir, prefer_dir, force_yes, [], 'hwpack1.tgz',
-            'hwpack2.tgz')
+            chroot_dir, tmp_dir, prefer_dir, force_yes, [], None, None,
+            'hwpack1.tgz', 'hwpack2.tgz')
         linaro_hwpack_install = find_command(
             'linaro-hwpack-install', prefer_dir=prefer_dir)
         expected = [
@@ -3072,7 +3072,7 @@
         try:
             install_hwpacks(
                 'chroot', '/tmp/dir', preferred_tools_dir(), force_yes, [],
-                'hwp.tgz', 'hwp2.tgz')
+                None, None, 'hwp.tgz', 'hwp2.tgz')
         except:
             exception_caught = True
         self.assertTrue(self.run_local_atexit_functions_called)