On Mon, 16 Apr 2012 21:01:25 -0000, Le Chi Thu <email address hidden> wrote:
> Le Chi Thu has proposed merging lp:~le-chi-thu/lava-dispatcher/cache-tarballs-v1 into lp:lava-dispatcher.
>
> Requested reviews:
>   Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~le-chi-thu/lava-dispatcher/cache-tarballs-v1/+merge/102181
>
> BP https://blueprints.launchpad.net/lava-dispatcher/+spec/cache-rootfs-boot-tarballs
>
> The solution I did was only caching the tarballs when the build is of
> type image which all health check jobs are using. I am not sure how
> much reuse of tarballs for jobs which are using hwpack and rootfs,
> they are right now mostly CI jobs. Maybe be we need to add a new
> blueprint to investigate the hwpack & rootfs case.

I'm a bit sad that this reuses the existing cache machinery, as I'd
hoped that we could delete all of that when we've got squid running
sensibly.  But I don't have any better ideas really.

I don't think it's a safe idea to cache the outputs of all jobs that run
with images -- we'll soon be running tests on images that Andy is
building in Jenkins.  I'd rather do something more explicit, like adding
a "cache_tarballs" option to the deploy_linaro action and changing
behaviour based on that...

> https://code.launchpad.net/~le-chi-thu/lava-dispatcher/cache-tarballs-v1/+merge/102181
> You are subscribed to branch lp:lava-dispatcher.
> === modified file 'lava_dispatcher/client/master.py'
> --- lava_dispatcher/client/master.py	2012-04-02 11:36:56 +0000
> +++ lava_dispatcher/client/master.py	2012-04-16 21:00:29 +0000
> @@ -37,7 +37,7 @@
>      logging_spawn,
>      logging_system,
>      string_to_list,
> -    )
> +    url_to_cache, link_or_copy_file)
>  from lava_dispatcher.client.base import (
>      CommandRunner,
>      CriticalError,
> @@ -58,13 +58,13 @@
>      :param partno: The index of the partition in the image
>      :param tarfile: path and filename of the tgz to output
>      """
> +
>      with image_partition_mounted(image, partno) as mntdir:
>          cmd = "sudo tar -C %s -czf %s ." % (mntdir, tarfile)
>          rc = logging_system(cmd)
>          if rc:
>              raise RuntimeError("Failed to create tarball: %s" % tarfile)
>  
> -
>  def _deploy_tarball_to_board(session, tarball_url, dest, timeout=-1):
>      decompression_char = ''
>      if tarball_url.endswith('.gz') or tarball_url.endswith('.tgz'):
> @@ -289,31 +289,79 @@
>                  return uncompressed_name
>          return image_file

I have a couple of English-language requests for the method names...

> +    def _tarball_url_to_cache(self, url, cachedir):
> +        cache_loc = url_to_cache(url, cachedir)
> +        return os.path.join(cache_loc.replace('.','-'), "tarballs")
> +
> +    def _is_tarballs_cached(self, image, lava_cachedir):

_are_tarballs_cached seems better to me.

> +        cache_loc = self._tarball_url_to_cache(image, lava_cachedir)
> +        return os.path.exists(os.path.join(cache_loc, "boot.tgz")) and \
> +               os.path.exists(os.path.join(cache_loc, "root.tgz"))
> +
> +    def _get_cached_tarballs(self, image, tarball_dir, lava_cachedir):
> +        cache_loc = self._tarball_url_to_cache(image, lava_cachedir)
> +
> +        boot_tgz = os.path.join(tarball_dir,"boot.tgz")
> +        root_tgz = os.path.join(tarball_dir,"root.tgz")
> +        link_or_copy_file(os.path.join(cache_loc, "root.tgz"), root_tgz)
> +        link_or_copy_file(os.path.join(cache_loc, "boot.tgz"), boot_tgz)
> +
> +        return (boot_tgz,root_tgz)
> +
> +    def _cached_tarballs(self, image, boot_tgz, root_tgz, lava_cachedir):

This should be _cache_tarballs.

> +        cache_loc = self._tarball_url_to_cache(image, lava_cachedir)
> +        if not os.path.exists(cache_loc):
> +              os.makedirs(cache_loc)
> +        c_boot_tgz = os.path.join(cache_loc, "boot.tgz")
> +        c_root_tgz = os.path.join(cache_loc, "root.tgz")
> +        shutil.copy(boot_tgz, c_boot_tgz)
> +        shutil.copy(root_tgz, c_root_tgz)
>  

Cheers,
mwh