I'm a bit sad that this reuses the existing cache machinery, as I'd
hoped that we could delete all of that when we've got squid running
sensibly. But I don't have any better ideas really.
I don't think it's a safe idea to cache the outputs of all jobs that run
with images -- we'll soon be running tests on images that Andy is
building in Jenkins. I'd rather do something more explicit, like adding
a "cache_tarballs" option to the deploy_linaro action and changing
behaviour based on that...
> https://code.launchpad.net/~le-chi-thu/lava-dispatcher/cache-tarballs-v1/+merge/102181
> You are subscribed to branch lp:lava-dispatcher.
> === modified file 'lava_dispatcher/client/master.py'
> --- lava_dispatcher/client/master.py 2012-04-02 11:36:56 +0000
> +++ lava_dispatcher/client/master.py 2012-04-16 21:00:29 +0000
> @@ -37,7 +37,7 @@
> logging_spawn,
> logging_system,
> string_to_list,
> - )
> + url_to_cache, link_or_copy_file)
> from lava_dispatcher.client.base import (
> CommandRunner,
> CriticalError,
> @@ -58,13 +58,13 @@
> :param partno: The index of the partition in the image
> :param tarfile: path and filename of the tgz to output
> """
> +
> with image_partition_mounted(image, partno) as mntdir:
> cmd = "sudo tar -C %s -czf %s ." % (mntdir, tarfile)
> rc = logging_system(cmd)
> if rc:
> raise RuntimeError("Failed to create tarball: %s" % tarfile)
>
> -
> def _deploy_tarball_to_board(session, tarball_url, dest, timeout=-1):
> decompression_char = ''
> if tarball_url.endswith('.gz') or tarball_url.endswith('.tgz'):
> @@ -289,31 +289,79 @@
> return uncompressed_name
> return image_file
I have a couple of English-language requests for the method names...
On Mon, 16 Apr 2012 21:01:25 -0000, Le Chi Thu <email address hidden> wrote: /code.launchpad .net/~le- chi-thu/ lava-dispatcher /cache- tarballs- v1/+merge/ 102181 /blueprints. launchpad. net/lava- dispatcher/ +spec/cache- rootfs- boot-tarballs
> Le Chi Thu has proposed merging lp:~le-chi-thu/lava-dispatcher/cache-tarballs-v1 into lp:lava-dispatcher.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https:/
>
> BP https:/
>
> The solution I did was only caching the tarballs when the build is of
> type image which all health check jobs are using. I am not sure how
> much reuse of tarballs for jobs which are using hwpack and rootfs,
> they are right now mostly CI jobs. Maybe be we need to add a new
> blueprint to investigate the hwpack & rootfs case.
I'm a bit sad that this reuses the existing cache machinery, as I'd
hoped that we could delete all of that when we've got squid running
sensibly. But I don't have any better ideas really.
I don't think it's a safe idea to cache the outputs of all jobs that run
with images -- we'll soon be running tests on images that Andy is
building in Jenkins. I'd rather do something more explicit, like adding
a "cache_tarballs" option to the deploy_linaro action and changing
behaviour based on that...
> https:/ /code.launchpad .net/~le- chi-thu/ lava-dispatcher /cache- tarballs- v1/+merge/ 102181 r/client/ master. py' /client/ master. py 2012-04-02 11:36:56 +0000 /client/ master. py 2012-04-16 21:00:29 +0000 .client. base import ( _mounted( image, partno) as mntdir: "Failed to create tarball: %s" % tarfile) tarball_ to_board( session, tarball_url, dest, timeout=-1): url.endswith( '.gz') or tarball_ url.endswith( '.tgz') :
> You are subscribed to branch lp:lava-dispatcher.
> === modified file 'lava_dispatche
> --- lava_dispatcher
> +++ lava_dispatcher
> @@ -37,7 +37,7 @@
> logging_spawn,
> logging_system,
> string_to_list,
> - )
> + url_to_cache, link_or_copy_file)
> from lava_dispatcher
> CommandRunner,
> CriticalError,
> @@ -58,13 +58,13 @@
> :param partno: The index of the partition in the image
> :param tarfile: path and filename of the tgz to output
> """
> +
> with image_partition
> cmd = "sudo tar -C %s -czf %s ." % (mntdir, tarfile)
> rc = logging_system(cmd)
> if rc:
> raise RuntimeError(
>
> -
> def _deploy_
> decompression_char = ''
> if tarball_
> @@ -289,31 +289,79 @@
> return uncompressed_name
> return image_file
I have a couple of English-language requests for the method names...
> + def _tarball_ url_to_ cache(self, url, cachedir): join(cache_ loc.replace( '.','-' ), "tarballs") cached( self, image, lava_cachedir):
> + cache_loc = url_to_cache(url, cachedir)
> + return os.path.
> +
> + def _is_tarballs_
_are_tarballs_ cached seems better to me.
> + cache_loc = self._tarball_ url_to_ cache(image, lava_cachedir) exists( os.path. join(cache_ loc, "boot.tgz")) and \ exists( os.path. join(cache_ loc, "root.tgz")) tarballs( self, image, tarball_dir, lava_cachedir): url_to_ cache(image, lava_cachedir) join(tarball_ dir,"boot. tgz") join(tarball_ dir,"root. tgz") copy_file( os.path. join(cache_ loc, "root.tgz"), root_tgz) copy_file( os.path. join(cache_ loc, "boot.tgz"), boot_tgz) tarballs( self, image, boot_tgz, root_tgz, lava_cachedir):
> + return os.path.
> + os.path.
> +
> + def _get_cached_
> + cache_loc = self._tarball_
> +
> + boot_tgz = os.path.
> + root_tgz = os.path.
> + link_or_
> + link_or_
> +
> + return (boot_tgz,root_tgz)
> +
> + def _cached_
This should be _cache_tarballs.
> + cache_loc = self._tarball_ url_to_ cache(image, lava_cachedir) exists( cache_loc) : cache_loc) join(cache_ loc, "boot.tgz") join(cache_ loc, "root.tgz") copy(boot_ tgz, c_boot_tgz) copy(root_ tgz, c_root_tgz)
> + if not os.path.
> + os.makedirs(
> + c_boot_tgz = os.path.
> + c_root_tgz = os.path.
> + shutil.
> + shutil.
>
Cheers,
mwh