Merge lp:~smoser/curtin/trunk.no-translations into lp:~curtin-dev/curtin/trunk

[Scott Moser]

apt-get update: do not download translation files and source files

getting translation files and source index files is simply waste in
an installer environment.

Additionally, Translation files are everybody's favorite source of
Hash Sum Mismatch errors.

[Scott Moser (smoser)]

apt-get update: do not download translation files and source files

getting translation files and source index files is simply waste in
an installer environment.

Additionally, Translation files are everybody's favorite source of
Hash Sum Mismatch errors.

[Ryan Harper (raharper)]

man, where is the dpkg/apt flag to ignore src instead of modifying *every* apt source file?

[Ryan Harper (raharper)]

would a sed -i.curtin -e "s,deb-src,#deb-src,g" /etc/apt/sources.list /etc/apt/sources.d/*.list

be faster?

[Scott Moser (smoser)]

wrt faster for the sed. possibly, although it would incur overhead of 'sed' fork.
the bigger issue is it is lossy. a sources.list file might have some 'deb-src' lines commented out and others not (in fact base cloud image does have that). so my saving of a copy is safer in that regard.

with regard to translations, though, the option is probably a better path.

apt-get -o 'Acquire::Languages=none'

seems to do the trick. I had previously believed (maybe it was true at some point) that you had to remove the files in /var/lib/apt/cache also, or you'd get them despite your config.

So, i'll fix that part. that is less temporary files.

[lp:~smoser/curtin/trunk.no-translations]

287. By Scott Moser on 2015-10-22

use apt-get option for no translations

Rather than a temporary config file, use the apt-get option to disable
translations. It really should be both more effective and more temporary.

288. By Scott Moser on 2015-10-22

remove the deletes as not necessary

[Ryan Harper (raharper)]

> wrt faster for the sed. possibly, although it would incur overhead of 'sed'
> fork.
> the bigger issue is it is lossy. a sources.list file might have some 'deb-src'
> lines commented out and others not (in fact base cloud image does have that).
> so my saving of a copy is safer in that regard.

This a moot point now, but sed -i<suffix> would retain the original, so we wouldn't be lossy; just messy with additional files and IO.

[lp:~smoser/curtin/trunk.no-translations]

289. By Scott Moser on 2015-10-23

simplify a bit. only use one temp file

[Scott Moser (smoser)]

To show the improvement here, just run a test. Note, this is on
a cloud with a local mirror, so bandwidth is high and latency low.

The difference shown below is:
download: 1023 kB -> 1881 kB
time: 4.7s -> 6.6s
http connections: 11 -> 25 (both GET and stat)
http GET: 3 -> 7

We basically clean out to clean out all '-updates' things in
/var/lib/apt/lists/. This is reasonable, versus a full 'apt-get clean', as
during the life of a stable release, the 'trusty' pocket wont change, but
at any point after initial media create, you're probably out of date
on the trusty-updates things. Note, a full 'apt-get clean' would mean
we'd download the 'trusty' pocket things again, so both cases would just
have additional number of downloads.

## make sure you're up to date lists on everything, then clean out updates
% clean() { for f in /var/lib/apt/lists/*; do case "$f" in (*-updates*) rm -f "$f";; esac; done; }
% reset() { apt-get update --quiet >/dev/null 2>&1; clean; }

% reset; time python -c 'from curtin.util import apt_update; apt_update(target="/", force=True)'
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://nova.clouds.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:2 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [635 kB]
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [323 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security InRelease
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Fetched 1,023 kB in 2s (495 kB/s)
Reading package lists...

real 0m4.731s
user 0m2.670s
sys 0m0.194s

% reset; time apt-get update --quiet
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://nova.clouds.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:2 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [240 kB]Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [140 kB]
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [635 kB]
Hit http://security.ubuntu.com trusty-security InRelease
Get:5 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [323 kB]
Get:6 http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en [308 kB]
Get:7 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en [170 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.arch...

[lp:~smoser/curtin/trunk.no-translations]

290. By Scott Moser on 2015-10-23

minor cleanups / remove print / fix comments

291. By Scott Moser on 2015-10-26

fix usage of os.path.join

os.path.join("/target", "/etc/sources.list") returns "/etc/sources.list"

Fix that usage, and only get 'listfiles' once.

=== modified file 'curtin/util.py'
--- curtin/util.py	2015-09-21 13:23:20 +0000
+++ curtin/util.py	2015-10-26 18:04:10 +0000
@@ -473,17 +473,47 @@
 
     marker = os.path.join(target, marker)
     # if marker exists, check if there are files that would make it obsolete
+    listfiles = [os.path.join(target, "etc/apt/sources.list")]
+    listfiles += glob.glob(
+        os.path.join(target, "etc/apt/sources.list.d/*.list"))
+
     if os.path.exists(marker) and not force:
-        listfiles = [os.path.join(target, "/etc/apt/sources.list")]
-        listfiles += glob.glob(
-            os.path.join(target, "/etc/apt/sources.list.d/*.list"))
         if len(find_newer(marker, listfiles)) == 0:
             return
 
-    # we're not using 'run_apt_command' so we can use 'retries' to subp
-    apt_update = ['apt-get', 'update', '--quiet']
-    with RunInChroot(target, allow_daemons=True) as inchroot:
-        inchroot(apt_update, env=env, retries=retries)
+    abs_tmpdir = tempfile.mkdtemp(dir=os.path.join(target, 'tmp'))
+    try:
+        abs_slist = abs_tmpdir + "/sources.list"
+        abs_slistd = abs_tmpdir + "/sources.list.d"
+        ch_tmpdir = "/tmp/" + os.path.basename(abs_tmpdir)
+        ch_slist = ch_tmpdir + "/sources.list"
+        ch_slistd = ch_tmpdir + "/sources.list.d"
+
+        # create tmpdir/sources.list with all lines other than deb-src
+        # avoid apt complaining by using existing and empty dir for sourceparts
+        os.mkdir(abs_slistd)
+        with open(abs_slist, "w") as sfp:
+            for sfile in listfiles:
+                with open(sfile, "r") as fp:
+                    contents = fp.read()
+                for line in contents.splitlines():
+                    line = line.lstrip()
+                    if not line.startswith("deb-src"):
+                        sfp.write(line + "\n")
+
+        update_cmd = [
+            'apt-get', '--quiet',
+            '--option=Acquire::Languages=none',
+            '--option=Dir::Etc::sourcelist=%s' % ch_slist,
+            '--option=Dir::Etc::sourceparts=%s' % ch_slistd,
+            'update']
+
+        # do not using 'run_apt_command' so we can use 'retries' to subp
+        with RunInChroot(target, allow_daemons=True) as inchroot:
+            inchroot(update_cmd, env=env, retries=retries)
+    finally:
+        if abs_tmpdir:
+            shutil.rmtree(abs_tmpdir)
 
     with open(marker, "w") as fp:
         fp.write(comment + "\n")