Merge lp:~smoser/charm-helpers/no-translation-errors into lp:charm-helpers

[Scott Moser]

Heavily reduce number of files downloaded by apt-get update

apt-get update downloads loads of files. 2 types of files not likely used on a system that is installing charms are apt-source files and translation files.

This will stop apt from downloading either of those, and by nature of reducing files downloaded, reduce the chance for the dreaded Hash sum mismatch.

[lp:~smoser/charm-helpers/no-translation-errors]

463. By Scott Moser on 2015-10-07

fix missing import

[lp:~smoser/charm-helpers/no-translation-errors]

464. By Scott Moser on 2015-10-07

Drop the source entries also.

[Scott Moser (smoser)]

In a less permanent path to the same end, curtin now has
  https://code.launchpad.net/~smoser/curtin/trunk.no-translations/+merge/275451

That basically filters out deb-src lines and and then runs apt-get update with
   --option=Acquire::Languages=none

[Stuart Bishop (stub)]

review: Abstain

Unmerged revisions

464. By Scott Moser on 2015-10-07

Drop the source entries also.

463. By Scott Moser on 2015-10-07

fix missing import

462. By Scott Moser on 2015-10-07

disable use of translation files by apt before an apt-get update

translation files are the largest source of Hash sum mismatch
These should be permanently fixed in the 16.04 cycle for 16.04
and later under LP: #804252 and LP: #1430011. But because of the
prevalence of errors and proxies that do not handle these specially
(LP: #1474417) we just configure apt to not use translation files.

Even in a world where 'Hash Sum mismatch' errors do not occur, this
is still a win in that we do not download data that no one would
ever read.

=== modified file 'charmhelpers/fetch/__init__.py'
--- charmhelpers/fetch/__init__.py	2015-08-10 13:34:31 +0000
+++ charmhelpers/fetch/__init__.py	2015-10-07 20:57:23 +0000
@@ -26,6 +26,7 @@
     config,
     log,
 )
+import glob
 import os
 
 import six
@@ -208,6 +209,37 @@
 
 def apt_update(fatal=False):
     """Update local apt cache"""
+
+    # translation files are the largest source of Hash sum mismatch
+    # These should be permanently fixed in the 16.04 cycle for 16.04
+    # and later under LP: #804252 and LP: #1430011.  But because of the
+    # prevalence of errors and proxies that do not handle these specially
+    # (LP: #1474417) we just configure apt to not use translation files.
+    notrans_f = "/etc/apt/apt.conf.d/99notranslations"
+    if not os.path.isfile(notrans_f):
+        with open(notrans_f, "w") as fp:
+            fp.write('Acquire::Languages "none";\n')
+        for tfile in glob.glob("/var/lib/apt/lists/*Translation*"):
+            if os.path.isfile(tfile):
+                os.unlink(tfile)
+
+    listfiles = glob.glob("/etc/apt/sources.list")
+    listfiles += glob.glob("/etc/apt/sources.list.d/*.list")
+    for sfile in listfiles:
+        if not os.path.isfile(sfile):
+            continue
+        with open(sfile, "r") as fp:
+            contents = fp.read()
+
+        startswith = contents.startswith("deb-src")
+        if not startswith and "\ndeb-src" not in contents:
+            continue
+
+        with open(sfile, "w") as fp:
+            if startswith:
+                fp.write("# ")
+            fp.write(contents.replace("\ndeb-src", "\n# deb-src"))
+
     cmd = ['apt-get', 'update']
     _run_apt_command(cmd, fatal)