Merge lp:~cwarner/unattended-upgrades/whitelisting into lp:unattended-upgrades

[Christopher Warner]

Added whitelist option, this allows one to whitelist a package, meaning it's the only package installed from origin/archive. Moved some functionality in calculate_upgradable_pkgs into a new method flag_for_upgrade and a new Unattended-Upgrade::Package-Whitelist option.

Looks like it works pretty well but would like some vetting for other package conditions.

[Michael Vogt (mvo)]

Thanks for this branch! It looks good overall, some made some comments inline.

One issue right now is that the tests need a update, i.e.:
$ (cd tests ; make)
fails right now and there is no test that checks if the whitelisting works.

It also seems that the test for the whitelisted strings needs to check if there are only whitelisted packages in the upgrade set to ensure that there is no package that is pulled in via a dependency there (unless that is not what you want).

I pushed lp:~mvo/unattended-upgrades/whitelisting with the test fixes and the whitelist verification changes. Please have a look and merge if it seems sensilble.

review: Needs Information

[Christopher Warner (cwarner)]

Yeah this looks good, about the whitelisted strings. I battled with the dependency issue as I can't tell if that would necessarily be an issue going forward for our needs specifically but for the greater good i'm assuming others wouldn't want to deal with the inconsistency. Operating on the notion that a dependency would be apart of an overall update maybe we should just allow it?

So for example someone wants to use whitelisting and they automatically assume deps are included: whitelist bash, then bash and any deps would be whitelisted. So in the case of Ubuntu, Canonical would test/smoke/vet an update of x.pkg with x.pkgs && deps. That should cover most cases, specifically if it's a security update. In the cases where one wants to hold back a dep of a whitelisted package, they could simply forcefully downgrade or link to the dep at runtime or install from source and point to the dep they care about or etc.

The opposite side of the coin is I assume all deps are upgraded and they aren't. If a lib is a dep, it's possible i'm loading upgraded executables looking for undef'd refs unattended or all the other nasty things. Of course I don't have the data to prove which use case would be a more common rationalization. It just seems like if you ok the whitelist pkg any deps surrounding it should be considered OK for unattended-upgrading.

DISCLAIMER: If this comes back to bite me in the ass at some point I'll whine about it, wave my hands and then blame you.

[Michael Vogt (mvo)]

Thanks for your reply and sorry (again) for my slow reply.

I was thinking the strict whitelist vs relaxed whitelist over and I think the best option is to provide both but default to your initial version with the relaxed whitelist as this is probably what most people expect. If you whitelist e.g. ^apt$ you will want libapt-pkg4.11 as well.

I implemented this in lp:~mvo/unattended-upgrades/whitelist - it has a Unattended-Upgrade::Package-Whitelist-Strict" option now for people who want the strict behavior.

If you could double check by branch and also if the documentation makes sense, that would be great. If its all looking good I will make a new release with this feature. Thanks again for working on this!

Cheers,
 Michael

[Christopher Warner (cwarner)]

Yeah reviewed this and it covers everything and the doc lines up. I've got nothing but thanks on this end.

=== modified file 'unattended-upgrade'
--- unattended-upgrade	2014-09-11 13:12:05 +0000
+++ unattended-upgrade	2014-10-02 19:25:32 +0000
@@ -118,6 +118,7 @@
                 if is_allowed_origin(ver, self.allowed_origins):
                     # leave as soon as we have the highest new candidate
                     new_cand = ver
+                    print "newest candidate version applied"
                     break
             if new_cand and new_cand != pkg.candidate:
                 logging.debug("adjusting candidate version: '%s'" % new_cand)
@@ -391,7 +392,7 @@
 
 
 def upgrade_in_minimal_steps(cache, pkgs_to_upgrade, blacklist,
-                             logfile_dpkg="", verbose=False):
+                             whitelist, logfile_dpkg="", verbose=False):
 
     install_log = LogInstallProgress(logfile_dpkg, verbose)
     install_log.progress_log += ".minimal-steps"
@@ -420,7 +421,8 @@
                 continue
             # double check that we are not running into side effects like
             # what could have been caused LP: #1020680
-            if not check_changes_for_sanity(cache, allowed_origins, blacklist):
+            if not check_changes_for_sanity(cache, allowed_origins, blacklist,
+                                            whitelist):
                 logging.info("While building minimal partition: "
                              "cache has not allowed changes")
                 cache.clear()
@@ -491,8 +493,16 @@
             return True
     return False
 
-
-def check_changes_for_sanity(cache, allowed_origins, blacklist):
+def is_pkgname_in_whitelist(pkgname, whitelist):
+    for whitelist_regexp in whitelist:
+        if re.match(whitelist_regexp, pkgname):
+            logging.debug("only upgrading the following package '%s'" %
+                          pkgname)
+            return True
+    return False
+
+
+def check_changes_for_sanity(cache, allowed_origins, blacklist, whitelist):
     if cache._depcache.broken_count != 0:
         return False
     for pkg in cache:
@@ -504,7 +514,13 @@
                 logging.debug("pkg '%s' not in allowed origin" % pkg.name)
                 return False
             if is_pkgname_in_blacklist(pkg.name, blacklist):
+                logging.debug("pkg '%s' package has been blacklisted" %
+                              pkg.name)
                 return False
+            if is_pkgname_in_whitelist(pkg.name, whitelist):
+                logging.debug("pkg '%s' package has been whitelisted" %
+                              pkg.name)
+                return True
             if pkg._pkg.selected_state == apt_pkg.SELSTATE_HOLD:
                 logging.debug("pkg '%s' is on hold" % pkg.name)
                 return False
@@ -778,8 +794,8 @@
     logging.debug("mail returned: %s" % ret)
 
 
-def do_install(cache, pkgs_to_upgrade, blacklisted_pkgs, options,
-               logfile_dpkg):
+def do_install(cache, pkgs_to_upgrade, blacklisted_pkgs, whitelisted_pkgs,
+               options, logfile_dpkg):
     # set debconf to NON_INTERACTIVE, redirect output
     os.putenv("DEBIAN_FRONTEND", "noninteractive")
     setup_apt_listchanges()
@@ -801,7 +817,7 @@
             # try upgrade all "pkgs" in minimal steps
             pkg_install_success = upgrade_in_minimal_steps(
                 cache, [pkg.name for pkg in pkgs_to_upgrade],
-                blacklisted_pkgs, logfile_dpkg,
+                blacklisted_pkgs, whitelisted_pkgs, logfile_dpkg,
                 options.verbose or options.debug)
         else:
             pkg_install_success = upgrade_normal(
@@ -885,6 +901,9 @@
 def get_blacklisted_pkgs():
     return apt_pkg.config.value_list("Unattended-Upgrade::Package-Blacklist")
 
+def get_whitelisted_pkgs():
+    return apt_pkg.config.value_list("Unattended-Upgrade::Package-Whitelist")
+
 
 def reboot_if_requested_and_needed(shutdown_lock=0):
     # auto-reboot (if required and the config for this is set
@@ -906,44 +925,77 @@
         pass
 
 
-def calculate_upgradable_pkgs(cache, options, allowed_origins,
-                              blacklisted_pkgs):
+def flag_for_upgrade(pkg, cache, allowed_origins,
+                     blacklisted_pkgs, whitelisted_pkgs):
+
     pkgs_to_upgrade = []
     pkgs_kept_back = []
+
+    try:
+        pkg.mark_upgrade()
+        if check_changes_for_sanity(cache, allowed_origins,
+                                    blacklisted_pkgs, whitelisted_pkgs):
+            # add to packages to upgrade
+            pkgs_to_upgrade.append(pkg)
+            # re-eval pkgs_kept_back as the resolver may fail to
+            # directly upgrade a pkg, but that may work during
+            # a subsequent operation, see debian bug #639840
+            for pkgname in pkgs_kept_back:
+                if (cache[pkgname].marked_install or
+                        cache[pkgname].marked_upgrade):
+                    pkgs_kept_back.remove(pkgname)
+                    pkgs_to_upgrade.append(cache[pkgname])
+        else:
+            logging.debug("sanity check failed")
+            rewind_cache(cache, pkgs_to_upgrade)
+            pkgs_kept_back.append(pkg.name)
+    except SystemError as e:
+        # can't upgrade
+        logging.warning(
+            _("package '%s' upgradable but fails to "
+                "be marked for upgrade (%s)") % (pkg.name, e))
+        rewind_cache(cache, pkgs_to_upgrade)
+        pkgs_kept_back.append(pkg.name)
+    return pkgs_to_upgrade, pkgs_kept_back
+
+def calculate_upgradable_pkgs(cache, options, allowed_origins,
+                              blacklisted_pkgs, whitelisted_pkgs):
     # now do the actual upgrade
     for pkg in cache:
         if options.debug and pkg.is_upgradable:
             logging.debug("Checking: %s (%s)" % (
                 pkg.name, getattr(pkg.candidate, "origins", [])))
         if (pkg.is_upgradable and
-                not is_pkgname_in_blacklist(pkg.name, blacklisted_pkgs) and
-                is_allowed_origin(pkg.candidate, allowed_origins)):
-            try:
-                pkg.mark_upgrade()
-                if check_changes_for_sanity(cache, allowed_origins,
-                                            blacklisted_pkgs):
-                    # add to packages to upgrade
-                    pkgs_to_upgrade.append(pkg)
-                    # re-eval pkgs_kept_back as the resolver may fail to
-                    # directly upgrade a pkg, but that may work during
-                    # a subsequent operation, see debian bug #639840
-                    for pkgname in pkgs_kept_back:
-                        if (cache[pkgname].marked_install or
-                                cache[pkgname].marked_upgrade):
-                            pkgs_kept_back.remove(pkgname)
-                            pkgs_to_upgrade.append(cache[pkgname])
-                else:
-                    logging.debug("sanity check failed")
-                    rewind_cache(cache, pkgs_to_upgrade)
-                    pkgs_kept_back.append(pkg.name)
-            except SystemError as e:
-                # can't upgrade
-                logging.warning(
-                    _("package '%s' upgradable but fails to "
-                      "be marked for upgrade (%s)") % (pkg.name, e))
-                rewind_cache(cache, pkgs_to_upgrade)
-                pkgs_kept_back.append(pkg.name)
-    return pkgs_to_upgrade, pkgs_kept_back
+              not is_pkgname_in_blacklist(pkg.name, blacklisted_pkgs) and
+              is_pkgname_in_whitelist(pkg.name, whitelisted_pkgs) and
+              is_allowed_origin(pkg.candidate, allowed_origins)):
+
+            logging.debug("Whitelisted package")
+            pkgs_to_upgrade, pkgs_kept_back = flag_for_upgrade(pkg,
+                                                               cache,
+                                                               allowed_origins,
+                                                               blacklisted_pkgs,
+                                                               whitelisted_pkgs)
+            return pkgs_to_upgrade, pkgs_kept_back
+
+        elif (pkg.is_upgradable and
+              	not is_pkgname_in_blacklist(pkg.name, blacklisted_pkgs) and
+              	is_allowed_origin(pkg.candidate, allowed_origins)):
+
+            pkgs_to_upgrade, pkgs_kept_back = flag_for_upgrade(pkg,
+                                                               cache,
+                                                               allowed_origins,
+                                                               blacklisted_pkgs,
+                                                               whitelisted_pkgs)
+            return pkgs_to_upgrade, pkgs_kept_back
+        else:
+
+            # Send back the empty lists
+
+            pkgs_to_upgrade = []
+            pkgs_kept_back = []
+
+            return pkgs_to_upgrade, pkgs_kept_back
 
 
 def main(options, rootdir=""):
@@ -958,10 +1010,16 @@
     # format (origin, archive), e.g. ("Ubuntu","dapper-security")
     allowed_origins = get_allowed_origins()
 
-    # pkgs that are (for some reason) not save to install
+    # pkgs that are (for some reason) not safe to install
     blacklisted_pkgs = get_blacklisted_pkgs()
     logging.info(_("Initial blacklisted packages: %s"),
                  " ".join(blacklisted_pkgs))
+
+    # install only these packages regardless of other upgrades available
+    whitelisted_pkgs = get_whitelisted_pkgs()
+    logging.info(_("Initial whitelisted packages: %s"),
+                 " ".join(whitelisted_pkgs))
+
     logging.info(_("Starting unattended upgrades script"))
 
     # display available origin
@@ -1025,7 +1083,7 @@
                                if pkg.is_auto_removable])
     # find out about the packages that are upgradable (in a allowed_origin)
     pkgs_to_upgrade, pkgs_kept_back = calculate_upgradable_pkgs(
-        cache, options, allowed_origins, blacklisted_pkgs)
+        cache, options, allowed_origins, blacklisted_pkgs, whitelisted_pkgs)
     pkgs_to_upgrade.sort(key=lambda p: p.name)
     pkgs = "\n".join([pkg.name for pkg in pkgs_to_upgrade])
     logging.debug("pkgs that look like they should be upgraded: %s" % pkgs)
@@ -1094,16 +1152,19 @@
         # redo the selection about the packages to upgrade based on the new
         # blacklist
         logging.debug("blacklist: %s" % blacklisted_pkgs)
+        # whitelist
+        logging.debug("whitelist: %s" % whitelisted_pkgs)
         # find out about the packages that are upgradable (in a allowed_origin)
-        if len(blacklisted_pkgs) > 0:
+        if len(blacklisted_pkgs) > 0 or len(whitelisted_pkgs) > 0:
             cache.clear()
             old_pkgs_to_upgrade = pkgs_to_upgrade[:]
             pkgs_to_upgrade = []
             for pkg in old_pkgs_to_upgrade:
-                logging.debug("Checking (blacklist): %s" % (pkg.name))
+                logging.debug("Checking the black and whitelist: %s" %
+                              (pkg.name))
                 pkg.mark_upgrade()
                 if check_changes_for_sanity(cache, allowed_origins,
-                                            blacklisted_pkgs):
+                                            blacklisted_pkgs, whitelisted_pkgs):
                     pkgs_to_upgrade.append(pkg)
                 else:
                     if not (pkg.name in pkgs_kept_back):
@@ -1169,7 +1230,7 @@
         shutdown_lock = apt_pkg.get_lock("/var/run/unattended-upgrades.lock")
         # do install
         pkg_install_success = do_install(
-            cache, pkgs_to_upgrade, blacklisted_pkgs, options, logfile_dpkg)
+            cache, pkgs_to_upgrade, blacklisted_pkgs, whitelisted_pkgs, options, logfile_dpkg)
 
     # send a mail (if needed)
     if not options.dry_run:
@@ -1193,7 +1254,7 @@
 
     # this ensures the commandline is logged in /var/log/apt/history.log
     apt_pkg.config.set("Commandline::AsString", " ".join(sys.argv))
- 
+
     # init the options
     parser = OptionParser()
     parser.add_option("-d", "--debug",