Merge lp:~sil2100/merge-o-matic/exclude-pkgs-from-stats into lp:merge-o-matic

[Łukasz Zemczak]

Experimental try on adding a 'soft blacklist' for excluding certain packages only from m-o-m statistics.

Experimental try on adding a 'soft blacklist' for excluding certain packages only from m-o-m statistics.

This is useful since when generating statistics for some teams, some packages might not be relevant for merging. In the Foundations team we would like to omit those in the statistics which we use for graphing our KPIs.

[Łukasz Zemczak (sil2100)]

Two things:

1) I download the exclude list in cron.daily as I saw this is what's done for the blacklist. Makes more sense than having it local-only or in the source tree.
2) I was considering adding a new, separate count of 'excluded' but I couldn't find any real use for something like that. There's always risk of one forgetting that some packages are excluded, yes, but the exclude file should anyway be always kept up-to-date.

[Brian Murray (brian-murray)]

> Two things:
>
> 1) I download the exclude list in cron.daily as I saw this is what's done for
> the blacklist. Makes more sense than having it local-only or in the source
> tree.
> 2) I was considering adding a new, separate count of 'excluded' but I couldn't
> find any real use for something like that. There's always risk of one
> forgetting that some packages are excluded, yes, but the exclude file should
> anyway be always kept up-to-date.

I think the excluded count would be easy to add and doesn't cost anything, additionally it'd be useful to have this count for grafana because if it is grows a bunch something could be wrong.

[Brian Murray (brian-murray)]

One in-line comment.

[lp:~sil2100/merge-o-matic/exclude-pkgs-from-stats]

303. By Łukasz Zemczak on 2018-01-08

Remove unnecessary conditional for read_excluded_packages()

304. By Łukasz Zemczak on 2018-01-08

Add an 'excluded' statistic to keep track of the number of explicitly exluded packages from the stats.

[Brian Murray (brian-murray)]

Okay, this seems good to me.

review: Approve

[Łukasz Zemczak (sil2100)]

I also prepared a list of d-i packages to exclude for foundations. I do not have access to snakefruit so I can't get it in place, but once I get someone who can validate the list and put it up, this branch here will be good for production.

You can see the list here:
http://people.canonical.com/~lzemczak/foundations.exclude-mom

[lp:~sil2100/merge-o-matic/exclude-pkgs-from-stats]

305. By Łukasz Zemczak on 2018-01-10

Switch to shipping the exclude list in MoM bzr, add initial version of the foundations blacklist.

306. By Łukasz Zemczak on 2018-01-10

I guess it's better if we have a separate directory for 'static' files like the exclude list.

[Brian Murray (brian-murray)]

Waiting for slangasek's approval of the exclude list.

[lp:~sil2100/merge-o-matic/exclude-pkgs-from-stats]

307. By Łukasz Zemczak on 2018-01-11

Slightly trim down the exclude list.

[Steve Langasek (vorlon)]

I have discussed the particulars of this blacklist with Łukasz, and this has my +1. We may want to extend the list further later as a result of manual analysis, but this is a correct starting point with no false-positives.

review: Approve

=== modified file 'cron.daily'
--- cron.daily	2017-10-16 14:37:36 +0000
+++ cron.daily	2018-01-11 20:01:03 +0000
@@ -83,7 +83,7 @@
 # Produce pretty reports
 ./stats.py $QUIET
 ./stats.py $QUIET -t ubuntu-server
-./stats.py $QUIET -t foundations-bugs
+./stats.py $QUIET -t foundations-bugs -E data/foundations.exclude
 ./stats.py $QUIET -t ubuntu-openstack
 ./stats-graphs.py $QUIET
 ./merge-status.py $QUIET

=== added directory 'data'
=== added file 'data/foundations.exclude'
--- data/foundations.exclude	1970-01-01 00:00:00 +0000
+++ data/foundations.exclude	2018-01-11 20:01:03 +0000
@@ -0,0 +1,65 @@
+anna
+apt-setup
+base-installer
+bterm-unifont
+cdebconf-entropy
+cdebconf-keystep
+cdebconf-terminal
+cdrom-checker
+cdrom-detect
+cdrom-retriever
+choose-mirror
+clock-setup
+debian-installer
+debian-installer-utils
+efi-reader
+elilo-installer
+finish-install
+grub-installer
+installation-locale
+iso-scan
+lilo-installer
+live-installer
+lvmcfg
+main-menu
+media-retriever
+migration-assistant
+mountmedia
+netcfg
+net-retriever
+network-console
+nobootloader
+oldsys-preseed
+partconf
+partman-auto
+partman-auto-crypto
+partman-auto-loop
+partman-auto-lvm
+partman-auto-raid
+partman-base
+partman-basicfilesystems
+partman-basicmethods
+partman-btrfs
+partman-crypto
+partman-efi
+partman-ext3
+partman-iscsi
+partman-jfs
+partman-lvm
+partman-md
+partman-multipath
+partman-nbd
+partman-partitioning
+partman-swapfile
+partman-target
+partman-uboot
+partman-xfs
+pkgsel
+preseed
+rescue
+rootskel
+rootskel-gtk
+system-integrity-check
+tzsetup
+udpkg
+usb-discover
\ No newline at end of file

=== modified file 'stats.py'
--- stats.py	2017-07-06 19:11:04 +0000
+++ stats.py	2018-01-11 20:01:03 +0000
@@ -20,6 +20,7 @@
 from __future__ import print_function, with_statement
 
 import logging
+import os
 import time
 
 from deb.version import Version
@@ -39,6 +40,27 @@
     )
 
 
+def read_excluded_packages(path):
+    if not os.path.isfile(path):
+        return set()
+
+    exclude = set()
+    with open(path) as excludelist:
+        for line in excludelist:
+            try:
+                line = line[:line.index("#")]
+            except ValueError:
+                pass
+
+            line = line.strip()
+            if not line:
+                continue
+
+            exclude.add(line)
+
+    return exclude
+
+
 def options(parser):
     parser.add_option("-D", "--source-distro", type="string", metavar="DISTRO",
                       default=SRC_DISTRO,
@@ -62,6 +84,9 @@
                       help="Process only these destination components")
     parser.add_option("-t", "--team", type="string", metavar="TEAM",
                       help="Process only packages owned by this team")
+    parser.add_option("-E", "--exclude-packages", type="string", metavar="FILE",
+                      help="Exclude packages listed in the following file")
+
 
 def main(options, args):
     src_distro = options.source_distro
@@ -72,6 +97,7 @@
 
     team = options.team
     team_package_list = get_team_packages(team)
+    exclude_packages = read_excluded_packages(options.exclude_packages)
     blacklist = read_blacklist()
 
     # For each package in the destination distribution, locate the latest in
@@ -85,6 +111,7 @@
         stats["needs-merge"] = 0
         stats["repackaged"] = 0
         stats["modified"] = 0
+        stats["excluded"] = 0
 
         if options.component is not None \
                and our_component not in options.component:
@@ -102,6 +129,11 @@
             our_version = Version(our_source["Version"])
             logging.debug("%s: %s is %s", package, our_distro, our_version)
 
+            if package in exclude_packages:
+                logging.debug("%s: excluded from statistics")
+                stats["excluded"] += 1
+                continue
+
             stats["total"] += 1
 
             if package in blacklist: