Merge lp:~psivaa/britney/announce-generic-testclient into lp:~canonical-ci-engineering/britney/queued-announce-and-collect

[Para Siva]

Functionality to post a package details to pm.candidates queue

Functionality to post a package details to pm.candidates queue. Still to do is to update the excuses file with the entries. This will be part of another MP.

[lp:~psivaa/britney/announce-generic-testclient]

439. By Para Siva on 2015-05-21

Move caching decision to testclient

[Celso Providelo (cprov)]

Psivaa,

We discussed this change in a hangout ...

Basically we should cleanup the configuration setup (remove DEBUG, ARCHES and FETCH and replace your 'products' by 'required_test', everything prefixed with TESTCLIENT_) and the TestClient logic (cache per series, plain dictionary keyed by (sourcename, version), json.{dump, load}, replace needs_announce with 'if (name, version) in cache.keys()', re-use connection & SimpleQueue to announce everything in that batch).

Let's give it another round of review before staging the changes.

review: Needs Fixing

[lp:~psivaa/britney/announce-generic-testclient]

440. By Para Siva on 2015-05-21

After review meeting first pass

441. By Para Siva on 2015-05-21

Remove an entry from the cache if new version comes up

442. By Para Siva on 2015-05-21

Remove entry from excuses

443. By Para Siva on 2015-05-21

Replace source with candidate name

[Para Siva (psivaa)]

Thanks cprov for the comments and the hangout. Greatly helped.

As requested in irc, i'd like a re-review please.
Thanks

[Celso Providelo (cprov)]

Psivaa,

When you arrive this morning, please take a look at https://code.launchpad.net/~cprov/britney/testclient-api/+merge/259668 and check if my refactoring of the announcer (based on the work I've done for the collector) makes sense.

My MP still pending integration tests and better error handling (logging, at least).

Unmerged revisions

443. By Para Siva on 2015-05-21

Replace source with candidate name

442. By Para Siva on 2015-05-21

Remove entry from excuses

441. By Para Siva on 2015-05-21

Remove an entry from the cache if new version comes up

440. By Para Siva on 2015-05-21

After review meeting first pass

439. By Para Siva on 2015-05-21

Move caching decision to testclient

438. By Para Siva on 2015-05-20

pep8 and pyflakes

437. By Para Siva on 2015-05-20

A couple of tests

436. By Para Siva on 2015-05-20

Second pass with dup package entries

435. By Para Siva on 2015-05-20

Announce source package initial commit

=== modified file 'britney.conf'
--- britney.conf	2015-03-05 14:57:03 +0000
+++ britney.conf	2015-05-21 18:43:46 +0000
@@ -70,3 +70,7 @@
 BOOTTEST_DEBUG    = yes
 BOOTTEST_ARCHES   = armhf amd64
 BOOTTEST_FETCH    = yes
+
+TESTCLIENT_ENABLE           = yes
+TESTCLIENT_REQUIRED_TESTS   = selftest
+TESTCLIENT_AMQP_URIS        = amqp://guest:guest@162.213.32.181:5672//

=== modified file 'britney.py'
--- britney.py	2015-02-20 19:02:00 +0000
+++ britney.py	2015-05-21 18:43:46 +0000
@@ -227,6 +227,7 @@
                    PROVIDES, RDEPENDS, RCONFLICTS, MULTIARCH, ESSENTIAL)
 from autopkgtest import AutoPackageTest, ADT_PASS, ADT_EXCUSES_LABELS
 from boottest import BootTest
+from testclient import TestClient
 
 
 __author__ = 'Fabio Tranchitella and the Debian Release Team'
@@ -1677,6 +1678,7 @@
             excuse.addhtml("Not considered")
         excuse.run_autopkgtest = run_autopkgtest
         excuse.run_boottest = run_boottest
+        #excuse.run_producttest = run_producttest
 
         self.excuses.append(excuse)
         return update_candidate
@@ -1984,6 +1986,56 @@
                         upgrade_me.remove(excuse.name)
                         unconsidered.append(excuse.name)
 
+        if (getattr(self.options, "testclient_enable", "no") == "yes" and
+            self.options.series):
+            testing_excuses = []
+            for excuse in self.excuses:
+                # Skip removals, binary-only candidates, proposed-updates
+                # and unknown versions.
+                if (excuse.name.startswith("-") or
+                    "/" in excuse.name or
+                    "_" in excuse.name or
+                    excuse.ver[1] == "-"):
+                    continue
+                testing_excuses.append(excuse)
+
+            testclient = TestClient(
+                self, self.options.distribution, self.options.series)
+
+            if not self.options.dry_run:
+                testclient.announce_src(testing_excuses,
+                     self.options.testclient_amqp_uris.strip())
+
+            required_tests = getattr(
+                self.options, "testclient_required_tests", "").split()
+
+            for excuse in testing_excuses:
+                hints = self.hints.search('force', package=excuse.name)
+                hints.extend(
+                    self.hints.search('force-badtest', package=excuse.name))
+                forces = [x for x in hints
+                          if same_source(excuse.ver[1], x.version)]
+                for test in testclient.getTests(excuse.name):
+                    excuse.addhtml(
+                        "%s result: %s (<a href=\"%s\">results</a>)" % (
+                            test.name, test.status, test.result_url))
+                    if forces:
+                        excuse.addhtml(
+                            "Should wait for %s %s %s, but forced by "
+                            "%s" % (excuse.name, excuse.ver[1],
+                                    test.name, forces[0].user))
+                        continue
+                    if test.name not in required_tests:
+                        continue
+                    if test.status not in TestClient.VALID_STATUSES:
+                        excuse.addreason(test.name)
+                        if excuse.is_valid:
+                            excuse.is_valid = False
+                            excuse.addhtml("Not considered")
+                            upgrade_me.remove(excuse.name)
+                            unconsidered.append(excuse.name)
+
+
         # invalidate impossible excuses
         for e in self.excuses:
             # parts[0] == package name

=== added file 'testclient.py'
--- testclient.py	1970-01-01 00:00:00 +0000
+++ testclient.py	2015-05-21 18:43:46 +0000
@@ -0,0 +1,118 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2015 Canonical Ltd.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+import json
+from kombu import Connection
+import os
+import time
+
+
+class TestClient(object):
+
+    VALID_STATUSES = ('PASS',)
+
+    EXCUSE_LABELS = {
+        "PASS": '<span style="background:#87d96c">Pass</span>',
+        "FAIL": '<span style="background:#ff6666">Regression</span>',
+        "RUNNING": '<span style="background:#99ddff">Test in progress</span>',
+    }
+
+    def __init__(self, britney, distribution, series):
+        self.britney = britney
+        self.distribution = distribution
+        self.series = series
+
+    def _cache_file_name(self):
+        """Returns the name of the cache file
+
+        Provides the file name for write_cache and read_cache
+        """
+        home = os.path.expanduser("~")
+        cache_file = os.path.abspath(os.path.join(home, 'testclient',
+                                     '{}.json'.format(self.series)))
+        return cache_file
+
+    def announce_src(self, testing_excuses, amqp_uris):
+        """Puts new src details in `pm.candidates` queue"""
+
+        cached_entries = self._read_cache()
+
+        to_be_announced = []
+        for excuse in testing_excuses:
+            if excuse in cached_entries:
+                continue
+            else:
+                to_be_announced.append(excuse)
+
+        with Connection(amqp_uris) as connection:
+            try:
+                queue = connection.SimpleQueue(
+                    'pm.candidates')
+                for candidate in to_be_announced:
+                    payload = {
+                        'source': candidate.name,
+                        'version': candidate.version,
+                        'distribution': self.distribution,
+                        'series': self.series,
+                    }
+                    queue.put(payload)
+                    self._write_cache({candidate.name, candidate.version})
+            except Exception as exc:
+                print("E: [%s] - Queueing package: %s erred with %s" %
+                      (time.asctime(), candidate.name, exc))
+            finally:
+                queue.close()
+
+    def _write_cache(self, candidate):
+        """Writes new src details in to the cache file
+
+        This helps decide if a package, version needs
+        to be announced
+        """
+        cache_file = self._cache_file_name()
+
+        if not os.path.exists(os.path.dirname(cache_file)):
+            os.makedirs(os.path.dirname(cache_file))
+
+        data = self._read_cache()
+        to_remove = []
+
+        with open(cache_file, 'w') as outfile:
+            for cached_items in data:
+                for key, value in cached_items.items():
+                    # Remove the entries from the cache if
+                    # another version of an entry is coming new
+                    if candidate.name == key and candidate.version != value:
+                        to_remove.append(cached_items)
+            if {candidate.name: candidate.version} not in data:
+                data.append({candidate.name: candidate.version})
+            for removable in to_remove:
+                data.remove(removable)
+            json.dump(data, outfile)
+
+    def _read_cache(self):
+        """Reads the cache file contents
+
+        This helps decide if a package, version needs
+        to be announced
+        """
+        cache_file = self._cache_file_name()
+        entries = []
+        if not os.path.exists(cache_file):
+            return entries
+        with open(cache_file) as infile:
+            data = json.load(infile)
+            for k in data:
+                entries.append(k)
+        return entries