Merge lp:~vila/uci-engine/lets-land-the-spreadsheet into lp:uci-engine

[Vincent Ladeuil]

Disable tests that requires swift and nova SLAs that are not currently met by bootstack.

<cprov> ev: FAIL: tests.test_test_runner.TestTestRunner.test_process_ticket, 3x consecutively ? I confess I don't even know which strings to pull on swift to get a proper context for this failure
<cprov> ev: you guys can give me a "can kicker" t-shirt, but what's the point of having a test that fails 90 % of the time ?

90% ! Woohoo ! I'd kill to have a bug reproducing recipe with such a successful rate !

But I suspect you're vastly exagerating here... I'd would be happy with a
30% rate.

Long story short: this test failure value is that it tells us that we *will*
encounter that failure in production and that we *must* fix it before going
there.

<cprov> I in favor of disabling the test temporarily ... just it

Wrong idea.

Our current tarmac setup skips far too many tests already and you will miss
them.

<cprov> since I will be next on the tarmac queue and don't want to spend my night trying to land my changes

You are misusing a shared resource.

The point of gating changes on trunk is to ensure that a change in one part
of the code base is not breaking another part.

If your needs doesn't match that, use a different approach.

There are several that come to mind:

1 - reduce the deployment to a minimum: include only the components you care
    about (focus on the ticket system and its bare minimal deps if that's what
    you're after),

2 - fix the test failures you encounter,

3 - disable the tests that require SLAs that are not met by bootstack
    currently,

4 - buy more hardware and set it up in bootstack to restore the beahvior
    we've been relying on for months.

Since I'll be offline for a few days, this MP implements (3).

<vila> cprov: it tells you that your cloud behaves differently. It passes for months. But be honest, just remove all the tests that are in your way. You don't need them, they tell you nothing you want to hear.

Hence this proposal.

<cprov> vila: just until we have more intel about the problem

We do have a quite precise understanding of what is going on and how it
affects our code.

We even have reproducing recipes for them.

We even know how to fix our code to cope with these bugs.

So, what kind of intel do you miss ?

<cprov> vila: otherwise we will keep banging our head against the wall for landing unrelated changes

Doctor, it hurts when I do $X...

<vila> cprov: sure, it's not as if I never encounter the issue nor mentioned that nova behavior has evolved just like the swift one and that the fix is not available because I'm working on something else
<vila> cprov: yeah, too bad we didn't spend more time on fixing tarmac right or the known failures that identify stuff we *will* hit in production?

Like, say, the current issueS with the nova volumes.

Do we have reproducing recipes for these issues ?

<cprov> vila: production is spreadsheet replacement with mojo, that buys us time to cope with nova/swift changes later

Yeah, let's land the spreadsheet, who cares about running tests ? It's not
as if CI purpose was to provide a reliable infrastructure to run tests isn't
it ?

<cprov> it would be lot more productive to care about things that are actually in sync to what we have to deliver next week.

Stop fighting the tools and blaming people, use the tools.

<vila> cprov: go further: test runner is your way, remove it *temporarily* from your deployment, as well as its tests. Done.

That was (1).

My preferred choice is (2). Especially when dealing with races, having
reproducing recipes is a god sent gift.

This proposal (3) express the newly acquired knowledge: our code can't cope
with the current swift and nova SLAs provided by bootstack.

(4) is not for me to decide ;)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:902
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1751/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1751/rebuild

review: Approve (continuous-integration)

=== modified file 'ci-utils/ci_utils/testing/features.py'
--- ci-utils/ci_utils/testing/features.py	2014-10-15 10:35:04 +0000
+++ ci-utils/ci_utils/testing/features.py	2014-11-26 08:32:50 +0000
@@ -36,6 +36,10 @@
 class Swift(features.Feature):
 
     def _probe(self):
+        # FIXME: swift needs to handle more transient failures, tests are
+        # currently failing when swift becomes too slow. Disable all tests that
+        # require a "perfect" swift -- vila 2014-11-26
+        return False
         conn = self.get_connection()
         if conn is None:
             return False
@@ -67,6 +71,10 @@
 class NovaCompute(features.Feature):
 
     def _probe(self):
+        # FIXME: nova needs to handle more transient failures, tests are
+        # currently failing for various known reasons. Disable all tests that
+        # require a "perfect" nova. -- vila 2014-11-26
+        return False
         client = self.get_client()
         if client is None:
             return False
@@ -131,7 +139,11 @@
         self.nova = nova_compute
 
     def _probe_local(self):
-        # We can connect to swift
+        # We can connect to swift.
+
+        # FIXME: Is that a hard requirement for the local provider ? I think
+        # tests should specify the dep if needed. In the openstack provider
+        # case, it *is* a requirement for the node 0. -- vila 2014-05-22
         if not self.swift.available():
             return False