Merge ~lgp171188/lpbuildbot:prevent-merges-buildbot-suceeds-false-positive-full-test-suite-not-run into lpbuildbot:main

[Guruprasad]

Prevent merges when build succeeds with a false positive and the full test suite is not run

For instance, buildbot considers
http://lpbuildbot.canonical.com/builders/lp-devel-focal/builds/1043 as
successful but the full test suite did not run and succeed. This change
tries to handle such false positive scenarios and prevents a merge to
the `*stable` branch.

[Guruprasad (lgp171188)]

It is possible to verify that this fix works by running the Python snippet in https://paste.ubuntu.com/p/sXK8gnJ8XH/ (requires `requests` as a dependency) and checking the details of the builds that have succeeded without the expected number of passing tests.

[Yuliy Schwartzburg (ikoruk)]

Looks good, just a few questions.

review: Approve

[Guruprasad (lgp171188)]

Thanks for the review, Yuliy. I will test this in the buildbot environment using a cowboy and some manual hacks and then merge it if everything works as expected.

diff --git a/buildbot-poll.py b/buildbot-poll.py
index 0a2b396..fee6b54 100755
--- a/buildbot-poll.py
+++ b/buildbot-poll.py
@@ -206,10 +206,36 @@ def check_builder_and_push_to_stable(options, builder, stable_branch):
 
     # Push to stable on success.
     if result == 'success':
-        if not options.quiet:
-            print('Builder %s at %s succeeded building r%s.' % (
-                builder, options.buildbot, revision))
-        run_git(options, 'push', 'origin', '%s:%s' % (revision, stable_branch))
+        latest_build_steps = latest_build["steps"]
+        tests_total = None
+        step_text = []
+        for step in latest_build_steps:
+            # This is the step where the test suite runs
+            if step.get("name") == "shell":
+                tests_total = step.get("statistics", {}).get("tests-total")
+                step_text = step.get("text")
+                break
+        # This check is required to prevent merges in the rare cases where buildbot
+        # thinks (due to subunit bugs?) that the run has succeeded but the expected
+        # number of tests have not been run by it.
+        if (
+            not tests_total
+            # At the time of implementing this check Launchpad had nearly 25000 tests
+            or tests_total < 24000
+            or " ".join(step_text) == "test no test results"
+        ):
+            if not options.quiet:
+                print(
+                    'Builder %s at %s succeeded building r%s '
+                    'but expected number of tests not run. So not merging.' % (
+                        builder, options.buildbot, revision
+                    )
+                )
+        else:
+            if not options.quiet:
+                print('Builder %s at %s succeeded building r%s.' % (
+                    builder, options.buildbot, revision))
+            run_git(options, 'push', 'origin', '%s:%s' % (revision, stable_branch))
     else:
         if not options.quiet:
             print('Builder %s at %s failed building r%s [%s].' % (