Merge ~cjwatson/launchpad:py3-readline-libedit-workaround into launchpad:master

[Colin Watson]

Work around readline crash in Python 3 on 18.04

On 18.04, importing GTK (which we do in lp.testing.html5browser) indirectly loads libedit, and versions of Python 3 before 3.8.1 crash when importing readline in a process that has libedit loaded (https://bugs.python.org/issue38634, https://bugs.launchpad.net/bugs/1899076). As a result the test suite crashes quite early on.

There's unfortunately no reasonable way to avoid this, because we aren't loading libedit directly and nor are we importing readline directly; they're both quite deeply-buried side-effects of other things. We have to take an unreasonable way instead: stub out the readline module in the test suite on affected systems. This is deliberately kept as narrow as possible, and we should drop it if and when the Python bug is fixed in Ubuntu 18.04 or we no longer care about trying to get Launchpad to run on Python 3 on 18.04.

731e8cc... by Colin Watson on 2020-10-09

Switch conditional order

six.PY3 does rather less work than distro.linux_distribution(), so
evaluate the cheaper one first.

[Tom Wardill (twom)]

review: Approve

diff --git a/lib/lp/scripts/utilities/test.py b/lib/lp/scripts/utilities/test.py
index e1e67b8..644c496 100755
--- a/lib/lp/scripts/utilities/test.py
+++ b/lib/lp/scripts/utilities/test.py
@@ -25,6 +25,7 @@ import sys
 import time
 import warnings
 
+import distro
 import six
 from zope.testrunner import options
 from zope.testrunner.feature import Feature
@@ -34,6 +35,7 @@ from lp.scripts.utilities import (
     importpedant,
     warninghandler,
     )
+from lp.services.compat import mock
 from lp.services.config import config
 
 
@@ -91,6 +93,21 @@ def configure_environment():
     # Suppress accessibility warning because the test runner does not have UI.
     os.environ['GTK_MODULES'] = ''
 
+    if six.PY3 and distro.linux_distribution()[:2] == ('Ubuntu', '18.04'):
+        # XXX cjwatson 2020-10-09: Certain versions of Python crash when
+        # importing readline into a process that has libedit loaded
+        # (https://bugs.python.org/issue38634,
+        # https://bugs.launchpad.net/bugs/1899076), so stub out readline to
+        # prevent this.  This unfortunately makes debugging less pleasant.
+        #
+        # So far the only LTS version of Ubuntu that exhibits this behaviour
+        # is 18.04; 16.04 doesn't seem to end up loading libedit because its
+        # libGL is laid out differently in a way that doesn't end up loading
+        # libedit, and 20.04 has the Python bug fixed.  We should drop this
+        # once 18.04 is fixed or once we no longer care about it, since this
+        # workaround is pretty nasty.
+        sys.modules['readline'] = mock.Mock()
+
 
 def filter_warnings():
     # Silence spurious warnings. Note that this does not propagate to
diff --git a/setup.py b/setup.py
index e88b299..4e76a25 100644
--- a/setup.py
+++ b/setup.py
@@ -163,6 +163,7 @@ setup(
         'cssselect',
         'cssutils',
         'defusedxml',
+        'distro',
         'dkimpy[ed25519]',
         'dulwich',
         'feedparser',