Merge lp:~gocept/landscape-client/fix-locale-dependency into lp:~landscape/landscape-client/trunk

[Steffen Allner]

This MP tackles the issue of the dependency on a given locale in tests.

With [0] the LC_ALL=C was introduced for the check target in the Makefile to solve a locale dependency with python 2 and Serbian locale.

This turned out later on to complicate things as the file system default encoding is changed with this setting in a way, which is not easily changed from within a python program. Problems arose at the interface with C-extensions which partly had some decoding errors or replaced unknown characters.

The late phenomenon is actually also present in Python 2 and was tested as landscape.package.tests.test_skeleton.test_build_skeleton_with_unicode_and_non_ascii
Comments [1] [2]

This is also the testcase which gets fixed by this MP.

I personally don't like this solution, I would rather like to make a conditional test based on the given locale as this allows to use the feature of direct decoding of non-ascii characters with Python 3 given and UTF-8 supporting locale.

[0] http://bazaar.launchpad.net/~landscape/landscape-client/trunk/files/940
[1] https://code.launchpad.net/~gocept/landscape-client/py3-package-skeleton/+merge/320610/comments/838871
[2] https://code.launchpad.net/~gocept/landscape-client/py3-package-skeleton/+merge/320610/comments/838930

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-check
Result: Success
Revno: 1001
Branch: lp:~gocept/landscape-client/fix-locale-dependency
Jenkins: https://ci.lscape.net/job/latch-test-xenial/3839/

review: Approve (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[Steffen Allner (sallner)]

Free is fixes this problem with conditional tests in [0], so it is yours to decide. I will close the card on that.

[0] https://code.launchpad.net/~free.ekanayaka/landscape-client/py3-registration/+merge/321986

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-check
Result: Success
Revno: 1001
Branch: lp:~gocept/landscape-client/fix-locale-dependency
Jenkins: https://ci.lscape.net/job/latch-test-xenial/3841/

review: Approve (test results)

[Eric Snow (ericsnowcurrently)]

Yeah, let's go with Free's change.

review: Disapprove

Unmerged revisions

1001. By Steffen Allner on 2017-04-05

Avoid locale dependency in tests.

=== modified file 'landscape/package/skeleton.py'
--- landscape/package/skeleton.py	2017-03-17 08:56:02 +0000
+++ landscape/package/skeleton.py	2017-04-05 13:10:32 +0000
@@ -152,11 +152,21 @@
         skeleton.size = version.size
         if version.installed_size > 0:
             skeleton.installed_size = version.installed_size
-        if with_unicode and not _PY3:
-            skeleton.section = skeleton.section.decode("utf-8")
-            skeleton.summary = skeleton.summary.decode("utf-8")
-            # Avoid double-decoding package descriptions in build_skeleton_apt,
-            # which causes an error with newer python-apt (Xenial onwards)
-            if not isinstance(skeleton.description, unicode):
-                skeleton.description = skeleton.description.decode("utf-8")
+        if with_unicode:
+            if not _PY3:
+                skeleton.section = skeleton.section.decode("utf-8")
+                skeleton.summary = skeleton.summary.decode("utf-8")
+                # Avoid double-decoding package descriptions in
+                # build_skeleton_apt, which causes an error with newer python-
+                # apt (Xenial onwards)
+                if not isinstance(skeleton.description, unicode):
+                    skeleton.description = skeleton.description.decode("utf-8")
+            # XXX This is a fix to keep the output consistent for different
+            # locales, i.e. LC_ALL=C vs. UTF-8. Tested in
+            # test_build_skeleton_with_unicode_and_non_ascii(). The replacement
+            # is done in apt.cache.Cache and cannot be controlled other then
+            # with FS default encoding. So we need to encode and decode here
+            # and replace non-ascii characters on the way.
+            skeleton.description = skeleton.description.encode(
+                "ascii", "replace").decode("ascii")
     return skeleton