pytz test failures with Python 3.8.0a2: PicklingTest.testDatabaseFixes fails

Bug #1819751 reported by Miro Hrončok
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
pytz
Fix Released
Undecided
Unassigned

Bug Description

With both pytz-2018.5 and pytz-2018.9 I get the following test failure with Python 3.8.0a2:

=================================== FAILURES ===================================
________________________ PicklingTest.testDatabaseFixes ________________________

self = <test_tzinfo.PicklingTest testMethod=testDatabaseFixes>

    def testDatabaseFixes(self):
        # Hack the pickle to make it refer to a timezone abbreviation
        # that does not match anything. The unpickler should be able
        # to repair this case
        tz = pytz.timezone('Australia/Melbourne')
        p = pickle.dumps(tz)
        tzname = tz._tzname
        hacked_p = p.replace(
            _byte_string(tzname),
            _byte_string('?' * len(tzname))
        )
        self.assertNotEqual(p, hacked_p)
        unpickled_tz = pickle.loads(hacked_p)
        self.assertTrue(tz is unpickled_tz)

        # Simulate a database correction. In this case, the incorrect
        # data will continue to be used.
        p = pickle.dumps(tz)
        new_utcoffset = tz._utcoffset.seconds + 42

        # Python 3 introduced a new pickle protocol where numbers are stored in
        # hexadecimal representation. Here we extract the pickle
        # representation of the number for the current Python version.
        old_pickle_pattern = pickle.dumps(tz._utcoffset.seconds)[3:-1]
        new_pickle_pattern = pickle.dumps(new_utcoffset)[3:-1]
        hacked_p = p.replace(old_pickle_pattern, new_pickle_pattern)

> self.assertNotEqual(p, hacked_p)
E AssertionError: b'\x80\x04\x955\x00\x00\x00\x00\x00\x00\x00\x8c\x04pytz\x94\x8c\x02_p\x94\x93\x94(\x8c\x13Australia/Melbourne\x94M\xf0\x87K\x00\x8c\x03LMT\x94t\x94R\x94.' == b'\x80\x04\x955\x00\x00\x00\x00\x00\x00\x00\x8c\x04pytz\x94\x8c\x02_p\x94\x93\x94(\x8c\x13Australia/Melbourne\x94M\xf0\x87K\x00\x8c\x03LMT\x94t\x94R\x94.'

pytz/tests/test_tzinfo.py:191: AssertionError
=============== 1 failed, 232 passed, 1 skipped in 1.61 seconds ================

Revision history for this message
Miro Hrončok (churchyard) wrote :
Revision history for this message
Victor Stinner (vstinner) wrote :

The failing test inspect pickle binary data... I'm not sure of the purpose of the test.

Python 3.8 changed to default pickle protocol from 3 to 4, whereas pytz test is written for pickle protocol 3 (or protocol 0 on Python 2).

The following change fix the test on Python 3.8. I tested my change on Python 2.7, 3.7 and 3.8:

--- pytz/tests/test_tzinfo.py.old 2019-03-14 15:02:49.694890849 +0100
+++ pytz/tests/test_tzinfo.py 2019-03-14 15:09:14.773704193 +0100
@@ -183,8 +183,11 @@
         # Python 3 introduced a new pickle protocol where numbers are stored in
         # hexadecimal representation. Here we extract the pickle
         # representation of the number for the current Python version.
- old_pickle_pattern = pickle.dumps(tz._utcoffset.seconds)[3:-1]
- new_pickle_pattern = pickle.dumps(new_utcoffset)[3:-1]
+ #
+ # Test protocol 3 on Python 3 and protocol 0 on Python 2.
+ protocol = (3 if sys.version_info >= (3,) else 0)
+ old_pickle_pattern = pickle.dumps(tz._utcoffset.seconds, protocol)[3:-1]
+ new_pickle_pattern = pickle.dumps(new_utcoffset, protocol)[3:-1]
         hacked_p = p.replace(old_pickle_pattern, new_pickle_pattern)

         self.assertNotEqual(p, hacked_p)

Revision history for this message
Victor Stinner (vstinner) wrote :

pytz/tests/test_docs.py also fails on Python 3.8 because repr(datetime.timedelta) changed to include parameter names. Example of failure:

File "pytz/tests/../../README.txt", line 213, in README.txt
Failed example:
    tz.dst(normal)
Expected:
    datetime.timedelta(0, 3600)
Got:
    datetime.timedelta(seconds=3600)

I propose the following fix which should work on all Python versions:

--- README.txt.old 2019-03-14 15:11:54.360631826 +0100
+++ README.txt 2019-03-14 15:13:31.809592925 +0100
@@ -177,29 +177,29 @@
 The ``is_dst`` parameter is ignored for most timestamps. It is only used
 during DST transition ambiguous periods to resolve that ambiguity.

->>> tz.utcoffset(normal, is_dst=True)
-datetime.timedelta(-1, 77400)
->>> tz.dst(normal, is_dst=True)
-datetime.timedelta(0, 3600)
+>>> print(tz.utcoffset(normal, is_dst=True))
+-1 day, 21:30:00
+>>> print(tz.dst(normal, is_dst=True))
+1:00:00
 >>> tz.tzname(normal, is_dst=True)
 'NDT'

->>> tz.utcoffset(ambiguous, is_dst=True)
-datetime.timedelta(-1, 77400)
->>> tz.dst(ambiguous, is_dst=True)
-datetime.timedelta(0, 3600)
+>>> print(tz.utcoffset(ambiguous, is_dst=True))
+-1 day, 21:30:00
+>>> print(tz.dst(ambiguous, is_dst=True))
+1:00:00
 >>> tz.tzname(ambiguous, is_dst=True)
 'NDT'

->>> tz.utcoffset(normal, is_dst=False)
-datetime.timedelta(-1, 77400)
->>> tz.dst(normal, is_dst=False)
-datetime.timedelta(0, 3600)
+>>> print(tz.utcoffset(normal, is_dst=False))
+-1 day, 21:30:00
+>>> print(tz.dst(normal, is_dst=False))
+1:00:00
 >>> tz.tzname(normal, is_dst=False)
 'NDT'

->>> tz.utcoffset(ambiguous, is_dst=False)
-datetime.timedelta(-1, 73800)
+>>> print(tz.utcoffset(ambiguous, is_dst=False))
+-1 day, 20:30:00
 >>> tz.dst(ambiguous, is_dst=False)
 datetime.timedelta(0)
 >>> tz.tzname(ambiguous, is_dst=False)
@@ -208,10 +208,10 @@
 If ``is_dst`` is not specified, ambiguous timestamps will raise
 an ``pytz.exceptions.AmbiguousTimeError`` exception.

->>> tz.utcoffset(normal)
-datetime.timedelta(-1, 77400)
->>> tz.dst(normal)
-datetime.timedelta(0, 3600)
+>>> print(tz.utcoffset(normal))
+-1 day, 21:30:00
+>>> print(tz.dst(normal))
+1:00:00
 >>> tz.tzname(normal)
 'NDT'

Revision history for this message
Gwyn Ciesla (limburgher) wrote :

First patch, refactored for 2019.1.

Revision history for this message
Stuart Bishop (stub) wrote :

The test ensures that an unpickled timestamp references the same instant in time (universal, not wallclock), even if changes to the timezone database have been made since it was pickled.

I've applied Victor's patches, thanks, with syntax adjusted for antique versions of Python.

Changed in pytz:
status: New → Fix Committed
Stuart Bishop (stub)
Changed in pytz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.