Autopilot lacks support for large timestamps

Autopilot needs to allow support for larger timestamps. The error above is a 32bit / c limitation that could be removed by using other datetime functions that support large timestamps (beyond 2038). I got it working by tweaking autopilot to use timedelta instead of fromtimestamp for instance . . .

Interesting about the use of timedelta, I need to look into it further as I notice this inconsistency (note the hour difference, also this is CnP from SO):

>>> datetime.datetime.fromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)

>>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 18, 27, 27)

I'll be looking into this further.

For calendar, this prevents using the time/date selection during the new event tests.

This has caused a lot of failures on the ubuntu ui toolklit on image #136.
http://ci.ubuntu.com/smokeng/utopic/touch/mako/136:20140717.1:20140717.1/9111/ubuntuuitoolkit/1385035/

Nothing seems to have changed, so I guess we just have a bigger timestamp now.

This (Leo's comment) means that we cannot land any changes for UITK because the autopilot tests fail

Veebers, the differences you are seeing are due to timezone.

>>> datetime.datetime.utcfromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)
>>> datetime.datetime.utcfromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)

There seems to be something weird on your machine, because without the utc calls you should get the same date on both calls, that's the part I don't understand.

>>> datetime.datetime.fromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 11, 27, 27)
>>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 11, 27, 27)

My tz is -6, so I correctly get 11 as the hour.

Still investigating this to see where the easiest fix lies, along with the real source of the issues. Looking at the toolkit, you could adjust the maximum property for the datepicker objects to avoid this issue. Adjusting the qml for the affected tests to set a lower maximum property does work, and the display is updated to reflect the new max year properly.

However, the default maxyear (currently 2064) object still exists. It's seemingly being created by AP? It's unclear.

I can confirm the maxyear is coming from the toolkit; the timepicker object also has these values. Setting the maximum property inside the tests will workaround the 32bit limitation.

Although autopilot should still add support, we can workaround the issue inside the testcases themselves by modifying these properties to ensure we don't hit the overrun error.

So, I've spent quite some time looking at this problem today.

There are actually several, related problems here:

First, on 32 bit platforms, you cannot create a timedelta object with large values either. This fails:

python3 -c "import datetime; datetime.timedelta(2983579200)"

so the proposed fix for autopilot doesn't, in fact, fix anything.

Second, adding a timedelta to a datetime object takes into account the local timezone, including the DST. So, for example:

>>> def do_test(num_seconds):
... dt1 = datetime.fromtimestamp(num_seconds)
... dt2 = datetime.fromtimestamp(0) + timedelta(seconds=num_seconds)
... print("Timestamps are %s the same: %s %s" % ("" if dt1 == dt2 else "not", dt1, dt2))
...
>>> do_test(2222121600)
Timestamps are the same: 2040-06-01 12:00:00 2040-06-01 12:00:00
>>> do_test(2208988800)
Timestamps are not the same: 2040-01-01 13:00:00 2040-01-01 12:00:00

This has nothing to do with the year of the timestamp, but rather because in New Zealand (which is my current locale), Jan has DST applied, but June does not (or the other way around, whatever).

So, sadly, both approaches have problems:

* Using datetime.fromtimestamp (current implementation) breaks because datetime uses C libraries that, on a 32 bit system, won't work with timestamps that overflow an int (which will be 32 bits).

* Using datetime.timedelta for some reason applies DST adjustments, which is frankly dumb. I'd expect this to hold true all the time:

datetime.fromtimestamp(N) == datetime.fromtimestamp(0) + timedelta(seconds=N)

but it only evaluates to true in local winter months it seems.

So this needs more investigation, is my final verdict :)

more investigation? we are not able to figure out how to do timestamp calcs properly in python? sigh...

who can fix this?

In the example:

python3 -c "import datetime; datetime.timedelta(2983579200)"

realize that the first argument is days, not seconds, so you're trying to create a date more than 8 million years in the future. In the abstract, I hope Ubuntu - and the human race - is around that long :). Try this:

python3 -c "import datetime; datetime.timedelta(seconds=2983579200)"

That should work even on 32 bit machines (it works in my i386 utopic chroot).

The documentation does describe that datetime.{utc,}fromtimestamp() is limited by the argument size to libc's gtime() so you'd have the same problems with a C program. How would you solve 64 bit timestamps on 32 bit platforms in C or C++? I'm not sure there is a solution for 64 bit datetimes on 32 bit platforms. E.g. numpy has a datetime64, but substituting the above I get a NaT (i.e. not-a-time). Maybe I'm using it wrong though. There's also egenix-datetime, but that's Python 2 only in the archive (and maybe upstream). It's probably worth investigating available options on PyPI.

On the second point, remember that Python datetimes have two "modes", a naive mode where timezones are ignored but local time is usually assumed, and timezone aware datetimes. IMHO, it's *always* a good idea to do your internal time calculations in UTC time, and create the appropriate timezone objects for conversion/display to local time at the edges. I think of it the same way as Unicode: convert from whatever encoding your bytes are in to Unicode at the edges, manipulate internally as unicode, and then encode to bytes on the way out. So by analogy, convert to UTC time on the way in and local time on the way out. E.g. datetime.utcfromteimstamp() is probably better for your use case.

Updating my MP after discussing with Barry and thomi. We should be able to deal with the timezone issue thomi is describing and continue to use timedelta to provide larger timestamp support.

Chris, Thomi and I discussed the need for a set of functional tests to assert what should be occuring with datetimestamps and timezones that go beyond what exists in autopilot trunk.

Namely, we'd like a suite of tests that can assert the same date + time in python as is present in a Qml. We'd like scenarios then to cover all 4 seasons (one every quarter to deal with daylight savings and other oddities), as well as several timezones around the globe.

The current MP does this on a unit testing level, but there is concern about how things correspond at a functional testing level. Chris and I agreed it would be useful to propose these functional level test enhancements as a separate MP against autopilot trunk to assert and understand what is occurring in autopilot today (filing bugs if found). These tests can then be used to verify the changes made in the MP and land them safely.

Is this bug still affecting Calendar? I understand there's nothing in the app that needs to be fixed, and the fixes would rather be on the test suite. Do we need a bug task for Calendar at all?

Fix committed into lp:autopilot at revision None, scheduled for release in autopilot, milestone 1.4

Both calendar and clock tests can now check for this. For calendar, this means the tests for specific dates and using the datefields should be re-added.

Tracking fixes for calendar and clock in individual bugs. The issue no longer affects them directly.