Merge lp:~mterry/geonames/translations into lp:geonames

[Michael Terry]

Add support for translations of city, states, and country names.

Support returning the translated name when asking for a city's name, state, and country. Also support searching over translated names when querying.

This can be tested in silo 33.

Most of this is straightforward. But here are some notable changes:

- I've added a 387MB text file, data/alternateNames.txt. So the code branch is quite big now. (And it broke LP's diff generator.) You can generate your own diff by doing "bzr diff --old lp:geonames" in a checkout of this branch.

- I've separated the translations into gettext po files (during cites.compiled generation). I did this so that we could save the memory from all the unused translations, if we had bundled them directly into the library like the city data is. And so that we can easily plug into existing language support (LANGUAGES/LANG/etc).

- I tried testing a version that searched all translations (not just your current locale's translations; this was with a version that built translations in, so no gettext overhead), but the time delay was extreme. It felt too sluggish on the phone. So I abandoned that approach and we just look at the current translation and English (as a sort of universal backup).

- Since I included alternateNames.txt, I decided to update the other data files to match (so that cross-file IDs would all match). They no longer ship admin1Codes.txt, instead only shipping admin1CodesASCII.txt. But no big deal, we only use the strings inside of it for English backup strings, in case alternateNames.txt doesn't have an English version.

[lp:~mterry/geonames/translations]

32. By Michael Terry on 2016-04-05

Add language-pack-en to Build-Deps for tests on s390x

33. By Michael Terry on 2016-04-05

Remove some unneeded changes

[Michael Terry (mterry)]

Poke.

If no one from the Desktop team wants to review this by a couple days from now, I'll dragoon some of the unity8 team into reviewing it.

[lp:~mterry/geonames/translations]

34. By Michael Terry on 2016-04-12

Try to fix s390x failure another way

35. By Michael Terry on 2016-04-12

Fix typo

[Josh Arenson (josharenson)]

Not quite done with the review, but I have the silo on my phone and I'm playing with it. I thought I'd change my phone to Hebrew as it is non-latin, right-to-left justified, and I can read it (the ultimate edge case). With my phone's language in Hebrew, and using a Hebrew keyboard, no results are returned for any of my queries. I've checked that the places I'm searching for are in the alternateNames file as well.

In addition:

The licenses in the modified files still list the author. I don't know if you left this on purpose. All the source files _are_ consistent.

** geonames-query.c **
line 142 - env_locale is unused
line 143 - env_lang is unused

(still have a bit to review, hang on till tomorrow)

review: Needs Fixing

[lp:~mterry/geonames/translations]

36. By Michael Terry on 2016-04-20

remove Authors section from copyright headers; not strictly needed and annoying to update

37. By Michael Terry on 2016-04-20

remove unused variables

38. By Michael Terry on 2016-04-20

try removing s390x workaround, now that bug is claimed fixed

39. By Michael Terry on 2016-04-20

Nope, not fixed yet

[Michael Terry (mterry)]

I can't reproduce the problem you see with Hebrew. I install the silo and if I type ב I see results, and I can type all the way up to בוסטון (Boston).

[Josh Arenson (josharenson)]

Looks good to me now

review: Approve