Merge lp:~stefanor/ibid/exchange-825217 into lp:ibid

[Stefano Rivera]

XE.com no longer has a nice helpful country to currency list, so we build our own, based on ISO-4127. This means many heuristics, so we include a reasonable test suite.

[lp:~stefanor/ibid/exchange-825217]

1033. By Stefano Rivera on 2011-08-12

Load country_codes in _load_currencies

1034. By Stefano Rivera on 2011-08-12

Comment data-structures, remove XE.com bracketed place name search heuristics, and return to pre r586 (exchange-343775) heuristics

1035. By Stefano Rivera on 2011-08-12

Plugin name in cacheable_download location

1036. By Stefano Rivera on 2011-08-12

Add test suite for resolve_currency (and rename to a public name)

1037. By Stefano Rivera on 2011-08-12

Filter non-exchangeable currencies

[Max Rabkin (max-rabkin)]

Some non-currencies are missing (search http://en.wikipedia.org/wiki/ISO_4217 for "funds code" for some more -- the funds codes are in table A.2 of the standard, we may be able to use this instead of hard coding).

You should use trial's tempdir functions instead of hardcoding /tmp.

The tests are networking tesst but it don't check for IBID_NETWORKLESS_TEST.

review: Needs Fixing

[lp:~stefanor/ibid/exchange-825217]

1038. By Stefano Rivera on 2011-08-14

Add ibid.test.TestCase, less full-stack than PluginTestCase

1039. By Stefano Rivera on 2011-08-14

use ibid.test.TestCase for conversions tests

1040. By Stefano Rivera on 2011-08-14

Use ibid.test.TestCase instead of unittest.TestCase for plugin tests

1041. By Stefano Rivera on 2011-08-14

More fund codes

[Stefano Rivera (stefanor)]

A.2 is of course a .doc file. I've just used it to hard-code some more fund codes.

The fund codes don't exactly hurt, they just cause confusion (and one of them matched euro before the euro currency :P )

I implemented mkdtemp, but throwing away the cache after every test is a bit silly, and seeing as our test.ini contains "cachedir = cache", I thought I'd just leave it using that.

[lp:~stefanor/ibid/exchange-825217]

1042. By Stefano Rivera on 2011-08-14

Missing super()

1043. By Stefano Rivera on 2011-08-14

Typos

1044. By Stefano Rivera on 2011-08-14

Add a full-stack CurrencyConversionTest

1045. By Stefano Rivera on 2011-08-14

Group 'currencies for country' output by country

1046. By Stefano Rivera on 2011-08-14

Omit 'No universal currency' entries

1047. By Stefano Rivera on 2011-08-14

Don't assume we have country_currencies for every country_code

1048. By Stefano Rivera on 2011-08-14

Try to determine the country code through country_codes if the currency does not begin with a country code

1049. By Stefano Rivera on 2011-08-14

Identify all countries into country_currencies or warn. Detect non_currencies early

1050. By Stefano Rivera on 2011-08-14

Do exact search before 'in' searches, fill more entries in country_currencies (including Euro contries)

1051. By Stefano Rivera on 2011-08-14

Strip whitespace from re-ordered names

1052. By Stefano Rivera on 2011-08-14

Make sure we get countries that primarily use another country's currency in country_currencies

1053. By Stefano Rivera on 2011-08-14

Snarky comment for same-currency conversions

[Max Rabkin (max-rabkin)]

<Taejo> tibid: currencies for Saint Helena
<tibid> Taejo: Ascension And Tristan Da Cunha Saint Helena uses Saint Helena Pound (SHP)
<Taejo> tumbleweed: ^^
<Taejo> should be called "Saint Helena, Ascension and Tristan da Cunha"
<Taejo> but your code to invert "foo, the republic of"
<Taejo> also "Taiwan, Province of China"
<Taejo> looking through the list, one solution is to check that it ends with " of" before inverting

review: Needs Fixing

[lp:~stefanor/ibid/exchange-825217]

1054. By Stefano Rivera on 2011-08-19

Tweak country comma-reordering to not reorder compound names.

1055. By Stefano Rivera on 2011-08-19

Document ibid.utils.get_country_codes

1056. By Stefano Rivera on 2011-08-19

Tweak country comma-reordering to not reorder compound names. in ibid.utils, add tests.

[Stefano Rivera (stefanor)]

Found a minutely better solution and added a test case. Also some missing documentation and stuff. Sorry the commits are rather unorganised.

[lp:~stefanor/ibid/exchange-825217]

1057. By Stefano Rivera on 2011-08-20

etree's iter() function is only in 2.7 (even if it makes lound deprecation noises)

1058. By Stefano Rivera on 2011-08-20

Ag screw it, let's stop rotating country names with commas in them. Too messy

1059. By Stefano Rivera on 2011-08-20

Country: Currency name

1060. By Stefano Rivera on 2011-08-20

More test cases for Currency Lookup

1061. By Stefano Rivera on 2011-08-20

Don't let currencies that don't start with country-codes fall through the gaps

1062. By Stefano Rivera on 2011-08-20

s/Sint/Saint/ only works if you do it everywhere

1063. By Stefano Rivera on 2011-08-20

Drop country names, the currency table we are using disambiguates currency names sufficiently

1064. By Stefano Rivera on 2011-08-20

Don't pre-compile regexes unecessarily

1065. By Stefano Rivera on 2011-08-20

Merge nested if pair

[Max Rabkin (max-rabkin)]

review: Approve

[lp:~stefanor/ibid/exchange-825217]

1066. By Stefano Rivera on 2011-10-16

Only do the inexact currency name search if rough

1067. By Stefano Rivera on 2011-10-16

Add fallthrough test case

1068. By Stefano Rivera on 2011-10-23

Handle reversed format conversion questions, such as 'convert GBP 1 to ZAR'. Otherwise units will grab them

=== modified file 'docs/api/ibid.utils.rst'
--- docs/api/ibid.utils.rst	2011-01-21 21:20:22 +0000
+++ docs/api/ibid.utils.rst	2011-10-23 20:17:24 +0000
@@ -125,9 +125,16 @@
 
    Returns the filename to the resource.
 
+.. function:: get_country_codes()
+
+   Retrieve and decode a list of ISO-3166-1 country codes.
+
+   Returns a dict of code -> country_name.
+   The codes are capitalised.
+
 .. function:: identity_name(event, identity)
 
-    Refer to *identity* naturally in response to *event*.
+   Refer to *identity* naturally in response to *event*.
 
 URL Functions
 -------------
 
=== modified file 'ibid/plugins/conversions.py'
--- ibid/plugins/conversions.py	2010-09-13 09:22:29 +0000
+++ ibid/plugins/conversions.py	2011-10-23 20:17:24 +0000
@@ -9,10 +9,10 @@
 
 import ibid
 from ibid.plugins import Processor, handler, match
-from ibid.compat import any, defaultdict
+from ibid.compat import any, defaultdict, ElementTree
 from ibid.config import Option
-from ibid.utils import file_in_path, get_country_codes, human_join, \
-                       unicode_output, generic_webservice
+from ibid.utils import (cacheable_download, file_in_path, get_country_codes,
+                        human_join, unicode_output, generic_webservice)
 from ibid.utils.html import get_html_parse_tree
 
 features = {}
@@ -312,86 +312,153 @@
     country_codes = {}
 
     def _load_currencies(self):
-        etree = get_html_parse_tree(
-                'http://www.xe.com/iso4217.php', headers = {
-                    'User-Agent': 'Mozilla/5.0',
-                    'Referer': 'http://www.xe.com/',
-                }, treetype='etree')
-
-        tbl_main = [x for x in etree.getiterator('table') if x.get('class') == 'tbl_main'][0]
-
+        iso4127_file = cacheable_download(
+                'http://www.currency-iso.org/dl_iso_table_a1.xml',
+                'conversions/iso4217.xml')
+        document = ElementTree.parse(iso4127_file)
+        # Code -> [Countries..., Currency Name]
         self.currencies = {}
-        for tbl_sub in tbl_main.getiterator('table'):
-            if tbl_sub.get('class') == 'tbl_sub':
-                for tr in tbl_sub.getiterator('tr'):
-                    code, place = [x.text for x in tr.getchildren()]
-                    name = u''
-                    if not place:
-                        place = u''
-                    if u',' in place[1:-1]:
-                        place, name = place.split(u',', 1)
-                    place = place.strip()
-                    if code in self.currencies:
-                        currency = self.currencies[code]
-                        # Are we using another country's currency?
-                        if place != u'' and name != u'' and (currency[1] == u'' or currency[1].rsplit(None, 1)[0] in place
-                                or (u'(also called' in currency[1] and currency[1].split(u'(', 1)[0].rsplit(None, 1)[0] in place)):
-                            currency[0].insert(0, place)
-                            currency[1] = name.strip()
-                        else:
-                            currency[0].append(place)
-                    else:
-                        self.currencies[code] = [[place], name.strip()]
+        # Country -> Code
+        self.country_currencies = {}
+        self.country_codes = get_country_codes()
+        # Non-currencies:
+        non_currencies = set(('BOV CLF COU MXV '
+                              'UYI XSU XUA '     # Various Fund codes
+                              'CHE CHW '         # Swiss WIR currencies
+                              'USN USS '         # US Dollar fund codes
+                              'XAG XAU XPD XPT ' # Metals
+                              'XBA XBB XBC XBD ' # Euro Bond Market
+                              'XDR XTS XXX '     # Other specials
+                             ).split())
+        no_country_codes = set(('Saint Martin',
+                                'Virgin Islands (Us)',
+                                'Virgin Islands (British)',))
+        accociated_all_countries = True
+        for currency in document.getiterator('ISO_CURRENCY'):
+            code = currency.findtext('ALPHABETIC_CODE').strip()
+            name = currency.findtext('CURRENCY').strip()
+            place = currency.findtext('ENTITY').strip().title()
+            if code == '' or code in non_currencies:
+                continue
+            # Fund codes
+            if re.match(r'^Zz[0-9]{2}', place, re.UNICODE):
+                continue
+            if code in self.currencies:
+                self.currencies[code][0].append(place)
+            else:
+                self.currencies[code] = [[place], name]
+            if place in no_country_codes:
+                continue
+            if (code[:2] in self.country_codes
+                        and code[:2] not in self.country_currencies):
+                    self.country_currencies[code[:2]] = code
+                    continue
+            ascii_place = (unicodedata.normalize('NFD', unicode(place))
+                           .encode('ASCII', 'ignore')
+                           .replace('-', ' ')
+                           .replace('Sint', 'Saint'))
+
+            # Countries with (alternative names)
+            swapped_place = None
+            m = re.match(r'^(.+?)\s+\((.+)\)$', ascii_place)
+            if m is not None:
+                swapped_place = '%s (%s)' % (m.group(2), m.group(1))
+
+            for ccode, country in self.country_codes.iteritems():
+                country = country.title()
+                ascii_country = (unicodedata.normalize('NFD', country)
+                                 .encode('ASCII', 'ignore')
+                                 .replace('-', ' ')
+                                 .replace('Sint', 'Saint'))
+                if ascii_country in (ascii_place, swapped_place):
+                    if ccode not in self.country_currencies:
+                        self.country_currencies[ccode] = code
+                    break
+            else:
+                log.info(u"ISO4127 parsing: Can't identify %s as a known "
+                         u"country", place)
+                accociated_all_countries = False
 
         # Special cases for shared currencies:
-        self.currencies['EUR'][0].insert(0, u'Euro Member Countries')
-        self.currencies['XOF'][0].insert(0, u'Communaut\xe9 Financi\xe8re Africaine')
-        self.currencies['XOF'][1] = u'Francs'
+        self.currencies['EUR'][0].append(u'Euro Member Countries')
+        self.currencies['XAF'][0].append(u"Communaut\xe9 financi\xe8re d'Afrique")
+        self.currencies['XCD'][0].append(u'Organisation of Eastern Caribbean States')
+        self.currencies['XOF'][0].append(u'Coop\xe9ration financi\xe8re en Afrique centrale')
+        self.currencies['XPF'][0].append(u'Comptoirs Fran\xe7ais du Pacifique')
+        return accociated_all_countries
 
-    def _resolve_currency(self, name, rough=True):
+    def resolve_currency(self, name, rough=True, plural_recursion=False):
         "Return the canonical name for a currency"
 
+        if not self.currencies:
+            self._load_currencies()
+
         if name.upper() in self.currencies:
             return name.upper()
 
-        strip_currency_re = re.compile(r'^[\.\s]*([\w\s]+?)s?$', re.UNICODE)
-        m = strip_currency_re.match(name)
-
+        # Strip leading dots (.TLD)
+        m = re.match(r'^[\.\s]*(.+)$', name, re.UNICODE)
         if m is None:
             return False
-
         name = m.group(1).lower()
 
-        # TLD -> country name
-        if rough and len(name) == 2 and name.upper() in self.country_codes:
-           name = self.country_codes[name.upper()].lower()
+        # TLD:
+        if rough and len(name) == 2 and name.upper() in self.country_currencies:
+            return self.country_currencies[name.upper()]
 
         # Currency Name
         if name == u'dollar':
             return "USD"
-
-        name_re = re.compile(r'^(.+\s+)?\(?%ss?\)?(\s+.+)?$' % name, re.I | re.UNICODE)
+        if name == u'pound':
+            return "GBP"
         for code, (places, currency) in self.currencies.iteritems():
-            if name_re.match(currency) or [True for place in places if name_re.match(place)]:
-                return code
-
+            if name == currency.lower():
+                return code
+            if name.title() in places:
+                return code
+
+        # There are also country names in country_codes:
+        for code, place in self.country_codes.iteritems():
+            if name == place.lower() and code in self.country_currencies:
+                return self.country_currencies[code]
+
+        # Second pass, not requiring exact match:
+        if rough:
+            for code, (places, currency) in self.currencies.iteritems():
+                if name in currency.lower():
+                    return code
+                if any(name in place.lower() for place in places):
+                    return code
+
+            for code, place in self.country_codes.iteritems():
+                if name in place.lower() and code in self.country_currencies:
+                    return self.country_currencies[code]
+
+        # Maybe it's a plural?
+        if name.endswith('s') and not plural_recursion:
+            return self.resolve_currency(name[:-1], rough, True)
         return False
 
     @match(r'^(exchange|convert)\s+([0-9.]+)\s+(.+)\s+(?:for|to|into)\s+(.+)$')
     def exchange(self, event, command, amount, frm, to):
-        if not self.currencies:
-            self._load_currencies()
-
-        if not self.country_codes:
-            self.country_codes = get_country_codes()
-
         rough = command.lower() == 'exchange'
 
-        canonical_frm = self._resolve_currency(frm, rough)
-        canonical_to = self._resolve_currency(to, rough)
+        canonical_frm = self.resolve_currency(frm, rough)
+        canonical_to = self.resolve_currency(to, rough)
         if not canonical_frm or not canonical_to:
             if rough:
-                event.addresponse(u"Sorry, I don't know about a currency for %s", (not canonical_frm and frm or to))
+                event.addresponse(
+                    u"Sorry, I don't know about a currency for %s",
+                    (not canonical_frm and frm or to))
+            return
+        if canonical_frm == canonical_to:
+            event.addresponse(
+                u"Um, that's the same currency. Tell you what, "
+                u"I can offer you my special rate of 0.5 %(currency)s for "
+                u"each %(code)s you sell me.", {
+                    'currency': self.currencies[canonical_frm][1],
+                    'code': canonical_frm,
+            })
             return
 
         data = generic_webservice(
@@ -407,14 +474,12 @@
             return
 
         event.addresponse(
-            u'%(fresult)s %(fcode)s (%(fcountry)s %(fcurrency)s) = '
-            u'%(tresult)0.2f %(tcode)s (%(tcountry)s %(tcurrency)s) '
+            u'%(fresult)s %(fcode)s (%(fcurrency)s) = '
+            u'%(tresult)0.2f %(tcode)s (%(tcurrency)s) '
             u'(Last trade rate: %(rate)s, Bid: %(bid)s, Ask: %(ask)s)', {
                 'fresult': amount,
                 'tresult': float(amount) * float(last_trade_rate),
-                'fcountry': self.currencies[canonical_frm][0][0],
                 'fcurrency': self.currencies[canonical_frm][1],
-                'tcountry': self.currencies[canonical_to][0][0],
                 'tcurrency': self.currencies[canonical_to][1],
                 'fcode': canonical_frm,
                 'tcode': canonical_to,
@@ -423,21 +488,28 @@
                 'ask': ask,
             })
 
+    @match(r'^(exchange|convert)\s+(.+)\s+([0-9.]+)\s+(?:for|to|into)\s+(.+)$')
+    def exchange_reversed(self, event, command, amount, frm, to):
+        self.exchange(event, command, frm, amount, to)
+
+
     @match(r'^(?:currency|currencies)\s+for\s+(?:the\s+)?(.+)$')
     def currency(self, event, place):
         if not self.currencies:
             self._load_currencies()
 
-        search = re.compile(place, re.I)
-        results = []
-        for code, (places, name) in self.currencies.iteritems():
-            for place in places:
-                if search.search(place):
-                    results.append(u'%s uses %s (%s)' % (place, name, code))
+        results = defaultdict(list)
+        for code, (c_places, name) in self.currencies.iteritems():
+            for c_place in c_places:
+                if re.search(place, c_place, re.I):
+                    results[c_place].append(u'%s (%s)' % (name, code))
                     break
 
         if results:
-            event.addresponse(human_join(results))
+            event.addresponse(human_join(
+                u'%s uses %s' % (place, human_join(currencies))
+                for place, currencies in results.iteritems()
+            ))
         else:
             event.addresponse(u'No currencies found')
 
 
=== modified file 'ibid/test/__init__.py'
--- ibid/test/__init__.py	2011-06-20 20:59:56 +0000
+++ ibid/test/__init__.py	2011-10-23 20:17:24 +0000
@@ -6,7 +6,7 @@
 import os
 from traceback import format_exception
 import re
-from shutil import copyfile
+import shutil
 import sys
 import tempfile
 
@@ -72,21 +72,37 @@
         return None
 
 
-class PluginTestCase(unittest.TestCase):
+class TestCase(unittest.TestCase):
+    """TestCase subclass, implementing:
+    * detection for tests using network resources
+    * basic Ibid configuration
+    """
+    network = False
+
+    def setUp(self):
+        super(TestCase, self).setUp()
+        if self.network and os.getenv('IBID_NETWORKLESS_TEST') is not None:
+            raise unittest.SkipTest('test uses network')
+        ibid.config = FileConfig(locate_resource('ibid.test', 'test.ini'))
+
+
+class PluginTestCase(TestCase):
+    """A full-stack plugin test, implementing:
+    * Loading of the specified plugins before running the tests, and cleanup
+      afterwards
+    * DB support (clean DB for each TestCase)
+    * Test events passed through the standard Ibid event dispatcher
+    """
     load = []
     noload = []
     load_base = True
     load_configured = None
     username = u'user'
     public = False
-    network = False
     empty_dbfile = None
 
     def setUp(self):
-        if self.network and os.getenv('IBID_NETWORKLESS_TEST') is not None:
-            raise unittest.SkipTest('test uses network')
-
-        ibid.config = FileConfig(locate_resource('ibid.test', 'test.ini'))
+        super(PluginTestCase, self).setUp()
 
         if self.load_configured is None:
             self.load_configured = not self.load
@@ -145,7 +161,7 @@
         if self.empty_dbfile is None:
             self._create_empty_database()
         self.dbfile = self.mktemp()
-        copyfile(self.empty_dbfile, self.dbfile)
+        shutil.copyfile(self.empty_dbfile, self.dbfile)
         ibid.config['databases']['ibid'] = 'sqlite:///' + self.dbfile
 
     def make_event(self, message=None, type=u'message'):
@@ -216,6 +232,8 @@
             self.fail("Event was expected to fail", event)
 
     def tearDown(self):
+        super(PluginTestCase, self).tearDown()
+
         for processor in ibid.processors:
             processor.shutdown()
         del ibid.processors[:]
 
=== modified file 'ibid/test/plugins/test_conversions.py'
--- ibid/test/plugins/test_conversions.py	2011-01-26 12:20:59 +0000
+++ ibid/test/plugins/test_conversions.py	2011-10-23 20:17:24 +0000
@@ -1,9 +1,11 @@
-# Copyright (c) 2010-2011, Max Rabkin
+# Copyright (c) 2010-2011, Max Rabkin, Stefano Rivera
 # Released under terms of the MIT/X/Expat Licence. See COPYING for details.
 
-from ibid.test import PluginTestCase
-
-class UnihanTest(PluginTestCase):
+import logging
+
+import ibid.test
+
+class UnihanTest(ibid.test.PluginTestCase):
     load = ['conversions']
     network = True
 
@@ -12,3 +14,49 @@
                                    u'.*the traditional form is \u99AC')
         self.assertResponseMatches(u'unihan \u99AC',
                                    u'.*the simplified form is \u9A6C')
+
+class CurrencyLookupTest(ibid.test.TestCase):
+    network = True
+
+    def setUp(self):
+        super(CurrencyLookupTest, self).setUp()
+        from ibid.plugins import conversions
+        self.processor = conversions.Currency(u'testplugin')
+
+    def test_country_association(self):
+        self.assertTrue(self.processor._load_currencies())
+
+    def test_common_currencies(self):
+        self.assertEqual(self.processor.resolve_currency('pound', True), 'GBP')
+        self.assertEqual(self.processor.resolve_currency('dollar', True), 'USD')
+        self.assertEqual(self.processor.resolve_currency('euro', True), 'EUR')
+        self.assertEqual(self.processor.resolve_currency('rand', True), 'ZAR')
+
+    def test_tld(self):
+        self.assertEqual(self.processor.resolve_currency('.za', True), 'ZAR')
+        self.assertEqual(self.processor.resolve_currency('.na', True), 'NAD')
+        self.assertEqual(self.processor.resolve_currency('.ch', True), 'CHF')
+        # Currency from a former country
+        self.assertEqual(self.processor.resolve_currency('.sx', True), 'ANG')
+        # X- Currency
+        self.assertEqual(self.processor.resolve_currency('.cm', True), 'XAF')
+
+    def test_country(self):
+        self.assertEqual(self.processor.resolve_currency('united kingdom', True), 'GBP')
+        self.assertEqual(self.processor.resolve_currency('south africa', True), 'ZAR')
+        self.assertEqual(self.processor.resolve_currency('bosnia', True), 'BAM')
+
+    def test_fallthrough(self):
+        self.assertFalse(self.processor.resolve_currency('oz', False))
+
+class CurrencyConversionTest(ibid.test.PluginTestCase):
+    load = ['conversions']
+    network = True
+
+    def test_conversion(self):
+        self.assertResponseMatches(u'exchange 1 Pound for ZAR',
+                r'1 GBP \(.+\) = [0-9.]+ ZAR \(.+\) .* Bid: [0-9.]+')
+        self.assertResponseMatches(u'exchange 1 France for Egypt',
+                r'1 EUR \(.+\) = [0-9.]+ EGP \(.+\) .* Bid: [0-9.]+')
+        self.assertResponseMatches(u'exchange 1 Virgin Islands for .tv',
+                r'1 USD \(.+\) = [0-9.]+ AUD \(.+\) .* Bid: [0-9.]+')
 
=== modified file 'ibid/test/plugins/test_core.py'
--- ibid/test/plugins/test_core.py	2010-05-01 10:02:44 +0000
+++ ibid/test/plugins/test_core.py	2011-10-23 20:17:24 +0000
@@ -1,22 +1,17 @@
 # Copyright (c) 2009-2010, Jeremy Thurgood and Max Rabkin
 # Released under terms of the MIT/X/Expat Licence. See COPYING for details.
 
-from twisted.trial import unittest
-
+import ibid
 import ibid.test
 from ibid.event import Event
 
-class TestAddressed(unittest.TestCase):
+class TestAddressed(ibid.test.TestCase):
 
     def setUp(self):
-        ibid.test.set_config({
-            u'botname': u'test_ibid',
-            u'plugins': {
-                u'testplugin': {
-                    u'names': [u'test_ibid', u'bot', u'ant']
-                    },
-                },
-            })
+        super(TestAddressed, self).setUp()
+        ibid.config.botname = u'test_ibid'
+        ibid.config.plugins['testplugin'] = ibid.test.FakeConfig(
+                {'names': [u'test_ibid', u'bot', u'ant']})
 
         from ibid.plugins import core
         self.processor = core.Addressed(u'testplugin')
 
=== modified file 'ibid/test/plugins/test_url.py'
--- ibid/test/plugins/test_url.py	2010-01-18 22:45:15 +0000
+++ ibid/test/plugins/test_url.py	2011-10-23 20:17:24 +0000
@@ -1,13 +1,13 @@
 # Copyright (c) 2009-2010, Stefano Rivera
 # Released under terms of the MIT/X/Expat Licence. See COPYING for details.
 
-from twisted.trial import unittest
-
+import ibid.test
 from ibid.plugins import urlgrab
 
-class TestURLGrabber(unittest.TestCase):
+class TestURLGrabber(ibid.test.TestCase):
 
     def setUp(self):
+        super(TestURLGrabber, self).setUp()
         self.grab = urlgrab.Grab(u'testplugin')
 
     good_grabs = [
 
=== added file 'ibid/test/test_utils.py'
--- ibid/test/test_utils.py	1970-01-01 00:00:00 +0000
+++ ibid/test/test_utils.py	2011-10-23 20:17:24 +0000
@@ -0,0 +1,19 @@
+# Copyright (c) 2011, Stefano Rivera
+# Released under terms of the MIT/X/Expat Licence. See COPYING for details.
+
+import datetime
+
+import ibid.test
+import ibid.utils
+
+class TestUtils(ibid.test.TestCase):
+    def test_ago(self):
+        self.assertEqual(ibid.utils.ago(datetime.timedelta(seconds=60)), u'1 minute')
+        self.assertEqual(ibid.utils.ago(datetime.timedelta(seconds=60000), 1), u'16 hours')
+
+class TestUtilsNetwork(ibid.test.TestCase):
+    network = True
+
+    def test_get_country_codes(self):
+        codes = ibid.utils.get_country_codes()
+        self.assertIn('ZA', codes)
 
=== modified file 'ibid/utils/__init__.py'
--- ibid/utils/__init__.py	2011-01-22 23:04:00 +0000
+++ ibid/utils/__init__.py	2011-10-23 20:17:24 +0000
@@ -376,9 +376,9 @@
         line = line.strip()
         if started and ';' in line:
             country, code = line.split(u';')
-            if u',' in country:
-                country = u' '.join(reversed(country.split(u',', 1)))
-            country = country.title()
+            country = country.lower()
+            # Hack around http://bugs.python.org/issue7008
+            country = country.title().replace(u"'S", u"'s")
             countries[code] = country
         elif line == u'':
             started = True
@@ -387,7 +387,7 @@
 
     return countries
 
-def identity_name (event, identity):
+def identity_name(event, identity):
     if event.identity == identity.id:
         return u'you'
     elif event.source == identity.source: