Merge lp:~stefanor/ibid/db-import-export into lp:~ibid-core/ibid/old-trunk-pack-0.92

[Stefano Rivera]

[lp:~stefanor/ibid/db-import-export]

728. By Stefano Rivera on 2009-08-08

DC: Treat a share of '' as 0
https://code.edge.launchpad.net/~stefanor/ibid/dc-null-share/+merge/9369

729. By Stefano Rivera on 2009-08-08

"dict" is now a synonym for "define" (for those command-driven-bot users)
https://code.edge.launchpad.net/~stefanor/ibid/dict-synonym-405702/+merge/9358

730. By Stefano Rivera on 2009-08-08

Some improvements to and use of human_join
https://code.edge.launchpad.net/~stefanor/ibid/human-join-generator/+merge/9317

731. By Stefano Rivera on 2009-08-08

Don't pick up e-mail addresses even with three domain components
https://code.edge.launchpad.net/~stefanor/ibid/url-za-net-403172/+merge/9158

732. By Stefano Rivera on 2009-08-08

MLIA page layout changed.
https://code.edge.launchpad.net/~stefanor/ibid/mlia-410481/+merge/9856

733. By Stefano Rivera on 2009-08-08

ibid-plugin wasn't unusable when load is specified in the config file
https://code.edge.launchpad.net/~stefanor/ibid/ibid-plugin-407386/+merge/9509

734. By Stefano Rivera on 2009-08-08

Bnet: Allow whitespace in game name and channel
https://code.edge.launchpad.net/~stefanor/ibid/gameservers-dota-match/+merge/9461

735. By Michael Gorven on 2009-08-08

Add Zen of Python factpack.
https://code.launchpad.net/~mgorven/ibid/zen-factpack/+merge/9237

736. By Stefano Rivera on 2009-08-10

DC: The bot was seeing itself in the user-list
https://code.edge.launchpad.net/~stefanor/ibid/dc-self-410782/+merge/9880

737. By Stefano Rivera on 2009-08-20

Surround URLs in <brackets>, makes for slightly more parseable output
https://code.edge.launchpad.net/~stefanor/ibid/url-output-404159/+merge/9250

738. By Stefano Rivera on 2009-08-20

Allow "also" on either side of the factoid setting verb
https://code.edge.launchpad.net/~stefanor/ibid/factoid-407752/+merge/9537

739. By Stefano Rivera on 2009-08-30

* Stop Old SQLAlchemy 0.4 from doing cartesian join instead of inner join
* Add configurable SQLAlchemy echo mode
* Don't cascade account deletion to identities
https://code.edge.launchpad.net/~stefanor/ibid/identity-421169/+merge/10875

740. By Stefano Rivera on 2009-09-01

Use the Event.sender values more consistently:
['nick'] The name to use in messages
['id'] Uniquely identifies the person. Use for direct (out of the blue) messages
['connection'] Identify a connection. Use for all private replies

https://code.launchpad.net/~stefanor/ibid/event-sender-401485/+merge/9852

741. By Stefano Rivera on 2009-09-01

Log with the correct timezone difference rather than +0000.
Changes internal timestamps to datetime instances.
https://code.edge.launchpad.net/~stefanor/ibid/log-tz-417182/+merge/10881

742. By Stefano Rivera on 2009-09-01

Put PostgreSQL into UTC mode at connection time
https://code.launchpad.net/~stefanor/ibid/pg-tz-417184/+merge/10880

743. By Stefano Rivera on 2009-09-01

Handle unknown modes in MyINFO
https://code.launchpad.net/~stefanor/ibid/dc-414577/+merge/10878

744. By Stefano Rivera on 2009-09-01

Add a permission that allows users to speak to the bot in public.
Useful in busy channels when newbie<->bot traffic dwarfs everything else

https://code.launchpad.net/~stefanor/ibid/public-ignore/+merge/10043

745. By Stefano Rivera on 2009-09-01

* Log exception-causing event
* Simplify tvrage date conversion to allow for year-only information
https://code.launchpad.net/~stefanor/ibid/tvshow-412447/+merge/10021

746. By Michael Gorven on 2009-09-02

Add explicit URL lengthen command, and recognise more URL shortening services.
https://code.launchpad.net/~mgorven/ibid/url-services/+merge/10164

747. By Stefano Rivera on 2009-09-02

Catch XML systax error in FML, and move quote URLs to end of output
https://code.launchpad.net/~stefanor/ibid/fml-421146/+merge/11005

748. By Stefano Rivera on 2009-09-02

Don't walk into strings, only dictionaries, in config get and set
https://code.launchpad.net/~stefanor/ibid/config-412937/+merge/11000

749. By Stefano Rivera on 2009-09-02

Mistake in TVRage merge

750. By Stefano Rivera on 2009-09-02

Reduce 'checking permission from cache' messages from INFO to DEBUG

751. By Michael Gorven on 2009-09-02

Make plugins.core.ignore a List Option.
https://code.launchpad.net/~mgorven/ibid/ignore-default-401009/+merge/11056

752. By Jonathan Hitchcock on 2009-09-03

minor update to INSTALL: wokkel is no longer shipped with ibid, needs to be installed - jabber is a core source

753. By Stefano Rivera on 2009-09-06

Don't try and download multiple copies of the same file.
https://code.launchpad.net/~stefanor/ibid/download-414574/+merge/11021

754. By Stefano Rivera on 2009-09-09

Don't include possibly ambiguous characters in auth token
https://code.launchpad.net/~stefanor/ibid/token-426349/+merge/11375

755. By Stefano Rivera on 2009-09-09

Don't use func.current_timestamp() - doesn't work when Postgres is used in UTC mode
and Timestamp columns are timezone-unaware.

https://code.launchpad.net/~stefanor/ibid/memo-date-426341/+merge/11374

756. By Stefano Rivera on 2009-09-09

Typo in memo-date-426341

757. By Stefano Rivera on 2009-10-16

Remove <brackets> from URLs. Use "text - URL" or "text. URL" instead
https://code.launchpad.net/~stefanor/ibid/prettyurl/+merge/12374

758. By Stefano Rivera on 2009-10-16

Add regex-based static-factoids.
https://code.launchpad.net/~stefanor/ibid/regex-factoids-397725/+merge/12379

759. By Stefano Rivera on 2009-10-16

Use Google's new country-specific API-search
https://code.launchpad.net/~stefanor/ibid/google-local-api/+merge/13270

760. By Stefano Rivera on 2009-10-16

Strip trailing white-space

763. By Stefano Rivera on 2009-08-07

Restore in correct dependancy order

764. By Stefano Rivera on 2009-08-07

Set sequence start values on restore for PostgreSQL

765. By Stefano Rivera on 2009-08-07

Increase index length on Unique identity constraint

766. By Stefano Rivera on 2009-08-11

Whitespace: setup.py

767. By Stefano Rivera on 2009-08-11

Install ibid-db

768. By Stefano Rivera on 2009-10-04

Whitespace

769. By Stefano Rivera on 2009-12-06

Unused import

770. By Stefano Rivera on 2009-12-06

Merge from trunk

771. By Stefano Rivera on 2009-12-06

Ignore header comment on import

772. By Stefano Rivera on 2009-12-09

ibid_import: Don't touch the Knab DB

773. By Stefano Rivera on 2009-12-09

ibid_import: Use local.ini

774. By Stefano Rivera on 2009-12-09

Check for over-length items

775. By Stefano Rivera on 2009-12-09

Double-guess chardet

776. By Stefano Rivera on 2009-12-09

Arbitrary length factoid names

777. By Stefano Rivera on 2009-12-10

Spelling

Unmerged revisions

777. By Stefano Rivera on 2009-12-10

Spelling

776. By Stefano Rivera on 2009-12-09

Arbitrary length factoid names

775. By Stefano Rivera on 2009-12-09

Double-guess chardet

774. By Stefano Rivera on 2009-12-09

Check for over-length items

773. By Stefano Rivera on 2009-12-09

ibid_import: Use local.ini

772. By Stefano Rivera on 2009-12-09

ibid_import: Don't touch the Knab DB

771. By Stefano Rivera on 2009-12-06

Ignore header comment on import

770. By Stefano Rivera on 2009-12-06

Merge from trunk

769. By Stefano Rivera on 2009-12-06

Unused import

768. By Stefano Rivera on 2009-10-04

Whitespace

=== modified file 'ibid/core.py'
--- ibid/core.py	2009-07-15 22:28:01 +0000
+++ ibid/core.py	2009-08-07 19:15:43 +0000
@@ -218,7 +218,10 @@
                     else:
                         self.log.debug("Skipping Processor: %s.%s", name, klass.__name__)
 
-            ibid.models.check_schema_versions(ibid.databases['ibid'])
+            try:
+                ibid.models.check_schema_versions(ibid.databases['ibid'])
+            except ibid.models.SchemaVersionException, e:
+                self.log.error(u'Tables out of date: %s. Run "ibid-db --upgrade"', e.message)
                 
         except Exception, e:
             self.log.exception(u"Couldn't instantiate %s processor of %s plugin", classname, name)
@@ -284,7 +287,11 @@
             self.load(database)
 
         if check_schema_versions:
-            ibid.models.check_schema_versions(self['ibid'])
+            try:
+                ibid.models.check_schema_versions(self['ibid'])
+            except ibid.models.SchemaVersionException, e:
+                self.log.error(u'Tables out of date: %s. Run "ibid-db --upgrade"', e.message)
+                raise
 
     def load(self, name):
         uri = ibid.config.databases[name]

=== modified file 'ibid/models.py'
--- ibid/models.py	2009-07-14 21:11:49 +0000
+++ ibid/models.py	2009-08-07 19:15:43 +0000
@@ -509,6 +509,14 @@
     def __repr__(self):
         return '<Account %s>' % self.username
 
+
+class SchemaVersionException(Exception):
+    """There is an out-of-date table.
+    The message should be a list of out of date tables.
+    """
+    pass
+
+
 def check_schema_versions(sessionmaker):
     """Pass through all tables, log out of date ones,
     and except if not all up to date"""
@@ -526,7 +534,7 @@
     if not upgrades:
         return
 
-    raise Exception(u"Tables %s are out of date. Run ibid-setup" % u", ".join(upgrades))
+    raise SchemaVersionException(u", ".join(upgrades))
 
 def upgrade_schemas(sessionmaker):
     "Pass through all tables and update schemas"

=== added file 'scripts/ibid-db'
--- scripts/ibid-db	1970-01-01 00:00:00 +0000
+++ scripts/ibid-db	2009-08-07 19:09:12 +0000
@@ -0,0 +1,167 @@
+#!/usr/bin/env python
+
+from datetime import datetime
+import gzip
+import logging
+from optparse import OptionParser, OptionGroup
+from os.path import exists
+import re
+from sys import stdin, stdout, stderr, exit
+
+from sqlalchemy import select, DateTime, Unicode, UnicodeText
+from twisted.python.modules import getModule
+# json only in Python >=2.6
+try:
+    import simplejson as json
+except ImportError:
+    import json
+
+import ibid
+from ibid.config import FileConfig
+from ibid.core import DatabaseManager
+from ibid.models import metadata, upgrade_schemas
+from ibid.utils import ibid_version
+
+parser = OptionParser(usage='%prog [options...]',
+        description="""Ibid Database management tool. Used for import, export,
+and upgrades. Export format is a header-line followed by JSON. FILE can be -
+for stdin/stdout or can end in .gz for automatic Gzip compression.""")
+commands = OptionGroup(parser, 'Modes')
+commands.add_option('-e', '--export', dest='export', metavar='FILE',
+        help='Export DB contents to FILE.')
+commands.add_option('-i', '--import', dest='import_', metavar='FILE',
+        help='Import DB contents from FILE. DB must be empty first.')
+commands.add_option('-u', '--upgrade', dest='upgrade', action='store_true',
+        help='Upgrade DB schema. You should backup first.')
+parser.add_option_group(commands)
+parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
+        default=False, help='Turn on debugging output to STDERR.')
+
+(options, args) = parser.parse_args()
+
+modes = sum(1 for mode in (options.import_, options.export, options.upgrade)
+        if mode is not None)
+if modes > 1:
+    parser.error('Only one mode can be specified.')
+elif modes == 0:
+    parser.error('You must specify a mode.')
+
+if options.verbose:
+    logging.basicConfig(level=logging.DEBUG)
+else:
+    logging.basicConfig(level=logging.ERROR)
+
+for module in getModule('ibid.plugins').iterModules():
+    try:
+        __import__(module.name)
+    except Exception, e:
+        print >> stderr, u"Couldn't load %s plugin: %s" % (
+                module.name.replace('ibid.plugins.', ''), unicode(e))
+
+ibid.options = {'base': '.'}
+ibid.config = FileConfig('ibid.ini')
+ibid.config.merge(FileConfig('local.ini'))
+
+def dump_table(table, db):
+    sql = select([table])
+    rows = []
+    for row in db.execute(sql):
+        out = {}
+        for key in row.iterkeys():
+            value = row[key]
+            if isinstance(value, datetime):
+                value = value.strftime('%Y-%m-%dT%H:%M:%SZ')
+            out[key] = value
+        rows.append(out)
+    return rows
+
+if options.upgrade is not None:
+    db = DatabaseManager(check_schema_versions=False)['ibid']
+    if not db.bind.has_table('schema'):
+        print >> stderr, ("Database doesn't appear to contain an Ibid. "
+                'Run ibid-setup.')
+        exit(1)
+    upgrade_schemas(db)
+
+elif options.export is not None:
+    if options.export == '-':
+        output = stdout
+    elif exists(options.export):
+        print >> stderr, (
+                'Output file (%s) exists, refusing to clobber it' %
+                options.export)
+        exit(1)
+    elif options.export.endswith('.gz'):
+        output = gzip.open(options.export, 'wb')
+    else:
+        output = open(options.export, 'wb')
+
+    output.write(('# Ibid %(version)s Database dump for %(botname)s '
+            'made on %(date)s\n') % {
+        'version': ibid_version() or 'bzr',
+        'botname': ibid.config['botname'],
+        'date': datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ'),
+    })
+
+    db = DatabaseManager()['ibid']
+    dump = dict((table.name, dump_table(table, db))
+            for table in metadata.tables.itervalues())
+
+    output.write(json.dumps(dump, sort_keys=True, indent=4))
+    output.close()
+
+elif options.import_ is not None:
+    db = DatabaseManager(check_schema_versions=False)['ibid']
+    if db.bind.has_table('schema'):
+        print >> stderr, ('Database looks like it already contains an Ibid. '
+                'Refusing to clobber it.')
+        exit(1)
+
+    upgrade_schemas(db)
+
+    if options.import_ == '-':
+        input = stdin
+    elif options.import_.endswith('.gz'):
+        input = gzip.open(options.import_, 'rb')
+    else:
+        input = open(options.import_, 'rb')
+
+    version = input.readline().strip()
+    version = re.match(r'^# Ibid (.+) Database dump for', version).group(1)
+    if version != (ibid_version() or 'bzr'):
+        print >> stderr, (
+            'Ibid version differs between dump (%(dump)s) and '
+            'local install (%(local)s). Aborting.') % {
+                'dump': version,
+                'local': ibid_version() or 'bzr',
+        }
+        exit(1)
+
+    dump = json.loads(input.read())
+
+    dump_schema = dict((table['table'], table['version'])
+            for table in dump['schema'])
+    db_schema = dict((table['table'], table['version'])
+            for table in dump_table(metadata.tables['schema'], db))
+    if dump_schema != db_schema:
+        print >> stderr, (
+            "Dump schema doesn't match DB Schema. "
+            'You must use the same version of ibid that you dumped with. '
+            'Aborting.')
+        exit(1)
+
+    for table_name, table in dump.iteritems():
+        if table_name == 'schema':
+            continue
+        dbtable = metadata.tables[table_name]
+        for row in table:
+            for field in row.iterkeys():
+                if isinstance(dbtable.c[field].type, DateTime):
+                    row[field] = datetime.strptime(row[field],
+                            '%Y-%m-%dT%H:%M:%SZ')
+                elif isinstance(dbtable.c[field].type, (Unicode, UnicodeText)):
+                    row[field] = unicode(row[field])
+            sql = dbtable.insert().values(**row)
+            db.execute(sql)
+        db.commit()
+    db.close()