Merge lp:~dbarth/zeitgeist-extensions/fts-cjk-support into lp:zeitgeist-extensions

[David Barth]

Add support for the new CJK tokenizer in Xapian

[lp:~dbarth/zeitgeist-extensions/fts-cjk-support]

59. By David Barth on 2011-08-25

Upgrade by forcing a reindex for older databases

[David Barth (dbarth)]

For reference, the upstream recommendation for activating the CJK tokenizer. This is in the perspective of future optional tokenizer, and to avoid an API change while a new official release is not ready yet.

[Mikkel Kamstrup Erlandsen (kamstrup)]

The condition os.environ['XAPIAN_CJK_NGRAM'] == 1 will always be false, use "1". But maybe we really want to check != None since I thin I recall Olly mentioning that Xapian activates CJK if the envvar is set (disregarding value).

Stylistically we use double quotes " and not single quotes ' for strings.

review: Needs Fixing

[David Barth (dbarth)]

This should be better now

review: Needs Resubmitting

[lp:~dbarth/zeitgeist-extensions/fts-cjk-support]

60. By David Barth on 2011-09-01

smarter test and better style

[Mikkel Kamstrup Erlandsen (kamstrup)]

I merged this with some slight tweaks. Now trying to convince the test harness to run :-)

review: Approve

=== modified file 'fts/fts.py'
--- fts/fts.py	2011-07-04 10:25:10 +0000
+++ fts/fts.py	2011-09-01 08:57:26 +0000
@@ -176,6 +176,9 @@
 	
 	def __init__ (self, engine):
 		self._engine = engine
+
+		# Activate support for the CJK tokenizer
+		os.environ["XAPIAN_CJK_NGRAM"] = "1"
 	
 		log.debug("Opening full text index: %s" % INDEX_FILE)
 		self._index = xapian.WritableDatabase(INDEX_FILE, xapian.DB_CREATE_OR_OPEN)
@@ -217,6 +220,12 @@
 		self._check_index ()
 		
 	def _check_index (self):
+		if self._index.get_metadata("cjk_ngram") != "1" and os.environ["XAPIAN_CJK_NGRAM"] != None:
+			# If the database was built prior to CJK support
+			# force of a reindex
+			self._index.set_metadata("cjk_ngram", "1")
+			gobject.idle_add (self._reindex)
+
 		if self._index.get_doccount() == 0:
 			# If the index is empty we trigger a rebuild
 			# We must delay reindexing until after the engine is done setting up