Merge lp:~koschmieder/zim/zim into lp:~jaap.karssenberg/zim/pyzim

[Lukas M. Koschmieder]

My Zim plugin which is inspired by Tomboy/Gnote and provides a search pane which shows results as you type.

[Jaap Karssenberg (jaap.karssenberg)]

Hi Lukas,

I'm very sorry, but I'm afraid I cannot accept this functionality in it's current form. The big problem I see is that you implement your own search method, rather than re-using the search module and the logic from the dialog. Specifically you load all pages in memory and then search on that - this will break when you have a large notebook on a small computer. The search module in zim uses the cached index and greps through pages one by one (the speed of the grep is still to be improved - I'm planning for that).

My suggestion would be that you make a new version where you re-use the widgets from the search dialog and embed these in the pane. This is trivial. Then you make any improvements for the interface on those widgets, such that both the dialog and the embedded pane benefit.

Regards,

Jaap

review: Disapprove

[Lukas M. Koschmieder (koschmieder)]

Hi Jaap,

thank you for your quick response.

I would also prefer to re-use the search module but I fear that it doesn't meet my requirements.

These are my main concerns:

First, I rate/weight search results by certain criteria and don't see how I could fit my rating system into your search logic.

Second, the key feature of my plug-in is that it shows search results as you type. Judging by how long the default search dialog takes to process a query, I suspect that the underlying logic is too slow/bloated for my needs.

Please let me know what you think. I will try to re-write my plug-in, if you're positive that my concerns are unjustified.

Regards,

Lukas

[Jaap Karssenberg (jaap.karssenberg)]

Hi Lukas,

With respect to the rating system, I'm open to improvements. Just specify
in more detail what you think should be different in the rating and we can
have a look at it. My objective is putting most relevant results on the
top, but I didn't optimize it much.

Search-as-you type for content search will indeed not work in the current
module. But loading the whole notebook in memory to solve that is not the
right way to go. You should either research using the SQLite database as a
search engine to achieve that, or think of optimizations like using
search-as-you-type for cached content like page names and searc hby word
for content. That could certainly work.

Have you looked at the search language we use? If you use keywords for
searches, searching names will be really fast.

Regards,

Jaap

On Sat, May 2, 2015 at 12:05 AM, Lukas M. Koschmieder <email address hidden>
wrote:

> Hi Jaap,
>
> thank you for your quick response.
>
> I would also prefer to re-use the search module but I fear that it doesn't
> meet my requirements.
>
> These are my main concerns:
>
> First, I rate/weight search results by certain criteria and don't see how
> I could fit my rating system into your search logic.
>
> Second, the key feature of my plug-in is that it shows search results as
> you type. Judging by how long the default search dialog takes to process a
> query, I suspect that the underlying logic is too slow/bloated for my needs.
>
> Please let me know what you think. I will try to re-write my plug-in, if
> you're positive that my concerns are unjustified.
>
> Regards,
>
> Lukas
> --
> https://code.launchpad.net/~koschmieder/zim/zim/+merge/257936
> You are reviewing the proposed merge of lp:~koschmieder/zim/zim into
> lp:zim.
>

[Lukas M. Koschmieder (koschmieder)]

Hi Jaap,

If I understand you correctly, then your main issue is the fact that my plug-in loads the entire notebook in memory. Would you allow me to keep my own search function / rating system if I got rid of the problematic page buffer / results cache?

> With respect to the rating system, I'm open to improvements. Just specify
> in more detail what you think should be different in the rating and we can
> have a look at it. My objective is putting most relevant results on the
> top, but I didn't optimize it much.

My rating system is inspired by Tomboy/Gnote. The search function prioritize "title matches" over "path matches" over "page matches". This concept may sound simple but it has proven to be useful in my daily routine. However, I'm not sure if this method is suited to improve our default results system.
With respect to improvements, it would be nice if ParseTree could also provide a case-sensitive count function.

> Have you looked at the search language we use? If you use keywords for
> searches, searching names will be really fast.

My plug-in iterates over all pages in the notebook, for each page greps its parse tree and then uses the count function to get the number of matches. Is there a faster alternative?

Unmerged revisions

758. By Lukas M. Koschmieder on 2015-04-30

Added plug-in

=== added file 'zim/plugins/quicksearch.py'
--- zim/plugins/quicksearch.py	1970-01-01 00:00:00 +0000
+++ zim/plugins/quicksearch.py	2015-04-30 17:05:53 +0000
@@ -0,0 +1,375 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# (c) 2015 Lukas M. Koschmieder ([e-mail])
+
+import gtk
+import logging
+import sys
+import gobject
+
+from zim.plugins import extends, PluginClass, WindowExtension
+from zim.notebook import Path
+from zim.gui.widgets import BOTTOM_PANE, PANE_POSITIONS, ScrolledWindow
+from zim.signals import ConnectorMixin
+
+logger = logging.getLogger('zim.plugins.quicksearch')
+
+class QuickSearchPlugin(PluginClass):	
+	plugin_info = {
+		'name': _('Quick Search'),
+		'description': _('''This plugin provides a search side pane which shows search results as you type (similar to Tomboy or Google).'''),
+		'author': 'Lukas M. Koschmieder ([e-mail])',
+		#'help': 'Plugins:Quick Search',
+	}
+	
+	plugin_preferences = (
+		('pane', 'choice', _('Position in the window'), BOTTOM_PANE, PANE_POSITIONS),
+	)
+
+@extends('MainWindow')
+class QuickSearchWindowExtension(WindowExtension):
+	
+	def __init__(self, plugin, window):
+		'''
+		Plug-in constructor.
+		
+		@plugin QuickSearchPlugin object
+		@window zim.plugins.MainWindow object
+		'''
+		
+		WindowExtension.__init__(self, plugin, window)
+		
+		self.widget = QuickSearchWidget(self.window.ui.notebook, self.uistate, self.window.ui)
+		
+		self.on_preferences_changed(plugin.preferences)
+		self.connectto(plugin.preferences, 'changed', self.on_preferences_changed)
+
+	def on_preferences_changed(self, preferences):
+		'''
+		This function is called when the user edits our plug-in preferences.
+		It re-adds our plug-in widget to Zim according to these new preferences.
+		
+		@preferences zim.dicts.ConfigDict object
+		'''
+		
+		if self.widget is None:
+			return
+		
+		try:
+			self.window.remove(self.widget)
+		except ValueError:
+			pass
+		
+		self.window.add_tab(_('Search'), self.widget, preferences['pane'])
+	
+	def teardown(self):
+		'''
+		Plug-in destructor.
+		'''
+		
+		self.window.remove(self.widget)
+		self.widget.disconnect_all()
+		self.widget = None
+
+class QuickSearchWidget(ConnectorMixin, gtk.VBox):
+	'''
+	This class is the main part of our plug-in.
+	It mainly comprises of the following GUI elements and the underlying logic:
+	- a gtk.Entry    (search field)
+	- a gtk.TreeView (results list)
+	- a gtk.Label    (status text)
+	
+	@todo Toggle between "search result view" (if a keyword has been entered)
+	      and "navigation view" (if the search field is empty).
+	'''
+	
+	TITLE_MATCH = int(sys.maxsize)
+	PATH_MATCH = TITLE_MATCH - 1
+	
+	'''Persistent UI state dictionary IDs'''
+	UISTATE_CASE_SENSITIVE = "case_sensitive"
+	
+	def __init__(self, notebook, uistate, ui):
+		'''
+		Constructor.
+		
+		@notebook zim.notebook object
+		@uistate  zim.dicts.ConfigDict object
+		@ui       zim.gui.GtkInterface object
+		'''
+		
+		gtk.VBox.__init__(self)
+		
+		self.notebook = notebook
+		self.ui = ui
+		self.uistate  = uistate
+		
+		'''States'''
+		self.is_searching = False
+		self.has_query_changed = False
+		self.has_index_changed = True
+		
+		'''Page buffer for faster page iteration'''
+		self.buffered_pages = list()
+		'''Results buffer to quickly restore previous search results'''
+		self.buffered_results = dict()
+		
+		'''Create GUI elements'''
+		self.query_entry = gtk.Entry()
+		self.options_hbox = gtk.HBox()
+		self.tomboy_mode = gtk.CheckButton("Show title/path matches")
+		self.case_sensitive_checkbox = gtk.CheckButton("Case-sensitive")
+		self.options_hbox.pack_start(self.tomboy_mode, expand=False)
+		self.options_hbox.pack_start(self.case_sensitive_checkbox, expand=False)
+		self.results_treeview = QuickSearchTreeView(self.ui)
+		self.results_scrolledwindow = ScrolledWindow(self.results_treeview)
+		self.status_label = gtk.Label("")
+		
+		'''Add GUI elements to widget'''
+		self.pack_start(self.query_entry, expand=False)
+		self.pack_start(self.options_hbox, expand=False)
+		self.pack_start(self.results_scrolledwindow, expand=True)
+		self.pack_start(self.status_label, expand=False)
+		
+		'''Edit GUI elements'''
+		self.results_scrolledwindow.set_sensitive(False)
+		self.tomboy_mode.set_active(True)
+		self.status_label.set_text("")
+		
+		'''Callback connections'''
+		self.query_entry.connect_object('changed', self.__class__.on_query_changed, self)
+		self.case_sensitive_checkbox.connect_object('clicked', self.__class__.on_case_sensitive_changed, self)
+		self.tomboy_mode.connect_object('clicked', self.__class__.on_tomboy_mode_changed, self)
+		self.connectto_all(self.notebook.index, (
+			('page-inserted', self.on_index_changed),
+			('page-deleted', self.on_index_changed),
+		))
+	
+	def on_tomboy_mode_changed(self):
+		self.on_query_changed()
+			
+	def on_case_sensitive_changed(self):
+		self.on_query_changed()
+	
+	def on_query_changed(self):
+		'''
+		This function is called when the user modifies the search field.
+		It (re)starts the search function. 
+		'''
+		
+		logging.debug("Callback: Query changed!")
+		
+		if self.is_searching:
+			self.has_query_changed = True
+		else:
+			gobject.idle_add(self.search)
+		
+		return True
+		
+	def on_index_changed(self, path, signal):
+		'''
+		This function is called when a page is added/removed to/from the notebook.
+		It tells our widget to update its page buffer.
+		
+		@path   zim.index.Index object
+		@signal zim.index.IndexPath object
+		'''
+		
+		logging.debug("Callback: Index changed!")
+		
+		self.has_index_changed = True
+		
+		return True
+		
+	def _update_buffered_pages(self):
+		'''
+		This function synchronizes our page buffer with the notebook.
+		
+		@todo Optimize function to only add/remove new/deleted page
+		'''
+		
+		self.has_index_changed = False
+		
+		logging.debug("Updating page buffer")
+		
+		self.buffered_pages = list()
+		for page in self.notebook.walk():
+			self.buffered_pages.append(page)
+	
+	def search(self):
+		'''
+		This is our search function.
+		'''
+		
+		self.is_searching = True
+		
+		is_case_sensitive = self.case_sensitive_checkbox.get_active()
+		is_tomboy_mode = self.tomboy_mode.get_active()
+		
+		if is_case_sensitive:
+			query = self.query_entry.get_text()
+		else:
+			query = self.query_entry.get_text().lower()
+		
+		logging.debug("Starting new query: " + query)
+		
+		'''Skip search if query is empty
+		@todo Enter "navigation tree mode"
+		'''
+		if(query == ""):
+			self.results_treeview.set_model(gtk.ListStore(str, str, int))
+			self.results_scrolledwindow.set_sensitive(False)
+			self.status_label.set_text("Total: " + str(len(self.buffered_pages)) + " pages")
+			self.is_searching = False
+			return False
+		
+		'''Update page buffer and clear results buffer if page index has changed'''
+		if self.has_index_changed:
+			self.has_index_changed = False
+			self.buffered_results.clear()
+			self._update_buffered_pages()
+		buffered_pages = self.buffered_pages
+		
+		'''Try to restore buffered results'''
+		liststore = self.buffered_results.get((is_case_sensitive, is_tomboy_mode, query))
+		if liststore != None:
+			self.results_treeview.set_model(liststore)
+			results_count = len(liststore)
+			if results_count == 0:
+				self.results_scrolledwindow.set_sensitive(False)
+			else:
+				self.results_scrolledwindow.set_sensitive(True)
+			self.status_label.set_text("Matches: " + str(results_count) + " pages")
+			logging.debug("Query results restored: " + query)
+			self.is_searching = False
+			return False			
+		
+		'''Start search'''
+		results = list()
+		
+		for page in buffered_pages:
+			
+			while gtk.events_pending():
+				gtk.main_iteration_do(False)
+			
+			if self.has_query_changed:
+				self.has_query_changed = False
+				logging.debug("Stopping out-dated query: " + query)
+				return True
+			
+			'''Title match'''
+			if is_tomboy_mode:
+			
+				if is_case_sensitive:
+					page_basename = page.basename
+				else:
+					page_basename = page.basename.lower()
+				
+				if query in page_basename:
+					results.append([page.name, "Title match", self.TITLE_MATCH])
+					continue
+			
+			'''Path match'''
+			if is_tomboy_mode:
+			
+				if is_case_sensitive:
+					page_name = page.name
+				else:
+					page_name = page.name.lower()
+				
+				if query in page_name:
+					results.append([page.name, "Path match", self.PATH_MATCH])
+					continue
+			
+			'''Text match'''
+			try:
+				parsetree = page.get_parsetree()
+				if parsetree is None:
+					continue
+				
+				query_count = 0
+				
+				if is_case_sensitive:
+					query_count = parsetree.count(query)
+				else:
+					for element in parsetree._etree.getiterator(): # Sorry!
+						if element.text:
+							query_count += element.text.lower().count(query)
+						if element.tail:
+							query_count += element.tail.lower().count(query)
+				
+				if query_count == 1:
+					results.append([page.name, str(query_count) + " match", query_count])
+					continue
+				if query_count > 0:
+					results.append([page.name, str(query_count) + " matches", query_count])
+					continue
+			except:
+				pass
+		
+		'''Sort results by 3rd column ("query score")'''
+		results.sort(key=lambda x: int(x[2]), reverse=True)
+		
+		liststore = gtk.ListStore(str, str, int)
+		for r in results:
+			liststore.append(r)		
+		self.results_treeview.set_model(liststore)
+		
+		'''Add result to results buffer'''
+		self.buffered_results[(is_case_sensitive, is_tomboy_mode, query)] = liststore
+		
+		results_count = len(results)
+		if results_count == 0:
+			self.results_scrolledwindow.set_sensitive(False)
+		else:
+			self.results_scrolledwindow.set_sensitive(True)
+		
+		self.status_label.set_text("Matches: " + str(results_count) + " pages")
+		
+		logging.debug("Query has been processed: " + query)
+			
+		self.is_searching = False
+		
+		return False
+
+class QuickSearchTreeView(gtk.TreeView):
+	'''
+	This class represents our results list GUI element. 
+	'''
+	
+	def __init__(self, ui):
+		'''
+		Constructor.
+		
+		@ui zim.gui.GtkInterface object: Used to open the corresponding page if the users clicks on a result
+		'''
+		gtk.TreeView.__init__(self, gtk.ListStore(str, str, int)) # page name, match description, match score
+		
+		self.ui = ui
+
+		p_col = gtk.TreeViewColumn('Page', gtk.CellRendererText(), text=0) # page name
+		m_col = gtk.TreeViewColumn('Matches', gtk.CellRendererText(), text=1) # match description
+		
+		p_col.set_sort_column_id(0) # page name
+		m_col.set_sort_column_id(2) # match score
+		
+		p_col.set_expand(True)
+		m_col.set_expand(False)
+		
+		self.append_column(p_col)
+		self.append_column(m_col)
+		
+		self.set_search_column(0)
+		self.set_search_column(1)
+		
+		self.connect('row-activated', self._do_open_page)
+		
+	def _do_open_page(self, view, path, col):
+		'''
+		This callback function is called when the user clicks on a row in the results list.
+		It opens the corresponding page.
+		'''
+		
+		page = Path(self.get_model()[path][0].decode('utf-8'))
+		self.ui.open_page(page)
+