Merge lp:~pmarchwiak/synapse-project/recoll-plugin into lp:synapse-project

[Patrick Marchwiak]

This plugin runs queries against a Recoll index providing search results based on file contents (document types supported by Recoll include msword and PDF).

Feature summary for the current implementation:
* returns up to 20 UriMatches, using the filename as the title and filename + abstract (sample of doc contents) as the description
* results are scored lower than results from the locate plugin to ensure file name matches show up first in the results
* plugin is only enabled when the recoll command is available

[Michal Hruby (mhr3)]

Thanks for the contribution, it's looking pretty good, although I do have one gripe:

I'm not a fan of spawning processes in the search() method of an ItemProvider plugin - this is called on every keystroke and is pretty heavy on the system. That's why "Locate" is hidden behind an action, and before calculating with `bc` the query is first checked against a regex. Could you please do something like that?

Otherwise, the code is clean, nice job ;)

[Patrick Marchwiak (pmarchwiak)]

I understand your concern.

I feel there's more motivation for the action approach with "Locate" since it is quite common for users to have a large enough number of files that it would take too long to run. I don't see how checking a regex would help in the search index situation, since anything a user types is a valid search. In my (limited) testing on a more recent i5 laptop as well as an older core duo laptop, results have been quick to return with this plugin. However, I do appreciate that spawning a process could be taxing on other systems. The other drawback of hiding it behind an action is that it would require additional keystrokes to get to the results, which makes it less useful to me.

I'm not sure when I'll find the free time, but I will consider making hiding it behind an action.

[Ari (ari-lp)]

Hi Patrick,

first of all thank you very much for this plugin. I have been searching for a way to integrate Recoll into Synapse for a while now and I am very glad I found your code. I am not a coder myself but I just wanted to say that I agree with Michal's remarks. If you're working with a large index - say thousands of documents - searching can be quite resource-intensive, both in terms of CPU cycles and hard disk activity. Engaging a search while the user is still typing is counter-productive in these use cases. That's why I would love to see this plugin implemented as an action. It would be fantastic if you could modify it in that regard.

Thank you for taking the time to read this.

Cheers
--Ari

Unmerged revisions

511. By Patrick Marchwiak on 2012-11-10

Set the description to file name + abstract in recoll plugin

510. By Patrick Marchwiak on 2012-11-10

Turn down match score and use Utils.FileInfo to build better results in recoll plugin

509. By Patrick Marchwiak on 2012-11-09

Remove python script for accessing recoll and invoke cmd line tool directly

508. By Patrick Marchwiak on 2012-11-08

Add initial version of recoll plugin

=== modified file 'src/plugins/Makefile.am'
--- src/plugins/Makefile.am	2012-03-18 19:11:23 +0000
+++ src/plugins/Makefile.am	2012-11-10 07:41:20 +0000
@@ -48,6 +48,7 @@
 	opensearch.vala \
 	pastebin-plugin.vala \
 	pidgin-plugin.vala \
+	recoll-plugin.vala \
 	rhythmbox-plugin.vala \
 	selection-plugin.vala \
 	test-slow-plugin.vala \

=== added file 'src/plugins/recoll-plugin.vala'
--- src/plugins/recoll-plugin.vala	1970-01-01 00:00:00 +0000
+++ src/plugins/recoll-plugin.vala	2012-11-10 07:41:20 +0000
@@ -0,0 +1,189 @@
+  /*
+   *
+   * Authored by Patrick Marchwiak <[e-mail]>
+   *
+   */
+   
+  namespace Synapse
+  {
+  
+  // Sends query to Recoll command line tool.
+  public class RecollPlugin : Object, Activatable, ItemProvider
+  {
+    // a mandatory property
+    public bool enabled { get; set; default = true; }
+ 
+    // this method is called when a plugin is enabled
+    // use it to initialize your plugin
+    public void activate ()
+    {
+    }
+ 
+    // this method is called when a plugin is disabled
+    // use it to free the resources you're using
+    public void deactivate ()
+    {
+    }
+ 
+    // register your plugin in the UI
+    static void register_plugin ()
+    {
+      DataSink.PluginRegistry.get_default ().register_plugin (
+        typeof (RecollPlugin),
+        _ ("Recoll"), // plugin title
+        _ ("Returns results of full text search against an existing Recoll index."), // description
+        "recoll", // icon name
+        register_plugin, // reference to this function
+        Environment.find_program_in_path ("recoll") != null, // true if user's system has all required components which the plugin needs
+        _ ("recoll is not installed") // error message
+      );
+    }
+ 
+    static construct
+    {
+      // register the plugin when the class is constructed
+      register_plugin ();
+    }
+ 
+    // an optional method to improve the speed of searches, 
+    // if you return false here, the search method won't be called
+    // for this query
+    public bool handles_query (Query query)
+    {
+      return (QueryFlags.FILES in query.query_type);
+    }
+
+    enum LineType {
+      FIELDS,
+      ABSTRACT_START,
+      ABSTRACT,
+      ABSTRACT_END;
+    }
+    
+    public async ResultSet? search (Query query) throws SearchError
+    {
+      Pid pid;
+      int read_fd, write_fd;
+      string[] argv = {"recoll", 
+                       "-t",   // command line mode
+                       "-n",   // indices of results
+                       "0-20", // return first 20 results
+                       "-a",   // ALL TERMS mode
+                       "-A",   // output abstracts
+                       query.query_string};
+
+      try
+      {
+        Process.spawn_async_with_pipes (null, argv, null,
+                                        SpawnFlags.SEARCH_PATH,
+                                        null, out pid, out write_fd, out read_fd);
+        UnixInputStream read_stream = new UnixInputStream (read_fd, true);
+        DataInputStream recoll_output = new DataInputStream (read_stream);
+
+        // Sample output from `recoll -t` :
+        // ===============================
+        // Recoll query: ((kernel:(wqf=11) OR kernels OR kernelize OR kernelized))
+        // 4725 results (printing  1 max):
+        // text/plain	[file:///home/patrick/code/sample-results.txt]	[sample-results.txt]	8806	bytes	
+        // ABSTRACT
+        // some text summarizing the document usually has the keyword (kernel)
+        // /ABSTRACT
+      
+        string line = null;
+        var next_line_type = LineType.FIELDS;
+        ResultSet results = new ResultSet ();
+        Utils.FileInfo result = null;
+        int line_idx = 0;
+        string description = null;
+        while ((line = yield recoll_output.read_line_async (Priority.DEFAULT)) != null) 
+        {
+          if (line_idx >= 2) // skip first two lines
+          {
+            if (next_line_type == LineType.FIELDS)
+            {
+              string[] fields = line.split("\t");
+              
+              //string mimetype = fields[0];
+              
+              string uri = fields[1];
+              uri = uri.substring(1, uri.length - 2);
+              
+              description = uri.split("://")[1];
+              
+              // FIXME: recoll already gives us the mimetype so FileInfo
+              // is doing extra work obtaining it from the file
+              result = new Utils.FileInfo (uri, typeof (MatchObject));
+              
+              yield result.initialize ();
+              
+              next_line_type = LineType.ABSTRACT_START;
+            }
+            else if (next_line_type == LineType.ABSTRACT_START)
+            {
+              // TODO check for the start of the abstract
+              next_line_type = LineType.ABSTRACT;
+            }
+            else if (next_line_type == LineType.ABSTRACT)
+            {
+              line = line.chug().chomp();
+              if (line != null && line != "")
+              { 
+                description = description + ": " + line;
+              }
+              next_line_type = LineType.ABSTRACT_END;
+            }
+            else if (next_line_type == LineType.ABSTRACT_END)
+            {
+              // TODO check for the end of the abstract
+              
+              // TODO use relevancy rating to set match score
+              
+              // score defaults to just under that of results from the locate plugin
+              result.match_obj.description = description;
+              results.add(result.match_obj, Match.Score.INCREMENT_MINOR * 2);
+              next_line_type = LineType.FIELDS;
+            }
+            else
+            {
+              // TODO handle unexpected output ?
+            }
+          }
+          line_idx++;
+        }
+
+        return results;
+      }
+      catch (Error err)
+      {
+        if (!query.is_cancelled ()) warning ("%s", err.message);
+      }
+ 
+      // make sure this method is called before returning any results
+      query.check_cancellable ();
+      return null;
+    }
+ 
+    private class MatchObject: Object, Match, UriMatch
+    {
+      // from Match interface
+      public string title { get; construct set; }
+      public string description { get; set; }
+      public string icon_name { get; construct set; }
+      public bool has_thumbnail { get; construct set; }
+      public string thumbnail_path { get; construct set; }
+      public MatchType match_type { get; construct set; }
+
+      // from UriMatch interface
+      public string uri { get; set; }
+      public QueryFlags file_type { get; set; }
+      public string mime_type { get; set; }
+
+      public int default_relevancy { get; set; default = 0; }
+
+      public MatchObject ()
+      {
+        Object (match_type: MatchType.GENERIC_URI);
+      }
+    }
+  }
+}

=== modified file 'src/ui/synapse-main.vala'
--- src/ui/synapse-main.vala	2012-03-18 16:03:54 +0000
+++ src/ui/synapse-main.vala	2012-11-10 07:41:20 +0000
@@ -186,6 +186,7 @@
 #if HAVE_LIBREST
         typeof (ImgUrPlugin),
 #endif
+        typeof (RecollPlugin),
         // action-only plugins
         typeof (DevhelpPlugin),
         typeof (OpenSearchPlugin),