Merge lp:~mhr3/zeitgeist/mimetypes into lp:zeitgeist/0.1

[Michal Hruby]

[Siegfried Gevatter (rainct)]

Looks good on a quick glance over the diff. Nice work.

[Mikkel Kamstrup Erlandsen (kamstrup)]

While we are at improving our mime handling situation we might as well go the extra mile - especially now that we want to make it public API.

We now have interpretations like Interpretation.SPREADSHEET/PAGINATED_TEXT_DOCUMENT so we can do a much finer grained mapping of the mimetypes. This in turn works out beautifully with the queries because we now expand queries to also match all child types.

As an example take a look at my mimetype mappings libzeitgeist: http://bazaar.launchpad.net/~libzeitgeist-developers/libzeitgeist/trunk/annotate/head:/src/zeitgeist-mimetypes.c#L239

Anoter note: The public API should be added to our generated sphinx docs too...

review: Needs Information (visual review)

[Markus Korn (thekorn)]

I did not do a review yet, but I've one first comment: I don't like that we have the same logic in two places, in zeitgeist and in libzeitgeist. (and in the future also in C# and JS, ...)

[Mikkel Kamstrup Erlandsen (kamstrup)]

Perhaps we can compile the mimetype->interpretation map from some XML or JSON schemas, like we do for the ontologies, and have that be the authoritative place? For libzeitgeist I compile the ontology C-bindings from the Zeitgeist's Python module so there is no code dupe there. I could easily do the same for the mimetypes, but that requires a bit more structure in the mimetype spec than what is in Michal's proposal.

However doing all of this probably requires more work than I think we should put in for 0.3.4 at this point. I think the best route would be to either:

 1) Defer this task for post 0.3.4, or
 2) Make sure the implementation is done in a way so that we can add the extra introspection API for post 0.3.4 without breaking the mimetypes API from 0.3.4

I am not entirely sure which route I prefer, but I would hate to delay 0.3.4 more than we already have...

[Seif Lotfy (seif)]

Ok I am up for a mimetype-interpretation map:
I would prefer JSON for this issue...
what do u guys think?

[Mikkel Kamstrup Erlandsen (kamstrup)]

Why JSON and not straight Python? JSON would only slow down startup time, and Python already supports a very JSON-like syntax for declaring maps...

[Seif Lotfy (seif)]

Beacuse I think we want libzeitgeist and other languages to be able to
parse it... Or do u want to parse python syntax ?

On Mon, Jul 19, 2010 at 10:07 AM, Mikkel Kamstrup Erlandsen
<email address hidden> wrote:
> Why JSON and not straight Python? JSON would only slow down startup time, and Python already supports a very JSON-like syntax for declaring maps...
> --
> https://code.launchpad.net/~mhr3/zeitgeist/mimetypes/+merge/26233
> You are subscribed to branch lp:zeitgeist.
>

--
This is me doing some advertisement for my blog http://seilo.geekyogre.com

[Markus Korn (thekorn)]

This is not a reviewer comment, but I would like to add another possible way to solve this, python applications which would like to use this kind of mapper, could depend on libzeitgeist and use ctypes to access the mapper functions:

from ctypes import CDLL, c_char_p

libzeitgeist = CDLL("libzeitgeist-1.0.so.0")

interpretation_for_mimetype = libzeitgeist.zeitgeist_interpretation_for_mimetype
interpretation_for_mimetype.argtypes = [c_char_p,]
interpretation_for_mimetype.restype = c_char_p

print interpretation_for_mimetype("text/html")
print interpretation_for_mimetype("image/png")

manifestation_for_uri = libzeitgeist.zeitgeist_manifestation_for_uri
manifestation_for_uri.argtypes = [c_char_p,]
manifestation_for_uri.restype = c_char_p

print manifestation_for_uri("file:///home")
print manifestation_for_uri("http://www.google.de")

[Siegfried Gevatter (rainct)]

2010/7/19 Markus Korn <email address hidden>:
> This is not a reviewer comment, but I would like to add another possible way to solve this, python applications which would like to use this kind of mapper, could depend on libzeitgeist and use ctypes to access the mapper functions:

Eerk, making a real C->Python interface is not so difficult.

--
Siegfried-Angel Gevatter Pujals (RainCT)
Free Software Developer       363DEAE3

[Michal Hruby (mhr3)]

I think the cross-dependency of zg->libzg->zg is not a good idea, +1 for making it like the ontologies.

[Mikkel Kamstrup Erlandsen (kamstrup)]

Markus, RainCT: Depending on libzeitgeist would add ~1mb to the Zeitgeist runtime I guess - which I think is a very steep price to pay for this.

As I noted in comment from 2010-05-28 I'd actually prefer a Python module for libzeitgeist, so I just write a small tool for libzeitgeist that outputs the C code at build time. Very easy for me, and fast and light for ZG.

Alternatively go full monty and write the autofoo magic that compiles some JSON/XML schema into Python code like we do for the ontology. In the end giving me a Python module to use in the libzeitgeist build process

[Seif Lotfy (seif)]

+1 for the alternative

On Mon, Jul 19, 2010 at 2:51 PM, Mikkel Kamstrup Erlandsen
<email address hidden> wrote:
> Markus, RainCT: Depending on libzeitgeist would add ~1mb to the Zeitgeist runtime I guess - which I think is a very steep price to pay for this.
>
> As I noted in comment from 2010-05-28 I'd actually prefer a Python module for libzeitgeist, so I just write a small tool for libzeitgeist that outputs the C code at build time. Very easy for me, and fast and light for ZG.
>
> Alternatively go full monty and write the autofoo magic that compiles some JSON/XML schema into Python code like we do for the ontology. In the end giving me a Python module to use in the libzeitgeist build process
> --
> https://code.launchpad.net/~mhr3/zeitgeist/mimetypes/+merge/26233
> You are subscribed to branch lp:zeitgeist.
>

--
This is me doing some advertisement for my blog http://seilo.geekyogre.com

[Siegfried Gevatter (rainct)]

Let's go with Python code, there's no reason to have it as JSON.

[Seif Lotfy (seif)]

Markus finished a module for that

Unmerged revisions

1460. By Michal Hruby on 2010-05-10

Interpretation.MUSIC -> Interpretation.AUDIO

1459. By Michal Hruby on 2010-05-07

Recent datasource: Use the zg.mimetypes module

1458. By Michal Hruby on 2010-05-07

Added mimetype to interpretation getter

=== modified file '_zeitgeist/loggers/datasources/recent.py'
--- _zeitgeist/loggers/datasources/recent.py	2010-04-29 11:33:01 +0000
+++ _zeitgeist/loggers/datasources/recent.py	2010-05-27 19:34:36 +0000
@@ -24,8 +24,6 @@
 
 from __future__ import with_statement
 import os
-import re
-import fnmatch
 import urllib
 import time
 import logging
@@ -34,6 +32,7 @@
 from zeitgeist import _config
 from zeitgeist.datamodel import Event, Subject, Interpretation, Manifestation, \
 	DataSource, get_timestamp_for_now
+from zeitgeist.mimetypes import get_interpretation_for_mimetype
 from _zeitgeist.loggers.zeitgeist_base import DataProvider
 
 log = logging.getLogger("zeitgeist.logger.datasources.recent")
@@ -51,166 +50,9 @@
 else:
 	enabled = True
 
-class SimpleMatch(object):
-	""" Wrapper around fnmatch.fnmatch which allows to define mimetype
-	patterns by using shell-style wildcards.
-	"""
-
-	def __init__(self, pattern):
-		self.__pattern = pattern
-
-	def match(self, text):
-		return fnmatch.fnmatch(text, self.__pattern)
-
-	def __repr__(self):
-		return "%s(%r)" %(self.__class__.__name__, self.__pattern)
-
-DOCUMENT_MIMETYPES = [
-		# Covers:
-		#	 vnd.corel-draw
-		#	 vnd.ms-powerpoint
-		#	 vnd.ms-excel
-		#	 vnd.oasis.opendocument.*
-		#	 vnd.stardivision.*
-		#	 vnd.sun.xml.*
-		SimpleMatch(u"application/vnd.*"),
-		# Covers: x-applix-word, x-applix-spreadsheet, x-applix-presents
-		SimpleMatch(u"application/x-applix-*"),
-		# Covers: x-kword, x-kspread, x-kpresenter, x-killustrator
-		re.compile(u"application/x-k(word|spread|presenter|illustrator)"),
-		u"application/ms-powerpoint",
-		u"application/msword",
-		u"application/pdf",
-		u"application/postscript",
-		u"application/ps",
-		u"application/rtf",
-		u"application/x-abiword",
-		u"application/x-gnucash",
-		u"application/x-gnumeric",
-		SimpleMatch(u"application/x-java*"),
-		SimpleMatch(u"*/x-tex"),
-		SimpleMatch(u"*/x-latex"),
-		SimpleMatch(u"*/x-dvi"),
-		u"text/plain"
-]
-
-IMAGE_MIMETYPES = [
-		# Covers:
-		#	 vnd.corel-draw
-		u"application/vnd.corel-draw",
-		# Covers: x-kword, x-kspread, x-kpresenter, x-killustrator
-		re.compile(u"application/x-k(word|spread|presenter|illustrator)"),
-		SimpleMatch(u"image/*"),
-]
-
-AUDIO_MIMETYPES = [
-		SimpleMatch(u"audio/*"),
-		u"application/ogg"
-]
-
-VIDEO_MIMETYPES = [
-		SimpleMatch(u"video/*"),
-		u"application/ogg"
-]
-
-DEVELOPMENT_MIMETYPES = [
-		u"application/ecmascript",
-		u"application/javascript",
-		u"application/x-csh",
-		u"application/x-designer",
-		u"application/x-desktop",
-		u"application/x-dia-diagram",
-		u"application/x-fluid",
-		u"application/x-glade",
-		u"application/xhtml+xml",
-		u"application/x-java-archive",
-		u"application/x-m4",
-		u"application/xml",
-		u"application/x-object",
-		u"application/x-perl",
-		u"application/x-php",
-		u"application/x-ruby",
-		u"application/x-shellscript",
-		u"application/x-sql",
-		u"text/css",
-		u"text/html",
-		u"text/x-c",
-		u"text/x-c++",
-		u"text/x-chdr",
-		u"text/x-copying",
-		u"text/x-credits",
-		u"text/x-csharp",
-		u"text/x-c++src",
-		u"text/x-csrc",
-		u"text/x-dsrc",
-		u"text/x-eiffel",
-		u"text/x-gettext-translation",
-		u"text/x-gettext-translation-template",
-		u"text/x-haskell",
-		u"text/x-idl",
-		u"text/x-java",
-		u"text/x-lisp",
-		u"text/x-lua",
-		u"text/x-makefile",
-		u"text/x-objcsrc",
-		u"text/x-ocaml",
-		u"text/x-pascal",
-		u"text/x-patch",
-		u"text/x-python",
-		u"text/x-sql",
-		u"text/x-tcl",
-		u"text/x-troff",
-		u"text/x-vala",
-		u"text/x-vhdl",
-]
-
-ALL_MIMETYPES = DOCUMENT_MIMETYPES + IMAGE_MIMETYPES + AUDIO_MIMETYPES + \
-				VIDEO_MIMETYPES + DEVELOPMENT_MIMETYPES
-
-class MimeTypeSet(set):
-	""" Set which allows to match against a string or an object with a
-	match() method.
-	"""
-
-	def __init__(self, *items):
-		super(MimeTypeSet, self).__init__()
-		self.__pattern = set()
-		for item in items:
-			if isinstance(item, (str, unicode)):
-				self.add(item)
-			elif hasattr(item, "match"):
-				self.__pattern.add(item)
-			else:
-				raise ValueError("Bad mimetype '%s'" %item)
-
-	def __contains__(self, mimetype):
-		result = super(MimeTypeSet, self).__contains__(mimetype)
-		if not result:
-			for pattern in self.__pattern:
-				if pattern.match(mimetype):
-					return True
-		return result
-		
-	def __len__(self):
-		return super(MimeTypeSet, self).__len__() + len(self.__pattern)
-
-	def __repr__(self):
-		items = ", ".join(sorted(map(repr, self | self.__pattern)))
-		return "%s(%s)" %(self.__class__.__name__, items)
-
 
 class RecentlyUsedManagerGtk(DataProvider):
 	
-	FILTERS = {
-		# dict of name as key and the matching mimetypes as value
-		# if the value is None this filter matches all mimetypes
-		"DOCUMENT": MimeTypeSet(*DOCUMENT_MIMETYPES),
-		"IMAGE": MimeTypeSet(*IMAGE_MIMETYPES),
-		"MUSIC": MimeTypeSet(*AUDIO_MIMETYPES),
-		"VIDEO": MimeTypeSet(*VIDEO_MIMETYPES),
-		"SOURCE_CODE": MimeTypeSet(*DEVELOPMENT_MIMETYPES),
-	}
-	
 	def __init__(self, client):
 		DataProvider.__init__(self,
 			unique_id="com.zeitgeist-project,datahub,recent",
@@ -269,16 +111,6 @@
 						pass # file may be a broken symlink (LP: #523761)
 		return None
 	
-	def _get_interpretation_for_mimetype(self, mimetype):
-		matching_filter = None
-		for filter_name, mimetypes in self.FILTERS.iteritems():
-			if mimetype and mimetype in mimetypes:
-				matching_filter = filter_name
-				break
-		if matching_filter:
-			return getattr(Interpretation, matching_filter).uri
-		return ""
-	
 	def _get_items(self):
 		# We save the start timestamp to avoid race conditions
 		last_seen = get_timestamp_for_now()
@@ -297,7 +129,7 @@
 				
 				subject = Subject.new_for_values(
 					uri = unicode(uri),
-					interpretation = self._get_interpretation_for_mimetype(
+					interpretation = get_interpretation_for_mimetype(
 						unicode(info.get_mime_type())),
 					manifestation = Manifestation.FILE_DATA_OBJECT.uri,
 					text = info.get_display_name(),

=== modified file 'test/engine-test.py'
--- test/engine-test.py	2010-05-15 13:15:44 +0000
+++ test/engine-test.py	2010-05-27 19:34:36 +0000
@@ -10,6 +10,7 @@
 from _zeitgeist.engine import constants
 from _zeitgeist.engine import get_engine
 from zeitgeist.datamodel import *
+from zeitgeist.mimetypes import *
 from testutils import import_events
 
 import unittest
@@ -366,7 +367,30 @@
 		self.assertEquals(2, len(result))
 		events = self.engine.get_events(result)
 		
-	
+	def testMimetypeToInterpretation(self):
+		mimetype = "text/plain"
+		interpretation = get_interpretation_for_mimetype(mimetype)
+
+		event = Event()
+		event.interpretation = Manifestation.USER_ACTIVITY
+		event.manifestation = Interpretation.CREATE_EVENT
+		event.actor = "/usr/share/applications/gnome-about.desktop"
+		
+		subject = Subject()
+		subject.uri = "file:///tmp/file.txt"
+		subject.manifestation = Manifestation.FILE_DATA_OBJECT
+		subject.interpretation = interpretation
+		subject.origin = "test://"
+		subject.mimetype = "text/plain"
+		subject.text = "This subject has no text"
+		
+		event.append_subject(subject)
+		ids = self.engine.insert_events([event,])
+
+		results = self.engine.get_events([1])
+		self.assertEquals(1, len(results))
+		self.assertEquals(
+			interpretation, results[0].subjects[0].interpretation)
 	
 	def testDontFindState(self):
 		# searchin by storage state is currently not implemented

=== added file 'test/test-mimetypes.py'
--- test/test-mimetypes.py	1970-01-01 00:00:00 +0000
+++ test/test-mimetypes.py	2010-05-27 19:34:36 +0000
@@ -0,0 +1,25 @@
+#! /usr/bin/python
+
+# Update python path to use local zeitgeist module
+import sys
+import os
+
+import zeitgeist.mimetypes
+
+# Test the mimetype -> Interpretation conversion
+mimetype = "image/png"
+i = zeitgeist.mimetypes.get_interpretation_for_mimetype(mimetype)
+print "Interpretation for %s: %s" % (mimetype, i)
+
+mimetype = "text/plain"
+i = zeitgeist.mimetypes.get_interpretation_for_mimetype(mimetype)
+print "Interpretation for %s: %s" % (mimetype, i)
+
+mimetype = "application/ogg"
+i = zeitgeist.mimetypes.get_interpretation_for_mimetype(mimetype)
+print "Interpretation for %s: %s" % (mimetype, i)
+
+mimetype = "unknown/invalid"
+i = zeitgeist.mimetypes.get_interpretation_for_mimetype(mimetype)
+print "Interpretation for %s: %s" % (mimetype, i)
+

=== modified file 'zeitgeist/Makefile.am'
--- zeitgeist/Makefile.am	2009-11-27 20:32:54 +0000
+++ zeitgeist/Makefile.am	2010-05-27 19:34:36 +0000
@@ -3,7 +3,8 @@
 app_PYTHON = \
 	__init__.py \
 	datamodel.py \
-	client.py
+	client.py \
+	mimetypes.py
 
 nodist_app_PYTHON = _config.py
 

=== added file 'zeitgeist/mimetypes.py'
--- zeitgeist/mimetypes.py	1970-01-01 00:00:00 +0000
+++ zeitgeist/mimetypes.py	2010-05-27 19:34:36 +0000
@@ -0,0 +1,194 @@
+# -.- coding: utf-8 -.-
+
+# Zeitgeist
+#
+# Copyright © 2009-2010 Siegfried-Angel Gevatter Pujals <[e-mail]>
+# Copyright © 2009 Mikkel Kamstrup Erlandsen <[e-mail]>
+# Copyright © 2009 Markus Korn <[e-mail]>
+# Copyright © 2010 Michal Hruby <[e-mail]>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+import fnmatch
+
+from zeitgeist.datamodel import Interpretation
+
+class SimpleMatch(object):
+    """ Wrapper around fnmatch.fnmatch which allows to define mimetype
+    patterns by using shell-style wildcards.
+    """
+
+    def __init__(self, pattern):
+        self.__pattern = pattern
+
+    def match(self, text):
+        return fnmatch.fnmatch(text, self.__pattern)
+
+    def __repr__(self):
+        return "%s(%r)" %(self.__class__.__name__, self.__pattern)
+
+class MimeTypeSet(set):
+    """ Set which allows to match against a string or an object with a
+    match() method.
+    """
+
+    def __init__(self, *items):
+        super(MimeTypeSet, self).__init__()
+        self.__pattern = set()
+        for item in items:
+            if isinstance(item, (str, unicode)):
+                self.add(item)
+            elif hasattr(item, "match"):
+                self.__pattern.add(item)
+            else:
+                raise ValueError("Bad mimetype '%s'" %item)
+
+    def __contains__(self, mimetype):
+        result = super(MimeTypeSet, self).__contains__(mimetype)
+        if not result:
+            for pattern in self.__pattern:
+                if pattern.match(mimetype):
+                    return True
+        return result
+
+    def __len__(self):
+        return super(MimeTypeSet, self).__len__() + len(self.__pattern)
+
+    def __repr__(self):
+        items = ", ".join(sorted(map(repr, self | self.__pattern)))
+        return "%s(%s)" %(self.__class__.__name__, items)
+
+DOCUMENT_MIMETYPES = [
+        # Covers:
+        #    vnd.corel-draw
+        #    vnd.ms-powerpoint
+        #    vnd.ms-excel
+        #    vnd.oasis.opendocument.*
+        #    vnd.stardivision.*
+        #    vnd.sun.xml.*
+        SimpleMatch(u"application/vnd.*"),
+        # Covers: x-applix-word, x-applix-spreadsheet, x-applix-presents
+        SimpleMatch(u"application/x-applix-*"),
+        # Covers: x-kword, x-kspread, x-kpresenter, x-killustrator
+        re.compile(u"application/x-k(word|spread|presenter|illustrator)"),
+        u"application/ms-powerpoint",
+        u"application/msword",
+        u"application/pdf",
+        u"application/postscript",
+        u"application/ps",
+        u"application/rtf",
+        u"application/x-abiword",
+        u"application/x-gnucash",
+        u"application/x-gnumeric",
+        SimpleMatch(u"application/x-java*"),
+        SimpleMatch(u"*/x-tex"),
+        SimpleMatch(u"*/x-latex"),
+        SimpleMatch(u"*/x-dvi"),
+        u"text/plain"
+]
+
+IMAGE_MIMETYPES = [
+        # Covers:
+        #    vnd.corel-draw
+        u"application/vnd.corel-draw",
+        # Covers: x-kword, x-kspread, x-kpresenter, x-killustrator
+        re.compile(u"application/x-k(word|spread|presenter|illustrator)"),
+        SimpleMatch(u"image/*"),
+]
+
+AUDIO_MIMETYPES = [
+        SimpleMatch(u"audio/*"),
+        u"application/ogg"
+]
+
+VIDEO_MIMETYPES = [
+        SimpleMatch(u"video/*"),
+        u"application/ogg"
+]
+
+DEVELOPMENT_MIMETYPES = [
+        u"application/ecmascript",
+        u"application/javascript",
+        u"application/x-csh",
+        u"application/x-designer",
+        u"application/x-desktop",
+        u"application/x-dia-diagram",
+        u"application/x-fluid",
+        u"application/x-glade",
+        u"application/xhtml+xml",
+        u"application/x-java-archive",
+        u"application/x-m4",
+        u"application/xml",
+        u"application/x-object",
+        u"application/x-perl",
+        u"application/x-php",
+        u"application/x-ruby",
+        u"application/x-shellscript",
+        u"application/x-sql",
+        u"text/css",
+        u"text/html",
+        u"text/x-c",
+        u"text/x-c++",
+        u"text/x-chdr",
+        u"text/x-copying",
+        u"text/x-credits",
+        u"text/x-csharp",
+        u"text/x-c++src",
+        u"text/x-csrc",
+        u"text/x-dsrc",
+        u"text/x-eiffel",
+        u"text/x-gettext-translation",
+        u"text/x-gettext-translation-template",
+        u"text/x-haskell",
+        u"text/x-idl",
+        u"text/x-java",
+        u"text/x-lisp",
+        u"text/x-lua",
+        u"text/x-makefile",
+        u"text/x-objcsrc",
+        u"text/x-ocaml",
+        u"text/x-pascal",
+        u"text/x-patch",
+        u"text/x-python",
+        u"text/x-sql",
+        u"text/x-tcl",
+        u"text/x-troff",
+        u"text/x-vala",
+        u"text/x-vhdl",
+]
+
+ALL_MIMETYPES = DOCUMENT_MIMETYPES + IMAGE_MIMETYPES + AUDIO_MIMETYPES + \
+                VIDEO_MIMETYPES + DEVELOPMENT_MIMETYPES
+
+FILTERS = {
+        # dict of name as key and the matching mimetypes as value
+        # if the value is None this filter matches all mimetypes
+        "DOCUMENT": MimeTypeSet(*DOCUMENT_MIMETYPES),
+        "IMAGE": MimeTypeSet(*IMAGE_MIMETYPES),
+        "AUDIO": MimeTypeSet(*AUDIO_MIMETYPES),
+        "VIDEO": MimeTypeSet(*VIDEO_MIMETYPES),
+        "SOURCE_CODE": MimeTypeSet(*DEVELOPMENT_MIMETYPES),
+}
+
+def get_interpretation_for_mimetype(mimetype):
+    matching_filter = None
+    for filter_name, mimetypes in FILTERS.iteritems():
+        if mimetype and mimetype in mimetypes:
+            matching_filter = filter_name
+            break
+    if matching_filter:
+        return getattr(Interpretation, matching_filter).uri
+    return ""
+