lp:~vcs-imports/tracker/+git/tracker

Author: Carlos Garnacho
Author Date: 2016-10-26 18:42:56 UTC

po: update POTFILES.in

A lot of files have gone away into some other repo.

Author: Sam Thursfield
Author Date: 2016-06-23 23:04:47 UTC

libtracker-sparql: Add public API to open a private Tracker store

Author: =?utf-8?q?Timm_B=C3=A4der?=
Author Date: 2016-04-17 10:19:39 UTC

miner-manager: Plug memory leak

https://bugzilla.gnome.org/show_bug.cgi?id=765172

Author: Cédric VALMARY (Tot en òc)
Author Date: 2016-04-18 19:23:35 UTC

Updated Occitan translation

Author: Cédric VALMARY (Tot en òc)
Author Date: 2016-04-18 19:22:58 UTC

Updated Occitan translation

Author: Cédric VALMARY (Tot en òc)
Author Date: 2016-04-18 19:22:21 UTC

Updated Occitan translation

Author: Carlos Garnacho
Author Date: 2016-02-27 17:12:36 UTC

Update to FTS5

Our old stale copy of the FTS3/4 module is now deleted, replaced by
a shinier FTS5 embedded module. If at configure time we detect that
SQLite doesn't offer the FTS5 module, we will load our own, just as
we used to do with FTS4.

FTS5 brings a few differences in the ways it's meant to be extended,
the tokenizer has been updated to cope with the differences. Also,
FTS5 offers no offsets() builtin function, nor matchinfo() which we
used to implement ranking. It offers though ways to implement
additional functions, and builtin rank support which can be tweaked
to achieve the same functional results than we did.

Other than that, the ways to interact with the FTS virtual table
are roughly similar to those in FTS4, insertions and deletions have
been updated to do things the FTS5 way.

Since it's not worth to bump the database format (data is reproducted
from the journal, so we drop some embedded data such as
nie:plainTextContent), the nco:hobby property has been modified to
no longer be fulltext indexed, AFAIK there's no users ever setting/
accessing that, and the FTS properties change will trigger the
regeneration of the FTS view and virtual tables, resulting in a
seamless update to FTS5.

However, we don't leave completely unscathed from the fts3_tokenizer()
change. Since the older FTS3/4 tokenizer is not registered, we can't
just drop the older FTS table. So it is left dangling and never
accessed again, in favor of the newer fts5 table. This is obviously
not a problem when creating the database from scratch.

In the way, a few bugs were found. per-property weights in ranking
were being given in a scrambled way (although stable across database
generations). And deletion of FTS properties (or entire rows) could
result in the tokens not being fully removed from the FTS table,
resulting in confused searches. These are now fixed.

Impact to users of tracker should be none. All the FTS Sparql-to-SQL
translation has been updated to just use FTS5 syntax and tables.

Author: Carlos Garnacho
Author Date: 2015-10-18 21:41:37 UTC

libtracker-miner: Optimize move operations

Instead of creating one delete+insert per file (indirectly) contained
in a folder, do it all at once in a DELETE...INSERT...WHERE. We only
now need to query and block if we're running a thumbnailer service,
which should be most uncommon.

This makes the whole update to happen in a much tighter loop within
tracker-store, eg. helps reduce the time spent in processing the
renaming of a linux kernel tree (51964 elements) from 23s to 12s.
(as measured locally by tracker-miner-fs on a previously idle
system).

Author: Sam Thursfield
Author Date: 2015-08-11 16:19:42 UTC

libtracker-miner: Add ownership tracking to Tracker indexing config

The TrackerIndexingTree class keeps track of each directory or file that
has been explicitly selected for indexing and monitoring.

As a new feature, we want to allow any application to add to the list of
directories being indexed and monitored, and to later be able to remove
things from the list again. Previously, the tracker-miner-fs process was
totally in control of what went in the IndexingTree.

In order to ensure that applications can't interfere with the
existing configuration, we need to track *who* added something to the
indexing tree. This is done with a string identifier.

The tracker_miner_fs_directory_add() functions now take an 'owner'
parameter so they can keep track of who is interested in each configured
indexing root. The tracker_miner_fs_ignore() function lets a given owner
that previously expressed interest in an indexing root now say that they
are not interested. If that indexing root has no 'owners' left, it is
deleted from the TrackerIndexingTree, which cancels any running indexing
tasks and removes any monitors.

We need to make sure that nobody can claim the internal
'tracker-miner-fs' ID through the external API -- that would make it
possible to override the internal configuration.

Author: Sam Thursfield
Author Date: 2015-08-12 16:33:48 UTC

TEMP: Add IndexFileForProcess, and functional tests

Author: Carlos Garnacho
Author Date: 2015-07-13 19:41:23 UTC

cli: Add collect-bug-info subcommand to "tracker" CLI tool

This command basically does everything we usually ask to the users, it:
- queries all info about a given resource or file
- Gets seemingly related journald logs
- Optionally, copy the file or
- Gets gvfs-info and file(1) output
- Gets the lowlevel database schema
- Packs everything on a tar.zx, ready to be attached to bz, shared via
email...

The --inspect-logs/-l switch can be used to check the logs from the last
30 days on the look for tracker errors with an uri/urn we can pull info
from.

Author: Cosimo Cecchi
Author Date: 2015-01-27 16:37:41 UTC

tracker-miner-fs: fix build

Author: Martyn Russell
Author Date: 2015-01-15 15:45:43 UTC

libtracker-miner: Upgrade unit tests for TrackerCrawler

Also create new TrackerDataProvider when creating crawler to avoid test
failures

Author: Sam Thursfield
Author Date: 2014-12-30 21:27:18 UTC

docs: Add an 'overview' UML diagram

Needs to be a bit clearer but the gist of it is there.

Author: Sam Thursfield
Author Date: 2014-12-29 17:46:46 UTC

functional-tests: Reformat code to correspond to consistent style

The standard coding style for Python is PEP-8. I used the 'autopep8'
tool to reformat all the tests to conform to PEP-8 style.

There should be no functional changes.

PEP-8: https://www.python.org/dev/peps/pep-0008/
autopep8: https://pypi.python.org/pypi/autopep8

Author: Sam Thursfield
Author Date: 2014-12-23 16:00:22 UTC

miner-fs: Give clearer warning for missing nfo:fileLastModified

This error should be rare after the previous commit, but it's still
possible.

Author: Philip Van Hoof
Author Date: 2014-12-19 14:18:12 UTC

Use crawler's max_depth param of start in file_system_traverse

Author: Carlos Garnacho
Author Date: 2014-12-07 20:34:46 UTC

tracker-miner-fs: Delete nie:dataSources on update

Decorators should be updating again all relevant information, so ensure
their nie:dataSource is deleted for these to kick in again.

This used to be unnecessary because updated items just went to the
internal decorator queue, without further checks. Now that the decorators
query the database for the next items missing that dataSource, it is
crucial to do this somewhere.

Author: Martyn Russell
Author Date: 2014-10-28 16:29:44 UTC

tracker: Make sure old commands still work with the new system

Author: Philip Van Hoof
Author Date: 2014-09-26 15:19:24 UTC

Remove misleading code comment

Author: Philip Van Hoof
Author Date: 2014-09-26 15:04:18 UTC

Remove even more! debugging codez

Author: Martyn Russell
Author Date: 2014-09-25 14:03:53 UTC

build: Removed remaining libstemmer Makefile generation

Found during git clean -dfx

Author: Martyn Russell
Author Date: 2014-09-04 14:05:11 UTC

tracker-preferences: Fixed deprecation warnings in UI file

Author: Martyn Russell
Author Date: 2014-08-14 11:15:52 UTC

libtracker-fts: Fixed memory leak for property_names variable

Author: Martyn Russell
Author Date: 2014-07-22 11:42:31 UTC

build: Don't use API version 1.2, only the major/minor version should

This avoids broken library and pkgconfig versions

Author: Roberto Guido
Author Date: 2014-06-08 21:40:02 UTC

tracker-miner-rss: properly handling updates from tracker-store

Author: Carlos Garnacho
Author Date: 2014-03-22 22:54:39 UTC

libtracker-miner: Perform leveled notification in TrackerFileNotifier

The current notification process involves crawling over index roots
without restrictions, and querying the state of every file in the
store. This is fastest, but can get memory hungry on huge directory
trees.

So split the process in 3 sequencial steps, that are repeated from
top to bottom over the directory hierarchy:

- A directory is crawled, contents that currently exist in the
  filesystem are extracted.
- Only if the directory is an index root, or was checked to exist
  in the store through previous iterations, the directory and all
  contents found are looked up on the store by their uri, new and
  updated contents are detected by comparing mtimes.
- Only if the directory passed #2, and its mtime changed (which
  usually implies something was added or removed, at this stage we
  only have to care of the latter), query all elements in the store
  that nfo:belongsToContainer to it, and check for those files that
  existed in the store but don't exist anymore. Deleted contents
  are detected in this stage.

The change has been done so there is certain compile-time granularity
on the directory processing, currently controlled through the MAX_DEPTH
define. This switch controls the maximum depth on crawled/queried chunks,
which establishes some indirect limit on the number of GFiles (and all
misc data around) that are in memory at the same time.

From testing, first-time crawling performance is completely unaffected,
and second-time crawling on an unchanged directory tree has negligible
decreases. The IN() match on an indexed property like nie:url looks
near constant, and the third more expensive step will only happen
when it is very likely that there are actual changes to process. So
the MAX_DEPTH value has been set to 1 to keep memory usage to a minimum
(tracker-miner-fs now peaks on massif at 24MB when it previously early
grew to ~180MB, indexing 11304 folders and 123428 files)

Author: Martyn Russell
Author Date: 2014-03-17 13:33:17 UTC

tracker-extract: Remove the need for fork() and sub-processes with PDF extraction

This fuctionality we're removing has been around since
be58d8da6da5a8f16ca83f78a9427ca4de723151 where Philip added code to kill a
forked process after n seconds for really complex PDFs (or PDFs that take
longer to process on embedded devices). This was covered under NB#290406.

Later I improved the code to use select() instead of signal handlers due to
IO blocking in some situations in commit
9c4e166ec101c79bdeac30696371ffe0abd4e046 and as part of GB#680897.

However, the extra complexity is no longer needed as far as I can tell. There
was a good reason for the IO blocking in the past and I believe it has been
fixed by Carlos in one commit or another and with the new GTask and
_run_in_thread() APIs we have now, it's unnecessary to use this approach. IF
anything, we should add some timeout or cancellation in the task issuing
(main?) thread instead of implementing this for each extractor. Currently we
don't do this though.

As an additional note, this may fix GB#726421.

I've tested this patch with files attached to previous bug reports related,
including GB#680897 and GB#685378.

Author: Martyn Russell
Author Date: 2014-03-10 11:13:02 UTC

all: Make GBusType configurable with TRACKER_BUS_TYPE

This works by using G_BUS_TYPE_SESSION if the env var TRACKER_BUS_TYPE is not
set and will allow for "system" to be used as a value. If it is set, the
system bus is used instead.

Author: Philip Van Hoof
Author Date: 2014-03-07 20:18:45 UTC

Fix handling when no match is found by libicu charset detector

Author: Martin Kampas
Author Date: 2014-02-24 16:22:48 UTC

functional-tests: fix invalid SPARQL s/nao:hasTag:prefLabel/nao:prefLabel/

The extractor output has changed. Similar fix can be seen in
commit aabc8a8e07e90b8fae0172824185e50b6af68228.

Author: Martyn Russell
Author Date: 2014-02-18 13:31:27 UTC

tracker-sparql: Add new command line features for help with the ontology

Specifically

  --tree $CLASS (to print the tree for $CLASS, or NULL for all ontology)
  --get-shorthand $CLASS
  --get-longhand $CLASS

Author: Martyn Russell
Author Date: 2014-02-11 19:56:45 UTC

tests: Updated gtester scripts to improve testing system

We imported Makefile.decl from glib when it was using Subversion, a long time
ago. The script had a few issues (like running all unit tests twice on make
distcheck) and needed to be updated.

I've stolen a latest version from GLib's glib.mk and also their m4 macros
which are required to make this all work too.

This gives us the advantage of defining installable and uninstallable test
cases as well as test data, scripts and other useful stuff.

The debug output also looks quite good.
And distcheck passes :)

Author: Carlos Garnacho
Author Date: 2014-02-20 11:14:53 UTC

libtracker-commit: Fix #if vs #ifdef typo

This can be amended with the previous commit.

Author: Carlos Garnacho
Author Date: 2014-01-20 19:17:40 UTC

extract: Plug a leak

If the string being processed is detected as UTF16, the "utf8"
variable would already contain a malloc()'ed string, so the
GString would be leaked altogether.

Then, ensure the string is freed if already converted from UTF16,
and spare the utf8 validation too, since we already know the string
is valid utf8.

Author: Matthias Clasen
Author Date: 2013-02-26 04:51:31 UTC

Avoid a double-free in the png extractor

The software member of the exif data was getting freed twice. So
whenever you deal with a png that has this field set to a non-NULL
value, you get a segfault.
https://bugzilla.gnome.org/show_bug.cgi?id=660965

Author: Carlos Garnacho
Author Date: 2013-02-12 17:59:34 UTC

fts: Obey min-word-length configuration on the tokenizer

Author: Sam Thursfield
Author Date: 2012-06-07 16:36:20 UTC

functional-tests: Add 302-miner-removable-devices test

(Currently contains a hack to cause Tracker to recognise the fake
mounts as removable devices; we can't create fake Volume or Drive
objects in the test mounter yet due to Vala bug #677769)

Author: Jürg Billeter
Author Date: 2012-03-21 10:47:05 UTC

tracker-miner-fs: Fix use after free in extractor_process_failsafe

Fixes NB#299221.

Author: Martyn Russell
Author Date: 2012-02-17 15:38:21 UTC

tracker-miner-evolution: Doesn't build with Evolution 3.3.5

This is based on the patch from Milan Crha <mcrha@redhat.com>

Fixes GB#669646.

Author: Alexandre Franke
Author Date: 2011-12-15 20:14:12 UTC

Update French translation

miner → collecteur
extractor → extracteur

Author: Philip Van Hoof
Author Date: 2011-10-25 09:44:26 UTC

tracker-extract, jpeg: Detect albumart

Author: Jürg Billeter
Author Date: 2011-01-11 17:00:59 UTC

miner-fs: Fix GDBusMessage leak in get_metadata_fast_async

Author: Adrien Bustany
Author Date: 2010-04-01 20:40:36 UTC

Twitter miner: Port to new miner API and fix crasher

Author: Petr Kovar
Author Date: 2010-02-18 18:32:25 UTC

Update Czech translation by Marek Cernocky