Merge lp:~zeitgeist/zeitgeist/bluebird-sig-crash into lp:~zeitgeist/zeitgeist/bluebird

[Siegfried Gevatter]

So, how will we do this?

(Consider this merge proposal a cheap blueprint.)

[lp:~zeitgeist/zeitgeist/bluebird-sig-crash]

226. By Siegfried Gevatter on 2011-09-05

Blacklist: implement D-Bus methods.

227. By Michal Hruby on 2011-09-05

Pass Engine instance to extensions

228. By Michal Hruby on 2011-09-05

Add Histogram extension

229. By Siegfried Gevatter on 2011-09-05

GetEvents: make sure event_ids[] isn't empty

230. By Michal Hruby on 2011-09-05

Fix more warnings

231. By Michal Hruby on 2011-09-05

Prettify make output

232. By Michal Hruby on 2011-09-05

Flow-analyse this Vala\!

233. By Michal Hruby on 2011-09-05

Remove debug on event insertion

234. By Seif Lotfy on 2011-09-05

added name_changed logic

235. By Seif Lotfy on 2011-09-07

set print for debugging the signal issue

236. By Siegfried Gevatter on 2011-09-07

Fix signal test cases so extensions are detected correctly.

237. By Seif Lotfy on 2011-09-08

Added destinction rules based on the old ZG

238. By Michal Hruby on 2011-09-12

Add a little bit of profiling

239. By Michal Hruby on 2011-09-14

Merge lp:~zeitgeist/zeitgeist/some-fixes branch with more fixes

240. By Michal Hruby on 2011-09-14

Fix up some issues with DSR

241. By Michal Hruby on 2011-09-15

Use find_custom instead of index, this will actually work with strings

242. By Michal Hruby on 2011-09-15

Hook up version info to config.h

243. By Seif Lotfy on 2011-09-15

remove fixed FIXMEs

244. By Seif Lotfy on 2011-09-16

replaced some asaasay with Utils.SIG_EVENT

245. By Seif Lotfy on 2011-09-16

removed unecessary fixme

246. By Seif Lotfy on 2011-09-16

removed unecessary fixme since the timestamp sanity is cecked upon converting the the event from variant to an Event object

247. By Michal Hruby on 2011-09-16

Fix payload marshalling (hopefully)

248. By Seif Lotfy on 2011-09-16

implement payload insertion
implement payload retrieval

249. By Seif Lotfy on 2011-09-16

fix payload insertion by referfing to the dat in the byte array
added FIXME to check if extension is disabled

250. By Seif Lotfy on 2011-09-16

merged with trunk

251. By Siegfried Gevatter on 2011-09-17

Make some code nicer.

252. By Michal Hruby on 2011-09-17

The nicer the better

253. By Siegfried Gevatter on 2011-09-17

Update extension base class docstring.

254. By Seif Lotfy on 2011-09-17

fixed critical error where timerange = null and we try to look for the start and end, this was due to the intersection returning null

255. By Seif Lotfy on 2011-09-17

removed redundant fixme -> this will be fixed in update_callback

256. By Siegfried Gevatter on 2011-09-17

Update TableLookup caches upon event deletion.

257. By Seif Lotfy on 2011-09-17

removed peers and made a hashtable with generic array

258. By Seif Lotfy on 2011-09-17

fix remove issue in notify.vala and replaced it with remove_index

259. By Seif Lotfy on 2011-09-17

move connections.lookup (peer) outside the loop

260. By Seif Lotfy on 2011-09-17

rename peerse to paths and use remove_index_fast instead of remove_index

261. By Seif Lotfy on 2011-09-17

added missing ;

262. By Seif Lotfy on 2011-09-18

prohibit the datamodel from assiging the current_uri if not set

263. By Siegfried Gevatter on 2011-09-18

testFindTwoOfThreeEvents: move event declarations to a new three_events.js

264. By Siegfried Gevatter on 2011-09-18

engine.vala: add comment about subject templates being AND'ed

265. By Siegfried Gevatter on 2011-09-18

Add test cases for events with multiple subjects and FindEvents (re DISTINCT)

266. By Siegfried Gevatter on 2011-09-18

Add FIXME about --replace

267. By Siegfried Gevatter on 2011-09-18

exit -> Posix.exit

268. By Seif Lotfy on 2011-09-18

added new extensions_conf table for extensions to store their data into

269. By Siegfried Gevatter on 2011-09-18

Handle SIGINT properly.

270. By Seif Lotfy on 2011-09-18

remove faulty fixme notice

271. By Siegfried Gevatter on 2011-09-19

Avoid using event_view's subqueries in WHERE clauses.

272. By Siegfried Gevatter on 2011-09-21

Add actor_uri field to event_view.

273. By Siegfried Gevatter on 2011-09-24

Fix segmentation fault removing a blacklist that doesn't exist.

274. By Siegfried Gevatter on 2011-09-25

datamodel: style fixes

275. By Siegfried Gevatter on 2011-09-25

Add API enabling extensions to store their state in the database. Use it
for blacklist.

276. By Siegfried Gevatter on 2011-09-25

 - Remove Extension.get_name method, use get_type().name() instead.
 - Document the store_config and retrieve_config functions.

277. By Siegfried Gevatter on 2011-09-28

 * DataSourceRegistry:
    - implement saving to (and restoring from) database.
    - fix use of dirty flag (backported to Zeitgeist) and some cleanup.
    - register_data_source(): require sender!=null
 * Engine:
    - Fix shutdown order (unload extensions, close DB, destroy engine).

278. By Siegfried Gevatter on 2011-09-29

Start implementing storage monitor extension.

279. By Seif Lotfy on 2011-09-30

add missing ';'

280. By Seif Lotfy on 2011-10-09

added temp fts solution
NEEDS FIXING

281. By Seif Lotfy on 2011-10-09

some more fixes

282. By Siegfried Gevatter on 2011-10-09

Add missing symlink and stuff.

283. By Michal Hruby on 2011-10-09

Workaround a Vala bug

284. By Michal Hruby on 2011-10-09

Fix the unique name

285. By Michal Hruby on 2011-10-09

And also fix the interface name

286. By Seif Lotfy on 2011-10-09

added print for search

287. By Michal Hruby on 2011-10-09

Fix the marshalling mess

288. By Michal Hruby on 2011-10-10

Catch ExtensionStore error

289. By Michal Hruby on 2011-10-10

A few small fixes

290. By Michal Hruby on 2011-10-10

Merge in the standalone fts indexer

291. By Michal Hruby on 2011-10-10

Damned typo

292. By Michal Hruby on 2011-10-10

Let fts write into bluebird directory

293. By Michal Hruby on 2011-10-10

Added missing file

294. By Seif Lotfy on 2011-10-11

use children of interpretations while generating queries

295. By Michal Hruby on 2011-10-11

Don't overuse Variants if we can just use a struct

296. By Seif Lotfy on 2011-10-11

fixed mutliple symbol negation with children

297. By Michal Hruby on 2011-10-11

Finish up command line options

298. By Michal Hruby on 2011-10-11

Throw exception all the way up to Engine

299. By Michal Hruby on 2011-10-12

Get rid of useless warning

300. By Michal Hruby on 2011-10-12

Let's require latest Vala

301. By Michal Hruby on 2011-10-12

Fix FTS handling

302. By Michal Hruby on 2011-10-12

Added copyright

303. By Siegfried Gevatter on 2011-10-12

Change copyright line to the format used everywhere else.

304. By Michal Hruby on 2011-10-12

Add one more test

305. By Michal Hruby on 2011-10-12

Speed up marshalling! Now 3% faster!

306. By Michal Hruby on 2011-10-12

Little fixes

307. By Seif Lotfy on 2011-10-13

fix double negation problem

308. By Michal Hruby on 2011-10-13

Make the bus name owning more robust

309. By Siegfried Gevatter on 2011-10-13

Change error acquiring name to descriptive string like in Python.

310. By Siegfried Gevatter on 2011-10-13

Fix test case.

311. By Michal Hruby on 2011-10-13

Merge from my branch

312. By Seif Lotfy on 2011-10-13

reverted changes from r237

313. By Michal Hruby on 2011-10-13

Make FTS quit when ZG disappears

314. By Seif Lotfy on 2011-10-14

added WAL pragma

315. By Siegfried Gevatter on 2011-10-14

Remove old comment, move PRAGMA to the right place.

316. By Michal Hruby on 2011-10-15

Fix distcheck

317. By Michal Hruby on 2011-10-17

A couple of fixes

318. By Michal Hruby on 2011-10-19

Don't create a copy when using get_value

319. By Siegfried Gevatter on 2011-10-20

Remove the post_get_events in prevision of possible direct-access.

320. By Michal Hruby on 2011-10-20

Merge lp:~mhr3/zeitgeist/bb-schema-ver-table/

321. By Michal Hruby on 2011-10-20

Handle null events in event array properly

322. By Manish Sinha (मनीष सिन्हा) on 2011-10-25

Added enable for AM_MAINTAINER_MODE

Our build system contains AM_MAINTAINER_MODE without enable. Changed it to
AM_MAINTAINER_MODE([enable])
Please see these links for better explanation
* https://mail.gnome.org/archives/desktop-devel-list/2011-September/msg00064.html
* http://blogs.gnome.org/desrt/2011/09/08/am_maintainer_mode-is-not-cool/

323. By Siegfried Gevatter on 2011-10-29

Merge Seif's lp:~seif/zeitgeist/bb-extract-methods:
  Extract event and subj generation out of get_events into
  get_event_from_row and get_subject_from_row.

324. By Siegfried Gevatter on 2011-10-29

Add Zeitgeist Python module.

325. By Siegfried Gevatter on 2011-10-29

Include Python tests in tarball. Change version number to include ~alpha.

326. By Siegfried Gevatter on 2011-10-29

And forgot a file.

327. By Michal Hruby on 2011-10-31

Add config.vapi to EXTRADIST

328. By Siegfried Gevatter on 2011-10-31

Include a copy of the LGPL, don't install everything into usr/doc,
rename test to test-dbus so it doesn't run by default.

329. By Siegfried Gevatter on 2011-10-31

It's cold and the birds fly away. But the Zeitgeist won't change.

330. By Siegfried Gevatter on 2011-11-01

Binary: zeitgeist -> zeitgeist-daemon

331. By Siegfried Gevatter on 2011-11-01

Forgot one change.

332. By Siegfried Gevatter on 2011-11-01

Add manpage.

333. By Siegfried Gevatter on 2011-11-01

autohell should burn in fire

334. By Siegfried Gevatter on 2011-11-01

Add TODO.

335. By Michal Hruby on 2011-11-01

Dont mark some fts scripts as executable

336. By Siegfried Gevatter on 2011-11-01

More TODO.

337. By Seif Lotfy on 2011-11-02

add missing import of NULL_EVENT in fts.py

338. By Michal Hruby on 2011-11-03

Don't create both gzip and bz2

339. By Siegfried Gevatter on 2011-12-04

Merge Stefano Candori's branch implementing network status
monitoring in storage-monitor.

340. By Siegfried Gevatter on 2011-12-04

Cleanup some stuff.

341. By Siegfried Gevatter on 2011-12-04

Add "+" as prefix operator for exact matches (not expanding
symbols to include their childs).

342. By Stefano Candori on 2011-12-07

[Histogram extension] Fix timezone's hell.

This commit fixes the timezone problem in the histogram extension and adds
the test for checking this.

343. By Stefano Candori on 2011-12-08

Updated TODO list.

344. By Seif Lotfy on 2011-12-09

Merged Stefano's work on fixing the history-extension

345. By Michal Hruby on 2011-12-11

Use statically allocated StringChunk to store the manifestations/interpretations/mimetypes/actors

346. By Michal Hruby on 2011-12-13

Fix one minor unnecessary allocation

347. By Michal Hruby on 2011-12-28

Fix warning about delegate copying

348. By Siegfried Gevatter on 2011-12-29

Add interpretation_for_mimetype and manifestation_for_uri functions,
ported from libzeitgeist.

This is needed to implement https://bugs.launchpad.net/zeitgeist/+bug/899602

349. By Siegfried Gevatter on 2011-12-29

Use compact classes instead of structs, return unowned strings and
sort mimetypes alphabetically.

350. By Siegfried Gevatter on 2011-12-29

Add test case to reproduce "Inserting event with two subjects with the same
URI breaks the following event" (LP: #909708).

351. By Siegfried Gevatter on 2011-12-29

 - Automatically determine interpretation from mime-type when it's empty
 - Automatically determine manifestation from URI when it's empty

352. By Siegfried Gevatter on 2011-12-29

Updated to shared-desktop-ontologies version 0.8.1. This introduces
WebDataObject and makes MusicPiece a child of Audio.

[Michal Hruby (mhr3)]

Let's change the warn_if_fail to the assert_sig and merge this

review: Needs Fixing

[lp:~zeitgeist/zeitgeist/bluebird-sig-crash]

353. By Siegfried Gevatter on 2011-12-31

Merge Seif's lp:~zeitgeist/zeitgeist/benchmark-tools branch, which adds an
extension to get timing information from FindEventsId and some plotting
tools.

354. By Siegfried Gevatter on 2011-12-31

Some test cases cleanup:
 - Rename marshalling to marshalling-test and add copyright header.
 - Make all tests exit with 0 on success.
 - Use :memory: database in query-operators-test (it's not used, but
   it's connected to).

355. By Siegfried Gevatter on 2011-12-31

benchmark-tools: fix symlink and usage example.

356. By Michal Hruby on 2011-12-31

Move stuff around

357. By Michal Hruby on 2011-12-31

Add a define to explain query plans

358. By Michal Hruby on 2011-12-31

Merge lp:~zeitgeist/zeitgeist/bluebird-no-distinct

359. By Seif Lotfy on 2011-12-31

Add Exclusive lock for secure writing

360. By Michal Hruby on 2011-12-31

Add real-world query samples

361. By Michal Hruby on 2012-01-02

Don't use fts when using in-memory database

362. By Michal Hruby on 2012-01-02

Don't display debug messages by default

363. By Michal Hruby on 2012-01-02

Add a way to disable extensions

364. By Seif Lotfy on 2012-01-03

Fix bug 909708 where inserting and event with 2 subjects.
Since we already parse the subjects before insertion I added a list with subj_uris to find if there is a duplicate and return 0 if one is found

365. By Michal Hruby on 2012-01-06

Fix Reindex in FTS indexer

366. By Michal Hruby on 2012-01-06

Disable the VolumeMonitor in StorageMonitor extension to avoid weird races

367. By Seif Lotfy on 2012-01-15

optimize allocations of variants

368. By Seif Lotfy on 2012-01-16

Seperate the allocation optimization into a new method optimize_variant_allocation.
This optimization effects marshalling by slowint it down by around 0.06s for 10k
events yet saves us 10MB in memory consumption.

369. By Siegfried Gevatter on 2012-01-25

Merge Seif's FTS changes removing database writes.

370. By Siegfried Gevatter on 2012-01-25

Change event structure assertions to errors so D-Bus doesn't crash Zeitgeist

[Michal Hruby (mhr3)]

156 assert (event_variant.get_type_string () == "(" +
157 Utils.SIG_EVENT + ")");

Let's turn this into assert_sig as well.

161 - assert (iter.n_children () >= 3);

Shouldn't this throw the error as well, instead of just removing it?

review: Needs Fixing

[Michal Hruby (mhr3)]

Also, some tests pls...

[Siegfried Gevatter (rainct)]

(Moving to https://code.launchpad.net/~zeitgeist/zeitgeist/bluebird-sig-crash2/+merge/90166)

Unmerged revisions

370. By Siegfried Gevatter on 2012-01-25

Change event structure assertions to errors so D-Bus doesn't crash Zeitgeist

=== modified file 'extensions/blacklist.vala'
--- extensions/blacklist.vala	2011-10-20 14:20:17 +0000
+++ extensions/blacklist.vala	2012-01-25 11:54:25 +0000
@@ -47,7 +47,7 @@
         private const string SIG_BLACKLIST = "a{s("+Utils.SIG_EVENT+")}";
 
         private static HashTable<string, Event> from_variant (
-            Variant templates_variant)
+            Variant templates_variant) throws EngineError
         {
             var blacklist = new HashTable<string, Event> (str_hash, str_equal);
 
@@ -100,8 +100,18 @@
             Variant? templates = retrieve_config ("blacklist",
                 BlacklistTemplates.SIG_BLACKLIST);
             if (templates != null)
-                blacklist = BlacklistTemplates.from_variant (templates);
-            else
+            {
+                try
+                {
+                    blacklist = BlacklistTemplates.from_variant (templates);
+                    warning ("The blacklist database is corrupt. Overwriting it.");
+                }
+                catch (EngineError err)
+                {
+                    templates = null;
+                }
+            }
+            if (templates != null)
                 blacklist = new HashTable<string, Event> (str_hash, str_equal);
 
             // This will be called after bus is acquired, so it shouldn't block

=== modified file 'extensions/ds-registry.vala'
--- extensions/ds-registry.vala	2011-09-28 12:51:08 +0000
+++ extensions/ds-registry.vala	2012-01-25 11:54:25 +0000
@@ -73,6 +73,7 @@
 
         public DataSource.from_variant (Variant variant,
             bool reset_running=false)
+            throws EngineError
         {
             warn_if_fail (
                 variant.get_type_string () == "(sssa("+Utils.SIG_EVENT+")bxb)"
@@ -128,6 +129,7 @@
 
         private static HashTable<string, DataSource> from_variant (
             Variant sources_variant, bool reset_running=false)
+            throws EngineError
         {
             var registry = new HashTable<string, DataSource> (
                 str_hash, str_equal);
@@ -186,8 +188,18 @@
             Variant? registry = retrieve_config ("registry",
                 DataSources.SIG_DATASOURCES);
             if (registry != null)
-                sources = DataSources.from_variant (registry, true);
-            else
+            {
+                try
+                {
+                    sources = DataSources.from_variant (registry, true);
+                    warning ("The data-source registry is corrupt. Overwriting it.");
+                }
+                catch (EngineError.INVALID_SIGNATURE err)
+                {
+                    registry = null;
+                }
+            }
+            if (registry == null)
                 sources = new HashTable<string, DataSource> (
                     str_hash, str_equal);
 
@@ -250,6 +262,7 @@
 
         public bool register_data_source (string unique_id, string name,
             string description, Variant event_templates, BusName? sender)
+            throws Error
         {
             debug ("%s: %s, %s, %s", Log.METHOD, unique_id, name, description);
             if (sender == null)

=== modified file 'extensions/storage-monitor.vala'
--- extensions/storage-monitor.vala	2012-01-06 10:20:42 +0000
+++ extensions/storage-monitor.vala	2012-01-25 11:54:25 +0000
@@ -333,9 +333,13 @@
             store_storage_medium_stmt.bind_text (3, icon);
             store_storage_medium_stmt.bind_text (4, display_name);
 
-            int rc = store_storage_medium_stmt.step ();
-            database.assert_query_success (rc, "add_storage_medium",
-                Sqlite.DONE);
+            try
+            {
+                int rc = store_storage_medium_stmt.step ();
+                database.assert_query_success (rc, "add_storage_medium",
+                    Sqlite.DONE);
+            }
+            catch (EngineError err) {}
 
             storage_available (medium_name, StorageMedia.to_variant (
                 medium_name, true, icon, display_name));
@@ -352,9 +356,13 @@
                 update_medium_state_stmt.reset ();
                 update_medium_state_stmt.bind_int (1, 0);
                 update_medium_state_stmt.bind_text (2, medium_name);
-                int rc = update_medium_state_stmt.step ();
-                database.assert_query_success (rc, "remove_storage_medium",
-                    Sqlite.DONE);
+                try
+                {
+                    int rc = update_medium_state_stmt.step ();
+                    database.assert_query_success (rc, "remove_storage_medium",
+                        Sqlite.DONE);
+                }
+                catch (EngineError err) {}
             }
             storage_unavailable (medium_name);
         }

=== modified file 'src/datamodel.vala'
--- src/datamodel.vala	2012-01-16 12:25:29 +0000
+++ src/datamodel.vala	2012-01-25 11:54:25 +0000
@@ -22,6 +22,13 @@
 
 namespace Zeitgeist
 {
+    private void assert_sig (bool condition, string error_message)
+        throws EngineError.INVALID_SIGNATURE
+    {
+        if (unlikely (!condition))
+            throw new EngineError.INVALID_SIGNATURE (error_message);
+    }
+
     namespace Timestamp
     {
         public static int64 now ()
@@ -252,7 +259,7 @@
                              // must be immediately available to the user
         ANY             = 2  // The event subjects may or may not be available
     }
-    
+
     private bool check_field_match (string property,
             string template_property, bool is_symbol = false,
             bool can_wildcard = false)
@@ -334,21 +341,22 @@
             subjects.add (subject);
         }
 
-        public Event.from_variant (Variant event_variant) {
+        public Event.from_variant (Variant event_variant)
+            throws EngineError.INVALID_SIGNATURE
+        {
             assert (event_variant.get_type_string () == "(" +
                 Utils.SIG_EVENT + ")");
 
             VariantIter iter = event_variant.iterator ();
 
-            assert (iter.n_children () >= 3);
-            VariantIter event_array = iter.next_value ().iterator ();
-            VariantIter subjects_array = iter.next_value ().iterator ();
+            VariantIter event_array = iter.next_value().iterator();
+            VariantIter subjects_array = iter.next_value().iterator();
             Variant payload_variant = iter.next_value ();
 
             var event_props = event_array.n_children ();
-            assert (event_props >= 5);
-            id = (uint32) uint64.parse (event_array.next_value ().get_string ());
-            var str_timestamp = event_array.next_value ().get_string ();
+            assert_sig (event_props >= 5, "Missing event information.");
+            id = (uint32) uint64.parse (event_array.next_value().get_string ());
+            var str_timestamp = event_array.next_value().get_string ();
             if (str_timestamp == "")
                 timestamp = Timestamp.now ();
             else
@@ -504,6 +512,7 @@
     {
 
         public static GenericArray<Event> from_variant (Variant vevents)
+            throws EngineError.INVALID_SIGNATURE
         {
             GenericArray<Event> events = new GenericArray<Event> ();
 
@@ -588,11 +597,12 @@
         }
 
         public Subject.from_variant (Variant subject_variant)
+            throws EngineError.INVALID_SIGNATURE
         {
             VariantIter iter = subject_variant.iterator();
 
             var subject_props = iter.n_children ();
-            assert (subject_props >= 7);
+            assert_sig (subject_props >= 7, "Missing subject information");
             uri = iter.next_value().get_string ();
             interpretation = iter.next_value().get_string ();
             manifestation = iter.next_value().get_string ();

=== modified file 'src/errors.vala'
--- src/errors.vala	2011-10-20 13:32:51 +0000
+++ src/errors.vala	2012-01-25 11:54:25 +0000
@@ -27,6 +27,7 @@
         INVALID_ARGUMENT,
         INVALID_KEY,
         BACKUP_FAILED,
+        INVALID_SIGNATURE, // FIXME: change from EngineError to sth. + public?
     }
 
     // vala doesn't include proper headers, this fixes it

=== modified file 'src/sql.vala'
--- src/sql.vala	2012-01-02 19:30:51 +0000
+++ src/sql.vala	2012-01-25 11:54:25 +0000
@@ -225,7 +225,7 @@
         public void assert_query_success (int rc, string msg,
             int success_code=Sqlite.OK) throws EngineError
         {
-            if (rc != success_code)
+            if (unlikely (rc != success_code))
             {
                 string error_message = "%s: %d, %s".printf(
                     msg, rc, database.errmsg ());