Merge lp:~robru/friends/persistence into lp:friends

Proposed by Robert Bruce Park on 2012-10-16
Status: Merged
Merged at revision: 19
Proposed branch: lp:~robru/friends/persistence
Merge into: lp:friends
Diff against target: 543 lines (+254/-31)
11 files modified
friends/main.py (+16/-0)
friends/service/dispatcher.py (+10/-0)
friends/tests/test_download.py (+1/-1)
friends/tests/test_flickr.py (+2/-2)
friends/tests/test_model.py (+68/-2)
friends/tests/test_protocols.py (+53/-5)
friends/tests/test_twitter.py (+11/-11)
friends/utils/base.py (+41/-2)
friends/utils/download.py (+0/-1)
friends/utils/model.py (+48/-2)
tools/debug_live.py (+4/-5)
To merge this branch: bzr merge lp:~robru/friends/persistence
Reviewer Review Type Date Requested Status
Robert Bruce Park Approve on 2012-10-18
Barry Warsaw 2012-10-16 Pending
Ken VanDine 2012-10-16 Pending
Review via email: mp+130009@code.launchpad.net

Description of the Change

This implements the Dee.ResourceManager for persisting our Dee.SharedModel across instances of friends-service.

So, the important thing is that this *works*. I confirmed it with debug_live.py script. You can run './tools/debug_live.py twitter home' and then run './tools/debug_live.py facebook receive' and you'll see the second invocation will still show you all the rows from the first invocation (eg, the twitter messages will show up with the facebook ones as well).

That said, I don't know what kind of performance implications this has. Probably this is going to make friends-service very slow to launch once there are lots of rows in the Model. We should consider some kind of row-expiry logic, dropping rows after they are older than some certain threshold. Or maybe just putting a hard limit on the number of rows that we'll allow and then deleting the oldest of the rows when that limit is exceeded. Somebody who knows more about Dee should decide what the maximum number of rows should be before we can expect performance to become an issue.

barry: I don't know how to write a test for this. Maybe TestDbus could be expanded to invoke friends-service, close it, invoke it again, and test that Model rows are still there? I don't know what that would look like.

kenvandine: when you ping mhr3 about this, please show him this mp so he can see what we're trying to do.

To post a comment you must log in.
lp:~robru/friends/persistence updated on 2012-10-18
8. By Robert Bruce Park on 2012-10-18

Add test case for _initialize_caches, and some cleanup.

9. By Robert Bruce Park on 2012-10-18

Stop persisting an unchanged model at launch.

Previously the code was unconditionally saving the Model to disk
immediately after loading it, even if there were no changes.

Now, we will save an empty model to disk at launch only if a) there
was no model saved before, or b) the schema has changed since the last
run, invalidating all existing data.

10. By Robert Bruce Park on 2012-10-18

Add a test case for persist_model function.

Robert Bruce Park (robru) wrote :

Alright, this is tested and working. Please merge ;-)

review: Approve
lp:~robru/friends/persistence updated on 2012-10-23
11. By Robert Bruce Park on 2012-10-20

Merged trunk.

12. By Robert Bruce Park on 2012-10-20

Add row expiry logic.

13. By Robert Bruce Park on 2012-10-20

Add test coverage for model pruning functionality.

14. By Robert Bruce Park on 2012-10-20

Move prune_model and initialize_caches calls into main.py

The reason for this is that a) defining a function only to call it
immediately is slightly goofy, and b) this gives main.py finer-grained
control on startup tasks, freeing us from import side-effects that
could potentially break if we had the wrong import order.

15. By Robert Bruce Park on 2012-10-22

Use Model.insert_sorted instead of Model.append, sorting by timestamp.

This makes it easier to delete oldest rows when pruning.

16. By Robert Bruce Park on 2012-10-22

Merged lp:friends

17. By Robert Bruce Park on 2012-10-23

Fix SharedModel sorting, thanks to mhr3.

Apparently you can't cmp() a GLib.Variant the way you would expect to
do so with a string, so you have to .get_string() from the Variant
first.

This commit also adds tests for _cmp and _cmp_date, and also fixes up
a bunch of other tests which were broken by the model data being sorted.

18. By Robert Bruce Park on 2012-10-23

Pyflakes cleanup.

19. By Robert Bruce Park on 2012-10-23

Prevent Dispatcher.Refresh from pointlessly saving the Model at launch.

20. By Robert Bruce Park on 2012-10-23

Merged lp:friends.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'friends/main.py'
2--- friends/main.py 2012-10-19 21:09:10 +0000
3+++ friends/main.py 2012-10-23 17:34:18 +0000
4@@ -33,8 +33,10 @@
5 from friends.service.connection import ConnectionMonitor
6 from friends.service.dispatcher import Dispatcher
7 from friends.service.shortener import URLShorten
8+from friends.utils.base import initialize_caches
9 from friends.utils.logging import initialize
10 from friends.utils.menus import MenuManager
11+from friends.utils.model import prune_model
12 from friends.utils.options import Options
13
14
15@@ -61,6 +63,20 @@
16 log = logging.getLogger(__name__)
17 log.info('Friends backend service starting')
18
19+ # mhr3 says that we should not let a Dee.SharedModel exceed 8mb in
20+ # size, because anything larger will have problems being transmitted
21+ # over DBus. I have conservatively calculated our average row length
22+ # to be 500 bytes, which means that we shouldn't let our model exceed
23+ # approximately 16,000 rows. However, that seems like a lot to me, so
24+ # I'm going to set it to 8,000 for now and we can tweak this later if
25+ # necessary. Do you really need more than 8,000 tweets in memory at
26+ # once? What are you doing with all these tweets?
27+ prune_model(8000)
28+
29+ # This builds two different indexes of our persisted Dee.Model
30+ # data for the purposes of faster duplicate checks.
31+ initialize_caches()
32+
33 # Set up the DBus main loop.
34 DBusGMainLoop(set_as_default=True)
35 loop = GLib.MainLoop()
36
37=== modified file 'friends/service/dispatcher.py'
38--- friends/service/dispatcher.py 2012-10-23 04:27:58 +0000
39+++ friends/service/dispatcher.py 2012-10-23 17:34:18 +0000
40@@ -33,6 +33,7 @@
41 from friends.utils.avatar import Avatar
42 from friends.utils.manager import protocol_manager
43 from friends.utils.signaler import signaler
44+from friends.utils.model import persist_model
45
46 log = logging.getLogger(__name__)
47
48@@ -54,7 +55,10 @@
49 signaler.add_signal('ConnectionOnline', self._on_connection_online)
50 signaler.add_signal('ConnectionOffline', self._on_connection_offline)
51 self._on_connection_online()
52+ # Don't persist the model on launch, before we have anything to save.
53+ self._do_persist_model = False
54 self.Refresh()
55+ self._do_persist_model = True
56
57 def _on_connection_online(self):
58 if self._timer_id is None:
59@@ -80,6 +84,12 @@
60 if thread != current:
61 thread.join()
62
63+ # Write the Dee.SharedModel to disk. We do this every refresh
64+ # for robustness, so if the computer loses power, we won't
65+ # lose all the messages.
66+ if self._do_persist_model:
67+ persist_model()
68+
69 if not self.online:
70 return
71
72
73=== modified file 'friends/tests/test_download.py'
74--- friends/tests/test_download.py 2012-10-19 19:27:21 +0000
75+++ friends/tests/test_download.py 2012-10-23 17:34:18 +0000
76@@ -27,7 +27,7 @@
77 import threading
78
79 from base64 import encodebytes
80-from urllib.error import HTTPError, URLError
81+from urllib.error import URLError
82 from urllib.parse import parse_qs
83 from urllib.request import urlopen
84 from wsgiref.simple_server import WSGIRequestHandler, make_server
85
86=== modified file 'friends/tests/test_flickr.py'
87--- friends/tests/test_flickr.py 2012-10-20 15:35:30 +0000
88+++ friends/tests/test_flickr.py 2012-10-23 17:34:18 +0000
89@@ -262,7 +262,7 @@
90 self.assertEqual(col('sender'), '123')
91 self.assertEqual(col('timestamp'), '2012-05-10T13:36:45')
92 self.assertFalse(col('from_me'))
93- row = list(TestModel.get_row(1))
94+ row = list(TestModel.get_row(2))
95 # Image 2 data. The image is from the account owner.
96 self.assertEqual(col('message'), 'bee')
97 self.assertEqual(col('html'), 'bee')
98@@ -272,7 +272,7 @@
99 self.assertTrue(col('from_me'))
100 # Image 3 data. This data set has some additional entries that allow
101 # various image urls and other keys to be added.
102- row = list(TestModel.get_row(2))
103+ row = list(TestModel.get_row(1))
104 self.assertEqual(col('message'), 'cat')
105 self.assertEqual(col('html'), 'cat')
106 self.assertEqual(
107
108=== modified file 'friends/tests/test_model.py'
109--- friends/tests/test_model.py 2012-10-13 01:27:15 +0000
110+++ friends/tests/test_model.py 2012-10-23 17:34:18 +0000
111@@ -26,19 +26,85 @@
112
113 import unittest
114
115-from friends.utils.model import Model
116+from friends.utils.model import Model, first_run, stale_schema
117+from friends.utils.model import prune_model, persist_model
118+from friends.testing.mocks import LogMock
119 from gi.repository import Dee
120
121
122+try:
123+ # Python 3.3
124+ from unittest import mock
125+except ImportError:
126+ import mock
127+
128+
129 class TestModel(unittest.TestCase):
130 """Test our Dee.SharedModel instance."""
131
132+ def setUp(self):
133+ self.log_mock = LogMock('friends.utils.model')
134+
135+ def tearDown(self):
136+ self.log_mock.stop()
137+
138 def test_basic_properties(self):
139 self.assertIsInstance(Model, Dee.SharedModel)
140 self.assertEqual(Model.get_n_columns(), 37)
141- self.assertEqual(Model.get_n_rows(), 0)
142 self.assertEqual(Model.get_schema(),
143 ['aas', 's', 's', 's', 'b', 's', 's', 's',
144 's', 's', 's', 's', 's', 's', 'd', 'b', 's', 's',
145 's', 's', 's', 's', 's', 's', 's', 's', 's', 's',
146 's', 's', 's', 's', 's', 'as', 's', 's', 's'])
147+ if first_run or stale_schema:
148+ # Then the Model should be brand-new and empty
149+ self.assertEqual(Model.get_n_rows(), 0)
150+
151+ @mock.patch('friends.utils.model._resource_manager')
152+ def test_persistence(self, resource_manager):
153+ persist_model()
154+ resource_manager.store.assert_called_once_with(
155+ Model, 'com.canonical.Friends.Streams')
156+
157+ @mock.patch('friends.utils.model.Model')
158+ @mock.patch('friends.utils.model.persist_model')
159+ def test_prune_one(self, persist, model):
160+ model.get_n_rows.return_value = 8001
161+ def side_effect(arg):
162+ model.get_n_rows.return_value -= 1
163+ model.remove.side_effect = side_effect
164+ prune_model(8000)
165+ persist.assert_called_once_with()
166+ model.get_first_iter.assert_called_once_with()
167+ model.remove.assert_called_once_with(model.get_first_iter())
168+ self.assertEqual(self.log_mock.empty(),
169+ 'Deleted 1 rows from Dee.SharedModel.\n')
170+
171+ @mock.patch('friends.utils.model.Model')
172+ @mock.patch('friends.utils.model.persist_model')
173+ def test_prune_one_hundred(self, persist, model):
174+ model.get_n_rows.return_value = 8100
175+ def side_effect(arg):
176+ model.get_n_rows.return_value -= 1
177+ model.remove.side_effect = side_effect
178+ prune_model(8000)
179+ persist.assert_called_once_with()
180+ self.assertEqual(model.get_first_iter.call_count, 100)
181+ model.remove.assert_called_with(model.get_first_iter())
182+ self.assertEqual(model.remove.call_count, 100)
183+ self.assertEqual(self.log_mock.empty(),
184+ 'Deleted 100 rows from Dee.SharedModel.\n')
185+
186+ @mock.patch('friends.utils.model.Model')
187+ @mock.patch('friends.utils.model.persist_model')
188+ def test_prune_none(self, persist, model):
189+ model.get_n_rows.return_value = 100
190+ def side_effect(arg):
191+ model.get_n_rows.return_value -= 1
192+ model.remove.side_effect = side_effect
193+ prune_model(8000)
194+ model.get_n_rows.assert_called_once_with()
195+ self.assertFalse(persist.called)
196+ self.assertFalse(model.get_first_iter.called)
197+ self.assertFalse(model.remove.called)
198+ self.assertEqual(self.log_mock.empty(), '')
199
200=== modified file 'friends/tests/test_protocols.py'
201--- friends/tests/test_protocols.py 2012-10-13 01:27:15 +0000
202+++ friends/tests/test_protocols.py 2012-10-23 17:34:18 +0000
203@@ -29,7 +29,7 @@
204 from friends.protocols.flickr import Flickr
205 from friends.protocols.twitter import Twitter
206 from friends.testing.helpers import FakeAccount
207-from friends.utils.base import Base, feature
208+from friends.utils.base import Base, feature, _cmp, _cmp_date, TIME_IDX
209 from friends.utils.manager import ProtocolManager
210 from friends.utils.model import (
211 COLUMN_INDICES, COLUMN_NAMES, COLUMN_TYPES, Model)
212@@ -146,16 +146,40 @@
213
214 @mock.patch('friends.utils.base.Model', TestModel)
215 def test_shared_model_successfully_mocked(self):
216- self.assertEqual(Model.get_n_rows(), 0)
217+ count = Model.get_n_rows()
218 self.assertEqual(TestModel.get_n_rows(), 0)
219 base = Base(FakeAccount())
220 base._publish('alpha', message='a')
221 base._publish('beta', message='b')
222 base._publish('omega', message='c')
223- self.assertEqual(Model.get_n_rows(), 0)
224+ self.assertEqual(Model.get_n_rows(), count)
225 self.assertEqual(TestModel.get_n_rows(), 3)
226
227 @mock.patch('friends.utils.base.Model', TestModel)
228+ @mock.patch('friends.utils.base._seen_ids', {})
229+ @mock.patch('friends.utils.base._seen_messages', {})
230+ def test_seen_dicts_successfully_instantiated(self):
231+ from friends.utils.base import _seen_ids, _seen_messages
232+ from friends.utils.base import initialize_caches
233+ self.assertEqual(TestModel.get_n_rows(), 0)
234+ base = Base(FakeAccount())
235+ base._publish('alpha', sender='a', message='a')
236+ base._publish('beta', sender='a', message='a')
237+ base._publish('omega', sender='a', message='b')
238+ self.assertEqual(TestModel.get_n_rows(), 2)
239+ _seen_ids.clear()
240+ _seen_messages.clear()
241+ initialize_caches()
242+ self.assertEqual(sorted(list(_seen_messages.keys())), ['aa', 'ab'])
243+ self.assertEqual(sorted(list(_seen_ids.keys())),
244+ [('base', 'faker/than fake', 'alpha'),
245+ ('base', 'faker/than fake', 'beta'),
246+ ('base', 'faker/than fake', 'omega')])
247+ # These two point at the same row because sender+message are identical
248+ self.assertEqual(_seen_ids[('base', 'faker/than fake', 'alpha')],
249+ _seen_ids[('base', 'faker/than fake', 'beta')])
250+
251+ @mock.patch('friends.utils.base.Model', TestModel)
252 def test_invalid_argument(self):
253 base = Base(FakeAccount())
254 self.assertEqual(0, TestModel.get_n_rows())
255@@ -342,10 +366,10 @@
256 # See? Two rows in the table.
257 self.assertEqual(2, TestModel.get_n_rows())
258 # The first row is the message from fred.
259- self.assertEqual(TestModel.get_row(0)[COLUMN_INDICES['sender']],
260+ self.assertEqual(TestModel.get_row(1)[COLUMN_INDICES['sender']],
261 'fred')
262 # The second row is the message from tedtholomew.
263- self.assertEqual(TestModel.get_row(1)[COLUMN_INDICES['sender']],
264+ self.assertEqual(TestModel.get_row(0)[COLUMN_INDICES['sender']],
265 'tedtholomew')
266
267 def test_basic_login(self):
268@@ -370,3 +394,27 @@
269
270 def test_features(self):
271 self.assertEqual(MyProtocol.get_features(), ['feature_1', 'feature_2'])
272+
273+ def test_cmp(self):
274+ self.assertEqual(_cmp('2007-05-15T16:45:00', '2007-05-15T16:45:01'), -1)
275+ self.assertEqual(_cmp('2007-05-15T16:45:00', '2007-05-15T16:45:00'), 0)
276+ self.assertEqual(_cmp('2007-05-15T16:45:01', '2007-05-15T16:45:00'), 1)
277+
278+ def test_cmp_date(self):
279+ """Ensure that our SharedModel sorting func sorts correctly."""
280+ row1 = ['foo'] * 100
281+ row2 = ['bar'] * 100
282+ class fake_variant:
283+ def __init__(self, string):
284+ self.string = string
285+ def get_string(self):
286+ return self.string
287+ row1[TIME_IDX] = fake_variant('2012-01-01T11:59:59')
288+ row2[TIME_IDX] = fake_variant('2012-01-01T11:59:59')
289+ self.assertEqual(_cmp_date(row1, len(row1), row2, len(row2), None), 0)
290+ row1[TIME_IDX] = fake_variant('2012-01-01T11:59:58')
291+ row2[TIME_IDX] = fake_variant('2012-01-01T11:59:59')
292+ self.assertEqual(_cmp_date(row1, len(row1), row2, len(row2), None), -1)
293+ row1[TIME_IDX] = fake_variant('2012-01-01T11:59:58')
294+ row2[TIME_IDX] = fake_variant('2012-01-01T11:59:57')
295+ self.assertEqual(_cmp_date(row1, len(row1), row2, len(row2), None), 1)
296
297=== modified file 'friends/tests/test_twitter.py'
298--- friends/tests/test_twitter.py 2012-10-16 18:31:06 +0000
299+++ friends/tests/test_twitter.py 2012-10-23 17:34:18 +0000
300@@ -126,11 +126,12 @@
301
302 # This test data was ripped directly from Twitter's API docs.
303 expected = [
304- [[['twitter', 'faker/than fake', '240558470661799936']],
305- 'messages', 'OAuth Dancer', 'oauth_dancer', False,
306- '2012-08-28T21:16:23', 'just another test', '',
307- 'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
308- 'https://twitter.com/oauth_dancer/status/240558470661799936', '',
309+ [[['twitter', 'faker/than fake', '240539141056638977']],
310+ 'messages', 'Taylor Singletary', 'episod', False,
311+ '2012-08-28T19:59:34',
312+ 'You\'d be right more often if you thought you were wrong.', '',
313+ 'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
314+ 'https://twitter.com/episod/status/240539141056638977', '',
315 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
316 '', '', '', '', '', '', '', [], '', '', '',
317 ],
318@@ -144,12 +145,11 @@
319 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
320 '', '', '', '', '', '', '', [], '', '', '',
321 ],
322- [[['twitter', 'faker/than fake', '240539141056638977']],
323- 'messages', 'Taylor Singletary', 'episod', False,
324- '2012-08-28T19:59:34',
325- 'You\'d be right more often if you thought you were wrong.', '',
326- 'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
327- 'https://twitter.com/episod/status/240539141056638977', '',
328+ [[['twitter', 'faker/than fake', '240558470661799936']],
329+ 'messages', 'OAuth Dancer', 'oauth_dancer', False,
330+ '2012-08-28T21:16:23', 'just another test', '',
331+ 'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
332+ 'https://twitter.com/oauth_dancer/status/240558470661799936', '',
333 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
334 '', '', '', '', '', '', '', [], '', '', '',
335 ],
336
337=== modified file 'friends/utils/base.py'
338--- friends/utils/base.py 2012-10-19 21:09:10 +0000
339+++ friends/utils/base.py 2012-10-23 17:34:18 +0000
340@@ -19,6 +19,7 @@
341 __all__ = [
342 'Base',
343 'feature',
344+ 'initialize_caches',
345 ]
346
347
348@@ -27,8 +28,11 @@
349 import logging
350 import threading
351
352+from datetime import datetime
353+
354 from friends.utils.authentication import Authentication
355 from friends.utils.model import COLUMN_INDICES, SCHEMA, DEFAULTS, Model
356+from friends.utils.time import ISO8601_FORMAT
357
358
359 IGNORED = string.punctuation + string.whitespace
360@@ -38,6 +42,7 @@
361 SENDER_IDX = COLUMN_INDICES['sender']
362 MESSAGE_IDX = COLUMN_INDICES['message']
363 IDS_IDX = COLUMN_INDICES['message_ids']
364+TIME_IDX = COLUMN_INDICES['timestamp']
365
366
367 # This is a mapping from Dee.SharedModel row keys to the DeeModelIters
368@@ -66,7 +71,7 @@
369
370 Then find all feature methods for a protocol with:
371
372- for feature_name in ProtocolClass.features:
373+ for feature_name in ProtocolClass.get_features():
374 # ...
375 """
376 method.is_feature = True
377@@ -102,6 +107,35 @@
378 return EMPTY_STRING.join(char for char in key if char not in IGNORED)
379
380
381+def _cmp(a, b):
382+ """Ressurrect cmp() because Dee.SharedModel demands it."""
383+ return (a > b) - (a < b)
384+
385+
386+def _cmp_date(row1, row1_length, row2, row2_length, user_data):
387+ """Comparison function that sorts Model rows by UTC timestamp."""
388+ row1_key = row1[TIME_IDX].get_string()
389+ row2_key = row2[TIME_IDX].get_string()
390+ return _cmp(row1_key, row2_key)
391+
392+
393+def initialize_caches():
394+ """Populate _seen_ids and _seen_messages with Model data.
395+
396+ Our Dee.SharedModel persists across instances, so we need to
397+ populate these caches at launch.
398+ """
399+ for i in range(Model.get_n_rows()):
400+ row_iter = Model.get_iter_at_row(i)
401+ row = Model.get_row(row_iter)
402+ _seen_messages[_make_key(row)] = row_iter
403+ for triple in row[IDS_IDX]:
404+ _seen_ids[tuple(triple)] = row_iter
405+ log.debug(
406+ '_seen_ids: {}, _seen_messages: {}'.format(
407+ len(_seen_ids), len(_seen_messages)))
408+
409+
410 class _OperationThread(threading.Thread):
411 """Catch, log, and swallow all exceptions in the sub-thread."""
412
413@@ -207,6 +241,11 @@
414 if len(kwargs) > 0:
415 raise TypeError('Unexpected keyword arguments: {}'.format(
416 COMMA_SPACE.join(sorted(kwargs))))
417+ if not args[TIME_IDX]:
418+ # We *need* a timestamp for sorting so badly that it's better
419+ # to use the current time than to fail too loudly.
420+ log.error('No timestamp for message: {!r}'.format(triple))
421+ args[TIME_IDX] = datetime.today().strftime(ISO8601_FORMAT)
422 with _publish_lock:
423 # Don't let duplicate messages into the model, but do record the
424 # unique message ids of each duplicate message.
425@@ -214,7 +253,7 @@
426 row_iter = _seen_messages.get(key)
427 if row_iter is None:
428 # We haven't seen this message before.
429- _seen_messages[key] = Model.append(*args)
430+ _seen_messages[key] = Model.insert_sorted(_cmp_date, *args)
431 else:
432 # We have seen this before, so append to the matching column's
433 # message_ids list, this message's id.
434
435=== modified file 'friends/utils/download.py'
436--- friends/utils/download.py 2012-10-19 19:10:54 +0000
437+++ friends/utils/download.py 2012-10-23 17:34:18 +0000
438@@ -28,7 +28,6 @@
439 from base64 import encodebytes
440 from contextlib import contextmanager
441 from gi.repository import Soup, SoupGNOME
442-from urllib.error import HTTPError
443 from urllib.parse import urlencode
444
445 log = logging.getLogger(__name__)
446
447=== modified file 'friends/utils/model.py'
448--- friends/utils/model.py 2012-10-13 01:27:15 +0000
449+++ friends/utils/model.py 2012-10-23 17:34:18 +0000
450@@ -29,11 +29,17 @@
451 'COLUMN_TYPES',
452 'COLUMN_INDICES',
453 'DEFAULTS',
454+ 'MODEL_DBUS_NAME',
455+ 'persist_model',
456+ 'prune_model',
457 ]
458
459
460 from gi.repository import Dee
461
462+import logging
463+log = logging.getLogger(__name__)
464+
465
466 # Most of this schema is very straightforward, but the 'message_ids' column
467 # needs a bit of explanation:
468@@ -108,5 +114,45 @@
469 }
470
471
472-Model = Dee.SharedModel.new('com.canonical.Friends.Streams')
473-Model.set_schema_full(COLUMN_TYPES)
474+MODEL_DBUS_NAME = 'com.canonical.Friends.Streams'
475+_resource_manager = Dee.ResourceManager.get_default()
476+Model = _resource_manager.load(MODEL_DBUS_NAME)
477+
478+
479+first_run = Model is None
480+if not first_run:
481+ stale_schema = Model.get_schema() != list(COLUMN_TYPES)
482+else:
483+ stale_schema = True
484+
485+
486+def persist_model():
487+ """Write our Dee.SharedModel instance to disk."""
488+ log.debug('Saving Dee.SharedModel with {} rows.'.format(len(Model)))
489+ _resource_manager.store(Model, MODEL_DBUS_NAME)
490+
491+
492+# If this is first run, or the schema has changed since last run,
493+# we'll need to make a new, empty Model.
494+if first_run or stale_schema:
495+ log.debug('Starting a new, empty Dee.SharedModel.')
496+ Model = Dee.SharedModel.new(MODEL_DBUS_NAME)
497+ Model.set_schema_full(COLUMN_TYPES)
498+
499+ # Calling this from here ensures that schema changes are persisted
500+ # ASAP, but we also call it periodically in the dispatcher in
501+ # order to ensure data is saved often in case of power loss.
502+ persist_model()
503+
504+
505+def prune_model(maximum):
506+ """If there are more than maximum rows, remove the oldest ones."""
507+ pruned = 0
508+ while Model.get_n_rows() > maximum:
509+ Model.remove(Model.get_first_iter())
510+ pruned += 1
511+
512+ if pruned:
513+ log.debug('Deleted {} rows from Dee.SharedModel.'.format(pruned))
514+ # Delete those messages from disk, too, not just memory.
515+ persist_model()
516
517=== modified file 'tools/debug_live.py'
518--- tools/debug_live.py 2012-10-20 00:32:54 +0000
519+++ tools/debug_live.py 2012-10-23 17:34:18 +0000
520@@ -20,20 +20,19 @@
521 """
522
523 import sys
524-import time
525 import logging
526
527 sys.path.insert(0, '.')
528
529 from gi.repository import GObject
530 from friends.utils.logging import initialize
531+
532+# Print all logs for debugging purposes
533+initialize(debug=True, console=True)
534+
535 from friends.utils.account import AccountManager
536-from friends.utils.base import Base
537 from friends.utils.model import Model
538
539-# Print all logs for debugging purposes
540-initialize(debug=True, console=True)
541-
542
543 log = logging.getLogger('friends.debug_live')
544

Subscribers

People subscribed via source and target branches