Ubuntu Software Center

Merge lp:~osomon/software-center/one_database_per_language into lp:software-center

Proposed by Olivier Tilloy on 2010-04-21
Status: Rejected
Rejected by: Rodney Dawes on 2013-05-23
Proposed branch: lp:~osomon/software-center/one_database_per_language
Merge into: lp:software-center
Diff against target: 498 lines (+95/-65) 19 files modified
To merge this branch: bzr merge lp:~osomon/software-center/one_database_per_language
Reviewer Review Type Date Requested Status
Michael Vogt 2010-04-21 Needs Fixing on 2010-05-06
Review via email: mp+23870@code.launchpad.net

Description of the Change

This branch intends to fix bug #434601 by implementing the solution proposed by mpt in comment #1: store one database per language code, since the contents of the database is localized. Where the database was previously stored in a "xapian" directory on disk, it is now stored in "xapian.{lang_code}", e.g. "xapian.fr_FR".

Comments on this approach are welcome.

To post a comment you must log in.
Michael Vogt (mvo) wrote :

Thanks for the patch. It will need similar logic in utils/update-sofware-center when it rebuilds the database. Best is probably to consolidate the name with language added in one place.

review: Needs Fixing
Olivier Tilloy (osomon) wrote :

Right, I overlooked this. I will re-factor the patch to have the database name computed in one place only.

Michael Vogt (mvo) wrote :

Thanks, this is good now. I wonder how we should deal with the following case though:
1. system locale is en_US and update-software-center creates /var/cache/software-center/xapian.en_US
2. a normal user has de_DE and runs software-center
   - it will not be able to create /var/cache/software-center
   - from looking at the code it does not currently fallback to the existing one

Ideally I guess we would have a dbus service that simply re-builds it in the language we want.

Michael Vogt (mvo) wrote :

I pushed a branch with some minor modifications (transition code in postinst) to lp:~mvo/software-center/one_database_per_language/

Olivier Tilloy (osomon) wrote :

Indeed there is currently no fallback mechanism. The DBus service solution sounds good to me, but in the short term we can implement a simple fallback mechanism that will pick the first database found in /var/cache/software-center.

749. By Olivier Tilloy on 2010-06-02

Merged from trunk.

750. By Olivier Tilloy on 2010-06-02

Merged Michael's changes to the post-installation script.

751. By Olivier Tilloy on 2010-06-02

When the database path for the current locale doesn't exist
or cannot be created, try to fall back gracefully to an existing database.

Olivier Tilloy (osomon) wrote :

I have merged Michael's changes and implemented a mechanism that should ensure we always try to fall back to an existing database when one cannot be created. Comments appreciated!

Olivier Tilloy (osomon) wrote :

This branch is too old and doesn’t merge cleanly any longer.

@Michael: do you think the original approach would still work? If so, I could take a stab at refreshing the code to apply to the current code base.

Michael Vogt (mvo) wrote :

On Mon, May 23, 2011 at 08:08:27AM -0000, Olivier Tilloy wrote:
> This branch is too old and doesn’t merge cleanly any longer.
>
> @Michael: do you think the original approach would still work? If so, I could take a stab at refreshing the code to apply to the current code base.
> --
> https://code.launchpad.net/~osomon/software-center/one_database_per_language/+merge/23870
> You are reviewing the proposed merge of lp:~osomon/software-center/one_database_per_language into lp:software-center.

I think the general approach is still good and sound, however I would
rather like to see your talents used in the new softwarecener/ui/qml
part of the source ;) Its currently empty, but there is a
lp:~mvo/software-center/refactor-with-qml branch already that
hopefully can move to trunk soonish.

Cheers,
 Michael

Olivier Tilloy (osomon) wrote :

> I think the general approach is still good and sound, however I would
> rather like to see your talents used in the new softwarecener/ui/qml
> part of the source ;) Its currently empty, but there is a
> lp:~mvo/software-center/refactor-with-qml branch already that
> hopefully can move to trunk soonish.

Sure, I’m also more interested in playing with the UI!
I only meant to do a bit of cleanup in pending tasks, this one has been pending forever and in view of the recent activity on the related bug report, it looks like it remains an important issue for users.

Rodney Dawes (dobey) wrote :

Setting this to Rejected since it's quite old, and hasn't been commented on in 2 years, or touched in almost 3.

Unmerged revisions

751. By Olivier Tilloy on 2010-06-02

When the database path for the current locale doesn't exist
or cannot be created, try to fall back gracefully to an existing database.

750. By Olivier Tilloy on 2010-06-02

Merged Michael's changes to the post-installation script.

749. By Olivier Tilloy on 2010-06-02

Merged from trunk.

748. By Olivier Tilloy on 2010-05-07

Use get_database_path everywhere to instantiate the DB.

747. By Olivier Tilloy on 2010-05-07

Merged from trunk.

746. By Olivier Tilloy on 2010-05-07

Separate function to compute the database path on disk, depending on the current locale.

745. By Olivier Tilloy on 2010-05-06

Merged from trunk.

744. By Olivier Tilloy on 2010-04-21

Handle nicely broken locales: fall back to C.

743. By Olivier Tilloy on 2010-04-21

Store one database per language code.

Preview Diff

1=== modified file '.bzrignore'
2--- .bzrignore 2009-09-16 16:37:16 +0000
3+++ .bzrignore 2010-06-02 15:09:30 +0000
4@@ -1,14 +1,1 @@
5-data/xapian/flintlock
6-data/xapian/iamflint
7-data/xapian/postlist.baseA
8-data/xapian/postlist.baseB
9-data/xapian/postlist.DB
10-data/xapian/record.baseA
11-data/xapian/record.baseB
12-data/xapian/record.DB
13-data/xapian/termlist.baseA
14-data/xapian/termlist.baseB
15-data/xapian/termlist.DB
16-data/xapian/value.baseA
17-data/xapian/value.baseB
18-data/xapian/value.DB
19+data/xapian.*
20
21=== removed directory 'data/xapian'
22=== modified file 'debian/software-center.postinst'
23--- debian/software-center.postinst 2010-03-11 16:15:42 +0000
24+++ debian/software-center.postinst 2010-06-02 15:09:30 +0000
25@@ -12,6 +12,11 @@
26 exit 0
27 fi
28
29+# in 2.1.2 we moved to one db per language so we can remove
30+# the old one
31+if dpkg --compare-versions "$2" lt-nl "2.1.2"; then
32+ rm -rf /var/cache/software-center/xapian
33+fi
34
35
36
37
38=== modified file 'softwarecenter/app.py'
39--- softwarecenter/app.py 2010-05-28 07:35:33 +0000
40+++ softwarecenter/app.py 2010-06-02 15:09:30 +0000
41@@ -36,7 +36,7 @@
42 from softwarecenter.enums import *
43 from softwarecenter.utils import *
44 from softwarecenter.version import *
45-from softwarecenter.db.database import StoreDatabase
46+from softwarecenter.db.database import get_database_path, StoreDatabase
47
48 import view.dialogs
49 from view.viewswitcher import ViewSwitcher, ViewSwitcherList
50@@ -120,8 +120,7 @@
51 self.backend.connect("transaction-stopped", self._on_transaction_stopped)
52 self.backend.connect("channels-changed", self.on_channels_changed)
53
54- # xapian
55- pathname = os.path.join(xapian_base_path, "xapian")
56+ pathname = get_database_path(xapian_base_path)
57 try:
58 self.db = StoreDatabase(pathname, self.cache)
59 self.db.open()
60
61=== modified file 'softwarecenter/db/database.py'
62--- softwarecenter/db/database.py 2010-05-18 11:14:59 +0000
63+++ softwarecenter/db/database.py 2010-06-02 15:09:30 +0000
64@@ -16,9 +16,12 @@
65 # this program; if not, write to the Free Software Foundation, Inc.,
66 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
67
68+import errno
69+import glob
70 import gobject
71 import locale
72 import logging
73+import os
74 import re
75 import xapian
76
77@@ -28,6 +31,59 @@
78 from softwarecenter.enums import *
79 from gettext import gettext as _
80
81+
82+def _fallback_existing_database(xapian_base_path):
83+ # Try to locate an existing database to fall back on.
84+ matches = glob.glob(os.path.join(xapian_base_path, "xapian*"))
85+ if matches:
86+ return matches[0]
87+ else:
88+ return None
89+
90+
91+def get_database_path(xapian_base_path=XAPIAN_BASE_PATH, create=True):
92+ """
93+ Return the path to the database for the current locale.
94+ If the path doesn't exist yet and the create parameter is True (default),
95+ it is created. If creation fails or if the create parameter is False,
96+ try to fall back gracefully to an existing database. If that fails, raise
97+ an exception.
98+ """
99+ # Store one database per language code, since the contents of the database
100+ # are localized ("xapian.{lang_code}", e.g. "xapian.fr_FR").
101+ try:
102+ lang_code, encoding = locale.getlocale()
103+ except ValueError:
104+ lang_code = None
105+ if lang_code is None:
106+ lang_code = "C"
107+ db_name = "xapian.%s" % lang_code
108+ pathname = os.path.join(xapian_base_path, db_name)
109+ if os.path.exists(pathname):
110+ return pathname
111+ elif create:
112+ try:
113+ os.makedirs(pathname)
114+ except OSError:
115+ # The database cannot be created, most likely because
116+ # software-center is being run installed by an unprivileged
117+ # user. Try to fall back gracefully to an existing database.
118+ pathname = _fallback_existing_database(xapian_base_path)
119+ if pathname is not None:
120+ return pathname
121+ else:
122+ raise
123+ else:
124+ return pathname
125+ else:
126+ # Try to fall back gracefully to an existing database.
127+ fallback = _fallback_existing_database(xapian_base_path)
128+ if fallback is not None:
129+ return fallback
130+ else:
131+ raise IOError(errno.ENOENT, os.strerror(errno.ENOENT), pathname)
132+
133+
134 class StoreDatabase(gobject.GObject):
135 """thin abstraction for the xapian database with convenient functions"""
136
137
138=== modified file 'softwarecenter/view/appview.py'
139--- softwarecenter/view/appview.py 2010-05-30 16:21:33 +0000
140+++ softwarecenter/view/appview.py 2010-06-02 15:09:30 +0000
141@@ -39,7 +39,7 @@
142 sys.path.insert(0, ".")
143 from softwarecenter.enums import *
144 from softwarecenter.utils import *
145-from softwarecenter.db.database import StoreDatabase, Application
146+from softwarecenter.db.database import get_database_path, StoreDatabase, Application
147 from softwarecenter.backend import get_install_backend
148 from softwarecenter.distro import get_distro
149
150@@ -1464,13 +1464,10 @@
151 if __name__ == "__main__":
152 logging.basicConfig(level=logging.DEBUG)
153
154- xapian_base_path = XAPIAN_BASE_PATH
155- pathname = os.path.join(xapian_base_path, "xapian")
156-
157 # the store
158 from softwarecenter.apt.aptcache import AptCache
159 cache = AptCache()
160- db = StoreDatabase(pathname, cache)
161+ db = StoreDatabase(get_database_path(), cache)
162 db.open()
163
164 # additional icons come from app-install-data
165
166=== modified file 'softwarecenter/view/availablepane.py'
167--- softwarecenter/view/availablepane.py 2010-05-30 15:53:11 +0000
168+++ softwarecenter/view/availablepane.py 2010-06-02 15:09:30 +0000
169@@ -595,8 +595,6 @@
170
171 if __name__ == "__main__":
172 #logging.basicConfig(level=logging.DEBUG)
173- xapian_base_path = XAPIAN_BASE_PATH
174- pathname = os.path.join(xapian_base_path, "xapian")
175
176 if len(sys.argv) > 1:
177 datadir = sys.argv[1]
178@@ -605,7 +603,8 @@
179 else:
180 datadir = "/usr/share/software-center"
181
182- db = xapian.Database(pathname)
183+ from softwarecenter.db.database import get_database_path
184+ db = xapian.Database(get_database_path())
185 icons = gtk.icon_theme_get_default()
186 icons.append_search_path("/usr/share/app-install/icons/")
187
188
189=== modified file 'softwarecenter/view/channelpane.py'
190--- softwarecenter/view/channelpane.py 2010-05-30 17:11:05 +0000
191+++ softwarecenter/view/channelpane.py 2010-06-02 15:09:30 +0000
192@@ -219,8 +219,6 @@
193
194 if __name__ == "__main__":
195 #logging.basicConfig(level=logging.DEBUG)
196- xapian_base_path = XAPIAN_BASE_PATH
197- pathname = os.path.join(xapian_base_path, "xapian")
198
199 if len(sys.argv) > 1:
200 datadir = sys.argv[1]
201@@ -229,7 +227,8 @@
202 else:
203 datadir = "/usr/share/software-center"
204
205- db = xapian.Database(pathname)
206+ from softwarecenter.db.database import get_database_path
207+ db = xapian.Database(get_database_path())
208 icons = gtk.icon_theme_get_default()
209 icons.append_search_path("/usr/share/app-install/icons/")
210 cache = apt.Cache(apt.progress.text.OpProgress())
211
212=== modified file 'softwarecenter/view/historypane.py'
213--- softwarecenter/view/historypane.py 2010-05-19 16:01:48 +0000
214+++ softwarecenter/view/historypane.py 2010-06-02 15:09:30 +0000
215@@ -36,7 +36,7 @@
216 from softwarecenter.enums import *
217 from softwarecenter.view.widgets.searchentry import SearchEntry
218 from softwarecenter.apt.aptcache import AptCache
219-from softwarecenter.db.database import StoreDatabase
220+from softwarecenter.db.database import get_database_path, StoreDatabase
221
222
223 class HistoryPane(gtk.VBox):
224@@ -287,9 +287,7 @@
225
226 if __name__ == '__main__':
227 cache = AptCache()
228-
229- db_path = os.path.join(XAPIAN_BASE_PATH, "xapian")
230- db = StoreDatabase(db_path, cache)
231+ db = StoreDatabase(get_database_path(), cache)
232 db.open()
233
234 icons = gtk.icon_theme_get_default()
235
236=== modified file 'softwarecenter/view/installedpane.py'
237--- softwarecenter/view/installedpane.py 2010-05-31 07:24:21 +0000
238+++ softwarecenter/view/installedpane.py 2010-06-02 15:09:30 +0000
239@@ -177,8 +177,6 @@
240
241 if __name__ == "__main__":
242 #logging.basicConfig(level=logging.DEBUG)
243- xapian_base_path = XAPIAN_BASE_PATH
244- pathname = os.path.join(xapian_base_path, "xapian")
245
246 if len(sys.argv) > 1:
247 datadir = sys.argv[1]
248@@ -187,7 +185,8 @@
249 else:
250 datadir = "/usr/share/software-center"
251
252- db = xapian.Database(pathname)
253+ from softwarecenter.db.database import get_database_path
254+ db = xapian.Database(get_database_path())
255 icons = gtk.icon_theme_get_default()
256 icons.append_search_path("/usr/share/app-install/icons/")
257 cache = apt.Cache(apt.progress.text.OpProgress())
258
259=== modified file 'softwarecenter/view/viewswitcher.py'
260--- softwarecenter/view/viewswitcher.py 2010-05-25 00:48:21 +0000
261+++ softwarecenter/view/viewswitcher.py 2010-06-02 15:09:30 +0000
262@@ -34,7 +34,7 @@
263 from softwarecenter.backend.channel import SoftwareChannel
264 from softwarecenter.backend import get_install_backend
265 from softwarecenter.distro import get_distro
266-from softwarecenter.db.database import StoreDatabase
267+from softwarecenter.db.database import get_database_path, StoreDatabase
268 from softwarecenter.enums import *
269
270 from widgets.animatedimage import CellRendererAnimatedImage, AnimatedImage
271@@ -403,10 +403,8 @@
272 scroll = gtk.ScrolledWindow()
273 icons = gtk.icon_theme_get_default()
274
275- xapian_base_path = XAPIAN_BASE_PATH
276- pathname = os.path.join(xapian_base_path, "xapian")
277 cache = apt.Cache(apt.progress.text.OpProgress())
278- db = StoreDatabase(pathname, cache)
279+ db = StoreDatabase(get_database_path(), cache)
280 db.open()
281
282 view = ViewSwitcher(datadir, db, icons)
283
284=== modified file 'test/test_appview.py'
285--- test/test_appview.py 2010-05-20 18:33:08 +0000
286+++ test/test_appview.py 2010-06-02 15:09:30 +0000
287@@ -10,9 +10,8 @@
288
289 from softwarecenter import Application
290 from softwarecenter.apt.aptcache import AptCache
291-from softwarecenter.db.database import StoreDatabase
292+from softwarecenter.db.database import get_database_path, StoreDatabase
293 from softwarecenter.view.appview import AppStore
294-from softwarecenter.enums import *
295
296 import xapian
297
298@@ -30,10 +29,8 @@
299 """ tests the AppStore GtkTreeViewModel """
300
301 def setUp(self):
302- xapian_base_path = XAPIAN_BASE_PATH
303- pathname = os.path.join(xapian_base_path, "xapian")
304 self.cache = AptCache()
305- self.db = StoreDatabase(pathname, self.cache)
306+ self.db = StoreDatabase(get_database_path(), self.cache)
307 self.db.open()
308 self.mock_icons = MockIconCache()
309 self.mock_filter = MockAppViewFilter()
310
311=== modified file 'utils/bench.py'
312--- utils/bench.py 2010-05-09 07:13:26 +0000
313+++ utils/bench.py 2010-06-02 15:09:30 +0000
314@@ -7,6 +7,7 @@
315 sys.path.insert(0, "../")
316 from softwarecenter.enums import *
317 from softwarecenter.utils import *
318+from softwarecenter.db.database import get_database_path
319
320 def run_benchmark(db):
321 # test postlist
322@@ -37,8 +38,7 @@
323
324 if __name__ == "__main__":
325
326- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
327- db = xapian.Database(pathname)
328+ db = xapian.Database(get_database_path())
329
330 print "app db only"
331 run_benchmark(db)
332
333=== modified file 'utils/installedapps.py'
334--- utils/installedapps.py 2010-05-09 07:13:26 +0000
335+++ utils/installedapps.py 2010-06-02 15:09:30 +0000
336@@ -8,13 +8,13 @@
337 sys.path.insert(0, "../")
338 from softwarecenter.enums import *
339 from softwarecenter.utils import *
340+from softwarecenter.db.database import get_database_path
341
342 if __name__ == "__main__":
343
344 cache = apt.Cache()
345
346- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
347- db = xapian.Database(pathname)
348+ db = xapian.Database(get_database_path())
349
350 installed = []
351 for m in db.postlist(""):
352
353=== modified file 'utils/query.py'
354--- utils/query.py 2010-05-09 07:13:26 +0000
355+++ utils/query.py 2010-06-02 15:09:30 +0000
356@@ -9,6 +9,7 @@
357 sys.path.insert(0, "../")
358 from softwarecenter.enums import *
359 from softwarecenter.utils import *
360+from softwarecenter.db.database import get_database_path
361
362 def parse_query(parser, search_strings, verbose=True):
363 str_to_prefix = { 'section' : 'AE',
364@@ -52,8 +53,7 @@
365 help="print found apps/pkgs too")
366 (options, args) = parser.parse_args()
367
368- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
369- db = xapian.Database(pathname)
370+ db = xapian.Database(get_database_path())
371
372 axi = xapian.Database("/var/lib/apt-xapian-index/index")
373 db.add_database(axi)
374
375=== modified file 'utils/search_query.py'
376--- utils/search_query.py 2010-05-09 07:13:26 +0000
377+++ utils/search_query.py 2010-06-02 15:09:30 +0000
378@@ -7,7 +7,6 @@
379 from optparse import OptionParser
380
381 sys.path.insert(0, "../")
382-from softwarecenter.enums import *
383 from softwarecenter.utils import *
384
385 def run_query(parser, search_terms, verbose):
386@@ -37,8 +36,8 @@
387 help="print found apps/pkgs too")
388 (options, args) = parser.parse_args()
389
390- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
391- db = xapian.Database(pathname)
392+ from softwarecenter.db.database import get_database_path
393+ db = xapian.Database(get_database_path())
394
395 axi = xapian.Database("/var/lib/apt-xapian-index/index")
396 db.add_database(axi)
397
398=== modified file 'utils/stats.py'
399--- utils/stats.py 2010-05-09 07:13:26 +0000
400+++ utils/stats.py 2010-06-02 15:09:30 +0000
401@@ -24,6 +24,7 @@
402
403 sys.path.insert(0, "../")
404 from softwarecenter.enums import *
405+from softwarecenter.db.database import get_database_path
406
407 if __name__ == "__main__":
408
409@@ -35,8 +36,7 @@
410 app_to_pkg = {}
411
412 # gather data
413- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
414- db = xapian.Database(pathname)
415+ db = xapian.Database(get_database_path())
416 for m in db.postlist(""):
417 doc = db.get_document(m.docid)
418 appname = doc.get_data()
419
420=== modified file 'utils/topapps.py'
421--- utils/topapps.py 2010-05-09 07:13:26 +0000
422+++ utils/topapps.py 2010-06-02 15:09:30 +0000
423@@ -8,6 +8,7 @@
424 sys.path.insert(0, "../")
425 from softwarecenter.enums import *
426 from softwarecenter.utils import *
427+from softwarecenter.db.database import get_database_path
428
429 if __name__ == "__main__":
430
431@@ -15,8 +16,7 @@
432 if len(sys.argv) > 1:
433 topn = int(sys.argv[1])
434
435- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
436- db = xapian.Database(pathname)
437+ db = xapian.Database(get_database_path())
438
439 heap = []
440 for m in db.postlist(""):
441
442=== modified file 'utils/update-software-center'
443--- utils/update-software-center 2010-05-09 07:13:26 +0000
444+++ utils/update-software-center 2010-06-02 15:09:30 +0000
445@@ -31,6 +31,7 @@
446 from optparse import OptionParser
447
448 from softwarecenter.enums import *
449+from softwarecenter.db.database import get_database_path
450 from softwarecenter.db.update import rebuild_database
451
452 # dbus may not be available during a upgrade so we
453@@ -97,7 +98,7 @@
454 logging.info("no translation information in database needed")
455 sys.exit(0)
456 mo_time = os.path.getctime(mofile)
457- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
458+ pathname = get_database_path(create=False)
459 if os.path.exists(pathname):
460 db = xapian.Database(pathname)
461 mo_db_time = db.get_metadata("app-install-mo-time")
462@@ -120,13 +121,9 @@
463
464 # rebuild and send signal when done
465 try:
466- # setup path
467- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
468- if not os.path.exists(pathname):
469- os.makedirs(pathname)
470 # rebuild the database, the default context is run to ensure
471 # dbus querries are processed
472- rebuild_database(pathname)
473+ rebuild_database(get_database_path())
474 finally:
475 # signal that the xapian db is valid again
476 if dbus_controller:
477
478=== modified file 'utils/wildcard_query_parser.py'
479--- utils/wildcard_query_parser.py 2010-05-09 07:13:26 +0000
480+++ utils/wildcard_query_parser.py 2010-06-02 15:09:30 +0000
481@@ -9,6 +9,7 @@
482 sys.path.insert(0, "../")
483 from softwarecenter.enums import *
484 from softwarecenter.utils import *
485+from softwarecenter.db.database import get_database_path
486
487 if __name__ == "__main__":
488
489@@ -18,8 +19,7 @@
490 help="print found apps/pkgs too")
491 (options, args) = parser.parse_args()
492
493- pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
494- db = xapian.Database(pathname)
495+ db = xapian.Database(get_database_path())
496
497 axi = xapian.Database("/var/lib/apt-xapian-index/index")
498 db.add_database(axi)