Merge lp:~osomon/software-center/one_database_per_language into lp:software-center

[Olivier Tilloy]

This branch intends to fix bug #434601 by implementing the solution proposed by mpt in comment #1: store one database per language code, since the contents of the database is localized. Where the database was previously stored in a "xapian" directory on disk, it is now stored in "xapian.{lang_code}", e.g. "xapian.fr_FR".

Comments on this approach are welcome.

[Michael Vogt (mvo)]

Thanks for the patch. It will need similar logic in utils/update-sofware-center when it rebuilds the database. Best is probably to consolidate the name with language added in one place.

review: Needs Fixing

[Olivier Tilloy (osomon)]

Right, I overlooked this. I will re-factor the patch to have the database name computed in one place only.

[Michael Vogt (mvo)]

Thanks, this is good now. I wonder how we should deal with the following case though:
1. system locale is en_US and update-software-center creates /var/cache/software-center/xapian.en_US
2. a normal user has de_DE and runs software-center
   - it will not be able to create /var/cache/software-center
   - from looking at the code it does not currently fallback to the existing one

Ideally I guess we would have a dbus service that simply re-builds it in the language we want.

[Michael Vogt (mvo)]

I pushed a branch with some minor modifications (transition code in postinst) to lp:~mvo/software-center/one_database_per_language/

[Olivier Tilloy (osomon)]

Indeed there is currently no fallback mechanism. The DBus service solution sounds good to me, but in the short term we can implement a simple fallback mechanism that will pick the first database found in /var/cache/software-center.

[lp:~osomon/software-center/one_database_per_language]

749. By Olivier Tilloy on 2010-06-02

Merged from trunk.

750. By Olivier Tilloy on 2010-06-02

Merged Michael's changes to the post-installation script.

751. By Olivier Tilloy on 2010-06-02

When the database path for the current locale doesn't exist
or cannot be created, try to fall back gracefully to an existing database.

[Olivier Tilloy (osomon)]

I have merged Michael's changes and implemented a mechanism that should ensure we always try to fall back to an existing database when one cannot be created. Comments appreciated!

[Olivier Tilloy (osomon)]

This branch is too old and doesn’t merge cleanly any longer.

@Michael: do you think the original approach would still work? If so, I could take a stab at refreshing the code to apply to the current code base.

[Michael Vogt (mvo)]

On Mon, May 23, 2011 at 08:08:27AM -0000, Olivier Tilloy wrote:
> This branch is too old and doesn’t merge cleanly any longer.
>
> @Michael: do you think the original approach would still work? If so, I could take a stab at refreshing the code to apply to the current code base.
> --
> https://code.launchpad.net/~osomon/software-center/one_database_per_language/+merge/23870
> You are reviewing the proposed merge of lp:~osomon/software-center/one_database_per_language into lp:software-center.

I think the general approach is still good and sound, however I would
rather like to see your talents used in the new softwarecener/ui/qml
part of the source ;) Its currently empty, but there is a
lp:~mvo/software-center/refactor-with-qml branch already that
hopefully can move to trunk soonish.

Cheers,
 Michael

[Olivier Tilloy (osomon)]

> I think the general approach is still good and sound, however I would
> rather like to see your talents used in the new softwarecener/ui/qml
> part of the source ;) Its currently empty, but there is a
> lp:~mvo/software-center/refactor-with-qml branch already that
> hopefully can move to trunk soonish.

Sure, I’m also more interested in playing with the UI!
I only meant to do a bit of cleanup in pending tasks, this one has been pending forever and in view of the recent activity on the related bug report, it looks like it remains an important issue for users.

[dobey (dobey)]

Setting this to Rejected since it's quite old, and hasn't been commented on in 2 years, or touched in almost 3.

Unmerged revisions

751. By Olivier Tilloy on 2010-06-02

When the database path for the current locale doesn't exist
or cannot be created, try to fall back gracefully to an existing database.

750. By Olivier Tilloy on 2010-06-02

Merged Michael's changes to the post-installation script.

749. By Olivier Tilloy on 2010-06-02

Merged from trunk.

748. By Olivier Tilloy on 2010-05-07

Use get_database_path everywhere to instantiate the DB.

747. By Olivier Tilloy on 2010-05-07

Merged from trunk.

746. By Olivier Tilloy on 2010-05-07

Separate function to compute the database path on disk, depending on the current locale.

745. By Olivier Tilloy on 2010-05-06

Merged from trunk.

744. By Olivier Tilloy on 2010-04-21

Handle nicely broken locales: fall back to C.

743. By Olivier Tilloy on 2010-04-21

Store one database per language code.

=== modified file '.bzrignore'
--- .bzrignore	2009-09-16 16:37:16 +0000
+++ .bzrignore	2010-06-02 15:09:30 +0000
@@ -1,14 +1,1 @@
-data/xapian/flintlock
-data/xapian/iamflint
-data/xapian/postlist.baseA
-data/xapian/postlist.baseB
-data/xapian/postlist.DB
-data/xapian/record.baseA
-data/xapian/record.baseB
-data/xapian/record.DB
-data/xapian/termlist.baseA
-data/xapian/termlist.baseB
-data/xapian/termlist.DB
-data/xapian/value.baseA
-data/xapian/value.baseB
-data/xapian/value.DB
+data/xapian.*
 
=== removed directory 'data/xapian'
=== modified file 'debian/software-center.postinst'
--- debian/software-center.postinst	2010-03-11 16:15:42 +0000
+++ debian/software-center.postinst	2010-06-02 15:09:30 +0000
@@ -12,6 +12,11 @@
     exit 0
 fi
 
+# in 2.1.2 we moved to one db per language so we can remove
+# the old one
+if dpkg --compare-versions "$2" lt-nl "2.1.2"; then
+    rm -rf /var/cache/software-center/xapian
+fi
 
 
 
 
=== modified file 'softwarecenter/app.py'
--- softwarecenter/app.py	2010-05-28 07:35:33 +0000
+++ softwarecenter/app.py	2010-06-02 15:09:30 +0000
@@ -36,7 +36,7 @@
 from softwarecenter.enums import *
 from softwarecenter.utils import *
 from softwarecenter.version import *
-from softwarecenter.db.database import StoreDatabase
+from softwarecenter.db.database import get_database_path, StoreDatabase
 
 import view.dialogs
 from view.viewswitcher import ViewSwitcher, ViewSwitcherList
@@ -120,8 +120,7 @@
         self.backend.connect("transaction-stopped", self._on_transaction_stopped)
         self.backend.connect("channels-changed", self.on_channels_changed)
 
-        # xapian
-        pathname = os.path.join(xapian_base_path, "xapian")
+        pathname = get_database_path(xapian_base_path)
         try:
             self.db = StoreDatabase(pathname, self.cache)
             self.db.open()
 
=== modified file 'softwarecenter/db/database.py'
--- softwarecenter/db/database.py	2010-05-18 11:14:59 +0000
+++ softwarecenter/db/database.py	2010-06-02 15:09:30 +0000
@@ -16,9 +16,12 @@
 # this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 
+import errno
+import glob
 import gobject
 import locale
 import logging
+import os
 import re
 import xapian
 
@@ -28,6 +31,59 @@
 from softwarecenter.enums import *
 from gettext import gettext as _
 
+
+def _fallback_existing_database(xapian_base_path):
+    # Try to locate an existing database to fall back on.
+    matches = glob.glob(os.path.join(xapian_base_path, "xapian*"))
+    if matches:
+        return matches[0]
+    else:
+        return None
+
+
+def get_database_path(xapian_base_path=XAPIAN_BASE_PATH, create=True):
+    """
+    Return the path to the database for the current locale.
+    If the path doesn't exist yet and the create parameter is True (default),
+    it is created. If creation fails or if the create parameter is False,
+    try to fall back gracefully to an existing database. If that fails, raise
+    an exception.
+    """
+    # Store one database per language code, since the contents of the database
+    # are localized ("xapian.{lang_code}", e.g. "xapian.fr_FR").
+    try:
+        lang_code, encoding = locale.getlocale()
+    except ValueError:
+        lang_code = None
+    if lang_code is None:
+        lang_code = "C"
+    db_name = "xapian.%s" % lang_code
+    pathname = os.path.join(xapian_base_path, db_name)
+    if os.path.exists(pathname):
+        return pathname
+    elif create:
+        try:
+            os.makedirs(pathname)
+        except OSError:
+            # The database cannot be created, most likely because
+            # software-center is being run installed by an unprivileged
+            # user. Try to fall back gracefully to an existing database.
+            pathname = _fallback_existing_database(xapian_base_path)
+            if pathname is not None:
+                return pathname
+            else:
+                raise
+        else:
+            return pathname
+    else:
+        # Try to fall back gracefully to an existing database.
+        fallback = _fallback_existing_database(xapian_base_path)
+        if fallback is not None:
+            return fallback
+        else:
+            raise IOError(errno.ENOENT, os.strerror(errno.ENOENT), pathname)
+
+
 class StoreDatabase(gobject.GObject):
     """thin abstraction for the xapian database with convenient functions"""
 
 
=== modified file 'softwarecenter/view/appview.py'
--- softwarecenter/view/appview.py	2010-05-30 16:21:33 +0000
+++ softwarecenter/view/appview.py	2010-06-02 15:09:30 +0000
@@ -39,7 +39,7 @@
     sys.path.insert(0, ".")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
-from softwarecenter.db.database import StoreDatabase, Application
+from softwarecenter.db.database import get_database_path, StoreDatabase, Application
 from softwarecenter.backend import get_install_backend
 from softwarecenter.distro import get_distro
 
@@ -1464,13 +1464,10 @@
 if __name__ == "__main__":
     logging.basicConfig(level=logging.DEBUG)
 
-    xapian_base_path = XAPIAN_BASE_PATH
-    pathname = os.path.join(xapian_base_path, "xapian")
-
     # the store
     from softwarecenter.apt.aptcache import AptCache
     cache = AptCache()
-    db = StoreDatabase(pathname, cache)
+    db = StoreDatabase(get_database_path(), cache)
     db.open()
 
     # additional icons come from app-install-data
 
=== modified file 'softwarecenter/view/availablepane.py'
--- softwarecenter/view/availablepane.py	2010-05-30 15:53:11 +0000
+++ softwarecenter/view/availablepane.py	2010-06-02 15:09:30 +0000
@@ -595,8 +595,6 @@
 
 if __name__ == "__main__":
     #logging.basicConfig(level=logging.DEBUG)
-    xapian_base_path = XAPIAN_BASE_PATH
-    pathname = os.path.join(xapian_base_path, "xapian")
 
     if len(sys.argv) > 1:
         datadir = sys.argv[1]
@@ -605,7 +603,8 @@
     else:
         datadir = "/usr/share/software-center"
 
-    db = xapian.Database(pathname)
+    from softwarecenter.db.database import get_database_path
+    db = xapian.Database(get_database_path())
     icons = gtk.icon_theme_get_default()
     icons.append_search_path("/usr/share/app-install/icons/")
 
 
=== modified file 'softwarecenter/view/channelpane.py'
--- softwarecenter/view/channelpane.py	2010-05-30 17:11:05 +0000
+++ softwarecenter/view/channelpane.py	2010-06-02 15:09:30 +0000
@@ -219,8 +219,6 @@
 
 if __name__ == "__main__":
     #logging.basicConfig(level=logging.DEBUG)
-    xapian_base_path = XAPIAN_BASE_PATH
-    pathname = os.path.join(xapian_base_path, "xapian")
 
     if len(sys.argv) > 1:
         datadir = sys.argv[1]
@@ -229,7 +227,8 @@
     else:
         datadir = "/usr/share/software-center"
 
-    db = xapian.Database(pathname)
+    from softwarecenter.db.database import get_database_path
+    db = xapian.Database(get_database_path())
     icons = gtk.icon_theme_get_default()
     icons.append_search_path("/usr/share/app-install/icons/")
     cache = apt.Cache(apt.progress.text.OpProgress())
 
=== modified file 'softwarecenter/view/historypane.py'
--- softwarecenter/view/historypane.py	2010-05-19 16:01:48 +0000
+++ softwarecenter/view/historypane.py	2010-06-02 15:09:30 +0000
@@ -36,7 +36,7 @@
 from softwarecenter.enums import *
 from softwarecenter.view.widgets.searchentry import SearchEntry
 from softwarecenter.apt.aptcache import AptCache
-from softwarecenter.db.database import StoreDatabase
+from softwarecenter.db.database import get_database_path, StoreDatabase
 
 
 class HistoryPane(gtk.VBox):
@@ -287,9 +287,7 @@
 
 if __name__ == '__main__':
     cache = AptCache()
-
-    db_path = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = StoreDatabase(db_path, cache)
+    db = StoreDatabase(get_database_path(), cache)
     db.open()
 
     icons = gtk.icon_theme_get_default()
 
=== modified file 'softwarecenter/view/installedpane.py'
--- softwarecenter/view/installedpane.py	2010-05-31 07:24:21 +0000
+++ softwarecenter/view/installedpane.py	2010-06-02 15:09:30 +0000
@@ -177,8 +177,6 @@
 
 if __name__ == "__main__":
     #logging.basicConfig(level=logging.DEBUG)
-    xapian_base_path = XAPIAN_BASE_PATH
-    pathname = os.path.join(xapian_base_path, "xapian")
 
     if len(sys.argv) > 1:
         datadir = sys.argv[1]
@@ -187,7 +185,8 @@
     else:
         datadir = "/usr/share/software-center"
 
-    db = xapian.Database(pathname)
+    from softwarecenter.db.database import get_database_path
+    db = xapian.Database(get_database_path())
     icons = gtk.icon_theme_get_default()
     icons.append_search_path("/usr/share/app-install/icons/")
     cache = apt.Cache(apt.progress.text.OpProgress())
 
=== modified file 'softwarecenter/view/viewswitcher.py'
--- softwarecenter/view/viewswitcher.py	2010-05-25 00:48:21 +0000
+++ softwarecenter/view/viewswitcher.py	2010-06-02 15:09:30 +0000
@@ -34,7 +34,7 @@
 from softwarecenter.backend.channel import SoftwareChannel
 from softwarecenter.backend import get_install_backend
 from softwarecenter.distro import get_distro
-from softwarecenter.db.database import StoreDatabase
+from softwarecenter.db.database import get_database_path, StoreDatabase
 from softwarecenter.enums import *
 
 from widgets.animatedimage import CellRendererAnimatedImage, AnimatedImage
@@ -403,10 +403,8 @@
     scroll = gtk.ScrolledWindow()
     icons = gtk.icon_theme_get_default()
 
-    xapian_base_path = XAPIAN_BASE_PATH
-    pathname = os.path.join(xapian_base_path, "xapian")
     cache = apt.Cache(apt.progress.text.OpProgress())
-    db = StoreDatabase(pathname, cache)
+    db = StoreDatabase(get_database_path(), cache)
     db.open()
 
     view = ViewSwitcher(datadir, db, icons)
 
=== modified file 'test/test_appview.py'
--- test/test_appview.py	2010-05-20 18:33:08 +0000
+++ test/test_appview.py	2010-06-02 15:09:30 +0000
@@ -10,9 +10,8 @@
 
 from softwarecenter import Application
 from softwarecenter.apt.aptcache import AptCache
-from softwarecenter.db.database import StoreDatabase
+from softwarecenter.db.database import get_database_path, StoreDatabase
 from softwarecenter.view.appview import AppStore
-from softwarecenter.enums import *
 
 import xapian
 
@@ -30,10 +29,8 @@
     """ tests the AppStore GtkTreeViewModel """
 
     def setUp(self):
-        xapian_base_path = XAPIAN_BASE_PATH
-        pathname = os.path.join(xapian_base_path, "xapian")
         self.cache = AptCache()
-        self.db = StoreDatabase(pathname, self.cache)
+        self.db = StoreDatabase(get_database_path(), self.cache)
         self.db.open()
         self.mock_icons = MockIconCache()
         self.mock_filter = MockAppViewFilter()
 
=== modified file 'utils/bench.py'
--- utils/bench.py	2010-05-09 07:13:26 +0000
+++ utils/bench.py	2010-06-02 15:09:30 +0000
@@ -7,6 +7,7 @@
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
+from softwarecenter.db.database import get_database_path
 
 def run_benchmark(db):
     # test postlist
@@ -37,8 +38,7 @@
 
 if __name__ == "__main__":
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
 
     print "app db only"
     run_benchmark(db)
 
=== modified file 'utils/installedapps.py'
--- utils/installedapps.py	2010-05-09 07:13:26 +0000
+++ utils/installedapps.py	2010-06-02 15:09:30 +0000
@@ -8,13 +8,13 @@
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
+from softwarecenter.db.database import get_database_path
 
 if __name__ == "__main__":
 
     cache = apt.Cache()
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
 
     installed = []
     for m in db.postlist(""):
 
=== modified file 'utils/query.py'
--- utils/query.py	2010-05-09 07:13:26 +0000
+++ utils/query.py	2010-06-02 15:09:30 +0000
@@ -9,6 +9,7 @@
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
+from softwarecenter.db.database import get_database_path
 
 def parse_query(parser, search_strings, verbose=True):
     str_to_prefix = { 'section' : 'AE',
@@ -52,8 +53,7 @@
                       help="print found apps/pkgs too")
     (options, args) = parser.parse_args()
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
 
     axi = xapian.Database("/var/lib/apt-xapian-index/index")
     db.add_database(axi)
 
=== modified file 'utils/search_query.py'
--- utils/search_query.py	2010-05-09 07:13:26 +0000
+++ utils/search_query.py	2010-06-02 15:09:30 +0000
@@ -7,7 +7,6 @@
 from optparse import OptionParser
 
 sys.path.insert(0, "../")
-from softwarecenter.enums import *
 from softwarecenter.utils import *
 
 def run_query(parser, search_terms, verbose):
@@ -37,8 +36,8 @@
                       help="print found apps/pkgs too")
     (options, args) = parser.parse_args()
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    from softwarecenter.db.database import get_database_path
+    db = xapian.Database(get_database_path())
 
     axi = xapian.Database("/var/lib/apt-xapian-index/index")
     db.add_database(axi)
 
=== modified file 'utils/stats.py'
--- utils/stats.py	2010-05-09 07:13:26 +0000
+++ utils/stats.py	2010-06-02 15:09:30 +0000
@@ -24,6 +24,7 @@
 
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
+from softwarecenter.db.database import get_database_path
 
 if __name__ == "__main__":
 
@@ -35,8 +36,7 @@
     app_to_pkg = {}
 
     # gather data
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
     for m in db.postlist(""):
         doc = db.get_document(m.docid)
         appname = doc.get_data()
 
=== modified file 'utils/topapps.py'
--- utils/topapps.py	2010-05-09 07:13:26 +0000
+++ utils/topapps.py	2010-06-02 15:09:30 +0000
@@ -8,6 +8,7 @@
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
+from softwarecenter.db.database import get_database_path
 
 if __name__ == "__main__":
 
@@ -15,8 +16,7 @@
     if len(sys.argv) > 1:
         topn = int(sys.argv[1])
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
 
     heap = []
     for m in db.postlist(""):
 
=== modified file 'utils/update-software-center'
--- utils/update-software-center	2010-05-09 07:13:26 +0000
+++ utils/update-software-center	2010-06-02 15:09:30 +0000
@@ -31,6 +31,7 @@
 from optparse import OptionParser
 
 from softwarecenter.enums import *
+from softwarecenter.db.database import get_database_path
 from softwarecenter.db.update import rebuild_database
 
 # dbus may not be available during a upgrade so we 
@@ -97,7 +98,7 @@
             logging.info("no translation information in database needed")
             sys.exit(0)
         mo_time = os.path.getctime(mofile)
-        pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
+        pathname = get_database_path(create=False)
         if os.path.exists(pathname):
             db = xapian.Database(pathname)
             mo_db_time = db.get_metadata("app-install-mo-time")
@@ -120,13 +121,9 @@
 
     # rebuild and send signal when done
     try:
-        # setup path
-        pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-        if not os.path.exists(pathname):
-            os.makedirs(pathname)
         # rebuild the database, the default context is run to ensure
         # dbus querries are processed
-        rebuild_database(pathname)
+        rebuild_database(get_database_path())
     finally:
         # signal that the xapian db is valid again
         if dbus_controller:
 
=== modified file 'utils/wildcard_query_parser.py'
--- utils/wildcard_query_parser.py	2010-05-09 07:13:26 +0000
+++ utils/wildcard_query_parser.py	2010-06-02 15:09:30 +0000
@@ -9,6 +9,7 @@
 sys.path.insert(0, "../")
 from softwarecenter.enums import *
 from softwarecenter.utils import *
+from softwarecenter.db.database import get_database_path
 
 if __name__ == "__main__":
 
@@ -18,8 +19,7 @@
                       help="print found apps/pkgs too")
     (options, args) = parser.parse_args()
 
-    pathname = os.path.join(XAPIAN_BASE_PATH, "xapian")
-    db = xapian.Database(pathname)
+    db = xapian.Database(get_database_path())
 
     axi = xapian.Database("/var/lib/apt-xapian-index/index")
     db.add_database(axi)