Merge lp:~izidor/gtg/performance into lp:~gtg/gtg/old-trunk

[Izidor Matušov]

I finally made my proposed performance optimization and hopefully, solve all new bugs I introduced. I did have to make changes to liblarch and there is a merge request as well: https://github.com/liblarch/liblarch/pull/2

How to test it? Run following commands:

mkdir fast-gtg && cd fast-gtg/
bzr branch lp:~izidor/gtg/performance gtg
git clone --branch=performance https://github.com/liblarch/liblarch liblarch
cd gtg/
./scripts/import_my_tasks_into_debug_tasks.sh
./scripts/debug.sh -l

Changes I made:
- added a new parameter -l which makes GTG to prefer a local installation of liblarch in ../liblarch rather than the system installation

- parameter -b works again! Run "time ./scripts/debug.sh -l -b" to test how fast your GTG boot up

- background colors are cached instead of recomputing every time. It makes GTG run smoother.

- quiet boot: GTG window is shown only when all tasks are loaded. By the size of your tasks, GTG would start in few seconds:

150 tasks: 4.6 seconds
My 450+ tasks: 10 seconds
Legendary Bryce set: 21 seconds

(The quiet boot should be okay because applications like Firefox takes some time to start too.)

- several improvement on side of liblarch

After merging this patch, GTG won't be perfectly fast. There is still room for improvement but it would make GTG more useable.

[Bertrand Rousseau (bertrand-rousseau)]

Wow, the performance boost is amazing! It's great to have GTG feeling snappy! Congrats Izidor, you've done a great work there!

The quiet boot option is ok it's WAY better than having the tasks and tags appearing progressively, and still not be able to interact with GTG. I guess we could expect not to have more bryce's task count (I expect most people to have less than 100 tasks - btw, we could make a survey about this sometimes), so it should be allright. If it bothers people, there are ways to help with this (e.g. popup with loading bar, show GTG brower but make the lists inactive, etc.)

For me, your GTG patch is good to go!

review: Approve

[Xuan (Sean) Hu (huxuan)]

Really really fantastic!

I do see huge preformance improvement on my laptop with bryce's dataset.
Cause I even can not open bryce before (My laptop is a little old too) but now it's able to see all tasks even with tag sidebar and modification. It's sure to be more useable now.

A little problem is that there will be "* active tasks -" in the window title before "Getting Things GNOME!", but it's missing in this branch.

I looked into the code and found the window_title is updated by GTG.gtk.browser.browser._update_windows_title
And it's only called by callback of 'node-added-inview' and 'node-deleted-inview', maybe there need one more callback something like 'node-loaded-inview'

[lp:~izidor/gtg/performance]

1182. By Izidor Matušov on 2012-05-04

Refresh title on start

[Izidor Matušov (izidor)]

Xuan> thanks for the notice about the title! I updated code and the title is manually refreshed on startup - no regression anymore. Would anybody find other glitches?

=== modified file 'CHANGELOG'
--- CHANGELOG	2012-04-22 15:16:24 +0000
+++ CHANGELOG	2012-05-04 06:22:17 +0000
@@ -18,6 +18,7 @@
     * Reimplement the tag context menu as a widget, in order to - hopefully - be able to implement an enhanced context menu with, for instance, a color picker.
     * Background Colors option was moved from View menu into preferences dialog
     * Reorganised notification area menu (New task, Show browser, <tasks>, Quit)
+    * Added an option to import local (development) version of liblarch instead of system installed one
 
 2012-02-13 Getting Things GNOME! 0.2.9
     * Big refractorization of code, now using liblarch

=== modified file 'GTG/core/datastore.py'
--- GTG/core/datastore.py	2012-03-22 14:19:18 +0000
+++ GTG/core/datastore.py	2012-05-04 06:22:17 +0000
@@ -39,7 +39,6 @@
 from GTG.backends.genericbackend import GenericBackend
 from GTG.tools                   import cleanxml
 from GTG.backends.backendsignals import BackendSignals
-from GTG.tools.synchronized      import synchronized
 from GTG.tools.borg              import Borg
 from GTG.core.search             import parse_search_query, search_filter, InvalidQuery
 
@@ -334,7 +333,6 @@
         self.__tasks.add_node(task)
         return task
 
-    @synchronized
     def push_task(self, task):
         '''
         Adds the given task object to the task tree. In other words, registers
@@ -647,19 +645,8 @@
         self.to_set_timer = None
 
     def start_get_tasks(self):
-        ''''
-        Maps the TaskSource to the backend and starts threading.
-        '''
-        self.start_get_tasks_thread = \
-             threading.Thread(target=self.__start_get_tasks)
-        self.start_get_tasks_thread.setDaemon(True)
-        self.start_get_tasks_thread.start()
-
-    def __start_get_tasks(self):
-        '''
-        Loads all task from the backend and connects its signals afterwards.
-        Launched as a thread by start_get_tasks
-        '''
+        """ Loads all task from the backend and connects its signals
+        afterwards. """
         self.backend.start_get_tasks()
         self._connect_signals()
         if self.backend.is_default():

=== modified file 'GTG/core/plugins/api.py'
--- GTG/core/plugins/api.py	2012-04-22 12:29:43 +0000
+++ GTG/core/plugins/api.py	2012-05-04 06:22:17 +0000
@@ -177,20 +177,19 @@
                           "%s" % e)
 
     def set_bgcolor_func(self, func=None):
-        """ Set a function which defines a background color for each task """
-        # NOTE: this function is strongly dependend of browser structure
-        # after refaractoring browser, this might need to review
+        """ Set a function which defines a background color for each task
+        
+        NOTE: This function stronglye depend on browser and could be easily
+        broken by changes in browser code
+        """
         browser = self.__ui
 
         # set default bgcolor?
         if func is None:
             func = browser.tv_factory.task_bg_color
-            info_col = 'tags'
-        else:
-            info_col = 'task_id'
 
         for pane in browser.vtree_panes.itervalues():
-            pane.set_bg_color(func, info_col)
+            pane.set_bg_color(func, 'bg_color')
             pane.basetree.get_basetree().refresh_all()
 
 #=== file saving/loading ======================================================

=== modified file 'GTG/gtk/browser/browser.py'
--- GTG/gtk/browser/browser.py	2012-04-23 21:50:54 +0000
+++ GTG/gtk/browser/browser.py	2012-05-04 06:22:17 +0000
@@ -93,7 +93,7 @@
 
         # Set up models
         # Active Tasks
-        self.activetree.apply_filter('active',refresh=False)
+        self.activetree.apply_filter('active')
         # Tags
         self.tagtree = None
         self.tagtreeview = None
@@ -131,6 +131,7 @@
         #Update the title when a task change
         self.activetree.register_cllbck('node-added-inview', self._update_window_title)
         self.activetree.register_cllbck('node-deleted-inview', self._update_window_title)
+        self._update_window_title()
 
 ### INIT HELPER FUNCTIONS #####################################################
 #
@@ -209,6 +210,9 @@
             self.on_tag_treeview_key_press_event)
         self.sidebar_container.add(self.tagtreeview)
 
+        # Refresh tree 
+        self.tagtree.reset_filters(transparent_only=True)
+
         # expanding search tag does not work automatically, request it
         self.expand_search_tag()
 

=== modified file 'GTG/gtk/browser/treeview_factory.py'
--- GTG/gtk/browser/treeview_factory.py	2012-04-22 14:06:51 +0000
+++ GTG/gtk/browser/treeview_factory.py	2012-05-04 06:22:17 +0000
@@ -85,9 +85,9 @@
                     real_count = real_count + 1
         return display_count < real_count
     
-    def task_bg_color(self,tags,bg):
+    def task_bg_color(self, node, default_color):
         if self.config.get('bg_color_enable'):
-            return colors.background_color(tags,bg)
+            return colors.background_color(node.get_tags(), default_color)
         else:
             return None
     
@@ -424,6 +424,13 @@
         col['order'] = 0
         desc[col_name] = col
 
+        #invisible 'bg_color' column
+        col_name = 'bg_color'
+        col = {}
+        col['value'] = [str, lambda node: None]
+        col['visible'] = False
+        desc[col_name] = col
+
         #invisible 'title' column
         col_name = 'title'
         col = {}
@@ -463,6 +470,7 @@
         desc[col_name] = col
         return desc
         
+    
     def build_task_treeview(self,tree,desc):
         treeview = AutoExpandTreeView(tree,desc)
         #Now that the treeview is done, we can polish
@@ -470,7 +478,7 @@
         treeview.set_expander_column('label')
         treeview.set_dnd_name('gtg/task-iter-str')
         #Background colors
-        treeview.set_bg_color(self.task_bg_color,'tags')
+        treeview.set_bg_color(self.task_bg_color, 'bg_color')
          # Global treeview properties
         treeview.set_property("enable-tree-lines", False)
         treeview.set_rules_hint(False)

=== modified file 'GTG/plugins/urgency_color/urgency_color.py'
--- GTG/plugins/urgency_color/urgency_color.py	2012-03-17 02:20:46 +0000
+++ GTG/plugins/urgency_color/urgency_color.py	2012-05-04 06:22:17 +0000
@@ -53,8 +53,7 @@
         else:
             return None
 
-    def bgcolor(self, node_id, standard_color):
-        node = self.req.get_task(node_id)
+    def bgcolor(self, node, standard_color):
         sdate = node.get_start_date()
         ddate = node.get_due_date()
         if (sdate != Date.no_date() != ddate):

=== modified file 'GTG/tools/import_liblarch.py'
--- GTG/tools/import_liblarch.py	2012-04-23 04:16:49 +0000
+++ GTG/tools/import_liblarch.py	2012-05-04 06:22:17 +0000
@@ -25,11 +25,11 @@
 REQUIRED_LIBLARCH_API = "1.0"
 GIT_CMD = "git clone https://github.com/liblarch/liblarch ../liblarch"
 
-
-def import_liblarch():
+def import_liblarch(use_local=False):
     """ Check if liblarch is installed and is compatible
 
-    If not, provide information how to obtain the newest version """
+    If not, provide information how to obtain the newest version.
+    If use_local, prioritize local (development) liblarch in ../liblarch"""
 
     def check_liblarch():
         """ Import liblarch and find out which one is missing """
@@ -49,8 +49,12 @@
 
         return has_libraries, " and ".join(missing)
 
+    if use_local:
+        sys.path.insert(0, "../liblarch")
+
     has_libraries, missing = check_liblarch()
-    if not has_libraries:
+
+    if not use_local and not has_libraries:
         sys.path.append("../liblarch/")
         has_libraries, missing = check_liblarch()
 
@@ -83,7 +87,9 @@
     return True
 
 if __name__ == "__main__":
-    if import_liblarch():
+    use_local = "-l" in sys.argv[1:] or "--local-liblarch" in sys.argv[1:]
+
+    if import_liblarch(use_local):
         sys.exit(0)
     else:
         sys.exit(1)

=== removed file 'GTG/tools/synchronized.py'
--- GTG/tools/synchronized.py	2012-03-05 15:23:05 +0000
+++ GTG/tools/synchronized.py	1970-01-01 00:00:00 +0000
@@ -1,14 +0,0 @@
-from __future__ import with_statement
-from threading import Lock
-
-def synchronized(fun):
-    the_lock = Lock()
-
-    def fwrap(function):
-        def newFunction(*args, **kw):
-            with the_lock:
-                return function(*args, **kw)
-
-        return newFunction
-
-    return fwrap(fun)

=== modified file 'gtg'
--- gtg	2012-04-30 05:56:11 +0000
+++ gtg	2012-05-04 06:22:17 +0000
@@ -50,6 +50,9 @@
       default=False)
     parser.add_option('-d', '--debug', action='store_true', dest='debug',
       help="Enable debug output", default=False)
+    parser.add_option('-l', '--local-liblarch', action='store_true',
+      dest='local_liblarch', default=False,
+      help="Use local liblarch located in ../liblarch if it is posible")
     parser.add_option('-v', '--version', action='store_true',
       dest='print_version', help="Print GTG's version number", default=False)
     (options, args) = parser.parse_args()
@@ -64,7 +67,7 @@
         print "Could not open X display"
         sys.exit(1)
 
-    elif import_liblarch():
+    elif import_liblarch(options.local_liblarch):
         from GTG import gtg
         sys.exit(gtg.main(options, args))
 

=== modified file 'scripts/debug.sh'
--- scripts/debug.sh	2012-03-26 17:43:15 +0000
+++ scripts/debug.sh	2012-05-04 06:22:17 +0000
@@ -15,14 +15,18 @@
 mkdir -p tmp
 
 # Interpret arguments
-while getopts bdnps: o
+while getopts bdlnps: o
 do  case "$o" in
     b)   args="$args --boot-test";;
     d)   args="$args -d";;
+    # Request usage local liblarch if it is possible
+    l)   args="$args -l"
+         liblarchArgs="$liblarchArgs -l"
+        ;;
     n)   norun=1;;
     p)   profile=1;;
     s)   set="$OPTARG";;
-    [?]) echo >&2 "Usage: $0 [-s dataset] [-b] [-d] [-n] [-p]"
+    [?]) echo >&2 "Usage: $0 [-s dataset] [-b] [-d] [-l] [-n] [-p]"
          exit 1;;
     esac
 done
@@ -46,19 +50,19 @@
     export XDG_CONFIG_HOME="./tmp/default/xdg/config"
 fi
 
-# Check for liblarch
-if ! ./GTG/tools/import_liblarch.py ; then
-    echo
-    echo -n "Download latest liblarch? [y/N] "
-    read answer
-    if [ "$answer" = "y" -o "$answer" = "Y" -o "$answer" = "yes" ]; then
-        git clone https://github.com/liblarch/liblarch ../liblarch
-    else
-        exit 1
-    fi
-fi
-
 if [ $norun -eq 0 ]; then
+    # Check for liblarch
+    if ! ./GTG/tools/import_liblarch.py $liblarchArgs; then
+        echo
+        echo -n "Download latest liblarch? [y/N] "
+        read answer
+        if [ "$answer" = "y" -o "$answer" = "Y" -o "$answer" = "yes" ]; then
+            git clone https://github.com/liblarch/liblarch ../liblarch
+        else
+            exit 1
+        fi
+    fi
+
     if [ $profile -eq 1 ]; then
 	python -m cProfile -o gtg.prof ./gtg $args
     python ./scripts/profile_interpret.sh