Merge lp:~brian-murray/arsenal/date_last_updated into lp:arsenal/trunk

[Brian Murray]

Loading the bug data for bugs that haven't changed is really unnecessary. This branch checks the date_last_updated for a bug in the json file and sees if the bug's date_last_updated is more recent than it, if it is then the bug data is retrieved.

I wanted to have another person look at it before merging it - just in case.

[Bryce Harrington (bryce)]

I'm not super familiar with this script yet, but conceptually the new options sound sensible. +1 from me.

review: Approve

[Brad Figg (brad-figg)]

On 11/29/2011 06:49 PM, Bryce Harrington wrote:
> Review: Approve
>
> I'm not super familiar with this script yet, but conceptually the new options sound sensible. +1 from me.

I'm not sure how you bzr folks normally deal with these
requests and reviews. I've emailed Brian separately with
some questions/suggestions. Overall, this is a good fix.
I'd almost prefer it to be the default rather than a
command line option but it's good either way.

Brad
--
Brad Figg <email address hidden> http://www.canonical.com

[Bryce Harrington (bryce)]

On Wed, Nov 30, 2011 at 02:58:24AM -0000, Brad Figg wrote:
> On 11/29/2011 06:49 PM, Bryce Harrington wrote:
> > Review: Approve
> >
> > I'm not super familiar with this script yet, but conceptually the new options sound sensible. +1 from me.
>
> I'm not sure how you bzr folks normally deal with these
> requests and reviews. I've emailed Brian separately with
> some questions/suggestions. Overall, this is a good fix.

If you go to code.launchpad.net/arsenal it shows all the branches. The
ones with the detour sign icon are ones that someone's proposed a merge
on.

  https://code.launchpad.net/arsenal/

If you click on the 'active reviews' link it shows you the merge
proposals relevant to you for reviewing.

  https://code.launchpad.net/arsenal/+activereviews

Click on the merge proposal you want to review. The green 'Diff against
target' link is useful to do a quick review of the diff. You can add a
comment and mark status to 'Approved', 'Needs Fixing', etc.

> I'd almost prefer it to be the default rather than a
> command line option but it's good either way.

I'm definitely open to trying it as the default. If it reduces our
cronjob loads against LP, that can only be a good thing.

Bryce

[Brian Murray (brian-murray)]

I've set updated_only to be the default and have merged my changes to trunk. Thanks for the feedback!

=== modified file 'reports/cbd'
--- reports/cbd	2011-11-21 18:01:57 +0000
+++ reports/cbd	2011-11-29 23:29:24 +0000
@@ -72,6 +72,9 @@
         stdo('                         contains the mappings from packages to teams and from               \n')
         stdo('                         milestone to milestone-date.                                        \n')
         stdo('                                                                                             \n')
+        stdo('        --privates      Include private bugs in the results.                                 \n')
+        stdo('                                                                                             \n')
+        stdo('        --updated-only  Only retrieve bug data for bugs which have been recently updated.    \n')
         stdo('                                                                                             \n')
         stdo('    Examples:                                                                                \n')
         stdo('        %s iso-testing-bugs.json                                                             \n' % defaults['app_name'])
@@ -88,7 +91,7 @@
         try:
             cfg = defaults
             optsShort = ''
-            optsLong  = ['help', 'debug=', 'cached-lp-resources=', 'quiet']
+            optsLong  = ['help', 'debug=', 'cached-lp-resources=', 'quiet', 'privates', 'updated-only']
             opts, args = getopt(argv[1:], optsShort, optsLong)
 
             for opt, val in opts:
@@ -104,6 +107,12 @@
                         if level not in Dbg.levels:
                             Dbg.levels.append(level)
 
+                elif opt in ('--privates'):
+                    cfg['show_private_bugs'] = True
+
+                elif opt in ('--updated-only'):
+                    cfg['updated_only'] = True
+
                 elif opt in ('--cached-lp-resources'):
                     cfg['cached_resources'] = val
 
@@ -332,6 +341,14 @@
             self.initialize()
 
             search_start = datetime.utcnow()
+            # old_tasks loaded from json file
+            if self.cfg['updated_only']:
+                last_run = json_load(self.cfg['json_file'])
+                try:
+                    old_tasks = last_run['tasks']
+                except KeyError:
+                    print("Old tasks not loaded as they aren't in the json file")
+                    old_tasks = {}
             iso_tasks = {}
 
             # For each 'search_criteria' section of the config file:
@@ -360,9 +377,23 @@
                                 #
                                 if bug_number not in iso_tasks:
                                     iso_tasks[bug_number] = []
+                                if self.cfg['updated_only']:
+                                    if str(bug_number) in old_tasks.keys():
+                                        try:
+                                            # this could be way better
+                                            old_bug = old_tasks['%s' % bug_number][0]
+                                        except IndexError:
+                                            old_bug = None
+                                        # the date_last_updated info gets stripped a lot in date_to_string
+                                        last_update = task.bug.date_last_updated.replace(second=0, microsecond=0, tzinfo=None)
+                                        if old_bug is not None and last_update <= string_to_date(old_bug['bug']['date_last_updated']):
+                                            if not self.cfg['run_quietly']:
+                                                stdo("LP: #%s is being skipped as it hasn't been updated" % bug_number)
+                                            iso_tasks[bug_number] = old_tasks['%s' % bug_number]
+                                            continue
 
                                 try:
-                                    if not task.bug.private:
+                                    if not task.bug.private or self.cfg['show_private_bugs']:
                                         t = self.get_task_info(task)
                                         t['debug']['series'] = s
                                         t_name = t['bug_target_name']
@@ -546,6 +577,8 @@
     defaults = {}
     defaults['app_name'] = argv[0]
     defaults['run_quietly'] = False
+    defaults['show_private_bugs'] = False
+    defaults['updated_only'] = False
 
     # The cmdline processing is done here partially so the debug options
     # can be processed right away.