Merge lp:~james-w/launchpad-work-items-tracker/multi-project into lp:launchpad-work-items-tracker

[James Westby]

Hi,

Apologies for the bug in the previous change, I hadn't tested with
a project base, just Ubuntu.

This is the proper fix that passes the LP object for both calls, which
will make it work in both cases.

Thanks,

James

[Martin Pitt (pitti)]

I reverted the multi-projects commit for now, as it wreaked havoc. It causes the ubuntu database to only contain the linaro milestones, and thus breaking all the milestone charts and sending tons of errors to everyone.

What is the rationale here in the first place? linaro has separate projects in Launchpad, thus we should also have separate WI reports/burndowns, such as

  http://people.canonical.com/~platform/workitems/linaro-pm-wg/all.html
  http://people.canonical.com/~platform/workitems/linaro-toolchain-wg/all.html

(you also use a different burndown chart style). Trying to merge different projects with different milestones into one set of charts will just create confusion. Either you need to ignore the milestones from either side (which is obvioulsy not acceptable), or you need to merge the milestones from all projects. But this will create a combinatorial explosion of charts, and also break the automatic reset of per-milestone burndown charts, so it's also not feasible.

So let's discuss that a bit further first.

[James Westby (james-w)]

> I reverted the multi-projects commit for now, as it wreaked havoc. It causes
> the ubuntu database to only contain the linaro milestones, and thus breaking
> all the milestone charts and sending tons of errors to everyone.

Sorry about that.

> What is the rationale here in the first place? linaro has separate projects in
> Launchpad, thus we should also have separate WI reports/burndowns, such as
>
> http://people.canonical.com/~platform/workitems/linaro-pm-wg/all.html
> http://people.canonical.com/~platform/workitems/linaro-toolchain-wg/all.html

We have changed where we put blueprints somewhat this cycle. For instance, the linaro
foundations team has blueprints against at least "ubuntu", and "linaro", and they
don't want to have to try and reconcile two burndown charts in their head.

> (you also use a different burndown chart style). Trying to merge different
> projects with different milestones into one set of charts will just create
> confusion. Either you need to ignore the milestones from either side (which is
> obvioulsy not acceptable), or you need to merge the milestones from all
> projects.

As the milestones are just name->date mappings I don't think that storing all of
them is a problem. I can arrange to have ubuntu's milestones take precedence, so
we can't do something like have a natty-alpha-2 with a different date.

> But this will create a combinatorial explosion of charts,

Because you generate a chart for every milestone? Would it be enough not to generate
empty charts?

> and also
> break the automatic reset of per-milestone burndown charts, so it's also not
> feasible.

I don't follow this part, can you point me to the auto-reset code so that I can
see what will break please?

> So let's discuss that a bit further first.

Sure, I'm happy to do that, but this is an urgent matter for Linaro, so I'm willing
and keen to make the changes to get this landed ASAP. If it's not acceptable in
Ubuntu's instance, then I imagine we will end up with our own instance with changes
similar to these.

Thanks,

James

[lp:~james-w/launchpad-work-items-tracker/multi-project]

246. By James Westby on 2010-12-10

Make sure we import milestones from all projects, not just the first.

[James Westby (james-w)]

Found the auto-reset:

   if milestone:
        # get milestone date range
        cur = db.cursor()
        cur.execute('SELECT due_date FROM milestones WHERE name = ?', (milestone,))
        date_end = cur.fetchone()[0]
        cur.execute('SELECT MAX(due_date), MAX(due_date) < date(CURRENT_TIMESTAMP) FROM milestones WHERE due_date < ?', (date_end,))
        (date_start, in_milestone) = cur.fetchone()

        # if we are already within the milestone, start at the previous
        # milestone's end date; if we are before, start at the beginning of the
        # entire cycle, to allow planning
        if in_milestone:
            ms_sql = "AND w.milestone='%s' AND date >= '%s' AND date <= '%s'" % (
                    escape_sql(milestone), date_start, date_end)
        else:
            ms_sql = "AND w.milestone='%s' AND date <= '%s'" % (
                    escape_sql(milestone), date_end)

I'll have to think about that a bit.

Thanks,

James

[Martin Pitt (pitti)]

Right, that's the auto-reset code. BTW, this is not just an issue for resetting the charts, the more important usage is that you need to determine what the "next" milestone is. Ubuntu alpha-2 needs to go until the day Ubuntu alpha-3 starts, not until an intermediate milestone of a different project source.

> > But this will create a combinatorial explosion of charts,
>Because you generate a chart for every milestone? Would it be enough not to generate empty charts?

The svgs are already skipped if they are empty, and empty HTML pages don't cost that much to generate. Of course this won't be a combinatorial explosion, that was a thinko, sorry. It's just additive, so no big deal.

So the main points to consider here are the merging of the milestone table (which is just a small implementation problem), and making milestones "source" specific, so that the individual charts can still know how long a milestone is, and when to reset it.

[James Westby (james-w)]

"Martin Pitt" <email address hidden> wrote:

>Right, that's the auto-reset code. BTW, this is not just an issue for
>resetting the charts, the more important usage is that you need to
>determine what the "next" milestone is. Ubuntu alpha-2 needs to go
>until the day Ubuntu alpha-3 starts, not until an intermediate
>milestone of a different project source.

Yep, got that now. I'd never realised what the milestone charts were doing, so thanks for the pointer.

>The svgs are already skipped if they are empty, and empty HTML pages
>don't cost that much to generate. Of course this won't be a
>combinatorial explosion, that was a thinko, sorry. It's just additive,
>so no big deal.

Ok, I'm going to see if I can come up with a cunning plan to reduce the number of these without configuration.

>So the main points to consider here are the merging of the milestone
>table (which is just a small implementation problem),

I've pushed up one approach to this, but I may revisit it depending on the solution to the next problem.

> and making
>milestones "source" specific, so that the individual charts can still
>know how long a milestone is,

I've just talked to Jamie and we want to go a little further than that to give unified burndowns for linaro and ubuntu alphas that are on the same day, but I've no idea if that is feasible. Either way I will aim for no change of behavior for ubuntu teams.

[James Westby (james-w)]

> Right, that's the auto-reset code. BTW, this is not just an issue for
> resetting the charts, the more important usage is that you need to determine
> what the "next" milestone is. Ubuntu alpha-2 needs to go until the day Ubuntu
> alpha-3 starts, not until an intermediate milestone of a different project
> source.

I hadn't understood what the milestone charts were doing before. Thanks for the pointer.

> The svgs are already skipped if they are empty, and empty HTML pages don't
> cost that much to generate. Of course this won't be a combinatorial explosion,
> that was a thinko, sorry. It's just additive, so no big deal.

Ok. I'm going to see if I can come up with a cunning way to avoid some of
this without configuration.

> So the main points to consider here are the merging of the milestone table
> (which is just a small implementation problem),

I've pushed up a small change that does this, but may alter the approach depending
on the solution to the next issue.

> and making milestones "source"
> specific, so that the individual charts can still know how long a milestone
> is, and when to reset it.

I've just chatted to Jamie, and this doesn't go quite as far as we would like
in Linaro, where we would like to have consolidated burndown charts for milestones
where the Ubuntu and Linaro milestones are on the same date. I will aim to have
no change for Ubuntu teams when doing this though.

Thanks,

James

[lp:~james-w/launchpad-work-items-tracker/multi-project]

247. By James Westby on 2010-12-13

Store the milestone where the project comes from.

This is only a partial solution.

248. By James Westby on 2010-12-14

Merge trunk.

[Martin Pitt (pitti)]

Updating status to get a better overview on the "active code reviews" page.

review: Needs Fixing

=== modified file 'burndown-chart'
--- burndown-chart	2010-12-01 15:52:29 +0000
+++ burndown-chart	2010-12-14 15:53:12 +0000
@@ -280,6 +280,8 @@
     if opts.milestone:
         cur.execute('SELECT due_date FROM milestones WHERE name = ?', (opts.milestone,))
     else:
+        # FIXME: this won't work for Ubuntu with Linaro's milestones in
+        # there.
         cur.execute('SELECT MAX(due_date) FROM milestones')
     end_date = cur.fetchone()[0]
 else:

=== modified file 'collect'
--- collect	2010-12-10 10:48:07 +0000
+++ collect	2010-12-14 15:53:12 +0000
@@ -471,7 +471,7 @@
         db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))',
                 (desc, bp_name, status, assignee, milestone))
 
-def lp_import_milestones(lp_project, db):
+def lp_import_milestones(milestones, db):
     '''Import milestones from Launchpad into DB.
 
     lp_project must be a Launchpad project or distro_series object.
@@ -483,10 +483,10 @@
         return
 
     dbg('lp_import_milestones(): importing from Launchpad')
-    for ms in lp_project.all_milestones:
+    for ms in milestones:
         if ms.date_targeted:
-            cur.execute('INSERT INTO milestones VALUES (?, ?)', (ms.name,
-                    ms.date_targeted.strftime('%Y-%m-%d')))
+            cur.execute('INSERT OR IGNORE INTO milestones VALUES (?, ?, ?)', (ms.name,
+                    ms.date_targeted.strftime('%Y-%m-%d'), ms.target.name))
     db.commit()
 
 def lp_import_teams(lp, db, cfg):
@@ -550,25 +550,43 @@
 
     lp = Launchpad.login_with('ubuntu-work-items', service_root=EDGE_SERVICE_ROOT)
 
+    urls = []
+    milestones = []
+
+    def add_milestones(project):
+        milestones.extend([ms for ms in project.all_milestones])
+
+    def import_project(project):
+        add_milestones(project)
+        url = '%s/%s/+specs?batch=300' % (
+                report_tools.blueprints_base_url, project.name)
+        urls.append(url)
+
     if 'release' in cfg:
         lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release'])
+        add_milestones(lp_project)
         url = '%s/ubuntu/%s/+specs?batch=300' % (
                 report_tools.blueprints_base_url, cfg['release'])
+        urls.append(url)
     else:
         assert 'project' in cfg, 'Configuration needs to specify project or release'
         lp_project = lp.projects[cfg['project']]
-        url = '%s/%s/+specs?batch=300' % (
-                report_tools.blueprints_base_url, cfg['project'])
-
-    lp_import_milestones(lp_project, db)
+        import_project(lp_project)
+
+    extra_projects = cfg.get('extra_projects', [])
+    for extra_project_name in extra_projects:
+        extra_project = lp.projects[extra_project_name]
+        import_project(extra_project)
+
+    lp_import_milestones(milestones, db)
     lp_import_teams(lp, db, cfg)
 
-    for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems():
-        dbg('lp_import(): downloading %s from %s' % (bp, url))
-        contents = urllib.urlopen(url).read().decode('UTF-8')
-        if lp_import_blueprint(db, bp, url, contents):
-            lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release'))
-
+    for url in urls:
+        for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems():
+            dbg('lp_import(): downloading %s from %s' % (bp, url))
+            contents = urllib.urlopen(url).read().decode('UTF-8')
+            if lp_import_blueprint(db, bp, url, contents):
+                lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release'))
     lp_import_bug_workitems(lp_project, db, cfg)
 
 ########################################################################
@@ -725,7 +743,7 @@
         cur.execute('''CREATE TABLE version (
             db_layout_ref INT NOT NULL
             )''')
-        cur.execute('''INSERT INTO version VALUES (6)''')
+        cur.execute('''INSERT INTO version VALUES (7)''')
 
         cur.execute('''CREATE TABLE specs (
             name VARCHAR(255) PRIMARY KEY,
@@ -759,7 +777,8 @@
 
         cur.execute('''CREATE TABLE milestones (
             name VARCHAR(50) PRIMARY KEY,
-            due_date TIMESTAMP NOT NULL
+            due_date TIMESTAMP NOT NULL,
+            project VARCHAR(50)
             )''')
 
         cur.execute('''CREATE TABLE meta (
@@ -840,6 +859,13 @@
             db.commit()
             ver = 6
 
+        if ver == 6:
+            dbg('Upgrading DB to layout version 7')
+            cur.execute('ALTER TABLE milestones ADD COLUMN project VARCHAR(50)')
+            cur.execute('UPDATE version SET db_layout_ref = 7')
+            db.commit()
+            ver = 7
+
     return db
 
 def create_v5_indexes(cur):

=== modified file 'report_tools.py'
--- report_tools.py	2010-12-03 12:41:46 +0000
+++ report_tools.py	2010-12-14 15:53:12 +0000
@@ -133,9 +133,11 @@
     if milestone:
         # get milestone date range
         cur = db.cursor()
-        cur.execute('SELECT due_date FROM milestones WHERE name = ?', (milestone,))
-        date_end = cur.fetchone()[0]
-        cur.execute('SELECT MAX(due_date), MAX(due_date) < date(CURRENT_TIMESTAMP) FROM milestones WHERE due_date < ?', (date_end,))
+        cur.execute('SELECT due_date, project FROM milestones WHERE name = ?', (milestone,))
+        row = cur.fetchone()
+        date_end = row[0]
+        project = row[1]
+        cur.execute('SELECT MAX(due_date), MAX(due_date) < date(CURRENT_TIMESTAMP) FROM milestones WHERE due_date < ? AND project = ?', (date_end, project))
         (date_start, in_milestone) = cur.fetchone()
 
         # if we are already within the milestone, start at the previous