Merge lp:~wgrant/launchpad/bugtaskflat-db into lp:launchpad/db-devel
| Status: | Merged | ||||
|---|---|---|---|---|---|
| Approved by: | William Grant on 2012-03-30 | ||||
| Approved revision: | no longer in the source branch. | ||||
| Merged at revision: | 11489 | ||||
| Proposed branch: | lp:~wgrant/launchpad/bugtaskflat-db | ||||
| Merge into: | lp:launchpad/db-devel | ||||
| Prerequisite: | lp:~wgrant/launchpad/ap-delete | ||||
| Diff against target: |
1103 lines (+971/-0) 3 files modified
database/schema/patch-2209-16-0.sql (+546/-0) database/schema/security.cfg (+19/-0) lib/lp/bugs/tests/test_bugtaskflat_triggers.py (+406/-0) |
||||
| To merge this branch: | bzr merge lp:~wgrant/launchpad/bugtaskflat-db | ||||
| Related bugs: |
|
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| j.c.sackett (community) | code | 2012-03-29 | Approve on 2012-03-29 |
| Richard Harding (community) | code* | 2012-03-29 | Approve on 2012-03-29 |
| Stuart Bishop | db | 2012-03-08 | Approve on 2012-03-29 |
| Robert Collins | db | 2012-03-29 | Pending |
|
Review via email:
|
|||
Commit Message
Create BugTaskFlat, a flattened version of Bug, BugTask, and some other bits and pieces.
Description of the Change
Bug filtering and sorting is split awkwardly across Bug and BugTask, causing fairly universally terrible performance. Further, access checks extend to joins on BugSubscription. Experiments have shown that putting most of the sorting and filtering criteria into a single table improves speed vastly.
This branch introduces a new table, BugTaskFlat, comprising relevant fields from Bug and BugTask, Product.active and arrays of relevant records from AccessPolicyArt
The table is maintained by a fairly impressive array of triggers and helper functions, but there is also a master function which is the canonical definition of BugTaskFlat and can correct rows to match reality. So the table is fairly disposable and easy to regenerate if we need to for any reason. There are reasonably thorough tests of these triggers.
The triggers have one hole: Product.active. Because products can have hilariously large numbers of bugs, updating BugTaskFlat.active inside a request is infeasible. It will probably be updated by a job later.
I've omitted foreign keys from the table. This seems to be a significant performance enhancement, and is fine from an integrity perspective because the table is only updated by triggers from FKed tables, and is disposable and cheap to regenerate if things end up going wrong. I also make use of arrays which can't be FKed in current PostgreSQL releases.
| William Grant (wgrant) wrote : | # |
| Stuart Bishop (stub) wrote : | # |
I guess we need a bug files about the Product.active hole. I agree it can't be done with triggers (well... we could schedule a celary job from a trigger).
Without foreign key references to the Person table, all those references will need special casing in the people merge code. We probably want a bug tracking this too, as we will need to rebuild BugTaskFlat after adding that code. Or perhaps the job that handles Product.active will maintain this too?
=== added file 'database/
+CREATE INDEX bugtaskflat_
This index is redundant - a UNIQUE index gets created on bugtask with the PRIMARY KEY.
+CREATE INDEX
Lots of other indexes for searching and reporting. I'm not going to second guess which columns should be DESC or not, and that things are structured the way they are to support the queries you need. My eyeballs don't pick up any naming errors or cut'n'paste glitches.
+ ELSIF new_flat_row != old_flat_row THEN
I think this comparison will fail if the rows are identical except for the order of the elements in the access_policies and access_grants arrays. A spurious update will be harmless here, but the return value will be incorrect. I think we need to guarantee the access cache helpers return the elements in a stable order.
=== added file 'lib/lp/
I haven't reviewed the tests. I can do the code portion later if nobody beats me too it.
| Richard Harding (rharding) wrote : | # |
Thanks William. From a code standpoint this looks ok. It's hard to say that all cases and corners are covered but it appears very well thought out and run through. I'd definitely expect the integration parts in the future exercising the current tests around the actual searching to help find any large holes in moving to this new flattened table setup.
| William Grant (wgrant) wrote : | # |
On 29/03/12 23:48, Stuart Bishop wrote:
> Review: Approve db
>
> I guess we need a bug files about the Product.active hole. I agree it
> can't be done with triggers (well... we could schedule a celary job
> from a trigger).
>
> Without foreign key references to the Person table, all those
> references will need special casing in the people merge code. We
> probably want a bug tracking this too, as we will need to rebuild
> BugTaskFlat after adding that code. Or perhaps the job that handles
> Product.active will maintain this too?
No need for special-casing -- person merging will update the FKs on the
base tables, and the usual triggers will update BugTaskFlat. Person
merging doesn't need to know it exists.
> === added file 'database/
>
>
> +CREATE INDEX bugtaskflat_
> (bugtask);
>
> This index is redundant - a UNIQUE index gets created on bugtask with
> the PRIMARY KEY.
Good point.
> +CREATE INDEX
>
> Lots of other indexes for searching and reporting. I'm not going to
> second guess which columns should be DESC or not, and that things are
> structured the way they are to support the queries you need. My
> eyeballs don't pick up any naming errors or cut'n'paste glitches.
>
>
> + ELSIF new_flat_row != old_flat_row THEN
>
> I think this comparison will fail if the rows are identical except
> for the order of the elements in the access_policies and
> access_grants arrays. A spurious update will be harmless here, but
> the return value will be incorrect. I think we need to guarantee the
> access cache helpers return the elements in a stable order.
Indeed. I'll add an order.
> === added file 'lib/lp/
>
> I haven't reviewed the tests. I can do the code portion later if
> nobody beats me too it.
>
| j.c.sackett (jcsackett) wrote : | # |
The tests seem pretty solid--everything I would worry about seems covered. This looks fine to land from my perspective.

I suspect that a later iteration will include a tag array in some form, but with PostgreSQL 8.4 it gets a bit awkward, and it needs lots of testing to see which method is best (an array of strings, or an array of integers as interned strings, a fake tsvector, etc.). This is a good start.
Also, access_policies and access_grants are null if the bug is public, as they don't really make sense. When the bug is private they're always (possibly empty) arrays.