Distribution:+ppas and Distribution:+questions issue unlimited queries when memo=0&direction=backwards

Bug #872086 reported by William Grant
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Raphaël Badin
Storm
Fix Released
High
Raphaël Badin
lazr.batchnavigator
Fix Released
Low
Raphaël Badin

Bug Description

The batch navigators on at least Distribution:+ppas and Distribution:+questions fail to LIMIT the queries when memo=0&direction=backwards. This causes them to read in huge volumes of data (three minutes worth, in the case of Distribution:+questions). Strangely, Distribution:+bugs appears to LIMIT appropriately.

OOPS-2109ED42 and OOPS-2109EC14 are examples. The only thing I've seen using this sort of URL (apart from me when testing) is Googlebot.

Related branches

Revision history for this message
Raphaël Badin (rvb) wrote :

Looks like a storm bug SQLObjectResultSet[0:0]:

{{{
>>> from lp.registry.interfaces.distribution import IDistributionSet
>>> from lp.registry.interfaces.distroseries import IDistroSeriesSet
>>> ubuntu = getUtility(IDistributionSet).getByName("ubuntu")
>>> distroseriesset = getUtility(IDistroSeriesSet)

>>> list(distroseriesset.search(distribution=ubuntu))
[<DistroSeries u'warty'>, <DistroSeries u'hoary'>, <DistroSeries
u'grumpy'>, <DistroSeries u'breezy-autotest'>]
# Good
# SELECT DistroSeries.backports_not_automatic,[...] FROM DistroSeries
WHERE distribution = 1 ORDER BY DistroSeries.distribution,
DistroSeries.version

>>> list(distroseriesset.search(distribution=ubuntu)[1:1])
[]
# Good
# SELECT DistroSeries.backports_not_automatic,[...] FROM DistroSeries
WHERE distribution = 1 ORDER BY DistroSeries.distribution,
DistroSeries.version LIMIT 0 OFFSET 1

>>> list(distroseriesset.search(distribution=ubuntu)[0:0])
[<DistroSeries u'warty'>, <DistroSeries u'hoary'>, <DistroSeries
u'grumpy'>, <DistroSeries u'breezy-autotest'>]
# No limit nor offset in the generated query ... ?!
# SELECT DistroSeries.backports_not_automatic,[...] FROM DistroSeries
WHERE distribution = 1 ORDER BY DistroSeries.distribution,
DistroSeries.version
}}}

Revision history for this message
Raphaël Badin (rvb) wrote :

The plan here is to:
- fix the bug in Storm
- fix the batchnavigator to not even issue a query if the slice is going to be of the form [N:N]

Thomas Herve (therve)
Changed in storm:
importance: Undecided → High
assignee: nobody → Raphaël Badin (rvb)
milestone: none → 0.20
status: New → In Progress
Revision history for this message
Robert Collins (lifeless) wrote :

This is nearly right :) the if block you remove needs to stay and be tweaked though:
>>> range(10)[None:None]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Raphaël Badin (rvb)
Changed in launchpad:
status: Triaged → In Progress
assignee: nobody → Raphaël Badin (rvb)
Raphaël Badin (rvb)
Changed in lazr.batchnavigator:
status: New → In Progress
assignee: nobody → Raphaël Badin (rvb)
importance: Undecided → Low
Revision history for this message
Raphaël Badin (rvb) wrote :

Profiling the queries issued by the page on qastaging confirms the limit is now there in the sql query ("ORDER BY Question.datecreated DESC LIMIT 0 OFFSET 0").

tags: added: qa-ok
Changed in lazr.batchnavigator:
status: In Progress → Fix Committed
Changed in storm:
status: In Progress → Fix Committed
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant)
Changed in launchpad:
status: Fix Committed → Fix Released
Revision history for this message
Robert Collins (lifeless) wrote :

Can we do a release of lazr.batchnavigator then, clear the inventory out?

Changed in storm:
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote :

I've released lazr.batchnavigator 1.2.11, and will upgrade Launchpad to that.

Changed in lazr.batchnavigator:
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote :

And I've confirmed that this does indeed get rid of the extra LIMIT 0 query.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.