Merge lp:~jtv/storm/profile-fetches into lp:storm

Proposed by Jeroen T. Vermeulen on 2010-12-10
Status: Needs review
Proposed branch: lp:~jtv/storm/profile-fetches
Merge into: lp:storm
Diff against target: 1109 lines (+903/-7) (has conflicts)
8 files modified
storm/database.py (+1/-0)
storm/fetch_profile.py (+255/-0)
storm/references.py (+12/-4)
storm/store.py (+97/-3)
tests/fetch_context.py (+164/-0)
tests/fetch_profile.py (+64/-0)
tests/fetch_statistics.py (+99/-0)
tests/store/base.py (+211/-0)
Text conflict in tests/store/base.py
To merge this branch: bzr merge lp:~jtv/storm/profile-fetches
Reviewer Review Type Date Requested Status
Storm Developers 2010-12-15 Pending
Storm Developers 2010-12-10 Pending
Review via email: mp+43323@code.launchpad.net

Description of the change

= Fetch-Profiling =

Profile dependencies between object fetches from the database.

This work has been raised on the launchpad-dev mailing list, and in a later stage, discussed with Jamu Kakar, Stuart Bishop, and others.

== The problem ==

Profiling will help map out and optimize data needs. For instance, consider this loop:

    def get_x_for(item):
        return item.other_object.x

# ...

    for item in query_items():
        total += get_x_for(item)

This will fetch item.other_object individually for each item coming out of query_items(). It's a common anti-pattern in ORM performance, and easy enough to optimize: just outer-join item.other_object inside query_items, so that it will already be in cache when get_x_for needs it.

Keeping track of all such dependencies is tedious, brittle, and a source of major abstraction leaks. Conventional ways of dealing with them involve profiling, analysis, matching query patterns to code paths, mapping out data requirements, and identifying downside risks of optimization. The result is a single "point-solution" fix. The effort produces experience as a side effect, but transfer of such experience from one human to others is relatively ineffective.

Then, after that's all done and the code has been optimized, it's difficult to keep track of which optimizations are still relevant. Code is alive, and the more intricate and beautiful an optimization is, the easier it is to break. Testing for such "semantically-neutral" breakage is often difficult, and monitoring the relevance of the tests themselves can be costly.

Refactorings in particular raise questions: will I need to port this optimization to the new structure? Will I still be hitting the right indexes? Could the new structure make the optimization unnecessary or less relevant? Am I prefetching a lot of objects that I don't need? And after I've made all those choices, how can I compare the new code's performance to the old code's performance as it's been tuned over time?

Fetch-profiling brings us closer to solving all those problems, but be patient. For now, it simplifies the mapping of data requirements by eliminating the tracing and the matching of query patterns to code paths. Read on for where we go next.

== Visibility improvements ==

Profiling would expose the problem pattern in the example very clearly. After running through the loop a few times, you'd inspect the store's statistics. The statistics will tell you how many item.other_object instances were loaded from the database (for "item"s returned by query_items) as well as how this number compares to the number of "item"s loaded by query_items, as a percentage.

The highest "item" numbers will identify the places most in need of pre-fetching optimizations. Among those, the highest percentages of "item.other_object" loads identify the places where simple prejoins are most likely to be beneficial. Lower percentages may indicate that many of the foreign-key references are null, or that most of the objects they refer to are already covered by other caching, or that most of the objects you might prefetch in the query would be irrelevant.

== Future improvements ==

This is phase 1 of a proposed multi-phase development. For phase 2 I'd like to automate the prefetching so that code like this (using features layered on top of the existing Storm API) will optimize itself, without requiring manual tweaking. After that come policy tuning and automated context definition (see below).

With automated prefetching, the most basic optimizations will no longer be specified in the application. They will work themselves out automatically based on feedback from a real, running production system.

It is at that point where the big problems resolve themselves. After a code change the individual optimizations will re-apply themselves as appropriate. There is no need to track their relevance manually.

== Concepts ==

Cast of characters:
 • A "fetch" is the retrieval and de-marshalling of an object from the database. Reading just a foreign key (e.g. to check for NULL) is not a fetch from the table it refers to; neither is retrieving an object from cache.
 • A "fetch context" is some indicator of what part of the application is executing a query. To the application, this is managed as a "call stack" of strings.
 • An "original fetch" is a free-form query, as performed using Store.find or Store.execute.
 • A "fetch origin" (within a context) is a class participating in an original fetch.
 • A "derived fetch" is the retrieval of objects that are clearly derived (directly or indirectly) from an original fetch through a chain of reference.

In the example loop, query_items() would contain at least an original fetch. The reference to item.other_object inside get_x_for is a derived fetch (derived directly from the original fetch, as it happens). Derived fetches can also be tracked across stores.

There's only room for one fetch context in this example, since derived fetches are associated with the same context as their original fetches. In a web application, the most useful context would probably be the request type, but for detailed optimization you'll want more fine-grained contexts. The typical ideal granularity for automated optimization would be just one original query per context.

Contexts form a hierarchy so as to suit all these use cases, as well as "drilling down" during analysis of data requirements. A context manager helps mark regions of code as being a specific context. Another idea would be a decorator (probably at the application level though, where it's easier to find the right store) and an optional argument to find() that selects a context for just one query.

Original fetches are identified by the fetched class as well as the context. This makes it possible to associate derived fetches with individual classes in a join, and track their dependent fetches separately.

== Implementation notes ==

I'm not planning to map out full dependency chains from fetch origins to derived fetches for now; that would probably become too costly. We'd have to see how useful that information is in practice.

You may note how fetch_context is tracked in Stores, in Results, and in ObjectInfos. The reason for this is that objects may be fetched from a result set long after the store has moved on to a different context. An object fetch should be associated with the context that the result set was produced in, which in turn is the context the store was in at the time.

All interesting analysis and optimization work will be done outside the performance-critical path of query execution. Profiling costs should be minimal, limited to simple dict lookups and counter increments.

Jeroen

To post a comment you must log in.
Robert Collins (lifeless) wrote :

Hi Jeroen, thanks for pointing me at this.

I think this is a very interesting project. I think the stats will be useful for manual optimisation in the short term in Launchpad.

As far as the autotuning goes long term, the jit-vm style learn-and-improve doesn't interest me for Launchpad : https://dev.launchpad.net/LEP/PersistenceLayer contains my broad plans for addressing systematic performance issues in Launchpad. I think a jit-vm auto tuning layer would be a fascinating project, but the warm-up time in many JIT's can be substantial, and is only ameliorated by loops running hundreds or thousands of times : and still at best only approaches the efficiency available by writing in a more efficient language. Thus my interest in providing a more efficient DSL than storms bound-object approach. I'd love to see storm become radically simpler in aid of that: faster short circuits in object caching - optionally no object caching at all. Constrained and parameterised references would be awesome too.

Cheers,
Rob

Jeroen T. Vermeulen (jtv) wrote :

I'd be more careful in assuming similarity with a JIT VM. Differences in relative overhead aside, JIT compilers don't specialize method calls much. Actually there is a JVM that combines JIT with inlining of all code, but from what I hear that actually yields fantastic results. A lot of the startup overhead would be in the inlining, which doesn't come up per se in fetch profiling.

As an example of specialization, if Launchpad used automatic optimization based on this profiler, a given query method somewhere deep down the call stack gets optimized separately when used from the web service API or when used from the web UI. The two calls have very different needs. We don't have any decent solution for that at the moment.

If you're willing to be as aggressive in fetching objects as to do it "statically," then you might as well use automatic optimization with a warmup time of 1: generate optimization advice after a first pass through a stretch of code, then repeat periodically to cover any objects that may also be needed but weren't referenced in that first run. To amortize startup cost over more requests, pickle the optimization choices and presto: profile-driven optimizations get reused across restarts.

Separately from that, the term "efficient" is treacherous. A static approach is almost guaranteed to be more efficient in terms of computing power, yes, but less efficient when it comes to the human factors: flexibility, legibility, conceptual cleanliness. (Isn't that why we're using python in the first place?) A dynamic approach on the other hand can narrow the gap in computational efficiency without sacrificing any of those other efficiencies. It's also easier to deploy and fine-tune such optimizations across the entire application.

Jeroen

lp:~jtv/storm/profile-fetches updated on 2011-06-20
419. By Jeroen T. Vermeulen on 2010-12-22

Don't support derived_from without a known reference.

420. By Jeroen T. Vermeulen on 2011-01-01

Record origin, source, and reference; ignore cross-store dependencies.

421. By Jeroen T. Vermeulen on 2011-01-01

Cosmetic.

422. By Jeroen T. Vermeulen on 2011-06-20

Documentation; made is_root a @property.

Unmerged revisions

422. By Jeroen T. Vermeulen on 2011-06-20

Documentation; made is_root a @property.

421. By Jeroen T. Vermeulen on 2011-01-01

Cosmetic.

420. By Jeroen T. Vermeulen on 2011-01-01

Record origin, source, and reference; ignore cross-store dependencies.

419. By Jeroen T. Vermeulen on 2010-12-22

Don't support derived_from without a known reference.

418. By Jeroen T. Vermeulen on 2010-12-16

Aggregate stats by context name; nicer cumulate API.

417. By Jeroen T. Vermeulen on 2010-12-16

Context iteration.

416. By Jeroen T. Vermeulen on 2010-12-16

Test add_to_dict separately.

415. By Jeroen T. Vermeulen on 2010-12-16

Move profiling functions out of Store.

414. By Jeroen T. Vermeulen on 2010-12-15

Context manager.

413. By Jeroen T. Vermeulen on 2010-12-15

Track derived fetches across stores.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'storm/database.py'
2--- storm/database.py 2010-04-16 07:14:25 +0000
3+++ storm/database.py 2011-06-20 13:09:26 +0000
4@@ -52,6 +52,7 @@
5 def __init__(self, connection, raw_cursor):
6 self._connection = connection # Ensures deallocation order.
7 self._raw_cursor = raw_cursor
8+ self.fetch_origin = None
9 if raw_cursor.arraysize == 1:
10 # Default of 1 is silly.
11 self._raw_cursor.arraysize = 10
12
13=== added file 'storm/fetch_profile.py'
14--- storm/fetch_profile.py 1970-01-01 00:00:00 +0000
15+++ storm/fetch_profile.py 2011-06-20 13:09:26 +0000
16@@ -0,0 +1,255 @@
17+#
18+# Copyright (c) 2011 Canonical
19+#
20+# Written by Jeroen Vermeulen at Canonical.
21+#
22+# This file is part of Storm Object Relational Mapper.
23+#
24+# Storm is free software; you can redistribute it and/or modify
25+# it under the terms of the GNU Lesser General Public License as
26+# published by the Free Software Foundation; either version 2.1 of
27+# the License, or (at your option) any later version.
28+#
29+# Storm is distributed in the hope that it will be useful,
30+# but WITHOUT ANY WARRANTY; without even the implied warranty of
31+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
32+# GNU Lesser General Public License for more details.
33+#
34+# You should have received a copy of the GNU Lesser General Public License
35+# along with this program. If not, see <http://www.gnu.org/licenses/>.
36+#
37+
38+"""Fetch-profiling support.
39+
40+This profiles how objects are pulled from the database into the local cache.
41+Accesses to objects that are already cached are ignored.
42+
43+
44+= Original and Derived Fetches =
45+
46+The profiler distinguishes two ways in which an object can be retrieved from
47+the database: "original fetches" and "derived fetches."
48+
49+An original fetch happens during a free-form query, such as...
50+
51+ employees = store.find(Employee, Employee.name.startswith(name_pattern))
52+
53+A derived fetch happens when the program follows a reference from an object
54+that needs to be retrieved from the database. These are often a problem in
55+ORM performance, because object-oriented programs can easily request large
56+numbers of fetches in small, inefficient queries. For instance, this could
57+query all your company's Department records one by one in separate queries:
58+
59+ for emp in employees:
60+ print(emp.department.name)
61+
62+Here, the fetch that pulls emp.department into the cache is "derived" from
63+the original fetch that retrieved emp itself. A derived fetch can also be
64+derived from another derived fetch, but ultimately the chain leads back to
65+either an original fetch or a new object in memory.
66+
67+When it comes to optimizing your application, one of the things you'll want to
68+look at is reducing derived fetches. That simple loop over employees might be
69+many times faster if you loaded all of the departments you needed into cache
70+in one single query:
71+
72+ # "Pre-fetch" the employees' departments into Storm's cache.
73+ dept_ids = set([emp.department_id for emp in employees])
74+ list(store.find(Department, Department.id.is_in(dept_ids)))
75+ for emp in employees:
76+ # Faster, because emp.department is in cache now!
77+ print(emp.department.name)
78+
79+Sometimes it may even be most efficient to join the derived fetch into your
80+original query:
81+
82+ emps_and_depts = store.find(
83+ (Employee, Department),
84+ Department.id == Employee.department_id,
85+ employee.name.startswith(name_pattern))
86+
87+ for emp, dept in emps_and_depts:
88+ print dept.name
89+
90+You'll recognize opportunities for this optimization when you see the same
91+derived fetch happen many times after the same original query. But it's
92+probably not worth doing this when the derived fetches are infrequent: the
93+code path that does the derived fetch could be rare, or the reference might
94+be None in most cases. Or more likely in this example, the query will return
95+many employees but only very few different departments. Most are probably
96+already in the cache, in which case the profile won't count them.
97+
98+The profile tells you how many objects were fetched at each step along any
99+data access path. So for example the profiler might report that the
100+original employees query pulled 1,233 objects into memory, but the
101+"emp.department" in the loop only fetched 62 departments: that means you can
102+save 61 queries by fetching all departments in one go. Or you might see zero
103+derived fetches because the departments are already in cache, in which case
104+the loop is not worth optimizing. You might even see about the same number
105+of departments fetched as you do employees, in which case your best option may
106+be to join Employee and Department together into one query. Maybe there is
107+too much data to hold in cache, and you need to get really creative.
108+
109+
110+= Fetch Contexts =
111+
112+When you spot a performance problem in the profile, you'll want to know
113+exactly where in your application it occurs. Sometimes the name of the
114+function is enough, but most of the time you'll want to know something more
115+about the context in which that function was called.
116+
117+To help you keep track of this information, the profiler lets you define
118+"fetch contexts," which are profiled separately. An original fetch is
119+counted in its current context, but a derived fetch is counted in the context
120+of the original fetch that it is ultimately derived from.
121+
122+So when you look at the profile for a particular context, you see not only the
123+data it queries (its original fetches) but also the "future" of that data: how
124+will this data be used after my function returns it? What can we start
125+prefetching here to speed up the code that consumes this data? It doesn't
126+matter whether that other code is in the same context or not.
127+
128+You can name contexts whatever you like. Contexts can also nest inside
129+contexts: you can have a context "process_salaries" with a context
130+"get_employees" inside it, and a different context "send_birthday_cards" also
131+with a "get_employees" inside it. Those two nested contexts are different
132+ones, even though they're both called "get_employees," and even though they
133+could actually be the same function.
134+
135+This lets you profile the same code separately in different situations, and
136+you can then decide whether they need the same optimizations or not. For
137+instance, a function in a web application could have different performance
138+characteristics depending on which page is being rendered or even who is
139+viewing it (an administrator might see more data than someone who is not
140+logged in, for example). In that case you could incorporate that information
141+in your context names, so that you'll be able to tell them apart in the
142+profile.
143+"""
144+
145+
146+__all__ = ["FetchContext", "FetchStatistics", "fetch_context"]
147+
148+
149+class fetch_context(object):
150+ """Context manager to mark a region of code as a fetch context."""
151+ def __init__(self, store, context_name):
152+ self.store = store
153+ self.context_name = context_name
154+
155+ def __enter__(self):
156+ """Start context `context_name`for the given `Store`."""
157+ self.store.push_fetch_context(self.context_name)
158+
159+ def __exit__(self, *args, **kwargs):
160+ """Close context."""
161+ self.store.pop_fetch_context()
162+
163+
164+class FetchContext(object):
165+ """A context in which database fetches are recorded for profiling.
166+
167+ Contexts nest in order to support detailed views and aggregation.
168+ However they need not exactly match the program's call stack.
169+
170+ Profiling is disabled in the root context.
171+
172+ :ivar name: The context's name.
173+ :ivar parent: The parent context.
174+ :ivar children: Child contexts, mapped by name.
175+ :ivar stats: `FetchStatistics` for this context.
176+ """
177+ def __init__(self, name, parent=None):
178+ self.name = name
179+ self.parent = parent
180+ self.children = {}
181+ self.stats = FetchStatistics()
182+
183+ def __iter__(self):
184+ for child in self.children.itervalues():
185+ yield child
186+ for grandchild in child:
187+ yield grandchild
188+
189+ @property
190+ def is_root(self):
191+ """Is this the root context?"""
192+ return self.parent is None
193+
194+ def get_child(self, name):
195+ """Find a child context of the given name, or create one."""
196+ child = self.children.get(name)
197+ if child is None:
198+ child = self.children[name] = FetchContext(name, parent=self)
199+ return child
200+
201+ def cumulate_stats(self):
202+ """Add `FetchStatistics` for this context and its children."""
203+ stats = self.stats.copy()
204+ for child in self:
205+ stats.merge(child.stats)
206+ return stats
207+
208+ def aggregate_stats_by_name(self):
209+ """Aggregate `FetchStatistics` for self and children by context name.
210+
211+ Returns a dict mapping context names to aggregated statistics for
212+ all `FetchContext`s of those respective names among self and its
213+ children.
214+ """
215+ names = set(child.name for child in self).union([self.name])
216+ stats = dict(
217+ (name, FetchStatistics())
218+ for name in names if name is not None)
219+ stats[self.name].merge(self.stats)
220+ for child in self:
221+ stats[child.name].merge(child.stats)
222+ return stats
223+
224+
225+def add_number_to_dict(dictionary, key, value=1):
226+ """Add `value` to `dictionary[key]`, defaulting to 0."""
227+ dictionary.setdefault(key, 0)
228+ dictionary[key] += value
229+
230+
231+def add_dict_to_dict(dest, addition):
232+ """Add the values from dict `addition` into dict `dest`."""
233+ for key, value in addition:
234+ add_number_to_dict(dest, key, value)
235+
236+
237+class FetchStatistics(object):
238+ """Fetch profiling statistics.
239+
240+ Statistics are recorded for each context, but they can also be
241+ aggregated.
242+
243+ :ivar original_fetches: Maps fetch origin (i.e. a class being fetched)
244+ to a count of the number of original fetches on that class.
245+ :ivar derived_fetches: Maps derived fetches to their respective fetch
246+ counts. A derived derived fetch is represented as a tuple
247+ (origin, source, Reference).
248+ """
249+ def __init__(self):
250+ self.original_fetches = {}
251+ self.derived_fetches = {}
252+
253+ def copy(self):
254+ new = FetchStatistics()
255+ new.original_fetches = self.original_fetches.copy()
256+ new.derived_fetches = self.derived_fetches.copy()
257+ return new
258+
259+ def record_original_fetch(self, origin):
260+ """Record an original fetch in the statistics."""
261+ add_number_to_dict(self.original_fetches, origin)
262+
263+ def record_derived_fetch(self, origin, source, reference):
264+ """Record a derived fetch in the statistics."""
265+ fetch = (origin, source, reference)
266+ add_number_to_dict(self.derived_fetches, fetch)
267+
268+ def merge(self, other_stats):
269+ """For aggregation purposes: merge `other_stats` into `self`."""
270+ add_dict_to_dict(self.original_fetches, other_stats.original_fetches)
271+ add_dict_to_dict(self.derived_fetches, other_stats.derived_fetches)
272
273=== modified file 'storm/references.py'
274--- storm/references.py 2010-06-01 08:33:33 +0000
275+++ storm/references.py 2011-06-20 13:09:26 +0000
276@@ -1,5 +1,5 @@
277 #
278-# Copyright (c) 2006, 2007 Canonical
279+# Copyright (c) 2006-2011 Canonical
280 #
281 # Written by Gustavo Niemeyer <gustavo@niemeyer.net>
282 #
283@@ -22,7 +22,8 @@
284
285 from storm.exceptions import (
286 ClassInfoError, FeatureError, NoStoreError, WrongStoreError)
287-from storm.store import Store, get_where_for_args, LostObjectError
288+from storm.store import (
289+ Store, get_where_for_args, LostObjectError, record_derived_fetch)
290 from storm.variables import LazyValue
291 from storm.expr import (
292 Select, Column, Exists, ComparableExpr, LeftJoin, Not, SQLRaw,
293@@ -133,7 +134,8 @@
294 def __get__(self, local, cls=None):
295 if local is not None:
296 # Don't use local here, as it might be security proxied.
297- local = get_obj_info(local).get_obj()
298+ local_obj_info = get_obj_info(local)
299+ local = local_obj_info.get_obj()
300
301 if self._cls is None:
302 self._cls = _find_descriptor_class(cls or local.__class__, self)
303@@ -154,11 +156,17 @@
304
305 if self._relation.remote_key_is_primary:
306 remote = store.get(self._relation.remote_cls,
307- self._relation.get_local_variables(local))
308+ self._relation.get_local_variables(local),
309+ derived_from=(local, self))
310 else:
311 where = self._relation.get_where_for_remote(local)
312 result = store.find(self._relation.remote_cls, where)
313+ result.fetch_context = local_obj_info["fetch_context"]
314+ if not result.fetch_context.is_root:
315+ result.fetch_origin = local_obj_info.get("fetch_origin")
316 remote = result.one()
317+ if remote is not None:
318+ record_derived_fetch(self, local_obj_info, get_obj_info(remote))
319
320 if remote is not None:
321 self._relation.link(local, remote)
322
323=== modified file 'storm/store.py'
324--- storm/store.py 2011-05-16 10:45:52 +0000
325+++ storm/store.py 2011-06-20 13:09:26 +0000
326@@ -1,5 +1,5 @@
327 #
328-# Copyright (c) 2006, 2007 Canonical
329+# Copyright (c) 2006-2011 Canonical
330 #
331 # Written by Gustavo Niemeyer <gustavo@niemeyer.net>
332 #
333@@ -37,6 +37,7 @@
334 from storm.exceptions import (
335 WrongStoreError, NotFlushedError, OrderLoopError, UnorderedError,
336 NotOneError, FeatureError, CompileError, LostObjectError, ClassInfoError)
337+from storm.fetch_profile import FetchContext
338 from storm import Undef
339 from storm.cache import Cache
340 from storm.event import EventSystem
341@@ -49,6 +50,48 @@
342 PENDING_REMOVE = 2
343
344
345+def record_unfetched_object(fetch_context, obj_info):
346+ """Record creation of an object not fetched from the database."""
347+ obj_info["fetch_context"] = fetch_context
348+ if not fetch_context.is_root:
349+ obj_info["fetch_origin"] = obj_info.cls_info.cls
350+
351+
352+def record_original_fetch(obj_info, fetch_context, cls):
353+ """Record an original fetch.
354+
355+ The fetch context may or may not be store's currently active context; the
356+ active context may have changed between the point where the query occurs
357+ in the program and the point where it is actually issued to the database.
358+
359+ :param obj_info: `ObjectInfo` for the object being fetched.
360+ :param fetch_context: The `FetchContext` issuing the fetch.
361+ :param cls: The class that's being fetched.
362+ """
363+ obj_info["fetch_context"] = fetch_context
364+ if not fetch_context.is_root:
365+ obj_info["fetch_origin"] = cls
366+ fetch_context.stats.record_original_fetch(cls)
367+
368+
369+def record_derived_fetch(reference, local, remote):
370+ """Record a derived fetch.
371+
372+ :param reference: The `Reference` whose dereference triggers the fetch.
373+ :param local: `ObjectInfo` for the object whose reference to the object is
374+ being followed.
375+ :param remote: `ObjectInfo` for the object that's being fetched.
376+ """
377+ fetch_context = local["fetch_context"]
378+ remote["fetch_context"] = fetch_context
379+ if not fetch_context.is_root:
380+ fetch_origin = local["fetch_origin"]
381+ remote["fetch_origin"] = fetch_origin
382+ fetch_context.stats.record_derived_fetch(fetch_origin,
383+ local.cls_info.cls,
384+ reference)
385+
386+
387 class Store(object):
388 """The Storm Store.
389
390@@ -80,6 +123,7 @@
391 self._cache = cache
392 self._implicit_flush_block_count = 0
393 self._sequence = 0 # Advisory ordering.
394+ self.fetch_context = FetchContext(None)
395
396 def get_database(self):
397 """Return this Store's Database object."""
398@@ -137,13 +181,16 @@
399 self.invalidate()
400 self._connection.rollback()
401
402- def get(self, cls, key):
403+ def get(self, cls, key, derived_from=None):
404 """Get object of type cls with the given primary key from the database.
405
406 If the object is alive the database won't be touched.
407
408 @param cls: Class of the object to be retrieved.
409 @param key: Primary key of object. May be a tuple for composed keys.
410+ @param derived_from: For profiling purposes, an optional tuple of the
411+ object that this fetch is derived from and its reference property
412+ that linked to this object.
413
414 @return: The object found with the given primary key, or None
415 if no object is found.
416@@ -176,10 +223,28 @@
417 default_tables=cls_info.table, limit=1)
418
419 result = self._connection.execute(select)
420+
421+ if derived_from is None:
422+ result.fetch_context = self.fetch_context
423+ else:
424+ origin_obj, origin_ref = derived_from
425+ origin_obj_info = get_obj_info(origin_obj)
426+ result.fetch_context = origin_obj_info["fetch_context"]
427+ if not result.fetch_context.is_root:
428+ result.fetch_origin = origin_obj_info.get("fetch_origin")
429+
430 values = result.get_one()
431 if values is None:
432 return None
433- return self._load_object(cls_info, result, values)
434+
435+ obj = self._load_object(cls_info, result, values)
436+ if derived_from is not None:
437+ obj_info = get_obj_info(obj)
438+ if origin_obj_info["store"] == self:
439+ record_derived_fetch(origin_ref, origin_obj_info, obj_info)
440+ else:
441+ record_original_fetch(obj_info, self.fetch_context, cls)
442+ return obj
443
444 def find(self, cls_spec, *args, **kwargs):
445 """Perform a query.
446@@ -260,6 +325,7 @@
447 obj_info["pending"] = PENDING_ADD
448 self._set_dirty(obj_info)
449 self._enable_lazy_resolving(obj_info)
450+ record_unfetched_object(self.fetch_context, obj_info)
451 obj_info.event.emit("added")
452
453 return obj
454@@ -710,6 +776,10 @@
455 self._set_values(obj_info, cls_info.columns, result, values,
456 replace_unknown_lazy=True)
457
458+ if result.fetch_origin is None:
459+ # This is an original fetch.
460+ record_original_fetch(obj_info, result.fetch_context, cls)
461+
462 self._add_to_alive(obj_info)
463 self._enable_change_notification(obj_info)
464 self._enable_lazy_resolving(obj_info)
465@@ -895,6 +965,23 @@
466 self._set_values(obj_info, autoreload_columns,
467 result, result.get_one())
468
469+ def push_fetch_context(self, context_name):
470+ """Enter a fetch context.
471+
472+ If no fetch context was active previously, this enables
473+ profiling.
474+ """
475+ self.fetch_context = self.fetch_context.get_child(context_name)
476+
477+ def pop_fetch_context(self):
478+ """Leave the current fetch context.
479+
480+ If the current context was the outermost one, this disables
481+ profiling.
482+ """
483+ assert not self.fetch_context.is_root, "Popped root fetch context."
484+ self.fetch_context = self.fetch_context.parent
485+
486
487 class ResultSet(object):
488 """The representation of the results of a query.
489@@ -920,6 +1007,8 @@
490 self._distinct = False
491 self._group_by = Undef
492 self._having = Undef
493+ self.fetch_context = store.fetch_context
494+ self.fetch_origin = None
495
496 def copy(self):
497 """Return a copy of this ResultSet object, with the same configuration.
498@@ -976,6 +1065,7 @@
499 """Iterate the results of the query.
500 """
501 result = self._store._connection.execute(self._get_select())
502+ result.fetch_context = self.fetch_context
503 for values in result:
504 yield self._load_objects(result, values)
505
506@@ -1068,6 +1158,7 @@
507 select.limit = 1
508 select.order_by = Undef
509 result = self._store._connection.execute(select)
510+ result.fetch_context = self.fetch_context
511 values = result.get_one()
512 if values:
513 return self._load_objects(result, values)
514@@ -1081,6 +1172,7 @@
515 select = self._get_select()
516 select.limit = 1
517 result = self._store._connection.execute(select)
518+ result.fetch_context = self.fetch_context
519 values = result.get_one()
520 if values:
521 return self._load_objects(result, values)
522@@ -1122,6 +1214,7 @@
523 else:
524 select.order_by.append(Desc(expr))
525 result = self._store._connection.execute(select)
526+ result.fetch_context = self.fetch_context
527 values = result.get_one()
528 if values:
529 return self._load_objects(result, values)
530@@ -1140,6 +1233,7 @@
531 if select.limit is not Undef and select.limit > 2:
532 select.limit = 2
533 result = self._store._connection.execute(select)
534+ result.fetch_context = self.fetch_context
535 values = result.get_one()
536 if result.get_one():
537 raise NotOneError("one() used with more than one result available")
538
539=== added file 'tests/fetch_context.py'
540--- tests/fetch_context.py 1970-01-01 00:00:00 +0000
541+++ tests/fetch_context.py 2011-06-20 13:09:26 +0000
542@@ -0,0 +1,164 @@
543+# -*- coding: utf-8 -*-
544+
545+from storm.fetch_profile import FetchContext, FetchStatistics, fetch_context
546+
547+from tests.helper import TestHelper
548+
549+
550+class FakeStats(object):
551+ def __init__(self, contents=None):
552+ if contents is None:
553+ self.contents = set()
554+ else:
555+ self.contents = set(contents)
556+
557+ def merge(self, other_stats):
558+ self.contents = self.contents.union(other_stats.contents)
559+
560+
561+class FakeStore(object):
562+ def __init__(self):
563+ self.fetch_context = FetchContext(None)
564+
565+ def push_fetch_context(self, name):
566+ self.fetch_context = FetchContext(name, parent=self.fetch_context)
567+
568+ def pop_fetch_context(self):
569+ self.fetch_context = self.fetch_context.parent
570+
571+
572+class FetchContextTest(TestHelper):
573+ def get_relatives(self, context):
574+ """Return a tuple of `context`s parent and children."""
575+ return (context.parent, context.children)
576+
577+ def test_initially_childless(self):
578+ self.assertEqual({}, FetchContext("context").children)
579+
580+ def test_iter_childless_context_yields_nothing(self):
581+ self.assertEqual([], list(FetchContext("context")))
582+
583+ def test_iter_does_not_yield_parent(self):
584+ parent = FetchContext("parent")
585+ child = parent.get_child("child")
586+ self.assertEqual([], list(child))
587+
588+ def test_iter_context_with_children_yields_children(self):
589+ root = FetchContext("root")
590+ one = root.get_child("one")
591+ two = root.get_child("two")
592+ self.assertEqual(set([one, two]), set(root))
593+
594+ def test_iter_context_includes_grandchildren(self):
595+ root = FetchContext("root")
596+ child = root.get_child("child")
597+ grandchild = child.get_child("grandchild")
598+ self.assertEqual(set([child, grandchild]), set(root))
599+
600+ def test_iter_context_includes_grand_grandchildren(self):
601+ root = FetchContext("root")
602+ child = root.get_child("child")
603+ grandchild = child.get_child("grandchild")
604+ grand_grandchild = grandchild.get_child("grand-grandchild")
605+ self.assertEqual(set([child, grandchild, grand_grandchild]), set(root))
606+
607+ def test_is_root_for_root(self):
608+ self.assertTrue(FetchContext("root").is_root())
609+
610+ def test_is_root_for_child(self):
611+ root = FetchContext("root")
612+ self.assertFalse(root.get_child("child").is_root())
613+
614+ def test_get_child_creates_first_child(self):
615+ parent = FetchContext("parent")
616+ child = parent.get_child("child")
617+
618+ self.assertEqual((None, {"child": child}), self.get_relatives(parent))
619+ self.assertEqual((parent, {}), self.get_relatives(child))
620+
621+ def test_get_child_adds_child(self):
622+ parent = FetchContext("parent")
623+ eldest = parent.get_child("eldest")
624+ youngest = parent.get_child("youngest")
625+
626+ children = {
627+ "eldest": eldest,
628+ "youngest": youngest,
629+ }
630+ self.assertEqual((None, children), self.get_relatives(parent))
631+ self.assertEqual((parent, {}), self.get_relatives(eldest))
632+ self.assertEqual((parent, {}), self.get_relatives(youngest))
633+ self.assertNotEqual(eldest, youngest)
634+
635+ def test_get_child_finds_child(self):
636+ parent = FetchContext("parent")
637+ child = parent.get_child("child")
638+ self.assertEqual(child, parent.get_child("child"))
639+
640+ def test_cumulate_stats_on_empty_context_yields_empty(self):
641+ stats = FetchContext("context").cumulate_stats()
642+ self.assertEqual({}, stats.original_fetches)
643+ self.assertEqual({}, stats.derived_fetches)
644+
645+ def test_cumulate_stats_includes_local_stats(self):
646+ context = FetchContext("context")
647+ context.stats.original_fetches = {("origin", "reference", "store"): 1}
648+ self.assertEqual(context.stats.original_fetches,
649+ context.cumulate_stats().original_fetches)
650+
651+ def test_cumulate_stats_includes_child_stats(self):
652+ parent = FetchContext("parent")
653+ child = parent.get_child("child")
654+ child.stats.original_fetches = {("origin", "reference", "store"): 1}
655+ self.assertEqual(child.stats.original_fetches,
656+ parent.cumulate_stats().original_fetches)
657+
658+ def test_aggregate_stats_by_name_includes_local_stats(self):
659+ context = FetchContext("context")
660+ fetch = ("origin", "reference", "store")
661+ context.stats.original_fetches[fetch] = 1
662+ stats = context.aggregate_stats_by_name()
663+ self.assertEqual(["context"], stats.keys())
664+ self.assertEqual({fetch: 1}, stats["context"].original_fetches)
665+
666+ def test_aggregate_stats_by_name_includes_child_stats(self):
667+ parent = FetchContext("parent")
668+ child = parent.get_child("child")
669+ fetch = ("origin", "reference", "store")
670+ child.stats.original_fetches[fetch] = 1
671+ stats = parent.aggregate_stats_by_name()
672+ self.assertEqual({fetch: 1}, stats["child"].original_fetches)
673+
674+ def test_aggregate_stats_by_name_aggregates(self):
675+ root = FetchContext("x")
676+ fetch = ("origin", "reference", "store")
677+ root.stats.original_fetches[fetch] = 1
678+ root.get_child("x").stats.original_fetches[fetch] = 1
679+ stats = root.aggregate_stats_by_name()
680+ self.assertEqual(["x"], stats.keys())
681+ self.assertEqual({fetch: 2}, stats["x"].original_fetches)
682+
683+ def test_context_manager_pushes_context(self):
684+ store = FakeStore()
685+ with fetch_context(store, "with"):
686+ current_context = store.fetch_context.name
687+ self.assertEqual("with", current_context)
688+
689+ def test_context_manager_pops_context_on_normal_exit(self):
690+ store = FakeStore()
691+ with fetch_context(store, "with"):
692+ pass
693+ self.assertTrue(store.fetch_context.is_root())
694+
695+ def test_context_manager_pops_context_on_exception(self):
696+ class ArbitraryException(Exception):
697+ pass
698+
699+ store = FakeStore()
700+ try:
701+ with fetch_context(store, "with"):
702+ raise ArbitraryException()
703+ except ArbitraryException:
704+ pass
705+
706+ self.assertTrue(store.fetch_context.is_root())
707
708=== added file 'tests/fetch_profile.py'
709--- tests/fetch_profile.py 1970-01-01 00:00:00 +0000
710+++ tests/fetch_profile.py 2011-06-20 13:09:26 +0000
711@@ -0,0 +1,64 @@
712+# -*- coding: utf-8 -*-
713+
714+from storm.store import Store, record_original_fetch, record_derived_fetch
715+
716+from tests.helper import TestHelper
717+
718+
719+class DummyDatabase(object):
720+
721+ def connect(self, event=None):
722+ return None
723+
724+
725+class FetchProfilingTest(TestHelper):
726+
727+ def test_initial_context_is_root(self):
728+ store = Store(DummyDatabase())
729+ self.assertTrue(store.fetch_context.is_root())
730+
731+ def test_push_fetch_context(self):
732+ store = Store(DummyDatabase())
733+ store.push_fetch_context("context")
734+ self.assertFalse(store.fetch_context.is_root())
735+
736+ def test_pop_fetch_context(self):
737+ store = Store(DummyDatabase())
738+ store.push_fetch_context("context")
739+ store.pop_fetch_context()
740+ self.assertTrue(store.fetch_context.is_root())
741+
742+ def record_original_fetch(self):
743+ store = Store(DummyDatabase())
744+ store.push_fetch_context("context")
745+ fake_object = {"store": store}
746+ record_original_fetch(fake_object, store.fetch_context, "class")
747+ self.assertEqual({"class": 1},
748+ store.fetch_context.stats.original_fetches)
749+ self.assertEqual(store.fetch_context, fake_object["fetch_context"])
750+
751+ def test_record_derived_fetch(self):
752+ class FakeObjInfo(dict):
753+ pass
754+ class FakeClsInfo(object):
755+ def __init__(self, cls):
756+ self.cls = cls
757+
758+ store = Store(DummyDatabase())
759+ store.push_fetch_context("context")
760+ fake_local_object = FakeObjInfo(store=store,
761+ fetch_context=store.fetch_context,
762+ fetch_origin="origin")
763+ fake_local_object.cls_info = FakeClsInfo("source")
764+ fake_remote_object = FakeObjInfo(store=store)
765+ record_derived_fetch("reference", fake_local_object, fake_remote_object)
766+
767+ self.assertEqual({("origin", "source", "reference"): 1},
768+ store.fetch_context.stats.derived_fetches)
769+ self.assertEqual("origin", fake_remote_object["fetch_origin"])
770+
771+ def test_root_context_does_not_profile(self):
772+ store = Store(DummyDatabase())
773+ fake_object = {"store": store}
774+ record_original_fetch(fake_object, store.fetch_context, "class")
775+ self.assertEqual({}, store.fetch_context.stats.original_fetches)
776
777=== added file 'tests/fetch_statistics.py'
778--- tests/fetch_statistics.py 1970-01-01 00:00:00 +0000
779+++ tests/fetch_statistics.py 2011-06-20 13:09:26 +0000
780@@ -0,0 +1,99 @@
781+# -*- coding: utf-8 -*-
782+
783+from storm.fetch_profile import add_to_dict, FetchStatistics
784+
785+from tests.helper import TestHelper
786+
787+
788+class AddToDictTest(TestHelper):
789+ def test_creates_entry(self):
790+ data = {}
791+ add_to_dict(data, "x", 1)
792+ self.assertEqual({"x": 1}, data)
793+
794+ def test_adds_to_entry(self):
795+ data = {"x": 1}
796+ add_to_dict(data, "x", 1)
797+ self.assertEqual({"x": 2}, data)
798+
799+
800+class FetchStatisticsTest(TestHelper):
801+ def test_initially_empty(self):
802+ empty = FetchStatistics()
803+ self.assertEqual({}, empty.original_fetches)
804+ self.assertEqual({}, empty.derived_fetches)
805+
806+ def test_record_original_fetch(self):
807+ stats = FetchStatistics()
808+ stats.record_original_fetch("origin")
809+ self.assertEqual({"origin": 1}, stats.original_fetches)
810+
811+ def test_record_derived_fetch(self):
812+ stats = FetchStatistics()
813+ fetch = ("origin", "source", "reference")
814+ stats.record_derived_fetch(*fetch)
815+ self.assertEqual({fetch: 1}, stats.derived_fetches)
816+
817+ def test_copy(self):
818+ stats = FetchStatistics()
819+ stats.record_original_fetch("origin")
820+ stats.record_derived_fetch("origin", "source", "reference")
821+ copy = stats.copy()
822+ self.assertEqual(stats.original_fetches, copy.original_fetches)
823+ self.assertEqual(stats.derived_fetches, copy.derived_fetches)
824+ self.assertNotEqual(stats, copy)
825+
826+ def test_merge_empty_does_nothing(self):
827+ stats = FetchStatistics()
828+ stats.original_fetches = {"origin": 1}
829+ derived_fetch = ("origin", "source", "reference")
830+ stats.derived_fetches = {derived_fetch: 1}
831+ empty = FetchStatistics()
832+ stats.merge(empty)
833+ self.assertEqual({"origin": 1}, stats.original_fetches)
834+ self.assertEqual({derived_fetch: 1}, stats.derived_fetches)
835+
836+ def test_merge_adds_counts(self):
837+ stats = FetchStatistics()
838+ other_stats = FetchStatistics()
839+ other_stats.original_fetches = {"other_origin": 1}
840+ derived_fetch = ("other_origin", "other_source", "reference")
841+ other_stats.derived_fetches = {derived_fetch: 1}
842+ stats.merge(other_stats)
843+ self.assertEqual(other_stats.original_fetches, stats.original_fetches)
844+ self.assertEqual(other_stats.derived_fetches, stats.derived_fetches)
845+
846+ def test_merge_leaves_existing_counts_in_place(self):
847+ stats = FetchStatistics()
848+ stats.original_fetches = {"origin": 1}
849+ derived_fetch = ("origin", "source", "reference")
850+ stats.derived_fetches = {derived_fetch: 1}
851+ other_stats = FetchStatistics()
852+ other_stats.original_fetches = {"other_origin": 1}
853+ other_derived_fetch = ("other_origin", "other_source", "reference")
854+ other_stats.derived_fetches = {other_derived_fetch: 1}
855+ stats.merge(other_stats)
856+
857+ cumulative_original_fetches = {
858+ "origin": 1,
859+ "other_origin": 1,
860+ }
861+ cumulative_derived_fetches = {
862+ derived_fetch: 1,
863+ other_derived_fetch: 1,
864+ }
865+ self.assertEqual(cumulative_original_fetches, stats.original_fetches)
866+ self.assertEqual(cumulative_derived_fetches, stats.derived_fetches)
867+
868+ def test_merge_sums_counts(self):
869+ stats = FetchStatistics()
870+ stats.original_fetches = {"origin": 1}
871+ derived_fetch = ("origin", "source", "reference")
872+ stats.derived_fetches = {derived_fetch: 1}
873+ other_stats = FetchStatistics()
874+ other_stats.original_fetches = {"origin": 1}
875+ other_stats.derived_fetches = {derived_fetch: 1}
876+ stats.merge(other_stats)
877+
878+ self.assertEqual({"origin": 2}, stats.original_fetches)
879+ self.assertEqual({derived_fetch: 2}, stats.derived_fetches)
880
881=== modified file 'tests/store/base.py'
882--- tests/store/base.py 2011-02-14 12:17:54 +0000
883+++ tests/store/base.py 2011-06-20 13:09:26 +0000
884@@ -29,8 +29,13 @@
885
886 from storm.references import Reference, ReferenceSet, Proxy
887 from storm.database import Result
888+<<<<<<< TREE
889 from storm.properties import (
890 Int, Float, RawStr, Unicode, Property, Pickle, UUID)
891+=======
892+from storm.fetch_profile import fetch_context
893+from storm.properties import Int, Float, RawStr, Unicode, Property, Pickle
894+>>>>>>> MERGE-SOURCE
895 from storm.properties import PropertyPublisherMeta, Decimal
896 from storm.variables import PickleVariable
897 from storm.expr import (
898@@ -6004,6 +6009,212 @@
899 result_to_remove = self.store.find(Foo, Foo.id <= 30)
900 self.assertEquals(result_to_remove.remove(), 3)
901
902+ def test_push_fetch_context(self):
903+ root = self.store.fetch_context
904+ self.store.push_fetch_context("child")
905+ self.assertEqual(root, self.store.fetch_context.parent)
906+
907+ def test_pop_fetch_context(self):
908+ root = self.store.fetch_context
909+ self.store.push_fetch_context("child")
910+ self.store.pop_fetch_context()
911+ self.assertEqual(root, self.store.fetch_context)
912+
913+ def test_fetch_context_manager(self):
914+ with fetch_context(self.store, "with-context"):
915+ context_name = self.store.fetch_context.name
916+ self.assertEqual("with-context", context_name)
917+
918+ def test_profile_find(self):
919+ self.store.push_fetch_context("test")
920+ obj = self.store.find(Foo).any()
921+ stats = self.store.fetch_context.stats
922+ self.assertEqual({Foo: 1}, stats.original_fetches)
923+ self.assertEqual({}, stats.derived_fetches)
924+
925+ def test_profile_get(self):
926+ self.store.push_fetch_context("test")
927+ obj = self.store.get(Foo, 10)
928+ stats = self.store.fetch_context.stats
929+ self.assertEqual({Foo: 1}, stats.original_fetches)
930+ self.assertEqual({}, stats.derived_fetches)
931+
932+ def test_profile_get_derived_from(self):
933+ self.store.push_fetch_context("test")
934+ bar = self.store.get(Bar, 100)
935+ foo = self.store.get(Foo, bar.foo_id, derived_from=(bar, Bar.foo))
936+ stats = self.store.fetch_context.stats
937+ self.assertEqual({Bar: 1}, stats.original_fetches)
938+ fetch = (Bar, Bar, Bar.foo)
939+ self.assertEqual({fetch: 1}, stats.derived_fetches)
940+
941+ def test_profile_dereference(self):
942+ self.store.push_fetch_context("test")
943+ bar = self.store.get(Bar, 100)
944+ foo = bar.foo
945+ stats = self.store.fetch_context.stats
946+ fetch = (Bar, Bar, Bar.foo)
947+ self.assertEqual({fetch: 1}, stats.derived_fetches)
948+
949+ def test_profile_indirect_derived_fetch_records_origin_and_source(self):
950+ self.store.execute("""
951+ CREATE TEMPORARY TABLE splat (id integer, bar_id integer)
952+ """)
953+ self.store.execute("INSERT INTO splat (id, bar_id) VALUES (1, 100)")
954+
955+ class Splat(object):
956+ __storm_table__ = "splat"
957+ id = Int(primary=True)
958+ bar_id = Int()
959+ bar = Reference(bar_id, Bar.id)
960+
961+ with fetch_context(self.store, "test"):
962+ splat = self.store.get(Splat, 1)
963+ context = self.store.fetch_context
964+
965+ foo = splat.bar.foo
966+
967+ expected_fetches = {
968+ (Splat, Splat, Splat.bar): 1,
969+ (Splat, Bar, Bar.foo): 1,
970+ }
971+ self.assertEqual(expected_fetches, context.stats.derived_fetches)
972+
973+ def test_profile_derived_get_records_origin_and_source(self):
974+ self.store.execute("""
975+ CREATE TEMPORARY TABLE splat (id integer, bar_id integer)
976+ """)
977+ self.store.execute("INSERT INTO splat (id, bar_id) VALUES (1, 100)")
978+
979+ class Splat(object):
980+ __storm_table__ = "splat"
981+ id = Int(primary=True)
982+ bar_id = Int()
983+ bar = Reference(bar_id, Bar.id)
984+
985+ with fetch_context(self.store, "test"):
986+ splat = self.store.get(Splat, 1)
987+ context = self.store.fetch_context
988+
989+ bar = splat.bar
990+ self.store.get(Foo, bar.foo_id, derived_from=(bar, Bar.foo))
991+
992+ expected_fetches = {
993+ (Splat, Splat, Splat.bar): 1,
994+ (Splat, Bar, Bar.foo): 1,
995+ }
996+ self.assertEqual(expected_fetches, context.stats.derived_fetches)
997+
998+
999+ def test_profile_new_object_is_origin_but_not_fetched(self):
1000+ self.store.push_fetch_context("test")
1001+ bar = Bar()
1002+ bar.id = 999
1003+ bar.foo_id = 10
1004+ self.store.add(bar)
1005+ foo = bar.foo
1006+ stats = self.store.fetch_context.stats
1007+ self.assertEqual({}, stats.original_fetches)
1008+ self.assertEqual({(Bar, Bar, Bar.foo): 1}, stats.derived_fetches)
1009+
1010+ def test_profile_cached_objects_not_fetched(self):
1011+ foo = self.store.get(Foo, 10)
1012+ bar = self.store.get(Bar, 100)
1013+ self.store.push_fetch_context("test")
1014+ same_foo = bar.foo
1015+ self.assertEqual({}, self.store.fetch_context.stats.original_fetches)
1016+ self.assertEqual({}, self.store.fetch_context.stats.derived_fetches)
1017+
1018+ def test_profile_derived_fetch_uses_original_context(self):
1019+ with fetch_context(self.store, "original-fetch-context"):
1020+ original_context = self.store.fetch_context
1021+ bar = self.store.get(Bar, 100)
1022+ with fetch_context(self.store, "later-fetch-context"):
1023+ later_context = self.store.fetch_context
1024+ foo = bar.foo
1025+ self.assertEqual({}, later_context.stats.derived_fetches)
1026+ self.assertEqual({(Bar, Bar, Bar.foo): 1},
1027+ original_context.stats.derived_fetches)
1028+
1029+ def test_profile_result_uses_original_context(self):
1030+ with fetch_context(self.store, "original-fetch-context"):
1031+ original_context = self.store.fetch_context
1032+ bar_result = self.store.find(Foo, Foo.id == 10)
1033+ with fetch_context(self.store, "later-fetch-context"):
1034+ later_context = self.store.fetch_context
1035+ bar = bar_result.one()
1036+ self.assertEqual({}, later_context.stats.original_fetches)
1037+ self.assertEqual({Foo: 1}, original_context.stats.original_fetches)
1038+
1039+ def test_profile_result_find_uses_original_context(self):
1040+ with fetch_context(self.store, "original-fetch-context"):
1041+ original_context = self.store.fetch_context
1042+ original_result = self.store.find(Foo, Foo.id == 10)
1043+ with fetch_context(self.store, "later-fetch-context"):
1044+ later_context = self.store.fetch_context
1045+ original_result.find(True).one()
1046+ self.assertEqual({}, later_context.stats.original_fetches)
1047+ self.assertEqual({Foo: 1}, original_context.stats.original_fetches)
1048+
1049+ def test_profile_contexts_persist(self):
1050+ with fetch_context(self.store, "context"):
1051+ foo = self.store.get(Foo, 10)
1052+ context = self.store.fetch_context
1053+ with fetch_context(self.store, "context"):
1054+ bar = self.store.get(Foo, 20)
1055+ self.assertEqual({Foo: 2}, context.stats.original_fetches)
1056+
1057+ def test_profile_does_not_count_empty_result(self):
1058+ self.store.push_fetch_context("context")
1059+ self.store.find(Foo, False).any()
1060+ self.assertEqual({}, self.store.fetch_context.stats.original_fetches)
1061+
1062+ def test_profile_counts_objects_fetched(self):
1063+ self.store.push_fetch_context("context")
1064+ list(self.store.find(Foo, Foo.id.is_in([10, 20])))
1065+ self.assertEqual({Foo: 2},
1066+ self.store.fetch_context.stats.original_fetches)
1067+
1068+ def test_profile_counts_all_objects_in_join(self):
1069+ self.store.push_fetch_context("context")
1070+ list(self.store.find((Foo, Bar), Foo.id == Bar.foo_id, Foo.id == 10))
1071+ expected_fetches = {
1072+ Foo: 1,
1073+ Bar: 1,
1074+ }
1075+ self.assertEqual(expected_fetches,
1076+ self.store.fetch_context.stats.original_fetches)
1077+
1078+ def test_profile_tracks_origin_within_join(self):
1079+ self.store.execute("UPDATE %s SET selfref_id = %d WHERE id = %d" % (
1080+ SelfRef.__storm_table__, 25, 15))
1081+ self.store.push_fetch_context("context")
1082+ query = self.store.find((Bar, SelfRef),
1083+ Bar.id == 100,
1084+ SelfRef.id == 15)
1085+ (bar, selfref) = query.one()
1086+ foo = bar.foo
1087+ expected_derived_fetches = {
1088+ (Bar, Bar, Bar.foo): 1,
1089+ }
1090+ self.assertEqual(expected_derived_fetches,
1091+ self.store.fetch_context.stats.derived_fetches)
1092+ other_selfref = selfref.selfref
1093+ expected_derived_fetches[(SelfRef, SelfRef, SelfRef.selfref)] = 1
1094+ self.assertEqual(expected_derived_fetches,
1095+ self.store.fetch_context.stats.derived_fetches)
1096+
1097+ def test_profile_derived_fetch_on_different_store_is_original_fetch(self):
1098+ self.store.push_fetch_context("context")
1099+ bar = self.store.get(Bar, 100)
1100+ other_store = self.create_store()
1101+ other_store.push_fetch_context("remote-context")
1102+ other_store.get(Foo, bar.foo_id, derived_from=(bar, Bar.foo))
1103+ self.assertEqual({}, self.store.fetch_context.stats.derived_fetches)
1104+ self.assertEqual({Foo: 1},
1105+ other_store.fetch_context.stats.original_fetches)
1106+ self.assertEqual({}, other_store.fetch_context.stats.derived_fetches)
1107+
1108
1109 class EmptyResultSetTest(object):
1110

Subscribers

People subscribed via source and target branches

to status/vote changes: