Merge ~cjwatson/launchpad:remove-tickcount into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 85a4bb4795ff19b480b3cf4996ac577b59c591ac
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:remove-tickcount
Merge into: launchpad:master
Diff against target: 594 lines (+206/-108)
3 files modified
lib/lp/services/webapp/doc/webapp-publication.txt (+121/-67)
lib/lp/services/webapp/publication.py (+78/-34)
lib/lp/services/webapp/servers.py (+7/-7)
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email: mp+373764@code.launchpad.net

Commit message

Remove tickcount and track traversal/publication durations instead

Description of the change

tickcount can't work on Python 3, so in the long term we don't have much choice here. We do want to be able to spot contention, but tracking thread time should be enough here. (Unfortunately, as far as I can tell this is only possible with Python >= 3.3.)

This does change the access log format slightly (logging durations rather than ticks), but as far as I can tell nothing cares.

This is essentially the same as https://code.launchpad.net/~cjwatson/launchpad/remove-tickcount/+merge/373347, converted to git and rebased on master.

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/lib/lp/services/webapp/doc/webapp-publication.txt b/lib/lp/services/webapp/doc/webapp-publication.txt
index 4406d41..a8ad43a 100644
--- a/lib/lp/services/webapp/doc/webapp-publication.txt
+++ b/lib/lp/services/webapp/doc/webapp-publication.txt
@@ -621,52 +621,62 @@ name of the context class and the view class name.
621 TestContext:TestView621 TestContext:TestView
622622
623623
624Tick counts624Durations
625-----------625---------
626626
627Similarly to our page IDs, our publication implementation will store the627Similarly to our page IDs, our publication implementation will store the
628tick counts for the traversal and object publication processes in WSGI628durations for the traversal and object publication processes in WSGI
629variables which aren't set originally.629variables which aren't set originally.
630630
631 >>> import tickcount631 >>> import time
632 >>> request, publication = get_request_and_publication()632 >>> request, publication = get_request_and_publication()
633 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())633 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())
634 >>> logout()634 >>> logout()
635635
636For traversal we start counting the ticks during the beforeTraversal()636For traversal we start counting the duration during the beforeTraversal()
637hook and stop the count in afterTraversal(). The tick count is then637hook and stop the count in afterTraversal(). The duration is then available
638available as launchpad.traversalticks in the WSGI environment.638as launchpad.traversalduration in the WSGI environment. On Python >= 3.3,
639there is also launchpad.traversalthreadduration with the time spent in the
640current thread.
639641
640 >>> 'launchpad.traversalticks' in request._orig_env642 >>> 'launchpad.traversalduration' in request._orig_env
643 False
644 >>> 'launchpad.traversalthreadduration' in request._orig_env
641 False645 False
642 >>> publication.beforeTraversal(request)646 >>> publication.beforeTraversal(request)
643 >>> tickcount.difference(
644 ... request._traversalticks_start, tickcount.tickcount()) > 0
645 True
646
647 >>> publication.afterTraversal(request, None)647 >>> publication.afterTraversal(request, None)
648 >>> request._orig_env['launchpad.traversalticks'] < 200648 >>> 'launchpad.traversalduration' in request._orig_env
649 True
650 >>> if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
651 ... 'launchpad.traversalthreadduration' in request._orig_env
652 ... else:
653 ... True
649 True654 True
650655
651For publication we start counting the ticks during the callObject()656For publication we start counting the duration during the callObject()
652hook and stop the count in afterCall(). The tick count is then657hook and stop the count in afterCall(). The duration is then available as
653available as launchpad.publicationticks in the WSGI environment.658launchpad.publicationduration in the WSGI environment. On Python >= 3.3,
659there is also launchpad.publicationthreadduration with the time spent in the
660current thread.
654661
655 >>> 'launchpad.publicationticks' in request._orig_env662 >>> 'launchpad.publicationduration' in request._orig_env
663 False
664 >>> 'launchpad.publicationthreadduration' in request._orig_env
656 False665 False
657 >>> publication.callObject(request, TestView(TestContext(), request))666 >>> publication.callObject(request, TestView(TestContext(), request))
658 u'Result'667 u'Result'
659 >>> tickcount.difference(
660 ... request._publicationticks_start, tickcount.tickcount()) > 0
661 True
662
663 >>> publication.afterCall(request, None)668 >>> publication.afterCall(request, None)
664 >>> request._orig_env['launchpad.publicationticks'] < 60669 >>> 'launchpad.publicationduration' in request._orig_env
670 True
671 >>> if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
672 ... 'launchpad.publicationthreadduration' in request._orig_env
673 ... else:
674 ... True
665 True675 True
666 >>> publication.endRequest(request, None)676 >>> publication.endRequest(request, None)
667677
668If an exception is raised during traversal or object publication, we'll678If an exception is raised during traversal or object publication, we'll
669store the ticks up to the point in which the exception is raised. This679store the durations up to the point in which the exception is raised. This
670is done inside the handleException() hook. (The hook also sets and resets680is done inside the handleException() hook. (The hook also sets and resets
671the request timer from lp.services.webapp.adapter, so you'll notice681the request timer from lp.services.webapp.adapter, so you'll notice
672some calls to prepare that code to what handleException expects.)682some calls to prepare that code to what handleException expects.)
@@ -687,26 +697,37 @@ there's nothing to store.
687 >>> set_request_started()697 >>> set_request_started()
688 >>> publication.handleException(698 >>> publication.handleException(
689 ... None, request, exc_info, retry_allowed=False)699 ... None, request, exc_info, retry_allowed=False)
690 >>> 'launchpad.traversalticks' in request._orig_env700 >>> 'launchpad.traversalduration' in request._orig_env
701 False
702 >>> 'launchpad.traversalthreadduration' in request._orig_env
691 False703 False
692 >>> 'launchpad.publicationticks' in request._orig_env704 >>> 'launchpad.publicationduration' in request._orig_env
705 False
706 >>> 'launchpad.publicationthreadduration' in request._orig_env
693 False707 False
694 >>> clear_request_started()708 >>> clear_request_started()
695709
696If we started the traversal, but haven't finished it, we'll only have710If we started the traversal, but haven't finished it, we'll only have
697the ticks for the traversal and not for the publication.711the duration for the traversal and not for the publication.
698712
699 >>> publication.beforeTraversal(request)713 >>> publication.beforeTraversal(request)
700 >>> publication.handleException(714 >>> publication.handleException(
701 ... None, request, exc_info, retry_allowed=False)715 ... None, request, exc_info, retry_allowed=False)
702 >>> request._orig_env['launchpad.traversalticks'] < 200716 >>> 'launchpad.traversalduration' in request._orig_env
717 True
718 >>> if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
719 ... 'launchpad.traversalthreadduration' in request._orig_env
720 ... else:
721 ... True
703 True722 True
704 >>> 'launchpad.publicationticks' in request._orig_env723 >>> 'launchpad.publicationduration' in request._orig_env
724 False
725 >>> 'launchpad.publicationthreadduration' in request._orig_env
705 False726 False
706 >>> clear_request_started()727 >>> clear_request_started()
707728
708If we started the publication (which means the traversal has been729If we started the publication (which means the traversal has been
709completed), we'll have the ticks for the traversal and the ticks for730completed), we'll have the duration for the traversal and the duration for
710the publication, up to the point where it was forcefully stopped.731the publication, up to the point where it was forcefully stopped.
711732
712 >>> publication.afterTraversal(request, None)733 >>> publication.afterTraversal(request, None)
@@ -715,22 +736,34 @@ the publication, up to the point where it was forcefully stopped.
715 >>> set_request_started()736 >>> set_request_started()
716 >>> publication.handleException(737 >>> publication.handleException(
717 ... None, request, exc_info, retry_allowed=False)738 ... None, request, exc_info, retry_allowed=False)
718 >>> request._orig_env['launchpad.traversalticks'] < 500739 >>> 'launchpad.traversalduration' in request._orig_env
740 True
741 >>> if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
742 ... 'launchpad.traversalthreadduration' in request._orig_env
743 ... else:
744 ... True
745 True
746 >>> 'launchpad.publicationduration' in request._orig_env
719 True747 True
720 >>> request._orig_env['launchpad.publicationticks'] < 50748 >>> if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
749 ... 'launchpad.publicationthreadduration' in request._orig_env
750 ... else:
751 ... True
721 True752 True
722 >>> publication.endRequest(request, None)753 >>> publication.endRequest(request, None)
723 >>> logger.setLevel(old_level)754 >>> logger.setLevel(old_level)
724755
725When a Retry or DisconnectionError exception is raised and the request756When a Retry or DisconnectionError exception is raised and the request
726supports retry, it will be retried with a copy of the WSGI environment.757supports retry, it will be retried with a copy of the WSGI environment.
727If that happens, though, we'll remove the [publication,traversal]ticks758If that happens, though, we'll remove the
728variables from there.759{publication,traversal}{,thread}duration variables from there.
729760
730 >>> request, publication = get_request_and_publication()761 >>> request, publication = get_request_and_publication()
731 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())762 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())
732 >>> request._orig_env['launchpad.traversalticks'] = 500763 >>> request._orig_env['launchpad.traversalduration'] = 0.5
733 >>> request._orig_env['launchpad.publicationticks'] = 500764 >>> request._orig_env['launchpad.traversalthreadduration'] = 0.4
765 >>> request._orig_env['launchpad.publicationduration'] = 0.5
766 >>> request._orig_env['launchpad.publicationthreadduration'] = 0.4
734 >>> request.supportsRetry()767 >>> request.supportsRetry()
735 True768 True
736 >>> from zope.publisher.interfaces import Retry769 >>> from zope.publisher.interfaces import Retry
@@ -744,15 +777,21 @@ variables from there.
744 ...777 ...
745 Retry: foo778 Retry: foo
746779
747 >>> 'launchpad.publicationticks' in request._orig_env780 >>> 'launchpad.publicationduration' in request._orig_env
748 False781 False
749 >>> 'launchpad.traversalticks' in request._orig_env782 >>> 'launchpad.publicationthreadduration' in request._orig_env
783 False
784 >>> 'launchpad.traversalduration' in request._orig_env
785 False
786 >>> 'launchpad.traversalthreadduration' in request._orig_env
750 False787 False
751788
752 >>> request, publication = get_request_and_publication()789 >>> request, publication = get_request_and_publication()
753 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())790 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())
754 >>> request._orig_env['launchpad.traversalticks'] = 500791 >>> request._orig_env['launchpad.traversalduration'] = 0.5
755 >>> request._orig_env['launchpad.publicationticks'] = 500792 >>> request._orig_env['launchpad.traversalthreadduration'] = 0.4
793 >>> request._orig_env['launchpad.publicationduration'] = 0.5
794 >>> request._orig_env['launchpad.publicationthreadduration'] = 0.4
756 >>> request.supportsRetry()795 >>> request.supportsRetry()
757 True796 True
758 >>> from storm.exceptions import DisconnectionError797 >>> from storm.exceptions import DisconnectionError
@@ -766,19 +805,25 @@ variables from there.
766 ...805 ...
767 Retry: foo DisconnectionError806 Retry: foo DisconnectionError
768807
769 >>> 'launchpad.publicationticks' in request._orig_env808 >>> 'launchpad.publicationduration' in request._orig_env
809 False
810 >>> 'launchpad.publicationthreadduration' in request._orig_env
770 False811 False
771 >>> 'launchpad.traversalticks' in request._orig_env812 >>> 'launchpad.traversalduration' in request._orig_env
813 False
814 >>> 'launchpad.traversalthreadduration' in request._orig_env
772 False815 False
773816
774Of course, any request can only be retried a certain number of times and817Of course, any request can only be retried a certain number of times and
775when we reach that number of retries we don't pop the tick conts from the818when we reach that number of retries we don't pop the durations from the
776WSGI env.819WSGI env.
777820
778 >>> request, publication = get_request_and_publication()821 >>> request, publication = get_request_and_publication()
779 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())822 >>> request.setPrincipal(auth_utility.unauthenticatedPrincipal())
780 >>> request._orig_env['launchpad.traversalticks'] = 500823 >>> request._orig_env['launchpad.traversalduration'] = 0.5
781 >>> request._orig_env['launchpad.publicationticks'] = 500824 >>> request._orig_env['launchpad.traversalthreadduration'] = 0.4
825 >>> request._orig_env['launchpad.publicationduration'] = 0.5
826 >>> request._orig_env['launchpad.publicationthreadduration'] = 0.4
782 >>> request.supportsRetry = lambda: False827 >>> request.supportsRetry = lambda: False
783 >>> request.supportsRetry()828 >>> request.supportsRetry()
784 False829 False
@@ -792,10 +837,14 @@ WSGI env.
792 ...837 ...
793 Retry: foo838 Retry: foo
794839
795 >>> request._orig_env['launchpad.publicationticks']840 >>> request._orig_env['launchpad.publicationduration']
796 500841 0.5
797 >>> request._orig_env['launchpad.traversalticks']842 >>> request._orig_env['launchpad.publicationthreadduration']
798 500843 0.4
844 >>> request._orig_env['launchpad.traversalduration']
845 0.5
846 >>> request._orig_env['launchpad.traversalthreadduration']
847 0.4
799848
800(A bit of cleanup so the test can continue)849(A bit of cleanup so the test can continue)
801850
@@ -818,11 +867,12 @@ The user attribute is an empty string, when no user is logged in.
818 >>> print request.principal867 >>> print request.principal
819 None868 None
820869
821 # Our afterCall() implementation expects to find a870 # Our afterCall() implementation expects to find _publication_start and
822 # _publicationticks_start in its request, which is set by871 # _publication_thread_start in its request, which are set by
823 # callObject(). Since we don't want to callObject() here, we'll872 # callObject(). Since we don't want to callObject() here, we'll
824 # have to change the request manually.873 # have to change the request manually.
825 >>> request._publicationticks_start = 1345874 >>> request._publication_start = 1.345
875 >>> request._publication_thread_start = None
826 >>> publication.afterCall(request, None)876 >>> publication.afterCall(request, None)
827 >>> txn.user877 >>> txn.user
828 ''878 ''
@@ -883,11 +933,12 @@ be automatically reverted in a GET request.
883933
884 >>> request, publication = get_request_and_publication(method='GET')934 >>> request, publication = get_request_and_publication(method='GET')
885935
886 # Our afterCall() implementation expects to find a936 # Our afterCall() implementation expects to find _publication_start and
887 # _publicationticks_start in its request, which is set by937 # _publication_thread_start in its request, which are set by
888 # callObject(). Since we don't want to callObject() here, we'll938 # callObject(). Since we don't want to callObject() here, we'll
889 # have to change the request manually.939 # have to change the request manually.
890 >>> request._publicationticks_start = 1345940 >>> request._publication_start = 1.345
941 >>> request._publication_thread_start = None
891 >>> publication.afterCall(request, None)942 >>> publication.afterCall(request, None)
892 >>> txn = transaction.begin()943 >>> txn = transaction.begin()
893 >>> foo_bar = get_foo_bar_person()944 >>> foo_bar = get_foo_bar_person()
@@ -901,11 +952,12 @@ But not if the request uses POST, the changes will be preserved.
901952
902 >>> request, publication = get_request_and_publication(method='POST')953 >>> request, publication = get_request_and_publication(method='POST')
903954
904 # Our afterCall() implementation expects to find a955 # Our afterCall() implementation expects to find _publication_start and
905 # _publicationticks_start in its request, which is set by956 # _publication_thread_start in its request, which are set by
906 # callObject(). Since we don't want to callObject() here, we'll957 # callObject(). Since we don't want to callObject() here, we'll
907 # have to change the request manually.958 # have to change the request manually.
908 >>> request._publicationticks_start = 1345959 >>> request._publication_start = 1.345
960 >>> request._publication_thread_start = None
909 >>> publication.afterCall(request, None)961 >>> publication.afterCall(request, None)
910 >>> txn = transaction.begin()962 >>> txn = transaction.begin()
911 >>> print get_foo_bar_person().description963 >>> print get_foo_bar_person().description
@@ -934,11 +986,12 @@ Doomed transactions are aborted.
934 >>> txn.isDoomed()986 >>> txn.isDoomed()
935 True987 True
936988
937 # Our afterCall() implementation expects to find a989 # Our afterCall() implementation expects to find _publication_start and
938 # _publicationticks_start in its request, which is set by990 # _publication_thread_start in its request, which are set by
939 # callObject(). Since we don't want to callObject() here, we'll991 # callObject(). Since we don't want to callObject() here, we'll
940 # have to change the request manually.992 # have to change the request manually.
941 >>> request._publicationticks_start = 1345993 >>> request._publication_start = 1.345
994 >>> request._publication_thread_start = None
942995
943 >>> publication.afterCall(request, None)996 >>> publication.afterCall(request, None)
944 Aborted997 Aborted
@@ -976,11 +1029,12 @@ afterCall() publication hook.)
976 >>> response = request.response1029 >>> response = request.response
977 >>> response.setResult('Content that will disappear.')1030 >>> response.setResult('Content that will disappear.')
9781031
979 # Our afterCall() implementation expects to find a1032 # Our afterCall() implementation expects to find _publication_start and
980 # _publicationticks_start in its request, which is set by1033 # _publication_thread_start in its request, which are set by
981 # callObject(). Since we don't want to callObject() here, we'll1034 # callObject(). Since we don't want to callObject() here, we'll
982 # have to change the request manually.1035 # have to change the request manually.
983 >>> request._publicationticks_start = 13451036 >>> request._publication_start = 1.345
1037 >>> request._publication_thread_start = None
984 >>> publication.afterCall(request, None)1038 >>> publication.afterCall(request, None)
985 >>> request.response.consumeBody()1039 >>> request.response.consumeBody()
986 ''1040 ''
@@ -992,11 +1046,12 @@ In other cases, like a GET, the body would be unchanged.
992 >>> response = request.response1046 >>> response = request.response
993 >>> response.setResult('Some boring content.')1047 >>> response.setResult('Some boring content.')
9941048
995 # Our afterCall() implementation expects to find a1049 # Our afterCall() implementation expects to find _publication_start and
996 # _publicationticks_start in its request, which is set by1050 # _publication_thread_start in its request, which are set by
997 # callObject(). Since we don't want to callObject() here, we'll1051 # callObject(). Since we don't want to callObject() here, we'll
998 # have to change the request manually.1052 # have to change the request manually.
999 >>> request._publicationticks_start = 13451053 >>> request._publication_start = 1.345
1054 >>> request._publication_thread_start = None
1000 >>> publication.afterCall(request, None)1055 >>> publication.afterCall(request, None)
1001 >>> print request.response.consumeBody()1056 >>> print request.response.consumeBody()
1002 Some boring content.1057 Some boring content.
@@ -1060,7 +1115,6 @@ associated with the access token specified in the request. The
1060principal's access_level and scope will match what was specified in the1115principal's access_level and scope will match what was specified in the
1061token.1116token.
10621117
1063 >>> import time
1064 >>> from lp.registry.interfaces.product import IProductSet1118 >>> from lp.registry.interfaces.product import IProductSet
1065 >>> from lp.services.database.policy import MasterDatabasePolicy1119 >>> from lp.services.database.policy import MasterDatabasePolicy
1066 >>> from lp.services.database.interfaces import IStoreSelector1120 >>> from lp.services.database.interfaces import IStoreSelector
diff --git a/lib/lp/services/webapp/publication.py b/lib/lp/services/webapp/publication.py
index 6ba60b5..1b01f53 100644
--- a/lib/lp/services/webapp/publication.py
+++ b/lib/lp/services/webapp/publication.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4__metaclass__ = type4__metaclass__ = type
@@ -11,6 +11,7 @@ import re
11import sys11import sys
12import thread12import thread
13import threading13import threading
14import time
14import traceback15import traceback
15import urllib16import urllib
1617
@@ -26,7 +27,6 @@ from storm.exceptions import (
26 IntegrityError,27 IntegrityError,
27 )28 )
28from storm.zope.interfaces import IZStorm29from storm.zope.interfaces import IZStorm
29import tickcount
30import transaction30import transaction
31from zc.zservertracelog.interfaces import ITraceLog31from zc.zservertracelog.interfaces import ITraceLog
32import zope.app.publication.browser32import zope.app.publication.browser
@@ -192,6 +192,17 @@ class LoginRoot:
192 return self192 return self
193193
194194
195def _get_thread_time():
196 """Get the CPU time spent in the current thread.
197
198 This requires Python >= 3.3, and otherwise returns None.
199 """
200 if hasattr(time, 'CLOCK_THREAD_CPUTIME_ID'):
201 return time.clock_gettime(time.CLOCK_THREAD_CPUTIME_ID)
202 else:
203 return None
204
205
195class LaunchpadBrowserPublication(206class LaunchpadBrowserPublication(
196 zope.app.publication.browser.BrowserPublication):207 zope.app.publication.browser.BrowserPublication):
197 """Subclass of z.a.publication.BrowserPublication that removes ZODB.208 """Subclass of z.a.publication.BrowserPublication that removes ZODB.
@@ -240,7 +251,8 @@ class LaunchpadBrowserPublication(
240251
241 def beforeTraversal(self, request):252 def beforeTraversal(self, request):
242 notify(StartRequestEvent(request))253 notify(StartRequestEvent(request))
243 request._traversalticks_start = tickcount.tickcount()254 request._traversal_start = time.time()
255 request._traversal_thread_start = _get_thread_time()
244 threadid = thread.get_ident()256 threadid = thread.get_ident()
245 threadrequestfile = open_for_writing(257 threadrequestfile = open_for_writing(
246 'logs/thread-%s.request' % threadid, 'w')258 'logs/thread-%s.request' % threadid, 'w')
@@ -416,7 +428,8 @@ class LaunchpadBrowserPublication(
416 It also sets the launchpad.userid and launchpad.pageid WSGI428 It also sets the launchpad.userid and launchpad.pageid WSGI
417 environment variables.429 environment variables.
418 """430 """
419 request._publicationticks_start = tickcount.tickcount()431 request._publication_start = time.time()
432 request._publication_thread_start = _get_thread_time()
420 if request.response.getStatus() in [301, 302, 303, 307]:433 if request.response.getStatus() in [301, 302, 303, 307]:
421 return ''434 return ''
422435
@@ -470,12 +483,17 @@ class LaunchpadBrowserPublication(
470 Because of this we cannot chain to the superclass and implement483 Because of this we cannot chain to the superclass and implement
471 the whole behaviour here.484 the whole behaviour here.
472 """485 """
473 assert hasattr(request, '_publicationticks_start'), (486 assert hasattr(request, '_publication_start'), (
474 'request._publicationticks_start, which should have been set by '487 'request._publication_start, which should have been set by '
475 'callObject(), was not found.')488 'callObject(), was not found.')
476 ticks = tickcount.difference(489 publication_duration = time.time() - request._publication_start
477 request._publicationticks_start, tickcount.tickcount())490 if request._publication_thread_start is not None:
478 request.setInWSGIEnvironment('launchpad.publicationticks', ticks)491 publication_thread_duration = (
492 _get_thread_time() - request._publication_thread_start)
493 else:
494 publication_thread_duration = None
495 request.setInWSGIEnvironment(
496 'launchpad.publicationduration', publication_duration)
479497
480 # Calculate SQL statement statistics.498 # Calculate SQL statement statistics.
481 sql_statements = da.get_request_statements()499 sql_statements = da.get_request_statements()
@@ -483,10 +501,16 @@ class LaunchpadBrowserPublication(
483 endtime - starttime501 endtime - starttime
484 for starttime, endtime, id, statement, tb in sql_statements)502 for starttime, endtime, id, statement, tb in sql_statements)
485503
486 # Log publication tickcount, sql statement count, and sql time504 # Log publication duration (in milliseconds), sql statement count,
487 # to the tracelog.505 # and sql time (in milliseconds) to the tracelog. If we have the
488 tracelog(request, 't', '%d %d %d' % (506 # publication time spent in this thread, then log that too (in
489 ticks, len(sql_statements), sql_milliseconds))507 # milliseconds).
508 tracelog_entry = '%d %d %d' % (
509 publication_duration * 1000,
510 len(sql_statements), sql_milliseconds)
511 if publication_thread_duration is not None:
512 tracelog_entry += ' %d' % (publication_thread_duration * 1000)
513 tracelog(request, 't', tracelog_entry)
490514
491 # Annotate the transaction with user data. That was done by515 # Annotate the transaction with user data. That was done by
492 # zope.app.publication.zopepublication.ZopePublication.516 # zope.app.publication.zopepublication.ZopePublication.
@@ -547,12 +571,17 @@ class LaunchpadBrowserPublication(
547 # Log the URL including vhost information to the ZServer tracelog.571 # Log the URL including vhost information to the ZServer tracelog.
548 tracelog(request, 'u', request.getURL())572 tracelog(request, 'u', request.getURL())
549573
550 assert hasattr(request, '_traversalticks_start'), (574 assert hasattr(request, '_traversal_start'), (
551 'request._traversalticks_start, which should have been set by '575 'request._traversal_start, which should have been set by '
552 'beforeTraversal(), was not found.')576 'beforeTraversal(), was not found.')
553 ticks = tickcount.difference(577 traversal_duration = time.time() - request._traversal_start
554 request._traversalticks_start, tickcount.tickcount())578 request.setInWSGIEnvironment(
555 request.setInWSGIEnvironment('launchpad.traversalticks', ticks)579 'launchpad.traversalduration', traversal_duration)
580 if request._traversal_thread_start is not None:
581 traversal_thread_duration = (
582 _get_thread_time() - request._traversal_thread_start)
583 request.setInWSGIEnvironment(
584 'launchpad.traversalthreadduration', traversal_thread_duration)
556585
557 def _maybePlacefullyAuthenticate(self, request, ob):586 def _maybePlacefullyAuthenticate(self, request, ob):
558 """ This should never be called because we've excised it in587 """ This should never be called because we've excised it in
@@ -569,22 +598,35 @@ class LaunchpadBrowserPublication(
569 db_policy = None598 db_policy = None
570599
571 orig_env = request._orig_env600 orig_env = request._orig_env
572 ticks = tickcount.tickcount()601 now = time.time()
573 if (hasattr(request, '_publicationticks_start') and602 thread_now = _get_thread_time()
574 ('launchpad.publicationticks' not in orig_env)):603 if (hasattr(request, '_publication_start') and
604 ('launchpad.publicationduration' not in orig_env)):
575 # The traversal process has been started but hasn't completed.605 # The traversal process has been started but hasn't completed.
576 assert 'launchpad.traversalticks' in orig_env, (606 assert 'launchpad.traversalduration' in orig_env, (
577 'We reached the publication process so we must have finished '607 'We reached the publication process so we must have finished '
578 'the traversal.')608 'the traversal.')
579 ticks = tickcount.difference(609 publication_duration = now - request._publication_start
580 request._publicationticks_start, ticks)610 request.setInWSGIEnvironment(
581 request.setInWSGIEnvironment('launchpad.publicationticks', ticks)611 'launchpad.publicationduration', publication_duration)
582 elif (hasattr(request, '_traversalticks_start') and612 if thread_now is not None:
583 ('launchpad.traversalticks' not in orig_env)):613 publication_thread_duration = (
614 thread_now - request._publication_thread_start)
615 request.setInWSGIEnvironment(
616 'launchpad.publicationthreadduration',
617 publication_thread_duration)
618 elif (hasattr(request, '_traversal_start') and
619 ('launchpad.traversalduration' not in orig_env)):
584 # The traversal process has been started but hasn't completed.620 # The traversal process has been started but hasn't completed.
585 ticks = tickcount.difference(621 traversal_duration = now - request._traversal_start
586 request._traversalticks_start, ticks)622 request.setInWSGIEnvironment(
587 request.setInWSGIEnvironment('launchpad.traversalticks', ticks)623 'launchpad.traversalduration', traversal_duration)
624 if thread_now is not None:
625 traversal_thread_duration = (
626 thread_now - request._traversal_thread_start)
627 request.setInWSGIEnvironment(
628 'launchpad.traversalthreadduration',
629 traversal_thread_duration)
588 else:630 else:
589 # The exception wasn't raised in the middle of the traversal nor631 # The exception wasn't raised in the middle of the traversal nor
590 # the publication, so there's nothing we need to do here.632 # the publication, so there's nothing we need to do here.
@@ -639,10 +681,12 @@ class LaunchpadBrowserPublication(
639 # is a normal part of operation.681 # is a normal part of operation.
640 if should_retry(exc_info):682 if should_retry(exc_info):
641 if request.supportsRetry():683 if request.supportsRetry():
642 # Remove variables used for counting ticks as this request is684 # Remove variables used for counting publication/traversal
643 # going to be retried.685 # durations as this request is going to be retried.
644 orig_env.pop('launchpad.traversalticks', None)686 orig_env.pop('launchpad.traversalduration', None)
645 orig_env.pop('launchpad.publicationticks', None)687 orig_env.pop('launchpad.traversalthreadduration', None)
688 orig_env.pop('launchpad.publicationduration', None)
689 orig_env.pop('launchpad.publicationthreadduration', None)
646 # Our endRequest needs to know if a retry is pending or not.690 # Our endRequest needs to know if a retry is pending or not.
647 request._wants_retry = True691 request._wants_retry = True
648 # Abort any in-progress transaction and reset any692 # Abort any in-progress transaction and reset any
diff --git a/lib/lp/services/webapp/servers.py b/lib/lp/services/webapp/servers.py
index 6f3e7b8..506f705 100644
--- a/lib/lp/services/webapp/servers.py
+++ b/lib/lp/services/webapp/servers.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2018 Canonical Ltd. This software is licensed under the1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Definition of the internet servers that Launchpad uses."""4"""Definition of the internet servers that Launchpad uses."""
@@ -1067,8 +1067,8 @@ class LaunchpadAccessLogger(CommonAccessLogger):
1067 response bytes written1067 response bytes written
1068 number of nonpython statements (sql, email, memcache, rabbit etc)1068 number of nonpython statements (sql, email, memcache, rabbit etc)
1069 request duration1069 request duration
1070 number of ticks during traversal1070 traversal duration
1071 number of ticks during publication1071 publication duration
1072 launchpad user id1072 launchpad user id
1073 launchpad page id1073 launchpad page id
1074 REFERER1074 REFERER
@@ -1088,8 +1088,8 @@ class LaunchpadAccessLogger(CommonAccessLogger):
1088 pageid = cgi_env.get('launchpad.pageid', '')1088 pageid = cgi_env.get('launchpad.pageid', '')
1089 nonpython_actions = cgi_env.get('launchpad.nonpythonactions', 0)1089 nonpython_actions = cgi_env.get('launchpad.nonpythonactions', 0)
1090 request_duration = cgi_env.get('launchpad.requestduration', 0)1090 request_duration = cgi_env.get('launchpad.requestduration', 0)
1091 traversal_ticks = cgi_env.get('launchpad.traversalticks', 0)1091 traversal_duration = cgi_env.get('launchpad.traversalduration', 0)
1092 publication_ticks = cgi_env.get('launchpad.publicationticks', 0)1092 publication_duration = cgi_env.get('launchpad.publicationduration', 0)
1093 referer = request_headers.get('REFERER', '')1093 referer = request_headers.get('REFERER', '')
1094 user_agent = request_headers.get('USER_AGENT', '')1094 user_agent = request_headers.get('USER_AGENT', '')
10951095
@@ -1106,8 +1106,8 @@ class LaunchpadAccessLogger(CommonAccessLogger):
1106 bytes_written,1106 bytes_written,
1107 nonpython_actions,1107 nonpython_actions,
1108 request_duration,1108 request_duration,
1109 traversal_ticks,1109 traversal_duration,
1110 publication_ticks,1110 publication_duration,
1111 userid,1111 userid,
1112 pageid,1112 pageid,
1113 referer,1113 referer,

Subscribers

People subscribed via source and target branches

to status/vote changes: