Merge lp:~cjwatson/launchpad/build-twisted-plugin-cache into lp:launchpad

Proposed by Colin Watson on 2017-09-23
Status: Merged
Merged at revision: 18466
Proposed branch: lp:~cjwatson/launchpad/build-twisted-plugin-cache
Merge into: lp:launchpad
Diff against target: 149 lines (+34/-26)
7 files modified
Makefile (+3/-1)
buildout.cfg (+0/-20)
lib/lp/scripts/runlaunchpad.py (+1/-2)
lib/lp/services/twistedsupport/plugincache.py (+24/-0)
lib/lp/services/txlongpoll/tests/test_server.py (+1/-2)
setup.py (+4/-0)
utilities/start-dev-soyuz.sh (+1/-1)
To merge this branch: bzr merge lp:~cjwatson/launchpad/build-twisted-plugin-cache
Reviewer Review Type Date Requested Status
William Grant code 2017-09-23 Approve on 2017-09-25
Review via email: mp+331240@code.launchpad.net

Commit Message

Build dropin.cache for all installed Twisted plugins in "make compile", and merge txlongpoll and txpkgupload back into the scripts recipe.

Description of the Change

If an ampoule subprocess finds itself needing to build dropin.cache for a Twisted plugin, and that plugin imports twisted.internet.reactor at some point and thus installs a default reactor, then twisted.application.reactors.installReactor (called by the ampoule bootstrap code) will crash with ReactorAlreadyInstalledError in twisted.internet.main.installReactor. This will then proceed to ruin everyone's day with incomprehensible buildbot failures.

https://twistedmatrix.com/documents/current/core/howto/plugin.html#plugin-caching recommends that we build the cache files when installing packages, and it certainly seems like a good idea not to rely on writing files into eggs at run-time.

With this, I believe we can go back to the merged recipes, and thus remove a blocker to conversion to pip.

To post a comment you must log in.
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2017-09-04 12:35:40 +0000
3+++ Makefile 2017-09-23 03:20:32 +0000
4@@ -47,7 +47,8 @@
5 # NB: It's important BUILDOUT_BIN only mentions things genuinely produced by
6 # buildout.
7 BUILDOUT_BIN = \
8- $(PY) bin/apiindex bin/bzr bin/combine-css bin/fl-build-report \
9+ $(PY) bin/apiindex bin/build-twisted-plugin-cache bin/bzr \
10+ bin/combine-css bin/fl-build-report \
11 bin/fl-credential-ctl bin/fl-install-demo bin/fl-monitor-ctl \
12 bin/fl-record bin/fl-run-bench bin/fl-run-test bin/googletestservice \
13 bin/harness bin/iharness bin/ipy bin/jsbuild bin/lpjsmin\
14@@ -236,6 +237,7 @@
15 compile: $(PY) $(VERSION_INFO)
16 ${SHHH} $(MAKE) -C sourcecode build PYTHON=${PYTHON} \
17 LPCONFIG=${LPCONFIG}
18+ ${SHHH} bin/build-twisted-plugin-cache
19 ${SHHH} LPCONFIG=${LPCONFIG} ${PY} -t buildmailman.py
20
21 test_build: build
22
23=== modified file 'buildout.cfg'
24--- buildout.cfg 2017-09-18 10:08:40 +0000
25+++ buildout.cfg 2017-09-23 03:20:32 +0000
26@@ -5,8 +5,6 @@
27 parts =
28 scripts
29 tags
30- txlongpoll
31- txpkgupload
32 unzip = true
33 eggs-directory = eggs
34 download-cache = download-cache
35@@ -53,21 +51,3 @@
36 [tags]
37 recipe = z3c.recipe.tag:tags
38 eggs = lp
39-
40-[txlongpoll]
41-recipe = z3c.recipe.scripts
42-eggs = ${scripts:eggs}
43- txlongpoll
44-include-site-packages = false
45-initialization = ${scripts:initialization}
46-entry-points = twistd-for-txlongpoll=twisted.scripts.twistd:run
47-scripts = twistd-for-txlongpoll
48-
49-[txpkgupload]
50-recipe = z3c.recipe.scripts
51-eggs = ${scripts:eggs}
52- txpkgupload
53-include-site-packages = false
54-initialization = ${scripts:initialization}
55-entry-points = twistd-for-txpkgupload=twisted.scripts.twistd:run
56-scripts = twistd-for-txpkgupload
57
58=== modified file 'lib/lp/scripts/runlaunchpad.py'
59--- lib/lp/scripts/runlaunchpad.py 2017-09-18 09:17:21 +0000
60+++ lib/lp/scripts/runlaunchpad.py 2017-09-23 03:20:32 +0000
61@@ -246,8 +246,7 @@
62 return config.txlongpoll.launch
63
64 def launch(self):
65- twistd_bin = os.path.join(
66- config.root, 'bin', 'twistd-for-txlongpoll')
67+ twistd_bin = os.path.join(config.root, 'bin', 'twistd')
68 broker_hostname, broker_port = as_host_port(
69 config.rabbitmq.host, None, None)
70 self.server = TxLongPollServer(
71
72=== added file 'lib/lp/services/twistedsupport/plugincache.py'
73--- lib/lp/services/twistedsupport/plugincache.py 1970-01-01 00:00:00 +0000
74+++ lib/lp/services/twistedsupport/plugincache.py 2017-09-23 03:20:32 +0000
75@@ -0,0 +1,24 @@
76+# Copyright 2017 Canonical Ltd. This software is licensed under the
77+# GNU Affero General Public License version 3 (see the file LICENSE).
78+
79+"""Build dropin.cache for all installed Twisted plugins.
80+
81+This would be built on the fly if we didn't do it here, but we want to make
82+sure to build it in a predictable environment. In particular, if a plugin's
83+cache is first built as a result of being run via ampoule, then ampoule will
84+fail if any part of the process of importing the plugin installs a default
85+reactor.
86+"""
87+
88+from __future__ import absolute_import, print_function, unicode_literals
89+
90+__metaclass__ = type
91+
92+from twisted.plugin import (
93+ getPlugins,
94+ IPlugin,
95+ )
96+
97+
98+def main():
99+ list(getPlugins(IPlugin))
100
101=== modified file 'lib/lp/services/txlongpoll/tests/test_server.py'
102--- lib/lp/services/txlongpoll/tests/test_server.py 2017-09-18 09:17:21 +0000
103+++ lib/lp/services/txlongpoll/tests/test_server.py 2017-09-23 03:20:32 +0000
104@@ -22,8 +22,7 @@
105 def test_service_config(self):
106 # TxLongPollServer pokes some .ini configuration into its
107 # service_config attributes.
108- twistd_bin = os.path.join(
109- config.root, 'bin', 'twistd-for-txlongpoll')
110+ twistd_bin = os.path.join(config.root, 'bin', 'twistd')
111 fixture = self.useFixture(TxLongPollServer(
112 broker_user='guest', broker_password='guest', broker_vhost='/',
113 broker_port=123, frontend_port=None,
114
115=== modified file 'setup.py'
116--- setup.py 2017-09-18 10:08:40 +0000
117+++ setup.py 2017-09-23 03:20:32 +0000
118@@ -104,7 +104,9 @@
119 'transaction',
120 'Twisted',
121 'txfixtures',
122+ 'txlongpoll',
123 'txlongpollfixture',
124+ 'txpkgupload',
125 'wadllib',
126 'z3c.pt',
127 'z3c.ptcompat',
128@@ -167,6 +169,8 @@
129 entry_points=dict(
130 console_scripts=[ # `console_scripts` is a magic name to setuptools
131 'apiindex = lp.scripts.utilities.apiindex:main',
132+ 'build-twisted-plugin-cache = '
133+ 'lp.services.twistedsupport.plugincache:main',
134 'bzr = lp.scripts.utilities.bzr:main',
135 'combine-css = lp.scripts.utilities.js.combinecss:main',
136 'googletestservice = '
137
138=== modified file 'utilities/start-dev-soyuz.sh'
139--- utilities/start-dev-soyuz.sh 2017-09-18 10:08:40 +0000
140+++ utilities/start-dev-soyuz.sh 2017-09-23 03:20:32 +0000
141@@ -20,7 +20,7 @@
142 plugin=$2
143 shift 2
144 echo "Starting $name."
145- "bin/twistd-for-$name" \
146+ bin/twistd \
147 --logfile "/var/tmp/development-$name.log" \
148 --pidfile "/var/tmp/development-$name.pid" \
149 "$plugin" "$@"