Merge lp:~allenap/maas/more-better-things--bug-1389007 into lp:maas/trunk

Proposed by Gavin Panella
Status: Rejected
Rejected by: MAAS Lander
Proposed branch: lp:~allenap/maas/more-better-things--bug-1389007
Merge into: lp:maas/trunk
Diff against target: 7895 lines (+6361/-339)
29 files modified
LICENSE.Twisted (+65/-0)
docs/development/rpc.rst (+12/-8)
scripts/ampclient.py (+1/-1)
src/maasserver/clusterrpc/power.py (+1/-1)
src/maasserver/clusterrpc/utils.py (+1/-1)
src/maasserver/models/node.py (+2/-2)
src/maasserver/models/tests/test_node.py (+6/-4)
src/maasserver/rpc/regionservice.py (+3/-3)
src/maasserver/rpc/testing/fixtures.py (+2/-2)
src/maasserver/rpc/tests/test_regionservice.py (+5/-5)
src/provisioningserver/pserv_services/dhcp_probe_service.py (+1/-1)
src/provisioningserver/rpc/amp32.py (+2646/-0)
src/provisioningserver/rpc/arguments.py (+11/-11)
src/provisioningserver/rpc/cluster.py (+119/-117)
src/provisioningserver/rpc/clusterservice.py (+4/-4)
src/provisioningserver/rpc/common.py (+17/-17)
src/provisioningserver/rpc/monitors.py (+5/-3)
src/provisioningserver/rpc/region.py (+127/-127)
src/provisioningserver/rpc/testing/__init__.py (+10/-8)
src/provisioningserver/rpc/testing/tls.py (+1/-1)
src/provisioningserver/rpc/tests/test_amp32.py (+3270/-0)
src/provisioningserver/rpc/tests/test_arguments.py (+6/-4)
src/provisioningserver/rpc/tests/test_clusterservice.py (+7/-7)
src/provisioningserver/rpc/tests/test_common.py (+5/-3)
src/provisioningserver/rpc/tests/test_docs.py (+3/-3)
src/provisioningserver/rpc/tests/test_monitors.py (+6/-4)
src/provisioningserver/utils/__init__.py (+1/-1)
src/provisioningserver/utils/shell.py (+11/-0)
src/provisioningserver/utils/tests/test_shell.py (+13/-1)
To merge this branch: bzr merge lp:~allenap/maas/more-better-things--bug-1389007
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) Approve
Christian Reis (community) Needs Information
Review via email: mp+241467@code.launchpad.net

Commit message

Adapt AMP to use 32-bit length prefixes for on-wire serialization.

This allows us to have much larger arguments and responses over RPC, though it's limited to 2MiB for now, up from 64kiB.

Description of the change

This is a *big* branch, I know. There's no good way of doing this in stages. I copied the AMP code and tests from Twisted, adapted them for 32-bit-length-prefixed strings, then fixed up the code to pass muster in MAAS's coding style, investigated a test isolation failure, and then also adapted the tests to run using MAASTestCase instead of Twisted's Trial code.

I think the best way to verify/test/review this code is in CI and use. I've not changed the behaviour of the AMP code except the switch from 16 to 32 bit, so that can be reviewed solely for style, rather than substance. I did have to write AMPTestCase for the tests, but again most of the behaviour is intact.

The bulk of the rest is mechanical, which can be skipped through fairly quickly.

Please review this?

To post a comment you must log in.
Revision history for this message
Christian Reis (kiko) wrote :

Hmm, do we really want to copy the code wholesale instead of changing AMT upstream to be fixed, parametrizable or monkey-patchable?

review: Needs Information
Revision history for this message
Christian Reis (kiko) wrote :

AMT on my mind, I obviously menat AMP.

Revision history for this message
Gavin Panella (allenap) wrote :

There's no way we can change upstream AMP to 32-bit prefixes; it's an incompatible change. We could propose amp32 as an additional protocol, or we could write negotiation code that starts on 16-bit and switches to 32-bit if supported.

The former would be fine: it's something we can try though it may be rejected, but it doesn't block this branch.

The latter is, at a guess, several days work instead of ~6 hours that went into this. I'm okay to do it, but I guess we've got more important things to do.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Not a full review yet, but some notes:

.

Typo: "This contain the".

.

If you're going to include Twisted code here, it may not be valid to say "See LICENSE for details." You may need to include their licence file under a recognisable name, and make the copyright header refer to it as such. Also, if you're creating a derived work with substantial changes, I guess it needs our copyright notice as well.

.

AMPTestCase (not AMP32TestCase?) and everything in it needs docstrings.

Although to be honest, I'd prefer not to have yet another test-case class in the first place! I see very little in there that actually needs to be in a class: shortDescription might be generic or it could be in a fixture. setUp *is* a fixture. getLoggedFailures could be in MAASTestCase. And assertWarns is a generic matcher disguised as a specific helper which makes non-transparent decisions about whether the test should continue after failure.

.

In AMPTestCase.assertWarns, if you know you've got a list of one item and you want the item, prefer extracting it using pattern-matching ("[item] = mylist") over indexing ("mylist[0]").

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

More notes:

.

It'd be worth noting in the documentation (or did I miss it?) that this is a modified version of the Twisted implementation, and that it does not change the on-the-wire protocol

.

I'd stick the module-level verifyClass invocations in tests.

.

I wasn't sure about AMPTestCase, but the AMPTest class almost certainly should say AMP32 instead of plain AMP.

.

For the bulk of the code, I can only assume that it's as good as the code we've been relying on anyway. So I'm voting Approve for the code as such — though I'll leave it to others (including Jenkins) to figure out how well the landing would fit into the grand plan.

review: Approve
3372. By Gavin Panella

Fix typo.

3373. By Gavin Panella

Include Twisted's LICENSE file which is references by amp32 and its tests.

3374. By Gavin Panella

shortDescription() is no longer needed.

3375. By Gavin Panella

Rename AMPTestCase to AMP32TestCase.

3376. By Gavin Panella

Docstrings for AMP32TestCase.

3377. By Gavin Panella

Use the pattern match, Luke.

Revision history for this message
Gavin Panella (allenap) wrote :

> Typo: "This contain the".

Well spotted, thanks. Fixed.

>
> .
>
> If you're going to include Twisted code here, it may not be valid to
> say "See LICENSE for details." You may need to include their licence
> file under a recognisable name, and make the copyright header refer to
> it as such. Also, if you're creating a derived work with substantial
> changes, I guess it needs our copyright notice as well.

I've pulled Twisted's LICENSE file in now, as LICENSE.Twisted, and
updated the headers in those files. Canonical is already listed in
LICENSE.Twisted so I think we can probably leave it as is.

>
> .
>
> AMPTestCase (not AMP32TestCase?) and everything in it needs
> docstrings.
>
> Although to be honest, I'd prefer not to have yet another test-case
> class in the first place! I see very little in there that actually
> needs to be in a class: shortDescription might be generic or it could
> be in a fixture. setUp *is* a fixture. getLoggedFailures could be in
> MAASTestCase. And assertWarns is a generic matcher disguised as a
> specific helper which makes non- transparent decisions about whether
> the test should continue after failure.

shortDescription() is a unittest method that gets called before the test
runs, but now that I've rebased on MAASTestCase instead of Trial's
TestCase it can go.

I've added docstrings which explain the other functions, and justify
their continued existence. I could spend time on getting rid of
AMP32TestCase, but the pay off doesn't seem worth it right now. I'm
biased, but I think it's already an improvement over the tests that work
with Trial.

>
> .
>
> In AMPTestCase.assertWarns, if you know you've got a list of one item
> and you want the item, prefer extracting it using pattern-matching
> ("[item] = mylist") over indexing ("mylist[0]").

Done.

Thanks!

3378. By Gavin Panella

Update amp32's docstring to draw attention to the differences from Twisted's amp.

3379. By Gavin Panella

Draw more attention to the difference between amp and amp32 in the docs.

3380. By Gavin Panella

Move module-level interface tests into a test case.

3381. By Gavin Panella

Rename AMPTest to AMP32Test.

Revision history for this message
Gavin Panella (allenap) wrote :

> It'd be worth noting in the documentation (or did I miss it?) that
> this is a modified version of the Twisted implementation, and that it
> does not change the on-the-wire protocol

Done.

>
> .
>
> I'd stick the module-level verifyClass invocations in tests.

Done.

>
> .
>
> I wasn't sure about AMPTestCase, but the AMPTest class almost
> certainly should say AMP32 instead of plain AMP.

Done.

>
> .
>
> For the bulk of the code, I can only assume that it's as good as the
> code we've been relying on anyway. So I'm voting Approve for the code
> as such — though I'll leave it to others (including Jenkins) to figure
> out how well the landing would fit into the grand plan.

Yeah, it's the same code, modified to our style conventions. The tests
have been modified to use testtools instead of Trial, but otherwise
they're the same.

Thanks again!

Revision history for this message
Christian Reis (kiko) wrote :

I think this needs a discussion upstream in order to establish the best path forward.

If upstream is willing to accept an upgrade-to-32bit patch, then we should invest energy into that because long-term it's much more sustainable for us. If they have another suggested solution we should use it.

We should only really take this fork in if there's really no other viable solution, which should be quite unlikely.

Revision history for this message
Gavin Panella (allenap) wrote :

> I think this needs a discussion upstream in order to establish the
> best path forward.
>
> If upstream is willing to accept an upgrade-to-32bit patch, then we
> should invest energy into that because long-term it's much more
> sustainable for us. If they have another suggested solution we should
> use it.
>
> We should only really take this fork in if there's really no other
> viable solution, which should be quite unlikely.

I think it's the right and proper thing to do to talk to the Twisted
project, and see if this can go upstream.

Twisted is a community project; we or I would likely be the de facto
maintainers.

Sustainably, my guess is that we're not in for a lot of work in either
case. AMP in Twisted is essentially done. This branch doesn't add new
features, just a change to an on-the-wire detail.

Even so, being upstream would give others the chance to use the code,
and give greater visibility to bugs and security issues.

Anecdotally, though from my own experience of the Twisted development
process, I can say that it's onerous to get a patch reviewed and landed.

If we get it upstream it could be a long time before that release makes
it back into Ubuntu. It's unlikely to get into the current LTS. We'd
need to carry this code anyway, for a while.

In all, I think we can make it a goal to get this upstream, but I don't
think we should block on it, except just to check there isn't a
ready-made alternative.

Revision history for this message
Christian Reis (kiko) wrote :

Yes, if we do agree that we want this patch and that it can go upstream,
then we should carry this in-tree until it is in a released version.

Please let us know what the immediate reaction upstream is and we'll
figure out what to do; I assume as you do that there's no obvious other
solution, but it would be nice to be surprised.
--
Christian Robottom Reis | [+1] 612 888 4935 | http://launchpad.net/~kiko
Canonical VP Hyperscale | [+55 16] 9 9112 6430 | http://async.com.br/~kiko

Revision history for this message
Gavin Panella (allenap) wrote :

The conversation with upstream developer exarkun [1] has put me off this branch. Not in a never-ever sense, but in the sense that I think this 64k limit is actually making us think. We can actually improve the power-poller (the trigger for this branch) by working within this limitation.

[1] http://twistedmatrix.com/pipermail/twisted-python/2014-November/028941.html

Revision history for this message
Julian Edwards (julian-edwards) wrote :

On Friday 14 Nov 2014 12:05:28 you wrote:
> The conversation with upstream developer exarkun [1] has put me off this
> branch. Not in a never-ever sense, but in the sense that I think this 64k
> limit is actually making us think. We can actually improve the power-poller
> (the trigger for this branch) by working within this limitation.
>
> [1]
> http://twistedmatrix.com/pipermail/twisted-python/2014-November/028941.html

I saw your mention of a "priority queue". I'm not sure we need that sort of
solution; we talked today about only querying the oldest 10 (say) power
statuses at a time. This will fit easily in the message size.

Revision history for this message
Gavin Panella (allenap) wrote :

> I saw your mention of a "priority queue". I'm not sure we need that
> sort of solution; we talked today about only querying the oldest 10
> (say) power statuses at a time. This will fit easily in the message
> size.

I was using big words to mean just that :)

Revision history for this message
MAAS Lander (maas-lander) wrote :

Transitioned to Git.

lp:maas has now moved from Bzr to Git.
Please propose your branches with Launchpad using Git.

git clone https://git.launchpad.net/maas

Unmerged revisions

3381. By Gavin Panella

Rename AMPTest to AMP32Test.

3380. By Gavin Panella

Move module-level interface tests into a test case.

3379. By Gavin Panella

Draw more attention to the difference between amp and amp32 in the docs.

3378. By Gavin Panella

Update amp32's docstring to draw attention to the differences from Twisted's amp.

3377. By Gavin Panella

Use the pattern match, Luke.

3376. By Gavin Panella

Docstrings for AMP32TestCase.

3375. By Gavin Panella

Rename AMPTestCase to AMP32TestCase.

3374. By Gavin Panella

shortDescription() is no longer needed.

3373. By Gavin Panella

Include Twisted's LICENSE file which is references by amp32 and its tests.

3372. By Gavin Panella

Fix typo.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'LICENSE.Twisted'
2--- LICENSE.Twisted 1970-01-01 00:00:00 +0000
3+++ LICENSE.Twisted 2014-11-12 12:20:32 +0000
4@@ -0,0 +1,65 @@
5+Copyright (c) 2001-2014
6+Allen Short
7+Andy Gayton
8+Andrew Bennetts
9+Antoine Pitrou
10+Apple Computer, Inc.
11+Ashwini Oruganti
12+Benjamin Bruheim
13+Bob Ippolito
14+Canonical Limited
15+Christopher Armstrong
16+David Reid
17+Donovan Preston
18+Eric Mangold
19+Eyal Lotem
20+Google Inc.
21+Hybrid Logic Ltd.
22+Hynek Schlawack
23+Itamar Turner-Trauring
24+James Knight
25+Jason A. Mobarak
26+Jean-Paul Calderone
27+Jessica McKellar
28+Jonathan Jacobs
29+Jonathan Lange
30+Jonathan D. Simms
31+Jürgen Hermann
32+Julian Berman
33+Kevin Horn
34+Kevin Turner
35+Laurens Van Houtven
36+Mary Gardiner
37+Matthew Lefkowitz
38+Massachusetts Institute of Technology
39+Moshe Zadka
40+Paul Swartz
41+Pavel Pergamenshchik
42+Ralph Meijer
43+Richard Wall
44+Sean Riley
45+Software Freedom Conservancy
46+Travis B. Hartwell
47+Thijs Triemstra
48+Thomas Herve
49+Timothy Allen
50+Tom Prince
51+
52+Permission is hereby granted, free of charge, to any person obtaining
53+a copy of this software and associated documentation files (the
54+"Software"), to deal in the Software without restriction, including
55+without limitation the rights to use, copy, modify, merge, publish,
56+distribute, sublicense, and/or sell copies of the Software, and to
57+permit persons to whom the Software is furnished to do so, subject to
58+the following conditions:
59+
60+The above copyright notice and this permission notice shall be
61+included in all copies or substantial portions of the Software.
62+
63+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
64+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
65+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
66+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
67+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
68+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
69+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
70
71=== modified file 'docs/development/rpc.rst'
72--- docs/development/rpc.rst 2014-08-21 20:10:47 +0000
73+++ docs/development/rpc.rst 2014-11-12 12:20:32 +0000
74@@ -5,7 +5,11 @@
75
76 MAAS contains an RPC mechanism such that every process in the region is
77 connected to every process in the cluster (strictly, every pserv
78-process). It's based on AMP_, specifically `Twisted's implementation`_.
79+process). It's based on AMP_, specifically `Twisted's implementation`_,
80+but MAAS incorporates an extended version that can transmit much larger
81+messages by using 32-bit length prefixes in place of 16-bit. This makes
82+it on-the-wire incompatible with standard AMP_ implementations, but its
83+essential behaviour is identical.
84
85 .. _AMP:
86 http://amp-protocol.net/
87@@ -17,22 +21,22 @@
88 Where do I start?
89 -----------------
90
91-Start in the :py:mod:`provisioningserver.rpc` package. The first two files to
92-look at are ``cluster.py`` and ``region.py``. This contain the
93+Start in the :py:mod:`provisioningserver.rpc` package. The first two
94+files to look at are ``cluster.py`` and ``region.py``. This contains the
95 declarations of what commands are available on clusters and regions
96 respectively.
97
98 A new command could be declared like so::
99
100- from twisted.protocols import amp
101+ from provisioningserver.rpc import amp32
102
103- class EatCheez(amp.Command):
104+ class EatCheez(amp32.Command):
105 arguments = [
106- (b"name", amp.Unicode()),
107- (b"origin", amp.Unicode()),
108+ (b"name", amp32.Unicode()),
109+ (b"origin", amp32.Unicode()),
110 ]
111 response = [
112- (b"rating", amp.Integer()),
113+ (b"rating", amp32.Integer()),
114 ]
115
116 It's also possible to map exceptions across the wire using an ``errors``
117
118=== modified file 'scripts/ampclient.py'
119--- scripts/ampclient.py 2014-01-31 16:15:00 +0000
120+++ scripts/ampclient.py 2014-11-12 12:20:32 +0000
121@@ -16,6 +16,7 @@
122
123 import sys
124
125+from provisioningserver.rpc.amp32 import AMP
126 from provisioningserver.rpc.cluster import ListBootImages
127 from provisioningserver.rpc.region import ReportBootImages
128 from twisted.internet import reactor
129@@ -23,7 +24,6 @@
130 connectProtocol,
131 TCP4ClientEndpoint,
132 )
133-from twisted.protocols.amp import AMP
134
135
136 def callRemote(command, port, **kwargs):
137
138=== modified file 'src/maasserver/clusterrpc/power.py'
139--- src/maasserver/clusterrpc/power.py 2014-09-16 21:00:33 +0000
140+++ src/maasserver/clusterrpc/power.py 2014-11-12 12:20:32 +0000
141@@ -38,7 +38,7 @@
142 Nodes can be in any cluster; the power calls will be directed to their
143 owning cluster.
144
145- :param command: The `amp.Command` to call.
146+ :param command: The `amp32.Command` to call.
147 :param nodes: A sequence of ``(system-id, hostname, cluster-uuid,
148 power-info)`` tuples.
149 :returns: A mapping of each node's system ID to a
150
151=== modified file 'src/maasserver/clusterrpc/utils.py'
152--- src/maasserver/clusterrpc/utils.py 2014-10-08 09:43:51 +0000
153+++ src/maasserver/clusterrpc/utils.py 2014-11-12 12:20:32 +0000
154@@ -38,7 +38,7 @@
155
156 :param nodegroups: The :class:`NodeGroup`s on which to make the RPC
157 call. If None, defaults to all :class:`NodeGroup`s.
158- :param command: An :class:`amp.Command` to call on the clusters.
159+ :param command: An :class:`amp32.Command` to call on the clusters.
160 :param ignore_errors: If True, errors encountered whilst calling
161 `command` on the clusters won't raise an exception.
162 :return: A generator of results, i.e. the dicts returned by the RPC
163
164=== modified file 'src/maasserver/models/node.py'
165--- src/maasserver/models/node.py 2014-11-10 03:22:59 +0000
166+++ src/maasserver/models/node.py 2014-11-12 12:20:32 +0000
167@@ -111,6 +111,7 @@
168 from piston.models import Token
169 from provisioningserver.logger import get_maas_logger
170 from provisioningserver.power.poweraction import UnknownPowerType
171+from provisioningserver.rpc import amp32
172 from provisioningserver.rpc.cluster import (
173 CancelMonitor,
174 StartMonitors,
175@@ -118,7 +119,6 @@
176 from provisioningserver.rpc.power import QUERY_POWER_TYPES
177 from provisioningserver.utils.enum import map_enum_reverse
178 from provisioningserver.utils.twisted import asynchronous
179-from twisted.protocols import amp
180
181
182 maaslog = get_maas_logger("node")
183@@ -584,7 +584,7 @@
184 'node_status': self.status,
185 'timeout': timeout,
186 }
187- deadline = datetime.now(tz=amp.utc) + timedelta(seconds=timeout)
188+ deadline = datetime.now(tz=amp32.utc) + timedelta(seconds=timeout)
189 monitors = [{
190 'deadline': deadline,
191 'id': self.system_id,
192
193=== modified file 'src/maasserver/models/tests/test_node.py'
194--- src/maasserver/models/tests/test_node.py 2014-11-10 03:22:53 +0000
195+++ src/maasserver/models/tests/test_node.py 2014-11-12 12:20:32 +0000
196@@ -101,7 +101,10 @@
197 )
198 from provisioningserver.power.poweraction import UnknownPowerType
199 from provisioningserver.power_schema import JSON_POWER_TYPE_PARAMETERS
200-from provisioningserver.rpc import cluster as cluster_module
201+from provisioningserver.rpc import (
202+ amp32,
203+ cluster as cluster_module,
204+ )
205 from provisioningserver.rpc.cluster import StartMonitors
206 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
207 from provisioningserver.rpc.power import QUERY_POWER_TYPES
208@@ -114,7 +117,6 @@
209 )
210 from twisted.internet import defer
211 from twisted.internet.defer import Deferred
212-from twisted.protocols import amp
213 from twisted.python.failure import Failure
214
215
216@@ -2123,7 +2125,7 @@
217
218 def test__start_transition_monitor_starts_monitor(self):
219 rpc_fixture = self.prepare_rpc()
220- now = datetime.now(tz=amp.utc)
221+ now = datetime.now(tz=amp32.utc)
222 self.patch_datetime_now(now)
223 node = factory.make_Node()
224 cluster = rpc_fixture.makeCluster(node.nodegroup, StartMonitors)
225@@ -2142,7 +2144,7 @@
226 )
227
228 def test__start_transition_monitor_copes_with_timeouterror(self):
229- now = datetime.now(tz=amp.utc)
230+ now = datetime.now(tz=amp32.utc)
231 self.patch_datetime_now(now)
232 node = factory.make_Node()
233 mock_client = Mock()
234
235=== modified file 'src/maasserver/rpc/regionservice.py'
236--- src/maasserver/rpc/regionservice.py 2014-11-10 15:11:58 +0000
237+++ src/maasserver/rpc/regionservice.py 2014-11-12 12:20:32 +0000
238@@ -57,6 +57,7 @@
239 from maasserver.utils.async import transactional
240 from netaddr import IPAddress
241 from provisioningserver.rpc import (
242+ amp32,
243 cluster,
244 common,
245 exceptions,
246@@ -86,7 +87,6 @@
247 from twisted.internet.error import ConnectionClosed
248 from twisted.internet.protocol import Factory
249 from twisted.internet.threads import deferToThread
250-from twisted.protocols import amp
251 from twisted.python import log
252 from zope.interface import implementer
253
254@@ -151,11 +151,11 @@
255 """
256 return deferToThread(leases.update_leases, uuid, mappings)
257
258- @amp.StartTLS.responder
259+ @amp32.StartTLS.responder
260 def get_tls_parameters(self):
261 """get_tls_parameters()
262
263- Implementation of :py:class:`~twisted.protocols.amp.StartTLS`.
264+ Implementation of :py:class:`~provisioningserver.rpc.amp32.StartTLS`.
265 """
266 try:
267 from provisioningserver.rpc.testing import tls
268
269=== modified file 'src/maasserver/rpc/testing/fixtures.py'
270--- src/maasserver/rpc/testing/fixtures.py 2014-10-08 21:59:23 +0000
271+++ src/maasserver/rpc/testing/fixtures.py 2014-11-12 12:20:32 +0000
272@@ -191,7 +191,7 @@
273 def addCluster(self, protocol):
274 """Add a new stub cluster using the given `protocol`.
275
276- The `protocol` should be an instance of `amp.AMP`.
277+ The `protocol` should be an instance of `amp32.AMP`.
278
279 :returns: py:class:`twisted.test.iosim.IOPump`
280 """
281@@ -315,7 +315,7 @@
282 def addCluster(self, protocol):
283 """Add a new stub cluster using the given `protocol`.
284
285- The `protocol` should be an instance of `amp.AMP`.
286+ The `protocol` should be an instance of `amp32.AMP`.
287
288 :returns: A `Deferred` that fires with the connected protocol
289 instance.
290
291=== modified file 'src/maasserver/rpc/tests/test_regionservice.py'
292--- src/maasserver/rpc/tests/test_regionservice.py 2014-11-10 15:11:58 +0000
293+++ src/maasserver/rpc/tests/test_regionservice.py 2014-11-12 12:20:32 +0000
294@@ -87,6 +87,7 @@
295 import netaddr
296 from provisioningserver.network import discover_networks
297 from provisioningserver.rpc import (
298+ amp32,
299 cluster,
300 common,
301 exceptions,
302@@ -163,7 +164,6 @@
303 from twisted.internet.interfaces import IStreamServerEndpoint
304 from twisted.internet.protocol import Factory
305 from twisted.internet.threads import deferToThread
306-from twisted.protocols import amp
307 from twisted.python import log
308 from twisted.python.failure import Failure
309 from zope.interface.verify import verifyObject
310@@ -284,7 +284,7 @@
311
312 def test_StartTLS_is_registered(self):
313 protocol = Region()
314- responder = protocol.locateResponder(amp.StartTLS.commandName)
315+ responder = protocol.locateResponder(amp32.StartTLS.commandName)
316 self.assertIsNotNone(responder)
317
318 def test_get_tls_parameters_returns_parameters(self):
319@@ -292,7 +292,7 @@
320 # However, locateResponder() returns a closure, so we have to
321 # side-step it.
322 protocol = Region()
323- cls, func = protocol._commandDispatch[amp.StartTLS.commandName]
324+ cls, func = protocol._commandDispatch[amp32.StartTLS.commandName]
325 self.assertThat(func(protocol), are_valid_tls_parameters)
326
327 @wait_for_reactor
328@@ -303,7 +303,7 @@
329 # travelling over the wire as part of an AMP message. However,
330 # the responder is not aware of this, and is called just like
331 # any other.
332- d = call_responder(Region(), amp.StartTLS, {})
333+ d = call_responder(Region(), amp32.StartTLS, {})
334
335 def check(response):
336 self.assertEqual({}, response)
337@@ -1119,7 +1119,7 @@
338 service.factory.protocol = HandshakingRegionServer
339 protocol = service.factory.buildProtocol(addr=None) # addr is unused.
340 protocol.connectionMade()
341- connectionLost_up_call = self.patch(amp.AMP, "connectionLost")
342+ connectionLost_up_call = self.patch(amp32.AMP, "connectionLost")
343 self.assertDictEqual(
344 {protocol.ident: {protocol}},
345 service.connections)
346
347=== modified file 'src/provisioningserver/pserv_services/dhcp_probe_service.py'
348--- src/provisioningserver/pserv_services/dhcp_probe_service.py 2014-10-02 11:53:59 +0000
349+++ src/provisioningserver/pserv_services/dhcp_probe_service.py 2014-11-12 12:20:32 +0000
350@@ -22,6 +22,7 @@
351
352 from provisioningserver.dhcp.detect import probe_interface
353 from provisioningserver.logger.log import get_maas_logger
354+from provisioningserver.rpc.amp32 import UnhandledCommand
355 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
356 from provisioningserver.rpc.region import (
357 GetClusterInterfaces,
358@@ -37,7 +38,6 @@
359 returnValue,
360 )
361 from twisted.internet.threads import deferToThread
362-from twisted.protocols.amp import UnhandledCommand
363
364
365 maaslog = get_maas_logger("dhcp.probe")
366
367=== added file 'src/provisioningserver/rpc/amp32.py'
368--- src/provisioningserver/rpc/amp32.py 1970-01-01 00:00:00 +0000
369+++ src/provisioningserver/rpc/amp32.py 2014-11-12 12:20:32 +0000
370@@ -0,0 +1,2646 @@
371+# Copyright (c) 2005 Divmod, Inc.
372+# Copyright (c) Twisted Matrix Laboratories.
373+# See LICENSE.Twisted for details.
374+
375+"""
376+This module implements a modified version of AMP, the Asynchronous Messaging
377+Protocol, which is referred to as AMP32 from now on where it's necessary to
378+differentiate between the two.
379+
380+The original AMP uses 16-bit length prefixes when sending messages, which
381+limits payloads to 64k. This version uses 32-bit prefixes so that messages can
382+be much larger, though in practice they're limit them to something much
383+smaller that the maximum. Unfortunately this makes AMP32 incompatible with
384+AMP, though their essential behaviour is identical.
385+
386+AMP is a protocol for sending multiple asynchronous request/response pairs over
387+the same connection. Requests and responses are both collections of key/value
388+pairs.
389+
390+AMP is a very simple protocol which is not an application. This module is a
391+"protocol construction kit" of sorts; it attempts to be the simplest wire-level
392+implementation of Deferreds. AMP provides the following base-level features:
393+
394+ - Asynchronous request/response handling (hence the name)
395+
396+ - Requests and responses are both key/value pairs
397+
398+ - Binary transfer of all data: all data is length-prefixed. Your
399+ application will never need to worry about quoting.
400+
401+ - Command dispatching (like HTTP Verbs): the protocol is extensible, and
402+ multiple AMP sub-protocols can be grouped together easily.
403+
404+The protocol implementation also provides a few additional features which are
405+not part of the core wire protocol, but are nevertheless very useful:
406+
407+ - Tight TLS integration, with an included StartTLS command.
408+
409+ - Handshaking to other protocols: because AMP has well-defined message
410+ boundaries and maintains all incoming and outgoing requests for you, you
411+ can start a connection over AMP and then switch to another protocol.
412+ This makes it ideal for firewall-traversal applications where you may
413+ have only one forwarded port but multiple applications that want to use
414+ it.
415+
416+Using AMP with Twisted is simple. Each message is a command, with a response.
417+You begin by defining a command type. Commands specify their input and output
418+in terms of the types that they expect to see in the request and response
419+key-value pairs. Here's an example of a command that adds two integers, 'a'
420+and 'b'::
421+
422+ class Sum(amp32.Command):
423+ arguments = [('a', amp32.Integer()),
424+ ('b', amp32.Integer())]
425+ response = [('total', amp32.Integer())]
426+
427+Once you have specified a command, you need to make it part of a protocol, and
428+define a responder for it. Here's a 'JustSum' protocol that includes a
429+responder for our 'Sum' command::
430+
431+ class JustSum(amp32.AMP):
432+ def sum(self, a, b):
433+ total = a + b
434+ print 'Did a sum: %d + %d = %d' % (a, b, total)
435+ return {'total': total}
436+ Sum.responder(sum)
437+
438+Later, when you want to actually do a sum, the following expression will return
439+a L{Deferred} which will fire with the result::
440+
441+ ClientCreator(reactor, amp32.AMP).connectTCP(...).addCallback(
442+ lambda p: p.callRemote(Sum, a=13, b=81)).addCallback(
443+ lambda result: result['total'])
444+
445+Command responders may also return Deferreds, causing the response to be
446+sent only once the Deferred fires::
447+
448+ class DelayedSum(amp32.AMP):
449+ def slowSum(self, a, b):
450+ total = a + b
451+ result = defer.Deferred()
452+ reactor.callLater(3, result.callback, {'total': total})
453+ return result
454+ Sum.responder(slowSum)
455+
456+This is transparent to the caller.
457+
458+You can also define the propagation of specific errors in AMP. For example,
459+for the slightly more complicated case of division, we might have to deal with
460+division by zero::
461+
462+ class Divide(amp32.Command):
463+ arguments = [('numerator', amp32.Integer()),
464+ ('denominator', amp32.Integer())]
465+ response = [('result', amp32.Float())]
466+ errors = {ZeroDivisionError: 'ZERO_DIVISION'}
467+
468+The 'errors' mapping here tells AMP that if a responder to Divide emits a
469+L{ZeroDivisionError}, then the other side should be informed that an error of
470+the type 'ZERO_DIVISION' has occurred. Writing a responder which takes
471+advantage of this is very simple - just raise your exception normally::
472+
473+ class JustDivide(amp32.AMP):
474+ def divide(self, numerator, denominator):
475+ result = numerator / denominator
476+ print 'Divided: %d / %d = %d' % (numerator, denominator, total)
477+ return {'result': result}
478+ Divide.responder(divide)
479+
480+On the client side, the errors mapping will be used to determine what the
481+'ZERO_DIVISION' error means, and translated into an asynchronous exception,
482+which can be handled normally as any L{Deferred} would be::
483+
484+ def trapZero(result):
485+ result.trap(ZeroDivisionError)
486+ print "Divided by zero: returning INF"
487+ return 1e1000
488+ ClientCreator(reactor, amp32.AMP).connectTCP(...).addCallback(
489+ lambda p: p.callRemote(Divide, numerator=1234,
490+ denominator=0)
491+ ).addErrback(trapZero)
492+
493+For a complete, runnable example of both of these commands, see the files in
494+the Twisted repository::
495+
496+ doc/core/examples/ampserver.py
497+ doc/core/examples/ampclient.py
498+
499+On the wire, AMP is a protocol which uses 2-byte lengths to prefix keys and
500+values, and empty keys to separate messages::
501+
502+ <2-byte length><key><2-byte length><value>
503+ <2-byte length><key><2-byte length><value>
504+ ...
505+ <2-byte length><key><2-byte length><value>
506+ <NUL><NUL> # Empty Key == End of Message
507+
508+And so on. Because it's tedious to refer to lengths and NULs constantly, the
509+documentation will refer to packets as if they were newline delimited, like
510+so::
511+
512+ C: _command: sum
513+ C: _ask: ef639e5c892ccb54
514+ C: a: 13
515+ C: b: 81
516+
517+ S: _answer: ef639e5c892ccb54
518+ S: total: 94
519+
520+Notes:
521+
522+In general, the order of keys is arbitrary. Specific uses of AMP may impose an
523+ordering requirement, but unless this is specified explicitly, any ordering may
524+be generated and any ordering must be accepted. This applies to the
525+command-related keys I{_command} and I{_ask} as well as any other keys.
526+
527+Values are limited to the maximum encodable size in a 32-bit length.
528+
529+Keys are limited to the maximum encodable size in a 8-bit length, 255 bytes.
530+Note that we still use 2-byte lengths to encode keys. This small redundancy
531+has several features:
532+
533+ - If an implementation becomes confused and starts emitting corrupt data,
534+ or gets keys confused with values, many common errors will be signalled
535+ immediately instead of delivering obviously corrupt packets.
536+
537+ - A single NUL will separate every key, and a double NUL separates
538+ messages. This provides some redundancy when debugging traffic dumps.
539+
540+ - NULs will be present at regular intervals along the protocol, providing
541+ some padding for otherwise braindead C implementations of the protocol,
542+ so that <stdio.h> string functions will see the NUL and stop.
543+
544+ - This makes it possible to run an AMP server on a port also used by a
545+ plain-text protocol, and easily distinguish between non-AMP clients (like
546+ web browsers) which issue non-NUL as the first byte, and AMP clients,
547+ which always issue NUL as the first byte.
548+
549+"""
550+
551+from __future__ import (
552+ absolute_import,
553+ print_function,
554+ # unicode_literals,
555+ )
556+
557+str = None
558+
559+__metaclass__ = type
560+
561+import datetime
562+import decimal
563+from io import BytesIO
564+from itertools import count
565+from struct import pack
566+import types
567+import warnings
568+
569+from twisted.internet.defer import (
570+ Deferred,
571+ fail,
572+ maybeDeferred,
573+ )
574+from twisted.internet.error import (
575+ ConnectionClosed,
576+ ConnectionLost,
577+ PeerVerifyError,
578+ )
579+from twisted.internet.interfaces import IFileDescriptorReceiver
580+from twisted.internet.main import CONNECTION_LOST
581+from twisted.protocols.basic import (
582+ Int32StringReceiver,
583+ StatefulStringProtocol,
584+ )
585+from twisted.python import (
586+ filepath,
587+ log,
588+ )
589+from twisted.python.failure import Failure
590+from twisted.python.reflect import accumulateClassDict
591+from zope.interface import (
592+ implements,
593+ Interface,
594+ )
595+
596+
597+try:
598+ from twisted.internet import ssl
599+except ImportError:
600+ ssl = None
601+else:
602+ if ssl.supported:
603+ from twisted.internet.ssl import (
604+ CertificateOptions,
605+ Certificate,
606+ DN,
607+ KeyPair,
608+ )
609+ else:
610+ ssl = None
611+
612+
613+ASK = '_ask'
614+ANSWER = '_answer'
615+COMMAND = '_command'
616+ERROR = '_error'
617+ERROR_CODE = '_error_code'
618+ERROR_DESCRIPTION = '_error_description'
619+UNKNOWN_ERROR_CODE = 'UNKNOWN'
620+UNHANDLED_ERROR_CODE = 'UNHANDLED'
621+
622+MAX_KEY_LENGTH = 0xff
623+MAX_VALUE_LENGTH = 2 * 1024 * 1024 # 2MiB
624+
625+
626+class IArgumentType(Interface):
627+ """
628+ An L{IArgumentType} can serialize a Python object into an AMP box and
629+ deserialize information from an AMP box back into a Python object.
630+
631+ @since: 9.0
632+ """
633+
634+ def fromBox(name, strings, objects, proto):
635+ """
636+ Given an argument name and an AMP box containing serialized values,
637+ extract one or more Python objects and add them to the C{objects}
638+ dictionary.
639+
640+ @param name: The name associated with this argument. Most commonly,
641+ this is the key which can be used to find a serialized value in
642+ C{strings} and which should be used as the key in C{objects} to
643+ associate with a structured Python object.
644+ @type name: C{bytes}
645+
646+ @param strings: The AMP box from which to extract one or more
647+ values.
648+ @type strings: C{dict}
649+
650+ @param objects: The output dictionary to populate with the value for
651+ this argument.
652+ @type objects: C{dict}
653+
654+ @param proto: The protocol instance which received the AMP box being
655+ interpreted. Most likely this is an instance of L{AMP}, but
656+ this is not guaranteed.
657+
658+ @return: C{None}
659+ """
660+
661+ def toBox(name, strings, objects, proto):
662+ """
663+ Given an argument name and a dictionary containing structured Python
664+ objects, serialize values into one or more strings and add them to
665+ the C{strings} dictionary.
666+
667+ @param name: The name associated with this argument. Most commonly,
668+ this is the key which can be used to find an object in
669+ C{objects} and which should be used as the key in C{strings} to
670+ associate with a C{bytes} giving the serialized form of that
671+ object.
672+ @type name: C{bytes}
673+
674+ @param strings: The AMP box into which to insert one or more
675+ strings.
676+ @type strings: C{dict}
677+
678+ @param objects: The input dictionary from which to extract Python
679+ objects to serialize.
680+ @type objects: C{dict}
681+
682+ @param proto: The protocol instance which will send the AMP box once
683+ it is fully populated. Most likely this is an instance of
684+ L{AMP}, but this is not guaranteed.
685+
686+ @return: C{None}
687+ """
688+
689+
690+class IBoxSender(Interface):
691+ """
692+ A transport which can send L{AmpBox} objects.
693+ """
694+
695+ def sendBox(box):
696+ """
697+ Send an L{AmpBox}.
698+
699+ @raise ProtocolSwitched: if the underlying protocol has been
700+ switched.
701+
702+ @raise ConnectionLost: if the underlying connection has already been
703+ lost.
704+ """
705+
706+ def unhandledError(failure):
707+ """
708+ An unhandled error occurred in response to a box. Log it
709+ appropriately.
710+
711+ @param failure: a L{Failure} describing the error that occurred.
712+ """
713+
714+
715+class IBoxReceiver(Interface):
716+ """
717+ An application object which can receive L{AmpBox} objects and dispatch them
718+ appropriately.
719+ """
720+
721+ def startReceivingBoxes(boxSender):
722+ """
723+ The L{ampBoxReceived} method will start being called; boxes may be
724+ responded to by responding to the given L{IBoxSender}.
725+
726+ @param boxSender: an L{IBoxSender} provider.
727+ """
728+
729+ def ampBoxReceived(box):
730+ """
731+ A box was received from the transport; dispatch it appropriately.
732+ """
733+
734+ def stopReceivingBoxes(reason):
735+ """
736+ No further boxes will be received on this connection.
737+
738+ @type reason: L{Failure}
739+ """
740+
741+
742+class IResponderLocator(Interface):
743+ """
744+ An application object which can look up appropriate responder methods for
745+ AMP commands.
746+ """
747+
748+ def locateResponder(name):
749+ """
750+ Locate a responder method appropriate for the named command.
751+
752+ @param name: the wire-level name (commandName) of the AMP command to be
753+ responded to.
754+
755+ @return: a 1-argument callable that takes an L{AmpBox} with argument
756+ values for the given command, and returns an L{AmpBox} containing
757+ argument values for the named command, or a L{Deferred} that fires the
758+ same.
759+ """
760+
761+
762+class AmpError(Exception):
763+ """
764+ Base class of all Amp-related exceptions.
765+ """
766+
767+
768+class ProtocolSwitched(Exception):
769+ """
770+ Connections which have been switched to other protocols can no longer
771+ accept traffic at the AMP level. This is raised when you try to send it.
772+ """
773+
774+
775+class OnlyOneTLS(AmpError):
776+ """
777+ This is an implementation limitation; TLS may only be started once per
778+ connection.
779+ """
780+
781+
782+class NoEmptyBoxes(AmpError):
783+ """
784+ You can't have empty boxes on the connection. This is raised when you
785+ receive or attempt to send one.
786+ """
787+
788+
789+class InvalidSignature(AmpError):
790+ """
791+ You didn't pass all the required arguments.
792+ """
793+
794+
795+class TooLong(AmpError):
796+ """
797+ One of the protocol's length limitations was violated.
798+
799+ @ivar isKey: true if the string being encoded in a key position, false if
800+ it was in a value position.
801+
802+ @ivar isLocal: Was the string encoded locally, or received too long from
803+ the network? (It's only physically possible to encode "too long" values on
804+ the network for keys.)
805+
806+ @ivar value: The string that was too long.
807+
808+ @ivar keyName: If the string being encoded was in a value position, what
809+ key was it being encoded for?
810+ """
811+
812+ def __init__(self, isKey, isLocal, value, keyName=None):
813+ AmpError.__init__(self)
814+ self.isKey = isKey
815+ self.isLocal = isLocal
816+ self.value = value
817+ self.keyName = keyName
818+
819+ def __repr__(self):
820+ hdr = self.isKey and "key" or "value"
821+ if not self.isKey:
822+ hdr += ' ' + repr(self.keyName)
823+ lcl = self.isLocal and "local" or "remote"
824+ return "%s %s too long: %d" % (lcl, hdr, len(self.value))
825+
826+
827+class BadLocalReturn(AmpError):
828+ """
829+ A bad value was returned from a local command; we were unable to coerce it.
830+ """
831+
832+ def __init__(self, message, enclosed):
833+ AmpError.__init__(self)
834+ self.message = message
835+ self.enclosed = enclosed
836+
837+ def __repr__(self):
838+ return self.message + " " + self.enclosed.getBriefTraceback()
839+
840+ __bytes__ = __repr__
841+
842+
843+class RemoteAmpError(AmpError):
844+ """
845+ This error indicates that something went wrong on the remote end of the
846+ connection, and the error was serialized and transmitted to you.
847+ """
848+
849+ def __init__(self, errorCode, description, fatal=False, local=None):
850+ """Create a remote error with an error code and description.
851+
852+ @param errorCode: the AMP error code of this error.
853+
854+ @param description: some text to show to the user.
855+
856+ @param fatal: a boolean, true if this error should terminate the
857+ connection.
858+
859+ @param local: a local Failure, if one exists.
860+ """
861+ if local:
862+ localwhat = ' (local)'
863+ othertb = local.getBriefTraceback()
864+ else:
865+ localwhat = ''
866+ othertb = ''
867+ Exception.__init__(
868+ self, "Code<%s>%s: %s%s" % (
869+ errorCode, localwhat,
870+ description, othertb))
871+ self.local = local
872+ self.errorCode = errorCode
873+ self.description = description
874+ self.fatal = fatal
875+
876+
877+class UnknownRemoteError(RemoteAmpError):
878+ """
879+ This means that an error whose type we can't identify was raised from the
880+ other side.
881+ """
882+
883+ def __init__(self, description):
884+ errorCode = UNKNOWN_ERROR_CODE
885+ RemoteAmpError.__init__(self, errorCode, description)
886+
887+
888+class MalformedAmpBox(AmpError):
889+ """
890+ This error indicates that the wire-level protocol was malformed.
891+ """
892+
893+
894+class UnhandledCommand(AmpError):
895+ """
896+ A command received via amp could not be dispatched.
897+ """
898+
899+
900+class IncompatibleVersions(AmpError):
901+ """
902+ It was impossible to negotiate a compatible version of the protocol with
903+ the other end of the connection.
904+ """
905+
906+
907+PROTOCOL_ERRORS = {UNHANDLED_ERROR_CODE: UnhandledCommand}
908+
909+
910+class AmpBox(dict):
911+ """
912+ I am a packet in the AMP protocol, much like a regular bytes:bytes
913+ dictionary.
914+ """
915+
916+ # be like a regular dictionary, don't magically acquire a __dict__...
917+ __slots__ = []
918+
919+ def copy(self):
920+ """
921+ Return another AmpBox just like me.
922+ """
923+ newBox = self.__class__()
924+ newBox.update(self)
925+ return newBox
926+
927+ def serialize(self):
928+ """
929+ Convert me into a wire-encoded string.
930+
931+ @return: a bytes encoded according to the rules described in the module
932+ docstring.
933+ """
934+ i = sorted(self.viewitems())
935+ L = []
936+ w = L.append
937+ for k, v in i:
938+ if type(k) == unicode:
939+ raise TypeError("Unicode key not allowed: %r" % k)
940+ if type(v) == unicode:
941+ raise TypeError(
942+ "Unicode value for key %r not allowed: %r" % (k, v))
943+ if len(k) > MAX_KEY_LENGTH:
944+ raise TooLong(True, True, k, None)
945+ if len(v) > MAX_VALUE_LENGTH:
946+ raise TooLong(False, True, v, k)
947+ for kv in k, v:
948+ w(pack("!I", len(kv)))
949+ w(kv)
950+ w(pack("!I", 0))
951+ return ''.join(L)
952+
953+ def _sendTo(self, proto):
954+ """
955+ Serialize and send this box to a Amp instance. By the time it is being
956+ sent, several keys are required. I must have exactly ONE of::
957+
958+ _ask
959+ _answer
960+ _error
961+
962+ If the '_ask' key is set, then the '_command' key must also be
963+ set.
964+
965+ @param proto: an AMP instance.
966+ """
967+ proto.sendBox(self)
968+
969+ def __repr__(self):
970+ return 'AmpBox(%s)' % (dict.__repr__(self),)
971+
972+
973+# amp32.Box => AmpBox
974+Box = AmpBox
975+
976+
977+class QuitBox(AmpBox):
978+ """
979+ I am an AmpBox that, upon being sent, terminates the connection.
980+ """
981+
982+ __slots__ = []
983+
984+ def __repr__(self):
985+ return 'QuitBox(**%s)' % (super(QuitBox, self).__repr__(),)
986+
987+ def _sendTo(self, proto):
988+ """
989+ Immediately call loseConnection after sending.
990+ """
991+ super(QuitBox, self)._sendTo(proto)
992+ proto.transport.loseConnection()
993+
994+
995+class _SwitchBox(AmpBox):
996+ """
997+ Implementation detail of ProtocolSwitchCommand: I am a AmpBox which sets
998+ up state for the protocol to switch.
999+ """
1000+
1001+ # DON'T set __slots__ here; we do have an attribute.
1002+
1003+ def __init__(self, innerProto, **kw):
1004+ """
1005+ Create a _SwitchBox with the protocol to switch to after being sent.
1006+
1007+ @param innerProto: the protocol instance to switch to.
1008+ @type innerProto: an IProtocol provider.
1009+ """
1010+ super(_SwitchBox, self).__init__(**kw)
1011+ self.innerProto = innerProto
1012+
1013+ def __repr__(self):
1014+ return '_SwitchBox(%r, **%s)' % (self.innerProto,
1015+ dict.__repr__(self),)
1016+
1017+ def _sendTo(self, proto):
1018+ """
1019+ Send me; I am the last box on the connection. All further traffic will
1020+ be over the new protocol.
1021+ """
1022+ super(_SwitchBox, self)._sendTo(proto)
1023+ proto._lockForSwitch()
1024+ proto._switchTo(self.innerProto)
1025+
1026+
1027+class BoxDispatcher:
1028+ """
1029+ A L{BoxDispatcher} dispatches '_ask', '_answer', and '_error' L{AmpBox}es,
1030+ both incoming and outgoing, to their appropriate destinations.
1031+
1032+ Outgoing commands are converted into L{Deferred}s and outgoing boxes, and
1033+ associated tracking state to fire those L{Deferred} when '_answer' boxes
1034+ come back. Incoming '_answer' and '_error' boxes are converted into
1035+ callbacks and errbacks on those L{Deferred}s, respectively.
1036+
1037+ Incoming '_ask' boxes are converted into method calls on a supplied method
1038+ locator.
1039+
1040+ @ivar _outstandingRequests: a dictionary mapping request IDs to
1041+ L{Deferred}s which were returned for those requests.
1042+
1043+ @ivar locator: an object with a L{locateResponder} method that locates a
1044+ responder function that takes a Box and returns a result (either a Box or a
1045+ Deferred which fires one).
1046+
1047+ @ivar boxSender: an object which can send boxes, via the L{_sendBox}
1048+ method, such as an L{AMP} instance.
1049+ @type boxSender: L{IBoxSender}
1050+ """
1051+
1052+ implements(IBoxReceiver)
1053+
1054+ _failAllReason = None
1055+ _outstandingRequests = None
1056+ _counter = 0
1057+ boxSender = None
1058+
1059+ def __init__(self, locator):
1060+ self._outstandingRequests = {}
1061+ self.locator = locator
1062+
1063+ def startReceivingBoxes(self, boxSender):
1064+ """
1065+ The given boxSender is going to start calling boxReceived on this
1066+ L{BoxDispatcher}.
1067+
1068+ @param boxSender: The L{IBoxSender} to send command responses to.
1069+ """
1070+ self.boxSender = boxSender
1071+
1072+ def stopReceivingBoxes(self, reason):
1073+ """
1074+ No further boxes will be received here. Terminate all currently
1075+ oustanding command deferreds with the given reason.
1076+ """
1077+ self.failAllOutgoing(reason)
1078+
1079+ def failAllOutgoing(self, reason):
1080+ """
1081+ Call the errback on all outstanding requests awaiting responses.
1082+
1083+ @param reason: the Failure instance to pass to those errbacks.
1084+ """
1085+ self._failAllReason = reason
1086+ OR = list(self._outstandingRequests.viewitems())
1087+ self._outstandingRequests = None # we can never send another request
1088+ for key, value in OR:
1089+ value.errback(reason)
1090+
1091+ def _nextTag(self):
1092+ """
1093+ Generate protocol-local serial numbers for _ask keys.
1094+
1095+ @return: a string that has not yet been used on this connection.
1096+ """
1097+ self._counter += 1
1098+ return '%x' % (self._counter,)
1099+
1100+ def _sendBoxCommand(self, command, box, requiresAnswer=True):
1101+ """
1102+ Send a command across the wire with the given C{amp32.Box}.
1103+
1104+ Mutate the given box to give it any additional keys (_command, _ask)
1105+ required for the command and request/response machinery, then send it.
1106+
1107+ If requiresAnswer is True, returns a C{Deferred} which fires when a
1108+ response is received. The C{Deferred} is fired with an C{amp32.Box} on
1109+ success, or with an C{amp32.RemoteAmpError} if an error is received.
1110+
1111+ If the Deferred fails and the error is not handled by the caller of
1112+ this method, the failure will be logged and the connection dropped.
1113+
1114+ @param command: a bytes, the name of the command to issue.
1115+
1116+ @param box: an AmpBox with the arguments for the command.
1117+
1118+ @param requiresAnswer: a boolean. Defaults to True. If True, return a
1119+ Deferred which will fire when the other side responds to this command.
1120+ If False, return None and do not ask the other side for
1121+ acknowledgement.
1122+
1123+ @return: a Deferred which fires the AmpBox that holds the response to
1124+ this command, or None, as specified by requiresAnswer.
1125+
1126+ @raise ProtocolSwitched: if the protocol has been switched.
1127+ """
1128+ if self._failAllReason is not None:
1129+ return fail(self._failAllReason)
1130+ box[COMMAND] = command
1131+ tag = self._nextTag()
1132+ if requiresAnswer:
1133+ box[ASK] = tag
1134+ box._sendTo(self.boxSender)
1135+ if requiresAnswer:
1136+ result = self._outstandingRequests[tag] = Deferred()
1137+ else:
1138+ result = None
1139+ return result
1140+
1141+ def callRemoteString(self, command, requiresAnswer=True, **kw):
1142+ """
1143+ This is a low-level API, designed only for optimizing simple messages
1144+ for which the overhead of parsing is too great.
1145+
1146+ @param command: a bytes naming the command.
1147+
1148+ @param kw: arguments to the amp box.
1149+
1150+ @param requiresAnswer: a boolean. Defaults to True. If True, return a
1151+ Deferred which will fire when the other side responds to this command.
1152+ If False, return None and do not ask the other side for
1153+ acknowledgement.
1154+
1155+ @return: a Deferred which fires the AmpBox that holds the response to
1156+ this command, or None, as specified by requiresAnswer.
1157+ """
1158+ box = Box(kw)
1159+ return self._sendBoxCommand(command, box, requiresAnswer)
1160+
1161+ def callRemote(self, commandType, *a, **kw):
1162+ """
1163+ This is the primary high-level API for sending messages via AMP.
1164+ Invoke it with a command and appropriate arguments to send a message
1165+ to this connection's peer.
1166+
1167+ @param commandType: a subclass of Command.
1168+ @type commandType: L{type}
1169+
1170+ @param a: Positional (special) parameters taken by the command.
1171+ Positional parameters will typically not be sent over the wire. The
1172+ only command included with AMP which uses positional parameters is
1173+ L{ProtocolSwitchCommand}, which takes the protocol that will be
1174+ switched to as its first argument.
1175+
1176+ @param kw: Keyword arguments taken by the command. These are the
1177+ arguments declared in the command's 'arguments' attribute. They will
1178+ be encoded and sent to the peer as arguments for the L{commandType}.
1179+
1180+ @return: If L{commandType} has a C{requiresAnswer} attribute set to
1181+ L{False}, then return L{None}. Otherwise, return a L{Deferred} which
1182+ fires with a dictionary of objects representing the result of this
1183+ call. Additionally, this L{Deferred} may fail with an exception
1184+ representing a connection failure, with L{UnknownRemoteError} if the
1185+ other end of the connection fails for an unknown reason, or with any
1186+ error specified as a key in L{commandType}'s C{errors} dictionary.
1187+ """
1188+
1189+ # XXX this takes command subclasses and not command objects on purpose.
1190+ # There's really no reason to have all this back-and-forth between
1191+ # command objects and the protocol, and the extra object being created
1192+ # (the Command instance) is pointless. Command is kind of like
1193+ # Interface, and should be more like it.
1194+
1195+ # In other words, the fact that commandType is instantiated here is an
1196+ # implementation detail. Don't rely on it.
1197+
1198+ try:
1199+ co = commandType(*a, **kw)
1200+ except:
1201+ return fail()
1202+ return co._doCommand(self)
1203+
1204+ def unhandledError(self, failure):
1205+ """
1206+ This is a terminal callback called after application code has had a
1207+ chance to quash any errors.
1208+ """
1209+ return self.boxSender.unhandledError(failure)
1210+
1211+ def _answerReceived(self, box):
1212+ """
1213+ An AMP box was received that answered a command previously sent with
1214+ L{callRemote}.
1215+
1216+ @param box: an AmpBox with a value for its L{ANSWER} key.
1217+ """
1218+ question = self._outstandingRequests.pop(box[ANSWER])
1219+ question.addErrback(self.unhandledError)
1220+ question.callback(box)
1221+
1222+ def _errorReceived(self, box):
1223+ """
1224+ An AMP box was received that answered a command previously sent with
1225+ L{callRemote}, with an error.
1226+
1227+ @param box: an L{AmpBox} with a value for its L{ERROR}, L{ERROR_CODE},
1228+ and L{ERROR_DESCRIPTION} keys.
1229+ """
1230+ question = self._outstandingRequests.pop(box[ERROR])
1231+ question.addErrback(self.unhandledError)
1232+ errorCode = box[ERROR_CODE]
1233+ description = box[ERROR_DESCRIPTION]
1234+ if errorCode in PROTOCOL_ERRORS:
1235+ exc = PROTOCOL_ERRORS[errorCode](errorCode, description)
1236+ else:
1237+ exc = RemoteAmpError(errorCode, description)
1238+ question.errback(Failure(exc))
1239+
1240+ def _commandReceived(self, box):
1241+ """
1242+ @param box: an L{AmpBox} with a value for its L{COMMAND} and L{ASK}
1243+ keys.
1244+ """
1245+ def formatAnswer(answerBox):
1246+ answerBox[ANSWER] = box[ASK]
1247+ return answerBox
1248+
1249+ def formatError(error):
1250+ if error.check(RemoteAmpError):
1251+ code = error.value.errorCode
1252+ desc = error.value.description
1253+ if error.value.fatal:
1254+ errorBox = QuitBox()
1255+ else:
1256+ errorBox = AmpBox()
1257+ else:
1258+ errorBox = QuitBox()
1259+ # here is where server-side logging happens if the error isn't
1260+ # handled
1261+ log.err(error)
1262+ code = UNKNOWN_ERROR_CODE
1263+ desc = "Unknown Error"
1264+ errorBox[ERROR] = box[ASK]
1265+ errorBox[ERROR_DESCRIPTION] = desc
1266+ errorBox[ERROR_CODE] = code
1267+ return errorBox
1268+
1269+ deferred = self.dispatchCommand(box)
1270+ if ASK in box:
1271+ deferred.addCallbacks(formatAnswer, formatError)
1272+ deferred.addCallback(self._safeEmit)
1273+ deferred.addErrback(self.unhandledError)
1274+
1275+ def ampBoxReceived(self, box):
1276+ """
1277+ An AmpBox was received, representing a command, or an answer to a
1278+ previously issued command (either successful or erroneous). Respond to
1279+ it according to its contents.
1280+
1281+ @param box: an AmpBox
1282+
1283+ @raise NoEmptyBoxes: when a box is received that does not contain an
1284+ '_answer', '_command' / '_ask', or '_error' key; i.e. one which does
1285+ not fit into the command / response protocol defined by AMP.
1286+ """
1287+ if ANSWER in box:
1288+ self._answerReceived(box)
1289+ elif ERROR in box:
1290+ self._errorReceived(box)
1291+ elif COMMAND in box:
1292+ self._commandReceived(box)
1293+ else:
1294+ raise NoEmptyBoxes(box)
1295+
1296+ def _safeEmit(self, aBox):
1297+ """
1298+ Emit a box, ignoring L{ProtocolSwitched} and L{ConnectionLost} errors
1299+ which cannot be usefully handled.
1300+ """
1301+ try:
1302+ aBox._sendTo(self.boxSender)
1303+ except (ProtocolSwitched, ConnectionLost):
1304+ pass
1305+
1306+ def dispatchCommand(self, box):
1307+ """
1308+ A box with a _command key was received.
1309+
1310+ Dispatch it to a local handler call it.
1311+
1312+ @param proto: an AMP instance.
1313+ @param box: an AmpBox to be dispatched.
1314+ """
1315+ cmd = box[COMMAND]
1316+ responder = self.locator.locateResponder(cmd)
1317+ if responder is None:
1318+ return fail(RemoteAmpError(
1319+ UNHANDLED_ERROR_CODE, "Unhandled Command: %r" % (cmd,),
1320+ False, local=Failure(UnhandledCommand())))
1321+ return maybeDeferred(responder, box)
1322+
1323+
1324+class CommandLocator:
1325+ """
1326+ A L{CommandLocator} is a collection of responders to AMP L{Command}s, with
1327+ the help of the L{Command.responder} decorator.
1328+ """
1329+
1330+ class __metaclass__(type):
1331+ """
1332+ This metaclass keeps track of all of the Command.responder-decorated
1333+ methods defined since the last CommandLocator subclass was defined. It
1334+ assumes (usually correctly, but unfortunately not necessarily so) that
1335+ those commands responders were all declared as methods of the class
1336+ being defined. Note that this list can be incorrect if users use the
1337+ Command.responder decorator outside the context of a CommandLocator
1338+ class declaration.
1339+
1340+ Command responders defined on subclasses are given precedence over
1341+ those inherited from a base class.
1342+
1343+ The Command.responder decorator explicitly cooperates with this
1344+ metaclass.
1345+ """
1346+
1347+ _currentClassCommands = []
1348+
1349+ def __new__(cls, name, bases, attrs):
1350+ commands = cls._currentClassCommands[:]
1351+ cls._currentClassCommands[:] = []
1352+ cd = attrs['_commandDispatch'] = {}
1353+ subcls = type.__new__(cls, name, bases, attrs)
1354+ ancestors = list(subcls.__mro__[1:])
1355+ ancestors.reverse()
1356+ for ancestor in ancestors:
1357+ cd.update(getattr(ancestor, '_commandDispatch', {}))
1358+ for commandClass, responderFunc in commands:
1359+ cd[commandClass.commandName] = (commandClass, responderFunc)
1360+ if (bases and (
1361+ subcls.lookupFunction != CommandLocator.lookupFunction)):
1362+ def locateResponder(self, name):
1363+ warnings.warn(
1364+ "Override locateResponder, not lookupFunction.",
1365+ category=PendingDeprecationWarning,
1366+ stacklevel=2)
1367+ return self.lookupFunction(name)
1368+ subcls.locateResponder = locateResponder
1369+ return subcls
1370+
1371+ implements(IResponderLocator)
1372+
1373+ def _wrapWithSerialization(self, aCallable, command):
1374+ """
1375+ Wrap aCallable with its command's argument de-serialization
1376+ and result serialization logic.
1377+
1378+ @param aCallable: a callable with a 'command' attribute, designed to be
1379+ called with keyword arguments.
1380+
1381+ @param command: the command class whose serialization to use.
1382+
1383+ @return: a 1-arg callable which, when invoked with an AmpBox, will
1384+ deserialize the argument list and invoke appropriate user code for the
1385+ callable's command, returning a Deferred which fires with the result or
1386+ fails with an error.
1387+ """
1388+ def doit(box):
1389+ kw = command.parseArguments(box, self)
1390+
1391+ def checkKnownErrors(error):
1392+ key = error.trap(*command.allErrors)
1393+ code = command.allErrors[key]
1394+ desc = bytes(error.value)
1395+ return Failure(RemoteAmpError(
1396+ code, desc, key in command.fatalErrors, local=error))
1397+
1398+ def makeResponseFor(objects):
1399+ try:
1400+ return command.makeResponse(objects, self)
1401+ except:
1402+ # let's helpfully log this.
1403+ originalFailure = Failure()
1404+ raise BadLocalReturn(
1405+ "%r returned %r and %r could not serialize it" % (
1406+ aCallable,
1407+ objects,
1408+ command),
1409+ originalFailure)
1410+
1411+ return maybeDeferred(aCallable, **kw).addCallback(
1412+ makeResponseFor).addErrback(
1413+ checkKnownErrors)
1414+
1415+ return doit
1416+
1417+ def lookupFunction(self, name):
1418+ """
1419+ Deprecated synonym for L{locateResponder}
1420+ """
1421+ if self.__class__.lookupFunction != CommandLocator.lookupFunction:
1422+ return CommandLocator.locateResponder(self, name)
1423+ else:
1424+ warnings.warn("Call locateResponder, not lookupFunction.",
1425+ category=PendingDeprecationWarning,
1426+ stacklevel=2)
1427+ return self.locateResponder(name)
1428+
1429+ def locateResponder(self, name):
1430+ """
1431+ Locate a callable to invoke when executing the named command.
1432+
1433+ @param name: the normalized name (from the wire) of the command.
1434+
1435+ @return: a 1-argument function that takes a Box and returns a box or a
1436+ Deferred which fires a Box, for handling the command identified by the
1437+ given name, or None, if no appropriate responder can be found.
1438+ """
1439+ # Try to find a high-level method to invoke, and if we can't find one,
1440+ # fall back to a low-level one.
1441+ cd = self._commandDispatch
1442+ if name in cd:
1443+ commandClass, responderFunc = cd[name]
1444+ responderMethod = types.MethodType(
1445+ responderFunc, self, self.__class__)
1446+ return self._wrapWithSerialization(responderMethod, commandClass)
1447+
1448+
1449+class SimpleStringLocator(object):
1450+ """
1451+ Implement the L{locateResponder} method to do simple, string-based
1452+ dispatch.
1453+ """
1454+
1455+ implements(IResponderLocator)
1456+
1457+ baseDispatchPrefix = 'amp_'
1458+
1459+ def locateResponder(self, name):
1460+ """
1461+ Locate a callable to invoke when executing the named command.
1462+
1463+ @return: a function with the name C{"amp_" + name} on L{self}, or None
1464+ if no such function exists. This function will then be called with the
1465+ L{AmpBox} itself as an argument.
1466+
1467+ @param name: the normalized name (from the wire) of the command.
1468+ """
1469+ fName = self.baseDispatchPrefix + (name.upper())
1470+ return getattr(self, fName, None)
1471+
1472+
1473+PYTHON_KEYWORDS = [
1474+ 'and', 'del', 'for', 'is', 'raise', 'assert', 'elif', 'from', 'lambda',
1475+ 'return', 'break', 'else', 'global', 'not', 'try', 'class', 'except',
1476+ 'if', 'or', 'while', 'continue', 'exec', 'import', 'pass', 'yield',
1477+ 'def', 'finally', 'in', 'print']
1478+
1479+
1480+def _wireNameToPythonIdentifier(key):
1481+ """
1482+ (Private) Normalize an argument name from the wire for use with Python
1483+ code. If the return value is going to be a python keyword it will be
1484+ capitalized. If it contains any dashes they will be replaced with
1485+ underscores.
1486+
1487+ The rationale behind this method is that AMP should be an inherently
1488+ multi-language protocol, so message keys may contain all manner of bizarre
1489+ bytes. This is not a complete solution; there are still forms of arguments
1490+ that this implementation will be unable to parse. However, Python
1491+ identifiers share a huge raft of properties with identifiers from many
1492+ other languages, so this is a 'good enough' effort for now. We deal
1493+ explicitly with dashes because that is the most likely departure: Lisps
1494+ commonly use dashes to separate method names, so protocols initially
1495+ implemented in a lisp amp dialect may use dashes in argument or command
1496+ names.
1497+
1498+ @param key: a bytes, looking something like 'foo-bar-baz' or 'from'
1499+
1500+ @return: a bytes which is a valid python identifier, looking something like
1501+ 'foo_bar_baz' or 'From'.
1502+ """
1503+ lkey = key.replace("-", "_")
1504+ if lkey in PYTHON_KEYWORDS:
1505+ return lkey.title()
1506+ return lkey
1507+
1508+
1509+class Argument:
1510+ """
1511+ Base-class of all objects that take values from Amp packets and convert
1512+ them into objects for Python functions.
1513+
1514+ This implementation of L{IArgumentType} provides several higher-level
1515+ hooks for subclasses to override. See L{toString} and L{fromString}
1516+ which will be used to define the behavior of L{IArgumentType.toBox} and
1517+ L{IArgumentType.fromBox}, respectively.
1518+ """
1519+
1520+ implements(IArgumentType)
1521+
1522+ optional = False
1523+
1524+ def __init__(self, optional=False):
1525+ """
1526+ Create an Argument.
1527+
1528+ @param optional: a boolean indicating whether this argument can be
1529+ omitted in the protocol.
1530+ """
1531+ self.optional = optional
1532+
1533+ def retrieve(self, d, name, proto):
1534+ """
1535+ Retrieve the given key from the given dictionary, removing it if found.
1536+
1537+ @param d: a dictionary.
1538+
1539+ @param name: a key in L{d}.
1540+
1541+ @param proto: an instance of an AMP.
1542+
1543+ @raise KeyError: if I am not optional and no value was found.
1544+
1545+ @return: d[name].
1546+ """
1547+ if self.optional:
1548+ value = d.get(name)
1549+ if value is not None:
1550+ del d[name]
1551+ else:
1552+ value = d.pop(name)
1553+ return value
1554+
1555+ def fromBox(self, name, strings, objects, proto):
1556+ """
1557+ Populate an 'out' dictionary with mapping names to Python values
1558+ decoded from an 'in' AmpBox mapping strings to string values.
1559+
1560+ @param name: the argument name to retrieve
1561+ @type name: bytes
1562+
1563+ @param strings: The AmpBox to read string(s) from, a mapping of
1564+ argument names to string values.
1565+ @type strings: AmpBox
1566+
1567+ @param objects: The dictionary to write object(s) to, a mapping of
1568+ names to Python objects.
1569+ @type objects: dict
1570+
1571+ @param proto: an AMP instance.
1572+ """
1573+ st = self.retrieve(strings, name, proto)
1574+ nk = _wireNameToPythonIdentifier(name)
1575+ if self.optional and st is None:
1576+ objects[nk] = None
1577+ else:
1578+ objects[nk] = self.fromStringProto(st, proto)
1579+
1580+ def toBox(self, name, strings, objects, proto):
1581+ """
1582+ Populate an 'out' AmpBox with strings encoded from an 'in' dictionary
1583+ mapping names to Python values.
1584+
1585+ @param name: the argument name to retrieve
1586+ @type name: bytes
1587+
1588+ @param strings: The AmpBox to write string(s) to, a mapping of
1589+ argument names to string values.
1590+ @type strings: AmpBox
1591+
1592+ @param objects: The dictionary to read object(s) from, a mapping of
1593+ names to Python objects.
1594+
1595+ @type objects: dict
1596+
1597+ @param proto: the protocol we are converting for.
1598+ @type proto: AMP
1599+ """
1600+ obj = self.retrieve(objects, _wireNameToPythonIdentifier(name), proto)
1601+ if self.optional and obj is None:
1602+ # strings[name] = None
1603+ pass
1604+ else:
1605+ strings[name] = self.toStringProto(obj, proto)
1606+
1607+ def fromStringProto(self, inString, proto):
1608+ """
1609+ Convert a string to a Python value.
1610+
1611+ @param inString: the string to convert.
1612+
1613+ @param proto: the protocol we are converting for.
1614+ @type proto: AMP
1615+
1616+ @return: a Python object.
1617+ """
1618+ return self.fromString(inString)
1619+
1620+ def toStringProto(self, inObject, proto):
1621+ """
1622+ Convert a Python object to a string.
1623+
1624+ @param inObject: the object to convert.
1625+
1626+ @param proto: the protocol we are converting for.
1627+ @type proto: AMP
1628+ """
1629+ return self.toString(inObject)
1630+
1631+ def fromString(self, inString):
1632+ """
1633+ Convert a string to a Python object. Subclasses must implement this.
1634+
1635+ @param inString: the string to convert.
1636+ @type inString: bytes
1637+
1638+ @return: the decoded value from inString
1639+ """
1640+
1641+ def toString(self, inObject):
1642+ """
1643+ Convert a Python object into a string for passing over the network.
1644+
1645+ @param inObject: an object of the type that this Argument is intended
1646+ to deal with.
1647+
1648+ @return: the wire encoding of inObject
1649+ @rtype: bytes
1650+ """
1651+
1652+
1653+class Integer(Argument):
1654+ """
1655+ Encode any integer values of any size on the wire as the string
1656+ representation.
1657+
1658+ Example: C{123} becomes C{"123"}
1659+ """
1660+
1661+ fromString = int
1662+
1663+ def toString(self, inObject):
1664+ return bytes(int(inObject))
1665+
1666+
1667+class String(Argument):
1668+ """
1669+ Don't do any conversion at all; just pass through 'bytes'.
1670+ """
1671+
1672+ def toString(self, inObject):
1673+ return inObject
1674+
1675+ def fromString(self, inString):
1676+ return inString
1677+
1678+
1679+class Float(Argument):
1680+ """
1681+ Encode floating-point values on the wire as their repr.
1682+ """
1683+
1684+ fromString = float
1685+ toString = repr
1686+
1687+
1688+class Boolean(Argument):
1689+ """
1690+ Encode True or False as "True" or "False" on the wire.
1691+ """
1692+
1693+ def fromString(self, inString):
1694+ if inString == 'True':
1695+ return True
1696+ elif inString == 'False':
1697+ return False
1698+ else:
1699+ raise TypeError("Bad boolean value: %r" % (inString,))
1700+
1701+ def toString(self, inObject):
1702+ if inObject:
1703+ return 'True'
1704+ else:
1705+ return 'False'
1706+
1707+
1708+class Unicode(String):
1709+ """
1710+ Encode a unicode string on the wire as UTF-8.
1711+ """
1712+
1713+ def toString(self, inObject):
1714+ # assert isinstance(inObject, unicode)
1715+ return String.toString(self, inObject.encode('utf-8'))
1716+
1717+ def fromString(self, inString):
1718+ # assert isinstance(inString, bytes)
1719+ return String.fromString(self, inString).decode('utf-8')
1720+
1721+
1722+class Path(Unicode):
1723+ """
1724+ Encode and decode L{filepath.FilePath} instances as paths on the wire.
1725+
1726+ This is really intended for use with subprocess communication tools:
1727+ exchanging pathnames on different machines over a network is not generally
1728+ meaningful, but neither is it disallowed; you can use this to communicate
1729+ about NFS paths, for example.
1730+ """
1731+
1732+ def fromString(self, inString):
1733+ return filepath.FilePath(Unicode.fromString(self, inString))
1734+
1735+ def toString(self, inObject):
1736+ return Unicode.toString(self, inObject.path)
1737+
1738+
1739+class ListOf(Argument):
1740+ """
1741+ Encode and decode lists of instances of a single other argument type.
1742+
1743+ For example, if you want to pass::
1744+
1745+ [3, 7, 9, 15]
1746+
1747+ You can create an argument like this::
1748+
1749+ ListOf(Integer())
1750+
1751+ The serialized form of the entire list is subject to the limit imposed by
1752+ L{MAX_VALUE_LENGTH}. List elements are represented as 32-bit length
1753+ prefixed strings. The argument type passed to the L{ListOf} initializer is
1754+ responsible for producing the serialized form of each element.
1755+
1756+ @ivar elementType: The L{Argument} instance used to encode and decode list
1757+ elements (note, not an arbitrary L{IArgument} implementation:
1758+ arguments must be implemented using only the C{fromString} and
1759+ C{toString} methods, not the C{fromBox} and C{toBox} methods).
1760+
1761+ @param optional: a boolean indicating whether this argument can be
1762+ omitted in the protocol.
1763+
1764+ @since: 10.0
1765+ """
1766+
1767+ def __init__(self, elementType, optional=False):
1768+ self.elementType = elementType
1769+ Argument.__init__(self, optional)
1770+
1771+ def fromString(self, inString):
1772+ """
1773+ Convert the serialized form of a list of instances of some type back
1774+ into that list.
1775+ """
1776+ strings = []
1777+ parser = Int32StringReceiver()
1778+ parser.MAX_LENGTH = MAX_VALUE_LENGTH
1779+ parser.stringReceived = strings.append
1780+ parser.dataReceived(inString)
1781+ return map(self.elementType.fromString, strings)
1782+
1783+ def toString(self, inObject):
1784+ """
1785+ Serialize the given list of objects to a single string.
1786+ """
1787+ strings = []
1788+ for obj in inObject:
1789+ serialized = self.elementType.toString(obj)
1790+ strings.append(pack('!I', len(serialized)))
1791+ strings.append(serialized)
1792+ return ''.join(strings)
1793+
1794+
1795+class AmpList(Argument):
1796+ """
1797+ Convert a list of dictionaries into a list of AMP boxes on the wire.
1798+
1799+ For example, if you want to pass::
1800+
1801+ [{'a': 7, 'b': u'hello'}, {'a': 9, 'b': u'goodbye'}]
1802+
1803+ You might use an AmpList like this in your arguments or response list::
1804+
1805+ AmpList([('a', Integer()),
1806+ ('b', Unicode())])
1807+ """
1808+
1809+ def __init__(self, subargs, optional=False):
1810+ """
1811+ Create an AmpList.
1812+
1813+ @param subargs: a list of 2-tuples of ('name', argument) describing the
1814+ schema of the dictionaries in the sequence of amp boxes.
1815+
1816+ @param optional: a boolean indicating whether this argument can be
1817+ omitted in the protocol.
1818+ """
1819+ self.subargs = subargs
1820+ Argument.__init__(self, optional)
1821+
1822+ def fromStringProto(self, inString, proto):
1823+ boxes = parseString(inString)
1824+ values = [_stringsToObjects(box, self.subargs, proto)
1825+ for box in boxes]
1826+ return values
1827+
1828+ def toStringProto(self, inObject, proto):
1829+ return ''.join(
1830+ _objectsToStrings(objects, self.subargs, Box(), proto).serialize()
1831+ for objects in inObject)
1832+
1833+
1834+class Descriptor(Integer):
1835+ """
1836+ Encode and decode file descriptors for exchange over a UNIX domain socket.
1837+
1838+ This argument type requires an AMP connection set up over an
1839+ L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>} provider (for
1840+ example, the kind of connection created by
1841+ L{IReactorUNIX.connectUNIX<
1842+ twisted.internet.interfaces.IReactorUNIX.connectUNIX>}
1843+ and L{UNIXClientEndpoint<twisted.internet.endpoints.UNIXClientEndpoint>}).
1844+
1845+ There is no correspondence between the integer value of the file
1846+ descriptor on the sending and receiving sides, therefore an alternate
1847+ approach is taken to matching up received descriptors with particular
1848+ L{Descriptor} parameters. The argument is encoded to an ordinal (unique
1849+ per connection) for inclusion in the AMP command or response box. The
1850+ descriptor itself is sent using
1851+ L{IUNIXTransport.sendFileDescriptor<
1852+ twisted.internet.interfaces.IUNIXTransport.sendFileDescriptor>}.
1853+ The receiver uses the order in which file descriptors are received and the
1854+ ordinal value to come up with the received copy of the descriptor.
1855+ """
1856+
1857+ def fromStringProto(self, inString, proto):
1858+ """
1859+ Take a unique identifier associated with a file descriptor which must
1860+ have been received by now and use it to look up that descriptor in a
1861+ dictionary where they are kept.
1862+
1863+ @param inString: The base representation (as a byte string) of an
1864+ ordinal indicating which file descriptor corresponds to this usage
1865+ of this argument.
1866+ @type inString: C{bytes}
1867+
1868+ @param proto: The protocol used to receive this descriptor. This
1869+ protocol must be connected via a transport providing
1870+ L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
1871+ @type proto: L{BinaryBoxProtocol}
1872+
1873+ @return: The file descriptor represented by C{inString}.
1874+ @rtype: C{int}
1875+ """
1876+ return proto._getDescriptor(int(inString))
1877+
1878+ def toStringProto(self, inObject, proto):
1879+ """
1880+ Send C{inObject}, an integer file descriptor, over C{proto}'s
1881+ connection and return a unique identifier which will allow the
1882+ receiver to associate the file descriptor with this argument.
1883+
1884+ @param inObject: A file descriptor to duplicate over an AMP connection
1885+ as the value for this argument.
1886+ @type inObject: C{int}
1887+
1888+ @param proto: The protocol which will be used to send this descriptor.
1889+ This protocol must be connected via a transport providing
1890+ L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
1891+
1892+ @return: A byte string which can be used by the receiver to reconstruct
1893+ the file descriptor.
1894+ @type: C{bytes}
1895+ """
1896+ identifier = proto._sendFileDescriptor(inObject)
1897+ outString = Integer.toStringProto(self, identifier, proto)
1898+ return outString
1899+
1900+
1901+class Command:
1902+ """
1903+ Subclass me to specify an AMP Command.
1904+
1905+ @cvar arguments: A list of 2-tuples of (name, Argument-subclass-instance),
1906+ specifying the names and values of the parameters which are required for
1907+ this command.
1908+
1909+ @cvar response: A list like L{arguments}, but instead used for the return
1910+ value.
1911+
1912+ @cvar errors: A mapping of subclasses of L{Exception} to wire-protocol tags
1913+ for errors represented as L{bytes}s. Responders which raise keys from this
1914+ dictionary will have the error translated to the corresponding tag on the
1915+ wire. Invokers which receive Deferreds from invoking this command with
1916+ L{AMP.callRemote} will potentially receive Failures with keys from this
1917+ mapping as their value. This mapping is inherited; if you declare a
1918+ command which handles C{FooError} as 'FOO_ERROR', then subclass it and
1919+ specify C{BarError} as 'BAR_ERROR', responders to the subclass may raise
1920+ either C{FooError} or C{BarError}, and invokers must be able to deal with
1921+ either of those exceptions.
1922+
1923+ @cvar fatalErrors: like 'errors', but errors in this list will always
1924+ terminate the connection, despite being of a recognizable error type.
1925+
1926+ @cvar commandType: The type of Box used to issue commands; useful only for
1927+ protocol-modifying behavior like startTLS or protocol switching. Defaults
1928+ to a plain vanilla L{Box}.
1929+
1930+ @cvar responseType: The type of Box used to respond to this command; only
1931+ useful for protocol-modifying behavior like startTLS or protocol switching.
1932+ Defaults to a plain vanilla L{Box}.
1933+
1934+ @ivar requiresAnswer: a boolean; defaults to True. Set it to False on your
1935+ subclass if you want callRemote to return None. Note: this is a hint only
1936+ to the client side of the protocol. The return-type of a command responder
1937+ method must always be a dictionary adhering to the contract specified by
1938+ L{response}, because clients are always free to request a response if they
1939+ want one.
1940+ """
1941+
1942+ class __metaclass__(type):
1943+ """
1944+ Metaclass hack to establish reverse-mappings for 'errors' and
1945+ 'fatalErrors' as class vars.
1946+ """
1947+ def __new__(cls, name, bases, attrs):
1948+ reverseErrors = attrs['reverseErrors'] = {}
1949+ er = attrs['allErrors'] = {}
1950+ if 'commandName' not in attrs:
1951+ attrs['commandName'] = name
1952+ newtype = type.__new__(cls, name, bases, attrs)
1953+ errors = {}
1954+ fatalErrors = {}
1955+ accumulateClassDict(newtype, 'errors', errors)
1956+ accumulateClassDict(newtype, 'fatalErrors', fatalErrors)
1957+ for v, k in errors.viewitems():
1958+ reverseErrors[k] = v
1959+ er[v] = k
1960+ for v, k in fatalErrors.viewitems():
1961+ reverseErrors[k] = v
1962+ er[v] = k
1963+ return newtype
1964+
1965+ arguments = []
1966+ response = []
1967+ extra = []
1968+ errors = {}
1969+ fatalErrors = {}
1970+
1971+ commandType = Box
1972+ responseType = Box
1973+
1974+ requiresAnswer = True
1975+
1976+ def __init__(self, **kw):
1977+ """
1978+ Create an instance of this command with specified values for its
1979+ parameters.
1980+
1981+ @param kw: a dict containing an appropriate value for each name
1982+ specified in the L{arguments} attribute of my class.
1983+
1984+ @raise InvalidSignature: if you forgot any required arguments.
1985+ """
1986+ self.structured = kw
1987+ givenArgs = set(kw)
1988+ forgotten = []
1989+ for name, arg in self.arguments:
1990+ pythonName = _wireNameToPythonIdentifier(name)
1991+ if pythonName not in givenArgs and not arg.optional:
1992+ forgotten.append(pythonName)
1993+ if forgotten:
1994+ raise InvalidSignature("forgot %s for %s" % (
1995+ ', '.join(forgotten), self.commandName))
1996+ forgotten = []
1997+
1998+ @classmethod
1999+ def makeResponse(cls, objects, proto):
2000+ """
2001+ Serialize a mapping of arguments using this L{Command}'s
2002+ response schema.
2003+
2004+ @param objects: a dict with keys matching the names specified in
2005+ self.response, having values of the types that the Argument objects in
2006+ self.response can format.
2007+
2008+ @param proto: an L{AMP}.
2009+
2010+ @return: an L{AmpBox}.
2011+ """
2012+ try:
2013+ responseType = cls.responseType()
2014+ except:
2015+ return fail()
2016+ return _objectsToStrings(objects, cls.response, responseType, proto)
2017+
2018+ @classmethod
2019+ def makeArguments(cls, objects, proto):
2020+ """
2021+ Serialize a mapping of arguments using this L{Command}'s
2022+ argument schema.
2023+
2024+ @param objects: a dict with keys similar to the names specified in
2025+ self.arguments, having values of the types that the Argument objects in
2026+ self.arguments can parse.
2027+
2028+ @param proto: an L{AMP}.
2029+
2030+ @return: An instance of this L{Command}'s C{commandType}.
2031+ """
2032+ allowedNames = set()
2033+ for (argName, ignored) in cls.arguments:
2034+ allowedNames.add(_wireNameToPythonIdentifier(argName))
2035+
2036+ for intendedArg in objects:
2037+ if intendedArg not in allowedNames:
2038+ raise InvalidSignature(
2039+ "%s is not a valid argument" % (intendedArg,))
2040+ return _objectsToStrings(objects, cls.arguments, cls.commandType(),
2041+ proto)
2042+
2043+ @classmethod
2044+ def parseResponse(cls, box, protocol):
2045+ """
2046+ Parse a mapping of serialized arguments using this
2047+ L{Command}'s response schema.
2048+
2049+ @param box: A mapping of response-argument names to the
2050+ serialized forms of those arguments.
2051+ @param protocol: The L{AMP} protocol.
2052+
2053+ @return: A mapping of response-argument names to the parsed
2054+ forms.
2055+ """
2056+ return _stringsToObjects(box, cls.response, protocol)
2057+
2058+ @classmethod
2059+ def parseArguments(cls, box, protocol):
2060+ """
2061+ Parse a mapping of serialized arguments using this
2062+ L{Command}'s argument schema.
2063+
2064+ @param box: A mapping of argument names to the seralized forms
2065+ of those arguments.
2066+ @param protocol: The L{AMP} protocol.
2067+
2068+ @return: A mapping of argument names to the parsed forms.
2069+ """
2070+ return _stringsToObjects(box, cls.arguments, protocol)
2071+
2072+ @classmethod
2073+ def responder(cls, methodfunc):
2074+ """
2075+ Declare a method to be a responder for a particular command.
2076+
2077+ This is a decorator.
2078+
2079+ Use like so::
2080+
2081+ class MyCommand(Command):
2082+ arguments = [('a', ...), ('b', ...)]
2083+
2084+ class MyProto(AMP):
2085+ def myFunMethod(self, a, b):
2086+ ...
2087+ MyCommand.responder(myFunMethod)
2088+
2089+ Notes: Although decorator syntax is not used within Twisted, this
2090+ function returns its argument and is therefore safe to use with
2091+ decorator syntax.
2092+
2093+ This is not thread safe. Don't declare AMP subclasses in other
2094+ threads. Don't declare responders outside the scope of AMP subclasses;
2095+ the behavior is undefined.
2096+
2097+ @param methodfunc: A function which will later become a method, which
2098+ has a keyword signature compatible with this command's L{argument} list
2099+ and returns a dictionary with a set of keys compatible with this
2100+ command's L{response} list.
2101+
2102+ @return: the methodfunc parameter.
2103+ """
2104+ CommandLocator._currentClassCommands.append((cls, methodfunc))
2105+ return methodfunc
2106+
2107+ # Our only instance method
2108+ def _doCommand(self, proto):
2109+ """
2110+ Encode and send this Command to the given protocol.
2111+
2112+ @param proto: an AMP, representing the connection to send to.
2113+
2114+ @return: a Deferred which will fire or error appropriately when the
2115+ other side responds to the command (or error if the connection is lost
2116+ before it is responded to).
2117+ """
2118+
2119+ def _massageError(error):
2120+ error.trap(RemoteAmpError)
2121+ rje = error.value
2122+ errorType = self.reverseErrors.get(rje.errorCode,
2123+ UnknownRemoteError)
2124+ return Failure(errorType(rje.description))
2125+
2126+ d = proto._sendBoxCommand(self.commandName,
2127+ self.makeArguments(self.structured, proto),
2128+ self.requiresAnswer)
2129+
2130+ if self.requiresAnswer:
2131+ d.addCallback(self.parseResponse, proto)
2132+ d.addErrback(_massageError)
2133+
2134+ return d
2135+
2136+
2137+class _NoCertificate:
2138+ """
2139+ This is for peers which don't want to use a local certificate. Used by
2140+ AMP because AMP's internal language is all about certificates and this
2141+ duck-types in the appropriate place; this API isn't really stable though,
2142+ so it's not exposed anywhere public.
2143+
2144+ For clients, it will use ephemeral DH keys, or whatever the default is for
2145+ certificate-less clients in OpenSSL. For servers, it will generate a
2146+ temporary self-signed certificate with garbage values in the DN and use
2147+ that.
2148+ """
2149+
2150+ def __init__(self, client):
2151+ """
2152+ Create a _NoCertificate which either is or isn't for the client side of
2153+ the connection.
2154+
2155+ @param client: True if we are a client and should truly have no
2156+ certificate and be anonymous, False if we are a server and actually
2157+ have to generate a temporary certificate.
2158+
2159+ @type client: bool
2160+ """
2161+ self.client = client
2162+
2163+ def options(self, *authorities):
2164+ """
2165+ Behaves like L{twisted.internet.ssl.PrivateCertificate.options}().
2166+ """
2167+ if not self.client:
2168+ # do some crud with sslverify to generate a temporary self-signed
2169+ # certificate. This is SLOOOWWWWW so it is only in the absolute
2170+ # worst, most naive case.
2171+
2172+ # We have to do this because OpenSSL will not let both the server
2173+ # and client be anonymous.
2174+ sharedDN = DN(CN='TEMPORARY CERTIFICATE')
2175+ key = KeyPair.generate()
2176+ cr = key.certificateRequest(sharedDN)
2177+ sscrd = key.signCertificateRequest(
2178+ sharedDN, cr, lambda dn: True, 1)
2179+ cert = key.newCertificate(sscrd)
2180+ return cert.options(*authorities)
2181+ options = dict()
2182+ if authorities:
2183+ options.update(
2184+ verify=True, requireCertificate=True,
2185+ caCerts=[auth.original for auth in authorities])
2186+ occo = CertificateOptions(**options)
2187+ return occo
2188+
2189+
2190+class _TLSBox(AmpBox):
2191+ """
2192+ I am an AmpBox that, upon being sent, initiates a TLS connection.
2193+ """
2194+
2195+ __slots__ = []
2196+
2197+ def __init__(self):
2198+ if ssl is None:
2199+ raise RemoteAmpError("TLS_ERROR", "TLS not available")
2200+ AmpBox.__init__(self)
2201+
2202+ def _keyprop(k, default):
2203+ return property(lambda self: self.get(k, default))
2204+
2205+ # These properties are described in startTLS
2206+ certificate = _keyprop('tls_localCertificate', _NoCertificate(False))
2207+ verify = _keyprop('tls_verifyAuthorities', None)
2208+
2209+ def _sendTo(self, proto):
2210+ """
2211+ Send my encoded value to the protocol, then initiate TLS.
2212+ """
2213+ ab = AmpBox(self)
2214+ for k in ['tls_localCertificate',
2215+ 'tls_verifyAuthorities']:
2216+ ab.pop(k, None)
2217+ ab._sendTo(proto)
2218+ proto._startTLS(self.certificate, self.verify)
2219+
2220+
2221+class _LocalArgument(String):
2222+ """
2223+ Local arguments are never actually relayed across the wire. This is just a
2224+ shim so that StartTLS can pretend to have some arguments: if arguments
2225+ acquire documentation properties, replace this with something nicer later.
2226+ """
2227+
2228+ def fromBox(self, name, strings, objects, proto):
2229+ pass
2230+
2231+
2232+class StartTLS(Command):
2233+ """
2234+ Use, or subclass, me to implement a command that starts TLS.
2235+
2236+ Callers of StartTLS may pass several special arguments, which affect the
2237+ TLS negotiation:
2238+
2239+ - tls_localCertificate: This is a
2240+ twisted.internet.ssl.PrivateCertificate which will be used to secure
2241+ the side of the connection it is returned on.
2242+
2243+ - tls_verifyAuthorities: This is a list of
2244+ twisted.internet.ssl.Certificate objects that will be used as the
2245+ certificate authorities to verify our peer's certificate.
2246+
2247+ Each of those special parameters may also be present as a key in the
2248+ response dictionary.
2249+ """
2250+
2251+ arguments = [("tls_localCertificate", _LocalArgument(optional=True)),
2252+ ("tls_verifyAuthorities", _LocalArgument(optional=True))]
2253+
2254+ response = [("tls_localCertificate", _LocalArgument(optional=True)),
2255+ ("tls_verifyAuthorities", _LocalArgument(optional=True))]
2256+
2257+ responseType = _TLSBox
2258+
2259+ def __init__(self, **kw):
2260+ """
2261+ Create a StartTLS command. (This is private. Use AMP.callRemote.)
2262+
2263+ @param tls_localCertificate: the PrivateCertificate object to use to
2264+ secure the connection. If it's None, or unspecified, an ephemeral DH
2265+ key is used instead.
2266+
2267+ @param tls_verifyAuthorities: a list of Certificate objects which
2268+ represent root certificates to verify our peer with.
2269+ """
2270+ if ssl is None:
2271+ raise RuntimeError("TLS not available.")
2272+ self.certificate = kw.pop('tls_localCertificate', _NoCertificate(True))
2273+ self.authorities = kw.pop('tls_verifyAuthorities', None)
2274+ Command.__init__(self, **kw)
2275+
2276+ def _doCommand(self, proto):
2277+ """
2278+ When a StartTLS command is sent, prepare to start TLS, but don't
2279+ actually do it; wait for the acknowledgement, then initiate the TLS
2280+ handshake.
2281+ """
2282+ d = Command._doCommand(self, proto)
2283+ proto._prepareTLS(self.certificate, self.authorities)
2284+
2285+ # XXX before we get back to user code we are going to start TLS...
2286+ def actuallystart(response):
2287+ proto._startTLS(self.certificate, self.authorities)
2288+ return response
2289+ d.addCallback(actuallystart)
2290+
2291+ return d
2292+
2293+
2294+class ProtocolSwitchCommand(Command):
2295+ """
2296+ Use this command to switch from something Amp-derived to a different
2297+ protocol mid-connection. This can be useful to use amp as the
2298+ connection-startup negotiation phase. Since TLS is a different layer
2299+ entirely, you can use Amp to negotiate the security parameters of your
2300+ connection, then switch to a different protocol, and the connection will
2301+ remain secured.
2302+ """
2303+
2304+ def __init__(self, _protoToSwitchToFactory, **kw):
2305+ """
2306+ Create a ProtocolSwitchCommand.
2307+
2308+ @param _protoToSwitchToFactory: a ProtocolFactory which will generate
2309+ the Protocol to switch to.
2310+
2311+ @param kw: Keyword arguments, encoded and handled normally as
2312+ L{Command} would.
2313+ """
2314+
2315+ self.protoToSwitchToFactory = _protoToSwitchToFactory
2316+ super(ProtocolSwitchCommand, self).__init__(**kw)
2317+
2318+ @classmethod
2319+ def makeResponse(cls, innerProto, proto):
2320+ return _SwitchBox(innerProto)
2321+
2322+ def _doCommand(self, proto):
2323+ """
2324+ When we emit a ProtocolSwitchCommand, lock the protocol, but don't
2325+ actually switch to the new protocol unless an acknowledgement is
2326+ received. If an error is received, switch back.
2327+ """
2328+ d = super(ProtocolSwitchCommand, self)._doCommand(proto)
2329+ proto._lockForSwitch()
2330+
2331+ def switchNow(ign):
2332+ innerProto = self.protoToSwitchToFactory.buildProtocol(
2333+ proto.transport.getPeer())
2334+ proto._switchTo(innerProto, self.protoToSwitchToFactory)
2335+ return ign
2336+
2337+ def handle(ign):
2338+ proto._unlockFromSwitch()
2339+ self.protoToSwitchToFactory.clientConnectionFailed(
2340+ None, Failure(CONNECTION_LOST))
2341+ return ign
2342+
2343+ return d.addCallbacks(switchNow, handle)
2344+
2345+
2346+class _DescriptorExchanger(object):
2347+ """
2348+ L{_DescriptorExchanger} is a mixin for L{BinaryBoxProtocol} which adds
2349+ support for receiving file descriptors, a feature offered by
2350+ L{IUNIXTransport<twisted.internet.interfaces.IUNIXTransport>}.
2351+
2352+ @ivar _descriptors: Temporary storage for all file descriptors received.
2353+ Values in this dictionary are the file descriptors (as integers). Keys
2354+ in this dictionary are ordinals giving the order in which each
2355+ descriptor was received. The ordering information is used to allow
2356+ L{Descriptor} to determine which is the correct descriptor for any
2357+ particular usage of that argument type.
2358+ @type _descriptors: C{dict}
2359+
2360+ @ivar _sendingDescriptorCounter: A no-argument callable which returns the
2361+ ordinals, starting from 0. This is used to construct values for
2362+ C{_sendFileDescriptor}.
2363+
2364+ @ivar _receivingDescriptorCounter: A no-argument callable which returns the
2365+ ordinals, starting from 0. This is used to construct values for
2366+ C{fileDescriptorReceived}.
2367+ """
2368+
2369+ implements(IFileDescriptorReceiver)
2370+
2371+ def __init__(self):
2372+ self._descriptors = {}
2373+ self._getDescriptor = self._descriptors.pop
2374+ self._sendingDescriptorCounter = count()
2375+ self._receivingDescriptorCounter = count()
2376+
2377+ def _sendFileDescriptor(self, descriptor):
2378+ """
2379+ Assign and return the next ordinal to the given descriptor after
2380+ sending the descriptor over this protocol's transport.
2381+ """
2382+ self.transport.sendFileDescriptor(descriptor)
2383+ return next(self._sendingDescriptorCounter)
2384+
2385+ def fileDescriptorReceived(self, descriptor):
2386+ """
2387+ Collect received file descriptors to be claimed later by L{Descriptor}.
2388+
2389+ @param descriptor: The received file descriptor.
2390+ @type descriptor: C{int}
2391+ """
2392+ self._descriptors[next(self._receivingDescriptorCounter)] = descriptor
2393+
2394+
2395+class BinaryBoxProtocol(StatefulStringProtocol, Int32StringReceiver,
2396+ _DescriptorExchanger):
2397+ """
2398+ A protocol for receiving L{AmpBox}es - key/value pairs - via
2399+ length-prefixed strings. A box is composed of:
2400+
2401+ - any number of key-value pairs, described by:
2402+ - a 4-byte network-endian packed key length (of which the first
2403+ 3 bytes must be null, and the last must be non-null: i.e. the
2404+ value of the length must be 1-255)
2405+ - a key, comprised of that many bytes
2406+ - a 4-byte network-endian unsigned value length (up to the maximum
2407+ of C{MAX_VALUE_LENGTH})
2408+ - a value, comprised of that many bytes
2409+ - 4 null bytes
2410+
2411+ In other words, an even number of strings prefixed with packed unsigned
2412+ 32-bit integers, and then a 0-length string to indicate the end of the box.
2413+
2414+ This protocol also implements 2 extra private bits of functionality related
2415+ to the byte boundaries between messages; it can start TLS between two given
2416+ boxes or switch to an entirely different protocol. However, due to some
2417+ tricky elements of the implementation, the public interface to this
2418+ functionality is L{ProtocolSwitchCommand} and L{StartTLS}.
2419+
2420+ @ivar _keyLengthLimitExceeded: A flag which is only true when the
2421+ connection is being closed because a key length prefix which was longer
2422+ than allowed by the protocol was received.
2423+
2424+ @ivar boxReceiver: an L{IBoxReceiver} provider, whose L{ampBoxReceived}
2425+ method will be invoked for each L{AmpBox} that is received.
2426+ """
2427+
2428+ implements(IBoxSender)
2429+
2430+ _justStartedTLS = False
2431+ _startingTLSBuffer = None
2432+ _locked = False
2433+ _currentKey = None
2434+ _currentBox = None
2435+
2436+ _keyLengthLimitExceeded = False
2437+
2438+ hostCertificate = None
2439+ noPeerCertificate = False # for tests
2440+ innerProtocol = None
2441+ innerProtocolClientFactory = None
2442+
2443+ def __init__(self, boxReceiver):
2444+ _DescriptorExchanger.__init__(self)
2445+ self.boxReceiver = boxReceiver
2446+
2447+ def _switchTo(self, newProto, clientFactory=None):
2448+ """
2449+ Switch this BinaryBoxProtocol's transport to a new protocol. You need
2450+ to do this 'simultaneously' on both ends of a connection; the easiest
2451+ way to do this is to use a subclass of ProtocolSwitchCommand.
2452+
2453+ @param newProto: the new protocol instance to switch to.
2454+
2455+ @param clientFactory: the ClientFactory to send the
2456+ L{clientConnectionLost} notification to.
2457+ """
2458+ # All the data that Int32StringReceiver has not yet dealt with belongs
2459+ # to our new protocol: luckily it's keeping that in a handy (although
2460+ # ostensibly internal) variable for us:
2461+ newProtoData = self.recvd
2462+ # We're quite possibly in the middle of a 'dataReceived' loop in
2463+ # Int32StringReceiver: let's make sure that the next iteration, the
2464+ # loop will break and not attempt to look at something that isn't a
2465+ # length prefix.
2466+ self.recvd = ''
2467+ # Finally, do the actual work of setting up the protocol and delivering
2468+ # its first chunk of data, if one is available.
2469+ self.innerProtocol = newProto
2470+ self.innerProtocolClientFactory = clientFactory
2471+ newProto.makeConnection(self.transport)
2472+ if newProtoData:
2473+ newProto.dataReceived(newProtoData)
2474+
2475+ def sendBox(self, box):
2476+ """
2477+ Send a amp32.Box to my peer.
2478+
2479+ Note: transport.write is never called outside of this method.
2480+
2481+ @param box: an AmpBox.
2482+
2483+ @raise ProtocolSwitched: if the protocol has previously been switched.
2484+
2485+ @raise ConnectionLost: if the connection has previously been lost.
2486+ """
2487+ if self._locked:
2488+ raise ProtocolSwitched(
2489+ "This connection has switched: no AMP traffic allowed.")
2490+ if self.transport is None:
2491+ raise ConnectionLost()
2492+ if self._startingTLSBuffer is not None:
2493+ self._startingTLSBuffer.append(box)
2494+ else:
2495+ self.transport.write(box.serialize())
2496+
2497+ def makeConnection(self, transport):
2498+ """
2499+ Notify L{boxReceiver} that it is about to receive boxes from this
2500+ protocol by invoking L{startReceivingBoxes}.
2501+ """
2502+ self.transport = transport
2503+ self.boxReceiver.startReceivingBoxes(self)
2504+ self.connectionMade()
2505+
2506+ def dataReceived(self, data):
2507+ """
2508+ Either parse incoming data as L{AmpBox}es or relay it to our nested
2509+ protocol.
2510+ """
2511+ if self._justStartedTLS:
2512+ self._justStartedTLS = False
2513+ # If we already have an inner protocol, then we don't deliver data to
2514+ # the protocol parser any more; we just hand it off.
2515+ if self.innerProtocol is not None:
2516+ self.innerProtocol.dataReceived(data)
2517+ return
2518+ return Int32StringReceiver.dataReceived(self, data)
2519+
2520+ def connectionLost(self, reason):
2521+ """
2522+ The connection was lost; notify any nested protocol.
2523+ """
2524+ if self.innerProtocol is not None:
2525+ self.innerProtocol.connectionLost(reason)
2526+ if self.innerProtocolClientFactory is not None:
2527+ self.innerProtocolClientFactory.clientConnectionLost(
2528+ None, reason)
2529+ if self._keyLengthLimitExceeded:
2530+ failReason = Failure(TooLong(True, False, None, None))
2531+ elif reason.check(ConnectionClosed) and self._justStartedTLS:
2532+ # We just started TLS and haven't received any data. This means
2533+ # the other connection didn't like our cert (although they may not
2534+ # have told us why - later Twisted should make 'reason' into a TLS
2535+ # error.)
2536+ failReason = PeerVerifyError(
2537+ "Peer rejected our certificate for an unknown reason.")
2538+ else:
2539+ failReason = reason
2540+ self.boxReceiver.stopReceivingBoxes(failReason)
2541+
2542+ # The longest key allowed
2543+ _MAX_KEY_LENGTH = MAX_KEY_LENGTH
2544+
2545+ # The longest value allowed (this is somewhat redundant, as longer values
2546+ # cannot be encoded - ah well).
2547+ _MAX_VALUE_LENGTH = MAX_VALUE_LENGTH
2548+
2549+ # The first thing received is a key.
2550+ MAX_LENGTH = _MAX_KEY_LENGTH
2551+
2552+ def proto_init(self, string):
2553+ """
2554+ String received in the 'init' state.
2555+ """
2556+ self._currentBox = AmpBox()
2557+ return self.proto_key(string)
2558+
2559+ def proto_key(self, string):
2560+ """
2561+ String received in the 'key' state. If the key is empty, a complete
2562+ box has been received.
2563+ """
2564+ if string:
2565+ self._currentKey = string
2566+ self.MAX_LENGTH = self._MAX_VALUE_LENGTH
2567+ return 'value'
2568+ else:
2569+ self.boxReceiver.ampBoxReceived(self._currentBox)
2570+ self._currentBox = None
2571+ return 'init'
2572+
2573+ def proto_value(self, string):
2574+ """
2575+ String received in the 'value' state.
2576+ """
2577+ self._currentBox[self._currentKey] = string
2578+ self._currentKey = None
2579+ self.MAX_LENGTH = self._MAX_KEY_LENGTH
2580+ return 'key'
2581+
2582+ def lengthLimitExceeded(self, length):
2583+ """
2584+ The key length limit was exceeded. Disconnect the transport and make
2585+ sure a meaningful exception is reported.
2586+ """
2587+ self._keyLengthLimitExceeded = True
2588+ self.transport.loseConnection()
2589+
2590+ def _lockForSwitch(self):
2591+ """
2592+ Lock this binary protocol so that no further boxes may be sent. This
2593+ is used when sending a request to switch underlying protocols. You
2594+ probably want to subclass ProtocolSwitchCommand rather than calling
2595+ this directly.
2596+ """
2597+ self._locked = True
2598+
2599+ def _unlockFromSwitch(self):
2600+ """
2601+ Unlock this locked binary protocol so that further boxes may be sent
2602+ again. This is used after an attempt to switch protocols has failed
2603+ for some reason.
2604+ """
2605+ if self.innerProtocol is not None:
2606+ raise ProtocolSwitched(
2607+ "Protocol already switched. Cannot unlock.")
2608+ self._locked = False
2609+
2610+ def _prepareTLS(self, certificate, verifyAuthorities):
2611+ """
2612+ Used by StartTLSCommand to put us into the state where we don't
2613+ actually send things that get sent, instead we buffer them. see
2614+ L{_sendBox}.
2615+ """
2616+ self._startingTLSBuffer = []
2617+ if self.hostCertificate is not None:
2618+ raise OnlyOneTLS(
2619+ "Previously authenticated connection between %s and %s "
2620+ "is trying to re-establish as %s" % (
2621+ self.hostCertificate,
2622+ self.peerCertificate,
2623+ (certificate, verifyAuthorities)))
2624+
2625+ def _startTLS(self, certificate, verifyAuthorities):
2626+ """
2627+ Used by TLSBox to initiate the SSL handshake.
2628+
2629+ @param certificate: a L{twisted.internet.ssl.PrivateCertificate} for
2630+ use locally.
2631+
2632+ @param verifyAuthorities: L{twisted.internet.ssl.Certificate} instances
2633+ representing certificate authorities which will verify our peer.
2634+ """
2635+ self.hostCertificate = certificate
2636+ self._justStartedTLS = True
2637+ if verifyAuthorities is None:
2638+ verifyAuthorities = ()
2639+ self.transport.startTLS(certificate.options(*verifyAuthorities))
2640+ stlsb = self._startingTLSBuffer
2641+ if stlsb is not None:
2642+ self._startingTLSBuffer = None
2643+ for box in stlsb:
2644+ self.sendBox(box)
2645+
2646+ def _getPeerCertificate(self):
2647+ if self.noPeerCertificate:
2648+ return None
2649+ return Certificate.peerFromTransport(self.transport)
2650+ peerCertificate = property(_getPeerCertificate)
2651+
2652+ def unhandledError(self, failure):
2653+ """
2654+ The buck stops here. This error was completely unhandled, time to
2655+ terminate the connection.
2656+ """
2657+ log.err(
2658+ failure,
2659+ "Amp server or network failure unhandled by client application. "
2660+ "Dropping connection! To avoid, add errbacks to ALL remote "
2661+ "commands!")
2662+ if self.transport is not None:
2663+ self.transport.loseConnection()
2664+
2665+ def _defaultStartTLSResponder(self):
2666+ """
2667+ The default TLS responder doesn't specify any certificate or anything.
2668+
2669+ From a security perspective, it's little better than a plain-text
2670+ connection - but it is still a *bit* better, so it's included for
2671+ convenience.
2672+
2673+ You probably want to override this by providing your own
2674+ C{StartTLS.responder}.
2675+ """
2676+ return {}
2677+ StartTLS.responder(_defaultStartTLSResponder)
2678+
2679+
2680+class AMP(BinaryBoxProtocol, BoxDispatcher,
2681+ CommandLocator, SimpleStringLocator):
2682+ """
2683+ This protocol is an AMP connection. See the module docstring for protocol
2684+ details.
2685+ """
2686+
2687+ _ampInitialized = False
2688+
2689+ def __init__(self, boxReceiver=None, locator=None):
2690+ # For backwards compatibility. When AMP did not separate parsing logic
2691+ # (L{BinaryBoxProtocol}), request-response logic (L{BoxDispatcher}) and
2692+ # command routing (L{CommandLocator}), it did not have a constructor.
2693+ # Now it does, so old subclasses might have defined their own that did
2694+ # not upcall. If this flag isn't set, we'll call the constructor in
2695+ # makeConnection before anything actually happens.
2696+ self._ampInitialized = True
2697+ if boxReceiver is None:
2698+ boxReceiver = self
2699+ if locator is None:
2700+ locator = self
2701+ BoxDispatcher.__init__(self, locator)
2702+ BinaryBoxProtocol.__init__(self, boxReceiver)
2703+
2704+ def locateResponder(self, name):
2705+ """
2706+ Unify the implementations of L{CommandLocator} and
2707+ L{SimpleStringLocator} to perform both kinds of dispatch, preferring
2708+ L{CommandLocator}.
2709+ """
2710+ firstResponder = CommandLocator.locateResponder(self, name)
2711+ if firstResponder is not None:
2712+ return firstResponder
2713+ secondResponder = SimpleStringLocator.locateResponder(self, name)
2714+ return secondResponder
2715+
2716+ def __repr__(self):
2717+ """
2718+ A verbose string representation which gives us information about this
2719+ AMP connection.
2720+ """
2721+ if self.innerProtocol is not None:
2722+ innerRepr = ' inner %r' % (self.innerProtocol,)
2723+ else:
2724+ innerRepr = ''
2725+ return '<%s%s at 0x%x>' % (
2726+ self.__class__.__name__, innerRepr, id(self))
2727+
2728+ def makeConnection(self, transport):
2729+ """
2730+ Emit a helpful log message when the connection is made.
2731+ """
2732+ if not self._ampInitialized:
2733+ # See comment in the constructor re: backward compatibility. I
2734+ # should probably emit a deprecation warning here.
2735+ AMP.__init__(self)
2736+ # Save these so we can emit a similar log message in L{connectionLost}.
2737+ self._transportPeer = transport.getPeer()
2738+ self._transportHost = transport.getHost()
2739+ log.msg("%s connection established (HOST:%s PEER:%s)" % (
2740+ self.__class__.__name__,
2741+ self._transportHost,
2742+ self._transportPeer))
2743+ BinaryBoxProtocol.makeConnection(self, transport)
2744+
2745+ def connectionLost(self, reason):
2746+ """
2747+ Emit a helpful log message when the connection is lost.
2748+ """
2749+ log.msg("%s connection lost (HOST:%s PEER:%s)" %
2750+ (self.__class__.__name__,
2751+ self._transportHost,
2752+ self._transportPeer))
2753+ BinaryBoxProtocol.connectionLost(self, reason)
2754+ self.transport = None
2755+
2756+
2757+class _ParserHelper:
2758+ """
2759+ A box receiver which records all boxes received.
2760+ """
2761+
2762+ def __init__(self):
2763+ self.boxes = []
2764+
2765+ def getPeer(self):
2766+ return 'string'
2767+
2768+ def getHost(self):
2769+ return 'string'
2770+
2771+ disconnecting = False
2772+
2773+ def startReceivingBoxes(self, sender):
2774+ """
2775+ No initialization is required.
2776+ """
2777+
2778+ def ampBoxReceived(self, box):
2779+ self.boxes.append(box)
2780+
2781+ # Synchronous helpers
2782+ @classmethod
2783+ def parse(cls, fileObj):
2784+ """
2785+ Parse some amp data stored in a file.
2786+
2787+ @param fileObj: a file-like object.
2788+
2789+ @return: a list of AmpBoxes encoded in the given file.
2790+ """
2791+ parserHelper = cls()
2792+ bbp = BinaryBoxProtocol(boxReceiver=parserHelper)
2793+ bbp.makeConnection(parserHelper)
2794+ bbp.dataReceived(fileObj.read())
2795+ return parserHelper.boxes
2796+
2797+ @classmethod
2798+ def parseString(cls, data):
2799+ """
2800+ Parse some amp data stored in a string.
2801+
2802+ @param data: a bytes holding some amp-encoded data.
2803+
2804+ @return: a list of AmpBoxes encoded in the given string.
2805+ """
2806+ return cls.parse(BytesIO(data))
2807+
2808+
2809+parse = _ParserHelper.parse
2810+parseString = _ParserHelper.parseString
2811+
2812+
2813+def _stringsToObjects(strings, arglist, proto):
2814+ """
2815+ Convert an AmpBox to a dictionary of python objects, converting through a
2816+ given arglist.
2817+
2818+ @param strings: an AmpBox (or dict of strings)
2819+
2820+ @param arglist: a list of 2-tuples of strings and Argument objects, as
2821+ described in L{Command.arguments}.
2822+
2823+ @param proto: an L{AMP} instance.
2824+
2825+ @return: the converted dictionary mapping names to argument objects.
2826+ """
2827+ objects = {}
2828+ myStrings = strings.copy()
2829+ for argname, argparser in arglist:
2830+ argparser.fromBox(argname, myStrings, objects, proto)
2831+ return objects
2832+
2833+
2834+def _objectsToStrings(objects, arglist, strings, proto):
2835+ """
2836+ Convert a dictionary of python objects to an AmpBox, converting through a
2837+ given arglist.
2838+
2839+ @param objects: a dict mapping names to python objects
2840+
2841+ @param arglist: a list of 2-tuples of strings and Argument objects, as
2842+ described in L{Command.arguments}.
2843+
2844+ @param strings: [OUT PARAMETER] An object providing the L{dict}
2845+ interface which will be populated with serialized data.
2846+
2847+ @param proto: an L{AMP} instance.
2848+
2849+ @return: The converted dictionary mapping names to encoded argument
2850+ strings (identical to C{strings}).
2851+ """
2852+ myObjects = objects.copy()
2853+ for argname, argparser in arglist:
2854+ argparser.toBox(argname, strings, myObjects, proto)
2855+ return strings
2856+
2857+
2858+class _FixedOffsetTZInfo(datetime.tzinfo):
2859+ """
2860+ Represents a fixed timezone offset (without daylight saving time).
2861+
2862+ @ivar name: A C{bytes} giving the name of this timezone; the name just
2863+ includes how much time this offset represents.
2864+
2865+ @ivar offset: A C{datetime.timedelta} giving the amount of time this
2866+ timezone is offset.
2867+ """
2868+
2869+ def __init__(self, sign, hours, minutes):
2870+ self.name = '%s%02i:%02i' % (sign, hours, minutes)
2871+ if sign == '-':
2872+ hours = -hours
2873+ minutes = -minutes
2874+ elif sign != '+':
2875+ raise ValueError('invalid sign for timezone %r' % (sign,))
2876+ self.offset = datetime.timedelta(hours=hours, minutes=minutes)
2877+
2878+ def utcoffset(self, dt):
2879+ """
2880+ Return this timezone's offset from UTC.
2881+ """
2882+ return self.offset
2883+
2884+ def dst(self, dt):
2885+ """
2886+ Return a zero C{datetime.timedelta} for the daylight saving time
2887+ offset, since there is never one.
2888+ """
2889+ return datetime.timedelta(0)
2890+
2891+ def tzname(self, dt):
2892+ """
2893+ Return a string describing this timezone.
2894+ """
2895+ return self.name
2896+
2897+
2898+utc = _FixedOffsetTZInfo('+', 0, 0)
2899+
2900+
2901+class Decimal(Argument):
2902+ """
2903+ Encodes C{decimal.Decimal} instances.
2904+
2905+ There are several ways in which a decimal value might be encoded.
2906+
2907+ Special values are encoded as special strings::
2908+
2909+ - Positive infinity is encoded as C{"Infinity"}
2910+ - Negative infinity is encoded as C{"-Infinity"}
2911+ - Quiet not-a-number is encoded as either C{"NaN"} or C{"-NaN"}
2912+ - Signalling not-a-number is encoded as either C{"sNaN"} or C{"-sNaN"}
2913+
2914+ Normal values are encoded using the base ten string representation, using
2915+ engineering notation to indicate magnitude without precision, and "normal"
2916+ digits to indicate precision. For example::
2917+
2918+ - C{"1"} represents the value I{1} with precision to one place.
2919+ - C{"-1"} represents the value I{-1} with precision to one place.
2920+ - C{"1.0"} represents the value I{1} with precision to two places.
2921+ - C{"10"} represents the value I{10} with precision to two places.
2922+ - C{"1E+2"} represents the value I{10} with precision to one place.
2923+ - C{"1E-1"} represents the value I{0.1} with precision to one place.
2924+ - C{"1.5E+2"} represents the value I{15} with precision to two places.
2925+
2926+ U{http://speleotrove.com/decimal/} should be considered the authoritative
2927+ specification for the format.
2928+ """
2929+
2930+ fromString = decimal.Decimal
2931+
2932+ def toString(self, inObject):
2933+ """
2934+ Serialize a C{decimal.Decimal} instance to the specified wire format.
2935+ """
2936+ if isinstance(inObject, decimal.Decimal):
2937+ # Hopefully decimal.Decimal.__bytes__ actually does what we want.
2938+ return bytes(inObject)
2939+ raise ValueError(
2940+ "amp32.Decimal can only encode instances of decimal.Decimal")
2941+
2942+
2943+class DateTime(Argument):
2944+ """
2945+ Encodes C{datetime.datetime} instances.
2946+
2947+ Wire format: '%04i-%02i-%02iT%02i:%02i:%02i.%06i%s%02i:%02i'. Fields in
2948+ order are: year, month, day, hour, minute, second, microsecond, timezone
2949+ direction (+ or -), timezone hour, timezone minute. Encoded string is
2950+ always exactly 32 characters long. This format is compatible with ISO 8601,
2951+ but that does not mean all ISO 8601 dates can be accepted.
2952+
2953+ Also, note that the datetime module's notion of a "timezone" can be
2954+ complex, but the wire format includes only a fixed offset, so the
2955+ conversion is not lossless. A lossless transmission of a C{datetime}
2956+ instance is not feasible since the receiving end would require a Python
2957+ interpreter.
2958+
2959+ @ivar _positions: A sequence of slices giving the positions of various
2960+ interesting parts of the wire format.
2961+ """
2962+
2963+ _positions = [
2964+ slice(0, 4), slice(5, 7), slice(8, 10), # year, month, day
2965+ slice(11, 13), slice(14, 16), slice(17, 19), # hour, minute, second
2966+ slice(20, 26), # microsecond
2967+ # intentionally skip timezone direction, as it is not an integer
2968+ slice(27, 29), slice(30, 32) # timezone hour, timezone minute
2969+ ]
2970+
2971+ def fromString(self, s):
2972+ """
2973+ Parse a string containing a date and time in the wire format into a
2974+ C{datetime.datetime} instance.
2975+ """
2976+ if len(s) != 32:
2977+ raise ValueError('invalid date format %r' % (s,))
2978+
2979+ values = [int(s[p]) for p in self._positions]
2980+ sign = s[26]
2981+ timezone = _FixedOffsetTZInfo(sign, *values[7:])
2982+ values[7:] = [timezone]
2983+ return datetime.datetime(*values)
2984+
2985+ def toString(self, i):
2986+ """
2987+ Serialize a C{datetime.datetime} instance to a string in the specified
2988+ wire format.
2989+ """
2990+ offset = i.utcoffset()
2991+ if offset is None:
2992+ raise ValueError(
2993+ 'amp32.DateTime cannot serialize naive datetime instances. '
2994+ 'You may find amp32.utc useful.')
2995+
2996+ minutesOffset = (offset.days * 86400 + offset.seconds) // 60
2997+
2998+ if minutesOffset > 0:
2999+ sign = '+'
3000+ else:
3001+ sign = '-'
3002+
3003+ # strftime has no way to format the microseconds, or put a ':' in the
3004+ # timezone. Suprise!
3005+
3006+ return '%04i-%02i-%02iT%02i:%02i:%02i.%06i%s%02i:%02i' % (
3007+ i.year,
3008+ i.month,
3009+ i.day,
3010+ i.hour,
3011+ i.minute,
3012+ i.second,
3013+ i.microsecond,
3014+ sign,
3015+ abs(minutesOffset) // 60,
3016+ abs(minutesOffset) % 60)
3017
3018=== modified file 'src/provisioningserver/rpc/arguments.py'
3019--- src/provisioningserver/rpc/arguments.py 2014-10-30 11:22:41 +0000
3020+++ src/provisioningserver/rpc/arguments.py 2014-11-12 12:20:32 +0000
3021@@ -25,10 +25,10 @@
3022 import zlib
3023
3024 from apiclient.utils import ascii_url
3025-from twisted.protocols import amp
3026-
3027-
3028-class Bytes(amp.Argument):
3029+from provisioningserver.rpc import amp32
3030+
3031+
3032+class Bytes(amp32.Argument):
3033 """Encode a structure on the wire as bytes.
3034
3035 In truth, this does nothing more than assert that the inputs are
3036@@ -41,11 +41,11 @@
3037 return inObject
3038
3039 def fromString(self, inString):
3040- # inString is always a byte string, as defined by amp.Argument.
3041+ # inString is always a byte string, as defined by amp32.Argument.
3042 return inString
3043
3044
3045-class Choice(amp.Argument):
3046+class Choice(amp32.Argument):
3047 """Encode a choice to a predefined bytestring on the wire."""
3048
3049 def __init__(self, choices, optional=False):
3050@@ -76,7 +76,7 @@
3051 return self._decode[inString]
3052
3053
3054-class ParsedURL(amp.Argument):
3055+class ParsedURL(amp32.Argument):
3056 """Encode a URL on the wire.
3057
3058 The URL should be an instance of :py:class:`~urlparse.ParseResult`
3059@@ -105,11 +105,11 @@
3060 return urlparse.urlparse(inString)
3061
3062
3063-class StructureAsJSON(amp.Argument):
3064+class StructureAsJSON(amp32.Argument):
3065 """Encode a structure on the wire as JSON, compressed with zlib.
3066
3067 The compressed size of the structure should not exceed
3068- :py:data:`~twisted.protocols.amp.MAX_VALUE_LENGTH`, or ``0xffff``
3069+ :py:data:`~provisioningserver.rpc.amp32.MAX_VALUE_LENGTH`, or ``0xffff``
3070 bytes. This is pretty hard to be sure of ahead of time, so only use
3071 this for small structures that won't go near the limit.
3072 """
3073@@ -121,8 +121,8 @@
3074 return json.loads(zlib.decompress(inString))
3075
3076
3077-class CompressedAmpList(amp.AmpList):
3078- """An :py:class:`amp.AmpList` that's compressed on the wire.
3079+class CompressedAmpList(amp32.AmpList):
3080+ """An :py:class:`amp32.AmpList` that's compressed on the wire.
3081
3082 The serialised form is transparently compressed and decompressed with
3083 zlib. This can be useful when there's a lot of repetition in the list
3084
3085=== modified file 'src/provisioningserver/rpc/cluster.py'
3086--- src/provisioningserver/rpc/cluster.py 2014-10-23 18:21:41 +0000
3087+++ src/provisioningserver/rpc/cluster.py 2014-11-12 12:20:32 +0000
3088@@ -37,7 +37,10 @@
3089 PowerActionFail,
3090 UnknownPowerType,
3091 )
3092-from provisioningserver.rpc import exceptions
3093+from provisioningserver.rpc import (
3094+ amp32,
3095+ exceptions,
3096+ )
3097 from provisioningserver.rpc.arguments import (
3098 Bytes,
3099 ParsedURL,
3100@@ -47,10 +50,9 @@
3101 Authenticate,
3102 Identify,
3103 )
3104-from twisted.protocols import amp
3105-
3106-
3107-class ListBootImages(amp.Command):
3108+
3109+
3110+class ListBootImages(amp32.Command):
3111 """List the boot images available on this cluster controller.
3112
3113 :since: 1.5
3114@@ -58,20 +60,20 @@
3115
3116 arguments = []
3117 response = [
3118- (b"images", amp.AmpList(
3119- [(b"osystem", amp.Unicode()),
3120- (b"architecture", amp.Unicode()),
3121- (b"subarchitecture", amp.Unicode()),
3122- (b"release", amp.Unicode()),
3123- (b"label", amp.Unicode()),
3124- (b"purpose", amp.Unicode()),
3125- (b"xinstall_type", amp.Unicode()),
3126- (b"xinstall_path", amp.Unicode())]))
3127+ (b"images", amp32.AmpList(
3128+ [(b"osystem", amp32.Unicode()),
3129+ (b"architecture", amp32.Unicode()),
3130+ (b"subarchitecture", amp32.Unicode()),
3131+ (b"release", amp32.Unicode()),
3132+ (b"label", amp32.Unicode()),
3133+ (b"purpose", amp32.Unicode()),
3134+ (b"xinstall_type", amp32.Unicode()),
3135+ (b"xinstall_path", amp32.Unicode())]))
3136 ]
3137 errors = []
3138
3139
3140-class DescribePowerTypes(amp.Command):
3141+class DescribePowerTypes(amp32.Command):
3142 """Get a JSON Schema describing this cluster's power types.
3143
3144 :since: 1.5
3145@@ -84,7 +86,7 @@
3146 errors = []
3147
3148
3149-class ListSupportedArchitectures(amp.Command):
3150+class ListSupportedArchitectures(amp32.Command):
3151 """Report the cluster's supported architectures.
3152
3153 :since: 1.5
3154@@ -92,15 +94,15 @@
3155
3156 arguments = []
3157 response = [
3158- (b"architectures", amp.AmpList([
3159- (b"name", amp.Unicode()),
3160- (b"description", amp.Unicode()),
3161+ (b"architectures", amp32.AmpList([
3162+ (b"name", amp32.Unicode()),
3163+ (b"description", amp32.Unicode()),
3164 ])),
3165 ]
3166 errors = []
3167
3168
3169-class ListOperatingSystems(amp.Command):
3170+class ListOperatingSystems(amp32.Command):
3171 """Report the cluster's supported operating systems.
3172
3173 :since: 1.7
3174@@ -108,34 +110,34 @@
3175
3176 arguments = []
3177 response = [
3178- (b"osystems", amp.AmpList([
3179- (b"name", amp.Unicode()),
3180- (b"title", amp.Unicode()),
3181- (b"releases", amp.AmpList([
3182- (b"name", amp.Unicode()),
3183- (b"title", amp.Unicode()),
3184- (b"requires_license_key", amp.Boolean()),
3185- (b"can_commission", amp.Boolean()),
3186+ (b"osystems", amp32.AmpList([
3187+ (b"name", amp32.Unicode()),
3188+ (b"title", amp32.Unicode()),
3189+ (b"releases", amp32.AmpList([
3190+ (b"name", amp32.Unicode()),
3191+ (b"title", amp32.Unicode()),
3192+ (b"requires_license_key", amp32.Boolean()),
3193+ (b"can_commission", amp32.Boolean()),
3194 ])),
3195- (b"default_release", amp.Unicode(optional=True)),
3196- (b"default_commissioning_release", amp.Unicode(optional=True)),
3197+ (b"default_release", amp32.Unicode(optional=True)),
3198+ (b"default_commissioning_release", amp32.Unicode(optional=True)),
3199 ])),
3200 ]
3201 errors = []
3202
3203
3204-class GetOSReleaseTitle(amp.Command):
3205+class GetOSReleaseTitle(amp32.Command):
3206 """Get the title for the operating systems release.
3207
3208 :since: 1.7
3209 """
3210
3211 arguments = [
3212- (b"osystem", amp.Unicode()),
3213- (b"release", amp.Unicode()),
3214+ (b"osystem", amp32.Unicode()),
3215+ (b"release", amp32.Unicode()),
3216 ]
3217 response = [
3218- (b"title", amp.Unicode()),
3219+ (b"title", amp32.Unicode()),
3220 ]
3221 errors = {
3222 exceptions.NoSuchOperatingSystem: (
3223@@ -143,19 +145,19 @@
3224 }
3225
3226
3227-class ValidateLicenseKey(amp.Command):
3228+class ValidateLicenseKey(amp32.Command):
3229 """Validate an OS license key.
3230
3231 :since: 1.7
3232 """
3233
3234 arguments = [
3235- (b"osystem", amp.Unicode()),
3236- (b"release", amp.Unicode()),
3237- (b"key", amp.Unicode()),
3238+ (b"osystem", amp32.Unicode()),
3239+ (b"release", amp32.Unicode()),
3240+ (b"key", amp32.Unicode()),
3241 ]
3242 response = [
3243- (b"is_valid", amp.Boolean()),
3244+ (b"is_valid", amp32.Boolean()),
3245 ]
3246 errors = {
3247 exceptions.NoSuchOperatingSystem: (
3248@@ -163,20 +165,20 @@
3249 }
3250
3251
3252-class GetPreseedData(amp.Command):
3253+class GetPreseedData(amp32.Command):
3254 """Get OS-specific preseed data.
3255
3256 :since: 1.7
3257 """
3258
3259 arguments = [
3260- (b"osystem", amp.Unicode()),
3261- (b"preseed_type", amp.Unicode()),
3262- (b"node_system_id", amp.Unicode()),
3263- (b"node_hostname", amp.Unicode()),
3264- (b"consumer_key", amp.Unicode()),
3265- (b"token_key", amp.Unicode()),
3266- (b"token_secret", amp.Unicode()),
3267+ (b"osystem", amp32.Unicode()),
3268+ (b"preseed_type", amp32.Unicode()),
3269+ (b"node_system_id", amp32.Unicode()),
3270+ (b"node_hostname", amp32.Unicode()),
3271+ (b"consumer_key", amp32.Unicode()),
3272+ (b"token_key", amp32.Unicode()),
3273+ (b"token_secret", amp32.Unicode()),
3274 (b"metadata_url", ParsedURL()),
3275 ]
3276 response = [
3277@@ -190,16 +192,16 @@
3278 }
3279
3280
3281-class ComposeCurtinNetworkPreseed(amp.Command):
3282+class ComposeCurtinNetworkPreseed(amp32.Command):
3283 """Generate Curtin network preseed for a node.
3284
3285 :since: 1.7
3286 """
3287
3288 arguments = [
3289- (b"osystem", amp.Unicode()),
3290+ (b"osystem", amp32.Unicode()),
3291 (b"config", StructureAsJSON()),
3292- (b"disable_ipv4", amp.Boolean()),
3293+ (b"disable_ipv4", amp32.Boolean()),
3294 ]
3295 response = [
3296 (b"data", StructureAsJSON()),
3297@@ -209,16 +211,16 @@
3298 }
3299
3300
3301-class _Power(amp.Command):
3302+class _Power(amp32.Command):
3303 """Base class for power control commands.
3304
3305 :since: 1.7
3306 """
3307
3308 arguments = [
3309- (b"system_id", amp.Unicode()),
3310- (b"hostname", amp.Unicode()),
3311- (b"power_type", amp.Unicode()),
3312+ (b"system_id", amp32.Unicode()),
3313+ (b"hostname", amp32.Unicode()),
3314+ (b"power_type", amp32.Unicode()),
3315 # We can't define a tighter schema here because this is a highly
3316 # variable bag of arguments from a variety of sources.
3317 (b"context", StructureAsJSON()),
3318@@ -256,29 +258,29 @@
3319 :since: 1.7
3320 """
3321 response = [
3322- (b"state", amp.Unicode()),
3323+ (b"state", amp32.Unicode()),
3324 ]
3325
3326
3327-class _ConfigureDHCP(amp.Command):
3328+class _ConfigureDHCP(amp32.Command):
3329 """Configure a DHCP server.
3330
3331 :since: 1.7
3332 """
3333 arguments = [
3334- (b"omapi_key", amp.Unicode()),
3335- (b"subnet_configs", amp.AmpList([
3336- (b"subnet", amp.Unicode()),
3337- (b"subnet_mask", amp.Unicode()),
3338- (b"subnet_cidr", amp.Unicode()),
3339- (b"broadcast_ip", amp.Unicode()),
3340- (b"interface", amp.Unicode()),
3341- (b"router_ip", amp.Unicode()),
3342- (b"dns_servers", amp.Unicode()),
3343- (b"ntp_server", amp.Unicode()),
3344- (b"domain_name", amp.Unicode()),
3345- (b"ip_range_low", amp.Unicode()),
3346- (b"ip_range_high", amp.Unicode()),
3347+ (b"omapi_key", amp32.Unicode()),
3348+ (b"subnet_configs", amp32.AmpList([
3349+ (b"subnet", amp32.Unicode()),
3350+ (b"subnet_mask", amp32.Unicode()),
3351+ (b"subnet_cidr", amp32.Unicode()),
3352+ (b"broadcast_ip", amp32.Unicode()),
3353+ (b"interface", amp32.Unicode()),
3354+ (b"router_ip", amp32.Unicode()),
3355+ (b"dns_servers", amp32.Unicode()),
3356+ (b"ntp_server", amp32.Unicode()),
3357+ (b"domain_name", amp32.Unicode()),
3358+ (b"ip_range_low", amp32.Unicode()),
3359+ (b"ip_range_high", amp32.Unicode()),
3360 ])),
3361 ]
3362 response = []
3363@@ -299,18 +301,18 @@
3364 """
3365
3366
3367-class CreateHostMaps(amp.Command):
3368+class CreateHostMaps(amp32.Command):
3369 """Create host maps in the DHCP server's configuration.
3370
3371 :since: 1.7
3372 """
3373
3374 arguments = [
3375- (b"mappings", amp.AmpList([
3376- (b"ip_address", amp.Unicode()),
3377- (b"mac_address", amp.Unicode()),
3378+ (b"mappings", amp32.AmpList([
3379+ (b"ip_address", amp32.Unicode()),
3380+ (b"mac_address", amp32.Unicode()),
3381 ])),
3382- (b"shared_key", amp.Unicode()),
3383+ (b"shared_key", amp32.Unicode()),
3384 ]
3385 response = []
3386 errors = {
3387@@ -319,15 +321,15 @@
3388 }
3389
3390
3391-class RemoveHostMaps(amp.Command):
3392+class RemoveHostMaps(amp32.Command):
3393 """Remove host maps from the DHCP server's configuration.
3394
3395 :since: 1.7
3396 """
3397
3398 arguments = [
3399- (b"ip_addresses", amp.ListOf(amp.Unicode())),
3400- (b"shared_key", amp.Unicode()),
3401+ (b"ip_addresses", amp32.ListOf(amp32.Unicode())),
3402+ (b"shared_key", amp32.Unicode()),
3403 ]
3404 response = []
3405 errors = {
3406@@ -336,7 +338,7 @@
3407 }
3408
3409
3410-class ImportBootImages(amp.Command):
3411+class ImportBootImages(amp32.Command):
3412 """Import boot images and report the final
3413 boot images that exist on the cluster.
3414
3415@@ -344,15 +346,15 @@
3416 """
3417
3418 arguments = [
3419- (b"sources", amp.AmpList(
3420- [(b"url", amp.Unicode()),
3421+ (b"sources", amp32.AmpList(
3422+ [(b"url", amp32.Unicode()),
3423 (b"keyring_data", Bytes()),
3424- (b"selections", amp.AmpList(
3425- [(b"os", amp.Unicode()),
3426- (b"release", amp.Unicode()),
3427- (b"arches", amp.ListOf(amp.Unicode())),
3428- (b"subarches", amp.ListOf(amp.Unicode())),
3429- (b"labels", amp.ListOf(amp.Unicode()))]))])),
3430+ (b"selections", amp32.AmpList(
3431+ [(b"os", amp32.Unicode()),
3432+ (b"release", amp32.Unicode()),
3433+ (b"arches", amp32.ListOf(amp32.Unicode())),
3434+ (b"subarches", amp32.ListOf(amp32.Unicode())),
3435+ (b"labels", amp32.ListOf(amp32.Unicode()))]))])),
3436 (b"http_proxy", ParsedURL(optional=True)),
3437 (b"https_proxy", ParsedURL(optional=True)),
3438 ]
3439@@ -360,80 +362,80 @@
3440 errors = []
3441
3442
3443-class StartMonitors(amp.Command):
3444+class StartMonitors(amp32.Command):
3445 """Starts monitors(s) on the cluster.
3446
3447 :since: 1.7
3448 """
3449
3450 arguments = [
3451- (b"monitors", amp.AmpList(
3452- [(b"deadline", amp.DateTime()),
3453+ (b"monitors", amp32.AmpList(
3454+ [(b"deadline", amp32.DateTime()),
3455 (b"context", StructureAsJSON()),
3456- (b"id", amp.Unicode()),
3457+ (b"id", amp32.Unicode()),
3458 ]))
3459 ]
3460 response = []
3461 errors = []
3462
3463
3464-class CancelMonitor(amp.Command):
3465+class CancelMonitor(amp32.Command):
3466 """Cancels an existing monitor on the cluster.
3467
3468 :since: 1.7
3469 """
3470
3471 arguments = [
3472- (b"id", amp.Unicode()),
3473+ (b"id", amp32.Unicode()),
3474 ]
3475 response = []
3476 error = []
3477
3478
3479-class EvaluateTag(amp.Command):
3480+class EvaluateTag(amp32.Command):
3481 """Evaluate a tag against all of the cluster's nodes.
3482
3483 :since: 1.7
3484 """
3485
3486 arguments = [
3487- (b"tag_name", amp.Unicode()),
3488- (b"tag_definition", amp.Unicode()),
3489- (b"tag_nsmap", amp.AmpList([
3490- (b"prefix", amp.Unicode()),
3491- (b"uri", amp.Unicode()),
3492+ (b"tag_name", amp32.Unicode()),
3493+ (b"tag_definition", amp32.Unicode()),
3494+ (b"tag_nsmap", amp32.AmpList([
3495+ (b"prefix", amp32.Unicode()),
3496+ (b"uri", amp32.Unicode()),
3497 ])),
3498 # A 3-part credential string for the web API.
3499- (b"credentials", amp.Unicode()),
3500+ (b"credentials", amp32.Unicode()),
3501 ]
3502 response = []
3503 errors = []
3504
3505
3506-class AddVirsh(amp.Command):
3507+class AddVirsh(amp32.Command):
3508 """Probe for and enlist virsh VMs attached to the cluster.
3509
3510 :since: 1.7
3511 """
3512
3513 arguments = [
3514- (b"poweraddr", amp.Unicode()),
3515- (b"password", amp.Unicode(optional=True)),
3516+ (b"poweraddr", amp32.Unicode()),
3517+ (b"password", amp32.Unicode(optional=True)),
3518 ]
3519 response = []
3520 errors = []
3521
3522
3523-class AddSeaMicro15k(amp.Command):
3524+class AddSeaMicro15k(amp32.Command):
3525 """Probe for and enlist seamicro15k machines attached to the cluster.
3526
3527 :since: 1.7
3528 """
3529 arguments = [
3530- (b"mac", amp.Unicode()),
3531- (b"username", amp.Unicode()),
3532- (b"password", amp.Unicode()),
3533- (b"power_control", amp.Unicode(optional=True)),
3534+ (b"mac", amp32.Unicode()),
3535+ (b"username", amp32.Unicode()),
3536+ (b"password", amp32.Unicode()),
3537+ (b"power_control", amp32.Unicode(optional=True)),
3538 ]
3539 response = []
3540 errors = {
3541@@ -441,41 +443,41 @@
3542 }
3543
3544
3545-class EnlistNodesFromMSCM(amp.Command):
3546+class EnlistNodesFromMSCM(amp32.Command):
3547 """Probe for and enlist mscm machines attached to the cluster.
3548
3549 :since: 1.7
3550 """
3551 arguments = [
3552- (b"host", amp.Unicode()),
3553- (b"username", amp.Unicode()),
3554- (b"password", amp.Unicode()),
3555+ (b"host", amp32.Unicode()),
3556+ (b"username", amp32.Unicode()),
3557+ (b"password", amp32.Unicode()),
3558 ]
3559 response = []
3560 errors = {}
3561
3562
3563-class EnlistNodesFromUCSM(amp.Command):
3564+class EnlistNodesFromUCSM(amp32.Command):
3565 """Probe for and enlist ucsm machines attached to the cluster.
3566
3567 :since: 1.7
3568 """
3569 arguments = [
3570- (b"url", amp.Unicode()),
3571- (b"username", amp.Unicode()),
3572- (b"password", amp.Unicode()),
3573+ (b"url", amp32.Unicode()),
3574+ (b"username", amp32.Unicode()),
3575+ (b"password", amp32.Unicode()),
3576 ]
3577 response = []
3578 errors = {}
3579
3580
3581-class IsImportBootImagesRunning(amp.Command):
3582+class IsImportBootImagesRunning(amp32.Command):
3583 """Check if the import boot images task is running on the cluster.
3584
3585 :since: 1.7
3586 """
3587 arguments = []
3588 response = [
3589- (b"running", amp.Boolean()),
3590+ (b"running", amp32.Boolean()),
3591 ]
3592 errors = {}
3593
3594=== modified file 'src/provisioningserver/rpc/clusterservice.py'
3595--- src/provisioningserver/rpc/clusterservice.py 2014-11-10 15:11:58 +0000
3596+++ src/provisioningserver/rpc/clusterservice.py 2014-11-12 12:20:32 +0000
3597@@ -44,6 +44,7 @@
3598 from provisioningserver.logger.utils import log_call
3599 from provisioningserver.network import discover_networks
3600 from provisioningserver.rpc import (
3601+ amp32,
3602 cluster,
3603 common,
3604 dhcp,
3605@@ -97,7 +98,6 @@
3606 ConnectionClosed,
3607 )
3608 from twisted.internet.threads import deferToThread
3609-from twisted.protocols import amp
3610 from twisted.python import log
3611 from twisted.web import http
3612 import twisted.web.client
3613@@ -312,11 +312,11 @@
3614 cancel_monitor(id)
3615 return {}
3616
3617- @amp.StartTLS.responder
3618+ @amp32.StartTLS.responder
3619 def get_tls_parameters(self):
3620 """get_tls_parameters()
3621
3622- Implementation of :py:class:`~twisted.protocols.amp.StartTLS`.
3623+ Implementation of :py:class:`~provisioningserver.rpc.amp32.StartTLS`.
3624 """
3625 try:
3626 from provisioningserver.rpc.testing import tls
3627@@ -551,7 +551,7 @@
3628
3629 @inlineCallbacks
3630 def secureConnection(self):
3631- yield self.callRemote(amp.StartTLS, **self.get_tls_parameters())
3632+ yield self.callRemote(amp32.StartTLS, **self.get_tls_parameters())
3633
3634 # For some weird reason (it's mentioned in Twisted's source),
3635 # TLS negotiation does not complete until we do something with
3636
3637=== modified file 'src/provisioningserver/rpc/common.py'
3638--- src/provisioningserver/rpc/common.py 2014-10-03 12:52:35 +0000
3639+++ src/provisioningserver/rpc/common.py 2014-11-12 12:20:32 +0000
3640@@ -19,22 +19,22 @@
3641 "RPCProtocol",
3642 ]
3643
3644+from provisioningserver.rpc import amp32
3645 from provisioningserver.rpc.interfaces import IConnection
3646 from provisioningserver.utils.twisted import asynchronous
3647 from twisted.internet.defer import Deferred
3648-from twisted.protocols import amp
3649-
3650-
3651-class Identify(amp.Command):
3652+
3653+
3654+class Identify(amp32.Command):
3655 """Request the identity of the remote side, e.g. its UUID.
3656
3657 :since: 1.5
3658 """
3659
3660- response = [(b"ident", amp.Unicode())]
3661-
3662-
3663-class Authenticate(amp.Command):
3664+ response = [(b"ident", amp32.Unicode())]
3665+
3666+
3667+class Authenticate(amp32.Command):
3668 """Authenticate the remote side.
3669
3670 The procedure is as follows:
3671@@ -63,19 +63,19 @@
3672 """
3673
3674 arguments = [
3675- (b"message", amp.String()),
3676+ (b"message", amp32.String()),
3677 ]
3678 response = [
3679- (b"digest", amp.String()),
3680- (b"salt", amp.String()), # Is 'salt' the right term here?
3681+ (b"digest", amp32.String()),
3682+ (b"salt", amp32.String()), # Is 'salt' the right term here?
3683 ]
3684 errors = []
3685
3686
3687 class Client:
3688- """Wrapper around an :class:`amp.AMP` instance.
3689+ """Wrapper around an :class:`amp32.AMP` instance.
3690
3691- Limits the API to a subset of the behaviour of :class:`amp.AMP`'s,
3692+ Limits the API to a subset of the behaviour of :class:`amp32.AMP`'s,
3693 with alterations to make it suitable for use from a thread outside
3694 of the reactor.
3695 """
3696@@ -107,7 +107,7 @@
3697 different stack from the caller's, e.g. when calling into the
3698 Twisted reactor from a thread.
3699
3700- :param cmd: The `amp.Command` child class representing the remote
3701+ :param cmd: The `amp32.Command` child class representing the remote
3702 method to be invoked.
3703 :param kwargs: Any parameters to the remote method. Only keyword
3704 arguments are accepted.
3705@@ -143,10 +143,10 @@
3706 return hash(self._conn)
3707
3708
3709-class RPCProtocol(amp.AMP, object):
3710- """A specialisation of `amp.AMP`.
3711+class RPCProtocol(amp32.AMP, object):
3712+ """A specialisation of `amp32.AMP`.
3713
3714- It's hard to track exactly when an `amp.AMP` protocol is connected to its
3715+ It's hard to track exactly when an `amp32.AMP` protocol is connected to its
3716 transport, or disconnected, from the "outside". It's necessary to subclass
3717 and override `connectionMade` and `connectionLost` and signal from there,
3718 which is what this class does.
3719
3720=== modified file 'src/provisioningserver/rpc/monitors.py'
3721--- src/provisioningserver/rpc/monitors.py 2014-10-16 11:12:48 +0000
3722+++ src/provisioningserver/rpc/monitors.py 2014-11-12 12:20:32 +0000
3723@@ -20,11 +20,13 @@
3724 from datetime import datetime
3725
3726 from provisioningserver.logger import get_maas_logger
3727-from provisioningserver.rpc import getRegionClient
3728+from provisioningserver.rpc import (
3729+ amp32,
3730+ getRegionClient,
3731+ )
3732 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
3733 from provisioningserver.rpc.region import MonitorExpired
3734 from twisted.internet import reactor
3735-from twisted.protocols import amp
3736
3737
3738 maaslog = get_maas_logger("monitors")
3739@@ -47,7 +49,7 @@
3740 monitor ID.
3741 """
3742 for monitor in monitors:
3743- delay = monitor["deadline"] - datetime.now(amp.utc)
3744+ delay = monitor["deadline"] - datetime.now(amp32.utc)
3745 monitor_id = monitor["id"]
3746 if monitor_id in running_monitors:
3747 dc, _ = running_monitors.pop(monitor_id)
3748
3749=== modified file 'src/provisioningserver/rpc/region.py'
3750--- src/provisioningserver/rpc/region.py 2014-10-30 11:29:22 +0000
3751+++ src/provisioningserver/rpc/region.py 2014-11-12 12:20:32 +0000
3752@@ -40,6 +40,7 @@
3753 "UpdateNodePowerState",
3754 ]
3755
3756+from provisioningserver.rpc import amp32
3757 from provisioningserver.rpc.arguments import (
3758 Bytes,
3759 Choice,
3760@@ -59,10 +60,9 @@
3761 NoSuchEventType,
3762 NoSuchNode,
3763 )
3764-from twisted.protocols import amp
3765-
3766-
3767-class Register(amp.Command):
3768+
3769+
3770+class Register(amp32.Command):
3771 """Register a cluster with the region controller.
3772
3773 This is the last part of the Authenticate and Register two-step. See
3774@@ -72,11 +72,11 @@
3775 """
3776
3777 arguments = [
3778- (b"uuid", amp.Unicode()),
3779- (b"networks", amp.AmpList([
3780- (b"interface", amp.Unicode()),
3781- (b"ip", amp.Unicode()),
3782- (b"subnet_mask", amp.Unicode()),
3783+ (b"uuid", amp32.Unicode()),
3784+ (b"networks", amp32.AmpList([
3785+ (b"interface", amp32.Unicode()),
3786+ (b"ip", amp32.Unicode()),
3787+ (b"subnet_mask", amp32.Unicode()),
3788 ], optional=True)),
3789 # The URL for the region as seen by the cluster.
3790 (b"url", ParsedURL(optional=True)),
3791@@ -87,7 +87,7 @@
3792 }
3793
3794
3795-class ReportBootImages(amp.Command):
3796+class ReportBootImages(amp32.Command):
3797 """Report boot images available on the invoking cluster controller.
3798
3799 :since: 1.5
3800@@ -95,18 +95,18 @@
3801
3802 arguments = [
3803 # The cluster UUID.
3804- (b"uuid", amp.Unicode()),
3805- (b"images", amp.AmpList(
3806- [(b"architecture", amp.Unicode()),
3807- (b"subarchitecture", amp.Unicode()),
3808- (b"release", amp.Unicode()),
3809- (b"purpose", amp.Unicode())])),
3810+ (b"uuid", amp32.Unicode()),
3811+ (b"images", amp32.AmpList(
3812+ [(b"architecture", amp32.Unicode()),
3813+ (b"subarchitecture", amp32.Unicode()),
3814+ (b"release", amp32.Unicode()),
3815+ (b"purpose", amp32.Unicode())])),
3816 ]
3817 response = []
3818 errors = []
3819
3820
3821-class GetBootSources(amp.Command):
3822+class GetBootSources(amp32.Command):
3823 """Report boot sources and selections for the given cluster.
3824
3825 :since: 1.6
3826@@ -115,22 +115,22 @@
3827
3828 arguments = [
3829 # The cluster UUID.
3830- (b"uuid", amp.Unicode()),
3831+ (b"uuid", amp32.Unicode()),
3832 ]
3833 response = [
3834- (b"sources", amp.AmpList(
3835- [(b"url", amp.Unicode()),
3836+ (b"sources", amp32.AmpList(
3837+ [(b"url", amp32.Unicode()),
3838 (b"keyring_data", Bytes()),
3839- (b"selections", amp.AmpList(
3840- [(b"release", amp.Unicode()),
3841- (b"arches", amp.ListOf(amp.Unicode())),
3842- (b"subarches", amp.ListOf(amp.Unicode())),
3843- (b"labels", amp.ListOf(amp.Unicode()))]))])),
3844+ (b"selections", amp32.AmpList(
3845+ [(b"release", amp32.Unicode()),
3846+ (b"arches", amp32.ListOf(amp32.Unicode())),
3847+ (b"subarches", amp32.ListOf(amp32.Unicode())),
3848+ (b"labels", amp32.ListOf(amp32.Unicode()))]))])),
3849 ]
3850 errors = []
3851
3852
3853-class GetBootSourcesV2(amp.Command):
3854+class GetBootSourcesV2(amp32.Command):
3855 """Report boot sources and selections for the given cluster.
3856
3857 Includes the new os field for the selections.
3858@@ -140,33 +140,33 @@
3859
3860 arguments = [
3861 # The cluster UUID.
3862- (b"uuid", amp.Unicode()),
3863+ (b"uuid", amp32.Unicode()),
3864 ]
3865 response = [
3866- (b"sources", amp.AmpList(
3867- [(b"url", amp.Unicode()),
3868+ (b"sources", amp32.AmpList(
3869+ [(b"url", amp32.Unicode()),
3870 (b"keyring_data", Bytes()),
3871- (b"selections", amp.AmpList(
3872- [(b"os", amp.Unicode()),
3873- (b"release", amp.Unicode()),
3874- (b"arches", amp.ListOf(amp.Unicode())),
3875- (b"subarches", amp.ListOf(amp.Unicode())),
3876- (b"labels", amp.ListOf(amp.Unicode()))]))])),
3877+ (b"selections", amp32.AmpList(
3878+ [(b"os", amp32.Unicode()),
3879+ (b"release", amp32.Unicode()),
3880+ (b"arches", amp32.ListOf(amp32.Unicode())),
3881+ (b"subarches", amp32.ListOf(amp32.Unicode())),
3882+ (b"labels", amp32.ListOf(amp32.Unicode()))]))])),
3883 ]
3884 errors = []
3885
3886
3887-class UpdateLeases(amp.Command):
3888+class UpdateLeases(amp32.Command):
3889 """Report DHCP leases on the invoking cluster controller.
3890
3891 :since: 1.7
3892 """
3893 arguments = [
3894 # The cluster UUID.
3895- (b"uuid", amp.Unicode()),
3896+ (b"uuid", amp32.Unicode()),
3897 (b"mappings", CompressedAmpList(
3898- [(b"ip", amp.Unicode()),
3899- (b"mac", amp.Unicode())]))
3900+ [(b"ip", amp32.Unicode()),
3901+ (b"mac", amp32.Unicode())]))
3902 ]
3903 response = []
3904 errors = {
3905@@ -174,7 +174,7 @@
3906 }
3907
3908
3909-class GetArchiveMirrors(amp.Command):
3910+class GetArchiveMirrors(amp32.Command):
3911 """Return the Main and Port mirrors to use.
3912
3913 :since: 1.7
3914@@ -187,7 +187,7 @@
3915 errors = []
3916
3917
3918-class GetProxies(amp.Command):
3919+class GetProxies(amp32.Command):
3920 """Return the HTTP and HTTPS proxies to use.
3921
3922 :since: 1.6
3923@@ -201,7 +201,7 @@
3924 errors = []
3925
3926
3927-class GetClusterStatus(amp.Command):
3928+class GetClusterStatus(amp32.Command):
3929 """Return the status of the given cluster.
3930
3931 :since: 1.7
3932@@ -209,7 +209,7 @@
3933
3934 arguments = [
3935 # The cluster UUID.
3936- (b"uuid", amp.Unicode()),
3937+ (b"uuid", amp32.Unicode()),
3938 ]
3939 _response_status_choices = {
3940 0: b"PENDING", # NODEGROUP_STATUS.PENDING
3941@@ -224,7 +224,7 @@
3942 }
3943
3944
3945-class MarkNodeFailed(amp.Command):
3946+class MarkNodeFailed(amp32.Command):
3947 """Mark a node as 'broken'.
3948
3949 :since: 1.7
3950@@ -232,9 +232,9 @@
3951
3952 arguments = [
3953 # The node's system_id.
3954- (b"system_id", amp.Unicode()),
3955+ (b"system_id", amp32.Unicode()),
3956 # The error description.
3957- (b"error_description", amp.Unicode()),
3958+ (b"error_description", amp32.Unicode()),
3959 ]
3960 response = []
3961 errors = {
3962@@ -243,7 +243,7 @@
3963 }
3964
3965
3966-class ListNodePowerParameters(amp.Command):
3967+class ListNodePowerParameters(amp32.Command):
3968 """Return the list of power parameters for nodes
3969 that this cluster controls.
3970
3971@@ -255,14 +255,14 @@
3972
3973 arguments = [
3974 # The cluster UUID.
3975- (b"uuid", amp.Unicode()),
3976+ (b"uuid", amp32.Unicode()),
3977 ]
3978 response = [
3979- (b"nodes", amp.AmpList(
3980- [(b"system_id", amp.Unicode()),
3981- (b"hostname", amp.Unicode()),
3982- (b"power_state", amp.Unicode()),
3983- (b"power_type", amp.Unicode()),
3984+ (b"nodes", amp32.AmpList(
3985+ [(b"system_id", amp32.Unicode()),
3986+ (b"hostname", amp32.Unicode()),
3987+ (b"power_state", amp32.Unicode()),
3988+ (b"power_type", amp32.Unicode()),
3989 # We can't define a tighter schema here because this is a highly
3990 # variable bag of arguments from a variety of sources.
3991 (b"context", StructureAsJSON())])),
3992@@ -272,7 +272,7 @@
3993 }
3994
3995
3996-class UpdateNodePowerState(amp.Command):
3997+class UpdateNodePowerState(amp32.Command):
3998 """Update Node Power State.
3999
4000 :since: 1.7
4001@@ -280,120 +280,120 @@
4002
4003 arguments = [
4004 # The node's system_id.
4005- (b"system_id", amp.Unicode()),
4006+ (b"system_id", amp32.Unicode()),
4007 # The node's power_state.
4008- (b"power_state", amp.Unicode()),
4009+ (b"power_state", amp32.Unicode()),
4010 ]
4011 response = []
4012 errors = {NoSuchNode: b"NoSuchNode"}
4013
4014
4015-class RegisterEventType(amp.Command):
4016+class RegisterEventType(amp32.Command):
4017 """Register an event type.
4018
4019 :since: 1.7
4020 """
4021
4022 arguments = [
4023- (b"name", amp.Unicode()),
4024- (b"description", amp.Unicode()),
4025- (b"level", amp.Integer()),
4026+ (b"name", amp32.Unicode()),
4027+ (b"description", amp32.Unicode()),
4028+ (b"level", amp32.Integer()),
4029 ]
4030 response = []
4031 errors = []
4032
4033
4034-class SendEvent(amp.Command):
4035- """Send an event.
4036-
4037- :since: 1.7
4038- """
4039-
4040- arguments = [
4041- (b"system_id", amp.Unicode()),
4042- (b"type_name", amp.Unicode()),
4043- (b"description", amp.Unicode()),
4044- ]
4045- response = []
4046- errors = {
4047- NoSuchNode: b"NoSuchNode",
4048- NoSuchEventType: b"NoSuchEventType"
4049- }
4050-
4051-
4052-class SendEventMACAddress(amp.Command):
4053- """Send an event.
4054-
4055- :since: 1.7
4056- """
4057-
4058- arguments = [
4059- (b"mac_address", amp.Unicode()),
4060- (b"type_name", amp.Unicode()),
4061- (b"description", amp.Unicode()),
4062- ]
4063- response = []
4064- errors = {
4065- NoSuchNode: b"NoSuchNode",
4066- NoSuchEventType: b"NoSuchEventType"
4067- }
4068-
4069-
4070-class ReportForeignDHCPServer(amp.Command):
4071+class SendEvent(amp32.Command):
4072+ """Send an event.
4073+
4074+ :since: 1.7
4075+ """
4076+
4077+ arguments = [
4078+ (b"system_id", amp32.Unicode()),
4079+ (b"type_name", amp32.Unicode()),
4080+ (b"description", amp32.Unicode()),
4081+ ]
4082+ response = []
4083+ errors = {
4084+ NoSuchNode: b"NoSuchNode",
4085+ NoSuchEventType: b"NoSuchEventType"
4086+ }
4087+
4088+
4089+class SendEventMACAddress(amp32.Command):
4090+ """Send an event.
4091+
4092+ :since: 1.7
4093+ """
4094+
4095+ arguments = [
4096+ (b"mac_address", amp32.Unicode()),
4097+ (b"type_name", amp32.Unicode()),
4098+ (b"description", amp32.Unicode()),
4099+ ]
4100+ response = []
4101+ errors = {
4102+ NoSuchNode: b"NoSuchNode",
4103+ NoSuchEventType: b"NoSuchEventType"
4104+ }
4105+
4106+
4107+class ReportForeignDHCPServer(amp32.Command):
4108 """Report a foreign DHCP server on the cluster's network.
4109
4110 :since: 1.7
4111 """
4112
4113 arguments = [
4114- (b"cluster_uuid", amp.Unicode()),
4115- (b"interface_name", amp.Unicode()),
4116- (b"foreign_dhcp_ip", amp.Unicode(optional=True)),
4117+ (b"cluster_uuid", amp32.Unicode()),
4118+ (b"interface_name", amp32.Unicode()),
4119+ (b"foreign_dhcp_ip", amp32.Unicode(optional=True)),
4120 ]
4121 response = []
4122 errors = []
4123
4124
4125-class GetClusterInterfaces(amp.Command):
4126+class GetClusterInterfaces(amp32.Command):
4127 """Fetch the known interfaces for a cluster from the region.
4128
4129 :since: 1.7
4130 """
4131
4132 arguments = [
4133- (b"cluster_uuid", amp.Unicode()),
4134+ (b"cluster_uuid", amp32.Unicode()),
4135 ]
4136 response = [
4137- (b"interfaces", amp.AmpList(
4138- [(b"name", amp.Unicode()),
4139- (b"interface", amp.Unicode()),
4140- (b"ip", amp.Unicode())]))
4141+ (b"interfaces", amp32.AmpList(
4142+ [(b"name", amp32.Unicode()),
4143+ (b"interface", amp32.Unicode()),
4144+ (b"ip", amp32.Unicode())]))
4145 ]
4146 errors = []
4147
4148
4149-class CreateNode(amp.Command):
4150+class CreateNode(amp32.Command):
4151 """Create a node on a given cluster.
4152
4153 :since: 1.7
4154 """
4155
4156 arguments = [
4157- (b'cluster_uuid', amp.Unicode()),
4158- (b'architecture', amp.Unicode()),
4159- (b'power_type', amp.Unicode()),
4160- (b'power_parameters', amp.Unicode()),
4161- (b'mac_addresses', amp.ListOf(amp.Unicode())),
4162+ (b'cluster_uuid', amp32.Unicode()),
4163+ (b'architecture', amp32.Unicode()),
4164+ (b'power_type', amp32.Unicode()),
4165+ (b'power_parameters', amp32.Unicode()),
4166+ (b'mac_addresses', amp32.ListOf(amp32.Unicode())),
4167 ]
4168 response = [
4169- (b'system_id', amp.Unicode()),
4170+ (b'system_id', amp32.Unicode()),
4171 ]
4172 errors = {
4173 NodeAlreadyExists: b"NodeAlreadyExists",
4174 }
4175
4176
4177-class MonitorExpired(amp.Command):
4178+class MonitorExpired(amp32.Command):
4179 """Called by a cluster when a running monitor hits its deadline.
4180
4181 The original context parameter from the StartMonitors call is returned.
4182@@ -402,14 +402,14 @@
4183 """
4184
4185 arguments = [
4186- (b"id", amp.Unicode()),
4187+ (b"id", amp32.Unicode()),
4188 (b"context", StructureAsJSON()),
4189 ]
4190 response = []
4191 errors = []
4192
4193
4194-class ReloadCluster(amp.Command):
4195+class ReloadCluster(amp32.Command):
4196 """Called by a cluster when it wants to reload its state.
4197
4198 The region may respond with many different calls to the cluster
4199@@ -424,30 +424,30 @@
4200 """
4201
4202 arguments = [
4203- (b"cluster_uuid", amp.Unicode()),
4204+ (b"cluster_uuid", amp32.Unicode()),
4205 ]
4206 response = []
4207 errors = []
4208
4209
4210-class RequestNodeInfoByMACAddress(amp.Command):
4211+class RequestNodeInfoByMACAddress(amp32.Command):
4212 """Request Node information by mac address.
4213
4214 :since: 1.7
4215 """
4216
4217 arguments = [
4218- (b"mac_address", amp.Unicode()),
4219+ (b"mac_address", amp32.Unicode()),
4220 ]
4221 response = [
4222- (b"system_id", amp.Unicode()),
4223- (b"hostname", amp.Unicode()),
4224- (b"status", amp.Integer()),
4225- (b"boot_type", amp.Unicode()),
4226- (b"osystem", amp.Unicode()),
4227- (b"distro_series", amp.Unicode()),
4228- (b"architecture", amp.Unicode()),
4229- (b"purpose", amp.Unicode()),
4230+ (b"system_id", amp32.Unicode()),
4231+ (b"hostname", amp32.Unicode()),
4232+ (b"status", amp32.Integer()),
4233+ (b"boot_type", amp32.Unicode()),
4234+ (b"osystem", amp32.Unicode()),
4235+ (b"distro_series", amp32.Unicode()),
4236+ (b"architecture", amp32.Unicode()),
4237+ (b"purpose", amp32.Unicode()),
4238 ]
4239 errors = {
4240 NoSuchNode: b"NoSuchNode",
4241
4242=== modified file 'src/provisioningserver/rpc/testing/__init__.py'
4243--- src/provisioningserver/rpc/testing/__init__.py 2014-10-10 16:44:15 +0000
4244+++ src/provisioningserver/rpc/testing/__init__.py 2014-11-12 12:20:32 +0000
4245@@ -45,7 +45,10 @@
4246 sentinel,
4247 )
4248 import provisioningserver
4249-from provisioningserver.rpc import region
4250+from provisioningserver.rpc import (
4251+ amp32,
4252+ region,
4253+ )
4254 from provisioningserver.rpc.clusterservice import (
4255 Cluster,
4256 ClusterClient,
4257@@ -80,7 +83,6 @@
4258 )
4259 from twisted.internet.protocol import Factory
4260 from twisted.internet.task import Clock
4261-from twisted.protocols import amp
4262 from twisted.python import (
4263 log,
4264 reflect,
4265@@ -100,11 +102,11 @@
4266 d.addCallback(command.parseResponse, protocol)
4267
4268 def eb_massage_error(error):
4269- if error.check(amp.RemoteAmpError):
4270+ if error.check(amp32.RemoteAmpError):
4271 # Convert remote errors back into local errors using the
4272 # command's error map if possible.
4273 error_type = command.reverseErrors.get(
4274- error.value.errorCode, amp.UnknownRemoteError)
4275+ error.value.errorCode, amp32.UnknownRemoteError)
4276 return Failure(error_type(error.value.description))
4277 else:
4278 # Exceptions raised in responders that aren't declared in that
4279@@ -112,7 +114,7 @@
4280 # in RemoteAmpError. This is because call_responder() bypasses the
4281 # network marshall/unmarshall steps, where these exceptions would
4282 # ordinarily get squashed.
4283- return Failure(amp.UnknownRemoteError("%s: %s" % (
4284+ return Failure(amp32.UnknownRemoteError("%s: %s" % (
4285 reflect.qual(error.type), reflect.safe_str(error.value))))
4286 d.addErrback(eb_massage_error)
4287
4288@@ -261,7 +263,7 @@
4289 def addEventLoop(self, protocol):
4290 """Add a new stub event-loop using the given `protocol`.
4291
4292- The `protocol` should be an instance of `amp.AMP`.
4293+ The `protocol` should be an instance of `amp32.AMP`.
4294
4295 :returns: py:class:`twisted.test.iosim.IOPump`
4296 """
4297@@ -281,8 +283,8 @@
4298 commands = commands + (region.Authenticate,)
4299 if region.Register not in commands:
4300 commands = commands + (region.Register,)
4301- if amp.StartTLS not in commands:
4302- commands = commands + (amp.StartTLS,)
4303+ if amp32.StartTLS not in commands:
4304+ commands = commands + (amp32.StartTLS,)
4305 protocol_factory = make_amp_protocol_factory(*commands)
4306 protocol = protocol_factory()
4307 eventloop = self.getEventLoopName(protocol)
4308
4309=== modified file 'src/provisioningserver/rpc/testing/tls.py'
4310--- src/provisioningserver/rpc/testing/tls.py 2014-09-03 12:23:11 +0000
4311+++ src/provisioningserver/rpc/testing/tls.py 2014-11-12 12:20:32 +0000
4312@@ -26,7 +26,7 @@
4313 def get_tls_parameters(private_cert_name, trust_cert_name):
4314 """get_tls_parameters()
4315
4316- Implementation of :py:class:`~twisted.protocols.amp.StartTLS`.
4317+ Implementation of :py:class:`~provisioningserver.rpc.amp32.StartTLS`.
4318 """
4319 testing = filepath.FilePath(__file__).parent()
4320 with testing.child(private_cert_name).open() as fin:
4321
4322=== added file 'src/provisioningserver/rpc/tests/test_amp32.py'
4323--- src/provisioningserver/rpc/tests/test_amp32.py 1970-01-01 00:00:00 +0000
4324+++ src/provisioningserver/rpc/tests/test_amp32.py 2014-11-12 12:20:32 +0000
4325@@ -0,0 +1,3270 @@
4326+# Copyright (c) 2005 Divmod, Inc.
4327+# Copyright (c) Twisted Matrix Laboratories.
4328+# See LICENSE.Twisted for details.
4329+
4330+"""
4331+Tests for L{provisioningserver.rpc.amp32}.
4332+"""
4333+
4334+from __future__ import (
4335+ absolute_import,
4336+ print_function,
4337+ # unicode_literals,
4338+ )
4339+
4340+str = None
4341+
4342+__metaclass__ = type
4343+__all__ = []
4344+
4345+import datetime
4346+import decimal
4347+from warnings import (
4348+ catch_warnings,
4349+ simplefilter,
4350+ )
4351+
4352+from maastesting.fixtures import TempDirectory
4353+from maastesting.testcase import (
4354+ MAASTestCase,
4355+ MAASTwistedRunTest,
4356+ )
4357+from provisioningserver.rpc import amp32
4358+from provisioningserver.rpc.testing import TwistedLoggerFixture
4359+from testtools.deferredruntest import assert_fails_with
4360+from testtools.matchers import (
4361+ Equals,
4362+ HasLength,
4363+ Is,
4364+ IsInstance,
4365+ StartsWith,
4366+ )
4367+from twisted.internet import (
4368+ defer,
4369+ error,
4370+ interfaces,
4371+ protocol,
4372+ reactor,
4373+ )
4374+from twisted.internet.address import UNIXAddress
4375+from twisted.internet.error import ConnectionLost
4376+from twisted.python import filepath
4377+from twisted.python.failure import Failure
4378+from twisted.test import iosim
4379+from twisted.test.proto_helpers import StringTransport
4380+from zope.interface import implements
4381+from zope.interface.verify import (
4382+ verifyClass,
4383+ verifyObject,
4384+ )
4385+
4386+
4387+try:
4388+ from twisted.internet import ssl
4389+except ImportError:
4390+ ssl = None
4391+else:
4392+ if not ssl.supported:
4393+ ssl = None
4394+
4395+if ssl is None:
4396+ skipSSL = "SSL not available"
4397+else:
4398+ skipSSL = None
4399+
4400+
4401+class AMP32TestCase(MAASTestCase):
4402+ """Common test class for testing AMP with 32-bit length prefixes.
4403+
4404+ All the tests in this module, as well as the implementation in `amp32`,
4405+ are derived from Twisted's `amp` implementation. Twisted uses Trial and
4406+ the AMP tests depend on it. Many of these dependencies have been recast
4407+ into uses of `testtools` or plain `unittest` and so on. This class remains
4408+ as the retirement home for the last few bits for which there are not yet
4409+ clean and uninvasive replacements.
4410+ """
4411+
4412+ run_tests_with = MAASTwistedRunTest.make_factory(timeout=5)
4413+
4414+ def setUp(self):
4415+ super(AMP32TestCase, self).setUp()
4416+ # Capture all Twisted logs. Normally this isn't necessary, but these
4417+ # tests are very noisy without this. This is also required for
4418+ # getLoggedFailures().
4419+ self.logger = self.useFixture(TwistedLoggerFixture())
4420+
4421+ def getLoggedFailures(self, error_type):
4422+ """Replacement for Trial's `flushLoggedErrors`.
4423+
4424+ This only queries the log for errors, whereas `flushLoggedErrors` also
4425+ removes those errors from the log. The reason for this is that Twisted
4426+ will fail a test if any errors are logged.
4427+
4428+ In time it would make sense to adopt Trial's approach. That's too
4429+ invasive at present, so this remains here as a reminder.
4430+
4431+ See :py:func:`twisted.trial.testcase.TestCase.flushLoggedErrors`.
4432+ """
4433+ errors = (log for log in self.logger.logs if log["isError"])
4434+ failures = (log['failure'] for log in errors if 'failure' in log)
4435+ return list(f for f in failures if f.check(error_type))
4436+
4437+ def assertWarns(self, category, message, filename, f, *args, **kwargs):
4438+ """Replacement for Trial's `assertWarns`.
4439+
4440+ Trial's implementation is slightly more featureful, but this does much
4441+ the same. Could be recast as a :py:class:`testtools.matchers.Matcher`.
4442+
4443+ See :py:func:`twisted.trial.testcase.TestCase.assertWarns`.
4444+ """
4445+ with catch_warnings(record=True) as log:
4446+ simplefilter('always')
4447+ result = f(*args, **kwargs)
4448+
4449+ self.assertThat(log, HasLength(1))
4450+ [warning] = log
4451+ self.expectThat(category, Is(warning.category))
4452+ self.expectThat(warning.message, IsInstance(Warning))
4453+ self.expectThat(message, Equals(warning.message[0]))
4454+ self.expectThat(filename, StartsWith(warning.filename))
4455+
4456+ return result
4457+
4458+
4459+class TestProto(protocol.Protocol):
4460+ """
4461+ A trivial protocol for use in testing where a L{Protocol} is expected.
4462+
4463+ @ivar instanceId: the id of this instance
4464+ @ivar onConnLost: deferred that will fired when the connection is lost
4465+ @ivar dataToSend: data to send on the protocol
4466+ """
4467+
4468+ instanceCount = 0
4469+
4470+ def __init__(self, onConnLost, dataToSend):
4471+ self.onConnLost = onConnLost
4472+ self.dataToSend = dataToSend
4473+ self.instanceId = TestProto.instanceCount
4474+ TestProto.instanceCount = TestProto.instanceCount + 1
4475+
4476+ def connectionMade(self):
4477+ self.data = []
4478+ self.transport.write(self.dataToSend)
4479+
4480+ def dataReceived(self, bytes):
4481+ self.data.append(bytes)
4482+
4483+ def connectionLost(self, reason):
4484+ self.onConnLost.callback(self.data)
4485+
4486+ def __repr__(self):
4487+ """
4488+ Custom repr for testing to avoid coupling amp tests with repr from
4489+ L{Protocol}
4490+
4491+ Returns a string which contains a unique identifier that can be looked
4492+ up using the instanceId property::
4493+
4494+ <TestProto #3>
4495+ """
4496+ return "<TestProto #%d>" % (self.instanceId,)
4497+
4498+
4499+class SimpleSymmetricProtocol(amp32.AMP):
4500+
4501+ def sendHello(self, text):
4502+ return self.callRemoteString("hello", hello=text)
4503+
4504+ def amp_HELLO(self, box):
4505+ return amp32.Box(hello=box['hello'])
4506+
4507+ def amp_HOWDOYOUDO(self, box):
4508+ return amp32.QuitBox(howdoyoudo='world')
4509+
4510+
4511+class UnfriendlyGreeting(Exception):
4512+ """Greeting was insufficiently kind.
4513+ """
4514+
4515+
4516+class DeathThreat(Exception):
4517+ """Greeting was insufficiently kind.
4518+ """
4519+
4520+
4521+class UnknownProtocol(Exception):
4522+ """Asked to switch to the wrong protocol.
4523+ """
4524+
4525+
4526+class TransportPeer(amp32.Argument):
4527+
4528+ # this serves as some informal documentation for how to get variables from
4529+ # the protocol or your environment and pass them to methods as arguments.
4530+ def retrieve(self, d, name, proto):
4531+ return ''
4532+
4533+ def fromStringProto(self, notAString, proto):
4534+ return proto.transport.getPeer()
4535+
4536+ def toBox(self, name, strings, objects, proto):
4537+ return
4538+
4539+
4540+class Hello(amp32.Command):
4541+
4542+ commandName = 'hello'
4543+
4544+ arguments = [
4545+ ('hello', amp32.String()),
4546+ ('optional', amp32.Boolean(optional=True)),
4547+ ('print', amp32.Unicode(optional=True)),
4548+ ('from', TransportPeer(optional=True)),
4549+ ('mixedCase', amp32.String(optional=True)),
4550+ ('dash-arg', amp32.String(optional=True)),
4551+ ('underscore_arg', amp32.String(optional=True)),
4552+ ]
4553+
4554+ response = [
4555+ ('hello', amp32.String()),
4556+ ('print', amp32.Unicode(optional=True)),
4557+ ]
4558+
4559+ errors = {UnfriendlyGreeting: 'UNFRIENDLY'}
4560+
4561+ fatalErrors = {DeathThreat: 'DEAD'}
4562+
4563+
4564+class NoAnswerHello(Hello):
4565+
4566+ commandName = Hello.commandName
4567+ requiresAnswer = False
4568+
4569+
4570+class FutureHello(amp32.Command):
4571+
4572+ commandName = 'hello'
4573+ arguments = [
4574+ ('hello', amp32.String()),
4575+ ('optional', amp32.Boolean(optional=True)),
4576+ ('print', amp32.Unicode(optional=True)),
4577+ ('from', TransportPeer(optional=True)),
4578+ # addt'l arguments should generally be added at the end, and be
4579+ # optional...
4580+ ('bonus', amp32.String(optional=True)),
4581+ ]
4582+
4583+ response = [
4584+ ('hello', amp32.String()),
4585+ ('print', amp32.Unicode(optional=True)),
4586+ ]
4587+
4588+ errors = {UnfriendlyGreeting: 'UNFRIENDLY'}
4589+
4590+
4591+class WTF(amp32.Command):
4592+ """
4593+ An example of an invalid command.
4594+ """
4595+
4596+
4597+class BrokenReturn(amp32.Command):
4598+ """
4599+ An example of a perfectly good command, but the handler is going to return
4600+ None...
4601+ """
4602+
4603+ commandName = 'broken_return'
4604+
4605+
4606+class Goodbye(amp32.Command):
4607+
4608+ # commandName left blank on purpose: this tests implicit command names.
4609+ response = [('goodbye', amp32.String())]
4610+ responseType = amp32.QuitBox
4611+
4612+
4613+class Howdoyoudo(amp32.Command):
4614+
4615+ commandName = 'howdoyoudo'
4616+ # responseType = amp32.QuitBox
4617+
4618+
4619+class WaitForever(amp32.Command):
4620+
4621+ commandName = 'wait_forever'
4622+
4623+
4624+class GetList(amp32.Command):
4625+
4626+ commandName = 'getlist'
4627+ arguments = [('length', amp32.Integer())]
4628+ response = [('body', amp32.AmpList([('x', amp32.Integer())]))]
4629+
4630+
4631+class DontRejectMe(amp32.Command):
4632+
4633+ commandName = 'dontrejectme'
4634+ arguments = [
4635+ ('magicWord', amp32.Unicode()),
4636+ ('list', amp32.AmpList([('name', amp32.Unicode())], optional=True)),
4637+ ]
4638+ response = [('response', amp32.Unicode())]
4639+
4640+
4641+class SecuredPing(amp32.Command):
4642+
4643+ # XXX TODO: actually make this refuse to send over an insecure connection
4644+ response = [('pinged', amp32.Boolean())]
4645+
4646+
4647+class TestSwitchProto(amp32.ProtocolSwitchCommand):
4648+
4649+ commandName = 'Switch-Proto'
4650+ arguments = [
4651+ ('name', amp32.String()),
4652+ ]
4653+ errors = {UnknownProtocol: 'UNKNOWN'}
4654+
4655+
4656+class SingleUseFactory(protocol.ClientFactory):
4657+
4658+ def __init__(self, proto):
4659+ self.proto = proto
4660+ self.proto.factory = self
4661+
4662+ def buildProtocol(self, addr):
4663+ p, self.proto = self.proto, None
4664+ return p
4665+
4666+ reasonFailed = None
4667+
4668+ def clientConnectionFailed(self, connector, reason):
4669+ self.reasonFailed = reason
4670+ return
4671+
4672+
4673+THING_I_DONT_UNDERSTAND = 'gwebol nargo'
4674+
4675+
4676+class ThingIDontUnderstandError(Exception):
4677+ pass
4678+
4679+
4680+class FactoryNotifier(amp32.AMP):
4681+
4682+ factory = None
4683+
4684+ def connectionMade(self):
4685+ if self.factory is not None:
4686+ self.factory.theProto = self
4687+ if hasattr(self.factory, 'onMade'):
4688+ self.factory.onMade.callback(None)
4689+
4690+ @SecuredPing.responder
4691+ def emitpong(self):
4692+ from twisted.internet.interfaces import ISSLTransport
4693+ if not ISSLTransport.providedBy(self.transport):
4694+ raise DeathThreat("only send secure pings over secure channels")
4695+ return {'pinged': True}
4696+
4697+
4698+class SimpleSymmetricCommandProtocol(FactoryNotifier):
4699+
4700+ maybeLater = None
4701+
4702+ def __init__(self, onConnLost=None):
4703+ amp32.AMP.__init__(self)
4704+ self.onConnLost = onConnLost
4705+
4706+ def sendHello(self, text):
4707+ return self.callRemote(Hello, hello=text)
4708+
4709+ def sendUnicodeHello(self, text, translation):
4710+ return self.callRemote(Hello, hello=text, Print=translation)
4711+
4712+ greeted = False
4713+
4714+ @Hello.responder
4715+ def cmdHello(self, hello, From, optional=None, Print=None,
4716+ mixedCase=None, dash_arg=None, underscore_arg=None):
4717+ assert From == self.transport.getPeer()
4718+ if hello == THING_I_DONT_UNDERSTAND:
4719+ raise ThingIDontUnderstandError()
4720+ if hello.startswith('fuck'):
4721+ raise UnfriendlyGreeting("Don't be a dick.")
4722+ if hello == 'die':
4723+ raise DeathThreat("aieeeeeeeee")
4724+ result = dict(hello=hello)
4725+ if Print is not None:
4726+ result.update(dict(Print=Print))
4727+ self.greeted = True
4728+ return result
4729+
4730+ @GetList.responder
4731+ def cmdGetlist(self, length):
4732+ return {'body': [dict(x=1)] * length}
4733+
4734+ @DontRejectMe.responder
4735+ def okiwont(self, magicWord, list=None):
4736+ if list is None:
4737+ response = u'list omitted'
4738+ else:
4739+ response = u'%s accepted' % (list[0]['name'])
4740+ return dict(response=response)
4741+
4742+ @WaitForever.responder
4743+ def waitforit(self):
4744+ self.waiting = defer.Deferred()
4745+ return self.waiting
4746+
4747+ @Howdoyoudo.responder
4748+ def howdo(self):
4749+ return dict(howdoyoudo='world')
4750+
4751+ @Goodbye.responder
4752+ def saybye(self):
4753+ return dict(goodbye="everyone")
4754+
4755+ def switchToTestProtocol(self, fail=False):
4756+ if fail:
4757+ name = 'no-proto'
4758+ else:
4759+ name = 'test-proto'
4760+ p = TestProto(self.onConnLost, SWITCH_CLIENT_DATA)
4761+ return self.callRemote(
4762+ TestSwitchProto,
4763+ SingleUseFactory(p), name=name).addCallback(lambda ign: p)
4764+
4765+ @TestSwitchProto.responder
4766+ def switchit(self, name):
4767+ if name == 'test-proto':
4768+ return TestProto(self.onConnLost, SWITCH_SERVER_DATA)
4769+ raise UnknownProtocol(name)
4770+
4771+ @BrokenReturn.responder
4772+ def donothing(self):
4773+ return None
4774+
4775+
4776+class DeferredSymmetricCommandProtocol(SimpleSymmetricCommandProtocol):
4777+
4778+ @TestSwitchProto.responder
4779+ def switchit(self, name):
4780+ if name == 'test-proto':
4781+ self.maybeLaterProto = TestProto(
4782+ self.onConnLost, SWITCH_SERVER_DATA)
4783+ self.maybeLater = defer.Deferred()
4784+ return self.maybeLater
4785+ raise UnknownProtocol(name)
4786+
4787+
4788+class BadNoAnswerCommandProtocol(SimpleSymmetricCommandProtocol):
4789+
4790+ @NoAnswerHello.responder
4791+ def badResponder(
4792+ self, hello, From, optional=None, Print=None, mixedCase=None,
4793+ dash_arg=None, underscore_arg=None):
4794+ """
4795+ This responder does nothing and forgets to return a dictionary.
4796+ """
4797+
4798+
4799+class NoAnswerCommandProtocol(SimpleSymmetricCommandProtocol):
4800+
4801+ @NoAnswerHello.responder
4802+ def goodNoAnswerResponder(
4803+ self, hello, From, optional=None, Print=None,
4804+ mixedCase=None, dash_arg=None, underscore_arg=None):
4805+ return dict(hello=hello + "-noanswer")
4806+
4807+
4808+def connectedServerAndClient(
4809+ ServerClass=SimpleSymmetricProtocol,
4810+ ClientClass=SimpleSymmetricProtocol,
4811+ *a, **kw):
4812+ """Returns a 3-tuple: (client, server, pump)
4813+ """
4814+ return iosim.connectedServerAndClient(
4815+ ServerClass, ClientClass,
4816+ *a, **kw)
4817+
4818+
4819+class TotallyDumbProtocol(protocol.Protocol):
4820+
4821+ buf = ''
4822+
4823+ def dataReceived(self, data):
4824+ self.buf += data
4825+
4826+
4827+class LiteralAmp(amp32.AMP):
4828+
4829+ def __init__(self):
4830+ self.boxes = []
4831+
4832+ def ampBoxReceived(self, box):
4833+ self.boxes.append(box)
4834+ return
4835+
4836+
4837+class AmpBoxTests(AMP32TestCase):
4838+ """
4839+ Test a few essential properties of AMP boxes, mostly with respect to
4840+ serialization correctness.
4841+ """
4842+
4843+ def test_serializeStr(self):
4844+ """
4845+ Make sure that strs serialize to strs.
4846+ """
4847+ a = amp32.AmpBox(key='value')
4848+ self.assertEqual(type(a.serialize()), bytes)
4849+
4850+ def test_serializeUnicodeKeyRaises(self):
4851+ """
4852+ Verify that TypeError is raised when trying to serialize Unicode keys.
4853+ """
4854+ a = amp32.AmpBox(**{u'key': 'value'})
4855+ self.assertRaises(TypeError, a.serialize)
4856+
4857+ def test_serializeUnicodeValueRaises(self):
4858+ """
4859+ Verify that TypeError is raised when trying to serialize Unicode
4860+ values.
4861+ """
4862+ a = amp32.AmpBox(key=u'value')
4863+ self.assertRaises(TypeError, a.serialize)
4864+
4865+
4866+class ParsingTest(AMP32TestCase):
4867+
4868+ def test_booleanValues(self):
4869+ """
4870+ Verify that the Boolean parser parses 'True' and 'False', but nothing
4871+ else.
4872+ """
4873+ b = amp32.Boolean()
4874+ self.assertEqual(b.fromString("True"), True)
4875+ self.assertEqual(b.fromString("False"), False)
4876+ self.assertRaises(TypeError, b.fromString, "ninja")
4877+ self.assertRaises(TypeError, b.fromString, "true")
4878+ self.assertRaises(TypeError, b.fromString, "TRUE")
4879+ self.assertEqual(b.toString(True), 'True')
4880+ self.assertEqual(b.toString(False), 'False')
4881+
4882+ def test_pathValueRoundTrip(self):
4883+ """
4884+ Verify the 'Path' argument can parse and emit a file path.
4885+ """
4886+ with TempDirectory() as tempdir:
4887+ fp = filepath.FilePath(tempdir.path)
4888+ p = amp32.Path()
4889+ s = p.toString(fp)
4890+ v = p.fromString(s)
4891+ self.assertIsNot(fp, v) # sanity check
4892+ self.assertEqual(fp, v)
4893+
4894+ def test_sillyEmptyThing(self):
4895+ """
4896+ Test that empty boxes raise an error; they aren't supposed to be sent
4897+ on purpose.
4898+ """
4899+ a = amp32.AMP()
4900+ return self.assertRaises(
4901+ amp32.NoEmptyBoxes, a.ampBoxReceived, amp32.Box())
4902+
4903+ def test_ParsingRoundTrip(self):
4904+ """
4905+ Verify that various kinds of data make it through the encode/parse
4906+ round-trip unharmed.
4907+ """
4908+ c, s, p = connectedServerAndClient(
4909+ ClientClass=LiteralAmp, ServerClass=LiteralAmp)
4910+
4911+ SIMPLE = ('simple', 'test')
4912+ CE = ('ceq', ': ')
4913+ CR = ('crtest', 'test\r')
4914+ LF = ('lftest', 'hello\n')
4915+ NEWLINE = ('newline', 'test\r\none\r\ntwo')
4916+ NEWLINE2 = ('newline2', 'test\r\none\r\n two')
4917+ BODYTEST = ('body', 'blah\r\n\r\ntesttest')
4918+
4919+ testData = [
4920+ [SIMPLE],
4921+ [SIMPLE, BODYTEST],
4922+ [SIMPLE, CE],
4923+ [SIMPLE, CR],
4924+ [SIMPLE, CE, CR, LF],
4925+ [CE, CR, LF],
4926+ [SIMPLE, NEWLINE, CE, NEWLINE2],
4927+ [BODYTEST, SIMPLE, NEWLINE]
4928+ ]
4929+
4930+ for test in testData:
4931+ jb = amp32.Box()
4932+ jb.update(dict(test))
4933+ jb._sendTo(c)
4934+ p.flush()
4935+ self.assertEqual(s.boxes[-1], jb)
4936+
4937+
4938+class FakeLocator(object):
4939+ """
4940+ This is a fake implementation of the interface implied by
4941+ L{CommandLocator}.
4942+ """
4943+
4944+ def __init__(self):
4945+ """
4946+ Remember the given keyword arguments as a set of responders.
4947+ """
4948+ self.commands = {}
4949+
4950+ def locateResponder(self, commandName):
4951+ """
4952+ Look up and return a function passed as a keyword argument of the given
4953+ name to the constructor.
4954+ """
4955+ return self.commands[commandName]
4956+
4957+
4958+class FakeSender:
4959+ """
4960+ This is a fake implementation of the 'box sender' interface implied by
4961+ L{AMP}.
4962+ """
4963+
4964+ def __init__(self):
4965+ """
4966+ Create a fake sender and initialize the list of received boxes and
4967+ unhandled errors.
4968+ """
4969+ self.sentBoxes = []
4970+ self.unhandledErrors = []
4971+ self.expectedErrors = 0
4972+
4973+ def expectError(self):
4974+ """
4975+ Expect one error, so that the test doesn't fail.
4976+ """
4977+ self.expectedErrors += 1
4978+
4979+ def sendBox(self, box):
4980+ """
4981+ Accept a box, but don't do anything.
4982+ """
4983+ self.sentBoxes.append(box)
4984+
4985+ def unhandledError(self, failure):
4986+ """
4987+ Deal with failures by instantly re-raising them for easier debugging.
4988+ """
4989+ self.expectedErrors -= 1
4990+ if self.expectedErrors < 0:
4991+ failure.raiseException()
4992+ else:
4993+ self.unhandledErrors.append(failure)
4994+
4995+
4996+class CommandDispatchTests(AMP32TestCase):
4997+ """
4998+ The AMP CommandDispatcher class dispatches converts AMP boxes into commands
4999+ and responses using Command.responder decorator.
5000+
The diff has been truncated for viewing.