Merge lp:~allenap/launchpad/ec2-test-race-bug-422433 into lp:launchpad

Proposed by Gavin Panella
Status: Merged
Merged at revision: not available
Proposed branch: lp:~allenap/launchpad/ec2-test-race-bug-422433
Merge into: lp:launchpad
Diff against target: 717 lines
8 files modified
lib/devscripts/ec2test/account.py (+58/-48)
lib/devscripts/ec2test/builtins.py (+40/-35)
lib/devscripts/ec2test/instance.py (+26/-18)
lib/devscripts/ec2test/session.py (+90/-0)
lib/devscripts/ec2test/tests/__init__.py (+2/-0)
lib/devscripts/ec2test/tests/test_session.py (+69/-0)
lib/devscripts/ec2test/tests/test_utils.py (+61/-0)
lib/devscripts/ec2test/utils.py (+55/-0)
To merge this branch: bzr merge lp:~allenap/launchpad/ec2-test-race-bug-422433
Reviewer Review Type Date Requested Status
Abel Deuring (community) Approve
Review via email: mp+12778@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote :
Download full text (3.4 KiB)

This branch attempts to resolve a race condition when starting two or
more ec2test instances at similar times. What can happen is that,
during setup, one session can delete the key pair for the other
session, or delete the security group that the other session has just
created, because a static name ("ec2-test-runner") is used to name
these objects within AWS.

Now, with this branch, a session-specific name is used to name key
pairs and security groups so that sessions don't clobber each
other. The name contains some random data to give a good guarantee of
uniqueness.

However, one useful feature of using a static name is that old stuff
gets cleared up, so the new session name also contains a expiry
timestamp after which time the object can be deleted. The methods
delete_previous_security_groups() and delete_previous_key_pairs()
respect this timestamp.

I cleaned up a big pile of lint in revision 9611, and it's basically
noise next to this feature (though it still needs review). I've
prepared two diffs to help review, one containing only the lint and
another with only the feature changes.

== The features ==

Diff: http://pastebin.ubuntu.com/283667/

lib/devscripts/ec2test/builtins.py

  Create EC2SessionName instances based on EC2TestRunner.name instead
  of passing the name in directly.

lib/devscripts/ec2test/instance.py

  Check that the name is an instance of EC2SessionName, and call new
  method account.collect_garbage() in place of just
  account.delete_previous_key_pairs(). Incidentally, the former calls
  the latter.

lib/devscripts/ec2test/account.py

  Don't try to delete old security groups in
  acquire_security_group(). Removing old security groups is now done
  in delete_previous_security_groups().

  The method delete_previous_security_groups() and
  delete_previous_key_pairs() are very similar in the way they handle
  expiry timestamps.

lib/devscripts/ec2test/session.py
lib/devscripts/ec2test/tests/test_session.py

  Implementation and tests for EC2SessionName. This is a subclass of
  str with a few useful properties and a classmethod to aid in
  generating a good and unique name.

lib/devscripts/ec2test/utils.py
lib/devscripts/ec2test/tests/test_utils.py

  A few functions used to help with generating and parsing session
  names, and their tests.

  make_random_string() is a little unfortunate because there's a uuid
  module in Python 2.5 and beyond that does a better job of
  this. However, this script needs to support Python 2.4, so it'll
  stay for now. It does no harm really I guess.

== The lint fixes ==

http://pastebin.ubuntu.com/283676/

There is some outstanding lint in this branch:

  lib/devscripts/ec2test/builtins.py
    223: [W0102, cmd_test.run] Dangerous default value [] as argument
    289: [W0102, cmd_demo.run] Dangerous default value [] as argument
    363: [W0102, cmd_update_image.run] Dangerous default value [] as argument

  lib/devscripts/ec2test/instance.py
    408: [W0703, EC2Instance.set_up_and_run] Catch "Exception"

There aren't any tests for these scripts other than the ones I've
written in the branch, so I'm going to leave this lint in place
because changing them could affect behaviour. I'm build...

Read more...

Revision history for this message
Abel Deuring (adeuring) wrote :
Download full text (3.9 KiB)

Hi Gavin,

a very nice branch! Just one suggestion:

> + def delete_previous_security_groups(self):
> + """Delete previously used security groups, if found."""
> + def try_delete_group(group):
> + try:
> + group.delete()
> + except EC2ResponseError, e:
> + if e.code != 'InvalidGroup.InUse':
> + raise
> + self.log('Cannot delete; security group '
> + '%r in use.\n' % group.name)
> + now = datetime.utcnow()
> + for group in self.conn.get_all_security_groups():
> + session_name = EC2SessionName(group.name)
> + if session_name in (self.name, self.name.base):
> + self.log('Deleting security group %r\n' % group.name)
> + try_delete_group(group)
> + elif session_name.base == self.name.base:
> + if session_name.expires is None:
> + self.log('Found security group %r without creation '
> + 'date; leaving.\n' % group.name)
> + elif session_name.expires >= now:
> + self.log('Found recent security group %r; '
> + 'leaving\n' % group.name)
> + else:
> + self.log('Deleting expired security '
> + 'group %r\n' % group.name)
> + try_delete_group(group)
> + else:
> + self.log('Found other security group %r; '
> + 'leaving.\n' % group.name)
> +

> + def delete_previous_key_pairs(self):
> + """Delete previously used keypairs, if found."""
> + def try_delete_key_pair(key_pair):
> + try:
> + key_pair.delete()
> + except EC2ResponseError, e:
> + if e.code != 'InvalidKeyPair.NotFound':
> + if e.code == 'AuthFailure':
> + # Inserted because of previous support issue.
> + self.log(
> + 'POSSIBLE CAUSES OF ERROR:\n'
> + ' Did you sign up for EC2?\n'
> + ' Did you put a credit card number in your AWS '
> + 'account?\n'
> + 'Please doublecheck before reporting a '
> + 'problem.\n')
> + raise
> + self.log('Cannot delete; key pair not '
> + 'found %r\n' % key_pair.name)
> + now = datetime.utcnow()
> + for key_pair in self.conn.get_all_key_pairs():
> + session_name = EC2SessionName(key_pair.name)
> + if session_name in (self.name, self.name.base):
> + self.log('Deleting key pair %r\n' % key_pair.name)
> + try_delete_key_pair(key_pair)
> + elif session_name.base == self.name.base:
> + if session_name.expires is None:
> + self.log('Found key pair %r without creation date; '
> + 'leaving.\n' % key_pair.name)
> + elif session_n...

Read more...

review: Approve
Revision history for this message
Gavin Panella (allenap) wrote :

Thanks for the review!

> Hi Gavin,
>
> a very nice branch! Just one suggestion:
>
...
>
> The for loops in both methods are nearly identical; the only differences I see
> are that the log messages contain "key pair" or "secuity group", and that
> try_delete_key_pair() or try_delete_security_group() is called. Would it make
> sense to move the code starting at "if session_name" and ending at "else:
> self.log(..)" into a common method?

Good point. I've done this, and also ditched a lot of the noisy
logging; it now says something only if a key pair or security group is
deleted or if the deletion fails.

Incremental diff: http://pastebin.ubuntu.com/283801/

Thanks, Gavin.

Revision history for this message
Abel Deuring (adeuring) wrote :

Hi Gavin,

On 02.10.2009 14:35, Gavin Panella wrote:
> Thanks for the review!
>
>> Hi Gavin,
>>
>> a very nice branch! Just one suggestion:
>>
> ...
>> The for loops in both methods are nearly identical; the only differences I see
>> are that the log messages contain "key pair" or "secuity group", and that
>> try_delete_key_pair() or try_delete_security_group() is called. Would it make
>> sense to move the code starting at "if session_name" and ending at "else:
>> self.log(..)" into a common method?
>
> Good point. I've done this, and also ditched a lot of the noisy
> logging; it now says something only if a key pair or security group is
> deleted or if the deletion fails.
>
> Incremental diff: http://pastebin.ubuntu.com/283801/

thanks for this change! Looks good

Abel

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/devscripts/ec2test/account.py'
2--- lib/devscripts/ec2test/account.py 2009-09-29 15:48:49 +0000
3+++ lib/devscripts/ec2test/account.py 2009-10-02 12:34:16 +0000
4@@ -13,7 +13,10 @@
5 import sys
6 import urllib
7
8+from datetime import datetime
9+
10 from boto.exception import EC2ResponseError
11+from devscripts.ec2test.session import EC2SessionName
12
13 import paramiko
14
15@@ -24,6 +27,13 @@
16 # ...anyone else want in on the fun?
17 )
18
19+AUTH_FAILURE_MESSAGE = """\
20+POSSIBLE CAUSES OF ERROR:
21+- Did you sign up for EC2?
22+- Did you put a credit card number in your AWS account?
23+Please double-check before reporting a problem.
24+"""
25+
26
27 def get_ip():
28 """Uses AWS checkip to obtain this machine's IP address.
29@@ -64,44 +74,29 @@
30 sys.stdout.write(msg)
31 sys.stdout.flush()
32
33+ def _find_expired_artifacts(self, artifacts):
34+ now = datetime.utcnow()
35+ for artifact in artifacts:
36+ session_name = EC2SessionName(artifact.name)
37+ if (session_name in (self.name, self.name.base) or (
38+ session_name.base == self.name.base and
39+ session_name.expires is not None and
40+ session_name.expires < now)):
41+ yield artifact
42+
43 def acquire_security_group(self, demo_networks=None):
44 """Get a security group with the appropriate configuration.
45
46 "Appropriate" means configured to allow this machine to connect via
47 SSH, HTTP and HTTPS.
48
49- If a group is already configured with this name for this connection,
50- then re-use that. Otherwise, create a new security group and configure
51- it appropriately.
52-
53 The name of the security group is the `EC2Account.name` attribute.
54
55 :return: A boto security group.
56 """
57 if demo_networks is None:
58 demo_networks = []
59- try:
60- group = self.conn.get_all_security_groups(self.name)[0]
61- except EC2ResponseError, e:
62- if e.code != 'InvalidGroup.NotFound':
63- raise
64- else:
65- # If an existing security group was configured, try deleting it
66- # since our external IP might have changed.
67- try:
68- group.delete()
69- except EC2ResponseError, e:
70- if e.code != 'InvalidGroup.InUse':
71- raise
72- # Otherwise, it means that an instance is already using
73- # it, so simply re-use it. It's unlikely that our IP changed!
74- #
75- # XXX: JonathanLange 2009-06-05: If the security group exists
76- # already, verify that the current IP is permitted; if it is
77- # not, make an INFO log and add the current IP.
78- self.log("Security group already in use, so reusing.\n")
79- return group
80-
81+ # Create the security group.
82 security_group = self.conn.create_security_group(
83 self.name, 'Authorization to access the test runner instance.')
84 # Authorize SSH and HTTP.
85@@ -117,34 +112,49 @@
86 security_group.authorize('tcp', 443, 443, network)
87 return security_group
88
89+ def delete_previous_security_groups(self):
90+ """Delete previously used security groups, if found."""
91+ expired_groups = self._find_expired_artifacts(
92+ self.conn.get_all_security_groups())
93+ for group in expired_groups:
94+ try:
95+ group.delete()
96+ except EC2ResponseError, e:
97+ if e.code != 'InvalidGroup.InUse':
98+ raise
99+ self.log('Cannot delete; security group '
100+ '%r in use.\n' % group.name)
101+ else:
102+ self.log('Deleted security group %r.' % group.name)
103+
104 def acquire_private_key(self):
105 """Create & return a new key pair for the test runner."""
106 key_pair = self.conn.create_key_pair(self.name)
107 return paramiko.RSAKey.from_private_key(
108 cStringIO.StringIO(key_pair.material.encode('ascii')))
109
110- def delete_previous_key_pair(self):
111- """Delete previously used keypair, if it exists."""
112- try:
113- # Only one keypair will match 'self.name' since it's a unique
114- # identifier.
115- key_pairs = self.conn.get_all_key_pairs(self.name)
116- assert len(key_pairs) == 1, (
117- "Should be only one keypair, found %d (%s)"
118- % (len(key_pairs), key_pairs))
119- key_pair = key_pairs[0]
120- key_pair.delete()
121- except EC2ResponseError, e:
122- if e.code != 'InvalidKeyPair.NotFound':
123- if e.code == 'AuthFailure':
124- # Inserted because of previous support issue.
125- self.log(
126- 'POSSIBLE CAUSES OF ERROR:\n'
127- ' Did you sign up for EC2?\n'
128- ' Did you put a credit card number in your AWS '
129- 'account?\n'
130- 'Please doublecheck before reporting a problem.\n')
131- raise
132+ def delete_previous_key_pairs(self):
133+ """Delete previously used keypairs, if found."""
134+ expired_key_pairs = self._find_expired_artifacts(
135+ self.conn.get_all_key_pairs())
136+ for key_pair in expired_key_pairs:
137+ try:
138+ key_pair.delete()
139+ except EC2ResponseError, e:
140+ if e.code != 'InvalidKeyPair.NotFound':
141+ if e.code == 'AuthFailure':
142+ # Inserted because of previous support issue.
143+ self.log(AUTH_FAILURE_MESSAGE)
144+ raise
145+ self.log('Cannot delete; key pair not '
146+ 'found %r\n' % key_pair.name)
147+ else:
148+ self.log('Deleted key pair %r.' % key_pair.name)
149+
150+ def collect_garbage(self):
151+ """Remove any old keys and security groups."""
152+ self.delete_previous_security_groups()
153+ self.delete_previous_key_pairs()
154
155 def acquire_image(self, machine_id):
156 """Get the image.
157
158=== modified file 'lib/devscripts/ec2test/builtins.py'
159--- lib/devscripts/ec2test/builtins.py 2009-09-30 12:57:13 +0000
160+++ lib/devscripts/ec2test/builtins.py 2009-10-02 12:34:16 +0000
161@@ -23,6 +23,7 @@
162 from devscripts.ec2test.credentials import EC2Credentials
163 from devscripts.ec2test.instance import (
164 AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
165+from devscripts.ec2test.session import EC2SessionName
166 from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
167
168
169@@ -168,52 +169,52 @@
170 help=('Store abridged test results in FILE.')),
171 ListOption(
172 'email', short_name='e', argname='EMAIL', type=str,
173- help=('Email address to which results should be mailed. Defaults to '
174- 'the email address from `bzr whoami`. May be supplied multiple '
175- 'times. The first supplied email address will be used as the '
176- 'From: address.')),
177+ help=('Email address to which results should be mailed. Defaults '
178+ 'to the email address from `bzr whoami`. May be supplied '
179+ 'multiple times. The first supplied email address will be '
180+ 'used as the From: address.')),
181 Option(
182 'noemail', short_name='n',
183 help=('Do not try to email results.')),
184 Option(
185 'test-options', short_name='o', type=str,
186- help=('Test options to pass to the remote test runner. Defaults to '
187- "``-o '-vv'``. For instance, to run specific tests, you might "
188- "use ``-o '-vvt my_test_pattern'``.")),
189+ help=('Test options to pass to the remote test runner. Defaults '
190+ "to ``-o '-vv'``. For instance, to run specific tests, "
191+ "you might use ``-o '-vvt my_test_pattern'``.")),
192 Option(
193 'submit-pqm-message', short_name='s', type=str, argname="MSG",
194- help=('A pqm message to submit if the test run is successful. If '
195- 'provided, you will be asked for your GPG passphrase before '
196- 'the test run begins.')),
197+ help=('A pqm message to submit if the test run is successful. '
198+ 'If provided, you will be asked for your GPG passphrase '
199+ 'before the test run begins.')),
200 Option(
201 'pqm-public-location', type=str,
202- help=('The public location for the pqm submit, if a pqm message is '
203- 'provided (see --submit-pqm-message). If this is not provided, '
204- 'for local branches, bzr configuration is consulted; for '
205- 'remote branches, it is assumed that the remote branch *is* '
206- 'a public branch.')),
207+ help=('The public location for the pqm submit, if a pqm message '
208+ 'is provided (see --submit-pqm-message). If this is not '
209+ 'provided, for local branches, bzr configuration is '
210+ 'consulted; for remote branches, it is assumed that the '
211+ 'remote branch *is* a public branch.')),
212 Option(
213 'pqm-submit-location', type=str,
214- help=('The submit location for the pqm submit, if a pqm message is '
215- 'provided (see --submit-pqm-message). If this option is not '
216- 'provided, the script will look for an explicitly specified '
217- 'launchpad branch using the -b/--branch option; if that branch '
218- 'was specified and is owned by the launchpad-pqm user on '
219- 'launchpad, it is used as the pqm submit location. Otherwise, '
220- 'for local branches, bzr configuration is consulted; for '
221- 'remote branches, it is assumed that the submit branch is %s.'
222- % (TRUNK_BRANCH,))),
223+ help=('The submit location for the pqm submit, if a pqm message '
224+ 'is provided (see --submit-pqm-message). If this option '
225+ 'is not provided, the script will look for an explicitly '
226+ 'specified launchpad branch using the -b/--branch option; '
227+ 'if that branch was specified and is owned by the '
228+ 'launchpad-pqm user on launchpad, it is used as the pqm '
229+ 'submit location. Otherwise, for local branches, bzr '
230+ 'configuration is consulted; for remote branches, it is '
231+ 'assumed that the submit branch is %s.' % (TRUNK_BRANCH,))),
232 Option(
233 'pqm-email', type=str,
234- help=('Specify the email address of the PQM you are submitting to. '
235- 'If the branch is local, then the bzr configuration is '
236+ help=('Specify the email address of the PQM you are submitting '
237+ 'to. If the branch is local, then the bzr configuration is '
238 'consulted; for remote branches "Launchpad PQM '
239 '<launchpad@pqm.canonical.com>" is used by default.')),
240 postmortem_option,
241 Option(
242 'headless',
243- help=('After building the instance and test, run the remote tests '
244- 'headless. Cannot be used with postmortem '
245+ help=('After building the instance and test, run the remote '
246+ 'tests headless. Cannot be used with postmortem '
247 'or file.')),
248 debug_option,
249 Option(
250@@ -252,9 +253,9 @@
251 'You have specified no way to get the results '
252 'of your headless test run.')
253
254-
255+ session_name = EC2SessionName.make(EC2TestRunner.name)
256 instance = EC2Instance.make(
257- EC2TestRunner.name, instance_type, machine)
258+ session_name, instance_type, machine)
259
260 runner = EC2TestRunner(
261 test_branch, email=email, file=file,
262@@ -381,8 +382,9 @@
263 branches, test_branch = _get_branches_and_test_branch(
264 trunk, branch, test_branch)
265
266+ session_name = EC2SessionName.make(EC2TestRunner.name)
267 instance = EC2Instance.make(
268- EC2TestRunner.name, instance_type, machine, demo)
269+ session_name, instance_type, machine, demo)
270
271 runner = EC2TestRunner(
272 test_branch, branches=branches,
273@@ -435,8 +437,9 @@
274 ListOption(
275 'extra-update-image-command', type=str,
276 help=('Run this command (with an ssh agent) on the image before '
277- 'running the default update steps. Can be passed more than '
278- 'once, the commands will be run in the order specified.')),
279+ 'running the default update steps. Can be passed more '
280+ 'than once, the commands will be run in the order '
281+ 'specified.')),
282 Option(
283 'public',
284 help=('Remove proprietary code from the sourcecode directory '
285@@ -453,8 +456,9 @@
286
287 credentials = EC2Credentials.load_from_file()
288
289+ session_name = EC2SessionName.make(EC2TestRunner.name)
290 instance = EC2Instance.make(
291- EC2TestRunner.name, instance_type, machine,
292+ session_name, instance_type, machine,
293 credentials=credentials)
294 instance.check_bundling_prerequisites()
295
296@@ -489,7 +493,8 @@
297 user_connection.run_with_ssh_agent(
298 'bzr pull -d /var/launchpad/test ' + TRUNK_BRANCH)
299 user_connection.run_with_ssh_agent(
300- 'bzr pull -d /var/launchpad/download-cache lp:lp-source-dependencies')
301+ 'bzr pull -d /var/launchpad/download-cache '
302+ 'lp:lp-source-dependencies')
303 if public:
304 update_sourcecode_options = '--public-only'
305 else:
306
307=== modified file 'lib/devscripts/ec2test/instance.py'
308--- lib/devscripts/ec2test/instance.py 2009-09-27 20:36:18 +0000
309+++ lib/devscripts/ec2test/instance.py 2009-10-02 12:34:16 +0000
310@@ -24,6 +24,7 @@
311 import paramiko
312
313 from devscripts.ec2test.credentials import EC2Credentials
314+from devscripts.ec2test.session import EC2SessionName
315
316
317 DEFAULT_INSTANCE_TYPE = 'c1.xlarge'
318@@ -159,6 +160,18 @@
319 """
320
321
322+postmortem_banner = """\
323+Postmortem Console. EC2 instance is not yet dead.
324+It will shut down when you exit this prompt (CTRL-D)
325+
326+Tab-completion is enabled.
327+EC2Instance is available as `instance`.
328+Also try these:
329+ http://%(dns)s/current_test.log
330+ ssh -A %(dns)s
331+"""
332+
333+
334 class EC2Instance:
335 """A single EC2 instance."""
336
337@@ -169,6 +182,7 @@
338
339 :param name: The name to use for the key pair and security group for
340 the instance.
341+ :type name: `EC2SessionName`
342 :param instance_type: One of the AVAILABLE_INSTANCE_TYPES.
343 :param machine_id: The AMI to use, or None to do the usual regexp
344 matching. If you put 'based-on:' before the AMI id, it is assumed
345@@ -179,6 +193,7 @@
346 to allow access to the instance.
347 :param credentials: An `EC2Credentials` object.
348 """
349+ assert isinstance(name, EC2SessionName)
350 if instance_type not in AVAILABLE_INSTANCE_TYPES:
351 raise ValueError('unknown instance_type %s' % (instance_type,))
352
353@@ -199,7 +214,7 @@
354 # We always recreate the keypairs because there is no way to
355 # programmatically retrieve the private key component, unless we
356 # generate it.
357- account.delete_previous_key_pair()
358+ account.collect_garbage()
359
360 if machine_id and machine_id.startswith('based-on:'):
361 from_scratch = True
362@@ -320,7 +335,8 @@
363 """
364 authorized_keys_file = conn.sftp.open(remote_filename, 'w')
365 authorized_keys_file.write(
366- "%s %s\n" % (self._user_key.get_name(), self._user_key.get_base64()))
367+ "%s %s\n" % (self._user_key.get_name(),
368+ self._user_key.get_base64()))
369 authorized_keys_file.close()
370
371 def _ensure_ec2test_user_has_keys(self, connection=None):
372@@ -345,8 +361,8 @@
373 if our_connection:
374 connection.close()
375 self.log(
376- 'You can now use ssh -A ec2test@%s to log in the instance.\n' %
377- self.hostname)
378+ 'You can now use ssh -A ec2test@%s to '
379+ 'log in the instance.\n' % self.hostname)
380 self._ec2test_user_has_keys = True
381
382 def connect(self):
383@@ -390,25 +406,17 @@
384 try:
385 return func(*args, **kw)
386 except Exception:
387- # When running in postmortem mode, it is really helpful to see if
388- # there are any exceptions before it waits in the console (in the
389- # finally block), and you can't figure out why it's broken.
390+ # When running in postmortem mode, it is really helpful to see
391+ # if there are any exceptions before it waits in the console
392+ # (in the finally block), and you can't figure out why it's
393+ # broken.
394 traceback.print_exc()
395 finally:
396 try:
397 if postmortem:
398 console = code.InteractiveConsole(locals())
399- console.interact((
400- 'Postmortem Console. EC2 instance is not yet dead.\n'
401- 'It will shut down when you exit this prompt (CTRL-D).\n'
402- '\n'
403- 'Tab-completion is enabled.'
404- '\n'
405- 'EC2Instance is available as `instance`.\n'
406- 'Also try these:\n'
407- ' http://%(dns)s/current_test.log\n'
408- ' ssh -A %(dns)s') %
409- {'dns': self.hostname})
410+ console.interact(
411+ postmortem_banner % {'dns': self.hostname})
412 print 'Postmortem console closed.'
413 finally:
414 if shutdown:
415
416=== added file 'lib/devscripts/ec2test/session.py'
417--- lib/devscripts/ec2test/session.py 1970-01-01 00:00:00 +0000
418+++ lib/devscripts/ec2test/session.py 2009-10-02 12:34:16 +0000
419@@ -0,0 +1,90 @@
420+# Copyright 2009 Canonical Ltd. This software is licensed under the
421+# GNU Affero General Public License version 3 (see the file LICENSE).
422+
423+"""Code to represent a single session of EC2 use."""
424+
425+__metaclass__ = type
426+__all__ = [
427+ 'EC2SessionName',
428+ ]
429+
430+from datetime import datetime, timedelta
431+
432+from devscripts.ec2test.utils import (
433+ find_datetime_string, make_datetime_string, make_random_string)
434+
435+
436+DEFAULT_LIFETIME = timedelta(hours=6)
437+
438+
439+class EC2SessionName(str):
440+ """A name for an EC2 session.
441+
442+ This is used when naming key pairs and security groups, so it's
443+ useful to be unique. However, to aid garbage collection of old key
444+ pairs and security groups, the name contains a common element and
445+ an expiry timestamp. The form taken should always be:
446+
447+ <base-name>/<expires-timestamp>/<random-data>
448+
449+ None of the parts should contain forward-slashes, and the
450+ timestamp should be acceptable input to `find_datetime_string`.
451+
452+ `EC2SessionName.make()` will generate a suitable name given a
453+ suitable base name.
454+ """
455+
456+ @classmethod
457+ def make(cls, base, expires=None):
458+ """Create an `EC2SessionName`.
459+
460+ This checks that `base` does not contain a forward-slash, and
461+ provides some convenient functionality for `expires`:
462+
463+ - If `expires` is None, it defaults to now (UTC) plus
464+ `DEFAULT_LIFETIME`.
465+
466+ - If `expires` is a `datetime`, it is converted to a timestamp
467+ in the correct form.
468+
469+ - If `expires` is a `timedelta`, it is added to now (UTC) then
470+ converted to a timestamp.
471+
472+ - Otherwise `expires` is assumed to be a string, so is checked
473+ for the absense of forward-slashes, and that a correctly
474+ formed timestamp can be discovered.
475+
476+ """
477+ assert '/' not in base
478+ if expires is None:
479+ expires = DEFAULT_LIFETIME
480+ if isinstance(expires, timedelta):
481+ expires = datetime.utcnow() + expires
482+ if isinstance(expires, datetime):
483+ expires = make_datetime_string(expires)
484+ else:
485+ assert '/' not in expires
486+ assert find_datetime_string(expires) is not None
487+ rand = make_random_string(8)
488+ return cls("%s/%s/%s" % (base, expires, rand))
489+
490+ @property
491+ def base(self):
492+ parts = self.split('/')
493+ if len(parts) != 3:
494+ return None
495+ return parts[0]
496+
497+ @property
498+ def expires(self):
499+ parts = self.split('/')
500+ if len(parts) != 3:
501+ return None
502+ return find_datetime_string(parts[1])
503+
504+ @property
505+ def rand(self):
506+ parts = self.split('/')
507+ if len(parts) != 3:
508+ return None
509+ return parts[2]
510
511=== added directory 'lib/devscripts/ec2test/tests'
512=== added file 'lib/devscripts/ec2test/tests/__init__.py'
513--- lib/devscripts/ec2test/tests/__init__.py 1970-01-01 00:00:00 +0000
514+++ lib/devscripts/ec2test/tests/__init__.py 2009-10-02 12:34:16 +0000
515@@ -0,0 +1,2 @@
516+# Copyright 2009 Canonical Ltd. This software is licensed under the
517+# GNU Affero General Public License version 3 (see the file LICENSE).
518
519=== added file 'lib/devscripts/ec2test/tests/test_session.py'
520--- lib/devscripts/ec2test/tests/test_session.py 1970-01-01 00:00:00 +0000
521+++ lib/devscripts/ec2test/tests/test_session.py 2009-10-02 12:34:16 +0000
522@@ -0,0 +1,69 @@
523+# Copyright 2009 Canonical Ltd. This software is licensed under the
524+# GNU Affero General Public License version 3 (see the file LICENSE).
525+
526+"""Test the session module."""
527+
528+__metaclass__ = type
529+
530+import re
531+import unittest
532+
533+from datetime import datetime, timedelta
534+
535+from devscripts.ec2test import session
536+
537+
538+class TestEC2SessionName(unittest.TestCase):
539+ """Tests for EC2SessionName."""
540+
541+ def test_make(self):
542+ # EC2SessionName.make() is the most convenient way to create
543+ # valid names.
544+ name = session.EC2SessionName.make("fred")
545+ check = re.compile(
546+ r'^fred/\d{4}-\d{2}-\d{2}-\d{4}/[0-9a-zA-Z]{8}$').match
547+ self.failIf(check(name) is None, "Did not match %r" % name)
548+ possible_expires = [
549+ None, '1986-04-26-0123', timedelta(hours=10),
550+ datetime(1986, 04, 26, 1, 23)
551+ ]
552+ for expires in possible_expires:
553+ name = session.EC2SessionName.make("fred", expires)
554+ self.failIf(check(name) is None, "Did not match %r" % name)
555+
556+ def test_properties(self):
557+ # A valid EC2SessionName has properies to access the three
558+ # components of its name.
559+ base = "fred"
560+ timestamp = datetime(1986, 4, 26, 1, 23)
561+ timestamp_string = '1986-04-26-0123'
562+ rand = 'abcdef123456'
563+ name = session.EC2SessionName(
564+ "%s/%s/%s" % (base, timestamp_string, rand))
565+ self.failUnlessEqual(base, name.base)
566+ self.failUnlessEqual(timestamp, name.expires)
567+ self.failUnlessEqual(rand, name.rand)
568+
569+ def test_invalid_base(self):
570+ # If the given base contains a forward-slash, an
571+ # AssertionError should be raised.
572+ self.failUnlessRaises(
573+ AssertionError, session.EC2SessionName.make, "forward/slash")
574+
575+ def test_invalid_timestamp(self):
576+ # If the given expiry timestamp contains a forward-slash, an
577+ # AssertionError should be raised.
578+ self.failUnlessRaises(
579+ AssertionError, session.EC2SessionName.make, "fred", "/")
580+ # If the given expiry timestamp does not contain a timestamp
581+ # in the correct form, an AssertionError should be raised.
582+ self.failUnlessRaises(
583+ AssertionError, session.EC2SessionName.make, "fred", "1986.04.26")
584+
585+ def test_form_not_correct(self):
586+ # If the form of the string is not base/timestamp/rand then
587+ # the corresponding properties should all return None.
588+ broken_name = session.EC2SessionName('bob')
589+ self.failUnless(broken_name.base is None)
590+ self.failUnless(broken_name.expires is None)
591+ self.failUnless(broken_name.rand is None)
592
593=== added file 'lib/devscripts/ec2test/tests/test_utils.py'
594--- lib/devscripts/ec2test/tests/test_utils.py 1970-01-01 00:00:00 +0000
595+++ lib/devscripts/ec2test/tests/test_utils.py 2009-10-02 12:34:16 +0000
596@@ -0,0 +1,61 @@
597+# Copyright 2009 Canonical Ltd. This software is licensed under the
598+# GNU Affero General Public License version 3 (see the file LICENSE).
599+
600+"""Test the utils module."""
601+
602+__metaclass__ = type
603+
604+import unittest
605+
606+from datetime import datetime
607+
608+from devscripts.ec2test import utils
609+
610+
611+class TestDateTimeUtils(unittest.TestCase):
612+ """Tests for date/time related utilities."""
613+
614+ example_date = datetime(1986, 4, 26, 1, 23)
615+ example_date_string = '1986-04-26-0123'
616+ example_date_text = (
617+ 'blah blah foo blah 23545 646 ' +
618+ example_date_string + ' 435 blah')
619+
620+ def test_make_datetime_string(self):
621+ self.failUnlessEqual(
622+ self.example_date_string,
623+ utils.make_datetime_string(self.example_date))
624+
625+ def test_find_datetime_string(self):
626+ self.failUnlessEqual(
627+ self.example_date,
628+ utils.find_datetime_string(self.example_date_string))
629+ self.failUnlessEqual(
630+ self.example_date,
631+ utils.find_datetime_string(self.example_date_text))
632+
633+
634+class TestRandomUtils(unittest.TestCase):
635+ """Tests for randomness related utilities."""
636+
637+ hex_chars = frozenset('0123456789abcdefABCDEF')
638+
639+ def test_make_random_string(self):
640+ rand_a = utils.make_random_string()
641+ rand_b = utils.make_random_string()
642+ self.failIfEqual(rand_a, rand_b)
643+ self.failUnlessEqual(32, len(rand_a))
644+ self.failUnlessEqual(32, len(rand_b))
645+ self.failUnless(self.hex_chars.issuperset(rand_a))
646+ self.failUnless(self.hex_chars.issuperset(rand_b))
647+
648+ def test_make_random_string_with_length(self):
649+ for length in (8, 16, 64):
650+ rand = utils.make_random_string(length)
651+ self.failUnlessEqual(length, len(rand))
652+ self.failUnless(self.hex_chars.issuperset(rand))
653+
654+ def test_make_random_string_with_bad_length(self):
655+ # length must be a multiple of 2.
656+ self.failUnlessRaises(
657+ AssertionError, utils.make_random_string, 15)
658
659=== added file 'lib/devscripts/ec2test/utils.py'
660--- lib/devscripts/ec2test/utils.py 1970-01-01 00:00:00 +0000
661+++ lib/devscripts/ec2test/utils.py 2009-10-02 12:34:16 +0000
662@@ -0,0 +1,55 @@
663+# Copyright 2009 Canonical Ltd. This software is licensed under the
664+# GNU Affero General Public License version 3 (see the file LICENSE).
665+
666+"""General useful stuff."""
667+
668+__metaclass__ = type
669+__all__ = [
670+ 'find_datetime_string',
671+ 'make_datetime_string',
672+ 'make_random_string',
673+ ]
674+
675+
676+import binascii
677+import datetime
678+import os
679+import re
680+
681+
682+def make_datetime_string(when=None):
683+ """Generate a simple formatted date and time string.
684+
685+ This is intended to be embedded in text to be later found by
686+ `find_datetime_string`.
687+ """
688+ if when is None:
689+ when = datetime.datetime.utcnow()
690+ return when.strftime('%Y-%m-%d-%H%M')
691+
692+
693+re_find_datetime = re.compile(
694+ r'(\d{4})-(\d{2})-(\d{2})-(\d{2})(\d{2})')
695+
696+def find_datetime_string(text):
697+ """Search for a simple date and time in arbitrary text.
698+
699+ The format searched for is %Y-%m-%d-%H%M - the same as produced by
700+ `make_datetime_string`.
701+ """
702+ match = re_find_datetime.search(text)
703+ if match is None:
704+ return None
705+ else:
706+ return datetime.datetime(
707+ *(int(part) for part in match.groups()))
708+
709+
710+def make_random_string(length=32):
711+ """Return a simple random UUID.
712+
713+ The uuid module is only available in Python 2.5 and above, but a
714+ simple non-RFC-compliant hack here is sufficient.
715+ """
716+ assert length % 2 == 0, "length must be a multiple of 2"
717+ return binascii.hexlify(os.urandom(length/2))