Merge lp:~frankban/launchpad/lpsetup-initial into lp:launchpad

Proposed by Francesco Banconi
Status: Rejected
Rejected by: William Grant
Proposed branch: lp:~frankban/launchpad/lpsetup-initial
Merge into: lp:launchpad
Diff against target: 1734 lines (+1730/-0)
1 file modified
utilities/lpsetup.py (+1730/-0)
To merge this branch: bzr merge lp:~frankban/launchpad/lpsetup-initial
Reviewer Review Type Date Requested Status
Gary Poster (community) Approve
Review via email: mp+94765@code.launchpad.net

Description of the change

= Summary =

This is the first step of *lpsetup* implementation.
It's basically a Python script created to eliminate code duplication between
*setuplxc* and *rocketfuel-setup/get/branch*.

PS: It's a brand new huge file, sorry about that.

== Proposed fix ==

Actually the script can be used to set up a Launchpad development or testing
environment in a real machine or inside an LXC container, with ability to
update the codebase once installed.

The next steps will be, in random order:

- implement lxc-update sub command: update the codebase inside a running or
  stopped LXC
- implement branch and lxc-branch sub commands: see rocketfuel-branch
- ability to specify a custom ssh key name to connect to LXC
- create a real *lpsetup* project, splitting code in separate files, etc.
- deb package?
- ec2 tests, tests, tests

== Implementation details ==

While *setuplxc* sets up the LXC connecting via ssh to the container,
*lpsetup* uses a different approach: the script calls itself using an ssh
connection to the container. This is possible thanks to the script ability
to be run specifying *actions*, i.e. a subset of internal functions to be
called. This way, the LXC can initialize itself and the code is reused.

*lpsetup* improves some of the functionalities present in *setuplxc*:

- errors in executing external commands are caught and reported, and they
  cause the script to stop running
- apt repositories are added in a gentler way, using *add-apt-repository*
  (this allows to get rid of `apt-get install --allow-unauthenticated`
  and needs to be back-ported to *setuplxc*)
- some helper objects were fixed and some others were added

A thin subcommands management layer (`BaseSubCommand`) was implemented to
allow extensibility.

Because the script can be executed in lucid containers,
it is now compatible with Python 2.6.

== Tests ==

python -m doctest -v lpsetup.py

To post a comment you must log in.
14876. By Francesco Banconi

Added tests to generate_ssh_keys helper.

Revision history for this message
Gary Poster (gary) wrote :
Download full text (3.6 KiB)

Hey Francesco. Thanks for doing this! This is a nice slack time project.

This review is through line 895. I will continue on this after lunch.

In join_command's docstring, there's a small typo, twice: containig -> containing.

Spelling niggle: in environ's docstring ("A context manager to temporary change environment variables."), change "temporary" to "temporarily". Same for su docstring ("A context manager to temporary run the script as a different user.").

Minor grammar niggle: in docstring of check_output, you need to convert the comma to one of three choices: semicolon, ", and", or period (new sentence): "The first argument is the path to the command to run, subsequent arguments are command-line arguments to be passed."

Unless you give me a good reason why not, I'd like all of the contextmanagers to have a try:finally block placed around the yield. environ and cd are the only ones without it, I think.

Why do you need your own check_output, rather than using subprocess' check_output? Ah! Because it is not in Python 2.6? If that's the reason, what would you think of trying to get it from the module, and if there's an AttributeError (or ImportError, however you want to work it) then you define your function? That way we get the newer version, with possible bugfixes, if it exists; and we have a chance to clarify why we are not using the stdlib's function.

I had never used the pipes module. The quotes function is small and convenient. I'm mildly concerned that it is not part of the module's documented API (http://docs.python.org/library/pipes.html). That stuff stays pretty stable across Python IME though; I'm fine with it.

So many of these helpers are generically useful. It's a shame they are not somewhere more central. Many of them look helpful for charms too. I don't have a suggested action for that; it's just an observation.

As we discussed already, in your shoes people have advocated to me that I should use bzr's command framework (and I mentioned Jamu Kakar's commandant; that project hasn't seen any updates, but it looks like Landscape is still using it because they have done some recent packaging work). I like the fact that what you have is self-contained and small. Again, I merely offer that as an avenue that I suggest you investigate in the future.

I look forward to this being a separate project with an ability to divide things up into support files and so on. In addition to making it possible to organize the code more obviously, it also might then be reasonable to reconsider the use of doctests. I am among the apparent minority that like them in the Python community, and I think what you have here is well done, but our team's standards are to use unit tests. As long as this is a single downloadable file, doctests make enough sense to me to be a defensible choice. Later, maybe less so.

Random thought: I was thinking about your other branch that had LANG=C. Probably obvious, but you could change apt_get_install to accept keyword arguments and include those as environment variables so you could easily do the LANG=C thing, maybe? That maybe looks too much like command options. Just a thought.

As I'm sure...

Read more...

Revision history for this message
Francesco Banconi (frankban) wrote :
Download full text (3.4 KiB)

Thank you Gary for reviewing this, it's a huge file.

> Unless you give me a good reason why not, I'd like all of the contextmanagers
> to have a try:finally block placed around the yield. environ and cd are the
> only ones without it, I think.

No good reason, I will add the try/finally block to them.

> Why do you need your own check_output, rather than using subprocess'
> check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> what would you think of trying to get it from the module, and if there's an
> AttributeError (or ImportError, however you want to work it) then you define
> your function? That way we get the newer version, with possible bugfixes, if
> it exists; and we have a chance to clarify why we are not using the stdlib's
> function.

That's one of the reasons, the others, like the first, aren't essential, just nice to have:
- the exception raised on errors is the one subcommand handlers trap (but of course I could also trap subprocess.CalledProcessError)
- the positional arguments thing (really not even a "nice to have", more a "look, we're lazy and cool")
- None is ignored if given as argument, so that you can write `check_output('ls', '-l' if something else None)
- the error contains a better representation of the executed command

I can get rid of check_output; my point is: like `subprocess.check_output`, it's just a little wrapper around `subprocess.Popen`, that does the real stuff (and we will not miss future bugfixes for that object).

> So many of these helpers are generically useful. It's a shame they are not
> somewhere more central. Many of them look helpful for charms too. I don't
> have a suggested action for that; it's just an observation.

I agree.

> As we discussed already, in your shoes people have advocated to me that I
> should use bzr's command framework (and I mentioned Jamu Kakar's commandant;
> that project hasn't seen any updates, but it looks like Landscape is still
> using it because they have done some recent packaging work). I like the fact
> that what you have is self-contained and small. Again, I merely offer that as
> an avenue that I suggest you investigate in the future.

Thanks Gary, sure, I will take a look at commandant, although I like my tiny subcommand objects.

> I look forward to this being a separate project with an ability to divide
> things up into support files and so on. In addition to making it possible to
> organize the code more obviously, it also might then be reasonable to
> reconsider the use of doctests.

Yes, you are right, I will add a slack time card for unit tests.

>
> Random thought: I was thinking about your other branch that had LANG=C.
> Probably obvious, but you could change apt_get_install to accept keyword
> arguments and include those as environment variables so you could easily do
> the LANG=C thing, maybe? That maybe looks too much like command options.
> Just a thought.

Nice idea IMHO.

> As I'm sure you've realized, we should use generate_ssh_keys with descriptive
> names per the webops request for setuplxc. If these become merely
> "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> want to know ...

Read more...

Revision history for this message
Gary Poster (gary) wrote :

Continuing on from where I left off...

I suggest deleting update_launchpad_lxc for now, though I see why you have it. Up to you.

You have your file nicely divided in sections by function. If you were going to keep everything in this one file, I'd ask for comment lines delineating the sections--in particular, I had to think to realize that we were in the arg parsing section when I saw handle_user. Since you are going to divide this up into files anyway, I won't worry about it now.

You are not using os.path.expanduser in handle_directories for a reason: you are using the home directory of the user that was passed in, not the current home directory. A comment explaining this might be helpful, or might not.

The ArgumentParser might be improved if it showed in the docstring how users would use the registered subcommands. I expect that kind of thing might fall out of the division into separate files.

I find the interaction between ArgumentParser (the actions attribute) and ActionsBasedSubCommand to not be obvious. I thought about it, and the main thing I found confusing was BaseSubCommand. ArgumentParser, AFAICT, has code specifically for ActionsBasedSubCommand, so it seems like it is specific to that kind of SubCommand. When would one actually use BaseSubCommand? The fact that we are not using the BaseSubCommand directly reinforces this feeling. Maybe a bit of elaboration would explain the value of having a separate BaseSubCommand, and not merging it directly with ActionsBasedSubCommand. In your cover letter you say it is for extensibility, but that does not convince me.

When I run the command's help, I get the following output:

"""
$ ./lpsetup.py -h
usage: lpsetup.py [-h] {install,update,lxc-install} ...

Create and update Launchpad development and testing environments.

optional arguments:
  -h, --help show this help message and exit

subcommands:
  valid subcommands

  {install,update,lxc-install}
                        -h, --help show subcommand help and exit
    install Install the Launchpad environment.
    update Update the Launchpad environment to latest version.
    lxc-install Install the Launchpad environment inside an LXC.
"""
- I find the text "valid subcommands {install,update,lxc-install}" to be unnecessary and confusing.
- I find the placement of "-h, --help..." to be confusing too. Perhaps put that text, written more explicitly, where the "calid subcommands" text is? "Each subcommand accepts --h or --help to describe it."? Also, this reminds me, and probably other people who will use lpsetup, of bzr. They will want to do the following:

$ lpsetup help
(this could be equivalent to lpsetup -h)

$ lpsetup help install
(this could be equivalent to lpsetup install -h)
- The first (usage) line looks like you ought to be able to write "lpsetup -h install". You can, but it shows the top-level help--not what I'd expect. I think this usage line is confusing. Ideally, I think, the usage would be "lpsetup {help,install,update,lxc-install} ..." or something like that.

That's it. Nice progress! Thank you.

review: Approve
Revision history for this message
Gary Poster (gary) wrote :

...
> > Why do you need your own check_output, rather than using subprocess'
> > check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> > what would you think of trying to get it from the module, and if there's an
> > AttributeError (or ImportError, however you want to work it) then you define
> > your function? That way we get the newer version, with possible bugfixes,
> if
> > it exists; and we have a chance to clarify why we are not using the stdlib's
> > function.
...
> I can get rid of check_output; my point is: like `subprocess.check_output`,
> it's just a little wrapper around `subprocess.Popen`, that does the real stuff
> (and we will not miss future bugfixes for that object).

Keep it. I think you ought to have a comment that explains why it is valuable. That's it.

...

> > As I'm sure you've realized, we should use generate_ssh_keys with
> descriptive
> > names per the webops request for setuplxc. If these become merely
> > "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> > want to know "why" the key is used (e.g., "lxc_communication_key" or
> similar).
> > If there's not a good name for this key that would distinguish it from other
> > keys a user might have, we should push back; but AFAICT there should be a
> good
> > name here.
>
> I propose 'id_for_lxc'... I know: we can do better.

"launchpad_lxc_id" perhaps? Getting the word "launchpad" in there would be valuable too, I think.

Thanks again

Gary

Revision history for this message
Robert Collins (lifeless) wrote :

Hi, a few points for you to consider:
 - this might be a lot better off in the lp-dev-tools project, its
clearly not part of LP itself. That would also make it easier to split
it into N files as you would not be stuck in the bootstrap stage of
the users environment (we could package that and mandate the package
be installed).
 - the environ context manager is duplicative of the
EnvironmentVariable fixture; it would be nice not to rewrite code that
has already been written and tested (and is used in our environment
too!) Yes, I know that this is small, but it does all add up, and
someone somewhere will eventually be cursing that they have to redo
this / fix a unicode escaping issue in the code or whatever.

 - I'm not fussed about reusing commandant or the bzrlib UI framework;
if I was writing something new that needed subcommands I would do it
differently than I did in bzrlib (and in fact I have since - with a
leaner, faster, more testable and machine drivable system in testr).
If you wanted to reuse that, its probably not quite there for reuse,
but we could split it into framework-and-user and make it so.

HTH,
Rob

Revision history for this message
Francesco Banconi (frankban) wrote :

Thank you for the review Gary. I agree with your suggestions and I will reply only to your doubts concerning BaseSubcommand.

> I find the interaction between ArgumentParser (the actions attribute) and
> ActionsBasedSubCommand to not be obvious. I thought about it, and the main
> thing I found confusing was BaseSubCommand. ArgumentParser, AFAICT, has code
> specifically for ActionsBasedSubCommand, so it seems like it is specific to
> that kind of SubCommand. When would one actually use BaseSubCommand? The
> fact that we are not using the BaseSubCommand directly reinforces this
> feeling. Maybe a bit of elaboration would explain the value of having a
> separate BaseSubCommand, and not merging it directly with
> ActionsBasedSubCommand. In your cover letter you say it is for extensibility,
> but that does not convince me.

ArgumentParser ignores the concept of *actions* as intended in ActionBasedSubcommand.
The register_subcommand method just:
- instantiates a sub command object
- calls subcommand.add_arguments to collect subcommand args
- stores a reference to subcommand.main that can be used later

This interface is implemented by BaseSubCommand. ActionBasedSubcommand is a specialization only useful when a sub command is subdivided in steps (internal functions) and we want to expose them in the ui.

I think the misunderstanding comes from the different meaning of the word *action* in argparse and ActionBasedSubcommand.
In argparse an action is an object containing info about a single argument (argparse.Action). Actions are stored in argparse.ArgumentParser._actions and returned by argparse.ArgumentParser.add_argument(...). In my ArgumentParser subclass I store those actions in a "public" attribute because I need to reuse them later: I preferred to populate a new attribute over using a "private" one (like in setuplxc).

I will agree if you suggest to change the name of ArgumentParser.actions to avoid confusion.

Revision history for this message
Francesco Banconi (frankban) wrote :

> - this might be a lot better off in the lp-dev-tools project, its
> clearly not part of LP itself. That would also make it easier to split
> it into N files as you would not be stuck in the bootstrap stage of
> the users environment (we could package that and mandate the package
> be installed).

I agree. Indeed we were thinking about creating an lpsetup project.
In general I think it's a good idea to have this separated from
launchpad itself and splitted into N files.

> - the environ context manager is duplicative of the
> EnvironmentVariable fixture; it would be nice not to rewrite code that
> has already been written and tested (and is used in our environment
> too!) Yes, I know that this is small, but it does all add up, and
> someone somewhere will eventually be cursing that they have to redo
> this / fix a unicode escaping issue in the code or whatever.

Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
Once this no longer is a single file, it will be easy to add dependencies.

Thanks again Robert.

Revision history for this message
Robert Collins (lifeless) wrote :

On Tue, Feb 28, 2012 at 11:47 PM, Francesco Banconi <email address hidden> wrote:
> Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
> Once this no longer is a single file, it will be easy to add dependencies.
>
> Thanks again Robert.

No probs! And seriously, feel free to ignore the testr subcommand bit
- as I mentioned, its likely still a bit too opinionated for reuse,
and we should focus on delivering this lovely improvement in the short
term. If its in lp-dev-tools (or a new project), the complexity of it
will be contained, and you can iterate on this sort of polish in later
steps. (Or do it in whatever order Gary prefers: I'm trying to leave
it wide open what you do).

-Rob

14877. By Francesco Banconi

Fixed spelling errors.

14878. By Francesco Banconi

Try/Finally blocks in context managers.

14879. By Francesco Banconi

Removed unused function.

14880. By Francesco Banconi

Help fix.

14881. By Francesco Banconi

First implementation of the help subcommand.

Unmerged revisions

14881. By Francesco Banconi

First implementation of the help subcommand.

14880. By Francesco Banconi

Help fix.

14879. By Francesco Banconi

Removed unused function.

14878. By Francesco Banconi

Try/Finally blocks in context managers.

14877. By Francesco Banconi

Fixed spelling errors.

14876. By Francesco Banconi

Added tests to generate_ssh_keys helper.

14875. By Francesco Banconi

Initial commit.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'utilities/lpsetup.py'
2--- utilities/lpsetup.py 1970-01-01 00:00:00 +0000
3+++ utilities/lpsetup.py 2012-02-28 17:19:19 +0000
4@@ -0,0 +1,1730 @@
5+#!/usr/bin/env python
6+# Copyright 2012 Canonical Ltd. This software is licensed under the
7+# GNU Affero General Public License version 3 (see the file LICENSE).
8+
9+"""Create and update Launchpad development and testing environments."""
10+
11+__metaclass__ = type
12+__all__ = [
13+ 'ActionsBasedSubCommand',
14+ 'apt_get_install',
15+ 'ArgumentParser',
16+ 'BaseSubCommand',
17+ 'bzr_whois',
18+ 'call',
19+ 'cd',
20+ 'check_output',
21+ 'CommandError',
22+ 'create_lxc',
23+ 'environ',
24+ 'file_append',
25+ 'file_prepend',
26+ 'generate_ssh_keys',
27+ 'get_container_path',
28+ 'get_lxc_gateway',
29+ 'get_su_command',
30+ 'get_user_home',
31+ 'get_user_ids',
32+ 'initialize',
33+ 'InstallSubCommand',
34+ 'join_command',
35+ 'LaunchpadError',
36+ 'link_sourcecode_in_branches',
37+ 'LXCInstallSubCommand',
38+ 'lxc_in_state',
39+ 'lxc_running',
40+ 'lxc_stopped',
41+ 'make_launchpad',
42+ 'mkdirs',
43+ 'setup_apt',
44+ 'setup_codebase',
45+ 'setup_external_sourcecode',
46+ 'setup_launchpad',
47+ 'setup_launchpad_lxc',
48+ 'ssh',
49+ 'SSHError',
50+ 'start_lxc',
51+ 'stop_lxc',
52+ 'su',
53+ 'this_command',
54+ 'update_launchpad',
55+ 'user_exists',
56+ 'ValidationError',
57+ 'wait_for_lxc',
58+ ]
59+
60+# To run doctests: python -m doctest -v lpsetup.py
61+
62+from collections import namedtuple
63+from contextlib import contextmanager
64+from email.Utils import parseaddr, formataddr
65+from functools import partial
66+import argparse
67+import errno
68+import os
69+import pipes
70+import platform
71+import pwd
72+import shutil
73+import subprocess
74+import sys
75+import time
76+
77+
78+APT_REPOSITORIES = (
79+ 'deb http://archive.ubuntu.com/ubuntu {distro} multiverse',
80+ 'deb http://archive.ubuntu.com/ubuntu {distro}-updates multiverse',
81+ 'deb http://archive.ubuntu.com/ubuntu {distro}-security multiverse',
82+ 'ppa:launchpad/ppa',
83+ 'ppa:bzr/ppa',
84+ )
85+BASE_PACKAGES = ['ssh', 'bzr', 'language-pack-en']
86+CHECKOUT_DIR = '~/launchpad/branches'
87+DEPENDENCIES_DIR = '~/launchpad/dependencies'
88+DHCP_FILE = '/etc/dhcp/dhclient.conf'
89+HOSTS_CONTENT = (
90+ ('127.0.0.88',
91+ 'launchpad.dev answers.launchpad.dev archive.launchpad.dev '
92+ 'api.launchpad.dev bazaar-internal.launchpad.dev beta.launchpad.dev '
93+ 'blueprints.launchpad.dev bugs.launchpad.dev code.launchpad.dev '
94+ 'feeds.launchpad.dev id.launchpad.dev keyserver.launchpad.dev '
95+ 'lists.launchpad.dev openid.launchpad.dev '
96+ 'ubuntu-openid.launchpad.dev ppa.launchpad.dev '
97+ 'private-ppa.launchpad.dev testopenid.dev translations.launchpad.dev '
98+ 'xmlrpc-private.launchpad.dev xmlrpc.launchpad.dev'),
99+ ('127.0.0.99', 'bazaar.launchpad.dev'),
100+ )
101+HOSTS_FILE = '/etc/hosts'
102+LP_APACHE_MODULES = 'proxy proxy_http rewrite ssl deflate headers'
103+LP_APACHE_ROOTS = (
104+ '/var/tmp/bazaar.launchpad.dev/static',
105+ '/var/tmp/bazaar.launchpad.dev/mirrors',
106+ '/var/tmp/archive',
107+ '/var/tmp/ppa',
108+ )
109+LP_BZR_LOCATIONS = (
110+ ('submit_branch', '{checkout_dir}'),
111+ ('public_branch', 'bzr+ssh://bazaar.launchpad.net/~{lpuser}/launchpad'),
112+ ('public_branch:policy', 'appendpath'),
113+ ('push_location', 'lp:~{lpuser}/launchpad'),
114+ ('push_location:policy', 'appendpath'),
115+ ('merge_target', '{checkout_dir}'),
116+ ('submit_to', 'merge@code.launchpad.net'),
117+ )
118+LP_CHECKOUT = 'devel'
119+LP_PACKAGES = [
120+ 'bzr', 'launchpad-developer-dependencies', 'apache2',
121+ 'apache2-mpm-worker', 'libapache2-mod-wsgi'
122+ ]
123+LP_REPOS = (
124+ 'http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel',
125+ 'lp:launchpad',
126+ )
127+LP_SOURCE_DEPS = (
128+ 'http://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk')
129+LXC_CONFIG_TEMPLATE = '/etc/lxc/local.conf'
130+LXC_GUEST_ARCH = 'i386'
131+LXC_GUEST_CHOICES = ('lucid', 'oneiric', 'precise')
132+LXC_GUEST_OS = LXC_GUEST_CHOICES[0]
133+LXC_NAME = 'lptests'
134+LXC_OPTIONS = """
135+lxc.network.type = veth
136+lxc.network.link = {interface}
137+lxc.network.flags = up
138+"""
139+LXC_PACKAGES = ['lxc', 'libvirt-bin']
140+LXC_PATH = '/var/lib/lxc/'
141+RESOLV_FILE = '/etc/resolv.conf'
142+
143+
144+Env = namedtuple('Env', 'uid gid home')
145+
146+
147+class LaunchpadError(Exception):
148+ """Base exception for lpsetup."""
149+
150+
151+class CommandError(LaunchpadError):
152+ """Errors occurred running shell commands."""
153+
154+
155+class SSHError(LaunchpadError):
156+ """Errors occurred during SSH connection."""
157+
158+
159+class ValidationError(LaunchpadError):
160+ """Argparse invalid arguments."""
161+
162+
163+def apt_get_install(*args):
164+ """Install given packages using apt."""
165+ with environ(DEBIAN_FRONTEND='noninteractive'):
166+ cmd = ('apt-get', '-y', 'install') + args
167+ return call(*cmd)
168+
169+
170+def join_command(args):
171+ """Return a valid Unix command line from `args`.
172+
173+ >>> join_command(['ls', '-l'])
174+ 'ls -l'
175+
176+ Arguments containing spaces and empty args are correctly quoted::
177+
178+ >>> join_command(['command', 'arg1', 'arg containing spaces', ''])
179+ "command arg1 'arg containing spaces' ''"
180+ """
181+ return ' '.join(pipes.quote(arg) for arg in args)
182+
183+
184+def bzr_whois(user):
185+ """Return full name and email of bzr `user`.
186+
187+ Return None if the given `user` does not have a bzr user id.
188+ """
189+ with su(user):
190+ try:
191+ whoami = check_output('bzr', 'whoami')
192+ except (CommandError, OSError):
193+ return None
194+ return parseaddr(whoami)
195+
196+
197+@contextmanager
198+def cd(directory):
199+ """A context manager to temporarily change current working dir, e.g.::
200+
201+ >>> import os
202+ >>> os.chdir('/tmp')
203+ >>> with cd('/bin'): print os.getcwd()
204+ /bin
205+ >>> print os.getcwd()
206+ /tmp
207+ """
208+ cwd = os.getcwd()
209+ os.chdir(directory)
210+ try:
211+ yield
212+ finally:
213+ os.chdir(cwd)
214+
215+
216+def check_output(*args, **kwargs):
217+ r"""Run the command with the given arguments.
218+
219+ The first argument is the path to the command to run.
220+ Subsequent arguments are command-line arguments to be passed.
221+
222+ Usually the output is captured by a pipe and returned::
223+
224+ >>> check_output('echo', 'output')
225+ 'output\n'
226+
227+ A `CommandError` exception is raised if the return code is not zero::
228+
229+ >>> check_output('ls', '--not a valid switch', stderr=subprocess.PIPE)
230+ Traceback (most recent call last):
231+ CommandError: Error running: ls '--not a valid switch'
232+
233+ None arguments are ignored::
234+
235+ >>> check_output(None, 'echo', None, 'output')
236+ 'output\n'
237+ """
238+ args = [i for i in args if i is not None]
239+ process = subprocess.Popen(
240+ args, stdout=kwargs.pop('stdout', subprocess.PIPE),
241+ close_fds=True, **kwargs)
242+ stdout, stderr = process.communicate()
243+ retcode = process.poll()
244+ if retcode:
245+ raise CommandError('Error running: ' + join_command(args))
246+ return stdout
247+
248+
249+call = partial(check_output, stdout=None)
250+
251+
252+@contextmanager
253+def environ(**kwargs):
254+ """A context manager to temporarily change environment variables.
255+
256+ If an existing environment variable is changed, it is restored during
257+ context cleanup::
258+
259+ >>> import os
260+ >>> os.environ['MY_VARIABLE'] = 'foo'
261+ >>> with environ(MY_VARIABLE='bar'): print os.getenv('MY_VARIABLE')
262+ bar
263+ >>> print os.getenv('MY_VARIABLE')
264+ foo
265+ >>> del os.environ['MY_VARIABLE']
266+
267+ If we are adding environment variables, they are removed during context
268+ cleanup::
269+
270+ >>> import os
271+ >>> with environ(MY_VAR1='foo', MY_VAR2='bar'):
272+ ... print os.getenv('MY_VAR1'), os.getenv('MY_VAR2')
273+ foo bar
274+ >>> os.getenv('MY_VAR1') == os.getenv('MY_VAR2') == None
275+ True
276+ """
277+ backup = {}
278+ for key, value in kwargs.items():
279+ backup[key] = os.getenv(key)
280+ os.environ[key] = value
281+ try:
282+ yield
283+ finally:
284+ for key, value in backup.items():
285+ if value is None:
286+ del os.environ[key]
287+ else:
288+ os.environ[key] = value
289+
290+
291+def file_append(filename, line):
292+ r"""Append given `line`, if not present, at the end of `filename`.
293+
294+ Usage example::
295+
296+ >>> import tempfile
297+ >>> f = tempfile.NamedTemporaryFile('w', delete=False)
298+ >>> f.write('line1\n')
299+ >>> f.close()
300+ >>> file_append(f.name, 'new line\n')
301+ >>> open(f.name).read()
302+ 'line1\nnew line\n'
303+
304+ Nothing happens if the file already contains the given `line`::
305+
306+ >>> file_append(f.name, 'new line\n')
307+ >>> open(f.name).read()
308+ 'line1\nnew line\n'
309+
310+ A new line is automatically added before the given `line` if it is not
311+ present at the end of current file content::
312+
313+ >>> import tempfile
314+ >>> f = tempfile.NamedTemporaryFile('w', delete=False)
315+ >>> f.write('line1')
316+ >>> f.close()
317+ >>> file_append(f.name, 'new line\n')
318+ >>> open(f.name).read()
319+ 'line1\nnew line\n'
320+
321+ The file is created if it does not exist::
322+
323+ >>> import tempfile
324+ >>> filename = tempfile.mktemp()
325+ >>> file_append(filename, 'line1\n')
326+ >>> open(filename).read()
327+ 'line1\n'
328+ """
329+ with open(filename, 'a+') as f:
330+ content = f.read()
331+ if line not in content:
332+ if content.endswith('\n') or not content:
333+ f.write(line)
334+ else:
335+ f.write('\n' + line)
336+
337+
338+def file_prepend(filename, line):
339+ r"""Insert given `line`, if not present, at the beginning of `filename`.
340+
341+ Usage example::
342+
343+ >>> import tempfile
344+ >>> f = tempfile.NamedTemporaryFile('w', delete=False)
345+ >>> f.write('line1\n')
346+ >>> f.close()
347+ >>> file_prepend(f.name, 'line0\n')
348+ >>> open(f.name).read()
349+ 'line0\nline1\n'
350+
351+ If the file starts with the given `line`, nothing happens::
352+
353+ >>> file_prepend(f.name, 'line0\n')
354+ >>> open(f.name).read()
355+ 'line0\nline1\n'
356+
357+ If the file contains the given `line`, but not at the beginning,
358+ the line is moved on top::
359+
360+ >>> file_prepend(f.name, 'line1\n')
361+ >>> open(f.name).read()
362+ 'line1\nline0\n'
363+ """
364+ with open(filename, 'r+') as f:
365+ lines = f.readlines()
366+ if lines[0] != line:
367+ if line in lines:
368+ lines.remove(line)
369+ lines.insert(0, line)
370+ f.seek(0)
371+ f.writelines(lines)
372+
373+
374+def generate_ssh_keys(directory, filename='id_rsa'):
375+ """Generate ssh key pair, saving them inside the given `directory`.
376+
377+ >>> generate_ssh_keys('/tmp/')
378+ >>> open('/tmp/id_rsa').readlines()[0].strip()
379+ '-----BEGIN RSA PRIVATE KEY-----'
380+ >>> open('/tmp/id_rsa.pub').read().startswith('ssh-rsa')
381+ True
382+ >>> os.remove('/tmp/id_rsa')
383+ >>> os.remove('/tmp/id_rsa.pub')
384+
385+ The key filename can be changed using the `filename` keyword argument
386+ (default is 'id_rsa')::
387+
388+ >>> generate_ssh_keys('/tmp/', 'custom_key')
389+ >>> os.path.exists('/tmp/custom_key')
390+ True
391+ >>> os.path.exists('/tmp/id_rsa')
392+ False
393+
394+ >>> os.remove('/tmp/custom_key')
395+ >>> os.remove('/tmp/custom_key.pub')
396+ """
397+ path = os.path.join(directory, filename)
398+ return call('ssh-keygen', '-q', '-t', 'rsa', '-N', '', '-f', path)
399+
400+
401+def get_container_path(lxc_name, path='', base_path=LXC_PATH):
402+ """Return the path of LXC container called `lxc_name`.
403+
404+ If a `path` is given, return that path inside the container, e.g.::
405+
406+ >>> get_container_path('mycontainer')
407+ '/var/lib/lxc/mycontainer/rootfs/'
408+ >>> get_container_path('mycontainer', '/etc/apt/')
409+ '/var/lib/lxc/mycontainer/rootfs/etc/apt/'
410+ >>> get_container_path('mycontainer', 'home')
411+ '/var/lib/lxc/mycontainer/rootfs/home'
412+ """
413+ return os.path.join(base_path, lxc_name, 'rootfs', path.lstrip('/'))
414+
415+
416+def get_lxc_gateway():
417+ """Return a tuple of gateway name and address.
418+
419+ The gateway name and address will change depending on which version
420+ of Ubuntu the script is running on.
421+ """
422+ release_name = platform.linux_distribution()[2]
423+ if release_name == 'oneiric':
424+ return 'virbr0', '192.168.122.1'
425+ else:
426+ return 'lxcbr0', '10.0.3.1'
427+
428+
429+def get_su_command(user, args):
430+ """Return a command line as a sequence, prepending "su" if necessary.
431+
432+ This can be used together with `call` or `check_output` when the `su`
433+ context manager is not enaugh (e.g. an external program uses uid rather
434+ than euid).
435+
436+ >>> import getpass
437+ >>> current_user = getpass.getuser()
438+
439+ If the su is requested as current user, the arguments are returned as
440+ given::
441+
442+ >>> get_su_command(current_user, ('ls', '-l'))
443+ ('ls', '-l')
444+
445+ Otherwise, "su" is prepended::
446+
447+ >>> get_su_command('nobody', ('ls', '-l', 'my file'))
448+ ('su', 'nobody', '-c', "ls -l 'my file'")
449+ """
450+ if get_user_ids(user)[0] != os.getuid():
451+ args = [i for i in args if i is not None]
452+ return ('su', user, '-c', join_command(args))
453+ return args
454+
455+
456+def get_user_home(user):
457+ """Return the home directory of the given `user`.
458+
459+ >>> get_user_home('root')
460+ '/root'
461+
462+ If the user does not exist, return a default /home/[username] home::
463+
464+ >>> get_user_home('_this_user_does_not_exist_')
465+ '/home/_this_user_does_not_exist_'
466+ """
467+ try:
468+ return pwd.getpwnam(user).pw_dir
469+ except KeyError:
470+ return os.path.join(os.path.sep, 'home', user)
471+
472+
473+def get_user_ids(user):
474+ """Return the uid and gid of given `user`, e.g.::
475+
476+ >>> get_user_ids('root')
477+ (0, 0)
478+ """
479+ userdata = pwd.getpwnam(user)
480+ return userdata.pw_uid, userdata.pw_gid
481+
482+
483+def lxc_in_state(state, lxc_name, timeout=30):
484+ """Return True if the LXC named `lxc_name` is in state `state`.
485+
486+ Return False otherwise.
487+ """
488+ while timeout:
489+ try:
490+ output = check_output(
491+ 'lxc-info', '-n', lxc_name, stderr=subprocess.STDOUT)
492+ except CommandError:
493+ pass
494+ else:
495+ if state in output:
496+ return True
497+ timeout -= 1
498+ time.sleep(1)
499+ return False
500+
501+
502+lxc_running = partial(lxc_in_state, 'RUNNING')
503+lxc_stopped = partial(lxc_in_state, 'STOPPED')
504+
505+
506+def mkdirs(*args):
507+ """Create leaf directories (given as `args`) and all intermediate ones.
508+
509+ >>> import tempfile
510+ >>> base_dir = tempfile.mktemp(suffix='/')
511+ >>> dir1 = tempfile.mktemp(prefix=base_dir)
512+ >>> dir2 = tempfile.mktemp(prefix=base_dir)
513+ >>> mkdirs(dir1, dir2)
514+ >>> os.path.isdir(dir1)
515+ True
516+ >>> os.path.isdir(dir2)
517+ True
518+
519+ If the leaf directory already exists the function returns without errors::
520+
521+ >>> mkdirs(dir1)
522+
523+ An `OSError` is raised if the leaf path exists and it is a file::
524+
525+ >>> f = tempfile.NamedTemporaryFile(
526+ ... 'w', delete=False, prefix=base_dir)
527+ >>> f.close()
528+ >>> mkdirs(f.name) # doctest: +ELLIPSIS
529+ Traceback (most recent call last):
530+ OSError: ...
531+ """
532+ for directory in args:
533+ try:
534+ os.makedirs(directory)
535+ except OSError as err:
536+ if err.errno != errno.EEXIST or os.path.isfile(directory):
537+ raise
538+
539+
540+def ssh(location, user=None, caller=subprocess.call):
541+ """Return a callable that can be used to run ssh shell commands.
542+
543+ The ssh `location` and, optionally, `user` must be given.
544+ If the user is None then the current user is used for the connection.
545+
546+ The callable internally uses the given `caller`::
547+
548+ >>> def caller(cmd):
549+ ... print cmd
550+ >>> sshcall = ssh('example.com', 'myuser', caller=caller)
551+ >>> root_sshcall = ssh('example.com', caller=caller)
552+ >>> sshcall('ls -l') # doctest: +ELLIPSIS
553+ ('ssh', '-t', ..., 'myuser@example.com', '--', 'ls -l')
554+ >>> root_sshcall('ls -l') # doctest: +ELLIPSIS
555+ ('ssh', '-t', ..., 'example.com', '--', 'ls -l')
556+
557+ If the ssh command exits with an error code, an `SSHError` is raised::
558+
559+ >>> ssh('loc', caller=lambda cmd: 1)('ls -l') # doctest: +ELLIPSIS
560+ Traceback (most recent call last):
561+ SSHError: ...
562+
563+ If ignore_errors is set to True when executing the command, no error
564+ will be raised, even if the command itself returns an error code.
565+
566+ >>> sshcall = ssh('loc', caller=lambda cmd: 1)
567+ >>> sshcall('ls -l', ignore_errors=True)
568+ """
569+ if user is not None:
570+ location = '{user}@{location}'.format(user=user, location=location)
571+
572+ def _sshcall(cmd, ignore_errors=False):
573+ sshcmd = (
574+ 'ssh',
575+ '-t',
576+ '-t', # Yes, this second -t is deliberate. See `man ssh`.
577+ '-o', 'StrictHostKeyChecking=no',
578+ '-o', 'UserKnownHostsFile=/dev/null',
579+ location,
580+ '--', cmd,
581+ )
582+ if caller(sshcmd) and not ignore_errors:
583+ raise SSHError('Error running command: ' + ' '.join(sshcmd))
584+
585+ return _sshcall
586+
587+
588+@contextmanager
589+def su(user):
590+ """A context manager to temporarily run the script as a different user."""
591+ uid, gid = get_user_ids(user)
592+ os.setegid(gid)
593+ os.seteuid(uid)
594+ home = get_user_home(user)
595+ with environ(HOME=home):
596+ try:
597+ yield Env(uid, gid, home)
598+ finally:
599+ os.setegid(os.getgid())
600+ os.seteuid(os.getuid())
601+
602+
603+def this_command(directory, args):
604+ """Return a command line to re-launch this script using given `args`.
605+
606+ The command returned will execute this script from `directory`::
607+
608+ >>> import os
609+ >>> script_name = os.path.basename(__file__)
610+
611+ >>> cmd = this_command('/home/user/lp/branches', ['install'])
612+ >>> cmd == ('cd /home/user/lp/branches && devel/utilities/' +
613+ ... script_name + ' install')
614+ True
615+
616+ Arguments are correctly quoted::
617+
618+ >>> cmd = this_command('/path', ['-arg', 'quote me'])
619+ >>> cmd == ('cd /path && devel/utilities/' +
620+ ... script_name + " -arg 'quote me'")
621+ True
622+ """
623+ script = join_command([
624+ os.path.join(LP_CHECKOUT, 'utilities', os.path.basename(__file__)),
625+ ] + list(args))
626+ return join_command(['cd', directory]) + ' && ' + script
627+
628+
629+def user_exists(username):
630+ """Return True if given `username` exists, e.g.::
631+
632+ >>> user_exists('root')
633+ True
634+ >>> user_exists('_this_user_does_not_exist_')
635+ False
636+ """
637+ try:
638+ pwd.getpwnam(username)
639+ except KeyError:
640+ return False
641+ return True
642+
643+
644+def setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir):
645+ """Set up Launchpad repository and source dependencies.
646+
647+ Return True if new changes are pulled from bzr repository.
648+ """
649+ # Using real su because bzr uses uid.
650+ if os.path.exists(checkout_dir):
651+ # Pull the repository.
652+ revno_args = ('bzr', 'revno', checkout_dir)
653+ revno = check_output(*revno_args)
654+ call(*get_su_command(user, ['bzr', 'pull', '-d', checkout_dir]))
655+ changed = revno != check_output(*revno_args)
656+ else:
657+ # Branch the repository.
658+ cmd = ('bzr', 'branch',
659+ LP_REPOS[1] if valid_ssh_keys else LP_REPOS[0], checkout_dir)
660+ call(*get_su_command(user, cmd))
661+ changed = True
662+ # Check repository integrity.
663+ if subprocess.call(['bzr', 'status', '-q', checkout_dir]):
664+ raise LaunchpadError(
665+ 'Repository {0} is corrupted.'.format(checkout_dir))
666+ # Set up source dependencies.
667+ with su(user):
668+ for subdir in ('eggs', 'yui', 'sourcecode'):
669+ mkdirs(os.path.join(dependencies_dir, subdir))
670+ download_cache = os.path.join(dependencies_dir, 'download-cache')
671+ if os.path.exists(download_cache):
672+ call('bzr', 'up', download_cache)
673+ else:
674+ call('bzr', 'co', '--lightweight', LP_SOURCE_DEPS, download_cache)
675+ return changed
676+
677+
678+def setup_external_sourcecode(
679+ user, valid_ssh_keys, checkout_dir, dependencies_dir):
680+ """Update and link external sourcecode."""
681+ cmd = (
682+ 'utilities/update-sourcecode',
683+ None if valid_ssh_keys else '--use-http',
684+ os.path.join(dependencies_dir, 'sourcecode'),
685+ )
686+ with cd(checkout_dir):
687+ # Using real su because update-sourcecode uses uid.
688+ call(*get_su_command(user, cmd))
689+ with su(user):
690+ call('utilities/link-external-sourcecode', dependencies_dir)
691+
692+
693+def make_launchpad(user, checkout_dir, install=False):
694+ """Make and optionally install Launchpad."""
695+ # Using real su because mailman make script uses uid.
696+ call(*get_su_command(user, ['make', '-C', checkout_dir]))
697+ if install:
698+ call('make', '-C', checkout_dir, 'install')
699+
700+
701+def initialize(
702+ user, full_name, email, lpuser, private_key, public_key, valid_ssh_keys,
703+ dependencies_dir, directory):
704+ """Initialize host machine."""
705+ # Install necessary deb packages. This requires Oneiric or later.
706+ call('apt-get', 'update')
707+ apt_get_install(*BASE_PACKAGES)
708+ # Create the user (if he does not exist).
709+ if not user_exists(user):
710+ call('useradd', '-m', '-s', '/bin/bash', '-U', user)
711+ # Generate root ssh keys if they do not exist.
712+ if not os.path.exists('/root/.ssh/id_rsa.pub'):
713+ generate_ssh_keys('/root/.ssh/')
714+ with su(user) as env:
715+ # Set up the user's ssh directory. The ssh key must be associated
716+ # with the lpuser's Launchpad account.
717+ ssh_dir = os.path.join(env.home, '.ssh')
718+ mkdirs(ssh_dir)
719+ # Generate user ssh keys if none are supplied.
720+ if not valid_ssh_keys:
721+ generate_ssh_keys(ssh_dir)
722+ private_key = open(os.path.join(ssh_dir, 'id_rsa')).read()
723+ public_key = open(os.path.join(ssh_dir, 'id_rsa.pub')).read()
724+ priv_file = os.path.join(ssh_dir, 'id_rsa')
725+ pub_file = os.path.join(ssh_dir, 'id_rsa.pub')
726+ auth_file = os.path.join(ssh_dir, 'authorized_keys')
727+ known_hosts = os.path.join(ssh_dir, 'known_hosts')
728+ known_host_content = check_output(
729+ 'ssh-keyscan', '-t', 'rsa', 'bazaar.launchpad.net')
730+ for filename, contents, mode in [
731+ (priv_file, private_key, 'w'),
732+ (pub_file, public_key, 'w'),
733+ (auth_file, public_key, 'a'),
734+ (known_hosts, known_host_content, 'a'),
735+ ]:
736+ with open(filename, mode) as f:
737+ f.write(contents + '\n')
738+ os.chmod(filename, 0644)
739+ os.chmod(priv_file, 0600)
740+ # Set up bzr and Launchpad authentication.
741+ call('bzr', 'whoami', formataddr([full_name, email]))
742+ if valid_ssh_keys:
743+ subprocess.call(['bzr', 'lp-login', lpuser])
744+ # Set up the repository.
745+ mkdirs(directory)
746+ call('bzr', 'init-repo', '--2a', directory)
747+ # Set up the codebase.
748+ checkout_dir = os.path.join(directory, LP_CHECKOUT)
749+ setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir)
750+ # Set up bzr locations
751+ tmpl = ''.join('{0} = {1}\n'.format(k, v) for k, v in LP_BZR_LOCATIONS)
752+ locations = tmpl.format(checkout_dir=checkout_dir, lpuser=lpuser)
753+ contents = '[{0}]\n'.format(directory) + locations
754+ with su(user) as env:
755+ bzr_locations = os.path.join(env.home, '.bazaar', 'locations.conf')
756+ file_append(bzr_locations, contents)
757+
758+
759+def setup_apt(no_repositories=True):
760+ """Setup, update and upgrade deb packages."""
761+ if not no_repositories:
762+ distro = check_output('lsb_release', '-cs').strip()
763+ # APT repository update.
764+ for reposirory in APT_REPOSITORIES:
765+ assume_yes = None if distro == 'lucid' else '-y'
766+ call('add-apt-repository', assume_yes,
767+ reposirory.format(distro=distro))
768+ # XXX frankban 2012-01-13 - Bug 892892: upgrading mountall in LXC
769+ # containers currently does not work.
770+ call("echo 'mountall hold' | dpkg --set-selections", shell=True)
771+ call('apt-get', 'update')
772+ # Install base and Launchpad deb packages.
773+ apt_get_install(*LP_PACKAGES)
774+
775+
776+def setup_launchpad(user, dependencies_dir, directory, valid_ssh_keys):
777+ """Set up the Launchpad environment."""
778+ # User configuration.
779+ subprocess.call(['adduser', user, 'sudo'])
780+ gid = pwd.getpwnam(user).pw_gid
781+ subprocess.call(['addgroup', '--gid', str(gid), user])
782+ # Set up Launchpad dependencies.
783+ checkout_dir = os.path.join(directory, LP_CHECKOUT)
784+ setup_external_sourcecode(
785+ user, valid_ssh_keys, checkout_dir, dependencies_dir)
786+ with su(user):
787+ # Create Apache document roots, to avoid warnings.
788+ mkdirs(*LP_APACHE_ROOTS)
789+ # Set up Apache modules.
790+ for module in LP_APACHE_MODULES.split():
791+ call('a2enmod', module)
792+ with cd(checkout_dir):
793+ # Launchpad database setup.
794+ call('utilities/launchpad-database-setup', user)
795+ # Make and install launchpad.
796+ make_launchpad(user, checkout_dir, install=True)
797+ # Set up container hosts file.
798+ lines = ['{0}\t{1}'.format(ip, names) for ip, names in HOSTS_CONTENT]
799+ file_append(HOSTS_FILE, '\n'.join(lines))
800+
801+
802+def update_launchpad(user, valid_ssh_keys, dependencies_dir, directory, apt):
803+ """Update the Launchpad environment."""
804+ if apt:
805+ setup_apt(no_repositories=True)
806+ checkout_dir = os.path.join(directory, LP_CHECKOUT)
807+ # Update the Launchpad codebase.
808+ changed = setup_codebase(
809+ user, valid_ssh_keys, checkout_dir, dependencies_dir)
810+ setup_external_sourcecode(
811+ user, valid_ssh_keys, checkout_dir, dependencies_dir)
812+ if changed:
813+ make_launchpad(user, checkout_dir, install=False)
814+
815+
816+def link_sourcecode_in_branches(user, dependencies_dir, directory):
817+ """Link external sourcecode for all branches in the project."""
818+ checkout_dir = os.path.join(directory, LP_CHECKOUT)
819+ cmd = os.path.join(checkout_dir, 'utilities', 'link-external-sourcecode')
820+ with su(user):
821+ for dirname in os.listdir(directory):
822+ branch = os.path.join(directory, dirname)
823+ sourcecode_dir = os.path.join(branch, 'sourcecode')
824+ if (branch != checkout_dir and
825+ os.path.exists(sourcecode_dir) and
826+ os.path.isdir(sourcecode_dir)):
827+ call(cmd, '--parent', dependencies_dir, '--target', branch)
828+
829+
830+def create_lxc(user, lxc_name, lxc_arch, lxc_os):
831+ """Create the LXC named `lxc_name` sharing `user` home directory.
832+
833+ The container will be used as development environment or as base template
834+ for parallel testing using ephemeral instances.
835+ """
836+ # Install necessary deb packages.
837+ apt_get_install(*LXC_PACKAGES)
838+ # XXX 2012-02-02 gmb bug=925024:
839+ # These calls need to be removed once the lxc vs. apparmor bug
840+ # is resolved, since having apparmor enabled for lxc is very
841+ # much a Good Thing.
842+ # Disable the apparmor profiles for lxc so that we don't have
843+ # problems installing postgres.
844+ call('ln', '-s',
845+ '/etc/apparmor.d/usr.bin.lxc-start', '/etc/apparmor.d/disable/')
846+ call('apparmor_parser', '-R', '/etc/apparmor.d/usr.bin.lxc-start')
847+ # Update resolv file in order to get the ability to ssh into the LXC
848+ # container using its name.
849+ lxc_gateway_name, lxc_gateway_address = get_lxc_gateway()
850+ file_prepend(RESOLV_FILE, 'nameserver {0}\n'.format(lxc_gateway_address))
851+ file_append(
852+ DHCP_FILE,
853+ 'prepend domain-name-servers {0};\n'.format(lxc_gateway_address))
854+ # Container configuration template.
855+ content = LXC_OPTIONS.format(interface=lxc_gateway_name)
856+ with open(LXC_CONFIG_TEMPLATE, 'w') as f:
857+ f.write(content)
858+ # Creating container.
859+ call(
860+ 'lxc-create',
861+ '-t', 'ubuntu',
862+ '-n', lxc_name,
863+ '-f', LXC_CONFIG_TEMPLATE,
864+ '--',
865+ '-r {os} -a {arch} -b {user}'.format(
866+ os=lxc_os, arch=lxc_arch, user=user),
867+ )
868+ # Set up root ssh key.
869+ user_authorized_keys = os.path.expanduser(
870+ '~' + user + '/.ssh/authorized_keys')
871+ with open(user_authorized_keys, 'a') as f:
872+ f.write(open('/root/.ssh/id_rsa.pub').read())
873+ dst = get_container_path(lxc_name, '/root/.ssh/')
874+ mkdirs(dst)
875+ shutil.copy(user_authorized_keys, dst)
876+
877+
878+def initialize_lxc(lxc_name, lxc_os):
879+ """Initialize LXC container."""
880+ base_packages = list(BASE_PACKAGES)
881+ if lxc_os == 'lucid':
882+ # Install argparse to be able to run this script inside a lucid lxc.
883+ base_packages.append('python-argparse')
884+ ssh(lxc_name)(
885+ 'DEBIAN_FRONTEND=noninteractive '
886+ 'apt-get install -y ' + ' '.join(base_packages))
887+
888+
889+def setup_launchpad_lxc(
890+ user, dependencies_dir, directory, valid_ssh_keys, lxc_name):
891+ """Set up the Launchpad environment inside an LXC."""
892+ # Use ssh to call this script from inside the container.
893+ args = [
894+ 'install', '-u', user, '-a', 'setup_apt', 'setup_launchpad',
895+ '-d', dependencies_dir, '-c', directory
896+ ]
897+ cmd = this_command(directory, args)
898+ ssh(lxc_name)(cmd)
899+
900+
901+def start_lxc(lxc_name):
902+ """Start the lxc instance named `lxc_name`."""
903+ call('lxc-start', '-n', lxc_name, '-d')
904+
905+
906+def stop_lxc(lxc_name):
907+ """Stop the lxc instance named `lxc_name`."""
908+ ssh(lxc_name)('poweroff')
909+ if not lxc_stopped(lxc_name):
910+ subprocess.call(['lxc-stop', '-n', lxc_name])
911+
912+
913+def wait_for_lxc(lxc_name, trials=60, sleep_seconds=1):
914+ """Try to ssh as `user` into the LXC container named `lxc_name`."""
915+ sshcall = ssh(lxc_name)
916+ while True:
917+ trials -= 1
918+ try:
919+ sshcall('true')
920+ except SSHError:
921+ if not trials:
922+ raise
923+ time.sleep(sleep_seconds)
924+ else:
925+ break
926+
927+
928+def handle_user(namespace):
929+ """Handle user argument.
930+
931+ This validator populates namespace with `home_dir` name::
932+
933+ >>> import getpass
934+ >>> username = getpass.getuser()
935+
936+ >>> namespace = argparse.Namespace(user=username)
937+
938+ >>> handle_user(namespace)
939+ >>> namespace.home_dir == '/home/' + username
940+ True
941+
942+ The validation fails if the current user is root and no user is provided::
943+
944+ >>> namespace = argparse.Namespace(user=None, euid=0)
945+ >>> handle_user(namespace) # doctest: +ELLIPSIS
946+ Traceback (most recent call last):
947+ ValidationError: argument user ...
948+ """
949+ if namespace.user is None:
950+ if not namespace.euid:
951+ raise ValidationError('argument user can not be omitted if '
952+ 'the script is run as root.')
953+ namespace.user = pwd.getpwuid(namespace.euid).pw_name
954+ namespace.home_dir = get_user_home(namespace.user)
955+
956+
957+def handle_lpuser(namespace):
958+ """Handle lpuser argument.
959+
960+ If lpuser is not provided by namespace, the user name is used::
961+
962+ >>> import getpass
963+ >>> username = getpass.getuser()
964+
965+ >>> namespace = argparse.Namespace(user=username, lpuser=None)
966+ >>> handle_lpuser(namespace)
967+ >>> namespace.lpuser == username
968+ True
969+ """
970+ if namespace.lpuser is None:
971+ namespace.lpuser = namespace.user
972+
973+
974+def handle_userdata(namespace, whois=bzr_whois):
975+ """Handle full_name and email arguments.
976+
977+ If they are not provided, this function tries to obtain them using
978+ the given `whois` callable::
979+
980+ >>> namespace = argparse.Namespace(
981+ ... full_name=None, email=None, user='root')
982+ >>> email = 'email@example.com'
983+ >>> handle_userdata(namespace, lambda user: (user, email))
984+ >>> namespace.full_name == namespace.user
985+ True
986+ >>> namespace.email == email
987+ True
988+
989+ The validation fails if full_name or email are not provided and
990+ they can not be obtained using the `whois` callable::
991+
992+ >>> namespace = argparse.Namespace(
993+ ... full_name=None, email=None, user='root')
994+ >>> handle_userdata(namespace, lambda user: None) # doctest: +ELLIPSIS
995+ Traceback (most recent call last):
996+ ValidationError: arguments full-name ...
997+
998+ >>> namespace = argparse.Namespace(
999+ ... full_name=None, email=None, user='this_user_does_not_exist')
1000+ >>> handle_userdata(namespace) # doctest: +ELLIPSIS
1001+ Traceback (most recent call last):
1002+ ValidationError: arguments full-name ...
1003+
1004+ It does not make sense to provide only one argument::
1005+
1006+ >>> namespace = argparse.Namespace(full_name='Foo Bar', email=None)
1007+ >>> handle_userdata(namespace) # doctest: +ELLIPSIS
1008+ Traceback (most recent call last):
1009+ ValidationError: arguments full-name ...
1010+ """
1011+ args = (namespace.full_name, namespace.email)
1012+ if not all(args):
1013+ if any(args):
1014+ raise ValidationError(
1015+ 'arguments full-name and email: '
1016+ 'either none or both must be provided.')
1017+ if user_exists(namespace.user):
1018+ userdata = whois(namespace.user)
1019+ if userdata is None:
1020+ raise ValidationError(
1021+ 'arguments full-name and email are required: '
1022+ 'bzr user id not found.')
1023+ namespace.full_name, namespace.email = userdata
1024+ else:
1025+ raise ValidationError(
1026+ 'arguments full-name and email are required: '
1027+ 'system user not found.')
1028+
1029+
1030+def handle_ssh_keys(namespace):
1031+ r"""Handle private and public ssh keys.
1032+
1033+ Keys contained in the namespace are escaped::
1034+
1035+ >>> private = r'PRIVATE\nKEY'
1036+ >>> public = r'PUBLIC\nKEY'
1037+ >>> namespace = argparse.Namespace(
1038+ ... private_key=private, public_key=public)
1039+ >>> handle_ssh_keys(namespace)
1040+ >>> namespace.private_key == private.decode('string-escape')
1041+ True
1042+ >>> namespace.public_key == public.decode('string-escape')
1043+ True
1044+ >>> namespace.valid_ssh_keys
1045+ True
1046+
1047+ Keys are None if they are not provided and can not be found in the
1048+ current home directory::
1049+
1050+ >>> namespace = argparse.Namespace(
1051+ ... private_key=None, home_dir='/tmp/__does_not_exists__')
1052+ >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS
1053+ >>> print namespace.private_key
1054+ None
1055+ >>> print namespace.public_key
1056+ None
1057+ >>> namespace.valid_ssh_keys
1058+ False
1059+
1060+ If only one of private_key and public_key is provided, a
1061+ ValidationError will be raised.
1062+
1063+ >>> namespace = argparse.Namespace(
1064+ ... private_key=private, public_key=None,
1065+ ... home_dir='/tmp/__does_not_exists__')
1066+ >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS
1067+ Traceback (most recent call last):
1068+ ValidationError: arguments private-key...
1069+ """
1070+ namespace.valid_ssh_keys = True
1071+ for attr, filename in (
1072+ ('private_key', 'id_rsa'),
1073+ ('public_key', 'id_rsa.pub')):
1074+ value = getattr(namespace, attr, None)
1075+ if value:
1076+ setattr(namespace, attr, value.decode('string-escape'))
1077+ else:
1078+ path = os.path.join(namespace.home_dir, '.ssh', filename)
1079+ try:
1080+ value = open(path).read()
1081+ except IOError:
1082+ value = None
1083+ namespace.valid_ssh_keys = False
1084+ setattr(namespace, attr, value)
1085+ if bool(namespace.private_key) != bool(namespace.public_key):
1086+ raise ValidationError(
1087+ "arguments private-key and public-key: "
1088+ "both must be provided or neither must be provided.")
1089+
1090+
1091+def handle_directories(namespace):
1092+ """Handle checkout and dependencies directories.
1093+
1094+ The ~ construction is automatically expanded::
1095+
1096+ >>> namespace = argparse.Namespace(
1097+ ... directory='~/launchpad', dependencies_dir='~/launchpad/deps',
1098+ ... home_dir='/home/foo')
1099+ >>> handle_directories(namespace)
1100+ >>> namespace.directory
1101+ '/home/foo/launchpad'
1102+ >>> namespace.dependencies_dir
1103+ '/home/foo/launchpad/deps'
1104+
1105+ The validation fails for directories not residing inside the home::
1106+
1107+ >>> namespace = argparse.Namespace(
1108+ ... directory='/tmp/launchpad',
1109+ ... dependencies_dir='~/launchpad/deps',
1110+ ... home_dir='/home/foo')
1111+ >>> handle_directories(namespace) # doctest: +ELLIPSIS
1112+ Traceback (most recent call last):
1113+ ValidationError: argument directory ...
1114+
1115+ The validation fails if the directory contains spaces::
1116+
1117+ >>> namespace = argparse.Namespace(directory='my directory')
1118+ >>> handle_directories(namespace) # doctest: +ELLIPSIS
1119+ Traceback (most recent call last):
1120+ ValidationError: argument directory ...
1121+ """
1122+ if ' ' in namespace.directory:
1123+ raise ValidationError('argument directory can not contain spaces.')
1124+ for attr in ('directory', 'dependencies_dir'):
1125+ directory = getattr(
1126+ namespace, attr).replace('~', namespace.home_dir)
1127+ if not directory.startswith(namespace.home_dir + os.path.sep):
1128+ raise ValidationError(
1129+ 'argument {0} does not reside under the home '
1130+ 'directory of the system user.'.format(attr))
1131+ setattr(namespace, attr, directory)
1132+
1133+
1134+class ArgumentParser(argparse.ArgumentParser):
1135+ """A customized argument parser for `argparse`."""
1136+
1137+ def __init__(self, *args, **kwargs):
1138+ self.actions = []
1139+ self.subparsers = None
1140+ super(ArgumentParser, self).__init__(*args, **kwargs)
1141+
1142+ def add_argument(self, *args, **kwargs):
1143+ """Override to store actions in a "public" instance attribute.
1144+
1145+ >>> parser = ArgumentParser()
1146+ >>> parser.add_argument('arg1')
1147+ >>> parser.add_argument('arg2')
1148+ >>> [action.dest for action in parser.actions]
1149+ ['help', 'arg1', 'arg2']
1150+ """
1151+ action = super(ArgumentParser, self).add_argument(*args, **kwargs)
1152+ self.actions.append(action)
1153+
1154+ def register_subcommand(self, name, subcommand_class, handler=None):
1155+ """Add a subcommand to this parser.
1156+
1157+ A sub command is registered giving the sub command `name` and class::
1158+
1159+ >>> parser = ArgumentParser()
1160+
1161+ >>> class SubCommand(BaseSubCommand):
1162+ ... def handle(self, namespace):
1163+ ... return self.name
1164+
1165+ >>> parser = ArgumentParser()
1166+ >>> _ = parser.register_subcommand('foo', SubCommand)
1167+
1168+ The `main` method of the subcommand class is added to namespace, and
1169+ can be used to actually handle the sub command execution.
1170+
1171+ >>> namespace = parser.parse_args(['foo'])
1172+ >>> namespace.main(namespace)
1173+ 'foo'
1174+
1175+ A `handler` callable can also be provided to handle the subcommand
1176+ execution::
1177+
1178+ >>> handler = lambda namespace: 'custom handler'
1179+
1180+ >>> parser = ArgumentParser()
1181+ >>> _ = parser.register_subcommand(
1182+ ... 'bar', SubCommand, handler=handler)
1183+
1184+ >>> namespace = parser.parse_args(['bar'])
1185+ >>> namespace.main(namespace)
1186+ 'custom handler'
1187+ """
1188+ if self.subparsers is None:
1189+ self.subparsers = self.add_subparsers(
1190+ title='subcommands',
1191+ help='Each subcommand accepts --h or --help to describe it.')
1192+ subcommand = subcommand_class(name, handler=handler)
1193+ parser = self.subparsers.add_parser(
1194+ subcommand.name, help=subcommand.help)
1195+ subcommand.add_arguments(parser)
1196+ parser.set_defaults(main=subcommand.main, get_parser=lambda: parser)
1197+ return subcommand
1198+
1199+ def get_args_from_namespace(self, namespace):
1200+ """Return a list of arguments taking values from `namespace`.
1201+
1202+ Having a parser defined as usual::
1203+
1204+ >>> parser = ArgumentParser()
1205+ >>> _ = parser.add_argument('--foo')
1206+ >>> _ = parser.add_argument('bar')
1207+ >>> namespace = parser.parse_args('--foo eggs spam'.split())
1208+
1209+ It is possible to recreate the argument list taking values from
1210+ a different namespace::
1211+
1212+ >>> namespace.foo = 'changed'
1213+ >>> parser.get_args_from_namespace(namespace)
1214+ ['--foo', 'changed', 'spam']
1215+ """
1216+ args = []
1217+ for action in self.actions:
1218+ dest = action.dest
1219+ option_strings = action.option_strings
1220+ value = getattr(namespace, dest, None)
1221+ if value:
1222+ if option_strings:
1223+ args.append(option_strings[0])
1224+ if isinstance(value, list):
1225+ args.extend(value)
1226+ elif not isinstance(value, bool):
1227+ args.append(value)
1228+ return args
1229+
1230+ def _handle_help(self, namespace):
1231+ """Help sub command handler."""
1232+ command = namespace.command
1233+ help = self.prefix_chars + 'h'
1234+ args = [help] if command is None else [command, help]
1235+ self.parse_args(args)
1236+
1237+ def _add_help_subcommand(self):
1238+ """Add an help sub command to this parser."""
1239+ name = 'help'
1240+ choices = self.subparsers.choices.keys()
1241+ if name not in choices:
1242+ choices.append(name)
1243+ parser = self.subparsers.add_parser(
1244+ name, help='More help on a command.')
1245+ parser.add_argument('command', nargs='?', choices=choices)
1246+ parser.set_defaults(main=self._handle_help)
1247+
1248+ def parse_args(self, *args, **kwargs):
1249+ """Override to add an help sub command.
1250+
1251+ The help sub command is added only if other sub commands exist::
1252+
1253+ >>> stderr, sys.stderr = sys.stderr, sys.stdout
1254+ >>> parser = ArgumentParser()
1255+ >>> parser.parse_args(['help'])
1256+ Traceback (most recent call last):
1257+ SystemExit: 2
1258+ >>> sys.stderr = stderr
1259+
1260+ >>> class SubCommand(BaseSubCommand): pass
1261+ >>> _ = parser.register_subcommand('command', SubCommand)
1262+ >>> namespace = parser.parse_args(['help'])
1263+ >>> namespace.main(namespace)
1264+ Traceback (most recent call last):
1265+ SystemExit: 0
1266+ """
1267+ if self.subparsers is not None:
1268+ self._add_help_subcommand()
1269+ return super(ArgumentParser, self).parse_args(*args, **kwargs)
1270+
1271+
1272+class BaseSubCommand(object):
1273+ """Base class defining a sub command.
1274+
1275+ Objects of this class can be used to easily add sub commands to this
1276+ script as plugins, providing arguments, validating them, restarting
1277+ as root if needed.
1278+
1279+ Override `add_arguments()` to add arguments, `validators` to add
1280+ namespace validators, and `handle()` to manage sub command execution::
1281+
1282+ >>> def validator(namespace):
1283+ ... namespace.bar = True
1284+
1285+ >>> class SubCommand(BaseSubCommand):
1286+ ... help = 'Sub command example.'
1287+ ... validators = (validator,)
1288+ ...
1289+ ... def add_arguments(self, parser):
1290+ ... super(SubCommand, self).add_arguments(parser)
1291+ ... parser.add_argument('--foo')
1292+ ...
1293+ ... def handle(self, namespace):
1294+ ... return namespace
1295+
1296+ Register the sub command using `ArgumentParser.register_subcommand`::
1297+
1298+ >>> parser = ArgumentParser()
1299+ >>> sub_command = parser.register_subcommand('spam', SubCommand)
1300+
1301+ Now the subcommand has a name::
1302+
1303+ >>> sub_command.name
1304+ 'spam'
1305+
1306+ The sub command handler can be called using `namespace.main()`::
1307+
1308+ >>> namespace = parser.parse_args('spam --foo eggs'.split())
1309+ >>> namespace = namespace.main(namespace)
1310+ >>> namespace.foo
1311+ 'eggs'
1312+ >>> namespace.bar
1313+ True
1314+
1315+ The help attribute of sub command instances is used to generate
1316+ the command usage message::
1317+ >>> help = parser.format_help()
1318+ >>> 'spam' in help
1319+ True
1320+ >>> 'Sub command example.' in help
1321+ True
1322+ """
1323+
1324+ help = ''
1325+ needs_root = False
1326+ validators = ()
1327+
1328+ def __init__(self, name, handler=None):
1329+ self.name = name
1330+ self.handler = handler or self.handle
1331+
1332+ def __repr__(self):
1333+ return '<{klass}: {name}>'.format(
1334+ klass=self.__class__.__name__, name=self.name)
1335+
1336+ def init_namespace(self, namespace):
1337+ """Add `run_as_root` and `euid` names to the given `namespace`."""
1338+ euid = os.geteuid()
1339+ namespace.euid, namespace.run_as_root = euid, not euid
1340+
1341+ def get_needs_root(self, namespace):
1342+ """Return True if root is needed to run this subcommand.
1343+
1344+ Subclasses can override this to dynamically change this value
1345+ based on the given `namespace`.
1346+ """
1347+ return self.needs_root
1348+
1349+ def get_validators(self, namespace):
1350+ """Return an iterable of namespace validators for this sub command.
1351+
1352+ Subclasses can override this to dynamically change validators
1353+ based on the given `namespace`.
1354+ """
1355+ return self.validators
1356+
1357+ def validate(self, parser, namespace):
1358+ """Validate the current namespace.
1359+
1360+ The method `self.get_validators` can contain an iterable of objects
1361+ that are called once the arguments namespace is fully populated.
1362+ This allows cleaning and validating arguments that depend on
1363+ each other, or on the current environment.
1364+
1365+ Each validator is a callable object, takes the current namespace
1366+ and can raise ValidationError if the arguments are not valid::
1367+
1368+ >>> import sys
1369+ >>> stderr, sys.stderr = sys.stderr, sys.stdout
1370+ >>> def validator(namespace):
1371+ ... raise ValidationError('nothing is going on')
1372+ >>> sub_command = BaseSubCommand('foo')
1373+ >>> sub_command.validators = [validator]
1374+ >>> sub_command.validate(ArgumentParser(), argparse.Namespace())
1375+ Traceback (most recent call last):
1376+ SystemExit: 2
1377+ >>> sys.stderr = stderr
1378+ """
1379+ for validator in self.get_validators(namespace):
1380+ try:
1381+ validator(namespace)
1382+ except ValidationError as err:
1383+ parser.error(err)
1384+
1385+ def restart_as_root(self, parser, namespace):
1386+ """Restart this script using *sudo*.
1387+
1388+ The arguments are recreated using the given `namespace`.
1389+ """
1390+ args = parser.get_args_from_namespace(namespace)
1391+ return subprocess.call(['sudo', sys.argv[0], self.name] + args)
1392+
1393+ def main(self, namespace):
1394+ """Entry point for subcommand execution.
1395+
1396+ This method takes care of:
1397+
1398+ - current argparse subparser retrieval
1399+ - namespace initialization
1400+ - namespace validation
1401+ - script restart as root (if this sub command needs to be run as root)
1402+
1403+ If everything is ok the sub command handler is called passing
1404+ the validated namespace.
1405+ """
1406+ parser = namespace.get_parser()
1407+ self.init_namespace(namespace)
1408+ self.validate(parser, namespace)
1409+ if self.get_needs_root(namespace) and not namespace.run_as_root:
1410+ return self.restart_as_root(parser, namespace)
1411+ return self.handler(namespace)
1412+
1413+ def handle(self, namespace):
1414+ """Default sub command handler.
1415+
1416+ Subclasses must either implement this method or provide another
1417+ callable handler during sub command registration.
1418+ """
1419+ raise NotImplementedError
1420+
1421+ def add_arguments(self, parser):
1422+ """Here subclasses can add arguments to the subparser."""
1423+ pass
1424+
1425+
1426+class ActionsBasedSubCommand(BaseSubCommand):
1427+ """A sub command that uses "actions" to handle its execution.
1428+
1429+ Actions are callables stored in the `actions` attribute, together
1430+ with the arguments they expect. Those arguments are strings
1431+ representing attributes of the argparse namespace::
1432+
1433+ >>> trace = []
1434+
1435+ >>> def action1(foo):
1436+ ... trace.append('action1 received ' + foo)
1437+
1438+ >>> def action2(foo, bar):
1439+ ... trace.append('action2 received {0} and {1}'.format(foo, bar))
1440+
1441+ >>> class SubCommand(ActionsBasedSubCommand):
1442+ ... actions = (
1443+ ... (action1, 'foo'),
1444+ ... (action2, 'foo', 'bar'),
1445+ ... )
1446+ ...
1447+ ... def add_arguments(self, parser):
1448+ ... super(SubCommand, self).add_arguments(parser)
1449+ ... parser.add_argument('--foo')
1450+ ... parser.add_argument('--bar')
1451+
1452+ This class implements an handler method that executes actions in the
1453+ order they are provided::
1454+
1455+ >>> parser = ArgumentParser()
1456+ >>> _ = parser.register_subcommand('sub', SubCommand)
1457+ >>> namespace = parser.parse_args('sub --foo eggs --bar spam'.split())
1458+ >>> namespace.main(namespace)
1459+ >>> trace
1460+ ['action1 received eggs', 'action2 received eggs and spam']
1461+
1462+ A special argument `-a` or `--actions` is automatically added to the
1463+ parser. It can be used to execute only one or a subset of actions::
1464+
1465+ >>> trace = []
1466+
1467+ >>> namespace = parser.parse_args('sub --foo eggs -a action1'.split())
1468+ >>> namespace.main(namespace)
1469+ >>> trace
1470+ ['action1 received eggs']
1471+
1472+ A special argument `--skip-actions` is automatically added to the
1473+ parser. It can be used to skip one or more actions::
1474+
1475+ >>> trace = []
1476+
1477+ >>> namespace = parser.parse_args(
1478+ ... 'sub --foo eggs --skip-actions action1'.split())
1479+ >>> namespace.main(namespace)
1480+ >>> trace
1481+ ['action2 received eggs and None']
1482+
1483+ The actions execution is stopped if an action raises `LaunchpadError`.
1484+ In that case, the error is returned by the handler.
1485+
1486+ >>> trace = []
1487+
1488+ >>> def erroneous_action(foo):
1489+ ... raise LaunchpadError('error')
1490+
1491+ >>> class SubCommandWithErrors(SubCommand):
1492+ ... actions = (
1493+ ... (action1, 'foo'),
1494+ ... (erroneous_action, 'foo'),
1495+ ... (action2, 'foo', 'bar'),
1496+ ... )
1497+
1498+ >>> parser = ArgumentParser()
1499+ >>> _ = parser.register_subcommand('sub', SubCommandWithErrors)
1500+ >>> namespace = parser.parse_args('sub --foo eggs'.split())
1501+ >>> error = namespace.main(namespace)
1502+ >>> error.message
1503+ 'error'
1504+
1505+ The action `action2` is not executed::
1506+
1507+ >>> trace
1508+ ['action1 received eggs']
1509+ """
1510+
1511+ actions = ()
1512+
1513+ def __init__(self, *args, **kwargs):
1514+ super(ActionsBasedSubCommand, self).__init__(*args, **kwargs)
1515+ self._action_names = []
1516+ self._actions = {}
1517+ for action_args in self.actions:
1518+ action, args = action_args[0], action_args[1:]
1519+ action_name = self._get_action_name(action)
1520+ self._action_names.append(action_name)
1521+ self._actions[action_name] = (action, args)
1522+
1523+ def _get_action_name(self, action):
1524+ """Return the string representation of an action callable.
1525+
1526+ The name is retrieved using attributes lookup for `action_name`
1527+ and then `__name__`::
1528+
1529+ >>> def action1():
1530+ ... pass
1531+ >>> action1.action_name = 'myaction'
1532+
1533+ >>> def action2():
1534+ ... pass
1535+
1536+ >>> sub_command = ActionsBasedSubCommand('foo')
1537+ >>> sub_command._get_action_name(action1)
1538+ 'myaction'
1539+ >>> sub_command._get_action_name(action2)
1540+ 'action2'
1541+ """
1542+ try:
1543+ return action.action_name
1544+ except AttributeError:
1545+ return action.__name__
1546+
1547+ def add_arguments(self, parser):
1548+ super(ActionsBasedSubCommand, self).add_arguments(parser)
1549+ parser.add_argument(
1550+ '-a', '--actions', nargs='+', choices=self._action_names,
1551+ help='Call one or more internal functions.')
1552+ parser.add_argument(
1553+ '--skip-actions', nargs='+', choices=self._action_names,
1554+ help='Skip one or more internal functions.')
1555+
1556+ def handle(self, namespace):
1557+ skip_actions = namespace.skip_actions or []
1558+ action_names = filter(
1559+ lambda action_name: action_name not in skip_actions,
1560+ namespace.actions or self._action_names)
1561+ for action_name in action_names:
1562+ action, arg_names = self._actions[action_name]
1563+ args = [getattr(namespace, i) for i in arg_names]
1564+ try:
1565+ action(*args)
1566+ except LaunchpadError as err:
1567+ return err
1568+
1569+
1570+class InstallSubCommand(ActionsBasedSubCommand):
1571+ """Install the Launchpad environment."""
1572+
1573+ actions = (
1574+ (initialize,
1575+ 'user', 'full_name', 'email', 'lpuser', 'private_key',
1576+ 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'),
1577+ (setup_apt, 'no_repositories'),
1578+ (setup_launchpad,
1579+ 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys'),
1580+ )
1581+ help = __doc__
1582+ needs_root = True
1583+ validators = (
1584+ handle_user,
1585+ handle_lpuser,
1586+ handle_userdata,
1587+ handle_ssh_keys,
1588+ handle_directories,
1589+ )
1590+
1591+ def add_arguments(self, parser):
1592+ super(InstallSubCommand, self).add_arguments(parser)
1593+ parser.add_argument(
1594+ '-u', '--user',
1595+ help='The name of the system user to be created or updated. '
1596+ 'The current user is used if this script is not run as '
1597+ 'root and this argument is omitted.')
1598+ parser.add_argument(
1599+ '-e', '--email',
1600+ help='The email of the user, used for bzr whoami. This argument '
1601+ 'can be omitted if a bzr id exists for current user.')
1602+ parser.add_argument(
1603+ '-f', '--full-name',
1604+ help='The full name of the user, used for bzr whoami. '
1605+ 'This argument can be omitted if a bzr id exists for '
1606+ 'current user.')
1607+ parser.add_argument(
1608+ '-l', '--lpuser',
1609+ help='The name of the Launchpad user that will be used to '
1610+ 'check out dependencies. If not provided, the system '
1611+ 'user name is used.')
1612+ parser.add_argument(
1613+ '-v', '--private-key',
1614+ help='The SSH private key for the Launchpad user (without '
1615+ 'passphrase). If this argument is omitted and a keypair is '
1616+ 'not found in the home directory of the system user a new '
1617+ 'SSH keypair will be generated and the checkout of the '
1618+ 'Launchpad code will use HTTP rather than bzr+ssh.')
1619+ parser.add_argument(
1620+ '-b', '--public-key',
1621+ help='The SSH public key for the Launchpad user. '
1622+ 'If this argument is omitted and a keypair is not found '
1623+ 'in the home directory of the system user a new SSH '
1624+ 'keypair will be generated and the checkout of the '
1625+ 'Launchpad code will use HTTP rather than bzr+ssh.')
1626+ parser.add_argument(
1627+ '-d', '--dependencies-dir', default=DEPENDENCIES_DIR,
1628+ help='The directory of the Launchpad dependencies to be created. '
1629+ 'The directory must reside under the home directory of the '
1630+ 'given user (see -u argument). '
1631+ '[DEFAULT={0}]'.format(DEPENDENCIES_DIR))
1632+ parser.add_argument(
1633+ '-c', '--directory', default=CHECKOUT_DIR,
1634+ help='The directory of the Launchpad repository to be created. '
1635+ 'The directory must reside under the home directory of the '
1636+ 'given user (see -u argument). '
1637+ '[DEFAULT={0}]'.format(CHECKOUT_DIR))
1638+ parser.add_argument(
1639+ '-N', '--no-repositories', action='store_true',
1640+ help='Do not add APT repositories.')
1641+
1642+
1643+class UpdateSubCommand(ActionsBasedSubCommand):
1644+ """Update the Launchpad environment to latest version."""
1645+
1646+ actions = (
1647+ (update_launchpad,
1648+ 'user', 'valid_ssh_keys', 'dependencies_dir', 'directory', 'apt'),
1649+ (link_sourcecode_in_branches,
1650+ 'user', 'dependencies_dir', 'directory'),
1651+ )
1652+ help = __doc__
1653+ validators = (
1654+ handle_user,
1655+ handle_ssh_keys,
1656+ handle_directories,
1657+ )
1658+
1659+ def get_needs_root(self, namespace):
1660+ # Root is needed only if an apt update/upgrade is requested.
1661+ return namespace.apt
1662+
1663+ def add_arguments(self, parser):
1664+ super(UpdateSubCommand, self).add_arguments(parser)
1665+ parser.add_argument(
1666+ '-u', '--user',
1667+ help='The name of the system user used to update Launchpad. '
1668+ 'The current user is used if this script is not run as '
1669+ 'root and this argument is omitted.')
1670+ parser.add_argument(
1671+ '-d', '--dependencies-dir', default=DEPENDENCIES_DIR,
1672+ help='The directory of the Launchpad dependencies to be updated. '
1673+ 'The directory must reside under the home directory of the '
1674+ 'given user (see -u argument). '
1675+ '[DEFAULT={0}]'.format(DEPENDENCIES_DIR))
1676+ parser.add_argument(
1677+ '-c', '--directory', default=CHECKOUT_DIR,
1678+ help='The directory of the Launchpad repository to be updated. '
1679+ 'The directory must reside under the home directory of the '
1680+ 'given user (see -u argument). '
1681+ '[DEFAULT={0}]'.format(CHECKOUT_DIR))
1682+ parser.add_argument(
1683+ '-D', '--apt', action='store_true',
1684+ help='Also update deb packages.')
1685+
1686+
1687+class LXCInstallSubCommand(InstallSubCommand):
1688+ """Install the Launchpad environment inside an LXC."""
1689+
1690+ actions = (
1691+ (initialize,
1692+ 'user', 'full_name', 'email', 'lpuser', 'private_key',
1693+ 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'),
1694+ (create_lxc,
1695+ 'user', 'lxc_name', 'lxc_arch', 'lxc_os'),
1696+ (start_lxc, 'lxc_name'),
1697+ (wait_for_lxc, 'lxc_name'),
1698+ (initialize_lxc,
1699+ 'lxc_name', 'lxc_os'),
1700+ (setup_launchpad_lxc,
1701+ 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys',
1702+ 'lxc_name'),
1703+ (stop_lxc, 'lxc_name'),
1704+ )
1705+ help = __doc__
1706+
1707+ def add_arguments(self, parser):
1708+ super(LXCInstallSubCommand, self).add_arguments(parser)
1709+ parser.add_argument(
1710+ '-n', '--lxc-name', default=LXC_NAME,
1711+ help='The LXC container name to setup. '
1712+ '[DEFAULT={0}]'.format(LXC_NAME))
1713+ parser.add_argument(
1714+ '-A', '--lxc-arch', default=LXC_GUEST_ARCH,
1715+ help='The LXC container architecture. '
1716+ '[DEFAULT={0}]'.format(LXC_GUEST_ARCH))
1717+ parser.add_argument(
1718+ '-r', '--lxc-os', default=LXC_GUEST_OS,
1719+ choices=LXC_GUEST_CHOICES,
1720+ help='The LXC container distro codename. '
1721+ '[DEFAULT={0}]'.format(LXC_GUEST_OS))
1722+
1723+
1724+def main():
1725+ parser = ArgumentParser(description=__doc__)
1726+ parser.register_subcommand('install', InstallSubCommand)
1727+ parser.register_subcommand('update', UpdateSubCommand)
1728+ parser.register_subcommand('lxc-install', LXCInstallSubCommand)
1729+ args = parser.parse_args()
1730+ return args.main(args)
1731+
1732+
1733+if __name__ == '__main__':
1734+ sys.exit(main())