Merge lp:~frankban/launchpad/lpsetup-initial into lp:launchpad
- lpsetup-initial
- Merge into devel
Status: | Rejected |
---|---|
Rejected by: | William Grant |
Proposed branch: | lp:~frankban/launchpad/lpsetup-initial |
Merge into: | lp:launchpad |
Diff against target: |
1734 lines (+1730/-0) 1 file modified
utilities/lpsetup.py (+1730/-0) |
To merge this branch: | bzr merge lp:~frankban/launchpad/lpsetup-initial |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Gary Poster (community) | Approve | ||
Review via email: mp+94765@code.launchpad.net |
Commit message
Description of the change
= Summary =
This is the first step of *lpsetup* implementation.
It's basically a Python script created to eliminate code duplication between
*setuplxc* and *rocketfuel-
PS: It's a brand new huge file, sorry about that.
== Proposed fix ==
Actually the script can be used to set up a Launchpad development or testing
environment in a real machine or inside an LXC container, with ability to
update the codebase once installed.
The next steps will be, in random order:
- implement lxc-update sub command: update the codebase inside a running or
stopped LXC
- implement branch and lxc-branch sub commands: see rocketfuel-branch
- ability to specify a custom ssh key name to connect to LXC
- create a real *lpsetup* project, splitting code in separate files, etc.
- deb package?
- ec2 tests, tests, tests
== Implementation details ==
While *setuplxc* sets up the LXC connecting via ssh to the container,
*lpsetup* uses a different approach: the script calls itself using an ssh
connection to the container. This is possible thanks to the script ability
to be run specifying *actions*, i.e. a subset of internal functions to be
called. This way, the LXC can initialize itself and the code is reused.
*lpsetup* improves some of the functionalities present in *setuplxc*:
- errors in executing external commands are caught and reported, and they
cause the script to stop running
- apt repositories are added in a gentler way, using *add-apt-
(this allows to get rid of `apt-get install --allow-
and needs to be back-ported to *setuplxc*)
- some helper objects were fixed and some others were added
A thin subcommands management layer (`BaseSubCommand`) was implemented to
allow extensibility.
Because the script can be executed in lucid containers,
it is now compatible with Python 2.6.
== Tests ==
python -m doctest -v lpsetup.py
- 14876. By Francesco Banconi
-
Added tests to generate_ssh_keys helper.
Gary Poster (gary) wrote : | # |
Francesco Banconi (frankban) wrote : | # |
Thank you Gary for reviewing this, it's a huge file.
> Unless you give me a good reason why not, I'd like all of the contextmanagers
> to have a try:finally block placed around the yield. environ and cd are the
> only ones without it, I think.
No good reason, I will add the try/finally block to them.
> Why do you need your own check_output, rather than using subprocess'
> check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> what would you think of trying to get it from the module, and if there's an
> AttributeError (or ImportError, however you want to work it) then you define
> your function? That way we get the newer version, with possible bugfixes, if
> it exists; and we have a chance to clarify why we are not using the stdlib's
> function.
That's one of the reasons, the others, like the first, aren't essential, just nice to have:
- the exception raised on errors is the one subcommand handlers trap (but of course I could also trap subprocess.
- the positional arguments thing (really not even a "nice to have", more a "look, we're lazy and cool")
- None is ignored if given as argument, so that you can write `check_output('ls', '-l' if something else None)
- the error contains a better representation of the executed command
I can get rid of check_output; my point is: like `subprocess.
> So many of these helpers are generically useful. It's a shame they are not
> somewhere more central. Many of them look helpful for charms too. I don't
> have a suggested action for that; it's just an observation.
I agree.
> As we discussed already, in your shoes people have advocated to me that I
> should use bzr's command framework (and I mentioned Jamu Kakar's commandant;
> that project hasn't seen any updates, but it looks like Landscape is still
> using it because they have done some recent packaging work). I like the fact
> that what you have is self-contained and small. Again, I merely offer that as
> an avenue that I suggest you investigate in the future.
Thanks Gary, sure, I will take a look at commandant, although I like my tiny subcommand objects.
> I look forward to this being a separate project with an ability to divide
> things up into support files and so on. In addition to making it possible to
> organize the code more obviously, it also might then be reasonable to
> reconsider the use of doctests.
Yes, you are right, I will add a slack time card for unit tests.
>
> Random thought: I was thinking about your other branch that had LANG=C.
> Probably obvious, but you could change apt_get_install to accept keyword
> arguments and include those as environment variables so you could easily do
> the LANG=C thing, maybe? That maybe looks too much like command options.
> Just a thought.
Nice idea IMHO.
> As I'm sure you've realized, we should use generate_ssh_keys with descriptive
> names per the webops request for setuplxc. If these become merely
> "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> want to know ...
Gary Poster (gary) wrote : | # |
Continuing on from where I left off...
I suggest deleting update_
You have your file nicely divided in sections by function. If you were going to keep everything in this one file, I'd ask for comment lines delineating the sections--in particular, I had to think to realize that we were in the arg parsing section when I saw handle_user. Since you are going to divide this up into files anyway, I won't worry about it now.
You are not using os.path.expanduser in handle_directories for a reason: you are using the home directory of the user that was passed in, not the current home directory. A comment explaining this might be helpful, or might not.
The ArgumentParser might be improved if it showed in the docstring how users would use the registered subcommands. I expect that kind of thing might fall out of the division into separate files.
I find the interaction between ArgumentParser (the actions attribute) and ActionsBasedSub
When I run the command's help, I get the following output:
"""
$ ./lpsetup.py -h
usage: lpsetup.py [-h] {install,
Create and update Launchpad development and testing environments.
optional arguments:
-h, --help show this help message and exit
subcommands:
valid subcommands
{install,
install Install the Launchpad environment.
update Update the Launchpad environment to latest version.
lxc-install Install the Launchpad environment inside an LXC.
"""
- I find the text "valid subcommands {install,
- I find the placement of "-h, --help..." to be confusing too. Perhaps put that text, written more explicitly, where the "calid subcommands" text is? "Each subcommand accepts --h or --help to describe it."? Also, this reminds me, and probably other people who will use lpsetup, of bzr. They will want to do the following:
$ lpsetup help
(this could be equivalent to lpsetup -h)
$ lpsetup help install
(this could be equivalent to lpsetup install -h)
- The first (usage) line looks like you ought to be able to write "lpsetup -h install". You can, but it shows the top-level help--not what I'd expect. I think this usage line is confusing. Ideally, I think, the usage would be "lpsetup {help,install,
That's it. Nice progress! Thank you.
Gary Poster (gary) wrote : | # |
...
> > Why do you need your own check_output, rather than using subprocess'
> > check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> > what would you think of trying to get it from the module, and if there's an
> > AttributeError (or ImportError, however you want to work it) then you define
> > your function? That way we get the newer version, with possible bugfixes,
> if
> > it exists; and we have a chance to clarify why we are not using the stdlib's
> > function.
...
> I can get rid of check_output; my point is: like `subprocess.
> it's just a little wrapper around `subprocess.Popen`, that does the real stuff
> (and we will not miss future bugfixes for that object).
Keep it. I think you ought to have a comment that explains why it is valuable. That's it.
...
> > As I'm sure you've realized, we should use generate_ssh_keys with
> descriptive
> > names per the webops request for setuplxc. If these become merely
> > "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> > want to know "why" the key is used (e.g., "lxc_communicat
> similar).
> > If there's not a good name for this key that would distinguish it from other
> > keys a user might have, we should push back; but AFAICT there should be a
> good
> > name here.
>
> I propose 'id_for_lxc'... I know: we can do better.
"launchpad_lxc_id" perhaps? Getting the word "launchpad" in there would be valuable too, I think.
Thanks again
Gary
Robert Collins (lifeless) wrote : | # |
Hi, a few points for you to consider:
- this might be a lot better off in the lp-dev-tools project, its
clearly not part of LP itself. That would also make it easier to split
it into N files as you would not be stuck in the bootstrap stage of
the users environment (we could package that and mandate the package
be installed).
- the environ context manager is duplicative of the
EnvironmentVariable fixture; it would be nice not to rewrite code that
has already been written and tested (and is used in our environment
too!) Yes, I know that this is small, but it does all add up, and
someone somewhere will eventually be cursing that they have to redo
this / fix a unicode escaping issue in the code or whatever.
- I'm not fussed about reusing commandant or the bzrlib UI framework;
if I was writing something new that needed subcommands I would do it
differently than I did in bzrlib (and in fact I have since - with a
leaner, faster, more testable and machine drivable system in testr).
If you wanted to reuse that, its probably not quite there for reuse,
but we could split it into framework-and-user and make it so.
HTH,
Rob
Francesco Banconi (frankban) wrote : | # |
Thank you for the review Gary. I agree with your suggestions and I will reply only to your doubts concerning BaseSubcommand.
> I find the interaction between ArgumentParser (the actions attribute) and
> ActionsBasedSub
> thing I found confusing was BaseSubCommand. ArgumentParser, AFAICT, has code
> specifically for ActionsBasedSub
> that kind of SubCommand. When would one actually use BaseSubCommand? The
> fact that we are not using the BaseSubCommand directly reinforces this
> feeling. Maybe a bit of elaboration would explain the value of having a
> separate BaseSubCommand, and not merging it directly with
> ActionsBasedSub
> but that does not convince me.
ArgumentParser ignores the concept of *actions* as intended in ActionBasedSubc
The register_subcommand method just:
- instantiates a sub command object
- calls subcommand.
- stores a reference to subcommand.main that can be used later
This interface is implemented by BaseSubCommand. ActionBasedSubc
I think the misunderstanding comes from the different meaning of the word *action* in argparse and ActionBasedSubc
In argparse an action is an object containing info about a single argument (argparse.Action). Actions are stored in argparse.
I will agree if you suggest to change the name of ArgumentParser.
Francesco Banconi (frankban) wrote : | # |
> - this might be a lot better off in the lp-dev-tools project, its
> clearly not part of LP itself. That would also make it easier to split
> it into N files as you would not be stuck in the bootstrap stage of
> the users environment (we could package that and mandate the package
> be installed).
I agree. Indeed we were thinking about creating an lpsetup project.
In general I think it's a good idea to have this separated from
launchpad itself and splitted into N files.
> - the environ context manager is duplicative of the
> EnvironmentVariable fixture; it would be nice not to rewrite code that
> has already been written and tested (and is used in our environment
> too!) Yes, I know that this is small, but it does all add up, and
> someone somewhere will eventually be cursing that they have to redo
> this / fix a unicode escaping issue in the code or whatever.
Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
Once this no longer is a single file, it will be easy to add dependencies.
Thanks again Robert.
Robert Collins (lifeless) wrote : | # |
On Tue, Feb 28, 2012 at 11:47 PM, Francesco Banconi <email address hidden> wrote:
> Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
> Once this no longer is a single file, it will be easy to add dependencies.
>
> Thanks again Robert.
No probs! And seriously, feel free to ignore the testr subcommand bit
- as I mentioned, its likely still a bit too opinionated for reuse,
and we should focus on delivering this lovely improvement in the short
term. If its in lp-dev-tools (or a new project), the complexity of it
will be contained, and you can iterate on this sort of polish in later
steps. (Or do it in whatever order Gary prefers: I'm trying to leave
it wide open what you do).
-Rob
- 14877. By Francesco Banconi
-
Fixed spelling errors.
- 14878. By Francesco Banconi
-
Try/Finally blocks in context managers.
- 14879. By Francesco Banconi
-
Removed unused function.
- 14880. By Francesco Banconi
-
Help fix.
- 14881. By Francesco Banconi
-
First implementation of the help subcommand.
Unmerged revisions
- 14881. By Francesco Banconi
-
First implementation of the help subcommand.
- 14880. By Francesco Banconi
-
Help fix.
- 14879. By Francesco Banconi
-
Removed unused function.
- 14878. By Francesco Banconi
-
Try/Finally blocks in context managers.
- 14877. By Francesco Banconi
-
Fixed spelling errors.
- 14876. By Francesco Banconi
-
Added tests to generate_ssh_keys helper.
- 14875. By Francesco Banconi
-
Initial commit.
Preview Diff
1 | === added file 'utilities/lpsetup.py' |
2 | --- utilities/lpsetup.py 1970-01-01 00:00:00 +0000 |
3 | +++ utilities/lpsetup.py 2012-02-28 17:19:19 +0000 |
4 | @@ -0,0 +1,1730 @@ |
5 | +#!/usr/bin/env python |
6 | +# Copyright 2012 Canonical Ltd. This software is licensed under the |
7 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
8 | + |
9 | +"""Create and update Launchpad development and testing environments.""" |
10 | + |
11 | +__metaclass__ = type |
12 | +__all__ = [ |
13 | + 'ActionsBasedSubCommand', |
14 | + 'apt_get_install', |
15 | + 'ArgumentParser', |
16 | + 'BaseSubCommand', |
17 | + 'bzr_whois', |
18 | + 'call', |
19 | + 'cd', |
20 | + 'check_output', |
21 | + 'CommandError', |
22 | + 'create_lxc', |
23 | + 'environ', |
24 | + 'file_append', |
25 | + 'file_prepend', |
26 | + 'generate_ssh_keys', |
27 | + 'get_container_path', |
28 | + 'get_lxc_gateway', |
29 | + 'get_su_command', |
30 | + 'get_user_home', |
31 | + 'get_user_ids', |
32 | + 'initialize', |
33 | + 'InstallSubCommand', |
34 | + 'join_command', |
35 | + 'LaunchpadError', |
36 | + 'link_sourcecode_in_branches', |
37 | + 'LXCInstallSubCommand', |
38 | + 'lxc_in_state', |
39 | + 'lxc_running', |
40 | + 'lxc_stopped', |
41 | + 'make_launchpad', |
42 | + 'mkdirs', |
43 | + 'setup_apt', |
44 | + 'setup_codebase', |
45 | + 'setup_external_sourcecode', |
46 | + 'setup_launchpad', |
47 | + 'setup_launchpad_lxc', |
48 | + 'ssh', |
49 | + 'SSHError', |
50 | + 'start_lxc', |
51 | + 'stop_lxc', |
52 | + 'su', |
53 | + 'this_command', |
54 | + 'update_launchpad', |
55 | + 'user_exists', |
56 | + 'ValidationError', |
57 | + 'wait_for_lxc', |
58 | + ] |
59 | + |
60 | +# To run doctests: python -m doctest -v lpsetup.py |
61 | + |
62 | +from collections import namedtuple |
63 | +from contextlib import contextmanager |
64 | +from email.Utils import parseaddr, formataddr |
65 | +from functools import partial |
66 | +import argparse |
67 | +import errno |
68 | +import os |
69 | +import pipes |
70 | +import platform |
71 | +import pwd |
72 | +import shutil |
73 | +import subprocess |
74 | +import sys |
75 | +import time |
76 | + |
77 | + |
78 | +APT_REPOSITORIES = ( |
79 | + 'deb http://archive.ubuntu.com/ubuntu {distro} multiverse', |
80 | + 'deb http://archive.ubuntu.com/ubuntu {distro}-updates multiverse', |
81 | + 'deb http://archive.ubuntu.com/ubuntu {distro}-security multiverse', |
82 | + 'ppa:launchpad/ppa', |
83 | + 'ppa:bzr/ppa', |
84 | + ) |
85 | +BASE_PACKAGES = ['ssh', 'bzr', 'language-pack-en'] |
86 | +CHECKOUT_DIR = '~/launchpad/branches' |
87 | +DEPENDENCIES_DIR = '~/launchpad/dependencies' |
88 | +DHCP_FILE = '/etc/dhcp/dhclient.conf' |
89 | +HOSTS_CONTENT = ( |
90 | + ('127.0.0.88', |
91 | + 'launchpad.dev answers.launchpad.dev archive.launchpad.dev ' |
92 | + 'api.launchpad.dev bazaar-internal.launchpad.dev beta.launchpad.dev ' |
93 | + 'blueprints.launchpad.dev bugs.launchpad.dev code.launchpad.dev ' |
94 | + 'feeds.launchpad.dev id.launchpad.dev keyserver.launchpad.dev ' |
95 | + 'lists.launchpad.dev openid.launchpad.dev ' |
96 | + 'ubuntu-openid.launchpad.dev ppa.launchpad.dev ' |
97 | + 'private-ppa.launchpad.dev testopenid.dev translations.launchpad.dev ' |
98 | + 'xmlrpc-private.launchpad.dev xmlrpc.launchpad.dev'), |
99 | + ('127.0.0.99', 'bazaar.launchpad.dev'), |
100 | + ) |
101 | +HOSTS_FILE = '/etc/hosts' |
102 | +LP_APACHE_MODULES = 'proxy proxy_http rewrite ssl deflate headers' |
103 | +LP_APACHE_ROOTS = ( |
104 | + '/var/tmp/bazaar.launchpad.dev/static', |
105 | + '/var/tmp/bazaar.launchpad.dev/mirrors', |
106 | + '/var/tmp/archive', |
107 | + '/var/tmp/ppa', |
108 | + ) |
109 | +LP_BZR_LOCATIONS = ( |
110 | + ('submit_branch', '{checkout_dir}'), |
111 | + ('public_branch', 'bzr+ssh://bazaar.launchpad.net/~{lpuser}/launchpad'), |
112 | + ('public_branch:policy', 'appendpath'), |
113 | + ('push_location', 'lp:~{lpuser}/launchpad'), |
114 | + ('push_location:policy', 'appendpath'), |
115 | + ('merge_target', '{checkout_dir}'), |
116 | + ('submit_to', 'merge@code.launchpad.net'), |
117 | + ) |
118 | +LP_CHECKOUT = 'devel' |
119 | +LP_PACKAGES = [ |
120 | + 'bzr', 'launchpad-developer-dependencies', 'apache2', |
121 | + 'apache2-mpm-worker', 'libapache2-mod-wsgi' |
122 | + ] |
123 | +LP_REPOS = ( |
124 | + 'http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel', |
125 | + 'lp:launchpad', |
126 | + ) |
127 | +LP_SOURCE_DEPS = ( |
128 | + 'http://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk') |
129 | +LXC_CONFIG_TEMPLATE = '/etc/lxc/local.conf' |
130 | +LXC_GUEST_ARCH = 'i386' |
131 | +LXC_GUEST_CHOICES = ('lucid', 'oneiric', 'precise') |
132 | +LXC_GUEST_OS = LXC_GUEST_CHOICES[0] |
133 | +LXC_NAME = 'lptests' |
134 | +LXC_OPTIONS = """ |
135 | +lxc.network.type = veth |
136 | +lxc.network.link = {interface} |
137 | +lxc.network.flags = up |
138 | +""" |
139 | +LXC_PACKAGES = ['lxc', 'libvirt-bin'] |
140 | +LXC_PATH = '/var/lib/lxc/' |
141 | +RESOLV_FILE = '/etc/resolv.conf' |
142 | + |
143 | + |
144 | +Env = namedtuple('Env', 'uid gid home') |
145 | + |
146 | + |
147 | +class LaunchpadError(Exception): |
148 | + """Base exception for lpsetup.""" |
149 | + |
150 | + |
151 | +class CommandError(LaunchpadError): |
152 | + """Errors occurred running shell commands.""" |
153 | + |
154 | + |
155 | +class SSHError(LaunchpadError): |
156 | + """Errors occurred during SSH connection.""" |
157 | + |
158 | + |
159 | +class ValidationError(LaunchpadError): |
160 | + """Argparse invalid arguments.""" |
161 | + |
162 | + |
163 | +def apt_get_install(*args): |
164 | + """Install given packages using apt.""" |
165 | + with environ(DEBIAN_FRONTEND='noninteractive'): |
166 | + cmd = ('apt-get', '-y', 'install') + args |
167 | + return call(*cmd) |
168 | + |
169 | + |
170 | +def join_command(args): |
171 | + """Return a valid Unix command line from `args`. |
172 | + |
173 | + >>> join_command(['ls', '-l']) |
174 | + 'ls -l' |
175 | + |
176 | + Arguments containing spaces and empty args are correctly quoted:: |
177 | + |
178 | + >>> join_command(['command', 'arg1', 'arg containing spaces', '']) |
179 | + "command arg1 'arg containing spaces' ''" |
180 | + """ |
181 | + return ' '.join(pipes.quote(arg) for arg in args) |
182 | + |
183 | + |
184 | +def bzr_whois(user): |
185 | + """Return full name and email of bzr `user`. |
186 | + |
187 | + Return None if the given `user` does not have a bzr user id. |
188 | + """ |
189 | + with su(user): |
190 | + try: |
191 | + whoami = check_output('bzr', 'whoami') |
192 | + except (CommandError, OSError): |
193 | + return None |
194 | + return parseaddr(whoami) |
195 | + |
196 | + |
197 | +@contextmanager |
198 | +def cd(directory): |
199 | + """A context manager to temporarily change current working dir, e.g.:: |
200 | + |
201 | + >>> import os |
202 | + >>> os.chdir('/tmp') |
203 | + >>> with cd('/bin'): print os.getcwd() |
204 | + /bin |
205 | + >>> print os.getcwd() |
206 | + /tmp |
207 | + """ |
208 | + cwd = os.getcwd() |
209 | + os.chdir(directory) |
210 | + try: |
211 | + yield |
212 | + finally: |
213 | + os.chdir(cwd) |
214 | + |
215 | + |
216 | +def check_output(*args, **kwargs): |
217 | + r"""Run the command with the given arguments. |
218 | + |
219 | + The first argument is the path to the command to run. |
220 | + Subsequent arguments are command-line arguments to be passed. |
221 | + |
222 | + Usually the output is captured by a pipe and returned:: |
223 | + |
224 | + >>> check_output('echo', 'output') |
225 | + 'output\n' |
226 | + |
227 | + A `CommandError` exception is raised if the return code is not zero:: |
228 | + |
229 | + >>> check_output('ls', '--not a valid switch', stderr=subprocess.PIPE) |
230 | + Traceback (most recent call last): |
231 | + CommandError: Error running: ls '--not a valid switch' |
232 | + |
233 | + None arguments are ignored:: |
234 | + |
235 | + >>> check_output(None, 'echo', None, 'output') |
236 | + 'output\n' |
237 | + """ |
238 | + args = [i for i in args if i is not None] |
239 | + process = subprocess.Popen( |
240 | + args, stdout=kwargs.pop('stdout', subprocess.PIPE), |
241 | + close_fds=True, **kwargs) |
242 | + stdout, stderr = process.communicate() |
243 | + retcode = process.poll() |
244 | + if retcode: |
245 | + raise CommandError('Error running: ' + join_command(args)) |
246 | + return stdout |
247 | + |
248 | + |
249 | +call = partial(check_output, stdout=None) |
250 | + |
251 | + |
252 | +@contextmanager |
253 | +def environ(**kwargs): |
254 | + """A context manager to temporarily change environment variables. |
255 | + |
256 | + If an existing environment variable is changed, it is restored during |
257 | + context cleanup:: |
258 | + |
259 | + >>> import os |
260 | + >>> os.environ['MY_VARIABLE'] = 'foo' |
261 | + >>> with environ(MY_VARIABLE='bar'): print os.getenv('MY_VARIABLE') |
262 | + bar |
263 | + >>> print os.getenv('MY_VARIABLE') |
264 | + foo |
265 | + >>> del os.environ['MY_VARIABLE'] |
266 | + |
267 | + If we are adding environment variables, they are removed during context |
268 | + cleanup:: |
269 | + |
270 | + >>> import os |
271 | + >>> with environ(MY_VAR1='foo', MY_VAR2='bar'): |
272 | + ... print os.getenv('MY_VAR1'), os.getenv('MY_VAR2') |
273 | + foo bar |
274 | + >>> os.getenv('MY_VAR1') == os.getenv('MY_VAR2') == None |
275 | + True |
276 | + """ |
277 | + backup = {} |
278 | + for key, value in kwargs.items(): |
279 | + backup[key] = os.getenv(key) |
280 | + os.environ[key] = value |
281 | + try: |
282 | + yield |
283 | + finally: |
284 | + for key, value in backup.items(): |
285 | + if value is None: |
286 | + del os.environ[key] |
287 | + else: |
288 | + os.environ[key] = value |
289 | + |
290 | + |
291 | +def file_append(filename, line): |
292 | + r"""Append given `line`, if not present, at the end of `filename`. |
293 | + |
294 | + Usage example:: |
295 | + |
296 | + >>> import tempfile |
297 | + >>> f = tempfile.NamedTemporaryFile('w', delete=False) |
298 | + >>> f.write('line1\n') |
299 | + >>> f.close() |
300 | + >>> file_append(f.name, 'new line\n') |
301 | + >>> open(f.name).read() |
302 | + 'line1\nnew line\n' |
303 | + |
304 | + Nothing happens if the file already contains the given `line`:: |
305 | + |
306 | + >>> file_append(f.name, 'new line\n') |
307 | + >>> open(f.name).read() |
308 | + 'line1\nnew line\n' |
309 | + |
310 | + A new line is automatically added before the given `line` if it is not |
311 | + present at the end of current file content:: |
312 | + |
313 | + >>> import tempfile |
314 | + >>> f = tempfile.NamedTemporaryFile('w', delete=False) |
315 | + >>> f.write('line1') |
316 | + >>> f.close() |
317 | + >>> file_append(f.name, 'new line\n') |
318 | + >>> open(f.name).read() |
319 | + 'line1\nnew line\n' |
320 | + |
321 | + The file is created if it does not exist:: |
322 | + |
323 | + >>> import tempfile |
324 | + >>> filename = tempfile.mktemp() |
325 | + >>> file_append(filename, 'line1\n') |
326 | + >>> open(filename).read() |
327 | + 'line1\n' |
328 | + """ |
329 | + with open(filename, 'a+') as f: |
330 | + content = f.read() |
331 | + if line not in content: |
332 | + if content.endswith('\n') or not content: |
333 | + f.write(line) |
334 | + else: |
335 | + f.write('\n' + line) |
336 | + |
337 | + |
338 | +def file_prepend(filename, line): |
339 | + r"""Insert given `line`, if not present, at the beginning of `filename`. |
340 | + |
341 | + Usage example:: |
342 | + |
343 | + >>> import tempfile |
344 | + >>> f = tempfile.NamedTemporaryFile('w', delete=False) |
345 | + >>> f.write('line1\n') |
346 | + >>> f.close() |
347 | + >>> file_prepend(f.name, 'line0\n') |
348 | + >>> open(f.name).read() |
349 | + 'line0\nline1\n' |
350 | + |
351 | + If the file starts with the given `line`, nothing happens:: |
352 | + |
353 | + >>> file_prepend(f.name, 'line0\n') |
354 | + >>> open(f.name).read() |
355 | + 'line0\nline1\n' |
356 | + |
357 | + If the file contains the given `line`, but not at the beginning, |
358 | + the line is moved on top:: |
359 | + |
360 | + >>> file_prepend(f.name, 'line1\n') |
361 | + >>> open(f.name).read() |
362 | + 'line1\nline0\n' |
363 | + """ |
364 | + with open(filename, 'r+') as f: |
365 | + lines = f.readlines() |
366 | + if lines[0] != line: |
367 | + if line in lines: |
368 | + lines.remove(line) |
369 | + lines.insert(0, line) |
370 | + f.seek(0) |
371 | + f.writelines(lines) |
372 | + |
373 | + |
374 | +def generate_ssh_keys(directory, filename='id_rsa'): |
375 | + """Generate ssh key pair, saving them inside the given `directory`. |
376 | + |
377 | + >>> generate_ssh_keys('/tmp/') |
378 | + >>> open('/tmp/id_rsa').readlines()[0].strip() |
379 | + '-----BEGIN RSA PRIVATE KEY-----' |
380 | + >>> open('/tmp/id_rsa.pub').read().startswith('ssh-rsa') |
381 | + True |
382 | + >>> os.remove('/tmp/id_rsa') |
383 | + >>> os.remove('/tmp/id_rsa.pub') |
384 | + |
385 | + The key filename can be changed using the `filename` keyword argument |
386 | + (default is 'id_rsa'):: |
387 | + |
388 | + >>> generate_ssh_keys('/tmp/', 'custom_key') |
389 | + >>> os.path.exists('/tmp/custom_key') |
390 | + True |
391 | + >>> os.path.exists('/tmp/id_rsa') |
392 | + False |
393 | + |
394 | + >>> os.remove('/tmp/custom_key') |
395 | + >>> os.remove('/tmp/custom_key.pub') |
396 | + """ |
397 | + path = os.path.join(directory, filename) |
398 | + return call('ssh-keygen', '-q', '-t', 'rsa', '-N', '', '-f', path) |
399 | + |
400 | + |
401 | +def get_container_path(lxc_name, path='', base_path=LXC_PATH): |
402 | + """Return the path of LXC container called `lxc_name`. |
403 | + |
404 | + If a `path` is given, return that path inside the container, e.g.:: |
405 | + |
406 | + >>> get_container_path('mycontainer') |
407 | + '/var/lib/lxc/mycontainer/rootfs/' |
408 | + >>> get_container_path('mycontainer', '/etc/apt/') |
409 | + '/var/lib/lxc/mycontainer/rootfs/etc/apt/' |
410 | + >>> get_container_path('mycontainer', 'home') |
411 | + '/var/lib/lxc/mycontainer/rootfs/home' |
412 | + """ |
413 | + return os.path.join(base_path, lxc_name, 'rootfs', path.lstrip('/')) |
414 | + |
415 | + |
416 | +def get_lxc_gateway(): |
417 | + """Return a tuple of gateway name and address. |
418 | + |
419 | + The gateway name and address will change depending on which version |
420 | + of Ubuntu the script is running on. |
421 | + """ |
422 | + release_name = platform.linux_distribution()[2] |
423 | + if release_name == 'oneiric': |
424 | + return 'virbr0', '192.168.122.1' |
425 | + else: |
426 | + return 'lxcbr0', '10.0.3.1' |
427 | + |
428 | + |
429 | +def get_su_command(user, args): |
430 | + """Return a command line as a sequence, prepending "su" if necessary. |
431 | + |
432 | + This can be used together with `call` or `check_output` when the `su` |
433 | + context manager is not enaugh (e.g. an external program uses uid rather |
434 | + than euid). |
435 | + |
436 | + >>> import getpass |
437 | + >>> current_user = getpass.getuser() |
438 | + |
439 | + If the su is requested as current user, the arguments are returned as |
440 | + given:: |
441 | + |
442 | + >>> get_su_command(current_user, ('ls', '-l')) |
443 | + ('ls', '-l') |
444 | + |
445 | + Otherwise, "su" is prepended:: |
446 | + |
447 | + >>> get_su_command('nobody', ('ls', '-l', 'my file')) |
448 | + ('su', 'nobody', '-c', "ls -l 'my file'") |
449 | + """ |
450 | + if get_user_ids(user)[0] != os.getuid(): |
451 | + args = [i for i in args if i is not None] |
452 | + return ('su', user, '-c', join_command(args)) |
453 | + return args |
454 | + |
455 | + |
456 | +def get_user_home(user): |
457 | + """Return the home directory of the given `user`. |
458 | + |
459 | + >>> get_user_home('root') |
460 | + '/root' |
461 | + |
462 | + If the user does not exist, return a default /home/[username] home:: |
463 | + |
464 | + >>> get_user_home('_this_user_does_not_exist_') |
465 | + '/home/_this_user_does_not_exist_' |
466 | + """ |
467 | + try: |
468 | + return pwd.getpwnam(user).pw_dir |
469 | + except KeyError: |
470 | + return os.path.join(os.path.sep, 'home', user) |
471 | + |
472 | + |
473 | +def get_user_ids(user): |
474 | + """Return the uid and gid of given `user`, e.g.:: |
475 | + |
476 | + >>> get_user_ids('root') |
477 | + (0, 0) |
478 | + """ |
479 | + userdata = pwd.getpwnam(user) |
480 | + return userdata.pw_uid, userdata.pw_gid |
481 | + |
482 | + |
483 | +def lxc_in_state(state, lxc_name, timeout=30): |
484 | + """Return True if the LXC named `lxc_name` is in state `state`. |
485 | + |
486 | + Return False otherwise. |
487 | + """ |
488 | + while timeout: |
489 | + try: |
490 | + output = check_output( |
491 | + 'lxc-info', '-n', lxc_name, stderr=subprocess.STDOUT) |
492 | + except CommandError: |
493 | + pass |
494 | + else: |
495 | + if state in output: |
496 | + return True |
497 | + timeout -= 1 |
498 | + time.sleep(1) |
499 | + return False |
500 | + |
501 | + |
502 | +lxc_running = partial(lxc_in_state, 'RUNNING') |
503 | +lxc_stopped = partial(lxc_in_state, 'STOPPED') |
504 | + |
505 | + |
506 | +def mkdirs(*args): |
507 | + """Create leaf directories (given as `args`) and all intermediate ones. |
508 | + |
509 | + >>> import tempfile |
510 | + >>> base_dir = tempfile.mktemp(suffix='/') |
511 | + >>> dir1 = tempfile.mktemp(prefix=base_dir) |
512 | + >>> dir2 = tempfile.mktemp(prefix=base_dir) |
513 | + >>> mkdirs(dir1, dir2) |
514 | + >>> os.path.isdir(dir1) |
515 | + True |
516 | + >>> os.path.isdir(dir2) |
517 | + True |
518 | + |
519 | + If the leaf directory already exists the function returns without errors:: |
520 | + |
521 | + >>> mkdirs(dir1) |
522 | + |
523 | + An `OSError` is raised if the leaf path exists and it is a file:: |
524 | + |
525 | + >>> f = tempfile.NamedTemporaryFile( |
526 | + ... 'w', delete=False, prefix=base_dir) |
527 | + >>> f.close() |
528 | + >>> mkdirs(f.name) # doctest: +ELLIPSIS |
529 | + Traceback (most recent call last): |
530 | + OSError: ... |
531 | + """ |
532 | + for directory in args: |
533 | + try: |
534 | + os.makedirs(directory) |
535 | + except OSError as err: |
536 | + if err.errno != errno.EEXIST or os.path.isfile(directory): |
537 | + raise |
538 | + |
539 | + |
540 | +def ssh(location, user=None, caller=subprocess.call): |
541 | + """Return a callable that can be used to run ssh shell commands. |
542 | + |
543 | + The ssh `location` and, optionally, `user` must be given. |
544 | + If the user is None then the current user is used for the connection. |
545 | + |
546 | + The callable internally uses the given `caller`:: |
547 | + |
548 | + >>> def caller(cmd): |
549 | + ... print cmd |
550 | + >>> sshcall = ssh('example.com', 'myuser', caller=caller) |
551 | + >>> root_sshcall = ssh('example.com', caller=caller) |
552 | + >>> sshcall('ls -l') # doctest: +ELLIPSIS |
553 | + ('ssh', '-t', ..., 'myuser@example.com', '--', 'ls -l') |
554 | + >>> root_sshcall('ls -l') # doctest: +ELLIPSIS |
555 | + ('ssh', '-t', ..., 'example.com', '--', 'ls -l') |
556 | + |
557 | + If the ssh command exits with an error code, an `SSHError` is raised:: |
558 | + |
559 | + >>> ssh('loc', caller=lambda cmd: 1)('ls -l') # doctest: +ELLIPSIS |
560 | + Traceback (most recent call last): |
561 | + SSHError: ... |
562 | + |
563 | + If ignore_errors is set to True when executing the command, no error |
564 | + will be raised, even if the command itself returns an error code. |
565 | + |
566 | + >>> sshcall = ssh('loc', caller=lambda cmd: 1) |
567 | + >>> sshcall('ls -l', ignore_errors=True) |
568 | + """ |
569 | + if user is not None: |
570 | + location = '{user}@{location}'.format(user=user, location=location) |
571 | + |
572 | + def _sshcall(cmd, ignore_errors=False): |
573 | + sshcmd = ( |
574 | + 'ssh', |
575 | + '-t', |
576 | + '-t', # Yes, this second -t is deliberate. See `man ssh`. |
577 | + '-o', 'StrictHostKeyChecking=no', |
578 | + '-o', 'UserKnownHostsFile=/dev/null', |
579 | + location, |
580 | + '--', cmd, |
581 | + ) |
582 | + if caller(sshcmd) and not ignore_errors: |
583 | + raise SSHError('Error running command: ' + ' '.join(sshcmd)) |
584 | + |
585 | + return _sshcall |
586 | + |
587 | + |
588 | +@contextmanager |
589 | +def su(user): |
590 | + """A context manager to temporarily run the script as a different user.""" |
591 | + uid, gid = get_user_ids(user) |
592 | + os.setegid(gid) |
593 | + os.seteuid(uid) |
594 | + home = get_user_home(user) |
595 | + with environ(HOME=home): |
596 | + try: |
597 | + yield Env(uid, gid, home) |
598 | + finally: |
599 | + os.setegid(os.getgid()) |
600 | + os.seteuid(os.getuid()) |
601 | + |
602 | + |
603 | +def this_command(directory, args): |
604 | + """Return a command line to re-launch this script using given `args`. |
605 | + |
606 | + The command returned will execute this script from `directory`:: |
607 | + |
608 | + >>> import os |
609 | + >>> script_name = os.path.basename(__file__) |
610 | + |
611 | + >>> cmd = this_command('/home/user/lp/branches', ['install']) |
612 | + >>> cmd == ('cd /home/user/lp/branches && devel/utilities/' + |
613 | + ... script_name + ' install') |
614 | + True |
615 | + |
616 | + Arguments are correctly quoted:: |
617 | + |
618 | + >>> cmd = this_command('/path', ['-arg', 'quote me']) |
619 | + >>> cmd == ('cd /path && devel/utilities/' + |
620 | + ... script_name + " -arg 'quote me'") |
621 | + True |
622 | + """ |
623 | + script = join_command([ |
624 | + os.path.join(LP_CHECKOUT, 'utilities', os.path.basename(__file__)), |
625 | + ] + list(args)) |
626 | + return join_command(['cd', directory]) + ' && ' + script |
627 | + |
628 | + |
629 | +def user_exists(username): |
630 | + """Return True if given `username` exists, e.g.:: |
631 | + |
632 | + >>> user_exists('root') |
633 | + True |
634 | + >>> user_exists('_this_user_does_not_exist_') |
635 | + False |
636 | + """ |
637 | + try: |
638 | + pwd.getpwnam(username) |
639 | + except KeyError: |
640 | + return False |
641 | + return True |
642 | + |
643 | + |
644 | +def setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir): |
645 | + """Set up Launchpad repository and source dependencies. |
646 | + |
647 | + Return True if new changes are pulled from bzr repository. |
648 | + """ |
649 | + # Using real su because bzr uses uid. |
650 | + if os.path.exists(checkout_dir): |
651 | + # Pull the repository. |
652 | + revno_args = ('bzr', 'revno', checkout_dir) |
653 | + revno = check_output(*revno_args) |
654 | + call(*get_su_command(user, ['bzr', 'pull', '-d', checkout_dir])) |
655 | + changed = revno != check_output(*revno_args) |
656 | + else: |
657 | + # Branch the repository. |
658 | + cmd = ('bzr', 'branch', |
659 | + LP_REPOS[1] if valid_ssh_keys else LP_REPOS[0], checkout_dir) |
660 | + call(*get_su_command(user, cmd)) |
661 | + changed = True |
662 | + # Check repository integrity. |
663 | + if subprocess.call(['bzr', 'status', '-q', checkout_dir]): |
664 | + raise LaunchpadError( |
665 | + 'Repository {0} is corrupted.'.format(checkout_dir)) |
666 | + # Set up source dependencies. |
667 | + with su(user): |
668 | + for subdir in ('eggs', 'yui', 'sourcecode'): |
669 | + mkdirs(os.path.join(dependencies_dir, subdir)) |
670 | + download_cache = os.path.join(dependencies_dir, 'download-cache') |
671 | + if os.path.exists(download_cache): |
672 | + call('bzr', 'up', download_cache) |
673 | + else: |
674 | + call('bzr', 'co', '--lightweight', LP_SOURCE_DEPS, download_cache) |
675 | + return changed |
676 | + |
677 | + |
678 | +def setup_external_sourcecode( |
679 | + user, valid_ssh_keys, checkout_dir, dependencies_dir): |
680 | + """Update and link external sourcecode.""" |
681 | + cmd = ( |
682 | + 'utilities/update-sourcecode', |
683 | + None if valid_ssh_keys else '--use-http', |
684 | + os.path.join(dependencies_dir, 'sourcecode'), |
685 | + ) |
686 | + with cd(checkout_dir): |
687 | + # Using real su because update-sourcecode uses uid. |
688 | + call(*get_su_command(user, cmd)) |
689 | + with su(user): |
690 | + call('utilities/link-external-sourcecode', dependencies_dir) |
691 | + |
692 | + |
693 | +def make_launchpad(user, checkout_dir, install=False): |
694 | + """Make and optionally install Launchpad.""" |
695 | + # Using real su because mailman make script uses uid. |
696 | + call(*get_su_command(user, ['make', '-C', checkout_dir])) |
697 | + if install: |
698 | + call('make', '-C', checkout_dir, 'install') |
699 | + |
700 | + |
701 | +def initialize( |
702 | + user, full_name, email, lpuser, private_key, public_key, valid_ssh_keys, |
703 | + dependencies_dir, directory): |
704 | + """Initialize host machine.""" |
705 | + # Install necessary deb packages. This requires Oneiric or later. |
706 | + call('apt-get', 'update') |
707 | + apt_get_install(*BASE_PACKAGES) |
708 | + # Create the user (if he does not exist). |
709 | + if not user_exists(user): |
710 | + call('useradd', '-m', '-s', '/bin/bash', '-U', user) |
711 | + # Generate root ssh keys if they do not exist. |
712 | + if not os.path.exists('/root/.ssh/id_rsa.pub'): |
713 | + generate_ssh_keys('/root/.ssh/') |
714 | + with su(user) as env: |
715 | + # Set up the user's ssh directory. The ssh key must be associated |
716 | + # with the lpuser's Launchpad account. |
717 | + ssh_dir = os.path.join(env.home, '.ssh') |
718 | + mkdirs(ssh_dir) |
719 | + # Generate user ssh keys if none are supplied. |
720 | + if not valid_ssh_keys: |
721 | + generate_ssh_keys(ssh_dir) |
722 | + private_key = open(os.path.join(ssh_dir, 'id_rsa')).read() |
723 | + public_key = open(os.path.join(ssh_dir, 'id_rsa.pub')).read() |
724 | + priv_file = os.path.join(ssh_dir, 'id_rsa') |
725 | + pub_file = os.path.join(ssh_dir, 'id_rsa.pub') |
726 | + auth_file = os.path.join(ssh_dir, 'authorized_keys') |
727 | + known_hosts = os.path.join(ssh_dir, 'known_hosts') |
728 | + known_host_content = check_output( |
729 | + 'ssh-keyscan', '-t', 'rsa', 'bazaar.launchpad.net') |
730 | + for filename, contents, mode in [ |
731 | + (priv_file, private_key, 'w'), |
732 | + (pub_file, public_key, 'w'), |
733 | + (auth_file, public_key, 'a'), |
734 | + (known_hosts, known_host_content, 'a'), |
735 | + ]: |
736 | + with open(filename, mode) as f: |
737 | + f.write(contents + '\n') |
738 | + os.chmod(filename, 0644) |
739 | + os.chmod(priv_file, 0600) |
740 | + # Set up bzr and Launchpad authentication. |
741 | + call('bzr', 'whoami', formataddr([full_name, email])) |
742 | + if valid_ssh_keys: |
743 | + subprocess.call(['bzr', 'lp-login', lpuser]) |
744 | + # Set up the repository. |
745 | + mkdirs(directory) |
746 | + call('bzr', 'init-repo', '--2a', directory) |
747 | + # Set up the codebase. |
748 | + checkout_dir = os.path.join(directory, LP_CHECKOUT) |
749 | + setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir) |
750 | + # Set up bzr locations |
751 | + tmpl = ''.join('{0} = {1}\n'.format(k, v) for k, v in LP_BZR_LOCATIONS) |
752 | + locations = tmpl.format(checkout_dir=checkout_dir, lpuser=lpuser) |
753 | + contents = '[{0}]\n'.format(directory) + locations |
754 | + with su(user) as env: |
755 | + bzr_locations = os.path.join(env.home, '.bazaar', 'locations.conf') |
756 | + file_append(bzr_locations, contents) |
757 | + |
758 | + |
759 | +def setup_apt(no_repositories=True): |
760 | + """Setup, update and upgrade deb packages.""" |
761 | + if not no_repositories: |
762 | + distro = check_output('lsb_release', '-cs').strip() |
763 | + # APT repository update. |
764 | + for reposirory in APT_REPOSITORIES: |
765 | + assume_yes = None if distro == 'lucid' else '-y' |
766 | + call('add-apt-repository', assume_yes, |
767 | + reposirory.format(distro=distro)) |
768 | + # XXX frankban 2012-01-13 - Bug 892892: upgrading mountall in LXC |
769 | + # containers currently does not work. |
770 | + call("echo 'mountall hold' | dpkg --set-selections", shell=True) |
771 | + call('apt-get', 'update') |
772 | + # Install base and Launchpad deb packages. |
773 | + apt_get_install(*LP_PACKAGES) |
774 | + |
775 | + |
776 | +def setup_launchpad(user, dependencies_dir, directory, valid_ssh_keys): |
777 | + """Set up the Launchpad environment.""" |
778 | + # User configuration. |
779 | + subprocess.call(['adduser', user, 'sudo']) |
780 | + gid = pwd.getpwnam(user).pw_gid |
781 | + subprocess.call(['addgroup', '--gid', str(gid), user]) |
782 | + # Set up Launchpad dependencies. |
783 | + checkout_dir = os.path.join(directory, LP_CHECKOUT) |
784 | + setup_external_sourcecode( |
785 | + user, valid_ssh_keys, checkout_dir, dependencies_dir) |
786 | + with su(user): |
787 | + # Create Apache document roots, to avoid warnings. |
788 | + mkdirs(*LP_APACHE_ROOTS) |
789 | + # Set up Apache modules. |
790 | + for module in LP_APACHE_MODULES.split(): |
791 | + call('a2enmod', module) |
792 | + with cd(checkout_dir): |
793 | + # Launchpad database setup. |
794 | + call('utilities/launchpad-database-setup', user) |
795 | + # Make and install launchpad. |
796 | + make_launchpad(user, checkout_dir, install=True) |
797 | + # Set up container hosts file. |
798 | + lines = ['{0}\t{1}'.format(ip, names) for ip, names in HOSTS_CONTENT] |
799 | + file_append(HOSTS_FILE, '\n'.join(lines)) |
800 | + |
801 | + |
802 | +def update_launchpad(user, valid_ssh_keys, dependencies_dir, directory, apt): |
803 | + """Update the Launchpad environment.""" |
804 | + if apt: |
805 | + setup_apt(no_repositories=True) |
806 | + checkout_dir = os.path.join(directory, LP_CHECKOUT) |
807 | + # Update the Launchpad codebase. |
808 | + changed = setup_codebase( |
809 | + user, valid_ssh_keys, checkout_dir, dependencies_dir) |
810 | + setup_external_sourcecode( |
811 | + user, valid_ssh_keys, checkout_dir, dependencies_dir) |
812 | + if changed: |
813 | + make_launchpad(user, checkout_dir, install=False) |
814 | + |
815 | + |
816 | +def link_sourcecode_in_branches(user, dependencies_dir, directory): |
817 | + """Link external sourcecode for all branches in the project.""" |
818 | + checkout_dir = os.path.join(directory, LP_CHECKOUT) |
819 | + cmd = os.path.join(checkout_dir, 'utilities', 'link-external-sourcecode') |
820 | + with su(user): |
821 | + for dirname in os.listdir(directory): |
822 | + branch = os.path.join(directory, dirname) |
823 | + sourcecode_dir = os.path.join(branch, 'sourcecode') |
824 | + if (branch != checkout_dir and |
825 | + os.path.exists(sourcecode_dir) and |
826 | + os.path.isdir(sourcecode_dir)): |
827 | + call(cmd, '--parent', dependencies_dir, '--target', branch) |
828 | + |
829 | + |
830 | +def create_lxc(user, lxc_name, lxc_arch, lxc_os): |
831 | + """Create the LXC named `lxc_name` sharing `user` home directory. |
832 | + |
833 | + The container will be used as development environment or as base template |
834 | + for parallel testing using ephemeral instances. |
835 | + """ |
836 | + # Install necessary deb packages. |
837 | + apt_get_install(*LXC_PACKAGES) |
838 | + # XXX 2012-02-02 gmb bug=925024: |
839 | + # These calls need to be removed once the lxc vs. apparmor bug |
840 | + # is resolved, since having apparmor enabled for lxc is very |
841 | + # much a Good Thing. |
842 | + # Disable the apparmor profiles for lxc so that we don't have |
843 | + # problems installing postgres. |
844 | + call('ln', '-s', |
845 | + '/etc/apparmor.d/usr.bin.lxc-start', '/etc/apparmor.d/disable/') |
846 | + call('apparmor_parser', '-R', '/etc/apparmor.d/usr.bin.lxc-start') |
847 | + # Update resolv file in order to get the ability to ssh into the LXC |
848 | + # container using its name. |
849 | + lxc_gateway_name, lxc_gateway_address = get_lxc_gateway() |
850 | + file_prepend(RESOLV_FILE, 'nameserver {0}\n'.format(lxc_gateway_address)) |
851 | + file_append( |
852 | + DHCP_FILE, |
853 | + 'prepend domain-name-servers {0};\n'.format(lxc_gateway_address)) |
854 | + # Container configuration template. |
855 | + content = LXC_OPTIONS.format(interface=lxc_gateway_name) |
856 | + with open(LXC_CONFIG_TEMPLATE, 'w') as f: |
857 | + f.write(content) |
858 | + # Creating container. |
859 | + call( |
860 | + 'lxc-create', |
861 | + '-t', 'ubuntu', |
862 | + '-n', lxc_name, |
863 | + '-f', LXC_CONFIG_TEMPLATE, |
864 | + '--', |
865 | + '-r {os} -a {arch} -b {user}'.format( |
866 | + os=lxc_os, arch=lxc_arch, user=user), |
867 | + ) |
868 | + # Set up root ssh key. |
869 | + user_authorized_keys = os.path.expanduser( |
870 | + '~' + user + '/.ssh/authorized_keys') |
871 | + with open(user_authorized_keys, 'a') as f: |
872 | + f.write(open('/root/.ssh/id_rsa.pub').read()) |
873 | + dst = get_container_path(lxc_name, '/root/.ssh/') |
874 | + mkdirs(dst) |
875 | + shutil.copy(user_authorized_keys, dst) |
876 | + |
877 | + |
878 | +def initialize_lxc(lxc_name, lxc_os): |
879 | + """Initialize LXC container.""" |
880 | + base_packages = list(BASE_PACKAGES) |
881 | + if lxc_os == 'lucid': |
882 | + # Install argparse to be able to run this script inside a lucid lxc. |
883 | + base_packages.append('python-argparse') |
884 | + ssh(lxc_name)( |
885 | + 'DEBIAN_FRONTEND=noninteractive ' |
886 | + 'apt-get install -y ' + ' '.join(base_packages)) |
887 | + |
888 | + |
889 | +def setup_launchpad_lxc( |
890 | + user, dependencies_dir, directory, valid_ssh_keys, lxc_name): |
891 | + """Set up the Launchpad environment inside an LXC.""" |
892 | + # Use ssh to call this script from inside the container. |
893 | + args = [ |
894 | + 'install', '-u', user, '-a', 'setup_apt', 'setup_launchpad', |
895 | + '-d', dependencies_dir, '-c', directory |
896 | + ] |
897 | + cmd = this_command(directory, args) |
898 | + ssh(lxc_name)(cmd) |
899 | + |
900 | + |
901 | +def start_lxc(lxc_name): |
902 | + """Start the lxc instance named `lxc_name`.""" |
903 | + call('lxc-start', '-n', lxc_name, '-d') |
904 | + |
905 | + |
906 | +def stop_lxc(lxc_name): |
907 | + """Stop the lxc instance named `lxc_name`.""" |
908 | + ssh(lxc_name)('poweroff') |
909 | + if not lxc_stopped(lxc_name): |
910 | + subprocess.call(['lxc-stop', '-n', lxc_name]) |
911 | + |
912 | + |
913 | +def wait_for_lxc(lxc_name, trials=60, sleep_seconds=1): |
914 | + """Try to ssh as `user` into the LXC container named `lxc_name`.""" |
915 | + sshcall = ssh(lxc_name) |
916 | + while True: |
917 | + trials -= 1 |
918 | + try: |
919 | + sshcall('true') |
920 | + except SSHError: |
921 | + if not trials: |
922 | + raise |
923 | + time.sleep(sleep_seconds) |
924 | + else: |
925 | + break |
926 | + |
927 | + |
928 | +def handle_user(namespace): |
929 | + """Handle user argument. |
930 | + |
931 | + This validator populates namespace with `home_dir` name:: |
932 | + |
933 | + >>> import getpass |
934 | + >>> username = getpass.getuser() |
935 | + |
936 | + >>> namespace = argparse.Namespace(user=username) |
937 | + |
938 | + >>> handle_user(namespace) |
939 | + >>> namespace.home_dir == '/home/' + username |
940 | + True |
941 | + |
942 | + The validation fails if the current user is root and no user is provided:: |
943 | + |
944 | + >>> namespace = argparse.Namespace(user=None, euid=0) |
945 | + >>> handle_user(namespace) # doctest: +ELLIPSIS |
946 | + Traceback (most recent call last): |
947 | + ValidationError: argument user ... |
948 | + """ |
949 | + if namespace.user is None: |
950 | + if not namespace.euid: |
951 | + raise ValidationError('argument user can not be omitted if ' |
952 | + 'the script is run as root.') |
953 | + namespace.user = pwd.getpwuid(namespace.euid).pw_name |
954 | + namespace.home_dir = get_user_home(namespace.user) |
955 | + |
956 | + |
957 | +def handle_lpuser(namespace): |
958 | + """Handle lpuser argument. |
959 | + |
960 | + If lpuser is not provided by namespace, the user name is used:: |
961 | + |
962 | + >>> import getpass |
963 | + >>> username = getpass.getuser() |
964 | + |
965 | + >>> namespace = argparse.Namespace(user=username, lpuser=None) |
966 | + >>> handle_lpuser(namespace) |
967 | + >>> namespace.lpuser == username |
968 | + True |
969 | + """ |
970 | + if namespace.lpuser is None: |
971 | + namespace.lpuser = namespace.user |
972 | + |
973 | + |
974 | +def handle_userdata(namespace, whois=bzr_whois): |
975 | + """Handle full_name and email arguments. |
976 | + |
977 | + If they are not provided, this function tries to obtain them using |
978 | + the given `whois` callable:: |
979 | + |
980 | + >>> namespace = argparse.Namespace( |
981 | + ... full_name=None, email=None, user='root') |
982 | + >>> email = 'email@example.com' |
983 | + >>> handle_userdata(namespace, lambda user: (user, email)) |
984 | + >>> namespace.full_name == namespace.user |
985 | + True |
986 | + >>> namespace.email == email |
987 | + True |
988 | + |
989 | + The validation fails if full_name or email are not provided and |
990 | + they can not be obtained using the `whois` callable:: |
991 | + |
992 | + >>> namespace = argparse.Namespace( |
993 | + ... full_name=None, email=None, user='root') |
994 | + >>> handle_userdata(namespace, lambda user: None) # doctest: +ELLIPSIS |
995 | + Traceback (most recent call last): |
996 | + ValidationError: arguments full-name ... |
997 | + |
998 | + >>> namespace = argparse.Namespace( |
999 | + ... full_name=None, email=None, user='this_user_does_not_exist') |
1000 | + >>> handle_userdata(namespace) # doctest: +ELLIPSIS |
1001 | + Traceback (most recent call last): |
1002 | + ValidationError: arguments full-name ... |
1003 | + |
1004 | + It does not make sense to provide only one argument:: |
1005 | + |
1006 | + >>> namespace = argparse.Namespace(full_name='Foo Bar', email=None) |
1007 | + >>> handle_userdata(namespace) # doctest: +ELLIPSIS |
1008 | + Traceback (most recent call last): |
1009 | + ValidationError: arguments full-name ... |
1010 | + """ |
1011 | + args = (namespace.full_name, namespace.email) |
1012 | + if not all(args): |
1013 | + if any(args): |
1014 | + raise ValidationError( |
1015 | + 'arguments full-name and email: ' |
1016 | + 'either none or both must be provided.') |
1017 | + if user_exists(namespace.user): |
1018 | + userdata = whois(namespace.user) |
1019 | + if userdata is None: |
1020 | + raise ValidationError( |
1021 | + 'arguments full-name and email are required: ' |
1022 | + 'bzr user id not found.') |
1023 | + namespace.full_name, namespace.email = userdata |
1024 | + else: |
1025 | + raise ValidationError( |
1026 | + 'arguments full-name and email are required: ' |
1027 | + 'system user not found.') |
1028 | + |
1029 | + |
1030 | +def handle_ssh_keys(namespace): |
1031 | + r"""Handle private and public ssh keys. |
1032 | + |
1033 | + Keys contained in the namespace are escaped:: |
1034 | + |
1035 | + >>> private = r'PRIVATE\nKEY' |
1036 | + >>> public = r'PUBLIC\nKEY' |
1037 | + >>> namespace = argparse.Namespace( |
1038 | + ... private_key=private, public_key=public) |
1039 | + >>> handle_ssh_keys(namespace) |
1040 | + >>> namespace.private_key == private.decode('string-escape') |
1041 | + True |
1042 | + >>> namespace.public_key == public.decode('string-escape') |
1043 | + True |
1044 | + >>> namespace.valid_ssh_keys |
1045 | + True |
1046 | + |
1047 | + Keys are None if they are not provided and can not be found in the |
1048 | + current home directory:: |
1049 | + |
1050 | + >>> namespace = argparse.Namespace( |
1051 | + ... private_key=None, home_dir='/tmp/__does_not_exists__') |
1052 | + >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS |
1053 | + >>> print namespace.private_key |
1054 | + None |
1055 | + >>> print namespace.public_key |
1056 | + None |
1057 | + >>> namespace.valid_ssh_keys |
1058 | + False |
1059 | + |
1060 | + If only one of private_key and public_key is provided, a |
1061 | + ValidationError will be raised. |
1062 | + |
1063 | + >>> namespace = argparse.Namespace( |
1064 | + ... private_key=private, public_key=None, |
1065 | + ... home_dir='/tmp/__does_not_exists__') |
1066 | + >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS |
1067 | + Traceback (most recent call last): |
1068 | + ValidationError: arguments private-key... |
1069 | + """ |
1070 | + namespace.valid_ssh_keys = True |
1071 | + for attr, filename in ( |
1072 | + ('private_key', 'id_rsa'), |
1073 | + ('public_key', 'id_rsa.pub')): |
1074 | + value = getattr(namespace, attr, None) |
1075 | + if value: |
1076 | + setattr(namespace, attr, value.decode('string-escape')) |
1077 | + else: |
1078 | + path = os.path.join(namespace.home_dir, '.ssh', filename) |
1079 | + try: |
1080 | + value = open(path).read() |
1081 | + except IOError: |
1082 | + value = None |
1083 | + namespace.valid_ssh_keys = False |
1084 | + setattr(namespace, attr, value) |
1085 | + if bool(namespace.private_key) != bool(namespace.public_key): |
1086 | + raise ValidationError( |
1087 | + "arguments private-key and public-key: " |
1088 | + "both must be provided or neither must be provided.") |
1089 | + |
1090 | + |
1091 | +def handle_directories(namespace): |
1092 | + """Handle checkout and dependencies directories. |
1093 | + |
1094 | + The ~ construction is automatically expanded:: |
1095 | + |
1096 | + >>> namespace = argparse.Namespace( |
1097 | + ... directory='~/launchpad', dependencies_dir='~/launchpad/deps', |
1098 | + ... home_dir='/home/foo') |
1099 | + >>> handle_directories(namespace) |
1100 | + >>> namespace.directory |
1101 | + '/home/foo/launchpad' |
1102 | + >>> namespace.dependencies_dir |
1103 | + '/home/foo/launchpad/deps' |
1104 | + |
1105 | + The validation fails for directories not residing inside the home:: |
1106 | + |
1107 | + >>> namespace = argparse.Namespace( |
1108 | + ... directory='/tmp/launchpad', |
1109 | + ... dependencies_dir='~/launchpad/deps', |
1110 | + ... home_dir='/home/foo') |
1111 | + >>> handle_directories(namespace) # doctest: +ELLIPSIS |
1112 | + Traceback (most recent call last): |
1113 | + ValidationError: argument directory ... |
1114 | + |
1115 | + The validation fails if the directory contains spaces:: |
1116 | + |
1117 | + >>> namespace = argparse.Namespace(directory='my directory') |
1118 | + >>> handle_directories(namespace) # doctest: +ELLIPSIS |
1119 | + Traceback (most recent call last): |
1120 | + ValidationError: argument directory ... |
1121 | + """ |
1122 | + if ' ' in namespace.directory: |
1123 | + raise ValidationError('argument directory can not contain spaces.') |
1124 | + for attr in ('directory', 'dependencies_dir'): |
1125 | + directory = getattr( |
1126 | + namespace, attr).replace('~', namespace.home_dir) |
1127 | + if not directory.startswith(namespace.home_dir + os.path.sep): |
1128 | + raise ValidationError( |
1129 | + 'argument {0} does not reside under the home ' |
1130 | + 'directory of the system user.'.format(attr)) |
1131 | + setattr(namespace, attr, directory) |
1132 | + |
1133 | + |
1134 | +class ArgumentParser(argparse.ArgumentParser): |
1135 | + """A customized argument parser for `argparse`.""" |
1136 | + |
1137 | + def __init__(self, *args, **kwargs): |
1138 | + self.actions = [] |
1139 | + self.subparsers = None |
1140 | + super(ArgumentParser, self).__init__(*args, **kwargs) |
1141 | + |
1142 | + def add_argument(self, *args, **kwargs): |
1143 | + """Override to store actions in a "public" instance attribute. |
1144 | + |
1145 | + >>> parser = ArgumentParser() |
1146 | + >>> parser.add_argument('arg1') |
1147 | + >>> parser.add_argument('arg2') |
1148 | + >>> [action.dest for action in parser.actions] |
1149 | + ['help', 'arg1', 'arg2'] |
1150 | + """ |
1151 | + action = super(ArgumentParser, self).add_argument(*args, **kwargs) |
1152 | + self.actions.append(action) |
1153 | + |
1154 | + def register_subcommand(self, name, subcommand_class, handler=None): |
1155 | + """Add a subcommand to this parser. |
1156 | + |
1157 | + A sub command is registered giving the sub command `name` and class:: |
1158 | + |
1159 | + >>> parser = ArgumentParser() |
1160 | + |
1161 | + >>> class SubCommand(BaseSubCommand): |
1162 | + ... def handle(self, namespace): |
1163 | + ... return self.name |
1164 | + |
1165 | + >>> parser = ArgumentParser() |
1166 | + >>> _ = parser.register_subcommand('foo', SubCommand) |
1167 | + |
1168 | + The `main` method of the subcommand class is added to namespace, and |
1169 | + can be used to actually handle the sub command execution. |
1170 | + |
1171 | + >>> namespace = parser.parse_args(['foo']) |
1172 | + >>> namespace.main(namespace) |
1173 | + 'foo' |
1174 | + |
1175 | + A `handler` callable can also be provided to handle the subcommand |
1176 | + execution:: |
1177 | + |
1178 | + >>> handler = lambda namespace: 'custom handler' |
1179 | + |
1180 | + >>> parser = ArgumentParser() |
1181 | + >>> _ = parser.register_subcommand( |
1182 | + ... 'bar', SubCommand, handler=handler) |
1183 | + |
1184 | + >>> namespace = parser.parse_args(['bar']) |
1185 | + >>> namespace.main(namespace) |
1186 | + 'custom handler' |
1187 | + """ |
1188 | + if self.subparsers is None: |
1189 | + self.subparsers = self.add_subparsers( |
1190 | + title='subcommands', |
1191 | + help='Each subcommand accepts --h or --help to describe it.') |
1192 | + subcommand = subcommand_class(name, handler=handler) |
1193 | + parser = self.subparsers.add_parser( |
1194 | + subcommand.name, help=subcommand.help) |
1195 | + subcommand.add_arguments(parser) |
1196 | + parser.set_defaults(main=subcommand.main, get_parser=lambda: parser) |
1197 | + return subcommand |
1198 | + |
1199 | + def get_args_from_namespace(self, namespace): |
1200 | + """Return a list of arguments taking values from `namespace`. |
1201 | + |
1202 | + Having a parser defined as usual:: |
1203 | + |
1204 | + >>> parser = ArgumentParser() |
1205 | + >>> _ = parser.add_argument('--foo') |
1206 | + >>> _ = parser.add_argument('bar') |
1207 | + >>> namespace = parser.parse_args('--foo eggs spam'.split()) |
1208 | + |
1209 | + It is possible to recreate the argument list taking values from |
1210 | + a different namespace:: |
1211 | + |
1212 | + >>> namespace.foo = 'changed' |
1213 | + >>> parser.get_args_from_namespace(namespace) |
1214 | + ['--foo', 'changed', 'spam'] |
1215 | + """ |
1216 | + args = [] |
1217 | + for action in self.actions: |
1218 | + dest = action.dest |
1219 | + option_strings = action.option_strings |
1220 | + value = getattr(namespace, dest, None) |
1221 | + if value: |
1222 | + if option_strings: |
1223 | + args.append(option_strings[0]) |
1224 | + if isinstance(value, list): |
1225 | + args.extend(value) |
1226 | + elif not isinstance(value, bool): |
1227 | + args.append(value) |
1228 | + return args |
1229 | + |
1230 | + def _handle_help(self, namespace): |
1231 | + """Help sub command handler.""" |
1232 | + command = namespace.command |
1233 | + help = self.prefix_chars + 'h' |
1234 | + args = [help] if command is None else [command, help] |
1235 | + self.parse_args(args) |
1236 | + |
1237 | + def _add_help_subcommand(self): |
1238 | + """Add an help sub command to this parser.""" |
1239 | + name = 'help' |
1240 | + choices = self.subparsers.choices.keys() |
1241 | + if name not in choices: |
1242 | + choices.append(name) |
1243 | + parser = self.subparsers.add_parser( |
1244 | + name, help='More help on a command.') |
1245 | + parser.add_argument('command', nargs='?', choices=choices) |
1246 | + parser.set_defaults(main=self._handle_help) |
1247 | + |
1248 | + def parse_args(self, *args, **kwargs): |
1249 | + """Override to add an help sub command. |
1250 | + |
1251 | + The help sub command is added only if other sub commands exist:: |
1252 | + |
1253 | + >>> stderr, sys.stderr = sys.stderr, sys.stdout |
1254 | + >>> parser = ArgumentParser() |
1255 | + >>> parser.parse_args(['help']) |
1256 | + Traceback (most recent call last): |
1257 | + SystemExit: 2 |
1258 | + >>> sys.stderr = stderr |
1259 | + |
1260 | + >>> class SubCommand(BaseSubCommand): pass |
1261 | + >>> _ = parser.register_subcommand('command', SubCommand) |
1262 | + >>> namespace = parser.parse_args(['help']) |
1263 | + >>> namespace.main(namespace) |
1264 | + Traceback (most recent call last): |
1265 | + SystemExit: 0 |
1266 | + """ |
1267 | + if self.subparsers is not None: |
1268 | + self._add_help_subcommand() |
1269 | + return super(ArgumentParser, self).parse_args(*args, **kwargs) |
1270 | + |
1271 | + |
1272 | +class BaseSubCommand(object): |
1273 | + """Base class defining a sub command. |
1274 | + |
1275 | + Objects of this class can be used to easily add sub commands to this |
1276 | + script as plugins, providing arguments, validating them, restarting |
1277 | + as root if needed. |
1278 | + |
1279 | + Override `add_arguments()` to add arguments, `validators` to add |
1280 | + namespace validators, and `handle()` to manage sub command execution:: |
1281 | + |
1282 | + >>> def validator(namespace): |
1283 | + ... namespace.bar = True |
1284 | + |
1285 | + >>> class SubCommand(BaseSubCommand): |
1286 | + ... help = 'Sub command example.' |
1287 | + ... validators = (validator,) |
1288 | + ... |
1289 | + ... def add_arguments(self, parser): |
1290 | + ... super(SubCommand, self).add_arguments(parser) |
1291 | + ... parser.add_argument('--foo') |
1292 | + ... |
1293 | + ... def handle(self, namespace): |
1294 | + ... return namespace |
1295 | + |
1296 | + Register the sub command using `ArgumentParser.register_subcommand`:: |
1297 | + |
1298 | + >>> parser = ArgumentParser() |
1299 | + >>> sub_command = parser.register_subcommand('spam', SubCommand) |
1300 | + |
1301 | + Now the subcommand has a name:: |
1302 | + |
1303 | + >>> sub_command.name |
1304 | + 'spam' |
1305 | + |
1306 | + The sub command handler can be called using `namespace.main()`:: |
1307 | + |
1308 | + >>> namespace = parser.parse_args('spam --foo eggs'.split()) |
1309 | + >>> namespace = namespace.main(namespace) |
1310 | + >>> namespace.foo |
1311 | + 'eggs' |
1312 | + >>> namespace.bar |
1313 | + True |
1314 | + |
1315 | + The help attribute of sub command instances is used to generate |
1316 | + the command usage message:: |
1317 | + >>> help = parser.format_help() |
1318 | + >>> 'spam' in help |
1319 | + True |
1320 | + >>> 'Sub command example.' in help |
1321 | + True |
1322 | + """ |
1323 | + |
1324 | + help = '' |
1325 | + needs_root = False |
1326 | + validators = () |
1327 | + |
1328 | + def __init__(self, name, handler=None): |
1329 | + self.name = name |
1330 | + self.handler = handler or self.handle |
1331 | + |
1332 | + def __repr__(self): |
1333 | + return '<{klass}: {name}>'.format( |
1334 | + klass=self.__class__.__name__, name=self.name) |
1335 | + |
1336 | + def init_namespace(self, namespace): |
1337 | + """Add `run_as_root` and `euid` names to the given `namespace`.""" |
1338 | + euid = os.geteuid() |
1339 | + namespace.euid, namespace.run_as_root = euid, not euid |
1340 | + |
1341 | + def get_needs_root(self, namespace): |
1342 | + """Return True if root is needed to run this subcommand. |
1343 | + |
1344 | + Subclasses can override this to dynamically change this value |
1345 | + based on the given `namespace`. |
1346 | + """ |
1347 | + return self.needs_root |
1348 | + |
1349 | + def get_validators(self, namespace): |
1350 | + """Return an iterable of namespace validators for this sub command. |
1351 | + |
1352 | + Subclasses can override this to dynamically change validators |
1353 | + based on the given `namespace`. |
1354 | + """ |
1355 | + return self.validators |
1356 | + |
1357 | + def validate(self, parser, namespace): |
1358 | + """Validate the current namespace. |
1359 | + |
1360 | + The method `self.get_validators` can contain an iterable of objects |
1361 | + that are called once the arguments namespace is fully populated. |
1362 | + This allows cleaning and validating arguments that depend on |
1363 | + each other, or on the current environment. |
1364 | + |
1365 | + Each validator is a callable object, takes the current namespace |
1366 | + and can raise ValidationError if the arguments are not valid:: |
1367 | + |
1368 | + >>> import sys |
1369 | + >>> stderr, sys.stderr = sys.stderr, sys.stdout |
1370 | + >>> def validator(namespace): |
1371 | + ... raise ValidationError('nothing is going on') |
1372 | + >>> sub_command = BaseSubCommand('foo') |
1373 | + >>> sub_command.validators = [validator] |
1374 | + >>> sub_command.validate(ArgumentParser(), argparse.Namespace()) |
1375 | + Traceback (most recent call last): |
1376 | + SystemExit: 2 |
1377 | + >>> sys.stderr = stderr |
1378 | + """ |
1379 | + for validator in self.get_validators(namespace): |
1380 | + try: |
1381 | + validator(namespace) |
1382 | + except ValidationError as err: |
1383 | + parser.error(err) |
1384 | + |
1385 | + def restart_as_root(self, parser, namespace): |
1386 | + """Restart this script using *sudo*. |
1387 | + |
1388 | + The arguments are recreated using the given `namespace`. |
1389 | + """ |
1390 | + args = parser.get_args_from_namespace(namespace) |
1391 | + return subprocess.call(['sudo', sys.argv[0], self.name] + args) |
1392 | + |
1393 | + def main(self, namespace): |
1394 | + """Entry point for subcommand execution. |
1395 | + |
1396 | + This method takes care of: |
1397 | + |
1398 | + - current argparse subparser retrieval |
1399 | + - namespace initialization |
1400 | + - namespace validation |
1401 | + - script restart as root (if this sub command needs to be run as root) |
1402 | + |
1403 | + If everything is ok the sub command handler is called passing |
1404 | + the validated namespace. |
1405 | + """ |
1406 | + parser = namespace.get_parser() |
1407 | + self.init_namespace(namespace) |
1408 | + self.validate(parser, namespace) |
1409 | + if self.get_needs_root(namespace) and not namespace.run_as_root: |
1410 | + return self.restart_as_root(parser, namespace) |
1411 | + return self.handler(namespace) |
1412 | + |
1413 | + def handle(self, namespace): |
1414 | + """Default sub command handler. |
1415 | + |
1416 | + Subclasses must either implement this method or provide another |
1417 | + callable handler during sub command registration. |
1418 | + """ |
1419 | + raise NotImplementedError |
1420 | + |
1421 | + def add_arguments(self, parser): |
1422 | + """Here subclasses can add arguments to the subparser.""" |
1423 | + pass |
1424 | + |
1425 | + |
1426 | +class ActionsBasedSubCommand(BaseSubCommand): |
1427 | + """A sub command that uses "actions" to handle its execution. |
1428 | + |
1429 | + Actions are callables stored in the `actions` attribute, together |
1430 | + with the arguments they expect. Those arguments are strings |
1431 | + representing attributes of the argparse namespace:: |
1432 | + |
1433 | + >>> trace = [] |
1434 | + |
1435 | + >>> def action1(foo): |
1436 | + ... trace.append('action1 received ' + foo) |
1437 | + |
1438 | + >>> def action2(foo, bar): |
1439 | + ... trace.append('action2 received {0} and {1}'.format(foo, bar)) |
1440 | + |
1441 | + >>> class SubCommand(ActionsBasedSubCommand): |
1442 | + ... actions = ( |
1443 | + ... (action1, 'foo'), |
1444 | + ... (action2, 'foo', 'bar'), |
1445 | + ... ) |
1446 | + ... |
1447 | + ... def add_arguments(self, parser): |
1448 | + ... super(SubCommand, self).add_arguments(parser) |
1449 | + ... parser.add_argument('--foo') |
1450 | + ... parser.add_argument('--bar') |
1451 | + |
1452 | + This class implements an handler method that executes actions in the |
1453 | + order they are provided:: |
1454 | + |
1455 | + >>> parser = ArgumentParser() |
1456 | + >>> _ = parser.register_subcommand('sub', SubCommand) |
1457 | + >>> namespace = parser.parse_args('sub --foo eggs --bar spam'.split()) |
1458 | + >>> namespace.main(namespace) |
1459 | + >>> trace |
1460 | + ['action1 received eggs', 'action2 received eggs and spam'] |
1461 | + |
1462 | + A special argument `-a` or `--actions` is automatically added to the |
1463 | + parser. It can be used to execute only one or a subset of actions:: |
1464 | + |
1465 | + >>> trace = [] |
1466 | + |
1467 | + >>> namespace = parser.parse_args('sub --foo eggs -a action1'.split()) |
1468 | + >>> namespace.main(namespace) |
1469 | + >>> trace |
1470 | + ['action1 received eggs'] |
1471 | + |
1472 | + A special argument `--skip-actions` is automatically added to the |
1473 | + parser. It can be used to skip one or more actions:: |
1474 | + |
1475 | + >>> trace = [] |
1476 | + |
1477 | + >>> namespace = parser.parse_args( |
1478 | + ... 'sub --foo eggs --skip-actions action1'.split()) |
1479 | + >>> namespace.main(namespace) |
1480 | + >>> trace |
1481 | + ['action2 received eggs and None'] |
1482 | + |
1483 | + The actions execution is stopped if an action raises `LaunchpadError`. |
1484 | + In that case, the error is returned by the handler. |
1485 | + |
1486 | + >>> trace = [] |
1487 | + |
1488 | + >>> def erroneous_action(foo): |
1489 | + ... raise LaunchpadError('error') |
1490 | + |
1491 | + >>> class SubCommandWithErrors(SubCommand): |
1492 | + ... actions = ( |
1493 | + ... (action1, 'foo'), |
1494 | + ... (erroneous_action, 'foo'), |
1495 | + ... (action2, 'foo', 'bar'), |
1496 | + ... ) |
1497 | + |
1498 | + >>> parser = ArgumentParser() |
1499 | + >>> _ = parser.register_subcommand('sub', SubCommandWithErrors) |
1500 | + >>> namespace = parser.parse_args('sub --foo eggs'.split()) |
1501 | + >>> error = namespace.main(namespace) |
1502 | + >>> error.message |
1503 | + 'error' |
1504 | + |
1505 | + The action `action2` is not executed:: |
1506 | + |
1507 | + >>> trace |
1508 | + ['action1 received eggs'] |
1509 | + """ |
1510 | + |
1511 | + actions = () |
1512 | + |
1513 | + def __init__(self, *args, **kwargs): |
1514 | + super(ActionsBasedSubCommand, self).__init__(*args, **kwargs) |
1515 | + self._action_names = [] |
1516 | + self._actions = {} |
1517 | + for action_args in self.actions: |
1518 | + action, args = action_args[0], action_args[1:] |
1519 | + action_name = self._get_action_name(action) |
1520 | + self._action_names.append(action_name) |
1521 | + self._actions[action_name] = (action, args) |
1522 | + |
1523 | + def _get_action_name(self, action): |
1524 | + """Return the string representation of an action callable. |
1525 | + |
1526 | + The name is retrieved using attributes lookup for `action_name` |
1527 | + and then `__name__`:: |
1528 | + |
1529 | + >>> def action1(): |
1530 | + ... pass |
1531 | + >>> action1.action_name = 'myaction' |
1532 | + |
1533 | + >>> def action2(): |
1534 | + ... pass |
1535 | + |
1536 | + >>> sub_command = ActionsBasedSubCommand('foo') |
1537 | + >>> sub_command._get_action_name(action1) |
1538 | + 'myaction' |
1539 | + >>> sub_command._get_action_name(action2) |
1540 | + 'action2' |
1541 | + """ |
1542 | + try: |
1543 | + return action.action_name |
1544 | + except AttributeError: |
1545 | + return action.__name__ |
1546 | + |
1547 | + def add_arguments(self, parser): |
1548 | + super(ActionsBasedSubCommand, self).add_arguments(parser) |
1549 | + parser.add_argument( |
1550 | + '-a', '--actions', nargs='+', choices=self._action_names, |
1551 | + help='Call one or more internal functions.') |
1552 | + parser.add_argument( |
1553 | + '--skip-actions', nargs='+', choices=self._action_names, |
1554 | + help='Skip one or more internal functions.') |
1555 | + |
1556 | + def handle(self, namespace): |
1557 | + skip_actions = namespace.skip_actions or [] |
1558 | + action_names = filter( |
1559 | + lambda action_name: action_name not in skip_actions, |
1560 | + namespace.actions or self._action_names) |
1561 | + for action_name in action_names: |
1562 | + action, arg_names = self._actions[action_name] |
1563 | + args = [getattr(namespace, i) for i in arg_names] |
1564 | + try: |
1565 | + action(*args) |
1566 | + except LaunchpadError as err: |
1567 | + return err |
1568 | + |
1569 | + |
1570 | +class InstallSubCommand(ActionsBasedSubCommand): |
1571 | + """Install the Launchpad environment.""" |
1572 | + |
1573 | + actions = ( |
1574 | + (initialize, |
1575 | + 'user', 'full_name', 'email', 'lpuser', 'private_key', |
1576 | + 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'), |
1577 | + (setup_apt, 'no_repositories'), |
1578 | + (setup_launchpad, |
1579 | + 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys'), |
1580 | + ) |
1581 | + help = __doc__ |
1582 | + needs_root = True |
1583 | + validators = ( |
1584 | + handle_user, |
1585 | + handle_lpuser, |
1586 | + handle_userdata, |
1587 | + handle_ssh_keys, |
1588 | + handle_directories, |
1589 | + ) |
1590 | + |
1591 | + def add_arguments(self, parser): |
1592 | + super(InstallSubCommand, self).add_arguments(parser) |
1593 | + parser.add_argument( |
1594 | + '-u', '--user', |
1595 | + help='The name of the system user to be created or updated. ' |
1596 | + 'The current user is used if this script is not run as ' |
1597 | + 'root and this argument is omitted.') |
1598 | + parser.add_argument( |
1599 | + '-e', '--email', |
1600 | + help='The email of the user, used for bzr whoami. This argument ' |
1601 | + 'can be omitted if a bzr id exists for current user.') |
1602 | + parser.add_argument( |
1603 | + '-f', '--full-name', |
1604 | + help='The full name of the user, used for bzr whoami. ' |
1605 | + 'This argument can be omitted if a bzr id exists for ' |
1606 | + 'current user.') |
1607 | + parser.add_argument( |
1608 | + '-l', '--lpuser', |
1609 | + help='The name of the Launchpad user that will be used to ' |
1610 | + 'check out dependencies. If not provided, the system ' |
1611 | + 'user name is used.') |
1612 | + parser.add_argument( |
1613 | + '-v', '--private-key', |
1614 | + help='The SSH private key for the Launchpad user (without ' |
1615 | + 'passphrase). If this argument is omitted and a keypair is ' |
1616 | + 'not found in the home directory of the system user a new ' |
1617 | + 'SSH keypair will be generated and the checkout of the ' |
1618 | + 'Launchpad code will use HTTP rather than bzr+ssh.') |
1619 | + parser.add_argument( |
1620 | + '-b', '--public-key', |
1621 | + help='The SSH public key for the Launchpad user. ' |
1622 | + 'If this argument is omitted and a keypair is not found ' |
1623 | + 'in the home directory of the system user a new SSH ' |
1624 | + 'keypair will be generated and the checkout of the ' |
1625 | + 'Launchpad code will use HTTP rather than bzr+ssh.') |
1626 | + parser.add_argument( |
1627 | + '-d', '--dependencies-dir', default=DEPENDENCIES_DIR, |
1628 | + help='The directory of the Launchpad dependencies to be created. ' |
1629 | + 'The directory must reside under the home directory of the ' |
1630 | + 'given user (see -u argument). ' |
1631 | + '[DEFAULT={0}]'.format(DEPENDENCIES_DIR)) |
1632 | + parser.add_argument( |
1633 | + '-c', '--directory', default=CHECKOUT_DIR, |
1634 | + help='The directory of the Launchpad repository to be created. ' |
1635 | + 'The directory must reside under the home directory of the ' |
1636 | + 'given user (see -u argument). ' |
1637 | + '[DEFAULT={0}]'.format(CHECKOUT_DIR)) |
1638 | + parser.add_argument( |
1639 | + '-N', '--no-repositories', action='store_true', |
1640 | + help='Do not add APT repositories.') |
1641 | + |
1642 | + |
1643 | +class UpdateSubCommand(ActionsBasedSubCommand): |
1644 | + """Update the Launchpad environment to latest version.""" |
1645 | + |
1646 | + actions = ( |
1647 | + (update_launchpad, |
1648 | + 'user', 'valid_ssh_keys', 'dependencies_dir', 'directory', 'apt'), |
1649 | + (link_sourcecode_in_branches, |
1650 | + 'user', 'dependencies_dir', 'directory'), |
1651 | + ) |
1652 | + help = __doc__ |
1653 | + validators = ( |
1654 | + handle_user, |
1655 | + handle_ssh_keys, |
1656 | + handle_directories, |
1657 | + ) |
1658 | + |
1659 | + def get_needs_root(self, namespace): |
1660 | + # Root is needed only if an apt update/upgrade is requested. |
1661 | + return namespace.apt |
1662 | + |
1663 | + def add_arguments(self, parser): |
1664 | + super(UpdateSubCommand, self).add_arguments(parser) |
1665 | + parser.add_argument( |
1666 | + '-u', '--user', |
1667 | + help='The name of the system user used to update Launchpad. ' |
1668 | + 'The current user is used if this script is not run as ' |
1669 | + 'root and this argument is omitted.') |
1670 | + parser.add_argument( |
1671 | + '-d', '--dependencies-dir', default=DEPENDENCIES_DIR, |
1672 | + help='The directory of the Launchpad dependencies to be updated. ' |
1673 | + 'The directory must reside under the home directory of the ' |
1674 | + 'given user (see -u argument). ' |
1675 | + '[DEFAULT={0}]'.format(DEPENDENCIES_DIR)) |
1676 | + parser.add_argument( |
1677 | + '-c', '--directory', default=CHECKOUT_DIR, |
1678 | + help='The directory of the Launchpad repository to be updated. ' |
1679 | + 'The directory must reside under the home directory of the ' |
1680 | + 'given user (see -u argument). ' |
1681 | + '[DEFAULT={0}]'.format(CHECKOUT_DIR)) |
1682 | + parser.add_argument( |
1683 | + '-D', '--apt', action='store_true', |
1684 | + help='Also update deb packages.') |
1685 | + |
1686 | + |
1687 | +class LXCInstallSubCommand(InstallSubCommand): |
1688 | + """Install the Launchpad environment inside an LXC.""" |
1689 | + |
1690 | + actions = ( |
1691 | + (initialize, |
1692 | + 'user', 'full_name', 'email', 'lpuser', 'private_key', |
1693 | + 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'), |
1694 | + (create_lxc, |
1695 | + 'user', 'lxc_name', 'lxc_arch', 'lxc_os'), |
1696 | + (start_lxc, 'lxc_name'), |
1697 | + (wait_for_lxc, 'lxc_name'), |
1698 | + (initialize_lxc, |
1699 | + 'lxc_name', 'lxc_os'), |
1700 | + (setup_launchpad_lxc, |
1701 | + 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys', |
1702 | + 'lxc_name'), |
1703 | + (stop_lxc, 'lxc_name'), |
1704 | + ) |
1705 | + help = __doc__ |
1706 | + |
1707 | + def add_arguments(self, parser): |
1708 | + super(LXCInstallSubCommand, self).add_arguments(parser) |
1709 | + parser.add_argument( |
1710 | + '-n', '--lxc-name', default=LXC_NAME, |
1711 | + help='The LXC container name to setup. ' |
1712 | + '[DEFAULT={0}]'.format(LXC_NAME)) |
1713 | + parser.add_argument( |
1714 | + '-A', '--lxc-arch', default=LXC_GUEST_ARCH, |
1715 | + help='The LXC container architecture. ' |
1716 | + '[DEFAULT={0}]'.format(LXC_GUEST_ARCH)) |
1717 | + parser.add_argument( |
1718 | + '-r', '--lxc-os', default=LXC_GUEST_OS, |
1719 | + choices=LXC_GUEST_CHOICES, |
1720 | + help='The LXC container distro codename. ' |
1721 | + '[DEFAULT={0}]'.format(LXC_GUEST_OS)) |
1722 | + |
1723 | + |
1724 | +def main(): |
1725 | + parser = ArgumentParser(description=__doc__) |
1726 | + parser.register_subcommand('install', InstallSubCommand) |
1727 | + parser.register_subcommand('update', UpdateSubCommand) |
1728 | + parser.register_subcommand('lxc-install', LXCInstallSubCommand) |
1729 | + args = parser.parse_args() |
1730 | + return args.main(args) |
1731 | + |
1732 | + |
1733 | +if __name__ == '__main__': |
1734 | + sys.exit(main()) |
Hey Francesco. Thanks for doing this! This is a nice slack time project.
This review is through line 895. I will continue on this after lunch.
In join_command's docstring, there's a small typo, twice: containig -> containing.
Spelling niggle: in environ's docstring ("A context manager to temporary change environment variables."), change "temporary" to "temporarily". Same for su docstring ("A context manager to temporary run the script as a different user.").
Minor grammar niggle: in docstring of check_output, you need to convert the comma to one of three choices: semicolon, ", and", or period (new sentence): "The first argument is the path to the command to run, subsequent arguments are command-line arguments to be passed."
Unless you give me a good reason why not, I'd like all of the contextmanagers to have a try:finally block placed around the yield. environ and cd are the only ones without it, I think.
Why do you need your own check_output, rather than using subprocess' check_output? Ah! Because it is not in Python 2.6? If that's the reason, what would you think of trying to get it from the module, and if there's an AttributeError (or ImportError, however you want to work it) then you define your function? That way we get the newer version, with possible bugfixes, if it exists; and we have a chance to clarify why we are not using the stdlib's function.
I had never used the pipes module. The quotes function is small and convenient. I'm mildly concerned that it is not part of the module's documented API (http:// docs.python. org/library/ pipes.html). That stuff stays pretty stable across Python IME though; I'm fine with it.
So many of these helpers are generically useful. It's a shame they are not somewhere more central. Many of them look helpful for charms too. I don't have a suggested action for that; it's just an observation.
As we discussed already, in your shoes people have advocated to me that I should use bzr's command framework (and I mentioned Jamu Kakar's commandant; that project hasn't seen any updates, but it looks like Landscape is still using it because they have done some recent packaging work). I like the fact that what you have is self-contained and small. Again, I merely offer that as an avenue that I suggest you investigate in the future.
I look forward to this being a separate project with an ability to divide things up into support files and so on. In addition to making it possible to organize the code more obviously, it also might then be reasonable to reconsider the use of doctests. I am among the apparent minority that like them in the Python community, and I think what you have here is well done, but our team's standards are to use unit tests. As long as this is a single downloadable file, doctests make enough sense to me to be a defensible choice. Later, maybe less so.
Random thought: I was thinking about your other branch that had LANG=C. Probably obvious, but you could change apt_get_install to accept keyword arguments and include those as environment variables so you could easily do the LANG=C thing, maybe? That maybe looks too much like command options. Just a thought.
As I'm sure...