Merge lp:~frankban/launchpad/lpsetup-initial into lp:launchpad

Proposed by Francesco Banconi
Status: Rejected
Rejected by: William Grant
Proposed branch: lp:~frankban/launchpad/lpsetup-initial
Merge into: lp:launchpad
Diff against target: 1734 lines (+1730/-0)
1 file modified
utilities/lpsetup.py (+1730/-0)
To merge this branch: bzr merge lp:~frankban/launchpad/lpsetup-initial
Reviewer Review Type Date Requested Status
Gary Poster (community) Approve
Review via email: mp+94765@code.launchpad.net

Description of the change

= Summary =

This is the first step of *lpsetup* implementation.
It's basically a Python script created to eliminate code duplication between
*setuplxc* and *rocketfuel-setup/get/branch*.

PS: It's a brand new huge file, sorry about that.

== Proposed fix ==

Actually the script can be used to set up a Launchpad development or testing
environment in a real machine or inside an LXC container, with ability to
update the codebase once installed.

The next steps will be, in random order:

- implement lxc-update sub command: update the codebase inside a running or
  stopped LXC
- implement branch and lxc-branch sub commands: see rocketfuel-branch
- ability to specify a custom ssh key name to connect to LXC
- create a real *lpsetup* project, splitting code in separate files, etc.
- deb package?
- ec2 tests, tests, tests

== Implementation details ==

While *setuplxc* sets up the LXC connecting via ssh to the container,
*lpsetup* uses a different approach: the script calls itself using an ssh
connection to the container. This is possible thanks to the script ability
to be run specifying *actions*, i.e. a subset of internal functions to be
called. This way, the LXC can initialize itself and the code is reused.

*lpsetup* improves some of the functionalities present in *setuplxc*:

- errors in executing external commands are caught and reported, and they
  cause the script to stop running
- apt repositories are added in a gentler way, using *add-apt-repository*
  (this allows to get rid of `apt-get install --allow-unauthenticated`
  and needs to be back-ported to *setuplxc*)
- some helper objects were fixed and some others were added

A thin subcommands management layer (`BaseSubCommand`) was implemented to
allow extensibility.

Because the script can be executed in lucid containers,
it is now compatible with Python 2.6.

== Tests ==

python -m doctest -v lpsetup.py

To post a comment you must log in.
14876. By Francesco Banconi

Added tests to generate_ssh_keys helper.

Revision history for this message
Gary Poster (gary) wrote :
Download full text (3.6 KiB)

Hey Francesco. Thanks for doing this! This is a nice slack time project.

This review is through line 895. I will continue on this after lunch.

In join_command's docstring, there's a small typo, twice: containig -> containing.

Spelling niggle: in environ's docstring ("A context manager to temporary change environment variables."), change "temporary" to "temporarily". Same for su docstring ("A context manager to temporary run the script as a different user.").

Minor grammar niggle: in docstring of check_output, you need to convert the comma to one of three choices: semicolon, ", and", or period (new sentence): "The first argument is the path to the command to run, subsequent arguments are command-line arguments to be passed."

Unless you give me a good reason why not, I'd like all of the contextmanagers to have a try:finally block placed around the yield. environ and cd are the only ones without it, I think.

Why do you need your own check_output, rather than using subprocess' check_output? Ah! Because it is not in Python 2.6? If that's the reason, what would you think of trying to get it from the module, and if there's an AttributeError (or ImportError, however you want to work it) then you define your function? That way we get the newer version, with possible bugfixes, if it exists; and we have a chance to clarify why we are not using the stdlib's function.

I had never used the pipes module. The quotes function is small and convenient. I'm mildly concerned that it is not part of the module's documented API (http://docs.python.org/library/pipes.html). That stuff stays pretty stable across Python IME though; I'm fine with it.

So many of these helpers are generically useful. It's a shame they are not somewhere more central. Many of them look helpful for charms too. I don't have a suggested action for that; it's just an observation.

As we discussed already, in your shoes people have advocated to me that I should use bzr's command framework (and I mentioned Jamu Kakar's commandant; that project hasn't seen any updates, but it looks like Landscape is still using it because they have done some recent packaging work). I like the fact that what you have is self-contained and small. Again, I merely offer that as an avenue that I suggest you investigate in the future.

I look forward to this being a separate project with an ability to divide things up into support files and so on. In addition to making it possible to organize the code more obviously, it also might then be reasonable to reconsider the use of doctests. I am among the apparent minority that like them in the Python community, and I think what you have here is well done, but our team's standards are to use unit tests. As long as this is a single downloadable file, doctests make enough sense to me to be a defensible choice. Later, maybe less so.

Random thought: I was thinking about your other branch that had LANG=C. Probably obvious, but you could change apt_get_install to accept keyword arguments and include those as environment variables so you could easily do the LANG=C thing, maybe? That maybe looks too much like command options. Just a thought.

As I'm sure...

Read more...

Revision history for this message
Francesco Banconi (frankban) wrote :
Download full text (3.4 KiB)

Thank you Gary for reviewing this, it's a huge file.

> Unless you give me a good reason why not, I'd like all of the contextmanagers
> to have a try:finally block placed around the yield. environ and cd are the
> only ones without it, I think.

No good reason, I will add the try/finally block to them.

> Why do you need your own check_output, rather than using subprocess'
> check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> what would you think of trying to get it from the module, and if there's an
> AttributeError (or ImportError, however you want to work it) then you define
> your function? That way we get the newer version, with possible bugfixes, if
> it exists; and we have a chance to clarify why we are not using the stdlib's
> function.

That's one of the reasons, the others, like the first, aren't essential, just nice to have:
- the exception raised on errors is the one subcommand handlers trap (but of course I could also trap subprocess.CalledProcessError)
- the positional arguments thing (really not even a "nice to have", more a "look, we're lazy and cool")
- None is ignored if given as argument, so that you can write `check_output('ls', '-l' if something else None)
- the error contains a better representation of the executed command

I can get rid of check_output; my point is: like `subprocess.check_output`, it's just a little wrapper around `subprocess.Popen`, that does the real stuff (and we will not miss future bugfixes for that object).

> So many of these helpers are generically useful. It's a shame they are not
> somewhere more central. Many of them look helpful for charms too. I don't
> have a suggested action for that; it's just an observation.

I agree.

> As we discussed already, in your shoes people have advocated to me that I
> should use bzr's command framework (and I mentioned Jamu Kakar's commandant;
> that project hasn't seen any updates, but it looks like Landscape is still
> using it because they have done some recent packaging work). I like the fact
> that what you have is self-contained and small. Again, I merely offer that as
> an avenue that I suggest you investigate in the future.

Thanks Gary, sure, I will take a look at commandant, although I like my tiny subcommand objects.

> I look forward to this being a separate project with an ability to divide
> things up into support files and so on. In addition to making it possible to
> organize the code more obviously, it also might then be reasonable to
> reconsider the use of doctests.

Yes, you are right, I will add a slack time card for unit tests.

>
> Random thought: I was thinking about your other branch that had LANG=C.
> Probably obvious, but you could change apt_get_install to accept keyword
> arguments and include those as environment variables so you could easily do
> the LANG=C thing, maybe? That maybe looks too much like command options.
> Just a thought.

Nice idea IMHO.

> As I'm sure you've realized, we should use generate_ssh_keys with descriptive
> names per the webops request for setuplxc. If these become merely
> "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> want to know ...

Read more...

Revision history for this message
Gary Poster (gary) wrote :

Continuing on from where I left off...

I suggest deleting update_launchpad_lxc for now, though I see why you have it. Up to you.

You have your file nicely divided in sections by function. If you were going to keep everything in this one file, I'd ask for comment lines delineating the sections--in particular, I had to think to realize that we were in the arg parsing section when I saw handle_user. Since you are going to divide this up into files anyway, I won't worry about it now.

You are not using os.path.expanduser in handle_directories for a reason: you are using the home directory of the user that was passed in, not the current home directory. A comment explaining this might be helpful, or might not.

The ArgumentParser might be improved if it showed in the docstring how users would use the registered subcommands. I expect that kind of thing might fall out of the division into separate files.

I find the interaction between ArgumentParser (the actions attribute) and ActionsBasedSubCommand to not be obvious. I thought about it, and the main thing I found confusing was BaseSubCommand. ArgumentParser, AFAICT, has code specifically for ActionsBasedSubCommand, so it seems like it is specific to that kind of SubCommand. When would one actually use BaseSubCommand? The fact that we are not using the BaseSubCommand directly reinforces this feeling. Maybe a bit of elaboration would explain the value of having a separate BaseSubCommand, and not merging it directly with ActionsBasedSubCommand. In your cover letter you say it is for extensibility, but that does not convince me.

When I run the command's help, I get the following output:

"""
$ ./lpsetup.py -h
usage: lpsetup.py [-h] {install,update,lxc-install} ...

Create and update Launchpad development and testing environments.

optional arguments:
  -h, --help show this help message and exit

subcommands:
  valid subcommands

  {install,update,lxc-install}
                        -h, --help show subcommand help and exit
    install Install the Launchpad environment.
    update Update the Launchpad environment to latest version.
    lxc-install Install the Launchpad environment inside an LXC.
"""
- I find the text "valid subcommands {install,update,lxc-install}" to be unnecessary and confusing.
- I find the placement of "-h, --help..." to be confusing too. Perhaps put that text, written more explicitly, where the "calid subcommands" text is? "Each subcommand accepts --h or --help to describe it."? Also, this reminds me, and probably other people who will use lpsetup, of bzr. They will want to do the following:

$ lpsetup help
(this could be equivalent to lpsetup -h)

$ lpsetup help install
(this could be equivalent to lpsetup install -h)
- The first (usage) line looks like you ought to be able to write "lpsetup -h install". You can, but it shows the top-level help--not what I'd expect. I think this usage line is confusing. Ideally, I think, the usage would be "lpsetup {help,install,update,lxc-install} ..." or something like that.

That's it. Nice progress! Thank you.

review: Approve
Revision history for this message
Gary Poster (gary) wrote :

...
> > Why do you need your own check_output, rather than using subprocess'
> > check_output? Ah! Because it is not in Python 2.6? If that's the reason,
> > what would you think of trying to get it from the module, and if there's an
> > AttributeError (or ImportError, however you want to work it) then you define
> > your function? That way we get the newer version, with possible bugfixes,
> if
> > it exists; and we have a chance to clarify why we are not using the stdlib's
> > function.
...
> I can get rid of check_output; my point is: like `subprocess.check_output`,
> it's just a little wrapper around `subprocess.Popen`, that does the real stuff
> (and we will not miss future bugfixes for that object).

Keep it. I think you ought to have a comment that explains why it is valuable. That's it.

...

> > As I'm sure you've realized, we should use generate_ssh_keys with
> descriptive
> > names per the webops request for setuplxc. If these become merely
> > "root_ssh_key" and "buildbot_ssh_key" then that's not what they want. They
> > want to know "why" the key is used (e.g., "lxc_communication_key" or
> similar).
> > If there's not a good name for this key that would distinguish it from other
> > keys a user might have, we should push back; but AFAICT there should be a
> good
> > name here.
>
> I propose 'id_for_lxc'... I know: we can do better.

"launchpad_lxc_id" perhaps? Getting the word "launchpad" in there would be valuable too, I think.

Thanks again

Gary

Revision history for this message
Robert Collins (lifeless) wrote :

Hi, a few points for you to consider:
 - this might be a lot better off in the lp-dev-tools project, its
clearly not part of LP itself. That would also make it easier to split
it into N files as you would not be stuck in the bootstrap stage of
the users environment (we could package that and mandate the package
be installed).
 - the environ context manager is duplicative of the
EnvironmentVariable fixture; it would be nice not to rewrite code that
has already been written and tested (and is used in our environment
too!) Yes, I know that this is small, but it does all add up, and
someone somewhere will eventually be cursing that they have to redo
this / fix a unicode escaping issue in the code or whatever.

 - I'm not fussed about reusing commandant or the bzrlib UI framework;
if I was writing something new that needed subcommands I would do it
differently than I did in bzrlib (and in fact I have since - with a
leaner, faster, more testable and machine drivable system in testr).
If you wanted to reuse that, its probably not quite there for reuse,
but we could split it into framework-and-user and make it so.

HTH,
Rob

Revision history for this message
Francesco Banconi (frankban) wrote :

Thank you for the review Gary. I agree with your suggestions and I will reply only to your doubts concerning BaseSubcommand.

> I find the interaction between ArgumentParser (the actions attribute) and
> ActionsBasedSubCommand to not be obvious. I thought about it, and the main
> thing I found confusing was BaseSubCommand. ArgumentParser, AFAICT, has code
> specifically for ActionsBasedSubCommand, so it seems like it is specific to
> that kind of SubCommand. When would one actually use BaseSubCommand? The
> fact that we are not using the BaseSubCommand directly reinforces this
> feeling. Maybe a bit of elaboration would explain the value of having a
> separate BaseSubCommand, and not merging it directly with
> ActionsBasedSubCommand. In your cover letter you say it is for extensibility,
> but that does not convince me.

ArgumentParser ignores the concept of *actions* as intended in ActionBasedSubcommand.
The register_subcommand method just:
- instantiates a sub command object
- calls subcommand.add_arguments to collect subcommand args
- stores a reference to subcommand.main that can be used later

This interface is implemented by BaseSubCommand. ActionBasedSubcommand is a specialization only useful when a sub command is subdivided in steps (internal functions) and we want to expose them in the ui.

I think the misunderstanding comes from the different meaning of the word *action* in argparse and ActionBasedSubcommand.
In argparse an action is an object containing info about a single argument (argparse.Action). Actions are stored in argparse.ArgumentParser._actions and returned by argparse.ArgumentParser.add_argument(...). In my ArgumentParser subclass I store those actions in a "public" attribute because I need to reuse them later: I preferred to populate a new attribute over using a "private" one (like in setuplxc).

I will agree if you suggest to change the name of ArgumentParser.actions to avoid confusion.

Revision history for this message
Francesco Banconi (frankban) wrote :

> - this might be a lot better off in the lp-dev-tools project, its
> clearly not part of LP itself. That would also make it easier to split
> it into N files as you would not be stuck in the bootstrap stage of
> the users environment (we could package that and mandate the package
> be installed).

I agree. Indeed we were thinking about creating an lpsetup project.
In general I think it's a good idea to have this separated from
launchpad itself and splitted into N files.

> - the environ context manager is duplicative of the
> EnvironmentVariable fixture; it would be nice not to rewrite code that
> has already been written and tested (and is used in our environment
> too!) Yes, I know that this is small, but it does all add up, and
> someone somewhere will eventually be cursing that they have to redo
> this / fix a unicode escaping issue in the code or whatever.

Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
Once this no longer is a single file, it will be easy to add dependencies.

Thanks again Robert.

Revision history for this message
Robert Collins (lifeless) wrote :

On Tue, Feb 28, 2012 at 11:47 PM, Francesco Banconi <email address hidden> wrote:
> Thanks for the hint. I will take a look at python-fixtures and testr integrated subcommands framework.
> Once this no longer is a single file, it will be easy to add dependencies.
>
> Thanks again Robert.

No probs! And seriously, feel free to ignore the testr subcommand bit
- as I mentioned, its likely still a bit too opinionated for reuse,
and we should focus on delivering this lovely improvement in the short
term. If its in lp-dev-tools (or a new project), the complexity of it
will be contained, and you can iterate on this sort of polish in later
steps. (Or do it in whatever order Gary prefers: I'm trying to leave
it wide open what you do).

-Rob

14877. By Francesco Banconi

Fixed spelling errors.

14878. By Francesco Banconi

Try/Finally blocks in context managers.

14879. By Francesco Banconi

Removed unused function.

14880. By Francesco Banconi

Help fix.

14881. By Francesco Banconi

First implementation of the help subcommand.

Unmerged revisions

14881. By Francesco Banconi

First implementation of the help subcommand.

14880. By Francesco Banconi

Help fix.

14879. By Francesco Banconi

Removed unused function.

14878. By Francesco Banconi

Try/Finally blocks in context managers.

14877. By Francesco Banconi

Fixed spelling errors.

14876. By Francesco Banconi

Added tests to generate_ssh_keys helper.

14875. By Francesco Banconi

Initial commit.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added file 'utilities/lpsetup.py'
--- utilities/lpsetup.py 1970-01-01 00:00:00 +0000
+++ utilities/lpsetup.py 2012-02-28 17:19:19 +0000
@@ -0,0 +1,1730 @@
1#!/usr/bin/env python
2# Copyright 2012 Canonical Ltd. This software is licensed under the
3# GNU Affero General Public License version 3 (see the file LICENSE).
4
5"""Create and update Launchpad development and testing environments."""
6
7__metaclass__ = type
8__all__ = [
9 'ActionsBasedSubCommand',
10 'apt_get_install',
11 'ArgumentParser',
12 'BaseSubCommand',
13 'bzr_whois',
14 'call',
15 'cd',
16 'check_output',
17 'CommandError',
18 'create_lxc',
19 'environ',
20 'file_append',
21 'file_prepend',
22 'generate_ssh_keys',
23 'get_container_path',
24 'get_lxc_gateway',
25 'get_su_command',
26 'get_user_home',
27 'get_user_ids',
28 'initialize',
29 'InstallSubCommand',
30 'join_command',
31 'LaunchpadError',
32 'link_sourcecode_in_branches',
33 'LXCInstallSubCommand',
34 'lxc_in_state',
35 'lxc_running',
36 'lxc_stopped',
37 'make_launchpad',
38 'mkdirs',
39 'setup_apt',
40 'setup_codebase',
41 'setup_external_sourcecode',
42 'setup_launchpad',
43 'setup_launchpad_lxc',
44 'ssh',
45 'SSHError',
46 'start_lxc',
47 'stop_lxc',
48 'su',
49 'this_command',
50 'update_launchpad',
51 'user_exists',
52 'ValidationError',
53 'wait_for_lxc',
54 ]
55
56# To run doctests: python -m doctest -v lpsetup.py
57
58from collections import namedtuple
59from contextlib import contextmanager
60from email.Utils import parseaddr, formataddr
61from functools import partial
62import argparse
63import errno
64import os
65import pipes
66import platform
67import pwd
68import shutil
69import subprocess
70import sys
71import time
72
73
74APT_REPOSITORIES = (
75 'deb http://archive.ubuntu.com/ubuntu {distro} multiverse',
76 'deb http://archive.ubuntu.com/ubuntu {distro}-updates multiverse',
77 'deb http://archive.ubuntu.com/ubuntu {distro}-security multiverse',
78 'ppa:launchpad/ppa',
79 'ppa:bzr/ppa',
80 )
81BASE_PACKAGES = ['ssh', 'bzr', 'language-pack-en']
82CHECKOUT_DIR = '~/launchpad/branches'
83DEPENDENCIES_DIR = '~/launchpad/dependencies'
84DHCP_FILE = '/etc/dhcp/dhclient.conf'
85HOSTS_CONTENT = (
86 ('127.0.0.88',
87 'launchpad.dev answers.launchpad.dev archive.launchpad.dev '
88 'api.launchpad.dev bazaar-internal.launchpad.dev beta.launchpad.dev '
89 'blueprints.launchpad.dev bugs.launchpad.dev code.launchpad.dev '
90 'feeds.launchpad.dev id.launchpad.dev keyserver.launchpad.dev '
91 'lists.launchpad.dev openid.launchpad.dev '
92 'ubuntu-openid.launchpad.dev ppa.launchpad.dev '
93 'private-ppa.launchpad.dev testopenid.dev translations.launchpad.dev '
94 'xmlrpc-private.launchpad.dev xmlrpc.launchpad.dev'),
95 ('127.0.0.99', 'bazaar.launchpad.dev'),
96 )
97HOSTS_FILE = '/etc/hosts'
98LP_APACHE_MODULES = 'proxy proxy_http rewrite ssl deflate headers'
99LP_APACHE_ROOTS = (
100 '/var/tmp/bazaar.launchpad.dev/static',
101 '/var/tmp/bazaar.launchpad.dev/mirrors',
102 '/var/tmp/archive',
103 '/var/tmp/ppa',
104 )
105LP_BZR_LOCATIONS = (
106 ('submit_branch', '{checkout_dir}'),
107 ('public_branch', 'bzr+ssh://bazaar.launchpad.net/~{lpuser}/launchpad'),
108 ('public_branch:policy', 'appendpath'),
109 ('push_location', 'lp:~{lpuser}/launchpad'),
110 ('push_location:policy', 'appendpath'),
111 ('merge_target', '{checkout_dir}'),
112 ('submit_to', 'merge@code.launchpad.net'),
113 )
114LP_CHECKOUT = 'devel'
115LP_PACKAGES = [
116 'bzr', 'launchpad-developer-dependencies', 'apache2',
117 'apache2-mpm-worker', 'libapache2-mod-wsgi'
118 ]
119LP_REPOS = (
120 'http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel',
121 'lp:launchpad',
122 )
123LP_SOURCE_DEPS = (
124 'http://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk')
125LXC_CONFIG_TEMPLATE = '/etc/lxc/local.conf'
126LXC_GUEST_ARCH = 'i386'
127LXC_GUEST_CHOICES = ('lucid', 'oneiric', 'precise')
128LXC_GUEST_OS = LXC_GUEST_CHOICES[0]
129LXC_NAME = 'lptests'
130LXC_OPTIONS = """
131lxc.network.type = veth
132lxc.network.link = {interface}
133lxc.network.flags = up
134"""
135LXC_PACKAGES = ['lxc', 'libvirt-bin']
136LXC_PATH = '/var/lib/lxc/'
137RESOLV_FILE = '/etc/resolv.conf'
138
139
140Env = namedtuple('Env', 'uid gid home')
141
142
143class LaunchpadError(Exception):
144 """Base exception for lpsetup."""
145
146
147class CommandError(LaunchpadError):
148 """Errors occurred running shell commands."""
149
150
151class SSHError(LaunchpadError):
152 """Errors occurred during SSH connection."""
153
154
155class ValidationError(LaunchpadError):
156 """Argparse invalid arguments."""
157
158
159def apt_get_install(*args):
160 """Install given packages using apt."""
161 with environ(DEBIAN_FRONTEND='noninteractive'):
162 cmd = ('apt-get', '-y', 'install') + args
163 return call(*cmd)
164
165
166def join_command(args):
167 """Return a valid Unix command line from `args`.
168
169 >>> join_command(['ls', '-l'])
170 'ls -l'
171
172 Arguments containing spaces and empty args are correctly quoted::
173
174 >>> join_command(['command', 'arg1', 'arg containing spaces', ''])
175 "command arg1 'arg containing spaces' ''"
176 """
177 return ' '.join(pipes.quote(arg) for arg in args)
178
179
180def bzr_whois(user):
181 """Return full name and email of bzr `user`.
182
183 Return None if the given `user` does not have a bzr user id.
184 """
185 with su(user):
186 try:
187 whoami = check_output('bzr', 'whoami')
188 except (CommandError, OSError):
189 return None
190 return parseaddr(whoami)
191
192
193@contextmanager
194def cd(directory):
195 """A context manager to temporarily change current working dir, e.g.::
196
197 >>> import os
198 >>> os.chdir('/tmp')
199 >>> with cd('/bin'): print os.getcwd()
200 /bin
201 >>> print os.getcwd()
202 /tmp
203 """
204 cwd = os.getcwd()
205 os.chdir(directory)
206 try:
207 yield
208 finally:
209 os.chdir(cwd)
210
211
212def check_output(*args, **kwargs):
213 r"""Run the command with the given arguments.
214
215 The first argument is the path to the command to run.
216 Subsequent arguments are command-line arguments to be passed.
217
218 Usually the output is captured by a pipe and returned::
219
220 >>> check_output('echo', 'output')
221 'output\n'
222
223 A `CommandError` exception is raised if the return code is not zero::
224
225 >>> check_output('ls', '--not a valid switch', stderr=subprocess.PIPE)
226 Traceback (most recent call last):
227 CommandError: Error running: ls '--not a valid switch'
228
229 None arguments are ignored::
230
231 >>> check_output(None, 'echo', None, 'output')
232 'output\n'
233 """
234 args = [i for i in args if i is not None]
235 process = subprocess.Popen(
236 args, stdout=kwargs.pop('stdout', subprocess.PIPE),
237 close_fds=True, **kwargs)
238 stdout, stderr = process.communicate()
239 retcode = process.poll()
240 if retcode:
241 raise CommandError('Error running: ' + join_command(args))
242 return stdout
243
244
245call = partial(check_output, stdout=None)
246
247
248@contextmanager
249def environ(**kwargs):
250 """A context manager to temporarily change environment variables.
251
252 If an existing environment variable is changed, it is restored during
253 context cleanup::
254
255 >>> import os
256 >>> os.environ['MY_VARIABLE'] = 'foo'
257 >>> with environ(MY_VARIABLE='bar'): print os.getenv('MY_VARIABLE')
258 bar
259 >>> print os.getenv('MY_VARIABLE')
260 foo
261 >>> del os.environ['MY_VARIABLE']
262
263 If we are adding environment variables, they are removed during context
264 cleanup::
265
266 >>> import os
267 >>> with environ(MY_VAR1='foo', MY_VAR2='bar'):
268 ... print os.getenv('MY_VAR1'), os.getenv('MY_VAR2')
269 foo bar
270 >>> os.getenv('MY_VAR1') == os.getenv('MY_VAR2') == None
271 True
272 """
273 backup = {}
274 for key, value in kwargs.items():
275 backup[key] = os.getenv(key)
276 os.environ[key] = value
277 try:
278 yield
279 finally:
280 for key, value in backup.items():
281 if value is None:
282 del os.environ[key]
283 else:
284 os.environ[key] = value
285
286
287def file_append(filename, line):
288 r"""Append given `line`, if not present, at the end of `filename`.
289
290 Usage example::
291
292 >>> import tempfile
293 >>> f = tempfile.NamedTemporaryFile('w', delete=False)
294 >>> f.write('line1\n')
295 >>> f.close()
296 >>> file_append(f.name, 'new line\n')
297 >>> open(f.name).read()
298 'line1\nnew line\n'
299
300 Nothing happens if the file already contains the given `line`::
301
302 >>> file_append(f.name, 'new line\n')
303 >>> open(f.name).read()
304 'line1\nnew line\n'
305
306 A new line is automatically added before the given `line` if it is not
307 present at the end of current file content::
308
309 >>> import tempfile
310 >>> f = tempfile.NamedTemporaryFile('w', delete=False)
311 >>> f.write('line1')
312 >>> f.close()
313 >>> file_append(f.name, 'new line\n')
314 >>> open(f.name).read()
315 'line1\nnew line\n'
316
317 The file is created if it does not exist::
318
319 >>> import tempfile
320 >>> filename = tempfile.mktemp()
321 >>> file_append(filename, 'line1\n')
322 >>> open(filename).read()
323 'line1\n'
324 """
325 with open(filename, 'a+') as f:
326 content = f.read()
327 if line not in content:
328 if content.endswith('\n') or not content:
329 f.write(line)
330 else:
331 f.write('\n' + line)
332
333
334def file_prepend(filename, line):
335 r"""Insert given `line`, if not present, at the beginning of `filename`.
336
337 Usage example::
338
339 >>> import tempfile
340 >>> f = tempfile.NamedTemporaryFile('w', delete=False)
341 >>> f.write('line1\n')
342 >>> f.close()
343 >>> file_prepend(f.name, 'line0\n')
344 >>> open(f.name).read()
345 'line0\nline1\n'
346
347 If the file starts with the given `line`, nothing happens::
348
349 >>> file_prepend(f.name, 'line0\n')
350 >>> open(f.name).read()
351 'line0\nline1\n'
352
353 If the file contains the given `line`, but not at the beginning,
354 the line is moved on top::
355
356 >>> file_prepend(f.name, 'line1\n')
357 >>> open(f.name).read()
358 'line1\nline0\n'
359 """
360 with open(filename, 'r+') as f:
361 lines = f.readlines()
362 if lines[0] != line:
363 if line in lines:
364 lines.remove(line)
365 lines.insert(0, line)
366 f.seek(0)
367 f.writelines(lines)
368
369
370def generate_ssh_keys(directory, filename='id_rsa'):
371 """Generate ssh key pair, saving them inside the given `directory`.
372
373 >>> generate_ssh_keys('/tmp/')
374 >>> open('/tmp/id_rsa').readlines()[0].strip()
375 '-----BEGIN RSA PRIVATE KEY-----'
376 >>> open('/tmp/id_rsa.pub').read().startswith('ssh-rsa')
377 True
378 >>> os.remove('/tmp/id_rsa')
379 >>> os.remove('/tmp/id_rsa.pub')
380
381 The key filename can be changed using the `filename` keyword argument
382 (default is 'id_rsa')::
383
384 >>> generate_ssh_keys('/tmp/', 'custom_key')
385 >>> os.path.exists('/tmp/custom_key')
386 True
387 >>> os.path.exists('/tmp/id_rsa')
388 False
389
390 >>> os.remove('/tmp/custom_key')
391 >>> os.remove('/tmp/custom_key.pub')
392 """
393 path = os.path.join(directory, filename)
394 return call('ssh-keygen', '-q', '-t', 'rsa', '-N', '', '-f', path)
395
396
397def get_container_path(lxc_name, path='', base_path=LXC_PATH):
398 """Return the path of LXC container called `lxc_name`.
399
400 If a `path` is given, return that path inside the container, e.g.::
401
402 >>> get_container_path('mycontainer')
403 '/var/lib/lxc/mycontainer/rootfs/'
404 >>> get_container_path('mycontainer', '/etc/apt/')
405 '/var/lib/lxc/mycontainer/rootfs/etc/apt/'
406 >>> get_container_path('mycontainer', 'home')
407 '/var/lib/lxc/mycontainer/rootfs/home'
408 """
409 return os.path.join(base_path, lxc_name, 'rootfs', path.lstrip('/'))
410
411
412def get_lxc_gateway():
413 """Return a tuple of gateway name and address.
414
415 The gateway name and address will change depending on which version
416 of Ubuntu the script is running on.
417 """
418 release_name = platform.linux_distribution()[2]
419 if release_name == 'oneiric':
420 return 'virbr0', '192.168.122.1'
421 else:
422 return 'lxcbr0', '10.0.3.1'
423
424
425def get_su_command(user, args):
426 """Return a command line as a sequence, prepending "su" if necessary.
427
428 This can be used together with `call` or `check_output` when the `su`
429 context manager is not enaugh (e.g. an external program uses uid rather
430 than euid).
431
432 >>> import getpass
433 >>> current_user = getpass.getuser()
434
435 If the su is requested as current user, the arguments are returned as
436 given::
437
438 >>> get_su_command(current_user, ('ls', '-l'))
439 ('ls', '-l')
440
441 Otherwise, "su" is prepended::
442
443 >>> get_su_command('nobody', ('ls', '-l', 'my file'))
444 ('su', 'nobody', '-c', "ls -l 'my file'")
445 """
446 if get_user_ids(user)[0] != os.getuid():
447 args = [i for i in args if i is not None]
448 return ('su', user, '-c', join_command(args))
449 return args
450
451
452def get_user_home(user):
453 """Return the home directory of the given `user`.
454
455 >>> get_user_home('root')
456 '/root'
457
458 If the user does not exist, return a default /home/[username] home::
459
460 >>> get_user_home('_this_user_does_not_exist_')
461 '/home/_this_user_does_not_exist_'
462 """
463 try:
464 return pwd.getpwnam(user).pw_dir
465 except KeyError:
466 return os.path.join(os.path.sep, 'home', user)
467
468
469def get_user_ids(user):
470 """Return the uid and gid of given `user`, e.g.::
471
472 >>> get_user_ids('root')
473 (0, 0)
474 """
475 userdata = pwd.getpwnam(user)
476 return userdata.pw_uid, userdata.pw_gid
477
478
479def lxc_in_state(state, lxc_name, timeout=30):
480 """Return True if the LXC named `lxc_name` is in state `state`.
481
482 Return False otherwise.
483 """
484 while timeout:
485 try:
486 output = check_output(
487 'lxc-info', '-n', lxc_name, stderr=subprocess.STDOUT)
488 except CommandError:
489 pass
490 else:
491 if state in output:
492 return True
493 timeout -= 1
494 time.sleep(1)
495 return False
496
497
498lxc_running = partial(lxc_in_state, 'RUNNING')
499lxc_stopped = partial(lxc_in_state, 'STOPPED')
500
501
502def mkdirs(*args):
503 """Create leaf directories (given as `args`) and all intermediate ones.
504
505 >>> import tempfile
506 >>> base_dir = tempfile.mktemp(suffix='/')
507 >>> dir1 = tempfile.mktemp(prefix=base_dir)
508 >>> dir2 = tempfile.mktemp(prefix=base_dir)
509 >>> mkdirs(dir1, dir2)
510 >>> os.path.isdir(dir1)
511 True
512 >>> os.path.isdir(dir2)
513 True
514
515 If the leaf directory already exists the function returns without errors::
516
517 >>> mkdirs(dir1)
518
519 An `OSError` is raised if the leaf path exists and it is a file::
520
521 >>> f = tempfile.NamedTemporaryFile(
522 ... 'w', delete=False, prefix=base_dir)
523 >>> f.close()
524 >>> mkdirs(f.name) # doctest: +ELLIPSIS
525 Traceback (most recent call last):
526 OSError: ...
527 """
528 for directory in args:
529 try:
530 os.makedirs(directory)
531 except OSError as err:
532 if err.errno != errno.EEXIST or os.path.isfile(directory):
533 raise
534
535
536def ssh(location, user=None, caller=subprocess.call):
537 """Return a callable that can be used to run ssh shell commands.
538
539 The ssh `location` and, optionally, `user` must be given.
540 If the user is None then the current user is used for the connection.
541
542 The callable internally uses the given `caller`::
543
544 >>> def caller(cmd):
545 ... print cmd
546 >>> sshcall = ssh('example.com', 'myuser', caller=caller)
547 >>> root_sshcall = ssh('example.com', caller=caller)
548 >>> sshcall('ls -l') # doctest: +ELLIPSIS
549 ('ssh', '-t', ..., 'myuser@example.com', '--', 'ls -l')
550 >>> root_sshcall('ls -l') # doctest: +ELLIPSIS
551 ('ssh', '-t', ..., 'example.com', '--', 'ls -l')
552
553 If the ssh command exits with an error code, an `SSHError` is raised::
554
555 >>> ssh('loc', caller=lambda cmd: 1)('ls -l') # doctest: +ELLIPSIS
556 Traceback (most recent call last):
557 SSHError: ...
558
559 If ignore_errors is set to True when executing the command, no error
560 will be raised, even if the command itself returns an error code.
561
562 >>> sshcall = ssh('loc', caller=lambda cmd: 1)
563 >>> sshcall('ls -l', ignore_errors=True)
564 """
565 if user is not None:
566 location = '{user}@{location}'.format(user=user, location=location)
567
568 def _sshcall(cmd, ignore_errors=False):
569 sshcmd = (
570 'ssh',
571 '-t',
572 '-t', # Yes, this second -t is deliberate. See `man ssh`.
573 '-o', 'StrictHostKeyChecking=no',
574 '-o', 'UserKnownHostsFile=/dev/null',
575 location,
576 '--', cmd,
577 )
578 if caller(sshcmd) and not ignore_errors:
579 raise SSHError('Error running command: ' + ' '.join(sshcmd))
580
581 return _sshcall
582
583
584@contextmanager
585def su(user):
586 """A context manager to temporarily run the script as a different user."""
587 uid, gid = get_user_ids(user)
588 os.setegid(gid)
589 os.seteuid(uid)
590 home = get_user_home(user)
591 with environ(HOME=home):
592 try:
593 yield Env(uid, gid, home)
594 finally:
595 os.setegid(os.getgid())
596 os.seteuid(os.getuid())
597
598
599def this_command(directory, args):
600 """Return a command line to re-launch this script using given `args`.
601
602 The command returned will execute this script from `directory`::
603
604 >>> import os
605 >>> script_name = os.path.basename(__file__)
606
607 >>> cmd = this_command('/home/user/lp/branches', ['install'])
608 >>> cmd == ('cd /home/user/lp/branches && devel/utilities/' +
609 ... script_name + ' install')
610 True
611
612 Arguments are correctly quoted::
613
614 >>> cmd = this_command('/path', ['-arg', 'quote me'])
615 >>> cmd == ('cd /path && devel/utilities/' +
616 ... script_name + " -arg 'quote me'")
617 True
618 """
619 script = join_command([
620 os.path.join(LP_CHECKOUT, 'utilities', os.path.basename(__file__)),
621 ] + list(args))
622 return join_command(['cd', directory]) + ' && ' + script
623
624
625def user_exists(username):
626 """Return True if given `username` exists, e.g.::
627
628 >>> user_exists('root')
629 True
630 >>> user_exists('_this_user_does_not_exist_')
631 False
632 """
633 try:
634 pwd.getpwnam(username)
635 except KeyError:
636 return False
637 return True
638
639
640def setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir):
641 """Set up Launchpad repository and source dependencies.
642
643 Return True if new changes are pulled from bzr repository.
644 """
645 # Using real su because bzr uses uid.
646 if os.path.exists(checkout_dir):
647 # Pull the repository.
648 revno_args = ('bzr', 'revno', checkout_dir)
649 revno = check_output(*revno_args)
650 call(*get_su_command(user, ['bzr', 'pull', '-d', checkout_dir]))
651 changed = revno != check_output(*revno_args)
652 else:
653 # Branch the repository.
654 cmd = ('bzr', 'branch',
655 LP_REPOS[1] if valid_ssh_keys else LP_REPOS[0], checkout_dir)
656 call(*get_su_command(user, cmd))
657 changed = True
658 # Check repository integrity.
659 if subprocess.call(['bzr', 'status', '-q', checkout_dir]):
660 raise LaunchpadError(
661 'Repository {0} is corrupted.'.format(checkout_dir))
662 # Set up source dependencies.
663 with su(user):
664 for subdir in ('eggs', 'yui', 'sourcecode'):
665 mkdirs(os.path.join(dependencies_dir, subdir))
666 download_cache = os.path.join(dependencies_dir, 'download-cache')
667 if os.path.exists(download_cache):
668 call('bzr', 'up', download_cache)
669 else:
670 call('bzr', 'co', '--lightweight', LP_SOURCE_DEPS, download_cache)
671 return changed
672
673
674def setup_external_sourcecode(
675 user, valid_ssh_keys, checkout_dir, dependencies_dir):
676 """Update and link external sourcecode."""
677 cmd = (
678 'utilities/update-sourcecode',
679 None if valid_ssh_keys else '--use-http',
680 os.path.join(dependencies_dir, 'sourcecode'),
681 )
682 with cd(checkout_dir):
683 # Using real su because update-sourcecode uses uid.
684 call(*get_su_command(user, cmd))
685 with su(user):
686 call('utilities/link-external-sourcecode', dependencies_dir)
687
688
689def make_launchpad(user, checkout_dir, install=False):
690 """Make and optionally install Launchpad."""
691 # Using real su because mailman make script uses uid.
692 call(*get_su_command(user, ['make', '-C', checkout_dir]))
693 if install:
694 call('make', '-C', checkout_dir, 'install')
695
696
697def initialize(
698 user, full_name, email, lpuser, private_key, public_key, valid_ssh_keys,
699 dependencies_dir, directory):
700 """Initialize host machine."""
701 # Install necessary deb packages. This requires Oneiric or later.
702 call('apt-get', 'update')
703 apt_get_install(*BASE_PACKAGES)
704 # Create the user (if he does not exist).
705 if not user_exists(user):
706 call('useradd', '-m', '-s', '/bin/bash', '-U', user)
707 # Generate root ssh keys if they do not exist.
708 if not os.path.exists('/root/.ssh/id_rsa.pub'):
709 generate_ssh_keys('/root/.ssh/')
710 with su(user) as env:
711 # Set up the user's ssh directory. The ssh key must be associated
712 # with the lpuser's Launchpad account.
713 ssh_dir = os.path.join(env.home, '.ssh')
714 mkdirs(ssh_dir)
715 # Generate user ssh keys if none are supplied.
716 if not valid_ssh_keys:
717 generate_ssh_keys(ssh_dir)
718 private_key = open(os.path.join(ssh_dir, 'id_rsa')).read()
719 public_key = open(os.path.join(ssh_dir, 'id_rsa.pub')).read()
720 priv_file = os.path.join(ssh_dir, 'id_rsa')
721 pub_file = os.path.join(ssh_dir, 'id_rsa.pub')
722 auth_file = os.path.join(ssh_dir, 'authorized_keys')
723 known_hosts = os.path.join(ssh_dir, 'known_hosts')
724 known_host_content = check_output(
725 'ssh-keyscan', '-t', 'rsa', 'bazaar.launchpad.net')
726 for filename, contents, mode in [
727 (priv_file, private_key, 'w'),
728 (pub_file, public_key, 'w'),
729 (auth_file, public_key, 'a'),
730 (known_hosts, known_host_content, 'a'),
731 ]:
732 with open(filename, mode) as f:
733 f.write(contents + '\n')
734 os.chmod(filename, 0644)
735 os.chmod(priv_file, 0600)
736 # Set up bzr and Launchpad authentication.
737 call('bzr', 'whoami', formataddr([full_name, email]))
738 if valid_ssh_keys:
739 subprocess.call(['bzr', 'lp-login', lpuser])
740 # Set up the repository.
741 mkdirs(directory)
742 call('bzr', 'init-repo', '--2a', directory)
743 # Set up the codebase.
744 checkout_dir = os.path.join(directory, LP_CHECKOUT)
745 setup_codebase(user, valid_ssh_keys, checkout_dir, dependencies_dir)
746 # Set up bzr locations
747 tmpl = ''.join('{0} = {1}\n'.format(k, v) for k, v in LP_BZR_LOCATIONS)
748 locations = tmpl.format(checkout_dir=checkout_dir, lpuser=lpuser)
749 contents = '[{0}]\n'.format(directory) + locations
750 with su(user) as env:
751 bzr_locations = os.path.join(env.home, '.bazaar', 'locations.conf')
752 file_append(bzr_locations, contents)
753
754
755def setup_apt(no_repositories=True):
756 """Setup, update and upgrade deb packages."""
757 if not no_repositories:
758 distro = check_output('lsb_release', '-cs').strip()
759 # APT repository update.
760 for reposirory in APT_REPOSITORIES:
761 assume_yes = None if distro == 'lucid' else '-y'
762 call('add-apt-repository', assume_yes,
763 reposirory.format(distro=distro))
764 # XXX frankban 2012-01-13 - Bug 892892: upgrading mountall in LXC
765 # containers currently does not work.
766 call("echo 'mountall hold' | dpkg --set-selections", shell=True)
767 call('apt-get', 'update')
768 # Install base and Launchpad deb packages.
769 apt_get_install(*LP_PACKAGES)
770
771
772def setup_launchpad(user, dependencies_dir, directory, valid_ssh_keys):
773 """Set up the Launchpad environment."""
774 # User configuration.
775 subprocess.call(['adduser', user, 'sudo'])
776 gid = pwd.getpwnam(user).pw_gid
777 subprocess.call(['addgroup', '--gid', str(gid), user])
778 # Set up Launchpad dependencies.
779 checkout_dir = os.path.join(directory, LP_CHECKOUT)
780 setup_external_sourcecode(
781 user, valid_ssh_keys, checkout_dir, dependencies_dir)
782 with su(user):
783 # Create Apache document roots, to avoid warnings.
784 mkdirs(*LP_APACHE_ROOTS)
785 # Set up Apache modules.
786 for module in LP_APACHE_MODULES.split():
787 call('a2enmod', module)
788 with cd(checkout_dir):
789 # Launchpad database setup.
790 call('utilities/launchpad-database-setup', user)
791 # Make and install launchpad.
792 make_launchpad(user, checkout_dir, install=True)
793 # Set up container hosts file.
794 lines = ['{0}\t{1}'.format(ip, names) for ip, names in HOSTS_CONTENT]
795 file_append(HOSTS_FILE, '\n'.join(lines))
796
797
798def update_launchpad(user, valid_ssh_keys, dependencies_dir, directory, apt):
799 """Update the Launchpad environment."""
800 if apt:
801 setup_apt(no_repositories=True)
802 checkout_dir = os.path.join(directory, LP_CHECKOUT)
803 # Update the Launchpad codebase.
804 changed = setup_codebase(
805 user, valid_ssh_keys, checkout_dir, dependencies_dir)
806 setup_external_sourcecode(
807 user, valid_ssh_keys, checkout_dir, dependencies_dir)
808 if changed:
809 make_launchpad(user, checkout_dir, install=False)
810
811
812def link_sourcecode_in_branches(user, dependencies_dir, directory):
813 """Link external sourcecode for all branches in the project."""
814 checkout_dir = os.path.join(directory, LP_CHECKOUT)
815 cmd = os.path.join(checkout_dir, 'utilities', 'link-external-sourcecode')
816 with su(user):
817 for dirname in os.listdir(directory):
818 branch = os.path.join(directory, dirname)
819 sourcecode_dir = os.path.join(branch, 'sourcecode')
820 if (branch != checkout_dir and
821 os.path.exists(sourcecode_dir) and
822 os.path.isdir(sourcecode_dir)):
823 call(cmd, '--parent', dependencies_dir, '--target', branch)
824
825
826def create_lxc(user, lxc_name, lxc_arch, lxc_os):
827 """Create the LXC named `lxc_name` sharing `user` home directory.
828
829 The container will be used as development environment or as base template
830 for parallel testing using ephemeral instances.
831 """
832 # Install necessary deb packages.
833 apt_get_install(*LXC_PACKAGES)
834 # XXX 2012-02-02 gmb bug=925024:
835 # These calls need to be removed once the lxc vs. apparmor bug
836 # is resolved, since having apparmor enabled for lxc is very
837 # much a Good Thing.
838 # Disable the apparmor profiles for lxc so that we don't have
839 # problems installing postgres.
840 call('ln', '-s',
841 '/etc/apparmor.d/usr.bin.lxc-start', '/etc/apparmor.d/disable/')
842 call('apparmor_parser', '-R', '/etc/apparmor.d/usr.bin.lxc-start')
843 # Update resolv file in order to get the ability to ssh into the LXC
844 # container using its name.
845 lxc_gateway_name, lxc_gateway_address = get_lxc_gateway()
846 file_prepend(RESOLV_FILE, 'nameserver {0}\n'.format(lxc_gateway_address))
847 file_append(
848 DHCP_FILE,
849 'prepend domain-name-servers {0};\n'.format(lxc_gateway_address))
850 # Container configuration template.
851 content = LXC_OPTIONS.format(interface=lxc_gateway_name)
852 with open(LXC_CONFIG_TEMPLATE, 'w') as f:
853 f.write(content)
854 # Creating container.
855 call(
856 'lxc-create',
857 '-t', 'ubuntu',
858 '-n', lxc_name,
859 '-f', LXC_CONFIG_TEMPLATE,
860 '--',
861 '-r {os} -a {arch} -b {user}'.format(
862 os=lxc_os, arch=lxc_arch, user=user),
863 )
864 # Set up root ssh key.
865 user_authorized_keys = os.path.expanduser(
866 '~' + user + '/.ssh/authorized_keys')
867 with open(user_authorized_keys, 'a') as f:
868 f.write(open('/root/.ssh/id_rsa.pub').read())
869 dst = get_container_path(lxc_name, '/root/.ssh/')
870 mkdirs(dst)
871 shutil.copy(user_authorized_keys, dst)
872
873
874def initialize_lxc(lxc_name, lxc_os):
875 """Initialize LXC container."""
876 base_packages = list(BASE_PACKAGES)
877 if lxc_os == 'lucid':
878 # Install argparse to be able to run this script inside a lucid lxc.
879 base_packages.append('python-argparse')
880 ssh(lxc_name)(
881 'DEBIAN_FRONTEND=noninteractive '
882 'apt-get install -y ' + ' '.join(base_packages))
883
884
885def setup_launchpad_lxc(
886 user, dependencies_dir, directory, valid_ssh_keys, lxc_name):
887 """Set up the Launchpad environment inside an LXC."""
888 # Use ssh to call this script from inside the container.
889 args = [
890 'install', '-u', user, '-a', 'setup_apt', 'setup_launchpad',
891 '-d', dependencies_dir, '-c', directory
892 ]
893 cmd = this_command(directory, args)
894 ssh(lxc_name)(cmd)
895
896
897def start_lxc(lxc_name):
898 """Start the lxc instance named `lxc_name`."""
899 call('lxc-start', '-n', lxc_name, '-d')
900
901
902def stop_lxc(lxc_name):
903 """Stop the lxc instance named `lxc_name`."""
904 ssh(lxc_name)('poweroff')
905 if not lxc_stopped(lxc_name):
906 subprocess.call(['lxc-stop', '-n', lxc_name])
907
908
909def wait_for_lxc(lxc_name, trials=60, sleep_seconds=1):
910 """Try to ssh as `user` into the LXC container named `lxc_name`."""
911 sshcall = ssh(lxc_name)
912 while True:
913 trials -= 1
914 try:
915 sshcall('true')
916 except SSHError:
917 if not trials:
918 raise
919 time.sleep(sleep_seconds)
920 else:
921 break
922
923
924def handle_user(namespace):
925 """Handle user argument.
926
927 This validator populates namespace with `home_dir` name::
928
929 >>> import getpass
930 >>> username = getpass.getuser()
931
932 >>> namespace = argparse.Namespace(user=username)
933
934 >>> handle_user(namespace)
935 >>> namespace.home_dir == '/home/' + username
936 True
937
938 The validation fails if the current user is root and no user is provided::
939
940 >>> namespace = argparse.Namespace(user=None, euid=0)
941 >>> handle_user(namespace) # doctest: +ELLIPSIS
942 Traceback (most recent call last):
943 ValidationError: argument user ...
944 """
945 if namespace.user is None:
946 if not namespace.euid:
947 raise ValidationError('argument user can not be omitted if '
948 'the script is run as root.')
949 namespace.user = pwd.getpwuid(namespace.euid).pw_name
950 namespace.home_dir = get_user_home(namespace.user)
951
952
953def handle_lpuser(namespace):
954 """Handle lpuser argument.
955
956 If lpuser is not provided by namespace, the user name is used::
957
958 >>> import getpass
959 >>> username = getpass.getuser()
960
961 >>> namespace = argparse.Namespace(user=username, lpuser=None)
962 >>> handle_lpuser(namespace)
963 >>> namespace.lpuser == username
964 True
965 """
966 if namespace.lpuser is None:
967 namespace.lpuser = namespace.user
968
969
970def handle_userdata(namespace, whois=bzr_whois):
971 """Handle full_name and email arguments.
972
973 If they are not provided, this function tries to obtain them using
974 the given `whois` callable::
975
976 >>> namespace = argparse.Namespace(
977 ... full_name=None, email=None, user='root')
978 >>> email = 'email@example.com'
979 >>> handle_userdata(namespace, lambda user: (user, email))
980 >>> namespace.full_name == namespace.user
981 True
982 >>> namespace.email == email
983 True
984
985 The validation fails if full_name or email are not provided and
986 they can not be obtained using the `whois` callable::
987
988 >>> namespace = argparse.Namespace(
989 ... full_name=None, email=None, user='root')
990 >>> handle_userdata(namespace, lambda user: None) # doctest: +ELLIPSIS
991 Traceback (most recent call last):
992 ValidationError: arguments full-name ...
993
994 >>> namespace = argparse.Namespace(
995 ... full_name=None, email=None, user='this_user_does_not_exist')
996 >>> handle_userdata(namespace) # doctest: +ELLIPSIS
997 Traceback (most recent call last):
998 ValidationError: arguments full-name ...
999
1000 It does not make sense to provide only one argument::
1001
1002 >>> namespace = argparse.Namespace(full_name='Foo Bar', email=None)
1003 >>> handle_userdata(namespace) # doctest: +ELLIPSIS
1004 Traceback (most recent call last):
1005 ValidationError: arguments full-name ...
1006 """
1007 args = (namespace.full_name, namespace.email)
1008 if not all(args):
1009 if any(args):
1010 raise ValidationError(
1011 'arguments full-name and email: '
1012 'either none or both must be provided.')
1013 if user_exists(namespace.user):
1014 userdata = whois(namespace.user)
1015 if userdata is None:
1016 raise ValidationError(
1017 'arguments full-name and email are required: '
1018 'bzr user id not found.')
1019 namespace.full_name, namespace.email = userdata
1020 else:
1021 raise ValidationError(
1022 'arguments full-name and email are required: '
1023 'system user not found.')
1024
1025
1026def handle_ssh_keys(namespace):
1027 r"""Handle private and public ssh keys.
1028
1029 Keys contained in the namespace are escaped::
1030
1031 >>> private = r'PRIVATE\nKEY'
1032 >>> public = r'PUBLIC\nKEY'
1033 >>> namespace = argparse.Namespace(
1034 ... private_key=private, public_key=public)
1035 >>> handle_ssh_keys(namespace)
1036 >>> namespace.private_key == private.decode('string-escape')
1037 True
1038 >>> namespace.public_key == public.decode('string-escape')
1039 True
1040 >>> namespace.valid_ssh_keys
1041 True
1042
1043 Keys are None if they are not provided and can not be found in the
1044 current home directory::
1045
1046 >>> namespace = argparse.Namespace(
1047 ... private_key=None, home_dir='/tmp/__does_not_exists__')
1048 >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS
1049 >>> print namespace.private_key
1050 None
1051 >>> print namespace.public_key
1052 None
1053 >>> namespace.valid_ssh_keys
1054 False
1055
1056 If only one of private_key and public_key is provided, a
1057 ValidationError will be raised.
1058
1059 >>> namespace = argparse.Namespace(
1060 ... private_key=private, public_key=None,
1061 ... home_dir='/tmp/__does_not_exists__')
1062 >>> handle_ssh_keys(namespace) # doctest: +ELLIPSIS
1063 Traceback (most recent call last):
1064 ValidationError: arguments private-key...
1065 """
1066 namespace.valid_ssh_keys = True
1067 for attr, filename in (
1068 ('private_key', 'id_rsa'),
1069 ('public_key', 'id_rsa.pub')):
1070 value = getattr(namespace, attr, None)
1071 if value:
1072 setattr(namespace, attr, value.decode('string-escape'))
1073 else:
1074 path = os.path.join(namespace.home_dir, '.ssh', filename)
1075 try:
1076 value = open(path).read()
1077 except IOError:
1078 value = None
1079 namespace.valid_ssh_keys = False
1080 setattr(namespace, attr, value)
1081 if bool(namespace.private_key) != bool(namespace.public_key):
1082 raise ValidationError(
1083 "arguments private-key and public-key: "
1084 "both must be provided or neither must be provided.")
1085
1086
1087def handle_directories(namespace):
1088 """Handle checkout and dependencies directories.
1089
1090 The ~ construction is automatically expanded::
1091
1092 >>> namespace = argparse.Namespace(
1093 ... directory='~/launchpad', dependencies_dir='~/launchpad/deps',
1094 ... home_dir='/home/foo')
1095 >>> handle_directories(namespace)
1096 >>> namespace.directory
1097 '/home/foo/launchpad'
1098 >>> namespace.dependencies_dir
1099 '/home/foo/launchpad/deps'
1100
1101 The validation fails for directories not residing inside the home::
1102
1103 >>> namespace = argparse.Namespace(
1104 ... directory='/tmp/launchpad',
1105 ... dependencies_dir='~/launchpad/deps',
1106 ... home_dir='/home/foo')
1107 >>> handle_directories(namespace) # doctest: +ELLIPSIS
1108 Traceback (most recent call last):
1109 ValidationError: argument directory ...
1110
1111 The validation fails if the directory contains spaces::
1112
1113 >>> namespace = argparse.Namespace(directory='my directory')
1114 >>> handle_directories(namespace) # doctest: +ELLIPSIS
1115 Traceback (most recent call last):
1116 ValidationError: argument directory ...
1117 """
1118 if ' ' in namespace.directory:
1119 raise ValidationError('argument directory can not contain spaces.')
1120 for attr in ('directory', 'dependencies_dir'):
1121 directory = getattr(
1122 namespace, attr).replace('~', namespace.home_dir)
1123 if not directory.startswith(namespace.home_dir + os.path.sep):
1124 raise ValidationError(
1125 'argument {0} does not reside under the home '
1126 'directory of the system user.'.format(attr))
1127 setattr(namespace, attr, directory)
1128
1129
1130class ArgumentParser(argparse.ArgumentParser):
1131 """A customized argument parser for `argparse`."""
1132
1133 def __init__(self, *args, **kwargs):
1134 self.actions = []
1135 self.subparsers = None
1136 super(ArgumentParser, self).__init__(*args, **kwargs)
1137
1138 def add_argument(self, *args, **kwargs):
1139 """Override to store actions in a "public" instance attribute.
1140
1141 >>> parser = ArgumentParser()
1142 >>> parser.add_argument('arg1')
1143 >>> parser.add_argument('arg2')
1144 >>> [action.dest for action in parser.actions]
1145 ['help', 'arg1', 'arg2']
1146 """
1147 action = super(ArgumentParser, self).add_argument(*args, **kwargs)
1148 self.actions.append(action)
1149
1150 def register_subcommand(self, name, subcommand_class, handler=None):
1151 """Add a subcommand to this parser.
1152
1153 A sub command is registered giving the sub command `name` and class::
1154
1155 >>> parser = ArgumentParser()
1156
1157 >>> class SubCommand(BaseSubCommand):
1158 ... def handle(self, namespace):
1159 ... return self.name
1160
1161 >>> parser = ArgumentParser()
1162 >>> _ = parser.register_subcommand('foo', SubCommand)
1163
1164 The `main` method of the subcommand class is added to namespace, and
1165 can be used to actually handle the sub command execution.
1166
1167 >>> namespace = parser.parse_args(['foo'])
1168 >>> namespace.main(namespace)
1169 'foo'
1170
1171 A `handler` callable can also be provided to handle the subcommand
1172 execution::
1173
1174 >>> handler = lambda namespace: 'custom handler'
1175
1176 >>> parser = ArgumentParser()
1177 >>> _ = parser.register_subcommand(
1178 ... 'bar', SubCommand, handler=handler)
1179
1180 >>> namespace = parser.parse_args(['bar'])
1181 >>> namespace.main(namespace)
1182 'custom handler'
1183 """
1184 if self.subparsers is None:
1185 self.subparsers = self.add_subparsers(
1186 title='subcommands',
1187 help='Each subcommand accepts --h or --help to describe it.')
1188 subcommand = subcommand_class(name, handler=handler)
1189 parser = self.subparsers.add_parser(
1190 subcommand.name, help=subcommand.help)
1191 subcommand.add_arguments(parser)
1192 parser.set_defaults(main=subcommand.main, get_parser=lambda: parser)
1193 return subcommand
1194
1195 def get_args_from_namespace(self, namespace):
1196 """Return a list of arguments taking values from `namespace`.
1197
1198 Having a parser defined as usual::
1199
1200 >>> parser = ArgumentParser()
1201 >>> _ = parser.add_argument('--foo')
1202 >>> _ = parser.add_argument('bar')
1203 >>> namespace = parser.parse_args('--foo eggs spam'.split())
1204
1205 It is possible to recreate the argument list taking values from
1206 a different namespace::
1207
1208 >>> namespace.foo = 'changed'
1209 >>> parser.get_args_from_namespace(namespace)
1210 ['--foo', 'changed', 'spam']
1211 """
1212 args = []
1213 for action in self.actions:
1214 dest = action.dest
1215 option_strings = action.option_strings
1216 value = getattr(namespace, dest, None)
1217 if value:
1218 if option_strings:
1219 args.append(option_strings[0])
1220 if isinstance(value, list):
1221 args.extend(value)
1222 elif not isinstance(value, bool):
1223 args.append(value)
1224 return args
1225
1226 def _handle_help(self, namespace):
1227 """Help sub command handler."""
1228 command = namespace.command
1229 help = self.prefix_chars + 'h'
1230 args = [help] if command is None else [command, help]
1231 self.parse_args(args)
1232
1233 def _add_help_subcommand(self):
1234 """Add an help sub command to this parser."""
1235 name = 'help'
1236 choices = self.subparsers.choices.keys()
1237 if name not in choices:
1238 choices.append(name)
1239 parser = self.subparsers.add_parser(
1240 name, help='More help on a command.')
1241 parser.add_argument('command', nargs='?', choices=choices)
1242 parser.set_defaults(main=self._handle_help)
1243
1244 def parse_args(self, *args, **kwargs):
1245 """Override to add an help sub command.
1246
1247 The help sub command is added only if other sub commands exist::
1248
1249 >>> stderr, sys.stderr = sys.stderr, sys.stdout
1250 >>> parser = ArgumentParser()
1251 >>> parser.parse_args(['help'])
1252 Traceback (most recent call last):
1253 SystemExit: 2
1254 >>> sys.stderr = stderr
1255
1256 >>> class SubCommand(BaseSubCommand): pass
1257 >>> _ = parser.register_subcommand('command', SubCommand)
1258 >>> namespace = parser.parse_args(['help'])
1259 >>> namespace.main(namespace)
1260 Traceback (most recent call last):
1261 SystemExit: 0
1262 """
1263 if self.subparsers is not None:
1264 self._add_help_subcommand()
1265 return super(ArgumentParser, self).parse_args(*args, **kwargs)
1266
1267
1268class BaseSubCommand(object):
1269 """Base class defining a sub command.
1270
1271 Objects of this class can be used to easily add sub commands to this
1272 script as plugins, providing arguments, validating them, restarting
1273 as root if needed.
1274
1275 Override `add_arguments()` to add arguments, `validators` to add
1276 namespace validators, and `handle()` to manage sub command execution::
1277
1278 >>> def validator(namespace):
1279 ... namespace.bar = True
1280
1281 >>> class SubCommand(BaseSubCommand):
1282 ... help = 'Sub command example.'
1283 ... validators = (validator,)
1284 ...
1285 ... def add_arguments(self, parser):
1286 ... super(SubCommand, self).add_arguments(parser)
1287 ... parser.add_argument('--foo')
1288 ...
1289 ... def handle(self, namespace):
1290 ... return namespace
1291
1292 Register the sub command using `ArgumentParser.register_subcommand`::
1293
1294 >>> parser = ArgumentParser()
1295 >>> sub_command = parser.register_subcommand('spam', SubCommand)
1296
1297 Now the subcommand has a name::
1298
1299 >>> sub_command.name
1300 'spam'
1301
1302 The sub command handler can be called using `namespace.main()`::
1303
1304 >>> namespace = parser.parse_args('spam --foo eggs'.split())
1305 >>> namespace = namespace.main(namespace)
1306 >>> namespace.foo
1307 'eggs'
1308 >>> namespace.bar
1309 True
1310
1311 The help attribute of sub command instances is used to generate
1312 the command usage message::
1313 >>> help = parser.format_help()
1314 >>> 'spam' in help
1315 True
1316 >>> 'Sub command example.' in help
1317 True
1318 """
1319
1320 help = ''
1321 needs_root = False
1322 validators = ()
1323
1324 def __init__(self, name, handler=None):
1325 self.name = name
1326 self.handler = handler or self.handle
1327
1328 def __repr__(self):
1329 return '<{klass}: {name}>'.format(
1330 klass=self.__class__.__name__, name=self.name)
1331
1332 def init_namespace(self, namespace):
1333 """Add `run_as_root` and `euid` names to the given `namespace`."""
1334 euid = os.geteuid()
1335 namespace.euid, namespace.run_as_root = euid, not euid
1336
1337 def get_needs_root(self, namespace):
1338 """Return True if root is needed to run this subcommand.
1339
1340 Subclasses can override this to dynamically change this value
1341 based on the given `namespace`.
1342 """
1343 return self.needs_root
1344
1345 def get_validators(self, namespace):
1346 """Return an iterable of namespace validators for this sub command.
1347
1348 Subclasses can override this to dynamically change validators
1349 based on the given `namespace`.
1350 """
1351 return self.validators
1352
1353 def validate(self, parser, namespace):
1354 """Validate the current namespace.
1355
1356 The method `self.get_validators` can contain an iterable of objects
1357 that are called once the arguments namespace is fully populated.
1358 This allows cleaning and validating arguments that depend on
1359 each other, or on the current environment.
1360
1361 Each validator is a callable object, takes the current namespace
1362 and can raise ValidationError if the arguments are not valid::
1363
1364 >>> import sys
1365 >>> stderr, sys.stderr = sys.stderr, sys.stdout
1366 >>> def validator(namespace):
1367 ... raise ValidationError('nothing is going on')
1368 >>> sub_command = BaseSubCommand('foo')
1369 >>> sub_command.validators = [validator]
1370 >>> sub_command.validate(ArgumentParser(), argparse.Namespace())
1371 Traceback (most recent call last):
1372 SystemExit: 2
1373 >>> sys.stderr = stderr
1374 """
1375 for validator in self.get_validators(namespace):
1376 try:
1377 validator(namespace)
1378 except ValidationError as err:
1379 parser.error(err)
1380
1381 def restart_as_root(self, parser, namespace):
1382 """Restart this script using *sudo*.
1383
1384 The arguments are recreated using the given `namespace`.
1385 """
1386 args = parser.get_args_from_namespace(namespace)
1387 return subprocess.call(['sudo', sys.argv[0], self.name] + args)
1388
1389 def main(self, namespace):
1390 """Entry point for subcommand execution.
1391
1392 This method takes care of:
1393
1394 - current argparse subparser retrieval
1395 - namespace initialization
1396 - namespace validation
1397 - script restart as root (if this sub command needs to be run as root)
1398
1399 If everything is ok the sub command handler is called passing
1400 the validated namespace.
1401 """
1402 parser = namespace.get_parser()
1403 self.init_namespace(namespace)
1404 self.validate(parser, namespace)
1405 if self.get_needs_root(namespace) and not namespace.run_as_root:
1406 return self.restart_as_root(parser, namespace)
1407 return self.handler(namespace)
1408
1409 def handle(self, namespace):
1410 """Default sub command handler.
1411
1412 Subclasses must either implement this method or provide another
1413 callable handler during sub command registration.
1414 """
1415 raise NotImplementedError
1416
1417 def add_arguments(self, parser):
1418 """Here subclasses can add arguments to the subparser."""
1419 pass
1420
1421
1422class ActionsBasedSubCommand(BaseSubCommand):
1423 """A sub command that uses "actions" to handle its execution.
1424
1425 Actions are callables stored in the `actions` attribute, together
1426 with the arguments they expect. Those arguments are strings
1427 representing attributes of the argparse namespace::
1428
1429 >>> trace = []
1430
1431 >>> def action1(foo):
1432 ... trace.append('action1 received ' + foo)
1433
1434 >>> def action2(foo, bar):
1435 ... trace.append('action2 received {0} and {1}'.format(foo, bar))
1436
1437 >>> class SubCommand(ActionsBasedSubCommand):
1438 ... actions = (
1439 ... (action1, 'foo'),
1440 ... (action2, 'foo', 'bar'),
1441 ... )
1442 ...
1443 ... def add_arguments(self, parser):
1444 ... super(SubCommand, self).add_arguments(parser)
1445 ... parser.add_argument('--foo')
1446 ... parser.add_argument('--bar')
1447
1448 This class implements an handler method that executes actions in the
1449 order they are provided::
1450
1451 >>> parser = ArgumentParser()
1452 >>> _ = parser.register_subcommand('sub', SubCommand)
1453 >>> namespace = parser.parse_args('sub --foo eggs --bar spam'.split())
1454 >>> namespace.main(namespace)
1455 >>> trace
1456 ['action1 received eggs', 'action2 received eggs and spam']
1457
1458 A special argument `-a` or `--actions` is automatically added to the
1459 parser. It can be used to execute only one or a subset of actions::
1460
1461 >>> trace = []
1462
1463 >>> namespace = parser.parse_args('sub --foo eggs -a action1'.split())
1464 >>> namespace.main(namespace)
1465 >>> trace
1466 ['action1 received eggs']
1467
1468 A special argument `--skip-actions` is automatically added to the
1469 parser. It can be used to skip one or more actions::
1470
1471 >>> trace = []
1472
1473 >>> namespace = parser.parse_args(
1474 ... 'sub --foo eggs --skip-actions action1'.split())
1475 >>> namespace.main(namespace)
1476 >>> trace
1477 ['action2 received eggs and None']
1478
1479 The actions execution is stopped if an action raises `LaunchpadError`.
1480 In that case, the error is returned by the handler.
1481
1482 >>> trace = []
1483
1484 >>> def erroneous_action(foo):
1485 ... raise LaunchpadError('error')
1486
1487 >>> class SubCommandWithErrors(SubCommand):
1488 ... actions = (
1489 ... (action1, 'foo'),
1490 ... (erroneous_action, 'foo'),
1491 ... (action2, 'foo', 'bar'),
1492 ... )
1493
1494 >>> parser = ArgumentParser()
1495 >>> _ = parser.register_subcommand('sub', SubCommandWithErrors)
1496 >>> namespace = parser.parse_args('sub --foo eggs'.split())
1497 >>> error = namespace.main(namespace)
1498 >>> error.message
1499 'error'
1500
1501 The action `action2` is not executed::
1502
1503 >>> trace
1504 ['action1 received eggs']
1505 """
1506
1507 actions = ()
1508
1509 def __init__(self, *args, **kwargs):
1510 super(ActionsBasedSubCommand, self).__init__(*args, **kwargs)
1511 self._action_names = []
1512 self._actions = {}
1513 for action_args in self.actions:
1514 action, args = action_args[0], action_args[1:]
1515 action_name = self._get_action_name(action)
1516 self._action_names.append(action_name)
1517 self._actions[action_name] = (action, args)
1518
1519 def _get_action_name(self, action):
1520 """Return the string representation of an action callable.
1521
1522 The name is retrieved using attributes lookup for `action_name`
1523 and then `__name__`::
1524
1525 >>> def action1():
1526 ... pass
1527 >>> action1.action_name = 'myaction'
1528
1529 >>> def action2():
1530 ... pass
1531
1532 >>> sub_command = ActionsBasedSubCommand('foo')
1533 >>> sub_command._get_action_name(action1)
1534 'myaction'
1535 >>> sub_command._get_action_name(action2)
1536 'action2'
1537 """
1538 try:
1539 return action.action_name
1540 except AttributeError:
1541 return action.__name__
1542
1543 def add_arguments(self, parser):
1544 super(ActionsBasedSubCommand, self).add_arguments(parser)
1545 parser.add_argument(
1546 '-a', '--actions', nargs='+', choices=self._action_names,
1547 help='Call one or more internal functions.')
1548 parser.add_argument(
1549 '--skip-actions', nargs='+', choices=self._action_names,
1550 help='Skip one or more internal functions.')
1551
1552 def handle(self, namespace):
1553 skip_actions = namespace.skip_actions or []
1554 action_names = filter(
1555 lambda action_name: action_name not in skip_actions,
1556 namespace.actions or self._action_names)
1557 for action_name in action_names:
1558 action, arg_names = self._actions[action_name]
1559 args = [getattr(namespace, i) for i in arg_names]
1560 try:
1561 action(*args)
1562 except LaunchpadError as err:
1563 return err
1564
1565
1566class InstallSubCommand(ActionsBasedSubCommand):
1567 """Install the Launchpad environment."""
1568
1569 actions = (
1570 (initialize,
1571 'user', 'full_name', 'email', 'lpuser', 'private_key',
1572 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'),
1573 (setup_apt, 'no_repositories'),
1574 (setup_launchpad,
1575 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys'),
1576 )
1577 help = __doc__
1578 needs_root = True
1579 validators = (
1580 handle_user,
1581 handle_lpuser,
1582 handle_userdata,
1583 handle_ssh_keys,
1584 handle_directories,
1585 )
1586
1587 def add_arguments(self, parser):
1588 super(InstallSubCommand, self).add_arguments(parser)
1589 parser.add_argument(
1590 '-u', '--user',
1591 help='The name of the system user to be created or updated. '
1592 'The current user is used if this script is not run as '
1593 'root and this argument is omitted.')
1594 parser.add_argument(
1595 '-e', '--email',
1596 help='The email of the user, used for bzr whoami. This argument '
1597 'can be omitted if a bzr id exists for current user.')
1598 parser.add_argument(
1599 '-f', '--full-name',
1600 help='The full name of the user, used for bzr whoami. '
1601 'This argument can be omitted if a bzr id exists for '
1602 'current user.')
1603 parser.add_argument(
1604 '-l', '--lpuser',
1605 help='The name of the Launchpad user that will be used to '
1606 'check out dependencies. If not provided, the system '
1607 'user name is used.')
1608 parser.add_argument(
1609 '-v', '--private-key',
1610 help='The SSH private key for the Launchpad user (without '
1611 'passphrase). If this argument is omitted and a keypair is '
1612 'not found in the home directory of the system user a new '
1613 'SSH keypair will be generated and the checkout of the '
1614 'Launchpad code will use HTTP rather than bzr+ssh.')
1615 parser.add_argument(
1616 '-b', '--public-key',
1617 help='The SSH public key for the Launchpad user. '
1618 'If this argument is omitted and a keypair is not found '
1619 'in the home directory of the system user a new SSH '
1620 'keypair will be generated and the checkout of the '
1621 'Launchpad code will use HTTP rather than bzr+ssh.')
1622 parser.add_argument(
1623 '-d', '--dependencies-dir', default=DEPENDENCIES_DIR,
1624 help='The directory of the Launchpad dependencies to be created. '
1625 'The directory must reside under the home directory of the '
1626 'given user (see -u argument). '
1627 '[DEFAULT={0}]'.format(DEPENDENCIES_DIR))
1628 parser.add_argument(
1629 '-c', '--directory', default=CHECKOUT_DIR,
1630 help='The directory of the Launchpad repository to be created. '
1631 'The directory must reside under the home directory of the '
1632 'given user (see -u argument). '
1633 '[DEFAULT={0}]'.format(CHECKOUT_DIR))
1634 parser.add_argument(
1635 '-N', '--no-repositories', action='store_true',
1636 help='Do not add APT repositories.')
1637
1638
1639class UpdateSubCommand(ActionsBasedSubCommand):
1640 """Update the Launchpad environment to latest version."""
1641
1642 actions = (
1643 (update_launchpad,
1644 'user', 'valid_ssh_keys', 'dependencies_dir', 'directory', 'apt'),
1645 (link_sourcecode_in_branches,
1646 'user', 'dependencies_dir', 'directory'),
1647 )
1648 help = __doc__
1649 validators = (
1650 handle_user,
1651 handle_ssh_keys,
1652 handle_directories,
1653 )
1654
1655 def get_needs_root(self, namespace):
1656 # Root is needed only if an apt update/upgrade is requested.
1657 return namespace.apt
1658
1659 def add_arguments(self, parser):
1660 super(UpdateSubCommand, self).add_arguments(parser)
1661 parser.add_argument(
1662 '-u', '--user',
1663 help='The name of the system user used to update Launchpad. '
1664 'The current user is used if this script is not run as '
1665 'root and this argument is omitted.')
1666 parser.add_argument(
1667 '-d', '--dependencies-dir', default=DEPENDENCIES_DIR,
1668 help='The directory of the Launchpad dependencies to be updated. '
1669 'The directory must reside under the home directory of the '
1670 'given user (see -u argument). '
1671 '[DEFAULT={0}]'.format(DEPENDENCIES_DIR))
1672 parser.add_argument(
1673 '-c', '--directory', default=CHECKOUT_DIR,
1674 help='The directory of the Launchpad repository to be updated. '
1675 'The directory must reside under the home directory of the '
1676 'given user (see -u argument). '
1677 '[DEFAULT={0}]'.format(CHECKOUT_DIR))
1678 parser.add_argument(
1679 '-D', '--apt', action='store_true',
1680 help='Also update deb packages.')
1681
1682
1683class LXCInstallSubCommand(InstallSubCommand):
1684 """Install the Launchpad environment inside an LXC."""
1685
1686 actions = (
1687 (initialize,
1688 'user', 'full_name', 'email', 'lpuser', 'private_key',
1689 'public_key', 'valid_ssh_keys', 'dependencies_dir', 'directory'),
1690 (create_lxc,
1691 'user', 'lxc_name', 'lxc_arch', 'lxc_os'),
1692 (start_lxc, 'lxc_name'),
1693 (wait_for_lxc, 'lxc_name'),
1694 (initialize_lxc,
1695 'lxc_name', 'lxc_os'),
1696 (setup_launchpad_lxc,
1697 'user', 'dependencies_dir', 'directory', 'valid_ssh_keys',
1698 'lxc_name'),
1699 (stop_lxc, 'lxc_name'),
1700 )
1701 help = __doc__
1702
1703 def add_arguments(self, parser):
1704 super(LXCInstallSubCommand, self).add_arguments(parser)
1705 parser.add_argument(
1706 '-n', '--lxc-name', default=LXC_NAME,
1707 help='The LXC container name to setup. '
1708 '[DEFAULT={0}]'.format(LXC_NAME))
1709 parser.add_argument(
1710 '-A', '--lxc-arch', default=LXC_GUEST_ARCH,
1711 help='The LXC container architecture. '
1712 '[DEFAULT={0}]'.format(LXC_GUEST_ARCH))
1713 parser.add_argument(
1714 '-r', '--lxc-os', default=LXC_GUEST_OS,
1715 choices=LXC_GUEST_CHOICES,
1716 help='The LXC container distro codename. '
1717 '[DEFAULT={0}]'.format(LXC_GUEST_OS))
1718
1719
1720def main():
1721 parser = ArgumentParser(description=__doc__)
1722 parser.register_subcommand('install', InstallSubCommand)
1723 parser.register_subcommand('update', UpdateSubCommand)
1724 parser.register_subcommand('lxc-install', LXCInstallSubCommand)
1725 args = parser.parse_args()
1726 return args.main(args)
1727
1728
1729if __name__ == '__main__':
1730 sys.exit(main())