Merge lp:~jimbaker/juju-jitsu/unit-test into lp:juju-jitsu
- unit-test
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Mark Mims | ||||
Approved revision: | 91 | ||||
Merged at revision: | 79 | ||||
Proposed branch: | lp:~jimbaker/juju-jitsu/unit-test | ||||
Merge into: | lp:juju-jitsu | ||||
Diff against target: |
404 lines (+315/-35) 5 files modified
sub-commands/Makefile.am (+3/-1) sub-commands/aiki/cli.py (+6/-1) sub-commands/files/juju (+75/-0) sub-commands/test (+231/-0) sub-commands/topodump (+0/-33) |
||||
To merge this branch: | bzr merge lp:~jimbaker/juju-jitsu/unit-test | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Mark Mims (community) | Approve | ||
Review via email: mp+133807@code.launchpad.net |
Commit message
Description of the change
Adds jitsu test subcommand
Adds new subcommand, jitsu test, that runs unit tests in a charm's tests/ directory.
$ jitsu test --logdir=
Executing this command then results in extensive logging (we may want to consider trimming this down); the key piece for each unit test is something like the following:
PASS: Verify top page is available for mediawiki/0
PASS: Verify top page is available for mediawiki/1
INFO: Completed test ./100_deploy.test
INFO: Passed test results for ./100_deploy.test in /tmp/test-
jitsu test ensures that the specified charm (mediawiki in this example) is deployed from the local repository, specifically the directory from which this test is being run. Otherwise any referenced but not qualified charms are deployed from the charm store.
Unit test ouput is captured in the specified LOGDIR. Note that unit tests can capture additional files by using JITSU_LOGDIR, which is passed as an environment variable to each unit test; it's LOGDIR/
/tmp/test-
.
├── 0
│ └── var
│ └── log
│ └── juju
│ ├── machine-agent.log
│ └── provision-agent.log
├── 1
│ └── var
│ ├── lib
│ │ └── juju
│ │ └── units
│ │ └── mediawiki-0
│ │ └── charm.log
│ └── log
│ └── juju
│ └── machine-agent.log
...
├── passed
└── wget.log
Other useful options include:
--no-bootstrap - so as to reuse an existing set of machines; note that jitsu test will not destroy services, so this would need to be automated outside jitsu test.
--isolate=
Jim Baker (jimbaker) wrote : | # |
Jim Baker (jimbaker) wrote : | # |
I also pushed up lp:~jimbaker/+junk/mediawiki-test-strawman, which shows how to write a unit test, to take advantage of JITSU_LOGDIR. Otherwise lib/test-helpers.sh will generate a temporary dir, but jitsu test won't know about this and therefore cannot capture.
Next up: propose the unit test as an update to the mediawiki charm!
Benji York (benji) wrote : | # |
Thanks for this subcommand, Jim. Nicola and I have been using it to write some tests (see a first cut at lp:~juju-gui/charms/precise/juju-gui/trunk). A few remarks follow.
The charm directory name has to be the same as the charm name itself, which makes testing different branches difficult. Is revisiting that requirement possible?
It seems to us that the exit code from each one of the *.test files should be propagated and reported by the test subcommand (e.g., if any test returns a non-zero then Jitsu should return non-zero).
Bootstrapping from within the command either causes errors or times out: one needs to bootstrap in advance and use the --no-bootstrap option.
When accessing machines for log retrieval, rsync asks for a password for ubuntu@localhost. We can not figure out what that password should be.
We would appreciate any information you can give us about the above. Thanks.
Kapil Thangavelu (hazmat) wrote : | # |
On Wed, Nov 14, 2012 at 1:10 PM, Benji York <email address hidden>wrote:
> Thanks for this subcommand, Jim. Nicola and I have been using it to write
> some tests (see a first cut at lp:~juju-gui/charms/precise/juju-gui/trunk).
> A few remarks follow.
>
> The charm directory name has to be the same as the charm name itself,
> which makes testing different branches difficult. Is revisiting that
> requirement possible?
>
This is a core requirement to guard against errant charm authorship. Its
not a jitsu check per se, but a juju core one. Its unlikely to be revisited
imo, but #juju-dev would be the appropriate place to confirm that.
>
> It seems to us that the exit code from each one of the *.test files should
> be propagated and reported by the test subcommand (e.g., if any test
> returns a non-zero then Jitsu should return non-zero).
>
> Bootstrapping from within the command either causes errors or times out:
> one needs to bootstrap in advance and use the --no-bootstrap option.
>
> When accessing machines for log retrieval, rsync asks for a password for
> ubuntu@localhost. We can not figure out what that password should be.
>
try setting up the key of your environment file as an authorized key for a
user ubuntu.
i assume based on the address that this is being tested with the local
provider, which is a bit of an oddball compared to other providers as the
default user isn't 'ubuntu' but whatever the user account is. ideally the
test runner could distinguish this based on provider type and use the local
provider user account. that would still need manual addition of the ssh key
to authorized keys as the local machine is not cloud-init initialized.
alternatively in the case of the local provider the testrunner eschews
machine log collection in favor of just unit log collection which should
still work as per other providers.
hth,
Kapil
Benji York (benji) wrote : | # |
> On Wed, Nov 14, 2012 at 1:10 PM, Benji York <email address hidden>wrote:
>
> > Thanks for this subcommand, Jim. Nicola and I have been using it to write
> > some tests (see a first cut at lp:~juju-gui/charms/precise/juju-gui/trunk).
> > A few remarks follow.
> >
> > The charm directory name has to be the same as the charm name itself,
> > which makes testing different branches difficult. Is revisiting that
> > requirement possible?
> >
>
> This is a core requirement to guard against errant charm authorship. Its
> not a jitsu check per se, but a juju core one. Its unlikely to be revisited
> imo, but #juju-dev would be the appropriate place to confirm that.
I can see how that would be important when handling submitted charms.
Our use case is a little different though, we want to run functional
tests of a charm in development. It seems a bit heavy-handed to force
all charm authors to name their development directory after the charm.
>
> > When accessing machines for log retrieval, rsync asks for a password for
> > ubuntu@localhost. We can not figure out what that password should be.
> >
>
> try setting up the key of your environment file as an authorized key for a
> user ubuntu.
I don't quite follow, but the next time I look at this I'll investigate
further.
> i assume based on the address that this is being tested with the local
> provider, which is a bit of an oddball compared to other providers as the
> default user isn't 'ubuntu' but whatever the user account is.
Right.
> alternatively in the case of the local provider the testrunner eschews
> machine log collection in favor of just unit log collection which should
> still work as per other providers.
It would be nice to have access to all the logs, but any way to avoid
the unanswerable password prompts in the middle of test runs would be an
improvement.
>
> hth,
> Kapil
Many thanks.
Jim Baker (jimbaker) wrote : | # |
> On Wed, Nov 14, 2012 at 1:10 PM, Benji York <email address hidden>wrote:
>
> > Thanks for this subcommand, Jim. Nicola and I have been using it to write
> > some tests (see a first cut at lp:~juju-gui/charms/precise/juju-gui/trunk).
> > A few remarks follow.
> >
> > The charm directory name has to be the same as the charm name itself,
> > which makes testing different branches difficult. Is revisiting that
> > requirement possible?
> >
>
> This is a core requirement to guard against errant charm authorship. Its
> not a jitsu check per se, but a juju core one. Its unlikely to be revisited
> imo, but #juju-dev would be the appropriate place to confirm that.
Correct, the test in jitsu test is only an extra check to avoid unnecessary test runs. I don't see this changing anytime soon, and if it does in juju, we can readily change jitsu test.
> > It seems to us that the exit code from each one of the *.test files should
> > be propagated and reported by the test subcommand (e.g., if any test
> > returns a non-zero then Jitsu should return non-zero).
I will add that in a future release, it sounds like a great feature.
> >
> > Bootstrapping from within the command either causes errors or times out:
> > one needs to bootstrap in advance and use the --no-bootstrap option.
> >
> > When accessing machines for log retrieval, rsync asks for a password for
> > ubuntu@localhost. We can not figure out what that password should be.
> >
>
> try setting up the key of your environment file as an authorized key for a
> user ubuntu.
>
> i assume based on the address that this is being tested with the local
> provider, which is a bit of an oddball compared to other providers as the
> default user isn't 'ubuntu' but whatever the user account is. ideally the
> test runner could distinguish this based on provider type and use the local
> provider user account. that would still need manual addition of the ssh key
> to authorized keys as the local machine is not cloud-init initialized.
> alternatively in the case of the local provider the testrunner eschews
> machine log collection in favor of just unit log collection which should
> still work as per other providers.
Based on discussion about this in #juju, I will do some additional work on the lxc provider support in the next release.
Mark Mims (mark-mims) wrote : | # |
charm_dir name is a #juju-dev issue.
Please clean up / consolidate log extraction in a future release.
Thanks!
Preview Diff
1 | === modified file 'sub-commands/Makefile.am' | |||
2 | --- sub-commands/Makefile.am 2012-08-09 16:40:53 +0000 | |||
3 | +++ sub-commands/Makefile.am 2012-11-10 21:15:24 +0000 | |||
4 | @@ -10,10 +10,11 @@ | |||
5 | 10 | export \ | 10 | export \ |
6 | 11 | import \ | 11 | import \ |
7 | 12 | open-port \ | 12 | open-port \ |
9 | 13 | provider-info \ | 13 | provider-info \ |
10 | 14 | run-as-hook \ | 14 | run-as-hook \ |
11 | 15 | search \ | 15 | search \ |
12 | 16 | setup-environment \ | 16 | setup-environment \ |
13 | 17 | test \ | ||
14 | 17 | upgrade-charm \ | 18 | upgrade-charm \ |
15 | 18 | watch \ | 19 | watch \ |
16 | 19 | wrap-juju | 20 | wrap-juju |
17 | @@ -31,6 +32,7 @@ | |||
18 | 31 | run-as-hook \ | 32 | run-as-hook \ |
19 | 32 | search \ | 33 | search \ |
20 | 33 | setup-environment \ | 34 | setup-environment \ |
21 | 35 | test \ | ||
22 | 34 | upgrade-charm \ | 36 | upgrade-charm \ |
23 | 35 | watch \ | 37 | watch \ |
24 | 36 | wrap-juju | 38 | wrap-juju |
25 | 37 | 39 | ||
26 | === modified file 'sub-commands/aiki/cli.py' | |||
27 | --- sub-commands/aiki/cli.py 2012-08-08 21:22:19 +0000 | |||
28 | +++ sub-commands/aiki/cli.py 2012-11-10 21:15:24 +0000 | |||
29 | @@ -34,7 +34,10 @@ | |||
30 | 34 | kwargs["help"] = kwargs["description"] # Fallback | 34 | kwargs["help"] = kwargs["description"] # Fallback |
31 | 35 | parser = root_parser.add_parser(subcommand, **kwargs) | 35 | parser = root_parser.add_parser(subcommand, **kwargs) |
32 | 36 | parser.add_argument( | 36 | parser.add_argument( |
34 | 37 | "-e", "--environment", default=None, help="Environment to act upon (otherwise uses default)", metavar="ENVIRONMENT") | 37 | "-e", "--environment", |
35 | 38 | default=os.environ.get("JUJU_ENV"), | ||
36 | 39 | help="Environment to act upon (otherwise uses default)", | ||
37 | 40 | metavar="ENVIRONMENT") | ||
38 | 38 | parser.add_argument( | 41 | parser.add_argument( |
39 | 39 | "--loglevel", default=None, choices=LOG_LEVELS, help="Log level", | 42 | "--loglevel", default=None, choices=LOG_LEVELS, help="Log level", |
40 | 40 | metavar="CRITICAL|ERROR|WARNING|INFO|DEBUG") | 43 | metavar="CRITICAL|ERROR|WARNING|INFO|DEBUG") |
41 | @@ -146,6 +149,8 @@ | |||
42 | 146 | of the script. | 149 | of the script. |
43 | 147 | """ | 150 | """ |
44 | 148 | filename = os.path.join(details["home"], cmd) | 151 | filename = os.path.join(details["home"], cmd) |
45 | 152 | if not (os.path.isfile(filename) and os.access(filename, os.X_OK)): | ||
46 | 153 | return | ||
47 | 149 | attempt_python_parse = subcommand_home == details["home"] | 154 | attempt_python_parse = subcommand_home == details["home"] |
48 | 150 | with open(filename) as f: | 155 | with open(filename) as f: |
49 | 151 | lines = f.readlines() | 156 | lines = f.readlines() |
50 | 152 | 157 | ||
51 | === added directory 'sub-commands/files' | |||
52 | === added file 'sub-commands/files/juju' | |||
53 | --- sub-commands/files/juju 1970-01-01 00:00:00 +0000 | |||
54 | +++ sub-commands/files/juju 2012-11-10 21:15:24 +0000 | |||
55 | @@ -0,0 +1,75 @@ | |||
56 | 1 | #!/usr/bin/env python | ||
57 | 2 | |||
58 | 3 | # A juju delegator that rewrites juju deploy|upgrade-charm ... charm [service] to use the desired respository | ||
59 | 4 | |||
60 | 5 | # Only matches the charm if it equals the environment variable | ||
61 | 6 | # $DELEGATE_CHARM and it is not qualified with cs: or local: | ||
62 | 7 | # specifiers. If so, it rewrites the arg list to include --repository | ||
63 | 8 | # $DELEGATE_REPOSITORY. | ||
64 | 9 | # | ||
65 | 10 | # Then call the real juju with the arg list. | ||
66 | 11 | # | ||
67 | 12 | # Requires that $PATH have the directory of the delegator as the first item in the search path. | ||
68 | 13 | |||
69 | 14 | |||
70 | 15 | import os | ||
71 | 16 | import sys | ||
72 | 17 | |||
73 | 18 | from juju.control import setup_parser, SUBCOMMANDS | ||
74 | 19 | |||
75 | 20 | |||
76 | 21 | def is_passthru(options): | ||
77 | 22 | # Only worry about juju deploy and upgrade-charm subcommands | ||
78 | 23 | if options.parser.prog not in ("juju deploy", "juju upgrade-charm"): | ||
79 | 24 | return True | ||
80 | 25 | |||
81 | 26 | if options.charm != os.environ.get("DELEGATE_CHARM"): | ||
82 | 27 | return True | ||
83 | 28 | |||
84 | 29 | # Charm name cannot be qualified with cs: or local: | ||
85 | 30 | if len(options.charm.split(":")) > 1: | ||
86 | 31 | return True | ||
87 | 32 | |||
88 | 33 | # Repository cannot be specified; this could be set by | ||
89 | 34 | # JUJU_REPOSITORY, so just iterate through the raw args | ||
90 | 35 | for arg in sys.argv[1:]: | ||
91 | 36 | if arg.startswith("--repository"): | ||
92 | 37 | return True | ||
93 | 38 | |||
94 | 39 | return False | ||
95 | 40 | |||
96 | 41 | |||
97 | 42 | def main(): | ||
98 | 43 | # Use juju itself to parse the command line options, to keep it simple; | ||
99 | 44 | # will need to be changed when moving to golang version | ||
100 | 45 | original_args = sys.argv[:] | ||
101 | 46 | parser = setup_parser( | ||
102 | 47 | subcommands=SUBCOMMANDS, | ||
103 | 48 | prog="juju", | ||
104 | 49 | description="juju cloud orchestration admin") | ||
105 | 50 | options, extra = parser.parse_known_args() # don't need to worry about more complex parse setups like juju ssh | ||
106 | 51 | |||
107 | 52 | # Create a new env with PATH that looks up the real juju; use the | ||
108 | 53 | # convention that we pop the first item in the search path to | ||
109 | 54 | # restore PATH | ||
110 | 55 | new_env = dict(os.environ) # make a copy | ||
111 | 56 | new_env["PATH"] = ":".join(new_env["PATH"].split(":")[1:]) # pop the first dir off the path | ||
112 | 57 | |||
113 | 58 | # Then exec in the real juju, with args modified as necessary | ||
114 | 59 | if is_passthru(options): | ||
115 | 60 | print "juju", " ".join(original_args) | ||
116 | 61 | os.execvpe("juju", original_args, new_env) | ||
117 | 62 | else: | ||
118 | 63 | if options.service_name: | ||
119 | 64 | args = original_args[0:-2] | ||
120 | 65 | else: | ||
121 | 66 | args = original_args[0:-1] | ||
122 | 67 | args.extend(["--repository", os.environ.get("DELEGATE_REPOSITORY")]) | ||
123 | 68 | args.append("local:" + options.charm) | ||
124 | 69 | if options.service_name: | ||
125 | 70 | args.append(options.service_name) | ||
126 | 71 | print "juju", " ".join(args) | ||
127 | 72 | os.execvpe("juju", args, new_env) | ||
128 | 73 | |||
129 | 74 | if __name__ == "__main__": | ||
130 | 75 | main() | ||
131 | 0 | 76 | ||
132 | === added file 'sub-commands/test' | |||
133 | --- sub-commands/test 1970-01-01 00:00:00 +0000 | |||
134 | +++ sub-commands/test 2012-11-10 21:15:24 +0000 | |||
135 | @@ -0,0 +1,231 @@ | |||
136 | 1 | #!/usr/bin/env python | ||
137 | 2 | |||
138 | 3 | # TODO snapshot logs at any time in the test | ||
139 | 4 | |||
140 | 5 | import argparse | ||
141 | 6 | import glob | ||
142 | 7 | import logging | ||
143 | 8 | import os.path | ||
144 | 9 | import subprocess | ||
145 | 10 | import sys | ||
146 | 11 | import textwrap | ||
147 | 12 | import uuid | ||
148 | 13 | |||
149 | 14 | import yaml | ||
150 | 15 | |||
151 | 16 | from aiki.cli import make_arg_parser, setup_logging | ||
152 | 17 | from juju.lib.format import YAMLFormat | ||
153 | 18 | |||
154 | 19 | |||
155 | 20 | log = logging.getLogger("jitsu.test") | ||
156 | 21 | |||
157 | 22 | |||
158 | 23 | class Tester(object): | ||
159 | 24 | |||
160 | 25 | def __init__(self, options): | ||
161 | 26 | self.options = options | ||
162 | 27 | |||
163 | 28 | # Create a search path that includes files/juju as the first | ||
164 | 29 | # item when searching for juju; this will enable delegating it | ||
165 | 30 | # so that juju deploy|upgrade-charm use the local repository | ||
166 | 31 | # for the tested charm | ||
167 | 32 | delegated_juju = [os.path.realpath(os.path.normpath(os.path.join(__file__, "..", "files")))] | ||
168 | 33 | delegated_juju.extend(os.environ["PATH"].split(":")) | ||
169 | 34 | self.delegated_path = ":".join(delegated_juju) | ||
170 | 35 | if self.options.isolate: | ||
171 | 36 | self.options.bootstrap = True # implied by --isolate | ||
172 | 37 | |||
173 | 38 | def install_required_test_packages(self): | ||
174 | 39 | if os.path.exists("tests/test.yaml"): | ||
175 | 40 | with open("tests/test.yaml") as f: | ||
176 | 41 | config = yaml.safe_load(f) | ||
177 | 42 | packages = config.get("packages") | ||
178 | 43 | if packages: | ||
179 | 44 | # Check if the test packages are already installed. If | ||
180 | 45 | # not, install them. We use this check to avoid an | ||
181 | 46 | # unnecessary requirement to sudo. | ||
182 | 47 | if subprocess.call(["dpkg", "-s"].extend(packages)): | ||
183 | 48 | subprocess.check_call(["sudo", "apt-get", "--yes", "install"].extend(packages), env=self.env) | ||
184 | 49 | |||
185 | 50 | def run_tests(self): | ||
186 | 51 | if not glob.glob("tests/*.test"): | ||
187 | 52 | log.info("Nothing to do: charm %s has no tests defined", self.options.charm) | ||
188 | 53 | return | ||
189 | 54 | |||
190 | 55 | if os.path.split(os.getcwd())[1] != self.options.charm: | ||
191 | 56 | log.error("Not in the directory of the charm being tested: %s", self.options.charm) | ||
192 | 57 | sys.exit(1) | ||
193 | 58 | |||
194 | 59 | self.install_required_test_packages() | ||
195 | 60 | os.chdir("tests") | ||
196 | 61 | unit_tests = sorted(glob.glob("*.test")) | ||
197 | 62 | for unit_test in unit_tests: | ||
198 | 63 | log.info("Running unit test: %s", unit_test) | ||
199 | 64 | runner = TestRunner(self, unit_test) | ||
200 | 65 | try: | ||
201 | 66 | runner.setup() | ||
202 | 67 | runner.run_unit_test() | ||
203 | 68 | except Exception: | ||
204 | 69 | log.exception("Unit test failure") | ||
205 | 70 | finally: | ||
206 | 71 | runner.teardown() | ||
207 | 72 | log.info("Completed unit test: %s", unit_test) | ||
208 | 73 | |||
209 | 74 | |||
210 | 75 | class TestRunner(object): | ||
211 | 76 | |||
212 | 77 | def __init__(self, tester, unit_test): | ||
213 | 78 | self.tester = tester | ||
214 | 79 | self.unit_test = unit_test | ||
215 | 80 | unit_test_name = os.path.splitext(self.unit_test)[0] | ||
216 | 81 | self.test_logdir = os.path.join(self.tester.options.logdir, unit_test_name) | ||
217 | 82 | if not os.path.exists(self.test_logdir): | ||
218 | 83 | os.makedirs(self.test_logdir) # maybe just do this in scope of an exception, to prevent races (but most unlikely) | ||
219 | 84 | self.env = dict(os.environ) | ||
220 | 85 | self.env["JITSU_LOGDIR"] = self.test_logdir | ||
221 | 86 | self.env["JITSU_UNIT_TEST"] = unit_test_name | ||
222 | 87 | self.env["PATH"] = self.tester.delegated_path | ||
223 | 88 | self.env["DELEGATE_CHARM"] = self.tester.options.charm | ||
224 | 89 | self.env["DELEGATE_REPOSITORY"] = os.path.realpath(os.path.normpath(os.path.join(os.getcwd(), "../../.."))) | ||
225 | 90 | self.bootstrapped = False | ||
226 | 91 | |||
227 | 92 | def setup(self): | ||
228 | 93 | if self.tester.options.isolate: | ||
229 | 94 | self.env["HOME"] = self.tester.options.logdir | ||
230 | 95 | self.juju_env = "jitsu-test-" + str(uuid.uuid1()) | ||
231 | 96 | with open(os.path.expanduser("~/.juju/environments.yaml")) as f: | ||
232 | 97 | env_config = yaml.safe_load(f.read()) | ||
233 | 98 | try: | ||
234 | 99 | cloned = dict(env_config["environments"][self.tester.options.isolate]) | ||
235 | 100 | except KeyError: | ||
236 | 101 | log.error("Isolated environment %s is not defined in ~/.juju/environments.yaml", self.tester.options.isolate) | ||
237 | 102 | sys.exit(1) | ||
238 | 103 | |||
239 | 104 | if "authorized-keys" not in cloned: | ||
240 | 105 | log.error("Isolated environment %s must define authorized-keys", self.tester.options.isolate) | ||
241 | 106 | sys.exit(1) | ||
242 | 107 | cloned.update({"control-bucket": self.juju_env}) | ||
243 | 108 | test_config = { | ||
244 | 109 | "default": self.juju_env, | ||
245 | 110 | "environments": { | ||
246 | 111 | self.juju_env: cloned | ||
247 | 112 | }} | ||
248 | 113 | juju_env_dir = os.path.join(self.tester.options.logdir, ".juju") | ||
249 | 114 | try: | ||
250 | 115 | os.makedirs(juju_env_dir) | ||
251 | 116 | except OSError: | ||
252 | 117 | pass # ignore existing directory | ||
253 | 118 | with open(os.path.join(juju_env_dir, "environments.yaml"), "w") as f: | ||
254 | 119 | f.write(YAMLFormat().format(test_config)) | ||
255 | 120 | else: | ||
256 | 121 | self.juju_env = self.tester.options.environment | ||
257 | 122 | if self.juju_env: | ||
258 | 123 | self.env["JUJU_ENV"] = self.juju_env | ||
259 | 124 | |||
260 | 125 | if self.tester.options.bootstrap: | ||
261 | 126 | log.info("Bootstrapping %s", self.juju_env or "default environment") | ||
262 | 127 | output = subprocess.check_output( | ||
263 | 128 | ["juju", "bootstrap"], env=self.env, stderr=subprocess.STDOUT) | ||
264 | 129 | if "juju environment previously bootstrapped" in output: | ||
265 | 130 | log.error("Environment previously bootstraped (use --no-bootstrap or different environment)") | ||
266 | 131 | sys.exit(1) | ||
267 | 132 | self.bootstrapped = True | ||
268 | 133 | sys.stderr.write(output) # After capture, simply redirect back to stderr | ||
269 | 134 | |||
270 | 135 | def teardown(self): | ||
271 | 136 | log.info("Tearing down unit test: %s", self.unit_test) | ||
272 | 137 | if self.tester.options.bootstrap and self.bootstrapped: | ||
273 | 138 | log.info("Destroying environment") | ||
274 | 139 | subprocess.check_call("echo y | juju destroy-environment", shell=True, env=self.env) | ||
275 | 140 | |||
276 | 141 | def run_unit_test(self): | ||
277 | 142 | try: | ||
278 | 143 | cmd = ["timeout", "--kill-after", "2m", self.tester.options.timeout, os.path.join(".", self.unit_test)] | ||
279 | 144 | output = subprocess.check_output(cmd, env=self.env) | ||
280 | 145 | log.info("Unit test %s: %s", " ".join(cmd), output) | ||
281 | 146 | except subprocess.CalledProcessError, e: | ||
282 | 147 | log.warn("Error running unit test %s: %s %s", self.unit_test, e.returncode, e.output) | ||
283 | 148 | except Exception, e: | ||
284 | 149 | log.error("Error running unit test %s: %s", self.unit_test, e) | ||
285 | 150 | try: | ||
286 | 151 | self.archive_logs() | ||
287 | 152 | except Exception, e: | ||
288 | 153 | log.error("Error archiving logs for %s: %s", self.unit_test, e) | ||
289 | 154 | |||
290 | 155 | def archive_logs(self): | ||
291 | 156 | status = yaml.safe_load(subprocess.check_output(["juju", "status"])) | ||
292 | 157 | for machine, info in status["machines"].iteritems(): | ||
293 | 158 | self.gather_logs(machine, info["dns-name"]) | ||
294 | 159 | |||
295 | 160 | def gather_logs(self, machine, host): | ||
296 | 161 | host_dir = os.path.abspath(os.path.join(self.test_logdir, str(machine))) | ||
297 | 162 | if not os.path.exists(host_dir): | ||
298 | 163 | os.makedirs(host_dir) | ||
299 | 164 | log.info("Gathering logs for %s:%s in %s", machine, host, host_dir) | ||
300 | 165 | self.retrieve_path(host, host_dir, "/var/log/juju") | ||
301 | 166 | if machine != 0: | ||
302 | 167 | self.retrieve_path(host, host_dir, "/var/lib/juju/units/*/charm.log") | ||
303 | 168 | |||
304 | 169 | def retrieve_path(self, host, local_path, remote_path): | ||
305 | 170 | args = ["rsync", "--archive", "--compress", "--relative", "--verbose", | ||
306 | 171 | "-e", "ssh", | ||
307 | 172 | "ubuntu@{host}:{remote_path}".format(host=host, remote_path=remote_path), | ||
308 | 173 | local_path] | ||
309 | 174 | try: | ||
310 | 175 | subprocess.check_call(args) | ||
311 | 176 | except Exception, e: | ||
312 | 177 | log.warn("Error retrieving log %s from %s: %s", remote_path, host, e) | ||
313 | 178 | |||
314 | 179 | |||
315 | 180 | def main(): | ||
316 | 181 | parser = make_parser() | ||
317 | 182 | options = parser.parse_args() | ||
318 | 183 | setup_logging(options) | ||
319 | 184 | tester = Tester(options) | ||
320 | 185 | if options.archive_only: | ||
321 | 186 | runner = TestRunner(tester, "archive-only") | ||
322 | 187 | runner.archive_logs() | ||
323 | 188 | else: | ||
324 | 189 | tester.run_tests() | ||
325 | 190 | |||
326 | 191 | |||
327 | 192 | def make_parser(root_parser=None): | ||
328 | 193 | main_parser = make_arg_parser( | ||
329 | 194 | root_parser, "test", | ||
330 | 195 | formatter_class=argparse.RawDescriptionHelpFormatter, | ||
331 | 196 | description="runs charm unit tests", | ||
332 | 197 | epilog=textwrap.dedent("""\ | ||
333 | 198 | Charms are resolved against the charm store if not otherwise | ||
334 | 199 | qualified, *except for the charm being tested*. | ||
335 | 200 | |||
336 | 201 | SSH host keys are problematic for testing with juju at this | ||
337 | 202 | time. Currently this cannot be mitigated by using per-environment | ||
338 | 203 | ssh_config. | ||
339 | 204 | |||
340 | 205 | Add the following in ~/.ssh/config | ||
341 | 206 | |||
342 | 207 | Host *.amazonaws.com # set for the specific cloud provider | ||
343 | 208 | StrictHostKeyChecking no | ||
344 | 209 | UserKnownHostsFile /dev/null | ||
345 | 210 | |||
346 | 211 | Remember: such setup is a potential security hole, and it should not | ||
347 | 212 | be used for any production systems. However, for testing purposes, | ||
348 | 213 | especially for unit testing, this configuration should be reasonable. | ||
349 | 214 | |||
350 | 215 | TODO: isolated environments should have support for env cleanup, | ||
351 | 216 | such as security group removal. | ||
352 | 217 | """)) | ||
353 | 218 | |||
354 | 219 | main_parser.add_argument("--archive-only", action="store_true", help="Archive all Juju logs") | ||
355 | 220 | main_parser.add_argument("--logdir", required=True, help="Write test logs and snapshots to this directory") | ||
356 | 221 | main_parser.add_argument("--timeout", default="15m", help="Timeout per unit test. Examples: 10m or 900s.") | ||
357 | 222 | main_parser.add_argument("--no-bootstrap", dest="bootstrap", default=True, action="store_false", | ||
358 | 223 | help="Do not bootstrap env before each test, then destroy") | ||
359 | 224 | main_parser.add_argument("--isolate", metavar="ENVIRONMENT", | ||
360 | 225 | help="Runs in an isolated, unique environment cloned from ENVIRONMENT in ~/.juju/environments.yaml") | ||
361 | 226 | main_parser.add_argument("charm", metavar="CHARM", help="Test for this charm") | ||
362 | 227 | return main_parser | ||
363 | 228 | |||
364 | 229 | |||
365 | 230 | if __name__ == '__main__': | ||
366 | 231 | main() | ||
367 | 0 | 232 | ||
368 | === removed file 'sub-commands/topodump' | |||
369 | --- sub-commands/topodump 2012-06-21 23:17:57 +0000 | |||
370 | +++ sub-commands/topodump 1970-01-01 00:00:00 +0000 | |||
371 | @@ -1,33 +0,0 @@ | |||
372 | 1 | #!/usr/bin/env python | ||
373 | 2 | # | ||
374 | 3 | # topodump - dumps the zookeeper topology | ||
375 | 4 | |||
376 | 5 | from twisted.internet.defer import inlineCallbacks | ||
377 | 6 | from aiki.cli import make_arg_parser, setup_logging, run_command | ||
378 | 7 | from juju.state.base import StateBase | ||
379 | 8 | |||
380 | 9 | |||
381 | 10 | def main(): | ||
382 | 11 | parser = make_arg_parser() | ||
383 | 12 | options = parser.parse_args() | ||
384 | 13 | setup_logging(options) | ||
385 | 14 | print "options", options | ||
386 | 15 | run_command(topodump, options) | ||
387 | 16 | |||
388 | 17 | |||
389 | 18 | @inlineCallbacks | ||
390 | 19 | def topodump(result, client, options): | ||
391 | 20 | Dumper = TopoDump(client) | ||
392 | 21 | yield Dumper.dump() | ||
393 | 22 | |||
394 | 23 | |||
395 | 24 | class TopoDump(StateBase): | ||
396 | 25 | |||
397 | 26 | @inlineCallbacks | ||
398 | 27 | def dump(self): | ||
399 | 28 | topology = yield self._read_topology() | ||
400 | 29 | print topology.dump() | ||
401 | 30 | |||
402 | 31 | |||
403 | 32 | if __name__ == '__main__': | ||
404 | 33 | main() |
Reviewers: mp+133807_ code.launchpad. net,
Message:
Please take a look.
Description:
Adds jitsu test subcommand
Adds new subcommand, jitsu test, that runs unit tests in a charm's
tests/ directory.
$ jitsu test --logdir= /tmp/test- results- 42 mediawiki # --logdir LOGDIR
must be specified
Executing this command then results in extensive logging (we may want to
consider trimming this down); the key piece for each unit test is
something like the following:
PASS: Verify top page is available for mediawiki/0 results- 42/100_ deploy, skips=0
PASS: Verify top page is available for mediawiki/1
INFO: Completed test ./100_deploy.test
INFO: Passed test results for ./100_deploy.test in
/tmp/test-
jitsu test ensures that the specified charm (mediawiki in this example)
is deployed from the local repository, specifically the directory from
which this test is being run. Otherwise any referenced but not qualified
charms are deployed from the charm store.
Unit test ouput is captured in the specified LOGDIR. Note that unit UNIT_TEST_ NAME:
tests can capture additional files by using JITSU_LOGDIR, which is
passed as an environment variable to each unit test; it's
LOGDIR/
/tmp/test- results- 42/100_ deploy$ tree
.
├── 0
│ └── var
│ └── log
│ └── juju
│ ├── machine-agent.log
│ └── provision-agent.log
├── 1
│ └── var
│ ├── lib
│ │ └── juju
│ │ └── units
│ │ └── mediawiki-0
│ │ └── charm.log
│ └── log
│ └── juju
│ └── machine-agent.log
...
├── passed
└── wget.log
Other useful options include:
--no-bootstrap - so as to reuse an existing set of machines; note that
jitsu test will not destroy services, so this would need to be automated
outside jitsu test.
--isolate= ENVIRONMENT - creates an isolated environment using a unique test-$( uuid)), including corresponding .juju/environme nts.yaml. Note that no cleanup of security groups;
name (jitsu-
LOGDIR/
this could be automated in testing through euca2ools.
https:/ /code.launchpad .net/~jimbaker/ juju-jitsu/ unit-test/ +merge/ 133807
(do not edit description out of merge proposal)
Please review this at https:/ /codereview. appspot. com/6821104/
Affected files: Makefile. am aiki/cli. py files/juju topodump
A [revision details]
M sub-commands/
M sub-commands/
A sub-commands/
A sub-commands/test
D sub-commands/