Merge lp:~elopio/u1-test-utils/setup-vm into lp:u1-test-utils
- setup-vm
- Merge into trunk
Proposed by
Leo Arias
Status: | Merged |
---|---|
Approved by: | Vincent Ladeuil |
Approved revision: | 54 |
Merged at revision: | 51 |
Proposed branch: | lp:~elopio/u1-test-utils/setup-vm |
Merge into: | lp:u1-test-utils |
Diff against target: |
3691 lines (+3544/-0) 28 files modified
setup_vm/ISSUES (+9/-0) setup_vm/NOTES (+259/-0) setup_vm/README (+241/-0) setup_vm/TODO (+129/-0) setup_vm/bin/setup_vm.py (+1203/-0) setup_vm/bin/ubuntu_admin.sh (+2/-0) setup_vm/pay/install (+40/-0) setup_vm/pay/run (+9/-0) setup_vm/pay/run-for-u1 (+57/-0) setup_vm/pay/test (+5/-0) setup_vm/selftest.py (+24/-0) setup_vm/sso/install (+45/-0) setup_vm/sso/run (+16/-0) setup_vm/sso/run-for-pay (+14/-0) setup_vm/sso/run-for-u1 (+43/-0) setup_vm/sso/test (+11/-0) setup_vm/tests/__init__.py (+75/-0) setup_vm/tests/test_setup_vm.py (+1074/-0) setup_vm/tests/test_test.py (+58/-0) setup_vm/u1/install (+71/-0) setup_vm/u1/run (+4/-0) setup_vm/u1/test (+15/-0) setup_vm/unity/install-sources (+19/-0) setup_vm/unity/run-sso-client (+11/-0) setup_vm/unity/run-syncdaemon (+4/-0) setup_vm/unity/run-unity-lens-music (+6/-0) setup_vm/unity/transient-dist-upgrade (+4/-0) setup_vm/vms.conf (+96/-0) |
To merge this branch: | bzr merge lp:~elopio/u1-test-utils/setup-vm |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Vincent Ladeuil (community) | Approve | ||
Review via email: mp+158828@code.launchpad.net |
Commit message
Merged the setup_vm project into u1testutils.
Description of the change
To post a comment you must log in.
Revision history for this message
Leo Arias (elopio) wrote : | # |
Revision history for this message
Leo Arias (elopio) wrote : | # |
Ready for review!
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | === added directory 'setup_vm' |
2 | === added file 'setup_vm/ISSUES' |
3 | --- setup_vm/ISSUES 1970-01-01 00:00:00 +0000 |
4 | +++ setup_vm/ISSUES 2013-04-17 01:29:27 +0000 |
5 | @@ -0,0 +1,9 @@ |
6 | +* while setting up sso.local .env/db/postgresql.log ends up with: |
7 | + |
8 | +FATAL: role "postgres" does not exist |
9 | + |
10 | +=> This may be "expected" according to mfoord |
11 | + |
12 | + |
13 | +* bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/isd-configs/sso-config/ is missing in the sso config and local.cfg should also be generated to refer to that. |
14 | + |
15 | |
16 | === added file 'setup_vm/NOTES' |
17 | --- setup_vm/NOTES 1970-01-01 00:00:00 +0000 |
18 | +++ setup_vm/NOTES 2013-04-17 01:29:27 +0000 |
19 | @@ -0,0 +1,259 @@ |
20 | +Here we keep notes about the tests we will run on these vms. Once finished all |
21 | +the TODOs, this will be probably moved to moztrap, or immediately and |
22 | +magically automated. |
23 | + |
24 | +== On the host == |
25 | +1. Install the requirements: |
26 | + $ sudo apt-get install libvirt-bin qemu virtinst virt-manager |
27 | + |
28 | +2. Install the apt cache: |
29 | + $ sudo apt-get install squid-deb-proxy |
30 | + |
31 | +3. Enable the cache for launchpad's private ppas: |
32 | + See point 9 of the README. |
33 | + |
34 | +4. Configure the vms: |
35 | + |
36 | + # This directory will be used as download cache for the Ubuntu images. |
37 | + $ mkdir ~/installers/ubuntu |
38 | + # This directory will store the disk images for the virtual machines. |
39 | + $ mkdir ~/images |
40 | + $ editor ~/vms.conf |
41 | + |
42 | + vm.ram_size=2048 |
43 | + vm.cpus=2 |
44 | + # Tweak the cpu model according to your needs |
45 | + vm.cpu_model=amd64 |
46 | + vm.download_cache=~/installers/ubuntu |
47 | + vm.images_dir=~/images |
48 | + # Tweak according to your squid-deb-proxy setup, 8000 is the default port. |
49 | + # Use {your-ip} or any address reachable by the vms (keeping in mind that |
50 | + # avahi's .local domain may not be up in the early stages of the install |
51 | + vm.apt_proxy = http://{your-ip}:800 |
52 | + vm.launchpad_id=your-launchpad-id |
53 | + # This is the ssh key of your host machine. Make sure that you have |
54 | + uploaded it to https://launchpad.net/~/+editsshkeys |
55 | + vm.ssh_authorized_keys = ~/.ssh/id_rsa.pub |
56 | + # This is the ssh key for your VMs. It might be safer if it's different from |
57 | + # your machine's key. Make sure that you have uploaded it to launchpad too. |
58 | + vm.ssh_keys=~/.ssh/rsa-vms |
59 | + # A default user (ubuntu) is created, here is its password |
60 | + vm.password = you-re-on-you-own-use-a-simple-or-complex-password |
61 | + # Go to https://launchpad.net/~/+archivesubscriptions to get the password for |
62 | + # the Ubuntu One hackers PPA. Click the View link on the PPA row, and on the |
63 | + # sources list entries you will see something like |
64 | + # https://your-launchpad-id:the-password@... |
65 | + ppa.ubuntuone_hackers.password=the-password |
66 | + |
67 | + sso.address=sso.local |
68 | + pay.address=pay.local |
69 | + u1.address=u1.local |
70 | + |
71 | +5. Get the branch: |
72 | + |
73 | + $ bzr branch lp:~online-services-qa/u1-test-utils/setup_vm |
74 | + $ cd setup_vm |
75 | + |
76 | +6. Download the image for the server (do so every time you want to use a |
77 | + fresh image): |
78 | + |
79 | + $ ./bin/setup_vm.py precise-server-pristine --download |
80 | + |
81 | +# TODO use LXCs insteal of virtual machines for the servers. |
82 | +7. Install the pristine server vm: |
83 | + |
84 | + $ ./bin/setup_vm.py raring-pristine --install |
85 | + |
86 | +8. Download the image for the desktop (do so every time you want to use a |
87 | + fresh image):: |
88 | + |
89 | + $ ./bin/setup_vm.py raring-desktop-pristine --download |
90 | + |
91 | +8. Install the pristine desktop vm: |
92 | + |
93 | + $ ./bin/setup_vm.py raring-pristine --install |
94 | + |
95 | +# TODO now it might be better to start all the servers on the same VM. |
96 | +9. Set the SSO server (Only needed for in-dash payments tests): |
97 | + |
98 | + $ ./bin/setup_vm.py sso --install |
99 | + $ virsh start sso |
100 | + $ ssh ubuntu@sso.local ~/bin/run-for-u1 |
101 | + |
102 | +10. Set up the Pay server (Only needed for in-dash payments tests): |
103 | + |
104 | + $ ./bin/setup_vm.py pay --install |
105 | + $ virsh start pay |
106 | + $ ssh ubuntu@pay.local ~/bin/run-for-u1 |
107 | + |
108 | +11. Set up the U1 server (Only needed for in-dash payments tests): |
109 | + |
110 | + $ ./bin/setup_vm.py u1 --install |
111 | + $ virsh start u1 |
112 | + $ ssh ubuntu@u1.local ~/bin/run |
113 | + |
114 | +12. Set up the Filesync server (Only needed for in-dash payments tests): |
115 | + |
116 | + TODO. See below the notes to set it up on the same u1 server. |
117 | + TODO. Explore how to set it up in a different server. |
118 | + |
119 | +13. Set up the Music Search server: |
120 | + |
121 | + TODO. Do we need it? |
122 | + |
123 | +13. Install the CurucĂș server (Only needed for Smart Scopes tests): |
124 | + |
125 | + TODO. |
126 | + |
127 | +17. Install the desktop machine that will run the tests: |
128 | + |
129 | + $ ./bin/setup_vm.py unity-prevalidation --install |
130 | + |
131 | + |
132 | +Set up the in-dash payments tests against the local servers |
133 | +=========================================================== |
134 | + |
135 | +This is for the happy path. |
136 | + |
137 | +1. Sign in to the unity-prevalidation machine using virt-manager. |
138 | + |
139 | +2. Kill syncdaemon if it is running. (As this is a pristine machine, this is not necessary) |
140 | + |
141 | +3. Open seahorse and delete the Ubuntu One credentials, if present. (As this is a pristine machine, this is not necessary) |
142 | + |
143 | +3. Log in with Staging Ubuntu SSO: |
144 | + |
145 | + # TODO: Currently we can just connect to production. This is a regression, |
146 | + # see bug http://pad.lv/1161067 |
147 | + $ ~/bin/run-sso-client |
148 | + |
149 | + Click the "Log-in with my existing account." link. |
150 | + Fill the form with: |
151 | + # TODO we need to create the user with the API helpers on u1-test-utils. |
152 | + Email address: u1test+local-only@canonical.com |
153 | + Password: Hola123* |
154 | + Click the "Sign In" button. |
155 | + |
156 | + Autopilot test: |
157 | + http://bazaar.launchpad.net/~elopio/ubuntu-sso-client/autopilot/view/head:/ubuntu_sso/tests/acceptance/test_ubuntu_sso_client.py |
158 | + |
159 | +4. Start syncdaemon: |
160 | + |
161 | + # TODO we currently don't have a filesync server. |
162 | + $ ~/bin/run-syncdaemon |
163 | + |
164 | +5. Start the unity musicstore daemon: |
165 | + |
166 | + $ ~/bin/run-unity-lens-music |
167 | + |
168 | +6. Start the control panel. |
169 | + |
170 | + # TODO probably not neccessary. |
171 | + |
172 | +7. Add a credit card to the user. |
173 | + |
174 | + # Use the API helpers on u1-test-utils. |
175 | + # We still need to log in to the pay webiste first. See http://pad.lv/1144523 |
176 | + |
177 | +8. Enable the automatic payments for the user. |
178 | + |
179 | + # TODO wait for http://ur1.ca/d6z4n to land and then extend the u1-test-utils |
180 | + # API helpers to do this. |
181 | + |
182 | +# For paypal payments, we would still need to do a lot of stuff on the website. |
183 | +# TODO do we need to test paypal payments? |
184 | + |
185 | +Run the in-dash payments tests against the local servers |
186 | +=========================================================== |
187 | + |
188 | +This is the happy path. |
189 | + |
190 | +1. Super+M. |
191 | +2. Search for 'hendrix'. |
192 | +3. Wait for the search to complete. |
193 | +4. Click the first album |
194 | +5. Click the Download button. |
195 | +6. Enter the password. |
196 | +7. Click the Purchase button. |
197 | + |
198 | +All the tests are now documented in moztrap. |
199 | + |
200 | +Set up the filesync server on the same machine as ubuntuone-servers |
201 | +==================================================================== |
202 | + |
203 | + # On the host |
204 | + $ ssh ubuntu@u1.local |
205 | + # On the vm. |
206 | + $ bzr branch lp:ubuntuone-filesync |
207 | + $ cd ubuntuone-filesync |
208 | + $ make link-sourcedeps |
209 | + $ editor lib/u1backends/db/config.py |
210 | + Change line 10 from: |
211 | + db_dir = os.path.abspath(os.path.join(get_tmpdir(), 'db1')) |
212 | + To: |
213 | + db_dir = os.path.abspath( |
214 | + os.path.join('/home/ubuntu/ubuntuone-servers/tmp', 'db1')) |
215 | + $ make start |
216 | + # Use ubuntuone-servers S4, statsd and AMQP. |
217 | + $ ln -s ../../ubuntuone-servers/tmp/rabbitmq-ubuntuone.port tmp/rabbitmq-ubuntuone.port |
218 | + $ ln -s ../../ubuntuone-servers/tmp/statsd.port tmp/statsd.port |
219 | + $ ln -s ../../ubuntuone-servers/tmp/s4.port tmp/s4.port |
220 | + $ make start-supervisor start-filesync-dummy-group |
221 | + # TODO we can also start-filesync-oauth-group. Ask #u1-di. |
222 | + |
223 | +Set up the curucu server |
224 | +======================== |
225 | + |
226 | + # There is a dummy website that accesses the server: |
227 | + # https://productsearch.ubuntu.com/smartscopes/v1/dashmock?geo_store=US |
228 | + # On ~pedronis/curucu/canonistack-deploy, there's a README with the |
229 | + # instructios to deploy curucu on a canonistack machine with Juju. |
230 | + # TODO try to deploy it with juju on our vms. |
231 | + # TODO we need an amazon key. |
232 | + # On the host |
233 | + $ ./bin/setup_vm.py precise-curucu-server --install |
234 | + $ virsh start precise-curucu-server |
235 | + $ ssh ubuntu@precise-curucu-server |
236 | + # On the vm. |
237 | + # TODO install the dependencies, what are the dependencies? |
238 | + $ bzr branch lp:curucu |
239 | + $ cd curucu |
240 | + $ editor try.cfg |
241 | + |
242 | + [amazon] |
243 | + key = amazon key |
244 | + secret = amazon secret |
245 | + |
246 | + [u1ms] |
247 | + service_url = http://musicsearch.ubuntu.com/v1/ |
248 | + |
249 | + [feedback_store] |
250 | + interval = 4 |
251 | + # when set to empty storing feedback is disabled |
252 | + store_directory = /tmp/feedback |
253 | + |
254 | + $ CURUCU_CFG=try.cfg U1CONFIG=configs/development-lazr.conf PYTHONPATH=.:lib gunicorn -w 2 -k gevent_wsgi -b 0.0.0.0:8000 curucu.wsgi:app |
255 | + |
256 | +Mounting guest disk images on the host |
257 | +====================================== |
258 | + |
259 | +This requires root access (what did you expect ;-p) and the current |
260 | +directory should contain the vm disk images. |
261 | + |
262 | +apt-get install qemu-nbd |
263 | + |
264 | +root@saw:/# modprobe nbd # once |
265 | +root@saw:/# mkdir /mnt/disk1 |
266 | +root@saw:/# mkdir /mnt/seed |
267 | + |
268 | +root@saw:/# qemu-nbd -c /dev/nbd0 raring-pristine.qcow2 |
269 | +root@saw:/# mount /dev/nbd0p1 /mnt/disk1 |
270 | +root@saw:/# umount /mnt/disk1/ |
271 | +root@saw:/# qemu-nbd -d /dev/nbd0 |
272 | + |
273 | +root@saw:/# qemu-nbd -c /dev/nbd1 raring-test.seed |
274 | +root@saw:/# mount /dev/nbd1 /mnt/seed |
275 | +mount: warning: /mnt/seed seems to be mounted read-only. |
276 | +root@saw:/# umount /mnt/seed |
277 | +root@saw:/# qemu-nbd -d /dev/nbd1 |
278 | +/dev/nbd1 disconnected |
279 | |
280 | === added file 'setup_vm/README' |
281 | --- setup_vm/README 1970-01-01 00:00:00 +0000 |
282 | +++ setup_vm/README 2013-04-17 01:29:27 +0000 |
283 | @@ -0,0 +1,241 @@ |
284 | +Getting started: |
285 | +================ |
286 | + |
287 | +1. Install the dependencies: |
288 | + |
289 | + sudo apt-get install bzr python-testtools python-yaml |
290 | + sudp apt-get install libvirt-bin qemu qemu-utils virtinst |
291 | + sudo apt-get install qemu-kvm-spice python-spice-client-gtk |
292 | + |
293 | + (Optional): |
294 | + To use a gui manager to see the desktop: |
295 | + sudo apt-get install virt-manager |
296 | + |
297 | + To use Apt proxy to speed up multiple downloads of the same packages: |
298 | + sudo apt-get install squid-deb-proxy |
299 | + (See point 9 about configuring the apt cache.) |
300 | + |
301 | +2. Reboot to allow kvm to activate on the running kernel. |
302 | + |
303 | +3. Get the code: |
304 | + |
305 | + bzr branch lp:~online-services-qa/u1-test-utils/setup_vm |
306 | + |
307 | +4. Run the tests: |
308 | + |
309 | + cd setup_vm |
310 | + ./selftest.py |
311 | + |
312 | + (The test_install_from_seed will require you to enter your password |
313 | + because it executes a command with sudo. Your user must be a sudoer.) |
314 | + |
315 | +5. Configure a virtual machine: |
316 | + |
317 | + Write the file ~/vms.conf with something like this: |
318 | + |
319 | + vm.ram_size=2048 |
320 | + vm.cpus=2 |
321 | + vm.cpu_model=amd64 |
322 | + |
323 | + [raring-pristine] |
324 | + vm.name = raring-pristine |
325 | + vm.update = True |
326 | + vm.packages = bzr, ubuntu-desktop, avahi-daemon |
327 | + vm.release=raring |
328 | + vm.ssh_authorized_keys = {your SSH public key path (~ is allowed, eg ~/.ssh/id_rsa.pub)} |
329 | + |
330 | + Create the ~/.config/setup_vm directory where cloud-init configuration |
331 | + files will be stored for each vm. Alternatively, you can create a |
332 | + directory (~/vms) where you want and add the following line in ~/vms.conf: |
333 | + |
334 | + vm.vms_dir=~/vms |
335 | + |
336 | + Optionally, you can setup scripts to be executed as root or as the |
337 | + ubuntu user just before the vm is powered off: |
338 | + |
339 | + vm.root_script = {path to a script on the host} |
340 | + vm.ubuntu_script = {path to a script on the host} |
341 | + |
342 | + These scripts *must* specify a shebang line and can be written in any |
343 | + language that can be run from a shebang line. |
344 | + |
345 | + You can also ask for some local scripts to be uploaded with: |
346 | + |
347 | + vm.uploaded_scripts = sso/run, sso/run-for-pay |
348 | + |
349 | + The config options in these scripts will be expanded before the upload. |
350 | + |
351 | + PPAS needs a bit of care to setup, as an example, the unity experimental |
352 | + prevalidation PPA is configured by going to |
353 | + https://launchpad.net/~ubuntu-unity/+archive/experimental-prevalidation |
354 | + Click the "Technical details about this PPA" link. |
355 | + Select the distribution from the combo box. |
356 | + Copy the apt line below: |
357 | + |
358 | + vm.apt_sources=deb http://ppa.launchpad.net/ubuntu-unity/experimental-prevalidation/ubuntu {vm.release} main|52D62F45 |
359 | + |
360 | + The page displays Signing Key: 1024R/52D62F45. Please note that only |
361 | + 52D62F45 should be specified, and that the url and the key are separated |
362 | + by '|' with no intervening spaces. |
363 | + |
364 | + For a private PPA, make sure to include your launchpad id and your |
365 | + password for that PPA in the URL. It would look something like this: |
366 | + |
367 | + vm.apt_sources = deb https://<lp id>:<ppa password>@private-ppa.launchpad.net/a-user/ppa-name/ubuntu {vm.release} main|<ppa key> |
368 | + |
369 | +6. (Optional) Create a system-wide vms.conf. |
370 | + |
371 | + In some cases, some options are better defined in a system-wide config |
372 | + file (/etc/libvirt/vms.conf). This file is queried if no definitions are |
373 | + found in ~/vms.conf and can define a no-name section and vm sections. |
374 | + |
375 | +7. (Optional) You can configure the location where the image will be |
376 | + downloaded with something like this in the vms.conf file: |
377 | + |
378 | + vm.download_cache=~/installers/ubuntu |
379 | + |
380 | +8. (Optional) You can configure the location where the virtual machines will |
381 | + be stored with something like this in the vms.conf file: |
382 | + |
383 | + vm.images_dir=~/images |
384 | + |
385 | +9. (Optional) Set up an apt cache, so repeated virtual machine installs will |
386 | + be faster, downloading the packages from the cache instead of an Ubuntu |
387 | + archive mirror: |
388 | + |
389 | + Add this to the vms.conf file: |
390 | + |
391 | + vm.apt_proxy = http://{your-squid-deb-proxy-ip}:8000 |
392 | + |
393 | + If you need to install packages from non official Ubuntu repositories, you |
394 | + will need to configure the proxy. For example, common tasks would require |
395 | + to access Launchpad public and private PPAs. For that, write the file |
396 | + /etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-local-vms with: |
397 | + |
398 | + # /etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-local-vms |
399 | + |
400 | + # network destinations that are allowed by this cache targeted at |
401 | + # locally installed vms |
402 | + |
403 | + # launchpad personal package archives |
404 | + ppa.launchpad.net |
405 | + # launchpad private personal package archives |
406 | + private-ppa.launchpad.net |
407 | + |
408 | + After that, restart the proxy: |
409 | + |
410 | + sudo restart squid-deb-proxy |
411 | + |
412 | + Each time you modify some file under /etc/squid-deb-proxy, don't forget to |
413 | + restart the service. |
414 | + |
415 | +10. Download the image: |
416 | + |
417 | + ./bin/setup_vm.py --download raring-pristine |
418 | + |
419 | + (This command will require you to enter your password because the |
420 | + directory where the image will be downloaded might be under control of |
421 | + the root user. Your user must be a sudoer. A pending task is to ask for |
422 | + the password just when needed.) |
423 | + |
424 | +11. Install the virtual machine: |
425 | + |
426 | + ./bin/setup_vm.py --install raring-pristine |
427 | + |
428 | + (This command will require you to enter your password because some of the |
429 | + operations it executes require root access. Your user must be a sudoer.) |
430 | + |
431 | +12. You can ssh into the virtual machine: |
432 | + |
433 | + virsh start raring-pristine |
434 | + ssh ubuntu@raring-pristine.local |
435 | + |
436 | + No password is needed because your SSH public key is authorized. |
437 | + |
438 | +13. You can run the virtual machine from virt-manager to get a graphical user |
439 | + interface: |
440 | + |
441 | + Open virt-manager. |
442 | + Right-click on the machine and select Run. |
443 | + You will be presented with the display manager greeter. |
444 | + Log in with the user ubuntu, password ubuntu. |
445 | + |
446 | + (You may need to do the following if virt-manager says it can't connect, |
447 | + sudo usermod -a -G libvirtd $USER (replace $USER with your username) |
448 | + and reboot the system) |
449 | + |
450 | +14. (Optional) You can set up a "throw away" virtual machine on top of |
451 | + another. We call it "throw away" because all modifications happening there |
452 | + won't affect the disk image of the backing on virtual machine. |
453 | + |
454 | + In the vms.conf described above add: |
455 | + |
456 | + [raring-test] |
457 | + vm.name = raring-test |
458 | + vm.update = False |
459 | + vm.release = raring |
460 | + vm.ssh_authorized_keys = {your SSH public key} |
461 | + # The name of the disk image used as a base |
462 | + vm.backing = raring-pristine.qcow2 |
463 | + |
464 | + Create the new vm with: |
465 | + |
466 | + ./bin/setup_vm.py --install raring-test |
467 | + |
468 | + The vm creation and boot should be faster. |
469 | + |
470 | +15. (Very optional) A few commands for virsh that may be of use: |
471 | + virsh list (shows what is running) |
472 | + virsh start x (start a vm with the name x) |
473 | + virsh destroy x (force shutdowns a vm with the name x) |
474 | + virsh undefine x [--remove-all-storage] (Deletes a vm with the name x) |
475 | + |
476 | +16. (Optional) Raise sudo timeout. |
477 | + |
478 | + If you run into vm installs taking too long and waiting for your |
479 | + password to fisnish, you can change the default value (15 minutes) by |
480 | + adding a file in /etc/sudoers.d containing: |
481 | + |
482 | + Defaults:<your login here> timestamp_timeout=60 |
483 | + |
484 | + This will setup the timeout to 60 minutes. |
485 | + |
486 | +17. (Optional) Setup launchpad access for the guests |
487 | + |
488 | + If you need to access launchpad private branches from the guests, you'll |
489 | + need to setup ssh launchpad access (if you only need access to public |
490 | + branches, http is good enough and you don't even need to 'bzr |
491 | + launchpad-login'): |
492 | + |
493 | + - You need to create an ssh key dedicated to the guests, it has to be |
494 | + passwordless and the public part uploaded to your launchpad profile. |
495 | + You can generate a new key pair (replacing <user> with your launchpad |
496 | + id) with: |
497 | + |
498 | + $ (cd ~/.ssh ; ssh-keygen -f <user>@setup_vm -N '' -C '<user>@setup_vm') |
499 | + |
500 | + This will create two files: '<user>@setup_vm' and |
501 | + '<user>@setup_vm.pub' in your ~/.ssh directory. |
502 | + |
503 | + Upload the later at https://launchpad.net/~/+editsshkeys |
504 | + |
505 | + The keys are created in your .ssh directory so you can test that they |
506 | + work against launchpad without involving a vm. |
507 | + |
508 | + Note that if you create vms from different hosts, you'll need to either |
509 | + copy the same keys on all the hosts or create a pair on each of them |
510 | + (or any combination as long as the public keys are uploaded to |
511 | + launchpad ;). |
512 | + |
513 | + - You need to set vm.launchpad_id to <user>. This will trigger running |
514 | + 'bzr launchpad-login <user>' in the guest and copy |
515 | + ~/.ssh/<user>@setup_vm (the private key) from your host to the guest. |
516 | + |
517 | + - The bazaar.launchpad.net host ssh key needs to be known or you'll get |
518 | + prompted to add it (which is not nice for scripts). This can be fixed |
519 | + by issuing the following command from the {vm.ubuntu_script}: |
520 | + |
521 | + $ ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts |
522 | + |
523 | + This will probably be automated at some point in the future. |
524 | + |
525 | |
526 | === added file 'setup_vm/TODO' |
527 | --- setup_vm/TODO 1970-01-01 00:00:00 +0000 |
528 | +++ setup_vm/TODO 2013-04-17 01:29:27 +0000 |
529 | @@ -0,0 +1,129 @@ |
530 | +* running setup_vm --install I-dont-exist raises an obscure error. Checking |
531 | + that the config section exist for the vm would allow reporting a better |
532 | + error. |
533 | + |
534 | +* Too many tests become too hard to write because their execution requires a |
535 | + real vm. This can be addressed in much the same way than |
536 | + requires_known_reference_image(), i.e. setup a real vm once (outside of |
537 | + selftest execution for now) and then use throw-away vms. But even that |
538 | + may be too costly and may need to wait for lxc/chroot support. |
539 | + |
540 | +* running --ssh-keygen twice gives an awful error message. |
541 | + |
542 | +* Find whether or not we should really support chroots or if lxcs are good |
543 | + enough (roughly: if they can be set up as fast as chroots but provide |
544 | + more features, just optimize the backing-on scenario). |
545 | + |
546 | +* Investigate btrfs support to use snapshots for nested backing. |
547 | + This may not be appropriate with kvms but will surely shine for |
548 | + lxc/chroot. |
549 | + |
550 | +* copying a file (with expanded options or not) from the host to the guest |
551 | + is hard (even internally). There should be a way to more simply describe |
552 | + a list of file/directories to install (with user:group and chmod bits). |
553 | + |
554 | +* Alternatively, we can allow the guest to access the host via ssh (after |
555 | + all, we're installing a private key in the guest so we trust it enough |
556 | + for that already). |
557 | + |
558 | +* As a first step, we can define vm.scripts as a list of relative paths on |
559 | + the host, that will be option expanded into vm.config_dir and uploaded |
560 | + from there into ~ubuntu/bin. |
561 | + |
562 | +* Provide the guest with config file containing the values used to build |
563 | + this vm. From there, the guest itself would be able to expanded options |
564 | + in files acquired from the host (including files modified after the vm |
565 | + has been built/started which will help during dev/debug). |
566 | + |
567 | +* vm.ubuntu_script is kind of an implementation leak from cloud-init, that's |
568 | + the default user there and comes with some nice properties but strictly |
569 | + speaking setup_vm cares about having *a* user, no matter how it is named |
570 | + so the option could be named vm.user_script. In any case, the features we |
571 | + rely on from using ubuntu should be tested if only to document them. |
572 | + |
573 | +* we need a way to run scripts on the host while expanding the config option |
574 | + for a given vm (see sso/test that needs at least the sso.url). |
575 | + |
576 | +* launchpad interaction requires the launchpad host key. |
577 | + |
578 | + => ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts does the |
579 | + trick but it would be nice to automate it. |
580 | + |
581 | +* document the /etc/avahi/ fix required to use avahi with vms |
582 | + |
583 | +* lag times between significant hosts should be collected (not specific to |
584 | + setup_vm but related to the use of the vms). |
585 | + |
586 | +* Add a way to display a vm configuration Ă la 'bzr config' |
587 | + |
588 | +* from the addresses below, find a way to test if some fixed subspace can be |
589 | + safely used (de:ad:be:ef or something...) or just steal some unused MAC |
590 | + prefix (vbox's one ? or vmware's one ? or... the sky is the limit ;) |
591 | + |
592 | + |
593 | +$ sudo grep -n 'mac address' /etc/libvirt/qemu/*.xml |
594 | +/etc/libvirt/qemu/essex-precise.xml:45: <mac address='52:54:00:26:3c:20'/> |
595 | +/etc/libvirt/qemu/freebsd8.xml:39: <mac address='08:00:27:5f:9f:06'/> |
596 | +/etc/libvirt/qemu/gentoo.xml:39: <mac address='08:00:27:da:65:cd'/> |
597 | +/etc/libvirt/qemu/indicator-sync.xml:45: <mac address='52:54:00:43:15:9c'/> |
598 | +/etc/libvirt/qemu/pkgimporter-lucid.xml:45: <mac address='52:54:00:95:4e:dc'/> |
599 | +/etc/libvirt/qemu/precise-cloud.xml:48: <mac address='52:54:00:68:aa:af'/> |
600 | +/etc/libvirt/qemu/precise-pristine.xml:48: <mac address='52:54:00:d5:52:06'/> |
601 | +/etc/libvirt/qemu/precise-pristine.xml:48: <mac address='52:54:00:70:a3:64'/> |
602 | +/etc/libvirt/qemu/precise-server-pristine.xml:51: <mac address='52:54:00:14:d5:be'/> |
603 | +/etc/libvirt/qemu/precise-test.xml:48: <mac address='52:54:00:25:8b:56'/> |
604 | +/etc/libvirt/qemu/quantal-cloud.xml:48: <mac address='52:54:00:e6:6a:df'/> |
605 | +/etc/libvirt/qemu/quantal-pristine.xml:48: <mac address='52:54:00:f5:95:0e'/> |
606 | +/etc/libvirt/qemu/quantal-pristine.xml:48: <mac address='52:54:00:b9:68:11'/> |
607 | +/etc/libvirt/qemu/quantal-test.xml:48: <mac address='52:54:00:21:3b:7b'/> |
608 | +/etc/libvirt/qemu/raring-current.xml <mac address='52:54:00:d9:ca:70'/> |
609 | +/etc/libvirt/qemu/raring-in-dash-pristine.xml:51: <mac address='52:54:00:c7:f9:ee'/> |
610 | +/etc/libvirt/qemu/raring-in-dash-test.xml:51: <mac address='52:54:00:a5:83:b9'/> |
611 | +/etc/libvirt/qemu/raring-pristine.xml:48: <mac address='52:54:00:07:09:cb'/> |
612 | +/etc/libvirt/qemu/raring-pristine.xml:48: <mac address='52:54:00:34:7f:62'/> |
613 | +/etc/libvirt/qemu/raring-scope-base.xml:42: <mac address='52:54:00:9c:0d:1c'/> |
614 | +/etc/libvirt/qemu/raring-scope-test.xml:42: <mac address='52:54:00:8a:b1:d6'/> |
615 | +/etc/libvirt/qemu/raring-test.xml:48: <mac address='52:54:00:23:b6:8d'/> |
616 | +/etc/libvirt/qemu/raring-test.xml:51: <mac address='52:54:00:e3:05:db'/> |
617 | +/etc/libvirt/qemu/sso.xml:51: <mac address='52:54:00:f1:88:84'/> |
618 | +/etc/libvirt/qemu/u1test-quantal.xml:48: <mac address='52:54:00:04:7b:45'/> |
619 | +/etc/libvirt/qemu/u1test-quantal.xml:48: <mac address='52:54:00:04:7b:45'/> |
620 | +/etc/libvirt/qemu/u1test2-quantal.xml:45: <mac address='52:54:00:df:35:6c'/> |
621 | +/etc/libvirt/qemu/u1test2-quantal.xml:45: <mac address='52:54:00:df:35:6c'/> |
622 | +/etc/libvirt/qemu/xp-32bits.xml:58: <mac address='52:54:00:86:aa:99'/> |
623 | +/etc/libvirt/qemu/xp64bits.xml:46: <mac address='52:54:00:0a:5e:7c'/> |
624 | + |
625 | +* while using fixed IP addresses is one way to address known_hosts |
626 | + stability, another way is to rely on `ssh-keygen -R` when installing the |
627 | + host (to remove the previous mapping between the key and the ip) and then |
628 | + add the new key with ssh-keyscan (or something else that doesn't require |
629 | + the guest to be up and running). |
630 | + |
631 | +* make actions verbose and obey -q (at least for tests) so we get some |
632 | + feeback about what is executed. |
633 | + |
634 | +* add a --delete action to make sure we clean up the config_dir |
635 | + |
636 | +* rework FileMonitor, ConsoleMonitor design, the actual result smells (pass |
637 | + in _wait_for_install_with_seed, really ?). |
638 | + |
639 | +* investigate using an upstart job like MAAS: |
640 | + |
641 | + write_poweroff_job() { |
642 | + cat >/etc/init/maas-poweroff.conf <<EOF |
643 | + description "poweroff when maas task is done" |
644 | + start on stopped cloud-final |
645 | + console output |
646 | + task |
647 | + script |
648 | + [ ! -e /tmp/block-poweroff ] || exit 0 |
649 | + poweroff |
650 | + end script |
651 | +EOF |
652 | + # reload required due to lack of inotify in overlayfs (LP: #882147) |
653 | + initctl reload-configuration |
654 | +} |
655 | + |
656 | +* look at PXE, interesting read may include: |
657 | + |
658 | + http://ubuntuforums.org/archive/index.php/t-1713845.html |
659 | |
660 | === added directory 'setup_vm/bin' |
661 | === added file 'setup_vm/bin/__init__.py' |
662 | === added file 'setup_vm/bin/setup_vm.py' |
663 | --- setup_vm/bin/setup_vm.py 1970-01-01 00:00:00 +0000 |
664 | +++ setup_vm/bin/setup_vm.py 2013-04-17 01:29:27 +0000 |
665 | @@ -0,0 +1,1203 @@ |
666 | +#!/usr/bin/env python |
667 | +""" |
668 | +Setup a virtual machine from a config file. |
669 | + |
670 | +Note: Most of the operations requires root access and this script uses ``sudo`` |
671 | +to get them. |
672 | + |
673 | +""" |
674 | +import argparse |
675 | +import base64 |
676 | +from cStringIO import StringIO |
677 | +import errno |
678 | +import os |
679 | +import subprocess |
680 | +import sys |
681 | +import tempfile |
682 | +import time |
683 | + |
684 | + |
685 | +from bzrlib import ( |
686 | + config, |
687 | + osutils, |
688 | + transport, |
689 | + urlutils, |
690 | + ) |
691 | +import yaml |
692 | + |
693 | +# Work around a bug in bzrlib.config forbidding some constructs in templates. |
694 | +# Namely, spaces are invalid as an identifier and therefore should not match |
695 | +# below. |
696 | +config._option_ref_re = config.lazy_regex.lazy_compile('({[^ {},\n]+})') |
697 | + |
698 | + |
699 | +class VmMatcher(config.NameMatcher): |
700 | + |
701 | + def match(self, section): |
702 | + if section.id is None: |
703 | + # The no name section contains default values |
704 | + return True |
705 | + return super(VmMatcher, self).match(section) |
706 | + |
707 | + def get_sections(self): |
708 | + matching_sections = super(VmMatcher, self).get_sections() |
709 | + return reversed(list(matching_sections)) |
710 | + |
711 | + |
712 | +class VmStore(config.LockableIniFileStore): |
713 | + """A config store for options specific to a directory.""" |
714 | + |
715 | + def __init__(self, directory, file_name, possible_transports=None): |
716 | + t = transport.get_transport_from_path( |
717 | + directory, possible_transports=possible_transports) |
718 | + super(VmStore, self).__init__(t, file_name) |
719 | + self.id = 'vm' |
720 | + |
721 | + |
722 | +def system_config_dir(): |
723 | + return '/etc/libvirt' |
724 | + |
725 | + |
726 | +class VmStack(config.Stack): |
727 | + """Per-directory options.""" |
728 | + |
729 | + def __init__(self, name): |
730 | + """Make a new stack for a given vm. |
731 | + |
732 | + The following sections are queried: |
733 | + |
734 | + * the ``name`` section in ./vms.conf, |
735 | + * the no-name section in ./vms.conf |
736 | + * the ``name`` section in ~/vms.conf, |
737 | + * the no-name section in ~/vms.conf |
738 | + * the ``name`` section in /etc/libvirt/vms.conf, |
739 | + * the no-name section in /etc/libvirt/vms.conf |
740 | + |
741 | + :param name: The name of a virtual machine. |
742 | + """ |
743 | + self.local_store = VmStore('.', 'vms.conf') |
744 | + user_store = VmStore(os.environ['HOME'], 'vms.conf') |
745 | + self.system_store = VmStore(system_config_dir(), 'vms.conf') |
746 | + # FIXME: Only available in bzr-2.6b3 :-/ -- vila 2012-01-31 |
747 | + # dstore = self.get_shared_store() |
748 | + super(VmStack, self).__init__( |
749 | + [VmMatcher(self.local_store, name).get_sections, |
750 | + VmMatcher(user_store, name).get_sections, |
751 | + VmMatcher(self.system_store, name).get_sections, |
752 | + ], |
753 | + user_store, mutable_section_id=name) |
754 | + |
755 | + |
756 | +def path_from_unicode(path_string): |
757 | + if not isinstance(path_string, basestring): |
758 | + raise TypeError |
759 | + return os.path.expanduser(path_string) |
760 | + |
761 | + |
762 | +class PathOption(config.Option): |
763 | + |
764 | + def __init__(self, *args, **kwargs): |
765 | + """A path option definition. |
766 | + |
767 | + This possibly expands the user home directory. |
768 | + """ |
769 | + super(PathOption, self).__init__( |
770 | + *args, from_unicode=path_from_unicode, **kwargs) |
771 | + |
772 | + |
773 | +def register(option): |
774 | + config.option_registry.register(option) |
775 | + |
776 | + |
777 | +register(config.Option( |
778 | + 'vm', default=None, |
779 | + help='''The name space defining a virtual machine. |
780 | + |
781 | +This option is a place holder to document the options that defines a virtual |
782 | +machine and the options defining the infrastructure used to manage them all. |
783 | + |
784 | +For qemu based vms, the definition of a vm is stored in an xml file under |
785 | +'/etc/libvirt/qemu/{vm.name}.xml'. This is under the libvirt package control |
786 | +and is out of scope for setup_vm.py. |
787 | + |
788 | +There are 3 other significant files used for a given vm: |
789 | + |
790 | +- a disk image mounted at '/' from '/dev/sda1': |
791 | + '{vm.images_dir}/{vm.name}.qcow2' |
792 | + |
793 | +- a iso image available from '/dev/sdb' labeled 'cidata': |
794 | + {vm.images_dir}/{vm.name}.seed which contains the cloud-init data used to |
795 | + configure/install/update the vm. |
796 | + |
797 | +- a console: {vm.images_dir}/{vm.name}.console which can be 'tail -f'ed from |
798 | + the host. |
799 | + |
800 | +The data used to create the seed above are stored in a vm specific |
801 | +configuration directory for easier debug and reference: |
802 | +- {vm.config_dir}/user-data |
803 | +- {vm.config_dir}/meta-data |
804 | +- {vm.config_dir}/ecdsa |
805 | +- {vm.config_dir}/ecdsa.pub |
806 | +''')) |
807 | + |
808 | +# The directory where we store vm files related to their configuration with |
809 | +# cloud-init (user-data, meta-data, ssh keys). |
810 | +register(config.Option( |
811 | + 'vm.vms_dir', default='~/.config/setup_vm', |
812 | + help='''Where vm related config files are stored. |
813 | + |
814 | +This includes user-data and meta-data for cloud-init and ssh server keys. |
815 | + |
816 | +This directory must exist. |
817 | + |
818 | +Each vm get a specific directory (automatically created) there based on its |
819 | +name. |
820 | +''')) |
821 | +# The base directories where vms are stored for kvm |
822 | +register(PathOption( |
823 | + 'vm.images_dir', default='/var/lib/libvirt/images', |
824 | + help="Where vm disk images are stored.", |
825 | + )) |
826 | +register(config.Option( |
827 | + 'vm.qemu_etc_dir', |
828 | + default='/etc/libvirt/qemu', |
829 | + help="Where libvirt (qemu) stores the vms config files." |
830 | + )) |
831 | + |
832 | +# Isos and images download handling |
833 | +register(config.Option( |
834 | + 'vm.iso_url', |
835 | + default='http://cdimage.ubuntu.com/daily-live/current/' , |
836 | + help="Where an iso can be downloaded from." |
837 | + )) |
838 | +register(config.Option( |
839 | + 'vm.iso_name', |
840 | + default='{vm.release}-desktop-{vm.cpu_model}.iso', |
841 | + help="The name of the iso." |
842 | + )) |
843 | +register(config.Option( |
844 | + 'vm.cloud_image_url', |
845 | + default='http://cloud-images.ubuntu.com/{vm.release}/current/', |
846 | + help="Where a cloud image can be downloaded from." |
847 | + )) |
848 | +register(config.Option( |
849 | + 'vm.cloud_image_name', |
850 | + default='{vm.release}-server-cloudimg-{vm.cpu_model}-disk1.img', |
851 | + help="The name of the cloud image." |
852 | + )) |
853 | +register(PathOption( |
854 | + 'vm.download_cache', |
855 | + default='{vm.images_dir}', |
856 | + help="Where downloads end up.", |
857 | + )) |
858 | + |
859 | +# The ubiquitous vm name |
860 | +register(config.Option( |
861 | + 'vm.name', default=None, invalid='error', |
862 | + help="The vm name, used as a prefix for related files." |
863 | + )) |
864 | +# The second most important bit to define a vm: which ubuntu release ? |
865 | +register(config.Option( |
866 | + 'vm.release', default=None, invalid='error', |
867 | + help="The ubuntu release name." |
868 | + )) |
869 | +# The third important piece to define a vm: where to store files like the |
870 | +# console, the user-data and meta-data files, the ssh server keys, etc. |
871 | +register(config.Option( |
872 | + 'vm.config_dir', default='{vm.vms_dir}/{vm.name}', |
873 | + invalid='error', |
874 | + help='''The directory where files specific to a vm are stored. |
875 | + |
876 | +This includes the user-data and meta-data files used at install time (for |
877 | +reference and easier debug) as well as the optional ssh server keys. |
878 | + |
879 | +By default this is {vm.vms_dir}/{vm.name}. You can put it somewhere else by |
880 | +redifining it as long as it ends up being unique for the vm. |
881 | + |
882 | +{vm.vms_dir}/{vm.release}/{vm.name} may better suit your taste for example. |
883 | +''' |
884 | + )) |
885 | +# The options defining the vm physical characteristics |
886 | +register(config.Option( |
887 | + 'vm.ram_size', default='1024', |
888 | + help="The ram size in megabytes." |
889 | + )) |
890 | +register(config.Option( |
891 | + 'vm.disk_size', default='8G', |
892 | + help='''The disk image size in bytes. |
893 | + |
894 | +Optional suffixes "k" or "K" (kilobyte, 1024) "M" (megabyte, 1024k) "G" |
895 | +(gigabyte, 1024M) and T (terabyte, 1024G) are supported. |
896 | +''')) |
897 | +register(config.Option( |
898 | + 'vm.cpus', default='1', |
899 | + help="The number of cpus." |
900 | + )) |
901 | +register(config.Option( |
902 | + 'vm.cpu_model', default=None, invalid='error', |
903 | + help="The number of cpus.")) |
904 | +register(config.Option( |
905 | + 'vm.network', default='network=default', invalid='error', |
906 | + help="""The --network parameter for virt-install. |
907 | + |
908 | +This can be specialized for each machine but the default should work in most |
909 | +setups. Watch for your DHCP server exhausting its address space if you create a |
910 | +lot of vms with random MAC addresses. |
911 | +""")) |
912 | + |
913 | +register(config.Option( |
914 | + 'vm.meta_data', default='''\ |
915 | +instance-id: {vm.name} |
916 | +local-hostname: {vm.name} |
917 | +''', |
918 | + invalid='error', |
919 | + help="The meta data for cloud-init to put in the seed." |
920 | + )) |
921 | + |
922 | +# Some bits that may added to user-data but are optional |
923 | + |
924 | +register(config.ListOption( |
925 | + 'vm.packages', default=None, |
926 | + help='''A list of package names to be installed. |
927 | +''')) |
928 | +register(config.Option( |
929 | + 'vm.apt_proxy', default=None, invalid='error', |
930 | + help='''A local proxy for apt to avoid repeated .deb downloads. |
931 | + |
932 | +Example: |
933 | + |
934 | + vm.apt_proxy = http://192.168.0.42:8000 |
935 | + |
936 | +''')) |
937 | +register(config.ListOption( |
938 | + 'vm.apt_sources', default=None, |
939 | + help='''A list of apt sources entries to be added to the default ones. |
940 | + |
941 | +Cloud-init already setup /etc/apt/sources.list with appropriate entries. Only |
942 | +additional entries need to be specified here. |
943 | +''')) |
944 | +register(config.ListOption( |
945 | + 'vm.ssh_authorized_keys', default=None, |
946 | + help='A list of paths to public ssh keys to be authorized for' |
947 | + ' the default user.')) |
948 | +register(config.ListOption( |
949 | + 'vm.ssh_keys', default=None, |
950 | + help='''A list of paths to server ssh keys. |
951 | + |
952 | +Both public and private keys can be provided. Accepted ssh key types are rsa, |
953 | +dsa and ecdsa. The file names should match <type>.*[.pub]. |
954 | +''')) |
955 | +register(config.Option( |
956 | + 'vm.update', default=False, |
957 | + from_unicode=config.bool_from_store, |
958 | + help='''Whether or not the vm should be updated. |
959 | +Both apt-get update and apt-get upgrade are called if this option is set. |
960 | +''')) |
961 | +register(config.Option( |
962 | + 'vm.password', default='ubuntu', invalid='error', |
963 | + help="The ubuntu user password." |
964 | + )) |
965 | +register(config.Option( |
966 | + 'vm.launchpad_id', |
967 | + help="The launchpad login used for launchpad ssh access from the guest." |
968 | + )) |
969 | +# The scripts that are executed before powering off |
970 | +register(PathOption( |
971 | + 'vm.root_script', default=None, |
972 | + help='''The path to a script executed as root before powering off. |
973 | + |
974 | +This script is executed before {vm.ubuntu_script}. |
975 | +''' |
976 | + )) |
977 | +register(PathOption( |
978 | + 'vm.ubuntu_script', default=None, |
979 | + help='''The path to a script executed as ubuntu before powering off. |
980 | + |
981 | +This script is excuted after {vm.root_script}. |
982 | +''')) |
983 | +register(config.ListOption( |
984 | + 'vm.uploaded_scripts', default=None, |
985 | + help='''A list of scripts to be uploaded to the guest. |
986 | + |
987 | +Scripts can use config options from their vm, they will be expanded before |
988 | +upload. All scripts are uploaded into {vm.uploaded_scripts.guest_dir} under |
989 | +their base name. |
990 | +''')) |
991 | +register(config.Option( |
992 | + 'vm.uploaded_scripts.guest_dir', default='~ubuntu/bin', |
993 | + help='''Where {vm.uploaded_scripts} are uploaded on the guest.''' |
994 | + )) |
995 | + |
996 | + |
997 | +class SetupVmError(Exception): |
998 | + |
999 | + msg = 'setup_vm Generic Error: %r' |
1000 | + |
1001 | + def __init__(self, msg=None, **kwds): |
1002 | + if msg is not None: |
1003 | + self.msg = msg |
1004 | + for key, value in kwds.items(): |
1005 | + setattr(self, key, value) |
1006 | + |
1007 | + def __str__(self): |
1008 | + return self.msg.format((), **self.__dict__) |
1009 | + |
1010 | + __repr__ = __str__ |
1011 | + |
1012 | + |
1013 | +class CommandError(SetupVmError): |
1014 | + |
1015 | + msg = ''' |
1016 | + command: {joined_cmd} |
1017 | + retcode: {retcode} |
1018 | + output: {out} |
1019 | + error: {err} |
1020 | +''' |
1021 | + |
1022 | + def __init__(self, cmd, retcode, out, err): |
1023 | + super(CommandError, self).__init__(joined_cmd=' '.join(cmd), |
1024 | + retcode=retcode, err=err, out=out) |
1025 | + |
1026 | +class ConfigValueError(SetupVmError): |
1027 | + |
1028 | + msg = 'Bad value "{value}" for option "{name}".' |
1029 | + |
1030 | + def __init__(self, name, value): |
1031 | + super(ConfigValueError, self).__init__(name=name, value=value) |
1032 | + |
1033 | + |
1034 | +class ConfigPathNotFound(SetupVmError): |
1035 | + |
1036 | + msg = 'No such file: {path} from {name}' |
1037 | + |
1038 | + def __init__(self, path, name): |
1039 | + super(ConfigPathNotFound, self).__init__(path=path, name=name) |
1040 | + |
1041 | + |
1042 | +def run_subprocess(args): |
1043 | + proc = subprocess.Popen(args, |
1044 | + stdout=subprocess.PIPE, |
1045 | + stderr=subprocess.PIPE, |
1046 | + stdin=subprocess.PIPE) |
1047 | + out, err = proc.communicate() |
1048 | + if proc.returncode: |
1049 | + raise CommandError(args, proc.returncode, out, err) |
1050 | + return proc.returncode, out, err |
1051 | + |
1052 | + |
1053 | +def pipe_subprocess(args): |
1054 | + proc = subprocess.Popen(args, |
1055 | + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) |
1056 | + return proc |
1057 | + |
1058 | +def ssh_infos_from_path(key_path): |
1059 | + """Analyze path to find ssh key type and kind. |
1060 | + |
1061 | + The basename should begin with ssh type used to create the key. and end |
1062 | + with '.pub' for a public key. |
1063 | + |
1064 | + If the type is neither of rds, dsa or ecdsa, None if returned. |
1065 | + |
1066 | + :param key_path: A path to an ssh key. |
1067 | + |
1068 | + :return: (type, kind) 'type' is the ssh key type or None if neither of rds, |
1069 | + dsa or ecdsa. 'kind' is 'public' if the path ends with '.pub', |
1070 | + 'private' otherwise. |
1071 | + """ |
1072 | + base = os.path.basename(key_path) |
1073 | + for p in ('rsa', 'dsa', 'ecdsa'): |
1074 | + if base.startswith(p): |
1075 | + ssh_type = p |
1076 | + break |
1077 | + else: |
1078 | + ssh_type = None |
1079 | + if base.endswith('.pub'): |
1080 | + kind = 'public' |
1081 | + else: |
1082 | + kind = 'private' |
1083 | + return ssh_type, kind |
1084 | + |
1085 | + |
1086 | +class ConsoleEOFError(SetupVmError): |
1087 | + |
1088 | + msg = 'Encountered EOF on console, something went wrong' |
1089 | + |
1090 | + |
1091 | +class CloudInitError(SetupVmError): |
1092 | + |
1093 | + msg = 'cloud-init reported: {line} check your config' |
1094 | + |
1095 | + def __init__(self, line): |
1096 | + super(CloudInitError, self).__init__(line=line) |
1097 | + |
1098 | + |
1099 | +class ConsoleMonitor(object): |
1100 | + """Monitor a console to identify known events.""" |
1101 | + |
1102 | + def __init__(self, stream): |
1103 | + super(ConsoleMonitor, self).__init__() |
1104 | + self.stream = stream |
1105 | + |
1106 | + |
1107 | + def parse(self): |
1108 | + while True: |
1109 | + line = self.stream.readline() |
1110 | + yield line |
1111 | + if not line: |
1112 | + raise ConsoleEOFError() |
1113 | + elif line.startswith(' * Will now halt'): |
1114 | + # That's our final_message, we're done |
1115 | + return |
1116 | + elif ('Failed loading yaml blob' in line |
1117 | + or 'Unhandled non-multipart userdata starting' in line |
1118 | + or 'failed to render string to stdout:' in line |
1119 | + or 'Failed loading of cloud config' in line): |
1120 | + raise CloudInitError(line) |
1121 | + |
1122 | + |
1123 | +class FileMonitor(ConsoleMonitor): |
1124 | + |
1125 | + def __init__(self, path): |
1126 | + cmd = ['tail', '-f', path] |
1127 | + proc = pipe_subprocess(cmd) |
1128 | + super(FileMonitor, self).__init__(proc.stdout) |
1129 | + self.path = path |
1130 | + self.cmd = cmd |
1131 | + self.proc = proc |
1132 | + self.lines = [] |
1133 | + |
1134 | + def parse(self): |
1135 | + try: |
1136 | + for line in super(FileMonitor, self).parse(): |
1137 | + self.lines.append(line) |
1138 | + yield line |
1139 | + finally: |
1140 | + self.proc.terminate() |
1141 | + |
1142 | + |
1143 | +class CIUserData(object): |
1144 | + """Maps configuration data into cloud-init user-data. |
1145 | + |
1146 | + This is a container for the data that will ultimately be encoded into a |
1147 | + cloud-config-archive user-data file. |
1148 | + """ |
1149 | + |
1150 | + def __init__(self, conf): |
1151 | + super(CIUserData, self).__init__() |
1152 | + self.conf = conf |
1153 | + # The objects we will populate before creating a yaml encoding as a |
1154 | + # cloud-config-archive file |
1155 | + self.cloud_config = {} |
1156 | + self.root_hook = None |
1157 | + self.ubuntu_hook = None |
1158 | + self.launchpad_hook = None |
1159 | + self.uploaded_scripts_hook = None |
1160 | + |
1161 | + def set(self, ud_name, conf_name=None, value=None): |
1162 | + """Set a user-data option from it's corresponding configuration one. |
1163 | + |
1164 | + :param ud_name: user-data key. |
1165 | + |
1166 | + :param conf_name: configuration key, If set to None, `value` should be |
1167 | + provided. |
1168 | + |
1169 | + :param value: value to use if `conf_name` is None. |
1170 | + """ |
1171 | + if value is None and conf_name is not None: |
1172 | + value = self.conf.get(conf_name) |
1173 | + if value is not None: |
1174 | + self.cloud_config[ud_name] = value |
1175 | + |
1176 | + def _file_content(self, path, option_name): |
1177 | + full_path = os.path.expanduser(path) |
1178 | + try: |
1179 | + with open(full_path) as f: |
1180 | + content = f.read() |
1181 | + except IOError, e: |
1182 | + if e.args[0] == errno.ENOENT: |
1183 | + raise ConfigPathNotFound(path, option_name) |
1184 | + else: |
1185 | + raise |
1186 | + return content |
1187 | + |
1188 | + def set_list_of_paths(self, ud_name, conf_name): |
1189 | + """Set a user-data option from its corresponding configuration one. |
1190 | + |
1191 | + The configuration option is a list of paths and the user-data option |
1192 | + will be a list of each file content. |
1193 | + |
1194 | + :param ud_name: user-data key. |
1195 | + |
1196 | + :param conf_name: configuration key. |
1197 | + """ |
1198 | + paths = self.conf.get(conf_name) |
1199 | + if paths: |
1200 | + contents = [] |
1201 | + for p in paths: |
1202 | + contents.append(self._file_content(p, conf_name)) |
1203 | + self.set(ud_name, None, contents) |
1204 | + |
1205 | + def _key_from_path(self, path, option_name): |
1206 | + """Infer user-data key from file name.""" |
1207 | + ssh_type, kind = ssh_infos_from_path(path) |
1208 | + if ssh_type is None: |
1209 | + raise ConfigValueError(option_name, path) |
1210 | + return '%s_%s' % (ssh_type, kind) |
1211 | + |
1212 | + def set_ssh_keys(self): |
1213 | + """Set the server ssh keys from a list of paths. |
1214 | + |
1215 | + Provided paths should respect some coding: |
1216 | + |
1217 | + - the base name should start with the ssh type of their key (rsa, dsa, |
1218 | + ecdsa), |
1219 | + |
1220 | + - base names ending with '.pub' are for public keys, the others are for |
1221 | + private keys. |
1222 | + """ |
1223 | + key_paths = self.conf.get('vm.ssh_keys') |
1224 | + if key_paths: |
1225 | + ssh_keys = {} |
1226 | + for p in key_paths: |
1227 | + key = self._key_from_path(p, 'vm.ssh_keys') |
1228 | + ssh_keys[key] = self._file_content(p, 'vm.ssh_keys') |
1229 | + self.set('ssh_keys', None, ssh_keys) |
1230 | + |
1231 | + def set_apt_sources(self): |
1232 | + sources = self.conf.get('vm.apt_sources') |
1233 | + if sources: |
1234 | + apt_sources = [] |
1235 | + for src in sources: |
1236 | + # '|' should not appear in urls nor keys so it should be safe |
1237 | + # to use it as a separator. |
1238 | + parts = src.split('|') |
1239 | + if len(parts) == 1: |
1240 | + apt_sources.append({'source': parts[0]}) |
1241 | + else: |
1242 | + # For PPAs, an additional GPG key should be imported in the |
1243 | + # guest. |
1244 | + apt_sources.append({'source': parts[0], 'keyid': parts[1]}) |
1245 | + self.cloud_config['apt_sources'] = apt_sources |
1246 | + |
1247 | + def append_cmd(self, cmd): |
1248 | + cmds = self.cloud_config.get('runcmd', []) |
1249 | + cmds.append(cmd) |
1250 | + self.cloud_config['runcmd'] = cmds |
1251 | + |
1252 | + def _hook_script_path(self, user): |
1253 | + return '~%s/setup_vm_post_install' % (user,) |
1254 | + |
1255 | + def _hook_content(self, option_name, user, hook_path, mode='0700'): |
1256 | + # FIXME: Add more tests towards properly creating a tree on the guest |
1257 | + # from a tree on the host. There seems to be three kind of items worth |
1258 | + # caring about here: file content (output path, owner, chmod), file |
1259 | + # (input and output paths, owner, chmod) and directory (path, owner, |
1260 | + # chmod). There are also some subtle traps involved about when files |
1261 | + # are created across various vm generations (one vm creates a dir, a mv |
1262 | + # on top of that one doesn't, but still creates a file in this dir, |
1263 | + # without realizing it can fail in a fresh vm). -- vila 2013-03-10 |
1264 | + host_path = self.conf.get(option_name) |
1265 | + if host_path is None: |
1266 | + return None |
1267 | + fcontent = self._file_content(host_path, option_name) |
1268 | + # Expand options in the provided content so we report better errors |
1269 | + expanded_content = self.conf.expand_options(fcontent) |
1270 | + # The following will generate an additional newline at the end of the |
1271 | + # script. I can't think of a case where it matters and it makes this |
1272 | + # code more robust (and/or simpler) if the script/file *doesn't* end up |
1273 | + # with a proper newline. |
1274 | + # FIXME: This may be worth fixing if we provide a more generic way to |
1275 | + # create a remote tree. -- vila 2013-03-10 |
1276 | + hook_content = '''#!/bin/sh |
1277 | +cat >{__guest_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
1278 | +{__fcontent} |
1279 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
1280 | +chown {__user}:{__user} {__guest_path} |
1281 | +chmod {__mode} {__guest_path} |
1282 | +''' |
1283 | + return hook_content.format(__user=user, __fcontent=expanded_content, |
1284 | + __mode=mode, |
1285 | + __guest_path=hook_path) |
1286 | + |
1287 | + def set_boot_hook(self): |
1288 | + # FIXME: Needs a test ensuring we execute as root -- vila 2013-03-07 |
1289 | + hook_path = self._hook_script_path('root') |
1290 | + content = self._hook_content('vm.root_script', 'root', hook_path) |
1291 | + if content is not None: |
1292 | + self.root_hook = content |
1293 | + self.append_cmd(hook_path) |
1294 | + |
1295 | + def set_ubuntu_hook(self): |
1296 | + # FIXME: Needs a test ensuring we execute as ubuntu -- vila 2013-03-07 |
1297 | + hook_path = self._hook_script_path('ubuntu') |
1298 | + content = self._hook_content('vm.ubuntu_script', 'ubuntu', hook_path) |
1299 | + if content is not None: |
1300 | + self.ubuntu_hook = content |
1301 | + self.append_cmd(['su', '-l', '-c', hook_path, 'ubuntu']) |
1302 | + |
1303 | + def set_launchpad_access(self): |
1304 | + # FIXME: Needs a test that we can really access launchpad properly via |
1305 | + # ssh. Can only be done as a real launchpad user and as such requires |
1306 | + # cooperation :) I.e. Some configuration option set by the user will |
1307 | + # trigger the test -- vila 2013-03-14 |
1308 | + lp_id = self.conf.get('vm.launchpad_id') |
1309 | + if lp_id is None: |
1310 | + return |
1311 | + # Use the specified ssh key found in ~/.ssh as the private key. The |
1312 | + # user is supposed to have uploaded the public one. |
1313 | + local_path = os.path.join('~', '.ssh', '%s@setup_vm' % (lp_id,)) |
1314 | + # Force id_rsa or we'll need a .ssh/config to point to user@setup_vm |
1315 | + # for .lauchpad.net. |
1316 | + hook_path = '/home/ubuntu/.ssh/id_rsa' |
1317 | + dir_path = os.path.dirname(hook_path) |
1318 | + try: |
1319 | + fcontent = self._file_content(local_path, 'vm.launchpad_id') |
1320 | + except ConfigPathNotFound, e: |
1321 | + e.msg = ('You need to create the {p} keypair and upload {p}.pub to' |
1322 | + ' launchpad.\n' |
1323 | + 'See vm.launchpad_id in README.'.format(p=local_path)) |
1324 | + raise e |
1325 | + # FIXME: ~Duplicated from _hook_content. -- vila 2013-03-10 |
1326 | + |
1327 | + # FIXME: If this hook is executed before the ubuntu user is created we |
1328 | + # need to chown/chmod ~ubuntu which is bad. This happens when a |
1329 | + # -pristine vm is created and lead to GUI login failing because it |
1330 | + # can't create any dir/file there. The fix is to only create a script |
1331 | + # that will be executed via runcmd so it will run later and avoid the |
1332 | + # issue. -- vila 2013-03-21 |
1333 | + hook_content = '''#!/bin/sh |
1334 | +mkdir -p {dir_path} |
1335 | +chown {user}:{user} ~ubuntu |
1336 | +chmod {dir_mode} ~ubuntu |
1337 | +chown {user}:{user} {dir_path} |
1338 | +chmod {dir_mode} {dir_path} |
1339 | +cat >{guest_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
1340 | +{fcontent} |
1341 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
1342 | +chown {user}:{user} {guest_path} |
1343 | +chmod {file_mode} {guest_path} |
1344 | +''' |
1345 | + self.launchpad_hook = self.conf.expand_options( |
1346 | + hook_content, |
1347 | + env=dict(user='ubuntu', fcontent=fcontent, |
1348 | + file_mode='0400', guest_path=hook_path, |
1349 | + dir_mode='0700', dir_path=dir_path)) |
1350 | + self.append_cmd(['sudo', '-u', 'ubuntu', |
1351 | + 'bzr', 'launchpad-login', lp_id]) |
1352 | + |
1353 | + def set_uploaded_scripts(self): |
1354 | + script_paths = self.conf.get('vm.uploaded_scripts') |
1355 | + if not script_paths: |
1356 | + return |
1357 | + hook_path = '~ubuntu/setup_vm_uploads' |
1358 | + bindir = self.conf.get('vm.uploaded_scripts.guest_dir') |
1359 | + out = StringIO() |
1360 | + out.write('''#!/bin/sh |
1361 | +cat >{hook_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
1362 | +mkdir -p {bindir} |
1363 | +cd {bindir} |
1364 | +'''.format(**locals())) |
1365 | + for path in script_paths: |
1366 | + fcontent = self._file_content(path, 'vm.uploaded_scripts') |
1367 | + expanded = self.conf.expand_options(fcontent) |
1368 | + base = os.path.basename(path) |
1369 | + # FIXME: ~Duplicated from _hook_content. -- vila 2012-03-15 |
1370 | + out.write('''cat >{base} <<'EOF{base}' |
1371 | +{expanded} |
1372 | +EOF{base} |
1373 | +chmod 0755 {base} |
1374 | +'''.format(**locals())) |
1375 | + |
1376 | + out.write('''EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
1377 | +chown {user}:{user} {hook_path} |
1378 | +chmod 0700 {hook_path} |
1379 | +'''.format(user='ubuntu',**locals())) |
1380 | + self.uploaded_scripts_hook = out.getvalue() |
1381 | + self.append_cmd(['su', '-l', '-c', hook_path, 'ubuntu']) |
1382 | + |
1383 | + def set_poweroff(self): |
1384 | + # We want to shutdown properly after installing. This is safe to set |
1385 | + # here as subsequent boots will ignore this setting, letting us use the |
1386 | + # vm ;) |
1387 | + if self.conf.get('vm.release') in ('precise', 'quantal'): |
1388 | + # Curse cloud-init lack of compatibility |
1389 | + self.append_cmd('halt') |
1390 | + else: |
1391 | + self.set('power_state', None, {'mode': 'poweroff'}) |
1392 | + |
1393 | + def populate(self): |
1394 | + # Common and non-configurable options |
1395 | + if self.conf.get('vm.release') == 'precise': |
1396 | + # Curse cloud-init lack of compatibility |
1397 | + msg = 'setup_vm finished installing in $UPTIME seconds.' |
1398 | + else: |
1399 | + msg = 'setup_vm finished installing in ${uptime} seconds.' |
1400 | + self.set('final_message', None, msg) |
1401 | + self.set('manage_etc_hosts', None, True) |
1402 | + self.set('chpasswd', None, dict(expire=False)) |
1403 | + # Configurable options |
1404 | + self.set('password', 'vm.password') |
1405 | + self.set_list_of_paths('ssh_authorized_keys', 'vm.ssh_authorized_keys') |
1406 | + self.set_ssh_keys() |
1407 | + self.set('apt_proxy', 'vm.apt_proxy') |
1408 | + # Both user-data keys are set from the same config key, we don't |
1409 | + # provide a finer access. |
1410 | + self.set('apt_update', 'vm.update') |
1411 | + self.set('apt_upgrade', 'vm.update') |
1412 | + self.set_apt_sources() |
1413 | + self.set('packages', 'vm.packages') |
1414 | + self.set_launchpad_access() |
1415 | + # uploaded scripts |
1416 | + self.set_uploaded_scripts() |
1417 | + # The commands executed before powering off |
1418 | + self.set_boot_hook() |
1419 | + self.set_ubuntu_hook() |
1420 | + # This must be called last so previous commands (for precise and |
1421 | + # quantal) can be executed before powering off |
1422 | + self.set_poweroff() |
1423 | + |
1424 | + def add_boot_hook(self, parts, hook): |
1425 | + if hook is not None: |
1426 | + parts.append({'content': '#cloud-boothook\n' + hook}) |
1427 | + |
1428 | + def dump(self): |
1429 | + parts = [{'content': '#cloud-config\n' |
1430 | + + yaml.safe_dump(self.cloud_config)}] |
1431 | + self.add_boot_hook(parts, self.root_hook) |
1432 | + self.add_boot_hook(parts, self.ubuntu_hook) |
1433 | + self.add_boot_hook(parts, self.launchpad_hook) |
1434 | + self.add_boot_hook(parts, self.uploaded_scripts_hook) |
1435 | + # Wrap the lot into a cloud config archive |
1436 | + return '#cloud-config-archive\n' + yaml.safe_dump(parts) |
1437 | + |
1438 | + |
1439 | +def vm_states(source=None): |
1440 | + """A dict of states for vms indexed by name. |
1441 | + |
1442 | + :param source: A list of lines as produced by virsh list --all without |
1443 | + decorations (header/footer). |
1444 | + """ |
1445 | + if source is None: |
1446 | + retcode, out, err = run_subprocess(['virsh', 'list', '--all']) |
1447 | + # Get rid of header/footer |
1448 | + source = out.splitlines()[2:-1] |
1449 | + states = {} |
1450 | + for line in source: |
1451 | + caret_or_id, name, state = line.split(None, 2) |
1452 | + states[name] = state |
1453 | + return states |
1454 | + |
1455 | + |
1456 | +class VM(object): |
1457 | + |
1458 | + def __init__(self, conf): |
1459 | + self.conf = conf |
1460 | + self._config_dir = None |
1461 | + |
1462 | + def ensure_dir(self, path): |
1463 | + try: |
1464 | + os.mkdir(path) |
1465 | + except OSError, e: |
1466 | + # FIXME: Try to create the parent dir ? |
1467 | + if e.errno == errno.EEXIST: |
1468 | + pass |
1469 | + else: |
1470 | + raise |
1471 | + |
1472 | + def ensure_config_dir(self): |
1473 | + if self._config_dir is None: |
1474 | + # FIXME: expanduser is not tested |
1475 | + self._config_dir = os.path.expanduser( |
1476 | + self.conf.get('vm.config_dir')) |
1477 | + self.ensure_dir(self._config_dir) |
1478 | + |
1479 | + def _ssh_keygen(self, key_path): |
1480 | + ssh_type, kind = ssh_infos_from_path(key_path) |
1481 | + path = os.path.expanduser(key_path) # Just in case |
1482 | + if kind == 'private': # public will be generated at the same time |
1483 | + run_subprocess( |
1484 | + ['ssh-keygen', '-f', path, '-N', '', '-t', ssh_type, |
1485 | + '-C', self.conf.get('vm.name')]) |
1486 | + |
1487 | + def ssh_keygen(self): |
1488 | + self.ensure_config_dir() |
1489 | + keys = self.conf.get('vm.ssh_keys') |
1490 | + for key in keys: |
1491 | + self._ssh_keygen(key) |
1492 | + |
1493 | + |
1494 | +class Kvm(VM): |
1495 | + |
1496 | + def __init__(self, conf): |
1497 | + super(Kvm, self).__init__(conf) |
1498 | + # Seed files |
1499 | + self._meta_data_path = None |
1500 | + self._user_data_path = None |
1501 | + # Disk paths |
1502 | + self._seed_path = None |
1503 | + self._disk_image_path = None |
1504 | + |
1505 | + self._console_path = None |
1506 | + |
1507 | + def _download_in_cache(self, source_url, name, force=False): |
1508 | + """Download ``name`` from ``source_url`` in ``vm.download_cache``. |
1509 | + |
1510 | + :param source_url: The url where the file to download is located |
1511 | + |
1512 | + :param name: The name of the file to download (also used as the name |
1513 | + for the downloaded file). |
1514 | + |
1515 | + :param force: Remove the file from the cache if present. |
1516 | + |
1517 | + :return: False if the file is in the download cache, True if a download |
1518 | + occurred. |
1519 | + """ |
1520 | + source = urlutils.join(source_url, name) |
1521 | + download_dir = self.conf.get('vm.download_cache') |
1522 | + if not os.path.exists(download_dir): |
1523 | + raise ConfigValueError('vm.download_cache', download_dir) |
1524 | + target = os.path.join(download_dir, name) |
1525 | + # FIXME: By default the download dir may be under root control, but if |
1526 | + # a user chose to use a different one under his own control, it would |
1527 | + # be nice to not require sudo usage. -- vila 2013-02-06 |
1528 | + if force: |
1529 | + run_subprocess(['sudo', 'rm', '-f', target]) |
1530 | + if not os.path.exists(target): |
1531 | + # FIXME: We do ask for a progress bar but it's not displayed |
1532 | + # (run_subprocess capture both stdout and stderr) ! At least while |
1533 | + # used interactively, it should. -- vila 2013-02-06 |
1534 | + run_subprocess(['sudo', 'wget', '--progress=dot:mega','-O', |
1535 | + target, source]) |
1536 | + return True |
1537 | + else: |
1538 | + return False |
1539 | + |
1540 | + def download_iso(self, force=False): |
1541 | + """Download the iso to install the vm. |
1542 | + |
1543 | + :return: False if the iso is in the download cache, True if a download |
1544 | + occurred. |
1545 | + """ |
1546 | + return self._download_in_cache(self.conf.get('vm.iso_url'), |
1547 | + self.conf.get('vm.iso_name'), |
1548 | + force=force) |
1549 | + |
1550 | + def download_cloud_image(self, force=False): |
1551 | + """Download the cloud image to install the vm. |
1552 | + |
1553 | + :return: False if the image is in the download cache, True if a |
1554 | + download occurred. |
1555 | + """ |
1556 | + return self._download_in_cache(self.conf.get('vm.cloud_image_url'), |
1557 | + self.conf.get('vm.cloud_image_name'), |
1558 | + force=force) |
1559 | + |
1560 | + def create_meta_data(self): |
1561 | + self.ensure_config_dir() |
1562 | + self._meta_data_path = os.path.join(self._config_dir, 'meta-data') |
1563 | + with open(self._meta_data_path, 'w') as f: |
1564 | + f.write(self.conf.get('vm.meta_data')) |
1565 | + |
1566 | + def create_user_data(self): |
1567 | + ci_user_data = CIUserData(self.conf) |
1568 | + ci_user_data.populate() |
1569 | + self.ensure_config_dir() |
1570 | + self._user_data_path = os.path.join(self._config_dir, 'user-data') |
1571 | + with open(self._user_data_path, 'w') as f: |
1572 | + f.write(ci_user_data.dump()) |
1573 | + |
1574 | + def create_seed(self): |
1575 | + if self._meta_data_path is None: |
1576 | + self.create_meta_data() |
1577 | + if self._user_data_path is None: |
1578 | + self.create_user_data() |
1579 | + images_dir = self.conf.get('vm.images_dir') |
1580 | + seed_path = os.path.join( |
1581 | + images_dir, self.conf.expand_options('{vm.name}.seed')) |
1582 | + run_subprocess( |
1583 | + # We create the seed in the ``vm.images_dir`` directory, so |
1584 | + # ``sudo`` is required |
1585 | + ['sudo', |
1586 | + 'genisoimage', '-output', seed_path, '-volid', 'cidata', |
1587 | + '-joliet', '-rock', '-input-charset', 'default', |
1588 | + '-graft-points', |
1589 | + 'user-data=%s' % (self._user_data_path,), |
1590 | + 'meta-data=%s' % (self._meta_data_path,), |
1591 | + ]) |
1592 | + self._seed_path = seed_path |
1593 | + |
1594 | + def create_disk_image(self): |
1595 | + raise NotImplementedError(self.create_disk_image) |
1596 | + |
1597 | + def _wait_for_install_with_seed(self): |
1598 | + # The console is created by virt-install which requires sudo but creates |
1599 | + # the file 0600 for libvirt-qemu. We give read access to all otherwise |
1600 | + # 'tail -f' requires sudo and can't be killed anymore. |
1601 | + run_subprocess(['sudo', 'chmod', '0644', self._console_path]) |
1602 | + # While `virt-install` is running, let's connect to the console |
1603 | + console = FileMonitor(self._console_path) |
1604 | + try: |
1605 | + for line in console.parse(): |
1606 | +# FIXME: We need some way to activate this dynamically (conf var defaulting to |
1607 | +# env var OR cmdline parameter ? -- vila 2013-02-11 |
1608 | +# print "read: [%s]" % (line,) # so useful for debug... |
1609 | + pass |
1610 | + except (ConsoleEOFError, CloudInitError), e: |
1611 | + # FIXME: No test covers this path -- vila 2013-02-15 |
1612 | + err_lines = ['Suspicious line from cloud-init.\n', |
1613 | + '\t' + console.lines[-1], |
1614 | + 'Check the configuration:\n'] |
1615 | + with open(self._meta_data_path) as f: |
1616 | + err_lines.append('meta-data content:\n') |
1617 | + err_lines.extend(f.readlines()) |
1618 | + with open(self._user_data_path) as f: |
1619 | + err_lines.append('user-data content:\n') |
1620 | + err_lines.extend(f.readlines()) |
1621 | + raise CommandError(console.cmd, console.proc.returncode, |
1622 | + '\n'.join(console.lines), |
1623 | + ''.join(err_lines)) |
1624 | + |
1625 | + def install(self): |
1626 | + # Create a kvm, relying on cloud-init to customize the base image. |
1627 | + # |
1628 | + # There are two processes involvded here: |
1629 | + # - virt-install creates the vm and boots it. |
1630 | + # - progress is monitored via the console to detect cloud-final. |
1631 | + # |
1632 | + # Once cloud-init has finished, the vm can be powered off. |
1633 | + |
1634 | + # FIXME: If the install doesn't finish after $time, emit a warning and |
1635 | + # terminate self.install_proc. |
1636 | + # FIXME: If we can't connect to the console, emit a warning and |
1637 | + # terminate console and self.install_proc. |
1638 | + # FIXME: If we don't receive anything on the console after $time2, emit |
1639 | + # a warning and terminate console and self.install_proc. |
1640 | + # -- vila 2013-02-07 |
1641 | + if self._seed_path is None: |
1642 | + self.create_seed() |
1643 | + if self._disk_image_path is None: |
1644 | + self.create_disk_image() |
1645 | + # FIXME: Install time is probably a good time to delete the |
1646 | + # console. While it makes sense to accumulate for all runs for a given |
1647 | + # install, keeping them without any limit nor roration is likely to |
1648 | + # cause issues at some point... -- vila 2013-02-20 |
1649 | + self._console_path = os.path.join( |
1650 | + self.conf.get('vm.images_dir'), |
1651 | + '%s.console' % (self.conf.get('vm.name'),)) |
1652 | + run_subprocess( |
1653 | + ['sudo', 'virt-install', |
1654 | + # To ensure we're not bitten again by http://pad.lv/1157272 where |
1655 | + # virt-install wrongly detect virtualbox. -- vila 2013-03-20 |
1656 | + '--connect', 'qemu:///system', |
1657 | + # Without --noautoconsole, virt-install will relay the console, |
1658 | + # that's not appropriate for our needs so we'll connect later |
1659 | + # ourselves |
1660 | + '--noautoconsole', |
1661 | + # We define the console as a file so we can monitor the install |
1662 | + # via 'tail -f' |
1663 | + '--serial', 'file,path=%s' % (self._console_path,), |
1664 | + '--network', self.conf.get('vm.network'), |
1665 | + # Anticipate that we'll need a graphic card defined |
1666 | + '--graphics', 'spice', |
1667 | + '--name', self.conf.get('vm.name'), |
1668 | + '--ram', self.conf.get('vm.ram_size'), |
1669 | + '--vcpus', self.conf.get('vm.cpus'), |
1670 | + '--disk', 'path=%s,format=qcow2' % (self._disk_image_path,), |
1671 | + '--disk', 'path=%s' % (self._seed_path,), |
1672 | + # We just boot, cloud-init will handle the installs we need |
1673 | + '--boot', 'hd', '--hvm', |
1674 | + ]) |
1675 | + self._wait_for_install_with_seed() |
1676 | + # We've seen the console signaling halt, but the vm will need a bit |
1677 | + # more time to get there so we help it a bit. |
1678 | + if self.conf.get('vm.release') in ('precise', 'quantal'): |
1679 | + # cloud-init doesn't implement power_state until raring and need a |
1680 | + # gentle nudge. |
1681 | + self.poweroff() |
1682 | + vm_name = self.conf.get('vm.name') |
1683 | + while True: |
1684 | + state = vm_states()[vm_name] |
1685 | + # We expect the vm's state to be 'in shutdown' but in some rare |
1686 | + # occasions we may catch 'running' before getting 'in shutdown'. |
1687 | + if state in ('in shutdown', 'running'): |
1688 | + # Ok, querying the state takes time, this regulates the polling |
1689 | + # enough that we don't need to sleep. |
1690 | + continue |
1691 | + elif state == 'shut off': |
1692 | + # Good, we're done |
1693 | + break |
1694 | + # FIXME: No idea on how to test the following. Manually tested by |
1695 | + # altering the expected state above and running 'selftest.py -v' |
1696 | + # which errors out for test_install_with_seed and |
1697 | + # test_install_backing. Also reproduced when 'running' wasn't |
1698 | + # expected before 'in shutdown' -- vila 2013-02-19 |
1699 | + # Unexpected state reached, bad. |
1700 | + raise SetupVmError('Something went wrong during {name} install\n' |
1701 | + 'The vm ended in state: {state}\n' |
1702 | + 'Check the console at {path}', |
1703 | + name=vm_name, state=state, |
1704 | + path=self._console_path) |
1705 | + |
1706 | + def poweroff(self): |
1707 | + return run_subprocess( |
1708 | + ['sudo', 'virsh', 'destroy', self.conf.get('vm.name')]) |
1709 | + |
1710 | + def undefine(self): |
1711 | + return run_subprocess( |
1712 | + ['sudo', 'virsh', 'undefine', self.conf.get('vm.name'), |
1713 | + '--remove-all-storage']) |
1714 | + |
1715 | + |
1716 | +class KvmFromCloudImage(Kvm): |
1717 | + |
1718 | + def create_disk_image(self, src_name=None, dst_name=None): |
1719 | + """Create a disk image from a cloud one.""" |
1720 | + if src_name is None: |
1721 | + src_name = self.conf.get('vm.cloud_image_name') |
1722 | + if dst_name is None: |
1723 | + dst_name = self.conf.expand_options('{vm.name}.qcow2') |
1724 | + cloud_image_path = os.path.join( |
1725 | + self.conf.get('vm.download_cache'), src_name) |
1726 | + disk_image_path = os.path.join( |
1727 | + self.conf.get('vm.images_dir'), dst_name) |
1728 | + run_subprocess( |
1729 | + ['sudo', 'qemu-img', 'convert', |
1730 | + '-O', 'qcow2', cloud_image_path, disk_image_path]) |
1731 | + run_subprocess( |
1732 | + ['sudo', 'qemu-img', 'resize', |
1733 | + disk_image_path, self.conf.get('vm.disk_size')]) |
1734 | + self._disk_image_path = disk_image_path |
1735 | + |
1736 | + |
1737 | +class KvmFromBacking(Kvm): |
1738 | + |
1739 | + def create_disk_image(self, src_name=None, dst_name=None): |
1740 | + """Create a disk image backed by an existing one.""" |
1741 | + backing_image_path = os.path.join( |
1742 | + self.conf.get('vm.images_dir'), |
1743 | + self.conf.expand_options('{vm.backing}')) |
1744 | + disk_image_path = os.path.join( |
1745 | + self.conf.get('vm.images_dir'), |
1746 | + self.conf.expand_options('{vm.name}.qcow2')) |
1747 | + run_subprocess( |
1748 | + ['sudo', 'qemu-img', 'create', '-f', 'qcow2', |
1749 | + '-b', backing_image_path, disk_image_path]) |
1750 | + run_subprocess( |
1751 | + ['sudo', 'qemu-img', 'resize', |
1752 | + disk_image_path, self.conf.get('vm.disk_size')]) |
1753 | + self._disk_image_path = disk_image_path |
1754 | + |
1755 | + |
1756 | +class ArgParser(argparse.ArgumentParser): |
1757 | + """A parser for the setup_vm script.""" |
1758 | + |
1759 | + def __init__(self): |
1760 | + description = 'Set up virtual machines from a configuration file.' |
1761 | + super(ArgParser, self).__init__( |
1762 | + prog='setup_vm.py', description=description) |
1763 | + self.add_argument( |
1764 | + 'name', help='Virtual machine section in the configuration file.') |
1765 | + self.add_argument('--download', '-d', action="store_true", |
1766 | + help='Force download of the required image.') |
1767 | + self.add_argument('--ssh-keygen', '-k', action="store_true", |
1768 | + help='Generate the ssh server keys (if any).') |
1769 | + self.add_argument('--install', '-i', action="store_true", |
1770 | + help='Install the virtual machine.') |
1771 | + |
1772 | + def parse_args(self, args=None, out=None, err=None): |
1773 | + """Parse arguments, overridding stdout/stderr if provided. |
1774 | + |
1775 | + Overridding stdout/stderr is provided for tests. |
1776 | + |
1777 | + :params args: Defaults to sys.argv[1:]. |
1778 | + |
1779 | + :param out: Defaults to sys.stdout. |
1780 | + |
1781 | + :param err: Defaults to sys.stderr. |
1782 | + """ |
1783 | + out_orig = sys.stdout |
1784 | + err_orig = sys.stderr |
1785 | + try: |
1786 | + if out is not None: |
1787 | + sys.stdout = out |
1788 | + if err is not None: |
1789 | + sys.stderr = err |
1790 | + return super(ArgParser, self).parse_args(args) |
1791 | + finally: |
1792 | + sys.stdout = out_orig |
1793 | + sys.stderr = err_orig |
1794 | + |
1795 | + |
1796 | + |
1797 | +arg_parser = ArgParser() |
1798 | + |
1799 | +class Command(object): |
1800 | + |
1801 | + def __init__(self, vm): |
1802 | + self.vm = vm |
1803 | + |
1804 | + |
1805 | +class Download(Command): |
1806 | + |
1807 | + def run(self): |
1808 | + # FIXME: what needs to be downloaded should depend on the type of the |
1809 | + # vm (possibly errors if there is nothing to download). -- vila |
1810 | + # 2013-02-06 |
1811 | + self.vm.download_cloud_image(force=True) |
1812 | + |
1813 | + |
1814 | +class SshKeyGen(Command): |
1815 | + |
1816 | + def run(self): |
1817 | + self.vm.ssh_keygen() |
1818 | + |
1819 | + |
1820 | +class Install(Command): |
1821 | + |
1822 | + def run(self): |
1823 | + vm_name = self.vm.conf.get('vm.name') |
1824 | + state = vm_states().get(vm_name, None) |
1825 | + if state == 'shut off': |
1826 | + self.vm.undefine() |
1827 | + elif state == 'running': |
1828 | + raise SetupVmError('{name} is running', name=vm_name) |
1829 | + # FIXME: The installation method may vary depending on the vm type. |
1830 | + # -- vila 2013-02-06 |
1831 | + self.vm.install() |
1832 | + |
1833 | + |
1834 | +def build_commands(args=None, out=None, err=None): |
1835 | + cmds = [] |
1836 | + if args is None: |
1837 | + args = sys.argv[1:] |
1838 | + |
1839 | + ns = arg_parser.parse_args(args, out=out, err=err) |
1840 | + |
1841 | + conf = VmStack(ns.name) |
1842 | + with_backing = conf.get('vm.backing') |
1843 | + if with_backing is None: |
1844 | + vm = KvmFromCloudImage(conf) |
1845 | + else: |
1846 | + vm = KvmFromBacking(conf) |
1847 | + if ns.download: |
1848 | + cmds.append(Download(vm)) |
1849 | + if ns.ssh_keygen: |
1850 | + cmds.append(SshKeyGen(vm)) |
1851 | + if ns.install: |
1852 | + cmds.append(Install(vm)) |
1853 | + return cmds |
1854 | + |
1855 | + |
1856 | +def run(args=None): |
1857 | + cmds = build_commands(args) |
1858 | + for cmd in cmds: |
1859 | + try: |
1860 | + cmd.run() |
1861 | + except SetupVmError, e: |
1862 | + # Stop on first error |
1863 | + print 'ERROR: %s' % e |
1864 | + exit(-1) |
1865 | + |
1866 | + |
1867 | +if __name__ == "__main__": |
1868 | + run() |
1869 | |
1870 | === added file 'setup_vm/bin/ubuntu_admin.sh' |
1871 | --- setup_vm/bin/ubuntu_admin.sh 1970-01-01 00:00:00 +0000 |
1872 | +++ setup_vm/bin/ubuntu_admin.sh 2013-04-17 01:29:27 +0000 |
1873 | @@ -0,0 +1,2 @@ |
1874 | +#!/bin/sh |
1875 | +adduser ubuntu admin |
1876 | |
1877 | === added directory 'setup_vm/pay' |
1878 | === added file 'setup_vm/pay/install' |
1879 | --- setup_vm/pay/install 1970-01-01 00:00:00 +0000 |
1880 | +++ setup_vm/pay/install 2013-04-17 01:29:27 +0000 |
1881 | @@ -0,0 +1,40 @@ |
1882 | +#!/bin/sh -ex |
1883 | + |
1884 | +# Allow ssh access to launchpad. |
1885 | +# This should probably be provided by setup_vm. -- vila 2013-03-10 |
1886 | +ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts |
1887 | +# Get the branch. |
1888 | +bzr branch lp:canonical-payment-service {pay.src_dir} |
1889 | +# Get the download cache. |
1890 | +bzr branch lp:~canonical-isd-hackers/+junk/download-cache |
1891 | +# Setup the environment. |
1892 | +cd {pay.src_dir} |
1893 | +# Get the version controlled configs. |
1894 | +bzr branch lp:~canonical-isd-hackers/isd-configs/payments-config branches/project |
1895 | +# Bootstrap the dependencies |
1896 | +fab bootstrap:download_cache_path=~/download-cache |
1897 | +# Set up the correct Django configuration. |
1898 | +cat <<EOF >django_project/local.cfg |
1899 | +[__noschema__] |
1900 | +db_host = /home/ubuntu/{pay.src_dir}/.env/db |
1901 | +hostname = {pay.address}:{pay.port} |
1902 | + |
1903 | +[__main__] |
1904 | +includes = |
1905 | + config/devel.cfg |
1906 | + ../branches/project/config/acceptance.cfg |
1907 | + |
1908 | +[django] |
1909 | +debug = false |
1910 | +internal_ips = |
1911 | + |
1912 | +[testing] |
1913 | +imap_server = {sso.address} |
1914 | +imap_port = {sso.imap_port} |
1915 | +imap_use_ssl = False |
1916 | + |
1917 | +[openid] |
1918 | +openid_sso_server_url = {sso.url} |
1919 | +openid_trust_root = {pay.url} |
1920 | + |
1921 | +EOF |
1922 | |
1923 | === added file 'setup_vm/pay/run' |
1924 | --- setup_vm/pay/run 1970-01-01 00:00:00 +0000 |
1925 | +++ setup_vm/pay/run 2013-04-17 01:29:27 +0000 |
1926 | @@ -0,0 +1,9 @@ |
1927 | +#!/bin/sh |
1928 | + |
1929 | +cd {pay.src_dir} |
1930 | + |
1931 | +# Setup the database. |
1932 | +fab setup_postgresql_server |
1933 | +fab manage:loaddata,test |
1934 | +# Start the PAY server, accessible from the local network. |
1935 | +fab run:0.0.0.0:{pay.port} |
1936 | |
1937 | === added file 'setup_vm/pay/run-for-u1' |
1938 | --- setup_vm/pay/run-for-u1 1970-01-01 00:00:00 +0000 |
1939 | +++ setup_vm/pay/run-for-u1 2013-04-17 01:29:27 +0000 |
1940 | @@ -0,0 +1,57 @@ |
1941 | +#!/bin/sh |
1942 | + |
1943 | +cd {pay.src_dir} |
1944 | + |
1945 | +# Setup the database. |
1946 | +fab setup_postgresql_server |
1947 | +# Add the U1 consumer. |
1948 | +cat <<EOF >src/paymentservice/fixtures/consumer.json |
1949 | +[ |
1950 | + { |
1951 | + "pk": "U1", |
1952 | + "model": "paymentservice.consumer", |
1953 | + "fields": { |
1954 | + "notification_url": "{u1.url}/notifications/", |
1955 | + "ip_address": "{u1.address}", |
1956 | + "name": "Ubuntu One", |
1957 | + "default_business_unit": "Online Services", |
1958 | + "email_footer": "Test footer.", |
1959 | + "theme": "ubuntuone" |
1960 | + } |
1961 | + } |
1962 | +] |
1963 | +EOF |
1964 | +fab manage:loaddata,consumer |
1965 | +# Add the API user for U1. |
1966 | +# We generated this json file with: |
1967 | +# Go to {pay.url}/admin |
1968 | +# Sign in with the admin/admin. |
1969 | +# Click the more link next to the Model Admin heading. |
1970 | +# On the Paymentservice section, click the +Add link next to API Users. |
1971 | +# Fill the form with: |
1972 | +# username: u1qauser |
1973 | +# password: u1qapassword |
1974 | +# Click the Save button. |
1975 | +# Select the Ubuntu One (U1) Consumer. |
1976 | +# Click the Save button. |
1977 | +# $ fab manage:dumpdata,paymentservice.APIUser |
1978 | +cat <<EOF >src/paymentservice/fixtures/apiuser.json |
1979 | +[ |
1980 | + { |
1981 | + "pk": 2, |
1982 | + "model": "paymentservice.apiuser", |
1983 | + "fields": { |
1984 | + "username": "u1qauser", |
1985 | + "created_at": "2013-04-15 00:09:48", |
1986 | + "password": "sha1\$b2a8e\$0e06d9cb46583aa53d3bf144ae07018a7546f737", |
1987 | + "consumer": "U1", |
1988 | + "updated_at": "2013-04-15 00:09:54" |
1989 | + } |
1990 | + } |
1991 | +] |
1992 | +EOF |
1993 | +fab manage:loaddata,apiuser |
1994 | +# Start the PAY server, accessible from the local network. |
1995 | +# We don't call the run task because it loads a fixture that overwrites our |
1996 | +# consumer. |
1997 | +fab manage:runserver,0.0.0.0:{pay.port} |
1998 | |
1999 | === added file 'setup_vm/pay/test' |
2000 | --- setup_vm/pay/test 1970-01-01 00:00:00 +0000 |
2001 | +++ setup_vm/pay/test 2013-04-17 01:29:27 +0000 |
2002 | @@ -0,0 +1,5 @@ |
2003 | +#!/bin/sh |
2004 | + |
2005 | +cd {pay.src_dir} |
2006 | + |
2007 | +SST_BASE_URL={pay.url} fab acceptance:screenshot=true,report=xml,extended=true |
2008 | |
2009 | === added file 'setup_vm/selftest.py' |
2010 | --- setup_vm/selftest.py 1970-01-01 00:00:00 +0000 |
2011 | +++ setup_vm/selftest.py 2013-04-17 01:29:27 +0000 |
2012 | @@ -0,0 +1,24 @@ |
2013 | +#!/usr/bin/env python |
2014 | + |
2015 | +import sys |
2016 | + |
2017 | +import testtools.run |
2018 | +import unittest |
2019 | + |
2020 | + |
2021 | +class TestProgram(testtools.run.TestProgram): |
2022 | + |
2023 | + def __init__(self, module, argv, stdout=None, testRunner=None, exit=True): |
2024 | + if testRunner is None: |
2025 | + testRunner = unittest.TextTestRunner |
2026 | + super(TestProgram, self).__init__(module, argv=argv, stdout=stdout, |
2027 | + testRunner=testRunner, exit=exit) |
2028 | + |
2029 | + |
2030 | +# We discover tests under './tests', the python 'load_test' protocol can be |
2031 | +# used in test modules for more fancy stuff. |
2032 | +discover_args = ['discover', |
2033 | + '--start-directory', './tests', |
2034 | + '--top-level-directory', '.', |
2035 | + ] |
2036 | +TestProgram(__name__, argv=[sys.argv[0]] + discover_args + sys.argv[1:]) |
2037 | |
2038 | === added directory 'setup_vm/sso' |
2039 | === added file 'setup_vm/sso/install' |
2040 | --- setup_vm/sso/install 1970-01-01 00:00:00 +0000 |
2041 | +++ setup_vm/sso/install 2013-04-17 01:29:27 +0000 |
2042 | @@ -0,0 +1,45 @@ |
2043 | +#!/bin/sh -ex |
2044 | + |
2045 | +# Allow ssh access to launchpad. |
2046 | +# This should probably be provided by setup_vm. -- vila 2013-03-10 |
2047 | +ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts |
2048 | +# Get the branch. |
2049 | +bzr branch lp:canonical-identity-provider {sso.src_dir} |
2050 | +# Get the download cache. |
2051 | +bzr branch lp:~canonical-isd-hackers/+junk/download-cache |
2052 | +# Setup the environment. |
2053 | +cd {sso.src_dir} |
2054 | +# Get the version controlled configs. |
2055 | +bzr branch lp:~canonical-isd-hackers/isd-configs/sso-config branches/project |
2056 | +# Bootstrap the dependencies |
2057 | +fab bootstrap:download_cache_path=~/download-cache |
2058 | +# Set up the correct Django configuration. |
2059 | +# In order to set the db_host to a directory in .env, we need to use the full |
2060 | +# path. Otherwise, fab setup_postgresql_server will fail. |
2061 | +# TODO we can either configure the postgresql authentication and pass db_host |
2062 | +# as empty, or use cat just to append to the end of the default local.cfg |
2063 | +# that will contain the full path we need, or pass the user name in a config |
2064 | +# variable. |
2065 | +cat <<EOF >django_project/local.cfg |
2066 | +[__noschema__] |
2067 | +basedir = . |
2068 | +db_host = /home/ubuntu/{sso.src_dir}/.env/db |
2069 | +hostname = {sso.address}:{sso.port} |
2070 | + |
2071 | +[__main__] |
2072 | +includes = |
2073 | + config/devel.cfg |
2074 | + ../branches/project/config/acceptance-dev.cfg |
2075 | + |
2076 | +[django] |
2077 | +debug = false |
2078 | +email_port = {sso.smtp_port} |
2079 | + |
2080 | +[testing] |
2081 | +imap_server = {sso.address} |
2082 | +imap_port = {sso.imap_port} |
2083 | +# needs to be a full email |
2084 | +imap_username = whatever@we.dont.care |
2085 | +imap_use_ssl = False |
2086 | + |
2087 | +EOF |
2088 | |
2089 | === added file 'setup_vm/sso/run' |
2090 | --- setup_vm/sso/run 1970-01-01 00:00:00 +0000 |
2091 | +++ setup_vm/sso/run 2013-04-17 01:29:27 +0000 |
2092 | @@ -0,0 +1,16 @@ |
2093 | +#!/bin/sh |
2094 | + |
2095 | +cd ~/{sso.src_dir} |
2096 | +# We need an SMTP server to send emails. |
2097 | +.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port} |
2098 | + |
2099 | +# Setup the database. |
2100 | +fab setup_postgresql_server |
2101 | +fab manage:loaddata,test |
2102 | +fab manage:create_test_team |
2103 | +# get gargoyle flags from their use in the code |
2104 | +SST_FLAGS=`grep -rho --exclude 'test_*.py' "is_active([\"']\(.*\)[\"']" identityprovider/ webui/ | sed -E "s/is_active\(['\"](.*)['\"]/\1/" | awk '{print tolower($0)}' | sort | uniq | tr '\n' ','` |
2105 | +# We need to remove the trailing ',' |
2106 | +fab gargoyle_flags:${SST_FLAGS%,} |
2107 | +# Start the SSO server, accessible from the local network. |
2108 | +fab run:0.0.0.0:{sso.port} |
2109 | |
2110 | === added file 'setup_vm/sso/run-for-pay' |
2111 | --- setup_vm/sso/run-for-pay 1970-01-01 00:00:00 +0000 |
2112 | +++ setup_vm/sso/run-for-pay 2013-04-17 01:29:27 +0000 |
2113 | @@ -0,0 +1,14 @@ |
2114 | +#!/bin/sh |
2115 | + |
2116 | +cd ~/{sso.src_dir} |
2117 | +# We need an SMTP server to send emails. |
2118 | +.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port} |
2119 | + |
2120 | +# Setup the database. |
2121 | +fab setup_postgresql_server |
2122 | +fab manage:loaddata,isdtest |
2123 | +fab manage:loaddata,allow_unverified |
2124 | +# Set the allow-unverified config for Pay. |
2125 | +fab manage:add_openid_rp_config,{pay.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language" |
2126 | +# Start the SSO server, accessible from the local network. |
2127 | +fab run:0.0.0.0:{sso.port} |
2128 | |
2129 | === added file 'setup_vm/sso/run-for-u1' |
2130 | --- setup_vm/sso/run-for-u1 1970-01-01 00:00:00 +0000 |
2131 | +++ setup_vm/sso/run-for-u1 2013-04-17 01:29:27 +0000 |
2132 | @@ -0,0 +1,43 @@ |
2133 | +#!/bin/sh |
2134 | + |
2135 | +cd ~/{sso.src_dir} |
2136 | +# We need an SMTP server to send emails. |
2137 | +.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port} |
2138 | + |
2139 | +# Setup the database. |
2140 | +fab setup_postgresql_server |
2141 | +fab manage:loaddata,allow_unverified |
2142 | +# Set the allow-unverified config for Pay. |
2143 | +fab manage:add_openid_rp_config,{pay.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language" |
2144 | +# Set the allow-unverified config for U1. |
2145 | +fab manage:add_openid_rp_config,{u1.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language" |
2146 | +# Add the API user for U1. |
2147 | +# We generated this json file with: |
2148 | +# $ fab manage:createsuperuser |
2149 | +# Go to {sso.url}/admin |
2150 | +# Sign in with the super user you have just created. |
2151 | +# Click the more link next to the Model Admin heading. |
2152 | +# On the Identityprovider section, click the +Add link next to API Users. |
2153 | +# Fill the form with: |
2154 | +# username: u1qauser |
2155 | +# password: u1qapassword |
2156 | +# Click the Save button. |
2157 | +# $ fab manage:dumpdata,identityprovider.APIUser |
2158 | +cat <<EOF >identityprovider/fixtures/apiuser.json |
2159 | +[ |
2160 | + { |
2161 | + "pk": 1, |
2162 | + "model": "identityprovider.apiuser", |
2163 | + "fields": { |
2164 | + "username": "u1qauser", |
2165 | + "created_at": "2013-04-14 21:09:43", |
2166 | + "password": "k1B7nUTaEsrqAPHF/bWsLlNIPUsH7mViraFQBZPgNRRuvsZlRq8CZg==", |
2167 | + "updated_at": "2013-04-14 21:09:43" |
2168 | + } |
2169 | + } |
2170 | +] |
2171 | + |
2172 | +EOF |
2173 | +fab manage:loaddata,apiuser |
2174 | +# Start the SSO server, accessible from the local network. |
2175 | +fab run:0.0.0.0:{sso.port} |
2176 | |
2177 | === added file 'setup_vm/sso/test' |
2178 | --- setup_vm/sso/test 1970-01-01 00:00:00 +0000 |
2179 | +++ setup_vm/sso/test 2013-04-17 01:29:27 +0000 |
2180 | @@ -0,0 +1,11 @@ |
2181 | +#!/bin/sh |
2182 | + |
2183 | +# FIXME: This should run on the host and get config options expanded |
2184 | +# -- vila 2013-03-12 |
2185 | + |
2186 | +cd {sso.src_dir} |
2187 | + |
2188 | +# get gargoyle flags from their use in the code |
2189 | +SST_FLAGS=`grep -rho --exclude 'test_*.py' "is_active([\"']\(.*\)[\"']" identityprovider/ webui/ | sed -E "s/is_active\(['\"](.*)['\"]/\1/" | awk '{print tolower($0)}' | sort | uniq | tr '\n' ';'` |
2190 | +# run tests |
2191 | +SST_BASE_URL={sso.url} fab acceptance:screenshot=true,report=xml,extended=true,flags=$SST_FLAGS |
2192 | |
2193 | === added directory 'setup_vm/tests' |
2194 | === added file 'setup_vm/tests/__init__.py' |
2195 | --- setup_vm/tests/__init__.py 1970-01-01 00:00:00 +0000 |
2196 | +++ setup_vm/tests/__init__.py 2013-04-17 01:29:27 +0000 |
2197 | @@ -0,0 +1,75 @@ |
2198 | +import os |
2199 | +import shutil |
2200 | +import tempfile |
2201 | + |
2202 | +from bzrlib import osutils |
2203 | + |
2204 | +def override_env_var(name, value): |
2205 | + """Modify the environment, setting or removing the env_variable. |
2206 | + |
2207 | + :param name: The environment variable to set. |
2208 | + |
2209 | + :param value: The value to set the environment to. If None, then |
2210 | + the variable will be removed. |
2211 | + |
2212 | + :return: The original value of the environment variable. |
2213 | + """ |
2214 | + orig = os.environ.get(name) |
2215 | + if value is None: |
2216 | + if orig is not None: |
2217 | + del os.environ[name] |
2218 | + else: |
2219 | + # FIXME: supporting unicode values requires a way to acquire the |
2220 | + # user encoding, punting for now -- vila 2013-01-30 |
2221 | + os.environ[name] = value |
2222 | + return orig |
2223 | + |
2224 | + |
2225 | +def override_env(test, name, new): |
2226 | + """Set an environment variable, and reset it after the test. |
2227 | + |
2228 | + :param name: The environment variable name. |
2229 | + |
2230 | + :param new: The value to set the variable to. If None, the |
2231 | + variable is deleted from the environment. |
2232 | + |
2233 | + :returns: The actual variable value. |
2234 | + """ |
2235 | + value = override_env_var(name, new) |
2236 | + test.addCleanup(override_env_var, name, value) |
2237 | + return value |
2238 | + |
2239 | + |
2240 | +isolated_environ = { |
2241 | + 'HOME': None, |
2242 | +} |
2243 | + |
2244 | + |
2245 | +def isolate_env(test, env=None): |
2246 | + """Isolate test from the environment variables. |
2247 | + |
2248 | + This is usually called in setUp for tests that needs to modify the |
2249 | + environment variables and restore them after the test is run. |
2250 | + |
2251 | + :param test: A test instance |
2252 | + |
2253 | + :param env: A dict containing variable definitions to be installed. Only |
2254 | + the variables present there are protected. They are initialized with |
2255 | + the provided values. |
2256 | + """ |
2257 | + if env is None: |
2258 | + env = isolated_environ |
2259 | + for name, value in env.items(): |
2260 | + override_env(test, name, value) |
2261 | + |
2262 | + |
2263 | +def set_cwd_to_tmp(test): |
2264 | + """Create a temp dir an cd into it for the test duration. |
2265 | + |
2266 | + This is generally called during a test setup. |
2267 | + """ |
2268 | + test.test_base_dir = tempfile.mkdtemp(prefix='mytests-', suffix='.tmp') |
2269 | + test.addCleanup(shutil.rmtree, test.test_base_dir, True) |
2270 | + current_dir = os.getcwdu() |
2271 | + test.addCleanup(os.chdir, current_dir) |
2272 | + os.chdir(test.test_base_dir) |
2273 | |
2274 | === added file 'setup_vm/tests/test_setup_vm.py' |
2275 | --- setup_vm/tests/test_setup_vm.py 1970-01-01 00:00:00 +0000 |
2276 | +++ setup_vm/tests/test_setup_vm.py 2013-04-17 01:29:27 +0000 |
2277 | @@ -0,0 +1,1074 @@ |
2278 | +from cStringIO import StringIO |
2279 | +import os |
2280 | + |
2281 | +from bzrlib import errors |
2282 | +import testtools |
2283 | + |
2284 | +import tests |
2285 | +from bin import setup_vm |
2286 | + |
2287 | + |
2288 | +def requires_known_reference_image(test): |
2289 | + # We need a pre-seeded download cache from the user running the tests |
2290 | + # as downloading the cloud image is too long. |
2291 | + user_conf = setup_vm.VmStack(None) |
2292 | + download_cache = user_conf.get('vm.download_cache') |
2293 | + if download_cache is None: |
2294 | + test.skip('vm.download_cache is not set') |
2295 | + # We use some known reference |
2296 | + reference_cloud_image_name = 'raring-server-cloudimg-amd64-disk1.img' |
2297 | + cloud_image_path = os.path.join( |
2298 | + download_cache, reference_cloud_image_name) |
2299 | + if not os.path.exists(cloud_image_path): |
2300 | + test.skip('%s is not available' % (cloud_image_path,)) |
2301 | + return download_cache, reference_cloud_image_name |
2302 | + |
2303 | + |
2304 | +class TestCaseWithHome(testtools.TestCase): |
2305 | + """Provide an isolated disk-based environment. |
2306 | + |
2307 | + A $HOME directory is setup as well as an /etc/ one so tests can setup |
2308 | + config files. |
2309 | + """ |
2310 | + |
2311 | + def setUp(self): |
2312 | + super(TestCaseWithHome, self).setUp() |
2313 | + tests.set_cwd_to_tmp(self) |
2314 | + tests.isolate_env(self) |
2315 | + # Isolate tests from the user environment |
2316 | + self.home_dir = os.path.join(self.test_base_dir, 'home') |
2317 | + os.mkdir(self.home_dir) |
2318 | + os.environ['HOME'] = self.home_dir |
2319 | + # Also isolate from the system environment |
2320 | + self.etc_dir = os.path.join(self.test_base_dir, 'etc') |
2321 | + os.mkdir(self.etc_dir) |
2322 | + self.patch(setup_vm, 'system_config_dir', lambda: self.etc_dir) |
2323 | + |
2324 | + |
2325 | +class TestVmMatcher(TestCaseWithHome): |
2326 | + |
2327 | + def setUp(self): |
2328 | + super(TestVmMatcher, self).setUp() |
2329 | + self.store = setup_vm.VmStore('.', 'foo.conf') |
2330 | + self.matcher = setup_vm.VmMatcher(self.store, 'test') |
2331 | + |
2332 | + def test_empty_section_always_matches(self): |
2333 | + self.store._load_from_string('foo=bar') |
2334 | + matching = list(self.matcher.get_sections()) |
2335 | + self.assertEqual(1, len(matching)) |
2336 | + |
2337 | + def test_specific_before_generic(self): |
2338 | + self.store._load_from_string('foo=bar\n[test]\nfoo=baz') |
2339 | + matching = list(self.matcher.get_sections()) |
2340 | + self.assertEqual(2, len(matching)) |
2341 | + # First matching section is for test |
2342 | + self.assertEqual(self.store, matching[0][0]) |
2343 | + base_section = matching[0][1] |
2344 | + self.assertEqual('test', base_section.id) |
2345 | + # Second matching section is the no-name one |
2346 | + self.assertEqual(self.store, matching[0][0]) |
2347 | + no_name_section = matching[1][1] |
2348 | + self.assertIs(None, no_name_section.id) |
2349 | + |
2350 | + |
2351 | +class TestVmStores(TestCaseWithHome): |
2352 | + |
2353 | + def setUp(self): |
2354 | + super(TestVmStores, self).setUp() |
2355 | + self.conf = setup_vm.VmStack('foo') |
2356 | + |
2357 | + |
2358 | + def test_default_in_empty_stack(self): |
2359 | + self.assertEqual('1024', self.conf.get('vm.ram_size')) |
2360 | + |
2361 | + |
2362 | + def test_system_overrides_internal(self): |
2363 | + self.conf.system_store._load_from_string('vm.ram_size = 42') |
2364 | + self.assertEqual('42', self.conf.get('vm.ram_size')) |
2365 | + |
2366 | + def test_user_overrides_system(self): |
2367 | + self.conf.system_store._load_from_string('vm.ram_size = 42') |
2368 | + self.conf.store._load_from_string('vm.ram_size = 4201') |
2369 | + self.assertEqual('4201', self.conf.get('vm.ram_size')) |
2370 | + |
2371 | + def test_local_overrides_user(self): |
2372 | + self.conf.system_store._load_from_string('vm.ram_size = 42') |
2373 | + self.conf.store._load_from_string('vm.ram_size = 4201') |
2374 | + self.conf.local_store._load_from_string('vm.ram_size = 8402') |
2375 | + self.assertEqual('8402', self.conf.get('vm.ram_size')) |
2376 | + |
2377 | + |
2378 | +class TestVmStack(TestCaseWithHome): |
2379 | + |
2380 | + def setUp(self): |
2381 | + super(TestVmStack, self).setUp() |
2382 | + self.conf = setup_vm.VmStack('foo') |
2383 | + self.conf.store._load_from_string(''' |
2384 | +vm.release=raring |
2385 | +vm.cpu_model=amd64 |
2386 | +''') |
2387 | + |
2388 | + def assertValue(self, expected_value, option): |
2389 | + self.assertEqual(expected_value, self.conf.get(option)) |
2390 | + |
2391 | + def test_raring_iso_url(self): |
2392 | + self.assertValue('http://cdimage.ubuntu.com/daily-live/current/', |
2393 | + 'vm.iso_url' ) |
2394 | + |
2395 | + def test_raring_iso_name(self): |
2396 | + self.assertValue( 'raring-desktop-amd64.iso', 'vm.iso_name') |
2397 | + |
2398 | + def test_raring_cloud_image_url(self): |
2399 | + self.assertValue('http://cloud-images.ubuntu.com/raring/current/', |
2400 | + 'vm.cloud_image_url') |
2401 | + |
2402 | + def test_raring_cloud_image_name(self): |
2403 | + self.assertValue('raring-server-cloudimg-amd64-disk1.img', |
2404 | + 'vm.cloud_image_name') |
2405 | + |
2406 | + def test_apt_proxy_set(self): |
2407 | + proxy = 'apt_proxy: http://example.org:4321' |
2408 | + self.conf.set('vm.apt_proxy', proxy) |
2409 | + self.assertEqual(proxy, self.conf.expand_options('{vm.apt_proxy}')) |
2410 | + |
2411 | + def test_download_cache_with_user_expansion(self): |
2412 | + download_cache = '~/installers' |
2413 | + self.conf.set('vm.download_cache', download_cache) |
2414 | + self.assertValue(os.path.join(self.home_dir, 'installers'), |
2415 | + 'vm.download_cache') |
2416 | + |
2417 | + def test_images_dir_with_user_expansion(self): |
2418 | + images_dir = '~/images' |
2419 | + self.conf.set('vm.images_dir', images_dir) |
2420 | + self.assertValue(os.path.join(self.home_dir, 'images'), |
2421 | + 'vm.images_dir') |
2422 | + |
2423 | + |
2424 | +class TestPathOption(TestCaseWithHome): |
2425 | + |
2426 | + def assertConverted(self, expected, value): |
2427 | + option = setup_vm.PathOption('foo', help='A path.') |
2428 | + self.assertEquals(expected, option.convert_from_unicode(None, value)) |
2429 | + |
2430 | + def test_absolute_path(self): |
2431 | + self.assertConverted('/test/path', '/test/path') |
2432 | + |
2433 | + def test_home_path_with_expansion(self): |
2434 | + self.assertConverted(self.home_dir, '~') |
2435 | + |
2436 | + def test_path_in_home_with_expansion(self): |
2437 | + self.assertConverted(os.path.join(self.home_dir, 'test/path'), |
2438 | + '~/test/path') |
2439 | + |
2440 | + |
2441 | +class TestDownload(TestCaseWithHome): |
2442 | + |
2443 | +# FIXME: Needs parametrization against vm.{cloud_image_name|iso_name} and |
2444 | +# {download_iso|download_cloud_image} -- vila 2013-02-07 |
2445 | + |
2446 | + def setUp(self): |
2447 | + # Downloading real isos or images is too long for tests, instead, we |
2448 | + # fake it by downloading a small but known to exist file: MD5SUMS |
2449 | + super(TestDownload, self).setUp() |
2450 | + download_cache = os.path.join(self.test_base_dir, 'downloads') |
2451 | + os.mkdir(download_cache) |
2452 | + self.conf = setup_vm.VmStack('foo') |
2453 | + self.conf.store._load_from_string(''' |
2454 | +vm.iso_name=MD5SUMS |
2455 | +vm.cloud_image_name=MD5SUMS |
2456 | +vm.release=raring |
2457 | +vm.cpu_model=amd64 |
2458 | +vm.download_cache=%s |
2459 | +''' % (download_cache,)) |
2460 | + |
2461 | + def test_download_iso(self): |
2462 | + vm = setup_vm.Kvm(self.conf) |
2463 | + self.assertTrue(vm.download_iso()) |
2464 | + # Trying to download again will find the file in the cache |
2465 | + self.assertFalse(vm.download_iso()) |
2466 | + # Forcing the download even when the file is present |
2467 | + self.assertTrue(vm.download_iso(force=True)) |
2468 | + |
2469 | + def test_download_cloud_image(self): |
2470 | + vm = setup_vm.Kvm(self.conf) |
2471 | + self.assertTrue(vm.download_cloud_image()) |
2472 | + # Trying to download again will find the file in the cache |
2473 | + self.assertFalse(vm.download_cloud_image()) |
2474 | + # Forcing the download even when the file is present |
2475 | + self.assertTrue(vm.download_cloud_image(force=True)) |
2476 | + |
2477 | + def test_download_unknown_iso_fail(self): |
2478 | + self.conf.set('vm.iso_name', 'I-dont-exist') |
2479 | + vm = setup_vm.Kvm(self.conf) |
2480 | + self.assertRaises(setup_vm.CommandError, vm.download_iso) |
2481 | + |
2482 | + def test_download_unknown_cloud_image_fail(self): |
2483 | + self.conf.set('vm.cloud_image_name', 'I-dont-exist') |
2484 | + vm = setup_vm.Kvm(self.conf) |
2485 | + self.assertRaises(setup_vm.CommandError, vm.download_cloud_image) |
2486 | + |
2487 | + def test_download_iso_with_unknown_cache_fail(self): |
2488 | + dl_cache = os.path.join(self.test_base_dir, 'I-dont-exist') |
2489 | + self.conf.set('vm.download_cache', dl_cache) |
2490 | + vm = setup_vm.Kvm(self.conf) |
2491 | + self.assertRaises(setup_vm.ConfigValueError, vm.download_iso) |
2492 | + |
2493 | + def test_download_cloud_image_with_unknown_cache_fail(self): |
2494 | + dl_cache = os.path.join(self.test_base_dir, 'I-dont-exist') |
2495 | + self.conf.set('vm.download_cache', dl_cache) |
2496 | + vm = setup_vm.Kvm(self.conf) |
2497 | + self.assertRaises(setup_vm.ConfigValueError, vm.download_cloud_image) |
2498 | + |
2499 | + |
2500 | +class TestMetaData(TestCaseWithHome): |
2501 | + |
2502 | + def setUp(self): |
2503 | + super(TestMetaData, self).setUp() |
2504 | + self.conf = setup_vm.VmStack('foo') |
2505 | + self.vm = setup_vm.Kvm(self.conf) |
2506 | + images_dir = os.path.join(self.test_base_dir, 'images') |
2507 | + os.mkdir(images_dir) |
2508 | + config_dir = os.path.join(self.test_base_dir, 'config') |
2509 | + self.conf.store._load_from_string(''' |
2510 | +vm.name=foo |
2511 | +vm.images_dir=%s |
2512 | +vm.config_dir=%s |
2513 | +''' % (images_dir, config_dir,)) |
2514 | + |
2515 | + def test_create_meta_data(self): |
2516 | + self.vm.create_meta_data() |
2517 | + self.assertTrue(os.path.exists(self.vm._config_dir)) |
2518 | + self.assertTrue(os.path.exists(self.vm._meta_data_path)) |
2519 | + with open(self.vm._meta_data_path) as f: |
2520 | + meta_data = f.readlines() |
2521 | + self.assertEqual(2, len(meta_data)) |
2522 | + self.assertEqual('instance-id: foo\n', meta_data[0]) |
2523 | + self.assertEqual('local-hostname: foo\n', meta_data[1]) |
2524 | + |
2525 | + |
2526 | +class TestYaml(testtools.TestCase): |
2527 | + |
2528 | + def yaml_load(self, *args, **kwargs): |
2529 | + return setup_vm.yaml.safe_load(*args, **kwargs) |
2530 | + |
2531 | + def yaml_dump(self, *args, **kwargs): |
2532 | + return setup_vm.yaml.safe_dump(*args, **kwargs) |
2533 | + |
2534 | + def test_load_scalar(self): |
2535 | + self.assertEqual({'foo': 'bar'}, self.yaml_load(StringIO('{foo: bar}'))) |
2536 | + # Surprisingly the enclosing braces are not needed, probably a special |
2537 | + # case for the highest level |
2538 | + self.assertEqual({'foo': 'bar'}, self.yaml_load(StringIO('foo: bar'))) |
2539 | + |
2540 | + def test_dump_scalar(self): |
2541 | + self.assertEqual('{foo: bar}\n', self.yaml_dump(dict(foo='bar'))) |
2542 | + |
2543 | + def test_load_list(self): |
2544 | + self.assertEqual({'foo': ['a', 'b', 'c']}, |
2545 | + # Single space indentation is enough |
2546 | + self.yaml_load(StringIO('''\ |
2547 | +foo: |
2548 | + - a |
2549 | + - b |
2550 | + - c |
2551 | +'''))) |
2552 | + |
2553 | + def test_dump_list(self): |
2554 | + # No more enclosing braces... yeah for consistency :-/ |
2555 | + self.assertEqual('foo: [a, b, c]\n', |
2556 | + self.yaml_dump(dict(foo=['a', 'b', 'c']))) |
2557 | + |
2558 | + def test_load_dict(self): |
2559 | + self.assertEqual({'foo': {'bar': 'baz'}}, |
2560 | + self.yaml_load(StringIO('{foo: {bar: baz}}'))) |
2561 | + multiple_lines = '''\ |
2562 | +foo: {bar: multiple |
2563 | + lines} |
2564 | +''' |
2565 | + self.assertEqual({'foo': {'bar': 'multiple lines'}}, |
2566 | + self.yaml_load(StringIO(multiple_lines))) |
2567 | + |
2568 | + |
2569 | + |
2570 | +class TestLaunchpadAccess(TestCaseWithHome): |
2571 | + |
2572 | + def setUp(self): |
2573 | + super(TestLaunchpadAccess, self).setUp() |
2574 | + self.conf = setup_vm.VmStack('foo') |
2575 | + self.vm = setup_vm.Kvm(self.conf) |
2576 | + self.ci_data = setup_vm.CIUserData(self.conf) |
2577 | + |
2578 | + def test_cant_find_private_key(self): |
2579 | + self.conf.store._load_from_string('vm.launchpad_id = I-dont-exist') |
2580 | + e = self.assertRaises(setup_vm.ConfigPathNotFound, |
2581 | + self.ci_data.set_launchpad_access) |
2582 | + key_path = '~/.ssh/I-dont-exist@setup_vm' |
2583 | + self.assertEqual(key_path, e.path) |
2584 | + self.assertTrue(unicode(e).startswith( |
2585 | + 'You need to create the {p} keypair'.format(p=key_path))) |
2586 | + |
2587 | + def test_id_with_key(self): |
2588 | + ssh_dir = os.path.join(self.home_dir, '.ssh') |
2589 | + os.mkdir(ssh_dir) |
2590 | + key_path = os.path.join(ssh_dir, 'user@setup_vm') |
2591 | + with open(key_path, 'w') as f: |
2592 | + f.write('key content') |
2593 | + self.conf.store._load_from_string('vm.launchpad_id = user') |
2594 | + self.assertEqual(None, self.ci_data.launchpad_hook) |
2595 | + self.ci_data.set_launchpad_access() |
2596 | + self.assertEqual('''\ |
2597 | +#!/bin/sh |
2598 | +mkdir -p /home/ubuntu/.ssh |
2599 | +chown ubuntu:ubuntu ~ubuntu |
2600 | +chmod 0700 ~ubuntu |
2601 | +chown ubuntu:ubuntu /home/ubuntu/.ssh |
2602 | +chmod 0700 /home/ubuntu/.ssh |
2603 | +cat >/home/ubuntu/.ssh/id_rsa <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
2604 | +key content |
2605 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
2606 | +chown ubuntu:ubuntu /home/ubuntu/.ssh/id_rsa |
2607 | +chmod 0400 /home/ubuntu/.ssh/id_rsa |
2608 | +''', |
2609 | + self.ci_data.launchpad_hook) |
2610 | + cc = self.ci_data.cloud_config |
2611 | + self.assertEquals([['sudo', '-u', 'ubuntu', |
2612 | + 'bzr', 'launchpad-login', 'user']], |
2613 | + cc['runcmd']) |
2614 | + |
2615 | + |
2616 | +class TestCIUserData(TestCaseWithHome): |
2617 | + |
2618 | + def setUp(self): |
2619 | + super(TestCIUserData, self).setUp() |
2620 | + self.conf = setup_vm.VmStack('foo') |
2621 | + self.ci_data = setup_vm.CIUserData(self.conf) |
2622 | + |
2623 | + def test_empty_config(self): |
2624 | + self.ci_data.populate() |
2625 | + # Check default values |
2626 | + self.assertIs(None, self.ci_data.root_hook) |
2627 | + self.assertIs(None, self.ci_data.ubuntu_hook) |
2628 | + cc = self.ci_data.cloud_config |
2629 | + self.assertFalse(cc['apt_update']) |
2630 | + self.assertFalse(cc['apt_upgrade']) |
2631 | + self.assertEqual({'expire': False}, cc['chpasswd']) |
2632 | + self.assertEqual('setup_vm finished installing in ${uptime} seconds.', |
2633 | + cc['final_message']) |
2634 | + self.assertTrue(cc['manage_etc_hosts']) |
2635 | + self.assertEqual('ubuntu', cc['password']) |
2636 | + self.assertEqual({'mode': 'poweroff'}, cc['power_state']) |
2637 | + |
2638 | + def test_password(self): |
2639 | + self.conf.store._load_from_string('vm.password = tagada') |
2640 | + self.ci_data.populate() |
2641 | + self.assertEquals('tagada', self.ci_data.cloud_config['password']) |
2642 | + |
2643 | + def test_apt_proxy(self): |
2644 | + self.conf.store._load_from_string('vm.apt_proxy = tagada') |
2645 | + self.ci_data.populate() |
2646 | + self.assertEquals('tagada', self.ci_data.cloud_config['apt_proxy']) |
2647 | + |
2648 | + def test_final_message_precise(self): |
2649 | + self.conf.store._load_from_string('vm.release = precise') |
2650 | + self.ci_data.populate() |
2651 | + self.assertEqual('setup_vm finished installing in $UPTIME seconds.', |
2652 | + self.ci_data.cloud_config['final_message']) |
2653 | + |
2654 | + def test_poweroff_precise(self): |
2655 | + self.conf.store._load_from_string('vm.release = precise') |
2656 | + self.ci_data.populate() |
2657 | + self.assertEqual(['halt'], self.ci_data.cloud_config['runcmd']) |
2658 | + |
2659 | + def test_poweroff_quantal(self): |
2660 | + self.conf.store._load_from_string('vm.release = quantal') |
2661 | + self.ci_data.populate() |
2662 | + self.assertEqual(['halt'], self.ci_data.cloud_config['runcmd']) |
2663 | + |
2664 | + def test_poweroff_other(self): |
2665 | + self.conf.store._load_from_string('vm.release = raring') |
2666 | + self.ci_data.populate() |
2667 | + self.assertEqual({'mode': 'poweroff'}, |
2668 | + self.ci_data.cloud_config['power_state']) |
2669 | + self.assertIs(None, self.ci_data.cloud_config.get('runcmd')) |
2670 | + |
2671 | + def test_update_true(self): |
2672 | + self.conf.store._load_from_string('vm.update = True') |
2673 | + self.ci_data.populate() |
2674 | + cc = self.ci_data.cloud_config |
2675 | + self.assertTrue(cc['apt_update']) |
2676 | + self.assertTrue(cc['apt_upgrade']) |
2677 | + |
2678 | + def test_packages(self): |
2679 | + self.conf.store._load_from_string('vm.packages = bzr,ubuntu-desktop') |
2680 | + self.ci_data.populate() |
2681 | + self.assertEqual(['bzr', 'ubuntu-desktop'], |
2682 | + self.ci_data.cloud_config['packages']) |
2683 | + |
2684 | + def test_apt_sources(self): |
2685 | + self.conf.store._load_from_string('''\ |
2686 | +vm.release = raring |
2687 | +# Ensure options are properly expanded (and comments supported ;) |
2688 | +_archive_url = http://archive.ubuntu.com/ubuntu |
2689 | +_ppa_url = https://u:p@ppa.lp.net/user/ppa/ubuntu |
2690 | +vm.apt_sources = deb {_archive_url} {vm.release} partner,\ |
2691 | + deb {_archive_url} {vm.release} main,\ |
2692 | + deb {_ppa_url} {vm.release} main|ABCDEF |
2693 | +''') |
2694 | + self.ci_data.populate() |
2695 | + self.assertEqual( |
2696 | + [{'source': 'deb http://archive.ubuntu.com/ubuntu raring partner'}, |
2697 | + {'source': 'deb http://archive.ubuntu.com/ubuntu raring main'}, |
2698 | + {'source': |
2699 | + 'deb https://u:p@ppa.lp.net/user/ppa/ubuntu raring main', |
2700 | + 'keyid': 'ABCDEF'}], |
2701 | + self.ci_data.cloud_config['apt_sources']) |
2702 | + |
2703 | + def create_file(self, path, content): |
2704 | + with open(path, 'wb') as f: |
2705 | + f.write(content) |
2706 | + |
2707 | + def test_good_ssh_keys(self): |
2708 | + paths = ('rsa', 'rsa.pub', 'dsa', 'dsa.pub', 'ecdsa', 'ecdsa.pub') |
2709 | + for path in paths: |
2710 | + self.create_file(path, '%s\ncontent\n' % (path,)) |
2711 | + paths_as_list = ','.join(paths) |
2712 | + self.conf.store._load_from_string( |
2713 | + 'vm.ssh_keys = %s' % (paths_as_list,)) |
2714 | + self.ci_data.populate() |
2715 | + self.assertEqual({'dsa_private': 'dsa\ncontent\n', |
2716 | + 'dsa_public': 'dsa.pub\ncontent\n', |
2717 | + 'ecdsa_private': 'ecdsa\ncontent\n', |
2718 | + 'ecdsa_public': 'ecdsa.pub\ncontent\n', |
2719 | + 'rsa_private': 'rsa\ncontent\n', |
2720 | + 'rsa_public': 'rsa.pub\ncontent\n'}, |
2721 | + self.ci_data.cloud_config['ssh_keys']) |
2722 | + |
2723 | + def test_bad_type_ssh_keys(self): |
2724 | + self.conf.store._load_from_string('vm.ssh_keys = I-dont-exist') |
2725 | + self.assertRaises(setup_vm.ConfigValueError, self.ci_data.populate) |
2726 | + |
2727 | + def test_unknown_ssh_keys(self): |
2728 | + self.conf.store._load_from_string('vm.ssh_keys = rsa.pub') |
2729 | + self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate) |
2730 | + |
2731 | + def test_good_ssh_authorized_keys(self): |
2732 | + paths = ('home.pub', 'work.pub') |
2733 | + for path in paths: |
2734 | + self.create_file(path, '%s\ncontent\n' % (path,)) |
2735 | + paths_as_list = ','.join(paths) |
2736 | + self.conf.store._load_from_string( |
2737 | + 'vm.ssh_authorized_keys = %s' % (paths_as_list,)) |
2738 | + self.ci_data.populate() |
2739 | + self.assertEqual(['home.pub\ncontent\n', 'work.pub\ncontent\n'], |
2740 | + self.ci_data.cloud_config['ssh_authorized_keys']) |
2741 | + |
2742 | + def test_unknown_ssh_authorized_keys(self): |
2743 | + self.conf.store._load_from_string('vm.ssh_authorized_keys = rsa.pub') |
2744 | + self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate) |
2745 | + |
2746 | + def test_unknown_root_script(self): |
2747 | + self.conf.store._load_from_string('vm.root_script = I-dont-exist') |
2748 | + self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate) |
2749 | + |
2750 | + def test_unknown_ubuntu_script(self): |
2751 | + self.conf.store._load_from_string('vm.ubuntu_script = I-dont-exist') |
2752 | + self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate) |
2753 | + |
2754 | + def test_expansion_error_in_script(self): |
2755 | + root_script_content = '''#!/bin/sh |
2756 | +echo Hello {I_dont_exist} |
2757 | +''' |
2758 | + with open('root_script.sh', 'w') as f: |
2759 | + f.write(root_script_content) |
2760 | + self.conf.store._load_from_string('''\ |
2761 | +vm.root_script = root_script.sh |
2762 | +''') |
2763 | + e = self.assertRaises(errors.ExpandingUnknownOption, |
2764 | + self.ci_data.populate) |
2765 | + self.assertEqual(root_script_content, e.string) |
2766 | + |
2767 | + def test_unknown_uploaded_scripts(self): |
2768 | + self.conf.store._load_from_string( |
2769 | + '''vm.uploaded_scripts = I-dont-exist ''') |
2770 | + e = self.assertRaises(setup_vm.ConfigPathNotFound, |
2771 | + self.ci_data.populate) |
2772 | + |
2773 | + def test_root_script(self): |
2774 | + with open('root_script.sh', 'w') as f: |
2775 | + f.write('''#!/bin/sh |
2776 | +echo Hello {user} |
2777 | +''') |
2778 | + self.conf.store._load_from_string('''\ |
2779 | +vm.root_script = root_script.sh |
2780 | +user=root |
2781 | +''') |
2782 | + self.ci_data.populate() |
2783 | + # The additional newline after the script is expected |
2784 | + self.assertEqual('''\ |
2785 | +#!/bin/sh |
2786 | +cat >~root/setup_vm_post_install <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
2787 | +#!/bin/sh |
2788 | +echo Hello root |
2789 | + |
2790 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
2791 | +chown root:root ~root/setup_vm_post_install |
2792 | +chmod 0700 ~root/setup_vm_post_install |
2793 | +''', self.ci_data.root_hook) |
2794 | + self.assertEqual(['~root/setup_vm_post_install'], |
2795 | + self.ci_data.cloud_config['runcmd']) |
2796 | + |
2797 | + def test_ubuntu_script(self): |
2798 | + with open('ubuntu_script.sh', 'w') as f: |
2799 | + f.write('''#!/bin/sh |
2800 | +echo Hello {user} |
2801 | +''') |
2802 | + self.conf.store._load_from_string('''\ |
2803 | +vm.ubuntu_script = ubuntu_script.sh |
2804 | +user = ubuntu |
2805 | +''') |
2806 | + self.ci_data.populate() |
2807 | + # The additional newline after the script is expected |
2808 | + self.assertEqual('''\ |
2809 | +#!/bin/sh |
2810 | +cat >~ubuntu/setup_vm_post_install <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
2811 | +#!/bin/sh |
2812 | +echo Hello ubuntu |
2813 | + |
2814 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
2815 | +chown ubuntu:ubuntu ~ubuntu/setup_vm_post_install |
2816 | +chmod 0700 ~ubuntu/setup_vm_post_install |
2817 | +''', self.ci_data.ubuntu_hook) |
2818 | + # The command is run as root, so we need to 'su ubuntu' first |
2819 | + self.assertEqual([['su', '-l', |
2820 | + '-c', '~ubuntu/setup_vm_post_install', |
2821 | + 'ubuntu']], |
2822 | + self.ci_data.cloud_config['runcmd']) |
2823 | + |
2824 | + def test_uploaded_scripts(self): |
2825 | + paths = ('foo', 'bar') |
2826 | + for path in paths: |
2827 | + self.create_file(path, '%s\ncontent\n' % (path,)) |
2828 | + paths_as_list = ','.join(paths) |
2829 | + self.conf.store._load_from_string( |
2830 | + 'vm.uploaded_scripts = %s' % (paths_as_list,)) |
2831 | + self.ci_data.populate() |
2832 | + self.assertEqual('''\ |
2833 | +#!/bin/sh |
2834 | +cat >~ubuntu/setup_vm_uploads <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN' |
2835 | +mkdir -p ~ubuntu/bin |
2836 | +cd ~ubuntu/bin |
2837 | +cat >foo <<'EOFfoo' |
2838 | +foo |
2839 | +content |
2840 | + |
2841 | +EOFfoo |
2842 | +chmod 0755 foo |
2843 | +cat >bar <<'EOFbar' |
2844 | +bar |
2845 | +content |
2846 | + |
2847 | +EOFbar |
2848 | +chmod 0755 bar |
2849 | +EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN |
2850 | +chown ubuntu:ubuntu ~ubuntu/setup_vm_uploads |
2851 | +chmod 0700 ~ubuntu/setup_vm_uploads |
2852 | +''', |
2853 | + self.ci_data.uploaded_scripts_hook) |
2854 | + self.assertEqual([['su', '-l', |
2855 | + '-c', '~ubuntu/setup_vm_uploads', |
2856 | + 'ubuntu']], |
2857 | + self.ci_data.cloud_config['runcmd']) |
2858 | + |
2859 | + |
2860 | +class TestCreateUserData(TestCaseWithHome): |
2861 | + |
2862 | + def setUp(self): |
2863 | + super(TestCreateUserData, self).setUp() |
2864 | + self.conf = setup_vm.VmStack('foo') |
2865 | + self.vm = setup_vm.Kvm(self.conf) |
2866 | + |
2867 | + def test_empty_config(self): |
2868 | + config_dir = os.path.join(self.test_base_dir, 'config') |
2869 | + os.mkdir(config_dir) |
2870 | + # The config is *almost* empty, we need to set config_dir though as the |
2871 | + # user-data needs to be stored there. |
2872 | + self.conf.store._load_from_string('vm.config_dir=%s' % (config_dir,)) |
2873 | + self.vm.create_user_data() |
2874 | + self.assertTrue(os.path.exists(self.vm._config_dir)) |
2875 | + self.assertTrue(os.path.exists(self.vm._user_data_path)) |
2876 | + with open(self.vm._user_data_path) as f: |
2877 | + user_data = f.readlines() |
2878 | + # We care about the two first lines only here, checking the format (or |
2879 | + # cloud-init is confused) |
2880 | + self.assertEqual('#cloud-config-archive\n', user_data[0]) |
2881 | + self.assertEqual("- {content: '#cloud-config\n", user_data[1]) |
2882 | + |
2883 | + |
2884 | +class TestSeed(TestCaseWithHome): |
2885 | + |
2886 | + def setUp(self): |
2887 | + super(TestSeed, self).setUp() |
2888 | + self.conf = setup_vm.VmStack('foo') |
2889 | + self.vm = setup_vm.Kvm(self.conf) |
2890 | + images_dir = os.path.join(self.test_base_dir, 'images') |
2891 | + os.mkdir(images_dir) |
2892 | + config_dir = os.path.join(self.test_base_dir, 'config') |
2893 | + self.conf.store._load_from_string(''' |
2894 | +vm.name=foo |
2895 | +vm.release=raring |
2896 | +vm.config_dir=%s |
2897 | +vm.images_dir=%s |
2898 | +''' % (config_dir, images_dir,)) |
2899 | + |
2900 | + def test_create_meta_data(self): |
2901 | + self.vm.create_meta_data() |
2902 | + self.assertTrue(os.path.exists(self.vm._meta_data_path)) |
2903 | + |
2904 | + def test_create_user_data(self): |
2905 | + self.vm.create_user_data() |
2906 | + self.assertTrue(os.path.exists(self.vm._user_data_path)) |
2907 | + |
2908 | + def test_create_seed(self): |
2909 | + self.assertTrue(self.vm._seed_path is None) |
2910 | + self.vm.create_seed() |
2911 | + self.assertFalse(self.vm._seed_path is None) |
2912 | + self.assertTrue(os.path.exists(self.vm._seed_path)) |
2913 | + |
2914 | + |
2915 | +class TestImageFromCloud(TestCaseWithHome): |
2916 | + |
2917 | + def setUp(self): |
2918 | + super(TestImageFromCloud, self).setUp() |
2919 | + self.conf = setup_vm.VmStack('foo') |
2920 | + self.vm = setup_vm.KvmFromCloudImage(self.conf) |
2921 | + images_dir = os.path.join(self.test_base_dir, 'images') |
2922 | + os.mkdir(images_dir) |
2923 | + download_cache_dir = os.path.join(self.test_base_dir, 'download') |
2924 | + os.mkdir(download_cache_dir) |
2925 | + self.conf.store._load_from_string(''' |
2926 | +vm.name=foo |
2927 | +vm.release=raring |
2928 | +vm.images_dir=%s |
2929 | +vm.download_cache=%s |
2930 | +vm.cloud_image_name=fake.img |
2931 | +vm.disk_size=1M |
2932 | +''' % (images_dir, download_cache_dir)) |
2933 | + |
2934 | + def test_create_disk_image(self): |
2935 | + cloud_image_path = os.path.join(self.conf.get('vm.download_cache'), |
2936 | + self.conf.get('vm.cloud_image_name')) |
2937 | + # We need a fake cloud image that can be converted |
2938 | + setup_vm.run_subprocess( |
2939 | + ['sudo', 'qemu-img', 'create', |
2940 | + cloud_image_path, self.conf.get('vm.disk_size')]) |
2941 | + self.assertTrue(self.vm._disk_image_path is None) |
2942 | + self.vm.create_disk_image() |
2943 | + self.assertFalse(self.vm._disk_image_path is None) |
2944 | + self.assertTrue(os.path.exists(self.vm._disk_image_path)) |
2945 | + |
2946 | + |
2947 | +class TestImageWithBacking(TestCaseWithHome): |
2948 | + |
2949 | + def setUp(self): |
2950 | + (download_cache_dir, |
2951 | + reference_cloud_image_name) = requires_known_reference_image(self) |
2952 | + super(TestImageWithBacking, self).setUp() |
2953 | + # We'll share the images_dir between vms |
2954 | + images_dir = os.path.join(self.test_base_dir, 'images') |
2955 | + os.mkdir(images_dir) |
2956 | + # Create a shared config |
2957 | + conf = setup_vm.VmStack(None) |
2958 | + conf.store._load_from_string(''' |
2959 | +vm.release=raring |
2960 | +vm.images_dir=%s |
2961 | +vm.download_cache=%s |
2962 | +vm.disk_size=2G |
2963 | +[selftest-from-cloud] |
2964 | +vm.name=selftest-from-cloud |
2965 | +vm.cloud_image_name=%s |
2966 | +[selftest-backing] |
2967 | +vm.name=selftest-backing |
2968 | +vm.backing=selftest-from-cloud.qcow2 |
2969 | +''' % (images_dir, download_cache_dir, reference_cloud_image_name)) |
2970 | + conf.store.save() |
2971 | + # To bypass creating a real vm, we start from the cloud image that is a |
2972 | + # real and bootable one, so we just convert it. That also makes it |
2973 | + # available in vm.images_dir |
2974 | + temp_vm = setup_vm.KvmFromCloudImage( |
2975 | + setup_vm.VmStack('selftest-from-cloud')) |
2976 | + temp_vm.create_disk_image() |
2977 | + |
2978 | + def test_create_image_with_backing(self): |
2979 | + vm = setup_vm.KvmFromBacking(setup_vm.VmStack('selftest-backing')) |
2980 | + self.assertTrue(vm._disk_image_path is None) |
2981 | + vm.create_disk_image() |
2982 | + self.assertFalse(vm._disk_image_path is None) |
2983 | + self.assertTrue(os.path.exists(vm._disk_image_path)) |
2984 | + |
2985 | + |
2986 | +class TestVmStates(testtools.TestCase): |
2987 | + |
2988 | + def assertStates(self, expected, lines): |
2989 | + self.assertEqual(expected, setup_vm.vm_states(lines)) |
2990 | + |
2991 | + def test_empty(self): |
2992 | + self.assertStates({},[]) |
2993 | + |
2994 | + def test_garbage(self): |
2995 | + self.assertRaises(ValueError, self.assertStates, None, ['']) |
2996 | + |
2997 | + def test_known_states(self): |
2998 | + # From a real life sample |
2999 | + self.assertStates({'foo': 'shut off', 'bar': 'running'}, |
3000 | + ['- foo shut off', |
3001 | + '19 bar running']) |
3002 | + |
3003 | + |
3004 | +class TestConsoleParsing(testtools.TestCase): |
3005 | + |
3006 | + def _parse_console_monitor(self, string): |
3007 | + mon = setup_vm.ConsoleMonitor(StringIO(string)) |
3008 | + lines = [] |
3009 | + for line in mon.parse(): |
3010 | + lines.append(line) |
3011 | + return lines |
3012 | + |
3013 | + def test_fails_on_empty(self): |
3014 | + self.assertRaises(setup_vm.ConsoleEOFError, |
3015 | + self._parse_console_monitor, '') |
3016 | + |
3017 | + def test_fail_on_knwon_cloud_init_errors(self): |
3018 | + self.assertRaises( |
3019 | + setup_vm.CloudInitError, |
3020 | + self._parse_console_monitor, |
3021 | + 'Failed loading yaml blob\n') |
3022 | + self.assertRaises( |
3023 | + setup_vm.CloudInitError, |
3024 | + self._parse_console_monitor, |
3025 | + 'Unhandled non-multipart userdata starting\n') |
3026 | + self.assertRaises( |
3027 | + setup_vm.CloudInitError, |
3028 | + self._parse_console_monitor, |
3029 | + "failed to render string to stdout: cannot find 'uptime'\n") |
3030 | + self.assertRaises( |
3031 | + setup_vm.CloudInitError, |
3032 | + self._parse_console_monitor, |
3033 | + "Failed loading of cloud config " |
3034 | + "'/var/lib/cloud/instance/cloud-config.txt'. " |
3035 | + "Continuing with empty config\n") |
3036 | + |
3037 | + def test_succeds_on_final_message(self): |
3038 | + lines = self._parse_console_monitor(''' |
3039 | +Lalala |
3040 | +I'm doing my work |
3041 | +It goes nicely |
3042 | +setup_vm finished installing in 1 seconds. |
3043 | +That was fast isn't it ? |
3044 | + * Will now halt |
3045 | +[ 33.204755] Power down. |
3046 | +''') |
3047 | + # We stop as soon as we get the final message and ignore the rest |
3048 | + self.assertEquals(' * Will now halt\n', |
3049 | + lines[-1]) |
3050 | + |
3051 | + |
3052 | +class TestConsoleParsingWithFile(TestCaseWithHome): |
3053 | + |
3054 | + def _parse_file_monitor(self, string): |
3055 | + with open('console', 'w') as f: |
3056 | + f.write(string) |
3057 | + mon = setup_vm.FileMonitor('console') |
3058 | + for line in mon.parse(): |
3059 | + pass |
3060 | + return mon.lines |
3061 | + |
3062 | + def test_succeeds_with_file(self): |
3063 | + content = '''\ |
3064 | +Yet another install |
3065 | +Going well |
3066 | +setup_vm finished installing in 0.5 seconds. |
3067 | +Wow, even faster ! |
3068 | + * Will now halt |
3069 | +Whatever, won't read that |
3070 | +''' |
3071 | + lines = self._parse_file_monitor(content) |
3072 | + |
3073 | + def xtest_fails_on_empty_file(self): |
3074 | + # FIXME: We need some sort of timeout there... |
3075 | + self.assertRaises(setup_vm.CommandError, self._parse_file_monitor, '') |
3076 | + |
3077 | + def test_fail_on_knwon_cloud_init_errors_with_file(self): |
3078 | + self.assertRaises( |
3079 | + setup_vm.CloudInitError, |
3080 | + self._parse_file_monitor, |
3081 | + 'Failed loading yaml blob\n') |
3082 | + self.assertRaises( |
3083 | + setup_vm.CloudInitError, |
3084 | + self._parse_file_monitor, |
3085 | + 'Unhandled non-multipart userdata starting\n') |
3086 | + self.assertRaises( |
3087 | + setup_vm.CloudInitError, |
3088 | + self._parse_file_monitor, |
3089 | + "failed to render string to stdout: cannot find 'uptime'\n") |
3090 | + |
3091 | + |
3092 | +class TestInstallWithSeed(TestCaseWithHome): |
3093 | + |
3094 | + def setUp(self): |
3095 | + (download_cache, |
3096 | + reference_cloud_image_name) = requires_known_reference_image(self) |
3097 | + super(TestInstallWithSeed, self).setUp() |
3098 | + # We need to allow other users to read this dir |
3099 | + os.chmod(self.test_base_dir, 0755) |
3100 | + # We also need to sudo rm it as root created some files there |
3101 | + self.addCleanup( |
3102 | + setup_vm.run_subprocess, |
3103 | + ['sudo', 'rm', '-fr', |
3104 | + os.path.join(self.test_base_dir, 'home', '.virtinst')]) |
3105 | + self.conf = setup_vm.VmStack('selftest-seed') |
3106 | + self.vm = setup_vm.KvmFromCloudImage(self.conf) |
3107 | + images_dir = os.path.join(self.test_base_dir, 'images') |
3108 | + os.mkdir(images_dir, 0755) |
3109 | + config_dir = os.path.join(self.test_base_dir, 'config') |
3110 | + self.conf.store._load_from_string(''' |
3111 | +vm.name=selftest-seed |
3112 | +vm.update=False # Shorten install time |
3113 | +vm.cpus=2, |
3114 | +vm.release=raring |
3115 | +vm.config_dir=%s |
3116 | +vm.images_dir=%s |
3117 | +vm.download_cache=%s |
3118 | +vm.cloud_image_name=%s |
3119 | +vm.disk_size=8G |
3120 | +''' % (config_dir, images_dir, download_cache, reference_cloud_image_name)) |
3121 | + |
3122 | + def assertVmState(self, expected): |
3123 | + states = setup_vm.vm_states() |
3124 | + self.assertEqual(expected, states[self.vm.conf.get('vm.name')]) |
3125 | + |
3126 | + def test_install_with_seed(self): |
3127 | + self.addCleanup(self.vm.undefine) |
3128 | + self.vm.install() |
3129 | + self.assertVmState('shut off') |
3130 | + |
3131 | + |
3132 | +class TestInstallWithBacking(TestCaseWithHome): |
3133 | + |
3134 | + def setUp(self): |
3135 | + (download_cache_dir, |
3136 | + reference_cloud_image_name) = requires_known_reference_image(self) |
3137 | + super(TestInstallWithBacking, self).setUp() |
3138 | + # We need to allow other users to read this dir |
3139 | + os.chmod(self.test_base_dir, 0755) |
3140 | + # We also need to sudo rm it as root created some files there |
3141 | + self.addCleanup( |
3142 | + setup_vm.run_subprocess, |
3143 | + ['sudo', 'rm', '-fr', |
3144 | + os.path.join(self.test_base_dir, 'home', '.virtinst')]) |
3145 | + self.conf = setup_vm.VmStack('selftest-backing') |
3146 | + self.vm = setup_vm.KvmFromBacking(self.conf) |
3147 | + # We'll share the images_dir between vms |
3148 | + images_dir = os.path.join(self.test_base_dir, 'images') |
3149 | + os.mkdir(images_dir, 0755) |
3150 | + config_dir = os.path.join(self.test_base_dir, 'config') |
3151 | + # Create a shared config |
3152 | + conf = setup_vm.VmStack(None) |
3153 | + conf.store._load_from_string(''' |
3154 | +vm.release=raring |
3155 | +vm.config_dir=%s |
3156 | +vm.images_dir=%s |
3157 | +vm.download_cache=%s |
3158 | +vm.disk_size=2G |
3159 | +vm.update=False # Shorten install time |
3160 | +[selftest-from-cloud] |
3161 | +vm.name=selftest-from-cloud |
3162 | +vm.cloud_image_name=%s |
3163 | +[selftest-backing] |
3164 | +vm.name=selftest-backing |
3165 | +vm.backing=selftest-from-cloud.qcow2 |
3166 | +''' % (config_dir, images_dir, download_cache_dir, reference_cloud_image_name)) |
3167 | + conf.store.save() |
3168 | + # Fake a previous install by just re-using the reference cloud image |
3169 | + temp_vm = setup_vm.KvmFromCloudImage( |
3170 | + setup_vm.VmStack('selftest-from-cloud')) |
3171 | + temp_vm.create_disk_image() |
3172 | + |
3173 | + def assertVmState(self, vm, expected): |
3174 | + states = setup_vm.vm_states() |
3175 | + self.assertEqual(expected, states[vm.conf.get('vm.name')]) |
3176 | + |
3177 | + def test_install_with_backing(self): |
3178 | + vm = setup_vm.KvmFromBacking(setup_vm.VmStack('selftest-backing')) |
3179 | + self.addCleanup(vm.undefine) |
3180 | + vm.install() |
3181 | + self.assertVmState(vm, 'shut off') |
3182 | + |
3183 | + |
3184 | +class TestSshKeyGen(TestCaseWithHome): |
3185 | + |
3186 | + def setUp(self): |
3187 | + super(TestSshKeyGen, self).setUp() |
3188 | + self.conf = setup_vm.VmStack(None) |
3189 | + self.vm = setup_vm.VM(self.conf) |
3190 | + self.config_dir = os.path.join(self.test_base_dir, 'config') |
3191 | + |
3192 | + def load_config(self, more): |
3193 | + content = '''\ |
3194 | +vm.config_dir=%s |
3195 | +vm.name=foo |
3196 | +''' % (self.config_dir,) |
3197 | + self.conf.store._load_from_string(content + more) |
3198 | + |
3199 | + def generate_key(self, ssh_type, upper_type=None): |
3200 | + if upper_type is None: |
3201 | + upper_type = ssh_type.upper() |
3202 | + self.load_config('vm.ssh_keys={vm.config_dir}/%s' % (ssh_type,)) |
3203 | + self.vm.ssh_keygen() |
3204 | + private_path = 'config/%s' % (ssh_type,) |
3205 | + self.assertTrue(os.path.exists(private_path)) |
3206 | + public_path = 'config/%s.pub' % (ssh_type,) |
3207 | + self.assertTrue(os.path.exists(public_path)) |
3208 | + public = file(public_path).read() |
3209 | + private = file(private_path).read() |
3210 | + self.assertTrue(private.startswith( |
3211 | + '-----BEGIN %s PRIVATE KEY-----\n' % (upper_type,))) |
3212 | + self.assertTrue(private.endswith( |
3213 | + '-----END %s PRIVATE KEY-----\n' % (upper_type,))) |
3214 | + return private, public |
3215 | + |
3216 | + def test_dsa(self): |
3217 | + private, public = self.generate_key('dsa') |
3218 | + self.assertTrue(public.startswith('ssh-dss ')) |
3219 | + self.assertTrue(public.endswith(' foo\n')) |
3220 | + |
3221 | + def test_rsa(self): |
3222 | + private, public = self.generate_key('rsa') |
3223 | + self.assertTrue(public.startswith('ssh-rsa ')) |
3224 | + self.assertTrue(public.endswith(' foo\n')) |
3225 | + |
3226 | + def test_ecdsa(self): |
3227 | + private, public = self.generate_key('ecdsa', 'EC') |
3228 | + self.assertTrue(public.startswith('ecdsa-sha2-nistp256 ')) |
3229 | + self.assertTrue(public.endswith(' foo\n')) |
3230 | + |
3231 | + |
3232 | +class TestOptionParsing(testtools.TestCase): |
3233 | + |
3234 | + def setUp(self): |
3235 | + super(TestOptionParsing, self).setUp() |
3236 | + self.out = StringIO() |
3237 | + self.err = StringIO() |
3238 | + |
3239 | + def parse_args(self, args): |
3240 | + return setup_vm.arg_parser.parse_args(args, self.out, self.err) |
3241 | + |
3242 | + def test_nothing(self): |
3243 | + self.assertRaises(SystemExit, self.parse_args, []) |
3244 | + |
3245 | + def test_install(self): |
3246 | + ns = self.parse_args(['foo', '--install']) |
3247 | + self.assertEquals('foo', ns.name) |
3248 | + self.assertTrue(ns.install) |
3249 | + self.assertFalse(ns.download) |
3250 | + |
3251 | + def test_download(self): |
3252 | + ns = self.parse_args(['foo', '--download']) |
3253 | + self.assertEquals('foo', ns.name) |
3254 | + self.assertFalse(ns.install) |
3255 | + self.assertTrue(ns.download) |
3256 | + |
3257 | +class TestBuildCommands(testtools.TestCase): |
3258 | + |
3259 | + def setUp(self): |
3260 | + super(TestBuildCommands, self).setUp() |
3261 | + self.out = StringIO() |
3262 | + self.err = StringIO() |
3263 | + |
3264 | + def build_commands(self, args): |
3265 | + return setup_vm.build_commands(args, self.out, self.err) |
3266 | + |
3267 | + def test_install(self): |
3268 | + cmds = self.build_commands(['--install', 'foo']) |
3269 | + self.assertEqual(1, len(cmds)) |
3270 | + self.assertTrue(isinstance(cmds[0], setup_vm.Install)) |
3271 | + |
3272 | + def test_download(self): |
3273 | + cmds = self.build_commands(['--download', 'foo']) |
3274 | + self.assertEqual(1, len(cmds)) |
3275 | + self.assertTrue(isinstance(cmds[0], setup_vm.Download)) |
3276 | + |
3277 | + def test_ssh_keygen(self): |
3278 | + cmds = self.build_commands(['--ssh-keygen', 'foo']) |
3279 | + self.assertEqual(1, len(cmds)) |
3280 | + self.assertTrue(isinstance(cmds[0], setup_vm.SshKeyGen)) |
3281 | + |
3282 | + def test_download_and_install(self): |
3283 | + cmds = self.build_commands(['--install', '--download', 'foo']) |
3284 | + self.assertEqual(2, len(cmds)) |
3285 | + # Download comes first |
3286 | + self.assertTrue(isinstance(cmds[0], setup_vm.Download)) |
3287 | + self.assertTrue(isinstance(cmds[1], setup_vm.Install)) |
3288 | + |
3289 | + |
3290 | +# FIXME: This needs to be parametrized for KvmFromCloudImage and |
3291 | +# KvmFromBacking. Since we don't define vm.backing below, we're only testing |
3292 | +# KvmFromCloudImage for now. -- vila 2013-02-13 |
3293 | +class TestInstall(TestCaseWithHome): |
3294 | + |
3295 | + def setUp(self): |
3296 | + super(TestInstall, self).setUp() |
3297 | + self.conf = setup_vm.VmStack('I-dont-exist') |
3298 | + self.conf.store._load_from_string(''' |
3299 | +vm.name=I-dont-exist |
3300 | +vm.release=raring |
3301 | +vm.cpu_model=amd64 |
3302 | +''') |
3303 | + self.states = [] |
3304 | + |
3305 | + def vm_states(source=None): |
3306 | + return self.states |
3307 | + self.patch(setup_vm, 'vm_states', vm_states) |
3308 | + self.vm = None |
3309 | + |
3310 | + def install(self): |
3311 | + class FakeKvm(setup_vm.Kvm): |
3312 | + |
3313 | + def __init__(self, conf): |
3314 | + super(FakeKvm, self).__init__(conf) |
3315 | + self.undefine_called = False |
3316 | + self.install_called = False |
3317 | + |
3318 | + # Make sure we avoid dangerous or costly calls |
3319 | + def poweroff(self): |
3320 | + pass |
3321 | + |
3322 | + def undefine(self): |
3323 | + self.undefine_called = True |
3324 | + |
3325 | + def install(self): |
3326 | + self.install_called = True |
3327 | + |
3328 | + |
3329 | + self.vm = FakeKvm(self.conf) |
3330 | + cmd = setup_vm.Install(self.vm) |
3331 | + cmd.run() |
3332 | + |
3333 | + def test_install_while_running(self): |
3334 | + self.conf.set('vm.name', 'foo') |
3335 | + self.states = {'foo': 'running'} |
3336 | + self.assertRaises(setup_vm.SetupVmError, self.install) |
3337 | + self.assertFalse(self.vm.install_called) |
3338 | + self.assertFalse(self.vm.undefine_called) |
3339 | + |
3340 | + def test_install_unknown(self): |
3341 | + self.states = {} |
3342 | + self.install() |
3343 | + self.assertTrue(self.vm.install_called) |
3344 | + self.assertFalse(self.vm.undefine_called) |
3345 | + |
3346 | + def test_install_shutoff(self): |
3347 | + self.conf.set('vm.name', 'foo') |
3348 | + self.states = {'foo': 'shut off'} |
3349 | + self.install() |
3350 | + self.assertTrue(self.vm.install_called) |
3351 | + self.assertTrue(self.vm.undefine_called) |
3352 | |
3353 | === added file 'setup_vm/tests/test_test.py' |
3354 | --- setup_vm/tests/test_test.py 1970-01-01 00:00:00 +0000 |
3355 | +++ setup_vm/tests/test_test.py 2013-04-17 01:29:27 +0000 |
3356 | @@ -0,0 +1,58 @@ |
3357 | +import os |
3358 | + |
3359 | +import testtools |
3360 | + |
3361 | +import tests |
3362 | + |
3363 | + |
3364 | +def assertTestSuccess(test, inner): |
3365 | + """The received test runs successfully.""" |
3366 | + result = testtools.TestResult() |
3367 | + inner.run(result) |
3368 | + test.assertEqual(0, len(result.errors) + len(result.failures)) |
3369 | + test.assertEqual(1, result.testsRun) |
3370 | + return result |
3371 | + |
3372 | + |
3373 | +class TestEnv(testtools.TestCase): |
3374 | + |
3375 | + |
3376 | + def test_env_preserved(self): |
3377 | + os.environ['NOBODY_USES_THIS'] = 'foo' |
3378 | + |
3379 | + class Inner(testtools.TestCase): |
3380 | + |
3381 | + def test_overridden(self): |
3382 | + tests.isolate_env(self, {'NOBODY_USES_THIS': 'bar'}) |
3383 | + self.assertEqual('bar', os.environ['NOBODY_USES_THIS']) |
3384 | + |
3385 | + assertTestSuccess(self, Inner('test_overridden')) |
3386 | + self.assertEqual('foo', os.environ['NOBODY_USES_THIS']) |
3387 | + |
3388 | + def test_env_var_deleted(self): |
3389 | + os.environ['NOBODY_USES_THIS'] = 'foo' |
3390 | + |
3391 | + class Inner(testtools.TestCase): |
3392 | + |
3393 | + def test_deleted(self): |
3394 | + tests.isolate_env(self, {'NOBODY_USES_THIS': None}) |
3395 | + self.assertIs('deleted', |
3396 | + os.environ.get('NOBODY_USES_THIS', 'deleted')) |
3397 | + assertTestSuccess(self, Inner('test_deleted')) |
3398 | + self.assertEqual('foo', os.environ['NOBODY_USES_THIS']) |
3399 | + |
3400 | + |
3401 | +class TestTmp(testtools.TestCase): |
3402 | + |
3403 | + def test_cwd_in_tmp(self): |
3404 | + |
3405 | + class Inner(testtools.TestCase): |
3406 | + |
3407 | + def setUp(self): |
3408 | + super(Inner, self).setUp() |
3409 | + tests.set_cwd_to_tmp(self) |
3410 | + |
3411 | + def test_cwd_in_tmp(self): |
3412 | + self.assertEqual(os.getcwdu(), self.test_base_dir) |
3413 | + |
3414 | + assertTestSuccess(self, Inner('test_cwd_in_tmp')) |
3415 | |
3416 | === added directory 'setup_vm/u1' |
3417 | === added file 'setup_vm/u1/install' |
3418 | --- setup_vm/u1/install 1970-01-01 00:00:00 +0000 |
3419 | +++ setup_vm/u1/install 2013-04-17 01:29:27 +0000 |
3420 | @@ -0,0 +1,71 @@ |
3421 | +#!/bin/sh -ex |
3422 | + |
3423 | +# Allow ssh access to launchpad. |
3424 | +# This should probably be provided by setup_vm. -- vila 2013-03-10 |
3425 | +ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts |
3426 | +# Use the openjdk. |
3427 | +sudo update-alternatives --set java /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java |
3428 | +# Get the branch. |
3429 | +bzr branch lp:ubuntuone-servers {u1.src_dir} |
3430 | +# Setup the environment. |
3431 | +cd {u1.src_dir} |
3432 | +# Set up the correct configuration. |
3433 | +cat <<EOF >configs/local.conf |
3434 | +[meta] |
3435 | +extends: development-appserver-lazr.conf |
3436 | + |
3437 | +[general] |
3438 | +port: {u1.port} |
3439 | +django_module: u1servers.web.localsettings |
3440 | + |
3441 | +[upay] |
3442 | +consumer_id: U1 |
3443 | +port: {pay.port} |
3444 | +hostname: {pay.address} |
3445 | +url_format: http://%(host)s:%(port)d/api/2.0 |
3446 | + |
3447 | +[upay_u1ms] |
3448 | +consumer_id: U1 |
3449 | +port: {pay.port} |
3450 | +hostname: {pay.address} |
3451 | +url_format: http://%(host)s:%(port)d/api/2.0 |
3452 | + |
3453 | +[url] |
3454 | +openid_sso_server: {sso.url} |
3455 | + |
3456 | +EOF |
3457 | +# XXX The secrets file is overlayed, so we can't use the config file. |
3458 | +# This is an ugly way to overwrite the default values. |
3459 | +cat <<EOF >>configs/dev_secrets-lazr.conf |
3460 | +ubuntu_pay_username: u1qauser |
3461 | +ubuntu_pay_password: u1qapassword |
3462 | +ubuntu_pay_username_u1ms: u1qauser |
3463 | +ubuntu_pay_password_u1ms: u1qapassword |
3464 | + |
3465 | +EOF |
3466 | +cat <<EOF >servers/u1servers/web/localsettings.py |
3467 | +from u1servers.web.devsettings import * |
3468 | + |
3469 | +OPENID_SSO_SERVER_URL = config.url.openid_sso_server |
3470 | +OPENID_SSO_LOGOUT_URL = '%s/+logout?return_to=%s' % ( |
3471 | + OPENID_SSO_SERVER_URL, BASE_URL) |
3472 | + |
3473 | +if __name__ == os.environ.get("DJANGO_SETTINGS_MODULE"): |
3474 | + # This only gets executed if the configured DJANGO_SETTINGS_MODULE matches |
3475 | + # the current module name. |
3476 | + from ubuntuone import dispatch |
3477 | + dispatch.connect_all(async=True) |
3478 | + |
3479 | + from u1servers.web import email |
3480 | + email.connect_receivers() |
3481 | + |
3482 | + # Triggered when the env variable U1_PAY_HOST is defined with |
3483 | + # "<hostname>:<port>" |
3484 | + if config.upay.hostname or config.upay_u1ms.hostname: |
3485 | + from u1backends.account.upayclient import init_payclient |
3486 | + init_payclient() |
3487 | +EOF |
3488 | +make update-sourcedeps |
3489 | +# TODO ask on #u1-ops if there's a better way. |
3490 | +sed -i 's/development-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl |
3491 | +sed -i 's/development-appserver-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl |
3492 | |
3493 | === added file 'setup_vm/u1/run' |
3494 | --- setup_vm/u1/run 1970-01-01 00:00:00 +0000 |
3495 | +++ setup_vm/u1/run 2013-04-17 01:29:27 +0000 |
3496 | @@ -0,0 +1,4 @@ |
3497 | +#!/bin/sh |
3498 | + |
3499 | +cd ~/{u1.src_dir} |
3500 | +HOSTNAME={u1.address} DJANGO_SETTINGS_MODULE=u1servers.web.localsettings U1CONFIG=`pwd`/configs/local.conf make start |
3501 | |
3502 | === added file 'setup_vm/u1/test' |
3503 | --- setup_vm/u1/test 1970-01-01 00:00:00 +0000 |
3504 | +++ setup_vm/u1/test 2013-04-17 01:29:27 +0000 |
3505 | @@ -0,0 +1,15 @@ |
3506 | +#!/bin/sh |
3507 | + |
3508 | + |
3509 | +cd {u1.src_dir} |
3510 | +# When run from the host against the u1 guest: |
3511 | +# sudo apt-get install python-mocker |
3512 | +# scp ubuntu@{u1.address}:~/ubuntuone-servers/configs/local.conf configs/local.conf |
3513 | +# scp ubuntu@{u1.address}:~/ubuntuone-servers/servers/u1servers/web/localsettings.py servers/u1servers/web/localsettings.py |
3514 | +# TODO ask on #u1-ops if there's a better way. |
3515 | +# sed -i 's/development-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl |
3516 | +# sed -i 's/development-appserver-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl |
3517 | +# echo 9999 >tmp/statsd.port |
3518 | +# make update-sourcedeps |
3519 | +U1CONFIG=`pwd`/configs/local.conf make smoke-test |
3520 | +U1CONFIG=`pwd`/configs/local.conf make acceptance-test |
3521 | |
3522 | === added directory 'setup_vm/unity' |
3523 | === added file 'setup_vm/unity/install-sources' |
3524 | --- setup_vm/unity/install-sources 1970-01-01 00:00:00 +0000 |
3525 | +++ setup_vm/unity/install-sources 2013-04-17 01:29:27 +0000 |
3526 | @@ -0,0 +1,19 @@ |
3527 | +#!/bin/sh |
3528 | +# Allow ssh access to launchpad. |
3529 | +# This should probably be provided by setup_vm. -- vila 2013-03-10 |
3530 | +ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts |
3531 | +# Install sst and dependencies from source |
3532 | +mkdir src |
3533 | +cd src |
3534 | +bzr branch lp:~ubuntuone-hackers/ubuntuone-servers/selenium |
3535 | +bzr branch lp:~canonical-isd-qa/selenium-simple-test/trunk selenium-simple-test |
3536 | +cd ~/src/selenium |
3537 | +python setup.py install --user |
3538 | +cd ~/src/selenium-simple-test |
3539 | +python setup.py install --user |
3540 | + |
3541 | +# If the need arise to use sst-run, it may become necessary to create a |
3542 | +# symlink to /home/ubuntu/.local/bin/sst-run |
3543 | + |
3544 | +# Also note that unittest2 is installed as a side-effect of installing sst |
3545 | +# even if we're already using pyton-2.7 (which includes unittest2 features). |
3546 | |
3547 | === added file 'setup_vm/unity/run-sso-client' |
3548 | --- setup_vm/unity/run-sso-client 1970-01-01 00:00:00 +0000 |
3549 | +++ setup_vm/unity/run-sso-client 2013-04-17 01:29:27 +0000 |
3550 | @@ -0,0 +1,11 @@ |
3551 | +#!/bin/sh -ex |
3552 | + |
3553 | +# We can use U1_DEBUG=True to get debug messages on the console. |
3554 | +USSOC_SERVICE_URL={sso.url}/api/1.0/ /usr/lib/ubuntu-sso-client/ubuntu-sso-login & |
3555 | +# XXX ugly sleep. |
3556 | +sleep 5s |
3557 | +# TODO x86_64 sounds like trouble. |
3558 | +# This has just stopped working on raring. See http://pad.lv/1161067 |
3559 | +# TODO in order for the application to be accessible with testability, we need |
3560 | +# TESTABILITY=1 |
3561 | +/usr/lib/ubuntu-sso-client/ubuntu-sso-login-qt --app_name 'Ubuntu One' --help_text '...' --ping_url '{u1.url}/oauth/sso-finished-so-get-tokens/%7Bemail%7D?platform_version=3.8.0-2-generic&platform=Linux&client_version=4.1.90&platform_arch=x86_64' --policy_url {u1.url}/privacy/ --tc_url {u1.url}/terms/ |
3562 | |
3563 | === added file 'setup_vm/unity/run-syncdaemon' |
3564 | --- setup_vm/unity/run-syncdaemon 1970-01-01 00:00:00 +0000 |
3565 | +++ setup_vm/unity/run-syncdaemon 2013-04-17 01:29:27 +0000 |
3566 | @@ -0,0 +1,4 @@ |
3567 | +#!/bin/sh -ex |
3568 | + |
3569 | +/usr/lib/ubuntuone-client/ubuntuone-syncdaemon --disable_ssl_verify --dns_srv=None --host={filesync.address} & |
3570 | +u1sdtool --connect |
3571 | |
3572 | === added file 'setup_vm/unity/run-unity-lens-music' |
3573 | --- setup_vm/unity/run-unity-lens-music 1970-01-01 00:00:00 +0000 |
3574 | +++ setup_vm/unity/run-unity-lens-music 2013-04-17 01:29:27 +0000 |
3575 | @@ -0,0 +1,6 @@ |
3576 | +#!/bin/sh -ex |
3577 | + |
3578 | +# TODO x86_64 sounds like trouble. |
3579 | +# We can use G_MESSAGES_DEBUG=all to get debug messages on the console. |
3580 | +# TODO change the name of the environment variables, it's not just staging. |
3581 | +pkill unity-music; U1_STAGING_WEBAPI={u1.url} U1_STAGING_AUTHENTICATION={sso.url} /usr/lib/x86_64-linux-gnu/unity-lens-music/unity-musicstore-daemon |
3582 | |
3583 | === added file 'setup_vm/unity/transient-dist-upgrade' |
3584 | --- setup_vm/unity/transient-dist-upgrade 1970-01-01 00:00:00 +0000 |
3585 | +++ setup_vm/unity/transient-dist-upgrade 2013-04-17 01:29:27 +0000 |
3586 | @@ -0,0 +1,4 @@ |
3587 | +#!/bin/sh |
3588 | +# For an unclear reason (probably a transient raring issue) we need to |
3589 | +# dist-upgrade instead of just upgrade |
3590 | +apt-get dist-upgrade -y |
3591 | |
3592 | === added file 'setup_vm/vms.conf' |
3593 | --- setup_vm/vms.conf 1970-01-01 00:00:00 +0000 |
3594 | +++ setup_vm/vms.conf 2013-04-17 01:29:27 +0000 |
3595 | @@ -0,0 +1,96 @@ |
3596 | +# This must be defined in some other vms.conf file (user or system) |
3597 | +# sso.address=sso.local |
3598 | +# pay.address=pay.local |
3599 | +# u1.address=u1.local |
3600 | +# ppa.ubuntuone-hackers.password |
3601 | + |
3602 | +sso.src_dir=canonical-identity-provider |
3603 | +sso.port=8001 |
3604 | +sso.url=http://{sso.address}:{sso.port} |
3605 | +sso.imap_port=2143 |
3606 | +sso.smtp_port=2025 |
3607 | + |
3608 | +pay.src_dir=canonical-payment-service |
3609 | +pay.port=8002 |
3610 | +pay.url=http://{pay.address}:{pay.port} |
3611 | + |
3612 | +ppa.ubuntuone_hackers=deb https://{vm.launchpad_id}:{ppa.ubuntuone_hackers.password}@private-ppa.launchpad.net/ubuntuone/hackers/ubuntu {vm.release} main|4BD0ECAE |
3613 | + |
3614 | +u1.src_dir=ubuntuone-servers |
3615 | +u1.port=8003 |
3616 | +u1.url=http://{u1.address}:{u1.port} |
3617 | + |
3618 | +[precise-server-pristine] |
3619 | +vm.name=precise-server-pristine |
3620 | +vm.release=precise |
3621 | +vm.packages=bzr, avahi-daemon, emacs23 |
3622 | +vm.update=True |
3623 | + |
3624 | +[sso] |
3625 | +vm.name=sso |
3626 | +vm.release=precise |
3627 | +vm.backing=precise-server-pristine.qcow2 |
3628 | +vm.packages=config-manager, fabric, libpq-dev, make, memcached, postgresql-plpython, python-m2crypto, python-dev, python-setuptools, python-virtualenv, swig, wget, libxml2-dev, libxslt1-dev |
3629 | +vm.ubuntu_script=sso/install |
3630 | +vm.update=True |
3631 | +vm.uploaded_scripts=sso/run, sso/run-for-pay, sso/run-for-u1 |
3632 | + |
3633 | +[pay] |
3634 | +vm.name=pay |
3635 | +vm.release=precise |
3636 | +vm.backing=precise-server-pristine.qcow2 |
3637 | +vm.packages=config-manager, fabric, libpq-dev, make, postgresql-plpython, python-dev, python-setuptools, python-virtualenv, wget, libxml2-dev, libxslt1-dev |
3638 | +vm.ubuntu_script=pay/install |
3639 | +vm.update=True |
3640 | +vm.uploaded_scripts=pay/run, pay/run-for-u1 |
3641 | + |
3642 | +[u1] |
3643 | +vm.name=u1 |
3644 | +vm.release=precise |
3645 | +vm.backing=precise-server-pristine.qcow2 |
3646 | +vm.apt_sources={ppa.ubuntuone_hackers} |
3647 | +vm.packages=openjdk-7-jre,ubuntuone-developer-dependencies |
3648 | +vm.ubuntu_script=u1/install |
3649 | +vm.update=True |
3650 | +vm.uploaded_scripts=u1/run |
3651 | + |
3652 | +[raring-desktop-pristine] |
3653 | +vm.name=raring-desktop-pristine |
3654 | +vm.release=raring |
3655 | +# python-unittest2 is not strictly required here but works around sst |
3656 | +# insisting on installing it locally. |
3657 | +vm.packages=bzr, emacs23, python-setuptools, python-unittest2, python-autopilot, unity-autopilot, ubuntu-desktop, avahi-daemon |
3658 | +vm.update=True |
3659 | +# Roughly all vms installing ubuntu-desktop need to complete the |
3660 | +# installation by making the ubuntu user part of the admin group. |
3661 | +vm.root_script = bin/ubuntu_admin.sh |
3662 | + |
3663 | +[purchase-testing] |
3664 | +vm.name=purchase-testing |
3665 | +vm.release=raring |
3666 | +vm.backing=raring-desktop-pristine.qcow2 |
3667 | +vm.apt_sources=deb http://ppa.launchpad.net/ubuntuone/dashpurchase-testing/ubuntu {vm.release} main|4BD0ECAE |
3668 | +vm.update=True |
3669 | +vm.ubuntu_script=purchase-testing/install |
3670 | + |
3671 | +[unity-prevalidation] |
3672 | +vm.name=unity-prevalidation |
3673 | +vm.release=raring |
3674 | +vm.backing=raring-desktop-pristine.qcow2 |
3675 | +vm.apt_sources=deb http://ppa.launchpad.net/ubuntu-unity/experimental-prevalidation/ubuntu {vm.release} main|52D62F45 |
3676 | +vm.uploaded_scripts=unity/run-sso-client, unity/run-unity-lens-music |
3677 | +# TODO unity/run-syncdaemon. We don't yet have the hermetic filesync server. |
3678 | +vm.update=True |
3679 | +vm.root_script=unity/transient-dist-upgrade |
3680 | +vm.ubuntu_script=unity/install-sources |
3681 | + |
3682 | +[indash-didrocks] |
3683 | +vm.name=indash-didrocks |
3684 | +vm.release=raring |
3685 | +vm.backing=raring-desktop-pristine.qcow2 |
3686 | +vm.apt_sources=ppa:didrocks/ppa |
3687 | +vm.uploaded_scripts=unity/run-sso-client, unity/run-unity-lens-music |
3688 | +# TODO unity/run-syncdaemon. We don't yet have the hermetic filesync server. |
3689 | +vm.update=True |
3690 | +vm.root_script=unity/transient-dist-upgrade |
3691 | +vm.ubuntu_script=unity/install-sources |
Currently blocked by: https:/ /bugs.launchpad .net/ubuntuone- servers/ +bug/1169218
We can work it around making a new Consumer and User on Ubuntu Pay.