Merge lp:~jtv/maas/import-to-tftp into lp:maas/trunk
| Status: | Merged |
|---|---|
| Approved by: | Jeroen T. Vermeulen on 2012-06-15 |
| Approved revision: | 640 |
| Merged at revision: | 649 |
| Proposed branch: | lp:~jtv/maas/import-to-tftp |
| Merge into: | lp:maas/trunk |
| Diff against target: |
426 lines (+396/-3) 3 files modified
HACKING.txt (+3/-3) scripts/maas-import-pxe-files (+191/-0) src/maas/tests/test_maas_import_pxe_files.py (+202/-0) |
| To merge this branch: | bzr merge lp:~jtv/maas/import-to-tftp |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Raphaël Badin (community) | 2012-06-15 | Approve on 2012-06-15 | |
|
Review via email:
|
|||
Commit Message
Script to import kernels/initrds for install netboots.
Description of the Change
See the “PXE boot requirements” document for the reasoning and conventions behind this branch. The branch adds a script that downloads the files that MAAS needs to provide over TFTP for a node to boot into the installer.
(Only the kernel and initrd are needed for this use-case; the minimal netbooted system will download the installer from an Ubuntu archive and start executing it.)
In a Cobblerless world this script replaces most, but not quite all, of what the existing maas-import-isos script does. There's still the preseeds and kernel parameters to be dealt with, but I think those will need to be done with knowledge from the database. My guess right now is that it can be done separately from the downloads.
There is still one problem with this script: a race condition. Updates are skipped when there are no changes, and the files are replaced as near-atomically as possible, but that doesn't help downloads that are ongoing at the time. Even if tftpd keeps file handles open for ongoing downloads (not too likely for a stateless UDP-based protocol!) so that each file will produce a consistent byte stream, the download is still 2 files, so a node could still see a mismatch between the kernel and the initrd. We agreed to tolerate this for now, but I'll file a bug.
For testing purposes I had to introduce a test dependency on curl. Amazingly, wget does not support the “file://” URLs I needed for my test. So for testing (and testing only) I substitute curl. The script is written such that it should work with either.
Jeroen
| Robie Basak (racb) wrote : | # |
| Raphaël Badin (rvb) wrote : | # |
Looks good. A few remarks:
[0]
99 + if [ -f $dest/pxelinux.0 ] && cmp pxelinux.0 $dest/pxelinux.0
I think you want to add "--quiet" to "cmp pxelinux.0 $dest/pxelinux.0" since you only care about the return code.
[1]
124 + rm -rf -- $dest/$name.old $dest/$name.new
How about using '-f' when moving files instead of cleaning up the files?
[2]
125 + # Put new downloads next to the old. If any file copying is needed
You've written the doc for this method without mentioning 'downloads'. I realize in practice this is used to move downloaded files but still, for consistency, I think you should s/downloads/
[3]
192 + cd -- $DOWNLOAD_DIR
Please consider using pushd/popd to restore go back to the original directory just before the script exits.
[4]
200 + # Replace the "base" with sub-architecture once we start supporting
201 + # those.
202 + arch_dir=
203 +
204 + mkdir -p -- $arch_dir
[…]
209 + rel_dir=
210 + mkdir -p -- $rel_dir
211 + update_
There is an additional white space on lines 202, 209 and 210.
[5]
259 + with open(os.
Please consider using "open(os.
[6]
196 + umask a+r
That looks like an important feature of the script. Don't you think it's worth a test?
[7]
66 +TFTPROOT=
and
202 + arch_dir=
arch_dir will be /var/lib/
[8]
Same "problem" with the download path: it downloads http://
[9]
54 +# Ubuntu releases that are to be downloaded.
55 +SUPPORTED_
Just a question here, is MAAS really supporting pre-precise releases?
[10]
70 +DOWNLOAD=
You've used --no-verbose and not --quiet here. This means that the output of this script looks like this: http://
[11]
I've noticed you've left the existing 'maas-import-isos' untouched. I understand we probably want to keep it around until the Cobbler replacement is done. But don't you think we should file bugs to make sure we don't forget to cleanup things when the replacement will be completed? I'd like your opinion on this because I've got the exact same dilemma about preseed templates/snippets.
| Jeroen T. Vermeulen (jtv) wrote : | # |
Thanks for the unexpected additional review. I've made the changes you asked for:
> Can we please use "generic" instead of "base"? "base" sounds like it is used
> as the foundation for something else which I think is a bit misleading. The
> armhf subarchitectures correspond to kernel flavours and "generic" is the
> standard i386/amd64 flavour that we are moving to so I think this makes sense.
Just goes to show that no matter how many people you sanity-check with, there's always something you don't know about. I updated the code, and the PXE requirements document as well.
> ARCHIVE=${ARCHIVE:-http://
> dists at the end - the standard way of specifying an archive is one level
> higher than this, without the dists. Then compose_
> insert "dists" into the URL instead.
Done. Gratifying to see how few changes that took.
You also mentioned some changes you might want to undertake yourself:
> The list of supported arches will need to grow from amd64 and i386 to
> armhf/highbank, armhf/armadaxp etc. At this point, it might be cleanest to
> specify ARCHES as "amd64/generic i386/generic armhf/highbank" etc.
Yes, something like that. Right now we can't quite do that because these settings are shared with the old, Cobbler-based scripts.
> The archive needs to be selected based on the architecture in use, in order to
> switch from archive.u.c/ubuntu to ports.u.
> into a function, rather than directly a variable?
I suppose. But it's a worry for later... Other solutions may become apparent in the meantime.
> compose_
> URL differs based on arch and release. The needed URLs don't necessary end in
> pxelinux.0, linux and initrd.gz though, so I think this needs to be a bit more
> specialised and return the actual URL of the kernel image, initrd image and
> pxelinux.0 instead. For example: the armadaxp kernel image is called uImage
> because it is in a different format.
A bit disheartening, but I guess we'll just have to deal with it when the time comes.
Jeroen
| Jeroen T. Vermeulen (jtv) wrote : | # |
And on to Raphaël's review!
> 99 + if [ -f $dest/pxelinux.0 ] && cmp pxelinux.0 $dest/pxelinux.0
>
> I think you want to add "--quiet" to "cmp pxelinux.0 $dest/pxelinux.0" since
> you only care about the return code.
Oh yes, absolutely. That also means I no longer need to check for the file's existence. I only did that to suppress the output — the documentation for --quiet suggested that the error message for a missing file would still be there, as you'd normally expect anyway, so I didn't think to use it.
> [1]
>
> 124 + rm -rf -- $dest/$name.old $dest/$name.new
>
> How about using '-f' when moving files instead of cleaning up the files?
Won't work for (non-empty) directories. There's even an option to make “mv” act as if the destination were a file, but I couldn't find a way to make that do what I wanted either.
> [2]
>
> 125 + # Put new downloads next to the old. If any file copying is
> needed
>
> You've written the doc for this method without mentioning 'downloads'. I
> realize in practice this is used to move downloaded files but still, for
> consistency, I think you should s/downloads/
Oops. Older incarnation of the code, when this wasn't extracted yet. Fixed.
> [3]
>
> 192 + cd -- $DOWNLOAD_DIR
>
> Please consider using pushd/popd to restore go back to the original directory
> just before the script exits.
I did consider it at the time, but bear in mind that “cd” will only affect the process of the shell that was forked for the purpose of running the script. The caller isn't affected.
> [4]
>
> 200 + # Replace the "base" with sub-architecture once we start
> supporting
> 201 + # those.
> 202 + arch_dir=
> 203 +
> 204 + mkdir -p -- $arch_dir
> […]
> 209 + rel_dir=
> 210 + mkdir -p -- $rel_dir
> 211 + update_
>
> There is an additional white space on lines 202, 209 and 210.
Took me a while to figure it out, since I didn't find any extra space in my editor: I got tabs I never asked for! All fixed now.
> [5]
>
> 259 + with open(os.
>
> Please consider using "open(os.
> compatibility.
Compatibility with what? Windows? Python 3? Unusual encodings? Given that this is for ASCII text files written from the same tests, what's the benefit of treating them as binary files?
> [6]
>
> 196 + umask a+r
>
> That looks like an important feature of the script. Don't you think it's
> worth a test?
Not particularly, no. It just means that the script will write world-readable files even if your umask doesn't normally do that. It doesn't affect its environment in any other way.
> [7]
>
> 66 +TFTPROOT=
> and
> 202 + arch_dir=
>
> arch_dir will be /var/lib/
> problem but maybe you'll want to remove that '//'.
I felt the trailing slash on $TFTPROOT was clearer, but if you have a problem...
| MAAS Lander (maas-lander) wrote : | # |
There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.
| Raphaël Badin (rvb) wrote : | # |
> Compatibility with what? Windows? Python 3? Unusual encodings? Given that this is for ASCII text files written from
> the same tests, what's the benefit of treating them as binary files?
Yes, Windows. I guess it's a little bit stupid in this instance as you mention but nonetheless, it's probably a good habit.
> Remember: cleanup is a noun, clean up is a verb!
Rarg!
> I don't like to worry too much about these small things in advance: if cleanup fits in naturally with the other work,
> we'll just do it and not worry about it. If we discover that it doesn't, then that is a good time to file a bug
> or a card so we don't remember. Until then, piling up detailed bugs or cards will just cloud our view of the work
> ahead. And as always: no matter how clear-cut a remote problem seems now, expect our view of it to change by the
> time it becomes relevant.
Fair enough.


This is really readable and shouldn't be too difficult to add ARM support to cleanly - thanks!
Can we please use "generic" instead of "base"? "base" sounds like it is used as the foundation for something else which I think is a bit misleading. The armhf subarchitectures correspond to kernel flavours and "generic" is the standard i386/amd64 flavour that we are moving to so I think this makes sense.
ARCHIVE=${ARCHIVE:-http:// archive. ubuntu. com/ubuntu/ dists/} should skip the dists at the end - the standard way of specifying an archive is one level higher than this, without the dists. Then compose_ installer_ download_ url would insert "dists" into the URL instead.
ARM support:
I'm happy to work on these as a separate merge request. Here are the changes I think we'll need:
The list of supported arches will need to grow from amd64 and i386 to armhf/highbank, armhf/armadaxp etc. At this point, it might be cleanest to specify ARCHES as "amd64/generic i386/generic armhf/highbank" etc.
The archive needs to be selected based on the architecture in use, in order to switch from archive.u.c/ubuntu to ports.u. c/ubuntu- ports. Can this be made into a function, rather than directly a variable?
compose_ installer_ download_ url will need to use a lookup table as the exact URL differs based on arch and release. The needed URLs don't necessary end in pxelinux.0, linux and initrd.gz though, so I think this needs to be a bit more specialised and return the actual URL of the kernel image, initrd image and pxelinux.0 instead. For example: the armadaxp kernel image is called uImage because it is in a different format.