Merge ~redriver/cloud-init:frbsd-azure-branch into cloud-init:master
- Git
- lp:~redriver/cloud-init
- frbsd-azure-branch
- Merge into master
Status: | Merged |
---|---|
Merged at revision: | 0a71d5a870b416f2c86c8bc196004bb3fc0768a0 |
Proposed branch: | ~redriver/cloud-init:frbsd-azure-branch |
Merge into: | cloud-init:master |
Diff against target: |
1374 lines (+782/-87) 22 files modified
cloudinit/config/cc_resizefs.py (+95/-0) cloudinit/distros/__init__.py (+3/-0) cloudinit/distros/freebsd.py (+261/-16) cloudinit/settings.py (+1/-1) cloudinit/sources/DataSourceAzure.py (+169/-11) cloudinit/sources/helpers/azure.py (+10/-1) cloudinit/stages.py (+1/-1) cloudinit/util.py (+51/-1) config/cloud.cfg-freebsd (+1/-1) setup.py (+6/-4) sysvinit/freebsd/cloudconfig (+0/-10) sysvinit/freebsd/cloudfinal (+0/-10) sysvinit/freebsd/cloudinit (+0/-10) sysvinit/freebsd/cloudinitlocal (+1/-11) tests/unittests/test_datasource/test_azure.py (+65/-0) tests/unittests/test_datasource/test_azure_helper.py (+2/-2) tests/unittests/test_datasource/test_cloudstack.py (+5/-0) tests/unittests/test_distros/test_netconfig.py (+46/-1) tests/unittests/test_handler/test_handler_resizefs.py (+59/-0) tests/unittests/test_net.py (+3/-3) tests/unittests/test_util.py (+2/-1) tools/build-on-freebsd (+1/-3) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Scott Moser | Approve | ||
Server Team CI bot | continuous-integration | Approve | |
Review via email: mp+314895@code.launchpad.net |
Commit message
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The modifications abide by the rule of
(1) making as less modification as possible
(2) delegating the platform or cloud specific operations to the specific classes instead of adding platform checking.
The main modifications constitute the following items:
1. Commands for network configuration are different from the ones on Ubuntu, for example, get ipv4/ipv6/mac, so, some of the functions called by apply_network_
2. Password setting through "pw" can only work through pipe, like "echo PASSWD|pw ... -H 0".
3. Default group for root user is "wheel" on FreeBSD. So, 'root:wheel' should be added syslog_fix_perms field.
4. There are some changes related to FreeBSD on Azure (DataSourceAzure):
(a) The leases file for dhclient is different from the one on Linux, so, the lease file path should be generated according to platform. The other similar platform related values: default primary network interface (hn0)and default file system (ufs)
(b) Mounting issues, including mounting CD, and get mount information
(c) Azure protocol depends on a field of dhclient lease file, but its name is different on FreeBSD. (azure.py, "option-245" vs. "unknown-245")
(d) Method to find resource (ephemeral) disk is a little complex because FreeBSD does not provide /dev/disk/
(e) Add a cloud.cfg for FreeBSD on Azure. Azure sets user information through ovf file, so the default configuration should not contain any user information.
Some of the features have not been fully implemented, for example, get seed from "/sys/firmware/
Description of the change
Scott Moser (smoser) wrote : | # |
Scott Moser (smoser) wrote : | # |
Note, I really dont have a lot of experience on FreeBSD. I relied heavily
on others who added freebsd support to tell me what made sense, and I'll do
the same for you.
Thank you for your efforts here... There are definitely some things we need
to re-work to better support different "distros" throughout the code base.
We'll see these same sorts of things as aixtools is working on getting better
aix support in.
One such example is your moving generate_
Overall, comments, much of the code needs some unit tests. We simply can't
avoid breaking things inadvertently without them get_resource_
and the networking code specifically.
Please bear with me... you have a lot of good things here, we just need
to work out some details.
- adding the specific config/
I'd much prefer we adjust the already freebsd specific cloud.cfg-freebsd.
The changes you have there are:
- add Azure to the datasource_list. I think its right to just put
Azure before Openstack and Ec2 so we have the list of 4 sources.
- you disabled growpart and resizefs
I wonder why these were not disabled by the original freebsd work.
Can you explain that? If they just aren't going to work, then lets
remove them from the list and update the docstring to mention lack
of support on freebsd. If they are only broken on Azure, lets figure
out why.
- default user changes
- removing of the default 'name'. This should not be necessary as
linux uses the datasource provided username on azure when its there.
- changing of shell to csh: my understanding (per git log) is that tcsh
is the default shell on freebsd, so using that makes more sense.
if Azure wishes to carry a different default shell, they can do that
in local config. Generally though, commonality is better.
- removal of lock_passwd: azure can carry that locally if they want.
- resizefs: it seems like you made some changes for this
in an attempt to make it work, but then disabled it in your
config. If it isn't going to work, lets not make ancillary changes there.
these include
tests/
cloudinit/
get_mount_info (i think)
I'm not opposed to doing this stuff at some point, but just want to make
sure all changes are intended.
- you moved 'get_mount_info' from the distro to the datasource?
that doesn't seem right. Possibly related only to resizefs.
- why the changes set_passwd?
You made 2 changes here:
a.) use -H 0 and -h 0 rather than just -H or -h
I'm fine to be explicit, but it seems like it shouldnt be necessary
or it would never have worked.
b.) you use a shell to 'echo ... |' when we were previously just
providing the password as standard in to the command (via subp 'data').
the shell is more processes to do the same work, did it not work for
you before?
- is there a reason you are not using set_passwd from add_user ?
- util.py:
freebsd really calls its x86_64 arch 'amd64' ? wow. maybe at this point we
should just s...
Jake Smith (c0by) wrote : | # |
Hi,
I have been doing some testing on FreeBSD.
The current growpart seems to work fine with FreeBSD 11.0 with GPT/UFS - the partition gets resized. The issue is with filesystem resize in resizefs. Its a simple fix, before you can modify the MBR you must first set kern.geom.
Basically this, I did a quick hack to test and seems ok:
current_debugflags = sysctl -n kern.geom.
sysctl kern.geom.
growfs -y /dev/vtbd0p2
sysctl kern.geom.
Hongjiang Zhang (redriver) wrote : | # |
Thanks for you careful review and valuable comments.
> Note, I really dont have a lot of experience on FreeBSD. I relied heavily
> on others who added freebsd support to tell me what made sense, and I'll do
> the same for you.
That is ok for me.
> Thank you for your efforts here... There are definitely some things we need
> to re-work to better support different "distros" throughout the code base.
> We'll see these same sorts of things as aixtools is working on getting better
> aix support in.
>
> One such example is your moving generate_
I agree. Azure supports both Linux and FreeBSD, so for the same data source, there are two distros and different distros sometimes have different native commands. So, we need to consider how to support them in an elegant way.
> Overall, comments, much of the code needs some unit tests. We simply can't
> avoid breaking things inadvertently without them get_resource_
> and the networking code specifically.
All right. I'll add some unit tests.
> Please bear with me... you have a lot of good things here, we just need
> to work out some details.
>
> - adding the specific config/
> I'd much prefer we adjust the already freebsd specific cloud.cfg-freebsd.
>
> The changes you have there are:
> - add Azure to the datasource_list. I think its right to just put
> Azure before Openstack and Ec2 so we have the list of 4 sources.
Ok.
> - you disabled growpart and resizefs
> I wonder why these were not disabled by the original freebsd work.
> Can you explain that? If they just aren't going to work, then lets
> remove them from the list and update the docstring to mention lack
> of support on freebsd. If they are only broken on Azure, lets figure
> out why.
On Azure, FreeBSD image used a fixed size VHD, so gpart is not necessary. Anyway, there is some error reported in the log if I enabled it. For the same reason, I disabled resizefs on Azure. Well, if we decide to use the same cloud-init.cfg for FreeBSD. I have to fix those error message and enable growpart and resizefs.
> - default user changes
> - removing of the default 'name'. This should not be necessary as
> linux uses the datasource provided username on azure when its there.
> - changing of shell to csh: my understanding (per git log) is that tcsh
> is the default shell on freebsd, so using that makes more sense.
> if Azure wishes to carry a different default shell, they can do that
> in local config. Generally though, commonality is better.
> - removal of lock_passwd: azure can carry that locally if they want.
For unknown reason, if I did not remove the default 'name' on Azure, it will stop the data source provided username. Ok, that may be a bug on Azure. In addition, the default username 'freebsd' is created but locked passwd, then why do we need it?
>
> - resizefs: it seems like you made some changes for this
> in an attempt to make it work, but then disabled it in your
> config. If it isn't going to work, lets not make ancillary changes there.
> these include
> tests/unit...
Scott Moser (smoser) wrote : | # |
> Thanks for you careful review and valuable comments.
>
> > Overall, comments, much of the code needs some unit tests. We simply can't
> > avoid breaking things inadvertently without them
> get_resource_
> > and the networking code specifically.
> All right. I'll add some unit tests.
Great.
>
> > Please bear with me... you have a lot of good things here, we just need
> > to work out some details.
> >
> > - adding the specific config/
> > I'd much prefer we adjust the already freebsd specific cloud.cfg-freebsd.
> >
> > The changes you have there are:
> > - add Azure to the datasource_list. I think its right to just put
> > Azure before Openstack and Ec2 so we have the list of 4 sources.
> Ok.
> > - you disabled growpart and resizefs
> > I wonder why these were not disabled by the original freebsd work.
> > Can you explain that? If they just aren't going to work, then lets
> > remove them from the list and update the docstring to mention lack
> > of support on freebsd. If they are only broken on Azure, lets figure
> > out why.
> On Azure, FreeBSD image used a fixed size VHD, so gpart is not necessary.
> Anyway, there is some error reported in the log if I enabled it. For the same
> reason, I disabled resizefs on Azure. Well, if we decide to use the same
> cloud-init.cfg for FreeBSD. I have to fix those error message and enable
> growpart and resizefs.
OK, Hopefully we could use Jake's suggestion above and get resizefs and growpart working even if they aren't needed on azure. They're in general good things.
> > - default user changes
> > - removing of the default 'name'. This should not be necessary as
> > linux uses the datasource provided username on azure when its there.
> > - changing of shell to csh: my understanding (per git log) is that tcsh
> > is the default shell on freebsd, so using that makes more sense.
> > if Azure wishes to carry a different default shell, they can do that
> > in local config. Generally though, commonality is better.
> > - removal of lock_passwd: azure can carry that locally if they want.
> For unknown reason, if I did not remove the default 'name' on Azure, it will
> stop the data source provided username. Ok, that may be a bug on Azure. In
> addition, the default username 'freebsd' is created but locked passwd, then
> why do we need it?
Maybe i wasn't clear. I was just saying that this all works as expected on Ubuntu (linux). We have a default config with a user 'ubuntu', but on azure, where a user is provided via the datasource, it overrides just that. So this should be able to work correctly.
> > - why the changes set_passwd?
> > You made 2 changes here:
> > a.) use -H 0 and -h 0 rather than just -H or -h
> > I'm fine to be explicit, but it seems like it shouldnt be necessary
> > or it would never have worked.
> > b.) you use a shell to 'echo ... |' when we were previously just
> > providing the password as standard in to the command (via subp
> > 'data').
> > the shell is more processes to do the same work, did it not work for
...
Hongjiang Zhang (redriver) wrote : | # |
> > Thanks for you careful review and valuable comments.
> >
> > > Overall, comments, much of the code needs some unit tests. We simply
> can't
> > > avoid breaking things inadvertently without them
> > get_resource_
> > > and the networking code specifically.
> > All right. I'll add some unit tests.
>
> Great.
>
> >
> > > Please bear with me... you have a lot of good things here, we just need
> > > to work out some details.
> > >
> > > - adding the specific config/
> > > I'd much prefer we adjust the already freebsd specific cloud.cfg-
> freebsd.
> > >
> > > The changes you have there are:
> > > - add Azure to the datasource_list. I think its right to just put
> > > Azure before Openstack and Ec2 so we have the list of 4 sources.
> > Ok.
> > > - you disabled growpart and resizefs
> > > I wonder why these were not disabled by the original freebsd work.
> > > Can you explain that? If they just aren't going to work, then lets
> > > remove them from the list and update the docstring to mention lack
> > > of support on freebsd. If they are only broken on Azure, lets figure
> > > out why.
> > On Azure, FreeBSD image used a fixed size VHD, so gpart is not necessary.
> > Anyway, there is some error reported in the log if I enabled it. For the
> same
> > reason, I disabled resizefs on Azure. Well, if we decide to use the same
> > cloud-init.cfg for FreeBSD. I have to fix those error message and enable
> > growpart and resizefs.
>
> OK, Hopefully we could use Jake's suggestion above and get resizefs and
> growpart working even if they aren't needed on azure. They're in general good
> things.
>
> > > - default user changes
> > > - removing of the default 'name'. This should not be necessary as
> > > linux uses the datasource provided username on azure when its
> there.
> > > - changing of shell to csh: my understanding (per git log) is that
> tcsh
> > > is the default shell on freebsd, so using that makes more sense.
> > > if Azure wishes to carry a different default shell, they can do
> that
> > > in local config. Generally though, commonality is better.
> > > - removal of lock_passwd: azure can carry that locally if they want.
> > For unknown reason, if I did not remove the default 'name' on Azure, it will
> > stop the data source provided username. Ok, that may be a bug on Azure. In
> > addition, the default username 'freebsd' is created but locked passwd, then
> > why do we need it?
>
> Maybe i wasn't clear. I was just saying that this all works as expected on
> Ubuntu (linux). We have a default config with a user 'ubuntu', but on azure,
> where a user is provided via the datasource, it overrides just that. So this
> should be able to work correctly.
>
Ok. I see. Anyway, I observed it does not work as expected with default user.
I'll investigate it.
> > > - why the changes set_passwd?
> > > You made 2 changes here:
> > > a.) use -H 0 and -h 0 rather than just -H or -h
> > > I'm fine to be explicit, but it seems like it shouldnt be
> necessary
> > > or it would never have worked.
> > > ...
Hongjiang Zhang (redriver) wrote : | # |
> Hi,
>
> I have been doing some testing on FreeBSD.
>
> The current growpart seems to work fine with FreeBSD 11.0 with GPT/UFS - the
> partition gets resized. The issue is with filesystem resize in resizefs. Its a
> simple fix, before you can modify the MBR you must first set
> kern.geom.
> You should then set debugflags back to it's original value.
>
> Basically this, I did a quick hack to test and seems ok:
>
> current_debugflags = sysctl -n kern.geom.
> sysctl kern.geom.
> growfs -y /dev/vtbd0p2
> sysctl kern.geom.
My problem is growfs always reports error since the disk is already full expanded.
# growfs -y /dev/da0p2
growfs: requested size 28GB is not larger than the current filesystem size 28GB
# echo $?
1
My disk is formatted to be UFS:
# gpart show
=> 40 62914480 da0 GPT (30G)
40 1024 1 freebsd-boot (512K)
1064 58719232 2 freebsd-ufs (28G)
58720296 3145728 3 freebsd-swap (1.5G)
61866024 1048496 - free - (512M)
=> 63 146800577 da1 MBR (70G)
63 1985 - free - (993K)
2048 146796544 1 ntfs (70G)
146798592 2048 - free - (1.0M)
Jake Smith (c0by) wrote : | # |
> > Hi,
> >
> > I have been doing some testing on FreeBSD.
> >
> > The current growpart seems to work fine with FreeBSD 11.0 with GPT/UFS - the
> > partition gets resized. The issue is with filesystem resize in resizefs. Its
> a
> > simple fix, before you can modify the MBR you must first set
> > kern.geom.
> flag.
> > You should then set debugflags back to it's original value.
> >
> > Basically this, I did a quick hack to test and seems ok:
> >
> > current_debugflags = sysctl -n kern.geom.
> > sysctl kern.geom.
> > growfs -y /dev/vtbd0p2
> > sysctl kern.geom.
> My problem is growfs always reports error since the disk is already full
> expanded.
>
> # growfs -y /dev/da0p2
> growfs: requested size 28GB is not larger than the current filesystem size
> 28GB
> # echo $?
> 1
>
> My disk is formatted to be UFS:
> # gpart show
> => 40 62914480 da0 GPT (30G)
> 40 1024 1 freebsd-boot (512K)
> 1064 58719232 2 freebsd-ufs (28G)
> 58720296 3145728 3 freebsd-swap (1.5G)
> 61866024 1048496 - free - (512M)
>
> => 63 146800577 da1 MBR (70G)
> 63 1985 - free - (993K)
> 2048 146796544 1 ntfs (70G)
> 146798592 2048 - free - (1.0M)
Not sure what the most elegant solution would be to avoid exit(1) when there is nothing to do...
Just thinking out loud here:
1.
Can resizefs be called to ignore exit status 1? Could be dangerous?
2.
Is there a way to pass back from growpart to see if gpart size did anything, if so then run growfs?
3.
You could do a dry run first and check the exit status before continuing:
# growfs -N -y /dev/da0p2
4.
Could we look at the current size of the file system and compare to gpart, (there is probably a better way to get the fs size):
# dumpfs -m /
# newfs command for / (/dev/da0p2)
newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -j -k 5240 -m 8 -o time -s 83885872 /dev/da0p2
Grab -s <size> round it and compare against
# gpart show
=> 40 83886000 da0 MBR (466G)
40 128 1 freebsd-boot (64K)
168 83886000 2 freebsd-ufs (40.0G)
Hongjiang Zhang (redriver) wrote : | # |
> > > Hi,
> > >
> > > I have been doing some testing on FreeBSD.
> > >
> > > The current growpart seems to work fine with FreeBSD 11.0 with GPT/UFS -
> the
> > > partition gets resized. The issue is with filesystem resize in resizefs.
> Its
> > a
> > > simple fix, before you can modify the MBR you must first set
> > > kern.geom.
> > flag.
> > > You should then set debugflags back to it's original value.
> > >
> > > Basically this, I did a quick hack to test and seems ok:
> > >
> > > current_debugflags = sysctl -n kern.geom.
> > > sysctl kern.geom.
> > > growfs -y /dev/vtbd0p2
> > > sysctl kern.geom.
> > My problem is growfs always reports error since the disk is already full
> > expanded.
> >
> > # growfs -y /dev/da0p2
> > growfs: requested size 28GB is not larger than the current filesystem size
> > 28GB
> > # echo $?
> > 1
> >
> > My disk is formatted to be UFS:
> > # gpart show
> > => 40 62914480 da0 GPT (30G)
> > 40 1024 1 freebsd-boot (512K)
> > 1064 58719232 2 freebsd-ufs (28G)
> > 58720296 3145728 3 freebsd-swap (1.5G)
> > 61866024 1048496 - free - (512M)
> >
> > => 63 146800577 da1 MBR (70G)
> > 63 1985 - free - (993K)
> > 2048 146796544 1 ntfs (70G)
> > 146798592 2048 - free - (1.0M)
>
> Not sure what the most elegant solution would be to avoid exit(1) when there
> is nothing to do...
>
> Just thinking out loud here:
>
> 1.
> Can resizefs be called to ignore exit status 1? Could be dangerous?
>
> 2.
> Is there a way to pass back from growpart to see if gpart size did anything,
> if so then run growfs?
>
> 3.
> You could do a dry run first and check the exit status before continuing:
> # growfs -N -y /dev/da0p2
>
> 4.
> Could we look at the current size of the file system and compare to gpart,
> (there is probably a better way to get the fs size):
>
> # dumpfs -m /
> # newfs command for / (/dev/da0p2)
> newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -j
> -k 5240 -m 8 -o time -s 83885872 /dev/da0p2
>
> Grab -s <size> round it and compare against
>
> # gpart show
> => 40 83886000 da0 MBR (466G)
> 40 128 1 freebsd-boot (64K)
> 168 83886000 2 freebsd-ufs (40.0G)
Firstly, I appreciate your help to resolve this issue. I prefer the 4th solution. Thanks.
Jake Smith (c0by) wrote : | # |
Just checked the code, you will need to round the output from gpart to the nearest whole block, see this example:
# gpart show md0
=> 34 297086 md0 GPT (145M)
34 297086 1 freebsd-ufs (145M)
# dumpfs -m /mnt
# newfs command for /mnt (/dev/md0p1)
newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -k 368 -m 8 -o time -s 297080 /dev/md0p1
So gpart reports 297086 but dumpfs reports 297080, making your code think it needs to do a resize.
Should be able to round the number using the fragment size (-f <frag-size>) like this, assuming a sector size of 512:
>>> gpart = 297086
>>> frag = 4096
>>> (gpart) - (gpart) % (frag / 512)
297080.0
Hongjiang Zhang (redriver) wrote : | # |
> Just checked the code, you will need to round the output from gpart to the
> nearest whole block, see this example:
>
> # gpart show md0
> => 34 297086 md0 GPT (145M)
> 34 297086 1 freebsd-ufs (145M)
>
> # dumpfs -m /mnt
> # newfs command for /mnt (/dev/md0p1)
> newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -k
> 368 -m 8 -o time -s 297080 /dev/md0p1
>
> So gpart reports 297086 but dumpfs reports 297080, making your code think it
> needs to do a resize.
>
> Should be able to round the number using the fragment size (-f <frag-size>)
> like this, assuming a sector size of 512:
>
> >>> gpart = 297086
> >>> frag = 4096
> >>> (gpart) - (gpart) % (frag / 512)
> 297080.0
Thanks. I'll apply it to my patch.
Hongjiang Zhang (redriver) wrote : | # |
> Just checked the code, you will need to round the output from gpart to the
> nearest whole block, see this example:
>
> # gpart show md0
> => 34 297086 md0 GPT (145M)
> 34 297086 1 freebsd-ufs (145M)
>
> # dumpfs -m /mnt
> # newfs command for /mnt (/dev/md0p1)
> newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -k
> 368 -m 8 -o time -s 297080 /dev/md0p1
>
> So gpart reports 297086 but dumpfs reports 297080, making your code think it
> needs to do a resize.
>
> Should be able to round the number using the fragment size (-f <frag-size>)
> like this, assuming a sector size of 512:
>
> >>> gpart = 297086
> >>> frag = 4096
> >>> (gpart) - (gpart) % (frag / 512)
> 297080.0
Another question, when I install cloud-init on FreeBSD, the cloud.cfg is used and I have to manually replace it with cloud.cfg-freebsd. Is it better select cloud configuration according to --init-system option. I want to implement like the following:
if (the value of --init-system option equal to sysvinit_freebsd)
then
replace cloud.cfg with cloud.cfg-FreeBSD
fi
But I also find tools/build-
Hongjiang Zhang (redriver) wrote : | # |
one more issue I found is the user information provided by Azure cannot override the default one. The reason is: Stages.Modules.cfg depends on helpers.
The quick fix is move "cfgs.extend(
I also wonder why Ubuntu on Azure does not encounter this issue. It looks like Ubuntu on Azure invokes cloudinitlocal, but I'm still confused how Ubuntu can override env_configs.
def _read_cfg(self):
# Input config files override
# env config files which
# override instance configs
# which override datasource
# configs which override
# base configuration
cfgs = []
if self._fns:
for c_fn in self._fns:
if self._base_cfg:
return util.mergemanyd
Scott Moser (smoser) wrote : | # |
I'm confused because Ubuntu's cloud.cfg is exactly what is in
config/cloud.cfg in trunk. which has:
users:
- default
...
system_info:
# This will affect which distro class gets used
distro: ubuntu
# Default user name + that default users groups (if added/used)
default_user:
name: ubuntu
lock_passwd: True
gecos: Ubuntu
groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
sudo: ["ALL=(ALL) NOPASSWD:ALL"]
shell: /bin/bash
Thats almost identical to what config/
branch.
I think that comment is actually wrong. If you look at how that method
works, it basically builds an array of configs (dictionaries), with
environment config (from environment CFG_ENV_NAME) first, then isntance
configs, then datasource, then base. Then calls mergemanydict with that.
So basically:
util.
{'foo': 'from enviroment'},
{'foo': 'from instance, no user override'},
{'
{'
])
Calling that returns:
{'foo': 'from enviroment',
'system_info': {'default_user': {'name': 'smoser-from-ds'}}}
So I'm pretty sure the comment is just wrong. Its probably because
'mergemanydict' has very odd usage in that precedence goes to the *first*
value, not the last.
Hongjiang Zhang (redriver) wrote : | # |
> I'm confused because Ubuntu's cloud.cfg is exactly what is in
> config/cloud.cfg in trunk. which has:
>
> users:
> - default
> ...
>
> system_info:
> # This will affect which distro class gets used
> distro: ubuntu
> # Default user name + that default users groups (if added/used)
> default_user:
> name: ubuntu
> lock_passwd: True
> gecos: Ubuntu
> groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev,
> sudo, video]
> sudo: ["ALL=(ALL) NOPASSWD:ALL"]
> shell: /bin/bash
>
> Thats almost identical to what config/
> branch.
>
> I think that comment is actually wrong. If you look at how that method
> works, it basically builds an array of configs (dictionaries), with
> environment config (from environment CFG_ENV_NAME) first, then isntance
> configs, then datasource, then base. Then calls mergemanydict with that.
>
> So basically:
> util.mergemanyd
> {'foo': 'from enviroment'},
> {'foo': 'from instance, no user override'},
> {'system_info': {'default_user': {'name': 'smoser-
> {'system_info': {'default_user': {'name': 'ubuntu'}}},
> ])
>
> Calling that returns:
> {'foo': 'from enviroment',
> 'system_info': {'default_user': {'name': 'smoser-from-ds'}}}
>
> So I'm pretty sure the comment is just wrong. Its probably because
> 'mergemanydict' has very odd usage in that precedence goes to the *first*
> value, not the last.
Ok. Let's ignore that comment. I debug this code snippet and still found it there.
I added a test function in helper.py, and specify two input files '/usr/local/
The content of ds.dat is as following.
{'disk_setup': {'ephemeral0': {'table_type': 'gpt', 'layout': [100], 'overwrite': True}}, 'fs_setup': [{'device': 'ephemeral0.1', 'replace_fs': 'ntfs', 'filesystem': 'freebsd-ufs'}], 'system_info': {'default_user': {'passwd': '$6$WSLBpwonNkC
The output indicates that the username is still "freebsd" even though the "passwd" field was updated. So, in my environment, there is something different from the environment on Ubuntu.
{"ssh_pwauth": true, "users": ["default"], "disable_root": false, "disk_setup": {"ephemeral0": {"table_type": "gpt", "layout": [100], "overwrite": true}}, "system_info": {"default_user": {"shell": "/bin/tcsh", "gecos": "FreeBSD", "name": "freebsd", "groups": ["wheel"], "passwd": "$6$WSLBpwonNkC
Hongjiang Zhang (redriver) wrote : | # |
> I'm confused because Ubuntu's cloud.cfg is exactly what is in
> config/cloud.cfg in trunk. which has:
>
> users:
> - default
> ...
>
> system_info:
> # This will affect which distro class gets used
> distro: ubuntu
> # Default user name + that default users groups (if added/used)
> default_user:
> name: ubuntu
> lock_passwd: True
> gecos: Ubuntu
> groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev,
> sudo, video]
> sudo: ["ALL=(ALL) NOPASSWD:ALL"]
> shell: /bin/bash
>
> Thats almost identical to what config/
> branch.
>
> I think that comment is actually wrong. If you look at how that method
> works, it basically builds an array of configs (dictionaries), with
> environment config (from environment CFG_ENV_NAME) first, then isntance
> configs, then datasource, then base. Then calls mergemanydict with that.
>
> So basically:
> util.mergemanyd
> {'foo': 'from enviroment'},
> {'foo': 'from instance, no user override'},
> {'system_info': {'default_user': {'name': 'smoser-
> {'system_info': {'default_user': {'name': 'ubuntu'}}},
> ])
>
> Calling that returns:
> {'foo': 'from enviroment',
> 'system_info': {'default_user': {'name': 'smoser-from-ds'}}}
>
> So I'm pretty sure the comment is just wrong. Its probably because
> 'mergemanydict' has very odd usage in that precedence goes to the *first*
> value, not the last.
Sorry, you are right, Ubuntu's cloud.cfg also specifies default user "ubuntu". That is not the root cause why cloud-init does not work on FreeBSD.
New finding is the helper.py will not load cloud.cfg as env_configs because only FreeBSD defines the CLOUD_CFG, which tells helpers.py to load env_configs.
I grep "CLOUD_CFG" from systemd and sysvinit and find:
grep "CLOUD_CFG" sysvinit/* -rn
sysvinit/
sysvinit/
sysvinit/
sysvinit/
in the helpers.py, I observed:
CFG_ENV_NAME = "CLOUD_CFG"
...
def _get_env_
e_cfgs = []
if CFG_ENV_NAME in os.environ:
e_fn = os.environ[
try:
except Exception:
return e_cfgs
I don't know why FreeBSD needs to export this environment, but none of the other distros need it.
Hongjiang Zhang (redriver) wrote : | # |
I have finished all the needed fixes.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:400a89bc938
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
There is a lot of good stuff here, thank you for your work and your patience.
Some good things:
- the generate_
- the unit tests added for that and for the resize.
There are some things that still need fixing though, and I have some in-line
comments to help elaborate.
a.) You've fixed some general things inside of the DataSourceAzureNet only.
Ie, you made devent2deev run through the datasource, which means
that that code wont work running freebsd on Ec2. Better to have
logic in it that does the right thing both places.
b.) I think we can simplify the '_can_skip_
c.) is your '_can_skip_
Hongjiang Zhang (redriver) : | # |
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:85137a5ffe6
https:/
Executed test runs:
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:316b4b389cd
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
> There is a lot of good stuff here, thank you for your work and your patience.
> Some good things:
> - the generate_
> - the unit tests added for that and for the resize.
>
> There are some things that still need fixing though, and I have some in-line
> comments to help elaborate.
> a.) You've fixed some general things inside of the DataSourceAzureNet only.
> Ie, you made devent2deev run through the datasource, which means
> that that code wont work running freebsd on Ec2. Better to have
> logic in it that does the right thing both places.
When I change devent2dev, I have considered other platforms. I have put a generic implementation of get_mount_info in sources/
def get_mount_
return util.get_
Why this function cannot be put to util?
FreeBSD on Azure uses glabel to label /dev/da0p2 with 'rootfs', so if we want to find '/' partition, we should first find /dev/label/rootfs through '/', and then map it to /dev/da0p2.
> b.) I think we can simplify the '_can_skip_
Yes, I have changed it per your suggestion.
> c.) is your '_can_skip_
It is used to check whether resize is necessary. On Azure, the partition has already fully covered the whole disk. It will report error in log if we resize it by force. This pre-check wants to avoid such error.
Scott Moser (smoser) wrote : | # |
> > There are some things that still need fixing though, and I have some in-line
> > comments to help elaborate.
> > a.) You've fixed some general things inside of the DataSourceAzureNet only.
> > Ie, you made devent2deev run through the datasource, which means
> > that that code wont work running freebsd on Ec2. Better to have
> > logic in it that does the right thing both places.
> When I change devent2dev, I have considered other platforms. I have put a
> generic implementation of get_mount_info in sources/
> platform, for example, EC2, it will invoke the util.get_
> def get_mount_
> return util.get_
devent2dev basically is supposed to turn either a filesystem location ("/home")
or a device into a device that contains that filesystem. Input of '/dev/sda'
should return '/dev/sda'. Input of "/home" should return the device that
/home's filesystem is on (what should be rezied).
Your made a change to resize_devices in cloudinit/
take a 'cloud'.
- resize_devices then calls devent2dev(devent, cloud)
- devent2dev calls cloud.datasourc
This change:
- result = util.get_
+ result = cloud.datasourc
I'd have preferred for 'util.get_
if is_FreeBSD:
# get the device one way
else:
# get it for linux.
Instead, since you're going through 'datasource', your fix will only work
on Azure.
> > c.) is your '_can_skip_
> needed?
> It is used to check whether resize is necessary. On Azure, the partition has
> already fully covered the whole disk. It will report error in log if we resize
> it by force. This pre-check wants to avoid such error.
OK, thats fine.
Scott Moser (smoser) wrote : | # |
I think you might be able to not change anything in the get_mount_info path, but rather just fix a bug in 'parse_
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -2056,6 +2056,8 @@ def parse_mount(path):
mount_locs = mountoutput.
for line in mount_locs:
m = re.search(
+ if not m:
+ continue
devpth = m.group(1)
fs_type = m.group(3)
then, in my testing on a freebsd instance in digital ocean :
$ python -c 'from cloudinit import util; print(util.
('/dev/gpt/rootfs', 'ufs', '/')
Actually, even without the change it works for me with 'mount' output like:
$ mount
/dev/gpt/rootfs on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
fdescfs on /dev/fd (fdescfs)
/dev/vtbd1 on /var/lib/
Note, that it does not work unless you give it a mount point as input. Ie, util.get_
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:cddf09f0a26
https:/
Executed test runs:
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:27ab16c21a1
https:/
Executed test runs:
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:1bafe24b285
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
I have done the fix. I also fixed the issue you mentioned for get_mount_info by providing a path.
> I think you might be able to not change anything in the get_mount_info path,
> but rather just fix a bug in 'parse_
>
> --- a/cloudinit/util.py
> +++ b/cloudinit/util.py
> @@ -2056,6 +2056,8 @@ def parse_mount(path):
> mount_locs = mountoutput.
> for line in mount_locs:
> m = re.search(
> + if not m:
> + continue
> devpth = m.group(1)
> mount_point = m.group(2)
> fs_type = m.group(3)
>
>
> then, in my testing on a freebsd instance in digital ocean :
> $ python -c 'from cloudinit import util; print(util.
> ('/dev/gpt/rootfs', 'ufs', '/')
>
> Actually, even without the change it works for me with 'mount' output like:
> $ mount
> /dev/gpt/rootfs on / (ufs, local, soft-updates)
> devfs on /dev (devfs, local, multilabel)
> fdescfs on /dev/fd (fdescfs)
> /dev/vtbd1 on /var/lib/
>
>
> Note, that it does not work unless you give it a mount point as input. Ie,
> util.get_
> could fix that though, by just looking through all the mount points and
> finding the one that 'path' is on.
Scott Moser (smoser) wrote : | # |
Hi,
Sorry for the slow reply.
We are getting closer.
I have some comments inline.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:1505ec9ccc5
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:35141b85edc
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
> Hi,
> Sorry for the slow reply.
> We are getting closer.
> I have some comments inline.
I have removed 'cloud' parameter from devent2dev, and I added more check in util.get_mount_info to specially handle the case for FreeBSD on Azure.
Is there any other mechanism for me to easily check whether it is running on Hyper-V?
Hongjiang Zhang (redriver) wrote : | # |
> Hi,
> Sorry for the slow reply.
> We are getting closer.
> I have some comments inline.
I have removed 'cloud' parameter from devent2dev, and I added more check in util.get_mount_info to specially handle the case for FreeBSD on Azure.
Is there any other mechanism for me to easily check whether it is running on Hyper-V?
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:a6a04d1d840
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:d91244316aa
https:/
Executed test runs:
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:9f509bee5f6
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
there are som ecomments in line.
I have a mp i will submit to you to cleanup some others.
Scott Moser (smoser) wrote : | # |
please see my changes at https:/
Scott Moser (smoser) wrote : | # |
Other than that and I think it looks really good.
thank you for adding the unit tests.
Hongjiang Zhang (redriver) wrote : | # |
> diff --git a/cloudinit/util.py b/cloudinit/util.py index
> 3301957..ef84e32 100644
> --- a/cloudinit/util.py
> +++ b/cloudinit/util.py
> @@ -2089,6 +2144,8 @@ def get_mount_
> #
> # So use /proc/$$/mountinfo to find the device underlying the
> # input path.
> + if is_FreeBSD_
>but why "freebsd on hyperv"
>why are you not just checking "is this freebsd".
>what is specific to Azure here?
FreeBSD on Azure used a special label on disk. If we want to
get the /dev/daXXX from mount information, we need to handle
it specially.
> + return get_mount_
> mountinfo_path = '/proc/
> if os.path.
> lines = load_file(
> diff --git a/tests/
> b/tests/
> new file mode 100644
> index 0000000..b5384e4
> --- /dev/null
> +++ b/tests/
> @@ -0,0 +1,73 @@
> +# This file is part of cloud-init. See LICENSE file for license information.
> +
> +from cloudinit.config import cc_resizefs
> +
> +import unittest
> +
> +try:
> + from unittest import mock
> +except ImportError:
> + import mock
> +
> +
> +class TestResizefs(
> + def setUp(self):
> + super(TestResizefs, self).setUp()
> + self.name = "resizefs"
> +
> + @mock.patch(
> + @mock.patch(
> + def test_skip_
> + fs_type = "ufs"
> + resize_what = "/"
> + devpth = "/dev/da0p2"
> + dumpfs_
> + "(/dev/
> + "newfs -O 2 -U -a 4 -b "\
> + "32768 -d 32768 -e 4096 "\
> + "-f 4096 -g 16384 -h 64 "\
> + "-i 8192 -j -k 6408 -m 8 "\
> + "-o time -s 58719232 "\
> + "/dev/label/
> + gpart_out.
> +=> 40 62914480 da0 GPT (30G)
> + 40 1024 1 freebsd-boot (512K)
> + 1064 58719232 2 freebsd-ufs (28G)
> + 58720296 3145728 3 freebsd-swap (1.5G)
> + 61866024 1048496 - free - (512M)
> +"""
> + res = cc_resizefs.
> + resize_what,
> + devpth)
> + self.assertTrue
> +
> + @mock.patch(
> + @mock.patch(
> + def test_skip_
> + fs_type = "ufs"
> + resize_what = "/"
> + devpth = "/dev/da0p2"
> + dumpfs_
> + ...
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:6a0c7e539c5
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:bba43d77e4c
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
I have manually merged your modifications to my branch and fixed a unit test issue.
Do you think it is ok for merging?
Thanks
Hongjiang Zhang
-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, April 13, 2017 2:51 AM
To: <email address hidden>
Subject: Re: [Merge] ~redriver/
Other than that and I think it looks really good.
thank you for adding the unit tests.
--
https:/
You are the owner of ~redriver/
Scott Moser (smoser) wrote : | # |
On Fri, 14 Apr 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I have manually merged your modifications to my branch and fixed a unit test issue.
> Do you think it is ok for merging?
The only thing I had left was that I'm still confused as what why we need
azure specific code in that code path.
I have to think about it some more and probably go play with it a little
on freebsd.
>
> Thanks
> Hongjiang Zhang
>
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
> Sent: Thursday, April 13, 2017 2:51 AM
> To: <email address hidden>
> Subject: Re: [Merge] ~redriver/
>
> Other than that and I think it looks really good.
> thank you for adding the unit tests.
>
> --
> https:/
> You are the owner of ~redriver/
>
Hongjiang Zhang (redriver) wrote : | # |
The reason is FreeBSD on Azure have more than 1 disks attached, and the rootfs located in one disk, for example, /dev/da0, but I cannot directly write something below:
/dev/da0 / ufs rw 1 1
Because the rootfs may be switched to /dev/da1. The solution is label the disk. For FreeBSD on Azure, I labeled /dev/da0 with "rootfs",
so for get_mount_info, I have to find "rootfs" first, and then find the corresponding /dev/da0.
But on Linux, there is a /proc/$$/mountinfo to help find the device. For Azure, there is also /dev/disk/
Both make the get_mount_info easy to implement.
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Friday, April 14, 2017 10:46 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
On Fri, 14 Apr 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I have manually merged your modifications to my branch and fixed a unit test issue.
> Do you think it is ok for merging?
The only thing I had left was that I'm still confused as what why we need azure specific code in that code path.
I have to think about it some more and probably go play with it a little on freebsd.
>
> Thanks
> Hongjiang Zhang
>
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Scott Moser
> Sent: Thursday, April 13, 2017 2:51 AM
> To: <email address hidden>
> Subject: Re: [Merge] ~redriver/
> cloud-init:master
>
> Other than that and I think it looks really good.
> thank you for adding the unit tests.
>
> --
> https:/
> unchpad.
> 2F314895&
> 08d481d4cf8c%
> 371945&
> ed=0 You are the owner of ~redriver/
>
Hongjiang Zhang (redriver) wrote : | # |
Hello Scott,
I'm not sure whether you are still confused about why we need Azure specific code, or is there anything I can do to help make any progress on this merge?
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Friday, April 14, 2017 10:58 AM
To: Scott Moser <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
The reason is FreeBSD on Azure have more than 1 disks attached, and the rootfs located in one disk, for example, /dev/da0, but I cannot directly write something below:
/dev/da0 / ufs rw 1 1
Because the rootfs may be switched to /dev/da1. The solution is label the disk. For FreeBSD on Azure, I labeled /dev/da0 with "rootfs", so for get_mount_info, I have to find "rootfs" first, and then find the corresponding /dev/da0.
But on Linux, there is a /proc/$$/mountinfo to help find the device. For Azure, there is also /dev/disk/
Both make the get_mount_info easy to implement.
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Friday, April 14, 2017 10:46 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
On Fri, 14 Apr 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I have manually merged your modifications to my branch and fixed a unit test issue.
> Do you think it is ok for merging?
The only thing I had left was that I'm still confused as what why we need azure specific code in that code path.
I have to think about it some more and probably go play with it a little on freebsd.
>
> Thanks
> Hongjiang Zhang
>
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Scott Moser
> Sent: Thursday, April 13, 2017 2:51 AM
> To: <email address hidden>
> Subject: Re: [Merge] ~redriver/
> cloud-init:master
>
> Other than that and I think it looks really good.
> thank you for adding the unit tests.
>
> --
> https:/
> unchpad.
> 2F314895&
> 08d481d4cf8c%
> 371945&
> ed=0 You are the owner of ~redriver/
>
- c1cf3ad... by Hongjiang Zhang
-
Remove Azure specific check
Take the get_mount_info for Azure as a common case on FreeBSD.
This was only triggered if the device info '/dev/xxx' is illegal.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:c1cf3ad1fc1
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
By the way, I removed the Azure specific check for "get_mount_info".
get_mount_
What do you think?
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -565,6 +565,10 @@ def is_ipv4(instr):
return len(toks) == 4
+def is_FreeBSD():
+ return system_
+
+
def get_cfg_
if key not in yobj:
return default
@@ -2055,11 +2059,56 @@ def parse_mtab(path):
return None
+def find_freebsd_
+ if label_part.
+ target_label = label_part[5:]
+ (label_part, err) = subp(['glabel', 'status', '-s'])
+ for labels in label_part.
+ items = labels.split()
+ if len(items) > 0 and items[0]
+ label_part = items[2]
+ break
+ label_part = str(label_part)
+ return label_part
+
+
+def get_path_
+ path_found = None
+ for line in mnt_list.
+ items = line.split()
+ if (len(items) > 2 and os.path.
+ path_found = line
+ break
+ return path_found
+
+
+def get_mount_
+ (result, err) = subp(['mount', '-p', path], rcs=[0, 1])
+ if len(err):
+ # find a path if the input is not a mounting point
+ (mnt_list, err) = subp(['mount', '-p'])
+ path_found = get_path_
+ if (path_found is None):
+ return None
+ result = path_found
+ ret = result.split()
+ label_part = find_freebsd_
+ return "/dev/" + label_part, ret[2], ret[1]
+
+
def parse_mount(path):
(mountoutput, _err) = subp("mount")
mount_locs = mountoutput.
for line in mount_locs:
m = re.search(
+ if not m:
+ continue
+ # check whether the dev refers to a label on FreeBSD
+ # for example, if dev is '/dev/label/
+ # continue finding the real device like '/dev/da0'.
+ devm = re.search(
+ if (not devm and is_FreeBSD()):
+ return get_mount_
devpth = m.group(1)
-----Original Message-----
From: Hongjiang Zhang
Sent: Tuesday, April 25, 2017 11:38 AM
To: 'Scott Moser' <email address hidden>
Cc: '<email address hidden>' <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
Hello Scott,
I'm not sure whether you are still confused about why we need Azure specific code, or is there anything I can do to help make any progress on this merge?
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Friday, April 14, 2017 10:58 AM
To: Scott Moser <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subj...
Kylie Liang (kyliel) wrote : | # |
Hi Scott,
Thank you for your effort. Could you please help take minutes to see any else we should do to get the patch merged? Feel free let us know your thoughts. It is our highest priority to get this work for BSD users. Your support is appreciated. Thank you.
Thanks,
Kylie Liang
-----Original Message-----
From: Hongjiang Zhang
Sent: Tuesday, April 25, 2017 11:38 AM
To: Scott Moser <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
Hello Scott,
I'm not sure whether you are still confused about why we need Azure specific code, or is there anything I can do to help make any progress on this merge?
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Friday, April 14, 2017 10:58 AM
To: Scott Moser <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
The reason is FreeBSD on Azure have more than 1 disks attached, and the rootfs located in one disk, for example, /dev/da0, but I cannot directly write something below:
/dev/da0 / ufs rw 1 1
Because the rootfs may be switched to /dev/da1. The solution is label the disk. For FreeBSD on Azure, I labeled /dev/da0 with "rootfs", so for get_mount_info, I have to find "rootfs" first, and then find the corresponding /dev/da0.
But on Linux, there is a /proc/$$/mountinfo to help find the device. For Azure, there is also /dev/disk/
Both make the get_mount_info easy to implement.
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Friday, April 14, 2017 10:46 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
On Fri, 14 Apr 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I have manually merged your modifications to my branch and fixed a unit test issue.
> Do you think it is ok for merging?
The only thing I had left was that I'm still confused as what why we need azure specific code in that code path.
I have to think about it some more and probably go play with it a little on freebsd.
>
> Thanks
> Hongjiang Zhang
>
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Scott Moser
> Sent: Thursday, April 13, 2017 2:51 AM
> To: <email address hidden>
> Subject: Re: [Merge] ~redriver/
> cloud-init:master
>
> Other than that and I think it looks really good.
> thank you for adding the unit tests.
>
> --
> https:/
> unchpad.
> 2F314895&
> 08d481d4cf8c%
> 371945&sdata=q...
- 1b1e0a8... by Hongjiang Zhang
-
fix the issue of 'ssh does not exist in /etc/rc.d'
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:1b1e0a8fe65
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
I have removed the Azure specific check for "get_mount_info".
get_mount_
What do you think?
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -565,6 +565,10 @@ def is_ipv4(instr):
return len(toks) == 4
+def is_FreeBSD():
+ return system_
+
+
def get_cfg_
if key not in yobj:
return default
@@ -2055,11 +2059,56 @@ def parse_mtab(path):
return None
+def find_freebsd_
+ if label_part.
+ target_label = label_part[5:]
+ (label_part, err) = subp(['glabel', 'status', '-s'])
+ for labels in label_part.
+ items = labels.split()
+ if len(items) > 0 and items[0]
+ label_part = items[2]
+ break
+ label_part = str(label_part)
+ return label_part
+
+
+def get_path_
+ path_found = None
+ for line in mnt_list.
+ items = line.split()
+ if (len(items) > 2 and os.path.
+ path_found = line
+ break
+ return path_found
+
+
+def get_mount_
+ (result, err) = subp(['mount', '-p', path], rcs=[0, 1])
+ if len(err):
+ # find a path if the input is not a mounting point
+ (mnt_list, err) = subp(['mount', '-p'])
+ path_found = get_path_
+ if (path_found is None):
+ return None
+ result = path_found
+ ret = result.split()
+ label_part = find_freebsd_
+ return "/dev/" + label_part, ret[2], ret[1]
+
+
def parse_mount(path):
(mountoutput, _err) = subp("mount")
mount_locs = mountoutput.
for line in mount_locs:
m = re.search(
+ if not m:
+ continue
+ # check whether the dev refers to a label on FreeBSD
+ # for example, if dev is '/dev/label/
+ # continue finding the real device like '/dev/da0'.
+ devm = re.search(
+ if (not devm and is_FreeBSD()):
+ return get_mount_
devpth = m.group(1)
-----Original Message-----
From: Hongjiang Zhang
Sent: Tuesday, April 25, 2017 11:38 AM
To: 'Scott Moser' <email address hidden>
Cc: '<email address hidden>' <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
Subject: RE: [Merge] ~redriver/
Hello Scott,
I'm not sure whether you are still confused about why we need Azure specific code, or is there anything I can do to help make any progress on this merge?
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Friday, April 14, 2017 10:58 AM
To: Scott Moser <email address hidden>
Cc: <email address hidden>; Kylie Liang <email address hidden>; Josh Poulson <email address hidden>
S...
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
I don't want to bother you again and again, but I really hope you can take 3~5 minutes to go through my patch, because it has taken at least 3 months on review since merge request was created.
Your main concern is why I need Azure specific code in get_mount_info, and I have removed this dependence.
Do you have any other concern?
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Tuesday, May 2, 2017 10:26 AM
To: 'Scott Moser' <email address hidden>; '<email address hidden>' <email address hidden>
Cc: '<email address hidden>' <email address hidden>
Subject: RE: [Merge] ~redriver/
Hi Scott,
I have removed the Azure specific check for "get_mount_info".
get_mount_
What do you think?
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -565,6 +565,10 @@ def is_ipv4(instr):
return len(toks) == 4
+def is_FreeBSD():
+ return system_
+
+
def get_cfg_
if key not in yobj:
return default
@@ -2055,11 +2059,56 @@ def parse_mtab(path):
return None
+def find_freebsd_
+ if label_part.
+ target_label = label_part[5:]
+ (label_part, err) = subp(['glabel', 'status', '-s'])
+ for labels in label_part.
+ items = labels.split()
+ if len(items) > 0 and items[0]
+ label_part = items[2]
+ break
+ label_part = str(label_part)
+ return label_part
+
+
+def get_path_
+ path_found = None
+ for line in mnt_list.
+ items = line.split()
+ if (len(items) > 2 and os.path.
+ path_found = line
+ break
+ return path_found
+
+
+def get_mount_
+ (result, err) = subp(['mount', '-p', path], rcs=[0, 1])
+ if len(err):
+ # find a path if the input is not a mounting point
+ (mnt_list, err) = subp(['mount', '-p'])
+ path_found = get_path_
+ if (path_found is None):
+ return None
+ result = path_found
+ ret = result.split()
+ label_part = find_freebsd_
+ return "/dev/" + label_part, ret[2], ret[1]
+
+
def parse_mount(path):
(mountoutput, _err) = subp("mount")
mount_locs = mountoutput.
for line in mount_locs:
m = re.search(
+ if not m:
+ continue
+ # check whether the dev refers to a label on FreeBSD
+ # for example, if dev is '/dev/label/
+ # continue finding the real device like '/dev/da0'.
+ devm = re.search(
+ if (not devm and is_FreeBSD()):
+ return get_mount_
devpth = m.group(1)
-----Original Message-----
From: Hongjiang Zhang
Sent: Tuesday, April 25, 2...
Scott Moser (smoser) wrote : | # |
On Thu, 4 May 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I don't want to bother you again and again, but I really hope you can take 3~5 minutes to go through my patch, because it has taken at least 3 months on review since merge request was created.
>
> Your main concern is why I need Azure specific code in get_mount_info, and I have removed this dependence.
>
> Do you have any other concern?
>
> Thanks
> Hongjiang Zhang
Hongjiang,
I'm really sorry for being so unresponsive.
I really do not mind you bothering me and appreciate your persistence.
I grabbed [1] and made some changes. I think that was your latest.
I pushed that to [2].
Please take a look, and integrate my changes. Then, please propose that
branch and write a good commit message that describes all the fixes you've
made.
I'll give it another look tomorrow. We are really close.
Thank you.
[1] https:/
[2] https:/
>
> -----Original Message-----
> From: Hongjiang Zhang
> Sent: Tuesday, May 2, 2017 10:26 AM
> To: 'Scott Moser' <email address hidden>; '<email address hidden>' <email address hidden>
> Cc: '<email address hidden>' <email address hidden>
> Subject: RE: [Merge] ~redriver/
>
> Hi Scott,
>
> I have removed the Azure specific check for "get_mount_info".
> get_mount_
> What do you think?
>
> --- a/cloudinit/util.py
> +++ b/cloudinit/util.py
> @@ -565,6 +565,10 @@ def is_ipv4(instr):
> return len(toks) == 4
>
>
> +def is_FreeBSD():
> + return system_
> +
> +
> def get_cfg_
> if key not in yobj:
> return default
> @@ -2055,11 +2059,56 @@ def parse_mtab(path):
> return None
>
>
> +def find_freebsd_
> + if label_part.
> + target_label = label_part[5:]
> + (label_part, err) = subp(['glabel', 'status', '-s'])
> + for labels in label_part.
> + items = labels.split()
> + if len(items) > 0 and items[0]
> + label_part = items[2]
> + break
> + label_part = str(label_part)
> + return label_part
> +
> +
> +def get_path_
> + path_found = None
> + for line in mnt_list.
> + items = line.split()
> + if (len(items) > 2 and os.path.
> + path_found = line
> + break
> + return path_found
> +
> +
> +def get_mount_
> + (result, err) = subp(['mount', '-p', path], rcs=[0, 1])
> + if len(err):
> + # find a path if the input is not a mounting point
> + (mnt_list, err) = subp(['mount', '-p'])
> + path_found = get_path_
> + if (path_found is None):
> + return None
> + result = path_found
> + ...
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
The branch cl_on_Azure_0.7.9 was created recently and used to patch on freebsd ports: cloud-init-azure (see the ports code review: https:/
I hope you can give some comments about another merge request: https:/
Since I did not see the hope to complete that merge request of 314895, I have to create another branch: cl_on_Azure_0.7.9 for FreeBSD cloud-init ports.
Now, the cloud-init-azure ports is almost ready, but I still hope you can take some time on reviewing https:/
Thanks
Hongjiang Zhang
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, May 4, 2017 11:56 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Josh Poulson <email address hidden>; Kylie Liang <email address hidden>
Subject: RE: [Merge] ~redriver/
On Thu, 4 May 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I don't want to bother you again and again, but I really hope you can take 3~5 minutes to go through my patch, because it has taken at least 3 months on review since merge request was created.
>
> Your main concern is why I need Azure specific code in get_mount_info, and I have removed this dependence.
>
> Do you have any other concern?
>
> Thanks
> Hongjiang Zhang
Hongjiang,
I'm really sorry for being so unresponsive.
I really do not mind you bothering me and appreciate your persistence.
I grabbed [1] and made some changes. I think that was your latest.
I pushed that to [2].
Please take a look, and integrate my changes. Then, please propose that branch and write a good commit message that describes all the fixes you've made.
I'll give it another look tomorrow. We are really close.
Thank you.
[1] https:/
[2] https:/
>
> -----Original Message-----
> From: Hongjiang Zhang
> Sent: Tuesday, May 2, 2017 10:26 AM
> To: 'Scott Moser' <email address hidden>; '<email address hidden>'
> <email address hidden>
> Cc: '<email address hidden>' <email address hidden>
> Subject: RE: [Merge] ~redriver/
> cloud-init:master
>
> Hi Scott,
>
> I have removed the Azure ...
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
Sorry for making you confused. Thanks for your reviewing on the branch cl_on_Azure_0.7.9, which is totally for FreeBSD ports, and that patch mainly comes from merge request of 314895.
I hope the 1st step is to make cloud-init head work for FreeBSD on Azure, and the 2nd step is to make FreeBSD ports work.
Could you please focus on my merge request of 314895?
>diff --git a/requirements.txt b/requirements.txt
>index 2559b31..0c4951f 100644
>--- a/requirements.txt
>+++ b/requirements.txt
>@@ -28,7 +28,7 @@ configobj>=5.0.2
> pyyaml
>
> # The new main entrypoint uses argparse instead of optparse
>-# argparse
>+argparse
>
> # Requests handles ssl correctly!
> requests
I removed "argparse" because I found FreeBSD ports will fail to run. The reason is python 2.7 has made argparse builtin library. FreeBSD currently python version is 2.7, so, that is a FreeBSD specific change.
That is why I create FreeBSD ports to hold that change.
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Thursday, May 4, 2017 12:49 PM
To: 'Scott Moser' <email address hidden>
Cc: <email address hidden>; Josh Poulson <email address hidden>; Kylie Liang <email address hidden>
Subject: RE: [Merge] ~redriver/
Hi Scott,
The branch cl_on_Azure_0.7.9 was created recently and used to patch on freebsd ports: cloud-init-azure (see the ports code review: https:/
I hope you can give some comments about another merge request: https:/
Since I did not see the hope to complete that merge request of 314895, I have to create another branch: cl_on_Azure_0.7.9 for FreeBSD cloud-init ports.
Now, the cloud-init-azure ports is almost ready, but I still hope you can take some time on reviewing https:/
Thanks
Hongjiang Zhang
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, May 4, 2017 11:56 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Josh Poulson <email address hidden>; Kylie Liang <email address hidden>
Subject: RE: [Merge] ~redriver/
On Thu, 4 May 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I don't want to bother you again and again, but I really hope you can take 3~5 minutes to go through my patch, because it has taken at least 3 months on review since merge request was created.
>
> Your main concern is why I need Azure specific code in get_mount_info, and I have removed this dependence.
>
> Do you have any other concern?
>
> Thanks
> Hongjiang Zhang
Hongjiang,
I'm really sorry for being so unresponsive.
I really do not mind you bothering me and appreciate your persistence.
I grabbed [1] and made some changes. I think that was your latest.
I pushed that to [2].
Please take a look, and integrate my changes. Then, please propose that ...
- 77669d5... by Hongjiang Zhang
-
modify according to Scott comments
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:77669d5afd5
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
I did not propose the branch of cl_on_Azure_0.7.9. Instead, I manually merged your diff to my existing merge request (https:/
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Thursday, May 4, 2017 1:09 PM
To: 'Scott Moser' <email address hidden>
Cc: '<email address hidden>' <email address hidden>
Subject: RE: [Merge] ~redriver/
Hi Scott,
Sorry for making you confused. Thanks for your reviewing on the branch cl_on_Azure_0.7.9, which is totally for FreeBSD ports, and that patch mainly comes from merge request of 314895.
I hope the 1st step is to make cloud-init head work for FreeBSD on Azure, and the 2nd step is to make FreeBSD ports work.
Could you please focus on my merge request of 314895?
>diff --git a/requirements.txt b/requirements.txt index 2559b31..0c4951f
>100644
>--- a/requirements.txt
>+++ b/requirements.txt
>@@ -28,7 +28,7 @@ configobj>=5.0.2
> pyyaml
>
> # The new main entrypoint uses argparse instead of optparse -#
>argparse
>+argparse
>
> # Requests handles ssl correctly!
> requests
I removed "argparse" because I found FreeBSD ports will fail to run. The reason is python 2.7 has made argparse builtin library. FreeBSD currently python version is 2.7, so, that is a FreeBSD specific change.
That is why I create FreeBSD ports to hold that change.
Thanks
Hongjiang Zhang
-----Original Message-----
From: Hongjiang Zhang
Sent: Thursday, May 4, 2017 12:49 PM
To: 'Scott Moser' <email address hidden>
Cc: <email address hidden>; Josh Poulson <email address hidden>; Kylie Liang <email address hidden>
Subject: RE: [Merge] ~redriver/
Hi Scott,
The branch cl_on_Azure_0.7.9 was created recently and used to patch on freebsd ports: cloud-init-azure (see the ports code review: https:/
I hope you can give some comments about another merge request: https:/
Since I did not see the hope to complete that merge request of 314895, I have to create another branch: cl_on_Azure_0.7.9 for FreeBSD cloud-init ports.
Now, the cloud-init-azure ports is almost ready, but I still hope you can take some time on reviewing https:/
Thanks
Hongjiang Zhang
-----Original Message-----
From: Scott Moser [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, May 4, 2017 11:56 AM
To: Hongjiang Zhang <email address hidden>
Cc: <email address hidden>; Josh Poulson <email address hidden>; Kylie Liang <email address hidden>
Subject: RE: [Merge] ~redriver/
On Thu, 4 May 2017, Hongjiang Zhang wrote:
> Hi Scott,
>
> I don't want to bother you again and again, but I really hope you can take 3~5 minutes to go through my patch, because it has taken at least 3 months o...
- 0ff61d4... by Hongjiang Zhang
-
remove Linux specific installation on FreeBSD
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:0ff61d4ed2e
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
I've merged this.
The diff from the merged commit (0a71d5a870b416
to your tip (0ff61d4) when rebased is just:
diff --git a/cloudinit/
index d9f31bfd..bad112fe 100644
--- a/cloudinit/
+++ b/cloudinit/
@@ -272,7 +272,7 @@ class Distro(
cmd = ['ifconfig', '-l']
(nics, err) = util.subp(cmd, rcs=[0, 1])
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return nics
@@ -281,7 +281,7 @@ class Distro(
cmd = ['ifconfig', ifname]
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return if_result
@@ -290,7 +290,7 @@ class Distro(
cmd = ['ifconfig', '-l', 'ether']
(nics, err) = util.subp(cmd, rcs=[0, 1])
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return nics
diff --git a/setup.py b/setup.py
index 46d05219..4616599b 100755
--- a/setup.py
+++ b/setup.py
@@ -173,8 +173,9 @@ else:
[f for f in glob('doc/
]
if os.uname()[0] != 'FreeBSD':
- data_files.append([
- (ETC + '/NetworkManage
+ data_files.extend([
+ (ETC + '/NetworkManage
+ ['tools/
(ETC + '/dhcp/
(LIB + '/udev/rules.d', [f for f in glob('udev/
])
Hongjiang Zhang (redriver) wrote : | # |
Hi Scott,
Great to see it is merged finally!
I appreciated all your review and help in the past 2 months.
Thanks
Hongjiang Zhang
-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, May 11, 2017 1:05 AM
To: <email address hidden>
Subject: Re: [Merge] ~redriver/
Review: Approve
I've merged this.
The diff from the merged commit (0a71d5a870b416
to your tip (0ff61d4) when rebased is just:
diff --git a/cloudinit/
--- a/cloudinit/
+++ b/cloudinit/
@@ -272,7 +272,7 @@ class Distro(
cmd = ['ifconfig', '-l']
(nics, err) = util.subp(cmd, rcs=[0, 1])
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return nics
@@ -281,7 +281,7 @@ class Distro(
cmd = ['ifconfig', ifname]
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return if_result
@@ -290,7 +290,7 @@ class Distro(
cmd = ['ifconfig', '-l', 'ether']
(nics, err) = util.subp(cmd, rcs=[0, 1])
if len(err):
- LOG.warn("Error running %s: %s", cmd, err)
+ LOG.warning("Error running %s: %s", cmd, err)
return None
return nics
diff --git a/setup.py b/setup.py
index 46d05219..4616599b 100755
--- a/setup.py
+++ b/setup.py
@@ -173,8 +173,9 @@ else:
[f for f in glob('doc/
]
if os.uname()[0] != 'FreeBSD':
- data_files.append([
- (ETC + '/NetworkManage
+ data_files.extend([
+ (ETC + '/NetworkManage
+ ['tools/
(ETC + '/dhcp/
(LIB + '/udev/rules.d', [f for f in glob('udev/
])
--
https:/
You are the owner of ~redriver/
Preview Diff
1 | diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py |
2 | index 60e3ab5..ceee952 100644 |
3 | --- a/cloudinit/config/cc_resizefs.py |
4 | +++ b/cloudinit/config/cc_resizefs.py |
5 | @@ -33,7 +33,10 @@ disabled altogether by setting ``resize_rootfs`` to ``false``. |
6 | """ |
7 | |
8 | import errno |
9 | +import getopt |
10 | import os |
11 | +import re |
12 | +import shlex |
13 | import stat |
14 | |
15 | from cloudinit.settings import PER_ALWAYS |
16 | @@ -58,6 +61,62 @@ def _resize_ufs(mount_point, devpth): |
17 | return ('growfs', devpth) |
18 | |
19 | |
20 | +def _get_dumpfs_output(mount_point): |
21 | + dumpfs_res, err = util.subp(['dumpfs', '-m', mount_point]) |
22 | + return dumpfs_res |
23 | + |
24 | + |
25 | +def _get_gpart_output(part): |
26 | + gpart_res, err = util.subp(['gpart', 'show', part]) |
27 | + return gpart_res |
28 | + |
29 | + |
30 | +def _can_skip_resize_ufs(mount_point, devpth): |
31 | + # extract the current fs sector size |
32 | + """ |
33 | + # dumpfs -m / |
34 | + # newfs command for / (/dev/label/rootfs) |
35 | + newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 |
36 | + -h 64 -i 8192 -j -k 6408 -m 8 -o time -s 58719232 /dev/label/rootf |
37 | + """ |
38 | + cur_fs_sz = None |
39 | + frag_sz = None |
40 | + dumpfs_res = _get_dumpfs_output(mount_point) |
41 | + for line in dumpfs_res.splitlines(): |
42 | + if not line.startswith('#'): |
43 | + newfs_cmd = shlex.split(line) |
44 | + opt_value = 'O:Ua:s:b:d:e:f:g:h:i:jk:m:o:' |
45 | + optlist, args = getopt.getopt(newfs_cmd[1:], opt_value) |
46 | + for o, a in optlist: |
47 | + if o == "-s": |
48 | + cur_fs_sz = int(a) |
49 | + if o == "-f": |
50 | + frag_sz = int(a) |
51 | + # check the current partition size |
52 | + """ |
53 | + # gpart show /dev/da0 |
54 | +=> 40 62914480 da0 GPT (30G) |
55 | + 40 1024 1 freebsd-boot (512K) |
56 | + 1064 58719232 2 freebsd-ufs (28G) |
57 | + 58720296 3145728 3 freebsd-swap (1.5G) |
58 | + 61866024 1048496 - free - (512M) |
59 | + """ |
60 | + expect_sz = None |
61 | + m = re.search('^(/dev/.+)p([0-9])$', devpth) |
62 | + gpart_res = _get_gpart_output(m.group(1)) |
63 | + for line in gpart_res.splitlines(): |
64 | + if re.search(r"freebsd-ufs", line): |
65 | + fields = line.split() |
66 | + expect_sz = int(fields[1]) |
67 | + # Normalize the gpart sector size, |
68 | + # because the size is not exactly the same as fs size. |
69 | + normal_expect_sz = (expect_sz - expect_sz % (frag_sz / 512)) |
70 | + if normal_expect_sz == cur_fs_sz: |
71 | + return True |
72 | + else: |
73 | + return False |
74 | + |
75 | + |
76 | # Do not use a dictionary as these commands should be able to be used |
77 | # for multiple filesystem types if possible, e.g. one command for |
78 | # ext2, ext3 and ext4. |
79 | @@ -68,9 +127,40 @@ RESIZE_FS_PREFIXES_CMDS = [ |
80 | ('ufs', _resize_ufs), |
81 | ] |
82 | |
83 | +RESIZE_FS_PRECHECK_CMDS = { |
84 | + 'ufs': _can_skip_resize_ufs |
85 | +} |
86 | + |
87 | NOBLOCK = "noblock" |
88 | |
89 | |
90 | +def rootdev_from_cmdline(cmdline): |
91 | + found = None |
92 | + for tok in cmdline.split(): |
93 | + if tok.startswith("root="): |
94 | + found = tok[5:] |
95 | + break |
96 | + if found is None: |
97 | + return None |
98 | + |
99 | + if found.startswith("/dev/"): |
100 | + return found |
101 | + if found.startswith("LABEL="): |
102 | + return "/dev/disk/by-label/" + found[len("LABEL="):] |
103 | + if found.startswith("UUID="): |
104 | + return "/dev/disk/by-uuid/" + found[len("UUID="):] |
105 | + |
106 | + return "/dev/" + found |
107 | + |
108 | + |
109 | +def can_skip_resize(fs_type, resize_what, devpth): |
110 | + fstype_lc = fs_type.lower() |
111 | + for i, func in RESIZE_FS_PRECHECK_CMDS.items(): |
112 | + if fstype_lc.startswith(i): |
113 | + return func(resize_what, devpth) |
114 | + return False |
115 | + |
116 | + |
117 | def handle(name, cfg, _cloud, log, args): |
118 | if len(args) != 0: |
119 | resize_root = args[0] |
120 | @@ -139,6 +229,11 @@ def handle(name, cfg, _cloud, log, args): |
121 | return |
122 | |
123 | resizer = None |
124 | + if can_skip_resize(fs_type, resize_what, devpth): |
125 | + log.debug("Skip resize filesystem type %s for %s", |
126 | + fs_type, resize_what) |
127 | + return |
128 | + |
129 | fstype_lc = fs_type.lower() |
130 | for (pfix, root_cmd) in RESIZE_FS_PREFIXES_CMDS: |
131 | if fstype_lc.startswith(pfix): |
132 | diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py |
133 | index 28650b8..f56c0cf 100755 |
134 | --- a/cloudinit/distros/__init__.py |
135 | +++ b/cloudinit/distros/__init__.py |
136 | @@ -155,6 +155,9 @@ class Distro(object): |
137 | ns, header=header, render_hwaddress=True) |
138 | return self.apply_network(contents, bring_up=bring_up) |
139 | |
140 | + def generate_fallback_config(self): |
141 | + return net.generate_fallback_config() |
142 | + |
143 | def apply_network_config(self, netconfig, bring_up=False): |
144 | # apply network config netconfig |
145 | # This method is preferred to apply_network which only takes |
146 | diff --git a/cloudinit/distros/freebsd.py b/cloudinit/distros/freebsd.py |
147 | index 183e445..d9f31bf 100644 |
148 | --- a/cloudinit/distros/freebsd.py |
149 | +++ b/cloudinit/distros/freebsd.py |
150 | @@ -30,6 +30,7 @@ class Distro(distros.Distro): |
151 | login_conf_fn_bak = '/etc/login.conf.orig' |
152 | resolv_conf_fn = '/etc/resolv.conf' |
153 | ci_sudoers_fn = '/usr/local/etc/sudoers.d/90-cloud-init-users' |
154 | + default_primary_nic = 'hn0' |
155 | |
156 | def __init__(self, name, cfg, paths): |
157 | distros.Distro.__init__(self, name, cfg, paths) |
158 | @@ -38,6 +39,8 @@ class Distro(distros.Distro): |
159 | # should only happen say once per instance...) |
160 | self._runner = helpers.Runners(paths) |
161 | self.osfamily = 'freebsd' |
162 | + self.ipv4_pat = re.compile(r"\s+inet\s+\d+[.]\d+[.]\d+[.]\d+") |
163 | + cfg['ssh_svcname'] = 'sshd' |
164 | |
165 | # Updates a key in /etc/rc.conf. |
166 | def updatercconf(self, key, value): |
167 | @@ -183,7 +186,6 @@ class Distro(distros.Distro): |
168 | "gecos": '-c', |
169 | "primary_group": '-g', |
170 | "groups": '-G', |
171 | - "passwd": '-h', |
172 | "shell": '-s', |
173 | "inactive": '-E', |
174 | } |
175 | @@ -193,19 +195,11 @@ class Distro(distros.Distro): |
176 | "no_log_init": '--no-log-init', |
177 | } |
178 | |
179 | - redact_opts = ['passwd'] |
180 | - |
181 | for key, val in kwargs.items(): |
182 | if (key in adduser_opts and val and |
183 | isinstance(val, six.string_types)): |
184 | adduser_cmd.extend([adduser_opts[key], val]) |
185 | |
186 | - # Redact certain fields from the logs |
187 | - if key in redact_opts: |
188 | - log_adduser_cmd.extend([adduser_opts[key], 'REDACTED']) |
189 | - else: |
190 | - log_adduser_cmd.extend([adduser_opts[key], val]) |
191 | - |
192 | elif key in adduser_flags and val: |
193 | adduser_cmd.append(adduser_flags[key]) |
194 | log_adduser_cmd.append(adduser_flags[key]) |
195 | @@ -226,19 +220,21 @@ class Distro(distros.Distro): |
196 | except Exception as e: |
197 | util.logexc(LOG, "Failed to create user %s", name) |
198 | raise e |
199 | + # Set the password if it is provided |
200 | + # For security consideration, only hashed passwd is assumed |
201 | + passwd_val = kwargs.get('passwd', None) |
202 | + if passwd_val is not None: |
203 | + self.set_passwd(name, passwd_val, hashed=True) |
204 | |
205 | def set_passwd(self, user, passwd, hashed=False): |
206 | - cmd = ['pw', 'usermod', user] |
207 | - |
208 | if hashed: |
209 | - cmd.append('-H') |
210 | + hash_opt = "-H" |
211 | else: |
212 | - cmd.append('-h') |
213 | - |
214 | - cmd.append('0') |
215 | + hash_opt = "-h" |
216 | |
217 | try: |
218 | - util.subp(cmd, passwd, logstring="chpasswd for %s" % user) |
219 | + util.subp(['pw', 'usermod', user, hash_opt, '0'], |
220 | + data=passwd, logstring="chpasswd for %s" % user) |
221 | except Exception as e: |
222 | util.logexc(LOG, "Failed to set password for %s", user) |
223 | raise e |
224 | @@ -271,6 +267,255 @@ class Distro(distros.Distro): |
225 | keys = set(kwargs['ssh_authorized_keys']) or [] |
226 | ssh_util.setup_user_keys(keys, name, options=None) |
227 | |
228 | + @staticmethod |
229 | + def get_ifconfig_list(): |
230 | + cmd = ['ifconfig', '-l'] |
231 | + (nics, err) = util.subp(cmd, rcs=[0, 1]) |
232 | + if len(err): |
233 | + LOG.warn("Error running %s: %s", cmd, err) |
234 | + return None |
235 | + return nics |
236 | + |
237 | + @staticmethod |
238 | + def get_ifconfig_ifname_out(ifname): |
239 | + cmd = ['ifconfig', ifname] |
240 | + (if_result, err) = util.subp(cmd, rcs=[0, 1]) |
241 | + if len(err): |
242 | + LOG.warn("Error running %s: %s", cmd, err) |
243 | + return None |
244 | + return if_result |
245 | + |
246 | + @staticmethod |
247 | + def get_ifconfig_ether(): |
248 | + cmd = ['ifconfig', '-l', 'ether'] |
249 | + (nics, err) = util.subp(cmd, rcs=[0, 1]) |
250 | + if len(err): |
251 | + LOG.warn("Error running %s: %s", cmd, err) |
252 | + return None |
253 | + return nics |
254 | + |
255 | + @staticmethod |
256 | + def get_interface_mac(ifname): |
257 | + if_result = Distro.get_ifconfig_ifname_out(ifname) |
258 | + for item in if_result.splitlines(): |
259 | + if item.find('ether ') != -1: |
260 | + mac = str(item.split()[1]) |
261 | + if mac: |
262 | + return mac |
263 | + |
264 | + @staticmethod |
265 | + def get_devicelist(): |
266 | + nics = Distro.get_ifconfig_list() |
267 | + return nics.split() |
268 | + |
269 | + @staticmethod |
270 | + def get_ipv6(): |
271 | + ipv6 = [] |
272 | + nics = Distro.get_devicelist() |
273 | + for nic in nics: |
274 | + if_result = Distro.get_ifconfig_ifname_out(nic) |
275 | + for item in if_result.splitlines(): |
276 | + if item.find("inet6 ") != -1 and item.find("scopeid") == -1: |
277 | + ipv6.append(nic) |
278 | + return ipv6 |
279 | + |
280 | + def get_ipv4(self): |
281 | + ipv4 = [] |
282 | + nics = Distro.get_devicelist() |
283 | + for nic in nics: |
284 | + if_result = Distro.get_ifconfig_ifname_out(nic) |
285 | + for item in if_result.splitlines(): |
286 | + print(item) |
287 | + if self.ipv4_pat.match(item): |
288 | + ipv4.append(nic) |
289 | + return ipv4 |
290 | + |
291 | + def is_up(self, ifname): |
292 | + if_result = Distro.get_ifconfig_ifname_out(ifname) |
293 | + pat = "^" + ifname |
294 | + for item in if_result.splitlines(): |
295 | + if re.match(pat, item): |
296 | + flags = item.split('<')[1].split('>')[0] |
297 | + if flags.find("UP") != -1: |
298 | + return True |
299 | + |
300 | + def _get_current_rename_info(self, check_downable=True): |
301 | + """Collect information necessary for rename_interfaces.""" |
302 | + names = Distro.get_devicelist() |
303 | + bymac = {} |
304 | + for n in names: |
305 | + bymac[Distro.get_interface_mac(n)] = { |
306 | + 'name': n, 'up': self.is_up(n), 'downable': None} |
307 | + |
308 | + if check_downable: |
309 | + nics_with_addresses = set() |
310 | + ipv6 = self.get_ipv6() |
311 | + ipv4 = self.get_ipv4() |
312 | + for bytes_out in (ipv6, ipv4): |
313 | + for i in ipv6: |
314 | + nics_with_addresses.update(i) |
315 | + for i in ipv4: |
316 | + nics_with_addresses.update(i) |
317 | + |
318 | + for d in bymac.values(): |
319 | + d['downable'] = (d['up'] is False or |
320 | + d['name'] not in nics_with_addresses) |
321 | + |
322 | + return bymac |
323 | + |
324 | + def _rename_interfaces(self, renames): |
325 | + if not len(renames): |
326 | + LOG.debug("no interfaces to rename") |
327 | + return |
328 | + |
329 | + current_info = self._get_current_rename_info() |
330 | + |
331 | + cur_bymac = {} |
332 | + for mac, data in current_info.items(): |
333 | + cur = data.copy() |
334 | + cur['mac'] = mac |
335 | + cur_bymac[mac] = cur |
336 | + |
337 | + def update_byname(bymac): |
338 | + return dict((data['name'], data) |
339 | + for data in bymac.values()) |
340 | + |
341 | + def rename(cur, new): |
342 | + util.subp(["ifconfig", cur, "name", new], capture=True) |
343 | + |
344 | + def down(name): |
345 | + util.subp(["ifconfig", name, "down"], capture=True) |
346 | + |
347 | + def up(name): |
348 | + util.subp(["ifconfig", name, "up"], capture=True) |
349 | + |
350 | + ops = [] |
351 | + errors = [] |
352 | + ups = [] |
353 | + cur_byname = update_byname(cur_bymac) |
354 | + tmpname_fmt = "cirename%d" |
355 | + tmpi = -1 |
356 | + |
357 | + for mac, new_name in renames: |
358 | + cur = cur_bymac.get(mac, {}) |
359 | + cur_name = cur.get('name') |
360 | + cur_ops = [] |
361 | + if cur_name == new_name: |
362 | + # nothing to do |
363 | + continue |
364 | + |
365 | + if not cur_name: |
366 | + errors.append("[nic not present] Cannot rename mac=%s to %s" |
367 | + ", not available." % (mac, new_name)) |
368 | + continue |
369 | + |
370 | + if cur['up']: |
371 | + msg = "[busy] Error renaming mac=%s from %s to %s" |
372 | + if not cur['downable']: |
373 | + errors.append(msg % (mac, cur_name, new_name)) |
374 | + continue |
375 | + cur['up'] = False |
376 | + cur_ops.append(("down", mac, new_name, (cur_name,))) |
377 | + ups.append(("up", mac, new_name, (new_name,))) |
378 | + |
379 | + if new_name in cur_byname: |
380 | + target = cur_byname[new_name] |
381 | + if target['up']: |
382 | + msg = "[busy-target] Error renaming mac=%s from %s to %s." |
383 | + if not target['downable']: |
384 | + errors.append(msg % (mac, cur_name, new_name)) |
385 | + continue |
386 | + else: |
387 | + cur_ops.append(("down", mac, new_name, (new_name,))) |
388 | + |
389 | + tmp_name = None |
390 | + while tmp_name is None or tmp_name in cur_byname: |
391 | + tmpi += 1 |
392 | + tmp_name = tmpname_fmt % tmpi |
393 | + |
394 | + cur_ops.append(("rename", mac, new_name, (new_name, tmp_name))) |
395 | + target['name'] = tmp_name |
396 | + cur_byname = update_byname(cur_bymac) |
397 | + if target['up']: |
398 | + ups.append(("up", mac, new_name, (tmp_name,))) |
399 | + |
400 | + cur_ops.append(("rename", mac, new_name, (cur['name'], new_name))) |
401 | + cur['name'] = new_name |
402 | + cur_byname = update_byname(cur_bymac) |
403 | + ops += cur_ops |
404 | + |
405 | + opmap = {'rename': rename, 'down': down, 'up': up} |
406 | + if len(ops) + len(ups) == 0: |
407 | + if len(errors): |
408 | + LOG.debug("unable to do any work for renaming of %s", renames) |
409 | + else: |
410 | + LOG.debug("no work necessary for renaming of %s", renames) |
411 | + else: |
412 | + LOG.debug("achieving renaming of %s with ops %s", |
413 | + renames, ops + ups) |
414 | + |
415 | + for op, mac, new_name, params in ops + ups: |
416 | + try: |
417 | + opmap.get(op)(*params) |
418 | + except Exception as e: |
419 | + errors.append( |
420 | + "[unknown] Error performing %s%s for %s, %s: %s" % |
421 | + (op, params, mac, new_name, e)) |
422 | + if len(errors): |
423 | + raise Exception('\n'.join(errors)) |
424 | + |
425 | + def apply_network_config_names(self, netcfg): |
426 | + renames = [] |
427 | + for ent in netcfg.get('config', {}): |
428 | + if ent.get('type') != 'physical': |
429 | + continue |
430 | + mac = ent.get('mac_address') |
431 | + name = ent.get('name') |
432 | + if not mac: |
433 | + continue |
434 | + renames.append([mac, name]) |
435 | + return self._rename_interfaces(renames) |
436 | + |
437 | + @classmethod |
438 | + def generate_fallback_config(self): |
439 | + nics = Distro.get_ifconfig_ether() |
440 | + if nics is None: |
441 | + LOG.debug("Fail to get network interfaces") |
442 | + return None |
443 | + potential_interfaces = nics.split() |
444 | + connected = [] |
445 | + for nic in potential_interfaces: |
446 | + pat = "^" + nic |
447 | + if_result = Distro.get_ifconfig_ifname_out(nic) |
448 | + for item in if_result.split("\n"): |
449 | + if re.match(pat, item): |
450 | + flags = item.split('<')[1].split('>')[0] |
451 | + if flags.find("RUNNING") != -1: |
452 | + connected.append(nic) |
453 | + if connected: |
454 | + potential_interfaces = connected |
455 | + names = list(sorted(potential_interfaces)) |
456 | + default_pri_nic = Distro.default_primary_nic |
457 | + if default_pri_nic in names: |
458 | + names.remove(default_pri_nic) |
459 | + names.insert(0, default_pri_nic) |
460 | + target_name = None |
461 | + target_mac = None |
462 | + for name in names: |
463 | + mac = Distro.get_interface_mac(name) |
464 | + if mac: |
465 | + target_name = name |
466 | + target_mac = mac |
467 | + break |
468 | + if target_mac and target_name: |
469 | + nconf = {'config': [], 'version': 1} |
470 | + nconf['config'].append( |
471 | + {'type': 'physical', 'name': target_name, |
472 | + 'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]}) |
473 | + return nconf |
474 | + else: |
475 | + return None |
476 | + |
477 | def _write_network(self, settings): |
478 | entries = net_util.translate_network(settings) |
479 | nameservers = [] |
480 | diff --git a/cloudinit/settings.py b/cloudinit/settings.py |
481 | index dbafead..411960d 100644 |
482 | --- a/cloudinit/settings.py |
483 | +++ b/cloudinit/settings.py |
484 | @@ -39,7 +39,7 @@ CFG_BUILTIN = { |
485 | ], |
486 | 'def_log_file': '/var/log/cloud-init.log', |
487 | 'log_cfgs': [], |
488 | - 'syslog_fix_perms': ['syslog:adm', 'root:adm'], |
489 | + 'syslog_fix_perms': ['syslog:adm', 'root:adm', 'root:wheel'], |
490 | 'system_info': { |
491 | 'paths': { |
492 | 'cloud_dir': '/var/lib/cloud', |
493 | diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py |
494 | index 04358b7..5254e18 100644 |
495 | --- a/cloudinit/sources/DataSourceAzure.py |
496 | +++ b/cloudinit/sources/DataSourceAzure.py |
497 | @@ -10,6 +10,7 @@ import crypt |
498 | from functools import partial |
499 | import os |
500 | import os.path |
501 | +import re |
502 | import time |
503 | from xml.dom import minidom |
504 | import xml.etree.ElementTree as ET |
505 | @@ -32,19 +33,160 @@ BOUNCE_COMMAND = [ |
506 | # azure systems will always have a resource disk, and 66-azure-ephemeral.rules |
507 | # ensures that it gets linked to this path. |
508 | RESOURCE_DISK_PATH = '/dev/disk/cloud/azure_resource' |
509 | +DEFAULT_PRIMARY_NIC = 'eth0' |
510 | +LEASE_FILE = '/var/lib/dhcp/dhclient.eth0.leases' |
511 | +DEFAULT_FS = 'ext4' |
512 | + |
513 | + |
514 | +def find_storvscid_from_sysctl_pnpinfo(sysctl_out, deviceid): |
515 | + # extract the 'X' from dev.storvsc.X. if deviceid matches |
516 | + """ |
517 | + dev.storvsc.1.%pnpinfo: |
518 | + classid=32412632-86cb-44a2-9b5c-50d1417354f5 |
519 | + deviceid=00000000-0001-8899-0000-000000000000 |
520 | + """ |
521 | + for line in sysctl_out.splitlines(): |
522 | + if re.search(r"pnpinfo", line): |
523 | + fields = line.split() |
524 | + if len(fields) >= 3: |
525 | + columns = fields[2].split('=') |
526 | + if (len(columns) >= 2 and |
527 | + columns[0] == "deviceid" and |
528 | + columns[1].startswith(deviceid)): |
529 | + comps = fields[0].split('.') |
530 | + return comps[2] |
531 | + return None |
532 | + |
533 | + |
534 | +def find_busdev_from_disk(camcontrol_out, disk_drv): |
535 | + # find the scbusX from 'camcontrol devlist -b' output |
536 | + # if disk_drv matches the specified disk driver, i.e. blkvsc1 |
537 | + """ |
538 | + scbus0 on ata0 bus 0 |
539 | + scbus1 on ata1 bus 0 |
540 | + scbus2 on blkvsc0 bus 0 |
541 | + scbus3 on blkvsc1 bus 0 |
542 | + scbus4 on storvsc2 bus 0 |
543 | + scbus5 on storvsc3 bus 0 |
544 | + scbus-1 on xpt0 bus 0 |
545 | + """ |
546 | + for line in camcontrol_out.splitlines(): |
547 | + if re.search(disk_drv, line): |
548 | + items = line.split() |
549 | + return items[0] |
550 | + return None |
551 | + |
552 | + |
553 | +def find_dev_from_busdev(camcontrol_out, busdev): |
554 | + # find the daX from 'camcontrol devlist' output |
555 | + # if busdev matches the specified value, i.e. 'scbus2' |
556 | + """ |
557 | + <Msft Virtual CD/ROM 1.0> at scbus1 target 0 lun 0 (cd0,pass0) |
558 | + <Msft Virtual Disk 1.0> at scbus2 target 0 lun 0 (da0,pass1) |
559 | + <Msft Virtual Disk 1.0> at scbus3 target 1 lun 0 (da1,pass2) |
560 | + """ |
561 | + for line in camcontrol_out.splitlines(): |
562 | + if re.search(busdev, line): |
563 | + items = line.split('(') |
564 | + if len(items) == 2: |
565 | + dev_pass = items[1].split(',') |
566 | + return dev_pass[0] |
567 | + return None |
568 | + |
569 | + |
570 | +def get_dev_storvsc_sysctl(): |
571 | + try: |
572 | + sysctl_out, err = util.subp(['sysctl', 'dev.storvsc']) |
573 | + except util.ProcessExecutionError: |
574 | + LOG.debug("Fail to execute sysctl dev.storvsc") |
575 | + return None |
576 | + return sysctl_out |
577 | + |
578 | + |
579 | +def get_camcontrol_dev_bus(): |
580 | + try: |
581 | + camcontrol_b_out, err = util.subp(['camcontrol', 'devlist', '-b']) |
582 | + except util.ProcessExecutionError: |
583 | + LOG.debug("Fail to execute camcontrol devlist -b") |
584 | + return None |
585 | + return camcontrol_b_out |
586 | + |
587 | + |
588 | +def get_camcontrol_dev(): |
589 | + try: |
590 | + camcontrol_out, err = util.subp(['camcontrol', 'devlist']) |
591 | + except util.ProcessExecutionError: |
592 | + LOG.debug("Fail to execute camcontrol devlist") |
593 | + return None |
594 | + return camcontrol_out |
595 | + |
596 | + |
597 | +def get_resource_disk_on_freebsd(port_id): |
598 | + g0 = "00000000" |
599 | + if port_id > 1: |
600 | + g0 = "00000001" |
601 | + port_id = port_id - 2 |
602 | + g1 = "000" + str(port_id) |
603 | + g0g1 = "{0}-{1}".format(g0, g1) |
604 | + """ |
605 | + search 'X' from |
606 | + 'dev.storvsc.X.%pnpinfo: |
607 | + classid=32412632-86cb-44a2-9b5c-50d1417354f5 |
608 | + deviceid=00000000-0001-8899-0000-000000000000' |
609 | + """ |
610 | + sysctl_out = get_dev_storvsc_sysctl() |
611 | + |
612 | + storvscid = find_storvscid_from_sysctl_pnpinfo(sysctl_out, g0g1) |
613 | + if not storvscid: |
614 | + LOG.debug("Fail to find storvsc id from sysctl") |
615 | + return None |
616 | + |
617 | + camcontrol_b_out = get_camcontrol_dev_bus() |
618 | + camcontrol_out = get_camcontrol_dev() |
619 | + # try to find /dev/XX from 'blkvsc' device |
620 | + blkvsc = "blkvsc{0}".format(storvscid) |
621 | + scbusx = find_busdev_from_disk(camcontrol_b_out, blkvsc) |
622 | + if scbusx: |
623 | + devname = find_dev_from_busdev(camcontrol_out, scbusx) |
624 | + if devname is None: |
625 | + LOG.debug("Fail to find /dev/daX") |
626 | + return None |
627 | + return devname |
628 | + # try to find /dev/XX from 'storvsc' device |
629 | + storvsc = "storvsc{0}".format(storvscid) |
630 | + scbusx = find_busdev_from_disk(camcontrol_b_out, storvsc) |
631 | + if scbusx: |
632 | + devname = find_dev_from_busdev(camcontrol_out, scbusx) |
633 | + if devname is None: |
634 | + LOG.debug("Fail to find /dev/daX") |
635 | + return None |
636 | + return devname |
637 | + return None |
638 | + |
639 | +# update the FreeBSD specific information |
640 | +if util.is_FreeBSD(): |
641 | + DEFAULT_PRIMARY_NIC = 'hn0' |
642 | + LEASE_FILE = '/var/db/dhclient.leases.hn0' |
643 | + DEFAULT_FS = 'freebsd-ufs' |
644 | + res_disk = get_resource_disk_on_freebsd(1) |
645 | + if res_disk is not None: |
646 | + LOG.debug("resource disk is not None") |
647 | + RESOURCE_DISK_PATH = "/dev/" + res_disk |
648 | + else: |
649 | + LOG.debug("resource disk is None") |
650 | |
651 | BUILTIN_DS_CONFIG = { |
652 | 'agent_command': AGENT_START_BUILTIN, |
653 | 'data_dir': "/var/lib/waagent", |
654 | 'set_hostname': True, |
655 | 'hostname_bounce': { |
656 | - 'interface': 'eth0', |
657 | + 'interface': DEFAULT_PRIMARY_NIC, |
658 | 'policy': True, |
659 | 'command': BOUNCE_COMMAND, |
660 | 'hostname_command': 'hostname', |
661 | }, |
662 | 'disk_aliases': {'ephemeral0': RESOURCE_DISK_PATH}, |
663 | - 'dhclient_lease_file': '/var/lib/dhcp/dhclient.eth0.leases', |
664 | + 'dhclient_lease_file': LEASE_FILE, |
665 | } |
666 | |
667 | BUILTIN_CLOUD_CONFIG = { |
668 | @@ -53,7 +195,7 @@ BUILTIN_CLOUD_CONFIG = { |
669 | 'layout': [100], |
670 | 'overwrite': True}, |
671 | }, |
672 | - 'fs_setup': [{'filesystem': 'ext4', |
673 | + 'fs_setup': [{'filesystem': DEFAULT_FS, |
674 | 'device': 'ephemeral0.1', |
675 | 'replace_fs': 'ntfs'}], |
676 | } |
677 | @@ -190,7 +332,11 @@ class DataSourceAzureNet(sources.DataSource): |
678 | for cdev in candidates: |
679 | try: |
680 | if cdev.startswith("/dev/"): |
681 | - ret = util.mount_cb(cdev, load_azure_ds_dir) |
682 | + if util.is_FreeBSD(): |
683 | + ret = util.mount_cb(cdev, load_azure_ds_dir, |
684 | + mtype="udf", sync=False) |
685 | + else: |
686 | + ret = util.mount_cb(cdev, load_azure_ds_dir) |
687 | else: |
688 | ret = load_azure_ds_dir(cdev) |
689 | |
690 | @@ -218,11 +364,12 @@ class DataSourceAzureNet(sources.DataSource): |
691 | LOG.debug("using files cached in %s", ddir) |
692 | |
693 | # azure / hyper-v provides random data here |
694 | - seed = util.load_file("/sys/firmware/acpi/tables/OEM0", |
695 | - quiet=True, decode=False) |
696 | - if seed: |
697 | - self.metadata['random_seed'] = seed |
698 | - |
699 | + if not util.is_FreeBSD(): |
700 | + seed = util.load_file("/sys/firmware/acpi/tables/OEM0", |
701 | + quiet=True, decode=False) |
702 | + if seed: |
703 | + self.metadata['random_seed'] = seed |
704 | + # TODO. find the seed on FreeBSD platform |
705 | # now update ds_cfg to reflect contents pass in config |
706 | user_ds_cfg = util.get_cfg_by_path(self.cfg, DS_CFG_PATH, {}) |
707 | self.ds_cfg = util.mergemanydict([user_ds_cfg, self.ds_cfg]) |
708 | @@ -633,8 +780,19 @@ def encrypt_pass(password, salt_id="$6$"): |
709 | def list_possible_azure_ds_devs(): |
710 | # return a sorted list of devices that might have a azure datasource |
711 | devlist = [] |
712 | - for fstype in ("iso9660", "udf"): |
713 | - devlist.extend(util.find_devs_with("TYPE=%s" % fstype)) |
714 | + if util.is_FreeBSD(): |
715 | + cdrom_dev = "/dev/cd0" |
716 | + try: |
717 | + util.subp(["mount", "-o", "ro", "-t", "udf", cdrom_dev, |
718 | + "/mnt/cdrom/secure"]) |
719 | + except util.ProcessExecutionError: |
720 | + LOG.debug("Fail to mount cd") |
721 | + return devlist |
722 | + util.subp(["umount", "/mnt/cdrom/secure"]) |
723 | + devlist.append(cdrom_dev) |
724 | + else: |
725 | + for fstype in ("iso9660", "udf"): |
726 | + devlist.extend(util.find_devs_with("TYPE=%s" % fstype)) |
727 | |
728 | devlist.sort(reverse=True) |
729 | return devlist |
730 | diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py |
731 | index 6e01aa4..e22409d 100644 |
732 | --- a/cloudinit/sources/helpers/azure.py |
733 | +++ b/cloudinit/sources/helpers/azure.py |
734 | @@ -29,6 +29,14 @@ def cd(newdir): |
735 | os.chdir(prevdir) |
736 | |
737 | |
738 | +def _get_dhcp_endpoint_option_name(): |
739 | + if util.is_FreeBSD(): |
740 | + azure_endpoint = "option-245" |
741 | + else: |
742 | + azure_endpoint = "unknown-245" |
743 | + return azure_endpoint |
744 | + |
745 | + |
746 | class AzureEndpointHttpClient(object): |
747 | |
748 | headers = { |
749 | @@ -235,8 +243,9 @@ class WALinuxAgentShim(object): |
750 | leases = [] |
751 | content = util.load_file(fallback_lease_file) |
752 | LOG.debug("content is %s", content) |
753 | + option_name = _get_dhcp_endpoint_option_name() |
754 | for line in content.splitlines(): |
755 | - if 'unknown-245' in line: |
756 | + if option_name in line: |
757 | # Example line from Ubuntu |
758 | # option unknown-245 a8:3f:81:10; |
759 | leases.append(line.strip(' ').split(' ', 2)[-1].strip(';\n"')) |
760 | diff --git a/cloudinit/stages.py b/cloudinit/stages.py |
761 | index f7191b0..ad55782 100644 |
762 | --- a/cloudinit/stages.py |
763 | +++ b/cloudinit/stages.py |
764 | @@ -624,7 +624,7 @@ class Init(object): |
765 | return (None, loc) |
766 | if ncfg: |
767 | return (ncfg, loc) |
768 | - return (net.generate_fallback_config(), "fallback") |
769 | + return (self.distro.generate_fallback_config(), "fallback") |
770 | |
771 | def apply_network_config(self, bring_up): |
772 | netcfg, src = self._find_networking_config() |
773 | diff --git a/cloudinit/util.py b/cloudinit/util.py |
774 | index 22af99d..27a9833 100644 |
775 | --- a/cloudinit/util.py |
776 | +++ b/cloudinit/util.py |
777 | @@ -565,6 +565,10 @@ def is_ipv4(instr): |
778 | return len(toks) == 4 |
779 | |
780 | |
781 | +def is_FreeBSD(): |
782 | + return system_info()['platform'].startswith('FreeBSD') |
783 | + |
784 | + |
785 | def get_cfg_option_bool(yobj, key, default=False): |
786 | if key not in yobj: |
787 | return default |
788 | @@ -2055,11 +2059,56 @@ def parse_mtab(path): |
789 | return None |
790 | |
791 | |
792 | +def find_freebsd_part(label_part): |
793 | + if label_part.startswith("/dev/label/"): |
794 | + target_label = label_part[5:] |
795 | + (label_part, err) = subp(['glabel', 'status', '-s']) |
796 | + for labels in label_part.split("\n"): |
797 | + items = labels.split() |
798 | + if len(items) > 0 and items[0].startswith(target_label): |
799 | + label_part = items[2] |
800 | + break |
801 | + label_part = str(label_part) |
802 | + return label_part |
803 | + |
804 | + |
805 | +def get_path_dev_freebsd(path, mnt_list): |
806 | + path_found = None |
807 | + for line in mnt_list.split("\n"): |
808 | + items = line.split() |
809 | + if (len(items) > 2 and os.path.exists(items[1] + path)): |
810 | + path_found = line |
811 | + break |
812 | + return path_found |
813 | + |
814 | + |
815 | +def get_mount_info_freebsd(path, log=LOG): |
816 | + (result, err) = subp(['mount', '-p', path], rcs=[0, 1]) |
817 | + if len(err): |
818 | + # find a path if the input is not a mounting point |
819 | + (mnt_list, err) = subp(['mount', '-p']) |
820 | + path_found = get_path_dev_freebsd(path, mnt_list) |
821 | + if (path_found is None): |
822 | + return None |
823 | + result = path_found |
824 | + ret = result.split() |
825 | + label_part = find_freebsd_part(ret[0]) |
826 | + return "/dev/" + label_part, ret[2], ret[1] |
827 | + |
828 | + |
829 | def parse_mount(path): |
830 | (mountoutput, _err) = subp("mount") |
831 | mount_locs = mountoutput.splitlines() |
832 | for line in mount_locs: |
833 | m = re.search(r'^(/dev/[\S]+) on (/.*) \((.+), .+, (.+)\)$', line) |
834 | + if not m: |
835 | + continue |
836 | + # check whether the dev refers to a label on FreeBSD |
837 | + # for example, if dev is '/dev/label/rootfs', we should |
838 | + # continue finding the real device like '/dev/da0'. |
839 | + devm = re.search('^(/dev/.+)p([0-9])$', m.group(1)) |
840 | + if (not devm and is_FreeBSD()): |
841 | + return get_mount_info_freebsd(path) |
842 | devpth = m.group(1) |
843 | mount_point = m.group(2) |
844 | fs_type = m.group(3) |
845 | @@ -2336,7 +2385,8 @@ def read_dmi_data(key): |
846 | uname_arch = os.uname()[4] |
847 | if not (uname_arch == "x86_64" or |
848 | (uname_arch.startswith("i") and uname_arch[2:] == "86") or |
849 | - uname_arch == 'aarch64'): |
850 | + uname_arch == 'aarch64' or |
851 | + uname_arch == 'amd64'): |
852 | LOG.debug("dmidata is not supported on %s", uname_arch) |
853 | return None |
854 | |
855 | diff --git a/config/cloud.cfg-freebsd b/config/cloud.cfg-freebsd |
856 | index be664f5..d666c39 100644 |
857 | --- a/config/cloud.cfg-freebsd |
858 | +++ b/config/cloud.cfg-freebsd |
859 | @@ -5,7 +5,7 @@ syslog_fix_perms: root:wheel |
860 | |
861 | # This should not be required, but leave it in place until the real cause of |
862 | # not beeing able to find -any- datasources is resolved. |
863 | -datasource_list: ['ConfigDrive', 'OpenStack', 'Ec2'] |
864 | +datasource_list: ['ConfigDrive', 'Azure', 'OpenStack', 'Ec2'] |
865 | |
866 | # A set of users which may be applied and/or used by various modules |
867 | # when a 'default' entry is found it will reference the 'default_user' |
868 | diff --git a/setup.py b/setup.py |
869 | index 32a44d9..46d0521 100755 |
870 | --- a/setup.py |
871 | +++ b/setup.py |
872 | @@ -89,7 +89,6 @@ LIB = "/lib" |
873 | if os.uname()[0] == 'FreeBSD': |
874 | USR = "/usr/local" |
875 | USR_LIB_EXEC = "/usr/local/lib" |
876 | - ETC = "/usr/local/etc" |
877 | elif os.path.isfile('/etc/redhat-release'): |
878 | USR_LIB_EXEC = "/usr/libexec" |
879 | |
880 | @@ -164,8 +163,6 @@ else: |
881 | (ETC + '/cloud', glob('config/*.cfg')), |
882 | (ETC + '/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')), |
883 | (ETC + '/cloud/templates', glob('templates/*')), |
884 | - (ETC + '/NetworkManager/dispatcher.d/', ['tools/hook-network-manager']), |
885 | - (ETC + '/dhcp/dhclient-exit-hooks.d/', ['tools/hook-dhclient']), |
886 | (USR_LIB_EXEC + '/cloud-init', ['tools/ds-identify', |
887 | 'tools/uncloud-init', |
888 | 'tools/write-ssh-key-fingerprints']), |
889 | @@ -174,8 +171,13 @@ else: |
890 | [f for f in glob('doc/examples/*') if is_f(f)]), |
891 | (USR + '/share/doc/cloud-init/examples/seed', |
892 | [f for f in glob('doc/examples/seed/*') if is_f(f)]), |
893 | - (LIB + '/udev/rules.d', [f for f in glob('udev/*.rules')]), |
894 | ] |
895 | + if os.uname()[0] != 'FreeBSD': |
896 | + data_files.append([ |
897 | + (ETC + '/NetworkManager/dispatcher.d/', ['tools/hook-network-manager']), |
898 | + (ETC + '/dhcp/dhclient-exit-hooks.d/', ['tools/hook-dhclient']), |
899 | + (LIB + '/udev/rules.d', [f for f in glob('udev/*.rules')]) |
900 | + ]) |
901 | # Use a subclass for install that handles |
902 | # adding on the right init system configuration files |
903 | cmdclass = { |
904 | diff --git a/sysvinit/freebsd/cloudconfig b/sysvinit/freebsd/cloudconfig |
905 | index 01bc061..e4064fa 100755 |
906 | --- a/sysvinit/freebsd/cloudconfig |
907 | +++ b/sysvinit/freebsd/cloudconfig |
908 | @@ -7,24 +7,14 @@ |
909 | . /etc/rc.subr |
910 | |
911 | PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" |
912 | -export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg |
913 | |
914 | name="cloudconfig" |
915 | command="/usr/local/bin/cloud-init" |
916 | start_cmd="cloudconfig_start" |
917 | stop_cmd=":" |
918 | rcvar="cloudinit_enable" |
919 | -start_precmd="cloudinit_override" |
920 | start_cmd="cloudconfig_start" |
921 | |
922 | -cloudinit_override() |
923 | -{ |
924 | - # If there exist sysconfig/defaults variable override files use it... |
925 | - if [ -f /etc/defaults/cloud-init ]; then |
926 | - . /etc/defaults/cloud-init |
927 | - fi |
928 | -} |
929 | - |
930 | cloudconfig_start() |
931 | { |
932 | echo "${command} starting" |
933 | diff --git a/sysvinit/freebsd/cloudfinal b/sysvinit/freebsd/cloudfinal |
934 | index 1b487aa..b6894c3 100755 |
935 | --- a/sysvinit/freebsd/cloudfinal |
936 | +++ b/sysvinit/freebsd/cloudfinal |
937 | @@ -7,24 +7,14 @@ |
938 | . /etc/rc.subr |
939 | |
940 | PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" |
941 | -export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg |
942 | |
943 | name="cloudfinal" |
944 | command="/usr/local/bin/cloud-init" |
945 | start_cmd="cloudfinal_start" |
946 | stop_cmd=":" |
947 | rcvar="cloudinit_enable" |
948 | -start_precmd="cloudinit_override" |
949 | start_cmd="cloudfinal_start" |
950 | |
951 | -cloudinit_override() |
952 | -{ |
953 | - # If there exist sysconfig/defaults variable override files use it... |
954 | - if [ -f /etc/defaults/cloud-init ]; then |
955 | - . /etc/defaults/cloud-init |
956 | - fi |
957 | -} |
958 | - |
959 | cloudfinal_start() |
960 | { |
961 | echo -n "${command} starting" |
962 | diff --git a/sysvinit/freebsd/cloudinit b/sysvinit/freebsd/cloudinit |
963 | index 862eeab..3326300 100755 |
964 | --- a/sysvinit/freebsd/cloudinit |
965 | +++ b/sysvinit/freebsd/cloudinit |
966 | @@ -7,24 +7,14 @@ |
967 | . /etc/rc.subr |
968 | |
969 | PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" |
970 | -export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg |
971 | |
972 | name="cloudinit" |
973 | command="/usr/local/bin/cloud-init" |
974 | start_cmd="cloudinit_start" |
975 | stop_cmd=":" |
976 | rcvar="cloudinit_enable" |
977 | -start_precmd="cloudinit_override" |
978 | start_cmd="cloudinit_start" |
979 | |
980 | -cloudinit_override() |
981 | -{ |
982 | - # If there exist sysconfig/defaults variable override files use it... |
983 | - if [ -f /etc/defaults/cloud-init ]; then |
984 | - . /etc/defaults/cloud-init |
985 | - fi |
986 | -} |
987 | - |
988 | cloudinit_start() |
989 | { |
990 | echo -n "${command} starting" |
991 | diff --git a/sysvinit/freebsd/cloudinitlocal b/sysvinit/freebsd/cloudinitlocal |
992 | index fb342a0..11a5eb1 100755 |
993 | --- a/sysvinit/freebsd/cloudinitlocal |
994 | +++ b/sysvinit/freebsd/cloudinitlocal |
995 | @@ -1,30 +1,20 @@ |
996 | #!/bin/sh |
997 | |
998 | # PROVIDE: cloudinitlocal |
999 | -# REQUIRE: mountcritlocal |
1000 | +# REQUIRE: ldconfig mountcritlocal |
1001 | # BEFORE: NETWORKING FILESYSTEMS cloudinit cloudconfig cloudfinal |
1002 | |
1003 | . /etc/rc.subr |
1004 | |
1005 | PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" |
1006 | -export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg |
1007 | |
1008 | name="cloudinitlocal" |
1009 | command="/usr/local/bin/cloud-init" |
1010 | start_cmd="cloudlocal_start" |
1011 | stop_cmd=":" |
1012 | rcvar="cloudinit_enable" |
1013 | -start_precmd="cloudinit_override" |
1014 | start_cmd="cloudlocal_start" |
1015 | |
1016 | -cloudinit_override() |
1017 | -{ |
1018 | - # If there exist sysconfig/defaults variable override files use it... |
1019 | - if [ -f /etc/defaults/cloud-init ]; then |
1020 | - . /etc/defaults/cloud-init |
1021 | - fi |
1022 | -} |
1023 | - |
1024 | cloudlocal_start() |
1025 | { |
1026 | echo -n "${command} starting" |
1027 | diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py |
1028 | index 8d22bb5..e6b0dcb 100644 |
1029 | --- a/tests/unittests/test_datasource/test_azure.py |
1030 | +++ b/tests/unittests/test_datasource/test_azure.py |
1031 | @@ -3,6 +3,8 @@ |
1032 | from cloudinit import helpers |
1033 | from cloudinit.util import b64e, decode_binary, load_file |
1034 | from cloudinit.sources import DataSourceAzure |
1035 | +from cloudinit.util import find_freebsd_part |
1036 | +from cloudinit.util import get_path_dev_freebsd |
1037 | |
1038 | from ..helpers import TestCase, populate_dir, mock, ExitStack, PY26, SkipTest |
1039 | |
1040 | @@ -95,6 +97,41 @@ class TestAzureDataSource(TestCase): |
1041 | for module, name, new in patches: |
1042 | self.patches.enter_context(mock.patch.object(module, name, new)) |
1043 | |
1044 | + def _get_mockds(self): |
1045 | + mod = DataSourceAzure |
1046 | + sysctl_out = "dev.storvsc.3.%pnpinfo: "\ |
1047 | + "classid=ba6163d9-04a1-4d29-b605-72e2ffb1dc7f "\ |
1048 | + "deviceid=f8b3781b-1e82-4818-a1c3-63d806ec15bb\n" |
1049 | + sysctl_out += "dev.storvsc.2.%pnpinfo: "\ |
1050 | + "classid=ba6163d9-04a1-4d29-b605-72e2ffb1dc7f "\ |
1051 | + "deviceid=f8b3781a-1e82-4818-a1c3-63d806ec15bb\n" |
1052 | + sysctl_out += "dev.storvsc.1.%pnpinfo: "\ |
1053 | + "classid=32412632-86cb-44a2-9b5c-50d1417354f5 "\ |
1054 | + "deviceid=00000000-0001-8899-0000-000000000000\n" |
1055 | + camctl_devbus = """ |
1056 | +scbus0 on ata0 bus 0 |
1057 | +scbus1 on ata1 bus 0 |
1058 | +scbus2 on blkvsc0 bus 0 |
1059 | +scbus3 on blkvsc1 bus 0 |
1060 | +scbus4 on storvsc2 bus 0 |
1061 | +scbus5 on storvsc3 bus 0 |
1062 | +scbus-1 on xpt0 bus 0 |
1063 | + """ |
1064 | + camctl_dev = """ |
1065 | +<Msft Virtual CD/ROM 1.0> at scbus1 target 0 lun 0 (cd0,pass0) |
1066 | +<Msft Virtual Disk 1.0> at scbus2 target 0 lun 0 (da0,pass1) |
1067 | +<Msft Virtual Disk 1.0> at scbus3 target 1 lun 0 (da1,pass2) |
1068 | + """ |
1069 | + self.apply_patches([ |
1070 | + (mod, 'get_dev_storvsc_sysctl', mock.MagicMock( |
1071 | + return_value=sysctl_out)), |
1072 | + (mod, 'get_camcontrol_dev_bus', mock.MagicMock( |
1073 | + return_value=camctl_devbus)), |
1074 | + (mod, 'get_camcontrol_dev', mock.MagicMock( |
1075 | + return_value=camctl_dev)) |
1076 | + ]) |
1077 | + return mod |
1078 | + |
1079 | def _get_ds(self, data, agent_command=None): |
1080 | |
1081 | def dsdevs(): |
1082 | @@ -177,6 +214,34 @@ class TestAzureDataSource(TestCase): |
1083 | return |
1084 | raise AssertionError("XML is the same") |
1085 | |
1086 | + def test_get_resource_disk(self): |
1087 | + ds = self._get_mockds() |
1088 | + dev = ds.get_resource_disk_on_freebsd(1) |
1089 | + self.assertEqual("da1", dev) |
1090 | + |
1091 | + @mock.patch('cloudinit.util.subp') |
1092 | + def test_find_freebsd_part_on_Azure(self, mock_subp): |
1093 | + glabel_out = ''' |
1094 | +gptid/fa52d426-c337-11e6-8911-00155d4c5e47 N/A da0p1 |
1095 | + label/rootfs N/A da0p2 |
1096 | + label/swap N/A da0p3 |
1097 | +''' |
1098 | + mock_subp.return_value = (glabel_out, "") |
1099 | + res = find_freebsd_part("/dev/label/rootfs") |
1100 | + self.assertEqual("da0p2", res) |
1101 | + |
1102 | + def test_get_path_dev_freebsd_on_Azure(self): |
1103 | + mnt_list = ''' |
1104 | +/dev/label/rootfs / ufs rw 1 1 |
1105 | +devfs /dev devfs rw,multilabel 0 0 |
1106 | +fdescfs /dev/fd fdescfs rw 0 0 |
1107 | +/dev/da1s1 /mnt/resource ufs rw 2 2 |
1108 | +''' |
1109 | + with mock.patch.object(os.path, 'exists', |
1110 | + return_value=True): |
1111 | + res = get_path_dev_freebsd('/etc', mnt_list) |
1112 | + self.assertNotEqual(res, None) |
1113 | + |
1114 | def test_basic_seed_dir(self): |
1115 | odata = {'HostName': "myhost", 'UserName': "myuser"} |
1116 | data = {'ovfcontent': construct_valid_ovf_env(data=odata), |
1117 | diff --git a/tests/unittests/test_datasource/test_azure_helper.py b/tests/unittests/test_datasource/test_azure_helper.py |
1118 | index aafdebd..b2d2971 100644 |
1119 | --- a/tests/unittests/test_datasource/test_azure_helper.py |
1120 | +++ b/tests/unittests/test_datasource/test_azure_helper.py |
1121 | @@ -3,7 +3,6 @@ |
1122 | import os |
1123 | |
1124 | from cloudinit.sources.helpers import azure as azure_helper |
1125 | - |
1126 | from ..helpers import ExitStack, mock, TestCase |
1127 | |
1128 | |
1129 | @@ -72,10 +71,11 @@ class TestFindEndpoint(TestCase): |
1130 | |
1131 | @staticmethod |
1132 | def _build_lease_content(encoded_address): |
1133 | + endpoint = azure_helper._get_dhcp_endpoint_option_name() |
1134 | return '\n'.join([ |
1135 | 'lease {', |
1136 | ' interface "eth0";', |
1137 | - ' option unknown-245 {0};'.format(encoded_address), |
1138 | + ' option {0} {1};'.format(endpoint, encoded_address), |
1139 | '}']) |
1140 | |
1141 | def test_from_dhcp_client(self): |
1142 | diff --git a/tests/unittests/test_datasource/test_cloudstack.py b/tests/unittests/test_datasource/test_cloudstack.py |
1143 | index e93d28d..1d3d2f1 100644 |
1144 | --- a/tests/unittests/test_datasource/test_cloudstack.py |
1145 | +++ b/tests/unittests/test_datasource/test_cloudstack.py |
1146 | @@ -15,6 +15,11 @@ class TestCloudStackPasswordFetching(TestCase): |
1147 | mod_name = 'cloudinit.sources.DataSourceCloudStack' |
1148 | self.patches.enter_context(mock.patch('{0}.ec2'.format(mod_name))) |
1149 | self.patches.enter_context(mock.patch('{0}.uhelp'.format(mod_name))) |
1150 | + default_gw = "192.201.20.0" |
1151 | + mod_name = 'cloudinit.sources.DataSourceCloudStack.get_default_gateway' |
1152 | + get_default_gw = mock.MagicMock(return_value=default_gw) |
1153 | + self.patches.enter_context( |
1154 | + mock.patch(mod_name, get_default_gw)) |
1155 | |
1156 | def _set_password_server_response(self, response_string): |
1157 | subp = mock.MagicMock(return_value=(response_string, '')) |
1158 | diff --git a/tests/unittests/test_distros/test_netconfig.py b/tests/unittests/test_distros/test_netconfig.py |
1159 | index 8837066..1e10a33 100644 |
1160 | --- a/tests/unittests/test_distros/test_netconfig.py |
1161 | +++ b/tests/unittests/test_distros/test_netconfig.py |
1162 | @@ -178,6 +178,20 @@ class WriteBuffer(object): |
1163 | |
1164 | class TestNetCfgDistro(TestCase): |
1165 | |
1166 | + frbsd_ifout = """\ |
1167 | +hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 |
1168 | + options=51b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,LRO> |
1169 | + ether 00:15:5d:4c:73:00 |
1170 | + inet6 fe80::215:5dff:fe4c:7300%hn0 prefixlen 64 scopeid 0x2 |
1171 | + inet 10.156.76.127 netmask 0xfffffc00 broadcast 10.156.79.255 |
1172 | + nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> |
1173 | + media: Ethernet autoselect (10Gbase-T <full-duplex>) |
1174 | + status: active |
1175 | +""" |
1176 | + |
1177 | + def setUp(self): |
1178 | + super(TestNetCfgDistro, self).setUp() |
1179 | + |
1180 | def _get_distro(self, dname, renderers=None): |
1181 | cls = distros.fetch(dname) |
1182 | cfg = settings.CFG_BUILTIN |
1183 | @@ -251,6 +265,7 @@ class TestNetCfgDistro(TestCase): |
1184 | |
1185 | def test_apply_network_config_v1_to_netplan_ub(self): |
1186 | renderers = ['netplan'] |
1187 | + devlist = ['eth0', 'lo'] |
1188 | ub_distro = self._get_distro('ubuntu', renderers=renderers) |
1189 | with ExitStack() as mocks: |
1190 | write_bufs = {} |
1191 | @@ -272,6 +287,9 @@ class TestNetCfgDistro(TestCase): |
1192 | mock.patch.object(util, 'subp', return_value=(0, 0))) |
1193 | mocks.enter_context( |
1194 | mock.patch.object(os.path, 'isfile', return_value=False)) |
1195 | + mocks.enter_context( |
1196 | + mock.patch("cloudinit.net.netplan.get_devicelist", |
1197 | + return_value=devlist)) |
1198 | |
1199 | ub_distro.apply_network_config(V1_NET_CFG, False) |
1200 | |
1201 | @@ -285,6 +303,7 @@ class TestNetCfgDistro(TestCase): |
1202 | |
1203 | def test_apply_network_config_v2_passthrough_ub(self): |
1204 | renderers = ['netplan'] |
1205 | + devlist = ['eth0', 'lo'] |
1206 | ub_distro = self._get_distro('ubuntu', renderers=renderers) |
1207 | with ExitStack() as mocks: |
1208 | write_bufs = {} |
1209 | @@ -306,7 +325,10 @@ class TestNetCfgDistro(TestCase): |
1210 | mock.patch.object(util, 'subp', return_value=(0, 0))) |
1211 | mocks.enter_context( |
1212 | mock.patch.object(os.path, 'isfile', return_value=False)) |
1213 | - |
1214 | + # FreeBSD does not have '/sys/class/net' file, |
1215 | + # so we need mock here. |
1216 | + mocks.enter_context( |
1217 | + mock.patch.object(os, 'listdir', return_value=devlist)) |
1218 | ub_distro.apply_network_config(V2_NET_CFG, False) |
1219 | |
1220 | self.assertEqual(len(write_bufs), 1) |
1221 | @@ -328,6 +350,29 @@ class TestNetCfgDistro(TestCase): |
1222 | for (k, v) in b1.items(): |
1223 | self.assertEqual(v, b2[k]) |
1224 | |
1225 | + @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_list') |
1226 | + @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ifname_out') |
1227 | + def test_get_ip_nic_freebsd(self, ifname_out, iflist): |
1228 | + frbsd_distro = self._get_distro('freebsd') |
1229 | + iflist.return_value = "lo0 hn0" |
1230 | + ifname_out.return_value = self.frbsd_ifout |
1231 | + res = frbsd_distro.get_ipv4() |
1232 | + self.assertEqual(res, ['lo0', 'hn0']) |
1233 | + res = frbsd_distro.get_ipv6() |
1234 | + self.assertEqual(res, []) |
1235 | + |
1236 | + @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ether') |
1237 | + @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ifname_out') |
1238 | + @mock.patch('cloudinit.distros.freebsd.Distro.get_interface_mac') |
1239 | + def test_generate_fallback_config_freebsd(self, mac, ifname_out, if_ether): |
1240 | + frbsd_distro = self._get_distro('freebsd') |
1241 | + |
1242 | + if_ether.return_value = 'hn0' |
1243 | + ifname_out.return_value = self.frbsd_ifout |
1244 | + mac.return_value = '00:15:5d:4c:73:00' |
1245 | + res = frbsd_distro.generate_fallback_config() |
1246 | + self.assertIsNotNone(res) |
1247 | + |
1248 | def test_simple_write_rh(self): |
1249 | rh_distro = self._get_distro('rhel') |
1250 | |
1251 | diff --git a/tests/unittests/test_handler/test_handler_resizefs.py b/tests/unittests/test_handler/test_handler_resizefs.py |
1252 | new file mode 100644 |
1253 | index 0000000..52591b8 |
1254 | --- /dev/null |
1255 | +++ b/tests/unittests/test_handler/test_handler_resizefs.py |
1256 | @@ -0,0 +1,59 @@ |
1257 | +# This file is part of cloud-init. See LICENSE file for license information. |
1258 | + |
1259 | +from cloudinit.config import cc_resizefs |
1260 | + |
1261 | +import textwrap |
1262 | +import unittest |
1263 | + |
1264 | +try: |
1265 | + from unittest import mock |
1266 | +except ImportError: |
1267 | + import mock |
1268 | + |
1269 | + |
1270 | +class TestResizefs(unittest.TestCase): |
1271 | + def setUp(self): |
1272 | + super(TestResizefs, self).setUp() |
1273 | + self.name = "resizefs" |
1274 | + |
1275 | + @mock.patch('cloudinit.config.cc_resizefs._get_dumpfs_output') |
1276 | + @mock.patch('cloudinit.config.cc_resizefs._get_gpart_output') |
1277 | + def test_skip_ufs_resize(self, gpart_out, dumpfs_out): |
1278 | + fs_type = "ufs" |
1279 | + resize_what = "/" |
1280 | + devpth = "/dev/da0p2" |
1281 | + dumpfs_out.return_value = ( |
1282 | + "# newfs command for / (/dev/label/rootfs)\n" |
1283 | + "newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 " |
1284 | + "-f 4096 -g 16384 -h 64 -i 8192 -j -k 6408 -m 8 " |
1285 | + "-o time -s 58719232 /dev/label/rootfs\n") |
1286 | + gpart_out.return_value = textwrap.dedent("""\ |
1287 | + => 40 62914480 da0 GPT (30G) |
1288 | + 40 1024 1 freebsd-boot (512K) |
1289 | + 1064 58719232 2 freebsd-ufs (28G) |
1290 | + 58720296 3145728 3 freebsd-swap (1.5G) |
1291 | + 61866024 1048496 - free - (512M) |
1292 | + """) |
1293 | + res = cc_resizefs.can_skip_resize(fs_type, resize_what, devpth) |
1294 | + self.assertTrue(res) |
1295 | + |
1296 | + @mock.patch('cloudinit.config.cc_resizefs._get_dumpfs_output') |
1297 | + @mock.patch('cloudinit.config.cc_resizefs._get_gpart_output') |
1298 | + def test_skip_ufs_resize_roundup(self, gpart_out, dumpfs_out): |
1299 | + fs_type = "ufs" |
1300 | + resize_what = "/" |
1301 | + devpth = "/dev/da0p2" |
1302 | + dumpfs_out.return_value = ( |
1303 | + "# newfs command for / (/dev/label/rootfs)\n" |
1304 | + "newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 " |
1305 | + "-f 4096 -g 16384 -h 64 -i 8192 -j -k 368 -m 8 " |
1306 | + "-o time -s 297080 /dev/label/rootfs\n") |
1307 | + gpart_out.return_value = textwrap.dedent("""\ |
1308 | + => 34 297086 da0 GPT (145M) |
1309 | + 34 297086 1 freebsd-ufs (145M) |
1310 | + """) |
1311 | + res = cc_resizefs.can_skip_resize(fs_type, resize_what, devpth) |
1312 | + self.assertTrue(res) |
1313 | + |
1314 | + |
1315 | +# vi: ts=4 expandtab |
1316 | diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py |
1317 | index 89e7536..5d2dd03 100644 |
1318 | --- a/tests/unittests/test_net.py |
1319 | +++ b/tests/unittests/test_net.py |
1320 | @@ -1120,14 +1120,14 @@ class TestNetplanPostcommands(CiTestCase): |
1321 | render_target = 'netplan.yaml' |
1322 | renderer = netplan.Renderer( |
1323 | {'netplan_path': render_target, 'postcmds': True}) |
1324 | - renderer.render_network_state(render_dir, ns) |
1325 | - |
1326 | expected = [ |
1327 | mock.call(['netplan', 'generate'], capture=True), |
1328 | mock.call(['udevadm', 'test-builtin', 'net_setup_link', |
1329 | '/sys/class/net/lo'], capture=True), |
1330 | ] |
1331 | - mock_subp.assert_has_calls(expected) |
1332 | + with mock.patch.object(os.path, 'islink', return_value=True): |
1333 | + renderer.render_network_state(render_dir, ns) |
1334 | + mock_subp.assert_has_calls(expected) |
1335 | |
1336 | |
1337 | class TestEniNetworkStateToEni(CiTestCase): |
1338 | diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py |
1339 | index 5d21b4b..189caca 100644 |
1340 | --- a/tests/unittests/test_util.py |
1341 | +++ b/tests/unittests/test_util.py |
1342 | @@ -596,7 +596,8 @@ class TestSubp(helpers.TestCase): |
1343 | def test_subp_capture_stderr(self): |
1344 | data = b'hello world' |
1345 | (out, err) = util.subp(self.stdin2err, capture=True, |
1346 | - decode=False, data=data) |
1347 | + decode=False, data=data, |
1348 | + update_env={'LC_ALL': 'C'}) |
1349 | self.assertEqual(err, data) |
1350 | self.assertEqual(out, b'') |
1351 | |
1352 | diff --git a/tools/build-on-freebsd b/tools/build-on-freebsd |
1353 | index 8436498..ccc10b4 100755 |
1354 | --- a/tools/build-on-freebsd |
1355 | +++ b/tools/build-on-freebsd |
1356 | @@ -10,9 +10,7 @@ depschecked=/tmp/c-i.dependencieschecked |
1357 | pkgs=" |
1358 | dmidecode |
1359 | e2fsprogs |
1360 | - gpart |
1361 | py27-Jinja2 |
1362 | - py27-argparse |
1363 | py27-boto |
1364 | py27-cheetah |
1365 | py27-configobj |
1366 | @@ -38,7 +36,7 @@ python setup.py build |
1367 | python setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd |
1368 | |
1369 | # Install the correct config file: |
1370 | -cp config/cloud.cfg-freebsd /usr/local/etc/cloud/cloud.cfg |
1371 | +cp config/cloud.cfg-freebsd /etc/cloud/cloud.cfg |
1372 | |
1373 | # Enable cloud-init in /etc/rc.conf: |
1374 | sed -i.bak -e "/cloudinit_enable=.*/d" /etc/rc.conf |
Hongjjang,
Thanks for submitting this, I will try to take a look next week.
Sorry for the slow response.