Merge ~james-hogarth/cloud-init:net-tools-deprecation into cloud-init:master
| Status: | Needs review | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Proposed branch: | ~james-hogarth/cloud-init:net-tools-deprecation | ||||||||
| Merge into: | cloud-init:master | ||||||||
| Diff against target: |
632 lines (+378/-124) 2 files modified
cloudinit/netinfo.py (+215/-64) cloudinit/tests/test_netinfo.py (+163/-60) |
||||||||
| Related bugs: |
|
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Chad Smith | 2017-11-14 | Needs Fixing on 2017-12-19 | |
| Server Team CI bot | continuous-integration | Needs Fixing on 2017-11-14 | |
|
Review via email:
|
|||
- ca9c666... by James Hogarth on 2017-11-14
| Ryan Harper (raharper) wrote : | # |
There's quite a bit of work to make the tables look the same.
Is that required or can we just dump a different format?
Please make sure the subp command used long options:
netstat --route --numeric --extended
ip --oneline ..
etc.
| James Hogarth (james-hogarth) wrote : | # |
I assumed we want to have the same output regardless of the tool... at
least initially.
It's the best way of ensuring both code paths are accurate at this time -
especially since new ifconfig syntax and route information is basically
completely broken.
The existing netstat and ifconfig used short options which is why I
continued that practice... switching to long is no big deal.
On 14 Nov 2017 20:08, "Ryan Harper" <email address hidden> wrote:
> There's quite a bit of work to make the tables look the same.
> Is that required or can we just dump a different format?
>
> Please make sure the subp command used long options:
>
> netstat --route --numeric --extended
> ip --oneline ..
>
> etc.
>
>
> Diff comments:
>
> > diff --git a/cloudinit/
> > index 993b26c..0b76301 100644
> > --- a/cloudinit/
> > +++ b/cloudinit/
> > @@ -74,6 +127,20 @@ def netdev_
> > pass
> > elif toks[i]
> > devs[curdev]
> > + return devs
> > +
> > +
> > +def netdev_
> > + devs = {}
> > + try:
> > + # Try iproute first of all
>
> iproute2
>
> > + (ipaddr_out, _err) = util.subp(["ip", "-o", "addr", "list"])
> > + (iplink_out, _err) = util.subp(["ip", "-o", "link", "list"])
> > + devs = netdev_
> > + except util.ProcessExe
> > + # Fall back to net-tools if iproute2 is not present
> > + (ifcfg_out, _err) = util.subp(
> > + devs = netdev_
> >
> > if empty != "":
> > for (_devname, dev) in devs.items():
> > @@ -84,14 +151,73 @@ def netdev_
> > return devs
> >
> >
> > -def route_info():
> > - (route_out, _err) = util.subp(
> > +def netdev_
> > + routes = {}
> > + routes['ipv4'] = []
> > + routes['ipv6'] = []
> > + entries = iproute_
> > + for line in entries:
> > + entry = {}
> > + if not line:
> > + continue
> > + toks = line.split()
> > + if toks[0] == "default":
> > + entry['
> > + entry['genmask'] = "0.0.0.0"
> > + entry['flags'] = "UG"
> > + else:
> > + (addr, cidr) = toks[0].split("/")
> > + entry['
> > + entry['genmask'] = netdev_
> > + entry['gateway'] = "0.0.0.0"
> > + entry['flags'] = "U"
> > + for i in range(len(toks)):
> > + if toks[i] == "via":
> > + entry['gateway'] = toks[i + 1]
> > + if toks[i] == "dev":
> > + entry["iface"] = toks[i + 1]
> > + if toks[i] == "metric":
> > + entry['metric'] = toks[i + 1]
> > + routes[
> > + try:
>
> it does seem odd to subp this here rather then above where you subp other
> ip commands
>
> > + (iproute_data6, _e...
| James Hogarth (james-hogarth) wrote : | # |
I did say that this wasn't ready for merge yet ;)
Once we have suitable test coverage in this area and actually have valid output, then I figure the time for tidiness will be valid (long opts rather than short, etc) for a final cleanup.
There will certainly be a lot of refactoring possible that I can already see which will cut the complexity ... but that has to come over tests are valid and passing.
I can already see a bunch of cleanup incoming in the process to refactor to handle multiple addresses.
| Ryan Harper (raharper) wrote : | # |
> I assumed we want to have the same output regardless of the tool... at
> least initially.
Right; I'm just asking the question (to other cloud-init devs as well as you)
if it's worth keeping the same format?
While I understand there may be tools that scrape the cloud-init log for
those values; I'd prefer to save execution time by dumping the iproute2
output values sort of as-is.
>
> It's the best way of ensuring both code paths are accurate at this time -
> especially since new ifconfig syntax and route information is basically
> completely broken.
For sure; and I think the unittests comparing both make sense.
That said, I do think we could save quite a bit of time and code
by dropping it.
>
> The existing netstat and ifconfig used short options which is why I
> continued that practice... switching to long is no big deal.
Yeah, makes sense; Since we're touching it, it's worth updating for
readers.
>
> On 14 Nov 2017 20:08, "Ryan Harper" <email address hidden> wrote:
>
> > There's quite a bit of work to make the tables look the same.
> > Is that required or can we just dump a different format?
> >
> > Please make sure the subp command used long options:
> >
> > netstat --route --numeric --extended
> > ip --oneline ..
> >
> > etc.
> >
> >
> > Diff comments:
> >
> > > diff --git a/cloudinit/
> > > index 993b26c..0b76301 100644
> > > --- a/cloudinit/
> > > +++ b/cloudinit/
> > > @@ -74,6 +127,20 @@ def netdev_
> > > pass
> > > elif toks[i]
> > > devs[curdev]
> > > + return devs
> > > +
> > > +
> > > +def netdev_
> > > + devs = {}
> > > + try:
> > > + # Try iproute first of all
> >
> > iproute2
> >
> > > + (ipaddr_out, _err) = util.subp(["ip", "-o", "addr", "list"])
> > > + (iplink_out, _err) = util.subp(["ip", "-o", "link", "list"])
> > > + devs = netdev_
> > > + except util.ProcessExe
> > > + # Fall back to net-tools if iproute2 is not present
> > > + (ifcfg_out, _err) = util.subp(
> > > + devs = netdev_
> > >
> > > if empty != "":
> > > for (_devname, dev) in devs.items():
> > > @@ -84,14 +151,73 @@ def netdev_
> > > return devs
> > >
> > >
> > > -def route_info():
> > > - (route_out, _err) = util.subp(
> > > +def netdev_
> > > + routes = {}
> > > + routes['ipv4'] = []
> > > + routes['ipv6'] = []
> > > + entries = iproute_
> > > + for line in entries:
> > > + entry = {}
> > > + if not line:
> > > + continue
> > > + toks = line.split()
> > > + if toks[0] == "default":
> > > + entry['
> > > + entry['genmask'] = "0.0.0.0"
> > > + entry['flags'] = "UG"
> > > + else:
> > > + (addr, cidr) = toks[0].split("/")
> > > + ...
| James Hogarth (james-hogarth) wrote : | # |
Is there anything else you're aware of that may call netdev_info() or route_info() directly?
If so we still need some gymnastics to ensure that the object returned is as expected, even if pformat isn't being called (which frankly is the simple bit).
A grep didn't find anything in the codebase, but I'm not familiar with cloud-init to know if it might be considered an interface or not... after all getgateway() isn't found via grep either.
You might want to wait for my refactor that handles new ifconfig output cleanly (nearly done with that) as it does clean up a lot of the ... uh ... awkward logic in parsing the ifconfig output.
- f87375c... by James Hogarth on 2017-11-14
FAILED: Continuous integration, rev:f87375c2ea2
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https:/
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
| Chad Smith (chad.smith) wrote : | # |
Thanks for this branch James, I've got a first round of comments here and have a minor fixup needed for running in an lxc that I'll post tomorrrow when I get to the bottom of what's happening.
but the following breaks on lxcs. The way routes are listed breaks your existing logic a bit:
it seems to fallover parsing the second line (probably because the route has an alias name @if64 in it and doesn't match the original key.
It might be as simple as truncating any aliases in the interface name after @.....
ip -o link list
1: lo: <LOOPBACK,
63: eth0@if64: <BROADCAST,
make deb
lxc launch ubuntu-daily:xenial x1
lxc file push cloud-init_
lxc exec x1 dpkg -i /cloud-init.deb
| Chad Smith (chad.smith) wrote : | # |
Here's the collection of review comments plus fix for lxc to truncate the ethX@interface# aliases
http://
| Chad Smith (chad.smith) wrote : | # |
Just saw it also lacks handling of host route targets routes and host gateway routes
On Azure cloud:
# old route format
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.4 metric 100
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4
10.0.0.1 dev eth0 proto dhcp scope link src 10.0.0.4 metric 100
168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp metric 100
169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp metric 100
# ip -o route list
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.4 metric 100
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4
10.0.0.1 dev eth0 proto dhcp scope link src 10.0.0.4 metric 100
168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp metric 100
169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp metric 100
The last 3 host routes cause tracebacksTraceback (most recent call last):
File "/usr/lib/
routes = route_info()
File "/usr/lib/
routes = netdev_
File "/usr/lib/
(addr, cidr) = toks[0].split("/")
| James Hogarth (james-hogarth) wrote : | # |
Yes as I mentioned it was a WIP not ready for merging, but given it's been
pending years and I'm on a drive to remove the net-tools dependency on
fedora I've picked it up.
If you can provide the ip route and ip addr output from your LXC container
that would be useful to add to the tests as I don't have that to hand.
Been busy with actual job and family past few weeks. Hopefully in the quiet
of Christmas I'll get this finished off.
On 20 Dec 2017 12:19, "Chad Smith" <email address hidden> wrote:
Just saw it also lacks handling of host route targets routes and host
gateway routes
On Azure cloud:
# old route format
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.4 metric 100
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4
10.0.0.1 dev eth0 proto dhcp scope link src 10.0.0.4 metric 100
168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp metric 100
169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp metric 100
# ip -o route list
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.4 metric 100
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4
10.0.0.1 dev eth0 proto dhcp scope link src 10.0.0.4 metric 100
168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp metric 100
169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp metric 100
The last 3 host routes cause tracebacksTraceback (most recent call last):
File "/usr/lib/
route_pformat
routes = route_info()
File "/usr/lib/
route_info
routes = netdev_
File "/usr/lib/
netdev_
(addr, cidr) = toks[0].split("/")
--
https:/
cloud-init/
You are the owner of ~james-
| Chad Smith (chad.smith) wrote : | # |
Expected cloud init ouput would have been
ci-info: +++++++
ci-info: +------
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +------
ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 2 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 3 | 10.0.0.1 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
ci-info: | 4 | 168.63.129.16 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH |
ci-info: | 5 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH |
ci-info: +------
Instead we got no route ci-info:
!!!!!!!
info.
here is the additional tweak needed to handlehost target routes
http://
| Chad Smith (chad.smith) wrote : | # |
collective patch
http://
Unmerged commits
- f87375c... by James Hogarth on 2017-11-14
- ca9c666... by James Hogarth on 2017-11-14
- 13f71fb... by James Hogarth on 2017-11-14
- f5c9f20... by James Hogarth on 2017-11-14
- 6412ac5... by James Hogarth on 2017-11-13
- 2dcfbde... by James Hogarth on 2017-11-13
- 27df90a... by James Hogarth on 2017-11-13
- ee51394... by James Hogarth on 2017-11-13
- 07be9d4... by James Hogarth on 2017-11-13
- db48caf... by James Hogarth on 2017-11-08


FAILED: Continuous integration, rev:13f71fbb9b0 90fc90e295e84ee d98719aa2252c8 /code.launchpad .net/~james- hogarth/ cloud-init/ +git/cloud- init/+merge/ 333657/ +edit-commit- message
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https:/
https:/ /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 489/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 489/rebuild
https:/