[2.1] netaddr raises IndexError when looking up some OUI registrations

Can you provide the first 3 octets of one of the bad MACs? If you're not sure which one, the following is sure to catch it:

sudo maas-region dbshell
select distinct substring(cast(mac_address as varchar) for 8) from maasserver_interface;

This would appear to be a bug in the python3-netaddr, which MAAS should probably armor itself against.

I found some that exhibit the issue (00-22-59, 00-23-89) for starters.

MAAS was incomplete in the workaround for Bug#1628761.

Hm. I think the root cause of this might be that the index is not updated often enough (that is, it could be a Debian package mismatch).

The `netaddr` package defines the following:

ls -la /usr/lib/python3/dist-packages/netaddr/eui/oui.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python3/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt

readlink -f /usr/lib/python3/dist-packages/netaddr/eui/oui.txt
/usr/share/ieee-data/oui.txt

The `ieee-data` provides the actual data:

dpkg -S /usr/share/ieee-data/oui.txt
ieee-data: /usr/share/ieee-data/oui.txt

... so if one package is updated but not the other, the index in `netaddr` will become incorrect. I ran some tests with `netaddr` upstream and it seems like the library works fine with any OUI, even OUIs with incorrectly encoded characters in them, etc.

To test this theory, you could run netaddr as follows to regenerate the indexes:

    sudo python3 -m netaddr.eui.ieee

You would need to restart MAAS after doing this, since netaddr keeps the indexes cached in memory.

Interesting point. Here is what I see:

[xenial]
ii ieee-data 20150531.1 all OUI and IAB listings
ii python3-netaddr 0.7.18-1 all manipulation of various common netwo
% python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
'Guangzhou New Postcom Equipment Co.,Ltd.'

[zesty]
ii ieee-data 20160613.1 all OUI and IAB listings
ii python3-netaddr 0.7.18-2 all manipulation of various common network
% python3
Python 3.5.2+ (default, Dec 13 2016, 14:16:35)
[GCC 6.2.1 20161212] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 478, in oui
    return OUI(self.value >> 24)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 98, in __init__
    self._parse_data(data, offset, size)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 145, in _parse_data
    record['org'] = line.split(None, 2)[2]
IndexError: list index out of range
>>>

That is netaddr is not sufficiently hardened against the newer ieee-data. While it's conceivable that the actual bug is in ieee-data, it would seem that hardening the code to better deal with it would be good.

The entry that breaks reads:
00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CN
while the same entry in the working file is:
  00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
  002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CHINA
(Note the leading pair of spaces...)

Thanks for the test case. Here's what I saw after running `sudo python3 -m netaddr.eui.ieee` on my system:

>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
'Guangzhou New Postcom Equipment Co.,Ltd.'

So I conclude that the indexes are out of date.

My guess is `netaddr` expects to ship with the entire OUI database, but the Debian maintainer didn't want several different copies of the OUI database around the filesystem.

Thanks for quick response, I see you already have the MACs which cause problems, if needed I can print the ones which caused problem at my side

The actual source of the issue seems to be that netaddr indexes a file that it doesn't own (from ieee-data package), and then assumes that it won't change underneath it (either because of an ieee-data update, or a netaddr update in the case of a running process), which results in various exceptions and other errors from the code. I believe that a fresh bug is being filed against python-netaddr on the issue.

I filed two additional bugs to track this.

(1) For the netaddr package, so that the indexes remain consistent with the IEEE data:

https://launchpad.net/bugs/1655447

(2) For the bug in netaddr which allows a long-running process to retain an old cached index:

https://bugs.launchpad.net/netaddr/+bug/1655452

... also filed upstream, at:

https://github.com/drkjam/netaddr/issues/146