[SRU] /usr/sbin/fanctl: arithmetic expression: expecting primary | unconfigured interfaces cause ifup failures

This might be related to ubuntu-fan and machines that have an unconfigured interface.
See also bug #1737721 which is a private bug with more information.

ubuntu-fan clearly triggers the problem.

Clean machine, no bridges after Juju installs ubuntu-fan and networking still works as bridges were not created yet => no ifup/down executions:

root@maas-xenial3:/home/ubuntu# ifdown -a ; ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# apt purge ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  ubuntu-fan*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 148 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 65843 files and directories currently installed.)
ubuntu-fan: removing default /etc/network/fan configuration
Purging configuration files for ubuntu-fan (0.12.8~16.04.1) ...

root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# ifdown ens3
ifdown: interface ens3 not configured

root@maas-xenial3:/home/ubuntu# ip a d 10.10.101.103/24 dev ens3
root@maas-xenial3:/home/ubuntu# ifup -a

root@maas-xenial3:/home/ubuntu# apt install ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  ubuntu-fan
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/35.6 kB of archives.
Selecting previously unselected package ubuntu-fan.ce will be used.
Processing triggers for systemd (229-4ubuntu21) ...urrently installed.)
Processing triggers for ureadahead (0.100.0-19) ...ll.deb ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21) ...
root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

Moved to field critical

Before and after ifdown -> ifup with fan:

http://paste.ubuntu.com/26169937/

After purging:

http://paste.ubuntu.com/26169950/

Status changed to 'Confirmed' because the bug affects multiple users.

This is reproducible on a machine with a single NIC:

https://paste.ubuntu.com/26170595/

Looks like we may have a fix for this. There is a test package in the PPA below for those who can test:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/fan-fixes/+packages

Tested that on a "broken" environment (after a reboot with a broken package):

root@maas-xenial3:~# apt install ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  ubuntu-fan
1 upgraded, 0 newly installed, 0 to remove and 94 not upgraded.
Need to get 36.4 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ppa.launchpad.net/canonical-kernel-team/fan-fixes/ubuntu xenial/main amd64 ubuntu-fan all 0.12.8~16.04.2~rc1 [36.4 kB]
Fetched 36.4 kB in 0s (0 B/s)
(Reading database ... 60144 files and directories currently installed.)
Preparing to unpack .../ubuntu-fan_0.12.8~16.04.2~rc1_all.deb ...
Unpacking ubuntu-fan (0.12.8~16.04.2~rc1) over (0.12.8~16.04.1) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up ubuntu-fan (0.12.8~16.04.2~rc1) ...

# after a reboot with a broken package an address was present but the interface was not considered as "configured" by ifupdown - had to manually delete an address to avoid RTNETLINK answers: File exists on ifup

root@maas-xenial3:~# ip a d 10.10.101.104/24 dev ens3

root@maas-xenial3:~# ifdown ens3
ifdown: interface ens3 not configured

root@maas-xenial3:~# ifup ens3

root@maas-xenial3:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:82:7d:16 brd ff:ff:ff:ff:ff:ff
    inet 10.10.101.104/24 brd 10.10.101.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe82:7d16/64 scope link
       valid_lft forever preferred_lft forever

# all good
root@maas-xenial3:~# ifdown -a
root@maas-xenial3:~# ifup -a

Will upload this to xenial-proposed urgently.

Hello Jason, or anyone else affected,

Accepted ubuntu-fan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.8~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

On xenial, deployed with maas 2.3.0. Node with two network interfaces: the first one (ens3) set to "auto" in maas, and the second one (ens8) left as "unconfigured" in MAAS:
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:58:04:57 brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.10/24 brd 10.0.5.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe58:457/64 scope link
       valid_lft forever preferred_lft forever
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:66:6c:1c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe66:6c1c/64 scope link
       valid_lft forever preferred_lft forever

With ubuntu-fan 0.12.8~16.04.1 installed:

root@04-57:~# ifdown -a; ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.

That also leaves the networking system in a bit of an inconsistent state:
root@04-57:~# ifdown ens3
ifdown: interface ens3 not configured

Bug confirmed.

I chose to reboot now, and then install the updated ubuntu-fan package from proposed:

SKIP - package not available yet - will continue sometime later

Ok, package available in proposed.
  Version table:
 *** 0.12.8~16.04.2 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages

Confusing result right after the upgrade:
root@04-57:~# ifdown -a; ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.
root@04-57:~# ifdown ens3
ifdown: interface ens3 not configured

rebooting

Now it works:root@04-57:~# ifdown -a; ifup -a
root@04-57:~#

If this test is acceptable, then consider this verified for xenial.

I can confirm that the proposed ubuntu-fan package resolves the issue.

If you are deploying with Juju and MaaS the Ubuntu proposed repository can be installed automatically on the nodes deployed by MaaS. This could be used as a temporary workaround for the issue.

Example for how it could be set up in MaaS:
https://gist.githubusercontent.com/fnordahl/f6e6b734484363e980a6e7fa2cfe51de/raw/b64d83225e0d11c1bd236e4f4625b0c466ad755f/maas-ubuntu-proposed.png

Changing tag to verified as per comments #13 & #14.

I just tested this also and can verify it fixed it in the environment/test where it was originally reported as broken.

thinking this is the same bug/duplicate I'm hitting here https://bugs.launchpad.net/juju/+bug/1736022

I'm deploying a few nodes to test the fix in proposed now

I can verify the fix works in my environment using whats in xenial-proposed http://paste.ubuntu.com/26178059/

Thanks!

Testing on arm64, the workaround of adding xenial-proposed via maas doesn't work - the newer ubuntu-fan package isn't being installed

http://paste.ubuntu.com/26178859/

I don't know how that can be, since the repo is being added (or should be added) before juju installs ubuntu-fan.

The workaround by adding the proposed repository into package repositories in MAAS worked for me. Thanks for the help!

The workaround by adding the proposed repository into package repositories in MAAS worked for me.

12:58 <rbasak> apw: no SRU paperwork in bug 1737640? I don't know how to check that the SRU verification minimises regression risk.

Blocked on:

 * SRU paperwork
 * Fixes for future stable releases
 * Fix for Bionic

I understand that this was being treated as urgent. But it's been eight days, and so if this were still the case I'd have expected the other essential pieces to have been completed by now. I'm not sure it can be treated as urgent any more.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug certainly needs to be treated as urgent. It's being tracked as part of the critical SLA process and blocks deployments of foundation bundles in the QA lab and in the field.

Is there anyone from ubuntu-fan developers willing to satisfy SRU requirements below?

> Blocked on:
>
> * SRU paperwork
> * Fixes for future stable releases
> * Fix for Bionic

I could try those, but it's not ideal since I have to go through sponsorship processes and guess the intention of the patch.

This bug is critical to me since the proposed workaround for MAAS deployments as adding -proposed in MAAS is not realistic in production cloud because:

- MAAS is not capable to add ubuntu-release aware apt line for -proposed using the custom repository management (trusty nodes will also have xenial-proposed apt line unexpectedly)
- MAAS cannot specify the pin-priority of xenial-proposed using the custom repository management so unnecessary kernels and grub will be installed from -proposed

I ended up using a little bit verbose late_commands in curtin_userdata which is not so handy:
https://paste.ubuntu.com/26319353/

This bug was fixed in the package ubuntu-fan - 0.12.8~16.04.2

---------------
ubuntu-fan (0.12.8~16.04.2) xenial; urgency=medium

  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Andy Whitcroft <email address hidden> Tue, 12 Dec 2017 15:34:26 +0000

The verification of the Stable Release Update for ubuntu-fan has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

As per earlier arrangements, I have published the package as-is as per the provided test feedback in comments. It's a one-off exception, bionic and other series will be prepared ASAP.

I can see that ubuntu-fan in zesty, artful and bionic needs this patch, but I can't determine if it's actually hit by this bug.

artful, for example, will use netplan in a fresh install, not e-n-i. Are these releases really affected by this bug?

I mean, I can see the error in netplan systems like artful, but maybe the impact there is different and local to ubuntu-fan only:

root@04-57:~# fanctl up -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "

Zesty will be EOL before we can push this fix out, marking it as "won't fix"

The bug exists in artful and bionic, it's probably just not as critical as it was ifupdown systems like xenial and zesty.

This bug was fixed in the package ubuntu-fan - 0.12.9

---------------
ubuntu-fan (0.12.9) bionic; urgency=medium

  [ Andy Whitcroft ]
  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Andreas Hasenack <email address hidden> Thu, 04 Jan 2018 15:41:32 -0200

Hello Jason, or anyone else affected,

Accepted ubuntu-fan into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.9~17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I did "systemctl restart NetworkManager" and observed errors from fanctl in my syslog. Then I upgraded ubuntu-fan to 0.12.9~17.10.1 and did "systemctl restart NetworkManager" again, and this time there were no such errors. However, I'm not certain that this is an adequate test, so I'm not changing the tags.

I think the previous test was as valid but to be completely sure I added an additional interface to a VM, installed ifupdown and assigned the additional interface to ifupdown control. Using 0.12.8~17.10.1 the error was observable and the status of the interface was incorrect. Upgrading to 0.12.9~17.10.1, ifup and ifdown worked correctly.

This bug was fixed in the package ubuntu-fan - 0.12.9~17.10.1

---------------
ubuntu-fan (0.12.9~17.10.1) artful; urgency=medium

  [ Andy Whitcroft ]
  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Stefan Bader <email address hidden> Tue, 30 Jan 2018 10:54:13 +0100