Ubuntu Cloud Archive

neutron-openvswitch-agent - error on startup

Bug #1630968 reported by James Page on 2016-10-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ubuntu Cloud Archive	Fix Released	High	James Page
	neutron (Ubuntu)	Fix Released	High	James Page

Bug Description

Looks like n-ovs-agent also needs a modprobe of br_netfilter on startup:

Oct 6 10:29:44 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:44.438 19914 ERROR neutron.agent.linux.utils [req-7e1b9f4f-92a7-4e10-8364-b20ec5fde487 - - - - -] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
Oct 6 10:29:45 ubuntu kernel: [ 1194.880868] Bridge firewalling registered
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7e1b9f4f-92a7-4e10-8364-b20ec5fde487 - - - - -] Error while processing VIF ports
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2047, in rpc_loop
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1655, in process_network_ports
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 265, in setup_port_filters
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 130, in decorated_function
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 138, in prepare_devices_filter
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 163, in _apply_port_filter
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/iptables_firewall.py", line 170, in prepare_port_filter
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._enable_netfilter_for_bridges()
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/iptables_firewall.py", line 114, in _enable_netfilter_for_bridges
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent run_as_root=True)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in execute
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(msg)
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Oct 6 10:29:46 ubuntu neutron-openvswitch-agent[19914]: 2016-10-06 10:29:46.976 19914 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: neutron-openvswitch-agent 2:9.0.0~rc3-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.4.0-38.57-generic 4.4.19
Uname: Linux 4.4.0-38-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CrashDB:
{
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Thu Oct 6 11:45:34 2016
Ec2AMI: ami-0000044a
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
SourcePackage: neutron
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.neutron.plugins.ml2.openvswitch_agent.ini: 2016-10-06T11:38:10.317795

Tags:

Revision history for this message

James Page (james-page) wrote on 2016-10-06:

Dependencies.txt Edit (9.5 KiB, text/plain; charset="utf-8")
JournalErrors.txt Edit (3.6 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (315 bytes, text/plain; charset="utf-8")
modified.conffile..etc.neutron.plugins.ml2.openvswitch_agent.ini.txt Edit (635 bytes, text/plain; charset="utf-8")

Changed in neutron (Ubuntu):
status:	New → Triaged
importance:	Undecided → High
Changed in cloud-archive:
status:	New → Triaged
importance:	Undecided → High
Changed in neutron (Ubuntu):
assignee:	nobody → James Page (james-page)
Changed in cloud-archive:
assignee:	nobody → James Page (james-page)

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2016-10-06:

Note armhf & s390x autopkgtests are executed in LXD containers, and thus one cannot modprobe kernel modules.

maybe one can check for the right CAPability to modprobe modules. Or e.g.

! systemd-detect-virt --container && exit 0

at the top of these scripts.

Revision history for this message

James Page (james-page) wrote on 2016-10-06:

The change to the systemd unit file should cope with running in a container; its an opportunistic load of the model, and the return code of the modprobe call is ignored.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-10-07:

This bug was fixed in the package neutron - 2:9.0.0-0ubuntu1

---------------
neutron (2:9.0.0-0ubuntu1) yakkety; urgency=medium

  [ Corey Bryant ]
  * d/control: oslo.log min version level in global-requirements is too low,
    so set min version to upper-constraints level (LP: #1628883).

  [ James Page ]
  * d/neutron-openvswitch-agent.service.in: Attempt load of br_netfilter
    module prior to starting service, ensuring that security rules will
    be applied to bridges associated with instances (LP: #1630968).

[ Corey Bryant ]
* New upstream release for OpenStack Newton.

-- Corey Bryant <email address hidden> Thu, 06 Oct 2016 13:36:38 -0400

Changed in neutron (Ubuntu):
status:	Triaged → Fix Released

James Page (james-page) on 2016-10-08

Changed in cloud-archive:
status:	Triaged → Fix Committed

Revision history for this message

James Page (james-page) wrote on 2016-10-11:

This bug was fixed in the package neutron - 2:9.0.0-0ubuntu1~cloud0
---------------

neutron (2:9.0.0-0ubuntu1~cloud0) xenial-newton; urgency=medium
.
   * New upstream release for the Ubuntu Cloud Archive.
.
neutron (2:9.0.0-0ubuntu1) yakkety; urgency=medium
.
   [ Corey Bryant ]
   * d/control: oslo.log min version level in global-requirements is too low,
     so set min version to upper-constraints level (LP: #1628883).
.
   [ James Page ]
   * d/neutron-openvswitch-agent.service.in: Attempt load of br_netfilter
     module prior to starting service, ensuring that security rules will
     be applied to bridges associated with instances (LP: #1630968).
.
   [ Corey Bryant ]
   * New upstream release for OpenStack Newton.