Ubuntu
openvswitch package

kernel fault ovs 1.10.1 + linux 3.11

Bug #1213879 reported by James Page
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
High
Unassigned
Saucy
Invalid
High
Unassigned
openvswitch (Ubuntu)
Fix Released
Critical
Unassigned
Saucy
Fix Released
Critical
Unassigned

Bug Description

I see this error when trying to execute the 'openflow' DEP-8 test with the latest 3.11 kernel on saucy:

Aug 19 10:14:06 armstrong kernel: [130348.598305] openvswitch: Open vSwitch switching datapath 1.10.1, built Aug 19 2013 10:13:35
Aug 19 10:15:43 armstrong kernel: [130445.653334] IPv6: ADDRCONF(NETDEV_UP): s1-eth1: link is not ready
Aug 19 10:15:43 armstrong kernel: [130445.659375] IPv6: ADDRCONF(NETDEV_UP): s1-eth1: link is not ready
Aug 19 10:15:43 armstrong kernel: [130445.662512] general protection fault: 0000 [#1] SMP
Aug 19 10:15:43 armstrong kernel: [130445.662543] Modules linked in: veth(F) openvswitch(OF) overlayfs(F) ipheth nls_utf8 isofs(F) ip6table_filter(F) ip6_tables(F) ebtable_nat(F) ebtables(F) xt_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) iptable_filter(F) dm_crypt(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) ip_tables(F) x_tables(F) bridge(F) stp(F) llc(F) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel(F) kvm(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) parport_pc(F) microcode(F) ppdev(F) psmouse(F) serio_raw(F) snd_hda_codec_hdmi snd_hda_codec_realtek arc4(F) joydev(F) rfcomm iwldvm mac80211 snd_usb_audio snd_usbmidi_lib lpc_ich uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core bnep videodev snd_hda_intel snd_hda_codec iwlwifi cfg80211 snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) thinkpad_acpi btusb nvram(F) bluetoot
Aug 19 10:15:43 armstrong kernel: h snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) mei_me mei snd(F) soundcore(F) tpm_tis mac_hid lp(F) parport(F) nls_iso8859_1(F) hid_generic hid_microsoft usbhid hid usb_storage(F) mmc_block i915 i2c_algo_bit drm_kms_helper drm sdhci_pci sdhci e1000e(F) ahci(F) libahci(F) ptp(F) pps_core(F) wmi video(F)
Aug 19 10:15:43 armstrong kernel: [130445.663077] CPU: 3 PID: 11974 Comm: ip Tainted: GF W O 3.11.0-2-generic #5-Ubuntu
Aug 19 10:15:43 armstrong kernel: [130445.663110] Hardware name: LENOVO 2324CTO/2324CTO, BIOS G2ET91WW (2.51 ) 01/14/2013
Aug 19 10:15:43 armstrong kernel: [130445.663142] task: ffff8803ba2b1770 ti: ffff8803d942c000 task.ti: ffff8803d942c000
Aug 19 10:15:43 armstrong kernel: [130445.663173] RIP: 0010:[<ffffffffa077d0d5>] [<ffffffffa077d0d5>] get_dpifindex+0x5/0x50 [openvswitch]
Aug 19 10:15:43 armstrong kernel: [130445.663215] RSP: 0018:ffff8803d942d698 EFLAGS: 00010286
Aug 19 10:15:43 armstrong kernel: [130445.663237] RAX: ffff88039656c014 RBX: ffff880402c60a00 RCX: 0000000000000000
Aug 19 10:15:43 armstrong kernel: [130445.663267] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0f90a2ebfffffb73
Aug 19 10:15:43 armstrong kernel: [130445.663296] RBP: ffff8803d942d710 R08: 0000000000000008 R09: ffff88039656c018
Aug 19 10:15:43 armstrong kernel: [130445.663322] R10: 0000000000000ec0 R11: 0000000000000000 R12: ffff88039656c014
Aug 19 10:15:43 armstrong kernel: [130445.663351] R13: ffffffff8115c109 R14: 0000000000000000 R15: 0000000000000002
Aug 19 10:15:43 armstrong kernel: [130445.663376] FS: 00007f8c9dfe5740(0000) GS:ffff88041e2c0000(0000) knlGS:0000000000000000
Aug 19 10:15:43 armstrong kernel: [130445.663406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 10:15:43 armstrong kernel: [130445.663430] CR2: 00007f29e7e8d005 CR3: 00000002e4bc0000 CR4: 00000000001407e0
Aug 19 10:15:43 armstrong kernel: [130445.663460] Stack:
Aug 19 10:15:43 armstrong kernel: [130445.663470] ffffffffa077d27e ffff880402c60a00 0000000000000020 00000000ffffffff
Aug 19 10:15:43 armstrong kernel: [130445.663504] ffff88040d803700 ffff880402c60a00 ffff8803d942d710 ffffffff815ec7a2
Aug 19 10:15:43 armstrong kernel: [130445.663538] 0000000000000001 00ff880406c1c000 ffff8803d942d7f8 ffff880402c60a00
Aug 19 10:15:43 armstrong kernel: [130445.663572] Call Trace:
Aug 19 10:15:43 armstrong kernel: [130445.663588] [<ffffffffa077d27e>] ? ovs_vport_cmd_fill_info+0x3e/0x170 [openvswitch]
Aug 19 10:15:43 armstrong kernel: [130445.663624] [<ffffffff815ec7a2>] ? __alloc_skb+0x92/0x2a0
Aug 19 10:15:43 armstrong kernel: [130445.663650] [<ffffffff8115c109>] ? zone_statistics+0x89/0xa0
Aug 19 10:15:43 armstrong kernel: [130445.663677] [<ffffffffa077fa28>] ovs_vport_cmd_build_info+0x58/0xa0 [openvswitch]
Aug 19 10:15:43 armstrong kernel: [130445.663712] [<ffffffff8115c109>] ? zone_statistics+0x89/0xa0
Aug 19 10:15:43 armstrong kernel: [130445.663739] [<ffffffffa0780042>] dp_device_event+0x72/0xe0 [openvswitch]
Aug 19 10:15:43 armstrong kernel: [130445.663771] [<ffffffff816fcd7c>] notifier_call_chain+0x4c/0x70
Aug 19 10:15:43 armstrong kernel: [130445.663799] [<ffffffff8108a026>] raw_notifier_call_chain+0x16/0x20
Aug 19 10:15:43 armstrong kernel: [130445.663828] [<ffffffff815f6b55>] call_netdevice_notifiers_info+0x35/0x60
Aug 19 10:15:43 armstrong kernel: [130445.663858] [<ffffffff815f9f13>] dev_change_net_namespace+0xc3/0x240
Aug 19 10:15:43 armstrong kernel: [130445.663889] [<ffffffff8160af53>] do_setlink+0x83/0x9f0
Aug 19 10:15:43 armstrong kernel: [130445.663914] [<ffffffff8137b906>] ? nla_parse+0x96/0xe0
Aug 19 10:15:43 armstrong kernel: [130445.663938] [<ffffffff8160c524>] rtnl_newlink+0x394/0x5e0
Aug 19 10:15:43 armstrong kernel: [130445.663964] [<ffffffff81314d94>] ? apparmor_capable+0x24/0xe0
Aug 19 10:15:43 armstrong kernel: [130445.663991] [<ffffffff81609159>] rtnetlink_rcv_msg+0x99/0x260
Aug 19 10:15:43 armstrong kernel: [130445.664018] [<ffffffff81190048>] ? __kmalloc_node_track_caller+0x58/0x1d0
Aug 19 10:15:43 armstrong kernel: [130445.664048] [<ffffffff8115c109>] ? zone_statistics+0x89/0xa0
Aug 19 10:15:43 armstrong kernel: [130445.664075] [<ffffffff815ec78e>] ? __alloc_skb+0x7e/0x2a0
Aug 19 10:15:43 armstrong kernel: [130445.664100] [<ffffffff816090c0>] ? rtnetlink_rcv+0x30/0x30
Aug 19 10:15:43 armstrong kernel: [130445.664124] [<ffffffff81626d29>] netlink_rcv_skb+0xa9/0xc0
Aug 19 10:15:43 armstrong kernel: [130445.664144] [<ffffffff816090b8>] rtnetlink_rcv+0x28/0x30
Aug 19 10:15:43 armstrong kernel: [130445.665267] [<ffffffff8162637d>] netlink_unicast+0xdd/0x190
Aug 19 10:15:43 armstrong kernel: [130445.666420] [<ffffffff8136c3cd>] ? memcpy_fromiovec+0x4d/0x90
Aug 19 10:15:43 armstrong kernel: [130445.667368] [<ffffffff8162672f>] netlink_sendmsg+0x2ff/0x740
Aug 19 10:15:43 armstrong kernel: [130445.668202] [<ffffffff815e42b9>] sock_sendmsg+0x99/0xd0
Aug 19 10:15:43 armstrong kernel: [130445.669030] [<ffffffff8135f2ae>] ? radix_tree_lookup_slot+0xe/0x10
Aug 19 10:15:43 armstrong kernel: [130445.670069] [<ffffffff815e46ac>] ___sys_sendmsg+0x36c/0x380
Aug 19 10:15:43 armstrong kernel: [130445.671144] [<ffffffff81168419>] ? handle_mm_fault+0x299/0x670
Aug 19 10:15:43 armstrong kernel: [130445.672079] [<ffffffff816fc9a4>] ? __do_page_fault+0x204/0x540
Aug 19 10:15:43 armstrong kernel: [130445.672887] [<ffffffff811ba68f>] ? __d_free+0x3f/0x60
Aug 19 10:15:43 armstrong kernel: [130445.673799] [<ffffffff8110434c>] ? acct_account_cputime+0x1c/0x20
Aug 19 10:15:43 armstrong kernel: [130445.674692] [<ffffffff8109795c>] ? account_user_time+0x9c/0xb0
Aug 19 10:15:43 armstrong kernel: [130445.675551] [<ffffffff815e5492>] __sys_sendmsg+0x42/0x80
Aug 19 10:15:43 armstrong kernel: [130445.676326] [<ffffffff815e54e2>] SyS_sendmsg+0x12/0x20
Aug 19 10:15:43 armstrong kernel: [130445.677144] [<ffffffff8170172f>] tracesys+0xe1/0xe6
Aug 19 10:15:43 armstrong kernel: [130445.677955] Code: 81 e1 ff 03 00 00 48 8d 14 ca 48 8b 0a 48 89 50 48 48 89 48 40 48 85 c9 48 89 32 74 04 48 89 71 08 5b 5d c3 66 90 0f 1f 44 00 00 <48> 8b 47 28 31 ff 48 8b 00 48 8d 50 c0 48 85 c0 48 0f 45 fa eb
Aug 19 10:15:43 armstrong kernel: [130445.679868] RIP [<ffffffffa077d0d5>] get_dpifindex+0x5/0x50 [openvswitch]
Aug 19 10:15:43 armstrong kernel: [130445.680742] RSP <ffff8803d942d698>
Aug 19 10:15:43 armstrong kernel: [130445.687447] ---[ end trace 98db0f68ac2d44b0 ]---
---
ApportVersion: 2.12-0ubuntu3
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: jamespage 2808 F.... pulseaudio
 /dev/snd/controlC0: jamespage 2808 F.... pulseaudio
DistroRelease: Ubuntu 13.10
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=9cabca8b-d2e3-4f65-a113-3c165c5ecbc6
InstallationDate: Installed on 2013-04-23 (117 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130423)
MachineType: LENOVO 2324CTO
MarkForUpload: True
Package: openvswitch
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.11.0-2-generic.efi.signed root=UUID=88b94abf-33eb-4e05-888d-9dacbe025054 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.11.0-2.5-generic 3.11.0-rc5
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-2-generic N/A
 linux-backports-modules-3.11.0-2-generic N/A
 linux-firmware 1.113
Tags: saucy saucy
Uname: Linux 3.11.0-2-generic x86_64
UpgradeStatus: Upgraded to saucy on 2013-06-15 (64 days ago)
UserGroups: adm cdrom dip libvirtd lpadmin plugdev sambashare sbuild sudo
dmi.bios.date: 01/14/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET91WW (2.51 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 Pro DPK TPG
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG2ET91WW(2.51):bd01/14/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvrWin8ProDPKTPG:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2324CTO
dmi.product.version: ThinkPad X230
dmi.sys.vendor: LENOVO

See original description
Revision history for this message
James Page (james-page) wrote :

DEP-8 testing looks like it sees the same:

https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/job/saucy-adt-openvswitch/35/

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1213879

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
James Page (james-page)
Changed in openvswitch (Ubuntu Saucy):
importance: Undecided → Critical
Revision history for this message
James Page (james-page) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected saucy
description: updated
Revision history for this message
James Page (james-page) wrote : BootDmesg.txt

apport information

Revision history for this message
James Page (james-page) wrote : CRDA.txt

apport information

Revision history for this message
James Page (james-page) wrote : CurrentDmesg.txt

apport information

Revision history for this message
James Page (james-page) wrote : IwConfig.txt

apport information

Revision history for this message
James Page (james-page) wrote : Lspci.txt

apport information

Revision history for this message
James Page (james-page) wrote : Lsusb.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcEnviron.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcInterrupts.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcModules.txt

apport information

Revision history for this message
James Page (james-page) wrote : PulseList.txt

apport information

Revision history for this message
James Page (james-page) wrote : RfKill.txt

apport information

Revision history for this message
James Page (james-page) wrote : UdevDb.txt

apport information

Revision history for this message
James Page (james-page) wrote : UdevLog.txt

apport information

Revision history for this message
James Page (james-page) wrote : WifiSyslog.txt

apport information

Changed in linux (Ubuntu Saucy):
status: Incomplete → Confirmed
penalvch (penalvch)
tags: added: bios-outdated-2.55 needs-upstream-testing regression-potential
Revision history for this message
penalvch (penalvch) wrote :

James Page, as per http://download.lenovo.com/express/ddfm.html an update is available for your BIOS (2.55). If you update to this, does it change anything?

If not, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Please note your current BIOS is already in the Bug Description, so posting this on the old BIOS would not be helpful.

Thank you for your understanding.

Changed in linux (Ubuntu Saucy):
status: Confirmed → Incomplete
Revision history for this message
James Page (james-page) wrote :
Download full text (5.1 KiB)

I'm aiming to fix this up in upstream trunk first and then backport the required commits to 1.10.

Right now I'm getting:

Aug 21 09:02:27 armstrong kernel: [54446.824413] ------------[ cut here ]------------
Aug 21 09:02:27 armstrong kernel: [54446.824425] WARNING: CPU: 1 PID: 8767 at /build/buildd/linux-3.11.0/net/core/dev.c:5078 rollback_registered_many+0x219/0x250()
Aug 21 09:02:27 armstrong kernel: [54446.824426] Modules linked in: openvswitch(OF) veth(F) mmc_block xt_REDIRECT(F) xt_hl(F) xt_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) ip_tables(F) x_tables(F) dm_crypt(F) bridge(F) stp(F) llc(F) gre(F) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel(F) kvm(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) ipheth snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec microcode(F) snd_usb_audio snd_usbmidi_lib snd_hwdep(F) thinkpad_acpi nvram(F) snd_pcm(F) snd_seq_midi(F) parport_pc(F) snd_seq_midi_event(F) ppdev(F) arc4(F) joydev(F) snd_rawmidi(F) uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev iwldvm snd_page_alloc(F) mac80211 btusb psmouse
Aug 21 09:02:27 armstrong kernel: (F) serio_raw(F) iwlwifi snd_seq(F) rfcomm bnep cfg80211 bluetooth lpc_ich snd_seq_device(F) snd_timer(F) snd(F) tpm_tis soundcore(F) mac_hid mei_me mei lp(F) parport(F) nls_iso8859_1(F) usb_storage(F) hid_generic hid_microsoft usbhid hid sdhci_pci i915 sdhci i2c_algo_bit ahci(F) libahci(F) drm_kms_helper drm e1000e(F) ptp(F) pps_core(F) wmi video(F) [last unloaded: openvswitch]
Aug 21 09:02:27 armstrong kernel: [54446.824498] CPU: 1 PID: 8767 Comm: ip Tainted: GF W O 3.11.0-2-generic #5-Ubuntu
Aug 21 09:02:27 armstrong kernel: [54446.824500] Hardware name: LENOVO 2324CTO/2324CTO, BIOS G2ET91WW (2.51 ) 01/14/2013
Aug 21 09:02:27 armstrong kernel: [54446.824502] 0000000000000009 ffff8803201518a8 ffffffff816f1668 0000000000000000
Aug 21 09:02:27 armstrong kernel: [54446.824504] ffff8803201518e0 ffffffff81061cdd ffff88025bc87000 ffff8803201518d8
Aug 21 09:02:27 armstrong kernel: [54446.824507] ffff880320151948 0000000000000000 ffffffff81cd3800 ffff8803201518f0
Aug 21 09:02:27 armstrong kernel: [54446.824510] Call Trace:
Aug 21 09:02:27 armstrong kernel: [54446.824515] [<ffffffff816f1668>] dump_stack+0x45/0x56
Aug 21 09:02:27 armstrong kernel: [54446.824519] [<ffffffff81061cdd>] warn_slowpath_common+0x7d/0xa0
Aug 21 09:02:27 armstrong kernel: [54446.824522] [<ffffffff81061dba>] warn_slowpath_null+0x1a/0x20
Aug 21 09:02:27 armstrong kernel: [54446.824524] [<ffffffff815f84b9>] rollback_registered_many+0x219/0x250
Aug 21 09:02:27 armstrong kernel: [54446.824527] [<ffffffff815f8547>] unregister_netdevice_many+0x17/0x70
Aug 21 09:02:27 armstrong kernel: [54446.824530] [<ffffffff8160bc6f>] rtnl_dellink+0xbf/0x130
Aug 21 09:02:27 armstrong kernel: [54446.824533] [<ffffffff81609159>] rtnetlink_rcv_msg+0x99/0x260
Aug 21 09:02:27...

Read more...

Revision history for this message
James Page (james-page) wrote :

forgot to note that openvswitch dkms appears to be functional

Revision history for this message
James Page (james-page) wrote :

@Christopher M. Penalver

Please read the context of bug reports before requesting actions which are not relevant to the bug.

The version of BIOS I have on my laptop is of no relevance to the fact that the openvswitch module dies horribly with 3.11.

If this is a bot please turn it off!

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

@James Page, Did this bug just start happening after a new kernel update? Was there a prior kernel that did not get this fault? If so, we can perform a bisect to identify the commit that introduced the regression.

Also, the 3.11.0-3.7 kernel is now available, which is rebased to upstream v3.11-rc6. Can you test this latest kernel to confirm it still exhibits this bug?

Changed in linux (Ubuntu Saucy):
importance: Undecided → High
tags: added: kernel-da-key performing-bisect
Revision history for this message
James Page (james-page) wrote :

@Joseph

3.10 was OK; the problem is that openvswitch expects to be able to register a protocol hander for GRE; however the kernel has already done this so it fails.

I think this then results in some badness later on openvswitch.

I'm trying to figure out how to replace the default GRE protocol hander from within the openvswitch module with its own implementation.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Thanks for the info, James. So in 3.10 and earlier the kernel did not register a protocol hander for GRE and now it does?

Revision history for this message
James Page (james-page) wrote :

Figured this out in the end - the native openvswitch module uses the gre module; during testing the gre module was not being unloaded during a force-reload of the kernel modules.

My last upload to saucy resolved this issue - stuck in -proposed right now...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 1.10.1+git20130823-0ubuntu2

---------------
openvswitch (1.10.1+git20130823-0ubuntu2) saucy; urgency=low

  * Fully support transitions from openvswitch module in 3.11 kernel to
    DKMS module:
    - d/openvswitch-switch.openvswitch-force-reload-mod.upstart: Drop the
      gre module when force reloading to make transitions from native kernel
      module to dkms module work.
    - d/NEWS: Drop information about DKMS/kernel compatibility with GRE
      tunnels.
 -- James Page <email address hidden> Sat, 24 Aug 2013 09:41:02 +0100

Changed in openvswitch (Ubuntu Saucy):
status: New → Fix Released
James Page (james-page)
Changed in linux (Ubuntu Saucy):
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.