libvirt default network doesn't start, iptables errors, bad rules

I have the same error after install KVM and use virtual-manager.
I try to add some iptables rules but nothing change.
We wait somebody help us to resolve this problem
thanks andrea

Hi there,

I'm hitting the same bug with an explanation there :
http://www.spinics.net/lists/kvm/msg42207.html

libvirt "faulty" change :
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=fd5b15ff1a2ec37e75609c091522ae1e2c74c811

and a quick way to test issue:
# /sbin/iptables --table mangle -A POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM

iptables v1.4.4: Couldn't load target `CHECKSUM':/lib/xtables/libipt_CHECKSUM.so: cannot open shared object file: No such file or directory

-------- Work around .....

.... Downgrade libvirt to 0.8.2 should let us use kvm with virt-manager ....

------------------- Extra Infos -------------------

------ Missing lib there:
# ls /lib/xtables/
libip6t_ah.so libipt_CLUSTERIP.so libipt_SNAT.so libxt_dscp.so libxt_owner.so libxt_tcpmss.so
libip6t_dst.so libipt_DNAT.so libipt_ttl.so libxt_DSCP.so libxt_physdev.so libxt_TCPMSS.so
libip6t_eui64.so libipt_ecn.so libipt_TTL.so libxt_esp.so libxt_pkttype.so libxt_TCPOPTSTRIP.so
libip6t_frag.so libipt_ECN.so libipt_ULOG.so libxt_hashlimit.so libxt_policy.so libxt_tcp.so
libip6t_hbh.so libipt_icmp.so libipt_unclean.so libxt_helper.so libxt_quota.so libxt_time.so
libip6t_hl.so libipt_LOG.so libxt_CLASSIFY.so libxt_iprange.so libxt_rateest.so libxt_tos.so
libip6t_HL.so libipt_MASQUERADE.so libxt_cluster.so libxt_length.so libxt_RATEEST.so libxt_TOS.so
libip6t_icmp6.so libipt_MIRROR.so libxt_comment.so libxt_limit.so libxt_recent.so libxt_TPROXY.so
libip6t_ipv6header.so libipt_NETMAP.so libxt_connbytes.so libxt_mac.so libxt_sctp.so libxt_TRACE.so
libip6t_LOG.so libipt_realm.so libxt_connlimit.so libxt_mark.so libxt_SECMARK.so libxt_u32.so
libip6t_mh.so libipt_REDIRECT.so libxt_connmark.so libxt_MARK.so libxt_socket.so libxt_udp.so
libip6t_REJECT.so libipt_REJECT.so libxt_CONNMARK.so libxt_multiport.so libxt_standard.so
libip6t_rt.so libipt_SAME.so libxt_CONNSECMARK.so libxt_NFLOG.so libxt_state.so
libipt_addrtype.so libipt_set.so libxt_conntrack.so libxt_NFQUEUE.so libxt_statistic.so
libipt_ah.so libipt_SET.so libxt_dccp.so libxt_NOTRACK.so libxt_string.so

--------- Kervel version
# uname -a
Linux ben-desktop 2.6.35-23-generic #36-Ubuntu SMP Tue Oct 26 17:13:06 UTC 2010 x86_64 GNU/Linux

-------- KVM version
# kvm --version
QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard

------- libvirt ve...

3 months ago this bug was filed and triaged and yet nothing has been done about it. You do realize that this prevents us from being able to configure networks in virt-manager, right? You know that this means that nothing except basic virtual guests can be created right?

Fixing this is simply a matter of building virt-manger 0.8.5 and putting it into maverick-updates. Can we please have that?

Brian, I'm sorry that you haven't received any direct response on this issue. This bug actually wasn't triaged fully, it was just set to medium importance. Since there are multiple users showing as affected, I'll mark the issue as Confirmed.

Can you please clarify something, is the git link from the description the problem, as in, did it cause this issue, or the solution to this bug? If its the solution, we can definitely mark this bug as Triaged and just need to apply the patch.

Generally we won't update to a new upstream version in a stable release, unless this one is just horribly broken, or all of next version are already cherry-picked into this one.

Anyway, marking Confirmed, and nominating for Maverick.

This patch fixes the problem.

libvirt0 and libvirt-bin packages with the proposed fix.

Please test the packages from the attachment in comment #6. (You can also get them from ppa:serge-hallyn/virt) If these do indeed fix the issue for you, then I'll issue a SRU to get this fix into maverick.

I've marked the main bug 'fix released' since natty has the fix from upstream (and since this is a prerequisite for SRU).

I get:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/host.py", line 401, in start_network
    net.start()
  File "/usr/share/virt-manager/virtManager/network.py", line 97, in start
    self.net.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 866, in create
    if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
libvirtError: internal error '/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' exited with non-zero status 2 and signal 0: iptables v1.4.4: unknown option `--checksum-fill'
Try `iptables -h' or 'iptables --help' for more information.

Error starting network: internal error '/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' exited with non-zero status 2 and signal 0: iptables v1.4.4: unknown option `--checksum-fill'
Try `iptables -h' or 'iptables --help' for more information.

:(

Regards
Alessandro

Fascinating - I'm looking at https://bugzilla.redhat.com/show_bug.cgi?id=642355.

Alessandro,

All references to this that I find online suggest that while you should see that message, it should only be a warning. Did things proceed after you saw that warning?

(If it did in fact fail, then I'll revert the offending patch http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=fd5b15ff1a2ec37e75609c091522ae1e2c74c811 as per http://bugs.gentoo.org/334921.

Hi Serge,

Yes I confirm it did fail. I am using the "Virtual Machine Manager" and when I enable the network it gives me the message I posted earlier.

When I sun it at the command line, (after updating my libvirt deb today from your ppa) I get:

ademaria@ademaria-laptop:~$ sudo virsh net-start default
error: Failed to start network default
error: internal error '/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' exited with non-zero status 2 and signal 0: iptables v1.4.4: unknown option `--checksum-fill'
Try `iptables -h' or 'iptables --help' for more information.

In both cases, it does not look like just a warning to me, as the network does not start.

Please let me know how can I assist more.

Regards
Alessandro

On Thu, 2011-02-03 at 21:04 +0000, Serge Hallyn wrote:
> (If it did in fact fail, then I'll revert the offending patch
> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=fd5b15ff1a2ec37e75609c091522ae1e2c74c811
> as per http://bugs.gentoo.org/334921.

Please take note that the fix that I supplied does nothing about
addressing this issue because it should not be an issue.

First of all, the fix I supplied only deals with error:

libvirtError: internal error '/sbin/iptables --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 69 --jump ACCEPT' exited with non-zero status 1 and signal 0: iptables: Bad rule (does a matching rule exist in that chain?).

This other error that Alle is getting:

error: internal error '/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' exited with non-zero status 2 and signal 0: iptables v1.4.4: unknown option `--checksum-fill'
Try `iptables -h' or 'iptables --help' for more information.

is not an actual error condition in the libvrit (0.8.3-1ubuntu14) that I
am looking at. The only code that I can find that tries to add a
checksum rule for port 68 is in networkAddIptablesRules() in the file
src/network/bridge_driver.c:

    if ((network->def->ipAddress || network->def->nranges) &&
        (iptablesAddOutputFixUdpChecksum(driver->iptables,
                                         network->def->bridge, 68) != 0)) {
        VIR_WARN("Could not add rule to fixup DHCP response checksums "
                 "on network '%s'.", network->def->name);
        VIR_WARN0("May need to update iptables package & kernel to support CHECKSUM rule.");
    }

Note that failure of iptablesAddOutputFixUdpChecksum() only emits
warnings.

The actual error string that Alle is seeing comes from virRunWithHook()
which is called to through the following sequence of functions:

iptablesAddOutputFixUdpChecksum
iptablesOutputFixUdpChecksum
iptablesAddRemoveRule
virRun
virRunWithHook

which propagates an error back up the stack to networkAddIptablesRules()
but per the above code snippet, the error is discarded and a couple of
warning messages have been printed.

At this point, seeing as there are two different issues in this one
ticket, I would suggest that Alle open a new ticket covering the second
issue.

I suspect that Alle's network is failing to come up for a reason other
than the message he is seeing and the message that he sees just happens
to be the last message printed. I have been fooled by libvirt's lack of
printing error messages and misunderstanding that the last message it
did print is not in fact what was causing the failure.

I would suggest that Alle runs libvirtd in the foreground with some
debug/verbosity perhaps to get to the real root of his problem.

On Thu, 2011-02-03 at 13:39 +0000, Serge Hallyn wrote:
> Please test the packages from the attachment in comment #6.

I would if I could but I can't because my libvirt has other patches
(i.e. bug 713071) for bugs which I have run into and filed reports
upstream in my libvirt. Running your above packages would break my
installation by triggering these other bugs.

Maybe if you can produce a package that also includes the above fix I
can test it.

@Brian:

> At this point, seeing as there are two different issues in this one
> ticket, I would suggest that Alle open a new ticket covering the second
> issue.

Yes, that would be ideal. The problem is that I can't ask for this patch to be merged into maverick without confirmation that it fixes a bug, and doesn't do harm.

Still,

@Alle:

Please open a new bug for your problem. It can only help in this case, process-wise.

A package is building at ppa:serge-hallyn/libvirt-mav, with source at https://code.launchpad.net/~serge-hallyn/ubuntu/maverick/libvirt/bugall/, with fixes for this bug (and for bug 713071). Please test when it is finished building, and let us know if it fixes the bug for you.

I installed this patch late last night, and got the error

error: internal error '/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' exited with non-zero status 2 and signal 0: iptables v1.4.4: unknown option `--checksum-fill'

I also confirmed that the interface was not started. I tried more than once, just to be sure.

However, I tried again this morning, and it worked, the interface was created, and no errors were reported.

The host has been running without reboot since last night, and before the patch was applied, no other updates have been applied.

I can see that apparmor did a "profile_replace" on libvirt some time after it was started, I don't know much about apparmor, but is it possible this should have been reloaded or something? This is the only thing I can see in my logs that might have changed between it not working and now, when it does.

-----------------------------
$ uname -a
Linux kea 2.6.35-25-generic #44-Ubuntu SMP Fri Jan 21 17:40:44 UTC 2011 x86_64 GNU/Linux

Serge, I tried installing your packages, but they're for 32 bit systems only. Can you make the 64 bit versions?

@Mark,

The ppa at https://launchpad.net/~serge-hallyn/+archive/libvirt-mav has 64-bit versions as well. You should be able to just

sudo add-apt-repository ppa:serge-hallyn/libvirt-mav
sudo apt-get update
sudo apt-get -y dist-upgrade

to install them

According to debian bug I just added in this one, the problem is actually in dnsmasq being already running before starting libvirt network.

I just purged dnsmasq package on my maverick destop and now libvirt network starts fine.

HTH

@Alex- in comment #20, is 'and now libvirt network starts fine' a confirmation of the proposed packages fixing the problem, or did you never actually experience this particular bug to begin with?

@Serge

i actually experienced the problem on my laptop but never tried the proposed packages. After removing dnsmasq package i had manually added, the problem disappeared.

Would you like me to reinstall dnsmasq again and try the proposed packages?

Quoting Alex Muntada (<email address hidden>):
> @Serge
>
> i actually experienced the problem on my laptop but never tried the
> proposed packages. After removing dnsmasq package i had manually added,
> the problem disappeared.

I see, thanks.

> Would you like me to reinstall dnsmasq again and try the proposed
> packages?

No, thanks - that actually is reproducing a different problem. I'm
trying to decide whether a separate bug should be filed for it, and,
if so, what should be done about it.

In fact, if you wouldn't mind, please do open a new bug for it. Best
would be to do as you suggested - reinstall dnsmasq, fire up
libvirt (no need to try the proposed packages - they don't do anything
for this), then when it fails run ubuntu-bug libvirt-bin.

Many thanks.

@Serge

While reporting the new bug as suggested Launchpad found it previously reported in bug #231060.

I'm using libvirt version: 0.8.8, this bug stille xists

12:29:45.910: 27562: info : libvirt version: 0.8.8
12:29:45.910: 27562: error : virCommandWait:1229 : internal error Child process (/sbin/iptables --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill) exited with status 2.
12:29:45.917: 27562: error : virCommandWait:1229 : internal error Child process (/sbin/iptables --table mangle --insert POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill) exited with status 2.
12:29:45.917: 27562: warning : networkAddGeneralIptablesRules:1128 : Could not add rule to fixup DHCP response checksums on network 'default'.
12:29:45.917: 27562: warning : networkAddGeneralIptablesRules:1129 : May need to update iptables package & kernel to support CHECKSUM rule.
^C12:43:16.904: 27565: warning : qemudDispatchSignalEvent:406 : Shutting down on signal 2

@Charles,

what release are you on? Are you using a backported libvirt 0.8.8 on maverick?

@Alex
thank you.
service dnsmasq stop
has helped to me.

As pointed out by Alex, starting the dnsmasq service makes the libvirt network to crash and not start.

So I found a work-around for this: you must set the option "bind-interfaces" in the "/etc/dnsmasq.conf" file to tell the dnsmasq daemon to bind itself to the interface specified in the config file.

This solved the problem for me.

Karmic has long since stopped to receive any updates. Marking the Karmic task for this ticket as "Won't Fix".