dhcp3-server reports many bad udp checksums to syslog using virtio NIC

Bug #930962 reported by Tim Miller Dyck
138
This bug affects 27 people
Affects Status Importance Assigned to Milestone
dhcp3 (Debian)
Fix Released
Unknown
dhcp3 (Ubuntu)
Fix Released
Undecided
Unassigned
Lucid
Fix Released
High
Stéphane Graber
isc-dhcp (Ubuntu)
Fix Released
Medium
Unassigned
Precise
Fix Released
High
Stéphane Graber
Quantal
Fix Released
Undecided
Unassigned
lxc (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

== Rationale ==
Machines using virtio (kvm/xen) and running a dhcp server in a VM won't receive the proper udp checksums which leads to dhcpd dropping the packets.
This patch forces the checksum to be calculated in such case.

== Test case ==
1) Install a dhcp server in a VM using kvm/xen
2) Make sure you don't have iptables re-calculating the checksums for you
3) Check that the requests go through as expected

== Regression potential ==
Raring has had that fix for a while, so does redhat, so it looks pretty safe to me.

--- original bug report ---
Tested with dhcp3-server ver. 3.1.3-2ubuntu3.3 in Ubuntu 10.04.3 Server, x64.

The DHCP server reports that UDP packets sent to it have bad checksums:

Feb 11 06:57:18 ... dhcpd: 5 bad udp checksums in 5 packets
Feb 11 06:58:22 ... dhcpd: last message repeated 7 times
Feb 11 06:59:17 ... dhcpd: last message repeated 7 times

The DHCP server host is a KVM virtual machine using a virtio-based virtual NIC.

This problem has been reported for other distros using KVM and virtio:

https://bugs.mageia.org/show_bug.cgi?id=1243

http://<email address hidden>/msg41958.html
 - suggests using iptables to write in a checksum

http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob;f=dhcp-4.2.2-xen-checksum.patch;h=038d346d726e131f1ab2579fe015a72b49733a0d;hb=HEAD
 - Fedora patch to dhcp to avoid this

The simplest workaround is to change the virtual NIC type from virtio to Intel e1000 in KVM. Apparently this driver calculates checksums. But virtio is the default driver type.

Thanks,
Tim Miller Dyck

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu):
status: New → Confirmed
Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

I just hit that bug and I might be wrong but I think the bug is critial since it prevent any precise cloud image to have
an IP address at boot time on OpenStack for example since like you said --libvirt_use_virtio_for_bridges is enabled
by default in precise.

Revision history for this message
Guilhem Lettron (guilhem-fr) wrote :

This bug IS critical.

We must add the fedora patch to official packages.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Subscribing Stefan Bader as he's known to have an interest in that sort of things ;)

Revision history for this message
Stefan Bader (smb) wrote :

Finally got some time to look into this. As much as my test reproduction of this goes it would be not critical in importance because clients seem to get an IP address even with those messages showing up.

I can see those bad crc messages in (Quantal, Precise and Lucid those just been the ones I set up to try). To summarize my understanding here:

- virtio (and xen netfron) would deliberately leave the checksum incomplete for
   packets between interfaces on the same host because the packet will always
  remain in memory. So calculating a checksum is a waste of time.
- There has been a kernel change that allows to check the checksum status
  ([PACKET]: Add optional checksum computation for recvmsg). This has been in
  the kernel since 2.6.26.
- The isc-dhcp code needs to be updated to use this new interface. It seems that the
   patch from Fedora was added about a year ago but would still apply to the latest
   Debian package. Whether it never was attempted to forward this to the isc-dhcp
   project, or whether it was rejected for some reason, I don't know.

I am working on updates for the Quantal and Precise packages (Lucid should follow but that needs a bit more work to apply the patch). But then we also should investigate why this is not upstream or at least in Debian.

Changed in isc-dhcp (Ubuntu):
importance: Undecided → Medium
assignee: nobody → Stefan Bader (stefan-bader-canonical)
Revision history for this message
Michal Suchanek (hramrach) wrote : Re: [Bug 930962] Re: dhcp3-server reports many bad udp checksums to syslog using virtio NIC

On 8 August 2012 18:35, Stefan Bader <email address hidden> wrote:
> Finally got some time to look into this. As much as my test reproduction
> of this goes it would be not critical in importance because clients seem
> to get an IP address even with those messages showing up.

That's not what I see here, they do not get an IP address.

>
> I can see those bad crc messages in (Quantal, Precise and Lucid those
> just been the ones I set up to try). To summarize my understanding here:
>
> - virtio (and xen netfron) would deliberately leave the checksum incomplete for
> packets between interfaces on the same host because the packet will always
> remain in memory. So calculating a checksum is a waste of time.

No, that's how any network card with checksum offloading behaves. It
may skip copying the checksum because it verified it already, and the
virtio device abuses this interface quirk to never produce any
checksum in the first place. On real network interfaces you can turn
that feature off, and then the checksum is always copied. On virtio
'turning off' would amount to producing a checksum which dhclient can
subsequently check.

Revision history for this message
Stefan Bader (smb) wrote :

On 09.08.2012 10:21, Michal Suchanek wrote:
>> of this goes it would be not critical in importance because clients seem
>> to get an IP address even with those messages showing up.
>
> That's not what I see here, they do not get an IP address.

Then what is your exact setup?

>> - virtio (and xen netfron) would deliberately leave the checksum incomplete for
>> packets between interfaces on the same host because the packet will always
>> remain in memory. So calculating a checksum is a waste of time.
>
> No, that's how any network card with checksum offloading behaves. It
> may skip copying the checksum because it verified it already, and the
> virtio device abuses this interface quirk to never produce any
> checksum in the first place. On real network interfaces you can turn
> that feature off, and then the checksum is always copied. On virtio
> 'turning off' would amount to producing a checksum which dhclient can
> subsequently check.

Yes, it a real card has checksum offloading it would do this in hardware. What I
try to say is that the fact that virtio leaves them incomplete when passed
within the virtual network looks like a deliberate decision there for
performance improvement. And since upstream has added that interface to check
for those incomplete checksums (and specifically mentioning dhcp) makes it clear
that the drivers will not change.

Revision history for this message
Michal Suchanek (hramrach) wrote :

On 9 August 2012 10:49, Stefan Bader <email address hidden> wrote:
> On 09.08.2012 10:21, Michal Suchanek wrote:
>>> of this goes it would be not critical in importance because clients seem
>>> to get an IP address even with those messages showing up.
>>
>> That's not what I see here, they do not get an IP address.
>
> Then what is your exact setup?

dnsmasq running on xen dom0 and xen dumUs trying to get IP address
with dhcp client.

I see server mentioned here so maybe on server the checksum error is
not considered fatal.

Revision history for this message
Stefan Bader (smb) wrote :

I added Quantal and Precise version of the isc-dhcp packages to http://people.canonical.com/~smb/lp930962/ for some broader testing. Are those also fixing the issues where no IP addresses were assigned?

Revision history for this message
Stefan Bader (smb) wrote :

On 09.08.2012 11:17, Michal Suchanek wrote:

>> Then what is your exact setup?
>
> dnsmasq running on xen dom0 and xen dumUs trying to get IP address
> with dhcp client.

Ah ok, I was using the isc-dhcp-server on a KVM guest to isc-dhcp-clinet on
another one.

But thought I also had been using the dnsmasq setup created by libvirt for Xen
dom0 to domU. Hm, wait, I may have seen something somewhere in all those links
and reports that mentioned libvirt having a quirk for that. Are you using
dnsmasq on your own or have libvirt in use?

(If libvirt does something it probably is the ipfilter based approach that is
mentioned in a few placec)
>
> I see server mentioned here so maybe on server the checksum error is
> not considered fatal.

The handling should be the same on server or on desktop. But then, maybe network
manager is doing something when it sees the checksum errors. That would not be
happening on server.

Revision history for this message
Stefan Bader (smb) wrote :

Right, so all the hosts I use already have this in "sudo iptables -L -t mangle"

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
CHECKSUM udp -- anywhere anywhere udp dpt:bootpc CHECKSUM fill

I suspect this is added in some way by libvirt (which I have installed on all of them). Manually this would be done by:

sudo iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill

Revision history for this message
Michal Suchanek (hramrach) wrote :

No, I am using some other manually installed workaround, and I am
using bare dnsmasq and dhcp client, not network manager nor libvirt.

Did not get to figuring out what the workaround was yet.

Revision history for this message
Stefan Bader (smb) wrote :

> Did not get to figuring out what the workaround was yet.

See the comment before your comment. :) Well, probably not that simple when
using the mail interface as ordering differs there. But in the report itself.

Revision history for this message
Peter Matulis (petermatulis) wrote :

This workaround succeeded for me (on my KVM hypervisor):

sudo iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill

Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

This workaround on bridged clients seems to make dhcpd happy

sudo iptables -A POSTROUTING -t mangle -p udp --dport 67 -j CHECKSUM --checksum-fill

(Luckily I only need it on my Icinga to check dhcp, all clients have fixed IPs)

Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

Note that the analog workaround on the server running dhcpd does not work. I assume the raw socket used by dhcpd is read before PREROUTING is hit. That makes this bug ugly, as the workaround must be done on *all* clients instead of only the server.

Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

According to my cursory reading of relevant RFCs and a quick wireshark session, the bug is in KVM virtio. An UDP packet should contain a checksum field of 0 for no checksum, or a correct checksum. I see bogus values coming in.

OTOH, wireshark warns that this may be due to offloading, so I may not be correct in this analysis.

Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

As mentioned in the first comment, switching to e1000 NIC on the dhcpd server also fixed the issue.

Revision history for this message
Michal Suchanek (hramrach) wrote :

On 7 January 2013 23:13, Martijn Lievaart <email address hidden> wrote:
> According to my cursory reading of relevant RFCs and a quick wireshark
> session, the bug is in KVM virtio. An UDP packet should contain a
> checksum field of 0 for no checksum, or a correct checksum. I see bogus
> values coming in.
>
> OTOH, wireshark warns that this may be due to offloading, so I may not
> be correct in this analysis.
>

This is due to virtio implementing 'offloading' by returning random
garbage in the checksum field.

Since the code to calculate the checksumis not present this offloading
cannot be turned off.

Since kernel has interface to inform applications that offloading is
on it is a bug in the application to interpret the bogus checksum.
Presumably the card has already checked it and did not bother passing
it on.

This same feature is available on real e1000 hardware and can be
turned off there.

Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

> On 7 January 2013 23:13, Martijn Lievaart <email address hidden> wrote:
>> According to my cursory reading of relevant RFCs and a quick wireshark
>> session, the bug is in KVM virtio. An UDP packet should contain a
>> checksum field of 0 for no checksum, or a correct checksum. I see bogus
>> values coming in.
>>
>> OTOH, wireshark warns that this may be due to offloading, so I may not
>> be correct in this analysis.
>
> This is due to virtio implementing 'offloading' by returning random
> garbage in the checksum field.
>
> Since the code to calculate the checksumis not present this offloading
> cannot be turned off.
>
> Since kernel has interface to inform applications that offloading is
> on it is a bug in the application to interpret the bogus checksum.
> Presumably the card has already checked it and did not bother passing
> it on.

Hi Michal,

I don't quite understand what you are saying here. Are you saying that because Linux has a kernel interface to check for offloading, any and all application (that cares about udp checksums) should use this? It would seem much simpler if the virtio driver zero'd the checksum field, any application already should be able to handle that. No Linux specific code needed.

Martijn

Revision history for this message
Michal Suchanek (hramrach) wrote :

Hello,

On 8 January 2013 15:01, Martijn Lievaart <email address hidden> wrote:

> Hi Michal,
>
> I don't quite understand what you are saying here. Are you saying that
> because Linux has a kernel interface to check for offloading, any and
> all application (that cares about udp checksums) should use this? It

Yes, reading several discussion about this issue it seems that
application that wants to work correctly on Linux has to do this. This
issue pops up on actual hardware with actual hardware checksum
offloading reportedly.

> would seem much simpler if the virtio driver zero'd the checksum field,
> any application already should be able to handle that. No Linux specific
> code needed.

Except virtio would have to do the work then. Also it does not solve
the issue for other hardware with offloading.

Also does the zero checksum not change the meaning of the message? It
would seem that setting the checksum to zero in driver and application
ignoring the offload status would imply interpretation that the sender
did not include any checksum at all whereas the correct interpretation
would be that the checksum was verified and stripped by network
hardware.

Thanks

Michal

Revision history for this message
Stéphane Graber (stgraber) wrote :

The next isc-dhcp upload will contain the RedHat patch for hardware/xen/virtio offloading.
Upstream was contacted through Debian but without any response yet.

Changed in isc-dhcp (Ubuntu):
status: Confirmed → Fix Committed
assignee: Stefan Bader (stefan-bader-canonical) → Stéphane Graber (stgraber)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.2.4-4ubuntu1

---------------
isc-dhcp (4.2.4-4ubuntu1) raring; urgency=low

  * Merge from Debian, remaining changes:
    - Apparmor profiles for dhclient and dhcpd.
    - Upstart jobs for isc-dhcp-server, isc-dhcp-server6, isc-dhcp-relay and
      isc-dhcp-relay6. (LP: #944849)
    - Separate default file for isc-dhcp-relay6. (LP: #944849)
    - Apport hook for isc-dhcp-client and isc-dhcp-server.
    - dhclient.conf: Set for IPv6 (dhcp6.*) (LP: #770324)
    - If /etc/ltsp/dhcpd.conf exists, use that instead of /etc/dhcp/dhcpd.conf
    - Drop isc-dhcp-server/new_auth_behavior question from high to medium
    - Create user/group dhcpd (LP: #727837)
    - Create /etc/dhcp/ddns-keys/ for DDNS updates (LP: #341817)
    - Build with extra hardening and use --enable-paranoia to run dhcpd as user
      (LP: #727837)
    - Add IPv6 support to dhclient-script.linux.udeb
    - Wait for /etc/resolv.conf to be writable in dhclient-script.linux.
      (LP: #856984)
    - Sanitize environment in dhclient-script.linux. (LP: #1045986)
    - Don't call 'ip addr flush' as it breaks IPv6 networking. (LP: #1023174)
    - debian/apparmor-profile.dhclient: update to add the new paths used by
      NetworkManager for its conf and leases files; standardized under
      /var/lib/NetworkManager.
    - Remaining Ubuntu patches:
      + dhclient-fix-backoff
      + dhclient-more-debug (LP: #35265)
      + dhclient-safer-timeout (LP: #838968)
      + dhcpd.conf-subnet-examples (LP: #26661)
      + multi-ip-addr-per-if (LP: #717166)
      + onetry_retry_after_initial_success (LP: #974284)
      + revert-next-server
    - Dropped Ubuntu patches:
      + dhclient-onetry-call-clientscript (now in Debian)
  * Fix isc-dhcp-server-ldap not actually containing ldap support.
    Thanks to Christian Hoffmann for tracking down the issue. (LP: #1071928)
  * Don't set fqdn.fqdn in dhclient.conf as that seems to confuse some DHCP
    servers. An alternative would have been to only set fqdn.fqdn and not
    host-name, but that appears to confuse another set of servers.
    For now go with just host-name which is the most common and if becomes a
    big problem for IPv6 (where fqdn.fqdn is apparently required), then we'll
    need to have a separate dhclient6.conf file and change all the calls to
    dhclient -6 to use that file instead. (LP: #1088682)
  * Include patch from RedHat/Fedora to deal with hardware/xen/virtio offload
    of UDP checksums. (LP: #930962)

isc-dhcp (4.2.4-4) unstable; urgency=medium

  * Run exit hooks when "dhclient -1" fails (closes: #486520).
  * Add dhcp6.name-servers and dhcp6.domain-search to the default request
    options in dhclient.conf (closes: #693315).
 -- Stephane Graber <email address hidden> Thu, 17 Jan 2013 17:09:14 -0500

Changed in isc-dhcp (Ubuntu):
status: Fix Committed → Fix Released
Changed in isc-dhcp (Ubuntu Quantal):
status: New → In Progress
assignee: nobody → Stéphane Graber (stgraber)
description: updated
Revision history for this message
Dave Walker (davewalker) wrote : Please test proposed package

Hello Tim, or anyone else affected,

Accepted isc-dhcp into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/isc-dhcp/4.2.4-1ubuntu10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in isc-dhcp (Ubuntu Quantal):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Markus Heberling (markus-tisoft) wrote :

The http://launchpad.net/ubuntu/+source/isc-dhcp/4.2.4-1ubuntu10.2 package fixes this for me on quantal.

Revision history for this message
Tim Miller Dyck (timmillerdyck) wrote :

Hi Dave, thanks for uploading that.

I have upgraded the DHCP server system to Quantal:

# dpkg -l | grep dhcp
ii isc-dhcp-client 4.2.4-1ubuntu10.1 amd64 ISC DHCP client
ii isc-dhcp-common 4.2.4-1ubuntu10.1 amd64 common files used by all the isc-dhcp* packages
ii isc-dhcp-server 4.2.4-1ubuntu10.1 amd64 ISC DHCP server for automatic IP address assignment

I switched the NIC type to virtio and confirmed the problem still exists.

I then installed the updated packages from quantal-proposed:

# aptitude install isc-dhcp-server/quantal-proposed isc-dhcp-client/quantal-proposed isc-dhcp-common/quantal-proposed

Confirming new versions:

# dpkg -l | grep dhcp
ii isc-dhcp-client 4.2.4-1ubuntu10.2 amd64 ISC DHCP client
ii isc-dhcp-common 4.2.4-1ubuntu10.2 amd64 common files used by all the isc-dhcp* packages
ii isc-dhcp-server 4.2.4-1ubuntu10.2 amd64 ISC DHCP server for automatic IP address assignment

The DHCP server log warnings do not appear any longer and DHCP is working as usual.

So, it looks fixed to me.

Thanks!
Tim Miller Dyck

tags: added: verification-done
removed: verification-needed
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.2.4-1ubuntu10.2

---------------
isc-dhcp (4.2.4-1ubuntu10.2) quantal-proposed; urgency=low

  * Fix isc-dhcp-server-ldap not actually containing ldap support.
    Thanks to Christian Hoffmann for tracking down the issue. (LP: #1071928)
  * Don't set fqdn.fqdn in dhclient.conf as that seems to confuse some DHCP
    servers. An alternative would have been to only set fqdn.fqdn and not
    host-name, but that appears to confuse another set of servers.
    For now go with just host-name which is the most common and if becomes a
    big problem for IPv6 (where fqdn.fqdn is apparently required), then we'll
    need to have a separate dhclient6.conf file and change all the calls to
    dhclient -6 to use that file instead. (LP: #1088682)
  * Include patch from RedHat/Fedora to deal with hardware/xen/virtio offload
    of UDP checksums. (LP: #930962)
 -- Stephane Graber <email address hidden> Fri, 01 Mar 2013 16:07:49 -0500

Changed in isc-dhcp (Ubuntu Quantal):
status: Fix Committed → Fix Released
Revision history for this message
Jon Schewe (jpschewe) wrote :

Any plans to get this fix into 12.04 LTS? Or can I run the new packages inside LTS?

Revision history for this message
Stéphane Graber (stgraber) wrote :

Recent kernel changes are now causing this issue to appear in LXC containers too where the libvirt mangle rule doesn't help at all.
As a result I'm now preparing SRUs for both lucid and precise so that all supported Ubuntu versions know to deal with missing checksums.

Changed in isc-dhcp (Ubuntu Lucid):
status: New → Triaged
Changed in isc-dhcp (Ubuntu Precise):
status: New → Triaged
no longer affects: isc-dhcp (Ubuntu Lucid)
no longer affects: dhcp3 (Ubuntu Precise)
no longer affects: dhcp3 (Ubuntu Quantal)
no longer affects: isc-dhcp (Ubuntu Lucid)
Changed in dhcp3 (Ubuntu):
status: New → Fix Released
Changed in dhcp3 (Ubuntu Lucid):
status: New → Triaged
assignee: nobody → Stéphane Graber (stgraber)
Changed in isc-dhcp (Ubuntu):
assignee: Stéphane Graber (stgraber) → nobody
Changed in dhcp3 (Ubuntu Lucid):
importance: Undecided → High
Changed in isc-dhcp (Ubuntu Precise):
importance: Undecided → High
assignee: nobody → Stéphane Graber (stgraber)
Changed in isc-dhcp (Ubuntu Quantal):
assignee: Stéphane Graber (stgraber) → nobody
Changed in isc-dhcp (Ubuntu Precise):
status: Triaged → In Progress
Changed in dhcp3 (Ubuntu Lucid):
status: Triaged → In Progress
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Tim, or anyone else affected,

Accepted isc-dhcp into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/isc-dhcp/4.1.ESV-R4-0ubuntu5.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in isc-dhcp (Ubuntu Precise):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Changed in dhcp3 (Ubuntu Lucid):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

Hello Tim, or anyone else affected,

Accepted dhcp3 into lucid-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/dhcp3/3.1.3-2ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Stéphane Graber (stgraber) wrote :

Tested in a precise LXC container running on saucy and confirmed that this fixes the issue there.

tags: added: verification-done-precise
Revision history for this message
William Grant (wgrant) wrote :

Tested in a lucid LXC container on saucy, also fixes the bug.

tags: added: verification-done verification-done-lucid
removed: verification-needed
Revision history for this message
Colin Watson (cjwatson) wrote :

Works well for me too with a lucid LXC container on saucy. Thanks!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dhcp3 - 3.1.3-2ubuntu3.5

---------------
dhcp3 (3.1.3-2ubuntu3.5) lucid-proposed; urgency=low

  * Include patch from RedHat/Fedora to deal with hardware/xen/virtio offload
    of UDP checksums. (LP: #930962)
  * Update apparmor profile to add required the "network packet raw" rule
    for the checksum change.
 -- Stephane Graber <email address hidden> Thu, 23 May 2013 19:58:28 -0400

Changed in dhcp3 (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.1.ESV-R4-0ubuntu5.8

---------------
isc-dhcp (4.1.ESV-R4-0ubuntu5.8) precise-proposed; urgency=low

  [ Dave Chiluk ]
  * Allow dhcpd to read /etc/ldap/ldap.conf for isc-dhcp-server-ldap.
    (LP: #1057358). Backported from Stéphane Graber's quantal patch.

  [ Stéphane Graber ]
  * Include patch from RedHat/Fedora to deal with hardware/xen/virtio offload
    of UDP checksums. (LP: #930962)
  * Update apparmor profile to add required the "network packet raw" rule
    for the checksum change.
 -- Stephane Graber <email address hidden> Thu, 23 May 2013 11:13:07 -0400

Changed in isc-dhcp (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Bart Heinsius (bheinsius) wrote :

new package was in precise-proposed this morning, it fixed the problem.

Revision history for this message
Robie Basak (racb) wrote :

This issue also affects the "debian" LXC template in Ubuntu (lxc 1.0.0~alpha1-0ubuntu11). If I create a Debian jessie container on Ubuntu Saucy, then the container cannot DHCP.

Workaround: sudo iptables -t mangle -A POSTROUTING -o lxcbr0 -p udp --dport bootpc -j CHECKSUM --checksum-fill

It looks like the Debian bug is http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717217

Is there any way a workaround could be applied to lxc in Ubuntu - perhaps something specific to the Debian template? I wonder if the template could arrange for an iptables inside the container to mangle the incoming DHCP packet or something.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'd be ok with adding a /usr/share/lxc/hooks/network-bootpc-checksum script which does the iptables add, and having the debian template add that to lxc.hook.pre-start. However, in that hook script we'd want to avoid having one rule added per container start. What's the cleanest and most robust way to detect if the rule has already been inserted? We can do a iptables -t mangle -L and look for

CHECKSUM udp -- anywhere anywhere udp dpt:bootpc CHECKSUM fill

but that feels fragile. Any better ideas?

Changed in lxc (Ubuntu):
status: New → Confirmed
importance: Undecided → High
status: Confirmed → Incomplete
Revision history for this message
Robie Basak (racb) wrote :

16:13 <rbasak> hallyn_: about bug 930962. I was suggesting doing the mangle the other way
               inside the container. Not sure if that's possible.
16:13 <rbasak> hallyn_: if not, then "iptables -D" can delete a rule by its specification,
               so perhaps "iptables -D ... || true; iptables -A ..." is a (hacky)
               workaround?

Changed in lxc (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 930962] Re: dhcp3-server reports many bad udp checksums to syslog using virtio NIC

> 16:13 <rbasak> hallyn_: if not, then "iptables -D" can delete a rule by its specification,
> so perhaps "iptables -D ... || true; iptables -A ..." is a (hacky)
> workaround?

I think I prefer to keep a file like /run/lxc/features/iptables-checksum
to show that it's been done, or just check the iptables -L output.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Changing to low priority as there is a workaround.

Changed in lxc (Ubuntu):
importance: High → Low
Revision history for this message
Robert Collins (lifeless) wrote :

See bug 1254338 - the 'fix' in the isc dhcp packages doesn't help at all for folk with existing containers: adding a mangle rule to the lxcbr0 in lxc-net.conf helps.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1.0.0~alpha3-0ubuntu8

---------------
lxc (1.0.0~alpha3-0ubuntu8) trusty; urgency=low

  * Add iptables rule to fix checksum of udp packets for dhcp (LP: #930962)
 -- Serge Hallyn <email address hidden> Tue, 10 Dec 2013 11:27:09 -0600

Changed in lxc (Ubuntu):
status: Confirmed → Fix Released
Changed in dhcp3 (Debian):
status: Unknown → New
Changed in dhcp3 (Debian):
status: New → Confirmed
Changed in dhcp3 (Debian):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.