Delay during PXE Boot, IP-Config gives up

This same bug appears to be affecting the PXE boot up process of MAAS nodes. It seems to only be an issue on machines with more than one NIC. It looks like the DHCP requests never make it to the MAAS cluster controller.

Status changed to 'Confirmed' because the bug affects multiple users.

Very weird bug;
I have PXE booted over VMWare VM w/o problem but when PXE booting on real hardware I got the problem.

I think I have found/solved this bug;
The problem is located at ipconfig (klibc-utils) not implementing the DHCP standard as it should at the file
/klibc-2.0.3/usr/kinit/ipconfig/dhcp_proto.c

old:
static int dhcp_send(struct netdev *dev, struct iovec *vec)
{
...
 bootp.yiaddr = dev->ip_addr; //this line must be replaced by the following two lines
...
}

new:
static int dhcp_send(struct netdev *dev, struct iovec *vec)
{
...
        bootp.yiaddr = INADDR_ANY;
        bootp.flags = htons(0x800);
...
}

This bug does not affect 100% of the DHCP servers as some of them are more forgiving than others with client protocol mistakes;
for those needing a quick solution for PXE booting see/get Serva complementary INITRD_N11.GZ which includes the already patched ipconfig

http://www.vercot.com/~serva/an/NonWindowsPXE3.html

Additional info
1)
When PXE installing on a VMware Workstation using the VMware DHCP server plus Serva as proxyDHCP the bug is not present.
When PXE installing on a real PC with a Netgear ISP router as DHCP plus Serva as proxyDHCP the bug is present.

In the first case the VMware DHCP module ignores the bad conformed ipconfig DHCP request but in the second case the NETGEAR DHCP refuses to provide the required DHCP parameters.

2) This bug has also been reported upstream
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756633

Please guys do something with this; today this issue affects virtually all the Debian/Ubuntu distros and their derivatives.

I think I'm seeing this bug as well. However, the patch from the debian bug report does not fix the problem for me.

This is the patch that "fix" this issue...
I have been using this patch for almost a year and works perfectly

static int dhcp_send(struct netdev *dev, struct iovec *vec)
{
...
 - bootp.yiaddr = dev->ip_addr;
 + bootp.yiaddr = INADDR_ANY;
 + bootp.flags = htons(0x800);
...
}

I have been talking to Debian people (Daniel Baumann, )who have proved their
complete lack of understanding/interest; they just do not care about this bug.
Talking to them is wasting my time; hopefully Ubuntu will fix this soon.

NOTE:
Please consider ipconfig needs to know (command line parameter)
if your server is of type "bootp" or "dhcp", if this parameter is wrong
ipconfig gives up.

We find similar issue. Does have any update from ubuntu?
https://bugs.launchpad.net/quantatw/+bug/1376588

no news from Ubuntu, your link seems broken...

patpat's fixed initramfs (INITRD_N11.2.GZ) doesn't work in ubuntu 15.10, system crash when trying launch ipconfig.

PXE booting w/o the ipconfig issue

; ubuntu-15.10-desktop-amd64.iso
; ubuntu-15.10-desktop-i386.iso
; ubuntukylin-15.10-desktop-amd64.iso
; ubuntukylin-15.10-desktop-i386.iso

requires INITRD_N11.1.GZ

source:
http://www.vercot.com/~serva/an/NonWindowsPXE3.html

I'm hitting the same bug. It's a show-stopper when it comes down to using PXE for network booting. I'm using Wily Werewolf and trying to boot off PXE, but no dice. The DHCP server receives the request, OFFERs an IP address and ipconfig (from klibc-utils) simply doesn't acknowledge it.

I first ran across the bug while using Clonezilla (which runs off a live Ubuntu 15.10 distro) booting off PXE. It turned the deployment into Hell on Earth.

I've opened the initrd cpio+xv image, snatched the binaries and also provided a bit more info about the system in which I performed the tests. Let me know if you need further info.

Right now, this is a show-stopper for PXE deployments. Please assist us with this issue.

Thanks!

This is an old bug never fixed by Ubuntu, never fixed by Debian either...
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756633

You have to recompile ipconfig with the patch I wrote above and everything works OK.
Hopefully some day Ubuntu and/or Debian will fix this ;-)

;-)

I kinda dread ask it here, but... it seems you've recompiled ipconfig quite a few times to get it working.
Have you tried recompiling the latest version from kernel.org? As in:

https://www.kernel.org/pub/linux/libs/klibc/2.0/

It would be interesting to know if it's a bug in the current source package *or* if it's directly related to its assembly in the Ubuntu environment + their initrd / initramfs environment.

I just recompiled ipconfig "once" and I got it working
I can confirm that /klibc-2.0.4/usr/kinit/ipconfig/dhcp_proto.c
did not fix this bug yet

static int dhcp_send(struct netdev *dev, struct iovec *vec)
{
 struct bootp_hdr bootp;
 char dhcp_hostname[SYS_NMLN+2];
 uint8_t padding[BOOTP_MIN_LEN - sizeof(struct bootp_hdr)];
 int padding_len;
 int i = 4;
 int j;

 memset(&bootp, 0, sizeof(struct bootp_hdr));

 bootp.op = BOOTP_REQUEST;
 bootp.htype = dev->hwtype;
 bootp.hlen = dev->hwlen;
 bootp.xid = dev->bootp.xid;
 bootp.ciaddr = INADDR_ANY;
 - bootp.yiaddr = dev->ip_addr; //<<< wrong >>>
 + bootp.yiaddr = INADDR_ANY; // needed
 + bootp.flags = htons(0x800); // needed
 bootp.giaddr = INADDR_ANY;
 bootp.secs = htons(time(NULL) - dev->open_time);
 memcpy(bootp.chaddr, dev->hwaddr, 16);

That's great - thank you (again) for testing this. :-) I'd help with the testing (compiling my own version), but I'm in a slow, expensive connection at the moment and can't grab the needed kernel packages.

If the latest version (from the most upstream this can get) also has this issue, what do you think about taking this issue straight to them as well?

klibc seems to have a development mailing list at:

http://www.zytor.com/mailman/listinfo/klibc/

It's been a bit empty lately, but I'm pretty sure the developers are still there lurking. There are very recent GIT source updates being mailed to the list.

It may be even easier to have it fixed, since you already have a proposed solution that seems to be working pretty well (I'm pretty sure they'll need to test it a lot in different settings, but it's part of the process). Feel free to use the files I attached to this bug if they need further evidence (even though we both know that your analysis of the code will be the juicy main course here :-)).

If we get them to fix it upstream, we may "simply" harass Debian and Ubuntu to update their sources for the fix.

As a workaround, you may also use "IPAPPEND 2" in the pxe config.

This bug was fixed in the package klibc - 2.0.4-8ubuntu1

---------------
klibc (2.0.4-8ubuntu1) xenial; urgency=medium

  * debian/patches/broadcast_dhcp_send.patch: set the broadcast bit when
    sending DHCPDISCOVER/DHCPREQUEST, as per RFC 2131; section 4.1. This
    helps when there is more than one interface trying to do DHCP at the same
    time. (LP: #1327412)

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 18 Mar 2016 15:35:37 -0400

It seems the fix is only partial ;

From RFC2131 (Dynamic Host Configuration Protocol) for a client we have:

Field DHCPDISCOVER DHCPREQUEST DHCPDECLINE,
                      DHCPINFORM DHCPRELEASE
----- ------------ ----------- -----------
'yiaddr' 0 0 0

but the mentioned patch does not include
- bootp.yiaddr = dev->ip_addr;
+ bootp.yiaddr = INADDR_ANY;
The patch only adds the flags... Considering it is the client DHCP request the yiaddr should be 0 (INADDR_ANY)

***debian/patches/broadcast_dhcp_send.patch***
--- a/usr/kinit/ipconfig/dhcp_proto.c
+++ b/usr/kinit/ipconfig/dhcp_proto.c
@@ -203,6 +203,7 @@ static int dhcp_send(struct netdev *dev,
  bootp.ciaddr = INADDR_ANY;
  bootp.yiaddr = dev->ip_addr;
  bootp.giaddr = INADDR_ANY;
+ bootp.flags = htons(0x800);
  bootp.secs = htons(time(NULL) - dev->open_time);
  memcpy(bootp.chaddr, dev->hwaddr, 16);

Best,
Patrick

Patch sent upstream
But I think that list is not really active....
http://www.zytor.com/pipermail/klibc/2016-March/003923.html

Best,
Patrick

yiaddr is set elsewhere in the code already (in dhcp_send_discover() or whatever); please check whether this actually fixes the issue.

>yiaddr is set elsewhere in the code already (in dhcp_send_discover()
If that is the case it is also wrong;
I wonder what IP are they using in a DHCP DISCOVERY??
that makes absolute not sense for a booting client trying to get an IP.

anyway it seems you are wrong:

int dhcp_send_discover(struct netdev *dev)
{
 dev->ip_addr = INADDR_ANY;
 dev->ip_gateway = INADDR_ANY;

 dprintf("-> dhcp discover ");

 return dhcp_send(dev, dhcp_discover_iov);
}

> please check whether this actually fixes the issue.
I encourage you to stop the guesswork and fix this upon what RFC2131 says
(which is pretty clear on this regard, just see its page 36)
yiaddr must be 0 on packets sent by the client.

I encourage you not to tag this bug as "fix released"

Best,
Patrick

Hello, Mathieu!

Thank you for looking into this.

Unfortunately, the bug is still present. I tested the same setups with the latest Xenial release (which natively includes your fix) and the behavior is the same: the DHCP server receives the request, OFFERs an IP address and ipconfig (from klibc-utils) simply doesn't acknowledge it.

Right now, this is a show-stopper for PXE deployments. Please assist us with this issue.

Thanks!

I'm still seeing this bug as well.
Both Ubuntu 16.04 LTS and 15.10 netboot kernels produce this behaviour.

When IP-Config starts configuring the NICs, I see the DHCPDISCOVER
arriving on the DHCP server, DHCPOFFER is going out, but no ACK is being received and the client continues telling me "IP-Config: no response after x secs - giving up".

@patpat; this is no guesswork. Bits can be missed. While you're right as to the correct value for yiaddr, I believe that you misunderstand how klibc on the whole works, and how Ubuntu processes work. The status change was automatic from the package upload.

Was I was getting at before is that the DHCPDISCOVER messages are going out correctly -- this is why we shouldn't change yiaddr directly in dhcp_send -- it affects all send operations indiscriminately. And obviously, the spec is extremely unclear on the expected behavior.

I've re-read the spec, and it looks like klibc only ever does a discover and request with the dhcp_send() method; so I'll rework the patch to make this clear and correctly fix the issue.

This is the kind of package in which you really need to know what you're doing when making changes. dhcp_send_discover() was doing the right thing for yiaddr: it sets dev->ip_addr = INADDR_ANY; which value will get copied in yiaddr in dhcp_send(), "as it should", except that's only good for dhcp_discover. dhcp_send_request() also needs to keep track of dev->ip_addr, and shouldn't overwrite it, but still ought to send with yiaddr set to INADDR_ANY.

@Mathieu Trudel-Lapierre
> please check whether this actually fixes the issue.
> this is no guesswork.
Yes it is; I'm saying here for almost 2 years what it has to be done to fix this problem and you recommend to see if a partial fix solves the problem or not. well...

>Bits can be missed.
Sure but do not make waste my time testing something I know that goes against the corresponding RFCs.

>I believe that you misunderstand how klibc on the whole works,
You are guessing again... I just said "please do not flag this issue as solved by a patch that does not work".

>this is why we shouldn't change yiaddr directly in dhcp_send
you wrong again; by RFC2131 all the DHCP traffic generated by a DHCP client "must" have yiaddr = 0, then you can VERY WELL define yiaddr = 0 at dhcp_send()

> I've re-read the spec, and it looks like klibc only ever does a discover and
> request with the dhcp_send() method; so I'll rework the patch to make this
> clear and correctly fix the issue.
You just want to re-invent the wheel here; just add the missing line and give all of us a break;

>This is the kind of package in which you really need to know what you're doing when making changes.
I do. do you?

>dhcp_send_discover() was doing the right thing for yiaddr: it sets dev->ip_addr = INADDR_ANY;
>which value will get copied in yiaddr in dhcp_send(), "as it should", except that's only good for dhcp_discover. >dhcp_send_request() also needs to keep track of dev->ip_addr, and shouldn't overwrite it,
>but still ought to send with yiaddr set to INADDR_ANY.
So??? Are you trying to tell me that you finally understood what's going on here ???
Did you finally see the problem? well it only takes one line of code to force yiaddr to 0 and
finish with this long story.

Best,
Patrick

This bug was fixed in the package klibc - 2.0.4-8ubuntu2

---------------
klibc (2.0.4-8ubuntu2) yakkety; urgency=medium

  * debian/patches/broadcast_dhcp_send.patch: yiaddr should always be set to 0
    (INADDR_ANY) when sending those messages we're sending as a DHCP client,
    so DHCPDISCOVER and DHCPREQUEST for now. DHCPDISCOVER already did the
    right thing since 2.0.4-8ubuntu1, this fixes DHCPREQUEST. (LP: #1327412)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 26 Apr 2016 12:59:59 -0400

--- a/usr/kinit/ipconfig/dhcp_proto.c
+++ b/usr/kinit/ipconfig/dhcp_proto.c
@@ -201,8 +201,14 @@ static int dhcp_send(struct netdev *dev,
  bootp.hlen = dev->hwlen;
  bootp.xid = dev->bootp.xid;
  bootp.ciaddr = INADDR_ANY;
- bootp.yiaddr = dev->ip_addr;
+ /* yiaddr should always be set to 0 for the messages we're likely
+ * to send as a DHCP client: DHCPDISCOVER, DHCPREQUEST, DHCPDECLINE,
+ * DHCPINFORM, DHCPRELEASE
+ * cf. RFC2131 section 4.1.1, table 5.
+ */
+ bootp.yiaddr = INADDR_ANY;
  bootp.giaddr = INADDR_ANY;
+ bootp.flags = htons(0x800);
  bootp.secs = htons(time(NULL) - dev->open_time);
  memcpy(bootp.chaddr, dev->hwaddr, 16);

Hallelujah;
you got it done; Thank you!

Thanks, Mathieu! I didn't get a chance to test this yet, but it's great to see it may be now fixed for good. :-)

Could you assist us once more?

We've been trying to send the fix for this bug upstream, so that all GNU/Linux distros can get the fixed versions. However, the original mailing list for klibc seems... kinda flaky. It's not dead, but it seems there's a lot of outside conversation before they do get patches.

By being (one of) the maintainer (s) of this package in Ubuntu, you might have more contact / closeness with H. Peter Anvin or any other dev who's involved with the upstream caretaking of klibc / klibc-utils. Could you send your fixes upstream? I know I'm asking for even more work regarding this, but it's a chance to have this fixed across all the Linux habitat instead of only Ubuntu (maybe Debian?).

Thanks. :-)

I have sent the patch to Debian long ago,
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756633
I found Daniel Baumann there who systematically refused to do anything
on this matter answering very funny things.

I have also sent the patch to the klibc list but it was completely ignored.

Best,
Patrick

Hello Jim, or anyone else affected,

Accepted klibc into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/klibc/2.0.3-0ubuntu1.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jim, or anyone else affected,

Accepted klibc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/klibc/2.0.3-0ubuntu1.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jim, or anyone else affected,

Accepted klibc into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/klibc/2.0.4-8ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

here you have also an old bug (2 years) affecting PXE scenarios when using CIFS. It was never fixed even when the solution is right there.

seen in:
 ubuntu-14.04-desktop-amd64.iso
 ubuntu-14.04-desktop-i386.iso
 ubuntu-16.04-desktop-amd64.iso
 ubuntu-16.04-desktop-i386.iso

https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1331547

Probably someone here can take a look at it. thanks.

The fix works for me. Thanks!

Forgot to say, I verified in 14.04.

This bug was fixed in the package klibc - 2.0.3-0ubuntu1.14.04.1

---------------
klibc (2.0.3-0ubuntu1.14.04.1) trusty; urgency=medium

  * debian/patches/broadcast_dhcp_send.patch: set the broadcast bit and yiaddr
    to 0 (INADDR_ANY) when sending DHCPDISCOVER/DHCPREQUEST, as per RFC 2131;
    section 4.1. This helps when there is more than one interface trying to do
    DHCP at the same time. (LP: #1327412)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 26 Apr 2016 13:15:15 -0400

> * debian/patches/broadcast_dhcp_send.patch: set the broadcast
> bit and yiaddr to 0 (INADDR_ANY) when sending DHCPDISCOVER/
> DHCPREQUEST, as per RFC 2131; section 4.1.

This is OK

>This helps when there is more than one interface
>trying to do DHCP at the same time.

This is complete nonsense.
The fixed bug has nothing to do with multi-homed PCs

Is it possible for anyone to test them in Wily/Xenial? It would be great to have those packages moved from Proposed to the current release for those versions.

Thanks! :)

This has broken DHCP in our environment. I suspect it has something to do with dhcp relaying. I am trying to get good data to confirm it.

Were you able to confirm this? It seems very odd, since regular DHCP operations are usually performed with userspace clients, not ipconfig (usually used in early userspace/startup environments).

>This has broken DHCP in our environment.
>I suspect it has something to do with dhcp relaying.

If your DHCP relay is broken after this change that means your DHCP relay needs the client's IP on the yiaddr field and that means you have a broken DHCP relay.

The wily update was not verified in three months, and wily is almost EOL. So I removed the -proposed package.

you know what
fixing 2 + 2 = 5
with 2 + 2 = 4
does not really need to be "confirmed" in every single Ubuntu flavor.
if I were you I would not remove anything because that would only add confusion.

Best,
Patrick

This bug was fixed in the package klibc - 2.0.4-8ubuntu1.16.04.1

---------------
klibc (2.0.4-8ubuntu1.16.04.1) xenial; urgency=medium

  * debian/patches/broadcast_dhcp_send.patch: yiaddr should always be set to 0
    (INADDR_ANY) when sending those messages we're sending as a DHCP client,
    so DHCPDISCOVER and DHCPREQUEST for now. DHCPDISCOVER already did the
    right thing since 2.0.4-8ubuntu1, this fixes DHCPREQUEST. (LP: #1327412)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 26 Apr 2016 12:59:59 -0400

The patch added to nominally fix this issue is incorrect; it is setting the wrong bit in the BOOTP flags field for broadcast:

+ bootp.flags = htons(0x800);

The correct value should be 0x8000. This is causing issues with switches that reject the packet as having bits set in a "must be zero" flag area.

RFC 1542 defines the flags field as 16 bits, and the broadcast bit is the most significant bit:

2.2 Definition of the 'flags' Field

   The standard BOOTP message format defined in [1] includes a two-octet
   field located between the 'secs' field and the 'ciaddr' field. This
   field is merely designated as "unused" and its contents left
   unspecified, although Section 7.1 of [1] does offer the following
   suggestion:

      "Before setting up the packet for the first time, it is a good
      idea to clear the entire packet buffer to all zeros; this will
      place all fields in their default state."

      This memo hereby designates this two-octet field as the 'flags'
      field.

      This memo hereby defines the most significant bit of the 'flags'
      field as the BROADCAST (B) flag. The semantics of this flag are
      discussed in Sections 3.1.1 and 4.1.2 of this memo.

      The remaining bits of the 'flags' field are reserved for future
      use. They MUST be set to zero by clients and ignored by servers
[...]
      and relay agents.

Opened bug 1624014 to patch the wrong-bit regression.

I stand corrected;
there's a typo on my patch the correct line should look like:
+ bootp.flags = htons(0x8000);
as mentioned by jvosburgh

Unfortunately this mistake was not previously detected because
all the net gear I've used on tests and Wireshark completely
ignored it.

Best,
Patrick

Hello, cyphermox!

Do you think you could send your patches upstream as well?

The same bug affects Debian (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756633) and, probably, the source packages. It's sad seeing this fixed in Ubuntu only.

Thanks!

The bug seems to be squashed in Xenial, Yakkety (and Zesty as well). Trusty seems to be the last one that needs verification.

trusty-proposed, klibc-utils 2.0.3-0ubuntu1.14.04.3 still suffers from this (or equally manifesting) issue.

We had a similar issue. We realized that (in our case) the problem was the Spanning Tree time to converge.

Explication: when PXE boots get an IP from DHCP server, and then when Linux starts, the interface go down and up, and Linux try to get IP from DHCP server again, but fails after 35 seconds aprox.
But the physical-switch delays 12 seconds to get the port up agian, and 30 seconds to converge the Spanning Tree. So changing the port configuration to: spanning-tree portfast, o spanning-tree portfast trunk (incase of a trunk), the Spanning Tree delay disappears, and Linux can get the IP from DHCP-server before the timeout ends.