Bug #1735331 “ec2: zesty tempfile sandbox dhclient.pid file can'...” : Bugs : cloud-init

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#1

cloud-init.log Edit (109.4 KiB, text/plain)

cloud-init log from Ec2Local failure to read dhclient.pid file

Chad Smith (chad.smith) on 2017-11-30

description:

updated

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#2

More detail:
this also happens on previous 17.1.27, we just never tried to read that pidfile to act on it.

Nov 30 04:54:32 ip-10-0-20-176 dhclient[683]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-zlfb5hsr/dhclient.pid: No such file or directory

Validated it doesn't happen on Artful. with 17.1.27 or 17.1.41

Chad Smith (chad.smith) on 2017-11-30

summary:

- ec2: temp dhclient.pid file issues
+ ec2: zesty tempfile sandbox dhclient.pid file can't be created

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#3

In further testing it is a timing thing with our tempdirectory being removed before dhclient attempts to write the actual pidfile.

sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'

This fails 100% of the time in zesty and artful emitting "Can't create dhclient[3296]: Can't create /var/tmp/cloud-init/.../dhclient.pid"

If I add a sleep(0.000001) in cloudinit/net/dhcp.py on line 123 just after the sandbox_dhclient_cmd, no pidfile write errors are produced by dhclient in /var/log/syslog.

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#4

Alternatively we could just have a busy wait there and only exit the tempdir context once we know we have a pid file and can clean ourselves up.

while not os.path.exists(pid_file):
pass

Changed in cloud-init:
status:	New → Incomplete
status:	Incomplete → Triaged
importance:	Undecided → High

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#5

This affectively causes Ec2Local to fail and we fall back to Ec2 in the init-network stage. It slows down ec2 instance boots a bit, but cloud-init will still function on ec2.

Revision history for this message

Scott Moser (smoser) wrote on 2017-11-30:

#6

OK. So dhclient is fun.

Background info
* we run dhclient like this:
   -1 (try to get an ip address only once exit fail on fail).
   -v (verbose)
   -lf <tmpdir>/dhcp.leases
   -pf <tmpdir>/pid
   <interface>
   -sf /bin/true
* dhclient.c source [1]

I'm pretty sure this is what is happening here.
1. when daemonizing, dhclient does not write its pidfile immediately (line 971). Instead it waits until it does "go_daemon" (line 4203). This does make some sense as it goes daemon by forking and the new process staying around.
2. on successful acquisition of address, cloud-init is removing the temp dir and looking for the pid before the fork()'ed process writes it.
2a. on failure, i think a pid is probably not written at all. I don't think we're handling the possible ProcessExecutionError that would be raised if dhclient returned non-zero.

Heres a suggestion for a fix
http://paste.ubuntu.com/26081537/

Another option would be for cloud-init to invoke dhclient with '-d' (do not background) which would avoid the background process entirely. The '-1' would allow it to exit on failure. Then on success we'd have to have the script file signal somehow that it was done or possibly just kill its parent.

The last option of course is to write a dhcp client in python :), which is looking more and more appealing.

[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/isc-dhcp/tree/client/dhclient.c

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-11-30:

#7

Interesting thing smoser is that dhclient doesn't actually return an error code where there are problems. Not the following when I hit selinux issues with /sbin/dhclient instead of our /tmp/dhclient

http://pastebin.ubuntu.com/26082502/ line 7 8 16 and 17 all point to busted dhclient, but supb wouldn't error for us in this case because retcode is 0

>>> from cloudinit.util import subp
>>> subp(['sudo', '/sbin/dhclient', '-1', '-v', '-lf', '/home/ubuntu/dhcp.leases', '-pf', '/home/ubuntu/dhclient.pid', 'eth0', '-sf', '/bin/true'])
('', "Internet Systems Consortium DHCP Client 4.3.3\nCopyright 2004-2015 Internet Systems Consortium.\nAll rights reserved.\nFor info, please visit https://www.isc.org/software/dhcp/\n\ncan't create /home/ubuntu/dhcp.leases: Permission denied\nexecve (/bin/true, ...): Permission denied\nListening on LPF/eth0/0a:42:9a:bd:dc:34\nSending on LPF/eth0/0a:42:9a:bd:dc:34\nSending on Socket/fallback\nDHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x9e09e019)\nDHCPREQUEST of 10.0.20.170 on eth0 to 255.255.255.255 port 67 (xid=0x19e0099e)\nDHCPOFFER of 10.0.20.170 from 10.0.20.1\nDHCPACK of 10.0.20.170 from 10.0.20.1\nexecve (/bin/true, ...): Permission denied\ncan't create /home/ubuntu/dhcp.leases: Permission denied\nbound to 10.0.20.170 -- renewal in 1636 seconds.\n")

Even parsing error output isn't an option (although we could catch SELinux type errors) because dhclient doesn't emit the 'Can't create pidfile' message until after the background process was forked.

So, I'll take part of your suggested patch, but I think us parsing or try/except handling of subp output is going to end up fruitless in this or other cases as dhclient doesn't really seem to exit in error upon the errors we care about.

Chad Smith (chad.smith) on 2017-11-30

Changed in cloud-init:
assignee:	nobody → Chad Smith (chad.smith)
status:	Triaged → In Progress

Scott Moser (smoser) on 2017-12-01

Changed in cloud-init:
status:	In Progress → Fix Committed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-12-01:

#8

This bug was fixed in the package cloud-init - 17.1-46-g7acc9e68-0ubuntu1

---------------
cloud-init (17.1-46-g7acc9e68-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot.
    - ec2: Fix sandboxed dhclient background process cleanup.
      [Chad Smith] (LP: #1735331)
    - tests: NoCloudKVMImage do not modify the original local cache image.
    - tests: Enable bionic in integration tests. [Joshua Powers]
    - tests: Use apt-get to install a deb so that depends get resolved.
    - sysconfig: Correctly render dns and dns search info.
      [Ryan McCabe] (LP: #1705804)

-- Scott Moser <email address hidden> Thu, 30 Nov 2017 21:59:55 -0500

Changed in cloud-init (Ubuntu Bionic):
status:	New → Fix Released

Scott Moser (smoser) on 2017-12-01

Changed in cloud-init (Ubuntu Xenial):
status:	New → Confirmed
Changed in cloud-init (Ubuntu Zesty):
status:	New → Confirmed
Changed in cloud-init (Ubuntu Artful):
status:	New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
importance:	Undecided → Medium
Changed in cloud-init (Ubuntu Zesty):
importance:	Undecided → Medium
Changed in cloud-init (Ubuntu Artful):
importance:	Undecided → Medium
Changed in cloud-init (Ubuntu Bionic):
importance:	Undecided → Medium

Revision history for this message

Steve Langasek (vorlon) wrote on 2017-12-01: Proposed package upload rejected

#9

An upload of cloud-init to artful-proposed has been rejected from the upload queue for the following reason: "to reupload with -v".

Revision history for this message

Steve Langasek (vorlon) wrote on 2017-12-01: Please test proposed package

#10

Hello Chad, or anyone else affected,

Accepted cloud-init into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-46-g7acc9e68-0ubuntu1~17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Artful):
status:	Confirmed → Fix Committed
tags:	added: verification-needed verification-needed-artful
Changed in cloud-init (Ubuntu Zesty):
status:	Confirmed → Fix Committed
tags:	added: verification-needed-zesty

Revision history for this message

Steve Langasek (vorlon) wrote on 2017-12-01:

#11

Hello Chad, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-46-g7acc9e68-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message

Chad Smith (chad.smith) wrote on 2017-12-04:

#12

ec2-sru-validate-17.1.46.log Edit (16.0 KiB, text/plain)

Download full text (17.5 KiB)

Thanks Steve. Attached are the verification logs for ec2 instance asserting that this bug is fixed.

# Launch instance under test
$ for release in xenial zesty artful; do
    echo "Handling $release";
    launch-ec2 --series $release;
    ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
    ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
    ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
    ssh ubuntu@<ec2-address> sudo apt update;
    ssh ubuntu@<ec2-address> sudo apt install cloud-init;
    # Show upgrade without restart doesn't break
    ssh ubuntu@<ec2-address> sudo cloud-init init;
    # Show clean install doesn't break
    ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
    ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
    ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
    # Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
    ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
    sudo ps -afe |grep dhclient;

done

[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.

[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=7acc9e68

=== End SRU Template ===

=== SRU Verification output ===

### EC2 Xenial upgrade and fresh install test
ubuntu@ip-10-0-20-54:~$ sudo vi /etc/apt/sources.list
ubuntu@ip-10-0-20-54:~$ sudo apt update
...
ubuntu@ip-10-0-20-54:~$ cat /run/cloud-init/result.json
{
"v1": {
"datasource": "DataSourceEc2Local",
"errors": []
}
}
ubuntu@ip-10-0-20-54:~$ grep Trace /var/log/cloud-init.log
ubuntu@ip-10-0-20-54:~$ sudo apt install cloud-init
Setting up cloud-init (17.1-46-g7acc9e68-0ubuntu1~16.04.1) ...
ubuntu@ip-10-0-20-54:~$ sudo cloud-init init
Cloud-init v. 17.1 running 'init' at Sat, 02 Dec 2017 04:10:13 +0000. Up 167.09 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 10.0.20.54 | 255.255.255.0 | . | 0a:a4:58:cb:62:8c |
ci-info: | eth0 | True | 2600:1f16:dc8:a120:73c6:9626:9c1d:2ecf/128 | . | global | 0a:a4:58:cb:62:8c |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++Rou...

Thanks Steve. Attached are the verification logs for ec2 instance asserting that this bug is fixed.

# Launch instance under test
$ for release in xenial zesty artful; do
    echo "Handling $release";
    launch-ec2 --series $release;
    ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
    ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
    ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
    ssh ubuntu@<ec2-address> sudo apt update;
    ssh ubuntu@<ec2-address> sudo apt install cloud-init;
    # Show upgrade without restart doesn't break
    ssh ubuntu@<ec2-address> sudo cloud-init init;
    # Show clean install doesn't break
    ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
    ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
    ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
    # Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
    ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
    sudo ps -afe |grep dhclient;
   
  done

[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=7acc9e68

=== End SRU Template ===

=== SRU Verification output ===

### EC2 Xenial upgrade and fresh install test
ubuntu@ip-10-0-20-54:~$ sudo vi /etc/apt/sources.list
ubuntu@ip-10-0-20-54:~$ sudo apt update
...
ubuntu@ip-10-0-20-54:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-0-20-54:~$ grep Trace /var/log/cloud-init.log 
ubuntu@ip-10-0-20-54:~$ sudo apt install cloud-init
Setting up cloud-init (17.1-46-g7acc9e68-0ubuntu1~16.04.1) ...
ubuntu@ip-10-0-20-54:~$ sudo cloud-init init
Cloud-init v. 17.1 running 'init' at Sat, 02 Dec 2017 04:10:13 +0000. Up 167.09 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |                  Address                   |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: |  eth0  | True |                 10.0.20.54                 | 255.255.255.0 |   .    | 0a:a4:58:cb:62:8c |
ci-info: |  eth0  | True | 2600:1f16:dc8:a120:73c6:9626:9c1d:2ecf/128 |       .       | global | 0a:a4:58:cb:62:8c |
ci-info: |   lo   | True |                 127.0.0.1                  |   255.0.0.0   |   .    |         .         |
ci-info: |   lo   | True |                  ::1/128                   |       .       |  host  |         .         |
ci-info: +--------+------+--------------------------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
ci-info: +-------+-------------+-----------+---------------+-----------+-------+
ci-info: | Route | Destination |  Gateway  |    Genmask    | Interface | Flags |
ci-info: +-------+-------------+-----------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 10.0.20.1 |    0.0.0.0    |    eth0   |   UG  |
ci-info: |   1   |  10.0.20.0  |  0.0.0.0  | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+-------------+-----------+---------------+-----------+-------+
ubuntu@ip-10-0-20-54:~$ # remove cloud-init artifacts for clean boot
ubuntu@ip-10-0-20-54:~$ sudo rm -rf /var/log/cloud-init* /var/lib/cloud/; sudo reboot
csmith@uptown:~/src/server/cloud-init/cloud-init ((2f2070f...))$   ssh-keygen -f "/home/csmith/.ssh/known_hosts" -R ec2-18-217-50-75.us-east-2.compute.amazonaws.com
csmith@uptown:~/src/server/cloud-init/cloud-init ((2f2070f...))$ ssh -i ~/Downloads/ubuntu.pem ubuntu@ec2-18-217-50-75.us-east-2.compute.amazonaws.com
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-18-217-50-75.us-east-2.compute.amazonaws.com,18.217.50.75' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1041-aws x86_64)

* Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

51 packages can be updated.
8 updates are security updates.

Last login: Sat Dec  2 04:08:08 2017 from 67.174.121.94
ubuntu@ip-10-0-20-54:~$ dpkg-query --show cloud-init
cloud-init	17.1-46-g7acc9e68-0ubuntu1~16.04.1
ubuntu@ip-10-0-20-54:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-0-20-54:~$ grep Trace /var/log/cloud-init.log

# No leaked dhclients from cloud-init's dhcp_discovery && No missing pidfile Tracebacks
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-0-20-54:~$ ps -afe | grep dhclient
root       980     1  0 04:11 ?        00:00:00 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
root      1076     1  0 04:11 ?        00:00:00 /sbin/dhclient -1 -6 -pf /run/dhclient6.eth0.pid -lf /var/lib/dhcp/dhclient6.eth0.leases -I -df /var/lib/dhcp/dhclient.eth0.leases eth0
ubuntu    1563  1464  0 04:16 pts/0    00:00:00 grep --color=auto dhclient

### EC2 Zesty upgrade and fresh install test
csmith@downtown:~/src/qa-scripts (master)$ ssh ubuntu@ec2-34-236-254-196.compute-1.amazonaws.com
The authenticity of host 'ec2-34-236-254-196.compute-1.amazonaws.com (34.236.254.196)' can't be established.
ECDSA key fingerprint is SHA256:iQqvMhiaS/xx0rzHxcNevpvtwgMgPpQoL8nwLM64QGo.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-34-236-254-196.compute-1.amazonaws.com,34.236.254.196' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-40-generic x86_64)
...

# Show leaked dhcpclient on older version of cloud-init in /var/tmp
ubuntu@ip-10-41-41-163:~$ ps -afe | grep dhcp
root       652     1  0 22:35 ?        00:00:00 /var/tmp/cloud-init/cloud-init-dhcp-ujw4y61p/dhclient -1 -v -lf /var/tmp/cloud-init/cloud-init-dhcp-ujw4y61p/dhcp.leases -pf /var/tmp/cloud-init/cloud-init-dhcp-ujw4y61p/dhclient.pid eth0 -sf /bin/true
root       901     1  0 22:35 ?        00:00:00 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
ubuntu    2015  1998  0 22:36 pts/0    00:00:00 grep --color=auto dhcp
ubuntu@ip-10-41-41-163:~$ grep Trace /var/log/cloud-init*
ubuntu@ip-10-41-41-163:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-163:~$ # Show upgrade path without restart does't Trace
ubuntu@ip-10-41-41-163:~$ sudo cloud-init init
Cloud-init v. 17.1 running 'init' at Mon, 04 Dec 2017 22:38:41 +0000. Up 199.69 seconds.
ci-info: ++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++
ci-info: +--------+------+--------------+---------------+-------+-------------------+
ci-info: | Device |  Up  |   Address    |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+--------------+---------------+-------+-------------------+
ci-info: | eth0:  | True | 10.41.41.163 | 255.255.255.0 |   .   | 02:49:62:e2:03:1c |
ci-info: | eth0:  | True |      .       |       .       |   d   | 02:49:62:e2:03:1c |
ci-info: |  lo:   | True |  127.0.0.1   |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |      .       |       .       |   d   |         .         |
ci-info: +--------+------+--------------+---------------+-------+-------------------+
ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+------------+---------------+-----------+-------+
ci-info: | Route | Destination |  Gateway   |    Genmask    | Interface | Flags |
ci-info: +-------+-------------+------------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 10.41.41.1 |    0.0.0.0    |    eth0   |   UG  |
ci-info: |   1   |  10.41.41.0 |  0.0.0.0   | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+-------------+------------+---------------+-----------+-------+
ubuntu@ip-10-41-41-163:~$ 
ubuntu@ip-10-41-41-163:~$ grep Trace /var/log/cloud-init*
ubuntu@ip-10-41-41-163:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-163:~$ # clean cloud-init artifacts to simulate fresh run
ubuntu@ip-10-41-41-163:~$ sudo rm -rf /var/log/cloud-init* /var/lib/cloud/; sudo reboot
Connection to ec2-34-236-254-196.compute-1.amazonaws.com closed by remote host.
Connection to ec2-34-236-254-196.compute-1.amazonaws.com closed.
csmith@downtown:~/src/qa-scripts (master)$   ssh-keygen -f "/home/csmith/.ssh/known_hosts" -R "ec2-34-236-254-196.compute-1.amazonaws.com"
# Host ec2-34-236-254-196.compute-1.amazonaws.com found: line 1168
/home/csmith/.ssh/known_hosts updated.
Original contents retained as /home/csmith/.ssh/known_hosts.old
csmith@downtown:~/src/qa-scripts (master)$ ssh ubuntu@ec2-34-236-254-196.compute-1.amazonaws.com
The authenticity of host 'ec2-34-236-254-196.compute-1.amazonaws.com (34.236.254.196)' can't be established.
ECDSA key fingerprint is SHA256:luwpDbt+/YHop4foKus+HQci4JAa0xDu2eTy82LpeiI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-34-236-254-196.compute-1.amazonaws.com,34.236.254.196' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-40-generic x86_64)
...
ubuntu@ip-10-41-41-163:~$ 
ubuntu@ip-10-41-41-163:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-163:~$ grep Trace /var/log/cloud-init*
# Show no leaked /var/tmp dhcpclients 
ubuntu@ip-10-41-41-163:~$ ps -afe | grep dhcp
root       807     1  0 22:39 ?        00:00:00 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
ubuntu    1442  1424  0 22:40 pts/0    00:00:00 grep --color=auto dhcp
ubuntu@ip-10-41-41-163:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-41-41-163:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
^[[Aubuntu@ip-10-41-41-163:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
^[[Aubuntu@ip-10-41-41-163:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
^[[Aubuntu@ip-10-41-41-163:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-41-41-163:~$ ps -afe | grep dhcp
root       807     1  0 22:39 ?        00:00:00 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
ubuntu    1486  1424  0 22:40 pts/0    00:00:00 grep --color=auto dhcp
ubuntu@ip-10-41-41-163:~$ # no leaked dhcpclients and no tracebacks
ubuntu@ip-10-41-41-163:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty
ubuntu@ip-10-41-41-163:~$ dpkg-query --show cloud-init
cloud-init	17.1-46-g7acc9e68-0ubuntu1~17.04.1

### EC2 Artful upgrade and fresh install test
ubuntu@ip-10-41-41-19:~$ dpkg-query --show cloud-init
cloud-init	17.1-27-geb292c18-0ubuntu1~17.10.1
# See stale dhcpclient from previous release
ubuntu@ip-10-41-41-19:~$ ps -afe | grep dhclient
root       571     1  0 21:42 ?        00:00:00 /var/tmp/cloud-init/cloud-init-dhcp-zro_t9xo/dhclient -1 -v -lf /var/tmp/cloud-init/cloud-init-dhcp-zro_t9xo/dhcp.leases -pf /var/tmp/cloud-init/cloud-init-dhcp-zro_t9xo/dhclient.pid eth0 -sf /bin/true
ubuntu    1249  1224  0 22:23 pts/0    00:00:00 grep --color=auto dhclient
ubuntu@ip-10-41-41-19:~$ sudo kill 571
ubuntu@ip-10-41-41-19:~$ # Upgrade to proposed
ubuntu@ip-10-41-41-19:~$ sudo sed -i 's/ artful / artful-proposed /' /etc/apt/sources.list
ubuntu@ip-10-41-41-19:~$ sudo apt update
...
ubuntu@ip-10-41-41-19:~$ sudo apt install cloud-init
...
Setting up cloud-init (17.1-46-g7acc9e68-0ubuntu1~17.10.1) ...
ubuntu@ip-10-41-41-19:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-19:~$ grep Trace /var/log/cloud-init.log 
# Check upgrade without restart doesn't fail
ubuntu@ip-10-41-41-19:~$ sudo cloud-init init
Cloud-init v. 17.1 running 'init' at Mon, 04 Dec 2017 22:24:51 +0000. Up 2544.87 seconds.
ci-info: ++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++
ci-info: +--------+------+-------------+---------------+-------+-------------------+
ci-info: | Device |  Up  |   Address   |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+-------------+---------------+-------+-------------------+
ci-info: | eth0:  | True | 10.41.41.19 | 255.255.255.0 |   .   | 02:05:74:57:47:f4 |
ci-info: | eth0:  | True |      .      |       .       |   d   | 02:05:74:57:47:f4 |
ci-info: |  lo:   | True |  127.0.0.1  |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |      .      |       .       |   d   |         .         |
ci-info: +--------+------+-------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ci-info: | Route | Destination |  Gateway   |     Genmask     | Interface | Flags |
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 10.41.41.1 |     0.0.0.0     |    eth0   |   UG  |
ci-info: |   1   |  10.41.41.0 |  0.0.0.0   |  255.255.255.0  |    eth0   |   U   |
ci-info: |   2   |  10.41.41.1 |  0.0.0.0   | 255.255.255.255 |    eth0   |   UH  |
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ubuntu@ip-10-41-41-19:~$ grep Trace /var/log/cloud-init.log 
ubuntu@ip-10-41-41-19:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-19:~$ # Clean /var/lib/cloud to allow cloud-init to run 'fresh'
ubuntu@ip-10-41-41-19:~$ sudo rm -rf /var/log/cloud-init* /var/lib/cloud/; sudo reboot
Connection to ec2-34-231-242-89.compute-1.amazonaws.com closed by remote host.
Connection to ec2-34-231-242-89.compute-1.amazonaws.com closed.
csmith@downtown:~/src/qa-scripts (master)$ ssh ubuntu@ec2-34-231-242-89.compute-1.amazonaws.com
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-34-231-242-89.compute-1.amazonaws.com,34.231.242.89' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 17.10 (GNU/Linux 4.13.0-17-generic x86_64)
...
Last login: Mon Dec  4 22:23:24 2017 from 67.174.121.94
ubuntu@ip-10-41-41-19:~$ 
ubuntu@ip-10-41-41-19:~$ grep Trace /var/log/cloud-init.log 
ubuntu@ip-10-41-41-19:~$ cat /run/cloud-init/result.json 
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}
ubuntu@ip-10-41-41-19:~$ ps -afe | grep dhclient
ubuntu    1082  1064  0 22:26 pts/0    00:00:00 grep --color=auto dhclient
# Show multiple dhcpclient sandbox attempts don't Traceback or leak processes
ubuntu@ip-10-41-41-19:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-41-41-19:~$ 
ubuntu@ip-10-41-41-19:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-41-41-19:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
^[[Aubuntu@ip-10-41-41-19:~$ sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()'
ubuntu@ip-10-41-41-19:~$ ps -afe | grep dhclient
ubuntu    1125  1064  0 22:28 pts/0    00:00:00 grep --color=auto dhclient
ubuntu@ip-10-41-41-19:~$ dpkg-query --show cloud-init
cloud-init	17.1-46-g7acc9e68-0ubuntu1~17.10.1

description:	updated
description:	updated
description:	updated
tags:	added: verification-done verification-done-artful verification-done-xenial verification-done-zesty removed: verification-needed verification-needed-artful verification-needed-zesty

Chad Smith (chad.smith) on 2017-12-05

Changed in cloud-init (Ubuntu Xenial):
status:	Confirmed → Fix Committed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-12-13:

#13

This bug was fixed in the package cloud-init - 17.1-46-g7acc9e68-0ubuntu1~16.04.1

---------------
cloud-init (17.1-46-g7acc9e68-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - ec2: Fix sandboxed dhclient background process cleanup.
      (LP: #1735331)
    - tests: NoCloudKVMImage do not modify the original local cache image.
    - tests: Enable bionic in integration tests. [Joshua Powers]
    - tests: Use apt-get to install a deb so that depends get resolved.
    - sysconfig: Correctly render dns and dns search info.
      [Ryan McCabe]

cloud-init (17.1-41-g76243487-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: Fix capitilazation in 'AliYun'
    (LP: #1728186)
  * New upstream snapshot (LP: #1733653)
    - integration test: replace curtin test ppa with cloud-init test ppa.
    - EC2: Fix bug using fallback_nic and metadata when restoring from cache.
    - EC2: Kill dhclient process used in sandbox dhclient.
    - ntp: fix configuration template rendering for openSUSE and SLES
    - centos: Provide the failed #include url in error messages
    - Catch UrlError when #include'ing URLs [Andrew Jorgensen]
    - hosts: Fix openSUSE and SLES setup for /etc/hosts and clarify docs.
      [Robert Schweikert]
    - rh_subscription: Perform null checks for enabled and disabled repos.
      [Dave Mulford]
    - Improve warning message when a template is not found.
      [Robert Schweikert]
    - Replace the temporary i9n.brickies.net with i9n.cloud-init.io.
    - Azure: don't generate network configuration for SRIOV devices
    - tests: address some minor feedback missed in last merge.
    - tests: integration test cleanup and full pass of nocloud-kvm.
    - Gentoo: chmod +x on all files in sysvinit/gentoo/
      [Carlos Konstanski]

-- Chad Smith <email address hidden> Fri, 01 Dec 2017 10:05:01 -0700

Changed in cloud-init (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Revision history for this message

Chris Halse Rogers (raof) wrote on 2017-12-13: Update Released

#14

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-12-13:

#15

This bug was fixed in the package cloud-init - 17.1-46-g7acc9e68-0ubuntu1~17.04.1

---------------
cloud-init (17.1-46-g7acc9e68-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

  * New upstream snapshot.
    - ec2: Fix sandboxed dhclient background process cleanup.
      (LP: #1735331)
    - tests: NoCloudKVMImage do not modify the original local cache image.
    - tests: Enable bionic in integration tests. [Joshua Powers]
    - tests: Use apt-get to install a deb so that depends get resolved.
    - sysconfig: Correctly render dns and dns search info.
      [Ryan McCabe]

cloud-init (17.1-41-g76243487-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

  * debian/cloud-init.templates: Fix capitilazation in 'AliYun'
    (LP: #1728186)
  * New upstream snapshot (LP: #1733653)
    - integration test: replace curtin test ppa with cloud-init test ppa.
    - EC2: Fix bug using fallback_nic and metadata when restoring from cache.
    - EC2: Kill dhclient process used in sandbox dhclient.
    - ntp: fix configuration template rendering for openSUSE and SLES
    - centos: Provide the failed #include url in error messages
    - Catch UrlError when #include'ing URLs [Andrew Jorgensen]
    - hosts: Fix openSUSE and SLES setup for /etc/hosts and clarify docs.
      [Robert Schweikert]
    - rh_subscription: Perform null checks for enabled and disabled repos.
      [Dave Mulford]
    - Improve warning message when a template is not found.
      [Robert Schweikert]
    - Replace the temporary i9n.brickies.net with i9n.cloud-init.io.
    - Azure: don't generate network configuration for SRIOV devices
    - tests: address some minor feedback missed in last merge.
    - tests: integration test cleanup and full pass of nocloud-kvm.
    - Gentoo: chmod +x on all files in sysvinit/gentoo/
      [Carlos Konstanski]

-- Chad Smith <email address hidden> Fri, 01 Dec 2017 10:02:24 -0700

Changed in cloud-init (Ubuntu Zesty):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-12-13:

#16

This bug was fixed in the package cloud-init - 17.1-46-g7acc9e68-0ubuntu1~17.10.1

---------------
cloud-init (17.1-46-g7acc9e68-0ubuntu1~17.10.1) artful-proposed; urgency=medium

  * New upstream snapshot.
    - ec2: Fix sandboxed dhclient background process cleanup.
      (LP: #1735331)
    - tests: NoCloudKVMImage do not modify the original local cache image.
    - tests: Enable bionic in integration tests. [Joshua Powers]
    - tests: Use apt-get to install a deb so that depends get resolved.
    - sysconfig: Correctly render dns and dns search info.
      [Ryan McCabe]

cloud-init (17.1-41-g76243487-0ubuntu1~17.10.1) artful-proposed; urgency=medium

  * debian/cloud-init.templates: Fix capitilazation in 'AliYun'
    (LP: #1728186)
  * New upstream snapshot (LP: #1733653)
    - integration test: replace curtin test ppa with cloud-init test ppa.
    - EC2: Fix bug using fallback_nic and metadata when restoring from cache.
    - EC2: Kill dhclient process used in sandbox dhclient.
    - ntp: fix configuration template rendering for openSUSE and SLES
    - centos: Provide the failed #include url in error messages
    - Catch UrlError when #include'ing URLs [Andrew Jorgensen]
    - hosts: Fix openSUSE and SLES setup for /etc/hosts and clarify docs.
      [Robert Schweikert]
    - rh_subscription: Perform null checks for enabled and disabled repos.
      [Dave Mulford]
    - Improve warning message when a template is not found.
      [Robert Schweikert]
    - Replace the temporary i9n.brickies.net with i9n.cloud-init.io.
    - Azure: don't generate network configuration for SRIOV devices
    - tests: address some minor feedback missed in last merge.
    - tests: integration test cleanup and full pass of nocloud-kvm.
    - Gentoo: chmod +x on all files in sysvinit/gentoo/
      [Carlos Konstanski]

-- Chad Smith <email address hidden> Fri, 01 Dec 2017 09:58:48 -0700

Changed in cloud-init (Ubuntu Artful):
status:	Fix Committed → Fix Released

Revision history for this message

Scott Moser (smoser) wrote on 2017-12-14: Fixed in Cloud-init 1705804

#17

This bug is believed to be fixed in cloud-init in 1705804. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status:	Fix Committed → Fix Released

Revision history for this message

Stretch (chriswright96) wrote on 2018-02-08:

#18

We are seeing errors very similar to this:

can't create /var/tmp/cloud-init/cloud-init-.../dhcp.leases: No such file or directory

The /var/tmp/cloud-init has been created, world writable.

Started seeing the errors a few days after the fix was released - around 5th December.

AWS EC2
ubuntu 16.04.1
cloud-init 17.1-46-g7acc9e68-0ubuntu1~16.04.1

Revision history for this message

Scott Moser (smoser) wrote on 2018-02-08:

#19

Stretch, can you please attach /var/log/cloud-init.log ?
Better would be to attach output of 'cloud-init collect-logs'

Thanks.

Revision history for this message

James Falcon (falcojr) wrote on 2023-05-11:

#20

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3073

cloud-init

ec2: zesty tempfile sandbox dhclient.pid file can't be created

Bug Description

Related branches

Other bug subscribers

Bug attachments

Remote bug watches

	Status	Importance	Assigned to
cloud-init	Fix Released	High	Chad Smith
cloud-init (Ubuntu)	Fix Released	Medium	Unassigned
Xenial	Fix Released	Medium	Unassigned
Zesty	Fix Released	Medium	Unassigned
Artful	Fix Released	Medium	Unassigned
Bionic	Fix Released	Medium	Unassigned