FUEL 10.0 Docker stuck in during installation fuel

Yeah, stuck with same problem.

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

I wonder, why we are still installing docker.

It is needed for upgrades from the 8.0, I think it should be removed completely in the 10.0.
Please consult with octane-team.

The problem is in fuel-menu.

Fuel-menu incorrectly configures something in puppets, so it disables hotplug and doesn't enable back (can be found in /var/log/puppet/fuelmenu-puppet.log)

Docker calls script docker-storage-setup, which in turn runs lvcreate. But lvcreate waits for udev to create devices in /dev (or something else), anyway, hotplug must be enabled for running lvcreate.

Dmitry, please paste here what can be found in /var/log/puppet/fuelmenu-puppet.log about hotplug.

Root cause analysis:

Symptoms:
When the <Save and Quite> key pressed, the script hangs on systemctl start docker.
Actually the docker.service wait docker-storage storage creating LVM volume, which in his turn
hang up on the semaphore.

Here the CLI investigation
=========================================================================
(gdb) bt
#0 0x00007f7ce75884e7 in semop () from /lib64/libc.so.6
#1 0x00007f7ce7cb40cc in dm_udev_wait () from /lib64/libdevmapper.so.1.02
#2 0x00007f7ce88537bd in fs_unlock ()
#3 0x00007f7ce8860a05 in _file_lock_resource ()
#4 0x00007f7ce87f3878 in _lock_vol ()
#5 0x00007f7ce87f41c3 in lock_vol ()
#6 0x00007f7ce8808b6c in wipe_lv ()
#7 0x00007f7ce880c54f in _lv_create_an_lv ()
#8 0x00007f7ce880d55f in lv_create_single ()
#9 0x00007f7ce879e13a in lvcreate ()
#10 0x00007f7ce87a63d8 in lvm_run_command ()
#11 0x00007f7ce87a6e40 in lvm2_main ()
#12 0x00007f7ce74b1b15 in __libc_start_main () from /lib64/libc.so.6
#13 0x00007f7ce878de41 in _start ()
(gdb)

# ipcs -a

------ Message Queues --------
key msqid owner perms used-bytes messages

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status

------ Semaphore Arrays --------
key semid owner perms nsems
0x0d4dbf3c 98304 root 600 1

# ipcs -sti 98304

Semaphore Array semid=98304
uid=0 gid=0 cuid=0 cgid=0
mode=0600, access_perms=0600
nsems = 1
otime = Wed Jul 6 12:29:04 2016
ctime = Wed Jul 6 12:29:04 2016
semnum value ncount zcount pid
0 1 0 0 25896

# ps xawf -eo pid,ppid,user,cgroup,args

    1 0 root - /usr/lib/systemd/systemd --switched-root --system --deserialize 21
  491 1 root 7:devices:/system.slice/sys /usr/lib/systemd/systemd-journald
  505 1 root 7:devices:/system.slice/lvm /usr/sbin/lvmetad -f
  512 1 root 7:devices:/system.slice/sys /usr/lib/systemd/systemd-udevd
  662 1 root 7:devices:/system.slice/aud /sbin/auditd -n
  686 1 root 7:devices:/system.slice/irq /usr/sbin/irqbalance --foreground
  687 1 dbus 7:devices:/system.slice/dbu /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  692 1 root 7:devices:/system.slice/sys /usr/lib/systemd/systemd-logind
  698 1 root 7:devices:/system.slice/cro /usr/sbin/crond -n
  715 1 root 7:devices:/system.slice/wpa /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplicant.log -c /etc/wpa_supplicant/wpa_supplicant.conf -u -f /var/log/wpa_supplica
  716 1 polkitd 7:devices:/system.slice/pol /usr/lib/polkit-1/polkitd --no-debug
10693 1 root 7:devices:/system.slice/rsy /usr/sbin/rsyslogd -n
11272 1 ntp 7:devices:/system.slice/ntp /usr/sbin/ntpd -u ntp:ntp -g
25796 1 root 7:devices:/system.slice,1:n /usr/sbin/sshd -D
26982 25796 root 1:name=systemd:/user.slice/ \_ sshd: root@pts/0
26986 26982 root 1:name=systemd:/user.slice/ | \_ -bash
27011 26986 root 1:name=systemd:/user.slice/ | \_ lvdispl...

Hello, Albert,

puppet disables hotplug and doesn't enable again because a error occur, because it assumes, that facts are objects, not strings. But on master node we don't set stringify_facts = false, so all facts are converted to strings.

The solution is to add stringify_facts = false to /etc/puppet/puppet.conf

Related fix proposed to branch: master
Review: https://review.openstack.org/341689

Reviewed: https://review.openstack.org/341689
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=ead49981075c7dc064e3aa67acd456c76aadefc5
Submitter: Jenkins
Branch: master

commit ead49981075c7dc064e3aa67acd456c76aadefc5
Author: Maksim Malchuk <email address hidden>
Date: Wed Jul 13 20:07:33 2016 +0300

    Setup stringify_facts for the puppet

    This change will setup stringify_facts to false for the puppet.
    Also, we need the tools to manage configuration files from the
    bash scripts, as we widely use augeas bindings in the puppet, lets
    use the same tools in the bash scripts.

    DocImpact
    Change-Id: I6c447f4a006825dd2dcdd8597f63f84eb2670556
    Closes-Bug: #1602774
    Closes-Bug: #1593207
    Signed-off-by: Maksim Malchuk <email address hidden>

This issue was fixed in the openstack/fuel-main 10.0.0rc1 release candidate.

This issue was fixed in the openstack/fuel-main 10.0.0 release.