MAAS

Failed to deploy with LVM layout

Bug #1487114 reported by Andres Rodriguez
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
MAAS 1.9.0

Bug Description

Using curtin: 0.1.0~bzr241-0ubuntu1~ppa0

Creating /etc/postfix/dynamicmaps.cf
Adding tcp map entry to /etc/postfix/dynamicmaps.cf
Adding sqlite map entry to /etc/postfix/dynamicmaps.cf
Adding group `postdrop' (GID 113) ...
Done.
setting myhostname: back-throne
setting alias maps
setting alias database
mailname is not a fully qualified domain name. Not changing /etc/mailname.
setting destinations: localdomain, localhost, localhost.localdomain, localhost
setting relayhost:
setting mynetworks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
setting mailbox_size_limit: 0
setting recipient_delimiter: +
setting inet_interfaces: all
/etc/aliases does not exist, creating it.
WARNING: /etc/aliases exists, but does not have a root alias.

Postfix is now set up with a default configuration. If you need to make
changes, edit
/etc/postfix/main.cf (and others) as needed. To view Postfix configuration
values, see postconf(1).

After modifying main.cf, be sure to run '/etc/init.d/postfix reload'.

Running newaliases
invoke-rc.d: policy-rc.d denied execution of restart.
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
Processing triggers for initramfs-tools (0.103ubuntu4.2) ...
update-initramfs: Generating /boot/initrd.img-3.13.0-35-generic
cryptsetup: WARNING: failed to detect canonical device of /media/root-ro/
cryptsetup: WARNING: could not determine root device from /etc/fstab
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for ufw (0.34~rc-0ubuntu2) ...
mdadm: No arrays found in config file or automatically
File descriptor 3 (socket:[11922]) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 4 (/curtin/configs/config-000.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 5 (/curtin/configs/config-001.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 6 (/curtin/configs/config-002.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 7 (/curtin/configs/config-003.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 8 (/curtin/configs/config-004.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 9 (/curtin/configs/config-005.cfg) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 10 (/var/log/curtin/install.log) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 11 (/var/log/curtin/install.log) leaked on vgcreate invocation. Parent PID 1493: python2
File descriptor 12 (/curtin/bin/curtin) leaked on vgcreate invocation. Parent PID 1493: python2
  No physical volume label read from /dev/sda1
  Physical volume "/dev/sda1" successfully created
  Volume group "vgroot" successfully created
File descriptor 3 (socket:[11922]) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 4 (/curtin/configs/config-000.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 5 (/curtin/configs/config-001.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 6 (/curtin/configs/config-002.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 7 (/curtin/configs/config-003.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 8 (/curtin/configs/config-004.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 9 (/curtin/configs/config-005.cfg) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 10 (/var/log/curtin/install.log) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 11 (/var/log/curtin/install.log) leaked on lvcreate invocation. Parent PID 1493: python2
File descriptor 12 (/curtin/bin/curtin) leaked on lvcreate invocation. Parent PID 1493: python2
  Rounding up size to full physical extent 119.24 GiB
  Volume group "vgroot" has insufficient free space (30525 extents): 30526 required.
An error occured handling 'vgroot-lvroot': ProcessExecutionError - Unexpected error while running command.
Command: ['lvcreate', 'vgroot', '-n', 'lvroot', '-L', '128031481856B']
Exit code: 5
Reason: -
Stdout: ''
Stderr: ''
Unexpected error while running command.
Command: ['lvcreate', 'vgroot', '-n', 'lvroot', '-L', '128031481856B']
Exit code: 5
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: 'Ign http://archive.ubuntu.com trusty InRelease\nIgn http://security.ubuntu.com trusty-security InRelease\nIgn http://ppa.launchpad.net trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nGet:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]\nHit http://archive.ubuntu.com trusty Release.gpg\nGet:2 http://security.ubuntu.com trusty-security Release [63.5 kB]\nGet:3 http://ppa.launchpad.net trusty Release.gpg [316 B]\nGet:4 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:5 http://ppa.launchpad.net trusty Release [15.1 kB]\nHit http://archive.ubuntu.com trusty Release\nGet:6 http://archive.ubuntu.com trusty-updates Release [63.5 kB]\nGet:7 http://security.ubuntu.com trusty-security/main Sources [92.2 kB]\nGet:8 http://ppa.launchpad.net trusty/main amd64 Packages [4871 B]\nGet:9 http://ppa.launchpad.net trusty/main Translation-en [3699 B]\nGet:10 http://archive.ubuntu.com trusty/main Sources [1064 kB]\nGet:11 http://security.ubuntu.com trusty-security/universe Sources [29.5 kB]\nGet:12 http://security.ubuntu.com trusty-security/main amd64 Packages [333 kB]\nGet:13 http://archive.ubuntu.com trusty/universe Sources [6399 kB]\nGet:14 http://security.ubuntu.com trusty-security/universe amd64 Packages [114 kB]\nGet:15 http://security.ubuntu.com trusty-security/main Translation-en [181 kB]\nGet:16 http://security.ubuntu.com trusty-security/universe Translation-en [66.6 kB]\nHit http://archive.ubuntu.com trusty/main amd64 Packages\nHit http://archive.ubuntu.com trusty/universe amd64 Packages\nHit http://archive.ubuntu.com trusty/main Translation-en\nHit http://archive.ubuntu.com trusty/universe Translation-en\nGet:17 http://archive.ubuntu.com trusty-updates/main Sources [229 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/universe Sources [133 kB]\nGet:19 http://archive.ubuntu.com trusty-updates/main amd64 Packages [605 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [308 kB]\nGet:21 http://archive.ubuntu.com trusty-updates/main Translation-en [292 kB]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [163 kB]\nFetched 10.2 MB in 34s (293 kB/s)\nReading package lists...\nReading package lists...\nBuilding dependency tree...\nReading state information...\nThe following extra packages will be installed:\n libdevmapper-event1.02.1 libreadline5 postfix ssl-cert watershed\nSuggested packages:\n thin-provisioning-tools procmail postfix-mysql postfix-pgsql postfix-ldap\n postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc\n openssl-blacklist\nRecommended packages:\n default-mta mail-transport-agent\nThe following NEW packages will be installed:\n bcache-tools libdevmapper-event1.02.1 libreadline5 lvm2 mdadm postfix\n ssl-cert watershed\n0 upgraded, 8 newly installed, 0 to remove and 175 not upgraded.\nNeed to get 2364 kB of archives.\nAfter this operation, 6837 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu/ trusty/main libdevmapper-event1.02.1 amd64 2:1.02.77-6ubuntu2 [10.8 kB]\nGet:2 http://ppa.launchpad.net/lamont/ppa/ubuntu/ trusty/main bcache-tools amd64 1.0.7-1~14.04 [17.2 kB]\nGet:3 http://archive.ubuntu.com/ubuntu/ trusty/main libreadline5 amd64 5.2+dfsg-2 [130 kB]\nGet:4 http://ppa.launchpad.net/lamont/ppa/ubuntu/ trusty/main postfix amd64 2.11.2-1~trusty1 [1347 kB]\nGet:5 http://archive.ubuntu.com/ubuntu/ trusty/main watershed amd64 7 [11.4 kB]\nGet:6 http://archive.ubuntu.com/ubuntu/ trusty/main lvm2 amd64 2.02.98-6ubuntu2 [470 kB]\nGet:7 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mdadm amd64 3.2.5-5ubuntu4.2 [361 kB]\nGet:8 http://archive.ubuntu.com/ubuntu/ trusty/main ssl-cert all 1.0.33 [16.6 kB]\nPreconfiguring packages ...\nFetched 2364 kB in 6s (360 kB/s)\nSelecting previously unselected package libdevmapper-event1.02.1:amd64.\n(Reading database ... 56007 files and directories currently installed.)\nPreparing to unpack .../libdevmapper-event1.02.1_2%3a1.02.77-6ubuntu2_amd64.deb ...\nUnpacking libdevmapper-event1.02.1:amd64 (2:1.02.77-6ubuntu2) ...\nSelecting previously unselected package libreadline5:amd64.\nPreparing to unpack .../libreadline5_5.2+dfsg-2_amd64.deb ...\nUnpacking libreadline5:amd64 (5.2+dfsg-2) ...\nSelecting previously unselected package bcache-tools.\nPreparing to unpack .../bcache-tools_1.0.7-1~14.04_amd64.deb ...\nUnpacking bcache-tools (1.0.7-1~14.04) ...\nSelecting previously unselected package watershed.\nPreparing to unpack .../archives/watershed_7_amd64.deb ...\nUnpacking watershed (7) ...\nSelecting previously unselected package lvm2.\nPreparing to unpack .../lvm2_2.02.98-6ubuntu2_amd64.deb ...\nUnpacking lvm2 (2.02.98-6ubuntu2) ...\nSelecting previously unselected package mdadm.\nPreparing to unpack .../mdadm_3.2.5-5ubuntu4.2_amd64.deb ...\nUnpacking mdadm (3.2.5-5ubuntu4.2) ...\nSelecting previously unselected package ssl-cert.\nPreparing to unpack .../ssl-cert_1.0.33_all.deb ...\nUnpacking ssl-cert (1.0.33) ...\nSelecting previously unselected package postfix.\nPreparing to unpack .../postfix_2.11.2-1~trusty1_amd64.deb ...\nUnpacking postfix (2.11.2-1~trusty1) ...\nProcessing triggers for man-db (2.6.7.1-1) ...\nProcessing triggers for ureadahead (0.100.0-16) ...\nProcessing triggers for ufw (0.34~rc-0ubuntu2) ...\nSetting up libdevmapper-event1.02.1:amd64 (2:1.02.77-6ubuntu2) ...\nSetting up libreadline5:amd64 (5.2+dfsg-2) ...\nSetting up bcache-tools (1.0.7-1~14.04) ...\nupdate-initramfs: deferring update (trigger activated)\nSetting up watershed (7) ...\nupdate-initramfs: deferring update (trigger activated)\nSetting up lvm2 (2.02.98-6ubuntu2) ...\nupdate-initramfs: deferring update (trigger activated)\nSetting up mdadm (3.2.5-5ubuntu4.2) ...\nGenerating mdadm.conf... done.\n Removing any system startup links for /etc/init.d/mdadm-raid ...\nupdate-initramfs: deferring update (trigger activated)\n/usr/sbin/grub-probe: error: failed to get canonical path of `overlayroot\'.\ninvoke-rc.d: policy-rc.d denied execution of start.\nSetting up ssl-cert (1.0.33) ...\nhostname: Name or service not known\nmake-ssl-cert: Could not get FQDN, using "back-throne".\nmake-ssl-cert: You may want to fix your /etc/hosts and/or DNS setup and run\nmake-ssl-cert: make-ssl-cert generate-default-snakeoil --force-overwrite\nmake-ssl-cert: again.\nSetting up postfix (2.11.2-1~trusty1) ...\nAdding group `postfix\' (GID 112) ...\nDone.\nAdding system user `postfix\' (UID 106) ...\nAdding new user `postfix\' (UID 106) with group `postfix\' ...\nNot creating home directory `/var/spool/postfix\'.\nCreating /etc/postfix/dynamicmaps.cf\nAdding tcp map entry to /etc/postfix/dynamicmaps.cf\nAdding sqlite map entry to /etc/postfix/dynamicmaps.cf\nAdding group `postdrop\' (GID 113) ...\nDone.\nsetting myhostname: back-throne\nsetting alias maps\nsetting alias database\nmailname is not a fully qualified domain name. Not changing /etc/mailname.\nsetting destinations: localdomain, localhost, localhost.localdomain, localhost\nsetting relayhost: \nsetting mynetworks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128\nsetting mailbox_size_limit: 0\nsetting recipient_delimiter: +\nsetting inet_interfaces: all\n/etc/aliases does not exist, creating it.\nWARNING: /etc/aliases exists, but does not have a root alias.\n\nPostfix is now set up with a default configuration. If you need to make \nchanges, edit\n/etc/postfix/main.cf (and others) as needed. To view Postfix configuration\nvalues, see postconf(1).\n\nAfter modifying main.cf, be sure to run \'/etc/init.d/postfix reload\'.\n\nRunning newaliases\ninvoke-rc.d: policy-rc.d denied execution of restart.\nProcessing triggers for libc-bin (2.19-0ubuntu6.3) ...\nProcessing triggers for initramfs-tools (0.103ubuntu4.2) ...\nupdate-initramfs: Generating /boot/initrd.img-3.13.0-35-generic\ncryptsetup: WARNING: failed to detect canonical device of /media/root-ro/\ncryptsetup: WARNING: could not determine root device from /etc/fstab\nW: mdadm: /etc/mdadm/mdadm.conf defines no arrays.\nProcessing triggers for ureadahead (0.100.0-16) ...\nProcessing triggers for ufw (0.34~rc-0ubuntu2) ...\nmdadm: No arrays found in config file or automatically\nFile descriptor 3 (socket:[11922]) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 4 (/curtin/configs/config-000.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 5 (/curtin/configs/config-001.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 6 (/curtin/configs/config-002.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 7 (/curtin/configs/config-003.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 8 (/curtin/configs/config-004.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 9 (/curtin/configs/config-005.cfg) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 10 (/var/log/curtin/install.log) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 11 (/var/log/curtin/install.log) leaked on vgcreate invocation. Parent PID 1493: python2\nFile descriptor 12 (/curtin/bin/curtin) leaked on vgcreate invocation. Parent PID 1493: python2\n No physical volume label read from /dev/sda1\n Physical volume "/dev/sda1" successfully created\n Volume group "vgroot" successfully created\nFile descriptor 3 (socket:[11922]) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 4 (/curtin/configs/config-000.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 5 (/curtin/configs/config-001.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 6 (/curtin/configs/config-002.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 7 (/curtin/configs/config-003.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 8 (/curtin/configs/config-004.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 9 (/curtin/configs/config-005.cfg) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 10 (/var/log/curtin/install.log) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 11 (/var/log/curtin/install.log) leaked on lvcreate invocation. Parent PID 1493: python2\nFile descriptor 12 (/curtin/bin/curtin) leaked on lvcreate invocation. Parent PID 1493: python2\n Rounding up size to full physical extent 119.24 GiB\n Volume group "vgroot" has insufficient free space (30525 extents): 30526 required.\nAn error occured handling \'vgroot-lvroot\': ProcessExecutionError - Unexpected error while running command.\nCommand: [\'lvcreate\', \'vgroot\', \'-n\', \'lvroot\', \'-L\', \'128031481856B\']\nExit code: 5\nReason: -\nStdout: \'\'\nStderr: \'\'\nUnexpected error while running command.\nCommand: [\'lvcreate\', \'vgroot\', \'-n\', \'lvroot\', \'-L\', \'128031481856B\']\nExit code: 5\nReason: -\nStdout: \'\'\nStderr: \'\'\n'
Stderr: ''

See original description
Tags: storage

Related branches

lp:~blake-rouse/maas/fix-1487114
Gavin Panella (community): Approve
Diff: 82 lines (+16/-17)
2 files modified
src/maasserver/models/filesystemgroup.py (+8/-12)
src/maasserver/models/tests/test_filesystemgroup.py (+8/-5)
Andres Rodriguez (andreserl)
description: updated
Revision history for this message
Ryan Harper (raharper) wrote :

Please append the config file passed to curin. In particular, we need to see the storage config passed in.

Changed in curtin:
status: New → Incomplete
Revision history for this message
Andres Rodriguez (andreserl) wrote :
Revision history for this message
Ryan Harper (raharper) wrote :

I can recreate this with curtin launch.

Changed in curtin:
status: Incomplete → Confirmed
Revision history for this message
Ryan Harper (raharper) wrote :

Curtin does not do any calculation for sizes, rather it relies on input from the config generator. So let's walk through the numbers.

We create a partition of size 128032530432B, offset by 2097152B.

There is a bug (I'll file this) that curtin isn't reading offset from config, so it's defaulting to 1M (1048576B).

Next, we add the partition as a LVM Physical Volume. LVM has its own alignment and overhead. After adding the above partition, pvdsplay says:

# pvdisplay --units B
               /run/lvm/lvmetad.socket: connect failed: No such file or directory
               WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
               --- Physical volume ---
               PV Name /dev/vdc1
               VG Name vgroot
               PV Size 128032530944 B / not usable 1401344 B
               Allocatable yes
               PE Size 4194304 B
               Total PE 30525
               Free PE 30525
               Allocated PE 0
               PV UUID T4OiMH-Zb7I-yy4N-PWZt-OR5M-PTPq-eeDTvN

notice the not usable bits.

The size of the partition created is not sufficient to hold the LV as it's not rounded to a
multiple of PE (4MB). The partition, sized 128035323904B, is only 1048576B larger.

Instead, if you keep the partition sized smaller, it will fit:

int(128032530432 / (4 * 1024 * 1024))
30525

Note, that even when we fix the offset this won't affect this calculation, instead MAAS LVM calculations need to account for default PE size (4MiB) and adjust the size of the partitions and lvs to fit.

If we adjust those sizes in the storage config to:

% diff -u storage.cfg storage-fix.cfg
--- storage.cfg 2015-08-21 09:53:57.737547864 -0500
+++ storage-fix.cfg 2015-08-21 09:54:26.681548701 -0500
@@ -11,7 +11,7 @@
     name: vgroot
     type: lvm_volgroup
     uuid: de8dc011-b95c-4b95-8406-09136172e54a
- - {id: vgroot-lvroot, name: lvroot, size: 128031481856B, type: lvm_partition, volgroup: vgroot}
+ - {id: vgroot-lvroot, name: lvroot, size: 127976603648B, type: lvm_partition, volgroup: vgroot}
   - {fstype: ext4, id: vgroot-lvroot_format, label: root, type: format, uuid: 1de4afe9-2637-4e5a-9ffa-d4ff96cef02e,
     volume: vgroot-lvroot}
   - {device: vgroot-lvroot_format, id: vgroot-lvroot_mount, path: /, type: mount}

The config is now deployable.

Changed in curtin:
status: Confirmed → Invalid
Blake Rouse (blake-rouse)
Changed in maas:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Blake Rouse (blake-rouse)
milestone: none → 1.9.0
tags: added: storage
Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
no longer affects: curtin
Revision history for this message
Scott Moser (smoser) wrote :

I just upgraded to 1.9, then installed a node and hit this.
http://paste.ubuntu.com/12639563/ has data on the node.

# dpkg-query --show python-curtin maas
maas 1.9.0~alpha2+bzr4322-0ubuntu1~trusty1
python-curtin 0.1.0~bzr270-0ubuntu1~trusty0

Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.