System cannot be enlisted with the ephemeral release images

Bug #1459865 reported by Larry Michel
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Unassigned
1.8
Fix Released
Critical
Unassigned
maas-images
Invalid
Undecided
Unassigned

Bug Description

We are seeing failure to enlist nodes in our lab. While booting, system drops to initramfs shell.

These are with ephemeral daily images and 1.8.0~beta8+bzr3951-0ubuntu1~trusty1, but we saw this with earlier beta versions (2 or 3).

Tags: oil

Related branches

Revision history for this message
Larry Michel (lmic) wrote :
Changed in maas:
importance: Undecided → Critical
Changed in maas:
status: New → Incomplete
Revision history for this message
Larry Michel (lmic) wrote :

Console output attached.

Changed in maas:
status: Incomplete → New
Revision history for this message
Larry Michel (lmic) wrote :

so issue seems to be that /root/dev does not exist. I created /root/dev and was mount /dev:

(initramfs) mount
rootfs on / type rootfs (rw,size=16445592k,nr_inodes=4111398)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=16445608k,nr_inodes=4111402,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
copymods on /root/lib/modules type tmpfs (rw,relatime)
(initramfs) ls /root/dev
ls: /root/dev: No such file or directory
(initramfs) mkdir /root/dev
(initramfs) mount
rootfs on / type rootfs (rw,size=16445592k,nr_inodes=4111398)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=16445608k,nr_inodes=4111402,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
copymods on /root/lib/modules type tmpfs (rw,relatime)
(initramfs) ls /root
lib dev
(initramfs) mount /dev /root/dev
(initramfs)

Revision history for this message
Larry Michel (lmic) wrote :

same for sys and proc

Changed in maas:
status: New → Incomplete
Revision history for this message
Narinder Gupta (narindergupta) wrote :

Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
mount: can't read '/etc/fstab': No such file or directory
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory
done.
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.

BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

Changed in maas-images:
status: New → Incomplete
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Narinder,

What hardware is this? What version of MAAS? How is your MAAS configured?

Revision history for this message
Scott Moser (smoser) wrote :

[ 0.000000] Command line: nomodeset netcfg/choose_interface=auto text priority=critical auto url=http://10.245.0.10/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed locale=en_US hostname=maas-enlist domain=oil log_host=10.245.0.10 log_port=514 -- console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/boot-initrd BOOT_IMAGE=ubuntu/amd64/generic/trusty/release/boot-kernel BOOTIF=01-d4-ae-52-cb-c3-b8

In the provided console log in comment 2, you booted with the above kernel command line.
that would seem to imply that you were booting d-i.
normal ephemeral or enlistment parameters would have iscsi params like:

  iscsi_target_name=${target_name} iscsi_target_ip=$ipaddr
  iscsi_initiator=maas-enlist
  ip=::::maas-enlist:BOOTIF ro
  root=/dev/disk/by-path/ip-$ipaddr:$iport-iscsi-${target_name}-lun-1

Revision history for this message
Narinder Gupta (narindergupta) wrote :

Andres,
I can reproduce this on all HP hardware Gen8/Gen9 as well as virtual machine with MAAS Version 1.8.0+bzr4001-0ubuntu2 (trusty1). I was running MAAS version 1.7.1 and configured manually then upgrade it using sudo apt-get dist-upgrade. Scott Moser i am also getting the same kernel parameters during enlistment.

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.13.0-61-generic (buildd@lgw01-50) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #100-Ubuntu SMP Wed Jul 29 11:21:34 UTC 2015 (Ubuntu 3.13.0-61.100-generic 3.13.11-ckt22)
[ 0.000000] Command line: nomodeset netcfg/choose_interface=auto text priority=critical auto url=http://10.10.2.11/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed locale=en_US hostname=maas-enlist domain=maas log_host=10.10.2.11 log_port=514 --- console=ttyS0,9600 initrd=ubuntu/amd64/generic/trusty/daily/boot-initrd BOOT_IMAGE=ubuntu/amd64/generic/trusty/daily/boot-kernel BOOTIF=01-a0-1d-48-b5-65-75
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] e820: BIOS-provided physical RAM map:

Looks like MAAS is not passing the iscsi parameters some how in this case.

Revision history for this message
Larry Michel (lmic) wrote :

Console files.. 1st one is from java console, 2nd is from serial console.

Changed in maas-images:
status: Incomplete → Invalid
Changed in maas:
milestone: none → 1.9.0
status: Incomplete → In Progress
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Larry Michel (lmic) wrote :

Andres, I tried patching the file with fix from this branch then restarting maas-regiond but I am still not seeing the server booting the ephemeral image during enlistment, and it eventually drops in initramfs shell. This is on a newly configured maas server on which I recreated issue. Maas version is 1.8.1.

Revision history for this message
Narinder Gupta (narindergupta) wrote : Re: [Bug 1459865] Re: System cannot be enlisted with the ephemeral release images

You need to restart clusterd as well then it worked for me.

On Wed, Aug 12, 2015, 17:35 Larry Michel <email address hidden> wrote:

> Andres, I tried patching the file with fix from this branch then
> restarting maas-regiond but I am still not seeing the server booting the
> ephemeral image during enlistment, and it eventually drops in initramfs
> shell. This is on a newly configured maas server on which I recreated
> issue. Maas version is 1.8.1.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1459865
>
> Title:
> System cannot be enlisted with the ephemeral release images
>
> Status in MAAS:
> Fix Committed
> Status in MAAS 1.8 series:
> Fix Committed
> Status in maas-images:
> Invalid
>
> Bug description:
> We are seeing failure to enlist nodes in our lab. While booting,
> system drops to initramfs shell.
>
> These are with ephemeral daily images and
> 1.8.0~beta8+bzr3951-0ubuntu1~trusty1, but we saw this with earlier
> beta versions (2 or 3).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1459865/+subscriptions
>

Revision history for this message
Larry Michel (lmic) wrote :

Great. Thanks Narinder. Works for me too!

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.