node powered off after reboot from rescue mode

Bug #1749210 reported by Scott Moser
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Lee Trager
2.3
Fix Released
Medium
Lee Trager

Bug Description

I acquired a node, and put it into rescue mode.
I used the system a bit, and then it hit a kernel crash.
I did a hard reset via ipmi, and expected the system to come back up into rescue mode.
Instead, the system booted and powered off.

On the console I saw:
[ 55.390845] cloud-init[2851]: Powering node off.

Ie, cloud-init was probably told to power off via power_state.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: maas 2.3.0-6434-gd354690-0ubuntu1~16.04.1
ProcVersionSignature: Ubuntu 4.4.0-112.135-generic 4.4.98
Uname: Linux 4.4.0-112-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
Date: Tue Feb 13 15:08:38 2018
JournalErrors:
 Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system.
       Users in the 'systemd-journal' group can see all messages. Pass -q to
       turn off this notice.
 No journal files were opened due to insufficient permissions.
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: maas
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Scott,

What was the state of the machine when it rebooted ?
When it rebooted, did it PXE boot again ?

Changed in maas:
status: New → Incomplete
Revision history for this message
Scott Moser (smoser) wrote :

Yes, it pxe booted, maas passed it IP address, rooturl....
i did not (could not) check, but i assume maas passed in its cloud-config 'power_state'.

Revision history for this message
Scott Moser (smoser) wrote :

attached is a console log of the system booting while it is in 'Rescue mode' on the web UI.

Changed in maas:
status: Incomplete → New
Revision history for this message
Scott Moser (smoser) wrote :

I've just now got a node back into rescue mode.
then just a simple 'reboot' from an ssh session and it will reboot, pxeboot, and then be power'd off.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Scott,

To clarify,

1. your machine was "allocated", not "deployed" right?
2. Only machines that are "deployed" should go back into booting from the disk.

no longer affects: maas (Ubuntu)
Changed in maas:
status: New → Incomplete
milestone: none → 2.4.0rc2
Revision history for this message
Scott Moser (smoser) wrote :

Please just try this:

a.) Go to a node's page
b.) Take action -> rescue mode
everything is happy, system goes into rescue mode like you'd expect.
c.) ssh to system, 'sudo reboot'
d.) instead of rebooting into rescue mode the system reboot, boot into ubuntu and power off.

At this point, the node page still says 'Rescue Mode'.

Note, that when I had first done this I 'acquired' the node first, but that does not seem necessary.

This maas is:
 MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1)

Changed in maas:
status: Incomplete → Confirmed
Changed in maas:
importance: Undecided → Medium
Changed in maas:
assignee: nobody → Lee Trager (ltrager)
Lee Trager (ltrager)
Changed in maas:
status: Confirmed → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.