Bug #1680197 “Zesty deployments failing sporadically” : Bugs : MAAS

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-05:

#1

/var/log/maas directory tree from the server Edit (10.7 MiB, application/x-tar)

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2017-04-05:

#2

This seems like a curtin bug (and may even be an issue with zesty itself).

Changed in maas:
status:	New → Incomplete
status:	Incomplete → Invalid

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2017-04-05:

#3

Hi Rod,

- Can you please provide the curtin version that you are using? On the maas server: dpkg -l | grep curtin
- That said, provide the full installation log from the UI. or via (maas <user> node-results read system_id=<your_system> result_type=1 | grep "\"data\"" | cut -d"\"" -f4 | base64 --decode)
- And please provide the config: maas <user> machine get-curtin-config <system_id_of_machine>

Revision history for this message

Ryan Harper (raharper) wrote on 2017-04-05:

#4

The lack of exclusive access on open suggests that the device has been claimed by some other block service (Like md or lvm, or bcache).

We'll need the curtin storage configuration as Andres suggests to help understand the underlying problem.

here's the original bug that lead to curtin testing for exclusive device
access when asked to wipe a device.

https://bugs.launchpad.net/curtin/+bug/1659509

Changed in curtin:
status:	New → Incomplete

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-05:

#5

Download full text (5.7 KiB)

Curtin version information:

$ dpkg -l | grep curtin
ii curtin-common 0.1.0~bzr470-0ubuntu1~16.04.1 all Library and tools for curtin installer
ii python3-curtin 0.1.0~bzr470-0ubuntu1~16.04.1 all Library and tools for curtin installer

Installation log:

$ maas admin node-results read system_id=xr36pd result_type=1 | grep "\"data\"" | cut -d"\"" -f4 | base64 --decode
curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
Failed to exclusively open path: /dev/sda1
Traceback (most recent call last):
  File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
    fd = os.open(path, os.O_RDWR | os.O_EXCL)
OSError: [Errno 16] Device or resource busy: '/dev/sda1'
Device holders with exclusive access: []
Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']
[Errno 16] Device or resource busy: '/dev/sda1'
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: Failed to exclusively open path: /dev/sda1
        Traceback (most recent call last):
          File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
            fd = os.open(path, os.O_RDWR | os.O_EXCL)
        OSError: [Errno 16] Device or resource busy: '/dev/sda1'
        Device holders with exclusive access: []
        Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']
        [Errno 16] Device or resource busy: '/dev/sda1'

Stderr: ''

Curtin machine config:

$ maas admin machine get-curtin-config xr36pd
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  primary:
  - arches:
    - default
    uri: http://nessus.rodsbooks.com/ubuntu/
  proxy: http://172.24.124.1:8000/
  security:
  - arches:
    - default
    uri: http://nessus.rodsbooks.com/ubuntu/
debconf_selections:
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

cloud-init cloud-init/maas-metadata-url string http://172.24.124.1:5240/MAAS/metadata/

cloud-init cloud-init/maas-metadata-credentials string oauth_consumer_key=EUDbzueWEvEUz3XsUS&oauth_token_key=Y3MUmumkezRCgtTM93&oauth_token_secret=zuseqQu9bvTHSBJN3wwDphFskTpff2FU

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://nessus.rodsbooks.com/ubuntu/\n proxy:
    http://172.24.124.1:8000/\n security:\n - arches: [default]\n uri: http://nessus.rodsbooks.com/ubuntu/\napt_preserve_sources_list:
    true\napt_proxy: http://172.24.124.1:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: EUDbzueWEvEUz3XsUS, endpoint: ''http://172.24.124.1:5240/MAAS/metadata/status/xr36pd'',\n token_key:
    Y3MUmumkezRCgtTM93, token_secret: zuseqQu9bvTHSBJN3wwDphFskTpff2FU,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
...

Curtin version information:

$ dpkg -l | grep curtin
ii  curtin-common                              0.1.0~bzr470-0ubuntu1~16.04.1                 all          Library and tools for curtin installer
ii  python3-curtin                             0.1.0~bzr470-0ubuntu1~16.04.1                 all          Library and tools for curtin installer

Installation log:

$ maas admin node-results read system_id=xr36pd result_type=1 | grep "\"data\"" | cut -d"\"" -f4 | base64 --decode
curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
Failed to exclusively open path: /dev/sda1
Traceback (most recent call last):
  File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
    fd = os.open(path, os.O_RDWR | os.O_EXCL)
OSError: [Errno 16] Device or resource busy: '/dev/sda1'
Device holders with exclusive access: []
Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']
[Errno 16] Device or resource busy: '/dev/sda1'
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: Failed to exclusively open path: /dev/sda1
        Traceback (most recent call last):
          File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
            fd = os.open(path, os.O_RDWR | os.O_EXCL)
        OSError: [Errno 16] Device or resource busy: '/dev/sda1'
        Device holders with exclusive access: []
        Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']
        [Errno 16] Device or resource busy: '/dev/sda1'
        
Stderr: ''

Curtin machine config:

$ maas admin machine get-curtin-config xr36pd
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  primary:
  - arches:
    - default
    uri: http://nessus.rodsbooks.com/ubuntu/
  proxy: http://172.24.124.1:8000/
  security:
  - arches:
    - default
    uri: http://nessus.rodsbooks.com/ubuntu/
debconf_selections:
  maas: 'cloud-init   cloud-init/datasources  multiselect MAAS

cloud-init   cloud-init/maas-metadata-url  string http://172.24.124.1:5240/MAAS/metadata/

cloud-init   cloud-init/maas-metadata-credentials  string oauth_consumer_key=EUDbzueWEvEUz3XsUS&oauth_token_key=Y3MUmumkezRCgtTM93&oauth_token_secret=zuseqQu9bvTHSBJN3wwDphFskTpff2FU

cloud-init   cloud-init/local-cloud-config  string apt:\n  preserve_sources_list:
    false\n  primary:\n  - arches: [default]\n    uri: http://nessus.rodsbooks.com/ubuntu/\n  proxy:
    http://172.24.124.1:8000/\n  security:\n  - arches: [default]\n    uri: http://nessus.rodsbooks.com/ubuntu/\napt_preserve_sources_list:
    true\napt_proxy: http://172.24.124.1:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n  maas: {consumer_key: EUDbzueWEvEUz3XsUS, endpoint: ''http://172.24.124.1:5240/MAAS/metadata/status/xr36pd'',\n    token_key:
    Y3MUmumkezRCgtTM93, token_secret: zuseqQu9bvTHSBJN3wwDphFskTpff2FU,\n    type:
    webhook}\nsystem_info:\n  package_mirrors:\n  - arches: [i386, amd64]\n    failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n    search:\n      primary:
    [''http://nessus.rodsbooks.com/ubuntu/'']\n      security: [''http://nessus.rodsbooks.com/ubuntu/'']\n  -
    arches: [default]\n    failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n    search:\n      primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n      security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

'
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}
  package: linux-generic
late_commands:
  maas:
  - wget
  - --no-proxy
  - http://172.24.124.1:5240/MAAS/metadata/latest/by-id/xr36pd/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: enp0s25
    mac_address: f0:92:1c:5e:1e:70
    mtu: 1500
    name: enp0s25
    subnets:
    - type: dhcp4
    type: physical
  - address:
    - 172.24.124.1
    search:
    - maas
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  - custom
partitioning_commands:
  builtin:
  - curtin
  - block-meta
  - custom
reporting:
  maas:
    consumer_key: EUDbzueWEvEUz3XsUS
    endpoint: http://172.24.124.1:5240/MAAS/metadata/status/xr36pd
    token_key: Y3MUmumkezRCgtTM93
    token_secret: zuseqQu9bvTHSBJN3wwDphFskTpff2FU
    type: webhook
storage:
  config:
  - grub_device: true
    id: sda
    model: FUJITSU MHV2060B
    name: sda
    ptable: gpt
    serial: NW84T6A2A9LH
    type: disk
    wipe: superblock
  - device: sda
    flag: boot
    id: sda-part1
    name: sda-part1
    number: 1
    offset: 4194304B
    size: 536870912B
    type: partition
    uuid: 2be68d8b-7e2c-44f1-b8b6-a0b80d076ea3
    wipe: superblock
  - device: sda
    id: sda-part2
    name: sda-part2
    number: 2
    size: 59466842112B
    type: partition
    uuid: b3a1130f-93d7-43a1-a9a6-a558a1e19985
    wipe: superblock
  - fstype: fat32
    id: sda-part1_format
    label: efi
    type: format
    uuid: 80bd9766-a8a9-49fb-943d-9da8c588b5ba
    volume: sda-part1
  - fstype: ext4
    id: sda-part2_format
    label: root
    type: format
    uuid: 49ec0137-d0bd-42eb-bb2e-858a4922e5f8
    volume: sda-part2
  - device: sda-part2_format
    id: sda-part2_mount
    path: /
    type: mount
  - device: sda-part1_format
    id: sda-part1_mount
    path: /boot/efi
    type: mount
  version: 1

Most of these failures have been on EFI machines, but at least one was on a BIOS machine. They're all configured for flat partitions, not LVM or bcache. The above output is for a system on my home network for which I'm using a local mirror, but I've seen failures on a MAAS server in 1SS that uses us.archive.ubuntu.com.

Revision history for this message

Ryan Harper (raharper) wrote on 2017-04-05: Re: [Bug 1680197] Re: Zesty deployments failing sporadically

#6

Download full text (6.9 KiB)

Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0
0']

That shouldn't be mounted during deployment; the ephemeral image that the
target machine netboots shouldn't mount anything at all.

Can you also include the full boot log (and if possible getting
/var/log/cloud-init* from
the ephemeral env would be helpful to see if somehow cloud-init mounted
this device).

Note, media/root-ro looks a *lot* like a custom kernel command line with
overlay root settings.

On Wed, Apr 5, 2017 at 2:07 PM, Rod Smith <email address hidden> wrote:

> Curtin version information:
>
> $ dpkg -l | grep curtin
> ii curtin-common
> 0.1.0~bzr470-0ubuntu1~16.04.1 all Library and
> tools for curtin installer
> ii python3-curtin
> 0.1.0~bzr470-0ubuntu1~16.04.1 all Library and
> tools for curtin installer
>
> Installation log:
>
> $ maas admin node-results read system_id=xr36pd result_type=1 | grep
> "\"data\"" | cut -d"\"" -f4 | base64 --decode
> curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
> Failed to exclusively open path: /dev/sda1
> Traceback (most recent call last):
> File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
> fd = os.open(path, os.O_RDWR | os.O_EXCL)
> OSError: [Errno 16] Device or resource busy: '/dev/sda1'
> Device holders with exclusive access: []
> Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0
> 0']
> [Errno 16] Device or resource busy: '/dev/sda1'
> curtin: Installation failed with exception: Unexpected error while running
> command.
> Command: ['curtin', 'block-meta', 'custom']
> Exit code: 3
> Reason: -
> Stdout: Failed to exclusively open path: /dev/sda1
> Traceback (most recent call last):
> File "/curtin/curtin/block/__init__.py", line 773, in
> exclusive_open
> fd = os.open(path, os.O_RDWR | os.O_EXCL)
> OSError: [Errno 16] Device or resource busy: '/dev/sda1'
> Device holders with exclusive access: []
> Device mounts: ['/dev/sda1 /media/root-ro ext4
> ro,relatime,data=ordered 0 0']
> [Errno 16] Device or resource busy: '/dev/sda1'
>
> Stderr: ''
>
> Curtin machine config:
>
> $ maas admin machine get-curtin-config xr36pd
> Success.
> Machine-readable output follows:
> apt:
> preserve_sources_list: false
> primary:
> - arches:
> - default
> uri: http://nessus.rodsbooks.com/ubuntu/
> proxy: http://172.24.124.1:8000/
> security:
> - arches:
> - default
> uri: http://nessus.rodsbooks.com/ubuntu/
> debconf_selections:
> maas: 'cloud-init cloud-init/datasources multiselect MAAS
>
> cloud-init cloud-init/maas-metadata-url string
> http://172.24.124.1:5240/MAAS/metadata/
>
> cloud-init cloud-init/maas-metadata-credentials string
> oauth_consumer_key=EUDbzueWEvEUz3XsUS&oauth_token_key=Y3MUmumkezRCgtTM93&
> oauth_token_secret=zuseqQu9bvTHSBJN3wwDphFskTpff2FU
>
> cloud-init cloud-init/local-cloud-config string apt:\n
> preserve_sources_list:
> false\n primary:\n - arches: [default]\n uri:
> http://nessus.rodsbooks.com/ubuntu/\n proxy:
> http://172.24.124.1:8000/\n security:\n - a...

Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0
0']

That shouldn't be mounted during deployment; the ephemeral image that the
target machine netboots shouldn't mount anything at all.

Can you also include the full boot log (and if possible getting
/var/log/cloud-init* from
the ephemeral env would be helpful to see if somehow cloud-init mounted
this device).

Note, media/root-ro looks a *lot* like a custom kernel command line with
overlay root settings.

On Wed, Apr 5, 2017 at 2:07 PM, Rod Smith <rod.smith@canonical.com> wrote:

> Curtin version information:
>
> $ dpkg -l | grep curtin
> ii  curtin-common
> 0.1.0~bzr470-0ubuntu1~16.04.1                 all          Library and
> tools for curtin installer
> ii  python3-curtin
>  0.1.0~bzr470-0ubuntu1~16.04.1                 all          Library and
> tools for curtin installer
>
> Installation log:
>
> $ maas admin node-results read system_id=xr36pd result_type=1 | grep
> "\"data\"" | cut -d"\"" -f4 | base64 --decode
> curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
> Failed to exclusively open path: /dev/sda1
> Traceback (most recent call last):
>   File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
>     fd = os.open(path, os.O_RDWR | os.O_EXCL)
> OSError: [Errno 16] Device or resource busy: '/dev/sda1'
> Device holders with exclusive access: []
> Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0
> 0']
> [Errno 16] Device or resource busy: '/dev/sda1'
> curtin: Installation failed with exception: Unexpected error while running
> command.
> Command: ['curtin', 'block-meta', 'custom']
> Exit code: 3
> Reason: -
> Stdout: Failed to exclusively open path: /dev/sda1
>         Traceback (most recent call last):
>           File "/curtin/curtin/block/__init__.py", line 773, in
> exclusive_open
>             fd = os.open(path, os.O_RDWR | os.O_EXCL)
>         OSError: [Errno 16] Device or resource busy: '/dev/sda1'
>         Device holders with exclusive access: []
>         Device mounts: ['/dev/sda1 /media/root-ro ext4
> ro,relatime,data=ordered 0 0']
>         [Errno 16] Device or resource busy: '/dev/sda1'
>
> Stderr: ''
>
> Curtin machine config:
>
> $ maas admin machine get-curtin-config xr36pd
> Success.
> Machine-readable output follows:
> apt:
>   preserve_sources_list: false
>   primary:
>   - arches:
>     - default
>     uri: http://nessus.rodsbooks.com/ubuntu/
>   proxy: http://172.24.124.1:8000/
>   security:
>   - arches:
>     - default
>     uri: http://nessus.rodsbooks.com/ubuntu/
> debconf_selections:
>   maas: 'cloud-init   cloud-init/datasources  multiselect MAAS
>
>     cloud-init   cloud-init/maas-metadata-url  string
> http://172.24.124.1:5240/MAAS/metadata/
>
>     cloud-init   cloud-init/maas-metadata-credentials  string
> oauth_consumer_key=EUDbzueWEvEUz3XsUS&oauth_token_key=Y3MUmumkezRCgtTM93&
> oauth_token_secret=zuseqQu9bvTHSBJN3wwDphFskTpff2FU
>
>     cloud-init   cloud-init/local-cloud-config  string apt:\n
> preserve_sources_list:
>     false\n  primary:\n  - arches: [default]\n    uri:
> http://nessus.rodsbooks.com/ubuntu/\n  proxy:
>     http://172.24.124.1:8000/\n  security:\n  - arches: [default]\n
> uri: http://nessus.rodsbooks.com/ubuntu/\napt_preserve_sources_list:
>     true\napt_proxy: http://172.24.124.1:8000/\nmanage_etc_hosts:
> false\nmanual_cache_clean:
>     true\nreporting:\n  maas: {consumer_key: EUDbzueWEvEUz3XsUS, endpoint:
> ''http://172.24.124.1:5240/MAAS/metadata/status/xr36pd'',\n    token_key:
>     Y3MUmumkezRCgtTM93, token_secret: zuseqQu9bvTHSBJN3wwDphFskTpff2FU,\n
>   type:
>     webhook}\nsystem_info:\n  package_mirrors:\n  - arches: [i386,
> amd64]\n    failsafe:
>     {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''
> http://security.ubuntu.com/ubuntu''}\n    search:\n      primary:
>     [''http://nessus.rodsbooks.com/ubuntu/'']\n      security: [''
> http://nessus.rodsbooks.com/ubuntu/'']\n  -
>     arches: [default]\n    failsafe: {primary: ''http://ports.ubuntu.com/
> ubuntu-ports'',
>     security: ''http://ports.ubuntu.com/ubuntu-ports''}\n    search:\n
>   primary:
>     [''http://ports.ubuntu.com/ubuntu-ports'']\n      security: [''
> http://ports.ubuntu.com/ubuntu-ports'']\n
>
>     '
> install:
>   log_file: /tmp/install.log
>   post_files:
>   - /tmp/install.log
> kernel:
>   mapping: {}
>   package: linux-generic
> late_commands:
>   maas:
>   - wget
>   - --no-proxy
>   - http://172.24.124.1:5240/MAAS/metadata/latest/by-id/xr36pd/
>   - --post-data
>   - op=netboot_off
>   - -O
>   - /dev/null
> network:
>   config:
>   - id: enp0s25
>     mac_address: f0:92:1c:5e:1e:70
>     mtu: 1500
>     name: enp0s25
>     subnets:
>     - type: dhcp4
>     type: physical
>   - address:
>     - 172.24.124.1
>     search:
>     - maas
>     type: nameserver
>   version: 1
> network_commands:
>   builtin:
>   - curtin
>   - net-meta
>   - custom
> partitioning_commands:
>   builtin:
>   - curtin
>   - block-meta
>   - custom
> reporting:
>   maas:
>     consumer_key: EUDbzueWEvEUz3XsUS
>     endpoint: http://172.24.124.1:5240/MAAS/metadata/status/xr36pd
>     token_key: Y3MUmumkezRCgtTM93
>     token_secret: zuseqQu9bvTHSBJN3wwDphFskTpff2FU
>     type: webhook
> storage:
>   config:
>   - grub_device: true
>     id: sda
>     model: FUJITSU MHV2060B
>     name: sda
>     ptable: gpt
>     serial: NW84T6A2A9LH
>     type: disk
>     wipe: superblock
>   - device: sda
>     flag: boot
>     id: sda-part1
>     name: sda-part1
>     number: 1
>     offset: 4194304B
>     size: 536870912B
>     type: partition
>     uuid: 2be68d8b-7e2c-44f1-b8b6-a0b80d076ea3
>     wipe: superblock
>   - device: sda
>     id: sda-part2
>     name: sda-part2
>     number: 2
>     size: 59466842112B
>     type: partition
>     uuid: b3a1130f-93d7-43a1-a9a6-a558a1e19985
>     wipe: superblock
>   - fstype: fat32
>     id: sda-part1_format
>     label: efi
>     type: format
>     uuid: 80bd9766-a8a9-49fb-943d-9da8c588b5ba
>     volume: sda-part1
>   - fstype: ext4
>     id: sda-part2_format
>     label: root
>     type: format
>     uuid: 49ec0137-d0bd-42eb-bb2e-858a4922e5f8
>     volume: sda-part2
>   - device: sda-part2_format
>     id: sda-part2_mount
>     path: /
>     type: mount
>   - device: sda-part1_format
>     id: sda-part1_mount
>     path: /boot/efi
>     type: mount
>   version: 1
>
> Most of these failures have been on EFI machines, but at least one was
> on a BIOS machine. They're all configured for flat partitions, not LVM
> or bcache. The above output is for a system on my home network for which
> I'm using a local mirror, but I've seen failures on a MAAS server in 1SS
> that uses us.archive.ubuntu.com.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1680197
>
> Title:
>   Zesty deployments failing sporadically
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1680197/+subscriptions
>

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-05:

#7

Can you provide me with pointers on how to get the information you want? I know there used to be some hackish way to gain access to MAAS ephemerals, but that was several versions ago and I don't know what the current method is.

Revision history for this message

Ryan Harper (raharper) wrote on 2017-04-05:

#8

On Wed, Apr 5, 2017 at 3:06 PM, Rod Smith <email address hidden> wrote:

> Can you provide me with pointers on how to get the information you want?
> I know there used to be some hackish way to gain access to MAAS
> ephemerals, but that was several versions ago and I don't know what the
> current method is.
>

https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b

If possible, Accessing during deployment, or stopping shutdown should
let you ssh in to examine the system.

In particular, the cloud-init.log (which actually shoud be available via
maas cli, you pasted one earlier where you base64 dumped and cut.

maas admin node-results read system_id=xr36pd result_type=1

decoding the whole blob should have the entire cloud-init log.

--
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1680197
>
> Title:
> Zesty deployments failing sporadically
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1680197/+subscriptions
>

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2017-04-06:

#9

Rod,

While the machine is deploying, ssh into it and run:

sudo touch /tmp/block-reboot

That will prevent the machine from being rebooted.

On Wed, Apr 5, 2017 at 4:06 PM, Rod Smith <email address hidden> wrote:

> Can you provide me with pointers on how to get the information you want?
> I know there used to be some hackish way to gain access to MAAS
> ephemerals, but that was several versions ago and I don't know what the
> current method is.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1680197
>
> Title:
> Zesty deployments failing sporadically
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1680197/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-06:

#10

/var/log from ephemeral failed deployment Edit (116.3 KiB, application/x-tar)

Here's the entire /var/log directory tree from the ephemeral on a failed deployment.

Revision history for this message

Rod Smith (rodsmith) wrote on 2017-04-06:

#11

FWIW, I've discovered that some nodes seem to be more susceptible than others to this bug. I've encountered it on at least four nodes, but some fail only once every several deployments, whereas others succeed in deploying only once every several tries (or maybe never). The worst seems to be an HP ProBook 6470b, which is an AMT-enabled laptop on my home network. (That's the system whose logs I've provided.) The next worse seems to be either a new Dell PowerEdge C6320p or an older Dell PowerEdge T610, both IPMI-enabled servers in 1SS. I've also encountered it, but with much less frequency, on an AMT-enabled Intel NUC on my home network. Of possible relevance: The NUC uses an SSD, whereas the other systems all have conventional spinning hard disks.

Revision history for this message

Ryan Harper (raharper) wrote on 2017-04-06:

#12

Something I'm looking further into is:

1) kernel command line using overlayroot=tmpfs

Kernel command line: BOOT_IMAGE=ubuntu/amd64/ga-17.04/zesty/daily/boot-kernel nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-ga-17.04-zesty-daily iscsi_target_ip=172.24.124.1 iscsi_target_port=3260 iscsi_initiator=kzanol ip=::::kzanol:BOOTIF ip6=off ro root=/dev/disk/by-path/ip-172.24.124.1:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-ga-17.04-zesty-daily-lun-1 overlayroot=tmpfs cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http://172.24.124.1:5240/MAAS/metadata/latest/by-id/xr36pd/?op=get_preseed log_host=172.24.124.1 log_port=514 BOOTIF=01-f0:92:1c:5e:1e:70

2) issues with systemd mount, likely triggered after partitioning

Apr 6 13:26:53 kzanol cloud-init[2457]: curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
Apr 6 13:26:57 kzanol kernel: [ 81.464816] sda: sda1 sda15
Apr 6 13:26:57 kzanol systemd[1]: Stopped target Local File Systems.
Apr 6 13:26:57 kzanol systemd[1]: Unmounting /media/root-ro...
Apr 6 13:26:57 kzanol systemd[1]: Unmounted /media/root-ro.
Apr 6 13:26:58 kzanol systemd[1]: Mounting /media/root-ro...
Apr 6 13:26:58 kzanol kernel: [ 82.109869] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Apr 6 13:26:58 kzanol systemd[1]: Mounted /media/root-ro.
Apr 6 13:26:58 kzanol cloud-init[2457]: Failed to exclusively open path: /dev/sda1
Apr 6 13:26:58 kzanol cloud-init[2457]: Traceback (most recent call last):
Apr 6 13:26:58 kzanol cloud-init[2457]: File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
Apr 6 13:26:58 kzanol cloud-init[2457]: fd = os.open(path, os.O_RDWR | os.O_EXCL)
Apr 6 13:26:58 kzanol cloud-init[2457]: OSError: [Errno 16] Device or resource busy: '/dev/sda1'
Apr 6 13:26:58 kzanol cloud-init[2457]: Device holders with exclusive access: []
Apr 6 13:26:58 kzanol cloud-init[2457]: Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']

It's likely that curtin is racying with systemd.mount files/entries related to initramfs configuring overlayroot.

Some details for that here:
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573

Something I'm looking further into is:

1) kernel command line using overlayroot=tmpfs

Kernel command line: BOOT_IMAGE=ubuntu/amd64/ga-17.04/zesty/daily/boot-kernel nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-ga-17.04-zesty-daily iscsi_target_ip=172.24.124.1 iscsi_target_port=3260 iscsi_initiator=kzanol ip=::::kzanol:BOOTIF ip6=off ro root=/dev/disk/by-path/ip-172.24.124.1:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-ga-17.04-zesty-daily-lun-1 overlayroot=tmpfs cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http://172.24.124.1:5240/MAAS/metadata/latest/by-id/xr36pd/?op=get_preseed log_host=172.24.124.1 log_port=514 BOOTIF=01-f0:92:1c:5e:1e:70

2) issues with systemd mount, likely triggered after partitioning

Apr  6 13:26:53 kzanol cloud-init[2457]: curtin: Installation started. (0.1.0~bzr470-0ubuntu1~16.04.1)
Apr  6 13:26:57 kzanol kernel: [   81.464816]  sda: sda1 sda15
Apr  6 13:26:57 kzanol systemd[1]: Stopped target Local File Systems.
Apr  6 13:26:57 kzanol systemd[1]: Unmounting /media/root-ro...
Apr  6 13:26:57 kzanol systemd[1]: Unmounted /media/root-ro.
Apr  6 13:26:58 kzanol systemd[1]: Mounting /media/root-ro...
Apr  6 13:26:58 kzanol kernel: [   82.109869] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Apr  6 13:26:58 kzanol systemd[1]: Mounted /media/root-ro.
Apr  6 13:26:58 kzanol cloud-init[2457]: Failed to exclusively open path: /dev/sda1
Apr  6 13:26:58 kzanol cloud-init[2457]: Traceback (most recent call last):
Apr  6 13:26:58 kzanol cloud-init[2457]:   File "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
Apr  6 13:26:58 kzanol cloud-init[2457]:     fd = os.open(path, os.O_RDWR | os.O_EXCL)
Apr  6 13:26:58 kzanol cloud-init[2457]: OSError: [Errno 16] Device or resource busy: '/dev/sda1'
Apr  6 13:26:58 kzanol cloud-init[2457]: Device holders with exclusive access: []
Apr  6 13:26:58 kzanol cloud-init[2457]: Device mounts: ['/dev/sda1 /media/root-ro ext4 ro,relatime,data=ordered 0 0']

It's likely that curtin is racying with systemd.mount files/entries related to initramfs configuring overlayroot.

Some details for that here:
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573

Revision history for this message

Ryan Harper (raharper) wrote on 2017-04-10:

#13

Download full text (3.3 KiB)

OK,

We believe this is related to deploying in simple mode where curtin formats
the root filesystem and label's it
'cloudimg-rootfs' which matches the /etc/fstab entry in the ephemeral
environment; subsequently, systemd's fstab generator creates a mount unit
for "cloudimg-rootfs" to be mounted at /media/root-ro; This races with
curtin's attempt to wipe the device.

To confirm, this should reproduce 100% if you deploy with no custom storage
config, followed by a custom config.
Also, it would be interesting to see if that also reproduces with Xenial in
the above scenario; that may help narrow down
changes to systemd that make this more likely in zesty.

On Thu, Apr 6, 2017 at 10:18 AM, Ryan Harper <email address hidden>
wrote:

> Something I'm looking further into is:
>
> 1) kernel command line using overlayroot=tmpfs
>
> Kernel command line: BOOT_IMAGE=ubuntu/amd64/ga-17.04/zesty/daily/boot-
> kernel nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas
> :ephemeral-ubuntu-amd64-ga-17.04-zesty-daily
> iscsi_target_ip=172.24.124.1 iscsi_target_port=3260
> iscsi_initiator=kzanol ip=::::kzanol:BOOTIF ip6=off ro root=/dev/disk
> /by-path/ip-172.24.124.1:3260-iscsi-iqn.2004-05.com.ubuntu:maas
> :ephemeral-ubuntu-amd64-ga-17.04-zesty-daily-lun-1 overlayroot=tmpfs
> cc:{datasource_list: [MAAS]}end_cc cloud-config-
> url=http://172.24.124.1:5240/MAAS/metadata/latest/by-
> id/xr36pd/?op=get_preseed log_host=172.24.124.1 log_port=514
> BOOTIF=01-f0:92:1c:5e:1e:70
>
> 2) issues with systemd mount, likely triggered after partitioning
>
> Apr 6 13:26:53 kzanol cloud-init[2457]: curtin: Installation started.
> (0.1.0~bzr470-0ubuntu1~16.04.1)
> Apr 6 13:26:57 kzanol kernel: [ 81.464816] sda: sda1 sda15
> Apr 6 13:26:57 kzanol systemd[1]: Stopped target Local File Systems.
> Apr 6 13:26:57 kzanol systemd[1]: Unmounting /media/root-ro...
> Apr 6 13:26:57 kzanol systemd[1]: Unmounted /media/root-ro.
> Apr 6 13:26:58 kzanol systemd[1]: Mounting /media/root-ro...
> Apr 6 13:26:58 kzanol kernel: [ 82.109869] EXT4-fs (sda1): mounted
> filesystem with ordered data mode. Opts: (null)
> Apr 6 13:26:58 kzanol systemd[1]: Mounted /media/root-ro.
> Apr 6 13:26:58 kzanol cloud-init[2457]: Failed to exclusively open path:
> /dev/sda1
> Apr 6 13:26:58 kzanol cloud-init[2457]: Traceback (most recent call last):
> Apr 6 13:26:58 kzanol cloud-init[2457]: File
> "/curtin/curtin/block/__init__.py", line 773, in exclusive_open
> Apr 6 13:26:58 kzanol cloud-init[2457]: fd = os.open(path, os.O_RDWR
> | os.O_EXCL)
> Apr 6 13:26:58 kzanol cloud-init[2457]: OSError: [Errno 16] Device or
> resource busy: '/dev/sda1'
> Apr 6 13:26:58 kzanol cloud-init[2457]: Device holders with exclusive
> access: []
> Apr 6 13:26:58 kzanol cloud-init[2457]: Device mounts: ['/dev/sda1
> /media/root-ro ext4 ro,relatime,data=ordered 0 0']
>
> It's likely that curtin is racying with systemd.mount files/entries
> related to initramfs configuring overlayroot.
>
> Some details for that here:
> https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1666573
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad....

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Invalid	Undecided	Unassigned
	curtin	Fix Released	Undecided	Unassigned

MAAS

Zesty deployments failing sporadically

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches