Unable to Deploy Focal with MAAS 2.6.1

Attached are the full curtin logs from the failed deploy

Here is the maas-curtin-config for the system

ubuntu@maas2:~$ maas sfeole machine get-curtin-config fgkgge
**********************************************************************
*** WARNING! The API on the server differs from the description that
*** is cached locally. This may result in failed API calls. Refresh
*** the local API description with `maas refresh`.
**********************************************************************
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  proxy: http://10.246.72.3:8000/
  sources_list: 'deb http://archive.ubuntu.com/ubuntu $RELEASE main multiverse restricted
    universe

    # deb-src http://archive.ubuntu.com/ubuntu $RELEASE main multiverse restricted
    universe

    deb http://archive.ubuntu.com/ubuntu $RELEASE-updates main multiverse restricted
    universe

    # deb-src http://archive.ubuntu.com/ubuntu $RELEASE-updates main multiverse restricted
    universe

    deb http://archive.ubuntu.com/ubuntu $RELEASE-security main multiverse restricted
    universe

    # deb-src http://archive.ubuntu.com/ubuntu $RELEASE-security main multiverse restricted
    universe

    deb http://archive.ubuntu.com/ubuntu $RELEASE-backports main multiverse restricted
    universe

    # deb-src http://archive.ubuntu.com/ubuntu $RELEASE-backports main multiverse
    restricted universe

    '
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n MAAS: {consumer_key: sVenNQPLGgV73bcDT5,\
      \ metadata_url: 'http://localhost:5240/MAAS/metadata/',\n token_key: SjHAwDWRHUVwBUhAN4,\
      \ token_secret: szFL5NFE8kkUcYy5wyumkkaB36yH4keX}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-reporting:
    content: "#cloud-config\nreporting:\n maas: {consumer_key: sVenNQPLGgV73bcDT5,\
      \ endpoint: 'http://localhost:5240/MAAS/metadata/status/fgkgge',\n token_key:\
      \ SjHAwDWRHUVwBUhAN4, token_secret: szFL5NFE8kkUcYy5wyumkkaB36yH4keX,\n type:\
      \ webhook}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_init_reporting.cfg
  maas-ubuntu-sso:
    content: '#cloud-config

      snappy: {email: <email address hidden>}

      '
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  grub2: grub2 grub2/update_nvram boolean false
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

    cloud-init cloud-init/maas-metadata-url string http://localhost:5240/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_consumer_key=sVenNQPLGgV73bcDT5&oauth_token_key=SjHAwDWRHUVwBUhAN4&oauth_token_secret=szFL5NFE8kkUcYy5wyumkkaB36yH4keX

    cloud-init cloud-init/local-cloud-config string manage_etc_hosts: true\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: sVenNQPLGgV73bcDT5, endpoint: ''http://localhost:5240/MAAS/metadata/status/fgkgge'',\n token_key:
    SjHAwDWRHUVwBUhAN4, token_secret: szFL5NFE8kkUcYy5wyumkkaB36yH4keX,\n type:
    webhook}\n

    '
early_commands:
  driver_00:
  - sh
  - -c
  - echo third party drivers not installed or ...

Thanks for the bug Sean

As you said in your description:
"linux-signed-* packages were transitional and were removed in the 5.4 upload therefore curtin should not be trying to install them."

Your curtin config that you attached from maas for those nodes is specifically requesting that linux-signed-generic kernel package gets installed on that machine due to the curtin config it provides to the node:

kernel:
  mapping: {}
  package: linux-signed-generic

So, something in MAAS, or in /etc/maas/preseeds is sending that config to curtin which curtin fails on because that package doesn't exist.

Sean, can you clarify in the description what "removed in the 5.4 upload" means? I'm missing the context there.

Do you just mean just the 5.4 kernel revision doesn't publish a linux-signed-generic transitional package anymore, or were you referring to a specific event that just happened in focal.

I meant that the 5.4 kernel revision doesn't publish a linux-signed-generic transitional package anymore, this took affect early Dec i believe.

> "linux-signed-* packages were transitional and were removed in the 5.4"

Shame we only find this out after it's done.

IRC chat with vorlon confirms that since 2019, linux-image-generic *will* have the signed bits already, this is change back through Trusty (precise does not have the right kernel bits to do this); so pending confirmation with Kernel team, but MAAS/Curtin should just use linux-image-generic.

Things to do

1) determine how long linux-signed-generic metapackage will live in previous releases (it's already gone in focal)
2) determine if other kernel varients (linux-image-lowlatency, etc) will also be signed

Curtin can change to installing linux-image-generic on focal and newer releases (in places where it would normally have used linux-signed-generic). MAAS will need to do the same if it's providing kernel-mappings (like it was in this bug).

Awaiting some details from Kernel team before we can move forward for certain.

We are only removing transitional packages in the new lts. Older releases have not changed. I believe these have been transitional for a long time so we should not be taken by surprise. Finally it is not clear how we would know you are referencing specific packages directly rather than using the seed references; to know there was a need to inform you. As you are not in the archive there is no protection from the archive to inform us of the link.

I think we confirmed in IRC #curtin that curtin's curthooks doesn't attempt to install anything other than linux-generic(-lts-*). So I'm not certain there is anything for curtin to do here. Minimally MAAS needs to know to to tell curtin to install via:

kernel:
    package: linux-signed-generic.

The fallback pkg that curtin installs if no "package: " is provided is either:
- linux-generic
OR

FLAVOR=`uname -r | cut -d '-' -f3`
- linux-$FLAVOR(-lts-*)?

> Finally it is not clear how we would know you are referencing
> specific packages directly rather than using the seed references;
> to know there was a need to inform you. As you are not in the
> archive there is no protection from the archive to inform us of the link.

Sure, I understand there wasn't a direct connection to us. Typically
ubuntu-devel or similar lists announce user-facing changes (like
archive package name changes, transitions); so I was just wondering if
we're not listening in the right place.

At this point have what other information is needed from the kernel team to get this moving along and fixed??

Has enough information been exchanged to possibly come up with an action plan for today?

No new information is needed. Two actions can be taken:

1) MAAS needs to fix the kernel it tells curtin to install, for focal, it should send linux-image-generic (rather than specifying 'signed')

2) You could ask if the kernel team would re-add the meta-package (linux-signed-generic) until (1) is fixed.

You may be able to workaround (1) by adding custom curtin config using the config:

kernel:
   package: linux-image-generic

When lp:maas-images creates the Ubuntu image stream it includes the kernel package name. When lp:maas imports the images it saves the kernel name in the database to pass to Curtin. No changes are needed for lp:maas, only lp:maas-images.

I've posted a branch to fix this. I'm just confirming with the kernel team that we can drop "-signed" from 14.04+.

Installs with the 20200116 focal image are working now. I think this can be closed.