18.10+ cloud images have the LXD group as gid 1000

I have added two other packages to this bug.

livecd-rootfs - For the ubuntu-server and ubuntu-cpc projects a chroot hook should be added to create the lxd system group.

cloud-init - We have the suggestion that user creation should not initially create user groups that do not exist. You may consider adding a second pass in a final cloud-init module which would add groups for the default user which did not exist during the first pass (or create them if they still do not exist; and adding a mechanism for identifying system groups would be a good enhancement). This 2-stage approach would allow packages and snaps that get installed during boot (after cloud-init creates the default user) to create their own groups and manage them while ensuring the default user does get added to these.

I don't love the idea of two passes on adding users/groups.

https://stackoverflow.com/questions/57605982/how-to-make-a-custom-gid-in-groupadd-cloud-init
has an idea for another solution. sort of.

This is complicated.

The root cause, I believe, is lxd moving from a deb (which created the lxd system group when installed during image build) to a snap (which creates the group after/during first boot but, crucially, after cloud-init has created users and therefore the lxd group itself). As a result, the expected Ubuntu experience (`ubuntu` user is in the `lxd` system group when created) degraded. This is another symptom of snaps being poorly integrated into classic Ubuntu systems, I think.

My feeling is that this is really a snap-seeding-on-classic issue. The lxd snap should have a way of expressing "I want a lxd system group" and the seeding process should ensure that the system is configured in a way that won't cause these problems. There are some changes to the seeding process in the works, so I've added a snapd task to get input on whether that's at all realistic to hope for.

In the absence of snapd changes, from a cloud-init perspective this feels like an image misconfiguration to me. lxd is a core part of the Ubuntu Server experience that we provide[0], and not having a lxd system group in the image was not an intentional change, but an accident brought about by the move to a snap.

[0] IIRC, the minimal cloud images don't include lxd, so we should also consider what the desired behaviour there should be.

> We have the suggestion that user creation should not initially create user groups that do not exist.

Changing the default behaviour would be a non-starter for backport to stable releases[0], but we could consider doing this with an additional configuration option in the user specification (e.g. "create_if_missing: False"). This wouldn't result in the `ubuntu` user being in the `lxd` group by default, though, without additional 2-stage changes also proposed.

Alternatively, we could allow specifying that groups should be created as system groups (e.g. "system: True"). This would mean that the lxd system group would exist on systems where lxd isn't seeded (such as the minimal cloud images), which is less than ideal IMO[1].

Either of these options would require a fundamental modification of the schema for groups[2] (and cloud-init SRUs), so should not be considered a quick route to resolving this issue, IMO. And, further, it's not clear to me that they are actually _correct_ ways to resolve this.

Cheers,

Dan

[0] People could be happily relying on cloud-init to create user groups they specify for users today, and we don't want to break those people.
[1] As a security-conscious user launching a minimal cloud image and manually installing lxd later, I probably wouldn't expect any users to be able to interact with lxd by default.
[2] The current schema doesn't have support for any options for groups; they're just strings. Furthermore, dicts already have specific meaning in that code (group name -> list of group members). So we would need to do the investigation into the schema changes and the work to add support for options to groups, in addition to support for the specific options we want.

I wonder if adding something to cloud.cfg that ensures there is a lxd group created as a system-group would be the way forward? Especially since we already reference this group in cloud.cfg.

On Thu, Sep 26, 2019 at 05:27:22PM -0000, Michael Vogt wrote:
> I wonder if adding something to cloud.cfg that ensures there is a lxd
> group created as a system-group would be the way forward? Especially
> since we already reference this group in cloud.cfg.

For the specific case of a preseeded snap which we know should have an
associated group, I think we should handle this in livecd-rootfs and
statically create the group in the image since both are static aspects of
the image mastering. We CAN instruct cloud-init to create this as a system
group, but the point is that this is a layering violation; we should not
have to keep the cloud.cfg and the lxd snap in sync with regards to the
properties of the groups being created.

The reason I think that there should be extensions to cloud-init for this at
all is for the case where you have a *non* preseeded package which you want
to install as part of the instance config, and you want to add your user to
the group that will be created by that package. That could be either a deb
or a snap, doesn't matter. I don't think it smells good to have cloud-init
creating the group in this case.

Per Steve's comment, I'm going to move the cloud-init task to Invalid. I don't fully understand the "extensions" referenced in comment #5, but it seems clear to me that they should be tracked in a separate bug. I'll let someone who _does_ understand that ask file that bug. :)

https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/373789

This bug was fixed in the package livecd-rootfs - 2.618

---------------
livecd-rootfs (2.618) eoan; urgency=medium

  [ Michael Hudson-Doyle ]
  * Do not create a hook manually for ubuntu-cpc builds, make-hooks does not
    like that.

 -- Steve Langasek <email address hidden> Tue, 08 Oct 2019 21:21:45 -0700

This has now added
lxd:x:101:

Into ubuntu-base tarballs http://cdimage.ubuntu.com/ubuntu-base/daily/pending/ which is a bit problematic.

Was that intended?

core20 snap builds started to fail now, as it installs a few more packages on top of base, and expects to maintain stable gids

Currently the below gids were already in use

netdev:x:101:
crontab:x:102:
messagebus:x:103:
snappypkg:x:104:
ssh:x:105:
systemd-journal:x:106:
systemd-timesync:x:108:
systemd-network:x:109:
systemd-resolve:x:110:
systemd-bus-proxy:x:111:
kvm:x:112:
docker:x:113:
syslog:x:114:
pkcs11:x:115:
tss:x:116:
input:x:107:
render:x:117:

Imho ubuntu-base should not contain any gids that are not declared statically by base-files.

https://travis-ci.org/snapcore/core20/builds/596036023?utm_source=github_status&utm_medium=notification

note that the build scripts for core do have an md5sum check to detect changes to /etc/passwd|group|shadow because this file is orignally readonly.
dirs created with specific UIDs/GIDs by package postinist scripts that get copied into the writable area of the rootfs during first boot would fail to be owned by the daemon users the packages set up for them during a core update...

adding a new entry to one of the readonly password db files is fine but requires changes in livecd-rootfs and an update of the expected md5sum ...

additionally to the above, since /etc/group is readonly, you can not add users to the lxd group if you add lxd there so unprivileged containers on core will become impossible, the GID of lxd should be transferred into /var/lib/extrausers/group to make it possible to add a local system user to this group.

On Thu, 10 Oct 2019, 23:41 Dimitri John Ledkov, <email address hidden>
wrote:

> This has now added
> lxd:x:101:
>
> Into ubuntu-base tarballs http://cdimage.ubuntu.com/ubuntu-
> base/daily/pending/ which is a bit problematic.
>
> Was that intended?
>

I think it was sort of known, vs actively intended.

core20 snap builds started to fail now, as it installs a few more
> packages on top of base, and expects to maintain stable gids
>

Ok. Making this hook not run for ubuntu-base seems ok to me. Can we make
autopkgtests fail in this case for next time?

>
>

> core20 snap builds started to fail now, as it installs a
> few more packages on top of base, and expects to maintain stable gids

In principle, the snap build should follow the changes in behavior in livecd-rootfs, rather than blocking us from making any changes; it's a sanity check against accidental drift. As Oliver says:

> adding a new entry to one of the readonly password
> db files is fine but requires changes in livecd-rootfs
> and an update of the expected md5sum ...

So it needs a separate confirmation for the snap builds to work, but I don't see any reason we should special case a revert for the ubuntu-base build.

Closing the ubuntu snapd task as this bug is already on the project snapd

If snapd should change for this I think we need to have some design to discuss how this should look. AIUI there is a desire to express that some snaps needs groups created on the host (like lxd) when a snap gets installed. We could make this work, but even then with our current model where snaps are seeded during boot we would still need to make sure that cloud-init does things in the right order, i.e. first wait for seeding and then do the user creation.