k8s workload container with non-root user confuses k8s charm operator

Related to bug 1878329

When a "stateless" charm is upgraded, the k8s deployment resource terminates the current pod and creates a new one. Juju in turn creates a new unit to represent this new pod. The original unit agent though can still be running and it looks like it tries to initialise a terminating pod with the upgraded charm, causing an error deleting the tools directory.

We're still investigating exactly the causal relation between the moving parts as it's been hard to reproduce. Having the mattermost charm which demonstrates the issue would help.

The charm is at https://github.com/canonical-is/charm-mattermost-k8s and the docker build is at https://github.com/canonical-is/mattermost-docker.

The issue here appears to be related to the docker image used for mattermost and a limitation of how juju is allowed to exec into the mattermost container.

There's this k8s bug

https://github.com/kubernetes/kubernetes/issues/30656

The mattermost docker image used by he charm sets up a non-root default user. However, when juju execs into the container as a result of upgrade charm, in order to unpack the new charm so actions are updated, the non-root user does not have permission to access the charm files, hence the permission error.

The only solution at the moment is to not have a default non-root user in the workload container.

The are options to work around this, eg we could provision a juju side car in addition to the init container; the side car would have root and could update the shared volume with the charm files as instructed by the charm operator.

Punting for now due to design flux.

Hello,

Got affected by this as well: see https://pastebin.ubuntu.com/p/jzzbscz5Hq/ for details.

I also use a docker image without a root user (as everyone should really!).

This is fixed in juju 2.8.2

https://github.com/juju/juju/pull/11840