Merge ~raharper/curtin:fix/curtainer-dont-wait-for-snap into curtin:master

[Ryan Harper]

tools/curtainer: do not wait for snapd.seeded.service

The curtainer tool is used in CI and other places to extract a certain
curtin source from the archive and then is used to pack/unpack the
curtin code in the correct environment. We have no need or use of
snapd in this container which currently blocks for longer than
curtainer waits for the system to boot due to waiting for snaps to
download and install.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:4cd1ea8cb8b1c0e0745f55c692f0976062ca1cac
https://jenkins.ubuntu.com/server/job/curtin-ci/163/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/163/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/163/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/163/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/163/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/163//rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

:-(

https://bugs.launchpad.net/cloud-images/+bug/1796137

This just works around containers that take too long to boot.

Perhaps maybe we move to using minimal image for this?

[Ryan Harper (raharper)]

> :-(
>
> https://bugs.launchpad.net/cloud-images/+bug/1796137

..:(╯°□°)╯彡┻━━┻

>
> This just works around containers that take too long to boot.

Yes. Yes it does.

> Perhaps maybe we move to using minimal image for this?

Already tried; out of the box, minimal does not work with
what we do in curtainer:

 - add-apt-repository is not present
 - unminimize performs the same long running snap stuff that you get in fat
   images
 - apt installing additional stuff (software-properties) in minimal isn not faster
   that just disabling the slow part of the fat image

This is the simplest fix that keeps curtainer working as it does without this
really long boot.

[Paride Legovini (paride)]

I wonder if there's a good reason for seeding the lxd snap in the lxd images given that nested lxd containers do not work, at least not with unprivileged containers.

I'm not against the proposed workaround, but avoiding the need for it would be of course better. What if we switch to minimal images and replace the call to add-apt-repository with dumping a couple of lines to a file in /etc/apt/sources.list.d/? This would make the CI runs even faster, as the minimal images are smaller. One tricky bit is that the ubuntu-minimal LXD remote is not available by default, so curtainer needs to assume a specific configuration is in place - not super nice. What do you think?

review: Needs Information

[Ryan Harper (raharper)]

@Paride

> One tricky bit is that the ubuntu-minimal LXD remote is not available by default, so curtainer needs to assume a specific configuration is in place

Why don't we combine both of these; disable seeded.service and if we don't have add-apt-repository, use sed to fix up repos. This way curtainer handles either ubuntu or minimal images.

[Ryan Harper (raharper)]

Ubuntu Minimal support here: https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/389431

Note that I don't believe there's a groovy minimal yet, so CI couldn't use minimal images for daily groovy testing AFAICT.

[Paride Legovini (paride)]

Thanks! The Groovy minimal images are here:

https://cloud-images.ubuntu.com/minimal/daily/groovy/

so I we can use them in CI.

I think there's a better way ignore the snapd.seeded service:

  lxc init ubuntu:focal test-f
  lxc file push /dev/null test-f/etc/systemd/system/snapd.seeded.service
  lxc start test-f

This will mask the service and prevent it from running at all, which is IMHO better than stopping it. For even better performance we could fully mask snapd.service instead, which also causes snapd.seeded.service to terminate immediately.

review: Needs Fixing

[Ryan Harper (raharper)]

> https://cloud-images.ubuntu.com/minimal/daily/groovy/

Yes, I added the releases, not daily.

lxc remote add --protocol simplestreams ubuntu-minimal-daily https://cloud-images.ubuntu.com/minimal/daily

> I think there's a better way ignore the snapd.seeded service:
>
> lxc init ubuntu:focal test-f
> lxc file push /dev/null test-f/etc/systemd/system/snapd.seeded.service
> lxc start test-f

Yes; that's nice

> For even better performance we could fully mask snapd.service instead, which
> also causes snapd.seeded.service to terminate immediately.gq

systemctl mask does exactly what you're saying (symlink)

You need to add --now to the systemctl mask to stop active units; but if
we use the pre-boot mask operation (your file push) that's not needed.

I'll update this PR with that.

The minimal feature is a separate PR now.

[Paride Legovini (paride)]

The "better performance" suggestion was to mask snapd.service instead of snapd.seeded.service.

[Ryan Harper (raharper)]

On Tue, Aug 18, 2020 at 9:41 AM Paride Legovini <
<email address hidden>> wrote:

> The "better performance" suggestion was to mask snapd.service instead of
> snapd.seeded.service.
>

I see. The long (really long) poll in the tent was the seeded operation
which may need to download
a new core snap or something. For this particular use-case (curtainer) I
don't think we need to do
a lot more.

As a general purpose (make the image boot faster; there's a long list of
things to mask/disable)
I may put up a gist with the details as the majority of our use does not
intersect snap at all.

--
> https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/389306
> You are the owner of ~raharper/curtin:fix/curtainer-dont-wait-for-snap.
>

ffe6408... by Ryan Harper on 2020-08-18

mask snapd.seeded.service before container starts

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ffe6408f7877383cec08848aa9ce19173c7fba6e
https://jenkins.ubuntu.com/server/job/curtin-ci/167/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/167/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/167/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/167/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/167/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/167//rebuild

review: Approve (continuous-integration)

[Paride Legovini (paride)]

Thanks!

review: Approve

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/230/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-autoland-test/nodes=metal-amd64/230/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-autoland-test/nodes=metal-arm64/230/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-autoland-test/nodes=metal-ppc64el/230/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-autoland-test/nodes=metal-s390x/230/

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

diff --git a/tools/curtainer b/tools/curtainer
index a6fdc1b..e469dcb 100755
--- a/tools/curtainer
+++ b/tools/curtainer
@@ -153,7 +153,13 @@ main() {
     fi
     getsource="${getsource%/}"
 
-    lxc launch "$src" "$name" || fail "failed lxc launch $src $name"
+    # launch container; mask snapd.seeded.service; not needed
+    {
+        lxc init "$src" "$name" &&
+        lxc file push \
+            /dev/null ${name}/etc/systemd/system/snapd.seeded.service &&
+        lxc start ${name}
+    } || fail "failed lxc launch $src $name"
     CONTAINER=$name
 
     wait_for_ready "$name" $maxwait $VERBOSITY ||