Canonical Juju

shutting down: catacomb 0xc000309680 is dying

Bug #1984060 reported by Haw Loeung
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Medium
Unassigned

Bug Description

Hi,

Seeing lots of these:

| ubuntu@machine-0:~$ grep catacomb /var/log/juju/unit-ubuntu-repository-cache-0.log | tail -n 20
| 2022-08-08 17:37:50 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000d21200 is dying
| 2022-08-08 17:37:55 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000244900 is dying
| 2022-08-08 18:15:13 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000448d80 is dying
| 2022-08-08 18:36:54 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000f6cd80 is dying
| 2022-08-08 19:58:59 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000b3c900 is dying
| 2022-08-08 21:27:20 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000448d80 is dying
| 2022-08-08 22:29:03 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: (re)starting watcher: catacomb 0xc00196d200 is dying
| 2022-08-08 23:27:38 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000abe480 is dying
| 2022-08-08 23:31:38 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc0012d8480 is dying
| 2022-08-09 00:30:04 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000e32000 is dying
| 2022-08-09 01:39:18 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: (re)starting watcher: catacomb 0xc0013d1b00 is dying
| 2022-08-09 03:56:58 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc001bd6900 is dying
| 2022-08-09 06:13:23 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000e33b00 is dying
| 2022-08-09 07:13:13 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc001ef6480 is dying
| 2022-08-09 07:37:36 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000aed680 is dying
| 2022-08-09 08:06:49 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc00105cd80 is dying
| 2022-08-09 09:15:07 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000abfb00 is dying
| 2022-08-09 09:15:48 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: (re)starting watcher: catacomb 0xc00044ed80 is dying
| 2022-08-09 09:19:46 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000309680 is dying
| 2022-08-09 09:54:25 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc00228ed80 is dying
| ubuntu@machine-0:~$

Staging Azure environment, running Juju 2.9.32.

Revision history for this message
Juan M. Tirado (tiradojm) wrote :

Could you please check with 2.9.33. There were some fixes regarding the Azure provider.

Changed in juju:
status: New → Invalid
Revision history for this message
Haw Loeung (hloeung) wrote :

Okay, upgraded a set of controllers to 2.9.33. Will observe and report back if we're still seeing frequent "shutting down: catacomb 0xc00228ed80 is dying"

Revision history for this message
Haw Loeung (hloeung) wrote :

Sadly, we're still seeing this:

| $ jsft ubuntu-repository-cache | awk '/^[0-9]+/ { print $3 }' | xargs -P10 -I{} ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@{} 'grep "^$(date -I) .*shutting down: catacomb" /var/log/juju/unit-ubuntu-repository-cache-*.log' 2>/dev/null | sort
| 2022-08-16 03:31:41 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc00145c900 is dying
| 2022-08-16 03:31:41 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/3" shutting down: catacomb 0xc000d85b00 is dying
| 2022-08-16 03:35:59 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc00304a900 is dying
| 2022-08-16 03:35:59 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/1" shutting down: catacomb 0xc001198000 is dying
| 2022-08-16 03:35:59 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/2" shutting down: catacomb 0xc0004a9680 is dying
| 2022-08-16 03:35:59 INFO juju.worker.uniter uniter.go:313 unit "ubuntu-repository-cache/3" shutting down: catacomb 0xc0008b7b00 is dying
| 2022-08-16 06:18:12 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc0004ab680 is dying
| 2022-08-16 08:49:40 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/1" shutting down: catacomb 0xc0009af680 is dying
| 2022-08-16 09:24:33 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000de6d80 is dying
| 2022-08-16 09:32:31 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc000a12000 is dying
| 2022-08-16 15:43:09 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc0019a2900 is dying
| 2022-08-16 19:37:25 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/0" shutting down: catacomb 0xc0004aa000 is dying
| 2022-08-16 19:37:25 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/1" shutting down: catacomb 0xc0000c7b00 is dying
| 2022-08-16 21:40:23 INFO juju.worker.uniter uniter.go:310 unit "ubuntu-repository-cache/1" shutting down: catacomb 0xc000c7a000 is dying

The upgrade to 2.9.33 happened at 03:29 UTC. So the events from 03:36 onwards are those happening while we're at 2.9.33.

Changed in juju:
status: Invalid → New
Revision history for this message
Haw Loeung (hloeung) wrote :

Relating to LP:1977798, I think the root cause is this where agents are respawning and causing leadership re-elections.

Revision history for this message
Ian Booth (wallyworld) wrote :

Can we get all the logs, not just the unit log grepping for catacomb. There's not enough to go on.

Revision history for this message
Haw Loeung (hloeung) wrote :

Was waiting for you to ask. Controller logs provided out of band (the usual juju-controller-reports).

Revision history for this message
Joseph Phillips (manadart) wrote :

Can you link to said reports?

Changed in juju:
status: New → Incomplete
Revision history for this message
Haw Loeung (hloeung) wrote :

https://juju-controller-reports.admin.canonical.com/ (will also highlight on Mattermost with exactly which).

Changed in juju:
status: Incomplete → New
Joseph Phillips (manadart)
Changed in juju:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.