Auto Package Testing

sudo systemctl restart autopkgtest.target loses currently running jobs with new kill test changes

Bug #2064582 reported by Tim Andersson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
Fix Committed
High
Tim Andersson

Bug Description

We can see (from a systemctl restart of worker services, by moi) that the recent changes which enable us to kill tests without them looping apparently can also cause the tests to be completely removed from the queue. This recent change requires more heuristics in order for this not to happen.

We need to investigate the behaviour of our signal handlers - currently I believe they're only fully functional before and after the autopkgtest process has run - whilst it's running, I've observed the signal handlers to not do their job whilst I was developing the test killing mechanism.

We also should investigate the behaviour of the autopkgtest exit code when running under a subprocess. I believe, from testing, that the autopkgtest subprocess for some reason inherits the code passed to a `kill` command.

We can leverage these two things to achieve the desired behaviour.

```
Apr 30 13:21:18 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98960]: INFO: Killing openstack server with uuid 0a2c0999-e137-4b00-b4f7-7873873b7afc
Apr 30 13:21:19 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98997]: WARNING: autopkgtest exited with code: 1
                                                                                                                       ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32376", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/72b2be8ffde36bca171c4744d37fc0a9f28343e3"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-20 07:34:36", "uuid": "657aabbd-4297-48db-bc9e-a809b0002a6c"}'
Apr 30 13:21:19 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98997]: INFO: Killing openstack server with uuid 657aabbd-4297-48db-bc9e-a809b0002a6c
Apr 30 13:21:20 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661636]: WARNING: autopkgtest exited with code: 1
                                                                                                                         ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32369", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/8c598553c21e2438910315b47934684fb8ae32e0"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 08:15:48", "uuid": "7b6cfed3-ec5c-456e-a308-33b322464f78"}'
Apr 30 13:21:20 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661636]: INFO: Killing openstack server with uuid 7b6cfed3-ec5c-456e-a308-33b322464f78
Apr 30 13:21:25 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98972]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-125829-juju-7f2275-prod-proposed-migration-environment-2-c2e0b8b5-adad-4b52-a94b-de0ddac192cd
Apr 30 13:21:26 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98976]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-130047-juju-7f2275-prod-proposed-migration-environment-2-96711e57-9e61-4d6c-b15f-0ae2ba6f804e
Apr 30 13:21:27 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98880]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-112657-juju-7f2275-prod-proposed-migration-environment-2-6f2e0d84-b18e-451b-bcec-6ebf9482f65c
Apr 30 13:21:35 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98931]: WARNING: autopkgtest exited with code: -15
                                                                                                                       ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32380", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/57c417d121ae0fc400e4e3e54bac642bdbcfa755"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 09:27:22", "uuid": "497eb00d-8fd1-4b38-bfb9-2850b8e5c1ec"}'
Apr 30 13:21:35 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98931]: INFO: Killing openstack server with uuid 497eb00d-8fd1-4b38-bfb9-2850b8e5c1ec
Apr 30 13:21:37 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660793]: INFO: Failed to find openstack server with uuid: 49f518db-d284-4c14-8058-e7f1ca63aef4
Apr 30 13:21:38 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661673]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-105017-juju-7f2275-prod-proposed-migration-environment-2-769595c9-f300-4eba-8a28-635efecbaa53
Apr 30 13:21:38 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661621]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-105532-juju-7f2275-prod-proposed-migration-environment-2-80b667d5-4f25-44f1-89db-411ec6df82bf
Apr 30 13:21:38 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98942]: INFO: Failed to find openstack server with uuid: 67703c4c-3e8e-4dbd-8d92-79cdc66d56fe
Apr 30 13:21:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660786]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-104927-juju-7f2275-prod-proposed-migration-environment-2-11b52002-d783-4f3d-a3c8-b8658ff342b2
Apr 30 13:21:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1624256]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-105237-juju-7f2275-prod-proposed-migration-environment-2-16fcc848-526b-4cf2-a66a-b80320b2ba4c
Apr 30 13:21:41 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661636]: INFO: Failed to find openstack server with uuid: 7b6cfed3-ec5c-456e-a308-33b322464f78
Apr 30 13:21:46 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1624209]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-101902-juju-7f2275-prod-proposed-migration-environment-2-8be88820-0e89-4e23-a06c-870437ba64fd
Apr 30 13:21:47 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98854]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-103749-juju-7f2275-prod-proposed-migration-environment-2-4424ca5f-97c9-4830-b131-0924129821c9
Apr 30 13:21:51 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660739]: WARNING: autopkgtest exited with code: 1
                                                                                                                         ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32393", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/bd89dab576f6ace02f9e5b1a5d91d8a13ca688df"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 08:53:54", "uuid": "0453257a-909b-413d-906c-4975fcfb9e2c"}'
Apr 30 13:21:53 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660739]: INFO: Killing openstack server with uuid 0453257a-909b-413d-906c-4975fcfb9e2c
Apr 30 13:21:54 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661729]: WARNING: autopkgtest exited with code: 1
                                                                                                                         ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32352", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/6507e8b760d556489826a3ad9ed375e6965d834f"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-21 17:38:22", "uuid": "2eccae90-42f2-4737-a7f5-7d79a2fad228"}'
Apr 30 13:21:54 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661742]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-123755-juju-7f2275-prod-proposed-migration-environment-2-36efbba3-0cb6-41a5-b7e1-886dbff2ef5f
Apr 30 13:21:55 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661729]: INFO: Killing openstack server with uuid 2eccae90-42f2-4737-a7f5-7d79a2fad228
Apr 30 13:21:59 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[99027]: WARNING: autopkgtest exited with code: 1
                                                                                                                       ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32383", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/6b03c84b75cf4793aca7dc7cdcd5e05fd3d605b3"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 08:49:43", "uuid": "b54c3868-f2fd-4f3a-9cb3-0a3db8a863c5"}'
Apr 30 13:22:00 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[99027]: INFO: Killing openstack server with uuid b54c3868-f2fd-4f3a-9cb3-0a3db8a863c5
Apr 30 13:22:06 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98959]: WARNING: autopkgtest exited with code: 1
                                                                                                                       ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32391", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/8308fcfec3153e0f48a8c1ab41a8304c14af27fb"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 08:50:07", "uuid": "31ef00d6-a2d1-4623-9472-12f178c4ed05"}'
Apr 30 13:22:06 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1624225]: INFO: Failed to find openstack server with uuid: c41aa627-6765-4538-881d-2b7f8823769d
Apr 30 13:22:07 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98996]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-101706-juju-7f2275-prod-proposed-migration-environment-2-b6f44c57-6763-408e-b255-b7e1349e3f6a
Apr 30 13:22:07 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98959]: INFO: Killing openstack server with uuid 31ef00d6-a2d1-4623-9472-12f178c4ed05
Apr 30 13:22:09 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98929]: WARNING: autopkgtest exited with code: 1
                                                                                                                       ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32393", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/57c3dc45b68eb65b8304118bb37792c7b39c5fea"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 09:07:07", "uuid": "8efc5a88-36d9-41dd-8584-c7393f8a9479"}'
Apr 30 13:22:10 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1623849]: INFO: Failed to find openstack server with uuid: 8bce7142-181c-4eb5-93b2-1aac819b8fc3
Apr 30 13:22:10 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661666]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-105016-juju-7f2275-prod-proposed-migration-environment-2-1d4c26b7-a8a4-4912-b3cc-43ed1c706143
Apr 30 13:22:10 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98929]: INFO: Killing openstack server with uuid 8efc5a88-36d9-41dd-8584-c7393f8a9479
Apr 30 13:22:12 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660964]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-114343-juju-7f2275-prod-proposed-migration-environment-2-c42f0458-ba87-4e9d-aa67-8c30d14288d9
Apr 30 13:22:19 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[98960]: INFO: Deleted test server: adt-noble-s390x-systemd-upstream-20240430-094003-juju-7f2275-prod-proposed-migration-environment-2-0a2c0999-e137-4b00-b4f7-7873873b7afc
Apr 30 13:22:20 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661756]: WARNING: autopkgtest exited with code: 1
                                                                                                                         ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32540", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/b45b497f4d45dbb4a12126676f22eee247ed4db6"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-29 12:15:19", "uuid": "b0dbf2c3-02f6-4b07-8141-11a5b38c3ba2"}'
Apr 30 13:22:23 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1661756]: INFO: Killing openstack server with uuid b0dbf2c3-02f6-4b07-8141-11a5b38c3ba2
Apr 30 13:22:26 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[1660883]: WARNING: autopkgtest exited with code: 1
                                                                                                                         ack-ing message: b'systemd-upstream\n{"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=32392", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/7f19c9de733b51a6dc56947af9a35d0876d58d70"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-04-22 09:09:15", "uuid": "4de3f5f2-0e06-47d6-9ae1-6925e4fbe2bb"}'
```

Related branches

~andersson123/autopkgtest-cloud:worker-dont-remove-queue-item-systemctl-restart
Paride Legovini: Approve
Diff: 55 lines (+26/-3)
2 files modified
charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker (+23/-3)
docs/administration.rst (+3/-0)
Tim Andersson (andersson123)
Changed in auto-package-testing:
assignee: nobody → Tim Andersson (andersson123)
importance: Undecided → High
Revision history for this message
Tim Andersson (andersson123) wrote :

looking into this rn

Changed in auto-package-testing:
status: New → In Progress
Revision history for this message
Tim Andersson (andersson123) wrote :

it seems that for some reason autopkgtest does inherit the signal from the kill command, and from systemctl restart:
```
May 02 16:13:52 juju-113c58-stg-proposed-migration-environment-35 /home/ubuntu/autopkgtest-cloud/worker/worker[2078485]: INFO: Caught SIGTERM, requesting exit
May 02 16:13:53 juju-113c58-stg-proposed-migration-environment-35 /home/ubuntu/autopkgtest-cloud/worker/worker[2078485]: WARNING: autopkgtest exited with code: -15
                                                                                                                         ack-ing message: b'autopkgtest\n{"triggers": ["autopkgtest/5.34ubuntu2"], "submit-time": "2024-05-02 16:12:58", "uuid": "5db3f9be-26f8-42a8-ae2b-2d3ec70cdfd9"}'

```

Tim Andersson (andersson123)
Changed in auto-package-testing:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.