Merge ~ahasenack/ubuntu/+source/samba:focal-samba-kb5028166-2027716 into ubuntu/+source/samba:ubuntu/focal-devel

[Andreas Hasenack]

PPA: https://launchpad.net/~ahasenack/+archive/ubuntu/samba-kb5028166/

Bug fix for #2027716. SRU template is filled in, including a test case.

I split the patch in two commits: one that introduces the upstream patch, pristine, and another that removes the hunks that changed the upstream test suite. We don't run that test suite, and I think a smaller patch is easier to review, specially when comparing to the other ubuntu releases which needed a small backport change.

I tried to make incremental changes to this branch when compared to lunar, so it's easier to review. But range-diff is still a bit noisy, specially because focal did not have d/t/util.

The DEP8 test needed even more tweaking for focal, and I tried to keep the differences as additional commits.

DEP8 is green. It doesn't exercise this bug in particular, but does exercise a domain join with linux<->linux, which is a good regression test.

[Lucas Kanashiro (lucaskanashiro)]

Thanks for this MP Andreas! I found no issue in the packaging changes, but the autopgktest execution against the package in your PPA has a failure in ppc64el:

  - samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa1
    + ✅ samba on focal for amd64 @ 20.07.23 17:52:59
    + ✅ samba on focal for arm64 @ 20.07.23 18:34:00
    + ✅ samba on focal for armhf @ 20.07.23 17:56:16
    + ❌ samba on focal for ppc64el @ 20.07.23 18:08:37
      • Status: FAIL
      • Log: https://autopkgtest.ubuntu.com/results/autopkgtest-focal-ahasenack-samba-kb5028166/focal/ppc64el/s/samba/20230720_180837_9f4b6@/log.gz
      • 2064s PASS 🟩
      • 2064s PASS 🟩
      • 2064s PASS 🟩
      • 2064s PASS 🟩
      • 2064s PASS 🟩
      • 2064s PASS 🟩
      • 2064s FAIL 🟥
    + ❌ samba on focal for ppc64el @ 20.07.23 20:21:15
      • Status: FAIL
      • Log: https://autopkgtest.ubuntu.com/results/autopkgtest-focal-ahasenack-samba-kb5028166/focal/ppc64el/s/samba/20230720_202115_59d84@/log.gz
      • 1929s PASS 🟩
      • 1929s PASS 🟩
      • 1929s PASS 🟩
      • 1929s PASS 🟩
      • 1929s PASS 🟩
      • 1929s PASS 🟩
      • 1929s FAIL 🟥
    + ✅ samba on focal for s390x @ 21.07.23 17:35:21

I just re-triggered the test to make sure it is failing again:

* Running:
  # time pkg release arch ppa trigger
  - 530 samba focal ppc64el ahasenack/samba-kb5028166 samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa1

Waiting for the results now.

[Andreas Hasenack (ahasenack)]

Thanks. I don't know why it failed, all of the previous, complicated, checks, passed, even kerberos ones, and it failed close to the end:

1927s ## wbinfo kerberos authentication check for user "<email address hidden>" inside member server
1927s plaintext kerberos password authentication for [<email address hidden>] failed (requesting cctype: FILE)
1927s Could not authenticate user [<email address hidden>%Passw0rd] with Kerberos (ccache: FILE)
1927s ## Something failed, gathering logs

Let's see what the new run brings.

[Andreas Hasenack (ahasenack)]

It failed again, let me see if I can get a ppc64el machine to poke around it...

[Andreas Hasenack (ahasenack)]

Ok, so when wbinfo --krb5auth fails in the DEP8 test, the winbind service logs this error on ppc64el:

[2023/07/26 18:21:42.215411, 0] ../../source3/winbindd/winbindd_cred_cache.c:836(store_memory_creds)
failed to mlock memory: Cannot allocate memory (12)

[Andreas Hasenack (ahasenack)]

In that situation, if I restart winbind, and then try again, then it works :/

This is a lxd container inside a 2Gb VM. Maybe ppc64el needs more memory...? I remember having to adjust some packages to use a bigger VM in the dep8 infrastructure.

[Andreas Hasenack (ahasenack)]

Tests passed on ppc64el this time, and kept passing in all other arches as well:

Results: (from http://autopkgtest.ubuntu.com/results/autopkgtest-focal-ahasenack-samba-kb5028166/?format=plain)
  samba @ amd64:
    26.07.23 22:31:46 Log 🗒️ ✅ Triggers: samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa2
  samba @ arm64:
    27.07.23 00:06:46 Log 🗒️ ✅ Triggers: samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa2
  samba @ armhf:
    26.07.23 22:15:14 Log 🗒️ ✅ Triggers: samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa2
  samba @ ppc64el:
    26.07.23 22:35:14 Log 🗒️ ✅ Triggers: samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa2
  samba @ s390x:
    26.07.23 22:33:21 Log 🗒️ ✅ Triggers: samba/2:4.15.13+dfsg-0ubuntu0.20.04.4~ppa2

[Lucas Kanashiro (lucaskanashiro)]

Thanks Andreas!

review: Approve

[git-ubuntu bot (git-ubuntu-bot)]

Approvers: ahasenack, lucaskanashiro
Uploaders: ahasenack, lucaskanashiro
MP auto-approved

review: Approve

[Andreas Hasenack (ahasenack)]

Thanks, uploaded with rich history:

Uploading samba_4.15.13+dfsg-0ubuntu0.20.04.4.dsc
Uploading samba_4.15.13+dfsg-0ubuntu0.20.04.4.debian.tar.xz
Uploading samba_4.15.13+dfsg-0ubuntu0.20.04.4_source.buildinfo
Uploading samba_4.15.13+dfsg-0ubuntu0.20.04.4_source.changes

diff --git a/debian/changelog b/debian/changelog
index 3ea7797..373ec2d 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,30 @@
+samba (2:4.15.13+dfsg-0ubuntu0.20.04.4) focal; urgency=medium
+
+  * d/p/secure-channel-faulty-kb5028166.patch: fix domain membership
+    after Windows KB5028166 update (LP: #2027716)
+  * Cherry pick samba AD DC provisioning DEP8 test from later Ubuntu
+    releases (LP: #1977746, LP: #2011745):
+    - d/t/control, d/t/util,d/t/samba-ad-dc-provisioning-internal-dns:
+      samba AD DC provisioning and domain join tests with internal DNS
+      + d/t/control: adjust package dependencies
+      + d/t/samba-ad-dc-provisioning-internal-dns: handle the case where
+        libnss-winbind does not automatically add winbind to
+        /etc/nsswitch.conf (that is done only in Lunar and later)
+      + d/t/samba-ad-dc-provisioning-internal-dns: use case insensitive
+        match when inspecting kerberos tickets, as the hostname may be
+        capitalized
+      + d/t/samba-ad-dc-provisioning-internal-dns: Adjust regexp for
+        slightly different resolvectl output
+      + d/t/util: several lxc command output parsing changes, needed for
+        this older version of the lxd snap
+      + d/t/samba-ad-dc-provisioning-internal-dns: more dependencies for
+        the winbind and sssd domain join tests, which don't get
+        installed automatically for us by this version of realmd
+      + d/t/util: increase the RLIMIT_MEMLOCK limit for lxd containers,
+        as the default of 64kb is too low for at least ppc64el on focal
+
+ -- Andreas Hasenack <[e-mail]>  Sun, 23 Jul 2023 17:19:48 -0300
+
 samba (2:4.15.13+dfsg-0ubuntu0.20.04.3) focal-security; urgency=medium
 
   * SECURITY UPDATE: Out-Of-Bounds read in winbind AUTH_CRAP
diff --git a/debian/patches/secure-channel-faulty-kb5028166.patch b/debian/patches/secure-channel-faulty-kb5028166.patch
new file mode 100644
index 0000000..c1367f7
--- /dev/null
+++ b/debian/patches/secure-channel-faulty-kb5028166.patch
@@ -0,0 +1,215 @@
+From 2150e7f3dc409b415ca8b6a541729a49932c5073 Mon Sep 17 00:00:00 2001
+From: Stefan Metzmacher <[e-mail]>
+Date: Sat, 15 Jul 2023 17:20:32 +0200
+Subject: [PATCH 1/4] netlogon.idl: add support for netr_LogonGetCapabilities
+ response level 2
+
+We don't have any documentation about this yet, but tests against
+a Windows Server 2022 patched with KB5028166 revealed that
+the response for query_level=2 is exactly the same as
+for querey_level=1.
+
+Until we know the reason for query_level=2 we won't
+use it as client nor support it in the server, but
+we want ndrdump to work.
+
+BUG: https://bugzilla.samba.org/show_bug.cgi?id=15418
+
+Signed-off-by: Stefan Metzmacher <[e-mail]>
+Reviewed-by: Andrew Bartlett <[e-mail]>
+(cherry picked from commit 5f87888ed53320538cf773d64868390d8641a40e)
+---
+ librpc/idl/netlogon.idl | 1 +
+ 1 file changed, 1 insertion(+)
+
+Ubuntu patch note: removed the parts that changed the upstream test suite
+
+Origin: backport, https://bugzilla.samba.org/attachment.cgi?id=17987
+Bug: https://bugzilla.samba.org/show_bug.cgi?id=15418
+Bug-Ubuntu: https://bugs.launchpad.net/ubuntu/+source/samba/+bug/2027716
+Last-Update: 2023-07-17
+
+diff --git a/librpc/idl/netlogon.idl b/librpc/idl/netlogon.idl
+index d956a661fff7..b51767136d3c 100644
+--- a/librpc/idl/netlogon.idl
++++ b/librpc/idl/netlogon.idl
+@@ -1241,6 +1241,7 @@ interface netlogon
+ 	/* Function 0x15 */
+ 	typedef [switch_type(uint32)] union {
+ 		[case(1)] netr_NegotiateFlags server_capabilities;
++		[case(2)] netr_NegotiateFlags server_capabilities;
+ 	} netr_Capabilities;
+ 
+ 	NTSTATUS netr_LogonGetCapabilities(
+-- 
+2.34.1
+
+
+From fa71e7b4b027dc8224fda7125f1faaefa4e71eae Mon Sep 17 00:00:00 2001
+From: Stefan Metzmacher <[e-mail]>
+Date: Sat, 15 Jul 2023 16:11:48 +0200
+Subject: [PATCH 3/4] s4:rpc_server:netlogon: generate FAULT_INVALID_TAG for
+ invalid netr_LogonGetCapabilities levels
+
+This is important as Windows clients with KB5028166 seem to
+call netr_LogonGetCapabilities with query_level=2 after
+a call with query_level=1.
+
+An unpatched Windows Server returns DCERPC_NCA_S_FAULT_INVALID_TAG
+for query_level values other than 1.
+While Samba tries to return NT_STATUS_NOT_SUPPORTED, but
+later fails to marshall the response, which results
+in DCERPC_FAULT_BAD_STUB_DATA instead.
+
+Because we don't have any documentation for level 2 yet,
+we just try to behave like an unpatched server and
+generate DCERPC_NCA_S_FAULT_INVALID_TAG instead of
+DCERPC_FAULT_BAD_STUB_DATA.
+Which allows patched Windows clients to keep working
+against a Samba DC.
+
+BUG: https://bugzilla.samba.org/show_bug.cgi?id=15418
+
+Signed-off-by: Stefan Metzmacher <[e-mail]>
+Reviewed-by: Andrew Bartlett <[e-mail]>
+(cherry picked from commit d5f1097b6220676d56ed5fc6707acf667b704518)
+---
+ .../knownfail.d/netr_LogonGetCapabilities     |  2 --
+ source4/rpc_server/netlogon/dcerpc_netlogon.c | 28 ++++++++++++++++---
+ 2 files changed, 24 insertions(+), 6 deletions(-)
+
+diff --git a/source4/rpc_server/netlogon/dcerpc_netlogon.c b/source4/rpc_server/netlogon/dcerpc_netlogon.c
+index 6a3e044eb9da..26be4f567513 100644
+--- a/source4/rpc_server/netlogon/dcerpc_netlogon.c
++++ b/source4/rpc_server/netlogon/dcerpc_netlogon.c
+@@ -2399,6 +2399,30 @@ static NTSTATUS dcesrv_netr_LogonGetCapabilities(struct dcesrv_call_state *dce_c
+ 	struct netlogon_creds_CredentialState *creds;
+ 	NTSTATUS status;
+ 
++	switch (r->in.query_level) {
++	case 1:
++		break;
++	case 2:
++		/*
++		 * Until we know the details behind KB5028166
++		 * just return DCERPC_NCA_S_FAULT_INVALID_TAG
++		 * like an unpatched Windows Server.
++		 */
++		FALL_THROUGH;
++	default:
++		/*
++		 * There would not be a way to marshall the
++		 * the response. Which would mean our final
++		 * ndr_push would fail an we would return
++		 * an RPC-level fault with DCERPC_FAULT_BAD_STUB_DATA.
++		 *
++		 * But it's important to match a Windows server
++		 * especially before KB5028166, see also our bug #15418
++		 * Otherwise Windows client would stop talking to us.
++		 */
++		DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG);
++	}
++
+ 	status = dcesrv_netr_creds_server_step_check(dce_call,
+ 						     mem_ctx,
+ 						     r->in.computer_name,
+@@ -2410,10 +2434,6 @@ static NTSTATUS dcesrv_netr_LogonGetCapabilities(struct dcesrv_call_state *dce_c
+ 	}
+ 	NT_STATUS_NOT_OK_RETURN(status);
+ 
+-	if (r->in.query_level != 1) {
+-		return NT_STATUS_NOT_SUPPORTED;
+-	}
+-
+ 	r->out.capabilities->server_capabilities = creds->negotiate_flags;
+ 
+ 	return NT_STATUS_OK;
+-- 
+2.34.1
+
+
+From 05f110e1a4d4b38bfbaaa3a92fda7a9127b3b456 Mon Sep 17 00:00:00 2001
+From: Stefan Metzmacher <[e-mail]>
+Date: Sat, 15 Jul 2023 16:11:48 +0200
+Subject: [PATCH 4/4] s3:rpc_server:netlogon: generate FAULT_INVALID_TAG for
+ invalid netr_LogonGetCapabilities levels
+
+This is important as Windows clients with KB5028166 seem to
+call netr_LogonGetCapabilities with query_level=2 after
+a call with query_level=1.
+
+An unpatched Windows Server returns DCERPC_NCA_S_FAULT_INVALID_TAG
+for query_level values other than 1.
+While Samba tries to return NT_STATUS_NOT_SUPPORTED, but
+later fails to marshall the response, which results
+in DCERPC_FAULT_BAD_STUB_DATA instead.
+
+Because we don't have any documentation for level 2 yet,
+we just try to behave like an unpatched server and
+generate DCERPC_NCA_S_FAULT_INVALID_TAG instead of
+DCERPC_FAULT_BAD_STUB_DATA.
+Which allows patched Windows clients to keep working
+against a Samba DC.
+
+BUG: https://bugzilla.samba.org/show_bug.cgi?id=15418
+
+Signed-off-by: Stefan Metzmacher <[e-mail]>
+Reviewed-by: Andrew Bartlett <[e-mail]>
+
+Autobuild-User(master): Stefan Metzmacher <[e-mail]>
+Autobuild-Date(master): Mon Jul 17 07:35:09 UTC 2023 on atb-devel-224
+
+(cherry picked from commit dfeabce44fbb78083fbbb2aa634fc4172cf83db9)
+---
+ .../knownfail.d/netr_LogonGetCapabilities     |  1 -
+ source3/rpc_server/netlogon/srv_netlog_nt.c   | 29 ++++++++++++++++---
+ 2 files changed, 25 insertions(+), 5 deletions(-)
+ delete mode 100644 selftest/knownfail.d/netr_LogonGetCapabilities
+
+diff --git a/source3/rpc_server/netlogon/srv_netlog_nt.c b/source3/rpc_server/netlogon/srv_netlog_nt.c
+index 5906464a9f3..35433ec6781 100644
+--- a/source3/rpc_server/netlogon/srv_netlog_nt.c
++++ b/source3/rpc_server/netlogon/srv_netlog_nt.c
+@@ -2421,6 +2421,31 @@ NTSTATUS _netr_LogonGetCapabilities(struct pipes_struct *p,
+ 	struct netlogon_creds_CredentialState *creds;
+ 	NTSTATUS status;
+ 
++	switch (r->in.query_level) {
++	case 1:
++		break;
++	case 2:
++		/*
++		 * Until we know the details behind KB5028166
++		 * just return DCERPC_NCA_S_FAULT_INVALID_TAG
++		 * like an unpatched Windows Server.
++		 */
++		FALL_THROUGH;
++	default:
++		/*
++		 * There would not be a way to marshall the
++		 * the response. Which would mean our final
++		 * ndr_push would fail an we would return
++		 * an RPC-level fault with DCERPC_FAULT_BAD_STUB_DATA.
++		 *
++		 * But it's important to match a Windows server
++		 * especially before KB5028166, see also our bug #15418
++		 * Otherwise Windows client would stop talking to us.
++		 */
++		p->fault_state = DCERPC_NCA_S_FAULT_INVALID_TAG;
++		return NT_STATUS_NOT_SUPPORTED;
++	}
++
+ 	become_root();
+ 	status = netr_creds_server_step_check(p, p->mem_ctx,
+ 					      r->in.computer_name,
+@@ -2432,10 +2457,6 @@ NTSTATUS _netr_LogonGetCapabilities(struct pipes_struct *p,
+ 		return status;
+ 	}
+ 
+-	if (r->in.query_level != 1) {
+-		return NT_STATUS_NOT_SUPPORTED;
+-	}
+-
+ 	r->out.capabilities->server_capabilities = creds->negotiate_flags;
+ 
+ 	return NT_STATUS_OK;
diff --git a/debian/patches/series b/debian/patches/series
index 54984e0..be2f88c 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -66,3 +66,4 @@ CVE-2023-34968-09.patch
 CVE-2023-34968-10.patch
 CVE-2023-34968-11.patch
 CVE-2023-34968-12.patch
+secure-channel-faulty-kb5028166.patch
diff --git a/debian/tests/control b/debian/tests/control
index efc23b7..5e1cd04 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -20,3 +20,7 @@ Restrictions: needs-root, allow-stderr, isolation-container
 Tests: reinstall-samba-common-bin
 Depends: samba-common, samba-common-bin
 Restrictions: needs-root, needs-reboot, isolation-machine, allow-stderr
+
+Tests: samba-ad-dc-provisioning-internal-dns
+Depends: samba, samba-dsdb-modules, samba-vfs-modules, winbind, smbclient, krb5-user, bind9-dnsutils, lxd | snapd, lsb-release, dctrl-tools
+Restrictions: needs-root, isolation-machine, allow-stderr, breaks-testbed
diff --git a/debian/tests/samba-ad-dc-provisioning-internal-dns b/debian/tests/samba-ad-dc-provisioning-internal-dns
new file mode 100755
index 0000000..f84372c
--- /dev/null
+++ b/debian/tests/samba-ad-dc-provisioning-internal-dns
@@ -0,0 +1,408 @@
+#!/bin/bash
+
+set -e
+set -o pipefail
+
+source debian/tests/util
+
+declare -r domain="EXAMPLE"
+declare -r realm="EXAMPLE.FAKE"
+declare -r adminpass="Passw0rd"
+declare -r test_user="test_user_${RANDOM}"
+declare -r test_pw="test_user_secret_${RANDOM}"
+declare -A user_pass
+user_pass[Administrator]="${adminpass}"
+user_pass[${test_user}]="${test_pw}"
+declare -A join_method_deps
+# Minimum set of deps: let realmd install the extra dependencies
+# as needed, depending on the join method.
+# sssd-dbus is needed by the sssctl tool, and is not installed automatically
+# via deps in focal
+join_method_deps[realmd_sssd]="realmd krb5-user smbclient sssd-dbus"
+# libnss-winbind needs to be explicitly listed because realmd only started
+# installing it in version 0.17.0, that's >= focal
+join_method_deps[realmd_winbind]="realmd krb5-user smbclient libnss-winbind"
+
+
+cleanup() {
+    rc=$?
+    set +e # so we don't exit midcleanup
+    if [ ${rc} -ne 0 ]; then
+        echo "## Something failed, gathering logs"
+        echo
+        echo "## smb.conf"
+        cat /etc/samba/smb.conf
+        echo
+        echo "## resolv.conf"
+        cat /etc/resolv.conf
+        echo
+        echo "## resolvectl status"
+        resolvectl status
+        echo "## journal for samba-ad-dc.service"
+        journalctl -u samba-ad-dc.service --lines 500
+        echo
+        for log in /var/log/samba/log.*; do
+            # skip compressed logrotated files
+            if [ "${log%.gz}" != "${log}" ]; then
+                continue
+            fi
+            [ -s "${log}" ] || continue
+            echo "## $(basename ${log}):"
+            tail -n 500 "${log}"
+            echo
+        done
+        echo "## syslog"
+        tail -n 500 /var/log/syslog
+    fi
+}
+
+trap cleanup EXIT
+
+assert_testparm() {
+    local parameter="${1}"
+    local expected_value="${2}"
+    local current_value=""
+    local -i retval=0
+
+    echo -n "Asserting ${parameter} is ${expected_value}: "
+    current_value=$(testparm -s --parameter-name "${parameter}" 2>/dev/null) || {
+        retval=$?
+        echo "FAIL"
+        return ${retval}
+    }
+    if [ "${current_value}" = "${expected_value}" ]; then
+        echo "OK"
+        return 0
+    else
+        echo "FAIL"
+        return 1
+    fi
+}
+
+basic_config_tests() {
+    echo "## Basic config tests"
+    testparm -s > /dev/null
+    assert_testparm "realm" "${realm}"
+    assert_testparm "workgroup" "${domain}"
+    assert_testparm "server role" "active directory domain controller"
+    echo
+}
+
+dns_tests() {
+    echo "## DNS tests"
+    echo "Obtaining administrator kerberos ticket"
+    echo "${adminpass}" | timeout --verbose 30 kinit Administrator
+    echo
+    echo "Querying server info"
+    samba-tool dns serverinfo "$(hostname)"
+    echo
+    echo "Checking we got a service ticket of type host/"
+    klist | grep -i "host/$(hostname)"
+    echo
+    echo "Checking specific DNS records"
+    for srv in _ldap._tcp _kerberos._tcp _kerberos._udp _kpasswd._udp; do
+        echo -n "${srv}.${realm,,}: "
+        dig @localhost +short -t SRV ${srv}.${realm,,}
+        echo
+    done
+    echo
+    echo -n "Checking that our hostname \"$(hostname)\" is in DNS: "
+    myip=$(dig @localhost +short -t A "$(hostname).${realm,,}")
+    echo "${myip}"
+    echo
+}
+
+user_creation_tests() {
+    echo "## User creation tests"
+    samba-tool domain passwordsettings set --complexity=off
+    echo "Creating user \"${test_user}\" with password ${test_pw}"
+    samba-tool user add "${test_user}" "${test_pw}"
+    echo
+    echo "Attempting to obtain kerberos ticket for user \"${test_user}\""
+    # just in case it ends up waiting at a prompt, we use "timeout"
+    echo "${test_pw}" | timeout --verbose 30 kinit "${test_user}"
+    echo "Ticket obtained"
+    klist
+    echo
+}
+
+smbclient_tests() {
+    echo "## smbclient tests"
+    kdestroy || :
+    echo
+    echo "Obtaining a TGT for ${test_user}"
+    echo "${test_pw}" | timeout --verbose 30 kinit "${test_user}"
+    klist | grep krbtgt
+    echo
+    echo "Attempting password-less authentication with smbclient"
+    echo
+    echo "Listing shares"
+    smbclient -L "$(hostname)" --use-kerberos=required -k
+    echo
+    echo "Listing the sysvol share"
+    smbclient "//$(hostname)/sysvol" --use-kerberos=required -k -c "ls"
+    echo
+    echo "Listing policies"
+    # lowercase the ${realm}
+    smbclient "//$(hostname)/sysvol" --use-kerberos=required -k -c "ls ${realm,,}/Policies/*"
+    echo
+    echo "Checking that we have a ticket for the cifs service after all these commands"
+    klist | grep cifs/
+    echo
+}
+
+server_join_tests() {
+    local member_server
+    # the join methods are the keys of the join_method_deps dict
+    local -a methods=("${!join_method_deps[@]}")
+    local member_server="member-server"
+
+    echo "## Server join tests"
+    echo "## Initializing lxd"
+    setup_lxd "${realm,,}"
+
+    for method in "${methods[@]}"; do
+        echo "## Setting up member server to join a domain using method ${method}"
+        setup_member_server "${member_server}" "${method}"
+        echo "## Joining domain with method ${method}"
+        join_domain "${member_server}" "${method}"
+        echo
+        echo "## Verifying join with method ${method}"
+        verify_join "${member_server}" "${method}"
+        echo
+        echo "## Leaving domain with method ${method}"
+        leave_domain "${member_server}" "${method}"
+        echo
+        echo "## Destroying member server"
+        lxc delete --force "${member_server}"
+    done
+}
+
+setup_member_server() {
+    local container_name="${1}"
+    local method="${2}"
+    local release
+
+    release="$(lsb_release -cs)"
+    if [ -z "${join_method_deps[${method}]}" ]; then
+        echo "## INTERNAL ERROR, invalid join method: ${method}"
+        return 1
+    fi
+    echo "## Got test dependencies: ${join_method_deps[${method}]}"
+    # can't use cloud-init here to install packages, because we first need to
+    # sync the apt config from the host to the container
+    echo "## Launching ${release} container"
+    lxc launch "ubuntu-daily:${release}" "${container_name}" -q
+    wait_container_ready "${container_name}"
+    send_apt_config "${container_name}"
+    copy_local_apt_files "${container_name}"
+    echo "## Installing dependencies in test container"
+    install_packages_in_container "${container_name}" ${join_method_deps[${method}]}
+}
+
+join_domain_realmd_winbind() {
+    local server="${1}"
+    local discover_cmd="realm discover -v --membership-software=samba --client-software=winbind ${realm,,}"
+    local join_cmd="realm join -v --membership-software=samba --client-software=winbind ${realm,,}"
+
+    echo "## Domain information"
+    lxc exec "${server}" -- ${discover_cmd}
+    echo
+    echo "## Running join command: ${join_cmd}"
+    echo "${adminpass}" | lxc exec "${server}" -- ${join_cmd}
+    # LP: #1980246
+    # So far, only lunar and later automatically add winbind to /etc/nsswitch.conf.
+    lxc exec "${server}" -- sed -r -i \
+          -e  '/^(passwd|group):.*[[:space:]]winbind\b/b' \
+          -e 's/^(passwd|group):.*/& winbind/' \
+          /etc/nsswitch.conf
+}
+
+verify_join_realmd_winbind() {
+    local server="${1}"
+    local member_domain
+
+    echo -n "## Verifying member server joined domain name: "
+    member_domain=$(lxc exec "${server}" -- wbinfo --own-domain)
+    echo "${member_domain}"
+    if [ "${member_domain}" != "${domain}" ]; then
+        echo "ERROR: expected member server domain to match the joined domain:"
+        echo "member server domain: ${member_domain}"
+        echo "AD domain: ${domain}"
+        return 1
+    fi
+    echo
+    # we just want to see the output, not parse it
+    echo "## Domain status in member server"
+    lxc exec "${server}" -- wbinfo --domain-info "${member_domain}"
+    echo
+    echo "## User status in member server"
+    for u in "${!user_pass[@]}"; do
+        echo "## User \"${u}@${realm}\" information:"
+        lxc exec "${server}" -- wbinfo --user-info "${u}@${realm}"
+        echo
+        echo "## id ${u}@${realm}"
+        lxc exec "${server}" -- id ${u}@${realm}
+        echo
+        echo "## kinit authentication check for user \"${u}@${realm}\" inside member server"
+        echo "${user_pass[${u}]}" | lxc exec "${server}" -- timeout --verbose 30 kinit "${u}@${realm}"
+        lxc exec "${server}" -- klist
+        echo
+        echo "## Listing shares with the obtained kerberos ticket"
+        lxc exec "${server}" -- smbclient -L "$(hostname)" --use-kerberos=required -k
+        lxc exec "${server}" -- kdestroy
+        echo
+        echo "## wbinfo authentication check for user \"${u}@${realm}\" inside member server"
+        # non-interactive format for username is user%password
+        lxc exec "${server}" -- wbinfo --authenticate="${u}@${realm}%${user_pass[${u}]}"
+        echo
+        echo "## wbinfo kerberos authentication check for user \"${u}@${realm}\" inside member server"
+        lxc exec "${server}" -- wbinfo --krb5auth="${u}@${realm}%${user_pass[${u}]}"
+        echo
+        echo "## Listing shares with the obtained kerberos ticket"
+        lxc exec "${server}" -- smbclient -L "$(hostname)" --use-kerberos=required -k
+        lxc exec "${server}" -- kdestroy
+    done
+}
+
+leave_domain_realmd_winbind() {
+    local server="${1}"
+    local leave_cmd="realm leave -v --remove --client-software=winbind"
+
+    echo "## Running leave command: ${leave_cmd}"
+    echo "${adminpass}" | lxc exec "${server}" -- ${leave_cmd}
+}
+
+join_domain_realmd_sssd() {
+    local server="${1}"
+    local discover_cmd="realm discover -v --membership-software=adcli --client-software=sssd ${realm,,}"
+    local join_cmd="realm join -v --membership-software=adcli --client-software=sssd ${realm,,}"
+
+    echo "## Domain information"
+    lxc exec "${server}" -- ${discover_cmd}
+    echo
+    echo "## Running join command: ${join_cmd}"
+    echo "${adminpass}" | lxc exec "${server}" -- ${join_cmd}
+    echo
+}
+
+verify_join_realmd_sssd() {
+    local server="${1}"
+    local samba_domain
+
+    echo -n "## Verifying member server joined domain name: "
+    samba_domain=$(lxc exec "${server}" -- sssctl domain-list)
+    echo "${samba_domain}"
+    if [ "${samba_domain}" != "${realm,,}" ]; then
+        echo "ERROR: expected member server domain to match the joined domain:"
+        echo "member server domain: ${samba_domain}"
+        echo "AD domain: ${realm,,}"
+        return 1
+    fi
+    echo
+    # we just want to see the output, not parse it
+    echo "## Domain status in member server"
+    lxc exec "${server}" -- sssctl domain-status "${realm}"
+    echo
+    echo "## User status in member server"
+    for u in "${!user_pass[@]}"; do
+        echo "## User \"${u}@${realm}\" information:"
+        lxc exec "${server}" -- sssctl user-checks "${u}@${realm}"
+        echo
+        echo "## id ${u}@${realm}"
+        lxc exec "${server}" -- id "${u}@${realm}"
+        echo
+        echo "## kinit authentication check for user \"${u}@${realm}\" inside member server"
+        echo "${user_pass[${u}]}" | lxc exec "${server}" -- timeout --verbose 30 kinit "${u}@${realm}"
+        lxc exec "${server}" -- klist
+        echo
+        echo "## Listing shares with the obtained kerberos ticket"
+        lxc exec "${server}" -- smbclient -L "$(hostname)" --use-kerberos=required -k
+        lxc exec "${server}" -- kdestroy
+    done
+}
+
+leave_domain_realmd_sssd() {
+    local server="${1}"
+    local leave_cmd="realm leave -v --remove --client-software=sssd"
+
+    echo "## Running leave command: ${leave_cmd}"
+    echo "${adminpass}" | lxc exec "${server}" -- ${leave_cmd}
+}
+
+join_domain() {
+    local server="${1}"
+    local m="${2}"
+
+    join_domain_${m} "${server}"
+}
+
+verify_join() {
+    local server="${1}"
+    local m="${2}"
+
+    verify_join_${m} "${server}"
+}
+
+leave_domain() {
+    local server="${1}"
+    local m="${2}"
+
+    leave_domain_${m} "${server}"
+}
+
+systemctl stop smbd nmbd winbind
+systemctl disable smbd nmbd winbind
+systemctl mask smbd nmbd winbind
+
+systemctl unmask samba-ad-dc
+systemctl enable samba-ad-dc
+
+if [ -f /etc/samba/smb.conf ]; then
+    mv /etc/samba/smb.conf{,.orig}
+fi
+
+# make sure we are starting fresh, as previous tests might left things around
+
+rm -rf /var/lib/samba/* /var/cache/samba/* /run/samba/*
+kdestroy || :
+
+samba-tool domain provision \
+    --domain="${domain}" \
+    --realm="${realm}" \
+    --adminpass="${adminpass}" \
+    --server-role=dc \
+    --use-rfc2307 \
+    --dns-backend=SAMBA_INTERNAL
+
+current_dns=$(resolvectl status | grep -E "^[[:blank:]]*Current DNS Server:" | awk '{print $4}')
+
+if [ -n "${current_dns}" ]; then
+    echo "## Setting dns forwarder to ${current_dns} in smb.conf"
+    sed -r -i "s,dns forwarder = .*,dns forwarder = ${current_dns}," \
+        /etc/samba/smb.conf
+    unlink /etc/resolv.conf
+    echo "nameserver 127.0.0.1" > /etc/resolv.conf
+    # lowercase substitution
+    echo "search ${realm,,}" >> /etc/resolv.conf
+    systemctl stop systemd-resolved
+    systemctl disable systemd-resolved
+else
+    echo "## Warning, couldn't detect the current DNS server to use as forwarder in smb.conf"
+    echo "## resolvectl status:"
+    resolvectl status
+    echo "## Continuing, and hoping for the best"
+fi
+
+cp -f /var/lib/samba/private/krb5.conf /etc/krb5.conf
+
+systemctl start samba-ad-dc
+
+# give it some time, it's a lot of services to start
+sleep 5s
+
+basic_config_tests
+dns_tests
+user_creation_tests
+smbclient_tests
+server_join_tests
diff --git a/debian/tests/util b/debian/tests/util
new file mode 100644
index 0000000..66ed247
--- /dev/null
+++ b/debian/tests/util
@@ -0,0 +1,178 @@
+#!/bin/sh
+
+# $1: share name
+# $2: comma separated list of vfs_objects to use, if any
+add_share() {
+    local share="$1"
+    local vfs="$2"
+    if ! testparm -s 2>&1 | grep -E "^\[${share}\]"; then
+        echo "Adding [${share}] share"
+        cat >> /etc/samba/smb.conf <<EOFEOF
+[${share}]
+  read only = no
+  guest ok = no
+  path = /${share}
+EOFEOF
+        if [ -n "${vfs}" ]; then
+            echo "vfs objects = ${vfs}" >> /etc/samba/smb.conf
+        fi
+        systemctl reload smbd.service
+    else
+        echo "Share [${share}] already exists, continuing"
+    fi
+}
+
+# $1: username
+# $2: password
+add_user() {
+    local username="$1"
+    local password="$2"
+
+    echo "Creating a local and samba user called ${username}"
+    useradd -m "${username}"
+    echo "Setting samba password for the ${username} user"
+    (echo "${password}"; echo "${password}") | smbpasswd -s -a ${username}
+}
+
+# $1: share name
+populate_share() {
+    local sharename="$1"
+    local usergroup="$2"
+    local sharepath="/${sharename}"
+
+    mkdir -p "${sharepath}"
+    dd if=/dev/urandom bs=4096 count=1000 2>/dev/null | base64 > "${sharepath}/data"
+    cd "${sharepath}"
+    md5sum data > data.md5
+    chown -R "${usergroup}:${usergroup}" "${sharepath}"
+}
+
+
+# $1: kernel version in the form major.minor.patch
+check_kernel_version() {
+    local k_ver=$1
+    local k_major=$(echo ${k_ver} | cut -d . -f 1)
+    local k_minor=$(echo ${k_ver} | cut -d . -f 2)
+
+    # uring is supported starting with kernel 5.1.x
+    if [ ${k_major} -eq 5 ] && [ ${k_minor} -ge 1 ]; then
+       return 0
+    elif [ ${k_major} -ge 6 ]; then
+       return 0
+    else
+        return 1
+    fi
+}
+
+wait_container_ready() {
+    local container="${1}"
+    local -i limit=120 # seconds
+    local -i i=0
+    local -i result=0
+    local ip
+    local output
+
+    while /bin/true; do
+        ip=$(lxc list "${container}" -c 4 --format=csv | tail -1 | awk '{print $1}')
+        if [ -n "${ip}" ]; then
+            break
+        fi
+        i=$((i+1))
+        if [ ${i} -ge ${limit} ]; then
+            return 1
+        fi
+        sleep 1s
+        echo -n "."
+    done
+    while ! nc -z "${ip}" 22; do
+        echo -n "."
+        i=$((i+1))
+        if [ ${i} -ge ${limit} ]; then
+            return 1
+        fi
+        sleep 1s
+    done
+    # cloud-init might still be doing things...
+    # this call blocks, so wrap it in its own little timeout
+    output=$(lxc exec "${container}" -- timeout --verbose $((limit-i)) cloud-init status --wait) || {
+        result=$?
+        echo "cloud-init status --wait failed on container ${container}"
+        echo "${output}"
+        return ${result}
+    }
+    echo
+}
+
+install_lxd() {
+    if ! command -v lxd > /dev/null 2>&1; then
+        # the test depends has "lxd | snapd", so if we don't have lxd, we must
+        # install the snap
+        snap list lxd > /dev/null 2>&1 || {
+            echo "Installing the LXD snap..."
+            snap install lxd
+        }
+    fi
+}
+
+setup_lxd() {
+    local dns_domain="${1}"
+    local nic
+    local dns_ip
+
+    install_lxd
+    # Stop samba while lxd is setup, to avoid conflicts on lxdbr0:53
+    systemctl stop samba-ad-dc
+    lxd init --auto
+    lxd waitready --timeout 600
+    # sample csv output. Columns are NAME,TYPE,MANAGED,DESCRIPTION,USED_BY
+    #enp1s0,physical,NO,,0
+    #lxdbr0,bridge,YES,,1
+    nic=$(lxc network list --format=csv | grep -E "bridge,YES,,1" | cut -d , -f 1)
+    dns_ip=$(lxc network info "${nic}" | grep -w inet | awk '{print $2}')
+    # port=0 effectively disables dnsmasq's DNS, so it doesn't conflict with samba's DNS
+    lxc network set "${nic:-lxdbr0}" ipv6.address=none dns.domain="${dns_domain}" raw.dnsmasq="$(echo -e port=0\\ndhcp-option=option:dns-server,${dns_ip})"
+    if [ -n "${http_proxy}" ]; then
+        lxc config set core.proxy_http "${http_proxy}"
+    fi
+    if [ -n "${https_proxy}" ]; then
+        lxc config set core.proxy_https "${https_proxy}"
+    fi
+    if [ -n "${noproxy}" ]; then
+        lxc config set core.proxy_ignore_hosts "${noproxy}"
+    fi
+    # the default of 64k is too low for, at least, ppc64el on focal
+    lxc profile set default limits.kernel.memlock 262144
+    systemctl start samba-ad-dc
+    # give it some time, it's a lot of services to start
+    sleep 5s
+}
+
+# Copy the local apt package archive over to the lxd container.
+copy_local_apt_files() {
+    local container_name="${1:-docker}"
+
+    for local_source in $(apt-get indextargets | grep-dctrl -F URI -e '^file:/' -sURI | awk '{print $2}'); do
+        local_source=${local_source#file:}
+        local_dir=$(dirname "${local_source}")
+        lxc exec "${container_name}" -- mkdir -p "${local_dir}"
+        tar -cC "${local_dir}" . | lxc exec "${container_name}" -- tar -xC "${local_dir}"
+    done
+}
+
+send_apt_config() {
+    echo "Copying over /etc/apt to container ${1}"
+    lxc exec "${1}" -- rm -rf /etc/apt
+    lxc exec "${1}" -- mkdir -p /etc/apt
+    tar -cC /etc/apt . | lxc exec "${1}" -- tar -xC /etc/apt
+}
+
+install_packages_in_container() {
+    local container="${1}"
+    shift
+    local packages="${*}"
+
+    echo "### Installing dependencies in member server container: ${packages}"
+    lxc exec "${container}" --env DEBIAN_FRONTEND=noninteractive -- apt-get update -q
+    lxc exec "${container}" --env DEBIAN_FRONTEND=noninteractive -- apt-get dist-upgrade -q -y
+    lxc exec "${container}" --env DEBIAN_FRONTEND=noninteractive -- apt-get install -q -y ${packages}
+}