Merge ~schopin/netplan/+git/ubuntu:hirsute-sru into ~ubuntu-core-dev/netplan/+git/ubuntu:ubuntu/hirsute

[Simon Chopin]

[Lukas Märdian (slyon)]

Thank you, LGTM!
Merged and sponsored.

I've removed the final dot "." at the end of the "Fix unset of a devtype subtree" line in d/changelog, to make it comply with lintian's 80 col limit (https://lintian.debian.org/tags/debian-changelog-line-too-long).

autopkgtest passed: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute-slyon-testing/hirsute/amd64/n/netplan.io/20211007_124048_49d33@/log.gz

review: Approve

diff --git a/debian/changelog b/debian/changelog
index ded1111..e7bf504 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,17 @@
+netplan.io (0.103-0ubuntu5~21.04.2) hirsute; urgency=medium
+
+  * Backport patches from impish:
+    + Add d/p/0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch:
+      Fix unset of a devtype subtree, e.g. "netplan set network.ethernets=null".
+      (LP: #1942930)
+    + Add d/p/0005-Implement-YAML-state-tracking-and-use-it-in-the-DBus.patch:
+      Allow to pass a state to netplan apply/try so it can cleanup unused
+      virtual network interfaces after itself. Make use of this functionality
+      inside the DBus Config.Try()/Apply() API and the 'netplan try' CLI.
+      (LP: #1943120)
+
+ -- Simon Chopin <[e-mail]>  Wed, 06 Oct 2021 12:57:35 +0200
+
 netplan.io (0.103-0ubuntu5~21.04.1) hirsute; urgency=medium
 
   * Backport netplan.io 0.103-0ubuntu5 to 21.04 (LP: #1938920)
diff --git a/debian/patches/0005-Implement-YAML-state-tracking-and-use-it-in-the-DBus.patch b/debian/patches/0005-Implement-YAML-state-tracking-and-use-it-in-the-DBus.patch
new file mode 100644
index 0000000..a51ee6c
--- /dev/null
+++ b/debian/patches/0005-Implement-YAML-state-tracking-and-use-it-in-the-DBus.patch
@@ -0,0 +1,583 @@
+From: =?utf-8?q?Lukas_M=C3=A4rdian?= <[e-mail]>
+Date: Mon, 27 Sep 2021 16:40:45 +0200
+Subject: Implement YAML state tracking and use it in the DBus API and
+ netplan-try (LP: #1943120) (FR-1745) (#231)
+
+Allow to pass an optional --state parameter to netplan try/apply describing a directory that contains a netplan configuration tree/state (i.e. /{etc,run,lib}/netplan/*.yaml).
+Netplan will make use of this "old state" to calculate the delta of dropped interface definitions, like bridges/bonds/vlans/tunnels that have been configured before but are not part of the current YAML configuration anymore. It will then try to delete those virtual interfaces (via ip link del dev IFACE) if they still exist.
+
+The same functionality is used to roll back a netplan try command that failed or was rejected.
+
+Generally, the state needs to be provided manually. The DBus API (using io.netplan.Netplan.Config.Try/Apply) is an exception, as the previous state can be backed up automatically in this case.
+
+COMMITS:
+* cli:apply:configmanager: clear_virtual_links during apply
+* tests:scenarios: check virtual interface cleanup
+* doc:apply: update manpage
+* cli:try: use clear_virtual_links from Apply()
+* dbus: make use of new YAML state tracking
+* dbus: properly create and clean the backup state dir
+* cli:apply:clear_virtual_links: make devices a named parameter
+---
+ doc/netplan-apply.md                | 18 ++++++-----
+ netplan/cli/commands/apply.py       | 40 +++++++++++++++++++++++-
+ netplan/cli/commands/try_command.py | 28 ++++++++---------
+ netplan/configmanager.py            | 10 ++++++
+ src/dbus.c                          | 62 ++++++++++++++++++++++++-------------
+ tests/dbus/test_dbus.py             | 24 ++++++++++----
+ tests/integration/base.py           |  7 +++--
+ tests/integration/scenarios.py      | 27 ++++++++++++++++
+ tests/test_cli_units.py             | 40 ++++++++++++++++++++++++
+ tests/test_configmanager.py         |  6 ++++
+ 10 files changed, 211 insertions(+), 51 deletions(-)
+
+diff --git a/doc/netplan-apply.md b/doc/netplan-apply.md
+index 153acb1..7811f04 100644
+--- a/doc/netplan-apply.md
++++ b/doc/netplan-apply.md
+@@ -47,13 +47,17 @@ see **netplan**(5).
+ 
+ # KNOWN ISSUES
+ 
+-**netplan apply** will not remove virtual devices such as bridges
+-and bonds that have been created, even if they are no longer described
+-in the netplan configuration.
+-
+-This can be resolved by manually removing the virtual device (for
+-example ``ip link delete dev bond0``) and then running **netplan
+-apply**, or by rebooting.
++**netplan apply** will not remove virtual devices such as bridges and bonds
++that have been created, even if they are no longer described in the netplan
++configuration. That is due to the fact that netplan operates statelessly and
++is not aware of the previously defined virtal devices.
++
++This can be resolved by manually removing the virtual device (for example
++``ip link delete dev bond0``) and then running **netplan apply**, by rebooting,
++or by creating a temporary backup of the YAML state in ``/etc/netplan``
++before modifying the configuration and passing this state to netplan (e.g.
++``mkdir -p /tmp/netplan_state_backup/etc && cp -r /etc/netplan /tmp/netplan_state_backup/etc/``
++then running **netplan apply --state /tmp/netplan_state_backup**)
+ 
+ 
+ # SEE ALSO
+diff --git a/netplan/cli/commands/apply.py b/netplan/cli/commands/apply.py
+index c3f5caa..477bc2f 100644
+--- a/netplan/cli/commands/apply.py
++++ b/netplan/cli/commands/apply.py
+@@ -48,14 +48,18 @@ class NetplanApply(utils.NetplanCommand):
+                                  help='Only apply SR-IOV related configuration and exit')
+         self.parser.add_argument('--only-ovs-cleanup', action='store_true',
+                                  help='Only clean up old OpenVSwitch interfaces and exit')
++        self.parser.add_argument('--state',
++                                 help='Directory containing previous YAML configuration')
+ 
+         self.func = self.command_apply
+ 
+         self.parse_args()
+         self.run_command()
+ 
+-    def command_apply(self, run_generate=True, sync=False, exit_on_error=True):  # pragma: nocover (covered in autopkgtest)
++    def command_apply(self, run_generate=True, sync=False, exit_on_error=True, state_dir=None):  # pragma: nocover
+         config_manager = ConfigManager()
++        if state_dir:
++            self.state = state_dir
+ 
+         # For certain use-cases, we might want to only apply specific configuration.
+         # If we only need SR-IOV configuration, do that and exit early.
+@@ -190,6 +194,14 @@ class NetplanApply(utils.NetplanCommand):
+         # for now, only applies to non-virtual (real) devices.
+         config_manager.parse()
+         changes = NetplanApply.process_link_changes(devices, config_manager)
++        # delete virtual interfaces that have been defined in a previous state
++        # but are not configured anymore in the current YAML
++        if self.state:
++            cm = ConfigManager(self.state)
++            cm.parse()  # get previous configuration state
++            prev_links = cm.virtual_interfaces.keys()
++            curr_links = config_manager.virtual_interfaces.keys()
++            NetplanApply.clear_virtual_links(prev_links, curr_links, devices)
+ 
+         # if the interface is up, we can still apply some .link file changes
+         # but we cannot apply the interface rename via udev, as it won't touch
+@@ -259,6 +271,32 @@ class NetplanApply(utils.NetplanCommand):
+ 
+         return False
+ 
++    @staticmethod
++    def clear_virtual_links(prev_links, curr_links, devices=[]):
++        """
++        Calculate the delta of virtual links. And remove the links that were
++        dropped from the YAML config, if they were not dropped by the backend
++        already.
++        We can make use of the netplan netdef ids, as those equal the interface
++        name for virtual links.
++        """
++        if not devices:
++            logging.warning('Cannot clear virtual links: no network interfaces provided.')
++            return []
++
++        dropped_interfaces = list(set(prev_links) - set(curr_links))
++        # some interfaces might have been cleaned up already, e.g. by the
++        # NetworkManager backend
++        interfaces_to_clear = list(set(dropped_interfaces).intersection(devices))
++        for link in interfaces_to_clear:
++            try:
++                cmd = ['ip', 'link', 'delete', 'dev', link]
++                subprocess.check_call(cmd)
++            except subprocess.CalledProcessError:
++                logging.warn('Could not delete interface {}'.format(link))
++
++        return dropped_interfaces
++
+     @staticmethod
+     def process_link_changes(interfaces, config_manager):  # pragma: nocover (covered in autopkgtest)
+         """
+diff --git a/netplan/cli/commands/try_command.py b/netplan/cli/commands/try_command.py
+index 198992f..5534596 100644
+--- a/netplan/cli/commands/try_command.py
++++ b/netplan/cli/commands/try_command.py
+@@ -19,10 +19,11 @@
+ 
+ import os
+ import time
++import shutil
+ import signal
+ import sys
++import tempfile
+ import logging
+-import subprocess
+ 
+ from netplan.configmanager import ConfigManager
+ import netplan.cli.utils as utils
+@@ -59,6 +60,8 @@ class NetplanTry(utils.NetplanCommand):
+         self.parser.add_argument('--timeout',
+                                  type=int, default=DEFAULT_INPUT_TIMEOUT,
+                                  help="Maximum number of seconds to wait for the user's confirmation")
++        self.parser.add_argument('--state',
++                                 help='Directory containing previous YAML configuration')
+ 
+         self.func = self.command_try
+ 
+@@ -81,7 +84,7 @@ class NetplanTry(utils.NetplanCommand):
+             self.backup()
+             self.setup()
+ 
+-            NetplanApply().command_apply(run_generate=True, sync=True, exit_on_error=False)
++            NetplanApply().command_apply(run_generate=True, sync=True, exit_on_error=False, state_dir=self.state)
+ 
+             self.t.get_confirmation_input(timeout=self.timeout)
+         except netplan.terminal.InputRejected:
+@@ -114,19 +117,16 @@ class NetplanTry(utils.NetplanCommand):
+         self.configuration_changed = True
+ 
+     def revert(self):  # pragma: nocover (requires user input)
++        # backup the state we just tried to apply
++        tempdir = tempfile.mkdtemp()
++        confdir = os.path.join(tempdir, 'etc', 'netplan')
++        os.makedirs(confdir)
++        shutil.copytree('/etc/netplan', confdir, dirs_exist_ok=True)
++        # restore previous state
+         self.config_manager.revert()
+-        NetplanApply().command_apply(run_generate=False, sync=True, exit_on_error=False)
+-        for ifname in self.new_interfaces:
+-            if ifname not in self.config_manager.bonds and \
+-               ifname not in self.config_manager.bridges and \
+-               ifname not in self.config_manager.vlans:
+-                logging.debug("{} will not be removed: not a virtual interface".format(ifname))
+-                continue
+-            try:
+-                cmd = ['ip', 'link', 'del', ifname]
+-                subprocess.check_call(cmd)
+-            except subprocess.CalledProcessError:
+-                logging.warn("Could not revert (remove) new interface '{}'".format(ifname))
++        NetplanApply().command_apply(run_generate=False, sync=True, exit_on_error=False, state_dir=tempdir)
++        # clear the backup
++        shutil.rmtree(tempdir)
+ 
+     def cleanup(self):  # pragma: nocover (requires user input)
+         self.config_manager.cleanup()
+diff --git a/netplan/configmanager.py b/netplan/configmanager.py
+index 9278d04..133dc5b 100644
+--- a/netplan/configmanager.py
++++ b/netplan/configmanager.py
+@@ -62,6 +62,16 @@ class ConfigManager(object):
+         interfaces.update(self.wifis)
+         return interfaces
+ 
++    @property
++    def virtual_interfaces(self):
++        interfaces = {}
++        # what about ovs_ports?
++        interfaces.update(self.bridges)
++        interfaces.update(self.bonds)
++        interfaces.update(self.tunnels)
++        interfaces.update(self.vlans)
++        return interfaces
++
+     @property
+     def ovs_ports(self):
+         return self.network['ovs_ports']
+diff --git a/src/dbus.c b/src/dbus.c
+index f0aa53a..a486b54 100644
+--- a/src/dbus.c
++++ b/src/dbus.c
+@@ -199,6 +199,30 @@ _clear_tmp_state(const char *config_id, NetplanData *d)
+     return TRUE;
+ }
+ 
++static int
++_backup_global_state(sd_bus_error *ret_error)
++{
++    int r = 0;
++    g_autofree gchar *path = NULL;
++    path = g_strdup_printf("%s/netplan-config-%s", g_get_tmp_dir(), NETPLAN_GLOBAL_CONFIG);
++    /* Create {etc,run,lib} subdirs with owner r/w permissions */
++    char *subdir = NULL;
++    for (int i = 0; i < 3; i++) {
++        subdir = g_strdup_printf("%s/%s/netplan", path, NETPLAN_SUBDIRS[i]);
++        r = g_mkdir_with_parents(subdir, 0700);
++        if (r < 0)
++            // LCOV_EXCL_START
++            return sd_bus_error_setf(ret_error, SD_BUS_ERROR_FAILED,
++                                    "Failed to create '%s': %s\n", subdir, strerror(errno));
++            // LCOV_EXCL_STOP
++        g_free(subdir);
++    }
++
++    /* Copy main *.yaml files from /{etc,run,lib}/netplan/ to GLOBAL backup dir */
++    _copy_yaml_state(NETPLAN_ROOT, path, ret_error);
++    return 0;
++}
++
+ /**
+  * io.netplan.Netplan methods
+  */
+@@ -209,6 +233,7 @@ method_apply(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+     g_autoptr(GError) err = NULL;
+     g_autofree gchar *stdout = NULL;
+     g_autofree gchar *stderr = NULL;
++    g_autofree gchar *state = NULL;
+     gint exit_status = 0;
+     NetplanData *d = userdata;
+ 
+@@ -216,8 +241,9 @@ method_apply(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+      * Otherwise execute 'netplan apply' directly. */
+     if (d->try_pid > 0)
+         return _try_accept(TRUE, m, userdata, ret_error);
+-
+-    gchar *argv[] = {SBINDIR "/" "netplan", "apply", NULL};
++    if (d->config_id)
++        state = g_strdup_printf("--state=%s/netplan-config-%s", g_get_tmp_dir(), NETPLAN_GLOBAL_CONFIG);
++    gchar *argv[] = {SBINDIR "/" "netplan", "apply", state, NULL};
+ 
+     // for tests only: allow changing what netplan to run
+     if (getenv("DBUS_TEST_NETPLAN_CMD") != 0)
+@@ -421,6 +447,7 @@ method_try(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+ {
+     g_autoptr(GError) err = NULL;
+     g_autofree gchar *timeout = NULL;
++    g_autofree gchar *state = NULL;
+     gint child_stdin = -1; /* child process needs an input to function correctly */
+     guint seconds = 0;
+     int r = -1;
+@@ -430,7 +457,9 @@ method_try(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+         return sd_bus_error_setf(ret_error, SD_BUS_ERROR_FAILED, "cannot extract timeout_seconds"); // LCOV_EXCL_LINE
+     if (seconds > 0)
+         timeout = g_strdup_printf("--timeout=%u", seconds);
+-    gchar *argv[] = {SBINDIR "/" "netplan", "try", timeout, NULL};
++    if (d->config_id)
++        state = g_strdup_printf("--state=%s/netplan-config-%s", g_get_tmp_dir(), NETPLAN_GLOBAL_CONFIG);
++    gchar *argv[] = {SBINDIR "/" "netplan", "try", timeout, state, NULL};
+ 
+     // for tests only: allow changing what netplan to run
+     if (getenv("DBUS_TEST_NETPLAN_CMD") != 0)
+@@ -482,6 +511,10 @@ method_config_apply(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+     d->config_dirty = g_strdup(d->config_id);
+ 
+     if (d->try_pid < 0) {
++        r = _backup_global_state(ret_error);
++        if (r < 0)
++            return r; // LCOV_EXCL_LINE
++
+         /* Delete GLOBAL state */
+         unlink_glob(NETPLAN_ROOT, "/{etc,run,lib}/netplan/*.yaml");
+         /* Copy current config state to GLOBAL */
+@@ -491,6 +524,8 @@ method_config_apply(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+     }
+ 
+     r = method_apply(m, d, ret_error);
++    /* Clear GLOBAL backup and config state */
++    _clear_tmp_state(NETPLAN_GLOBAL_CONFIG, d);
+     _clear_tmp_state(d->config_id, d);
+ 
+     /* unlock current config ID and handler ID */
+@@ -535,7 +570,6 @@ static int
+ method_config_try(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+ {
+     NetplanData *d = userdata;
+-    g_autofree gchar *path = NULL;
+     g_autofree gchar *state_dir = NULL;
+     const char *config_id = sd_bus_message_get_path(m) + 27;
+     if (d->try_pid > 0)
+@@ -551,23 +585,9 @@ method_config_try(sd_bus_message *m, void *userdata, sd_bus_error *ret_error)
+     d->try_pid = G_MAXINT;
+     d->config_id = config_id;
+ 
+-    /* Backup GLOBAL state */
+-    path = g_strdup_printf("%s/netplan-config-%s", g_get_tmp_dir(), NETPLAN_GLOBAL_CONFIG);
+-    /* Create {etc,run,lib} subdirs with owner r/w permissions */
+-    char *subdir = NULL;
+-    for (int i = 0; i < 3; i++) {
+-        subdir = g_strdup_printf("%s/%s/netplan", path, NETPLAN_SUBDIRS[i]);
+-        r = g_mkdir_with_parents(subdir, 0700);
+-        if (r < 0)
+-            // LCOV_EXCL_START
+-            return sd_bus_error_setf(ret_error, SD_BUS_ERROR_FAILED,
+-                                    "Failed to create '%s': %s\n", subdir, strerror(errno));
+-            // LCOV_EXCL_STOP
+-        g_free(subdir);
+-    }
+-
+-    /* Copy main *.yaml files from /{etc,run,lib}/netplan/ to GLOBAL backup dir */
+-    _copy_yaml_state(NETPLAN_ROOT, path, ret_error);
++    r = _backup_global_state(ret_error);
++    if (r < 0)
++        return r; // LCOV_EXCL_LINE
+ 
+     /* Clear main *.yaml files */
+     unlink_glob(NETPLAN_ROOT, "/{etc,run,lib}/netplan/*.yaml");
+diff --git a/tests/dbus/test_dbus.py b/tests/dbus/test_dbus.py
+index 87abf22..5c2f5b7 100644
+--- a/tests/dbus/test_dbus.py
++++ b/tests/dbus/test_dbus.py
+@@ -361,6 +361,7 @@ class TestNetplanDBus(unittest.TestCase):
+     def test_netplan_dbus_config_cancel(self):
+         cid = self._new_config_object()
+         tmpdir = '/tmp/netplan-config-{}'.format(cid)
++        backup = '/tmp/netplan-config-BACKUP'
+ 
+         # Verify .Config.Cancel() teardown of the config object and state dirs
+         BUSCTL_NETPLAN_CMD = [
+@@ -380,9 +381,14 @@ class TestNetplanDBus(unittest.TestCase):
+         err = self._check_dbus_error(BUSCTL_NETPLAN_CMD)
+         self.assertIn('Unknown object \'/io/netplan/Netplan/config/{}\''.format(cid), err)
+ 
++        # Verify the backup and config state dir are gone
++        self.assertFalse(os.path.isdir(backup))
++        self.assertFalse(os.path.isdir(tmpdir))
++
+     def test_netplan_dbus_config_apply(self):
+         cid = self._new_config_object()
+         tmpdir = '/tmp/netplan-config-{}'.format(cid)
++        backup = '/tmp/netplan-config-BACKUP'
+         with open(os.path.join(tmpdir, 'etc', 'netplan', 'apply_test.yaml'), 'w') as f:
+             f.write('TESTING-apply')
+         with open(os.path.join(tmpdir, 'lib', 'netplan', 'apply_test.yaml'), 'w') as f:
+@@ -400,7 +406,7 @@ class TestNetplanDBus(unittest.TestCase):
+         ]
+         out = subprocess.check_output(BUSCTL_NETPLAN_CMD)
+         self.assertEqual(b'b true\n', out)
+-        self.assertEquals(self.mock_netplan_cmd.calls(), [["netplan", "apply"]])
++        self.assertEquals(self.mock_netplan_cmd.calls(), [["netplan", "apply", "--state=/tmp/netplan-config-BACKUP"]])
+         time.sleep(1)  # Give some time for 'Apply' to clean up
+         self.assertFalse(os.path.isdir(tmpdir))
+ 
+@@ -413,6 +419,10 @@ class TestNetplanDBus(unittest.TestCase):
+         err = self._check_dbus_error(BUSCTL_NETPLAN_CMD)
+         self.assertIn('Unknown object \'/io/netplan/Netplan/config/{}\''.format(cid), err)
+ 
++        # Verify the backup and config state dir are gone
++        self.assertFalse(os.path.isdir(backup))
++        self.assertFalse(os.path.isdir(tmpdir))
++
+     def test_netplan_dbus_config_try_cancel(self):
+         # self-terminate after 30 dsec = 3 sec, if not cancelled before
+         self.mock_netplan_cmd.set_timeout(30)
+@@ -478,7 +488,8 @@ class TestNetplanDBus(unittest.TestCase):
+         self.assertIn('Unknown object \'/io/netplan/Netplan/config/{}\''.format(cid), err)
+ 
+         # Verify 'netplan try' has been called
+-        self.assertEquals(self.mock_netplan_cmd.calls(), [["netplan", "try", "--timeout=3"]])
++        self.assertEquals(self.mock_netplan_cmd.calls(),
++                          [["netplan", "try", "--timeout=3", "--state=/tmp/netplan-config-BACKUP"]])
+ 
+     def test_netplan_dbus_config_try_cb(self):
+         self.mock_netplan_cmd.set_timeout(1)  # actually self-terminate after 0.1 sec
+@@ -503,7 +514,7 @@ class TestNetplanDBus(unittest.TestCase):
+         self.assertEqual(b'b true\n', out)
+         time.sleep(1.5)  # Give some time for the timeout to happen
+ 
+-        # Verify the backup andconfig state dir are gone
++        # Verify the backup and config state dir are gone
+         self.assertFalse(os.path.isdir(backup))
+         self.assertFalse(os.path.isdir(tmpdir))
+ 
+@@ -518,7 +529,8 @@ class TestNetplanDBus(unittest.TestCase):
+         self.assertIn('Unknown object \'/io/netplan/Netplan/config/{}\''.format(cid), err)
+ 
+         # Verify 'netplan try' has been called
+-        self.assertEquals(self.mock_netplan_cmd.calls(), [["netplan", "try", "--timeout=1"]])
++        self.assertEquals(self.mock_netplan_cmd.calls(),
++                          [["netplan", "try", "--timeout=1", "--state=/tmp/netplan-config-BACKUP"]])
+ 
+     def test_netplan_dbus_config_try_apply(self):
+         self.mock_netplan_cmd.set_timeout(30)  # 30 dsec = 3 sec
+@@ -639,7 +651,7 @@ class TestNetplanDBus(unittest.TestCase):
+              "--root-dir=/tmp/netplan-config-{}".format(cid)],
+             ["netplan", "set", "ethernets.eth0.dhcp4=yes", "--origin-hint=70-snapd",
+              "--root-dir=/tmp/netplan-config-{}".format(cid)],
+-            ["netplan", "apply"]
++            ["netplan", "apply", "--state=/tmp/netplan-config-BACKUP"]
+         ])
+ 
+         # Now it works again
+@@ -756,7 +768,7 @@ class TestNetplanDBus(unittest.TestCase):
+         self.assertEquals(self.mock_netplan_cmd.calls(), [
+             ["netplan", "set", "ethernets.eth0.dhcp4=true", "--origin-hint=70-snapd",
+              "--root-dir=/tmp/netplan-config-{}".format(cid)],
+-            ["netplan", "try", "--timeout=1"],
++            ["netplan", "try", "--timeout=1", "--state=/tmp/netplan-config-BACKUP"],
+             ["netplan", "set", "ethernets.eth0.dhcp4=false", "--origin-hint=70-snapd",
+              "--root-dir=/tmp/netplan-config-{}".format(cid2)]
+         ])
+diff --git a/tests/integration/base.py b/tests/integration/base.py
+index 1054059..ec72d3c 100644
+--- a/tests/integration/base.py
++++ b/tests/integration/base.py
+@@ -287,11 +287,14 @@ class IntegrationTestsBase(unittest.TestCase):
+         if 'bond' not in iface:
+             self.assertIn('state UP', out)
+ 
+-    def generate_and_settle(self, wait_interfaces=None):
++    def generate_and_settle(self, wait_interfaces=None, state_dir=None):
+         '''Generate config, launch and settle NM and networkd'''
+ 
+         # regenerate netplan config
+-        out = subprocess.check_output(['netplan', 'apply'], stderr=subprocess.STDOUT, universal_newlines=True)
++        cmd = ['netplan', 'apply']
++        if state_dir:
++            cmd = cmd + ['--state', state_dir]
++        out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True)
+         if 'Run \'systemctl daemon-reload\' to reload units.' in out:
+             self.fail('systemd units changed without reload')
+         # start NM so that we can verify that it does not manage anything
+diff --git a/tests/integration/scenarios.py b/tests/integration/scenarios.py
+index 93f8a4a..e37dd18 100644
+--- a/tests/integration/scenarios.py
++++ b/tests/integration/scenarios.py
+@@ -22,8 +22,11 @@
+ # You should have received a copy of the GNU General Public License
+ # along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ 
++import os
++import shutil
+ import sys
+ import subprocess
++import tempfile
+ import unittest
+ 
+ from base import IntegrationTestsBase, test_backends
+@@ -107,6 +110,30 @@ class _CommonTests():
+         self.assert_iface_up(self.dev_e2_client, ['master br1'], ['inet '])
+         self.assert_iface_up('bond0', ['master br0'])
+ 
++    # https://bugs.launchpad.net/netplan/+bug/1943120
++    def test_remove_virtual_interfaces(self):
++        tempdir = tempfile.mkdtemp()
++        self.addCleanup(shutil.rmtree, tempdir)
++        self.addCleanup(subprocess.call, ['ip', 'link', 'delete', 'br54'], stderr=subprocess.DEVNULL)
++        confdir = os.path.join(tempdir, 'etc', 'netplan')
++        os.makedirs(confdir)
++        with open(self.config, 'w') as f:
++                    f.write('''network:
++  renderer: %(r)s
++  version: 2
++  bridges:
++    br54:
++      addresses: [1.2.3.4/24]''' % {'r': self.backend})
++        self.generate_and_settle(['br54'])
++        self.assert_iface('br54', ['inet 1.2.3.4/24'])
++        # backup the current YAML state (incl. br54)
++        shutil.copytree('/etc/netplan', confdir, dirs_exist_ok=True)
++        # drop br54 interface
++        subprocess.check_call(['netplan', 'set', 'network.bridges.br54.addresses=null'])
++        self.generate_and_settle([], state_dir=tempdir)
++        res = subprocess.run(['ip', 'link', 'show', 'dev', 'br54'], capture_output=True, text=True)
++        self.assertIn('not exist', res.stderr)
++
+ 
+ @unittest.skipIf("networkd" not in test_backends,
+                      "skipping as networkd backend tests are disabled")
+diff --git a/tests/test_cli_units.py b/tests/test_cli_units.py
+index 0814c18..90075e1 100644
+--- a/tests/test_cli_units.py
++++ b/tests/test_cli_units.py
+@@ -18,7 +18,9 @@
+ # along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ 
+ import unittest
++import subprocess
+ 
++from unittest.mock import patch
+ from netplan.cli.commands.apply import NetplanApply
+ 
+ 
+@@ -39,3 +41,41 @@ class TestCLI(unittest.TestCase):
+     def test_is_composite_member_with_renderer(self):
+         res = NetplanApply.is_composite_member([{'renderer': 'networkd', 'br0': {'interfaces': ['eth0']}}], 'eth0')
+         self.assertTrue(res)
++
++    @patch('subprocess.check_call')
++    def test_clear_virtual_links(self, mock):
++        # simulate as if 'tun3' would have already been delete another way,
++        # e.g. via NetworkManager backend
++        res = NetplanApply.clear_virtual_links(['br0', 'vlan2', 'bond1', 'tun3'],
++                                               ['br0', 'vlan2'],
++                                               devices=['br0', 'vlan2', 'bond1', 'eth0'])
++        mock.assert_called_with(['ip', 'link', 'delete', 'dev', 'bond1'])
++        self.assertIn('bond1', res)
++        self.assertIn('tun3', res)
++        self.assertNotIn('br0', res)
++        self.assertNotIn('vlan2', res)
++
++    @patch('subprocess.check_call')
++    def test_clear_virtual_links_failure(self, mock):
++        mock.side_effect = subprocess.CalledProcessError(1, '', 'Cannot find device "br0"')
++        res = NetplanApply.clear_virtual_links(['br0'], [], devices=['br0', 'eth0'])
++        mock.assert_called_with(['ip', 'link', 'delete', 'dev', 'br0'])
++        self.assertIn('br0', res)
++        self.assertNotIn('eth0', res)
++
++    @patch('subprocess.check_call')
++    def test_clear_virtual_links_no_delta(self, mock):
++        res = NetplanApply.clear_virtual_links(['br0', 'vlan2'],
++                                               ['br0', 'vlan2'],
++                                               devices=['br0', 'vlan2', 'eth0'])
++        mock.assert_not_called()
++        self.assertEquals(res, [])
++
++    @patch('subprocess.check_call')
++    def test_clear_virtual_links_no_devices(self, mock):
++        with self.assertLogs('', level='INFO') as ctx:
++            res = NetplanApply.clear_virtual_links(['br0', 'br1'],
++                                                   ['br0'])
++            self.assertEquals(res, [])
++            self.assertEqual(ctx.output, ['WARNING:root:Cannot clear virtual links: no network interfaces provided.'])
++        mock.assert_not_called()
+diff --git a/tests/test_configmanager.py b/tests/test_configmanager.py
+index e910f4e..625a1f1 100644
+--- a/tests/test_configmanager.py
++++ b/tests/test_configmanager.py
+@@ -151,6 +151,12 @@ class TestConfigManager(unittest.TestCase):
+         self.assertEquals(2, self.configmanager.version)
+         self.assertEquals('networkd', self.configmanager.renderer)
+         self.assertIn('fallback', self.configmanager.nm_devices)
++        self.assertIn('vlan2', self.configmanager.virtual_interfaces)
++        self.assertIn('br3', self.configmanager.virtual_interfaces)
++        self.assertIn('br4', self.configmanager.virtual_interfaces)
++        self.assertIn('bond5', self.configmanager.virtual_interfaces)
++        self.assertIn('bond6', self.configmanager.virtual_interfaces)
++        self.assertIn('he-ipv6', self.configmanager.virtual_interfaces)
+ 
+     def test_parse_merging(self):
+         self.configmanager.parse(extra_config=[os.path.join(self.workdir.name, "newfile_merging.yaml")])
diff --git a/debian/patches/0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch b/debian/patches/0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch
new file mode 100644
index 0000000..50e9e1c
--- /dev/null
+++ b/debian/patches/0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch
@@ -0,0 +1,368 @@
+From: Simon Chopin <[e-mail]>
+Date: Mon, 4 Oct 2021 14:18:50 +0200
+Subject: netplan: set: make it possible to unset a whole devtype subtree (LP:
+ #1942930) (FR-1685) (#236)
+MIME-Version: 1.0
+Content-Type: text/plain; charset="utf-8"
+Content-Transfer-Encoding: 8bit
+
+BACKPORT (from upstream): moving names.c ENUM_FUNCTION logic into util.c
+
+Previously, trying to unset a whole devtype subtree, such as netplan set network.ethernets=null would fail. This PR fixes it by detecting this special case and manually deleting each netdef assigned to the subtype.
+
+This approach is far from being performant, but it has the merit of working without needing too much new code.
+
+COMMITS:
+* tests: cli_get_set: use assertRaises to check for exceptions
+This allows the test machinery to do some better error reporting when an
+exception is raised, as it understands that the issue is an uncaught
+exception, not a failed comparison.
+
+* lib: names: add a function to parse a string into a devtype
+As I'm expecting this kind of function to be useful for the parser, the
+implementation is directly in the form of a macro modelled after its
+counterpart.
+
+* lib,netplan: tell Python about all the netdefs for a given devtype
+This is implemented via iterators to limit the knowledge Python has of
+the underlying memory model. The new symbols are prefixed with '_' to
+denote that they are not to be used by public consumers.
+
+We also expose process_yaml_hierarchy which was already part of the ABI,
+as it seems needed for the Python helper implementation.
+
+Contrary to the usual FFI calls, this one explicitly checks the
+existence of the symbols and raises an exception if the symbols are not
+found, as it is not totally unlikely that netplan.io and libnetplan0 be
+upgraded out of sync.
+
+V2: rename the C internal iterator struct, which hadn't evolved along
+with the successive iterations on the code
+
+fixup itar
+
+* netplan: set: allow the removal of an entire devtype subtree
+See LP: #1942930
+
+Note that performance for this are horrendous, as we parse the whole
+tree once for *each* removed netdef!
+
+V2:
+* Lukas as co-author, as he wrote the test!
+* Clean up commented-out code in the tests
+
+Co-authored-by: Lukas Märdian <[e-mail]>
+---
+ netplan/cli/commands/set.py |  9 +++++++-
+ netplan/cli/utils.py        | 46 +++++++++++++++++++++++++++++++++++++
+ src/util.c                  | 55 ++++++++++++++++++++++++++++++++++++++++++++
+ tests/test_cli_get_set.py   | 56 ++++++++++++++++++++++++++++-----------------
+ tests/test_utils.py         | 33 ++++++++++++++++++++++++++
+ 5 files changed, 177 insertions(+), 22 deletions(-)
+
+diff --git a/netplan/cli/commands/set.py b/netplan/cli/commands/set.py
+index 3bf7dc6..2d75cac 100644
+--- a/netplan/cli/commands/set.py
++++ b/netplan/cli/commands/set.py
+@@ -59,7 +59,14 @@ class NetplanSet(utils.NetplanCommand):
+         for devtype in network:
+             if devtype in GLOBAL_KEYS:
+                 continue  # special handling of global keys down below
+-            for netdef in network.get(devtype, []):
++            devtype_content = network.get(devtype, [])
++            # Special case: removal of a whole devtype.
++            # We replace the devtype null node with a dict of all defined netdefs
++            # set to None.
++            if devtype_content is None:
++                devtype_content = {dev: None for dev in utils.netplan_get_ids_for_devtype(devtype, self.root_dir)}
++                network[devtype] = devtype_content
++            for netdef in devtype_content:
+                 hint = FALLBACK_HINT
+                 filename = utils.netplan_get_filename_by_id(netdef, self.root_dir)
+                 if filename:
+diff --git a/netplan/cli/utils.py b/netplan/cli/utils.py
+index b0c87c9..656d60f 100644
+--- a/netplan/cli/utils.py
++++ b/netplan/cli/utils.py
+@@ -32,6 +32,10 @@ NM_SERVICE_NAME = 'NetworkManager.service'
+ NM_SNAP_SERVICE_NAME = 'snap.network-manager.networkmanager.service'
+ 
+ 
++class LibNetplanException(Exception):
++    pass
++
++
+ class _GError(ctypes.Structure):
+     _fields_ = [("domain", ctypes.c_uint32), ("code", ctypes.c_int), ("message", ctypes.c_char_p)]
+ 
+@@ -39,6 +43,8 @@ class _GError(ctypes.Structure):
+ lib = ctypes.CDLL(ctypes.util.find_library('netplan'))
+ lib.netplan_parse_yaml.argtypes = [ctypes.c_char_p, ctypes.POINTER(ctypes.POINTER(_GError))]
+ lib.netplan_get_filename_by_id.restype = ctypes.c_char_p
++lib.process_yaml_hierarchy.argtypes = [ctypes.c_char_p]
++lib.process_yaml_hierarchy.restype = ctypes.c_int
+ 
+ 
+ def netplan_parse(path):
+@@ -59,6 +65,46 @@ def netplan_get_filename_by_id(netdef_id, rootdir):
+     return res.decode('utf-8') if res else None
+ 
+ 
++class _NetdefIdIterator:
++    def __init__(self, devtype):
++        self.iterator = lib._netplan_iter_defs_per_devtype_init(devtype.encode('utf-8'))
++
++    def __del__(self):
++        lib._netplan_iter_defs_per_devtype_free(self.iterator)
++
++    def __iter__(self):
++        return self
++
++    def __next__(self):
++        next_value = lib._netplan_iter_defs_per_devtype_next(self.iterator)
++        if next_value is None:
++            raise StopIteration
++        return next_value
++
++
++def netplan_get_ids_for_devtype(devtype, rootdir):
++    if not hasattr(lib, '_netplan_iter_defs_per_devtype_init'):  # pragma: nocover (hard to unit-test against the WRONG lib)
++        raise LibNetplanException('''
++            The current version of libnetplan does not allow iterating by devtype.
++            Please ensure that both the netplan CLI package and its library are up to date.
++        ''')
++    lib._netplan_iter_defs_per_devtype_init.argtypes = [ctypes.c_char_p]
++    lib._netplan_iter_defs_per_devtype_init.restype = ctypes.c_void_p
++
++    lib._netplan_iter_defs_per_devtype_next.argtypes = [ctypes.c_void_p]
++    lib._netplan_iter_defs_per_devtype_next.restype = ctypes.c_void_p
++
++    lib._netplan_iter_defs_per_devtype_free.argtypes = [ctypes.c_void_p]
++    lib._netplan_iter_defs_per_devtype_free.restype = None
++
++    lib._netplan_netdef_id.argtypes = [ctypes.c_void_p]
++    lib._netplan_netdef_id.restype = ctypes.c_char_p
++
++    lib.process_yaml_hierarchy(rootdir.encode('utf-8'))
++    nds = list(_NetdefIdIterator(devtype))
++    return [lib._netplan_netdef_id(nd).decode('utf-8') for nd in nds]
++
++
+ def get_generator_path():
+     return os.environ.get('NETPLAN_GENERATE_PATH', '/lib/netplan/generate')
+ 
+diff --git a/src/util.c b/src/util.c
+index a4c0dba..4c8d716 100644
+--- a/src/util.c
++++ b/src/util.c
+@@ -327,3 +327,58 @@ get_global_network(int ip_family)
+     else
+         return "::/0";
+ }
++
++#define ENUM_FUNCTION(_radical, _type) _type netplan_ ## _radical ## _from_name(const char* val) \
++{ \
++    for (int i = 0; i < sizeof(netplan_ ## _radical ## _to_str); ++i) { \
++        if (g_strcmp0(val, netplan_ ## _radical ## _to_str[i]) == 0) \
++            return i; \
++    } \
++    return -1; \
++}
++
++ENUM_FUNCTION(def_type, NetplanDefType);
++
++struct netdef_pertype_iterator {
++    NetplanDefType type;
++    GHashTableIter iter;
++};
++
++struct netdef_pertype_iterator*
++_netplan_iter_defs_per_devtype_init(const char *devtype)
++{
++    NetplanDefType type = netplan_def_type_from_name(devtype);
++    struct netdef_pertype_iterator *iter = g_malloc0(sizeof(*iter));
++    iter->type = type;
++    if (netdefs)
++        g_hash_table_iter_init(&iter->iter, netdefs);
++    return iter;
++}
++
++NetplanNetDefinition*
++_netplan_iter_defs_per_devtype_next(struct netdef_pertype_iterator* it)
++{
++    gpointer key, value;
++
++    if (!netdefs)
++        return NULL;
++
++    while (g_hash_table_iter_next(&it->iter, &key, &value)) {
++        NetplanNetDefinition* netdef = value;
++        if (netdef->type == it->type)
++            return netdef;
++    }
++    return NULL;
++}
++
++void
++_netplan_iter_defs_per_devtype_free(struct netdef_pertype_iterator* it)
++{
++    g_free(it);
++}
++
++const char*
++_netplan_netdef_id(NetplanNetDefinition* nd)
++{
++    return nd->id;
++}
+diff --git a/tests/test_cli_get_set.py b/tests/test_cli_get_set.py
+index 7a1799b..b5206e7 100644
+--- a/tests/test_cli_get_set.py
++++ b/tests/test_cli_get_set.py
+@@ -31,13 +31,11 @@ from netplan.cli.core import Netplan
+ def _call_cli(args):
+     old_sys_argv = sys.argv
+     sys.argv = [old_sys_argv[0]] + args
++    f = io.StringIO()
+     try:
+-        f = io.StringIO()
+         with redirect_stdout(f):
+             Netplan().main()
+             return f.getvalue()
+-    except Exception as e:
+-        return e
+     finally:
+         sys.argv = old_sys_argv
+ 
+@@ -110,20 +108,20 @@ class TestSet(unittest.TestCase):
+             self.assertEquals('network:\n  ethernets:\n    eth0:\n      dhcp4: true\n', f.read())
+ 
+     def test_set_empty_origin_hint(self):
+-        err = self._set(['ethernets.eth0.dhcp4=true', '--origin-hint='])
+-        self.assertIsInstance(err, Exception)
+-        self.assertIn('Invalid/empty origin-hint', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set(['ethernets.eth0.dhcp4=true', '--origin-hint='])
++        self.assertTrue('Invalid/empty origin-hint' in str(context.exception))
+ 
+     def test_set_invalid(self):
+-        err = self._set(['xxx.yyy=abc'])
+-        self.assertIsInstance(err, Exception)
+-        self.assertIn('unknown key \'xxx\'\n  xxx:\n', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set(['xxx.yyy=abc'])
++        self.assertIn('unknown key \'xxx\'\n  xxx:\n', str(context.exception))
+         self.assertFalse(os.path.isfile(self.path))
+ 
+     def test_set_invalid_validation(self):
+-        err = self._set(['ethernets.eth0.set-name=myif0'])
+-        self.assertIsInstance(err, Exception)
+-        self.assertIn('eth0: \'set-name:\' requires \'match:\' properties', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set(['ethernets.eth0.set-name=myif0'])
++        self.assertIn('eth0: \'set-name:\' requires \'match:\' properties', str(context.exception))
+         self.assertFalse(os.path.isfile(self.path))
+ 
+     def test_set_invalid_validation2(self):
+@@ -134,9 +132,9 @@ class TestSet(unittest.TestCase):
+       mode: sit
+       local: 1.2.3.4
+       remote: 5.6.7.8''')
+-        err = self._set(['tunnels.tun0.keys.input=12345'])
+-        self.assertIsInstance(err, Exception)
+-        self.assertIn('tun0: \'input-key\' is not required for this tunnel type', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set(['tunnels.tun0.keys.input=12345'])
++        self.assertIn('tun0: \'input-key\' is not required for this tunnel type', str(context.exception))
+ 
+     def test_set_append(self):
+         with open(self.path, 'w') as f:
+@@ -195,6 +193,20 @@ class TestSet(unittest.TestCase):
+             self.assertNotIn('addresses:', out)
+             self.assertNotIn('eth0:', out)
+ 
++    def test_set_delete_subtree(self):
++        with open(self.path, 'w') as f:
++            f.write('''network:\n  version: 2\n  renderer: NetworkManager
++  ethernets:
++    eth0: {addresses: [1.2.3.4/24]}''')
++        self._set(['network.ethernets=null'])
++        self.assertTrue(os.path.isfile(self.path))
++        with open(self.path, 'r') as f:
++            out = f.read()
++        self.assertIn('network:\n', out)
++        self.assertIn(' version: 2\n', out)
++        self.assertIn(' renderer: NetworkManager\n', out)
++        self.assertNotIn('ethernets:', out)
++
+     def test_set_delete_file(self):
+         with open(self.path, 'w') as f:
+             f.write('''network:
+@@ -220,9 +232,9 @@ class TestSet(unittest.TestCase):
+             f.write('''network:\n  version: 2\n  renderer: NetworkManager
+   ethernets:
+     eth0: {addresses: [1.2.3.4]}''')
+-        err = self._set(['ethernets.eth0.addresses'])
+-        self.assertIsInstance(err, Exception)
+-        self.assertEquals('Invalid value specified', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set(['ethernets.eth0.addresses'])
++        self.assertEquals('Invalid value specified', str(context.exception))
+ 
+     def test_set_escaped_dot(self):
+         self._set([r'ethernets.eth0\.123.dhcp4=false'])
+@@ -231,9 +243,11 @@ class TestSet(unittest.TestCase):
+             self.assertIn('network:\n  ethernets:\n    eth0.123:\n      dhcp4: false', f.read())
+ 
+     def test_set_invalid_input(self):
+-        err = self._set([r'ethernets.eth0={dhcp4:false}'])
+-        self.assertIsInstance(err, Exception)
+-        self.assertEquals('Invalid input: {\'network\': {\'ethernets\': {\'eth0\': {\'dhcp4:false\': None}}}}', str(err))
++        with self.assertRaises(Exception) as context:
++            self._set([r'ethernets.eth0={dhcp4:false}'])
++        self.assertEquals(
++                'Invalid input: {\'network\': {\'ethernets\': {\'eth0\': {\'dhcp4:false\': None}}}}',
++                str(context.exception))
+ 
+     def test_set_override_existing_file(self):
+         override = os.path.join(self.workdir.name, 'etc', 'netplan', 'some-file.yaml')
+diff --git a/tests/test_utils.py b/tests/test_utils.py
+index 7954ec7..5c97ca2 100644
+--- a/tests/test_utils.py
++++ b/tests/test_utils.py
+@@ -196,3 +196,36 @@ class TestUtils(unittest.TestCase):
+       remote: 0.0.0.0
+       key: 0.0.0.0''')
+         self.assertIsNone(utils.netplan_get_filename_by_id('some-id', self.workdir.name))
++
++    def test_netplan_get_ids_for_devtype(self):
++        path = os.path.join(self.workdir.name, 'etc/netplan/a.yaml')
++        with open(path, 'w') as f:
++            f.write('''network:
++  ethernets:
++    id_b:
++      dhcp4: true
++    id_a:
++      dhcp4: true
++  vlans:
++    en-intra:
++      id: 3
++      link: id_b
++      dhcp4: true''')
++        self.assertSetEqual(
++                set(utils.netplan_get_ids_for_devtype("ethernets", self.workdir.name)),
++                set(["id_a", "id_b"]))
++
++    def test_netplan_get_ids_for_devtype_no_dev(self):
++        path = os.path.join(self.workdir.name, 'etc/netplan/a.yaml')
++        with open(path, 'w') as f:
++            f.write('''network:
++  ethernets:
++    id_b:
++      dhcp4: true''')
++        self.assertSetEqual(
++                set(utils.netplan_get_ids_for_devtype("tunnels", self.workdir.name)),
++                set([]))
++
++    def test_NetdefIdIterator_with_clear_netplan(self):
++        utils.lib.netplan_clear_netdefs()
++        self.assertSequenceEqual(list(utils._NetdefIdIterator("ethernets")), [])
diff --git a/debian/patches/series b/debian/patches/series
index 28914f9..475bdf7 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -3,3 +3,5 @@
 0003-Mute-gateway4-6-deprecation-warnings.patch
 autopkgtest-fixes.patch
 nm-1.32.10-compat.patch
+0005-Implement-YAML-state-tracking-and-use-it-in-the-DBus.patch
+0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch