Merge ~calvinmwadime/ubuntu/+source/walinuxagent:bionic-2.9.1.1 into ubuntu/+source/walinuxagent:ubuntu/bionic-devel

[Calvin Mwadime Makokha]

[Lucas Kanashiro (lucaskanashiro)]

See my comments in the mantic MP.

review: Needs Fixing

[Lukas Märdian (slyon)]

Please address comments from https://code.launchpad.net/~calvinmwadime/ubuntu/+source/walinuxagent/+git/walinuxagent/+merge/461115/comments/1242656 and re-request ~ubuntu-sponsors review afterwards.

Unmerged commits

9a2106f... by Calvin Mwadime Makokha on 2023-12-15

debian/changelog: Update version

de5dc3e... by Calvin Mwadime Makokha on 2023-11-30

d/install: move remaining things to /usr

4bc466b... by Calvin Mwadime Makokha on 2023-11-30

d/control: fix lintian priority-extra-is-replaced-by-priority-optional

a5ec50e... by Calvin Mwadime Makokha on 2023-11-30

d/control: add iproute2 build depdency (test only)

b4817c2... by Calvin Mwadime Makokha on 2023-11-30

d/rules: stop using the old systemd dh sequence

Matching our switch to dh_installsystemd we need to stop calling
for the older systemd-sequence in dh.

This fixes:
error: The systemd-sequence is no longer provided in compat >= 11,
please rely on dh_installsystemd instead

88304be... by Calvin Mwadime Makokha on 2023-11-30

d/control: drop old unneeded X-Python3-Version statement

c9c0e6a... by Calvin Mwadime Makokha on 2023-11-30

d/rules, d/install: install services once

Since "7d793bf debian: Remove upstart config" --no-restart-on-upgrade
behavior was lost. Add it back with the new name --no-stop-on-upgrade.
Furthermore the install was a mix of the upstream putting files
under /lib and then debian/install putting it under /usr/lib leading
to file-in-root-and-usr warnings.
Furthermore dh_systemd_enable and the formerly dropped dh_installinit
have been consumed into dh_installsystemd for systemd units.
Finally dh_installsystemd will drop the executable bit which is
wrong in the upstream archive and triggers executable-not-elf-or-script.
So overall to hopefully do it right we do:
- ephemeral-disk-warning.service gets installed by d/install
- walinuxagent.service gets installed badly by upstream and removed in
  d/rules
- both services are then picked up and configured by dh_installsystemd

eb9ef58... by Calvin Mwadime Makokha on 2023-11-30

d/control, d/compat: switch to compat level 13

638ab36... by Calvin Mwadime Makokha on 2024-01-30

d/manpages: add man pages.

2b6ece0... by Calvin Mwadime Makokha on 2023-11-27

debian/patches: Update the cgroup handling to include v2

WALA only supported cgroup v1 systems. This causes failures when running some
commands on cgroup v2 systems. Newer ubuntu versions are support cgroup v2 by
default

diff --git a/debian/changelog b/debian/changelog
index 42e8156..d25a30e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,32 @@
+walinuxagent (2.9.1.1-0ubuntu1) bionic; urgency=medium
+
+  [ Calvin Mwadime ]
+  * New upstream version 2.9.1.1 (LP: #2035331)
+  * debian/watch: Fix package version regex pattern
+  * debian/docs: Remove the changelog
+  * debian/install: add new udev rules
+  * debian/patches:
+    - Remove deprecated patches
+    - Update list of supported ubuntu versions
+    - Correctly point to the right dhcp client lease file
+    - Update the cgroup handling to include v2
+  * disable_udev_overrides.patch: Remove udev overrides in setup.py
+  * debian: Make python unit tests work
+  * debian: Remove upstart config
+  * debian/control: Remove isc-dhcp-client
+  * debian/manpages: Added manpages
+
+  [ Christian Ehrhardt ]
+  * d/control, d/compat: switch to compat level 13
+  * d/rules, d/install: install services once
+  * d/control: drop old unneeded X-Python3-Version statement
+  * d/rules: stop using the old systemd dh sequence
+  * d/control: add iproute2 build dependency (test only)
+  * d/control: fix lintian priority-extra-is-replaced-by-priority-optional
+  * d/install: move remaining things to /usr
+
+ -- Calvin Mwadime <[e-mail]>  Thu, 30 Nov 2023 11:07:05 +0300
+
 walinuxagent (2.2.45-0ubuntu1~18.04.3) bionic; urgency=medium
 
   * Add patch to fix handling of gen2 disks
diff --git a/debian/compat b/debian/compat
deleted file mode 100644
index 45a4fb7..0000000
--- a/debian/compat
+++ /dev/null
@@ -1 +0,0 @@
-8
diff --git a/debian/control b/debian/control
index 89403a4..cb3f700 100644
--- a/debian/control
+++ b/debian/control
@@ -1,23 +1,31 @@
 Source: walinuxagent
 Section: python
-Priority: extra
+Priority: optional
 Maintainer: Ubuntu Developers <[e-mail]>
 XSBC-Original-Maintainer: Microsoft Corporation <[e-mail]>
-Build-Depends: debhelper (>= 8),
-               dh-systemd,
+Build-Depends: debhelper-compat (= 13),
+               dh-python,
+               debhelper (>= 9.20160709) | dh-systemd,
+               go-md2man,
+               iproute2,
                python3-all,
                python3-setuptools,
                python3-pyasn1,
                python3-mock,
+               python3-nose,
+               python3-nose-timer,
+               python3-wrapt,
+               pylint,
+               python3-distro,
+               python3-coverage,
+               iptables,
                openssl (>= 1.0),
 Standards-Version: 3.9.8
-X-Python3-Version: >= 3.2
 Homepage: http://go.microsoft.com/fwlink/?LinkId=250998
 
 Package: walinuxagent
 Architecture: amd64 arm64
 Depends: cloud-init (>= 0.7.3~bzr826-0ubuntu2),
-         isc-dhcp-client,
          openssh-server (>= 1:5.9p1),
          openssl (>= 1.0),
          passwd (>= 4.1.4.2),
diff --git a/debian/docs b/debian/docs
index ae1ac0d..5cc071c 100644
--- a/debian/docs
+++ b/debian/docs
@@ -2,4 +2,3 @@ debian/99-cloud-init-disable-diskprovisioning.conf
 NOTICE
 README.md
 MANIFEST
-Changelog
diff --git a/debian/ephemeral-disk-warning.conf b/debian/ephemeral-disk-warning.conf
deleted file mode 100644
index 7f60cfe..0000000
--- a/debian/ephemeral-disk-warning.conf
+++ /dev/null
@@ -1,55 +0,0 @@
-# ephemeral-disk-warning - warns user that the disk is really, really ephemeral
-#
-# On Azure, the ephemeral disk is extremely ephemeral; the ephemeral disk is
-# unsafe between boots. This places a file on /mnt that warns the user
-# that the disk is a dangerous place for storing data of any importance.
-
-env RESOURCE_DISK=/dev/disk/azure/resource-part1
-
-start on (stopped rc RUNLEVEL=[2345] and stopped cloud-config)
-task
-script
-    if [ ! -e $RESOURCE_DISK ]; then
-            logger "Disk $RESOURCE_DISK does not exist, skipping ephemeral warning"
-            exit 0
-    fi
-
-    ephemeral_kdev=$(readlink -f $RESOURCE_DISK)
-    ephemeral_mp=$(awk '$1==kd {print$2}' "kd=$ephemeral_kdev" /proc/mounts)
-    warn_file="$ephemeral_mp/DATALOSS_WARNING_README.txt"
-
-    if [ -z "$ephemeral_mp" ]; then
-            logger "Unable to discover mount point of $ephemeral_kdev. Ephemeral warning will not be written"
-            exit 0
-    else
-            logger "Ephemeral disk $ephemeral_kdev located at $ephemeral_mp"
-    fi
-
-    if [ ! -e "$warn_file" ]; then
-        cat >> $warn_file <<EOF
-WARNING: THIS IS A TEMPORARY DISK.
-
-Any data stored on this drive is SUBJECT TO LOSS and THERE IS NO WAY TO
-RECOVER IT.
-
-Please do not use this disk for storing any personal or application data.
-
-For additional details to please refer to the MSDN documentation at:
-http://msdn.microsoft.com/en-us/library/windowsazure/jj672979.aspx
-
-To remove this warning run:
-    sudo chattr -i $warn_file
-    sudo rm $warn_file
-
-This warning is written each boot; to disable it:
-    echo "manual" | sudo tee /etc/init/ephemeral-disk-warning.override
-    sudo systemctl disable ephemeral-disk-warning.service
-
-EOF
-        chmod 0444 $warn_file
-        chattr +i $warn_file
-        logger "Added ephemeral disk warning to $warn_file"
-    fi
-    logger "WARNING: $ephemeral_mp is an ephemeral disk. See $warn_file for more information"
-
-end script
diff --git a/debian/install b/debian/install
index 8915177..98c4b97 100644
--- a/debian/install
+++ b/debian/install
@@ -1,6 +1,6 @@
-config/66-azure-storage.rules lib/udev/rules.d
-debian/ephemeral-disk-warning.service lib/systemd/system
-debian/ephemeral-disk-warning.conf etc/init
+config/66-azure-storage.rules usr/lib/udev/rules.d
+config/99-azure-product-uuid.rules usr/lib/udev/rules.d
+debian/ephemeral-disk-warning.service usr/lib/systemd/system
 debian/ephemeral-disk-warning usr/sbin
 debian/67-azure-console.rules lib/udev/rules.d
 debian/68-azure-nm-unmanaged.rules lib/udev/rules.d
diff --git a/debian/patches/disable_import_test.patch b/debian/patches/disable_import_test.patch
deleted file mode 100644
index 558a7be..0000000
--- a/debian/patches/disable_import_test.patch
+++ /dev/null
@@ -1,43 +0,0 @@
-Index: walinuxagent-2.2.32/config/waagent.conf
-===================================================================
---- walinuxagent-2.2.32.orig/config/waagent.conf
-+++ walinuxagent-2.2.32/config/waagent.conf
-@@ -3,7 +3,7 @@
- #
- 
- # Enable instance creation
--Provisioning.Enabled=y
-+Provisioning.Enabled=n
- 
- # Enable extension handling. Do not disable this unless you do not need password reset,
- # backup, monitoring, or any extension handling whatsoever.
-@@ -13,10 +13,10 @@ Extensions.Enabled=y
- Provisioning.UseCloudInit=n
- 
- # Password authentication for root account will be unavailable.
--Provisioning.DeleteRootPassword=y
-+Provisioning.DeleteRootPassword=n
- 
- # Generate fresh host key pair.
--Provisioning.RegenerateSshHostKeyPair=y
-+Provisioning.RegenerateSshHostKeyPair=n
- 
- # Supported values are "rsa", "dsa", "ecdsa", "ed25519", and "auto".
- # The "auto" option is supported on OpenSSH 5.9 (2011) and later.
-@@ -41,14 +41,14 @@ Provisioning.ExecuteCustomData=n
- Provisioning.AllowResetSysUser=n
- 
- # Format if unformatted. If 'n', resource disk will not be mounted.
--ResourceDisk.Format=y
-+ResourceDisk.Format=n
- 
- # File system on the resource disk
- # Typically ext3 or ext4. FreeBSD images should use 'ufs2' here.
- ResourceDisk.Filesystem=ext4
- 
- # Mount point for the resource disk
--ResourceDisk.MountPoint=/mnt/resource
-+ResourceDisk.MountPoint=/mnt
- 
- # Create and use swapfile on resource disk.
- ResourceDisk.EnableSwap=n
diff --git a/debian/patches/disable_udev_overrides.patch b/debian/patches/disable_udev_overrides.patch
index 7c344d0..cc25ba0 100644
--- a/debian/patches/disable_udev_overrides.patch
+++ b/debian/patches/disable_udev_overrides.patch
@@ -10,3 +10,19 @@
  VarLibDhcpDirectories = ["/var/lib/dhclient", "/var/lib/dhcpcd", "/var/lib/dhcp"]
  EtcDhcpClientConfFiles = ["/etc/dhcp/dhclient.conf", "/etc/dhcp3/dhclient.conf"]
  global LibDir
+
+Description: Remove udev overrides in setup.py
+Forwaded: not-needed
+Author: Calvin Mwadime Makokha <[e-mail]>
+Index: walinuxagent/setup.py
+===================================================================
+--- walinuxagent.orig/setup.py
++++ walinuxagent/setup.py
+@@ -82,8 +82,7 @@ def set_openbsd_rc_files(data_files, des
+
+ def set_udev_files(data_files, dest="/etc/udev/rules.d/", src=None):
+     if src is None:
+-        src = ["config/66-azure-storage.rules",
+-               "config/99-azure-product-uuid.rules"]
++        src = []
+     data_files.append((dest, src))
diff --git a/debian/patches/fix-udev-rules-for-azure.patch b/debian/patches/fix-udev-rules-for-azure.patch
deleted file mode 100644
index 497c74b..0000000
--- a/debian/patches/fix-udev-rules-for-azure.patch
+++ /dev/null
@@ -1,69 +0,0 @@
-From d33f1f810728dbe4e2891149936195d1ec7aaa27 Mon Sep 17 00:00:00 2001
-From: Thomas Stringer <[e-mail]>
-Date: Mon, 20 Jul 2020 14:24:22 -0400
-Subject: [PATCH] Fix handling of gen2 disks with udev rules (#1954)
-
----
- config/66-azure-storage.rules | 40 ++++++++++++++++++++---------------
- 1 file changed, 23 insertions(+), 17 deletions(-)
-
-Origin: upstream, https://github.com/Azure/WALinuxAgent/commit/d33f1f810728dbe4e2891149936195d1ec7aaa27
-Bug: https://github.com/Azure/WALinuxAgent/issues/1867
-Bug-Ubuntu: https://bugs.launchpad.net/ubuntu/+source/walinuxagent/+bug/1971141
-Applied-Upstream: https://github.com/Azure/WALinuxAgent/commit/d33f1f810728dbe4e2891149936195d1ec7aaa27 (v2.7.0.6)
-Last-Updated: 2022-05-12
-
---- a/config/66-azure-storage.rules
-+++ b/config/66-azure-storage.rules
-@@ -1,28 +1,34 @@
--ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="Msft", ENV{ID_MODEL}=="Virtual_Disk", GOTO="azure_disk"
--GOTO="azure_end"
-+# Azure specific rules.
-+ACTION!="add|change", GOTO="walinuxagent_end"
-+SUBSYSTEM!="block", GOTO="walinuxagent_end"
-+ATTRS{ID_VENDOR}!="Msft", GOTO="walinuxagent_end"
-+ATTRS{ID_MODEL}!="Virtual_Disk", GOTO="walinuxagent_end"
- 
--LABEL="azure_disk"
--# Root has a GUID of 0000 as the second value
--# The resource/resource has GUID of 0001 as the second value
--ATTRS{device_id}=="?00000000-0000-*", ENV{fabric_name}="root", GOTO="azure_names"
--ATTRS{device_id}=="?00000000-0001-*", ENV{fabric_name}="resource", GOTO="azure_names"
--ATTRS{device_id}=="?00000001-0001-*", ENV{fabric_name}="BEK", GOTO="azure_names"
--# Wellknown SCSI controllers
-+# Match the known ID parts for root and resource disks.
-+ATTRS{device_id}=="?00000000-0000-*", ENV{fabric_name}="root", GOTO="wa_azure_names"
-+ATTRS{device_id}=="?00000000-0001-*", ENV{fabric_name}="resource", GOTO="wa_azure_names"
-+
-+# Gen2 disk.
- ATTRS{device_id}=="{f8b3781a-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi0", GOTO="azure_datadisk"
-+# Create symlinks for data disks attached.
- ATTRS{device_id}=="{f8b3781b-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi1", GOTO="azure_datadisk"
- ATTRS{device_id}=="{f8b3781c-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi2", GOTO="azure_datadisk"
- ATTRS{device_id}=="{f8b3781d-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi3", GOTO="azure_datadisk"
--GOTO="azure_end"
-+GOTO="walinuxagent_end"
- 
--# Retrieve LUN number for datadisks
-+# Parse out the fabric n ame based off of scsi indicators.
- LABEL="azure_datadisk"
--ENV{DEVTYPE}=="partition", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/../device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result", GOTO="azure_names"
--PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result", GOTO="azure_names"
--GOTO="azure_end"
-+ENV{DEVTYPE}=="partition", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/../device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
-+ENV{DEVTYPE}=="disk", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
-+
-+ENV{fabric_name}=="scsi0/lun0", ENV{fabric_name}="root"
-+ENV{fabric_name}=="scsi0/lun1", ENV{fabric_name}="resource"
-+# Don't create a symlink for the cd-rom.
-+ENV{fabric_name}=="scsi0/lun2", GOTO="walinuxagent_end"
- 
--# Create the symlinks
--LABEL="azure_names"
-+# Create the symlinks.
-+LABEL="wa_azure_names"
- ENV{DEVTYPE}=="disk", SYMLINK+="disk/azure/$env{fabric_name}"
- ENV{DEVTYPE}=="partition", SYMLINK+="disk/azure/$env{fabric_name}-part%n"
- 
--LABEL="azure_end"
-+LABEL="walinuxagent_end"
diff --git a/debian/patches/fix_cgroup_v2_mounting_and_systemd_process.patch b/debian/patches/fix_cgroup_v2_mounting_and_systemd_process.patch
new file mode 100644
index 0000000..cbcdec2
--- /dev/null
+++ b/debian/patches/fix_cgroup_v2_mounting_and_systemd_process.patch
@@ -0,0 +1,232 @@
+Description: WALA was only supporting cgroup v1 which is not the default
+    in the latest ubuntu versions, which had cgroup v2 on. This led to
+    some tasks failing like periodic collection of the logs
+Forwarded: yes, https://github.com/Azure/WALinuxAgent/pull/2989
+Author: Calvin Mwadime Makokha <[e-mail]>
+
+--- walinuxagent-2.9.1.1.orig/azurelinuxagent/common/cgroup.py
++++ walinuxagent-2.9.1.1/azurelinuxagent/common/cgroup.py
+@@ -220,16 +220,25 @@ class CpuCgroup(CGroup):
+             with open(os.path.join(self.path, 'cpu.stat')) as cpu_stat:
+                 #
+                 # Sample file:
+-                #
++                #   In CGroup v1
+                 #   # cat /sys/fs/cgroup/cpuacct/azure.slice/walinuxagent.service/cpu.stat
+                 #   nr_periods  51660
+                 #   nr_throttled 19461
+                 #   throttled_time 1529590856339
++                #   In CGroup v2
++                #   # cat /sys/fs/cgroup/azure.slice/walinuxagent.service/cpu.stat
++                #   nr_periods  51660
++                #   nr_throttled 19461
++                #   throttled_usec 152959085
+                 #
+                 for line in cpu_stat:
+                     match = re.match(r'throttled_time\s+(\d+)', line)
++                    multiplier = 1
++                    if match is None:
++                        match = re.match(r'throttled_usec\s+(\d+)', line)
++                        multiplier = 100
+                     if match is not None:
+-                        return int(match.groups()[0])
++                        return int(match.groups()[0] * multiplier)
+                 raise Exception("Cannot find throttled_time")
+         except (IOError, OSError) as e:
+             if e.errno == errno.ENOENT:
+--- walinuxagent-2.9.1.1.orig/azurelinuxagent/common/cgroupapi.py
++++ walinuxagent-2.9.1.1/azurelinuxagent/common/cgroupapi.py
+@@ -171,10 +171,14 @@ class SystemdCgroupsApi(CGroupsApi):
+         The values returned can be None if the process is not in a cgroup for that controller (e.g. the controller is not mounted).
+         """
+         # The contents of the file are similar to
++        # In Cgroup v1
+         #    # cat /proc/1218/cgroup
+         #    10:memory:/system.slice/walinuxagent.service
+         #    3:cpu,cpuacct:/system.slice/walinuxagent.service
+         #    etc
++        # In Cgroup v2
++        #    # cat /proc/1218/cgroup
++        #    10::/system.slice/walinuxagent.service
+         cpu_path = None
+         memory_path = None
+         for line in fileutil.read_file("/proc/{0}/cgroup".format(process_id)).splitlines():
+@@ -186,6 +190,13 @@ class SystemdCgroupsApi(CGroupsApi):
+                     memory_path = path
+                 else:
+                     cpu_path = path
++            else:
++                match = re.match(r'\d+::(?P<path>.+)', line)
++                if match:
++                    path = match.group('path').lstrip('/') if match.group('path') != '/' else None
++                    cpu_path = path
++                    memory_path = path
++
+ 
+         return cpu_path, memory_path
+ 
+@@ -197,6 +208,20 @@ class SystemdCgroupsApi(CGroupsApi):
+         cpu_cgroup_relative_path, memory_cgroup_relative_path = self.get_process_cgroup_relative_paths(process_id)
+ 
+         cpu_mount_point, memory_mount_point = self.get_cgroup_mount_points()
++        if cpu_mount_point is None or memory_mount_point is None:
++            logger.warn("Cgroup v1 mount points not found")
++            cgroup2_mount_point, cgroup2_controllers = self.get_cgroup2_controllers()
++            if cgroup2_mount_point is not None:
++                controller_missing = False
++                if "memory" not in cgroup2_controllers or "-memory" in cgroup2_controllers:
++                    logger.warn("Memory controller not active in cgroup v2")
++                    controller_missing = True
++                if "cpu" not in cgroup2_controllers or "-cpu" in cgroup2_controllers:
++                    logger.warn("CPU controller not active in cgroup v2")
++                    controller_missing = True
++                if not controller_missing:
++                    cpu_mount_point = cgroup2_mount_point
++                    memory_mount_point = cgroup2_mount_point
+ 
+         cpu_cgroup_path = os.path.join(cpu_mount_point, cpu_cgroup_relative_path) \
+             if cpu_mount_point is not None and cpu_cgroup_relative_path is not None else None
+@@ -239,7 +264,8 @@ class SystemdCgroupsApi(CGroupsApi):
+             if match is not None:
+                 mount_point = match.group('path')
+                 controllers = None
+-                controllers_file = os.path.join(mount_point, 'cgroup.controllers')
++                # Check for enabled controllers only
++                controllers_file = os.path.join(mount_point, 'cgroup.subtree_control')
+                 if os.path.exists(controllers_file):
+                     controllers = fileutil.read_file(controllers_file)
+                 return mount_point, controllers
+--- walinuxagent-2.9.1.1.orig/azurelinuxagent/common/cgroupconfigurator.py
++++ walinuxagent-2.9.1.1/azurelinuxagent/common/cgroupconfigurator.py
+@@ -76,7 +76,7 @@ CPUQuota={cpu_quota}
+ MemoryAccounting=yes
+ """
+ _LOGCOLLECTOR_CPU_QUOTA = "5%"
+-LOGCOLLECTOR_MEMORY_LIMIT = 30 * 1024 ** 2  # 30Mb
++LOGCOLLECTOR_MEMORY_LIMIT = 40 * 1024 ** 2  # 40Mb
+ 
+ _AGENT_DROP_IN_FILE_SLICE = "10-Slice.conf"
+ _AGENT_DROP_IN_FILE_SLICE_CONTENTS = """
+@@ -299,24 +299,37 @@ class CGroupConfigurator(object):
+             # check v1 controllers
+             #
+             cpu_controller_root, memory_controller_root = self._cgroups_api.get_cgroup_mount_points()
++            controller_missing = False
+ 
+             if cpu_controller_root is not None:
+                 logger.info("The CPU cgroup controller is mounted at {0}", cpu_controller_root)
+             else:
+                 _log_cgroup_warning("The CPU cgroup controller is not mounted")
++                controller_missing = True
+ 
+             if memory_controller_root is not None:
+                 logger.info("The memory cgroup controller is mounted at {0}", memory_controller_root)
+             else:
+                 _log_cgroup_warning("The memory cgroup controller is not mounted")
++                controller_missing = True
+ 
+             #
+             # check v2 controllers
+             #
+-            cgroup2_mount_point, cgroup2_controllers = self._cgroups_api.get_cgroup2_controllers()
+-            if cgroup2_mount_point is not None:
+-                _log_cgroup_info("cgroups v2 mounted at {0}.  Controllers: [{1}]", cgroup2_mount_point,
+-                                 cgroup2_controllers)
++            if controller_missing:
++                cgroup2_mount_point, cgroup2_controllers = self._cgroups_api.get_cgroup2_controllers()
++                if cgroup2_mount_point is not None:
++                    _log_cgroup_info("cgroups v2 mounted at {0}.  Controllers: [{1}]", cgroup2_mount_point,
++                                     cgroup2_controllers)
++                    controller_missing = False
++                    if "memory" not in cgroup2_controllers or "-memory" in cgroup2_controllers:
++                        _log_cgroup_warning("Memory controller not active in cgroup v2")
++                        controller_missing = True
++                    if "cpu" not in cgroup2_controllers or "-cpu" in cgroup2_controllers:
++                        _log_cgroup_warning("CPU controller not active in cgroup v2")
++                        controller_missing = True
++                    if not controller_missing:
++                        return cgroup2_mount_point, cgroup2_mount_point
+ 
+             return cpu_controller_root, memory_controller_root
+ 
+--- walinuxagent-2.9.1.1.orig/azurelinuxagent/ga/collect_logs.py
++++ walinuxagent-2.9.1.1/azurelinuxagent/ga/collect_logs.py
+@@ -171,12 +171,11 @@ class CollectLogsHandler(ThreadHandlerIn
+     def _collect_logs(self):
+         logger.info("Starting log collection...")
+ 
+-        # Invoke the command line tool in the agent to collect logs, with resource limits on CPU.
+-        # Some distros like ubuntu20 by default cpu and memory accounting enabled. Thus create nested cgroups under the logcollector slice
+-        # So disabling CPU and Memory accounting prevents from creating nested cgroups, so that all the counters will be present in logcollector Cgroup
+-
++        # Changing both CPUAccounting and  MemoryAccounting to yes and adding CPUWeight so that cpu,cpuacct and memory controllers are enabled.
++        # According to this https://docs.kernel.org/admin-guide/cgroup-v2.html#controllers some values in cpu.stat are only available if cpu controller
++        # is enabled
+         systemd_cmd = [
+-            "systemd-run", "--property=CPUAccounting=no", "--property=MemoryAccounting=no",
++            "systemd-run", "--property=CPUAccounting=yes", "--property=CPUWeight=100", "--property=MemoryAccounting=yes",
+             "--unit={0}".format(logcollector.CGROUPS_UNIT),
+             "--slice={0}".format(cgroupconfigurator.LOGCOLLECTOR_SLICE), "--scope"
+         ]
+
+--- a/tests/common/mock_cgroup_environment.py
++++ b/tests/common/mock_cgroup_environment.py
+@@ -80,7 +80,7 @@ Thu 28 May 2020 07:25:55 AM PDT
+ _MOCKED_FILES = [
+     ("/proc/self/cgroup", os.path.join(data_dir, 'cgroups', 'proc_self_cgroup')),
+     (r"/proc/[0-9]+/cgroup", os.path.join(data_dir, 'cgroups', 'proc_pid_cgroup')),
+-    ("/sys/fs/cgroup/unified/cgroup.controllers", os.path.join(data_dir, 'cgroups', 'sys_fs_cgroup_unified_cgroup.controllers'))
++    ("/sys/fs/cgroup/unified/cgroup.subtree_control", os.path.join(data_dir, 'cgroups', 'sys_fs_cgroup_unified_cgroup.controllers'))
+ ]
+ 
+ _MOCKED_PATHS = [
+
+--- a/tests/common/test_cgroupconfigurator.py
++++ b/tests/common/test_cgroupconfigurator.py
+@@ -50,7 +50,7 @@ class CGroupConfiguratorSystemdTestCase(AgentTestCase):
+         AgentTestCase.tearDownClass()
+ 
+     @contextlib.contextmanager
+-    def _get_cgroup_configurator(self, initialize=True, enable=True, mock_commands=None):
++    def _get_cgroup_configurator(self, initialize=True, enable=True, mock_commands=None, mock_files=None):
+         CGroupConfigurator._instance = None
+         configurator = CGroupConfigurator.get_instance()
+         CGroupsTelemetry.reset()
+@@ -58,6 +58,9 @@ class CGroupConfiguratorSystemdTestCase(AgentTestCase):
+             if mock_commands is not None:
+                 for command in mock_commands:
+                     mock_environment.add_command(command)
++            if mock_files is not None:
++                for file in mock_files:
++                    mock_environment.add_file(*file)
+             configurator.mocks = mock_environment
+             if initialize:
+                 if not enable:
+@@ -115,7 +118,8 @@ cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
+ cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
+ cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
+ ''')]
+-        with self._get_cgroup_configurator(mock_commands=command_mocks) as configurator:
++        v2_files = [("/sys/fs/cgroup/unified/cgroup.subtree_control", os.path.join(data_dir, 'cgroups', 'sys_fs_cgroup_unified_cgroup_no_mem_cpu.controllers'))]
++        with self._get_cgroup_configurator(mock_commands=command_mocks, mock_files=v2_files) as configurator:
+             tracked = CGroupsTelemetry._tracked
+ 
+             self.assertFalse(configurator.enabled(), "Cgroups should not be enabled")
+@@ -135,7 +139,8 @@ cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,de
+ cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
+ ''')]
+ 
+-        with self._get_cgroup_configurator(mock_commands=command_mocks) as configurator:
++        v2_files = [("/sys/fs/cgroup/unified/cgroup.subtree_control", os.path.join(data_dir, 'cgroups', 'sys_fs_cgroup_unified_cgroup_no_mem_cpu.controllers'))]
++        with self._get_cgroup_configurator(mock_commands=command_mocks, mock_files=v2_files) as configurator:
+             tracked = CGroupsTelemetry._tracked
+             agent_drop_in_file_cpu_quota = configurator.mocks.get_mapped_path(UnitFilePaths.cpu_quota)
+ 
+--- /dev/null
++++ b/tests/data/cgroups/sys_fs_cgroup_unified_cgroup_no_mem_cpu.controllers
+@@ -0,0 +1,5 @@
++io
++pids
++perf_event
++rdma
++freezer
diff --git a/debian/patches/fix_systemd_networkd_lease_file_path b/debian/patches/fix_systemd_networkd_lease_file_path
new file mode 100644
index 0000000..85fda52
--- /dev/null
+++ b/debian/patches/fix_systemd_networkd_lease_file_path
@@ -0,0 +1,66 @@
+Description: Ubuntu versions >= 18.04 look for systemd-networkd lease files.
+    Currently, the package was configuring systemd-networkd but
+    checking for dhclient lease files.
+Forwarded: yes, https://github.com/Azure/WALinuxAgent/pull/2979
+Author: Calvin Mwadime Makokha <[e-mail]>
+
+--- a/azurelinuxagent/common/logcollector_manifests.py
++++ b/azurelinuxagent/common/logcollector_manifests.py
+@@ -83,6 +83,7 @@
+ copy,/var/lib/dhcp/dhclient.eth0.leases
+ copy,/var/lib/dhclient/dhclient-eth0.leases
+ copy,/var/lib/wicked/lease-eth0-dhcp-ipv4.xml
++copy,/run/systemd/netif/leases/2
+ echo,
+
+ echo,### Gathering Log Files ###
+--- a/azurelinuxagent/common/osutil/ubuntu.py
++++ b/azurelinuxagent/common/osutil/ubuntu.py
+@@ -16,6 +16,8 @@
+ # Requires Python 2.6+ and Openssl 1.0+
+ #
+
++import glob
++import textwrap
+ import time
+
+ import azurelinuxagent.common.logger as logger
+@@ -132,6 +134,25 @@
+     def stop_agent_service(self):
+         return shellutil.run("systemctl stop {0}".format(self.service_name), chk_err=False)
+
++    def get_dhcp_lease_endpoint(self):
++        pathglob = "/run/systemd/netif/leases/*"
++        logger.info("looking for leases in path [{0}]".format(pathglob))
++        endpoint = None
++        for lease_file in glob.glob(pathglob):
++            with open(lease_file) as f:
++                lease = f.read()
++            for line in lease.splitlines():
++                if line.startswith("OPTION_245"):
++                    option_245 = line.split("=")[1]
++                    options = [int(i, 16) for i in textwrap.wrap(option_245, 2)]
++                    endpoint = "{0}.{1}.{2}.{3}".format(*options)
++                    logger.info("found endpoint [{0}]".format(endpoint))
++        if endpoint is not None:
++            logger.info("cached endpoint found [{0}]".format(endpoint))
++        else:
++            logger.info("cached endpoint not found")
++        return endpoint
++
+
+ class UbuntuOSUtil(Ubuntu16OSUtil):
+     def __init__(self):  # pylint: disable=W0235
+--- a/azurelinuxagent/pa/deprovision/default.py
++++ b/azurelinuxagent/pa/deprovision/default.py
+@@ -131,6 +131,10 @@
+         actions.append(DeprovisionAction(fileutil.rm_files,
+                                          ["/var/lib/NetworkManager/dhclient-*.lease"]))
+
++        # For Ubuntu >= 18.04, using systemd-networkd
++        actions.append(DeprovisionAction(fileutil.rm_files,
++                                         ["/run/systemd/netif/leases/*"]))
++
+     def del_ext_handler_files(self, warnings, actions):  # pylint: disable=W0613
+         ext_dirs = [d for d in os.listdir(conf.get_lib_dir())
+                     if os.path.isdir(os.path.join(conf.get_lib_dir(), d))
diff --git a/debian/patches/series b/debian/patches/series
index 6217ba6..8c9f6bb 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,3 +1,4 @@
-disable_import_test.patch
+fix_systemd_networkd_lease_file_path
+update_dhcp_client_ubuntu_supported_versions.patch
 disable_udev_overrides.patch
-fix-udev-rules-for-azure.patch
+fix_cgroup_v2_mounting_and_systemd_process.patch
diff --git a/debian/patches/update_dhcp_client_ubuntu_supported_versions.patch b/debian/patches/update_dhcp_client_ubuntu_supported_versions.patch
new file mode 100644
index 0000000..c8208c7
--- /dev/null
+++ b/debian/patches/update_dhcp_client_ubuntu_supported_versions.patch
@@ -0,0 +1,39 @@
+Description: Update the latest supported ubuntu version
+Forwarded: yes, https://github.com/Azure/WALinuxAgent/pull/2980
+Author: Calvin Mwadime Makokha <[e-mail]>
+
+--- a/azurelinuxagent/common/osutil/factory.py
++++ b/azurelinuxagent/common/osutil/factory.py
+@@ -66,15 +66,14 @@
+         return ClearLinuxUtil()
+
+     if distro_name == "ubuntu":
+-        if Version(distro_version) in [Version("12.04"), Version("12.10")]:
++        ubuntu_version = Version(distro_version)
++        if ubuntu_version in [Version("12.04"), Version("12.10")]:
+             return Ubuntu12OSUtil()
+-        if Version(distro_version) in [Version("14.04"), Version("14.10")]:
++        if ubuntu_version in [Version("14.04"), Version("14.10")]:
+             return Ubuntu14OSUtil()
+-        if Version(distro_version) in [Version('16.04'), Version('16.10'), Version('17.04')]:
++        if ubuntu_version in [Version('16.04'), Version('16.10'), Version('17.04')]:
+             return Ubuntu16OSUtil()
+-        if Version(distro_version) in [Version('18.04'), Version('18.10'),
+-                                         Version('19.04'), Version('19.10'),
+-                                         Version('20.04')]:
++        if ubuntu_version >= Version('18.04') and ubuntu_version <= Version('24.04'):
+             return Ubuntu18OSUtil()
+         if distro_full_name == "Snappy Ubuntu Core":
+             return UbuntuSnappyOSUtil()
+
+--- a/azurelinuxagent/common/osutil/ubuntu.py
++++ b/azurelinuxagent/common/osutil/ubuntu.py
+@@ -88,7 +88,7 @@ class Ubuntu16OSUtil(Ubuntu14OSUtil):
+
+ class Ubuntu18OSUtil(Ubuntu16OSUtil):
+     """
+-    Ubuntu 18.04, 18.10, 19.04, 19.10, 20.04
++    Ubuntu >= 18.04 and <= 24.04
+     """
+     def __init__(self):
+         super(Ubuntu18OSUtil, self).__init__()
diff --git a/debian/rules b/debian/rules
index c06ee11..0e7bd5a 100755
--- a/debian/rules
+++ b/debian/rules
@@ -12,22 +12,19 @@ get-packaged-orig-source:
 	rm -rf orig_source
 
 %:
-	dh $@ --with python3,systemd --buildsystem=pybuild
+	dh $@ --with python3 --buildsystem=pybuild
 
+execute_before_dh_installman:
+	go-md2man -in=README.md -out=debian/walinuxagent.1
 
-override_dh_installinit:
-	dh_installinit --no-restart-on-upgrade --name walinuxagent
-	dh_installinit --no-restart-on-upgrade --name ephemeral-disk-warning
-
-override_dh_systemd_enable:
-	dh_systemd_enable --name walinuxagent walinuxagent.service
-	dh_systemd_enable --name ephemeral-disk-warning ephemeral-disk-warning.service
+override_dh_installsystemd:
+	# remove service placed in the wrong path by upstream install
+	rm -f debian/walinuxagent/lib/systemd/system/walinuxagent.service
+	dh_installsystemd --name walinuxagent --no-stop-on-upgrade walinuxagent.service
+	dh_installsystemd --name walinuxagent --no-stop-on-upgrade ephemeral-disk-warning.service
 
 override_dh_python3:
 	dh_python3 -O--buildsystem=pybuild --shebang "/usr/bin/env python3"
 
 override_dh_auto_test:
-	# No test were run on any of the previous versions, temporarily
-	# ignoring the failures as they're not regressions (LP: #1800499)
-	# XXX: Those need to be fixed ASAP.
-	-dh_auto_test -O--buildsystem=pybuild
+	nosetests3 -a '!requires_sudo' tests
diff --git a/debian/walinuxagent.manpages b/debian/walinuxagent.manpages
new file mode 100644
index 0000000..8b6ead3
--- /dev/null
+++ b/debian/walinuxagent.manpages
@@ -0,0 +1 @@
+debian/walinuxagent.1
diff --git a/debian/walinuxagent.upstart b/debian/walinuxagent.upstart
deleted file mode 100644
index a5f8a95..0000000
--- a/debian/walinuxagent.upstart
+++ /dev/null
@@ -1,25 +0,0 @@
-description "Windows Azure Linux agent"
-author "Ben Howard <[e-mail]>"
-
-start on runlevel [2345]
-stop on runlevel [!2345]
-
-pre-start script
-
-    [ -r /etc/default/walinuxagent ] && . /etc/default/walinuxagent
-
-    if [ "$WALINUXAGENT_ENABLED" != "1" ]; then
-        stop ; exit 0
-    fi
-
-    if [ ! -x /usr/sbin/waagent ]; then
-        stop ; exit 0
-    fi
-
-    #Load the udf module
-    modprobe -b udf
-
-end script
-
-exec /usr/bin/python3 /usr/sbin/waagent -daemon
-
diff --git a/debian/watch b/debian/watch
index 1cbe705..7892b9e 100644
--- a/debian/watch
+++ b/debian/watch
@@ -1,3 +1,4 @@
-version=3
-opts=filenamemangle=s/.+\/v?(\d\S*)\.tar\.gz/<project>-$1\.tar\.gz/ \
- https://github.com/Azure/WALinuxAgent/tags .*/?(\d\.\d.\d{1,})\.tar\.gz
+version=4
+ opts="searchmode=plain,\
+ filenamemangle=s/.+?v?(\d+(?:\.\d+)*)/walinuxagent-$1\.tar\.gz/" \
+ https://api.github.com/repos/Azure/WALinuxAgent/releases https://api.github.com/repos/Azure/WALinuxAgent/tarball/v@ANY_VERSION@