Merge ~paelzer/ubuntu/+source/libvirt:post-disco-sru-set-cosmic into ubuntu/+source/libvirt:ubuntu/cosmic-devel

[Christian Ehrhardt ]

[Christian Ehrhardt  (paelzer)]

PPA is ready at: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3620

[Christian Ehrhardt  (paelzer)]

dep8 tests are only hitting known flaky tests, so they are ok

Also the automated virt regression tests ran fine on all architectures.

Remaining: I'm waiting for the bug reporters to confirm test-ability and the MP review here

[Andreas Hasenack (ahasenack)]

Just an incorrect bug number in one of the dep3 headers.

I also wondered if lp-1771662-4-util-Fixing-invalid-error-checking-from-virPCIGetNet.patch warranted a "backport" note, since it's just a whitespace change on a single line. I would have fixed the whitespace and at most added a comment to the patch header, but that's just a personal opinion.

+1 with the bug number fixed.

review: Approve

[Christian Ehrhardt  (paelzer)]

Thanks, I fixed the bug reference.

For the backport note, I strictly follow "if it applies as-is then upstream, otherwise backport".
That way I don't have to make decisions where "real" backports start because any other line we draw can be just as wrong from other peoples POV.

diff --git a/debian/changelog b/debian/changelog
index 0cf4887..9cafb4c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,12 @@
+libvirt (4.6.0-2ubuntu3.3) cosmic; urgency=medium
+
+  * d/p/ubuntu/lp-1811198-utils-Remove-arbitrary-limit-on-socket_id-core_id
+    .patch: fix arm servers with high core_id (LP: #1811198)
+  * d/p/ubuntu/lp-1771662-*: fix assumption that all VFs have PFs assigned
+    (LP: #1771662)
+
+ -- Christian Ehrhardt <[e-mail]>  Thu, 31 Jan 2019 12:29:37 +0100
+
 libvirt (4.6.0-2ubuntu3.2) cosmic; urgency=medium
 
   * d/p/ubuntu/lp1787405-0008-qemu-mdev-Use-vfio-pci-display-property-only
diff --git a/debian/patches/series b/debian/patches/series
index 69ff83a..5e60dd2 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -70,3 +70,8 @@ ubuntu/lp1787405-0005-qemu-Extract-MDEV-VFIO-PCI-validation-code-into-a-se.patch
 ubuntu/lp1787405-0006-conf-Move-VFIO-AP-validation-from-post-parse-to-QEMU.patch
 ubuntu/lp1787405-0007-virt-aa-helper-mdevs-need-vfio.patch
 ubuntu/lp1787405-0008-qemu-mdev-Use-vfio-pci-display-property-only-with-vf.patch
+ubuntu/lp-1811198-utils-Remove-arbitrary-limit-on-socket_id-core_id.patch
+ubuntu/lp-1771662-1-util-fixing-wrong-assumption-that-PF-has-to-have-net.patch
+ubuntu/lp-1771662-2-util-Code-simplification.patch
+ubuntu/lp-1771662-3-util-Fix-for-NULL-dereference.patch
+ubuntu/lp-1771662-4-util-Fixing-invalid-error-checking-from-virPCIGetNet.patch
diff --git a/debian/patches/ubuntu/lp-1771662-1-util-fixing-wrong-assumption-that-PF-has-to-have-net.patch b/debian/patches/ubuntu/lp-1771662-1-util-fixing-wrong-assumption-that-PF-has-to-have-net.patch
 new file mode 100644
index 0000000..a7944e1
--- /dev/null
+++ b/debian/patches/ubuntu/lp-1771662-1-util-fixing-wrong-assumption-that-PF-has-to-have-net.patch
@@ -0,0 +1,62 @@
+From 6452e2f5e1837bd753ee465e6607ed3c4f62b815 Mon Sep 17 00:00:00 2001
+From: Radoslaw Biernacki <[e-mail]>
+Date: Tue, 22 Jan 2019 12:26:12 -0700
+Subject: [PATCH 1/4] util: fixing wrong assumption that PF has to have netdev
+ assigned
+
+libvirt wrongly assumes that VF netdev has to have the
+netdev assigned to PF. There is no such requirement in SRIOV standard.
+This patch change the virNetDevSwitchdevFeature() function to deal
+with SRIOV devices which does not have netdev on PF. Also corrects
+one comment about PF netdev assumption.
+
+One example of such devices is ThunderX VNIC.
+By applying this change, VF device is used for virNetlinkCommand() as
+it is the only netdev assigned to VNIC.
+
+Signed-off-by: Radoslaw Biernacki <[e-mail]>
+Signed-off-by: dann frazier <[e-mail]>
+Signed-off-by: Michal Privoznik <[e-mail]>
+
+Origin: upstream, https://libvirt.org/git/?p=libvirt.git;a=commit;h=6452e2f5e1837bd753ee465e6607ed3c4f62b815
+Bug-Ubuntu: https://bugs.launchpad.net/bugs/1771662
+Last-Update: 2019-01-31
+
+---
+ src/util/virnetdev.c | 13 ++++++++-----
+ 1 file changed, 8 insertions(+), 5 deletions(-)
+
+diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c
+index 2111b3ada9..d3e69b6f60 100644
+--- a/src/util/virnetdev.c
++++ b/src/util/virnetdev.c
+@@ -1355,9 +1355,8 @@ virNetDevGetPhysicalFunction(const char *ifname, char **pfname)
+     }
+ 
+     if (!*pfname) {
+-        /* this shouldn't be possible. A VF can't exist unless its
+-         * PF device is bound to a network driver
+-         */
++        /* The SRIOV standard does not require VF netdevs to have
++         * the netdev assigned to a PF. */
+         virReportError(VIR_ERR_INTERNAL_ERROR,
+                        _("The PF device for VF %s has no network device name"),
+                        ifname);
+@@ -3178,8 +3177,12 @@ virNetDevSwitchdevFeature(const char *ifname,
+     if ((is_vf = virNetDevIsVirtualFunction(ifname)) < 0)
+         return ret;
+ 
+-    if (is_vf == 1 && virNetDevGetPhysicalFunction(ifname, &pfname) < 0)
+-        goto cleanup;
++    if (is_vf == 1) {
++        /* Ignore error if PF does not have netdev assigned.
++         * In that case pfname == NULL. */
++        if (virNetDevGetPhysicalFunction(ifname, &pfname) < 0)
++            virResetLastError();
++    }
+ 
+     pci_device_ptr = pfname ? virNetDevGetPCIDevice(pfname) :
+                               virNetDevGetPCIDevice(ifname);
+-- 
+2.17.1
+
diff --git a/debian/patches/ubuntu/lp-1771662-2-util-Code-simplification.patch b/debian/patches/ubuntu/lp-1771662-2-util-Code-simplification.patch
 new file mode 100644
index 0000000..bcb8a6f
--- /dev/null
+++ b/debian/patches/ubuntu/lp-1771662-2-util-Code-simplification.patch
@@ -0,0 +1,104 @@
+From 10bca495e040ef834760a244a31f8b87391c2378 Mon Sep 17 00:00:00 2001
+From: Radoslaw Biernacki <[e-mail]>
+Date: Tue, 22 Jan 2019 12:26:13 -0700
+Subject: [PATCH 2/4] util: Code simplification
+
+Removing redundant sections of the code
+
+Signed-off-by: Radoslaw Biernacki <[e-mail]>
+Signed-off-by: dann frazier <[e-mail]>
+Signed-off-by: Michal Privoznik <[e-mail]>
+
+Origin: upstream, https://libvirt.org/git/?p=libvirt.git;a=commit;h=10bca495e040ef834760a244a31f8b87391c2378
+Bug-Ubuntu: https://bugs.launchpad.net/bugs/1771662
+Last-Update: 2019-01-31
+
+---
+ src/util/virnetdev.c | 33 ++++++---------------------------
+ 1 file changed, 6 insertions(+), 27 deletions(-)
+
+diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c
+index d3e69b6f60..92ef008ca1 100644
+--- a/src/util/virnetdev.c
++++ b/src/util/virnetdev.c
+@@ -1443,29 +1443,20 @@ int
+ virNetDevGetVirtualFunctionInfo(const char *vfname, char **pfname,
+                                 int *vf)
+ {
+-    char *pf_sysfs_path = NULL, *vf_sysfs_path = NULL;
+     int ret = -1;
+ 
+     *pfname = NULL;
+ 
+     if (virNetDevGetPhysicalFunction(vfname, pfname) < 0)
+-        return ret;
+-
+-    if (virNetDevSysfsFile(&pf_sysfs_path, *pfname, "device") < 0)
+-        goto cleanup;
++        return -1;
+ 
+-    if (virNetDevSysfsFile(&vf_sysfs_path, vfname, "device") < 0)
++    if (virNetDevGetVirtualFunctionIndex(*pfname, vfname, vf) < 0)
+         goto cleanup;
+ 
+-    ret = virPCIGetVirtualFunctionIndex(pf_sysfs_path, vf_sysfs_path, vf);
+-
++    ret = 0;
+  cleanup:
+     if (ret < 0)
+         VIR_FREE(*pfname);
+-
+-    VIR_FREE(vf_sysfs_path);
+-    VIR_FREE(pf_sysfs_path);
+-
+     return ret;
+ }
+ 
+@@ -1861,13 +1852,9 @@ virNetDevSaveNetConfig(const char *linkdev, int vf,
+          * it to PF + VFname
+          */
+ 
+-        if (virNetDevGetPhysicalFunction(linkdev, &pfDevOrig) < 0)
++        if (virNetDevGetVirtualFunctionInfo(linkdev, &pfDevOrig, &vf) < 0)
+             goto cleanup;
+-
+         pfDevName = pfDevOrig;
+-
+-        if (virNetDevGetVirtualFunctionIndex(pfDevName, linkdev, &vf) < 0)
+-            goto cleanup;
+     }
+ 
+     if (pfDevName) {
+@@ -2019,13 +2006,9 @@ virNetDevReadNetConfig(const char *linkdev, int vf,
+          * it to PF + VFname
+          */
+ 
+-        if (virNetDevGetPhysicalFunction(linkdev, &pfDevOrig) < 0)
++        if (virNetDevGetVirtualFunctionInfo(linkdev, &pfDevOrig, &vf) < 0)
+             goto cleanup;
+-
+         pfDevName = pfDevOrig;
+-
+-        if (virNetDevGetVirtualFunctionIndex(pfDevName, linkdev, &vf) < 0)
+-            goto cleanup;
+     }
+ 
+     /* if there is a PF, it's now in pfDevName, and linkdev is either
+@@ -2224,13 +2207,9 @@ virNetDevSetNetConfig(const char *linkdev, int vf,
+          * it to PF + VFname
+          */
+ 
+-        if (virNetDevGetPhysicalFunction(linkdev, &pfDevOrig) < 0)
++        if (virNetDevGetVirtualFunctionInfo(linkdev, &pfDevOrig, &vf))
+             goto cleanup;
+-
+         pfDevName = pfDevOrig;
+-
+-        if (virNetDevGetVirtualFunctionIndex(pfDevName, linkdev, &vf) < 0)
+-            goto cleanup;
+     }
+ 
+ 
+-- 
+2.17.1
+
diff --git a/debian/patches/ubuntu/lp-1771662-3-util-Fix-for-NULL-dereference.patch b/debian/patches/ubuntu/lp-1771662-3-util-Fix-for-NULL-dereference.patch
 new file mode 100644
index 0000000..4a8cd91
--- /dev/null
+++ b/debian/patches/ubuntu/lp-1771662-3-util-Fix-for-NULL-dereference.patch
@@ -0,0 +1,74 @@
+From 8fac64db5e7941efb820626f0043f5e0a31c79ee Mon Sep 17 00:00:00 2001
+From: Radoslaw Biernacki <[e-mail]>
+Date: Tue, 22 Jan 2019 12:26:14 -0700
+Subject: [PATCH 3/4] util: Fix for NULL dereference
+
+The device xml parser code does not set "model" while parsing the
+following XML:
+
+  <interface type='hostdev'>
+    <source>
+      <address type='pci' domain='0x0002' bus='0x01' slot='0x00' function='0x2'/>
+    </source>
+  </interface>
+
+The net->model can be NULL and therefore must be compared using
+STREQ_NULLABLE instead of plain STREQ.
+
+Fixes: ac47e4a6225 (qemu: replace "def->nets[i]" with "net" and "def->sounds[i]" with "sound")
+Fixes: c7fc151eec7 (qemu: assign virtio devices to PCIe slot when appropriate)
+Signed-off-by: Radoslaw Biernacki <[e-mail]>
+Signed-off-by: dann frazier <[e-mail]>
+Reviewed-by: Michal Privoznik <[e-mail]>
+
+Author: Christian Ehrhardt <[e-mail]>
+Original-Author: Radoslaw Biernacki <[e-mail]>
+Origin: backport, https://libvirt.org/git/?p=libvirt.git;a=commit;h=8fac64db5e7941efb820626f0043f5e0a31c79ee
+Bug-Ubuntu: https://bugs.launchpad.net/bugs/1771662
+Last-Update: 2019-01-31
+
+---
+ src/qemu/qemu_domain_address.c | 13 +++++--------
+ 1 file changed, 5 insertions(+), 8 deletions(-)
+
+--- a/src/qemu/qemu_domain_address.c
++++ b/src/qemu/qemu_domain_address.c
+@@ -232,10 +232,8 @@ qemuDomainAssignSpaprVIOAddresses(virDom
+     for (i = 0; i < def->nnets; i++) {
+         virDomainNetDefPtr net = def->nets[i];
+ 
+-        if (net->model &&
+-            STREQ(net->model, "spapr-vlan")) {
++        if (STREQ_NULLABLE(net->model, "spapr-vlan"))
+             net->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_SPAPRVIO;
+-        }
+ 
+         if (qemuDomainAssignSpaprVIOAddress(def, &net->info, VIO_ADDR_NET) < 0)
+             goto cleanup;
+@@ -323,7 +321,7 @@ qemuDomainPrimeVirtioDeviceAddresses(vir
+     for (i = 0; i < def->nnets; i++) {
+         virDomainNetDefPtr net = def->nets[i];
+ 
+-        if (STREQ(net->model, "virtio") &&
++        if (STREQ_NULLABLE(net->model, "virtio") &&
+             net->info.type == VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE) {
+             net->info.type = type;
+         }
+@@ -604,14 +602,14 @@ qemuDomainDeviceCalculatePCIConnectFlags
+          * addresses for other hostdev devices.
+          */
+         if (net->type == VIR_DOMAIN_NET_TYPE_HOSTDEV ||
+-            STREQ(net->model, "usb-net")) {
++            STREQ_NULLABLE(net->model, "usb-net")) {
+             return 0;
+         }
+ 
+-        if (STREQ(net->model, "virtio"))
++        if (STREQ_NULLABLE(net->model, "virtio"))
+             return  virtioFlags;
+ 
+-        if (STREQ(net->model, "e1000e"))
++        if (STREQ_NULLABLE(net->model, "e1000e"))
+             return pcieFlags;
+ 
+         return pciFlags;
diff --git a/debian/patches/ubuntu/lp-1771662-4-util-Fixing-invalid-error-checking-from-virPCIGetNet.patch b/debian/patches/ubuntu/lp-1771662-4-util-Fixing-invalid-error-checking-from-virPCIGetNet.patch
 new file mode 100644
index 0000000..6b002c7
--- /dev/null
+++ b/debian/patches/ubuntu/lp-1771662-4-util-Fixing-invalid-error-checking-from-virPCIGetNet.patch
@@ -0,0 +1,35 @@
+From 04983c3c6a821f67994b1c65d4d6175f3ac49d69 Mon Sep 17 00:00:00 2001
+From: Radoslaw Biernacki <[e-mail]>
+Date: Tue, 22 Jan 2019 12:26:15 -0700
+Subject: [PATCH 4/4] util: Fixing invalid error checking from
+ virPCIGetNetname()
+
+The @linkdev is In/Out function parameter as second order
+reference pointer so requires first order dereference for
+checking NULL which can be the result of virPCIGetNetName().
+
+Fixes: d6ee56d7237 (util: change virPCIGetNetName() to not return error if device has no net name)
+Signed-off-by: Radoslaw Biernacki <[e-mail]>
+Signed-off-by: dann frazier <[e-mail]>
+
+Author: Christian Ehrhardt <[e-mail]>
+Original-Author: Radoslaw Biernacki <[e-mail]>
+Origin: backport, https://libvirt.org/git/?p=libvirt.git;a=commit;h=04983c3c6a821f67994b1c65d4d6175f3ac49d69
+Bug-Ubuntu: https://bugs.launchpad.net/bugs/1771662
+Last-Update: 2019-01-31
+
+---
+ src/util/virhostdev.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+--- a/src/util/virhostdev.c
++++ b/src/util/virhostdev.c
+@@ -319,7 +319,7 @@ virHostdevNetDevice(virDomainHostdevDefP
+         if (virPCIGetNetName(sysfs_path, 0, NULL, linkdev) < 0)
+             return -1;
+ 
+-        if (!linkdev) {
++        if (!(*linkdev)) {
+             virReportError(VIR_ERR_INTERNAL_ERROR,
+                            _("The device at %s has no network device name"),
+                              sysfs_path);
diff --git a/debian/patches/ubuntu/lp-1811198-utils-Remove-arbitrary-limit-on-socket_id-core_id.patch b/debian/patches/ubuntu/lp-1811198-utils-Remove-arbitrary-limit-on-socket_id-core_id.patch
 new file mode 100644
index 0000000..81ae0fd
--- /dev/null
+++ b/debian/patches/ubuntu/lp-1811198-utils-Remove-arbitrary-limit-on-socket_id-core_id.patch
@@ -0,0 +1,91 @@
+From ba35ac2ebbc7f94abc50ffbf1d681458e2406444 Mon Sep 17 00:00:00 2001
+From: Andrea Bolognani <[e-mail]>
+Date: Fri, 3 Aug 2018 10:15:16 +0200
+Subject: [PATCH] utils: Remove arbitrary limit on socket_id/core_id
+
+While in most cases the values are going to be much
+smaller than our arbitrary 4096 limit, there is really
+no guarantee that would be the case: in fact, a few
+aarch64 servers have been spotted in the wild with
+core_id as high as 6216.
+
+Take advantage of virBitmap's ability to automatically
+alter its size at runtime to accomodate such values.
+
+Signed-off-by: Andrea Bolognani <[e-mail]>
+
+Origin: upstream, https://libvirt.org/git/?p=libvirt.git;a=commit;h=ba35ac2ebbc7f94abc50ffbf1d681458e2406444
+Bug-Ubuntu: https://bugs.launchpad.net/bugs/1811198
+Last-Update: 2019-01-31
+
+---
+ src/util/virhostcpu.c | 23 ++++-------------------
+ 1 file changed, 4 insertions(+), 19 deletions(-)
+
+diff --git a/src/util/virhostcpu.c b/src/util/virhostcpu.c
+index 6e79445abc..2337ad7d61 100644
+--- a/src/util/virhostcpu.c
++++ b/src/util/virhostcpu.c
+@@ -293,9 +293,6 @@ virHostCPUParseNode(const char *node,
+                     int *threads,
+                     int *offline)
+ {
+-    /* Biggest value we can expect to be used as either socket id
+-     * or core id. Bitmaps will need to be sized accordingly */
+-    const int ID_MAX = 4095;
+     int ret = -1;
+     int processors = 0;
+     DIR *cpudir = NULL;
+@@ -324,7 +321,7 @@ virHostCPUParseNode(const char *node,
+         goto cleanup;
+ 
+     /* enumerate sockets in the node */
+-    if (!(sockets_map = virBitmapNew(ID_MAX + 1)))
++    if (!(sockets_map = virBitmapNewEmpty()))
+         goto cleanup;
+ 
+     while ((direrr = virDirRead(cpudir, &cpudirent, node)) > 0) {
+@@ -343,14 +340,8 @@ virHostCPUParseNode(const char *node,
+ 
+         if (virHostCPUGetSocket(cpu, &sock) < 0)
+             goto cleanup;
+-        if (sock > ID_MAX) {
+-            virReportError(VIR_ERR_INTERNAL_ERROR,
+-                           _("Socket %d can't be handled (max socket is %d)"),
+-                           sock, ID_MAX);
+-            goto cleanup;
+-        }
+ 
+-        if (virBitmapSetBit(sockets_map, sock) < 0)
++        if (virBitmapSetBitExpand(sockets_map, sock) < 0)
+             goto cleanup;
+ 
+         if (sock > sock_max)
+@@ -367,7 +358,7 @@ virHostCPUParseNode(const char *node,
+         goto cleanup;
+ 
+     for (i = 0; i < sock_max; i++)
+-        if (!(cores_maps[i] = virBitmapNew(ID_MAX + 1)))
++        if (!(cores_maps[i] = virBitmapNewEmpty()))
+             goto cleanup;
+ 
+     /* Iterate over all CPUs in the node, in ascending order */
+@@ -411,14 +402,8 @@ virHostCPUParseNode(const char *node,
+             if (virHostCPUGetCore(cpu, &core) < 0)
+                 goto cleanup;
+         }
+-        if (core > ID_MAX) {
+-            virReportError(VIR_ERR_INTERNAL_ERROR,
+-                           _("Core %d can't be handled (max core is %d)"),
+-                           core, ID_MAX);
+-            goto cleanup;
+-        }
+ 
+-        if (virBitmapSetBit(cores_maps[sock], core) < 0)
++        if (virBitmapSetBitExpand(cores_maps[sock], core) < 0)
+             goto cleanup;
+ 
+         if (!(siblings = virHostCPUCountThreadSiblings(cpu)))
+-- 
+2.17.1
+