[FFe] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Bug #1541902 reported by bugproxy
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
Fix Released
High
Jon Grimm

Bug Description

== Comment: #0 - Breno Henrique Leitao <email address hidden> - 2016-02-04 10:48:25 ==
This is a feature to track DDW support on Qemu/kvm passthrough. This is a feature to improve performance when using GPUs on virtual machines.

The kernel pieces are already on version 4.2, and now we need the pieces on qemu.

This should be sent upstream in 2 weeks, and submission URL pasted here.

bugproxy (bugproxy)
tags: added: architecture-ppc64 bugnameltc-136627 severity-high targetmilestone-inin1604
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1541902/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Steve Langasek (vorlon)
affects: ubuntu → qemu (Ubuntu)
Revision history for this message
Jon Grimm (jgrimm) wrote :

What's the status on these patches? Have they been accepted upstream yet?

Changed in qemu (Ubuntu):
status: New → Incomplete
Revision history for this message
Breno Leitão (breno-leitao) wrote :

We should be able to send this patches soon.

Revision history for this message
Breno Leitão (breno-leitao) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1541902] Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

> The patches were just sent upstream.
>
> http://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00057.html

Thanks. The feedback also seems generally good, so I'm comfortable
with this. The main problem now is timing, since we are well past
feature freeze for 16.04. So if we want this in time for 16.04
we'll need a strong justification for its high priority, and to
mark this bug [FFE] in the Subject and subscribe the ubuntu-release
team per https://wiki.ubuntu.com/FreezeExceptionProcess .

Changed in qemu (Ubuntu):
status: Incomplete → Confirmed
summary: - Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM
+ [FFE] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu
+ KVM
Revision history for this message
Breno Leitão (breno-leitao) wrote :

I would like to justify this FFe as this is an important feature for Ubuntu 16.04/ppc64el

[Technical Justification]
This feature enables GPU pass through to work decently on Ubuntu KVM. Without this feature, GPU pass through works with a very bad performance, which is completely unusable in production. This feature enables a huge performance improvement, allowing Ubuntu KVM deployment in production environments.

[ Customer justification]
There are numerous partners interested in this feature, and a real set of customers that will be taking benefit directly if this feature makes Ubuntu 16.04 release.

[Testing plan]
This set of patch is being tested internally at IBM.

bugproxy (bugproxy)
tags: added: targetmilestone-inin16041
removed: targetmilestone-inin1604
Revision history for this message
Breno Leitão (breno-leitao) wrote :

Dear Canonical,

A new patchset (version 15) was sent upstream, and already reviewed by David Gibson.

http://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg00360.html

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-05-11 10:53 EDT-------
Reproducing Alexey's comment from 5/5:

"[PATCH qemu v16 00/19] spapr: vfio: Enable Dynamic DMA windows (DDW)" has been posted, it can be picked from https://patchwork.ozlabs.org/patch/618275/ , etc.

Mathew Hodson (mhodson)
Changed in qemu (Ubuntu):
importance: Undecided → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Seems like the upstream discussion is still unresolved in some areas (and no patches are upstream in the git tree) and I don't understand why.

I'm happy to pull these patches in now. Just as a sanity check, @rharper - any definite badness that you see?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

(Make that @raharper ^ )

Revision history for this message
Ryan Harper (raharper) wrote : Re: [Bug 1541902] Re: [FFE] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

I'd say we should hold off:

http://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg03092.html

Includes this bit:

>> Why 64-bit math here, but 128 bit math in the region_add path?

>

> Hm. Good question. It has been copied from the common VFIO code where it was
since 7532d3cbf1, back those days I did not even think of VFIO devices
hot(un)plugging
so
> region_del() was never executed so this seems to be a bug there and here.
Yet another patch is coming then

There are other things that aren't nearly as troublesome but it certainly
feels like another round of patches are forthcoming.
I didn't get a feeling from the patchset that this was functional, no
Tested-by: or Acked-by: either.

On Thu, May 12, 2016 at 1:11 PM, Serge Hallyn <email address hidden>
wrote:

> (Make that @raharper ^ )
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1541902
>
> Title:
> [FFE] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu
> KVM
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1541902/+subscriptions
>

bugproxy (bugproxy)
tags: removed: bot-comment bugnameltc-136627 severity-high
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks, Ryan. We'll hold off until the next round.

Revision history for this message
Ryan Harper (raharper) wrote :

Serge, we may want to pull them in and put in a PPA to see if IBM and
others can give the patchset a try. And ideally they'd report to
qemu-devel of their success/problems.

On Fri, May 13, 2016 at 8:17 AM, Serge Hallyn <email address hidden>
wrote:

> Thanks, Ryan. We'll hold off until the next round.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1541902
>
> Title:
> [FFE] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu
> KVM
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1541902/+subscriptions
>

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Happy to. I'll take the patchset from May 11,

Subject: [Qemu-devel] [for-2.7 PATCH v3 00/15] Core based CPU hotplug for PowerPC sPAPR

Please shout if that is not the best one to use.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

No that seems to be the entirely wrong patchset.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Pushed a package to ppa:ubuntu-virt/ppa with the patchset.

summary: - [FFE] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu
- KVM
+ Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM
Revision history for this message
Jon Grimm (jgrimm) wrote :

We should try to get this in for 16.04.1. IBM can you give this PPA a spin?

Revision history for this message
Breno Leitão (breno-leitao) wrote :

Hi Jon, Serge,

I think the PPA is somehow corrupted. I am not able to see the ddw package on ppc64el arch. It shows up in amd64 (and the source) but not on ppc64el. Take a look:

$ apt-cache madison qemu
      qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/universe ppc64el Packages
      qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/main Sources
      qemu | 1:2.6+dfsg-1ubuntu1ddw1 | http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu yakkety/main Sources

It is missing the binary for ppc64el. I looked at repository itself, and the Packages file for yakkety in ppc64el is empty. Take a look:

http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu/dists/yakkety/main/binary-ppc64el/Packages.xz

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1541902] Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

I've enabled ppc64le builds for that ppa. Hopefully it proceeds to
automatically build the packages now, though I suspect the pkgs will
need to be re-pushed.

On Fri, Jun 10, 2016 at 5:14 PM, Breno Leitão
<email address hidden> wrote:
> Hi Jon, Serge,
>
> I think the PPA is somehow corrupted. I am not able to see the ddw
> package on ppc64el arch. It shows up in amd64 (and the source) but not
> on ppc64el. Take a look:
>
> $ apt-cache madison qemu
> qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/universe ppc64el Packages
> qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/main Sources
> qemu | 1:2.6+dfsg-1ubuntu1ddw1 | http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu yakkety/main Sources
>
> It is missing the binary for ppc64el. I looked at repository itself, and
> the Packages file for yakkety in ppc64el is empty. Take a look:
>
> http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu/dists/yakkety/main
> /binary-ppc64el/Packages.xz
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1541902
>
> Title:
> Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM
>
> Status in qemu package in Ubuntu:
> Confirmed
>
> Bug description:
> == Comment: #0 - Breno Henrique Leitao <email address hidden> - 2016-02-04 10:48:25 ==
> This is a feature to track DDW support on Qemu/kvm passthrough. This is a feature to improve performance when using GPUs on virtual machines.
>
> The kernel pieces are already on version 4.2, and now we need the
> pieces on qemu.
>
> This should be sent upstream in 2 weeks, and submission URL pasted
> here.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1541902/+subscriptions

Revision history for this message
Breno Leitão (breno-leitao) wrote : Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Hi Serge,

yes, no ppc64el binaries so far:

$ apt-cache madison qemu
      qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/universe ppc64el Packages
      qemu | 1:2.6+dfsg-1ubuntu1 | http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/main Sources
      qemu | 1:2.6+dfsg-1ubuntu1ddw1 | http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu yakkety/main Sources

Revision history for this message
Ryan Harper (raharper) wrote : Re: [Bug 1541902] Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

On Mon, Jun 13, 2016 at 8:06 AM, Breno Leitão <email address hidden>
wrote:

> Hi Serge,
>
> yes, no ppc64el binaries so far:
>

Working on it. Reconfigured PPA to enable PPC64le, and re-triggered the
package build.
I'll follow up and make sure we see the ppc64le build show up.

Ryan

>
> $ apt-cache madison qemu
> qemu | 1:2.6+dfsg-1ubuntu1 |
> http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/universe ppc64el
> Packages
> qemu | 1:2.6+dfsg-1ubuntu1 |
> http://www.las.ic.unicamp.br/pub/ubuntu-ports yakkety/main Sources
> qemu | 1:2.6+dfsg-1ubuntu1ddw1 |
> http://ppa.launchpad.net/ubuntu-virt/ppa/ubuntu yakkety/main Sources
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1541902
>
> Title:
> Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1541902/+subscriptions
>

Revision history for this message
Breno Leitão (breno-leitao) wrote : Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Just an update:

We are still working to validate and debug the PPA fixes...

Revision history for this message
Breno Leitão (breno-leitao) wrote :

Tested on IBM and the package in the PPA works fine. Thanks

We can migrate it from the PPA to the main archive.

bugproxy (bugproxy)
tags: added: bugnameltc-136627 severity-high
Revision history for this message
Ryan Harper (raharper) wrote :

Can you provide some details on how to test the GPU pass-through? I'm working on the SRU text for this and would like to include those details.

Revision history for this message
Breno Leitão (breno-leitao) wrote :

Hi Ryan,
Leonardo will provide the details regarding GPU pass-through here soon.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-08-30 14:40 EDT-------
(In reply to comment #24)
> Can you provide some details on how to test the GPU pass-through? I'm
> working on the SRU text for this and would like to include those details.

Sorry for the delay here, Ryan. GPU pass through is no different from any other PCI pass through.

On the host, the GPU appears as a regular PCI device. For instance:

$ lspci
...
0002:01:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40m] (rev a1)
...

To pass through this GPU to a VM, you'll need to use a libvirt XML such as the one in the following example:

<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0002' bus='0x01' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>

There is currently a limitation when you want to pass through two or more GPUs to the same VM. In these cases, you will need to increase the MMIO memory size of the VM. The MMIO memory size needs to be set to 32 GB * <number of GPUs being passed through to the VM>. For instance, if we are passing through two GPUs to the same VM, we will need to set the MMIO memory size to 64 GB (0x1000000000) by adding this to the QEMU command line:

-global spapr-pci-host-bridge.mem_win_size=0x1000000000

Or, equivalently, adding this to the libvirt domain XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
...
<qemu:commandline>
<qemu:arg value='-global'/>
<qemu:arg value='spapr-pci-host-bridge.mem_win_size=0x1000000000'/>
</qemu:commandline>
<domain>

If you are passing through 4 GPUs to the same VM, you will need to set the MMIO memory size to 128GB (0x2000000000) and so on.

Inside the VM the GPUs will appear as regular PCI devices.

Please, let me know if this is sufficient.

Revision history for this message
Ryan Harper (raharper) wrote : Re: [Bug 1541902] Comment bridged from LTC Bugzilla

>
>
> If you are passing through 4 GPUs to the same VM, you will need to set
> the MMIO memory size to 128GB (0x2000000000) and so on.
>
> Inside the VM the GPUs will appear as regular PCI devices.
>
> Please, let me know if this is sufficient.
>

And do you run any program or check on the device in the guest?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-08-31 13:15 EDT-------
(In reply to comment #27)
> And do you run any program or check on the device in the guest?

The GPU will be listed by lspci, just as it is in the host when it is not passed through. Also you can check if the GPU driver has been correctly loaded by the kernel with lsmod.

I don't particularly run any specific program in the guest.

Revision history for this message
bugproxy (bugproxy) wrote : 1 spapr: ensure device trees are always associated with DRC

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:50 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 2 spapr_pci: Use correct DMA LIOBN when composing the device tree

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:51 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 3 spapr_iommu: Finish renaming vfio_accel to need_vfio

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:52 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 4 spapr_iommu: Move table allocation to helpers

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:52 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 5 vmstate: Define VARRAY with VMS_ALLOC

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:53 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 6 spapr_iommu: Introduce "enabled" state for TCE table

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:53 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 7 spapr_iommu: Migrate full state

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:54 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 8 spapr_iommu: Add root memory region

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:55 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 9 spapr_pci: Reset DMA config on PHB reset

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:56 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 10 spapr_pci: Add and export DMA resetting helper

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:56 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 11 memory: Add reporting of supported page sizes

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:57 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 12 memory: Add MemoryRegionIOMMUOps.notify_started/stopped callbacks

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:58 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 13 intel_iommu: Throw hw_error on notify_started

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:58 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 14 spapr_iommu: Realloc guest visible TCE table when starting/stopping listening

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:59 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 15 vfio: spapr: Add DMA memory preregistering (SPAPR IOMMU v2)

------- Comment (attachment only) From <email address hidden> 2016-09-06 11:59 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-09-06 12:16 EDT-------
The attached series (1 - 21) is a backport of the latest DDW code (and other supporting patches), which was applied upstream in QEMU 2.7, to QEMU 2.6.1, which we're assuming is the base for 16.10. The patches should apply fairly easily to 2.6.0 as well if needed, but if there are any issues I can prepare another series.

Our hope is that this series be shipped as part of QEMU 16.10, and possibly included as an update for DDW PPA for 16.04, which used an earlier version of Alexey's DDW series.

These patches are also available via github

https://github.com/mdroth/qemu/commits/backport-ddw-2.6.1

(also, if a tarball would have been preferred instead of individual patches, please let me know so i can avoid flooding the bz/lp next time)

Revision history for this message
bugproxy (bugproxy) wrote : 16 vfio: Add host side DMA window capabilities

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:00 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 17 vfio/spapr: Create DMA window dynamically (SPAPR IOMMU v2)

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:00 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 18 spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW)

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:01 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 19 vfio/spapr: Remove stale ioctl() call

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:02 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 20 spapr: Fix undefined behaviour in spapr_tce_reset()

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:03 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 21 memory: Fix IOMMU replay base address

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:03 EDT-------

Revision history for this message
Jon Grimm (jgrimm) wrote : Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Patch set applied clean, thanks.
Uploaded to PPA here for testing (not finished building yet), to allow for some testing.

Revision history for this message
Jon Grimm (jgrimm) wrote :
Revision history for this message
bugproxy (bugproxy) wrote : 17 vfio/spapr: Create DMA window dynamically (SPAPR IOMMU v2)

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:00 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 18 spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW)

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:01 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 19 vfio/spapr: Remove stale ioctl() call

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:02 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 20 spapr: Fix undefined behaviour in spapr_tce_reset()

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:03 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : 21 memory: Fix IOMMU replay base address

------- Comment (attachment only) From <email address hidden> 2016-09-06 12:03 EDT-------

Revision history for this message
Jon Grimm (jgrimm) wrote : Re: Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM

Package built successfully across arm64, ppc64el, amd64, and i386.

As I do not have the required hardware to test the specific use-case, would you be able to kindly test the GPU passthru?

Please document the steps and tests done. Thank you.

To get the fixed package:
 sudo add-apt-repository ppa:jgrimm/qemu-1541902
 sudo apt update
 sudo apt dist-upgrade

Revision history for this message
Jon Grimm (jgrimm) wrote :

qa-regression-testing tests reported successful against my PPA by hallyn.

Waiting on reporter to test of functionality working as expected as I do not have hardware to test.

Jon Grimm (jgrimm)
Changed in qemu (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Jon Grimm (jgrimm)
Revision history for this message
Jon Grimm (jgrimm) wrote :

Filing FFe while waiting on final testing by IBM on ppc64el.

For release team:
Rationale: Request from partner for improved performance with KVM/QMU GPU passthru on ppc64el arch.
All patches from upstream 2.7, backports provided by IBM above.
Regression tests from qa-regression-testing tested by hallyn on amd64.
PPA available: https://launchpad.net/~jgrimm/+archive/ubuntu/qemu-1541902
  (amd64, i386, arm64, ppc64el)

Seeking FFe.

Revision history for this message
Breno Leitão (breno-leitao) wrote :

We were able to install this qemu from the PPA on different machines. We didn't find any regression, and DDW worked as expected.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (3.5 KiB)

------- Comment From <email address hidden> 2016-09-13 16:35 EDT-------
Some additional information from our testing:

System was a Firestone / S822LC, 2 sockets, 8 cores each (POWER8), 64GB memory, running Ubuntu 16.10 with SMT disabled (for KVM), using qemu-system-common/qemu-system-pcc/qemu-kvm packages from Jon's PPA.

We tested 2 kernels, a local build of 4.7, and 4.4.0-30-generic from the install ISO.

We tested physical function passthrough of 2 adapters, a Mellanox ConnectX-3 and ConnectX-4

We don't have any GPUs on this system. I'm still trying to get some results with GPUs, but we're having some issues with the machine we have available (unrelated to the DDW code). From IBM's perspective however the Mellanox test case is the more important driver of the DDW work/feature request so our hope is verification against Mellanox adapters is enough confirm the DDW functionality is working as intended, and that any additional requirements for GPU passthrough be handled as separate issues.

Test steps are below (retried for each adapter/kernel):

# Bind device to vfio-pci module

root@ltc-fire1:~/mdroth# lspci | grep Mellanox
0001:01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0001:01:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0004:01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

root@ltc-fire1:~/mdroth# echo 0004:01:00.0 >/sys/bus/pci/devices/0004:01:00.0/driver/unbind
root@ltc-fire1:~/mdroth# echo 0x15b3 0x1003 >/sys/bus/pci/drivers/vfio-pci/
bind module/ new_id remove_id uevent unbind
root@ltc-fire1:~/mdroth# echo 0004:01:00.0 >/sys/bus/pci/drivers/vfio-pci/bind

# Start guest

root@ltc-fire1:~/mdroth# qemu-system-ppc64 --smp sockets=1,cores=2,threads=8 --enable-kvm -M pseries -m 3G,slots=32,maxmem=8G -device virtio-blk-pci,id=disk1,drive=drive1 -drive file=ubuntu16.04.base.img.3,aio=threads,cache=none,if=none,id=drive1 -device spapr-vscsi,id=scsi0,reg=0x2000 -monitor unix:/tmp/mdroth-vm3-hmp.sock,server,nowait -qmp unix:/tmp/mdroth-vm3-qmp.sock,server,nowait -device virtio-net-pci,mac=52:54:00:12:34:01,netdev=net1,id=nic1 -netdev user,hostfwd=tcp:0.0.0.0:2223-:22,id=net1 -device vfio-pci,host=0004:01:00.0,id=vfio0 -L /root/mdroth-qemu/pc-bios -boot c -vga none -nographic

# Verify that DDW window ("window#1") gets created after guest kernel boots and passthrough device driver enables 64-bit DMA

root@ltc-fire1:~/mdroth# dmesg | tail -9
[ 5801.875253] pci 0004:01 : [PE# 000] Disabling 64-bit DMA bypass
[ 5801.875262] pci 0004:01 : [PE# 000] Removing DMA window #0
[ 5801.875840] pci 0004:01 : [PE# 000] Setting up window#0 0..7fffffff pg=1000
[ 5802.053473] pci 0004:01 : [PE# 000] Removing DMA window #0
[ 5802.101984] vfio-pci 0004:01:00.0: enabling device (0400 -> 0402)
[ 5806.123998] vfio_ecap_init: 0004:01:00.0 hiding ecap 0x19@0x18c
[ 5806.137354] KVM guest htab at c000000f2b000000 (order 26), LPID 1
[ 5810.159668] pci 0004:01 : [PE# 000] Setting up window#0 0..3fffffff pg=1000
[ 5862.518987] pci 0004:01 : [PE# 000] Setting up window#1 800000000000000..8000001ffffffff pg=10000

# Shutdown guest, verify tha...

Read more...

Revision history for this message
Jon Grimm (jgrimm) wrote :

Thank you. Awaiting FFe approval now. But looks like I may need to reapply on top of pending 2.6.1 that has an FFe in progress atm.

summary: - Enable DDW on qemu aiming GPU passthrough performance on Ubuntu KVM
+ [FFe] Enable DDW on qemu aiming GPU passthrough performance on Ubuntu
+ KVM
Revision history for this message
Steve Langasek (vorlon) wrote :

Reviewing the diff, the changes are all well-isolated to hw/ppc + target-ppc, with the following exceptions:
 - trace-events
 - a few additions to generic headers
 - 98295803b5a7929232fd92ecf17137c991273678, fc8865d5c3e733c7b03070017778b31de22c329f, 94582d197efb7030f5d0dccab6165c28fb62d5b5 and 561e77f740999600449223c912126a575c7ec3ce, which make changes to hw/vfio/common.c and memory.c which appear to be reasonably self-contained changes
 - b6dc0429cb690572984e34aa6bfe557beed82282, which makes intel iommu handling more strict; this doesn't appear to be an actual prerequisite for the rest of the ppc64el patches, but it also appears to be harmless in the sense that it will cause earlier failure in the case of driver errors
 - changes to hw/vfio/common.c in 330619403e536e0e31671ae5bcd9f9921e73d9c9 which are spapr-specific (and therefore ppc-specific in practice)

This looks like a reasonably safe change, given the testing that's been done in the ppa.

As part of this FFe, these patches should be re-tested on top of 2.6.1 (LP: #1617055).

Changed in qemu (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Jon Grimm (jgrimm) wrote :

PPA rebased on 2.6.1. I'm running qa-regression-testing libvirt/qemu tests on it now. Will upload debdiff for sponsorship once testing complete.

Revision history for this message
Jon Grimm (jgrimm) wrote :

Rebase on 2.6.1. Re-ran qa-regression-testing libvirt/qemu.

Changed in qemu (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Jon Grimm (jgrimm) wrote :

On Hold. 2.6.1 seems to be having issues landing. Removed sponsors and will likely have to rebase again.

Revision history for this message
Jon Grimm (jgrimm) wrote :

Rebased on new 2.6.1.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.6.1+dfsg-0ubuntu2

---------------
qemu (1:2.6.1+dfsg-0ubuntu2) yakkety; urgency=medium

  * Enable GPU Passthru for ppc64le (LP: #1541902)
    - 0001-spapr-ensure-device-trees-are-always-associated-with.patch
    - 0002-spapr_pci-Use-correct-DMA-LIOBN-when-composing-the-d.patch
    - 0003-spapr_iommu-Finish-renaming-vfio_accel-to-need_vfio.patch
    - 0004-spapr_iommu-Move-table-allocation-to-helpers.patch
    - 0005-vmstate-Define-VARRAY-with-VMS_ALLOC.patch
    - 0006-spapr_iommu-Introduce-enabled-state-for-TCE-table.patch
    - 0007-spapr_iommu-Migrate-full-state.patch
    - 0008-spapr_iommu-Add-root-memory-region.patch
    - 0009-spapr_pci-Reset-DMA-config-on-PHB-reset.patch
    - 0010-spapr_pci-Add-and-export-DMA-resetting-helper.patch
    - 0011-memory-Add-reporting-of-supported-page-sizes.patch
    - 0012-memory-Add-MemoryRegionIOMMUOps.notify_started-stopp.patch
    - 0013-intel_iommu-Throw-hw_error-on-notify_started.patch
    - 0014-spapr_iommu-Realloc-guest-visible-TCE-table-when-sta.patch
    - 0015-vfio-spapr-Add-DMA-memory-preregistering-SPAPR-IOMMU.patch
    - 0016-vfio-Add-host-side-DMA-window-capabilities.patch
    - 0017-vfio-spapr-Create-DMA-window-dynamically-SPAPR-IOMMU.patch
    - 0018-spapr_pci-spapr_pci_vfio-Support-Dynamic-DMA-Windows.patch
    - 0019-vfio-spapr-Remove-stale-ioctl-call.patch
    - 0020-spapr-Fix-undefined-behaviour-in-spapr_tce_reset.patch
    - 0021-memory-Fix-IOMMU-replay-base-address.patch

 -- Jon Grimm <email address hidden> Fri, 16 Sep 2016 14:14:47 -0500

Changed in qemu (Ubuntu):
status: In Progress → Fix Released
bugproxy (bugproxy)
tags: added: targetmilestone-inin1610
removed: targetmilestone-inin16041
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.