--vhost-owner --vhost-perm no more transferred to DPDK EAL commandline

No need to change dpdk identified yet, setting to incomplete for now.

FYI - I just chimed in on the discussion at https://<email address hidden>/msg69706.html

IMHO that would be the cleanest solution, no delta for any package and the same solution for all distributions.
So anybody getting here with the same issues, please feel free to "vote" for its inclusion by letting the mailing list know that you are affected.

In fact this is the new way

other_config : dpdk-extra: optional string
              Specifies additional eal command line arguments for DPDK.

              The default is empty. Changing this value requires restarting
              the daemon

So it would be something like this before starting ovs with dpdk enabled.
1. follow all you need from http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.html
2. follow all you need from http://openvswitch.org/support/dist-docs/INSTALL.DPDK-ADVANCED.md.html
3. to fixup the permissions you'd do
   ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init="--vhost-owner libvirt-qemu:kvm
--vhost-perm 0664"

I'll keep the bug open until I verified that works

See my results below.
When restarting ovs-vswitchd it complains: unrecognized option '--vhost-owner'

I use ovs master (2.6.90) and dpdk 16.07. Do I still need to apply fix-vhost-user-socket-permission.patch?

Result:
---------
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="--vhost-owner libvirt-qemu:kvm --vhost-perm 0664"

root@compute29:/var/run/openvswitch# ovs-vsctl get Open_vSwitch . other_config
{dpdk-extra="--vhost-owner libvirt-qemu:kvm --vhost-perm 0664", dpdk-init="true", dpdk-lcore-mask="0x300", dpdk-socket-mem="4096,4096"}

root@compute29:/var/run/openvswitch# export DB_SOCK=/var/run/openvswitch/db.sock
root@compute29:/var/run/openvswitch# ovs-vswitchd unix:$DB_SOCK -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir \
> --log-file=/var/log/openvswitch/ovs-vswitchd.log \
> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid \
> --detach \
> --monitor
ovs-vswitchd: unrecognized option '--vhost-owner'
EAL: Detected 32 lcore(s)
ovs-vswitchd: unrecognized option '--vhost-owner'

Versions:
----------
root@compute29:/usr/src/ovs# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.6.90
DB Schema 7.14.0

dpdk 16.07
(from: http://dpdk.org/browse/dpdk/snapshot/dpdk-16.07.zip)

Hi Timo,
thanks for the feedback.

> Do I still need to apply fix-vhost-user-socket-permission.patch?

Yes, this is to make DPDK EAL having these options.
I'd recommend the packaged versions of dpdk/openvswitch available in Yakkety as-is - especially once bumped to OVS-2.6-final once released.
But if not possible for now - yes you still have to apply that patch.

> ovs-vswitchd: unrecognized option '--vhost-owner'

I wonder that it is ovs-vswitchd that complains in your example.
It shouldn't get these as options, but maybe only the message is broken and it already is DPDK complaining since it doesn't have the patches applied.

Hi Christian,
I added the code from fix-vhost-user-socket-permission.patch to my sources above (running into build issues with 2.6.0~git20160912.dc61b4e-0ubuntu4). This fixed the unrecognized option '--vhost-owner'.. issue. But the new code is not used - a new vhu socket is still root:root. I nova booted a vm which creates vhu6ef65d50-d7 but does not change owner and permissions; ovs-vswitchd.log does not show any error. See attachment.
Thanks,
Timo

Hi Timo,
it is good that the issue with the unknown parameter are gone now.
The question is now, are these parms:
a) no more passed to OVS, but to DPDK instead (good)
or
b) no more passed to OVS, but neither to DPDK (bad)

If a) is true you should see log entries of these kind each time a DPDK socket is created:
Socket %s changed permissions to %s
Socket %s changed ownership to %s

There are also a bunch of failing paths with messages as you can see in the patch e.g. if it can't change the permission or doesn't find the user.

Do you see any of those in the EAL log?

After a reboot and restart of ovs-vswitchd I have seen these EAL logs pointing to the problem:

EAL: vhost-user socket cannot set permissions to 0664 (Unknown error -1).
EAL: vhost-user socket unable to set ownership to libvirt-qemu:kvm (Unknown error -1).
VHOST_CONFIG: vhost-user server: socket created, fd: 41
VHOST_CONFIG: bind to /var/run/openvswitch/vhu6ef65d50-d7

Then I looked into vhost-net-user.c:rte_vhost_driver_register(). I have seen that rte_eal_set_socket_permissions(path) is done before the socket actually created and then I moved this call into vhost_user_create_server() after bind() and it started to work. Attached is the patch. I hope that makes sense (also need this in vhost_user_create_client() ?) . After rebuilding dpdk and restart ovs-vswitchd it looks better and nova boot now works.

Starting ovs-vswitchd:
:::
Zone 0: name:<rte_eth_dev_data>, phys:0x1bfffced40, len:0x30100, virt:0x7f40fffced40, socket_id:1, flags:0
VHOST_CONFIG: vhost-user server: socket created, fd: 41
VHOST_CONFIG: bind to /var/run/openvswitch/vhu6c018db8-c2
EAL: Socket /var/run/openvswitch/vhu6c018db8-c2 changed permissions to 0664
EAL: Socket /var/run/openvswitch/vhu6c018db8-c2 changed ownership to libvirt-qemu:kvm.
VHOST_CONFIG: vhost-user server: socket created, fd: 48
VHOST_CONFIG: bind to /var/run/openvswitch/vhu6ef65d50-d7
EAL: Socket /var/run/openvswitch/vhu6ef65d50-d7 changed permissions to 0664
EAL: Socket /var/run/openvswitch/vhu6ef65d50-d7 changed ownership to libvirt-qemu:kvm.

root@compute29:~# ll /var/run/openvswitch/
:::
srw-rw-r-- 1 libvirt-qemu kvm 0 Sep 26 11:09 vhu6c018db8-c2=
srw-rw-r-- 1 libvirt-qemu kvm 0 Sep 26 11:09 vhu6ef65d50-d7=
srw-rw-r-- 1 libvirt-qemu kvm 0 Sep 26 11:16 vhu927b950f-f1= <<---- nova booted new vm

The attachment "Moves rte_eal_set_socket_permissions() into vhost_user_create_server()" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thanks for the continued test Timo!
Thanks for the patch suggestion and the report that this works in your case then.

I need to evaluate if the place to call that really changed or if there is more at stake.
I hope to finalize the setup I need to evaluate that today/tomorrow.

FYI - I got to pass all other bits of my DPDK tests and got to the point where this is recreated.
I collected a few other minor bugs along the way that need fixes (which is what made it taking a while), but now I'm able to take a deeper look at your patch.

horsea ovs-vswitchd[31620]: EAL: vhost-user socket cannot set permissions to 0666 (Unknown error -1).
horsea ovs-vswitchd[31620]: EAL: vhost-user socket unable to set ownership to libvirt-qemu:kvm (Unknown error -1).

If anyone comes by the initial question of this bug.
There is a log entry in "/var/log/openvswitch/ovs-vswitchd.log" that states the EAL commandline constructed out of the options one configured in the ovsdb.
Example:
"EAL ARGS: ovs-vswitchd -c 0x1 -m 8192 --pci-whitelist 0000:04:00.0 --vhost-owner libvirt-qemu:kvm --vhost-perm 0666"

Ok, I could adapt the patch in a way to cover server&client sockets IMHO.
Verification was fine against a ppa at https://launchpad.net/~paelzer/+archive/ubuntu/dpdk-bug-1625542/+packages.

I'll take a look at preparing an upload if I can complete all other open tasks tomorrow that would fall into the same upload.

@jamespage - I think about adding some comment replacing the current DPDK= config in openvswitch.
So much for the OVS portion of this bug - therefor assigning to myself for now.

For OVS the change to the config is ready for review.
On update a user of the former DPDK_OPTS will get the common:

Configuration file '/etc/default/openvswitch-switch'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:

And when looking at it is is obvious what was going on for the user:
--- /etc/default/openvswitch-switch 2016-10-04 10:06:19.043458386 +0000
+++ /etc/default/openvswitch-switch.dpkg-new 2016-10-04 09:07:08.000000000 +0000
@@ -7,5 +7,13 @@
 # a suitable place to specify --ovs-vswitchd-wrapper=valgrind.
 # OVS_CTL_OPTS=

-# DPDK options - see /usr/share/doc/openvswitch-common/INSTALL.DPDK.md.gz
-DPDK_OPTS='--dpdk -c 0x1 -n 4'
+# DPDK options are now configured via ovs-vsctl/ovsdb, see:
+# - /usr/share/doc/openvswitch-common/INSTALL.DPDK.md.gz
+# - /usr/share/doc/openvswitch-common/INSTALL.DPDK-ADVANCED.md.gz
+# dpdk-extra is the fallback for users who formerly set e.g.
+# '--vhost-owner libvirt-qemu:kvm --vhost-perm 0664'
+# here or any other DPDK EAL arguments via DPDK_OPTS which are not covered by
+# the default switches provided by ovs-vsctl.
+# This is an arbitrary string that will get appended to the EAL commandline.
+# For more on the new ovsdb based configuration of DPDK features, see:
+# `man ovs-vswitchd.conf.db`

This bug was fixed in the package dpdk - 16.07-0ubuntu5

---------------
dpdk (16.07-0ubuntu5) yakkety; urgency=medium

  [ Christian Ehrhardt ]
  * Fix component mismatch by dropping the optional dpdk-pmdinfo tool
    (LP: #1630073).

  [ Gowrishankar Muthukrishnan ]
  * update d/p/dpdk-dev-examples-ip_pipeline-fix-pmd-driver-parameter.patch to
    fix dlopen issue (LP: #1630119)

 -- Christian Ehrhardt <email address hidden> Tue, 04 Oct 2016 09:27:54 +0200

The changes are purely non-functional.
DPDK was working with the new config as described fine before, to be sure I was running a few hings over "normal" OVS as built from this just to be sure. This is the logfile.

Fix for openvswitch in the upload queue

This bug was fixed in the package openvswitch - 2.6.0-0ubuntu2

---------------
openvswitch (2.6.0-0ubuntu2) yakkety; urgency=medium

  * Updates to cover the new way Openvswitch is configurging DPDK (LP: #1625542)
    - d/openvswitch-switch.default
    - d/openvswitch-switch-dpdk.README.Debian
    - drop d/p/ovs-ctl-dpdk.patch

 -- Christian Ehrhardt <email address hidden> Tue, 04 Oct 2016 11:07:08 +0200