Openvpn routing issue

Bug #1787208 reported by Martin Wolf
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenVPN
Unknown
Unknown
openvpn (Debian)
Fix Released
Unknown
openvpn (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * non-default but still common openvpn setups use callout scripts with
   sudo (if the openvpn user was set up to work with sudo). That breaks in
   >=Bionic since CAP_AUDIT_WRITE was dropped which makes pam/sudo denying
   the call.

 * We brought the change upstream and want to backport into Cosmic/Bionic
   to avoid Xenial upgrades to hit this.

 * Interesting is that the upstream .deb is not affected by still having
   Xenial rules:
 =>https://github.com/OpenVPN/sbuild_wrapper/tree/master/packaging/xenial/debian

[Test Case]

 * details in https://www.techgrube.de/tutorials/openvpn-server-mit-ipv4-
und-ipv6 which the reporter and I followed (warning: non commands are
   german)

 * there is no need to do any of the IPV6 stuff in the guide nor the
   iptables actions

 TL;DR would be:
 * apt install openvpn (on client and server)
   $ sudo apt install openvpn easy-rsa
   Use easy-rsa to create 1 server and 1 client certificate
   See the link above for commands to do so if you are unfamiliar
 * add "openvpn" user and grant him sudo permission for your test script
   $ addgroup --system --no-create-home --disabled-login --group openvpn
   $ adduser --system --no-create-home --disabled-login --ingroup openvpn openvpn
 * add server/client config as outlined in comment #25
   the important bit is to have a sudo call to a helper like:
     learn-address "/usr/bin/sudo -u root /etc/openvpn/scripts/test.sh"
client.conf
client
dev tun
proto udp
remote 192.168.122.29 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/easy-rsa/pki/ca.crt
cert /etc/openvpn/easy-rsa/pki/issued/guest1.crt
key /etc/openvpn/easy-rsa/pki/private/guest1.key
remote-cert-tls server
cipher AES-256-CBC
tls-version-min 1.2
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
auth SHA512
comp-lzo
verb 6
explicit-exit-notify

server.conf
port 1194
proto udp
dev tun
ca /etc/openvpn/easy-rsa/pki/ca.crt
cert /etc/openvpn/easy-rsa/pki/issued/server.crt
key /etc/openvpn/easy-rsa/pki/private/server.key
dh /etc/openvpn/easy-rsa/pki/dh.pem
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
script-security 2
push "redirect-gateway def1"
push "dhcp-option DNS 8.8.8.8"
keepalive 10 120
tls-version-min 1.2
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
auth SHA512
cipher AES-256-CBC
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 6
user openvpn
group openvpn

 * Create the test script
   $ sudo mkdir -p /etc/openvpn/scripts/
   $ sudo echo "id" >> /etc/openvpn/scripts/test.sh
   $ sudo chmod +x /etc/openvpn/scripts/test.sh
 * Start the server service and run journalctl -f
 * Let the client connect (you will see the denies on the server)

[Regression Potential]

 * It adds one allowed capability (a rather safe one btw) to the service
   of openvpn. There should be no regression risk breaking functional
   setups.
   If anything security concerns, but since it was this way in Xenial even
   that should not apply

[Other Info]

 * This was in Xenial, picked by upstream for their own .deb package but
   not integrated in their actual repository. Debian by aligning with
   upstream dropped it and we followed. This time we made sure it gets
   upstream and therefore hopefully should not reoccur again

---

I updated my Server from xenial to bionic today. on xenial I was using the openvpn repo from the openvpn developers (https://community.openvpn.net/openvpn/wiki/OpenvpnSoftwareRepos)

now that bionic ships a more recent version I removed the ppa and switched to the distro version (2.4.4)

my openvpn server assings a real ipv6 address and does nat for ipv4 forevery client. Also i push a route so a /64 ipv6 net and one ipv4 address is reachable through the tunnel.
(I have firewalled a server so it is only reachable through the tunnels ips)
With openvpn 2.4.4 from bionic repo this does not work anymore, aka the server is not reachable anymore.
I quicky reactivated the xenial repo from https://community.openvpn.net/openvpn/wiki/OpenvpnSoftwareRepos and upgraded the server again (to 2.4.6)
after a restart I was able to reach my server again.

so most likely there is a bug in bionics 2.4.4 version of openvpn

client config:

client
dev tun
proto udp
remote <ipv4-address> 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert martin-pc.crt
key martin-pc.key
remote-cert-tls server
tls-crypt ta.key
cipher AES-256-GCM
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384
auth SHA512
comp-lzo
explicit-exit-notify
pull-filter ignore "route"
pull-filter ignore "dhcp"
pull-filter ignore "redirect"
route-ipv6 <ipv6-net i want to reach>/64 <ipv6 ip of server> 1
route <server i want to reach ipv4> 255.255.255.255 10.8.0.1 1

server config:

port 1194
proto udp
dev tun
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh4096.pem
topology subnet
server 10.8.0.0 255.255.255.0
server-ipv6 <ipv6 net usable for clients>/112
ifconfig-pool-persist ipp.txt
push "route-ipv6 2000::/3 <ipv6 server ip> 1"
script-security 2
learn-address "/usr/bin/sudo -u root /etc/openvpn/scripts/ndp-proxy-setup.sh"
push "redirect-gateway def1"
push "redirect-gateway ipv6"
push "dhcp-option DNS 1.1.1.1"
keepalive 10 120
tls-crypt /etc/openvpn/easy-rsa/keys/ta.key
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384
auth SHA512
cipher AES-256-GCM
#compress lz4
comp-lzo
persist-key
persist-tun
status openvpn-status.log
#verb 6
user openvpn
group openvpn

Related branches

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvpn (Ubuntu):
status: New → Confirmed
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Odd, Ubuntu has no particular Delta on the 2.4.4 package.
I'd try to make you an 2.4.6 package available - that I'd want to do for the upcoming Ubuntu Cosmic release anyway.

If it is good in there we at least know there really is an upstream fix between 2.4.4 and 2.4.6 that has to be found to fix this issue here in Bionic as well.

I'll take a look at that later today.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hey, could you test Ubuntu Cosmic plus the 2.4.6 version in [1].
I would hope to get that in just before Feature Freeze and if it would contain the fix you need at least Cosmic would be good. From there we would need to find what between 2.4.4 and 2.4.6 is missing for your case.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3363

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

hello,
with your version I get these errors when I connect with my windows client (just tested the windows one):

Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: unable to send audit message
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: pam_open_session: System error
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: policy plugin failed session initialization
Aug 20 17:37:42 name_of_my_vpn_server ovpn-server[1111]: martin-pc/<client-ipv4-address>:51120 WARNING: Failed running command (--learn-address): external program exited with error status: 1
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: unable to send audit message
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: pam_open_session: System error
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: policy plugin failed session initialization
Aug 20 17:37:42 name_of_my_vpn_server ovpn-server[1111]: martin-pc/<client-ipv4-address>:51120 WARNING: Failed running command (--learn-address): external program exited with error status: 1

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hmm, that would mean the new upstream isn't as good as we 'd have hoped :-/
And you said the same 2.4.6 but from a PPA or so works fine?

Could you refer me to that PPA, maybe I can pull in some modifications from there that would help Ubuntu as well.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

I use the original openvpn repo for 16.04 xenial
https://community.openvpn.net/openvpn/wiki/OpenvpnSoftwareRepos

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: [Bug 1787208] Re: Openvpn routing issue

On Tue, Aug 21, 2018 at 12:30 PM Martin Wolf <email address hidden>
wrote:

> I use the original openvpn repo for 16.04 xenial
> https://community.openvpn.net/openvpn/wiki/OpenvpnSoftwareRepos

Unfortunately they don't seem to publish deb-src to learn what they did.
I need to ask there ...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'm discussing with upstream what they might do different, but so far all seems that those should be the same - not sure what you hit.
If we can identify a bug we can still work on it down the road after Feature Freeze, but in general the 2.4.6 looks good and fixes plenty of other things for the overall Ubuntu community.

You'll soon find 2.4.6 in Ubuntu cosmic and can try the other clients you had if they behave differently.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

Do I have to do something else beside remove the old package with apt remove openvpn, service openvpn@server stop, install the new one with dpkg - i openvpn_2.4.6-1ubuntu1_amd64.deb and server openvpn@server start ?

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :
Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Well other than the openvpn repo which builds/pushes for Xenial and asks you to just try on other releases this is build for Ubuntu 18.10 Cosmic.

It is not intended to work anywhere else and the SRU [1] policy forbids to backport the full new version.

So TL;DR to be able to use it you'd upgrade to the very latest Ubuntu release.
That said 18.10 Cosmic is in development right now, so some areas might be incomplete until released around October.

Good for testing, maybe not yet for production use.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

I am not sure if this helps, but if you look at line 16 to 20 in that diff the xenial init file contains a "script security" passage while the cosmic does not.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

I hit return too fast, I mentioned it since I use "script-security 2" in my server config

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The init script isn't used anymore instead it is systemd.
Therefore since Bionic the script security setting is at the systemd service:

--- a/debian/openvpn@.service
+++ b/debian/openvpn@.service
@@ -13,7 +13,7 @@ Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
 Type=notify
 PrivateTmp=true
 WorkingDirectory=/etc/openvpn
-ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
+ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
 PIDFile=/run/openvpn/%i.pid
 KillMode=process
 ExecReload=/bin/kill -HUP $MAINPID

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

yeah you are right,
so what can I do now?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

There won't be an ubuntu provided 2.4.6 for the older Ubuntu Releases for all the potential regression risks that has for all the other users [1].

I think we have three options now:
1. If you strictly need 2.4.6 which only is in Ubuntu Cosmic you'd have to upgrade to that version
2. If you can identify what was changed in between 2.4.x and 2.4.6 fixing your issue we can evaluate if backporting just that change would be ok for [1] or not.
3. you switch back to the upstream deb not supported by Ubuntu if that works for you
4. you pull in just openvpn from cosmic repositories with apt pinning (as not-recommended as #3 IMHO)

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

I downloaded the openvpn 2.4.6 from here: https://launchpad.net/ubuntu/+source/openvpn/2.4.6-1ubuntu1/+build/15288801
and I have the same issues.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'm never a fan for direct download deb installs, but thanks for the test.
So 2.4.6 has the same issues as 2.4.4 - is that what I read in the last update?

Hmm, the case itself is too complex for my openvpn-foo :-/

Again I'm still puzzled - to confirm - the upstream provided 2.4.6 deb is good on the same system/setup as the one you just had issues with the 2.4.6 you downloaded from Cosmic?

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

let me clarify it for you.
the package from repo: deb http://build.openvpn.net/debian/openvpn/stable xenial main
 works without any issues
while the package from bionic repo, the repo you provided and the cosmic deb file produce the same errors as listed in #4

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Odd, thanks for the clarification - that was absolutely helpful.

I'll try a repro with the config you initially added here tomorrow.

You were suspicious about the script security parameter before.
In your case with the 2.4.6 version, it clearly should set that from the systemd service script.
Could you check when the service is running how it was started.

I think a simple output of:
 "systemctl status openvpn"
should show all we need.

Just to make sure it actually is started with --script-security in your case as well.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :
Download full text (4.5 KiB)

This is "systemctl status <email address hidden>" with the openvpn - xenial repo (the functional one) - openvpn (2.4.6-xenial0)

● <email address hidden> - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-23 13:15:11 CEST; 6h ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
 Main PID: 620 (openvpn)
    Tasks: 1 (limit: 2260)
   CGroup: /<email address hidden>
           └─620 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/server.conf --writepid /run/openvpn/server.pid

Aug 23 20:20:18 name_of_my_vpn_server ovpn-server[13088]: <client-ipv4-address>:61079 peer info: IV_TCPNL=1
Aug 23 20:20:18 name_of_my_vpn_server ovpn-server[13088]: <client-ipv4-address>:61079 peer info: IV_GUI_VER=OpenVPN_GUI_11
Aug 23 20:20:18 name_of_my_vpn_server ovpn-server[13088]: <client-ipv4-address>:61079 [martin-pc] Peer Connection Initiated with [AF_INET]<client-ipv4-address>:61079
Aug 23 20:20:18 name_of_my_vpn_server ovpn-server[13088]: martin-pc/<client-ipv4-address>:61079 MULTI_sva: pool returned IPv4=10.8.0.2, IPv6=<openvpn-client-ipv6-address>
Aug 23 20:20:18 name_of_my_vpn_server sudo[13117]: openvpn : TTY=unknown ; PWD=/etc/openvpn ; USER=root ; COMMAND=/etc/openvpn/scripts/ndp-proxy-setup.sh add 10.8.0.2 martin-pc
Aug 23 20:20:18 name_of_my_vpn_server sudo[13117]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:20:18 name_of_my_vpn_server sudo[13117]: pam_unix(sudo:session): session closed for user root
Aug 23 20:20:18 name_of_my_vpn_server sudo[13119]: openvpn : TTY=unknown ; PWD=/etc/openvpn ; USER=root ; COMMAND=/etc/openvpn/scripts/ndp-proxy-setup.sh add <openvpn-client-ipv6-address> martin-pc
Aug 23 20:20:18 name_of_my_vpn_server sudo[13119]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:20:18 name_of_my_vpn_server sudo[13119]: pam_unix(sudo:session): session closed for user root

This is "systemctl status <email address hidden>" with the cosmic openvpn_2.4.6-1ubuntu1_amd64.deb
● <email address hidden> - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-23 20:14:48 CEST; 1s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
 Main PID: 10980 (openvpn)
   Status: "Initialization Sequence Completed"
    Tasks: 1 (limit: 2260)
   CGroup: /<email address hidden>
           └─10980 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/server.conf --writepid /run/openvpn/server.pid

Aug 23 20:16:13 name_of_my_vpn_server openvpn[10980]: sudo: policy plugin failed session initialization
Aug 23 20:16:13 name_of_my_vpn_ser...

Read more...

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

both "status" reports show a connect from one of my clients.

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

and since you want to try out my config. I followed this guide here:
https://www.techgrube.de/tutorials/openvpn-server-mit-ipv4-und-ipv6

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (28.1 KiB)

I followed the linked howto sans the ipv6 pieces as I had no ipv6 target range anywhere to be used.
I can connect just fine with that.
Before I do that it would be easier for you to check if dropping the ipv6 lines from the server conf make it work.
If it does - then we know it only affects the ipv6 handling that you have set up.
If it does not work we can kill all thoughts on ipv6 while searching for the root cause.

I'm still wondering why the version from upstream should work, at least that is a lead we can continue to try to sort out. The systemd file of Debian/Ubunut seems a bit more evolved to me, but maybe one of these configs is the reasons that your specific case fails.
I compared the services:
--- upstream.service 2018-08-24 07:40:11.302369502 +0000
+++ ubuntu.service 2018-08-24 07:37:09.014566529 +0000
@@ -3,25 +3,28 @@
 PartOf=openvpn.service
 ReloadPropagatedFrom=openvpn.service
 Before=systemd-user-sessions.service
+After=network-online.target
+Wants=network-online.target
 Documentation=man:openvpn(8)
-Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
+Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
 Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

 [Service]
+Type=notify
 PrivateTmp=true
-KillMode=mixed
-Type=forking
+WorkingDirectory=/etc/openvpn
 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
 PIDFile=/run/openvpn/%i.pid
+KillMode=process
 ExecReload=/bin/kill -HUP $MAINPID
-WorkingDirectory=/etc/openvpn
-ProtectSystem=yes
-CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
+CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
 LimitNPROC=10
 DeviceAllow=/dev/null rw
 DeviceAllow=/dev/net/tun rw
+ProtectSystem=true
+ProtectHome=true
+RestartSec=5s
+Restart=on-failure

 [Install]
 WantedBy=multi-user.target

In many cases I think the changes are fine for sure, but a few would be candidates for you to try.
IMHO Those worth to try are:
#1
-CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
+CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
#2
+ProtectSystem=true
#3
+ProtectHome=true

I'd ask you to check the following:
1. please verify without ipv6 settings (you see in my configs below which ones I removed) if the issue persists
2. Try the three changes - best would be to set up the upstream version that works and then add the changes #1/#2/#3 one by one restarting and retrying. Maybe one of those blocks something that is needed in your case?

Looking forward to hear from you ...

My test setup:

client.conf
client
dev tun
proto udp
remote 192.168.122.29 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/easy-rsa/pki/ca.crt
cert /...

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

I found it far more easy to test the three test cases first.

I. (first I test them seperately)

1. CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE

this one already breaks it:
Failed running command (--learn-address): external program exited with error status: 1

2. ProtectSystem=true
this line seems to work fine. I get no visible errors and the routing works.

3. ProtectHome=true
this line seems to work fine as well

II. (now I test the two working lines together)

ProtectSystem=true
ProtectHome=true

these two lines seem to work fine together, I still can reach my firewalled server

III. (all three together - since the error message on I.1. is only a fraction of bionic / cosmic repo build.)

CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
ProtectSystem=true
ProtectHome=true

again I get
"WARNING: Failed running command (--learn-address): external program exited with error status: 1" and therefore I cant reach my firewalled server.

I just want to mention, that this errormessage is just a fraction of the original error message I got in #4 (the sudo part is missing here)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Nice, thanks for the verification of my suggestions.

So effectively we gain CAP_DAC_OVERRIDE and loose CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE.
I'll need to check the hostory how/why they were added/dropped respectively, but that sounds like a fix that certainly can be done - much better then when we only knew "something changed" :-)

I won't get to that today thou ...

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

Just please keep in mind, that there are also these errors in the bionic repo, that are not present in the upstream repo.
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: unable to send audit message
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: pam_open_session: System error
Aug 20 17:37:42 name_of_my_vpn_server openvpn[1111]: sudo: policy plugin failed session initialization

About disabling ipv6 ...
maybe I find the time to set up another Vserver @Hetzner with 18.04 for testing this specific issue since I dont want to stop/alter my production system for too long.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Martin,
I'd have thought that the change of the allowed capabilities would at least fix the first of those errors maybe even more than just the first - as we are explicitly closing the gap of CAP_AUDIT_WRITE.

If you still have the logs of the checks for #1/#2/#3 that you did in comment #26 - it would be interesting which minimal combination of changes made all these errors go away.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

For now we have:
- CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE - helps to call external programs as needed for some setups

The above IMHO does not jeopardize the default config as much as dropping all boundary caps limiting would. - so the change above can be pursued.

I was re-reading a few times: I read your comment #26 - especially the last paragraph as "upstream deb package + #1/#2/#3 set" does not trigger the sudo/audit related messages.

I think the sudo issues are a consequence of the non-root setup.
There is a discussion on it in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792653 trying very similar changes.

The TL;DR is that the default configuration protects the default setup (which is with privileged users) and local overrides can be added if needed.

I have not found a good way to help with the other messages yet on top on what was discussed on the referenced Debian bug - i.e. nothing that I'd want to change in packaging.

Changing the Caps I'd consider reasonable, I might get in touch with a few others thou to be sure.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I focused a bit on learn-address.
It behaves quite normal.

There were more tests involved, but I always went back and try to find the minimal changes.

- without sudo setup I see "openvpn : user NOT in sudoers" (as expected)
# Change: add sudo to openvpn
- with sudo I get
  - sudo: unable to send audit message
  - sudo: /etc/openvpn/scripts/test.sh: command not found
# Change: make sure the scripts are +x (I verified that CAP_DAC_READ_SEARCH was not needed to avoid this)
  - sudo: unable to send audit message
  - sudo[6345]: PAM audit_log_acct_message() failed: Operation not permitted
  - sudo[6345]: pam_unix(sudo:session): session opened for user root by (uid=0)
  - sudo[6345]: openvpn : pam_open_session: System error ; TTY=unknown ; PWD=/etc/openvpn ;
       USER=root ; COMMAND=/etc/openvpn/scripts/test.sh add 10.8.0.2 guest1
  - sudo: pam_open_session: System error
  - sudo: policy plugin failed session initialization
# Change: add CAP_AUDIT_WRITE
  - as before (even the audit messages stayed as is)
# Change: CAP_DAC_READ_SEARCH
  - as before
# Change: Drop all Bounding set (=~ to have no bounds)
  - as before
# Change ProtectSystem=false
  - as before

Now that we are here that brings us to this being more of a re-occuring thins than I thought.
See: https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1511524
And related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=866523
Which lead to: https://community.openvpn.net/openvpn/ticket/918

But as of my tests atm, adding CAP_AUDIT_WRITE ?no more? is enough to get it to work these days.
Also Upstream hasn't changed anything for ticket 918.
Upstream did take it into This repo though: https://github.com/OpenVPN/sbuild_wrapper/tree/master/packaging/xenial/debian

Which is a copy of that times Ubuntu/Debian packaging.

Now by using what upstream provides in the main repo we differ in our systemd unit, and upstream packaging not their own ./distro/systemd/openvpn-server@.service.in into their .deb but instead a different service makes it work (at least for your case here with "CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE".

But neither of these avoids the pam/audit errors that we see.
IMHO we will need to:
- identify a path to get sudo working again in these cases
- once known need to decide if that can be a default ocnfig or needs to be an admins choice
- check how much of CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE or similar we need
- wake up the upstream bug on this and provide a PR to include the rules that they include in their own .dbe
- Adapt Ubuntu and Debian to do so as well.

If time permits I'll try more configs, but I can't make any promises.
This becomes more interesting, but also seems harder to solve the longer I look at it :-)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I was wondering, since I even set the Caps to =~ (=no cap limitation) and still get the Audit/Pam issues I'm wondering what the remaining difference would be - to confirm these messages/issues go away with the upstream 2.4.6 package as well for you or are they affected by this symptom just as much?

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

sorry for being silent for so long.
Today I tested openvpn_2.4.6-1ubuntu1_amd64.deb from cosmic with the openvpn@.service from upstream and I get no errors when my clients connect.
If we change the openvpn@.service file in cosmic / bionic according to the upstream file we should have a solution ...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (3.5 KiB)

Hi Martin,
no problem.

Exchaning the file "completely" might be too much unless we can reason all the changes.
Some I'd think were made in Debian/Ubuntu for valid reasons and should be kept.
So if I might ask you continue your test for combinations of the delta.

I only reordered things to make it more readable - overall the change from Ubuntu/Debian to upstream would be:

--- debian.openvpn@.service 2018-08-28 14:58:25.145712079 +0200
+++ upstream.openvpn@.service 2018-08-28 15:03:00.221966742 +0200
@@ -3,28 +3,23 @@
 PartOf=openvpn.service
 ReloadPropagatedFrom=openvpn.service
 Before=systemd-user-sessions.service
-After=network-online.target
-Wants=network-online.target
 Documentation=man:openvpn(8)
-Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
+Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
 Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

 [Service]
-Type=notify
+Type=forking
 PrivateTmp=true
 WorkingDirectory=/etc/openvpn
 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
 PIDFile=/run/openvpn/%i.pid
-KillMode=process
+KillMode=mixed
 ExecReload=/bin/kill -HUP $MAINPID
-CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
+CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
 LimitNPROC=10
 DeviceAllow=/dev/null rw
 DeviceAllow=/dev/net/tun rw
-ProtectSystem=true
-ProtectHome=true
-RestartSec=5s
-Restart=on-failure
+ProtectSystem=yes

 [Install]
 WantedBy=multi-user.target

Let us break this into categories:

1. Distro is better, we should not take this change (in fact we should file upstream issue to change there)
-After=network-online.target
-Wants=network-online.target
-Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
+Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage

2. we already know we want to follow this change in the Distro:
-CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
+CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
=> Probably a fusion of the capabilities here eventually

3. I think these are better than upstream, but we need to check which one of it is the reason for your issue
-ProtectSystem=true
+ProtectSystem=yes
(should be a no-op IMO)

-RestartSec=5s
-Restart=on-failure
-KillMode=process
+KillMode=mixed
These two are in fact copied over from ustraem (remember the actual upstream, not their deb package creation - there I'd hope that these are not the reason)

-ProtectHome=true

-Type=notify
+Type=forking

If you could apply #2 in any case and then test the sections of #3 one by one and report back here which one solve it for you that would be great.

I realized that all our changes ...

Read more...

Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

"CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE"

This line is sufficient to get me connected to my firewalled server. I dont get any "sudo pam/policy" or "Warning --learn" errors.

my second test was with this line:
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_AUDIT_WRITE

I just added CAP_AUDIT_WRITE and I still have no errors and a working connection to my firewalled machine, so we basically need just that, right?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Yes in that case "CAP_AUDIT_WRITE" is what I'd add back then - especially for an SRU minimal reasonable changes are required.

Thanks for working with me to sort this out (again since https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868806 is the same)

I beg your pardon, but let me open an Upstream discussion to agree on this move to avoid even more confusion. Actually lets jump onto the existing bug, I did so at [1].

Upstream bug: https://community.openvpn.net/openvpn/ticket/918
Debian Bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868806

[1]: https://community.openvpn.net/openvpn/ticket/918#comment:5

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Changed in openvpn (Debian):
status: Unknown → Confirmed
Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

is there an ETA for the new version?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The discussion upstream has not settled, so no ETA yet.
But since the change will just be a modification of the config file you can until then help yourself by overriding the capability boundaries.

Changed in openvpn (Ubuntu Bionic):
status: New → Triaged
Changed in openvpn (Ubuntu Cosmic):
status: Confirmed → Triaged
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I updated the Debian bug to pick the change as upstream accepted now (my submit was committed a few minutes ago).

We can pick that up on the next merge, but actually lets fix it for Cosmic and later Bionic (SRU policy).

Change is trivial, only a bugfix (Feature Freeze is in place).
I prepared a MP to review at https://code.launchpad.net/~paelzer/ubuntu/+source/openvpn/+git/openvpn/+merge/354174

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Also prepped the SRU Template to help reviewers ...

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I also submitted a request upstream to sync their .debs with their own main source repo.
Maybe that will eliminate some confusion later on.

description: updated
Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

Great, I am already running openvpn 2.4.4 with "CAP_AUDIT_WRITE" for a day or two. I am looking forward to see it fixed without having to edit the file manually.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.4.6-1ubuntu2

---------------
openvpn (2.4.6-1ubuntu2) cosmic; urgency=medium

  * d/openvpn@.service: Add CAP_AUDIT_WRITE to avoid issues with callout
    scripts breaking due to sudo/pam being unable to audit the action.
    Fixed in upstream issue #918, suggested to Debian in #868806 (LP: #1787208)

 -- Christian Ehrhardt <email address hidden> Mon, 03 Sep 2018 10:57:35 +0200

Changed in openvpn (Ubuntu Cosmic):
status: Triaged → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I think all prep is complete, uploaded for the consideration by the SRU team

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Martin, or anyone else affected,

Accepted openvpn into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.4-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openvpn (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

Hi Brian,
sorry that it took longer than expected to test the new package.
I have installed it right now and it seems to be working. I have no errors in my openvpn logfile and I can reach my firewalled machine through the tunnel.
If you dont hear till monday from me then I would give this a go.
Ty for Christians and your help.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Verified by reporter, setting tags ...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for openvpn has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.4.4-2ubuntu1.1

---------------
openvpn (2.4.4-2ubuntu1.1) bionic; urgency=medium

  * d/openvpn@.service: Add CAP_AUDIT_WRITE to avoid issues with callout
    scripts breaking due to sudo/pam being unable to audit the action.
    Fixed in upstream issue #918, suggested to Debian in #868806 (LP: #1787208)

 -- Christian Ehrhardt <email address hidden> Wed, 05 Sep 2018 14:43:16 +0200

Changed in openvpn (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in openvpn (Debian):
status: Confirmed → Fix Released
Revision history for this message
Martin Wolf (mwolf-adiumentum) wrote :

can you please put this fix also in the openvpn-server@ script?
because the error seems to be there as well.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Martin - I filed this in bug 1828771 to have a clear tracking not affecting the old report too much. If you happen to have a nice setup howto how to trigger the issue please feel encouraged to add it there. Otherwise I'll try to modify what we have used here which hopefully will work.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.