strongswan ipsec fails to finish connection (hangs after installing DNS server via resolvconf)

Hi Fermulator,
while I regularly pick up new strongswan versions I'm not enough of an expert to give good suggestions on what the reason might be.

But fortunately there are often a few community members here using strongswan that can chime in.
Never the less for a technical discussion on the very low details of your connection hang an upstream issue report might be useful.

I'd ask you to report the ID you get here so that we can link it and follow the discussion.
Also include as much of your setup (simplified and stripped of confidential things) to the bug here and there. As it would help a lot if we'd get it to a list of steps to locally reproduce the issue.

Version-wise you said 17.10 was good but in 18.04 it is showing this behavior.
In terms of versions that would be 5.5.1 -> 5.6.2 - maybe that rings a bell for upstream?

FYI upstream issue tracker is at https://wiki.strongswan.org/projects/strongswan/issues

Upstream report: https://wiki.strongswan.org/issues/2724

I am having a similar problem, but I have some additional tidbits:
1) I had the same issue after upgrading from 17.10 to 18.04, but before rebooting
2) I have the same problem if I downgrade all the strongswan/charon packages to 5.5.1-4ubuntu2.2
3) First connection attempt after reboot does not always work
4) "sudo service strongswan restart" has some probability that the next connection attempt will work, but no guarantee

Thanks Fermulator, no auto-import from this type of bug tracker, but the URL is great to check!.
Since remote tracking can't be done automatically, can you ping here once something interesting is coming back there?

(yes, the upstream report back-refs this downstream report, myself or someone else should do the needful communication once upstream is fixed)

So, I've noticed something else that may be relevant. My work's configuration actually has two DNS servers, which show up like so:
installing DNS server 10.10.0.8 via resolvconf
installing DNS server 10.10.0.2 via resolvconf
However, I saw that there was some probability that it failed after either log message. In addition to that, there is a "leftupdown" script that makes an _additional_ call to resolvconf to setup the DNS search (and tear it down afterwards). I also (sometimes) locks up at these times, so I disabled that script and noticed more successful runs.

So, suspecting that the problem was with resolvconf I downgraded it to 1.79ubuntu8. That didn't do the trick, but these days resolvconf is managed by systemd - so I then downgraded systemd to 234-2ubuntu12.1 (except for a conflict with netplan.io, which I ignored). That "worked" in an interesting way, now it reliably connects and finishes - but sometimes it takes about 10 seconds to complete each resolvconf transaction.

Based on this, I suspect that the issue is actually somehow in calling resolvconf (if I call resolvconf in a terminal then I don't see a lockup).

I think I may have somewhat figured it out. @fermulator, could you get it to lockup and then run "sudo killall host" in another terminal window?

It looks like this is some sort of integration issue between the latest bind9-host (used by avahi-daemon) and the latest systemd, downgrading either one of these packages appears to resolve the issue for me. Further, digging through the resolvconf scripts I found that it appears to be hanging when "host" is called by /usr/lib/avahi/avahi-daemon-check-dns.sh, so interrupting that tasks seems to bypass the issue.

@ehoover; great isolation! that's absolutely it

I too noticed it hangs in several other ways randomly.

One other example I had not mentioned was failing here:
{{{
Aug 20 09:02:49 fermmy charon[3698]: 14[IKE] maximum IKE_SA lifetime 56591s
Aug 20 09:02:49 fermmy charon[3698]: 14[IKE] processing INTERNAL_IP4_ADDRESS attribute
Aug 20 09:02:49 fermmy charon[3698]: 14[IKE] processing INTERNAL_IP4_DNS attribute
<<HANGS>>
}}}

During this state:
{{{
$ ps wuaxxx | grep host
root 8526 0.0 0.0 187628 8792 ? Sl 09:02 0:00 host -t soa local.
}}}

Then:
{{{
$ sudo killall host
}}}

tada: (this immediately followed log message then appears)
{{{
Aug 20 09:04:24 fermmy charon[3698]: 10[IKE] removing DNS server 192.168.194.20 via resolvconf
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] installing DNS server 192.168.194.20 via resolvconf
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] processing INTERNAL_IP4_DNS attribute
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] installing DNS server 192.168.196.20 via resolvconf
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] installing new virtual IP 192.168.130.4
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] CHILD_SA wtlsecondary{72} established with SPIs c557c437_i 9ae567d1_o and TS 192.168.130.4/32 === 10.0.0.0/8 172.16.0.0/12 192.168.128.0/17
Aug 20 09:04:24 fermmy charon[3698]: 14[IKE] CHILD_SA wtlsecondary{72} established with SPIs c557c437_i 9ae567d1_o and TS 192.168.130.4/32 === 10.0.0.0/8 172.16.0.0/12 192.168.128.0/17
}}}

it's definitely resolvconf related :( -- and as you suggest

we should move this defect to the appropriate sub-category then

fwiw
{{{
$ dpkg --list | grep avahi-daemon
ii avahi-daemon 0.7-3.1ubuntu1 amd64 Avahi mDNS/DNS-SD daemon
$ dpkg --list | grep resolvconf
ii resolvconf 1.79ubuntu10 all name server information handler
}}}

bind9-host is deprecated anyway isn't it?
Could the avahi shell script easily be modified to try dig at the same place?

Also as requested since you isolated it towards resolveconf I'm adding a bug task for that as well.

@paelzer, it is making an usual call to host and checking the return:
LC_ALL=C host -t soa local. 2>&1
and theoretically this maps to:
LC_ALL=C dig -t soa local. 2>&1

Practically, I don't think this works. I do not have a "local." start of authority record and when I run these commands I get a 1 return value from host and a 0 return value from dig. It might be possible that you can parse the output of dig to get the same information, but I don't know what dig outputs when "local." shows up with an SOA record. If I recall correctly, this should only happen on _very_ broken networks. The avahi folks might know more about how to proceed.

Hi,
I just realized with you tracking it down so far - could it be "another symptom" of bug 1752411 ?

If so we should add strongswan there as affected and make this bug a Dup, so that ALL effects of this one issue are in one place.

I'd be glad if one of you could make the detail check if you think it really is the same.

$ dpkg --list | grep bind9
ii bind9-host 1:9.11.3+dfsg-1ubuntu1.1 amd64 DNS lookup utility (deprecated)
ii libbind9-160:amd64 1:9.11.3+dfsg-1ubuntu1.1 amd64 BIND9 Shared Library used by BIND
rc libbind9-80 1:9.8.1.dfsg.P1-4ubuntu0.9 amd64 BIND9 Shared Library used by BIND
rc libbind9-90 1:9.9.5.dfsg-3ubuntu0.8 amd64 BIND9 Shared Library used by BIND

Yes, this is definitely another symptom (duplicate) of bug 1752411 . The folks on that bug might be people who will know if dig can be used instead.

The relationship to LP #1752411 certainly feels valid.
(I think I agree to the duplication)

Check this out btw (perhaps better submitted to the other bug) -- but --

despite "host" claiming a default timeout of a few seconds, this NEVER returns!!
{{{
$ LC_ALL=C host -t soa local.
}}}

$ man host
{{{
       -W wait
           Timeout: Wait for up to wait seconds for a reply. If wait is less than one, the wait interval is set to one second.

           By default, host will wait for 5 seconds for UDP responses and 10 seconds for TCP connections. These defaults can be overridden by the timeout option in
           /etc/resolv.conf.

           See also the -w option.
}}}

But I even tried to manually specify how long to wait, it isn't honoured
{{{
$ time LC_ALL=C host -W 1 -t soa local.

<<HUNG>>
}}}

--
we're talking about THIS method btw

/usr/lib/avahi/avahi-daemon-check-dns.sh
{{{
dns_has_local() {
  # Some magic to do tests
  if [ -n "${FAKE_HOST_RETURN}" ] ; then
    if [ "${FAKE_HOST_RETURN}" = "true" ]; then
      return 0;
    else
      return 1;
    fi
  fi

  OUT=`LC_ALL=C host -t soa local. 2>&1` <<<---- HERE
  if [ $? -eq 0 ] ; then
    if echo "$OUT" | egrep -vq 'has no|not found'; then
      return 0
    fi
  else
    # Checking the dns servers failed. Assuming no .local unicast dns, but
    # remove the nameserver cache so we recheck the next time we're triggered
    rm -f ${NS_CACHE}
  fi
  return 1
}
}}}

---

Steps to reproduce:
 1. fresh boot
 2. run "host -t soa local." (works fine)
 {{{
$ LC_ALL=C host -t soa local.
Host local. not found: 3(NXDOMAIN)
 }}}
 2. connect to strongswan vpn
 3. disconnect the session
 4. , now that command hangs forever
{{{
$ time LC_ALL=C host -t soa local.
<HUNG>
}}}
 (tried timing it ...)

I also note;

I think this is (at least partially) due to strongswan leaving a dangling duplicate DNS entry in resolve.conf.

It's 100% consistent, that after step #3 above, there is a dangling DNS entry in resolve.conf, and this script hangs.

More :

 1. fresh boot
 2. script checks:
 - "/usr/lib/avahi/avahi-daemon-check-dns.sh" is fine
 - "host -t soa local." returns
 3. activate strongswan connection = SUCCESS

{{{
fermulator@fermmy:~$ sudo /usr/lib/avahi/avahi-daemon-check-dns.sh

fermulator@fermmy:~$ LC_ALL=C host -t soa local.
Host local. not found: 3(NXDOMAIN)

resolv.conf contains:
nameserver 192.168.194.20
nameserver 192.168.196.20
nameserver 127.0.0.53
}}}

then;
 4. disconnect VPN,

{{{
resolv.conf dangling:

nameserver 192.168.194.20
nameserver 127.0.0.53
}}}

 5. script checks:
 - "/usr/lib/avahi/avahi-daemon-check-dns.sh" HANGS
 - "host -t soa local." HANGS

 6. killall host

back to normal;

resolv.conf properly only has the local nameserver now (no more dangling DNS),
{{{
nameserver 127.0.0.53
}}}
 7. script checks:
 - "/usr/lib/avahi/avahi-daemon-check-dns.sh" works
 - "host -t soa local." works

$ host -t soa local.
Host local not found: 2(SERVFAIL)

**accepting duplication**