KDC/kadmind explicit wildcard listener addresses do not use pktinfo

Bug #1688121 reported by Andreas Hasenack
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
krb5 (Ubuntu)
Fix Released
Medium
Unassigned
Zesty
Fix Released
Medium
Andreas Hasenack

Bug Description

This is fixed in artful in krb5 1.15-2

- upstream: http://krbdev.mit.edu/rt/Ticket/Display.html?id=8530
- debian: conflated into https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860767
- debian patch in artful's krb5: 0012-Use-pktinfo-for-explicit-UDP-wildcard-listeners.patch

[Impact]

When the KDC is configured to listen on the wildcard address (0.0.0.0) and receives a kinit request via UDP on an aliased interface, the response is sent with the wrong source IP and never received by kinit.
After a short timeout, kinit tries again with TCP, in which case it works. But if using PREAUTH (the default), that means this first request will correctly fail, with the server demanding PREAUTH, and the client will try again with a changed request. The whole dance starts again: first UDP, ignored, then TCP, and finally we have a ticket.

Most clients will just see an increased lag when obtaining tickets. If for some reason 88/TCP is blocked on the KDC and clients are expected to use UDP at all times, then kinit requests will just fail.

A workaround is to list the aliased interface's address in kdc_listen besides the wildcard (0.0.0.0) address.

The provided patch is applied upstream and in Debian testing.

[Test Case]

On zesty:
a) install krb5-kdc and krb5-admin-server
$ sudo apt install krb5-kdc krb5-admin-server
when prompted, use EXAMPLE.ORG (all caps) as the default realm
when prompted, select your own IP for the KDC and the Admin servers

b) configure a new realm called EXAMPLE.ORG
$ sudo krb5_newrealm
use any password of your liking when prompted

b.1) Add "kdc_listen = 0.0.0.0" to the EXAMPLE.ORG [realm] section in /etc/krb5kdc/kdc.conf and restart the kerberos services:
[realms]
    EXAMPLE.ORG = {
        kdc_listen = 0.0.0.0
(...)
$ sudo service krb5-kdc restart
$ sudo service krb5-admin-server restart

c) run kadmin.local to create a principal "ubuntu" with password "ubuntu" and with mandatory PREAUTH:
$ sudo kadmin.local addprinc -pw ubuntu +requires_preauth ubuntu

d) extract the ubuntu principal keytab and time how long it takes to obtain a ticket:
$ sudo kadmin.local ktadd -k /home/ubuntu/ubuntu.keytab ubuntu
$ sudo chown ubuntu:ubuntu /home/ubuntu/ubuntu.keytab
$ time kinit -k -t /home/ubuntu/ubuntu.keytab ubuntu
real 0m0.022s
$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: <email address hidden>

Valid starting Expires Service principal
05/03/2017 21:22:08 05/04/2017 07:22:08 <email address hidden>
 renew until 05/04/2017 21:22:08

e) add another IP to your network interface. For example, this adds 10.0.5.155 to ens3 (it has 10.0.5.55/24 already in my case):
$ sudo ip addr add 10.0.5.155/24 dev ens3

f) Edit the EXAMPLE.ORG realm section in /etc/krb5.conf and configure the kdc and admin server's IP to this new IP you just added in step (e):
[realms]
        EXAMPLE.ORG = {
                kdc = 10.0.5.155
                admin_server = 10.0.5.155

g) Time again how long it takes to obtain a ticket:
$ time kinit -k -t /home/ubuntu/ubuntu.keytab ubuntu
real 0m2.017s

Step (g) shows the bug.

On a more technical level, we can see that the server responds to kinit's UDP request using an incorrect source IP, therefore kinit never "sees" it. It quickly times out and switches to TCP, where the server responds using the correct source IP:
    1 0.000000000 10.0.5.55 → 10.0.5.155 KRB5 216 AS-REQ
    2 0.000566682 10.0.5.55 → 10.0.5.55 KRB5 298 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED
(2) has the incorrect source ip!

After roughly 1s, kinit switches to tcp and tries again:
    3 1.003231507 10.0.5.55 → 10.0.5.155 TCP 76 55588 → 88 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=3523453804 TSecr=0 WS=128
    4 1.003269692 10.0.5.155 → 10.0.5.55 TCP 76 88 → 55588 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=2572724273 TSecr=3523453804 WS=128
    5 1.003302614 10.0.5.55 → 10.0.5.155 TCP 68 55588 → 88 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=3523453804 TSecr=2572724273
    6 1.003545204 10.0.5.55 → 10.0.5.155 KRB5 244 AS-REQ
    7 1.003567693 10.0.5.155 → 10.0.5.55 TCP 68 88 → 55588 [ACK] Seq=1 Ack=177 Win=44800 Len=0 TSval=2572724273 TSecr=3523453804
    8 1.003799664 10.0.5.155 → 10.0.5.55 KRB5 326 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED
(continues)
(8) and the whole tcp handshake happens with the correct IP addresses and the exchange happens and we get the ticket, but not before kinit repeats the request with PREAUTH and UDP again. That's why it takes 2 seconds in the end :)

h) repeat step (g) with the updated packages. Timing should be similar to the one in step (d), and a traffic capture should show UDP (and not TCP) being used.

Alternativaly, you can also prefix the kinit command with KRB5_TRACE=/dev/stderr and verify in the debug logs that UDP instead of TCP is being used.

[Regression Potential]
This affects only UDP sockets bound to a wildcard address and makes these sockets work correctly when there are aliased NICs (eth0:1, eth0:2) and/or just multiple IPs on the same NIC.

description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

With the fix applied, we get this:

$ time kinit -k -t /home/ubuntu/ubuntu.keytab ubuntu
real 0m0.023s

And the traffic happens all in UDP, since kinit got the "PREAUTH required" response (because now it came from the correct source IP) and just issued the updated request right away:
    1 0.000000000 10.0.5.55 → 10.0.5.155 KRB5 216 AS-REQ
    2 0.002060386 10.0.5.155 → 10.0.5.55 KRB5 298 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED
    3 0.005412046 10.0.5.55 → 10.0.5.155 KRB5 311 AS-REQ
    4 0.012516720 10.0.5.155 → 10.0.5.55 KRB5 793 AS-REP

Changed in krb5 (Ubuntu Zesty):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in krb5 (Ubuntu):
assignee: Andreas Hasenack (ahasenack) → nobody
Changed in krb5 (Ubuntu):
status: In Progress → Fix Released
Changed in krb5 (Ubuntu Zesty):
status: New → In Progress
description: updated
Changed in krb5 (Debian):
status: Unknown → Fix Released
description: updated
Mathew Hodson (mhodson)
Changed in krb5 (Ubuntu):
importance: Undecided → High
Changed in krb5 (Ubuntu Zesty):
importance: Undecided → High
Mathew Hodson (mhodson)
affects: krb5 (Debian) → ubuntu-translations
Changed in ubuntu-translations:
importance: Unknown → Undecided
status: Fix Released → New
no longer affects: ubuntu-translations
Changed in krb5 (Ubuntu):
importance: High → Medium
Changed in krb5 (Ubuntu Zesty):
importance: High → Medium
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Andreas, or anyone else affected,

Accepted krb5 into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/krb5/1.15-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in krb5 (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Gianfranco Costamagna (costamagnagianfranco) wrote :

ahasenack, can you please test it?

thanks!

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

On it.

description: updated
description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I updated the test case with step (b.1) which I had forgotten. Here it goes:

Reproducing the error case with 1.15-1, we can see that UDP is tried first, is ignored, and then TCP is used one second later:
$ apt-cache policy krb5-kdc
krb5-kdc:
  Installed: 1.15-1
  Candidate: 1.15-1
  Version table:
 *** 1.15-1 500
        500 http://br.archive.ubuntu.com/ubuntu zesty/universe amd64 Packages
        100 /var/lib/dpkg/status

$ KRB5_TRACE=/dev/stdout kinit -k -t /home/ubuntu/ubuntu.keytab ubuntu
[2848] 1494852873.104617: Getting initial credentials for <email address hidden>
[2848] 1494852873.105449: Looked up etypes in keytab: aes256-cts, aes128-cts
[2848] 1494852873.105633: Sending request (172 bytes) to EXAMPLE.ORG
[2848] 1494852873.105684: Resolving hostname 10.0.100.249
[2848] 1494852873.105840: Sending initial UDP request to dgram 10.0.100.249:88
[2848] 1494852874.108235: Initiating TCP connection to stream 10.0.100.249:88
[2848] 1494852874.108528: Sending TCP request to stream 10.0.100.249:88
[2848] 1494852874.110518: Received answer (254 bytes) from stream 10.0.100.249:88
[2848] 1494852874.110549: Terminating TCP connection to stream 10.0.100.249:88
[2848] 1494852874.285214: Response was not from master KDC
[2848] 1494852874.285346: Received error from KDC: -1765328359/Additional pre-authentication required
...

After installing the update, UDP is again tried first but this time kinit receives an immediate answer and the exchange remains on UDP:
$ apt-cache policy krb5-kdc
krb5-kdc:
  Installed: 1.15-1ubuntu0.1
  Candidate: 1.15-1ubuntu0.1
  Version table:
 *** 1.15-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu zesty-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.15-1 500
        500 http://br.archive.ubuntu.com/ubuntu zesty/universe amd64 Packages

$ KRB5_TRACE=/dev/stdout kinit -k -t /home/ubuntu/ubuntu.keytab ubuntu
[10150] 1494853325.393939: Getting initial credentials for <email address hidden>
[10150] 1494853325.395247: Looked up etypes in keytab: aes256-cts, aes128-cts
[10150] 1494853325.395665: Sending request (172 bytes) to EXAMPLE.ORG
[10150] 1494853325.395851: Resolving hostname 10.0.100.249
[10150] 1494853325.396225: Sending initial UDP request to dgram 10.0.100.249:88
[10150] 1494853325.398161: Received answer (254 bytes) from dgram 10.0.100.249:88
[10150] 1494853325.648728: Response was not from master KDC
[10150] 1494853325.648835: Received error from KDC: -1765328359/Additional pre-authentication required

tags: added: verification-done-zesty
tags: removed: verification-needed
tags: added: verification-done
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package krb5 - 1.15-1ubuntu0.1

---------------
krb5 (1.15-1ubuntu0.1) zesty; urgency=medium

  * Pulled in Debian fixes from Sam Hartman for:
    - kinit fails for OTP user when using kdc discovery via DNS
      (LP: #1683237)
    - KDC/kadmind explicit wildcard listener addresses do not use pktinfo
      (LP: #1688121)
    - KDC/kadmind may fail to start on IPv4-only systems (LP: #1688310)

 -- Andreas Hasenack <email address hidden> Fri, 05 May 2017 14:05:38 +0000

Changed in krb5 (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for krb5 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.