keepalived reload buggy due to improper ipv4 address comparison
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| keepalived (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
| Precise |
Fix Released
|
High
|
Adam Stokes | ||
| Quantal |
Fix Released
|
High
|
Adam Stokes | ||
| Raring |
Fix Released
|
High
|
Adam Stokes | ||
Bug Description
[Impact]
There are at least 2 identified problems with keepalived on Precise when trying to reload the configuration (/etc/keepalive
(a) The removal of a server from the cluster does not work when the port of that server is in use by another member server
(b) The change of weight of a cluster member does not work
Reload is done by either:
1. sudo service keepalived reload
2. sudo kill -HUP `pidof keepalived`
The upstream devel list suggests that issue (a) [1] has been resolved in version 1.2.3 which is currently only in Debian unstable via version 1.2.7. I can't find the actual bug supposedly filed by 'Ronie Gilberto Henrich'. I have not tested 1.2.7 as it is not easily available to me.
The pertinent entry for (a) in the upstream changelog [2] I believe is here:
{{{ * check : Fix IPv4 address comparison routine. }}}
This bug is a request to have fixes for both issues ([3] for (a)) introduced into Ubuntu 12.04.
[1]: http://
[2]: http://
[3]: https:/
[Test Case]
4A. Steps to reproduce issue #1:
(a) Stop keepalived
$ sudo service keepalived stop
(b) Begin with a simple configuration file (attached: keepalived.conf)
$ cat keepalived.conf | sudo tee /etc/keepalived
$ sudo service keepalived start
$ sudo ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
(c) Add a stanza to the configuration file (a server with a port that is in use by another server)
--------------
real_server 192.168.1.103 8080 {
weight 1
}
--------------
(d) Reload the configuration
$ sudo service keepalived reload
$ sudo ipvsadm -Ln
--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
-> 192.168.1.103:8080 Masq 1 0 0
--------------
(e) Remove the .103 stanza
(f) Perform step (d) above
--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
-> 192.168.1.103:8080 Masq 1 0 0
--------------
Note the .103 server remains.
4B. Steps to reproduce issue #2:
(a) Perform steps (a) and (b) from 4A
(b) Edit a stanza in the configuration file (change weight from '1' to '0' in the .102 server)
--------------
real_server 192.168.1.102 8080 {
weight 0
}
--------------
(c) Perform step (d) from 4A
--------------
P Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
--------------
Note the original weight of '1' for .102 server remains.
4C. Steps to counterprove issue #1:
(a) Perform steps (a) and (b) from 4A
(b) Add a stanza to the configuration file (a server with a port that is not in use by another server)
--------------
real_server 192.168.1.103 8081 {
weight 1
}
--------------
(c) Perform step (d) from 4A
--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
-> 192.168.1.103:8081 Masq 1 0 0
--------------
(d) Remove the .103 stanza from the configuration file
(e) Perform step (d) from 4A
--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
-> 192.168.1.101:8080 Masq 1 0 0
-> 192.168.1.102:8080 Masq 1 0 0
--------------
Note the .103 server does not remain.
5. Known Workaround:
- Restart the daemon:
$ sudo service keepalived restart
OR
- Manage the daemon with ipvsadm:
(a) Add a server
$ sudo ipvsadm -a -t 192.168.0.100:80 -r 192.168.1.103:8080 -m
(b) Edit a server (change weight)
$ sudo ipvsadm -e -t 192.168.0.100:80 -r 192.168.1.103:8080 -m -w 0
(c) Delete a server
$ sudo ipvsadm -d -t 192.168.0.100:80 -r 192.168.1.103:8080
[Regression Potential]
This introduces the proper behavior for comparing ipv4 addresses. We may see issues where keepalived was working in an environment and after upgrade fails. This probably indicates that the environment may have issues outside of keepalived as the previous behavior was incorrect.
| summary: |
- [Precise] keepalived reload buggy + keepalived reload buggy |
| Changed in keepalived (Ubuntu): | |
| importance: | Undecided → High |
| Launchpad Janitor (janitor) wrote Re: keepalived reload buggy | #1 |
| Changed in keepalived (Ubuntu Precise): | |
| status: | New → Confirmed |
| Changed in keepalived (Ubuntu Quantal): | |
| status: | New → Confirmed |
| Changed in keepalived (Ubuntu Raring): | |
| status: | New → Confirmed |
| Changed in keepalived (Ubuntu): | |
| status: | New → Confirmed |
| Adam Stokes (adam-stokes) wrote | #5 |
| Changed in keepalived (Ubuntu Precise): | |
| assignee: | nobody → Adam Stokes (adam-stokes) |
| Changed in keepalived (Ubuntu Quantal): | |
| assignee: | nobody → Adam Stokes (adam-stokes) |
| Changed in keepalived (Ubuntu Raring): | |
| assignee: | nobody → Adam Stokes (adam-stokes) |
| Adam Stokes (adam-stokes) wrote | #7 |
| Adam Stokes (adam-stokes) wrote | #8 |
| summary: |
- keepalived reload buggy + keepalived reload buggy due to improper ipv4 address comparison |
| description: | updated |
| Changed in keepalived (Ubuntu): | |
| status: | Confirmed → Fix Released |
| Changed in keepalived (Ubuntu Quantal): | |
| importance: | Undecided → High |
| Changed in keepalived (Ubuntu Raring): | |
| importance: | Undecided → High |
| Changed in keepalived (Ubuntu Precise): | |
| importance: | Undecided → High |
| Brian Murray (brian-murray) wrote Please test proposed package | #9 |
| Changed in keepalived (Ubuntu Raring): | |
| status: | Confirmed → Fix Committed |
| tags: | added: verification-needed |
| Changed in keepalived (Ubuntu Quantal): | |
| status: | Confirmed → Fix Committed |
| Brian Murray (brian-murray) wrote | #10 |
| Changed in keepalived (Ubuntu Precise): | |
| status: | Confirmed → Fix Committed |
| Brian Murray (brian-murray) wrote | #11 |
| Peter Matulis (petermatulis) wrote | #12 |
| tags: | added: verification-done-precise |
| Peter Matulis (petermatulis) wrote | #13 |
| Peter Matulis (petermatulis) wrote | #14 |
| tags: | added: verification-done-quantal |
| Peter Matulis (petermatulis) wrote | #15 |
| Peter Matulis (petermatulis) wrote | #16 |
| Adam Stokes (adam-stokes) wrote | #17 |
| Peter Matulis (petermatulis) wrote | #18 |
| Evan Callicoat (diopter) wrote | #19 |
| Peter Matulis (petermatulis) wrote | #20 |
| tags: |
added: verification-done-raring removed: verification-needed |
| Peter Matulis (petermatulis) wrote | #21 |
| Launchpad Janitor (janitor) wrote | #22 |
| Changed in keepalived (Ubuntu Raring): | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Update Released | #23 |
| Launchpad Janitor (janitor) wrote | #24 |
| Changed in keepalived (Ubuntu Quantal): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #25 |
| Changed in keepalived (Ubuntu Precise): | |
| status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.