iptables connlimit allows more connections than the limit when using multiple CPUs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Trusty |
Fix Released
|
Medium
|
Unassigned | ||
Xenial |
Fix Released
|
Medium
|
Unassigned | ||
Bionic |
Fix Released
|
Medium
|
Unassigned | ||
Cosmic |
Fix Released
|
Medium
|
Unassigned |
Bug Description
[Impact]
* The iptables connection count/limit rules can be breached
with multithreaded network driver/
due to a race in the conncount/connlimit code.
* For example:
# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 \
-m connlimit --connlimit-above 2000 --connlimit-mask 0 \
-j DROP
* The fix is a backport from an upstream commit that resolves
the problem (plus dependencies for a cleaner backport) that
address the race condition:
commit b36e4523d4d5 ("netfilter: nf_conncount: fix garbage
collection confirm race").
[Test Case]
* Server-side: (relevant kernel side)
(limit TCP port 7777 to only 2000 connections)
# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 \
-m connlimit --connlimit-above 2000 --connlimit-mask 0 \
-j DROP
# ulimit -SHn 65000 # increase number of open files
# ruby server.rb # multi-threaded server
* Client-side:
# ulimit -SHn 65000
# ruby client.rb <server ip> <port> <target # connections> <# threads>
<test output>
* Results with Original kernel:
(client achieves target of 6000 connections > limit of 2000 connections)
# ruby client.rb 10.230.56.100 7777 6000 3
1
2
3
<...>
6000
Target reached. Thread finishing
6001
Target reached. Thread finishing
6002
Target reached. Thread finishing
Threads done. 6002 connections
press enter to exit
* Results with Modified kernel:
(client is limited to 2000 connections, and times out afterward)
# ruby client.rb 10.230.56.100 7777 6000 3
1
2
3
<...>
2000
<... blocks for a few minutes ...>
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
Threads done. 2000 connections
press enter to exit
* Test cases possibly available upon request,
depending on original author's permission.
[Regression Potential]
* The patchset has been reviewed by a netfilter maintainer [1] in
stable mailing list, and was considered OK for 4.14, and that's
essentially the same backport for 4.15 and 4.4.
* The changes are limited to netfilter connlimit/conncount (names
change between older/newer kernel versions).
[Other Info]
* The backport for 4.14 [2] is applied as of 4.14.92.
[1] https:/
[2] https:/
Changed in linux (Ubuntu): | |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
status: | New → Confirmed |
description: | updated |
description: | updated |
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Cosmic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Bionic): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Cosmic): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Xenial): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Trusty): | |
importance: | Undecided → Medium |
status: | New → Fix Committed |
[SRU T][PATCH 0/3] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097878. html
https:/
[SRU X][PATCH 0/6] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097698. html
https:/
[SRU B][PATCH 0/5] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097705. html
https:/
[SRU C, D/Unstable][PATCH 0/1] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097711. html
https:/