Ubuntu
apache2 package

ProxyErrorOverride leads to slow 404 responses

Bug #1495988 reported by Sven on 2015-09-15

This bug affects 1 person

	Status	Importance	Assigned to
Apache2 Web Server	Fix Released	Medium	auto-bz.apache.org #53420
apache2 (Ubuntu)	Fix Released	Medium	Christian Ehrhardt 
Trusty	Fix Released	Medium	Christian Ehrhardt 

Bug Description

[Impact]
* hang until ProxyTimeout timeout if ProxyErrorOverride is set
* see https://bz.apache.org/bugzilla/show_bug.cgi?id=53420 for more

[Test Case]
1. spawn 2x trusty container
2. install apache2 on both
3. Mark /var/www/html/index.html to verify frontend/backend setup
4. enable mod_proxy and ProxyErrorOverride Directive
   ln -s ln -s /etc/apache2/mods-available/xml2enc.load /etc/apache2/mods-enabled/
   ln -s /etc/apache2/mods-available/proxy* /etc/apache2/mods-enabled/
   rm /etc/apache2/mods-enabled/proxy_balancer.*
  On Frontend in /etc/apache2/mods-enabled/proxy.conf add (IP is backend)
    ProxyPass "/proxme" "http://10.0.4.44/"
    ProxyPassReverse "/proxme" "http://10.0.4.44/"
    ProxyErrorOverride On
    ProxyTimeout 5
5. verify that:
  on http:/Frontend/proxme you get the backend index.html page
  local and remote pages still work fine
  local errors are still fast
  proxied errors are slowed by ProxyTimeout (5 sec)
   time wget http://10.0.4.95/proxme/index-notthere.html
   real 0m4.987s
  proxied errors are slow now
   time wget http://10.0.4.95/proxme/index-notthere.html
   real 0m4.987s
   ab -n 100 -c 100 http://10.0.4.95/proxme/index-notthere.html
   Total: 4931 5186 433.9 4934 5936
6. On Frontend enable proposed, update and restart apache2
7. Verify things are now fast
  time wget http://10.0.4.95/proxme/index-notthere.html
  ab -n 100 -c 100 http://10.0.4.95/proxme/index-notthere.html

[Regression Potential]
* the patch is small and already backported by Apache people which is good
* The changes are local to mod_proxy only which should limit the potential fallout
* all basic tests I did still work, but we have to rely on a real test by the reporter from proposed for the actual Test

[Original Description]

Like described in #53420 [1] there is a bug in Apache 2.4.x which is fixed in 2.4.11.

This bug leads to slow 404 responses when ProxyErrorOverride is activated.

--------------------
When setting ProxyErrorOverride to "On" to get the httpd-ErrorDocument
instead of the backend-errordoc, httpd is waiting "ProxyTimeout" seconds
to respond to the client, even though the response is already read from
backend server.
--------------------

The bug is fixed in 2.4.11 [2]. Is there any chance to get this fix into the Ubuntu package for 14.04 LTS?

[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53420
[2] https://svn.apache.org/viewvc?view=revision&revision=1621601

-----------
$ apt-cache policy apache2
apache2:
  Installed: 2.4.7-1ubuntu4.5
  Candidate: 2.4.7-1ubuntu4.5
  Version table:
*** 2.4.7-1ubuntu4.5 0
        500 http://de.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.7-1ubuntu4 0
        500 http://de.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
-----------

See original description

Tags:

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Zl-k (zl-k) wrote on 2012-06-15:

When setting ProxyErrorOverride to "On" to get the httpd-ErrorDocument
instead of the backend-errordoc, httpd is waiting "ProxyTimeout" seconds
to respond to the client, even though the response is already read from
backend server.

The request/response is hanging somewhere in
  module/proxy/mod_proxy_http
    -> ap_proxy_http_process_response
       -> ap_discard_request_body(backend->r) (modules/http/http_filters)
before timing out with "ProxyTimeout".

ProxyErrorOverride Directive:
http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxyerroroverride

I can reproduce this error only in 2.4.x.
2.2.x works correctly.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Luka-dolenec (luka-dolenec) wrote on 2013-11-18:

HI.

We come across with same behavior with version 2.4.6
Our configuration is:
ProxyErrorOverride On
ProxyPass /test http://xxxxxx:8080/test
ProxyPassReverse /test http://xxxxxx:8080/test
ErrorDocument 404 "Page does not exist"
When we call proxy URL /test/nonexistingpage from browser, browser start to load page few minutes, then display "Page does not exist"
Same configuration on version 2.2.3 it works without problem.

Best regards.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Vijesh Shetty (shetty-vijesh82) wrote on 2014-04-29:

I am running Apache 2.4.7 and I find the same problem here too.
Anything other than 1xx, 2xx, 3xx response, proxy module waits for the default timeout before serving the ErrorDocument page if I have ProxyErrorOverride On.

Did anyone get a solution to this problem?

Thanks,
Vijesh

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Viktor-stolbin (viktor-stolbin) wrote on 2014-07-16:

Does anyone have any update on this? Is there any ticket to fix this?

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Rainer Jung (rainer-jung-kippdata) wrote on 2014-08-01:

First steps of analysis for trunk. The problem happens with all MPMs, especially also with prefork and event.

The hang until timeout happens in ap_discard_request_body called from mod_proxy_http.c in line 1640:

1629 /* PR 41646: get HEAD right with ProxyErrorOverride */
1630 if (ap_is_HTTP_ERROR(r->status) && dconf->error_override) {
1631 /* clear r->status for override error, otherwise ErrorDocument
1632 * thinks that this is a recursive error, and doesn't find the
1633 * custom error page
1634 */
1635 r->status = HTTP_OK;
1636 /* Discard body, if one is expected */
1637 if (!r->header_only && /* not HEAD request */
1638 (proxy_status != HTTP_NO_CONTENT) && /* not 204 */
1639 (proxy_status != HTTP_NOT_MODIFIED)) { /* not 304 */
1640 ap_discard_request_body(backend->r);
1641 }
1642 proxy_run_detach_backend(r, backend);
1643 return proxy_status;
1644 }

In ap_discard_request_body - which is used here to discard the reponse body from the origin server - I can see two brigades being read. The first has one bucket which contains the original error page coming form the origin server (here 206 bytes), the second brigade only has a EOS bucket. Reading the second brigade is where it blocks until proxy timeout.

For 2.2 the situation is different: the first brigade read has two buckets, the first again containing the origin server response body, the second one the EOS. The brigade with both buckets could be read immediately without waiting for timeout.

Any ideas what's going one? Hints in which direction to proceed?

Regards,

Rainer

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Rainer Jung (rainer-jung-kippdata) wrote on 2014-08-01:

The reason is broken header handling. Headers are read from the origin server and then directly put into r->headers_out. In 2.2 those headers are copied to the request struct (rp) used in ap_discard_request_body. therefore the http infput filter can detect the end of the body using the Content-Length header and sets EOS as soon as all the expected bytes are read.

In 2.4 and trunk, copying the headers from r->headers_out to backend->r-headers_in happens *after* the error_override handling. Thus the http input filter must read until timeout.

Either we rearrange the original order of code, or we at least copy Content-Length and Transfer-Encoding headers from r->headers_out to backend->r_headers_in before doing ap_discard_request_body. Somethin like

                const char *tmp;
                if (tmp = apr_table_get(r->headers_out, "Content-Length")) {
                    apr_table_set(backend->r->headers_in, "Content-Length", tmp);
                }
                else if (tmp = apr_table_get(r->headers_out, "Transfer-Encoding")) {
                    apr_table_set(backend->r->headers_in, "Transfer-Encoding", tmp);
                }
                else if (te) {
                    apr_table_set(backend->r->headers_in, "Transfer-Encoding", te);
                }
                ap_discard_request_body(backend->r);
                ...

I will check svn history to get an idea, why the code order was changed.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Rainer Jung (rainer-jung-kippdata) wrote on 2014-08-01:

The bug was introduced by r912063 to fix PR41646.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Rainer Jung (rainer-jung-kippdata) wrote on 2014-08-02:

Fix committed in r1615289.
Will propose for backport to 2.4 next week if no negative reactions come up in the meantime.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Vijesh Shetty (shetty-vijesh82) wrote on 2014-08-12:

Is the fix backported to 2.4.x?

Revision history for this message

In bz.apache.org/bugzilla/ #53420, tititou (christophe-jaillet) wrote on 2014-08-12:

#10

It has been proposed for backport for 2.4.x in r1615346 but it has not been accepted yet.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Ylavic-dev (ylavic-dev) wrote on 2015-01-23:

#11

Backported to 2.4.11 in r1621601.

Revision history for this message

In bz.apache.org/bugzilla/ #53420, Cristian-httpd (cristian-httpd) wrote on 2015-03-19:

#12

Hi, there are some additional issues with ProxyErrorOverride set to on, please see https://bz.apache.org/bugzilla/show_bug.cgi?id=56925

Anyone else experienced this, looking at this ticket it seems like the turnaround time may be up to 2 years until it will be solved.

Sven (muffl0n) on 2015-09-15

description:

updated

Robie Basak (racb) on 2015-09-22

Changed in apache2 (Ubuntu):
status:	New → Triaged
importance:	Undecided → Medium
tags:	added: server-next

Bug Watch Updater (bug-watch-updater) on 2015-09-22

Changed in apache2:
importance:	Unknown → Medium
status:	Unknown → Fix Released

Christian Ehrhardt  (paelzer) on 2016-05-31

Changed in apache2 (Ubuntu):
assignee:	nobody → ChristianEhrhardt (paelzer)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-07:

#13

Took a look at it, the backport already done by Apache applies nicely.
I ran it through various sbuild's and adt's and it always built fine (also the change is rather minimal - which is good for a SRU).

I installed the resulting deb's in a trusty container and the apache still works fine after the upgrade.

I lack a sophisticated setup for e.g. apache performance tests; but given the small size of the change I'd think it is ready to be considered as SRU.

So I'll upload the debdiff and fill the template, but would leave the complex test of the issue to the reporter who has a matching setup (once in proposed).

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-07:

#14

debdiff - fix timeout that can happen with mod proxy Edit (3.7 KiB, text/plain)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-07:

#15

Request for SRU sponsoring

[Impact]
* hang until ProxyTimeout timeout if ProxyErrorOverride is set
* see https://bz.apache.org/bugzilla/show_bug.cgi?id=53420 for more

[Test Case]
* use the feature of https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxyerroroverride

Mathew Hodson (mhodson) on 2016-06-07

Changed in apache2 (Ubuntu Trusty):
importance:	Undecided → Medium

Revision history for this message

Robie Basak (racb) wrote on 2016-06-08:

#16

Hi Christian,

Looks good. Thank you for preparing this. A couple of questions:

Do you have permission to edit the bug description at the top? We usually put the SRU paperwork there, but I don't know who can do it. I'll copy it up there for you now.

The "Test Case" is pretty minimal. Usually we want enough instructions that someone not familiar with the package can verify both the failure and success cases without having to infer things about the test environment which may lead to failure to reproduce. You said "but we have to rely on a real test by the reporter". Do you mean that this bug cannot easily be reproduced by someone else?

description:

updated

Revision history for this message

Robie Basak (racb) wrote on 2016-06-08:

#17

Marking Fix Released in Yakkety as this is reported fixed in 2.4.11 and Yakkety has 2.4.18.

Changed in apache2 (Ubuntu):
status:	Triaged → Fix Released

Revision history for this message

Robie Basak (racb) wrote on 2016-06-08:

#18

Looks like the SRU for 2.4.7-1ubuntu4.10 is still in flight so we'll need for that to clear before uploading this SRU.

Changed in apache2 (Ubuntu Trusty):
status:	New → Triaged
assignee:	nobody → ChristianEhrhardt (paelzer)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-09:

#19

Hi Robie,
I'll copy the SRU Teamplate up next time - just wasn't aware yet it should go there.

Regarding the Test, yes I was only able to Test apache in general.
But not easily regarding the bug - this could be done thou given more time investment.
I wanted to say with my "but we have to rely on a real test by the reporter" that I consider the test doable but as you phrase it "cannot easily be reproduced by someone else".

Given more time, it might turn out as easy and I could write something more up (TBD).
If that is a requirement for the SRU that I/we provide that I'm gonna try and do it - let me know.

Revision history for this message

Robie Basak (racb) wrote on 2016-06-09:

#20

It is an SRU requirement - "A [Test Case] section with detailed instructions how to reproduce the bug. These should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem."

Where this is impossible I have seen exceptions made, for example for difficult to reproduce bugs such as non-deterministic ones or ones that require certain hardware, and proceed with just an assurance that the reporter will perform SRU verification. But without such an exceptional reason I'd like a full test case before uploading the fix please. Otherwise we often get stuck at SRU verification stage, the SRU fails to land and we will still have disrupted others who run -proposed for us for no good reason.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-21:

#21

Thanks Robie for the help driving the other bug stalling in SRU - as a note it is bug 1286882

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-27:

#22

FYI - The other bug is now cleared, also I prepared a simplified test that I'll add now to the SRU description.
Could this be reevaluated for upload to proposed?

description:

updated

Revision history for this message

Robie Basak (racb) wrote on 2016-06-27:

#23

Uploaded. Thanks!

Changed in apache2 (Ubuntu Trusty):
status:	Triaged → In Progress

Revision history for this message

Chris J Arges (arges) wrote on 2016-06-29: Please test proposed package

#24

Hello Sven, or anyone else affected,

Accepted apache2 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.7-1ubuntu4.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apache2 (Ubuntu Trusty):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-07-06:

#25

As there was no activity after a week I decided to do the verification on my own to unblock the transition.
Followed the listed Test Case steps, triggered the issue, enabled and updated from proposed - issue fixed.

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-07-12:

#26

This bug was fixed in the package apache2 - 2.4.7-1ubuntu4.11

---------------
apache2 (2.4.7-1ubuntu4.11) trusty; urgency=medium

* Fix hang until proxy timeout for Proxy responses with error status and
"ProxyErrorOverride On" being set (LP: #1495988).

-- Christian Ehrhardt <email address hidden> Tue, 07 Jun 2016 16:28:05 +0200

Changed in apache2 (Ubuntu Trusty):
status:	Fix Committed → Fix Released

Revision history for this message

Martin Pitt (pitti) wrote on 2016-07-12: Update Released

#27

The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.