Apache2: BalancerMember worker hostname (65.character.host.name) too long

Bug #1750356 reported by Graham Leggett
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apache2 Web Server
Fix Released
Medium
apache2 (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Low
Andreas Hasenack

Bug Description

[Impact]

If the BalancerMember directive contains a URL with a hostname longer than X characters, apache2 will fail to start with the following error:

BalancerMember worker hostname (65.character.host.name) too long

RFC1035 allows for longer hostnames, and apache upstream has this fix already.

[Test Case]

* Install the packages:
sudo apt install apache2

* Edit /etc/apache2/sites-available/000-default.conf and add the following block inside the VirtualHost block:
  <Proxy "balancer://test">
    BalancerMember "http://xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx:90/"
    BalancerMember "http://xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx:91/"
  </Proxy>
  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  RewriteEngine On
  RewriteRule ^/foo balancer://test/foo [P,L]

* Enable the necessary apache modules:
sudo a2enmod proxy_balancer proxy lbmethod_byrequests proxy_http rewrite

* Restart apache2, which will fail:
sudo systemctl restart apache2

* Run the status action and expect an error like this:
sudo systemctl status apache2.service
...
Jun 27 18:31:16 bionic-apache-1750356 apachectl[2218]: BalancerMember worker hostname (xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx) too long

* Update the apache2 packages to the ones available in proposed. As part of the upgrade, apache2 will be restarted again, and in this time it will work. Confirm with systemctl status apache2 that there are no errors this time:
sudo systemctl status apache2

* Try to access http://localhost/foo to trigger the load balancer configuration. It will trigger a DNS error as we don't have an entry for the BalancerMember hostname, but it shows that the configuration worked:

ubuntu@bionic-apache-1750356:~$ wget http://localhost/foo
--2018-06-27 18:39:58-- http://localhost/foo
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 502 Proxy Error
2018-06-27 18:39:58 ERROR 502: Proxy Error.

ubuntu@bionic-apache-1750356:~$ tail /var/log/apache2/error.log -n 1
[Wed Jun 27 18:39:58.732097 2018] [proxy:error] [pid 3628:tid 139981565716224] [client 127.0.0.1:42508] AH00898: DNS lookup failure for: xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx returned by /foo

[Regression Potential]
This change is twofold: it allows for longer hostnames, and apache won't fail to start anymore if that length is exceeded. In the latter case, however, the hostname is truncated. With that in mind, here are some scenarios:
- hostnames larger than 65 characters but less than 255: before apache2 would fail to start, now it works.
- hostname larger than 255 characters. Before apache would fail to start; now, it starts but truncates that hostname, logging a warning. The configuration is likely to not work due to the truncation, which will lead to DNS errors. If the admin was only relying on (re)start errors to become aware of configuration problems, he/she might miss this until it's too late. But at least the log will be clear about what happened.
- third party modules that use apache's mod_proxy structure might not be aware of the new hostname_ex member which can hold the longer string, since we didn't update the MODULE_MAGIC_NUMBER_MINOR number with this patchset, and will probably remain exhibiting the problem described in this bug.

[Other Info]
The security team's regression test suite for apache2 (http://launchpad.net/qa-regression-testing) was run with the test packages from the PPA at https://launchpad.net/~ahasenack/+archive/ubuntu/apache-balance-member-hostname-1750356/+packages and passed: https://pastebin.ubuntu.com/p/nZ6GGHXgwQ/

== Original Description ==

If the BalancerMember directive contains a URL with a hostname longer than X characters, we fail as follows:

BalancerMember worker hostname (65.character.host.name) too long

The size of the hostname needs to be raised so it is RFC1035 compliant.

Bug fixed upstream at https://bz.apache.org/bugzilla/show_bug.cgi?id=62085, patches backported to v2.4.30:

http://svn.apache.org/r1824455
http://svn.apache.org/r1824504

(Both patches required, first is extended by second).

Related branches

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

If the BalancerMember directive contains a URL with a hostname longer than X characters, we fail as follows:

BalancerMember worker hostname (65.character.host.name) too long

The same fix for https://bz.apache.org/bugzilla/show_bug.cgi?id=53218 on the worker name needs to be applied to the hostname, and separately the size of the hostname needs to be raised so it is RFC1035 compliant in trunk.

Revision history for this message
In , Graham Leggett (minfrin-y) wrote :
Revision history for this message
In , Graham Leggett (minfrin-y) wrote :

Backported to v2.4.30:

http://svn.apache.org/r1824455
http://svn.apache.org/r1824504

(Both patches required, first is extended by second).

tags: added: xenial
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Just triaging right now, but I agree.
And in Bionic we only have 2.4.29 at the moment.

If you have a test at hand could you also outline a steps to reproduce?
Those will eventually come handy to verify the fix.

tags: added: server-next
Changed in apache2 (Ubuntu):
status: New → Triaged
Revision history for this message
Graham Leggett (minfrin-y) wrote :

This breaks things for us:

<Proxy "balancer://test">
  BalancerMember "https://xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx:443/"
</Proxy>

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Confirmed broken in bionic, and working in cosmic.

Changed in apache2 (Ubuntu Bionic):
status: New → Triaged
Changed in apache2 (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Graham Leggett (minfrin-y) wrote :

Any chance of a fix for xenial? This is where we hit the issue.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hm, what should happen with this hunk of the patch?

--- 2.4.x/include/ap_mmn.h 2018/02/16 14:58:32 1824503
+++ 2.4.x/include/ap_mmn.h 2018/02/16 15:04:41 1824504
@@ -517,7 +518,7 @@
 #ifndef MODULE_MAGIC_NUMBER_MAJOR
 #define MODULE_MAGIC_NUMBER_MAJOR 20120211
 #endif
-#define MODULE_MAGIC_NUMBER_MINOR 74 /* 0...n */
+#define MODULE_MAGIC_NUMBER_MINOR 75 /* 0...n */

 /**
  * Determine if the server's current MODULE_MAGIC_NUMBER is at least a

Apache in bionic (2.4.29) has MODULE_MAGIC_NUMBER_MINOR set to 68

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'm inclined to not change it, because 75 implies everything between 68 and 75, and 69 was probably used for something else. And it's a sequential number.

Revision history for this message
Graham Leggett (minfrin-y) wrote :

The module magic number gives code that depends on the apache API an indication of whether a particular function or variable is available or not.

I imagine Ubuntu already has a policy for the module magic number, and I suspect it's that the MMN doesn't change over the lifetime of a version of the distro. So the fields will be physically there, but code won't get any hint that they are, and nothing will break.

Changed in apache2:
importance: Unknown → Medium
status: Unknown → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I prepared test packages here: https://launchpad.net/~ahasenack/+archive/ubuntu/apache-balance-member-hostname-1750356/+packages

will propose a branch soon for bionic.

Changed in apache2 (Ubuntu Bionic):
assignee: nobody → Andreas Hasenack (ahasenack)
status: Triaged → In Progress
importance: Undecided → Low
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Robie Basak (racb) wrote :

I'm not very keen on this from an SRU perspective. RFC or not, it's really a feature addition (the support for longer hostnames for BalancerMember) and the patch implements it as such (across nine source files). This makes the patch difficult to review comprehensively.

Do we really need this in an SRU? Can affected users use shorter hostnames instead?

Revision history for this message
Graham Leggett (minfrin-y) wrote :

> I'm not very keen on this from an SRU perspective. RFC or not, it's really a feature addition

This is a bug. RFC1035 describes how long a hostname must be, and httpd was not honouring the RFC. This doesn't implement "longer hostnames", this implements RFC compliant hostnames.

The httpd project has ABI guarantees in the v2.4 release series. This meant we had to support the older fields unchanged, while at the same time introducing new compliant fields. This is well established and well understood at httpd and is not out of the ordinary.

> Do we really need this in an SRU? Can affected users use shorter hostnames instead?

No.

The longer answer is "epically no". We have a huge estate of machines, all with names that are both descriptive and unique, and therefore long. There is no option to make them shorter. Instead of hacking around bugs, let's fix them.

Revision history for this message
Robie Basak (racb) wrote :

> This is a bug.

I'm not interested in arguments about semantics. My point is that from the point of view of the proposed code change it's the implementation of a new feature, with all the wide-ranging changes and regression risk that goes with it.

> This meant we had to support the older fields unchanged, while at the same time introducing new compliant fields. This is well established and well understood at httpd and is not out of the ordinary.

Ah - do you represent upstream? Part of my issue is in gaining confidence that the patch does the right thing in the absence of the rest of upstream microrelease changes. If someone familiar with the patch can vouch for it, that would help.

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Graham, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apache2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Graham, is this something you can easily test? You sounded like you have a deployment where this bug happens.

If not, I can do the verification, it would just be more reassuring if someone other than who prepared the SRU could verify it.

description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Bionic verification. Starting with the packages that show the bug:

ubuntu@bionic-apache-1750356:~$ apt-cache policy apache2
apache2:
  Installed: 2.4.29-1ubuntu4.2
  Candidate: 2.4.29-1ubuntu4.2
  Version table:
 *** 2.4.29-1ubuntu4.2 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status

status shows the expected error:
(...)
Aug 28 17:20:37 bionic-apache-1750356 systemd[1]: Starting The Apache HTTP Server...
Aug 28 17:20:37 bionic-apache-1750356 apachectl[2040]: AH00526: Syntax error on line 24 of /etc/apache2/sites-enabled/000-default.conf:
Aug 28 17:20:37 bionic-apache-1750356 apachectl[2040]: BalancerMember worker hostname (xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx) too long
Aug 28 17:20:37 bionic-apache-1750356 apachectl[2040]: Action 'start' failed.

Upgrading to the packages in proposed:
ubuntu@bionic-apache-1750356:~$ apt-cache policy apache2
apache2:
  Installed: 2.4.29-1ubuntu4.3
  Candidate: 2.4.29-1ubuntu4.3
  Version table:
 *** 2.4.29-1ubuntu4.3 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.29-1ubuntu4.2 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages

Service is running right after the upgrade:
ubuntu@bionic-apache-1750356:~$ sudo systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Tue 2018-08-28 17:22:31 UTC; 25s ago
 Main PID: 2847 (apache2)
    Tasks: 54 (limit: 4915)
   CGroup: /system.slice/apache2.service
           ├─2847 /usr/sbin/apache2 -k start
           ├─2849 /usr/sbin/apache2 -k start
           └─2850 /usr/sbin/apache2 -k start

Aug 28 17:22:31 bionic-apache-1750356 systemd[1]: Starting The Apache HTTP Server...
Aug 28 17:22:31 bionic-apache-1750356 systemd[1]: Started The Apache HTTP Server.

Accessing the balancer url triggers the expected error:
ubuntu@bionic-apache-1750356:~$ wget http://localhost/foo
--2018-08-28 17:23:34-- http://localhost/foo
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 502 Proxy Error
2018-08-28 17:23:35 ERROR 502: Proxy Error.

And the logs confirm that the server tried to reach the balancer member:
ubuntu@bionic-apache-1750356:~$ tail /var/log/apache2/error.log -n 1
[Tue Aug 28 17:23:35.277191 2018] [proxy:error] [pid 2850:tid 139871414892288] [client 127.0.0.1:49520] AH00898: DNS lookup failure for: xxxxxx-xx-xxxxxxxx-xxxxx-xxxxx.xx-xxxx-x.xxxxx-xxx.xxx.xxxxx.xxxx returned by /foo

Verification complete and successfull for bionic.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.3

---------------
apache2 (2.4.29-1ubuntu4.3) bionic; urgency=medium

  * d/p/balance-member-long-hostname-part{1,2}.patch: Provide an RFC1035
    compliant version of the hostname in the
    proxy_worker_shared structure. A hostname that is too long is no longer a
    fatal error. (LP: #1750356)

 -- Andreas Hasenack <email address hidden> Wed, 27 Jun 2018 14:05:04 -0300

Changed in apache2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.