Apache segfaults on more than 50% of requests when running mod_perl2 application on Dell Xeon servers

Hi Robin,
are you using a local %ENV variable, or the Sys::HostIP module?

This is definitely a modperl problem.

Gary: I don't use the code "local %ENV", if that's what you're asking, but I DO fetch variables from %ENV to determine what kind of environment I'm in. I do not use the Sys::HostIP package.

I'm using the code below (I've cut out the lines that mentions ENV):

<code>
$pf->var('docroot',$ENV{'DOCUMENT_ROOT'} . $uri->path);
my $ip_addr=inet_ntoa(inet_aton($q->remote_host)) || $ENV{'REMOTE_ADDR'};
my $ip_addr=inet_ntoa(inet_aton($q->remote_host)) || $ENV{'REMOTE_ADDR'};
</code>

I also use the following snippet of code to identify if I'm running CGI or mod_perl mode:

<code>
    if ($ENV{'MOD_PERL'}) {
        eval "use Apache2::ServerUtil ();";

        if ($@) {
            die __"Apache2/ServerUtil.pm unavilable. Please verify your mod_perl 2.x setup.";
        }

        if (Apache2::ServerUtil->server->dir_config('PORTFOLIO_HOME')) {
            $homedir=Apache2::ServerUtil->server->dir_config('PORTFOLIO_HOME');
            $bindir=$homedir . '/bin';
        } else {
            die __"Please set PORTFOLIO_HOME when using mod_perl.";
        }
        $hosting_environment=$ENV{'GATEWAY_INTERFACE'} . "; " . $ENV{'MOD_PERL'};
    } elsif ( $ENV{'GATEWAY_INTERFACE'}=~/^CGI\/.*$/ and $ENV{'SCRIPT_FILENAME'}) {
        (my $dummy,$bindir,my $dummy2)=File::Spec->splitpath($ENV{'SCRIPT_FILENAME'}); # Split filename from directory
        if ( -d $bindir) {
            $bindir=~s/^(.*)$/$1/;
            $homedir=$bindir . '/../';
        }
        $hosting_environment=$ENV{'GATEWAY_INTERFACE'};
    } else {
        $bindir=$FindBin::Bin;
        # Untaint directory if it's valid
        if ( -d $bindir) {
            $bindir=~s/^(.*)$/$1/;
            $homedir=$bindir . '/../';
        }
        $hosting_environment='CLI';
    }
</code>

I'm also using this code in /etc/apache2/mods-enabled/perl.conf:

<code>
# Make sure files with .pl extension are run as mod_perl scripts

AddHandler perl-script .pl

# Make sure these point to the base folder of the application

PerlSetVar PORTFOLIO_HOME "/home/portfolio/portfolio"
PerlSwitches -w -I/home/portfolio/portfolio/lib

# Portfolio is a full ModPerl::Registry script.

PerlOptions +ParseHeaders
PerlOptions +SetupEnv
PerlResponseHandler ModPerl::Registry

# Automatic reload check of changed modules at each request.
# Useful in development, but shouldn't be used in production.

PerlInitHandler Apache2::Reload
</code>

How would I go about setting up some mod_perl tracing (mods-available/perl.conf) to get some more details on this problem. I've read the docs on perl.apache.org about tracing, but I'm unsure how to exactly use it. I tried to enable all tracing, but for some reason it doesn't do as I intend.

Do one of you have a working apache/mod_perl trace setup that traps this kind of error?

Robin Smidsrød wrote:
> How would I go about setting up some mod_perl tracing (mods-available/perl.conf) to get some more details on this problem. I've read the docs on perl.apache.org about tracing, but I'm unsure how to exactly use it. I tried to enable all tracing, but for some reason it doesn't do as I intend.
>
> Do one of you have a working apache/mod_perl trace setup that traps this kind of error?

Please try the following:
download mod-perl:
$ apt-get source libapache2-mod-perl2
get the build dependencies:
$ apt-get build-dep libapache2-mod-perl2

$ cd libapache2-mod-perl2-2.0.2

Edit debian/rules to set MP_USE_GTOP to 0 (compilation error otherwise)
and MP_TRACE to 1.

Preferably, install the fakeroot package, then
$ DEB_BUILD_OPTIONS="noopt nostrip" dpkg-buildpackage -rfakeroot

Install the resulting package...

Add
PerlTrace all
to your configuration.

When running, you should see extra tracing in the apache2 error log.
Check in the error log for
"localizing", referring to the name of an environment variable.

This might show a use of "local %ENV" somewhere in a module, as this is
a known reproducible crash (I can't access the patch for it at the
moment because the apache repository is down at the moment).

Otherwise, there's another known reason for a crash in this area: if
your $r object is a closure, the code may fail to work, and crash.

If neither suggestion helps, please send the backtrace resulting from
the crash.

I just recompiled mod_perl 2.0.2 from source, and used perl Makefile.PL MP_DEBUG=1 MP_TRACE=1 to enable debugging.

And after using the curinfo gdb macro mentioned in the debugging document I got this trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1214097728 (LWP 4241)]
0xb7a01912 in modperl_env_request_tie (my_perl=0x8186388, r=0x82110b0) at modperl_env.c:404
404 EnvMgObj = (char *)r;
(gdb) curinfo
0Cannot access memory at address 0x600d
(gdb) bt
#0 0xb7a01912 in modperl_env_request_tie (my_perl=0x8186388, r=0x82110b0) at modperl_env.c:404
#1 0xb79ed82e in modperl_response_handler_cgi (r=0x82110b0) at mod_perl.c:1087
#2 0x08077056 in ap_run_handler ()
#3 0x08077452 in ap_invoke_handler ()
#4 0x0806941d in ap_process_request ()
#5 0x08064a19 in _start ()
#6 0x082110b0 in ?? ()
#7 0x00000004 in ?? ()
#8 0x082110b0 in ?? ()
#9 0x082110b0 in ?? ()
#10 0x00000000 in ?? ()
#11 0x0820b170 in ?? ()
#12 0x00000001 in ?? ()
#13 0x080c143c in ?? ()
#14 0x00000001 in ?? ()
#15 0x0820b170 in ?? ()
#16 0xbfbb3118 in ?? ()
#17 0x080818c6 in ap_run_process_connection ()
#18 0x080818c6 in ap_run_process_connection ()
#19 0x080755f9 in ap_graceful_stop_signalled ()
#20 0x080757bd in ap_graceful_stop_signalled ()
#21 0x080758aa in ap_graceful_stop_signalled ()
#22 0x080760ce in ap_mpm_run ()
#23 0x0807bb1e in main ()

During the PerlTrace all I noticed these lines relating to localization of variables (I have changed all identificators and hostnames/IPs, but it shouldn't really matter).

modperl_env_magic_set_all: [4741/0x8185958] localizing: SCRIPT_NAME => /object.pl
modperl_env_magic_set_all: [4741/0x8185958] localizing: SERVER_NAME => www.example.com
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_REFERER => http://www.example.com/pin
modperl_env_magic_set_all: [4741/0x8185958] localizing: SERVER_ADMIN => <email address hidden>
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_CONNECTION => close
modperl_env_magic_set_all: [4741/0x8185958] localizing: REQUEST_METHOD => GET
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_ACCEPT => */*
modperl_env_magic_set_all: [4741/0x8185958] localizing: SCRIPT_FILENAME => /home/portfolio/portfolio/cgi/object.pl
modperl_env_magic_set_all: [4741/0x8185958] localizing: SERVER_SOFTWARE => Apache/2.0.54 (Ubuntu) mod_perl/2.0.2 Perl/v5.8.7
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_X_BLUECOAT_VIA => 72B41EF8A2248DEF
modperl_env_magic_set_all: [4741/0x8185958] localizing: QUERY_STRING => cmd=read;object=e3b23b59-3ec2-458f-a0d9-fed3b96e4e61
modperl_env_magic_set_all: [4741/0x8185958] localizing: REMOTE_PORT => 32700
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_USER_AGENT => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
modperl_env_magic_set_all: [4741/0x8185958] localizing: SERVER_PORT => 80
modperl_env_magic_set_all: [4741/0x8185958] localizing: SERVER_SIGNATURE =>
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_CACHE_CONTROL => max-stale=0
modperl_env_magic_set_all: [4741/0x8185958] localizing: HTTP_ACCEP...

Robin Smidsrød wrote:
> During the PerlTrace all I noticed these lines relating to localization of variables (I have changed all identificators and hostnames/IPs, but it shouldn't really matter).
>
> modperl_env_magic_set_all: [4741/0x8185958] localizing: SCRIPT_NAME => /object.pl

This definitely looks like something you're using is localizing the
environment. I'm attaching a patch, can you try it out?
You can apply the patch with
patch -p1 < mod_perl_r357236.patch

>> Otherwise, there's another known reason for a crash in this area: if
>> your $r object is a closure, the code may fail to work, and crash.
>
> Could you explain this problem a bit more in detail? I'm uncertain if I'm affected by this.

No, it's doubtful then - you would have to go out of your way to do this
since it causes crashes :-)

------------------------------------------------------------------------
r357236 | gozer | 2005-12-16 21:20:23 +0000 (Fri, 16 Dec 2005) | 6 lines

"local %ENV;" caused Segmentation fault in modperl_env_request_tie()

Reported-By: Peter Walsham <email address hidden>
Thread: http://thread.gmane.org/gmane.comp.apache.mod-perl/22236

Weeee! This seems to have fixed the problem!

I just recompiled mod_perl 2.0.2 from source with the included patch and voila, no more segfaults!

I will of course have to do some more thorough testing, but so far it looks good.

What is the correct way to build a new binary deb with this patch included?

And secondly, will this patch be included in Ubuntu's release version of mod_perl2 for 5.10 server or will I have to manually patch this in everytime you upgrade the release package?

I will install the release version of apache2 again and build a binary deb of mod_perl with this patch included. I will then report back if it works with the release version of apache2 in breezy. Also need to check if mpm-prefork and mpm-worker both work without hickups.

Sidenote: I wonder why this bug is only apparent on the Xeon machines, and not on the older P3 CPUs? Do you see any logic in this? Not really important at this point, but I'm just curious...

I won't be updating breezy with this, no, but dapper will be released on June 1, and I'll make sure this patch gets into dapper.

Fixed in dapper in libapache2-mod-perl2_2.0.2-2ubuntu1