[SRU] Using `fastcgi_cache` or `proxy_cache` with nginx-extras causes the push module to throw errors.

Confirmed. I'm seeing this issue also on nginx-extras 1.1.19-1ubuntu0.4

Arnaud, are you still getting this with 1.4.3? If so, I will consult with Debian about this, as the packaging in the PPA is pretty much directly from them.

------

Joe Clifford, please open a bug against the Ubuntu package specifically, as that needs to be handled differently from this bug.

Reported to be seen in Saucy as well.

This is a two part bug: This bug has been reported to be present in NGINX team's PPAs as well as the version(s) in the Ubuntu repositories.

To all impacted by this bug: Please test the version in Trusty (1.4.3) either with the nginx team's PPA or by testing on Trusty itself (in a VM or otherwise). I need to know whether 1.4.3 is impacted by this bug.

I've just quickly installed nginx-extras and nginx-common 1.4.3 from the
nginx PPA on to a 12.04 LTS VPS serving a couple of web sites. It is
running without showing those errors in the error.log any more which is
good news. I don't have time to dig too deep and test properly at the
moment but I'll leave the new nginx install running over the next few days
and let you know if anything goes wrong.

On 19 November 2013 16:47, Thomas Ward <email address hidden> wrote:

> To all impacted by this bug: Please test the version in Trusty (1.4.3)
> either with the nginx team's PPA or by testing on Trusty itself (in a VM
> or otherwise). I need to know whether 1.4.3 is impacted by this bug.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1216817
>
> Title:
> issue with nginx cache and nginx push module in nginx-extra package
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nginx/+bug/1216817/+subscriptions
>

Hi,

using 1.4.3-2~precise0:

2013/11/28 13:00:59 [info] 2441#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:84
2013/11/28 13:01:00 [alert] 2475#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2475#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2476#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2476#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2453#0: cache manager process 2475 exited with fatal code 2 and cannot be respawned

But the fastcgi_cache seems to work...
I have cache files generated in the cache directory...
weird...

Status changed to 'Confirmed' because the bug affects multiple users.

I get the same error in an Ubuntu 13.10 machine running the latest mainline reléase:
2013/12/15 07:57:57 [info] 1621#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:11:27 [info] 1674#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:11:28 [alert] 1691#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1691#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1686#0: cache manager process 1691 exited with fatal code 2 and cannot be respawned
2013/12/15 08:11:28 [alert] 1692#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1692#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:22 [info] 1702#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:16:23 [alert] 1719#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1719#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1720#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1720#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1714#0: cache manager process 1719 exited with fatal code 2 and cannot be respawned
2013/12/15 08:21:48 [info] 1749#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:21:49 [alert] 1767#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1767#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1766#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1766#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)2013/12/15 06:48:30 [info] 1155#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78

On a different computer were don't use proxy_cache I see only the info entry:
2013/12/15 06:59:01 [info] 1160#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 07:27:51 [info] 1213#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 07:37:24 [info] 1175#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:02:34 [info] 1799#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:02:42 [info] 1805#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:21:49 [alert] 1761#0: cache manager process 1766 exited with fatal code 2 and cannot be respawned

I also enabled fast cgi cache on the computer that was working fine and now it is experiencing the same issue as with the computer that was using proxy cache:
2013/12/15 17:05:09 [info] 3939#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:87
2013/12/15 17:05:10 [alert] 3956#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3956#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3951#0: cache manager process 3956 exited with fatal code 2 and cannot be respawned
2013/12/15 17:05:10 [alert] 3957#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3957#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)

Can someone post an example of their configuration, so I can attempt to replicate this in Debian's packages? If this breakage also occurs in Debian then it can be fixed there, and depending on how they fix it, I may be able to SRU the updates for Ubuntu, but I will be able to fix the issues in the PPAs.

I have used a plain test example from Digital River on getting cache working with php5-fpm, and can replicate this error with `fastcgi_cache` in Debian. This is being forwarded to Debian for a fix, and I will attach a bug number here later.

This is reported in Debian against the nginx-extras package as Debian bug 732251. (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732251)

As this has been sent up to Debian for a fix, because this is replicated in *their* packages first and foremost, I am marking both the Nginx project bug and the bug against the nginx packages in Ubuntu as "triaged", and the severity has been set to "Low" for both as this does not actually impact the ability for the system to cache data for `proxy_cache` or `fastcgi_cache` arguments, based on testing done by myself and one of the Debian maintainers.

Hello. I'm the writer of the push module. This is a problem with my code, not nginx, and I am working on a fix. See https://github.com/slact/nginx_http_push_module/issues/83 for the github discussion on this bug.

Fixed in version 0.711, available from https://github.com/slact/nginx_http_push_module/archive/v0.711.tar.gz . checksums at https://pushmodule.slact.net

Hello!
I'm using Ubuntu Server 14.04.1 in VM

# cat /etc/issue
Ubuntu 14.04.1 LTS \n \l
# uname -a
Linux vm-newsru-fe-02 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I'm using nginx
# dpkg -l | grep nginx
ii nginx-common 1.4.6-1ubuntu3.1 all small, powerful, scalable web/proxy server - common files
ii nginx-extras 1.4.6-1ubuntu3.1 amd64 nginx web/proxy server (extended version)

I use nginx for proxying with cache.
Cache size set to 1024M but it is growing much more because nginx cache manager do not work:

# tail /var/log/nginx/error.log
2014/10/23 07:47:47 [info] 11822#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:109
2014/10/23 07:47:49 [alert] 11840#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11840#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11839#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11839#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11827#0: cache manager process 11839 exited with fatal code 2 and cannot be respawned

This should be fixed now that the packages have been resynced with Debian which I believe had an updated version of the module included in it.

The package in Ubuntu is still affected.

Marking as "Fix Released" for other releases - this was fixed in Debian as 1.4.7-1, and Utopic and later have 1.6.x. As such, this is fixed in Utopic and later.

Upstream diff from the nginx-http-push module developers is located here: https://github.com/slact/nginx_http_push_module/commit/836e8319c93681386fb00e6bd34d9e37612f3334.diff

Precise debdiff for fixing this bug, using the upstream changes.

Trusty debdiff for fixing this bug

To everyone affected: We need example configurations to use here, so we can test this getting fixed.

To the SRU team: I am build-testing this now in a ppa, and will let you know if it builds correctly.

To the SRU team: The changes outlined in the debdiffs have successfully built on i386 and amd64, the two primary architectures. I do not have the alternate architectures available on my test PPA. https://launchpad.net/~teward/+archive/ubuntu/nginx-sru/+packages is the test PPA, with a focus on the details of the packages in the PPA.

Hello, Thomas!
I sent work nginx configuration to your e-mail (on ubuntu.com).

Vorobyoff: Thanks for providing the conf, that gives me the basic information I needed to start building a test configuration.

To all others: I estimate I'll have a test configuration available by the end of the day, and I will upload the test configuration here accordingly when finished.

Here's a test procedure:

(1) Use the attached test configuration file as a base, to test with localhost. (Partly derived from digital ocean's tutorials)

(2) Create a PHP file in the specified path. Alternatively, use an alternate root path.

(3) `tail -f /var/log/nginx/error.log`. Watch for any error outputs.

(4) Load nginx with the config. There should not be any errors triggered similar to above.

(5) To further test, load the PHP file you created. It should be rendered correctly and there should not be any errors in the error log similar to those observed with this bug.

Confirmed that this testcase works.

You may use ppa:teward/nginx-sru as a test build if you wish to really test this before it's uploaded to the proposed repositories.

I tested this on Trusty, but it should also work for precise.

REMEMBER: nginx-extras has the module so this needs to be tested with nginx-extras and not any other package.

Please use the test configuration for your OS version - the only difference is the fastcgi_pass line because the php5-fpm listener doesn't listen in the same place on Precise as it does in Trusty (and vice versa)

Hello!
Can anybody say when this bug will be fixed?
Now nginx-extras in 14.04 is in unusable state. We are stopping to use 14.04 in production.

Vorobyoff:

This will be fixed whenever the SRU team sees the upload - it's in the queue.

In the mean time, you can use my PPA with these changes already present - https://launchpad.net/~teward/+archive/ubuntu/nginx-sru/

It's ultimately the same builds that the debdiffs help make. If you can confirm the changes work that is one step in this - confirming the fixes actually work.

We use nginx with patches in production 2 days.
Everything is fine - no errors in logs, cache is not growing.

Removed old debdiffs because they're outdated and superseded by other updates. Also removed the sponsors team since I have upload rights now.

Thanks to Robie Basak for giving me a second-opinion technical review of the SRUs, before I work on pushing them up.

Updated the debdiffs and am reattaching them here.

Severity increased for Trusty, Precise, because in further testing I was able to replicate the comments from #16 where the cache can grow beyond the maximum set bounds.

Hello Arnaud, or anyone else affected,

Accepted nginx into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nginx/1.1.19-1ubuntu0.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Arnaud, or anyone else affected,

Accepted nginx into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nginx/1.4.6-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Been pulling my hair out on this...

Rolled the trusty-proposed and it fixed the crazy logs.
From basic usage it looks like this is good on 14.04 x64
Upgraded both nginx_common & extras (1.4.6-1ubuntu3.2)

Thanks!

J

(Added trusty-verification-done per #38 which used trusty-proposed and confirmed this is fixed there.)

I can also confirm that trusty-proposed fixed this same issue for me.

Can someone test this on Precise as well? Precise needs to be tested to see if it is fixed with the package in proposed there.

This bug was fixed in the package nginx - 1.4.6-1ubuntu3.2

---------------
nginx (1.4.6-1ubuntu3.2) trusty-proposed; urgency=medium

  * d/modules/nginx-http-push: Apply upstream bugfix. (LP: #1216817)
    * src/ngx_http_push_module_setup.c: Modify push module code with
      upstream changes to fix an issue with initialization when using
      `fastcgi_cache` or `proxy_cache`.
    * tests/nginx-cachemanager.conf: (new file) Include upstream change
      of adding an nginx-cachemanager.conf file to the tests.
 -- Thomas Ward <email address hidden> Mon, 09 Feb 2015 12:08:50 -0500

The verification of the Stable Release Update for nginx has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Please note to everyone: This is only fixed in TRUSTY so far - we need Precise tests to be done with the package in precise-proposed to get it from precise-proposed to precise-updates

Tested in Precise and it removes the error output and problems as well for Precise.

This bug was fixed in the package nginx - 1.1.19-1ubuntu0.8

---------------
nginx (1.1.19-1ubuntu0.8) precise-proposed; urgency=medium

  * d/modules/nginx-http-push: Apply upstream bugfix. (LP: #1216817)
    * src/ngx_http_push_module_setup.c: Modify push module code with
      upstream changes to fix an issue with initialization when using
      `fastcgi_cache` or `proxy_cache`.
    * tests/nginx-cachemanager.conf: (new file) Include upstream change
      of adding an nginx-cachemanager.conf file to the tests.
 -- Thomas Ward <email address hidden> Mon, 09 Feb 2015 12:02:52 -0500