_gpgme_io_select breaks when more than 1024 file descriptors are open

Bug #2037595 reported by Colin Watson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
gpgme1.0 (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
High
Colin Watson
Jammy
Fix Released
High
Colin Watson

Bug Description

[Impact]

Launchpad's buildd-manager is a complex event-driven daemon that manages several hundred builders on production. As such, it can often end up with more than 1024 file descriptors open. It also uses gpgme from time to time to fetch signing keys for PPAs when dispatching builds that involve PPAs.

On focal (and also on jammy), this breaks because `_gpgme_io_select` is implemented using select(2), whose manual page says: "WARNING: select() can monitor only file descriptors numbers that are less than FD_SETSIZE (1024)—an unreasonably low limit for many modern applications—and this limitation will not change. All modern applications should instead use poll(2) or epoll(7), which do not suffer this limitation." This is exactly the limitation we're running into, resulting in mysterious "Too many open files" errors despite resource limits being set to more than comfortable values.

Fortunately, this was fixed upstream in https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=commitdiff;h=8148237cb4ae20755c06a44d71761c7030973c3d, and lunar has that fix. focal and jammy do not (there may be a similar problem with older LTSes too, but I only care about it back to focal). buildd-manager currently runs on focal.

We've been running buildd-manager with a cherry-picked fix from upstream for a few weeks now, and I'd like to get that SRUed back into Ubuntu so that we don't accidentally lose this patch in the event of a security update or similar.

[Test Plan]

William Grant provided this minimal reproducer:

```
root@tmp:~# python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gpg
>>> c = gpg.Context()
>>> list(c.keylist())
[]
>>> fs = []
>>> for i in range(1024):
... fs.append(open('/dev/null'))
...
>>> list(c.keylist())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 751, in keylist
    key = self.op_keylist_next()
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1195, in op_keylist_next
    raise excp
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1190, in op_keylist_next
    errorcheck(gpgme.gpgme_op_keylist_next(self.wrapped, ptr))
  File "/usr/lib/python3/dist-packages/gpg/errors.py", line 129, in errorcheck
    raise GPGMEError(retval, extradata)
gpg.errors.GPGMEError: GPGME: Too many open files
```

[Where problems could occur]

`_gpgme_io_select` is pretty central to gpgme, so if something is wrong the package will be pretty broken. On the other hand, that also means any glaring problems are likely to be caught by the package's test suite.

Revision history for this message
Colin Watson (cjwatson) wrote :
Changed in gpgme1.0 (Ubuntu):
status: New → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote :
Changed in gpgme1.0 (Ubuntu Focal):
assignee: nobody → Colin Watson (cjwatson)
Changed in gpgme1.0 (Ubuntu Jammy):
assignee: nobody → Colin Watson (cjwatson)
Changed in gpgme1.0 (Ubuntu Focal):
status: New → In Progress
importance: Undecided → High
Changed in gpgme1.0 (Ubuntu Jammy):
status: New → In Progress
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Colin, or anyone else affected,

Accepted gpgme1.0 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gpgme1.0/1.16.0-1.2ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in gpgme1.0 (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Changed in gpgme1.0 (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Colin, or anyone else affected,

Accepted gpgme1.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gpgme1.0/1.13.1-7ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (gpgme1.0/1.16.0-1.2ubuntu4.2)

All autopkgtests for the newly accepted gpgme1.0 (1.16.0-1.2ubuntu4.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

samba/2:4.15.13+dfsg-0ubuntu1.5 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#gpgme1.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (gpgme1.0/1.13.1-7ubuntu2.2)

All autopkgtests for the newly accepted gpgme1.0 (1.13.1-7ubuntu2.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

samba/2:4.15.13+dfsg-0ubuntu0.20.04.7 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#gpgme1.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [gpgme1.0/focal] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for focal for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Chris Halse Rogers (raof) wrote :

ubuntu@sru-jammy:~$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gpg
>>> c = gpg.Context()
>>> list(c.keylist())
[]
>>> fs = []
>>> for i in range(1024):
... fs.append(open('/dev/null'))
...
>>> list(c.keylist())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 762, in keylist
    key = self.op_keylist_next()
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1206, in op_keylist_next
    raise excp
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1201, in op_keylist_next
    errorcheck(gpgme.gpgme_op_keylist_next(self.wrapped, ptr))
  File "/usr/lib/python3/dist-packages/gpg/errors.py", line 129, in errorcheck
    raise GPGMEError(retval, extradata)
gpg.errors.GPGMEError: GPGME: Too many open files
>>>
ubuntu@sru-jammy:~$ sudo apt install -t jammy-proposed libgpgme11
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  libgpgme11
1 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
Need to get 136 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 libgpgme11 amd64 1.16.0-1.2ubuntu4.2 [136 kB]
Fetched 136 kB in 1s (104 kB/s)
(Reading database ... 34017 files and directories currently installed.)
Preparing to unpack .../libgpgme11_1.16.0-1.2ubuntu4.2_amd64.deb ...
Unpacking libgpgme11:amd64 (1.16.0-1.2ubuntu4.2) over (1.16.0-1.2ubuntu4.1) ...
Setting up libgpgme11:amd64 (1.16.0-1.2ubuntu4.2) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
ubuntu@sru-jammy:~$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gpg
>>> c = gpg.Context()
>>> list(c.keylist())
[]
>>> fs = []
>>> for i in range(1024)
  File "<stdin>", line 1
    for i in range(1024)
                        ^
SyntaxError: expected ':'
>>> for i in range(1024):
... fs.append(open('/dev/null'))
...
>>> list(c.keylist())
[]
>>>

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Chris Halse Rogers (raof) wrote :

ubuntu@sru-focal:~$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gpg
>>> c = gpg.Context()
>>> list(c.keylist())
[]
>>> fs = []
>>> for i in range(1024):
... fs.append(open('/dev/null'))
...
>>> list(c.keylist())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 751, in keylist
    key = self.op_keylist_next()
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1195, in op_keylist_next
    raise excp
  File "/usr/lib/python3/dist-packages/gpg/core.py", line 1190, in op_keylist_next
    errorcheck(gpgme.gpgme_op_keylist_next(self.wrapped, ptr))
  File "/usr/lib/python3/dist-packages/gpg/errors.py", line 129, in errorcheck
    raise GPGMEError(retval, extradata)
gpg.errors.GPGMEError: GPGME: Too many open files
>>>

ubuntu@sru-focal:~$ sudo apt install -t focal-proposed libgpgme11
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  libgpgme11
1 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.
Need to get 120 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libgpgme11 amd64 1.13.1-7ubuntu2.2 [120 kB]
Fetched 120 kB in 1s (95.6 kB/s)
(Reading database ... 32247 files and directories currently installed.)
Preparing to unpack .../libgpgme11_1.13.1-7ubuntu2.2_amd64.deb ...
Unpacking libgpgme11:amd64 (1.13.1-7ubuntu2.2) over (1.13.1-7ubuntu2.1) ...
Setting up libgpgme11:amd64 (1.13.1-7ubuntu2.2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...

ubuntu@sru-focal:~$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gpg
>>> c = gpg.Context()
>>> list(c.keylist())
[]
>>> fs = []
>>> for i in range(1024):
... fs.append(open('/dev/null'))
...
>>> list(c.keylist())
[]
>>>

tags: added: verification-done verification-done-focal
removed: removal-candidate verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package gpgme1.0 - 1.16.0-1.2ubuntu4.2

---------------
gpgme1.0 (1.16.0-1.2ubuntu4.2) jammy; urgency=medium

  * Cherry-pick from upstream:
    - posix: Use poll instead, when available, removing use of select
      (LP: #2037595).

 -- Colin Watson <email address hidden> Thu, 28 Sep 2023 01:17:40 +0100

Changed in gpgme1.0 (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for gpgme1.0 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package gpgme1.0 - 1.13.1-7ubuntu2.2

---------------
gpgme1.0 (1.13.1-7ubuntu2.2) focal; urgency=medium

  * Cherry-pick from upstream:
    - posix: Use poll instead, when available, removing use of select
      (LP: #2037595).

 -- Colin Watson <email address hidden> Thu, 28 Sep 2023 01:16:09 +0100

Changed in gpgme1.0 (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.