Regression finding system certificates

Bug #2016439 reported by Sergio Durigan Junior
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
curl (Debian)
Fix Released
Unknown
curl (Ubuntu)
Fix Released
High
Sergio Durigan Junior
Lunar
Fix Released
High
Sergio Durigan Junior
Mantic
Fix Released
High
Sergio Durigan Junior

Bug Description

[ Impact ]

Users of applications that link against libcurl's NSS flavour might experience issues when trying to contact HTTPS servers. This can lead to scenarios where the application is unable to connect.

[ Test Plan ]

First, let's verify that the GNUTLS flavour of libcurl does the right thing:

$ lxc launch ubuntu-daily:lunar curl-bug2016439
$ lxc shell curl-bug2016439
# apt update && apt install -y libcurl4-gnutls-dev gcc
# cat > curl-test.c << __EOF__
#include <stdio.h>
#include <curl/curl.h>

int main(void)
{
  CURL *curl;
  CURLcode res;

  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
    /* example.com is redirected, so we tell libcurl to follow redirection */
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);

    /* Perform the request, res will get the return code */
    res = curl_easy_perform(curl);
    /* Check for errors */
    if(res != CURLE_OK)
      fprintf(stderr, "curl_easy_perform() failed: %s\n",
              curl_easy_strerror(res));

    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  return 0;
}
__EOF__
# gcc curl-test.c -o curl-test -lcurl
# ./curl-test
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...
#

You should see the HTML dump of example.com. Now, let's install the NSS flavour of libcurl and recompile the test program against it:

# apt install -y libcurl4-nss-dev
# gcc curl-test.c -o curl-test -lcurl
# ./curl-test
curl_easy_perform() failed: SSL peer certificate or SSH remote key was not OK

As we can see, there was an error when validating the TLS certificate.

[ Where problems could occur ]

The adjustment needed to the downstream patch is pretty simple and has been tested extensively. The original reporter mentioned that the issue did not happen before this patch was applied, so in the unlikely event of a regression the best route would be to revert the patch entirely.

[ More Info ]

This happens because of an error in one of our patches (authored by me) to teach libcurl where to properly find libnsspem.so and libnssckbi.so. The problem is that libnsspem.so is installed under /usr/lib/$(DEB_HOST_ARCH)/nss/, while libnssckbi.so is installed under /usr/lib/$(DEB_HOST_ARCH)/, but I mistakenly pointed libcurl to look under the "/nss/" directory for both libraries. As it turns out, libnssckbi.so is necessary in order to use the Mozilla's root certificate.

[ Original Description ]

[ Clone of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034359 ]

Between 7.88.1-2 and 7.88.1-5, there was a change to where curl with
nss looks for loadable libraries:

curl (7.88.1-4) unstable; urgency=medium

  * d/p/Use-correct-path-when-loading-libnss-pem-ckbi-.so.patch:
    Prepend "/nss/" before the library name.

Before the change to the load path, curl could find
/lib/x86_64-linux-gnu/libnssckbi.so but not
/lib/x86_64-linux-gnu/nss/libnsspem.so, after the change it's the
reverse.

libnssckbi.so is enough to get a trust root (the mozilla certificate
store is compiled inside that library), whereas libnsspem.so
(1.0.8+1-1) isn't.
This makes it impossible to connect to https servers by default for
programs that use curl with NSS.

Here is a way to test the regression:
debbisect -v --cache=./cache \
   --depends=libcurl4-nss-dev,git,pkg-config,libssl-dev,ca-certificates,cargo,nss-plugin-pem,p11-kit-modules,strace
\
  20230306T145638Z 20230306T203828Z \
    'chroot "$1" bash -exuc "
git clone --depth 1 https://github.com/alexcrichton/curl-rust.git
cd curl-rust
time cargo fetch
time cargo build --offline --example https
strace -efile target/debug/examples/https >/dev/null
"'

Changed in curl (Debian):
status: Unknown → Fix Released
tags: added: server-todo
description: updated
Changed in curl (Ubuntu):
status: Triaged → In Progress
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

As I understand it, libnsspem.so is the one in the "wrong" place:

/usr/lib/$ARCH/nss/libnsspem.so

A quick apt-file search on lunar shows that is the only package that puts something in /usr/lib/$ARCH/nss.

I appended this config to my ~/.pki/nssdb/pkcs11.txt file to test search paths:

  library=libnsspem.so
  name=PEM_plugin

And ran this test command with strace:

  certutil -d ~/.pki/nssdb -U

It shows the search path for the library:
  143458 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnsspem.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
  143458 openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libnsspem.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
  143458 openat(AT_FDCWD, "/lib/libnsspem.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
  143458 openat(AT_FDCWD, "/usr/lib/libnsspem.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

So yes, /usr/lib/$ARCH/nss is not searched for. Which means it's an odd placement choice decision from the src:nss-pem package. Or maybe src:nss should include that nss/ subdirectory in its search path.

In any case, back to the current bug.

It looks like a minimal patch would be to leave libnssckbi.so alone, as that will be found in the standard paths if I define it without an absolute path in pkcs11.txt:

  library=libnssckbi.so
  name=Mozilla Root Certs
  NSS=trustOrder=100

And strace:

  143993 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnssckbi.so", O_RDONLY|O_CLOEXEC) = 7

So how about just patching the libnsspem.so path? And reverting the change to the
`static const char *trust_library` path all the way back so it's just "libnssckbi.so"?

On a second note, has this been tested in other architectures, to be sure the _DEB_HOST_ARCH replacement is correct there as well? I'm thinking ppc64el vs ppc64le, for example, and whether the linux-gnu suffix is always used correctly.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

We talked about it and agreed on the following changes:
- don't patch the libnssckbi.so library path anymore, as that one is working in the original code
- for libnsspem.so, use a debian architecture macro that respects cross compiles and uses the target arch instead of the host one.

Changed in curl (Ubuntu Lunar):
assignee: nobody → Sergio Durigan Junior (sergiodj)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Thanks for the new upload. Please also fix this in mantic, either via a mantic upload with the same change, or a debian change/upload and later re-merge in mantic.

Changed in curl (Ubuntu Lunar):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-lunar
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Sergio, or anyone else affected,

Accepted curl into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curl/7.88.1-8ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the review and for accepting the SRU, Andreas.

The bug can be considered fixed in Mantic, although the change present there is not exactly the same as the one I uploaded to Lunar (it still uses $(DEB_HOST_ARCH) instead of $(DEB_TARGET_ARCH), and it unnecessarily patches the load path for libnssckbi.so).

I'm a bit weary of introducing another delta to the package only to address these minor details, so I will instead upload a small change to the patch in Debian and later merge it into Mantic.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Verifying the bug for Lunar.

First, make sure we can reproduce the problem. After following the steps outlined in the Test Plan, we see:

# ./curl-test
curl_easy_perform() failed: SSL peer certificate or SSH remote key was not OK
# apt policy libcurl4-nss-dev
libcurl4-nss-dev:
  Installed: 7.88.1-8ubuntu1
  Candidate: 7.88.1-8ubuntu1
  Version table:
 *** 7.88.1-8ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu lunar/universe amd64 Packages
        100 /var/lib/dpkg/status

Now, install libcurl4-nss-dev from -proposed and verify that the new package fixes the issue:

# apt policy libcurl4-nss-dev
libcurl4-nss-dev:
  Installed: 7.88.1-8ubuntu2
  Candidate: 7.88.1-8ubuntu2
  Version table:
 *** 7.88.1-8ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu lunar-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     7.88.1-8ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu lunar/universe amd64 Packages
# gcc curl-test.c -o curl-test -lcurl
# ./curl-test
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
...
<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

This concludes the verification for Lunar.

tags: added: verification-done verification-done-lunar
removed: verification-needed verification-needed-lunar
Changed in curl (Ubuntu Mantic):
status: In Progress → Fix Released
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (curl/7.88.1-8ubuntu2)

All autopkgtests for the newly accepted curl (7.88.1-8ubuntu2) for lunar have finished running.
The following regressions have been reported in tests triggered by the package:

curl/7.88.1-8ubuntu2 (ppc64el)
dgit/10.7 (amd64, ppc64el, s390x)
glance/2:26.0.0-0ubuntu1 (armhf)
gnocchi/4.5.0-0ubuntu1 (armhf)
ironic/1:21.4.0-0ubuntu1 (armhf)
libsoup3/3.4.0-1 (amd64, arm64)
lighttpd/1.4.67-1ubuntu2 (ppc64el)
mailman3/3.3.8-1ubuntu1 (arm64, ppc64el)
mediawiki/1:1.39.2-1 (armhf)
mediawiki-extension-codemirror/4.0.0~git20221204.b897975-1 (armhf)
mediawiki-extension-youtube/1.9.3~git20221020.e005c0b-1 (armhf)
mediawiki-skin-greystuff/1.2.5~git20220922.60bda8c-2 (armhf)
redmine/5.0.4-5 (armhf)
rsyslog/8.2302.0-1ubuntu3 (armhf)
rust-curl-sys/0.4.58-1 (arm64)
sahara/2:18.0.0-0ubuntu1 (armhf)
senlin/1:15.0.0-0ubuntu1 (armhf)
systemd/252.5-2ubuntu3 (amd64, arm64, s390x)
tang/11-2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/lunar/update_excuses.html#curl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

All dep8 failures have been resolved.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I just checked and a rebuild of the test app is not necessary, which is good, otherwise we would have to chase down anything in the archive that linked with the libcurl3-nss library package and rebuild those.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curl - 7.88.1-8ubuntu2

---------------
curl (7.88.1-8ubuntu2) lunar; urgency=medium

  * d/p/Use-correct-path-when-loading-libnss-pem-ckbi-.so.patch:
    Don't prepend "nss" when opening libnssckbi.so. Rename definition
    to _DEB_TARGET_ARCH. (LP: #2016439)
  * d/rules: Declare DEB_TARGET_MULTIARCH. Rename definition to
    _DEB_TARGET_ARCH.

 -- Sergio Durigan Junior <email address hidden> Thu, 20 Apr 2023 17:30:44 -0400

Changed in curl (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for curl has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.