libpam-mysql undefined symbol: make_scrambled_password

Disregard the proposed solution. It doesn't work. For some reason the patch from Fedora's version is causing stack overflows. In any event, this package needs to be fixed somehow. Legacy apps won't work anymore without it.

We need some kind of solution for:

PAM unable to dlopen(pam_mysql.so): /lib/security/pam_mysql.so: undefined symbol: make_scrambled_password

Status changed to 'Confirmed' because the bug affects multiple users.

I found the possible reason in this MySQL bug report: https://bugs.mysql.com/bug.php?id=80974

> Noted in 5.7.13 changelog
> The my_make_scrambled_password() function in the C client library was
> restricted earlier in MySQL 5.7 (not exported to client programs).
> The function has once again been made visible to client programs.

And since Ubuntu 16.04 use 5.7.12 LOL the fix was not introduced to Ubuntu. I believe that unless we program a make_scrambled_password to pam_mysql, this bug needs to be reported to libmysqlclient20.

After reading MySQL's code, I was able to make it work with this patch (almost the same as the Fedora patch). The patch switches from the missing make_scrambled_password() to my_make_scrambled_password(). Although my_make_scrambled_password() was not visible to client programs in MySQL 5.7, somehow libmysqlclient20 had fixed it earlier so my_make_scrambled_password() is available. And as the MySQL source code stated make_scrambled_password() as the wrapper around my_make_scrambled_password() to maintain client lib ABI compatibility, this switch changes should work.

P.S. I have tested the patch for 2 hours and hadn't noticed any stack overflows. I will post if the overflow happens in the future.

The attachment "libpam-mysql.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thanks for looking into it Kaijia. Unfortunately, after applying your patch, libpam-mysql still doesn't work. When I try to use it for VSFTPD authentication, with this patch and the patch I tried to apply original from Fedora, it crashes with the following:

*** stack smashing detected ***: /usr/sbin/vsftpd terminated

I can't seem to get it to work. I don't have any other log entries besides that though. I looked in syslog, auth.log, and vsftpd.log.

More information for my problem here:

https://bugs.launchpad.net/ubuntu/+source/vsftpd/+bug/1574911

Hi,
this item affects me and te .deb files provided above were actually helpful. Can you provide a src deb package?
It the adressed somewhere and will be patched (officially) for 16.04 ?

Thanks a lot!

Markus

Hi Markus,

libpam-mysql is alive again as a new maintainer take over the project. As for this bug, somehow another user named stewjohn committed a patch exactly the same to mine above to the new repo. So you may use the latest version in Ubuntu 17.04 that had the bug fixed from http://packages.ubuntu.com/zesty/libpam-mysql

I've been using my patch for more than half year and it's running without any problem so I think this bug can be closed now.

Sorry, but using it with VSFTPD doesn't work. It used to, but even with the patch, it doesn't work, so for me the package is worthless.

I've moved on to using libpam-python anyways. However, it would be nice if this package actually worked with VSFTPD as it used to.

So there appear to be two moving parts here.

a) If building against a version of libmysqlclient without make_scrambled_password, then https://github.com/NigelCunningham/pam-MySQL/commit/2f6837b5fb8ef4f5d8a561f3bf82b36cc79e77f4 is needed. This commit is present in 17.04 but not earlier releases.

b) libpam-mysql had a b-d on libmysqlclient15-dev. This was fixed in Ubuntu in 0.7~RC1-4ubuntu2 for 16.04 and in Debian in 0.7~RC1-4.1 (which is present in 16.10+). So this is Fix-Released everywhere already and not tracked in this bug.

For a) in my prior comment, I have provided test builds at https://launchpad.net/~nacc/+archive/ubuntu/lp1574900. Please try them once the PPA is done building and report back.

Hello OwN, or anyone else affected,

Accepted pam-mysql into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam-mysql/0.7~RC1-4ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello OwN, or anyone else affected,

Accepted pam-mysql into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam-mysql/0.7~RC1-4.1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I'm still running into the same issue.

USER test
331 Please specify the password.
PASS xxxx
*** stack smashing detected ***: /usr/sbin/vsftpd terminated
500 OOPS: priv_sock_get_result
Disconnecting from site localhost

Contents of /etc/pam.d/vsftpd:

auth required pam_mysql.so user={DBUSERHERE} passwd={DBPASSHERE} host=localhost db={DBNAMEHERE} table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2
account required pam_mysql.so user={DBUSERHERE} passwd={DBPASSHERE} host=localhost db={DBNAMEHERE} table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2

It's not working for me.

I installed the proposed package like this:

wget -N "http://launchpadlibrarian.net/311161289/libpam-mysql_0.7~RC1-4.1ubuntu1.1_amd64.deb"
dpkg -i libpam-mysql_0.7~RC1-4.1ubuntu1.1_amd64.deb

I hope that's correct. If so, it's still broken. VSFTPD login doesn't work when using libpam-mysql.

Granted, it turns out I am using MariaDB (10.0.29-MariaDB-0ubuntu0.16.04.1 Ubuntu 16.04). Is this package supposed to work with MySQL and its drop-in replacement MariaDB?

Trying official MySQL...

Hello OwN,

vsftpd having a stack smashing issue would appear to be its own bug.

This bug is purely about solving the mysql auth issues with 'undefined symbols'.

Using the official MySQL didn't work either with VSFTPD. The undefined symbols problem seems to be solved though.

I will file a bug against VSFTPD then since it appears they are not related... though this used to work just fine in older versions of Ubuntu.

Reopened:

https://bugs.launchpad.net/ubuntu/+source/vsftpd/+bug/1574911

But yes, the fix for libpam-mysql looks good... I guess. My only usage was for VSFTPD though.

While re-triaging the update in other bug I also came to verify this fix in Xenial at least.

I've been following this thread for a while now, as this issue also exists in Trusty. I am using someone's custom-patched vsftpd deb. Is there any plan to backport this release to trusty?

@jlellis: There is no MySQL 5.7 in Trusty. Are you using a PPA? Or are you saying that MySQL 5.5 also has this problem?

This issue with libpam predates mysql 5.7. It's present in trusty's vsftpd 3.0.2-1ubuntu2.14.04.1, mysql 5.5.54-0ubuntu0.14.04.1, and the standard libpam package. My workaround last year when I encountered the issue was to remove both vsftpd and libpam-mysql from the default repos and to manually install vsdfpd from

http://dinofly.com/files/linux/vsftpd_3.0.2-1ubuntu2_amd64.deb

https://ubuntuforums.org/archive/index.php/t-2218804.html is the spot where I found this fix a year ago. I've been waiting for an official fix ever since but if one has been released, I missed it. I'm still running this custom version.

It is entirely possible that the bug I'm experiencing with 5.5 is different but related to what is causing the problem covered in 1574900, but as the symptoms seem to be identical, I tend to assume it is the same bug.

The host involved here is my employer's primary public ftp server where we give 3rd parties (customers, contractors, etc) the ability to upload large files and we have a simple mysql db where their virtual users are kept. If you need a guinea pig host for testing, I can easily spin up a clone of this system for testing purposes.

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Nish Aravamudan
Sent: Monday, March 20, 2017 11:56 AM
To: Joi L. Ellis <email address hidden>
Subject: [Bug 1574900] Re: libpam-mysql undefined symbol: make_scrambled_password

@jlellis: There is no MySQL 5.7 in Trusty. Are you using a PPA? Or are you saying that MySQL 5.5 also has this problem?

--
You received this bug notification because you are subscribed to pam- mysql in Ubuntu.
Matching subscriptions: pam-mysql
https://bugs.launchpad.net/bugs/1574900

Title:
  libpam-mysql undefined symbol: make_scrambled_password

Status in pam-mysql package in Ubuntu:
  Fix Released
Status in pam-mysql source package in Xenial:
  Fix Committed
Status in pam-mysql source package in Yakkety:
  Fix Committed
Status in pam-mysql package in Fedora:
  Unknown

Bug description:
  [Impact]

   * libpam-mysql is unable to be used because of an upstream symbol
  change in libmysqlclient.

   * Upstream libpam-mysql has fixed this by updating the source to use
  the now-exported API.

  [Test Case]

  Steps to reproduce (with vsftpd):
  Installation requirements: mysqlserver 5.7.x, libpam-mysql, vsftpd

  create vsftpd user:
  useradd --home /home/vsftpd --gid nogroup -m --shell /bin/false vsftpd

  create mysql database and user:
  == // Start SQL

      CREATE DATABASE vsftpd;
      GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP ON vsftpd.* TO 'vsftpd'@'localhost' IDENTIFIED BY 'ftpdpass';
      GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP ON vsftpd.* TO 'vsftpd'@'localhost.localdomain' IDENTIFIED BY 'ftpdpass';
      FLUSH PRIVILEGES;

      USE vsftpd;

      CREATE TABLE `accounts` (
        `id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
        `username` VARCHAR( 30 ) NOT NULL ,
        `pass` VARCHAR( 50 ) NOT NULL ,
      UNIQUE (
        `username`
      ));
      insert into accounts(username,pass) values('...

@jlellis: you get "libpam-mysql undefined symbol: make_scrambled_password" on Trusty?

This bug is not about fixing vsftpd itself. Probably a new bug should be opened, or the vsftpd bug should be used for trusty as well.

The VSFTPD bug regarding libpam-mysql stack smashing is here:

https://bugs.launchpad.net/ubuntu/+source/vsftpd/+bug/1574911

Someone reproduced it today and left much better instructions on how to reproduce the issue than I did.

Should I open a new bug with those findings tagged under libpam-mysql?

@Brian and other SRU team members, let's see if we can debug the vsftpd stack smashing issue before releasing pam-mysql (it will probably be a follow-on patch for pam-mysql), if that's ok with the SRU team?

TL;DR
- pam_mysql.c buf in pam_mysql_check_passwd() is overflowing
- my_make_scrambled_password() is NOT returning content that can be compared to what is stored in the mysql DB when using PASSWORD().
- my_make_scrambled_password_sha1() seems to be the right one to use, as it returns a string of hex values, but it's not exported

Details:

It's buf that is overflowing in pam_mysql.c:
/* PASSWORD */
case 2: {
        char buf[42];
...
        my_make_scrambled_password(buf, passwd, strlen(passwd));
        syslog(LOG_AUTHPRIV | LOG_ERR, PAM_MYSQL_LOG_PREFIX "andreas: row0=%s buf=%s passwd=%s", row[0], buf, passwd);

I added some simple debugging above, and got;
May 11 22:08:42 yakkety-pam-mysql vsftpd: pam_mysql - andreas: row0=*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 buf=$5$EI#015W7T^j'x#00173VPb#016g#025J$8cyy7LNV0Uhg0RCY1OV0OcrbqCB7eaYBsRmWnBJmzT2 passwd=password

row0 matches what's in the DB:
mysql> SELECT pass FROM accounts WHERE username = 'user';
+-------------------------------------------+
| pass |
+-------------------------------------------+
| *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |

You can see that buf contains something in a very different format than what is stored in mysql. It's definitely not in hex format, it's much larger than 42 and I'm not even sure if it's supposed to be \0 terminated. I haven't found docs for my_make_scrambled_password() yet.

That magic number 42 comes from mysql's sql/auth/password.c for a *different* function:
/*
    MySQL 4.1.1 password hashing: SHA conversion (see RFC 2289, 3174) twice
    applied to the password string, and then produced octet sequence is
    converted to hex string.
    The result of this function is used as return value from PASSWORD() and
    is stored in the database.
  SYNOPSIS
    my_make_scrambled_password_sha1()
    buf OUT buffer of size 2*SHA1_HASH_SIZE + 2 to store hex string
    password IN password string
    pass_len IN length of password string
*/

Where:
./include/sha1.h:#define SHA1_HASH_SIZE 20 /* Hash size in bytes */

so buf has size 2*20+2 = 42. (what about null termination?)

BUT that Synopsis is for my_make_scrambled_password_sha1(), *NOT* my_make_scrambled_password().

my_make_scrambled_password() seems to be something very different:
void my_make_scrambled_password(char *to, const char *password,
                                size_t pass_len)
{

  char salt[CRYPT_SALT_LENGTH + 1];

  generate_user_salt(salt, CRYPT_SALT_LENGTH + 1);
  my_crypt_genhash(to,
                     CRYPT_MAX_PASSWORD_SIZE,
                     password,
                     pass_len,
                     salt,
                     0);

}

CRYPT_MAX_PASSWORD_SIZE is much bigger than 42, and in any case the result is not a string of hex values.

Further digging needed, but I'm EOD now :)

I'm going to post this over in #1574911 too

Marking this as verification-failed as it seems this is definitivly not the right solution.

Since mysql 5.6, there is no exported function that mimics the server-side PASSWORD() hash. my_make_scrambled_password() doesn't do it, and make_scrambled_password() isn't exported anymore.

The solution here is to reimplement that hash algorithm inside pam_mysql. One of the upstream forks did it, but still tries to use my_make_scrambled_password() if it's available, for which I opened bug https://github.com/NigelCunningham/pam-MySQL/issues/29.

Hello,

As of the following bug is there a working fix for xenial?
Because i installed the unofficial package at first comment but turns on error "Error in service module" if i try either https://launchpad.net/ubuntu/+source/pam-mysql/0.7~RC1-4ubuntu2.1 still have error "PAM unable to dlopen(pam_mysql.so): /lib/security/pam_mysql.so: undefined symbol: make_scrambled_password"

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

I have a branch with a patch for xenial, but I'm afraid the pam-mysql source package is very much broken there. The build process de-applies the patches, then builds the binaries, then applies the patches, and finishes. Not even the two existing patches are applied in that package, much less my third patch.

The xenial branch is at https://code.launchpad.net/~ahasenack/ubuntu/+source/pam-mysql/+git/pam-mysql/+ref/xenial-pam-mysql-scrambled-1574911 if someone wants to take over. The patch is backported in there but untested.

Can someone please release this fix?!? As far as I can tell, the code was fixed but never released?