artful libpam-mysql-0.8.0-1
TL;DR
pam_mysql in artful will in the best case scenario just fail to authenticate users whose passwords were hashed with the server-side PASSWORD() SQL function. There is a buffer overflow happening, but it doesn't trigger a crash for some reason.
Detailed explanation follows.
pam_mysql, when crypt=2 is set in its configuration, it expects the password to be hashed according to the server-side PASSWORD() SQL function. From its README:
2 (or "mysql") = Use MySQL PASSWORD() function. It is possible that the encryption function used by PAM-MySQL is different from that of the MySQL server, as PAM-MySQL uses the function defined in MySQL's C-client API instead of using PASSWORD() SQL function in the query.
pam_mysql is indeed using an incorrect hash function: it's using my_make_scrambled_password() as a replacement for make_scrambled_password() to locally hash the given password and compare it with what is stored in the database:
char buf[42];
my_make_scrambled_password(buf, passwd, strlen(passwd));
vresult = strcmp(row[0], buf);
row[0] is the result of the SQL query that fetches the user's password hash
There are two problems with this:
a) my_make_scrambled_password() writes CRYPT_MAX_PASSWORD_SIZE bytes to buf, and that's way more than 42. From the mysql source code:
#define CRYPT_SALT_LENGTH 20
#define CRYPT_MAGIC_LENGTH 3
#define CRYPT_PARAM_LENGTH 13
#define SHA256_HASH_LENGTH 43
#define CRYPT_MAX_PASSWORD_SIZE (CRYPT_SALT_LENGTH + \
SHA256_HASH_LENGTH + \
CRYPT_MAGIC_LENGTH + \
CRYPT_PARAM_LENGTH)
42 is the length of the hexified hash produced by make_scrambled_password(), not my_make_scrambled_password().
b) the output of my_make_scrambled_password() is not a hex string like "*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19", but something like "$5$9Ws#033Q.TZtI4^?X#026y@@{e2$OxTGgW3PiJUVZ/AChiJgAdIWQ2u2B8kA/hHZgqNj.y.". So even if buf had the correct size, the comparison would never match what's produced by PASSWORD() on the server side. As the documentation admitted could happen.
If my_make_scrambled_password() is not found in the system mysqlclient library, pam_mysql will reimplement it, and funnily enough this reimplementation actually mimicks the desired behavior of make_scrambled_password() and produces an hexified hash compatible with the server's PASSWORD() function and with the right length of 42 bytes.
So, if mysqlclient doesn't export my_make_scrambled_password(), pam_mysql will work because it will use its own implementation. But in the ubuntu case, my_make_scrambled_password() is exported and used, and leads to this bug.
To reproduce this problem, setup mysql, vsftpd and libpam-mysql on artful as explained in bug #1574900.
I cannot explain why vsftpd doesn't crash in this scenario in artful: gcc's stack protector isn't triggered, nor is a segfault. In debugging I can see the buf variable getting way more than 42 bytes written to it, and if I add another stack variable next to it, it gets corrupted. But no crashes, just an authentication error.
Nevermind, this is caused by the broken libpam-mysql package. Close bug.