'man' command fails with lseek error opening cross-architecture index.db file (on network share)

Thanks for the package identification, result is:

$ apt-cache policy man-db
man-db:
  Installed: 2.6.1-2
  Candidate: 2.6.1-2
  Version table:
 *** 2.6.1-2 0
        500 http://gb.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
        100 /var/lib/dpkg/status

It seems to be a permissions issue, in that if I su to root (we have a root account enabled on this machine) it works, and if I su to a local user it also works, but not with the NIS log-in identity.

So there seems to be two issues here:
(1) Why can't a NIS account holder access the man pages?
(2) Why, in the event it can't access them, does it not give a more helpful message to say what went wrong?

Just to add that 'man' works fine on 10.04 LTS using NIS user accounts, so I am guessing it is something that has changed with 12.04 file locations/permissions, and/or the NIS package for 12.04

When you are logged in as an NIS user, what does the "id" command show?

Similarly, do you get an error message if you say "cp /var/cache/man/index.db /dev/null" (as the NIS user) ? (This just checks to see if that user has read access to the man index database file.)

The id command shows:

$ id
uid=xxx05(opr) gid=xxx00(local) groups=xxx00(local),4(adm),20(dialout),109(lpadmin),501(operadores),502(vboxuser),xxx03(hrpt),xxx04(dosgroup),xxx07(vboxsf)

(with xxx replacing various numeric values that our sysadmin don't want on a public forum)

The copy works without issue:

$ \cp --verbose /var/cache/man/index.db /dev/null
`/var/cache/man/index.db' -> `/dev/null'

Where 'cp' is normally aliased to 'cp -ip' so it squawks about /dev/null existing, and that it can't preserve timestamps, hence the '\cp'.

The obvious permissions are fine, all users can read the file:

$ ls -l /var/cache/man/index.db
-rw-r--r-- 1 man root 634842 May 18 12:30 /var/cache/man/index.db

Given those files are readable, could it be some file format change between the working-on-NIS version with 10.04 (2.5.7-2ubuntu1) and the problem version of 12.04 (2.6.1-2), say if it is seeing extra man files via the NIS/automount environment?

I'm not particularly familiar with the inner workings of man-db, but as far as I can see both versions of man-db use "libgdbm3 >= 1.8.3", so I wouldn't expect there to be a .db-file format change between Lucid and Precise, off hand. But it does seem like "extra" man files are in play, somehow...

You might check for differences in the environment variable lists between the different environments, in particular the LANG and LC_* variables.

Based on what the "man" man page says: you could try "man -u" and see if that complains about any of the index files. Also, I guess "man" looks for a config file in $HOME/.manpath; do such files exist in your NIS accounts?

If not, the /etc/manpath.config file's MANDB_MAP lines appear to list the directories where man looks for the index files (in the CATPATH column).

Or, it might be easier to run man under "strace" and just see directly what file it's trying to open or process when it dies...

Nathan

Running it with strace produced this near the end:

open("/packages/local/share/man/index.db", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=3208201, ...}) = 0
fcntl64(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) = 0
read(3, "\316\232W\23\0\0\20\0\0\0\20\0\0\0\0\0\0\0\20\0\21\0\0\0\0\0\20\0\246\252\0\0"..., 52) = 52
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb72b7000
read(3, "\0\0\0\0\5\0\0\0\0\0\0\0\253\0300\0\0\0\0\0\17\0\0\0\0\0\0\0003\2370\0"..., 1048524) = 1048524
lseek(3, 1048576, SEEK_SET) = 1048576
read(3, "", 0) = 0
lseek(3, 3078034552, SEEK_SET) = -1 EINVAL (Invalid argument)
write(2, "gdbm fatal: ", 12gdbm fatal: ) = 12

The file it read is nowhere near as big as the lssek(3, 3078034552, SEEK_SET) tried to position, and it should know that from the fstat64() call just above! The file is:

$ ls -l /packages/local/share/man/index.db
-rw-r--r-- 1 man daemon 3208201 Nov 30 10:15 /packages/local/share/man/index.db

I don't know why it attempted to go that far, presumably it is reading an offset from the database and that is now wrong?

There is no $HOME/.manpath file in any of the NIS accounts I have tried, or for the local user accounts on my 10.04 box either.
Actually it is slightly more bizarre, that NIS account (opr) is braking 'man' but another NIS account on the 12.04 machine is OK, while opr (and others) accounts are all OK on my 10.04 box, so maybe there is some environment value, etc, that causes opr on the 12.04 box to access /packages/local/share/man/index.db which is triggering the fault.
But why is it seeking so far?
If the version of DB was wrong, is there not some header value to flag this?

On Sat, May 19, 2012 at 13:15:48 -0000, Paul Crawford wrote:
> Actually it is slightly more bizarre, that NIS account (opr) is braking
> 'man' but another NIS account on the 12.04 machine is OK, while opr (and
> others) accounts are all OK on my 10.04 box, so maybe there is some
> environment value, etc, that causes opr on the 12.04 box to access
> /packages/local/share/man/index.db which is triggering the fault.

> But why is it seeking so far?

> If the version of DB was wrong, is there not some header value to flag
> this?

Yes, it seems like there are two separate (but related) questions here:
  1) why does the one NIS-account-on-Precise have this problem but not
     your others?

  2) why is man/gdbm trying to seek past the end of the file
     (and failing in such an unhelpful way when it does)?

As far as the first question: Does "/packages/local/share/man/" get opened
and/or stat-ed in the strace when you run "man" on the Lucid box and from
other accounts on the Precise box?

Is $MANPATH set differently in the different accounts?

Is /packages/local/ a shared network mount available on the Lucid box, too
(or something like that), or is it unique to the Precise box?

What are the access permissions on the directories in the
/packages/local/share/man path? (I'm wondering if somehow "opr" is the only
one able to access that index.db file?)

I don't have a lot of ideas about the second question, but I notice that
"file" gives me the following for the standard index.db file:

  $ file /var/cache/man/index.db
  /var/cache/man/index.db: GNU dbm 1.x or ndbm database, little endian

What does it report for your /packages index.db file?

      Nathan

> As far as the first question: Does "/packages/local/share/man/" get opened
> and/or stat-ed in the strace when you run "man" on the Lucid box and from
> other accounts on the Precise box?

It seems not.

> Is $MANPATH set differently in the different accounts?

There is no $MANPATH variable set on either machine, and the
/etc/manpath.config files appear identical on 10.04 and 12.04

> Is /packages/local/ a shared network mount available on the Lucid box, too
> (or something like that), or is it unique to the Precise box?

Yes, it is nfs auto-mounted and available on both machines and to all
NIS accounts.

> What are the access permissions on the directories in the
> /packages/local/share/man path? (I'm wondering if somehow "opr" is the only
> one able to access that index.db file?)

I can access it with other accounts, so it is not a permission issue.

> $ file /var/cache/man/index.db
> /var/cache/man/index.db: GNU dbm 1.x or ndbm database, little endian
>
> What does it report for your /packages index.db file?

The same description:
$ file /packages/local/share/man/index.db
/packages/local/share/man/index.db: GNU dbm 1.x or ndbm database, little
endian
However, I don't know if there are other variations in the DB format
that are not reported by the 'file' command.

There is something controlling the order of reading index files though,
as if I run strace and then look for index.db in the results I get this
for the opr account on 12.04:

$ grep 'index\.db' /tmp/man.txt
stat64("/packages/local/man/index.db", {st_mode=S_IFREG|0644,
st_size=3208201, ...}) = 0
stat64("/packages/local/man/index.db", {st_mode=S_IFREG|0644,
st_size=3208201, ...}) = 0
stat64("/packages/local/man/index.db", {st_mode=S_IFREG|0644,
st_size=3208201, ...}) = 0
open("/var/cache/man/local/index.db", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("/packages/local/share/man/index.db", O_RDONLY) = 3

But with another account I get:

$ grep 'index\.db' /tmp/man1.txt
open("/var/cache/man/local/index.db", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3
open("/var/cache/man/en_GB/index.db", O_RDONLY) = 3

And with opr an my 10.04 box I get:

$ grep 'index\.db' /tmp/man.txt
open("/var/cache/man/local/index.db", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("/var/cache/man/index.db", O_RDONLY) = 3

So in all of these cases, it tries /var/cache/man/local/index.db and
fails, but then the next one it tries differs, and in the case of opr...

One thing I noticed that is different is $PATH but I expected that to be the search order for programs, not for mand pages and/or index.db files!

On Sun, May 20, 2012 at 16:47:21 -0000, Paul Crawford wrote:
> One thing I noticed that is different is $PATH but I expected that to be
> the search order for programs, not for mand pages and/or index.db files!

Actually, I see that man-db does do some mapping between $PATH and the
man-page directory trees...

What are the $PATHs for your "opr" and a no-error account on the Precise
machine? Is $PATH for "opr" on your Lucid machine the same as on the
Precise machine?

Do you have any idea how the /package/local/share/man/index.db file was
originally generated? (Could it have been generated on some non-Linux
system, for example.)

      Nathan

The $PATH variables are different, with the 12.04 (32-bit) system having /packages/local/bin included in the list. If I remove that from the 12.04 system it then avoids the /packages/local/share/man/index.db file and it works.

However, if I add /packages/local/bin to the 10.04 system (64-bit, in case it matters) it takes noticeably longer to present the page, and it reads the /packages/local/share/man/index.db file (according to strace), but it still works!

So we have found why they the accounts behave differently, but there still seems to be an issue where the 12.04 system is not handling the network mounted index.db file correctly, or at least, not handling database/format errors in any sort of elegant manner.

On Sun, May 20, 2012 at 18:27:11 -0000, Paul Crawford wrote:
> The $PATH variables are different, with the 12.04 (32-bit) system having
> /packages/local/bin included in the list. If I remove that from the
> 12.04 system it then avoids the /packages/local/share/man/index.db file
> and it works.

Cool. (See below.)

> However, if I add /packages/local/bin to the 10.04 system (64-bit, in
> case it matters) it takes noticeably longer to present the page, and it
> reads the /packages/local/share/man/index.db file (according to strace),
> but it still works!

Out of curiousity: does the strace show that after opening at the
/packages/.../index.db file man proceeds to open the
/var/cache/.../index.db file before showing you the man page?

(Also, does the strace show the same "lseek" after the
/packages/.../index.db open() call as the one you reported earlier on
Precise?)

Finally, are you able to open any of the man pages found under
/packages/.../man/ ? (Just wondering if that index.db file is actually
usable by e.g. the Lucid machine, or if it also treats it as corrupt but
just fails more gracefully somehow.)

Anyway, here's a quote from the
/usr/share/doc/man-db/man-db-manual.txt.gz file that describes the $PATH-
to-manpath mapping (and I think explains the behavior that you found):

  Assuming that a $PATH exists, each path element it contains is
  scanned for in the config file. If found, the corresponding manpath
  element is appended to the internal manpath. However, if the element
  is not mentioned in the config file, a man directory relative to it
  will be sought. The subdirectories ../man, man, ../share/man, or
  share/man relative to the path component are appended to the internal
  manpath if they exist.

Running the "manpath" command (or "man --path" in the Precise version of
man-db) will print out the auto-generated manpath search list.

      Nathan

On Sun, May 20, 2012 at 18:27:11 -0000, Paul Crawford wrote:
> still seems to be an issue where the 12.04 system is not handling the
> network mounted index.db file correctly, or at least, not handling
> database/format errors in any sort of elegant manner.

Okay, moving on to this question... it turns out there is an "accessdb"
command included with the man-db package that dumpts the contents of a
database file.

So, what happens when you run
  $ /usr/sbin/accessdb /packages/local/share/man/index.db
on your two machines?

I'm guessing it will fail on at least the Precise machine, but if it
succeeds on both, does it produce the same output in both cases?

What do the $mtime$ and $version$ fields show? (Looks like
Debian/Ubuntu have had $version$ of "2.4.1" for quite a while.)

       Nathan

If I run the accessdb command on the 32-bit 12.04 machine it core dumps:

$ /usr/sbin/accessdb /packages/local/share/man/index.db
Segmentation fault (core dumped)

Doing the same on the 64-bit 10.04 machine produces something intelligible:

$ /usr/sbin/accessdb /packages/local/share/man/index.db
$mtime$ -> "1322648132"
$version$ -> "2.4.1"
Data::AMF::Type::NULL~3pm -> "- 3pm 3 1293669463 C Data::AMF::Type::Null - - "
Data::AMF::Type::Null~3pm -> "- 3pm 3 1293669463 A - - - "
NEdit~1 -> "- 1 1 1069626004 C nedit - - "
animate -> "- 1 1 1178029761 A - - - animates an image or image sequence on any X server."
any::moose -> "Any::Moose 3pm 3 1293668769 A - - - use Moose or Mouse modules"
appletviewer -> "- 1 1 1096427362 A - - - Java applet viewer"
attribute::params::validate -> "Attribute::Params::Validate 3pm 3 1293669227 A - - - Validate method/function parameters using attributes"
biosdecode -> "- 8 8 1172493847 A - - - BIOS information decoder"
bonnie++ -> "- 8 8 993754830 A - - - program to test hard drive performance."

As you suspected, the $version$ value is the same 2.4.1
But trying this on a 32-bit 10.04 machine (not normally with this file available) I get something vaguely familiar:

$ /usr/sbin/accessdb /packages/local/share/man/index.db
gdbm fatal: read error

So it seems the underlying issue is the database has no indication (or handling) of 64-bit versus 32-bit data! That seems to me to be a spectacularly dumb omission for anything that performs such an aggressive search for index.db files, as I expect a lot of organisations will have both 32-bit and 64-bit OS in use with networking as well :(

It seems this dumbness has been recognised since 2005 according to this bug report, but not taken seriously:

https://bugzilla.redhat.com/show_bug.cgi?id=162416

Apparently "gdbm-1.9.1 (already in Rawhide) provides different magic values for 32 and 64 bits, so we can discover what system the file was created on if we use this new version", though I see that Ubuntu 12.04 is still using an older version:

$ apt-cache policy libgdbm3
libgdbm3:
  Installed: 1.8.3-10
  Candidate: 1.8.3-10
  Version table:
 *** 1.8.3-10 0
        500 http://gb.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
        100 /var/lib/dpkg/status

On Sun, May 20, 2012 at 20:07:40 -0000, Paul Crawford wrote:
> But trying this on a 32-bit 10.04 machine (not normally with this file
> available) I get something vaguely familiar:
>
> $ /usr/sbin/accessdb /packages/local/share/man/index.db
> gdbm fatal: read error
>
> So it seems the underlying issue is the database has no indication (or
> handling) of 64-bit versus 32-bit data! That seems to me to be a

I can confirm that if I copy /var/cache/man/index.db from a 64-bit
machine to a 32-bit Lucid machine, I get the same error:

  $ uname -m
  i686
  $ lsb_release -r
  Release: 10.04
  $ accessdb mandb_index.db_x86_64
  gdbm fatal: read error

That same operation on a Precise 32-bit machine fails with a slightly
different error message:

  $ uname -m
  i686
  $ lsb_release -r
  Release: 12.04
  $ accessdb mandb_index.db_x86_64
  accessdb: can't open mandb_index.db_x86_64 for reading

Opening this _x86_64 file on a 64-bit Lucid machine works fine.... but
on that machine the _i686 file fails:

  $ uname -m
  x86_64
  $ lsb_release -r
  Release: 10.04
  $ accessdb mandb_index.db_i686 | less
  accessdb: can't open mandb_index.db_i686 for reading: Invalid argument

      Nathan

Trying to be positive, we have a possible work-around for our own system (drop /packages/local/bin from $PATH) but it is not really a decent fix. The underlying problem is difficult in that all current 32-bit and 64-bit DB files look similar, so work on an improved libgdbm3 is going to be difficult to manage that discrimination.
So what is the best option for the 'man' command? Should it ignore the $PATH search for index.db files and stick to those locations configured locally (e.g. by $MANPATH and/or /etc/manpath.config)?
Or still scan $PATH but ignore any files on network drives (that are potentially generated by another word-size machine)?

Status changed to 'Confirmed' because the bug affects multiple users.

On Sun, May 20, 2012 at 21:02:42 -0000, Paul Crawford wrote:
> Trying to be positive, we have a possible work-around for our own
> system (drop /packages/local/bin from $PATH) but it is not really a
> decent fix.

Another option might be to simply delete/rename the index.db file from
that directory tree; according to the man-db-manual.txt file, "man"
would then fall back to doing direct file searching within that tree
(which might be good enough in the case of the /packages/ man pages --
and certainly better than the current situation for 32-bit machines...).

       Nathan

> Apparently "gdbm-1.9.1 (already in Rawhide) provides different magic values for 32 and 64 bits, so we can
> discover what system the file was created on if we use this new version"

Yes... for example, search for the string "Is the magic number good?" in
  http://git.gnu.org.ua/cgit/gdbm.git/tree/src/gdbmopen.c

So with a newer version of libgdbm3, man-db should get a GDBM_BYTE_SWAPPED or GDBM_BAD_FILE_OFFSET error code when attempting to open an wrong-architecture index.db file, rather than a "successful" open() followed by a seg fault or a fatal gdbm error when attempting to read from the file, as happens now....

In the long run, migrating all of the DB files to newer ones that identify the size/swap nature is the right approach, however, I wonder if a simpler fix for 'man' which would be useful for current systems would be to keep everything in the in local DB (say the /var/cache/man/index.db file or similar) but to allow that to index any files discovered by its optimistic searching of anywhere along the $PATH locations?

Not really. I don't want man-db to be in the business of knowing what's local vs. not, and I very much want to preserve the property of having one database per MANDB_MAP entry. I think it would actually be much more complicated to attempt to unwind that - for instance there'd be interesting effects on localisation.

Having gdbm detect this is the right answer here, and if it does so then I think everything in man-db should just work, gracefully falling back to non-database-backed operations. So gdbm 1.9.1 sounds like the right answer; but of course that will do nothing as such to address the specific case here, because gdbm 1.9.1 can still read older database files (excellent for most other purposes) and so we're still going to crash on trying to read a database file created on a 64-bit 10.04 system on a 32-bit system. I therefore think we still need to figure out some more graceful handling of this in man-db.

The awkward bit is that the error goes through _gdbm_fatal, which does have a mechanism for the application to provide a different function to print the error message, but calls exit(1) either way. I can think of three ways to deal with this:

 1) Poke around in the database file ourselves, trying to guess whether the "dir" offset looks plausible given the file size.
 2) Fork to do an initial test open of the database.
 3) Pass in a fatal error handling function that uses longjmp to avoid the exit(1) call in the library.

1) is fragile and foul on general principles. 2) has performance problems at scale, perhaps not a problem in practice but I'd prefer to avoid it. I think 3) is the most practical, although it will take some fiddling to get it right.

Mon Jun 18 04:20:41 BST 2012 Colin Watson <email address hidden>

        Avoid fatal errors when opening a 64-bit GDBM database from a 32-bit
        process (Ubuntu bug #1001189).

        * libdb/db_gdbm.c (trap_error): New function.
          (man_gdbm_open_wrapper): Rearrange interface to call gdbm_open
          rather than taking its return value as an argument. Fetch a test
          value from the database after opening it, inside a setjmp/longjmp
          guard to prevent GDBM from exiting.
        * include/mydbm.h (man_gdbm_open_wrapper): Update prototype.
          (MYDBM_CTRWOPEN, MYDBM_CRWOPEN, MYDBM_RWOPEN, MYDBM_RDOPEN):
          Update man_gdbm_open_wrapper calls, since it now calls gdbm_open
          itself.
        * NEWS: Document this.

It sounds like it might not be a bad idea to backport this to 12.04 as well, once this has landed in 12.10.

This bug was fixed in the package man-db - 2.6.2-1

---------------
man-db (2.6.2-1) unstable; urgency=low

  * New upstream release:
    - Optimise apropos when given many arguments (LP: #927028).
    - apropos prints an error message and returns non-zero when it finds no
      matches (closes: #672661).
    - Avoid fatal errors when opening a 64-bit GDBM database from a 32-bit
      process (LP: #1001189).
  * Configure with --with-xz=xz --with-lzip=lzip.
  * Adjust debian/watch to track .tar.xz releases.
  * Convert debian/copyright to copyright-format 1.0.
  * Override hardening-no-fortify-functions Lintian warning for
    /usr/bin/manpath, as a false positive.

 -- Colin Watson <email address hidden> Mon, 18 Jun 2012 22:56:56 +0100

Hello Paul, or anyone else affected,

Accepted man-db into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/man-db/2.6.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This all seems to be behaving correctly now, with the version in precise-proposed: man falls back as specified. (accessdb still doesn't work, but there's no way to make it work with the old gdbm and in any case that's a relatively minor issue.)

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package man-db - 2.6.1-2ubuntu1

---------------
man-db (2.6.1-2ubuntu1) precise-proposed; urgency=low

  * Avoid fatal errors when opening a 64-bit GDBM database from a 32-bit
    process (LP: #1001189).
  * Link with -Wl,--enable-new-dtags, so that LD_LIBRARY_PATH can be used to
    override our private library directory. (This fixes a build failure
    when attempting to build the above change.)
 -- Colin Watson <email address hidden> Thu, 22 Nov 2012 11:57:21 +0000

This bug was fixed in the package gdbm - 1.14.1-2ubuntu1

---------------
gdbm (1.14.1-2ubuntu1) bionic; urgency=medium

  * Don't build using dietlibc (universe).

 -- Matthias Klose <email address hidden> Thu, 01 Feb 2018 13:02:01 +0100