Permission Denied on nfs clients for only some files

This may be caused by bug 57543.

Could you please provide a copy of your /var/log/kern.log when this error occurs. Attach it using the web form on the bug page.

Also, can you downgrade to 2.6.17-6 and confirm that this bug disappears?

Thanks.

Nothing new appears in my kern.log when this happens. :-( There is nothing in my kern.log about NFS at all. The only thing in there regarding any filesystem activity it loads of:

BUG: warning at fs/inotify.c:538/inotify_d_instantiate()
  <c01958ef> inotify_d_instantiate+0x6f/0x80 <c01842b5> d_splice_alias+0x45/0x100
  <f89168a6> ext3_lookup+0xa6/0x100 [ext3] <c01831c9> d_alloc+0x119/0x1b0
  <c0179d5c> __lookup_hash+0xbc/0xf0 <c017a257> lookup_one_len+0x87/0x90
  <f8e508f7> nfsd_lookup+0xe7/0x4f0 [nfsd] <f8ce936f> sunrpc_cache_lookup+0x4f/0x130 [sunrpc]
  <f8e55677> nfsd3_proc_lookup+0xa7/0x100 [nfsd] <f8e58d80> nfs3svc_decode_diropargs+0x90/0xf0 [nfsd]
  <f8e4a0ac> nfsd_dispatch+0x8c/0x1d0 [nfsd] <f8ce2431> svc_process+0x3e1/0x6d0 [sunrpc]
  <f8ce4fbc> svc_recv+0x3ec/0x4c0 [sunrpc] <f8e4a625> nfsd+0x185/0x310 [nfsd]
  <f8e4a4a0> nfsd+0x0/0x310 [nfsd] <c0101005> kernel_thread_helper+0x5/0x10

But I think is has to do with beagle and it's use of inotify. I have had this problem for a long time as opposed to this new problem which has only appeared since the -6 release of this kernel.

Could you downgrade to -6 and verify that it wasn't a problem there?

There may still be a version of that package in /var/cache/apt/archives.

On Mon, 2006-25-09 at 19:43 +0000, Simon Law wrote:
> Could you downgrade to -6 and verify that it wasn't a problem there?

Sure. If I had it still.

> There may still be a version of that package in /var/cache/apt/archives.

Unfortunately not. :-( Due to space issues, I had to apt-get clean
just the other day. Anywhere I can get it? Is there an archive kept at
ubuntu or anything like that? Is it still in the pool of a current
mirror or does that get removed when updated packages for a given
release (i.e. edgy) are uploaded?

b.

--
My other computer is your Microsoft Windows server.

Brian J. Murrell

Hmm. It looks like -6 is off the mirrors now.

However, http://mirror.cs.umn.edu/ubuntu/pool/main/l/linux-source-2.6.17/ has
-7. If you can reproduce it with that, it would point to -7 as having the NFS patch that caused the regression.

On Mon, 2006-25-09 at 20:24 +0000, Simon Law wrote:
> Hmm. It looks like -6 is off the mirrors now.
>
> However, http://mirror.cs.umn.edu/ubuntu/pool/main/l/linux-source-2.6.17/ has
> -7. If you can reproduce it with that, it would point to -7 as having the NFS patch that caused the regression.

I do see kernel-image-2.6.17-7-386-di_2.6.17-7.20_i386.udeb there. Is
that what I want? What is udeb?

b.

--
My other computer is your Microsoft Windows server.

Brian J. Murrell

OK. The earliest kernel I could find around here was 2.6.17-7 (linux-image-2.6.17-7-generic) and it has the problem too. I am pretty damn sure -6 didn't have the problem although without one to actually test I cannot be absolutely positive. I did run -6 for quite a while so I am sure I would have noticed this problem.

Just to further confirm it being an Ubuntu kernel problem, I have these same problems even if I mount the exported on the same on the same host that's exporting it.

Is there an SCCS repo kept somewhere that I can view the changes from -6 to -7?

There's a git repo at:

http://git.kernel.org/git/

Look for ubuntu-edgy.git

So, how do we proceed on trying to fix this bug?

I have a -7 installed that has the bug and no access to a -6 to confirm
my assertion that -6 did not have the bug.

b.

--
My other computer is your Microsoft Windows server.

Brian J. Murrell

FYI - I am having the same problem on 2.6.17-10.

(matt@case) ~/workspace$ cd tivo/
(matt@case) ~/workspace/tivo$ ls
tivo_backup/ tivo_mfs_tools/
(matt@case) ~/workspace/tivo$ ls
ls: .: Permission denied
(matt@case) ~/workspace/tivo$

No logfile messages are produced.

I'm not sure this is related, but I'm seeing these log messages from a courier-imap server on a dapper server which nfs-mounts user home directories:

Oct 29 13:00:57 pong imapd-ssl: Failed to create cache file: maildir_lock (gjc)
Oct 29 13:00:57 pong imapd-ssl: Error: Permission denied

I checked the permissions in the home dir; everything is ok.

This only started happening since upgrading the nfs server from dapper to edgy. From then on, the imap server (on the dapper host) is very flaky, keeps giving strange errors. But the imap server wasn't touched, and it was working well before the nfs server upgrade, so it has to be the fault of the nfs server.

uname -a gives: Linux ferrari 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686 GNU/Linux

This happens in the following 2.6.17-10 versions:
-386
-generic
-server

So it looks to be pretty architecture independent.

This happens both on nfsv3 and nfsv4.

Is there anything I can do to help? I've tried just about everything I can think of, and I'm kind of dead in the water until this is fixed. Make no mistake, this completely breaks NFS on Ubuntu.

On my Edgy NFS server I had to revert to booting the older dapper kernel image... not nice! :-/

I agree this completely breaks NFS on Ubuntu. Also severely compromises the credibility of Ubuntu as a server OS, IMHO.

Also here the NFS server is unusable.
The bug is reported weeks before the edgy was released. A server without reliable nfs is useless and unacceptable for the majority of installations.

Now edgy is released 2 weeks ago and this mission critical bug remains unfixed.
Also the former kernel 2.6.17-6 has disappeared.

Please provide a download link to the old version for a short time solution.

If I couldn't solve the problem in my installation here, I have to drop edgy and go back to dapper (or maybe debian).

I didn't notice this bug until after I'd removed the old dapper kernels.

This is especially difficult because I'm pitching Ubuntu + Xen as a server consolidation solution to management. Because they're dragging their feet on hardware purchases, I have some time, but as it stands right now, we cannot deploy Ubuntu.

I'm thinking of perhaps using AFS instead, and selling it as a "look at all these whiz features AFS has over NFS".

Maybe AFS will also fail. Here the SMBFS infrastructure is also broken.

So what, you're saying that the two major network file sharing protocols are broken in the release version of Edgy?

Ouch.

In my network NFS isn't reliable any more.
Access from the notebook via Samba fails also.

I agree with the classification "high" for this bug. But I can't understand, why it isn't assigned to anybody after 6 weeks.

Just a note that you can compile the latest stuck kernel 2.6.18.2 and it doesn't have this bug. I know that might not be ideal for everybody but if you need your server to work it might be the only way.

Here is a link to how to compile a kernel the ubuntu way for anyone who hasn't done it before.

http://www.howtoforge.com/kernel_compilation_ubuntu

upgrade from dapper to edgy = nfs not working. Often users get permissions denied, or stale nfs file, in their own (automounted) homedirs. Makes edgy useless as nfs server.

ls
#:~/.openoffice.org2/user/basic$ ls -l
ls: .: Permission denied

lsof:
bash 12238 petekh cwd unknown 0,23 /h/petekh/.openoffice.org2/user/basic (deleted) (nfs.server.com:/home/petekh)

linux-image-2.6.17-10-server 2.6.17-10.33

vanilla 2.6.18.3 works.

Looks like linux-image-2.6.17-10-server 2.6.17-10.33 only have serious problems with dot-file or files in dot directories.

I'm seeing this problem with non-dot files and files in non-dot directories in 2.6.17-10 kernels.

Please try the kernel located here:
  http://people.ubuntu.com/~kyle/kernels/etienneg/2006-12-07-2/

(amd64 already built, i386 will be uploaded when they finish building)
And see if they resolved the permission denied problems. I'm hesitant to revert the NFSv4 patch without confirmation that it resolves this more important issue for users.

Thanks!
  Kyle

Will do. For what it's worth, I have an i686 server and a k7 client. So, I'm waiting for the i386 one. :-)

Ok, they're uploaded. Please let this bug know whether they are an improvement or not.

These kernels just made it worse.

yt is the server, case is the client.

(matt@yt) ~$ uname -rv
2.6.17-10-server #1 SMP Fri Dec 8 00:18:03 UTC 2006

matt@case:/var/log$ uname -rv
2.6.17-10-server #1 SMP Fri Dec 8 00:18:03 UTC 2006

Case no longer mounts NFS at boot (this was flaky to begin with and is addressed by another bug). Anyway, if you go to Case and do a:

sudo mount -a

The mount process hangs. The machine is still responsive, but nothing useful happens, and no useful error messages are printed.

I reboot, so as to not have concurrent mount processes running and try to mount /usr/local, I get a " BUG: unable to handle kernel paging request at virtual address 5f637072" message. (Attached complete output).

Same here!

Ubuntu serve Edgy with kernel 2.6.17-10-server #2 SMP results in a completelly unusable nfs exports. Randomly gives permision denied to clients (with breezy).

It seems that can be solved remounting with nfsver=3 nfsmountver=3.

Nah... Also problems with nfsver=3...

It takes longer, but again start appearing random permision denied:

ls: ./firefox1.5/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}: Permiso denegado
ls: ./firefox1.5/defaults: Permiso denegado
ls: ./firefox1.5/res: Permiso denegado
ls: ./firefox1.5/icons: Permiso denegado
ls: ./firefox1.5/components: Permiso denegado

Seems rather strange here as well. I migrated a few user homes to a newly set up Edgy server running the 2.6.17-10 generic kernel. The first two users were quite happy, the third had the Permission Denied error, then I switched back to the Dapper kernel. All three users use KDE and nfs mounted their homes on Dapper clients. When checking the third users home directory on the server, the Permission-Denied-files did not exist. When comparing the contents of the directories, the only strange thing was that the third user had made a backup of all his dot-files in an extra directory. Very odd.
Regards Michael

TEMPORARY SOLUTION (while ubuntu releases a new kernel package):

Guillermo Pérez and me solved this compiling a new kernel (2.6.18.5) and installing it. For all the people that find difficult to do that task, or doesn't want to loose time compiling, i'm hosting for a while a compiled kernel. You only have to donwload and install:

1) Visit http://www.eurielec.etsit.upm.es/~chesko/nfs_solution and download the two deb files
2) install them with dpkg -i <name_of_the_file>
3) check if your /boot/grub/menu.1st is properly modified
4) reboot
5) enjoy

Instructions on how to easily compile the kernel can be found at http://www.howtoforge.com/kernel_compilation_ubuntu

I have an easier solution: use the dapper kernel :)

For us it was not an option: It was giving an APIC error and refusing to boot. Perhaps it's too recent hardware that needs an updated kernel.

Anyway this must be solved soon... musn't it?

Guillermo/anonym - the kernel you posted seems to be working for me. Thanks for taking the time to compile it.

There has been an update:
2.6.17-10.33 ---> 2.6.17.1-10.34
Anybody knows if that update it's supposed to solve this? or it's unrelated?

I only see this talking about nfs in the release-info (http://www.ubuntu.com/usn/usn-395-1) and I dunno if it's worth a try or not:

Matthias Andree discovered that the NFS locking management daemon
(lockd) did not correctly handle mixing of 'lock' and 'nolock' option
mounts on the same client. A remote attacker could exploit this to
crash lockd and thus rendering the NFS imports inaccessible. This only
affects Ubuntu 5.10. (CVE-2006-5158)

I have recently upgraded fro dapper to edgy and believe that I also now have this problem, the symptoms are as follows:

1. Gnome users settings often revert to default gnome settings when the user logs in to hosts which mount /home over NFS;
2. The following error is often reported when starting evolution on hosts which mount /home over NFS: “Failed to get lock using fcntl(2): Resource temporarily unavailable”;
3. The following error is often reported when starting firefox on hosts which mount /home over NFS: “Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system”;
4. Thumbnails displayed by nautilus (either on the desktop or in file system browser) occasionally revert to basic icons before the preview reappears again a few seconds later. Once again this only occurs on hosts which mount /home over NFS.

Users either logging into the NFS server directly or logging in remotely using the gdm XDMCP chooser experience none of these problems.

Unfortunately my home network is now virtually useless :-(

Try using the kernel that anonym posted. It's working for me.

That's the nice thing about a community - when the maintainers drop the ball, someone with the time and skills can step up.

On Sat, 2007-01-06 at 03:44 +0000, Matthew Caron wrote:
> Try using the kernel that anonym posted. It's working for me.
>
> That's the nice thing about a community - when the maintainers drop the
> ball, someone with the time and skills can step up.

Yeah, but the question remains, that if somebody has found a solution to
such an basic issue (serving nfs) why is the official package not being
fixed too? Everybody who wants to use NFS on Edgy has to use a
non-official kernel to do it?

b.

--
My other computer is your Microsoft Windows server.

Brian J. Murrell

This problem certainly does appear to be related to the edgy kernel. The best solution for me was to reinstall the the dapper kernel and restricted modules package on the server – this fixed my NFS and also keeps my nVidia graphics card happy. All NFS clients can continue to run the edgy kernel.

In my case I fixed the problem as follows:
1.Edit /etc/apt/sources.list to add dapper repositories (I added the following)
deb http://gb.archive.ubuntu.com/ubuntu/ dapper main restricted universe
deb http://gb.archive.ubuntu.com/ubuntu/ dapper-updates main restricted universe
deb http://security.ubuntu.com/ubuntu dapper-security main restricted universe
deb http://archive.ubuntu.com/ubuntu/ dapper-backports restricted main universe

2.Install linux-image-2.6.15-27-686 and linux-restricted-modules-2.6.15-27-686 using either apt-get or synaptic.

3.Edit /boot/grub/menu.lst to move the 2.6.15-27-686 kernel entry above the 2.6.17-10-generic entry.

4.Reboot.

This seems like a bit of a hack but it does work.

Previously when running the edgy kernel on the NFS server, I found that network shares did not always show up when executing “sudo mount” on remote hosts (although they were mounted). Since installing the dapper kernel on the NFS server, the shares show up as expected.

As you say, good job the community is on the case here – there is no way I would have thought of installing another kernel to fix this without the information posted here, so thank you.

In view of the seriousness of this issue, it is very disappointing that no official fix has been made available since the bug was first reported in September.

Sorry for the late info, but this is being worked on for Edgy already.

I experienced the bug in 2.6.17.1-10.34.

On the linux-nfs mailing list, it was suggested that adding no_subtree_check to the options in /etc/exports is a workaround:

http://marc.theaimsgroup.com/?l=linux-nfs&m=116038104221946&w=2

In my situation I was able to do that, and have not seen the error since.
Make sure you understand the implications of that option (man 5 exports) for your configuration before you decide to use it.

bristleburger: I agree with your disappointment. I am equally dismayed that something this important has remained unfixed for so long.

Ben: Thanks for the update.

James: I've tried no_subtree_check when I first encountered this issue. I did not notice a difference. However, if it works for you, great!

I just wanted to point out that -6 actually still is available. Launchpad has the -386 version at https://launchpad.net/+builds/+build/236152/linux-image-2.6.17-6-386 and the -686 version at http://librarian.launchpad.net/3880531/linux-image-2.6.17-6-686_2.6.17-6.18_i386.deb

This is still broken in the new one - 2.6.17-11. I'm not saying it was supposed to be fixed, just confirming that it's still broken for me.

Our main fileserver here (which is based on dapper, but uses an edgy 2.6.17 kernel (version 2.6.17-11-server, backported to dapper) because we need NFSv4 with working krb5p support) is also hit by the problem. Using "no_subtree_check" as export option seems to work around the problem, but as I consider this to be just a better hack, a more thorough solution is needed (see below for my analysis and suggestion).

The problem manifests itself not only in "permission denied" errors, but also stale NFS handles. Googling around finally leads to Novell's bug 195040 in SuSE Linux (https://bugzilla.novell.com/show_bug.cgi?id=195040), for which a patch is already available (https://bugzilla.novell.com/attachment.cgi?id=94729&action=view). We are currently preparing a kernel rebuild using this patch and have thus not yet verified it (rebooting is currently not an option, as we have active users and the functioning workaround in place), but from the comments found on Novell's page it seems to work fine, and I thought it made sense to share this information; additionally, the patch seems to have found its way into Linus' git version of the kernel (commit d1bbf14f37261c2c0dba71404602e1ddcec069d2 from NeilBrown <email address hidden>, dating Sun, 30 Jul 2006 10:03:16 +0000, just one month after the commit introducing the problem), and it resembles the situation of fs/nfsd/nfsfh.c in dapper, which is known to be working without these problems.

So, for a quick workaround, I suggest using the "no_subtree_check" export option; it would certainly be nice to have above (simple!) patch (which I also added as an attachment to this comment) or (even better) the correcting commit mentioned above included in some updated kernel in edgy in the near (!) future, as the need to have this workaround makes ubuntu look less professional than we might want it to. (SuSE has it included for *quite some time* now...)

And, by the way, the fix mentioned above is included in the version of linux-source-2.6.17 that is in edgy-proposed (2.6.17-50.50). The newer (by one week, timestamp-wise) release of linux-source-2.6.17 in edgy-security (2.6.17-11.35) still contains the bug. The feisty kernel (2.6.20) is also fixed.

Update: The version of 2.6.17 in edgy-proposed (linux-source-2.6.17-50.50) is *not fixed*. Although it should (IMHO) not cause the problems concerning permissions or stale NFS handles, it is missing another security fix (the one that originally caused the problem, from commit 7fc90ec93a5eb71f4b08403baf5ba7176b3ec6b1 in Linus' git tree). Users affected by the bug should therefore not try to solve the problem by installing 50.50 on their systems. Instead, pull commit d1bbf14f37261c2c0dba71404602e1ddcec069d2 mentioned above and patch the kernel source. This puts the problematic section of code in its original place again (the situation also seen in 50.50), but additionally adds similar code in the "else" brance of the "if (!fhp->fh_dentry)".

For those users unfamiliar with git, I have appended the patch for the correcting commit.

Oops... The patch in the attachment above obviously mixed up spaces and tabs... Use the "--ignore-whitespace" patch option when applying it.

Problem still occurring as of 3rd May 2007.

> Problem still occurring as of 3rd May 2007.

The server is still running Edgy, right?

It appears to be fixed on feisty w/ 2.6.20-15-generic.

I have been running Feisty for a week or so now and can confirm that this problem has been fixed in the 2.6.20-15-generic kernel (for me at least).
Feisty also fixes some other kernel related problems that I experienced with Edgy so I think it is probably a good idea to upgrade.

Per the last sets of comments this appears to be resolved with the Feisty 2.6.20 kernel so I am marking as "Fix Released" there. Against 2.6.17 this will be closed. Thanks.

Please don't reopen already fixed bugs without explanation.