OSError at /oops.py/ when using lp-oops

I investigated this issue with Stefan, we found out a few things but still no
root cause:

- the librarian appinstance is creating those unreadable oops directories.

    <Chex> librarian@mizuho:/srv/launchpadlibrarian.net/production-logs/2010-09-29$ ls -la
    <Chex> total 216
    <Chex> drwx--S--- 2 librarian librarian 4096 Sep 29 09:14 .
    <Chex> drwxr-sr-x 6 librarian librarian 196608 Sep 30 20:37 ..
    <Chex> -rw------- 1 librarian librarian 4348 Sep 29 09:02 32568.LIBRARIANA1
    <Chex> -rw------- 1 librarian librarian 4348 Sep 29 09:14 33248.LIBRARIANA2

- the user running the librarian appinstance seems to have the correct
  permission to create new files.

    <Chex> librarian@mizuho:/tmp$ umask
    <Chex> 0022
    <Chex> so that looks OK

- there's no cronjob changing the permissions afeterwards.

    <Chex> I wonder if there is a cron changing the perms.. checking
    <Chex> no, there is a cron that deletes old librarian.log.* but thats it

- content of one of the oopses: https://pastebin.canonical.com/37969/

- If we find a reliable way of triggereing the oops above we could add some
  cowboy code to the librarian to log the permission at the time an oops is
  being generated. that way we would know if it's the appserver generating the
  content with the wrong permission or if it's some process later on changing
  it.

- Any other ideas to help find out the cause?

the librarian possibly is changing its own umask. if so (and its
likely) the oops writing code may want to override the perms.

-Rob

Robert, is there a way to verify if the librarian is changin its own umask with the process already running?

Dunno :) I'd look at the twisted daemonisation code though. I suspect
it defaults to 077 (which actually we like for the librarian).

I suggest that the oops logging code, if it needs some specific
permissions, should set them.

e.g.
os.fchmod(outfile.fd, 0440)

-Rob

Today this happened again. I cowboyed the following to the production
oops-tools instance: https://pastebin.canonical.com/38351/

This means there won't be a fallback to look for the OOPS in the filesystem.
The view will try to find the OOPS in the database, and if it's not there,
it'll show a message saying the the OOPS can't be found.

The cowboyed code can be removed once the permissions in the source directories are fixed and errorlog.py is patched to set the permission in the error_dir and oops file to world readable.

As Robert described, the fix for this is to change ``os.makedirs(result)`` in lp/services/log/uniquefileallocator.py to specify a more friendly permission value, and then do the same for the OOPS file generation in errorlog.py. We need both.

Fixed in stable r11700 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11700>.