Lucid: system becomes unstable randomly, seems related with apparmor

Bug #581525 reported by Vreixo Formoso
54
This bug affects 8 people
Affects Status Importance Assigned to Milestone
apparmor (Ubuntu)
Fix Released
Undecided
John Johansen
Lucid
Fix Released
Undecided
Jamie Strandboge
Maverick
Fix Released
Undecided
John Johansen
linux (Ubuntu)
Fix Released
Undecided
John Johansen
Lucid
Fix Released
Undecided
John Johansen
Maverick
Fix Released
Undecided
John Johansen

Bug Description

SRU Justification (apparmor)

1. impact of the bug is medium for stable releases. There are two parts to this bug: the kernel side OOPSing when a the parser generates invalid tables, and the parser generating correct tables. The lucid kernel should receive the fix sometime in the future, but the userspace should also be fixed.

The kernel bug was a broken test in verifying the dfa next/check table size (so the userspace bug was not caught when it should have been). This means that it can at times reference beyond the dfa table (by at most 255 entries).

The userspace bug is that the next/check table is not correctly padded with 0 entries, so that it is impossible to reference beyond the end of the table when in the states that use the end of the table for their references.

2. This has been addressed during the maverick development cycle.

3. This is r1392 from the apparmor-2.5 branch. The commit mistakenly references a different bug (599450), but the text is: "Changes the table resizing so that there is always sufficient high entries in the table, preventing bounds violations from occurring."

4. TEST CASE: there are multiple possible test cases
4.1 Load a profile against a patched kernel (the maverick kernel can be used for this or a patched Lucid Kernel). The kernel will reject the profile with the following message in the logs
    AppArmor DFA next/check upper bounds error fixed, upgrade user space tools

4.2 The dfa verifier can be run against a profiles dfa in user space, but the checker is not part of the distro or easy to use atm as it requires manually extracting the tables from the profile. The full userspace profile verifier isn't available yet.

4.3 A profile can be compiled using the parser pre and post patching, and compared using a hex editor. The components of the profile that are changed are the size of the table and at the end of dfa table several 0 entries padding out the table. To do this choose a small profile eg. usr.sbin.tcpdump and run
./apparmor_parser -S <profile> >out.file
./apparmor_parser-patched -S <profile> >out.file2

The dfa table generated starts with the string aadfa\0 followed by a 4 byte (little endian blob size - this will differ), follow by the actual table header with various table size (some of these will change) and then the actual tables which almost fill the rest of the profile. Towards the end of the profile there should be extra 0's. And then the closing data of the profile which should not change. The data within the profile should not change beyond the couple of size entries and the 0 padding at the end.

5. The regression potential is considered low as the patch just pads out the table to make sure there are no bounds violations. The patch was pushed in maverick during its development cycle and showed no regressions. This is an important reliability fix for people who are affected (this has affected at least one Canonical server).

Hi,

Since last week I am experiencing a problem which seems related to apparmor. Kernel is crashing at aa_dfa_match_len+0xd9/0xf0, and a trace like the the following appears on my system logs:

May 17 01:57:04 mplaptop kernel: [ 6430.314093] PGD 1002063 PUD 0
May 17 01:57:04 mplaptop kernel: [ 6430.314101] CPU 1
May 17 01:57:04 mplaptop kernel: [ 6430.314103] Modules linked in: xts gf128mul binfmt_misc ppdev vboxnetadp vboxnetflt vboxdrv sha256_generic cryptd aes_x86_64 aes_generic dm_crypt joydev snd_hda_codec_realtek ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype xt_state dell_wmi arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm ip6table_filter ip6_tables snd_seq_dummy nf_nat_irc snd_seq_oss nf_conntrack_irc snd_seq_midi nf_nat_ftp snd_rawmidi nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 snd_seq_midi_event nf_conntrack_ftp snd_seq nf_conntrack iwlagn iptable_filter snd_timer snd_seq_device iwlcore ip_tables snd uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 x_tables mac80211 sdhci_pci dell_laptop dcdbas sdhci led_class nvidia(P) soundcore snd_page_alloc cfg80211 psmouse serio_raw uinput lp parport usbhid hid fbcon tileblit font bitblit ohci1394 softcursor ieee1394 r8169 mii ahci vga16fb vgastate intel_agp video output
May 17 01:57:04 mplaptop kernel: [ 6430.314159] Pid: 5065, comm: gnome-panel Tainted: P D 2.6.32-22-generic #33-Ubuntu Vostro1710
May 17 01:57:04 mplaptop kernel: [ 6430.314161] RIP: 0010:[<ffffffff8127dc49>] [<ffffffff8127dc49>] aa_dfa_match_len+0xd9/0xf0
May 17 01:57:04 mplaptop kernel: [ 6430.314170] RSP: 0018:ffff880116649d20 EFLAGS: 00010216
May 17 01:57:04 mplaptop kernel: [ 6430.314172] RAX: 0000000000000039 RBX: ffff880051285a8c RCX: 0000000000000039
May 17 01:57:04 mplaptop kernel: [ 6430.314174] RDX: ffff88011e65a4f1 RSI: 0000000053726599 RDI: ffff88011e65a4f1
May 17 01:57:04 mplaptop kernel: [ 6430.314176] RBP: ffff880116649d38 R08: 0000000000000000 R09: ffff88012bbfc40c
May 17 01:57:04 mplaptop kernel: [ 6430.314177] R10: ffff88009697606c R11: ffff88011e65a4ff R12: ffff88012bbfc20c
May 17 01:57:04 mplaptop kernel: [ 6430.314179] R13: ffff88011e65a4de R14: ffff88011e65a4de R15: 0000000000000000
May 17 01:57:04 mplaptop kernel: [ 6430.314181] FS: 00007f689ffe17e0(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
May 17 01:57:04 mplaptop kernel: [ 6430.314183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 17 01:57:04 mplaptop kernel: [ 6430.314185] CR2: ffff8801d2a48f3e CR3: 0000000111c91000 CR4: 00000000000026e0
May 17 01:57:04 mplaptop kernel: [ 6430.314187] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 17 01:57:04 mplaptop kernel: [ 6430.314189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 17 01:57:04 mplaptop kernel: [ 6430.314191] Process gnome-panel (pid: 5065, threadinfo ffff880116648000, task ffff8801360a8000)
May 17 01:57:04 mplaptop kernel: [ 6430.314194] ffff880096976ea0 0000000000000001 ffff88011e65a4de ffff880116649d68
May 17 01:57:04 mplaptop kernel: [ 6430.314197] <0> ffffffff8127dc9a ffff880116649db8 ffff88012e58b800 0000000000000000
May 17 01:57:04 mplaptop kernel: [ 6430.314200] <0> ffff88013fc022a8 ffff880116649db8 ffffffff8127e7d3 ffff88012e58b818
May 17 01:57:04 mplaptop kernel: [ 6430.314206] [<ffffffff8127dc9a>] aa_dfa_match+0x3a/0x50
May 17 01:57:04 mplaptop kernel: [ 6430.314209] [<ffffffff8127e7d3>] aa_find_attach+0x93/0xf0
May 17 01:57:04 mplaptop kernel: [ 6430.314211] [<ffffffff8127f80b>] apparmor_bprm_set_creds+0x36b/0x530
May 17 01:57:04 mplaptop kernel: [ 6430.314215] [<ffffffff8108998e>] ? up_write+0xe/0x10
May 17 01:57:04 mplaptop kernel: [ 6430.314219] [<ffffffff812507e3>] security_bprm_set_creds+0x13/0x20
May 17 01:57:04 mplaptop kernel: [ 6430.314223] [<ffffffff81149431>] prepare_binprm+0xb1/0x110
May 17 01:57:04 mplaptop kernel: [ 6430.314225] [<ffffffff8114a29c>] do_execve+0x1ac/0x300
May 17 01:57:04 mplaptop kernel: [ 6430.314229] [<ffffffff812bbdda>] ? strncpy_from_user+0x4a/0x90
May 17 01:57:04 mplaptop kernel: [ 6430.314233] [<ffffffff810115ba>] sys_execve+0x4a/0x80
May 17 01:57:04 mplaptop kernel: [ 6430.314236] [<ffffffff8101360a>] stub_execve+0x6a/0xc0
May 17 01:57:04 mplaptop kernel: [ 6430.314265] RSP <ffff880116649d20>
May 17 01:57:04 mplaptop kernel: [ 6430.314268] ---[ end trace 2b51de9f06402b92 ]---

Sometimes it does not seem to have visible effects, other times it renders the system unusable. When that happens, I often need to reboot several times, as the issue appears again on the next boot process. My system is an up-to-date lucid, installation mostly by default but with several dm_crypt partitions over LVM, and virtualbox-ose installed. I have also enabled the firefox apparmor profile and several other custom profiles.
Note that I am sometimes experienced another extrange apparmor behavior, as it attaches (randomly) a profile to a process that has not a profile defined (lets say, for example, it attaches the firefox profile to gedit). I experienced that 2 or 3 times, I will try to give you more information next time I see it, maybe it is related to this.
Finally, just note that this problem seems related to bug #529288.

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :
tags: added: kj-triage
Revision history for this message
Johannes Rudolph (johannes-rudolph) wrote :

I'm experiencing the same. Didn't do any config changes for apparmor.

After using the computer some time, suddenly no processes can be spawned any more, because all fail with a message similar to the one above.

Any hints how this could be debugged?

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

I'm suffering this issue continuously, and the stack trace always shows aa_dfa_match, so I think this is an apparmor bug.

affects: linux (Ubuntu) → apparmor (Ubuntu)
Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (5.2 KiB)

I'm pretty sure I triggered this bug today:

Jun 20 23:58:04 haig kernel: [18128.952910] type=1505 audit(1277103484.764:181): operation="profile_load" pid=19615 name="/home/sarnold/Local/io/build/_build/binaries/io"
Jun 20 23:59:06 haig kernel: [18190.703695] type=1505 audit(1277103546.554:182): operation="profile_load" pid=19630 name="/home/sarnold/Local/io/**"
Jun 20 23:59:12 haig kernel: [18196.757170] PGD 1002063 PUD 0
Jun 20 23:59:12 haig kernel: [18196.757176] CPU 1
Jun 20 23:59:12 haig kernel: [18196.757177] Modules linked in: usb_storage nls_cp437 cifs binfmt_misc ppdev lp parport kvm_intel kvm snd_hda_codec_atihdmi coretemp snd_hda_codec_realtek fbcon tileblit font bitblit softcursor snd_hda_intel it87 hwmon_vid i2c_i801 i2c_dev snd_seq_dummy snd_seq_oss vga16fb snd_seq_midi vgastate snd_rawmidi snd_hda_codec snd_seq_midi_event snd_seq snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer radeon ttm drm_kms_helper drm i2c_algo_bit snd soundcore snd_page_alloc serio_raw hid_apple usbhid hid ohci1394 ieee1394 pata_jmicron r8169 mii ahci
Jun 20 23:59:12 haig kernel: [18196.757203] Pid: 19634, comm: bash Not tainted 2.6.32-22-generic #36-Ubuntu EX58-UD5
Jun 20 23:59:12 haig kernel: [18196.757205] RIP: 0010:[<ffffffff8127dc59>] [<ffffffff8127dc59>] aa_dfa_match_len+0xd9/0xf0
Jun 20 23:59:12 haig kernel: [18196.757207] RSP: 0000:ffff8801551f3d20 EFLAGS: 00010202
Jun 20 23:59:12 haig kernel: [18196.757209] RAX: 0000000000006573 RBX: ffff8801491ba28c RCX: 0000000000006573
Jun 20 23:59:12 haig kernel: [18196.757210] RDX: ffff8801a66d61fe RSI: 000000004841b77f RDI: ffff8801a66d61fe
Jun 20 23:59:12 haig kernel: [18196.757212] RBP: ffff8801551f3d38 R08: 0000000000000000 R09: ffff88014fc2810c
Jun 20 23:59:12 haig kernel: [18196.757213] R10: ffff8801a6901f0c R11: ffff8801a66d61ff R12: ffff88014fc28a0c
Jun 20 23:59:12 haig kernel: [18196.757215] R13: ffff8801a66d61f2 R14: ffff8801a66d61f2 R15: 0000000000000000
Jun 20 23:59:12 haig kernel: [18196.757216] FS: 00007f4b43417700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
Jun 20 23:59:12 haig kernel: [18196.757218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 20 23:59:12 haig kernel: [18196.757220] CR2: ffff8801e045f00a CR3: 0000000155212000 CR4: 00000000000026e0
Jun 20 23:59:12 haig kernel: [18196.757221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 20 23:59:12 haig kernel: [18196.757223] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 20 23:59:12 haig kernel: [18196.757225] Process bash (pid: 19634, threadinfo ffff8801551f2000, task ffff8801a91b8000)
Jun 20 23:59:12 haig kernel: [18196.757227] ffff880165507660 0000000000000001 ffff8801a66d61f2 ffff8801551f3d68
Jun 20 23:59:12 haig kernel: [18196.757229] <0> ffffffff8127dcaa ffff8801551f3db8 ffff88014fdebc00 0000000000000000
Jun 20 23:59:12 haig kernel: [18196.757232] <0> ffff8801afc18228 ffff8801551f3db8 ffffffff8127e7e3 ffff8801551f3db8
Jun 20 23:59:12 haig kernel: [18196.757237] [<ffffffff8127dcaa>] aa_dfa_match+0x3a/0x50
Jun 20 23:59:12 haig kernel: [18196.757239] [<ffffffff8127e7e3>] aa_find_attach+0x93/0xf0
Jun 20 23:59:12 haig kernel: [18196.757241] [<ff...

Read more...

Revision history for this message
Seth Arnold (seth-arnold) wrote :

I triggered this bug again while refining my /etc/init.d/origami profile. (I am breaking apart the giant profile into one profile with several child profiles.)

This time, I used /etc/init.d/apparmor restart to recompile and load policies. I successfully changed policy several times before the stacktrace.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

This is the profile I was working on the second time my kernel dumped stack. (It's not done yet.)

Linux haig 2.6.32-22-generic #36-Ubuntu SMP Thu Jun 3 19:31:57 UTC 2010 x86_64 GNU/Linux

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Vreixo, Johannes, I just discovered a profile for:

/home/sarnold/Local/Io/**

was being attached to /usr/bin/grotty.

I think there's something broken with regexps in profile names. (But my firefox profile appears to be working fine, and has for .. six weeks? eight weeks? with the profile name /usr/lib/firefox-3.5.*/firefox.)

I also had trouble with my /etc/init.d/origami profile when I significantly tightened it up, using child profiles. Perhaps child profiles also give AppArmor trouble. (My one other profile with child profiles only executes them once, at boot, so they weren't getting nearly as much use as my origami profile, when I was actively developing it.)

I hope this can help you guys find problem profiles and remove them. My system appears to stable again, now that I've removed the offending profiles. (It was very easy for me to stop running the confined programs, too, since it's just Folding at Home, far from a critical service for me, and the Io Language interpreter, something I was just playing with anyway.)

Revision history for this message
John Johansen (jjohansen) wrote :

Seth,

this is really distirbing, this is the first I have heard of problems with children profiles, what problems where you having with child profiles? Where they problems with enforcement, or problems with the tools and development?

Also did your problems with the /home/sarnold/Local/Io/** profile occur when just enforcing the profile? ie. did you see problems if you loaded the profile and didn't replace it/do development on it?

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

> I just discovered a profile for:
> /home/sarnold/Local/Io/**
> was being attached to /usr/bin/grotty.

I've also experienced the same issue. Sometimes profiles are attached to completely different processes.
It seems some kind of bug on profile loading.

I doubt there is a problem with profile syntax itself, as they work most times. However, sometimes they just throw the error I've pasted above, and system becomes unusable.
I think it is something related with profile loading, as it happens either at boot time or when I am working on profile definition. Problems at boot time happen around 20-30% of time, which is imho a really high frequency.
Maybe it is some kind of race condition, as it does not happen always.

I work regularly on 3 PCs, all of them with Lucid and apparmor enabled with default configuration. No problems at all. However, in my laptop, also an Ubuntu Lucid, but with custom profiles, problems happen frequently, as reported above.
This bug is making me crazy!

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

> I think there's something broken with regexps in profile names.

mmm, now you are saying this... It might be a problem with profiles that end with "*". I would swear my problems appeared after creating the /usr/bin/totem* profile... but I don't remember. I will try to test without it.

Revision history for this message
Seth Arnold (seth-arnold) wrote : Re: [Bug 581525] Re: Lucid: system becomes unstable randomly, seems related with apparmor
Download full text (9.4 KiB)

On Wed, Jun 23, 2010 at 6:39 AM, John Johansen
<email address hidden> wrote:
> this is really distirbing, this is the first I have heard of problems
> with children profiles, what problems where you having with child
> profiles?  Where they problems with enforcement, or problems with the
> tools and development?

Most of the profiles I write I do by hand. I've had enough cases of
the tools missing entries, or re-suggesting entries that I had done in
a previous pass, and all the new children null-complain-profile
replacements, that for the most part doing things by hand is faster.
(But I'm not sure I've tried them under 10.04.)

When I used child profiles for /etc/init.d/ushare, they appeared to
work perfectly:

$ cat /etc/apparmor.d/etc.init.d.ushare
# Last Modified: Wed Mar 3 22:29:57 2010
#include <tunables/global>

/etc/init.d/ushare {
  #include <abstractions/base>

  capability sys_tty_config,

  owner /bin/dash ix,
  owner /bin/readlink rix,

  owner /etc/init.d/ushare rix,
  owner /etc/default/rcS r,
  owner /etc/lsb-base-logging.sh r,
  owner /etc/ushare.conf r,

  owner /sbin/start-stop-daemon cx,
  owner /sbin/usplash_write px,
  owner /usr/bin/expr cx,
  owner /usr/bin/tput px,
  owner /bin/touch cx,

  owner /var/run/ushare.pid r,

  profile /sbin/start-stop-daemon {
    #include <abstractions/base>

    capability sys_ptrace,

    owner /dev/tty rw,
    owner /var/run/ushare.pid rw,
    owner /usr/bin/ushare px,
  }

  profile /usr/bin/expr {
    #include <abstractions/base>
  }

  profile /bin/touch {
    #include <abstractions/base>
    owner /var/run/ushare.pid w,
  }

  profile /usr/bin/tput {
    #include <abstractions/base>
    capability sys_tty_config,
  }
}

However, my system was _very_ unstable with my /etc/init.d/origami
profile loaded:

$ cat etc.init.d.origami
# Last Modified: Wed Mar 3 22:04:49 2010
#include <tunables/global>

/etc/init.d/origami {
  #include <abstractions/base>
  #include <abstractions/bash>

  capability dac_override,

  network inet dgram,
  network inet stream,

  /bin/pidof cx,
  /bin/ps cx,
  /bin/su cx,
  /sbin/killall5 cx,
  /usr/bin/taskset cx,

  /bin/bash ix,
  /bin/dash ix,
  /bin/grep mrix,
  /bin/sleep mrix,
  /bin/which mrix,
  /dev/tty rw,
  /etc/hosts r,
  /etc/init.d/origami r,
  /etc/nsswitch.conf r,
  /etc/resolv.conf r,
  /proc/sys/kernel/pid_max r,
  /proc/tty/drivers r,
  /proc/uptime r,
  /proc/version r,
  /tmp/fah/ rw,
  /tmp/fah/** rw,
  /tmp/fah/f* k,
  /usr/bin/cut mrix,
  /usr/bin/expr mrix,
  /usr/bin/getent mrix,
  /usr/bin/wc mrix,
  /var/lib/origami/** r,
  owner /var/lib/origami/foldingathome/CPU*/* r,
  owner /var/lib/origami/foldingathome/CPU*/Core_78.exe mwix,
  owner /var/lib/origami/foldingathome/CPU*/Core_78.fah wk,
  owner /var/lib/origami/foldingathome/CPU*/Core_b4.fah wk,
  owner /var/lib/origami/foldingathome/CPU*/FAHlog-Prev.txt wk,
  owner /var/lib/origami/foldingathome/CPU*/FAHlog.txt w,
  owner /var/lib/origami/foldingathome/CPU*/FaH mix,
  owner /var/lib/origami/foldingathome/CPU*/FahCore_78.exe mwkix,
  owner /var/lib/origami/foldingathome/CPU*/FahCore_b4.exe mwkix,
  owner /var/lib/origami/foldingathome/CPU*/MyFolding.html w,
  owner /va...

Read more...

Revision history for this message
John Johansen (jjohansen) wrote :

Okay, thank you for the profiles, from what I have been able to gather replacement maybe involved in triggering this. I will set up a fresh install and try replicating it again.

Vrexio,
if you could attach your custom profiles that would be helpful.

Seth
The comm output is coming back in Maverick

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

John, my profiles were already attached, please refer to first comment on the thread.

Cheers,
Vreixo

Revision history for this message
John Johansen (jjohansen) wrote :

Alright I think I have finally tracked this one down, and we should have some packages for testing soon. This is a user space fix to make the table actually work but will also get a kernel side test to ensure the table is properly bounded at load time.

What is happening is the created dfa is not being properly padded on the one side of the table, so that indexing can reference memory outside the bounds of the table. This is pretty much limited to the small profile with regex dfas.

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

John, great!!! Thank you so much for taking care of this, I am awaiting for your test packages.

Revision history for this message
John Johansen (jjohansen) wrote :

Alright I have finally got some test kernels up
Lucid
http://kernel.ubuntu.com/~jj/linux-image-2.6.32-24-generic_2.6.32-24.38~jj_amd64.deb
http://kernel.ubuntu.com/~jj/linux-image-2.6.32-24-generic_2.6.32-24.38~jj_i386.deb

Maverick
http://kernel.ubuntu.com/~jj/linux-image-2.6.35-7-generic_2.6.35-7.12~jj_amd64.deb
http://kernel.ubuntu.com/~jj/linux-image-2.6.35-7-generic_2.6.35-7.12~jj_i386.deb

Verify that you get a warning message about fixing next/check bounds error, and then test that the system is stable for you. There will be followup builds of the tools that should remove the next/check bounds error warning and also make the shipped distro kernels stable (this bug has 2 parts kernel had broken bounds check, and user space wasn't setting up the bounds correctly.

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

Thanks John, it seems to work:

...
Jul 10 02:36:38 metalpain-laptop kernel: [ 69.287814] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
...
Jul 10 02:36:38 metalpain-laptop kernel: [ 69.292180] type=1505 audit(1278722198.248:54): operation="profile_replace" pid=1416 name="/usr/share/gdm/guest-session/Xsession"
Jul 10 02:36:38 metalpain-laptop kernel: [ 70.019802] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
Jul 10 02:36:38 metalpain-laptop kernel: [ 70.024396] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
...

but the system remains stable (note I have not tested it too much yet).
I hope this fix gets merged on official kernel asap. Thank you very much.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

John, thanks for this updated kernel; I received 39 or so "AppArmor DFA next/check upper bounds error fixed, upgrade user space tools" per /etc/init.d/apparmor reload command.

I was able to re-install my /home/sarnold/Local/io/** profile with no obvious problems: both the Io interpreter appears to run fine and grotty was _not_ mistakenly attached. :) So great success there.

I also put my /etc/init.d/origami profile back in place and continued development of the profile. (This made my machine very unstable before; no system stability issues yet. Good work.)

I'm getting some very funny errors though:

5013 execve("/bin/bash", ["sh", "-", "/bin/bash", "-c", "/bin/bash -c \"cd /var/lib/origam"...], [/* 12 vars */]) = -1 ENOENT (No such file or directory)

This error is _not_ accompanied by any AppArmor log messages. It is a silent fail. Removing the AppArmor profile allows origami to start as normal.

I'll attach my /etc/init.d/origami profile, maybe it'll be obvious to you.

Revision history for this message
John Johansen (jjohansen) wrote :

Seth, can you turn on AppArmor debugging

as root
> echo 1 > /sys/module/apparmor/parameters/debug

And see what apparmor outputs to dmesg when you try that

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Sorry John, even after turning on the debug parameter, I don't get any messages from AppArmor when trying to start origami.

Revision history for this message
Gabriel de Perthuis (g2p) wrote :

Thank you for the fixed kernel. I don't have any special profiles except those that come with standard packages.
Random, non-contained processes were being killed in aa_dfa_match_len.

dlocate /etc/apparmor |sed 's#:.*##' |uniq
evince
tcpdump
apparmor-profiles
cups
dhcp3-client
gdm-guest-session
apparmor-utils
apparmor
libvirt-bin
firefox
ntp

With the new kernel, on apparmor reload, I get the "next/check upper bounds error fixed" message 37 times, and the crashes don't reappear.

Changed in apparmor (Ubuntu):
status: New → Fix Committed
Revision history for this message
John Johansen (jjohansen) wrote :

Gabriel,

yes the error could actually occur if any profile contained regexs (firefox), however certain profile patterns would trigger the bug more than others.

Changed in linux-meta (Ubuntu):
status: New → In Progress
assignee: nobody → John Johansen (jjohansen)
milestone: none → lucid-updates
Changed in apparmor (Ubuntu):
status: Fix Committed → Fix Released
Kees Cook (kees)
affects: linux-meta (Ubuntu) → linux (Ubuntu)
Changed in linux (Ubuntu Maverick):
milestone: lucid-updates → ubuntu-10.10
Changed in apparmor (Ubuntu Lucid):
status: New → In Progress
milestone: none → lucid-updates
Changed in linux (Ubuntu Lucid):
milestone: none → lucid-updates
status: New → In Progress
assignee: nobody → John Johansen (jjohansen)
Changed in linux (Ubuntu Maverick):
status: In Progress → Fix Released
Changed in apparmor (Ubuntu Maverick):
assignee: nobody → John Johansen (jjohansen)
Changed in apparmor (Ubuntu Lucid):
assignee: nobody → Kees Cook (kees)
Changed in apparmor (Ubuntu Maverick):
milestone: none → ubuntu-10.10
Changed in apparmor (Ubuntu Lucid):
assignee: Kees Cook (kees) → Jamie Strandboge (jdstrand)
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

SRU Justification (apparmor)

1. impact of the bug is medium for stable releases. There are two parts to this bug: the kernel side OOPSing when a the parser generates invalid tables, and the parser generating correct tables. The lucid kernel should receive the fix sometime in the future, but the userspace should also be fixed.

2. This has been addressed during the maverick development cycle.

3. This is r1392 from the apparmor-2.5 branch. The commit mistakenly references a different bug (599450), but the text is: "Changes the table resizing so that there is always sufficient high entries in the table, preventing bounds violations from occurring."

4. TEST CASE: jjohansen will document the test case in a separate comment

5. The regression potential is considered low as the patch just pads out the table to make sure there are no bounds violations. The patch was pushed in maverick during its development cycle and showed no regressions. This is an important reliability fix for people who are affected (this has affected at least one Canonical server).

description: updated
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted apparmor into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in apparmor (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Using John's test case 4.1, if in install the linux-image-generic-lts-backport-maverick without upgrading AppArmor, I can reproduce the bug easily by just rebooting. The dhclient3, guest-session and evince profiles all cause errors like this in dmesg:
[ 5.020139] type=1400 audit(1292354133.218:10): apparmor="STATUS" operation="profile_load" name="/usr/bin/evince-previewer" pid=586 comm="apparmor_parser"
[ 5.045714] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
[ 5.066967] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
[ 5.067433] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools
[ 5.076746] AppArmor DFA next/check upper bounds error fixed, upgrade user space tools

Without rebooting, I can continue to generate them with:
$ sudo apparmor_parser -r -T -W /etc/apparmor.d/usr.bin.evince

After installing 2.5.1-0ubuntu0.10.04.1 from lucid-proposed, I no longer see the error messages on boot or with apparmor_parser.

Martin Pitt (pitti)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (10.1 KiB)

This bug was fixed in the package apparmor - 2.5.1-0ubuntu0.10.04.1

---------------
apparmor (2.5.1-0ubuntu0.10.04.1) lucid-proposed; urgency=low

  * Backport 2.5.1-0ubuntu0.10.10.1 from maverick for userspace tools to work
    with newer kernels (LP: #660077)
    NOTE: user-tmp now uses 'owner' match, so non-default profiles will have
    to be adjusted when 2 separately confined applications that both use the
    user-tmp abstraction depend on being able to cooperatively share files
    with each other in /tmp or /var/tmp.
  * remove the following patches (features not appropriate for SRU):
    - 0002-add-chromium-browser.patch
    - 0003-local-includes.patch
    - 0004-ubuntu-abstractions-updates.patch
  * debian/rules (this makes it the same as what was shipped in 10.04 LTS
    release):
    - don't ship aa-update-browser and its man page (requires
      0004-ubuntu-abstractions-updates.patch)
    - don't ship apparmor.d/local/ (requires 0003-local-includes.patch)
    - don't use dh_apparmor (not in Ubuntu 10.04 LTS)
    - don't ship chromium profile
  * remove debian/profiles/chromium-browser
  * remove debian/aa-update-browser*
  * debian/apparmor-profiles.postinst: revert to that in lucid release
    (requires dh_apparmor and 0002-add-chromium-browser.patch)
  * remove debian/apparmor-profiles.postrm: doesn't make sense without
    0002-add-chromium-browser.patch
  * debian/control:
    - revert Build-Depends on debhelper (>= 5)
    - revert Standards-Version to 3.8.4
    - revert Vcs-Bzr
    - use Conflicts/Replaces version that was in Ubuntu 10.04 LTS
  * debian/patches/0011-lucid-compat-dbus.patch: move /var/lib/dbus/machine-id
    back into dbus, since profiles on 10.04 LTS expect it there
  * debian/patches/0012-lucid-compat-kde.patch: add kde4-config to kde
    abstraction, since the firefox profile on Ubuntu 10.04 LTS expects it to
    be there

apparmor (2.5.1-0ubuntu0.10.10.2) maverick-proposed; urgency=low

  * New upstream release (LP: #660077)
    - The following patches were refreshed:
      + 0001-fix-release.patch
      + 0003-local-includes.patch
      + 0004-ubuntu-abstractions-updates.patch
      + 0008-lp648900.patch: renamed as 0005-lp648900.patch
    - The following patches were dropped (included upstream):
      + 0005-lp601583.patch
      + 0006-network-interface-enumeration.patch
      + 0007-gnome-updates.patch
  * debian/patches/0006-testsuite-fixes.patch: testsuite fixes from head
    of 2.5 branch. These are needed for QRT and SRU testing (LP: #652211)
  * debian/patches/0007-honor-cflags.patch: have the parser makefile honor
    CFLAGS environment variable. Brings back missing symbols for the retracer
  * debian/patches/0008-lp652674.patch: fix warnings for messages without
    denied or requested masks (LP: #652674)
  * debian/apparmor.init: fix path to aa-status (LP: #654841)
  * debian/apport/source_apparmor.py: apport hook should use
    root_command_hook() for running apparmor_status (LP: #655529)
  * debian/apport/source_apparmor.py: use ProcKernelCmdline and don't clobber
    cmdline details (LP: #657091)
  * debian/{rules,control}: move apache2 abstractions into the base package
    so we can put ...

Changed in apparmor (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted linux into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in linux (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Steve Conklin (sconklin) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed' to 'verification-done'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Brad Figg (brad-figg)
tags: added: verification-done
removed: verification-needed
Revision history for this message
John Johansen (jjohansen) wrote :

I have verified that the proposed kernel catches bad policy loads, causing them to fail and outputs the expected

   AppArmor DFA next/check upper bounds error

message in dmesg.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (10.1 KiB)

This bug was fixed in the package linux - 2.6.32-28.55

---------------
linux (2.6.32-28.55) lucid-proposed; urgency=low

  * Another version bump because of abi check failure
  * Tracking Bug
    - LP: #699885

linux (2.6.32-28.54) lucid-proposed; urgency=low

  * Another version bump because of upload failure

linux (2.6.32-28.53) lucid-proposed; urgency=low

  * Another version bump because of upload failure

linux (2.6.32-28.52) lucid-proposed; urgency=low

  [ Steve Conklin ]

  * (removed old tracking bug link)

linux (2.6.32-28.51) lucid-proposed; urgency=low

  [ Steve Conklin ]

  * bumped version due to build fail

linux (2.6.32-28.50) lucid-proposed; urgency=low

  [ Tim Gardner ]

  * SAUCE: Change nodelayacct boot parameter polarity.
    - LP: #493156
  * [Config] CONFIG_TASK_DELAY_ACCT=y
    - LP: #493156

  [ Upstream Kernel Changes ]

  * ipc: initialize structure memory to zero for compat functions
  * tcp: Increase TCP_MAXSEG socket option minimum.
    - CVE-2010-4165
  * perf_events: Fix perf_counter_mmap() hook in mprotect()
    - CVE-2010-4169
  * af_unix: limit unix_tot_inflight
    - CVE-2010-4249
  * AppArmor: fix the upper bound check for the next/check table
    - LP: #581525
  * NFS: Fix panic after nfs_umount()
    - LP: #683938
  * block: Ensure physical block size is unsigned int
    - LP: #688669
  * block: limit vec count in bio_kmalloc() and bio_alloc_map_data()
    - LP: #688669
  * block: take care not to overflow when calculating total iov length
    - LP: #688669
  * block: check for proper length of iov entries in blk_rq_map_user_iov()
    - LP: #688669
  * jme: Fix PHY power-off error
    - LP: #688669
  * irda: Fix parameter extraction stack overflow
    - LP: #688669
  * irda: Fix heap memory corruption in iriap.c
    - LP: #688669
  * i2c-pca-platform: Change device name of request_irq
    - LP: #688669
  * microblaze: Fix build with make 3.82
    - LP: #688669
  * Staging: asus_oled: fix up some sysfs attribute permissions
    - LP: #688669
  * Staging: asus_oled: fix up my fixup for some sysfs attribute
    permissions
    - LP: #688669
  * Staging: line6: fix up some sysfs attribute permissions
    - LP: #688669
  * hpet: fix unwanted interrupt due to stale irq status bit
    - LP: #688669
  * hpet: unmap unused I/O space
    - LP: #688669
  * olpc_battery: Fix endian neutral breakage for s16 values
    - LP: #688669
  * percpu: fix list_head init bug in __percpu_counter_init()
    - LP: #688669
  * um: remove PAGE_SIZE alignment in linker script causing kernel
    segfault.
    - LP: #688669
  * um: fix global timer issue when using CONFIG_NO_HZ
    - LP: #688669
  * numa: fix slab_node(MPOL_BIND)
    - LP: #688669
  * hwmon: (lm85) Fix ADT7468 frequency table
    - LP: #688669
  * mm: fix return value of scan_lru_pages in memory unplug
    - LP: #688669
  * mm: fix is_mem_section_removable() page_order BUG_ON check
    - LP: #688669
  * ssb: b43-pci-bridge: Add new vendor for BCM4318
    - LP: #688669
  * sgi-xpc: XPC fails to discover partitions with all nasids above 128
    - LP: #688669
  * xen: ensure that all event channels start off bound to VCPU 0
    - LP: #688669
  * xen: don't bother to...

Changed in linux (Ubuntu Lucid):
status: Fix Committed → Fix Released
tags: added: testcase
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.