perf trace sleep 1 is throwing segmentation fault

Bug #1410673 reported by bugproxy
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Chris J Arges
Vivid
Fix Released
Medium
Chris J Arges

Bug Description

[Impact]
Running 'perf trace sleep 1' throws a segmentation fault.

[Test Case]
Install a Ubuntu 15.04 guest VM on Power KVM host on a P8 machine.
Then execute the below command on the same:

root@ubuntu:~# perf trace sleep 1
Segmentation fault (core dumped)

[Fix]
commit 7951722da2963cc1f1a7831a37aa2311ac927056

$ git describe 7951722da2963cc1f1a7831a37aa2311ac927056
v4.1-rc5-310-g7951722

--

== Comment: #0 - PAVAMAN SUBRAMANIYAM <email address hidden> - 2015-01-12 02:21:16 ==
---Problem Description---
perf trace sleep 1 is throwing segmentation fault

Contact Information = <email address hidden>

---uname output---
Linux ubuntu 3.18.0-8-generic #9-Ubuntu SMP Mon Jan 5 22:52:15 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = P8

---Debugger---
A debugger is not configured

---Steps to Reproduce---
Install a Ubuntu 15.04 guest VM on Power KVM host on a P8 machine.
Then execute the below command on the same:

root@ubuntu:~# perf trace sleep 1
Segmentation fault (core dumped)
root@ubuntu:~# echo $?
139

root@ubuntu:~# perf trace -v sleep 1
mmap size 67174400B
Problems reading syscall 3 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 11 information
Problems reading syscall 0 information
Problems reading syscall 45 information
Problems reading syscall 45 information
Problems reading syscall 33 information
Problems reading syscall 33 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 33 information
Problems reading syscall 33 information
Problems reading syscall 5 information
Problems reading syscall 5 information
Problems reading syscall 108 information
Problems reading syscall 108 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 6 information
Problems reading syscall 6 information
Problems reading syscall 33 information
Problems reading syscall 33 information
Problems reading syscall 5 information
Problems reading syscall 5 information
Problems reading syscall 3 information
Problems reading syscall 3 information
Problems reading syscall 108 information
Problems reading syscall 108 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 6 information
Problems reading syscall 6 information
Problems reading syscall 91 information
Problems reading syscall 91 information
Problems reading syscall 45 information
Problems reading syscall 45 information
Problems reading syscall 45 information
Problems reading syscall 45 information
Problems reading syscall 5 information
Problems reading syscall 5 information
Problems reading syscall 108 information
Problems reading syscall 108 information
Problems reading syscall 90 information
Problems reading syscall 90 information
Problems reading syscall 6 information
Problems reading syscall 6 information
Problems reading syscall 162 information
Problems reading syscall 162 information
Segmentation fault (core dumped)

Stack trace output:
 no

Oops output:
 no

Userspace tool common name: /usr/bin/perf

The userspace tool has the following bit modes: 64-bit

System Dump Info:
  The system is not configured to capture a system dump.

Userspace rpm: linux-tools-common

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach sysctl -a output output to the bug.
-Attach ltrace and strace of userspace application.

== Comment: #3 - SANDHYA VENUGOPALA <email address hidden> - 2015-01-13 06:00:18 ==
Segmentation Fault happens intermittently. Execute "perf trace sleep 1" multiple times to ensure there is no segmentation fault on the version you are testing.

This failure is seen on the upstream version(3.19) also. GDB output -

root@ubuntu:~/linux/tools/perf# ./perf trace sleep 1
Segmentation fault (core dumped)
root@ubuntu:~/linux/tools/perf# gdb ./perf core
GNU gdb (Ubuntu 7.8.1-1ubuntu2) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perf...done.
[New LWP 24493]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `./perf trace sleep 1 '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 perf_mmap__read_head (mm=0x3fff95730058) at util/evlist.h:193
193 int head = ACCESS_ONCE(pc->data_head);
(gdb) bt
#0 perf_mmap__read_head (mm=0x3fff95730058) at util/evlist.h:193
#1 perf_evlist__mmap_read (evlist=0x1000f851e70, idx=<optimized out>) at util/evlist.c:638
#2 0x000000001004318c in trace__run (argv=<optimized out>, argc=<optimized out>, trace=0x3fffdca33668) at builtin-trace.c:2123
#3 cmd_trace (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-trace.c:2601
#4 0x0000000010009328 in run_builtin (p=0x10178210 <commands+504>, argc=3, argv=0x3fffdca36800) at perf.c:341
#5 0x00000000100085c8 in handle_internal_command (argv=0x3fffdca36800, argc=3) at perf.c:400
#6 run_argv (argv=0x3fffdca36220, argcp=0x3fffdca3622c) at perf.c:444
#7 main (argc=3, argv=0x3fffdca36800) at perf.c:559

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-120252 severity-high targetmilestone-inin1504
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1410673/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → linux (Ubuntu)
tags: added: vivid
tags: added: kernel-da-key
Chris J Arges (arges)
Changed in linux (Ubuntu):
assignee: nobody → Chris J Arges (arges)
importance: Undecided → Medium
assignee: Chris J Arges (arges) → nobody
Revision history for this message
bugproxy (bugproxy) wrote : strace perf trace sleep 1 (ERROR)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:08 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace perf trace sleep 1 (OK)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:09 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace -f perf trace sleep 1 (ERROR)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:10 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace -f perf trace sleep 1 (OK)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:10 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-01-29 20:31 EDT-------
It appears that all fds get POLLHUP and hence get filtered out
by the perf_evlist__filter_pollfd(evlist, POLLERR|POLLHUP), check.
This happens most of the time but not always. When this happens
the mmap is freed (in perf_evlist__munmap_filtered()), but then we
do the "goto again" which tries to read from the freed mmap.

With the following patch, I was able to run the 'perf trace sleep 1'
several hundred times, but this defeats the purpose of the "draining"
check.

diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index fb12645..ac25e16 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2173,8 +2173,10 @@ next_event:
int timeout = done ? 100 : -1;
if (!draining && perf_evlist__poll(evlist, timeout) > 0) {
- if (perf_evlist__filter_pollfd(evlist, POLLERR | POLLHUP) == 0)
+ if (perf_evlist__filter_pollfd(evlist, POLLERR | POLLHUP) == 0) {
draining = true;
+ goto out_disable;
+ }
goto again;
}

Li, can you look into why they added the 'draining' check (IOW, why
do they go back and read even if perf_evlist___filter_pollfd() returns
0 ?

It maybe interesting to see if the mainline code, which at a quick
glance seems similar to this code in Ubuntu Vivid, behaves the same
way on this system.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-01-31 01:17 EDT-------
This is being discussed on LKML:

https://lkml.org/lkml/2015/1/29/1021

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-02-02 03:15 EDT-------
(In reply to comment #25)
> It appears that all fds get POLLHUP and hence get filtered out
> by the perf_evlist__filter_pollfd(evlist, POLLERR|POLLHUP), check.
> This happens most of the time but not always. When this happens
> the mmap is freed (in perf_evlist__munmap_filtered()), but then we
> do the "goto again" which tries to read from the freed mmap.
>
> With the following patch, I was able to run the 'perf trace sleep 1'
> several hundred times, but this defeats the purpose of the "draining"
> check.
>
>
> diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> index fb12645..ac25e16 100644
> --- a/tools/perf/builtin-trace.c
> +++ b/tools/perf/builtin-trace.c
> @@ -2173,8 +2173,10 @@ next_event:
> int timeout = done ? 100 : -1;
>
> if (!draining && perf_evlist__poll(evlist, timeout) > 0) {
> - if (perf_evlist__filter_pollfd(evlist, POLLERR |
> POLLHUP) == 0)
> + if (perf_evlist__filter_pollfd(evlist, POLLERR |
> POLLHUP) == 0) {
> draining = true;
> + goto out_disable;
> + }
>
> goto again;
> }
>
> Li, can you look into why they added the 'draining' check (IOW, why
> do they go back and read even if perf_evlist___filter_pollfd() returns
> 0 ?

Sure, still looking at this and your discussion with community. :)

>
> It maybe interesting to see if the mainline code, which at a quick
> glance seems similar to this code in Ubuntu Vivid, behaves the same
> way on this system.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-05-12 09:37 EDT-------
I carefully looked at kernel code and the patch
> + if (perf_evlist__filter_pollfd(evlist, POLLERR |
> POLLHUP) == 0) {
> draining = true;
> + goto out_disable;
> + }
doesn't seem to be there , the "goto out_disable" line doesn't exist.

Please check that a patch is needed on top of current kernel code and then it has been submitted upstream.

Revision history for this message
bugproxy (bugproxy) wrote : strace perf trace sleep 1 (ERROR)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:08 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace perf trace sleep 1 (OK)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:09 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace -f perf trace sleep 1 (ERROR)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:10 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : strace -f perf trace sleep 1 (OK)

------- Comment (attachment only) From <email address hidden> 2015-01-26 16:10 EDT-------

Chris J Arges (arges)
Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-06-12 06:12 EDT-------
Have not heard back from maintainer, so did some investigation
myself. Posted another patch to the community today:

https://lkml.org/lkml/2015/6/12/64

which reads:

[PATCH] perf, tools: Fix crash with perf trace

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-06-18 16:10 EDT-------
This fix has been merged into linux-tip:

http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=7951722da2963cc1f1a7831a37aa2311ac927056

Chris J Arges (arges)
Changed in linux (Ubuntu):
assignee: nobody → Chris J Arges (arges)
status: Confirmed → In Progress
Changed in linux (Ubuntu Vivid):
assignee: nobody → Chris J Arges (arges)
importance: Undecided → Medium
status: New → In Progress
description: updated
Brad Figg (brad-figg)
Changed in linux (Ubuntu Vivid):
status: In Progress → Fix Committed
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-vivid
Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (4.3 KiB)

------- Comment From <email address hidden> 2015-07-10 16:12 EDT-------
I have installed Ubuntu 15.04 ISO as a PKVM guest on P8 Hardware.
Then upgraded the guest VM to the latest kernel and rebooted the guest VM.

root@ubuntu:~# apt-get update
root@ubuntu:~# apt-get dist-upgrade

root@ubuntu:~# uname -a
Linux ubuntu 3.19.0-23-generic #24-Ubuntu SMP Tue Jul 7 20:49:14 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
root@ubuntu:~# cat /etc/issue
Ubuntu 15.04 \n \l

root@ubuntu:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="15.04 (Vivid Vervet)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 15.04"
VERSION_ID="15.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

The test scenario is executed again and it is successful without any issues.

root@ubuntu:~# perf trace sleep 1
0.221 ( 0.221 ms): read(fd: 5<pipe:[12702]>, buf: 0x3fffffc1caaf, count: 1 ) = 1
0.983 ( 0.157 ms): execve(filename: 0x3fffffc1c974, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
1.295 ( 0.011 ms): execve(filename: 0x3fffffc1c964, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
1.462 ( 0.011 ms): execve(filename: 0x3fffffc1c97b, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
1.629 ( 0.011 ms): execve(filename: 0x3fffffc1c97c, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
1.803 ( 0.011 ms): execve(filename: 0x3fffffc1c981, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
1.986 ( 0.011 ms): execve(filename: 0x3fffffc1c982, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
2.176 ( 0.010 ms): execve(filename: 0x3fffffc1c985, argv: 0x3fffffc1fe40, envp: 0x1002a6209f0) = -2
2.944 ( 0.114 ms): brk( ) = 0x100287c0000
3.252 ( 0.131 ms): access(filename: 0x3fffb3d3faa8 ) = -1 ENOENT No such file or directory
3.427 ( 0.010 ms): access(filename: 0x3fffb3d3ea28, mode: R ) = -1 ENOENT No such file or directory
3.732 ( 0.147 ms): open(filename: 0x3fffb3d40a30, flags: CLOEXEC ) = 3
3.892 ( 0.002 ms): fstat(fd: 3</etc/ld.so.cache>, statbuf: 0x3fffd4590320 ) = 0
4.188 ( 0.298 ms): ... [continued]: mmap()) = 0x3fffb3ce0000
4.346 ( 0.002 ms): close(fd: 3</etc/ld.so.cache> ) = 0
4.606 ( 0.009 ms): access(filename: 0x3fffb3d3faa8 ) = -1 ENOENT No such file or directory
4.774 ( 0.011 ms): open(filename: 0x3fffb3d641e8, flags: CLOEXEC ) = 3
4.971 ( 0.016 ms): read(fd: 3</lib/powerpc64le-linux-gnu/libc-2.21.so>, buf: 0x3fffd4590528, count: 832) = 832
5.148 ( 0.002 ms): fstat(fd: 3</lib/powerpc64le-linux-gnu/libc-2.21.so>, statbuf: 0x3fffd4590340) = 0
5.334 ( 0.013 ms): mmap(arg0: 0, arg1: 1925664, arg2: 5, arg3: 2050, arg4: 3, arg5: 0 ) = 0x3fffb3b00000
5.523 ( 0.013 ms): mmap(arg0: 70367465701376, arg1: 131072, arg2: 3, arg3: 2066, arg4: 3, arg5: 1769472) = 0x3fffb3cc0000
5.716 ( 0.002 ms): close(fd: 3</lib/powerpc64le-linux-gnu/libc-2.21.so> ) = 0
5.960 ( 0.141 ms): mprotect(start: 0x3fffb3cc0000, len: 65536, prot: READ ...

Read more...

Chris J Arges (arges)
tags: added: verification-done-vivid
removed: verification-needed-vivid
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (17.6 KiB)

This bug was fixed in the package linux - 3.19.0-23.24

---------------
linux (3.19.0-23.24) vivid; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1472346

  [ Chris J Arges ]

  * SAUCE: Don't use atomic read in evlist.c
    - LP: #1410673

linux (3.19.0-23.23) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1472048

  [ Chris J Arges ]

  * [Config] Add CRYPTO_DEV_NX_*, 842_* as modules
    - LP: #1454687

  [ Lu, Han ]

  * SAUCE: i915_bpo: drm/i915/audio: add codec wakeup override
    enabled/disable callback
    - LP: #1460674

  [ Timo Aaltonen ]

  * SAUCE: Backport I915_OVERLAY_DISABLE_DEST_COLORKEY
    - LP: #1460674
  * SAUCE: i915_bpo: Rebase to drm-intel-next-fixes-2015-05-29
    - LP: #1460674
  * SAUCE: i915_bpo: Revert "drm/i915: Implement the intel_dp_autotest_edid
    function for DP EDID complaince tests"
    - LP: #1460674
  * SAUCE: i915_bpo: Revert "drm/i915: Add debugfs test control files for
    Displayport compliance testing"
    - LP: #1460674
  * SAUCE: Load i915_bpo from the hda driver on SKL/CHV
    - LP: #1460674
  * SAUCE: i915_bpo: Don't try to support BXT
    - LP: #1460674
  * SAUCE: i915_bpo: drm/i915/skl: Fix DMC API version.

  [ Upstream Kernel Changes ]

  * Revert "usb: dwc2: add bus suspend/resume for dwc2"
    - LP: #1471252
  * Revert "HID: logitech-hidpp: support combo keyboard touchpad TK820"
    - LP: #1471252
  * Revert "KVM: x86: drop fpu_activate hook"
    - LP: #1471252
  * Revert "libceph: clear r_req_lru_item in __unregister_linger_request()"
    - LP: #1471252
  * drm/i915: add component support
    - LP: #1460661
  * ALSA: hda: export struct hda_intel
    - LP: #1460661
  * ALSA: hda: pass intel_hda to all i915 interface functions
    - LP: #1460661
  * ALSA: hda: add component support
    - LP: #1460661
  * drm/atomic-helpers: Fix documentation typos and wrong copy&paste
    - LP: #1460674
  * drm/atomic: Rename drm_atomic_helper_commit_pre_planes() state argument
    - LP: #1460674
  * drm/atomic-helper: Rename commmit_post/pre_planes
    - LP: #1460674
  * drm/atomic-helpers: make mode_set hooks optional
    - LP: #1460674
  * drm/atomic-helper: Fix kerneldoc for prepare_planes
    - LP: #1460674
  * drm: Complete moving rotation property to core
    - LP: #1460674
  * drm: Share plane pixel format check code between legacy and atomic
    - LP: #1460674
  * drm/atomic: Constify a bunch of functions pointer structs
    - LP: #1460674
  * drm: Fix some typo mistake of the annotations
    - LP: #1460674
  * drm: change connector to tmp_connector
    - LP: #1460674
  * drm: atomic: Expose CRTC active property
    - LP: #1460674
  * drm: atomic: Allow setting CRTC active property
    - LP: #1460674
  * drm/atomic-helpers: Properly avoid full modeset dance
    - LP: #1460674
  * drm/atomic: Add helpers for state-subclassing drivers
    - LP: #1460674
  * drm: Fix some typos
    - LP: #1460674
  * drm/atomic: Add for_each_{connector,crtc,plane}_in_state helper macros
    - LP: #1460674
  * drm/atomic-helper: Don't call atomic_update_plane when it stays off
    - LP: #1460674
  * drm/atomic-helper: Really recover pre-atomic plane/cursor behavior
 ...

Changed in linux (Ubuntu Vivid):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Marking Fix Released for Wily as well. Thanks.

Changed in linux (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.