[kgtp] r997 committed - Update doc about stack dump.
- From: kgtp@xxxxxxxxxxxxxx
- To: kgtp@xxxxxxxxxxxxx
- Date: Wed, 28 Mar 2012 12:49:35 +0000
Revision: 997
Author: teawater
Date: Wed Mar 28 05:48:47 2012
Log: Update doc about stack dump.
http://code.google.com/p/kgtp/source/detail?r=997
Modified:
/trunk/howto.txt
/wiki/HOWTO.wiki
=======================================
--- /trunk/howto.txt Wed Mar 28 03:08:41 2012
+++ /trunk/howto.txt Wed Mar 28 05:48:47 2012
@@ -44,11 +44,12 @@
Use $ignore_error and $last_errno to ignore the error of tstart
Use $cooked_clock and $cooked_rdtsc the time without KGTP used
Use $xtime_sec and $xtime_nsec get the timespec
+ Howto backtrace (stack dump)
+ Collect stack and use GDB command "backtrace"
+ Output stack dump through printk
How to use performance counters
Show all the traced data of current frame
- Get backtrace info(stack dump) from Kernel
Howto let tracepoint output value directly
- Output stack dump directly
Switch collect to output the value directly
Get status of KGTP from Kernel
Set the trace buffer into a circular buffer
@@ -815,6 +816,104 @@
+Howto backtrace (stack dump)
+----------------------------
+
+Collect stack and use GDB command "backtrace"
+---------------------------------------------
+
+We can get a backtrace(stack dump) by collecting the stack.
+In x86_32, following action command will collect 512 bytes of stack.
+
+-----------------------------------
+collect *(unsigned char *)$esp@512
+-----------------------------------
+
+In x86_64, following command will collect 512 bytes of stack.
+
+-----------------------------------
+collect *(unsigned char *)$rsp@512
+-----------------------------------
+
+In MIPS or ARM, following command will collect 512 bytes of stack.
+----------------------------------
+collect *(unsigned char *)$sp@512
+-----------------------------------
+
+Following part is an example about howto backtrace in x86_64:
+
+--------------------------------------------------------------------------------
+(gdb) target remote /sys/kernel/debug/gtp
+(gdb) trace vfs_readdir
+Tracepoint 1 at 0xffffffff8113f7fc: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
+(gdb) actions
+Enter actions for tracepoint 1, one per line.
+End with a line saying just "end".
+>collect *(unsigned char *)$rsp@512
+>end
+(gdb) tstart
+(gdb) shell ls
+2 block firmware i ipc Makefile
modules.order scripts source t~ vmlinux
+a.out crypto fs include kernel mm
Module.symvers security System.map usr vmlinux.o
+arch drivers gdb.txt init lib modules.builtin
net sound t virt
+(gdb) tstop
+(gdb) tfind
+Found trace frame 0, tracepoint 1
+#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
+ at ./linux-2.6/fs/readdir.c:24
+24 {
+(gdb) bt
+#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
+ at ./linux-2.6/fs/readdir.c:24
+#1 0xffffffff8113fa14 in sys_getdents (fd=<value optimized out>,
dirent=0x801108, count=32768)
+ at ./linux-2.6/fs/readdir.c:214
+#2 0xffffffff8100af42 in ?? ()
at ./linux-2.6/arch/x86/kernel/entry_64.S:487
+--------------------------------------------------------------------------------
+
+Output stack dump through printk
+--------------------------------
+
+KGTP has special trace state variable $dump_stack, "collect" it will let
Linux
+Kernel output stack dump through printk.
+Following example lets Linux Kernel show the stack dump of vfs_readdir:
+
+--------------------------------------------------------------------------------
+target remote /sys/kernel/debug/gtp
+trace vfs_readdir
+ commands
+ collect $dump_stack
+ end
+--------------------------------------------------------------------------------
+
+Then your kernel will printk like:
+
+--------------------------------------------------------------------------------
+[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
+[22779.208068] Call Trace:
+[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
+[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
+[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
+[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
+[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
+[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
+[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
+[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
+[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
+[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
+[22779.208112] [<c02db51f>] ? __jbd2_journal_temp_unlink_buffer+0x2f/0x110
+[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
+[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
+[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
+[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
+[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
+[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
+[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
+[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
+[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
+[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
+[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
+--------------------------------------------------------------------------------
+
How to use performance counters
-------------------------------
@@ -1047,59 +1146,6 @@
-Get backtrace info(stack dump) from Kernel
-------------------------------------------
-
-We can get a backtrace(stack dump) by collecting the stack.
-In x86_32, following action command will collect 512 bytes of stack.
-
------------------------------------
-collect *(unsigned char *)$esp@512
------------------------------------
-
-In x86_64, following command will collect 512 bytes of stack.
-
------------------------------------
-collect *(unsigned char *)$rsp@512
------------------------------------
-
-In MIPS or ARM, following command will collect 512 bytes of stack.
-----------------------------------
-collect *(unsigned char *)$sp@512
------------------------------------
-
-Following part is an example about howto backtrace in x86_64:
-
---------------------------------------------------------------------------------
-(gdb) target remote /sys/kernel/debug/gtp
-(gdb) trace vfs_readdir
-Tracepoint 1 at 0xffffffff8113f7fc: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
-(gdb) actions
-Enter actions for tracepoint 1, one per line.
-End with a line saying just "end".
->collect *(unsigned char *)$rsp@512
->end
-(gdb) tstart
-(gdb) shell ls
-2 block firmware i ipc Makefile
modules.order scripts source t~ vmlinux
-a.out crypto fs include kernel mm
Module.symvers security System.map usr vmlinux.o
-arch drivers gdb.txt init lib modules.builtin
net sound t virt
-(gdb) tstop
-(gdb) tfind
-Found trace frame 0, tracepoint 1
-#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
- at ./linux-2.6/fs/readdir.c:24
-24 {
-(gdb) bt
-#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
- at ./linux-2.6/fs/readdir.c:24
-#1 0xffffffff8113fa14 in sys_getdents (fd=<value optimized out>,
dirent=0x801108, count=32768)
- at ./linux-2.6/fs/readdir.c:214
-#2 0xffffffff8100af42 in ?? ()
at ./linux-2.6/arch/x86/kernel/entry_64.S:487
---------------------------------------------------------------------------------
-
-
-
Howto let tracepoint output value directly
------------------------------------------
@@ -1111,50 +1157,6 @@
output values directly.
-Output stack dump directly
---------------------------
-KGTP has special trace state variable $dump_stack, "collect" it will let
Linux
-Kernel output stack dump directly.
-Following example lets Linux Kernel show the stack dump of vfs_readdir:
-
---------------------------------------------------------------------------------
-target remote /sys/kernel/debug/gtp
-trace vfs_readdir
- commands
- collect $dump_stack
- end
---------------------------------------------------------------------------------
-
-Then your kernel will printk like:
-
---------------------------------------------------------------------------------
-[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
-[22779.208068] Call Trace:
-[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
-[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
-[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
-[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
-[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
-[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
-[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
-[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
-[22779.208112] [<c02db51f>] ? __jbd2_journal_temp_unlink_buffer+0x2f/0x110
-[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
-[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
-[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
-[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
-[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
-[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
-[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
-[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
-[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
-[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
-[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
---------------------------------------------------------------------------------
-
-
Switch collect to output the value directly
-------------------------------------------
=======================================
--- /wiki/HOWTO.wiki Wed Mar 28 03:08:41 2012
+++ /wiki/HOWTO.wiki Wed Mar 28 05:48:47 2012
@@ -509,6 +509,86 @@
Please goto read the file tools/perf/design.txt in Linux Kernel to get
more info about perf event.
+== Howto backtrace (stack dump) ==
+=== Collect stack and use GDB command "backtrace" ===
+We can get a backtrace(stack dump) by collecting the stack.<br>
+In x86_32, following command will collect 512 bytes of stack.
+{{{
+collect *(unsigned char *)$esp@512
+}}}
+In x86_64, following command will collect 512 bytes of stack.
+{{{
+collect *(unsigned char *)$rsp@512
+}}}
+In MIPS or ARM, following command will collect 512 bytes of stack.
+{{{
+collect *(unsigned char *)$sp@512
+}}}
+Following part is an example about howto backtrace in x86_64:
+{{{
+(gdb) target remote /sys/kernel/debug/gtp
+(gdb) trace vfs_readdir
+Tracepoint 1 at 0xffffffff8113f7fc: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
+(gdb) actions
+Enter actions for tracepoint 1, one per line.
+End with a line saying just "end".
+>collect *(unsigned char *)$rsp@512
+>end
+(gdb) tstart
+(gdb) shell ls
+2 block firmware i ipc Makefile modules.order scripts
source t~ vmlinux
+a.out crypto fs include kernel mm Module.symvers security
System.map usr vmlinux.o
+arch drivers gdb.txt init lib modules.builtin net
sound t virt
+(gdb) tstop
+(gdb) tfind
+Found trace frame 0, tracepoint 1
+#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
+ at ./linux-2.6/fs/readdir.c:24
+24 {
+(gdb) bt
+#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
+ at ./linux-2.6/fs/readdir.c:24
+#1 0xffffffff8113fa14 in sys_getdents (fd=<value optimized out>,
dirent=0x801108, count=32768)
+ at ./linux-2.6/fs/readdir.c:214
+#2 0xffffffff8100af42 in ?? ()
at ./linux-2.6/arch/x86/kernel/entry_64.S:487
+}}}
+=== Output stack dump through printk ===
+KGTP has special trace state variable $dump_stack, "collect" it will let
Linux Kernel output stack dump through printk.<br>
+Following example lets Linux Kernel show the stack dump of vfs_readdir:
+{{{
+target remote /sys/kernel/debug/gtp
+trace vfs_readdir
+ commands
+ collect $dump_stack
+ end
+}}}
+Then your kernel will printk like:
+{{{
+[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
+[22779.208068] Call Trace:
+[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
+[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
+[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
+[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
+[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
+[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
+[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
+[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
+[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
+[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
+[22779.208112] [<c02db51f>] ? __jbd2_journal_temp_unlink_buffer+0x2f/0x110
+[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
+[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
+[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
+[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
+[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
+[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
+[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
+[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
+[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
+[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
+[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
+}}}
=== Define a perf event trace state variable ===
Access an performance counter need define following trace state variable:
{{{
@@ -682,88 +762,9 @@
file->f_path.dentry->d_iname
= "gtp\000.google.chrome.g05ZYO\000\235\337\000\000\000\000\200\067k\364\200\067",
<incomplete sequence \364>
jiffies_64 = 4319751455
}}}
-== Get backtrace info(stack dump) from Kernel ==
-We can get a backtrace(stack dump) by collecting the stack.<br>
-In x86_32, following command will collect 512 bytes of stack.
-{{{
-collect *(unsigned char *)$esp@512
-}}}
-In x86_64, following command will collect 512 bytes of stack.
-{{{
-collect *(unsigned char *)$rsp@512
-}}}
-In MIPS or ARM, following command will collect 512 bytes of stack.
-{{{
-collect *(unsigned char *)$sp@512
-}}}
-Following part is an example about howto backtrace in x86_64:
-{{{
-(gdb) target remote /sys/kernel/debug/gtp
-(gdb) trace vfs_readdir
-Tracepoint 1 at 0xffffffff8113f7fc: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
-(gdb) actions
-Enter actions for tracepoint 1, one per line.
-End with a line saying just "end".
->collect *(unsigned char *)$rsp@512
->end
-(gdb) tstart
-(gdb) shell ls
-2 block firmware i ipc Makefile modules.order scripts
source t~ vmlinux
-a.out crypto fs include kernel mm Module.symvers security
System.map usr vmlinux.o
-arch drivers gdb.txt init lib modules.builtin net
sound t virt
-(gdb) tstop
-(gdb) tfind
-Found trace frame 0, tracepoint 1
-#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
- at ./linux-2.6/fs/readdir.c:24
-24 {
-(gdb) bt
-#0 0xffffffff8113f7fd in vfs_readdir (file=0xffff880075f00780,
filler=0xffffffff8113f630 <filldir>, buf=0xffff880005785f38)
- at ./linux-2.6/fs/readdir.c:24
-#1 0xffffffff8113fa14 in sys_getdents (fd=<value optimized out>,
dirent=0x801108, count=32768)
- at ./linux-2.6/fs/readdir.c:214
-#2 0xffffffff8100af42 in ?? ()
at ./linux-2.6/arch/x86/kernel/entry_64.S:487
-}}}
== Howto let tracepoint output value directly ==
In the previous parts, you may understand that to get a value from Linux
kernel, you need to use a tracepoint "collect" action to save the value to
the tracepoint frame and use the GDB command "tfind" to parse the value
from the frame data.<br>
But we want get the value directly sometimes, so KGTP supports two ways to
output values directly.<br>
-=== Output stack dump directly ===
-KGTP has special trace state variable $dump_stack, "collect" it will let
Linux Kernel output stack dump directly.<br>
-Following example lets Linux Kernel show the stack dump of vfs_readdir:
-{{{
-target remote /sys/kernel/debug/gtp
-trace vfs_readdir
- commands
- collect $dump_stack
- end
-}}}
-Then your kernel will printk like:
-{{{
-[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
-[22779.208068] Call Trace:
-[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
-[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
-[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
-[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
-[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
-[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
-[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
-[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
-[22779.208112] [<c02db51f>] ? __jbd2_journal_temp_unlink_buffer+0x2f/0x110
-[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
-[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
-[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
-[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
-[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
-[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
-[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
-[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
-[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
-[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
-[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
-}}}
=== Switch collect to output the value directly ===
KGTP has special trace state variables $printk_level, $printk_format and
$printk_tmp to support this function.<br>
$printk_level: if its value is 8 (this is the default value), "collect"
action will save value to the tracepoint frame in the simple behavior.<br>
Other related posts:
- » [kgtp] r997 committed - Update doc about stack dump. - kgtp
