[kgtp] r1394 committed - Add gtp_3.7_to_upstream.patch...
- From: kgtp@xxxxxxxxxxxxxx
- To: kgtp@xxxxxxxxxxxxx
- Date: Sat, 01 Dec 2012 08:38:47 +0000
Revision: 1394
Author: teawater
Date: Sat Dec 1 00:38:08 2012
Log: Add gtp_3.7_to_upstream.patch
Rename gtp_3.0_to_upstream.patch to gtp_3.0_to_3.6.patch
http://code.google.com/p/kgtp/source/detail?r=1394
Added:
/trunk/gtp_3.0_to_3.6.patch
/trunk/gtp_3.7_to_upstream.patch
Deleted:
/trunk/gtp_3.0_to_upstream.patch
=======================================
--- /dev/null
+++ /trunk/gtp_3.0_to_3.6.patch Sat Dec 1 00:38:08 2012
@@ -0,0 +1,16507 @@
+--- /dev/null
++++ b/Documentation/gtp/howto.txt
+@@ -0,0 +1,1525 @@
++ Linux Kernel GDB tracepoint module (KGTP)
++ =========================================
++ By Hui Zhu <teawater@xxxxxxxxx>
++ https://code.google.com/p/kgtp/wiki/HOWTO
++
++Update in 2012-09-20
++
++What is KGTP
++Get help or report issues about KGTP
++Preparatory work before use KGTP
++Linux kernel
++If your system use the Linux kernel that is built by yourself
++If your system use the Linux kernel from distribution
++Make sure current Linux kernel debug image is right
++Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
++Get KGTP
++Get KGTP through http
++Get KGTP through svn
++Config KGTP
++Compile KGTP
++Normal compile
++Compile KGTP with some special config
++Install and uninstall KGTP
++Use KGTP with DKMS
++Use KGTP patch for Linux kernel
++Install GDB for KGTP
++Howto use
++Exec it
++Make GDB connect to gtp
++GDB on the current machine
++If GDB on remote machine
++Add module symbols to GDB
++How to use getmod.py
++How to use getmod
++Access memory directly
++GDB tracepoint
++set tracepoint
++How to set tracepoint condition
++actions [num]
++Start and stop the tracepoint
++Enable and disable the tracepoint
++Use "tfind" select the entry inside the trace frame info
++Show and save the tracepoint
++Delete tracepoint
++Use tracepoint get register info from a point of kernel
++Use tracepoint get the value of variable from a point of kernel
++How to handle "Unsupported operator (null) (52) in expression."
++Show all the traced data of current frame
++Get status of tracepoint
++Set the trace buffer into a circular buffer
++Do not stop tracepoint when the GDB disconnects
++kprobes-optimization and the execution speed of tracepoint
++How to use trace state variables
++Simple trace state variables
++Per_cpu trace state variables
++Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
++Special trace state variable $no_self_trace
++Trace the function return with $kret
++Use $ignore_error and $last_errno to ignore the error of tstart
++Use $cooked_clock and $cooked_rdtsc the time without KGTP used
++Use $xtime_sec and $xtime_nsec get the timespec
++Howto backtrace (stack dump)
++Collect stack with $bt and use GDB command "backtrace"
++Collect stack of current function's caller with $_ret
++Use $dump_stack to output stack dump through printk
++How to use performance counters
++Define a perf event trace state variable
++Define a per_cpu perf event trace state variable
++The perf event type and config
++Enable and disable all the perf event in a CPU with $p_pe_en
++GDB scripts to help with set and get the perf event trace state variables
++Howto let tracepoint output value directly
++Switch collect to output the value directly
++Howto show a variable whose value has been optimized away
++Update your GCC
++How to get the function pointer point to
++If the debug info of the function pointer is not optimized out
++If the debug info of the function pointer is optimized out
++/sys/kernel/debug/gtpframe and offline debug
++How to use /sys/kernel/debug/gtpframe_pipe
++Get the frame info with GDB
++Get the frame info with cat
++Get the frame info with getframe
++Use $pipe_trace
++Use KGTP with user space program
++Read memory of user space program directly
++collect stack (for backtrace) of user space program in tracepoint
++How to use add-ons/hotcode.py
++How to add plugin in C
++API
++Example
++How to use
++What is KGTP
++KGTP is a flexible , lightweight and realtime Linux debugger and tracer.
++To use it, you don't need patch or rebuild the Linux kernel. Just build
KGTP module and insmod it is OK.
++
++It makes Linux Kernel supply a GDB remote debug interface. Then GDB in
current machine or remote machine can debug and trace Linux through GDB
tracepoint and some other functions without stopping the Linux Kernel.
++And even if the board doesn't have GDB on it and doesn't have interface
for remote debug. It can debug the Linux Kernel using offline debug (See
HOWTO#/sys/kernel/debug/gtpframe_and_offline_debug).
++KGTP supports X86-32, X86-64, MIPS and ARM.
++KGTP is tested on Linux kernel 2.6.18 to upstream.
++And it can work with Android (See HowToUseKGTPinAndroid).
++
++For new user of KGTP, please go to see Quickstart.
++
++Please go to UPDATE to get more info about KGTP update.
++
++Get help or report issues about KGTP
++You can post it to http://code.google.com/p/kgtp/issues/list, write Email
to kgtp@xxxxxxxxxxxxx or write Email to teawater@xxxxxxxxx .
++The KGTP team will try our best to help you.
++
++Preparatory work before use KGTP
++Linux kernel
++If your system use the Linux kernel that is built by yourself
++To use KGTP, your Linux kernel need open following options:
++
++General setup --->
++ [*] Kprobes
++
++[*] Enable loadable module support --->
++
++Kernel hacking --->
++ [*] Debug Filesystem
++ [*] Compile the kernel with debug info
++If your system use the Linux kernel from distribution
++You need install some Linux kernel package.
++
++Ubuntu
++Install the Linux kernel debug image
++1) Add debug source to the sources list of Ubuntu.
++
++Create an /etc/apt/sources.list.d/ddebs.list by running the following
line at a terminal:
++echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted
universe multiverse" | \
++sudo tee -a /etc/apt/sources.list.d/ddebs.list
++Stable releases (not alphas and betas) require three more lines adding to
the same file, which is done by the following terminal command:
++echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main
restricted universe multiverse
++deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted
universe multiverse
++deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted
universe multiverse" | \
++sudo tee -a /etc/apt/sources.list.d/ddebs.list
++Import the debug symbol archive signing key:
++sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
++Then run:
++sudo apt-get update
++
++2) Get Linux kernel debug image.
++
++sudo apt-get install linux-image-$(uname -r)-dbgsym
++Then you can find Linux kernel debug image
in "/usr/lib/debug/boot/vmlinux-$(uname -r)".
++
++Install the Linux kernel headers
++sudo apt-get install linux-headers-generic
++Install the Linux kernel source
++Install the source package:
++sudo apt-get install linux-source
++Uncompress the source package:
++sudo mkdir -p /build/buildd/
++sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2
-C /build/buildd/
++sudo rm -rf /build/buildd/linux-$(uname -r | sed 's/-.*//')
++sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//')
/build/buildd/linux-$(uname -r | sed 's/-.*//')
++Fedora
++Install the Linux kernel debug image
++sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo
++Then you can find Linux kernel debug image
in "/usr/lib/debug/lib/modules/$(uname -r)/vmlinux".
++
++Install the Linux kernel devel package
++sudo yum install kernel-devel-$(uname -r)
++Others
++You need install the Linux kernel debug image package and the Linux
Kernel source.
++
++Make sure current Linux kernel debug image is right
++GDB open the right Linux kernel debug image is an very important because
GDB will get the debug info and address info from it. So before you use
KGTP, please do the check to make sure about it.
++There are 2 ways to do the check, what I suggest is do both of them to
make sure Linux kernel debug image is right.
++
++Please note that if you determine you use the right Linux kernel debug
image, but cannot pass these ways. Please see
HOWTO#Handle_the_issue_that_Linux_kernel_debug_image's_address_in.
++
++Use /proc/kallsyms
++In the system that its Linux kernel is what you want to trace, use
following command to get the address of sys_read and sys_write:
++
++sudo cat /proc/kallsyms | grep sys_read
++ffffffff8117a520 T sys_read
++sudo cat /proc/kallsyms | grep sys_write
++ffffffff8117a5b0 T sys_write
++Then we can get that the address of sys_read is 0xffffffff8117a520 and
the address of sys_write is 0xffffffff8117a5b0.
++After that use GDB get address of sys_read and sys_write from Linux
kernel debug image:
++
++gdb ./vmlinux
++(gdb) p sys_read
++$1 = {long int (unsigned int, char *, size_t)} 0xffffffff8117a520
<sys_read>
++(gdb) p sys_write
++$2 = {long int (unsigned int, const char *, size_t)} 0xffffffff8117a5b0
<sys_write>
++The address of sys_read and sys_write is same, so the Linux kernel debug
image is right.
++
++Use linux_banner
++sudo gdb ./vmlinux
++(gdb) p linux_banner
++$1 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
++This linux_banner is the kernel info inside the Linux kernel debug image.
++After that, connect to KGTP following the way in
HOWTO#Make_GDB_connect_to_gtp connect to KGTP and print linux_banner again.
++
++(gdb) target remote /sys/kernel/debug/gtp
++Remote debugging using /sys/kernel/debug/gtp
++0x0000000000000000 in irq_stack_union ()
++(gdb) p linux_banner
++$2 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
++This linux_banner is the kernel info that Linux kernel that KGTP is
tracing. If it is same with the prev kernel info, the Linux kernel debug
image is right.
++
++Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
++In X86_32, you will found that the Linux kernel debug image's address
info is not same with Linux kernel when it running through the ways in
HOWTO#Make_sure_current_Linux_kernel_debug_image_is_right. And you
determine the Linux kernel debug image is right.
++This issue is because:
++
++Processor type and features --->
++ (0x1000000) Physical address where the kernel is loaded
++ (0x100000) Alignment value to which kernel should be aligned
++The values of these two options are different. Please note that
the "Physical address where the kernel is loaded" is not showed in config
sometimes. You can get its value through search "PHYSICAL_START".
++
++You can handle this issue through change "Alignment value to which kernel
should be aligned" same with "Physical address where the kernel is loaded".
++This issue doesn't affect X86_64.
++
++Get KGTP
++Get KGTP through http
++Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to
download the package.
++
++Get KGTP through svn
++Some people have trouble with access to KGTP website. You can access kgtp
through svn:
++
++svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only
++kgtp-read-only/tags/ Present for each release of KGTP.
++kgtp-read-only/trunk/ Present for the main trunk of KGTP.
++
++Config KGTP
++Following part is the default config of KGTP inside the Makefile. With
this config, KGTP will build together with current kernel that running on
this machine.
++
++KERNELDIR := /lib/modules/`uname -r`/build
++CROSS_COMPILE :=
++KERELDIR is set to the directory which holds the kernel you want to build
for. By default, it is set to the kernel that you are running.
++Please note that this directory should be Linux kernel build directory or
linux-headers directory but not the source directory but not the Linux
kernel source directory. And the Linux kernel build directory should be
used after build successful.
++CROSS_COMPILE is set to the prefix name of compiler that you want to
build KGTP. Empty to compile with your default compiler.
++ARCH is the architecture.
++
++Or you can choose which kernel you want build with and which compiler you
want use by change Makefile.
++For example:
++
++KERNELDIR := /home/teawater/kernel/bamd64
++CROSS_COMPILE :=x86_64-glibc_std-
++ARCH := x86_64
++KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will use
x86_64-glibc_std-gcc.
++
++Compile KGTP
++Normal compile
++cd kgtp/
++make
++In some build environment (for example Android) will get some error with
user space program getmod or getframe. Please ignore this error and use the
gtp.ko in this directory.
++
++Compile KGTP with some special config
++Most of time, KGTP can auto select right options to build with Various
versions of Linux kernel.
++But if you want config special options with yourself, you can read
following part:
++With this option, KGTP will not auto select any build options.
++
++make AUTO=0
++With this option, KGTP will use simple frame instead of KGTP ring buffer.
++The simple frame doesn't support gtpframe_pipe. It just for debug KGTP.
++
++make AUTO=0 FRAME_SIMPLE=1
++With this option, $clock will return rdtsc value instead of local_clock.
++
++make AUTO=0 CLOCK_CYCLE=1
++With this option, KGTP will use procfs instead of debugfs.
++
++make AUTO=0 USE_PROC=1
++The options can use together, for example:
++
++make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1
++Install and uninstall KGTP
++KGTP don't need to be install because it can insmod directly inside its
directory (See HOWTO#Exec_it). But if you need, you can install it to your
system.
++Install:
++
++cd kgtp/
++sudo make install
++Uninstall:
++
++cd kgtp/
++sudo make uninstall
++Use KGTP with DKMS
++You can use KGTP with DKMS if you want it.
++Following commands will copy the files of KGTP to the directory that DKMS
need.
++
++cd kgtp/
++sudo make dkms
++Then you can use DKMS commands to control KGTP. Please goto
http://linux.dell.com/dkms/manpage.html to see how to use DKMS.
++
++Use KGTP patch for Linux kernel
++Most of time, you don't need KGTP patch because KGTP can build as a LKM
and very easy to use. But to help some people include KGTP to them special
Linux Kernel tree, KGTP supply patches for Linux kernel.
++In the KGTP directory:
++gtp_3.0_to_upstream.patch is the patch for Linux kernel from 3.0 to
upstream.
++gtp_2.6.39.patch is the patch for Linux kernel 2.6.39.
++gtp_2.6.33_to_2.6.38.patch is the patch for Linux kernel from 2.6.33 to
2.6.38.
++gtp_2.6.20_to_2.6.32.patch is the patch for Linux kernel from 2.6.20 to
2.6.32.
++gtp_older_to_2.6.19.patch is the patch for Linux kernel 2.6.19 and older
version.
++Install GDB for KGTP
++The GDB that older than 7.3 have some bugs of tracepoint. And some
functions of GDB are not very well.
++So if your GDB is older than 7.3 please go to
https://code.google.com/p/gdbt/ to get howto install GDB for KGTP.
++If you have issue about GDB please get help according to
HOWTO#Report_issues_about_KGTP.
++
++Howto use
++Exec it
++If you have installed KGTP in your system, you can:
++
++sudo modprobe gtp
++Or you can use the kgtp module in the directory.
++
++cd kgtp/
++sudo insmod gtp.ko
++Make GDB connect to gtp
++GDB on the current machine
++sudo gdb ./vmlinux
++(gdb) target remote /sys/kernel/debug/gtp
++Remote debugging using /sys/kernel/debug/gtp
++0x0000000000000000 in ?? ()
++After that, you can begin to use GDB command trace and debug the Linux
Kernel.
++
++If GDB on remote machine
++#Open the KGTP interface in current machine.
++sudo su
++nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp
++(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old
version netcat.)
++#Let gdb connect to the port 1234
++gdb ./vmlinux
++(gdb) target remote xxx.xxx.xxx.xxx:1234
++After that, you can begin to use GDB command trace and debug the Linux
Kernel.
++
++Add module symbols to GDB
++Sometimes you need to add a Linux kernel module's symbols to GDB to debug
it.
++Add symbols with hand is not very easy, so KGTP package include an GDB
python script "getmod.py" and a program "getmod" can help you.
++
++How to use getmod.py
++Connect to KGTP before use the getmod.py.
++
++(gdb) source ~/kgtp/getmod.py
++Then this script will auto load the Linux kernel module's symbols to GDB.
++
++How to use getmod
++"getmod" is written by C so you can use it anywhere even if in an
embedded environment.
++For example:
++
++#Following command save Linux Kernel module info to the file ~/tmp/mi in
GDB
++#command format.
++sudo getmod >~/tmp/mi
++#in gdb part:
++(gdb) source ~/tmp/mi
++add symbol table from
file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
++ .text_addr = 0xf80de000
++ .note.gnu.build-id_addr = 0xf80de088
++ .exit.text_addr = 0xf80de074
++ .init.text_addr = 0xf8118000
++ .rodata.str1.1_addr = 0xf80de0ac
++ .rodata_addr = 0xf80de0c0
++ __mcount_loc_addr = 0xf80de9c0
++ .data_addr = 0xf80de9e0
++ .gnu.linkonce.this_module_addr = 0xf80dea00
++#After this GDB command, all the Linux Kernel module info is loaded into
GDB.
++If you use remote debug or offline debug, maybe you need change the base
directory. Following example is for it.
++
++#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r
/home/teawater/kernel/b26
++sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi
++Access memory directly
++After connect the KGTP, you can access most of memory directly.
++For example, you can access to "jiffies_64" with following command:
++
++(gdb) p jiffies_64
++Or you can access to the first entry of "static LIST_HEAD(modules)" with
following command:
++
++(gdb) p *((struct module *)((char *)modules->next - ((size_t) &(((struct
module *)0)->list))))
++Or you can access to the CPU0 memory info of "DEFINE_PER_CPU(struct
device *, mce_device);":`
++
++p *(struct device *)(__per_cpu_offset[0]+(uint64_t)(&mce_device))
++If you want show more than one variables with one GDB command, please use
following example:
++
++(gdb) printf "%4d %4d %4d %4d %4d %4d %18d %lu\n", this_rq->cpu,
this_rq->nr_running, this_rq->nr_uninterruptible, nr_active,
calc_load_tasks->counter, this_rq->calc_load_active, delta,
this_rq->calc_load_update
++2 1 0 0 0 0 673538312 717077240
++GDB tracepoint
++Tracepoint is that GDB define some addresses and some actions and put
them to the target (KGTP). After tracepoint start, , KGTP will do these
actions (Some of them will collect data and save them to tracepoint frame
buffer) when Linux kernel execution to there addresses. After that, Linux
kernel will keep execution.
++KGTP supply some interfaces that GDB or other programe can take the data
of tracepoint frame buffer out to parse.
++About these interfaces, this doc have introduced "/sys/kernel/debug/gtp".
And will introduce "/sys/kernel/debug/gtpframe"
and "/sys/kernel/debug/gtpframe_pipe" later.
++
++Doc of GDB tracepoint in
http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html.
++
++set tracepoint
++The trace command is very similar to the break command. Its argument
location can be a source line, a function name, or an address in the target
program. The trace command defines a tracepoint, which is a address or some
addresses that KGTP do some actions in it.
++Here are some examples of using the trace command:
++
++(gdb) trace foo.c:121 // a source file and line number
++
++(gdb) trace +2 // 2 lines forward
++
++(gdb) trace my_function // first source line of function
++
++(gdb) trace *my_function // EXACT start address of function
++
++(gdb) trace *0x2117c4 // an address
++Howto handle the function is there but set tracepoint on it got fail
++GCC will inline some static function to increase the performance. You
cannot set tracepoint on the function name because object file doesn't have
symbol of inline function.
++You can use "trace filename:line" to set tracepoint on it.
++
++How to set tracepoint condition
++http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html
++Like breakpoints, we can set conditions on tracepoints. The speed of
tracepoints is faster than breakpoints because KGTP can do all the
condition checks.
++For example:
++
++(gdb) trace handle_irq if (irq == 47)
++This action of tracepoint 1 will work only when irq number is 47.
++
++And you can use GDB command "condition" to specify the condition of a
tracepoint. GDB command "condition N COND" will specify tracepoint number N
to trace only if COND is true.
++For example:
++
++(gdb) trace handle_irq
++(gdb) condition 1 (irq == 47)
++GDB command "info tracepoint" will show the ID of the tracepoint.
++
++Value of $bpnum is the last ID of GDB tracepoint. Then you can use GDB
command "condtion" set the condition of last tracepoint without get its ID.
For example:
++
++(gdb) trace handle_irq
++(gdb) condition $bpnum (irq == 47)
++actions [num]
++This command will prompt for a list of actions to be taken when the
tracepoint is hit. If the tracepoint number num is not specified, this
command sets the actions for the one that was most recently defined (so
that you can define a tracepoint and then say actions without bothering
about its number). You specify the actions themselves on the following
lines, one action at a time, and terminate the actions list with a line
containing just end. So far, the only defined actions are collect, teval,
and while-stepping.
++
++collect expr1, expr2, ...
++Collect values of the given expressions when the tracepoint is hit. This
command accepts a comma-separated list of any valid expressions. In
addition to global, static, or local variables, the following special
arguments are supported:
++
++$regs Collect all registers.
++$args Collect all function arguments.
++$locals Collect all local variables.
++Please note that collect an pointer (collect ptr) will just collect the
address of this pointer. Add a * before ptr will make action collect the
data that pointer point to(collect *ptr).
++
++teval expr1, expr2, ...
++Evaluate the given expressions when the tracepoint is hit. This command
accepts a comma-separated list of expressions. The results are discarded,
so this is mainly useful for assigning values to trace state variables (see
HOWTO#Simple_trace_state_variables) without adding those values to the
trace buffer, as would be the case if the collect action were used.
++
++while-stepping n
++Please goto
HOWTO#If_the_debug_info_of_the_function_pointer_is_optimized_out see howto
use it.
++
++Start and stop the tracepoint
++Tracepoint will exec actions only when it is starting use this GDB
command:
++
++(gdb) tstart
++It will stop by this GDB command:
++
++(gdb) tstop
++Enable and disable the tracepoint
++Like breakpoint, tracepoint can be control by GDB commands "enable"
and "disable". But please note that it only useful when tracepoint stop.
++
++Use "tfind" select the entry inside the trace frame info
++GDB command "tfind" is used to select a entry of trace frame bufffer when
tracepoint stop.
++When GDB inside "tfind" mode, it will just show the values of this entry
that the tracepoint action collect. So it will output some error when print
some values that action doesn't collect for example the argument of
function. That is not a bug, please don't worry about it.
++Use "tfind" again will select next entry. "tfind id" will select entry id.
++To go back to normal mode, please use GDB command "tfind -1". Please goto
http://sourceware.org/gdb/current/onlinedocs/gdb/tfind.html get more info
about it.
++
++Show and save the tracepoint
++You can use GDB command "info tracepoints" to show all the tracepoints.
++You can use GDB command "save tracepoints filename" to save the commands
that setup the tracepoints and actions into file filename. Then you use use
GDB commands "source filename" to setup this tracepints again.
++
++Delete tracepoint
++GDB command "delete id" will delete tracepoint id. If "delete" without
argument, it will delete all the tracepoint.
++
++Use tracepoint get register info from a point of kernel
++The following is an example that records the value of all registers
when "vfs_readdir" is called.
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xc01a1ac0: file
++/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect $reg
++>end
++(gdb) tstart
++(gdb) shell ls
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
<filldir64>,
++ buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
++23 /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
directory.
++ in /home/teawater/kernel/linux-2.6/fs/readdir.c
++(gdb) info reg
++eax 0xc5528d00 -984445696
++ecx 0xc0d09f90 -1060069488
++edx 0xc01a1900 -1072031488
++ebx 0xfffffff7 -9
++esp 0xc0d09f8c 0xc0d09f8c
++ebp 0x0 0x0
++esi 0x8061480 134616192
++edi 0xc5528d00 -984445696
++eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
++eflags 0x286 [ PF SF IF ]
++cs 0x60 96
++ss 0x8061480 134616192
++ds 0x7b 123
++es 0x7b 123
++fs 0x0 0
++gs 0x0 0
++(gdb) tfind
++Found trace frame 1, tracepoint 1
++0xc01a1ac1 23 in /home/teawater/kernel/linux-2.6/fs/readdir.c
++(gdb) info reg
++eax 0xc5528d00 -984445696
++ecx 0xc0d09f90 -1060069488
++edx 0xc01a1900 -1072031488
++ebx 0xfffffff7 -9
++esp 0xc0d09f8c 0xc0d09f8c
++ebp 0x0 0x0
++esi 0x8061480 134616192
++edi 0xc5528d00 -984445696
++eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
++eflags 0x286 [ PF SF IF ]
++cs 0x60 96
++ss 0x8061480 134616192
++ds 0x7b 123
++es 0x7b 123
++fs 0x0 0
++gs 0x0 0
++Use tracepoint get the value of variable from a point of kernel
++The following is an example that records the value of "jiffies_64" when
the function "vfs_readdir" is called:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xc01ed740: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect jiffies_64
++>collect file->f_path.dentry->d_iname
++>end
++(gdb) tstart
++(gdb) shell ls
++arch drivers include kernel mm Module.symvers
security System.map virt
++block firmware init lib modules.builtin net
sound t vmlinux
++crypto fs ipc Makefile modules.order scripts
source usr vmlinux.o
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580
<filldir64>, buf=0xd6dfdf90)
++ at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
++24 {
++(gdb) p jiffies_64
++$1 = 4297248706
++(gdb) p file->f_path.dentry->d_iname
++$1 = "b26", '\000' <repeats 28 times>
++How to handle "Unsupported operator (null) (52) in expression."
++If you use condition about string, you will got this error when you
call "tstart".
++To handle it, you can convent the char to int to handle this issue, for
example:
++
++(gdb) p/x 'A'
++$4 = 0x41
++(gdb) condition 1 (buf[0] == 0x41)
++Show all the traced data of current frame
++After use "tfind" select an entry, you can use "tdump" to do it.
++
++(gdb) tdump
++Data collected at tracepoint 1, trace frame 0:
++$cr = void
++file->f_path.dentry->d_iname
= "gtp\000.google.chrome.g05ZYO\000\235\337\000\000\000\000\200\067k\364\200\067",
<incomplete sequence \364>
++jiffies_64 = 4319751455
++Get status of tracepoint
++Please use GDB command "tstatus".
++
++Set the trace buffer into a circular buffer
++http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
++The frame buffer is not a circular buffer by default. When the buffer is
full, the tracepoint will stop.
++Following command will set frame buffer to a circular buffer. When the
buffer is full, it will auto discard traceframes (oldest first) and keep
trace.
++
++(gdb) set circular-trace-buffer on
++Do not stop tracepoint when the GDB disconnects
++http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
++KGTP will stop tracepoint and delete the trace frame when GDB disconnects
with it by default.
++Following command will open the KGTP disconnect-trace. After that, when
GDB disconnects with KGTP, KGTP will not stop tracepoint. And after GDB
reconnects to KGTP, it can keep control of KGTP like nothing happened.
++
++(gdb) set disconnected-tracing on
++kprobes-optimization and the execution speed of tracepoint
++The tracepoint is execution together with Linux kernel. So it speed will
affect the speed the system.
++The KGTP tracepoint base on Linux kernel kprobe. Because the normal
kprobe base on breakpoint instruction, so it is not very fast.
++
++But if arch of kernel is X86_64 or X86_32 and kernel config didn't
open "Preemptible Kernel" (PREEMPT), the kprobe is speed up by
kprobes-optimization (CONFIG_OPTPROBES) that make kprobe very fast.
++To make sure about that, you can use following command in terminal:
++
++sysctl -A | grep kprobe
++debug.kprobes-optimization = 1
++That means that your kernel support kprobes-optimization.
++Please note that some KGTP functions will make this tracepoint use simple
kprobe even if this Kernel support kprobes-optimization. This doc will add
note when introduce these functions. Please avoid using them when you
really care about the tracepoint speed.
++
++How to use trace state variables
++http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html
++Tracepoints have variables that can be traced directly, or used in
tracepoint conditions.
++Please note that just GDB 7.2.1 and later versions support use trace
state variables directly, the old version of GDB can show the value of
trace state variables through command "info tvariables".
++
++Simple trace state variables
++Define a trace state variable $c.
++
++(gdb) tvariable $c
++Trace state variable $c is created with initial value 0. The following
action uses $c to count how many irqs happened in the kernel.
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace handle_irq
++(gdb) actions
++Enter actions for tracepoint 3, one per line.
++End with a line saying just "end".
++>collect $c #Save current value of $c to the trace frame buffer.
++>teval $c=$c+1 #Increase the $c.
++>end
++Also, you can set a value of variable to trace state variable, but don't
forget covert variable to "uint64_t".
++
++>teval $c=(uint64_t)a
++You can get the current value of $c while the trace is running or stopped.
++
++(gdb) tstart
++(gdb) info tvariables
++$c 0 31554
++(gdb) p $c
++$5 = 33652
++(gdb) tstop
++(gdb) p $c
++$9 = 105559
++When using tfind, you can parse the trace frame buffer. If the value of a
trace state variable is collected, you can parse it out.
++
++(gdb) tstop
++(gdb) tfind
++(gdb) info tvariables
++$c 0 0
++(gdb) p $c
++$6 = 0
++(gdb) tfind 100
++(gdb) p $c
++$7 = 100
++If need, the tracepoint action that access the simple trace state
variables will auto lock a spin lock for trace state variables. So it can
handle race condition issue about trace state variables.
++The following example is OK even if it running a machine that have more
than one CPU.
++
++>teval $c=$c+1
++Per_cpu trace state variables
++Per_cpu trace state variables are special simple trace state variables.
++When tracepoint action access to it, it will access to this CPU special
trace state variables.
++It have 2 advantages:
++1. The tracepoint actions that access to per_cpu trace state variables
don't have the race conditon issue. So it don't need lock the spin lock for
trace state variables. It is faster than simple trace state variables on
multi-core machine.
++2. Write the action that count some CPU special thing with it is easier
than simple trace state variables.
++
++How to define
++Per_cpu trace state variables have two types:
++
++Local CPU variables
++"per_cpu_"+string
++or
++
++"p_"+string
++For example:
++
++(gdb) tvariable $p_count
++When access this trace state variable in tracepoint actions, it will
return the variable's value of CPU that this tracepoint actions running on.
++
++CPU id variables
++"per_cpu_"+string+CPU_id
++or
++
++"p_"+string+CPU_id
++For example:
++
++(gdb) tvariable $p_count0
++(gdb) tvariable $p_count1
++(gdb) tvariable $p_count2
++(gdb) tvariable $p_count3
++When access this trace state variable in tracepoint actions or GDB
command line, it will return the variable's value of CPU CPU_id.
++Follow example can auto define a CPU id variables for each CPU of this
machine. (Please note that need let GDB connect to KGTP before use these
commands.)
++
++(gdb) set $tmp=0
++(gdb) while $tmp<$cpu_number
++ >eval "tvariable $p_count%d",$tmp
++ >set $tmp=$tmp+1
++ >end
++Example 1
++This example define a tracepoint that count the times that call vfs_read
of each CPU.
++
++tvariable $p_count
++set $tmp=0
++while $tmp<$cpu_number
++ eval "tvariable $p_count%d",$tmp
++ set $tmp=$tmp+1
++ end
++trace vfs_read
++actions
++ teval $p_count=$p_count+1
++ end
++Then you can show how many vfs_read in each CPU after "tstart":
++
++(gdb) p $p_count0
++$3 = 44802
++(gdb) p $p_count1
++$4 = 55272
++(gdb) p $p_count2
++$5 = 102085
++(gdb) p $p_count3
++Example 2
++This example record stack dump of the function that close IRQ longest
time of each CPU.
++
++set pagination off
++
++tvariable $bt=1024
++tvariable $p_count
++tvariable $p_cc
++set $tmp=0
++while $tmp<$cpu_number
++eval "tvariable $p_cc%d",$tmp
++set $tmp=$tmp+1
++end
++
++tvariable $ignore_error=1
++
++trace arch_local_irq_disable
++ commands
++ teval $p_count=$clock
++ end
++trace arch_local_irq_enable if ($p_count && $p_cc < $clock - $p_count)
++ commands
++ teval $p_cc = $clock - $p_count
++ collect $bt
++ collect $p_cc
++ teval $p_count=0
++ end
++
++enable
++set pagination on
++Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
++KGTP special trace state variables $current_task, $current_thread_info,
$cpu_id and $clock can very easy to access to some special value. You can
see them when GDB connects to the KGTP. You can use them in tracepoint
conditions or actions.
++Access $current_task in tracepoint condition and action will get that
returns of get_current().
++Access $current_task_pid in tracepoint condition and action will get that
returns of get_current()->pid.
++Access $current_thread_info in tracepoint condition and action will get
that returns of current_thread_info().
++Access $cpu_id in tracepoint condition and action will get that returns
of smp_processor_id().
++Access $clock in tracepoint condition and action will get that returns of
local_clock() that return the timestamp in nanoseconds.
++$rdtsc is only available on X86 and X86_64 architecture. Access it in
anytime will get current value of TSC with instruction RDTSC.
++Access $hardirq_count in tracepoint condition and action will get that
returns of hardirq_count().
++Access $softirq_count in tracepoint condition and action will get that
returns of softirq_count().
++Access $irq_count in tracepoint condition and action will get that
returns of irq_count().
++And KGTP has other special trace state variables $dump_stack,
$printk_level, $printk_format and $printk_tmp. All of them output their
values directly, as can be seen in
HOWTO#Howto_let_tracepoint_output_value_directly.
++The following example counts in $c how many vfs_read calls that process
16663 does and collects the struct thread_info of current task:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid ==
16663)
++(gdb) tvariable $c
++(gdb) actions
++Enter actions for tracepoint 4, one per line.
++End with a line saying just "end".
++>teval $c=$c+1
++>collect (*(struct thread_info *)$current_thread_info)
++>end
++(gdb) tstart
++(gdb) info tvariables
++Name Initial Current
++$c 0 184
++$current_task 0 <unknown>
++$current_thread_info 0 <unknown>
++$cpu_id 0 <unknown>
++(gdb) tstop
++(gdb) tfind
++(gdb) p *(struct thread_info *)$current_thread_info
++$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status =
0, cpu = 1, preempt_count = 2, addr_limit = {
++ seg = 4294967295}, restart_block = {fn = 0xc0159fb0
<do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11,
++ arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags
= 1, bitset = 78, time = 977063750,
++ uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb,
expires = 335007449089}, poll = {
++ ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78,
tv_nsec = 977063750}}},
++ sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack =
0xef340044 "", uaccess_err = 0}
++Another example shows how much sys_read() executes in each CPU.
++
++(gdb) tvariable $c0
++(gdb) tvariable $c1
++(gdb) trace sys_read
++(gdb) condition $bpnum ($cpu_id == 0)
++(gdb) actions
++>teval $c0=$c0+1
++>end
++(gdb) trace sys_read
++(gdb) condition $bpnum ($cpu_id == 1)
++(gdb) actions
++>teval $c1=$c1+1
++>end
++(gdb) info tvariables
++Name Initial Current
++$current_task 0 <unknown>
++$cpu_id 0 <unknown>
++$c0 0 3255
++$c1 0 1904
++sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. Please note
that this example just to howto use $cpu_id. Actially, this example use
per_cpu trace state variables is better.
++
++Special trace state variable $no_self_trace
++$no_self_trace is different with the special trace state variables in the
previous section. It is used to control the behavior of tracepoint.
++If the action of a tracepoint include a command access to the
$no_self_trace. The tracepoint will not trace anything if the current_task
is the a KGTP self process (GDB, netcat, getframe or some others process
that access to the interface of KGTP).
++For example, if we want trace vfs_read or something that have process
context, and we don't want trace the operation of KGTP self process. Add
following command to the action:
++
++>collect $no_self_trace
++Please note that the code that doesn't about process context (Irq
handler, softirq) doesn't need set this variable.
++
++Trace the function return with $kret
++Sometime, set the tracepoint to the end of function is hard because the
Kernel is compiled with optimization. At this time, you can get help from
$kret.
++$kret is a special trace state variable like $no_self_trace. When you set
value of it inside the action of tracepoint, this tracepoint be set with
kretprobe instead of kprobe. Then it can trace the end of this function.
++Please note that this tracepoint must set in the first address of the
function in format "function_name".
++
++Following part is an example:
++
++#"*(function_name)" format can make certain that GDB send the first
address of function to KGTP.
++(gdb) trace *vfs_read
++(gdb) actions
++>teval $kret=0
++#Following part you can set commands that you want.
++Use $ignore_error and $last_errno to ignore the error of tstart
++If KGTP got any error of tstart, this command will get fail.
++But sometime we need ignore this error and let KGTP keep work. For
example: If you set tracepoint on the inline function spin_lock. This
tracepoint will be set to a lot of addresses that some of them cannot be
set kprobe. It will make tstart get fail. You can use "$ignore_error"
ignore this error.
++And the last error number will available in "$last_errno".
++
++(gdb) tvariable $ignore_error=1
++This command will open ignore.
++
++(gdb) tvariable $ignore_error=0
++This command will close ignore.
++
++Use $cooked_clock and $cooked_rdtsc the time without KGTP used
++Access these two trace state variables can get the time without KGTP
used. Then we can get more close to really time that a part of code used
even if the actions of tracepoint is very complex. They will be introduce
in Cookbook (coming soon).
++
++Use $xtime_sec and $xtime_nsec get the timespec
++Access these two trace state variables will return the time of day in a
timespec that use getnstimeofday.
++$xtime_sec will access to the second part of a timespec.
++$xtime_nsec will access to the nanosecond part of a timespec.
++
++Howto backtrace (stack dump)
++Each time your program performs a function call, information about the
call is generated. That information includes the location of the call in
your program, the arguments of the call, and the local variables of the
function being called. The information is saved in a block of data called a
stack frame. The stack frames are allocated in a region of memory called
the call stack.
++
++Collect stack with $bt and use GDB command "backtrace"
++Because this way is faster (just collect the stack when trace) and parse
out most of info inside the call stack (it can show all the stack info that
I introduce). So I suggest you use this way to do the stack dump.
++First we need add the collect the stack command to the tracepoint action.
++The general collect the stack command in GDB tracepoint is: In x86_32,
following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$esp@512
++In x86_64, following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$rsp@512
++In MIPS or ARM, following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$sp@512
++These commands is so hard to remember, and the different arch need
different command.
++KGTP have an special tracepoint trace state variable $bt. If tracepoint
action access it, KGTP will auto collect the $bt size (default value is
512) stack. For example, this command will collect 512 bytes stack memory:
++
++>collect $bt
++If you want to change size of $bt, you can use following GDB command
before "tstart":
++
++(gdb) tvariable $bt=1024
++Following part is an example about howto collect stack and howto use GDB
parse it:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xffffffff8118c300: file
/home/teawater/kernel2/linux/fs/readdir.c, line 24.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect $bt
++>end
++(gdb) tstart
++(gdb) shell ls
++1 crypto fs include kernel mm
Module.symvers security System.map vmlinux
++arch drivers hotcode.html init lib modules.builtin
net sound usr vmlinux.o
++block firmware hotcode.html~ ipc Makefile modules.order
scripts source virt
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++24 {
++(gdb) bt
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
++#2 <signal handler called>
++#3 0x00007f00253848a5 in ?? ()
++#4 0x00003efd32cddfc9 in ?? ()
++#5 0x00002c15b7d04101 in ?? ()
++#6 0x000019c0c5704bf1 in ?? ()
++#7 0x0000000900000000 in ?? ()
++#8 0x000009988cc8d269 in ?? ()
++#9 0x000009988cc9b8d1 in ?? ()
++#10 0x0000000000000000 in ?? ()
++(gdb) up
++#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
++214 error = vfs_readdir(file, filldir, &buf);
++(gdb) p buf
++$1 = {current_dir = 0x1398c58, previous = 0x0, count = 32768, error = 0}
++(gdb) p error
++$3 = -9
++(gdb) frame 0
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++24 {
++From this example, we can see some GDB commands that parse the the call
stack:
++bt is the alias of GDB commands backtrace that print a backtrace of the
entire stack: one line per frame for all frames in the stack.
++up n is move n frames up the stack. For positive numbers n, this advances
toward the outermost frame, to higher frame numbers, to frames that have
existed longer. n defaults to one.
++down n is move n frames down the stack. For positive numbers n, this
advances toward the innermost frame, to lower frame numbers, to frames that
were created more recently. n defaults to one. You may abbreviate down as
do.
++frame n is select frame number n. Recall that frame zero is the innermost
(currently executing) frame, frame one is the frame that called the
innermost one, and so on. The highest-numbered frame is the one for main.
++You can see that when you use up, down or frame to the different calll
stack frame, you can output the value of the arguments and local variables
of different call stack frame.
++To get the more info about howto use GDB parse the call stack, please see
http://sourceware.org/gdb/current/onlinedocs/gdb/Stack.html
++Collect stack of current function's caller with $_ret
++If you just want to collect stack of current function's caller, please
use $_ret.
++Please note that set the tracepoint that collect $_ret cannot in the
first address of function.
++For example:
++
++(gdb) list vfs_read
++360 }
++361
++362 EXPORT_SYMBOL(do_sync_read);
++363
++364 ssize_t vfs_read(struct file *file, char __user *buf, size_t
count, loff_t *pos)
++365 {
++366 ssize_t ret;
++367
++368 if (!(file->f_mode & FMODE_READ))
++369 return -EBADF;
++(gdb) trace 368
++Tracepoint 2 at 0xffffffff8117a244: file
/home/teawater/kernel2/linux/fs/read_write.c, line 368.
++(gdb) actions
++Enter actions for tracepoint 2, one per line.
++End with a line saying just "end".
++>collect $_ret
++>end
++(gdb) tstart
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 2
++#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
++ at /home/teawater/kernel2/linux/fs/read_write.c:368
++368 if (!(file->f_mode & FMODE_READ))
++(gdb) bt
++#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
++ at /home/teawater/kernel2/linux/fs/read_write.c:368
++#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
++ at /home/teawater/kernel2/linux/fs/read_write.c:469
++Backtrace stopped: not enough registers or memory available to unwind
further
++(gdb) up
++#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
++ at /home/teawater/kernel2/linux/fs/read_write.c:469
++469 ret = vfs_read(file, buf, count, &pos);
++(gdb) p ret
++$2 = -9
++You see that the caller of function vfs_read is sys_read. And the local
variable ret of sys_read is -9.
++
++Use $dump_stack to output stack dump through printk
++Because this way need parse the stack when tracing and call printk
inside, so it will be slow, unsafe, unclear and cannot access a lot of info
of call stack. So I suggest you use the prev way to do stack dump.
++KGTP has special trace state variable $dump_stack, "collect" it will let
Linux Kernel output stack dump through printk.
++Following example lets Linux Kernel show the stack dump of vfs_readdir:
++
++target remote /sys/kernel/debug/gtp
++trace vfs_readdir
++ commands
++ collect $dump_stack
++ end
++Then your kernel will printk like:
++
++[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
++[22779.208068] Call Trace:
++[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
++[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
++[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
++[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
++[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
++[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
++[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
++[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
++[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
++[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
++[22779.208112] [<c02db51f>] ?
__jbd2_journal_temp_unlink_buffer+0x2f/0x110
++[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
++[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
++[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
++[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
++[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
++[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
++[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
++[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
++[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
++[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
++[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
++How to use performance counters
++Performance counters are special hardware registers available on most
modern CPUs. These registers count the number of certain types of hw
events: such as instructions executed, cachemisses suffered, or branches
mis-predicted - without slowing down the kernel or applications. These
registers can also trigger interrupts when a threshold number of events
have passed - and can thus be used to profile the code that runs on that
CPU.
++The Linux Performance Counter subsystem called perf event can get the
value of performance counter. You can access it through KGTP perf event
trace state variables.
++Please goto read the file tools/perf/design.txt in Linux Kernel to get
more info about perf event.
++
++Define a perf event trace state variable
++Access an performance counter need define following trace state variable:
++
++"pe_cpu_"+tv_name Define the the CPU id of the performance counter.
++"pe_type_"+tv_name Define the the type of the performance counter.
++"pe_config_"+tv_name Define the the config of the performance counter.
++"pe_en_"+tv_name This the switch to enable or disable the
performance counter.
++ The performance counter is disable in default.
++"pe_val_"+tv_name Access this variable can get the value of the
performance counter.
++Define a per_cpu perf event trace state variable
++Define a per_cpu perf event trace state variable is same with define
HOWTO#Per_cpu_trace_state_variables.
++
++"p_pe_"+perf_event type+string+CPU_id
***The diff for this file has been truncated for email.***
=======================================
--- /dev/null
+++ /trunk/gtp_3.7_to_upstream.patch Sat Dec 1 00:38:08 2012
@@ -0,0 +1,16506 @@
+--- /dev/null
++++ b/Documentation/gtp/howto.txt
+@@ -0,0 +1,1525 @@
++ Linux Kernel GDB tracepoint module (KGTP)
++ =========================================
++ By Hui Zhu <teawater@xxxxxxxxx>
++ https://code.google.com/p/kgtp/wiki/HOWTO
++
++Update in 2012-09-20
++
++What is KGTP
++Get help or report issues about KGTP
++Preparatory work before use KGTP
++Linux kernel
++If your system use the Linux kernel that is built by yourself
++If your system use the Linux kernel from distribution
++Make sure current Linux kernel debug image is right
++Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
++Get KGTP
++Get KGTP through http
++Get KGTP through svn
++Config KGTP
++Compile KGTP
++Normal compile
++Compile KGTP with some special config
++Install and uninstall KGTP
++Use KGTP with DKMS
++Use KGTP patch for Linux kernel
++Install GDB for KGTP
++Howto use
++Exec it
++Make GDB connect to gtp
++GDB on the current machine
++If GDB on remote machine
++Add module symbols to GDB
++How to use getmod.py
++How to use getmod
++Access memory directly
++GDB tracepoint
++set tracepoint
++How to set tracepoint condition
++actions [num]
++Start and stop the tracepoint
++Enable and disable the tracepoint
++Use "tfind" select the entry inside the trace frame info
++Show and save the tracepoint
++Delete tracepoint
++Use tracepoint get register info from a point of kernel
++Use tracepoint get the value of variable from a point of kernel
++How to handle "Unsupported operator (null) (52) in expression."
++Show all the traced data of current frame
++Get status of tracepoint
++Set the trace buffer into a circular buffer
++Do not stop tracepoint when the GDB disconnects
++kprobes-optimization and the execution speed of tracepoint
++How to use trace state variables
++Simple trace state variables
++Per_cpu trace state variables
++Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
++Special trace state variable $no_self_trace
++Trace the function return with $kret
++Use $ignore_error and $last_errno to ignore the error of tstart
++Use $cooked_clock and $cooked_rdtsc the time without KGTP used
++Use $xtime_sec and $xtime_nsec get the timespec
++Howto backtrace (stack dump)
++Collect stack with $bt and use GDB command "backtrace"
++Collect stack of current function's caller with $_ret
++Use $dump_stack to output stack dump through printk
++How to use performance counters
++Define a perf event trace state variable
++Define a per_cpu perf event trace state variable
++The perf event type and config
++Enable and disable all the perf event in a CPU with $p_pe_en
++GDB scripts to help with set and get the perf event trace state variables
++Howto let tracepoint output value directly
++Switch collect to output the value directly
++Howto show a variable whose value has been optimized away
++Update your GCC
++How to get the function pointer point to
++If the debug info of the function pointer is not optimized out
++If the debug info of the function pointer is optimized out
++/sys/kernel/debug/gtpframe and offline debug
++How to use /sys/kernel/debug/gtpframe_pipe
++Get the frame info with GDB
++Get the frame info with cat
++Get the frame info with getframe
++Use $pipe_trace
++Use KGTP with user space program
++Read memory of user space program directly
++collect stack (for backtrace) of user space program in tracepoint
++How to use add-ons/hotcode.py
++How to add plugin in C
++API
++Example
++How to use
++What is KGTP
++KGTP is a flexible , lightweight and realtime Linux debugger and tracer.
++To use it, you don't need patch or rebuild the Linux kernel. Just build
KGTP module and insmod it is OK.
++
++It makes Linux Kernel supply a GDB remote debug interface. Then GDB in
current machine or remote machine can debug and trace Linux through GDB
tracepoint and some other functions without stopping the Linux Kernel.
++And even if the board doesn't have GDB on it and doesn't have interface
for remote debug. It can debug the Linux Kernel using offline debug (See
HOWTO#/sys/kernel/debug/gtpframe_and_offline_debug).
++KGTP supports X86-32, X86-64, MIPS and ARM.
++KGTP is tested on Linux kernel 2.6.18 to upstream.
++And it can work with Android (See HowToUseKGTPinAndroid).
++
++For new user of KGTP, please go to see Quickstart.
++
++Please go to UPDATE to get more info about KGTP update.
++
++Get help or report issues about KGTP
++You can post it to http://code.google.com/p/kgtp/issues/list, write Email
to kgtp@xxxxxxxxxxxxx or write Email to teawater@xxxxxxxxx .
++The KGTP team will try our best to help you.
++
++Preparatory work before use KGTP
++Linux kernel
++If your system use the Linux kernel that is built by yourself
++To use KGTP, your Linux kernel need open following options:
++
++General setup --->
++ [*] Kprobes
++
++[*] Enable loadable module support --->
++
++Kernel hacking --->
++ [*] Debug Filesystem
++ [*] Compile the kernel with debug info
++If your system use the Linux kernel from distribution
++You need install some Linux kernel package.
++
++Ubuntu
++Install the Linux kernel debug image
++1) Add debug source to the sources list of Ubuntu.
++
++Create an /etc/apt/sources.list.d/ddebs.list by running the following
line at a terminal:
++echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted
universe multiverse" | \
++sudo tee -a /etc/apt/sources.list.d/ddebs.list
++Stable releases (not alphas and betas) require three more lines adding to
the same file, which is done by the following terminal command:
++echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main
restricted universe multiverse
++deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted
universe multiverse
++deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted
universe multiverse" | \
++sudo tee -a /etc/apt/sources.list.d/ddebs.list
++Import the debug symbol archive signing key:
++sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
++Then run:
++sudo apt-get update
++
++2) Get Linux kernel debug image.
++
++sudo apt-get install linux-image-$(uname -r)-dbgsym
++Then you can find Linux kernel debug image
in "/usr/lib/debug/boot/vmlinux-$(uname -r)".
++
++Install the Linux kernel headers
++sudo apt-get install linux-headers-generic
++Install the Linux kernel source
++Install the source package:
++sudo apt-get install linux-source
++Uncompress the source package:
++sudo mkdir -p /build/buildd/
++sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2
-C /build/buildd/
++sudo rm -rf /build/buildd/linux-$(uname -r | sed 's/-.*//')
++sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//')
/build/buildd/linux-$(uname -r | sed 's/-.*//')
++Fedora
++Install the Linux kernel debug image
++sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo
++Then you can find Linux kernel debug image
in "/usr/lib/debug/lib/modules/$(uname -r)/vmlinux".
++
++Install the Linux kernel devel package
++sudo yum install kernel-devel-$(uname -r)
++Others
++You need install the Linux kernel debug image package and the Linux
Kernel source.
++
++Make sure current Linux kernel debug image is right
++GDB open the right Linux kernel debug image is an very important because
GDB will get the debug info and address info from it. So before you use
KGTP, please do the check to make sure about it.
++There are 2 ways to do the check, what I suggest is do both of them to
make sure Linux kernel debug image is right.
++
++Please note that if you determine you use the right Linux kernel debug
image, but cannot pass these ways. Please see
HOWTO#Handle_the_issue_that_Linux_kernel_debug_image's_address_in.
++
++Use /proc/kallsyms
++In the system that its Linux kernel is what you want to trace, use
following command to get the address of sys_read and sys_write:
++
++sudo cat /proc/kallsyms | grep sys_read
++ffffffff8117a520 T sys_read
++sudo cat /proc/kallsyms | grep sys_write
++ffffffff8117a5b0 T sys_write
++Then we can get that the address of sys_read is 0xffffffff8117a520 and
the address of sys_write is 0xffffffff8117a5b0.
++After that use GDB get address of sys_read and sys_write from Linux
kernel debug image:
++
++gdb ./vmlinux
++(gdb) p sys_read
++$1 = {long int (unsigned int, char *, size_t)} 0xffffffff8117a520
<sys_read>
++(gdb) p sys_write
++$2 = {long int (unsigned int, const char *, size_t)} 0xffffffff8117a5b0
<sys_write>
++The address of sys_read and sys_write is same, so the Linux kernel debug
image is right.
++
++Use linux_banner
++sudo gdb ./vmlinux
++(gdb) p linux_banner
++$1 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
++This linux_banner is the kernel info inside the Linux kernel debug image.
++After that, connect to KGTP following the way in
HOWTO#Make_GDB_connect_to_gtp connect to KGTP and print linux_banner again.
++
++(gdb) target remote /sys/kernel/debug/gtp
++Remote debugging using /sys/kernel/debug/gtp
++0x0000000000000000 in irq_stack_union ()
++(gdb) p linux_banner
++$2 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
++This linux_banner is the kernel info that Linux kernel that KGTP is
tracing. If it is same with the prev kernel info, the Linux kernel debug
image is right.
++
++Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
++In X86_32, you will found that the Linux kernel debug image's address
info is not same with Linux kernel when it running through the ways in
HOWTO#Make_sure_current_Linux_kernel_debug_image_is_right. And you
determine the Linux kernel debug image is right.
++This issue is because:
++
++Processor type and features --->
++ (0x1000000) Physical address where the kernel is loaded
++ (0x100000) Alignment value to which kernel should be aligned
++The values of these two options are different. Please note that
the "Physical address where the kernel is loaded" is not showed in config
sometimes. You can get its value through search "PHYSICAL_START".
++
++You can handle this issue through change "Alignment value to which kernel
should be aligned" same with "Physical address where the kernel is loaded".
++This issue doesn't affect X86_64.
++
++Get KGTP
++Get KGTP through http
++Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to
download the package.
++
++Get KGTP through svn
++Some people have trouble with access to KGTP website. You can access kgtp
through svn:
++
++svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only
++kgtp-read-only/tags/ Present for each release of KGTP.
++kgtp-read-only/trunk/ Present for the main trunk of KGTP.
++
++Config KGTP
++Following part is the default config of KGTP inside the Makefile. With
this config, KGTP will build together with current kernel that running on
this machine.
++
++KERNELDIR := /lib/modules/`uname -r`/build
++CROSS_COMPILE :=
++KERELDIR is set to the directory which holds the kernel you want to build
for. By default, it is set to the kernel that you are running.
++Please note that this directory should be Linux kernel build directory or
linux-headers directory but not the source directory but not the Linux
kernel source directory. And the Linux kernel build directory should be
used after build successful.
++CROSS_COMPILE is set to the prefix name of compiler that you want to
build KGTP. Empty to compile with your default compiler.
++ARCH is the architecture.
++
++Or you can choose which kernel you want build with and which compiler you
want use by change Makefile.
++For example:
++
++KERNELDIR := /home/teawater/kernel/bamd64
++CROSS_COMPILE :=x86_64-glibc_std-
++ARCH := x86_64
++KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will use
x86_64-glibc_std-gcc.
++
++Compile KGTP
++Normal compile
++cd kgtp/
++make
++In some build environment (for example Android) will get some error with
user space program getmod or getframe. Please ignore this error and use the
gtp.ko in this directory.
++
++Compile KGTP with some special config
++Most of time, KGTP can auto select right options to build with Various
versions of Linux kernel.
++But if you want config special options with yourself, you can read
following part:
++With this option, KGTP will not auto select any build options.
++
++make AUTO=0
++With this option, KGTP will use simple frame instead of KGTP ring buffer.
++The simple frame doesn't support gtpframe_pipe. It just for debug KGTP.
++
++make AUTO=0 FRAME_SIMPLE=1
++With this option, $clock will return rdtsc value instead of local_clock.
++
++make AUTO=0 CLOCK_CYCLE=1
++With this option, KGTP will use procfs instead of debugfs.
++
++make AUTO=0 USE_PROC=1
++The options can use together, for example:
++
++make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1
++Install and uninstall KGTP
++KGTP don't need to be install because it can insmod directly inside its
directory (See HOWTO#Exec_it). But if you need, you can install it to your
system.
++Install:
++
++cd kgtp/
++sudo make install
++Uninstall:
++
++cd kgtp/
++sudo make uninstall
++Use KGTP with DKMS
++You can use KGTP with DKMS if you want it.
++Following commands will copy the files of KGTP to the directory that DKMS
need.
++
++cd kgtp/
++sudo make dkms
++Then you can use DKMS commands to control KGTP. Please goto
http://linux.dell.com/dkms/manpage.html to see how to use DKMS.
++
++Use KGTP patch for Linux kernel
++Most of time, you don't need KGTP patch because KGTP can build as a LKM
and very easy to use. But to help some people include KGTP to them special
Linux Kernel tree, KGTP supply patches for Linux kernel.
++In the KGTP directory:
++gtp_3.0_to_upstream.patch is the patch for Linux kernel from 3.0 to
upstream.
++gtp_2.6.39.patch is the patch for Linux kernel 2.6.39.
++gtp_2.6.33_to_2.6.38.patch is the patch for Linux kernel from 2.6.33 to
2.6.38.
++gtp_2.6.20_to_2.6.32.patch is the patch for Linux kernel from 2.6.20 to
2.6.32.
++gtp_older_to_2.6.19.patch is the patch for Linux kernel 2.6.19 and older
version.
++Install GDB for KGTP
++The GDB that older than 7.3 have some bugs of tracepoint. And some
functions of GDB are not very well.
++So if your GDB is older than 7.3 please go to
https://code.google.com/p/gdbt/ to get howto install GDB for KGTP.
++If you have issue about GDB please get help according to
HOWTO#Report_issues_about_KGTP.
++
++Howto use
++Exec it
++If you have installed KGTP in your system, you can:
++
++sudo modprobe gtp
++Or you can use the kgtp module in the directory.
++
++cd kgtp/
++sudo insmod gtp.ko
++Make GDB connect to gtp
++GDB on the current machine
++sudo gdb ./vmlinux
++(gdb) target remote /sys/kernel/debug/gtp
++Remote debugging using /sys/kernel/debug/gtp
++0x0000000000000000 in ?? ()
++After that, you can begin to use GDB command trace and debug the Linux
Kernel.
++
++If GDB on remote machine
++#Open the KGTP interface in current machine.
++sudo su
++nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp
++(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old
version netcat.)
++#Let gdb connect to the port 1234
++gdb ./vmlinux
++(gdb) target remote xxx.xxx.xxx.xxx:1234
++After that, you can begin to use GDB command trace and debug the Linux
Kernel.
++
++Add module symbols to GDB
++Sometimes you need to add a Linux kernel module's symbols to GDB to debug
it.
++Add symbols with hand is not very easy, so KGTP package include an GDB
python script "getmod.py" and a program "getmod" can help you.
++
++How to use getmod.py
++Connect to KGTP before use the getmod.py.
++
++(gdb) source ~/kgtp/getmod.py
++Then this script will auto load the Linux kernel module's symbols to GDB.
++
++How to use getmod
++"getmod" is written by C so you can use it anywhere even if in an
embedded environment.
++For example:
++
++#Following command save Linux Kernel module info to the file ~/tmp/mi in
GDB
++#command format.
++sudo getmod >~/tmp/mi
++#in gdb part:
++(gdb) source ~/tmp/mi
++add symbol table from
file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
++ .text_addr = 0xf80de000
++ .note.gnu.build-id_addr = 0xf80de088
++ .exit.text_addr = 0xf80de074
++ .init.text_addr = 0xf8118000
++ .rodata.str1.1_addr = 0xf80de0ac
++ .rodata_addr = 0xf80de0c0
++ __mcount_loc_addr = 0xf80de9c0
++ .data_addr = 0xf80de9e0
++ .gnu.linkonce.this_module_addr = 0xf80dea00
++#After this GDB command, all the Linux Kernel module info is loaded into
GDB.
++If you use remote debug or offline debug, maybe you need change the base
directory. Following example is for it.
++
++#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r
/home/teawater/kernel/b26
++sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi
++Access memory directly
++After connect the KGTP, you can access most of memory directly.
++For example, you can access to "jiffies_64" with following command:
++
++(gdb) p jiffies_64
++Or you can access to the first entry of "static LIST_HEAD(modules)" with
following command:
++
++(gdb) p *((struct module *)((char *)modules->next - ((size_t) &(((struct
module *)0)->list))))
++Or you can access to the CPU0 memory info of "DEFINE_PER_CPU(struct
device *, mce_device);":`
++
++p *(struct device *)(__per_cpu_offset[0]+(uint64_t)(&mce_device))
++If you want show more than one variables with one GDB command, please use
following example:
++
++(gdb) printf "%4d %4d %4d %4d %4d %4d %18d %lu\n", this_rq->cpu,
this_rq->nr_running, this_rq->nr_uninterruptible, nr_active,
calc_load_tasks->counter, this_rq->calc_load_active, delta,
this_rq->calc_load_update
++2 1 0 0 0 0 673538312 717077240
++GDB tracepoint
++Tracepoint is that GDB define some addresses and some actions and put
them to the target (KGTP). After tracepoint start, , KGTP will do these
actions (Some of them will collect data and save them to tracepoint frame
buffer) when Linux kernel execution to there addresses. After that, Linux
kernel will keep execution.
++KGTP supply some interfaces that GDB or other programe can take the data
of tracepoint frame buffer out to parse.
++About these interfaces, this doc have introduced "/sys/kernel/debug/gtp".
And will introduce "/sys/kernel/debug/gtpframe"
and "/sys/kernel/debug/gtpframe_pipe" later.
++
++Doc of GDB tracepoint in
http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html.
++
++set tracepoint
++The trace command is very similar to the break command. Its argument
location can be a source line, a function name, or an address in the target
program. The trace command defines a tracepoint, which is a address or some
addresses that KGTP do some actions in it.
++Here are some examples of using the trace command:
++
++(gdb) trace foo.c:121 // a source file and line number
++
++(gdb) trace +2 // 2 lines forward
++
++(gdb) trace my_function // first source line of function
++
++(gdb) trace *my_function // EXACT start address of function
++
++(gdb) trace *0x2117c4 // an address
++Howto handle the function is there but set tracepoint on it got fail
++GCC will inline some static function to increase the performance. You
cannot set tracepoint on the function name because object file doesn't have
symbol of inline function.
++You can use "trace filename:line" to set tracepoint on it.
++
++How to set tracepoint condition
++http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html
++Like breakpoints, we can set conditions on tracepoints. The speed of
tracepoints is faster than breakpoints because KGTP can do all the
condition checks.
++For example:
++
++(gdb) trace handle_irq if (irq == 47)
++This action of tracepoint 1 will work only when irq number is 47.
++
++And you can use GDB command "condition" to specify the condition of a
tracepoint. GDB command "condition N COND" will specify tracepoint number N
to trace only if COND is true.
++For example:
++
++(gdb) trace handle_irq
++(gdb) condition 1 (irq == 47)
++GDB command "info tracepoint" will show the ID of the tracepoint.
++
++Value of $bpnum is the last ID of GDB tracepoint. Then you can use GDB
command "condtion" set the condition of last tracepoint without get its ID.
For example:
++
++(gdb) trace handle_irq
++(gdb) condition $bpnum (irq == 47)
++actions [num]
++This command will prompt for a list of actions to be taken when the
tracepoint is hit. If the tracepoint number num is not specified, this
command sets the actions for the one that was most recently defined (so
that you can define a tracepoint and then say actions without bothering
about its number). You specify the actions themselves on the following
lines, one action at a time, and terminate the actions list with a line
containing just end. So far, the only defined actions are collect, teval,
and while-stepping.
++
++collect expr1, expr2, ...
++Collect values of the given expressions when the tracepoint is hit. This
command accepts a comma-separated list of any valid expressions. In
addition to global, static, or local variables, the following special
arguments are supported:
++
++$regs Collect all registers.
++$args Collect all function arguments.
++$locals Collect all local variables.
++Please note that collect an pointer (collect ptr) will just collect the
address of this pointer. Add a * before ptr will make action collect the
data that pointer point to(collect *ptr).
++
++teval expr1, expr2, ...
++Evaluate the given expressions when the tracepoint is hit. This command
accepts a comma-separated list of expressions. The results are discarded,
so this is mainly useful for assigning values to trace state variables (see
HOWTO#Simple_trace_state_variables) without adding those values to the
trace buffer, as would be the case if the collect action were used.
++
++while-stepping n
++Please goto
HOWTO#If_the_debug_info_of_the_function_pointer_is_optimized_out see howto
use it.
++
++Start and stop the tracepoint
++Tracepoint will exec actions only when it is starting use this GDB
command:
++
++(gdb) tstart
++It will stop by this GDB command:
++
++(gdb) tstop
++Enable and disable the tracepoint
++Like breakpoint, tracepoint can be control by GDB commands "enable"
and "disable". But please note that it only useful when tracepoint stop.
++
++Use "tfind" select the entry inside the trace frame info
++GDB command "tfind" is used to select a entry of trace frame bufffer when
tracepoint stop.
++When GDB inside "tfind" mode, it will just show the values of this entry
that the tracepoint action collect. So it will output some error when print
some values that action doesn't collect for example the argument of
function. That is not a bug, please don't worry about it.
++Use "tfind" again will select next entry. "tfind id" will select entry id.
++To go back to normal mode, please use GDB command "tfind -1". Please goto
http://sourceware.org/gdb/current/onlinedocs/gdb/tfind.html get more info
about it.
++
++Show and save the tracepoint
++You can use GDB command "info tracepoints" to show all the tracepoints.
++You can use GDB command "save tracepoints filename" to save the commands
that setup the tracepoints and actions into file filename. Then you use use
GDB commands "source filename" to setup this tracepints again.
++
++Delete tracepoint
++GDB command "delete id" will delete tracepoint id. If "delete" without
argument, it will delete all the tracepoint.
++
++Use tracepoint get register info from a point of kernel
++The following is an example that records the value of all registers
when "vfs_readdir" is called.
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xc01a1ac0: file
++/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect $reg
++>end
++(gdb) tstart
++(gdb) shell ls
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
<filldir64>,
++ buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
++23 /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
directory.
++ in /home/teawater/kernel/linux-2.6/fs/readdir.c
++(gdb) info reg
++eax 0xc5528d00 -984445696
++ecx 0xc0d09f90 -1060069488
++edx 0xc01a1900 -1072031488
++ebx 0xfffffff7 -9
++esp 0xc0d09f8c 0xc0d09f8c
++ebp 0x0 0x0
++esi 0x8061480 134616192
++edi 0xc5528d00 -984445696
++eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
++eflags 0x286 [ PF SF IF ]
++cs 0x60 96
++ss 0x8061480 134616192
++ds 0x7b 123
++es 0x7b 123
++fs 0x0 0
++gs 0x0 0
++(gdb) tfind
++Found trace frame 1, tracepoint 1
++0xc01a1ac1 23 in /home/teawater/kernel/linux-2.6/fs/readdir.c
++(gdb) info reg
++eax 0xc5528d00 -984445696
++ecx 0xc0d09f90 -1060069488
++edx 0xc01a1900 -1072031488
++ebx 0xfffffff7 -9
++esp 0xc0d09f8c 0xc0d09f8c
++ebp 0x0 0x0
++esi 0x8061480 134616192
++edi 0xc5528d00 -984445696
++eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
++eflags 0x286 [ PF SF IF ]
++cs 0x60 96
++ss 0x8061480 134616192
++ds 0x7b 123
++es 0x7b 123
++fs 0x0 0
++gs 0x0 0
++Use tracepoint get the value of variable from a point of kernel
++The following is an example that records the value of "jiffies_64" when
the function "vfs_readdir" is called:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xc01ed740: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect jiffies_64
++>collect file->f_path.dentry->d_iname
++>end
++(gdb) tstart
++(gdb) shell ls
++arch drivers include kernel mm Module.symvers
security System.map virt
++block firmware init lib modules.builtin net
sound t vmlinux
++crypto fs ipc Makefile modules.order scripts
source usr vmlinux.o
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580
<filldir64>, buf=0xd6dfdf90)
++ at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
++24 {
++(gdb) p jiffies_64
++$1 = 4297248706
++(gdb) p file->f_path.dentry->d_iname
++$1 = "b26", '\000' <repeats 28 times>
++How to handle "Unsupported operator (null) (52) in expression."
++If you use condition about string, you will got this error when you
call "tstart".
++To handle it, you can convent the char to int to handle this issue, for
example:
++
++(gdb) p/x 'A'
++$4 = 0x41
++(gdb) condition 1 (buf[0] == 0x41)
++Show all the traced data of current frame
++After use "tfind" select an entry, you can use "tdump" to do it.
++
++(gdb) tdump
++Data collected at tracepoint 1, trace frame 0:
++$cr = void
++file->f_path.dentry->d_iname
= "gtp\000.google.chrome.g05ZYO\000\235\337\000\000\000\000\200\067k\364\200\067",
<incomplete sequence \364>
++jiffies_64 = 4319751455
++Get status of tracepoint
++Please use GDB command "tstatus".
++
++Set the trace buffer into a circular buffer
++http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
++The frame buffer is not a circular buffer by default. When the buffer is
full, the tracepoint will stop.
++Following command will set frame buffer to a circular buffer. When the
buffer is full, it will auto discard traceframes (oldest first) and keep
trace.
++
++(gdb) set circular-trace-buffer on
++Do not stop tracepoint when the GDB disconnects
++http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
++KGTP will stop tracepoint and delete the trace frame when GDB disconnects
with it by default.
++Following command will open the KGTP disconnect-trace. After that, when
GDB disconnects with KGTP, KGTP will not stop tracepoint. And after GDB
reconnects to KGTP, it can keep control of KGTP like nothing happened.
++
++(gdb) set disconnected-tracing on
++kprobes-optimization and the execution speed of tracepoint
++The tracepoint is execution together with Linux kernel. So it speed will
affect the speed the system.
++The KGTP tracepoint base on Linux kernel kprobe. Because the normal
kprobe base on breakpoint instruction, so it is not very fast.
++
++But if arch of kernel is X86_64 or X86_32 and kernel config didn't
open "Preemptible Kernel" (PREEMPT), the kprobe is speed up by
kprobes-optimization (CONFIG_OPTPROBES) that make kprobe very fast.
++To make sure about that, you can use following command in terminal:
++
++sysctl -A | grep kprobe
++debug.kprobes-optimization = 1
++That means that your kernel support kprobes-optimization.
++Please note that some KGTP functions will make this tracepoint use simple
kprobe even if this Kernel support kprobes-optimization. This doc will add
note when introduce these functions. Please avoid using them when you
really care about the tracepoint speed.
++
++How to use trace state variables
++http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html
++Tracepoints have variables that can be traced directly, or used in
tracepoint conditions.
++Please note that just GDB 7.2.1 and later versions support use trace
state variables directly, the old version of GDB can show the value of
trace state variables through command "info tvariables".
++
++Simple trace state variables
++Define a trace state variable $c.
++
++(gdb) tvariable $c
++Trace state variable $c is created with initial value 0. The following
action uses $c to count how many irqs happened in the kernel.
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace handle_irq
++(gdb) actions
++Enter actions for tracepoint 3, one per line.
++End with a line saying just "end".
++>collect $c #Save current value of $c to the trace frame buffer.
++>teval $c=$c+1 #Increase the $c.
++>end
++Also, you can set a value of variable to trace state variable, but don't
forget covert variable to "uint64_t".
++
++>teval $c=(uint64_t)a
++You can get the current value of $c while the trace is running or stopped.
++
++(gdb) tstart
++(gdb) info tvariables
++$c 0 31554
++(gdb) p $c
++$5 = 33652
++(gdb) tstop
++(gdb) p $c
++$9 = 105559
++When using tfind, you can parse the trace frame buffer. If the value of a
trace state variable is collected, you can parse it out.
++
++(gdb) tstop
++(gdb) tfind
++(gdb) info tvariables
++$c 0 0
++(gdb) p $c
++$6 = 0
++(gdb) tfind 100
++(gdb) p $c
++$7 = 100
++If need, the tracepoint action that access the simple trace state
variables will auto lock a spin lock for trace state variables. So it can
handle race condition issue about trace state variables.
++The following example is OK even if it running a machine that have more
than one CPU.
++
++>teval $c=$c+1
++Per_cpu trace state variables
++Per_cpu trace state variables are special simple trace state variables.
++When tracepoint action access to it, it will access to this CPU special
trace state variables.
++It have 2 advantages:
++1. The tracepoint actions that access to per_cpu trace state variables
don't have the race conditon issue. So it don't need lock the spin lock for
trace state variables. It is faster than simple trace state variables on
multi-core machine.
++2. Write the action that count some CPU special thing with it is easier
than simple trace state variables.
++
++How to define
++Per_cpu trace state variables have two types:
++
++Local CPU variables
++"per_cpu_"+string
++or
++
++"p_"+string
++For example:
++
++(gdb) tvariable $p_count
++When access this trace state variable in tracepoint actions, it will
return the variable's value of CPU that this tracepoint actions running on.
++
++CPU id variables
++"per_cpu_"+string+CPU_id
++or
++
++"p_"+string+CPU_id
++For example:
++
++(gdb) tvariable $p_count0
++(gdb) tvariable $p_count1
++(gdb) tvariable $p_count2
++(gdb) tvariable $p_count3
++When access this trace state variable in tracepoint actions or GDB
command line, it will return the variable's value of CPU CPU_id.
++Follow example can auto define a CPU id variables for each CPU of this
machine. (Please note that need let GDB connect to KGTP before use these
commands.)
++
++(gdb) set $tmp=0
++(gdb) while $tmp<$cpu_number
++ >eval "tvariable $p_count%d",$tmp
++ >set $tmp=$tmp+1
++ >end
++Example 1
++This example define a tracepoint that count the times that call vfs_read
of each CPU.
++
++tvariable $p_count
++set $tmp=0
++while $tmp<$cpu_number
++ eval "tvariable $p_count%d",$tmp
++ set $tmp=$tmp+1
++ end
++trace vfs_read
++actions
++ teval $p_count=$p_count+1
++ end
++Then you can show how many vfs_read in each CPU after "tstart":
++
++(gdb) p $p_count0
++$3 = 44802
++(gdb) p $p_count1
++$4 = 55272
++(gdb) p $p_count2
++$5 = 102085
++(gdb) p $p_count3
++Example 2
++This example record stack dump of the function that close IRQ longest
time of each CPU.
++
++set pagination off
++
++tvariable $bt=1024
++tvariable $p_count
++tvariable $p_cc
++set $tmp=0
++while $tmp<$cpu_number
++eval "tvariable $p_cc%d",$tmp
++set $tmp=$tmp+1
++end
++
++tvariable $ignore_error=1
++
++trace arch_local_irq_disable
++ commands
++ teval $p_count=$clock
++ end
++trace arch_local_irq_enable if ($p_count && $p_cc < $clock - $p_count)
++ commands
++ teval $p_cc = $clock - $p_count
++ collect $bt
++ collect $p_cc
++ teval $p_count=0
++ end
++
++enable
++set pagination on
++Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
++KGTP special trace state variables $current_task, $current_thread_info,
$cpu_id and $clock can very easy to access to some special value. You can
see them when GDB connects to the KGTP. You can use them in tracepoint
conditions or actions.
++Access $current_task in tracepoint condition and action will get that
returns of get_current().
++Access $current_task_pid in tracepoint condition and action will get that
returns of get_current()->pid.
++Access $current_thread_info in tracepoint condition and action will get
that returns of current_thread_info().
++Access $cpu_id in tracepoint condition and action will get that returns
of smp_processor_id().
++Access $clock in tracepoint condition and action will get that returns of
local_clock() that return the timestamp in nanoseconds.
++$rdtsc is only available on X86 and X86_64 architecture. Access it in
anytime will get current value of TSC with instruction RDTSC.
++Access $hardirq_count in tracepoint condition and action will get that
returns of hardirq_count().
++Access $softirq_count in tracepoint condition and action will get that
returns of softirq_count().
++Access $irq_count in tracepoint condition and action will get that
returns of irq_count().
++And KGTP has other special trace state variables $dump_stack,
$printk_level, $printk_format and $printk_tmp. All of them output their
values directly, as can be seen in
HOWTO#Howto_let_tracepoint_output_value_directly.
++The following example counts in $c how many vfs_read calls that process
16663 does and collects the struct thread_info of current task:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid ==
16663)
++(gdb) tvariable $c
++(gdb) actions
++Enter actions for tracepoint 4, one per line.
++End with a line saying just "end".
++>teval $c=$c+1
++>collect (*(struct thread_info *)$current_thread_info)
++>end
++(gdb) tstart
++(gdb) info tvariables
++Name Initial Current
++$c 0 184
++$current_task 0 <unknown>
++$current_thread_info 0 <unknown>
++$cpu_id 0 <unknown>
++(gdb) tstop
++(gdb) tfind
++(gdb) p *(struct thread_info *)$current_thread_info
++$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status =
0, cpu = 1, preempt_count = 2, addr_limit = {
++ seg = 4294967295}, restart_block = {fn = 0xc0159fb0
<do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11,
++ arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags
= 1, bitset = 78, time = 977063750,
++ uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb,
expires = 335007449089}, poll = {
++ ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78,
tv_nsec = 977063750}}},
++ sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack =
0xef340044 "", uaccess_err = 0}
++Another example shows how much sys_read() executes in each CPU.
++
++(gdb) tvariable $c0
++(gdb) tvariable $c1
++(gdb) trace sys_read
++(gdb) condition $bpnum ($cpu_id == 0)
++(gdb) actions
++>teval $c0=$c0+1
++>end
++(gdb) trace sys_read
++(gdb) condition $bpnum ($cpu_id == 1)
++(gdb) actions
++>teval $c1=$c1+1
++>end
++(gdb) info tvariables
++Name Initial Current
++$current_task 0 <unknown>
++$cpu_id 0 <unknown>
++$c0 0 3255
++$c1 0 1904
++sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. Please note
that this example just to howto use $cpu_id. Actially, this example use
per_cpu trace state variables is better.
++
++Special trace state variable $no_self_trace
++$no_self_trace is different with the special trace state variables in the
previous section. It is used to control the behavior of tracepoint.
++If the action of a tracepoint include a command access to the
$no_self_trace. The tracepoint will not trace anything if the current_task
is the a KGTP self process (GDB, netcat, getframe or some others process
that access to the interface of KGTP).
++For example, if we want trace vfs_read or something that have process
context, and we don't want trace the operation of KGTP self process. Add
following command to the action:
++
++>collect $no_self_trace
++Please note that the code that doesn't about process context (Irq
handler, softirq) doesn't need set this variable.
++
++Trace the function return with $kret
++Sometime, set the tracepoint to the end of function is hard because the
Kernel is compiled with optimization. At this time, you can get help from
$kret.
++$kret is a special trace state variable like $no_self_trace. When you set
value of it inside the action of tracepoint, this tracepoint be set with
kretprobe instead of kprobe. Then it can trace the end of this function.
++Please note that this tracepoint must set in the first address of the
function in format "function_name".
++
++Following part is an example:
++
++#"*(function_name)" format can make certain that GDB send the first
address of function to KGTP.
++(gdb) trace *vfs_read
++(gdb) actions
++>teval $kret=0
++#Following part you can set commands that you want.
++Use $ignore_error and $last_errno to ignore the error of tstart
++If KGTP got any error of tstart, this command will get fail.
++But sometime we need ignore this error and let KGTP keep work. For
example: If you set tracepoint on the inline function spin_lock. This
tracepoint will be set to a lot of addresses that some of them cannot be
set kprobe. It will make tstart get fail. You can use "$ignore_error"
ignore this error.
++And the last error number will available in "$last_errno".
++
++(gdb) tvariable $ignore_error=1
++This command will open ignore.
++
++(gdb) tvariable $ignore_error=0
++This command will close ignore.
++
++Use $cooked_clock and $cooked_rdtsc the time without KGTP used
++Access these two trace state variables can get the time without KGTP
used. Then we can get more close to really time that a part of code used
even if the actions of tracepoint is very complex. They will be introduce
in Cookbook (coming soon).
++
++Use $xtime_sec and $xtime_nsec get the timespec
++Access these two trace state variables will return the time of day in a
timespec that use getnstimeofday.
++$xtime_sec will access to the second part of a timespec.
++$xtime_nsec will access to the nanosecond part of a timespec.
++
++Howto backtrace (stack dump)
++Each time your program performs a function call, information about the
call is generated. That information includes the location of the call in
your program, the arguments of the call, and the local variables of the
function being called. The information is saved in a block of data called a
stack frame. The stack frames are allocated in a region of memory called
the call stack.
++
++Collect stack with $bt and use GDB command "backtrace"
++Because this way is faster (just collect the stack when trace) and parse
out most of info inside the call stack (it can show all the stack info that
I introduce). So I suggest you use this way to do the stack dump.
++First we need add the collect the stack command to the tracepoint action.
++The general collect the stack command in GDB tracepoint is: In x86_32,
following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$esp@512
++In x86_64, following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$rsp@512
++In MIPS or ARM, following command will collect 512 bytes of stack.
++>collect *(unsigned char *)$sp@512
++These commands is so hard to remember, and the different arch need
different command.
++KGTP have an special tracepoint trace state variable $bt. If tracepoint
action access it, KGTP will auto collect the $bt size (default value is
512) stack. For example, this command will collect 512 bytes stack memory:
++
++>collect $bt
++If you want to change size of $bt, you can use following GDB command
before "tstart":
++
++(gdb) tvariable $bt=1024
++Following part is an example about howto collect stack and howto use GDB
parse it:
++
++(gdb) target remote /sys/kernel/debug/gtp
++(gdb) trace vfs_readdir
++Tracepoint 1 at 0xffffffff8118c300: file
/home/teawater/kernel2/linux/fs/readdir.c, line 24.
++(gdb) actions
++Enter actions for tracepoint 1, one per line.
++End with a line saying just "end".
++>collect $bt
++>end
++(gdb) tstart
++(gdb) shell ls
++1 crypto fs include kernel mm
Module.symvers security System.map vmlinux
++arch drivers hotcode.html init lib modules.builtin
net sound usr vmlinux.o
++block firmware hotcode.html~ ipc Makefile modules.order
scripts source virt
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 1
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++24 {
++(gdb) bt
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
++#2 <signal handler called>
++#3 0x00007f00253848a5 in ?? ()
++#4 0x00003efd32cddfc9 in ?? ()
++#5 0x00002c15b7d04101 in ?? ()
++#6 0x000019c0c5704bf1 in ?? ()
++#7 0x0000000900000000 in ?? ()
++#8 0x000009988cc8d269 in ?? ()
++#9 0x000009988cc9b8d1 in ?? ()
++#10 0x0000000000000000 in ?? ()
++(gdb) up
++#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
++214 error = vfs_readdir(file, filldir, &buf);
++(gdb) p buf
++$1 = {current_dir = 0x1398c58, previous = 0x0, count = 32768, error = 0}
++(gdb) p error
++$3 = -9
++(gdb) frame 0
++#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
++ at /home/teawater/kernel2/linux/fs/readdir.c:24
++24 {
++From this example, we can see some GDB commands that parse the the call
stack:
++bt is the alias of GDB commands backtrace that print a backtrace of the
entire stack: one line per frame for all frames in the stack.
++up n is move n frames up the stack. For positive numbers n, this advances
toward the outermost frame, to higher frame numbers, to frames that have
existed longer. n defaults to one.
++down n is move n frames down the stack. For positive numbers n, this
advances toward the innermost frame, to lower frame numbers, to frames that
were created more recently. n defaults to one. You may abbreviate down as
do.
++frame n is select frame number n. Recall that frame zero is the innermost
(currently executing) frame, frame one is the frame that called the
innermost one, and so on. The highest-numbered frame is the one for main.
++You can see that when you use up, down or frame to the different calll
stack frame, you can output the value of the arguments and local variables
of different call stack frame.
++To get the more info about howto use GDB parse the call stack, please see
http://sourceware.org/gdb/current/onlinedocs/gdb/Stack.html
++Collect stack of current function's caller with $_ret
++If you just want to collect stack of current function's caller, please
use $_ret.
++Please note that set the tracepoint that collect $_ret cannot in the
first address of function.
++For example:
++
++(gdb) list vfs_read
++360 }
++361
++362 EXPORT_SYMBOL(do_sync_read);
++363
++364 ssize_t vfs_read(struct file *file, char __user *buf, size_t
count, loff_t *pos)
++365 {
++366 ssize_t ret;
++367
++368 if (!(file->f_mode & FMODE_READ))
++369 return -EBADF;
++(gdb) trace 368
++Tracepoint 2 at 0xffffffff8117a244: file
/home/teawater/kernel2/linux/fs/read_write.c, line 368.
++(gdb) actions
++Enter actions for tracepoint 2, one per line.
++End with a line saying just "end".
++>collect $_ret
++>end
++(gdb) tstart
++(gdb) tstop
++(gdb) tfind
++Found trace frame 0, tracepoint 2
++#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
++ at /home/teawater/kernel2/linux/fs/read_write.c:368
++368 if (!(file->f_mode & FMODE_READ))
++(gdb) bt
++#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
++ at /home/teawater/kernel2/linux/fs/read_write.c:368
++#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
++ at /home/teawater/kernel2/linux/fs/read_write.c:469
++Backtrace stopped: not enough registers or memory available to unwind
further
++(gdb) up
++#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
++ at /home/teawater/kernel2/linux/fs/read_write.c:469
++469 ret = vfs_read(file, buf, count, &pos);
++(gdb) p ret
++$2 = -9
++You see that the caller of function vfs_read is sys_read. And the local
variable ret of sys_read is -9.
++
++Use $dump_stack to output stack dump through printk
++Because this way need parse the stack when tracing and call printk
inside, so it will be slow, unsafe, unclear and cannot access a lot of info
of call stack. So I suggest you use the prev way to do stack dump.
++KGTP has special trace state variable $dump_stack, "collect" it will let
Linux Kernel output stack dump through printk.
++Following example lets Linux Kernel show the stack dump of vfs_readdir:
++
++target remote /sys/kernel/debug/gtp
++trace vfs_readdir
++ commands
++ collect $dump_stack
++ end
++Then your kernel will printk like:
++
++[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
++[22779.208068] Call Trace:
++[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
++[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
++[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
++[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
++[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
++[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
++[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
++[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
++[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
++[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
++[22779.208112] [<c02db51f>] ?
__jbd2_journal_temp_unlink_buffer+0x2f/0x110
++[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
++[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
++[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
++[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
++[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
++[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
++[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
++[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
++[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
++[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
++[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
++How to use performance counters
++Performance counters are special hardware registers available on most
modern CPUs. These registers count the number of certain types of hw
events: such as instructions executed, cachemisses suffered, or branches
mis-predicted - without slowing down the kernel or applications. These
registers can also trigger interrupts when a threshold number of events
have passed - and can thus be used to profile the code that runs on that
CPU.
++The Linux Performance Counter subsystem called perf event can get the
value of performance counter. You can access it through KGTP perf event
trace state variables.
++Please goto read the file tools/perf/design.txt in Linux Kernel to get
more info about perf event.
++
++Define a perf event trace state variable
++Access an performance counter need define following trace state variable:
++
++"pe_cpu_"+tv_name Define the the CPU id of the performance counter.
++"pe_type_"+tv_name Define the the type of the performance counter.
++"pe_config_"+tv_name Define the the config of the performance counter.
++"pe_en_"+tv_name This the switch to enable or disable the
performance counter.
++ The performance counter is disable in default.
++"pe_val_"+tv_name Access this variable can get the value of the
performance counter.
++Define a per_cpu perf event trace state variable
++Define a per_cpu perf event trace state variable is same with define
HOWTO#Per_cpu_trace_state_variables.
++
++"p_pe_"+perf_event type+string+CPU_id
***The diff for this file has been truncated for email.***
=======================================
--- /trunk/gtp_3.0_to_upstream.patch Thu Sep 20 18:16:39 2012
+++ /dev/null
@@ -1,16507 +0,0 @@
---- /dev/null
-+++ b/Documentation/gtp/howto.txt
-@@ -0,0 +1,1525 @@
-+ Linux Kernel GDB tracepoint module (KGTP)
-+ =========================================
-+ By Hui Zhu <teawater@xxxxxxxxx>
-+ https://code.google.com/p/kgtp/wiki/HOWTO
-+
-+Update in 2012-09-20
-+
-+What is KGTP
-+Get help or report issues about KGTP
-+Preparatory work before use KGTP
-+Linux kernel
-+If your system use the Linux kernel that is built by yourself
-+If your system use the Linux kernel from distribution
-+Make sure current Linux kernel debug image is right
-+Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
-+Get KGTP
-+Get KGTP through http
-+Get KGTP through svn
-+Config KGTP
-+Compile KGTP
-+Normal compile
-+Compile KGTP with some special config
-+Install and uninstall KGTP
-+Use KGTP with DKMS
-+Use KGTP patch for Linux kernel
-+Install GDB for KGTP
-+Howto use
-+Exec it
-+Make GDB connect to gtp
-+GDB on the current machine
-+If GDB on remote machine
-+Add module symbols to GDB
-+How to use getmod.py
-+How to use getmod
-+Access memory directly
-+GDB tracepoint
-+set tracepoint
-+How to set tracepoint condition
-+actions [num]
-+Start and stop the tracepoint
-+Enable and disable the tracepoint
-+Use "tfind" select the entry inside the trace frame info
-+Show and save the tracepoint
-+Delete tracepoint
-+Use tracepoint get register info from a point of kernel
-+Use tracepoint get the value of variable from a point of kernel
-+How to handle "Unsupported operator (null) (52) in expression."
-+Show all the traced data of current frame
-+Get status of tracepoint
-+Set the trace buffer into a circular buffer
-+Do not stop tracepoint when the GDB disconnects
-+kprobes-optimization and the execution speed of tracepoint
-+How to use trace state variables
-+Simple trace state variables
-+Per_cpu trace state variables
-+Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
-+Special trace state variable $no_self_trace
-+Trace the function return with $kret
-+Use $ignore_error and $last_errno to ignore the error of tstart
-+Use $cooked_clock and $cooked_rdtsc the time without KGTP used
-+Use $xtime_sec and $xtime_nsec get the timespec
-+Howto backtrace (stack dump)
-+Collect stack with $bt and use GDB command "backtrace"
-+Collect stack of current function's caller with $_ret
-+Use $dump_stack to output stack dump through printk
-+How to use performance counters
-+Define a perf event trace state variable
-+Define a per_cpu perf event trace state variable
-+The perf event type and config
-+Enable and disable all the perf event in a CPU with $p_pe_en
-+GDB scripts to help with set and get the perf event trace state variables
-+Howto let tracepoint output value directly
-+Switch collect to output the value directly
-+Howto show a variable whose value has been optimized away
-+Update your GCC
-+How to get the function pointer point to
-+If the debug info of the function pointer is not optimized out
-+If the debug info of the function pointer is optimized out
-+/sys/kernel/debug/gtpframe and offline debug
-+How to use /sys/kernel/debug/gtpframe_pipe
-+Get the frame info with GDB
-+Get the frame info with cat
-+Get the frame info with getframe
-+Use $pipe_trace
-+Use KGTP with user space program
-+Read memory of user space program directly
-+collect stack (for backtrace) of user space program in tracepoint
-+How to use add-ons/hotcode.py
-+How to add plugin in C
-+API
-+Example
-+How to use
-+What is KGTP
-+KGTP is a flexible , lightweight and realtime Linux debugger and tracer.
-+To use it, you don't need patch or rebuild the Linux kernel. Just build
KGTP module and insmod it is OK.
-+
-+It makes Linux Kernel supply a GDB remote debug interface. Then GDB in
current machine or remote machine can debug and trace Linux through GDB
tracepoint and some other functions without stopping the Linux Kernel.
-+And even if the board doesn't have GDB on it and doesn't have interface
for remote debug. It can debug the Linux Kernel using offline debug (See
HOWTO#/sys/kernel/debug/gtpframe_and_offline_debug).
-+KGTP supports X86-32, X86-64, MIPS and ARM.
-+KGTP is tested on Linux kernel 2.6.18 to upstream.
-+And it can work with Android (See HowToUseKGTPinAndroid).
-+
-+For new user of KGTP, please go to see Quickstart.
-+
-+Please go to UPDATE to get more info about KGTP update.
-+
-+Get help or report issues about KGTP
-+You can post it to http://code.google.com/p/kgtp/issues/list, write Email
to kgtp@xxxxxxxxxxxxx or write Email to teawater@xxxxxxxxx .
-+The KGTP team will try our best to help you.
-+
-+Preparatory work before use KGTP
-+Linux kernel
-+If your system use the Linux kernel that is built by yourself
-+To use KGTP, your Linux kernel need open following options:
-+
-+General setup --->
-+ [*] Kprobes
-+
-+[*] Enable loadable module support --->
-+
-+Kernel hacking --->
-+ [*] Debug Filesystem
-+ [*] Compile the kernel with debug info
-+If your system use the Linux kernel from distribution
-+You need install some Linux kernel package.
-+
-+Ubuntu
-+Install the Linux kernel debug image
-+1) Add debug source to the sources list of Ubuntu.
-+
-+Create an /etc/apt/sources.list.d/ddebs.list by running the following
line at a terminal:
-+echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted
universe multiverse" | \
-+sudo tee -a /etc/apt/sources.list.d/ddebs.list
-+Stable releases (not alphas and betas) require three more lines adding to
the same file, which is done by the following terminal command:
-+echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main
restricted universe multiverse
-+deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted
universe multiverse
-+deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted
universe multiverse" | \
-+sudo tee -a /etc/apt/sources.list.d/ddebs.list
-+Import the debug symbol archive signing key:
-+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
-+Then run:
-+sudo apt-get update
-+
-+2) Get Linux kernel debug image.
-+
-+sudo apt-get install linux-image-$(uname -r)-dbgsym
-+Then you can find Linux kernel debug image
in "/usr/lib/debug/boot/vmlinux-$(uname -r)".
-+
-+Install the Linux kernel headers
-+sudo apt-get install linux-headers-generic
-+Install the Linux kernel source
-+Install the source package:
-+sudo apt-get install linux-source
-+Uncompress the source package:
-+sudo mkdir -p /build/buildd/
-+sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2
-C /build/buildd/
-+sudo rm -rf /build/buildd/linux-$(uname -r | sed 's/-.*//')
-+sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//')
/build/buildd/linux-$(uname -r | sed 's/-.*//')
-+Fedora
-+Install the Linux kernel debug image
-+sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo
-+Then you can find Linux kernel debug image
in "/usr/lib/debug/lib/modules/$(uname -r)/vmlinux".
-+
-+Install the Linux kernel devel package
-+sudo yum install kernel-devel-$(uname -r)
-+Others
-+You need install the Linux kernel debug image package and the Linux
Kernel source.
-+
-+Make sure current Linux kernel debug image is right
-+GDB open the right Linux kernel debug image is an very important because
GDB will get the debug info and address info from it. So before you use
KGTP, please do the check to make sure about it.
-+There are 2 ways to do the check, what I suggest is do both of them to
make sure Linux kernel debug image is right.
-+
-+Please note that if you determine you use the right Linux kernel debug
image, but cannot pass these ways. Please see
HOWTO#Handle_the_issue_that_Linux_kernel_debug_image's_address_in.
-+
-+Use /proc/kallsyms
-+In the system that its Linux kernel is what you want to trace, use
following command to get the address of sys_read and sys_write:
-+
-+sudo cat /proc/kallsyms | grep sys_read
-+ffffffff8117a520 T sys_read
-+sudo cat /proc/kallsyms | grep sys_write
-+ffffffff8117a5b0 T sys_write
-+Then we can get that the address of sys_read is 0xffffffff8117a520 and
the address of sys_write is 0xffffffff8117a5b0.
-+After that use GDB get address of sys_read and sys_write from Linux
kernel debug image:
-+
-+gdb ./vmlinux
-+(gdb) p sys_read
-+$1 = {long int (unsigned int, char *, size_t)} 0xffffffff8117a520
<sys_read>
-+(gdb) p sys_write
-+$2 = {long int (unsigned int, const char *, size_t)} 0xffffffff8117a5b0
<sys_write>
-+The address of sys_read and sys_write is same, so the Linux kernel debug
image is right.
-+
-+Use linux_banner
-+sudo gdb ./vmlinux
-+(gdb) p linux_banner
-+$1 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
-+This linux_banner is the kernel info inside the Linux kernel debug image.
-+After that, connect to KGTP following the way in
HOWTO#Make_GDB_connect_to_gtp connect to KGTP and print linux_banner again.
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+Remote debugging using /sys/kernel/debug/gtp
-+0x0000000000000000 in irq_stack_union ()
-+(gdb) p linux_banner
-+$2 = "Linux version 3.4.0-rc4+ (teawater@teawater-Precision-M4600) (gcc
version 4.6.3 (GCC) ) #3 SMP Tue Apr 24 13:29:05 CST 2012\n"
-+This linux_banner is the kernel info that Linux kernel that KGTP is
tracing. If it is same with the prev kernel info, the Linux kernel debug
image is right.
-+
-+Handle the issue that Linux kernel debug image's address info is not same
with Linux kernel when it running
-+In X86_32, you will found that the Linux kernel debug image's address
info is not same with Linux kernel when it running through the ways in
HOWTO#Make_sure_current_Linux_kernel_debug_image_is_right. And you
determine the Linux kernel debug image is right.
-+This issue is because:
-+
-+Processor type and features --->
-+ (0x1000000) Physical address where the kernel is loaded
-+ (0x100000) Alignment value to which kernel should be aligned
-+The values of these two options are different. Please note that
the "Physical address where the kernel is loaded" is not showed in config
sometimes. You can get its value through search "PHYSICAL_START".
-+
-+You can handle this issue through change "Alignment value to which kernel
should be aligned" same with "Physical address where the kernel is loaded".
-+This issue doesn't affect X86_64.
-+
-+Get KGTP
-+Get KGTP through http
-+Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to
download the package.
-+
-+Get KGTP through svn
-+Some people have trouble with access to KGTP website. You can access kgtp
through svn:
-+
-+svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only
-+kgtp-read-only/tags/ Present for each release of KGTP.
-+kgtp-read-only/trunk/ Present for the main trunk of KGTP.
-+
-+Config KGTP
-+Following part is the default config of KGTP inside the Makefile. With
this config, KGTP will build together with current kernel that running on
this machine.
-+
-+KERNELDIR := /lib/modules/`uname -r`/build
-+CROSS_COMPILE :=
-+KERELDIR is set to the directory which holds the kernel you want to build
for. By default, it is set to the kernel that you are running.
-+Please note that this directory should be Linux kernel build directory or
linux-headers directory but not the source directory but not the Linux
kernel source directory. And the Linux kernel build directory should be
used after build successful.
-+CROSS_COMPILE is set to the prefix name of compiler that you want to
build KGTP. Empty to compile with your default compiler.
-+ARCH is the architecture.
-+
-+Or you can choose which kernel you want build with and which compiler you
want use by change Makefile.
-+For example:
-+
-+KERNELDIR := /home/teawater/kernel/bamd64
-+CROSS_COMPILE :=x86_64-glibc_std-
-+ARCH := x86_64
-+KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will use
x86_64-glibc_std-gcc.
-+
-+Compile KGTP
-+Normal compile
-+cd kgtp/
-+make
-+In some build environment (for example Android) will get some error with
user space program getmod or getframe. Please ignore this error and use the
gtp.ko in this directory.
-+
-+Compile KGTP with some special config
-+Most of time, KGTP can auto select right options to build with Various
versions of Linux kernel.
-+But if you want config special options with yourself, you can read
following part:
-+With this option, KGTP will not auto select any build options.
-+
-+make AUTO=0
-+With this option, KGTP will use simple frame instead of KGTP ring buffer.
-+The simple frame doesn't support gtpframe_pipe. It just for debug KGTP.
-+
-+make AUTO=0 FRAME_SIMPLE=1
-+With this option, $clock will return rdtsc value instead of local_clock.
-+
-+make AUTO=0 CLOCK_CYCLE=1
-+With this option, KGTP will use procfs instead of debugfs.
-+
-+make AUTO=0 USE_PROC=1
-+The options can use together, for example:
-+
-+make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1
-+Install and uninstall KGTP
-+KGTP don't need to be install because it can insmod directly inside its
directory (See HOWTO#Exec_it). But if you need, you can install it to your
system.
-+Install:
-+
-+cd kgtp/
-+sudo make install
-+Uninstall:
-+
-+cd kgtp/
-+sudo make uninstall
-+Use KGTP with DKMS
-+You can use KGTP with DKMS if you want it.
-+Following commands will copy the files of KGTP to the directory that DKMS
need.
-+
-+cd kgtp/
-+sudo make dkms
-+Then you can use DKMS commands to control KGTP. Please goto
http://linux.dell.com/dkms/manpage.html to see how to use DKMS.
-+
-+Use KGTP patch for Linux kernel
-+Most of time, you don't need KGTP patch because KGTP can build as a LKM
and very easy to use. But to help some people include KGTP to them special
Linux Kernel tree, KGTP supply patches for Linux kernel.
-+In the KGTP directory:
-+gtp_3.0_to_upstream.patch is the patch for Linux kernel from 3.0 to
upstream.
-+gtp_2.6.39.patch is the patch for Linux kernel 2.6.39.
-+gtp_2.6.33_to_2.6.38.patch is the patch for Linux kernel from 2.6.33 to
2.6.38.
-+gtp_2.6.20_to_2.6.32.patch is the patch for Linux kernel from 2.6.20 to
2.6.32.
-+gtp_older_to_2.6.19.patch is the patch for Linux kernel 2.6.19 and older
version.
-+Install GDB for KGTP
-+The GDB that older than 7.3 have some bugs of tracepoint. And some
functions of GDB are not very well.
-+So if your GDB is older than 7.3 please go to
https://code.google.com/p/gdbt/ to get howto install GDB for KGTP.
-+If you have issue about GDB please get help according to
HOWTO#Report_issues_about_KGTP.
-+
-+Howto use
-+Exec it
-+If you have installed KGTP in your system, you can:
-+
-+sudo modprobe gtp
-+Or you can use the kgtp module in the directory.
-+
-+cd kgtp/
-+sudo insmod gtp.ko
-+Make GDB connect to gtp
-+GDB on the current machine
-+sudo gdb ./vmlinux
-+(gdb) target remote /sys/kernel/debug/gtp
-+Remote debugging using /sys/kernel/debug/gtp
-+0x0000000000000000 in ?? ()
-+After that, you can begin to use GDB command trace and debug the Linux
Kernel.
-+
-+If GDB on remote machine
-+#Open the KGTP interface in current machine.
-+sudo su
-+nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp
-+(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old
version netcat.)
-+#Let gdb connect to the port 1234
-+gdb ./vmlinux
-+(gdb) target remote xxx.xxx.xxx.xxx:1234
-+After that, you can begin to use GDB command trace and debug the Linux
Kernel.
-+
-+Add module symbols to GDB
-+Sometimes you need to add a Linux kernel module's symbols to GDB to debug
it.
-+Add symbols with hand is not very easy, so KGTP package include an GDB
python script "getmod.py" and a program "getmod" can help you.
-+
-+How to use getmod.py
-+Connect to KGTP before use the getmod.py.
-+
-+(gdb) source ~/kgtp/getmod.py
-+Then this script will auto load the Linux kernel module's symbols to GDB.
-+
-+How to use getmod
-+"getmod" is written by C so you can use it anywhere even if in an
embedded environment.
-+For example:
-+
-+#Following command save Linux Kernel module info to the file ~/tmp/mi in
GDB
-+#command format.
-+sudo getmod >~/tmp/mi
-+#in gdb part:
-+(gdb) source ~/tmp/mi
-+add symbol table from
file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
-+ .text_addr = 0xf80de000
-+ .note.gnu.build-id_addr = 0xf80de088
-+ .exit.text_addr = 0xf80de074
-+ .init.text_addr = 0xf8118000
-+ .rodata.str1.1_addr = 0xf80de0ac
-+ .rodata_addr = 0xf80de0c0
-+ __mcount_loc_addr = 0xf80de9c0
-+ .data_addr = 0xf80de9e0
-+ .gnu.linkonce.this_module_addr = 0xf80dea00
-+#After this GDB command, all the Linux Kernel module info is loaded into
GDB.
-+If you use remote debug or offline debug, maybe you need change the base
directory. Following example is for it.
-+
-+#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r
/home/teawater/kernel/b26
-+sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi
-+Access memory directly
-+After connect the KGTP, you can access most of memory directly.
-+For example, you can access to "jiffies_64" with following command:
-+
-+(gdb) p jiffies_64
-+Or you can access to the first entry of "static LIST_HEAD(modules)" with
following command:
-+
-+(gdb) p *((struct module *)((char *)modules->next - ((size_t) &(((struct
module *)0)->list))))
-+Or you can access to the CPU0 memory info of "DEFINE_PER_CPU(struct
device *, mce_device);":`
-+
-+p *(struct device *)(__per_cpu_offset[0]+(uint64_t)(&mce_device))
-+If you want show more than one variables with one GDB command, please use
following example:
-+
-+(gdb) printf "%4d %4d %4d %4d %4d %4d %18d %lu\n", this_rq->cpu,
this_rq->nr_running, this_rq->nr_uninterruptible, nr_active,
calc_load_tasks->counter, this_rq->calc_load_active, delta,
this_rq->calc_load_update
-+2 1 0 0 0 0 673538312 717077240
-+GDB tracepoint
-+Tracepoint is that GDB define some addresses and some actions and put
them to the target (KGTP). After tracepoint start, , KGTP will do these
actions (Some of them will collect data and save them to tracepoint frame
buffer) when Linux kernel execution to there addresses. After that, Linux
kernel will keep execution.
-+KGTP supply some interfaces that GDB or other programe can take the data
of tracepoint frame buffer out to parse.
-+About these interfaces, this doc have introduced "/sys/kernel/debug/gtp".
And will introduce "/sys/kernel/debug/gtpframe"
and "/sys/kernel/debug/gtpframe_pipe" later.
-+
-+Doc of GDB tracepoint in
http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html.
-+
-+set tracepoint
-+The trace command is very similar to the break command. Its argument
location can be a source line, a function name, or an address in the target
program. The trace command defines a tracepoint, which is a address or some
addresses that KGTP do some actions in it.
-+Here are some examples of using the trace command:
-+
-+(gdb) trace foo.c:121 // a source file and line number
-+
-+(gdb) trace +2 // 2 lines forward
-+
-+(gdb) trace my_function // first source line of function
-+
-+(gdb) trace *my_function // EXACT start address of function
-+
-+(gdb) trace *0x2117c4 // an address
-+Howto handle the function is there but set tracepoint on it got fail
-+GCC will inline some static function to increase the performance. You
cannot set tracepoint on the function name because object file doesn't have
symbol of inline function.
-+You can use "trace filename:line" to set tracepoint on it.
-+
-+How to set tracepoint condition
-+http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html
-+Like breakpoints, we can set conditions on tracepoints. The speed of
tracepoints is faster than breakpoints because KGTP can do all the
condition checks.
-+For example:
-+
-+(gdb) trace handle_irq if (irq == 47)
-+This action of tracepoint 1 will work only when irq number is 47.
-+
-+And you can use GDB command "condition" to specify the condition of a
tracepoint. GDB command "condition N COND" will specify tracepoint number N
to trace only if COND is true.
-+For example:
-+
-+(gdb) trace handle_irq
-+(gdb) condition 1 (irq == 47)
-+GDB command "info tracepoint" will show the ID of the tracepoint.
-+
-+Value of $bpnum is the last ID of GDB tracepoint. Then you can use GDB
command "condtion" set the condition of last tracepoint without get its ID.
For example:
-+
-+(gdb) trace handle_irq
-+(gdb) condition $bpnum (irq == 47)
-+actions [num]
-+This command will prompt for a list of actions to be taken when the
tracepoint is hit. If the tracepoint number num is not specified, this
command sets the actions for the one that was most recently defined (so
that you can define a tracepoint and then say actions without bothering
about its number). You specify the actions themselves on the following
lines, one action at a time, and terminate the actions list with a line
containing just end. So far, the only defined actions are collect, teval,
and while-stepping.
-+
-+collect expr1, expr2, ...
-+Collect values of the given expressions when the tracepoint is hit. This
command accepts a comma-separated list of any valid expressions. In
addition to global, static, or local variables, the following special
arguments are supported:
-+
-+$regs Collect all registers.
-+$args Collect all function arguments.
-+$locals Collect all local variables.
-+Please note that collect an pointer (collect ptr) will just collect the
address of this pointer. Add a * before ptr will make action collect the
data that pointer point to(collect *ptr).
-+
-+teval expr1, expr2, ...
-+Evaluate the given expressions when the tracepoint is hit. This command
accepts a comma-separated list of expressions. The results are discarded,
so this is mainly useful for assigning values to trace state variables (see
HOWTO#Simple_trace_state_variables) without adding those values to the
trace buffer, as would be the case if the collect action were used.
-+
-+while-stepping n
-+Please goto
HOWTO#If_the_debug_info_of_the_function_pointer_is_optimized_out see howto
use it.
-+
-+Start and stop the tracepoint
-+Tracepoint will exec actions only when it is starting use this GDB
command:
-+
-+(gdb) tstart
-+It will stop by this GDB command:
-+
-+(gdb) tstop
-+Enable and disable the tracepoint
-+Like breakpoint, tracepoint can be control by GDB commands "enable"
and "disable". But please note that it only useful when tracepoint stop.
-+
-+Use "tfind" select the entry inside the trace frame info
-+GDB command "tfind" is used to select a entry of trace frame bufffer when
tracepoint stop.
-+When GDB inside "tfind" mode, it will just show the values of this entry
that the tracepoint action collect. So it will output some error when print
some values that action doesn't collect for example the argument of
function. That is not a bug, please don't worry about it.
-+Use "tfind" again will select next entry. "tfind id" will select entry id.
-+To go back to normal mode, please use GDB command "tfind -1". Please goto
http://sourceware.org/gdb/current/onlinedocs/gdb/tfind.html get more info
about it.
-+
-+Show and save the tracepoint
-+You can use GDB command "info tracepoints" to show all the tracepoints.
-+You can use GDB command "save tracepoints filename" to save the commands
that setup the tracepoints and actions into file filename. Then you use use
GDB commands "source filename" to setup this tracepints again.
-+
-+Delete tracepoint
-+GDB command "delete id" will delete tracepoint id. If "delete" without
argument, it will delete all the tracepoint.
-+
-+Use tracepoint get register info from a point of kernel
-+The following is an example that records the value of all registers
when "vfs_readdir" is called.
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+(gdb) trace vfs_readdir
-+Tracepoint 1 at 0xc01a1ac0: file
-+/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
-+(gdb) actions
-+Enter actions for tracepoint 1, one per line.
-+End with a line saying just "end".
-+>collect $reg
-+>end
-+(gdb) tstart
-+(gdb) shell ls
-+(gdb) tstop
-+(gdb) tfind
-+Found trace frame 0, tracepoint 1
-+#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
<filldir64>,
-+ buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
-+23 /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
directory.
-+ in /home/teawater/kernel/linux-2.6/fs/readdir.c
-+(gdb) info reg
-+eax 0xc5528d00 -984445696
-+ecx 0xc0d09f90 -1060069488
-+edx 0xc01a1900 -1072031488
-+ebx 0xfffffff7 -9
-+esp 0xc0d09f8c 0xc0d09f8c
-+ebp 0x0 0x0
-+esi 0x8061480 134616192
-+edi 0xc5528d00 -984445696
-+eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
-+eflags 0x286 [ PF SF IF ]
-+cs 0x60 96
-+ss 0x8061480 134616192
-+ds 0x7b 123
-+es 0x7b 123
-+fs 0x0 0
-+gs 0x0 0
-+(gdb) tfind
-+Found trace frame 1, tracepoint 1
-+0xc01a1ac1 23 in /home/teawater/kernel/linux-2.6/fs/readdir.c
-+(gdb) info reg
-+eax 0xc5528d00 -984445696
-+ecx 0xc0d09f90 -1060069488
-+edx 0xc01a1900 -1072031488
-+ebx 0xfffffff7 -9
-+esp 0xc0d09f8c 0xc0d09f8c
-+ebp 0x0 0x0
-+esi 0x8061480 134616192
-+edi 0xc5528d00 -984445696
-+eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
-+eflags 0x286 [ PF SF IF ]
-+cs 0x60 96
-+ss 0x8061480 134616192
-+ds 0x7b 123
-+es 0x7b 123
-+fs 0x0 0
-+gs 0x0 0
-+Use tracepoint get the value of variable from a point of kernel
-+The following is an example that records the value of "jiffies_64" when
the function "vfs_readdir" is called:
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+(gdb) trace vfs_readdir
-+Tracepoint 1 at 0xc01ed740: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
-+(gdb) actions
-+Enter actions for tracepoint 1, one per line.
-+End with a line saying just "end".
-+>collect jiffies_64
-+>collect file->f_path.dentry->d_iname
-+>end
-+(gdb) tstart
-+(gdb) shell ls
-+arch drivers include kernel mm Module.symvers
security System.map virt
-+block firmware init lib modules.builtin net
sound t vmlinux
-+crypto fs ipc Makefile modules.order scripts
source usr vmlinux.o
-+(gdb) tstop
-+(gdb) tfind
-+Found trace frame 0, tracepoint 1
-+#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580
<filldir64>, buf=0xd6dfdf90)
-+ at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
-+24 {
-+(gdb) p jiffies_64
-+$1 = 4297248706
-+(gdb) p file->f_path.dentry->d_iname
-+$1 = "b26", '\000' <repeats 28 times>
-+How to handle "Unsupported operator (null) (52) in expression."
-+If you use condition about string, you will got this error when you
call "tstart".
-+To handle it, you can convent the char to int to handle this issue, for
example:
-+
-+(gdb) p/x 'A'
-+$4 = 0x41
-+(gdb) condition 1 (buf[0] == 0x41)
-+Show all the traced data of current frame
-+After use "tfind" select an entry, you can use "tdump" to do it.
-+
-+(gdb) tdump
-+Data collected at tracepoint 1, trace frame 0:
-+$cr = void
-+file->f_path.dentry->d_iname
= "gtp\000.google.chrome.g05ZYO\000\235\337\000\000\000\000\200\067k\364\200\067",
<incomplete sequence \364>
-+jiffies_64 = 4319751455
-+Get status of tracepoint
-+Please use GDB command "tstatus".
-+
-+Set the trace buffer into a circular buffer
-+http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
-+The frame buffer is not a circular buffer by default. When the buffer is
full, the tracepoint will stop.
-+Following command will set frame buffer to a circular buffer. When the
buffer is full, it will auto discard traceframes (oldest first) and keep
trace.
-+
-+(gdb) set circular-trace-buffer on
-+Do not stop tracepoint when the GDB disconnects
-+http://sourceware.org/gdb/current/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html
-+KGTP will stop tracepoint and delete the trace frame when GDB disconnects
with it by default.
-+Following command will open the KGTP disconnect-trace. After that, when
GDB disconnects with KGTP, KGTP will not stop tracepoint. And after GDB
reconnects to KGTP, it can keep control of KGTP like nothing happened.
-+
-+(gdb) set disconnected-tracing on
-+kprobes-optimization and the execution speed of tracepoint
-+The tracepoint is execution together with Linux kernel. So it speed will
affect the speed the system.
-+The KGTP tracepoint base on Linux kernel kprobe. Because the normal
kprobe base on breakpoint instruction, so it is not very fast.
-+
-+But if arch of kernel is X86_64 or X86_32 and kernel config didn't
open "Preemptible Kernel" (PREEMPT), the kprobe is speed up by
kprobes-optimization (CONFIG_OPTPROBES) that make kprobe very fast.
-+To make sure about that, you can use following command in terminal:
-+
-+sysctl -A | grep kprobe
-+debug.kprobes-optimization = 1
-+That means that your kernel support kprobes-optimization.
-+Please note that some KGTP functions will make this tracepoint use simple
kprobe even if this Kernel support kprobes-optimization. This doc will add
note when introduce these functions. Please avoid using them when you
really care about the tracepoint speed.
-+
-+How to use trace state variables
-+http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html
-+Tracepoints have variables that can be traced directly, or used in
tracepoint conditions.
-+Please note that just GDB 7.2.1 and later versions support use trace
state variables directly, the old version of GDB can show the value of
trace state variables through command "info tvariables".
-+
-+Simple trace state variables
-+Define a trace state variable $c.
-+
-+(gdb) tvariable $c
-+Trace state variable $c is created with initial value 0. The following
action uses $c to count how many irqs happened in the kernel.
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+(gdb) trace handle_irq
-+(gdb) actions
-+Enter actions for tracepoint 3, one per line.
-+End with a line saying just "end".
-+>collect $c #Save current value of $c to the trace frame buffer.
-+>teval $c=$c+1 #Increase the $c.
-+>end
-+Also, you can set a value of variable to trace state variable, but don't
forget covert variable to "uint64_t".
-+
-+>teval $c=(uint64_t)a
-+You can get the current value of $c while the trace is running or stopped.
-+
-+(gdb) tstart
-+(gdb) info tvariables
-+$c 0 31554
-+(gdb) p $c
-+$5 = 33652
-+(gdb) tstop
-+(gdb) p $c
-+$9 = 105559
-+When using tfind, you can parse the trace frame buffer. If the value of a
trace state variable is collected, you can parse it out.
-+
-+(gdb) tstop
-+(gdb) tfind
-+(gdb) info tvariables
-+$c 0 0
-+(gdb) p $c
-+$6 = 0
-+(gdb) tfind 100
-+(gdb) p $c
-+$7 = 100
-+If need, the tracepoint action that access the simple trace state
variables will auto lock a spin lock for trace state variables. So it can
handle race condition issue about trace state variables.
-+The following example is OK even if it running a machine that have more
than one CPU.
-+
-+>teval $c=$c+1
-+Per_cpu trace state variables
-+Per_cpu trace state variables are special simple trace state variables.
-+When tracepoint action access to it, it will access to this CPU special
trace state variables.
-+It have 2 advantages:
-+1. The tracepoint actions that access to per_cpu trace state variables
don't have the race conditon issue. So it don't need lock the spin lock for
trace state variables. It is faster than simple trace state variables on
multi-core machine.
-+2. Write the action that count some CPU special thing with it is easier
than simple trace state variables.
-+
-+How to define
-+Per_cpu trace state variables have two types:
-+
-+Local CPU variables
-+"per_cpu_"+string
-+or
-+
-+"p_"+string
-+For example:
-+
-+(gdb) tvariable $p_count
-+When access this trace state variable in tracepoint actions, it will
return the variable's value of CPU that this tracepoint actions running on.
-+
-+CPU id variables
-+"per_cpu_"+string+CPU_id
-+or
-+
-+"p_"+string+CPU_id
-+For example:
-+
-+(gdb) tvariable $p_count0
-+(gdb) tvariable $p_count1
-+(gdb) tvariable $p_count2
-+(gdb) tvariable $p_count3
-+When access this trace state variable in tracepoint actions or GDB
command line, it will return the variable's value of CPU CPU_id.
-+Follow example can auto define a CPU id variables for each CPU of this
machine. (Please note that need let GDB connect to KGTP before use these
commands.)
-+
-+(gdb) set $tmp=0
-+(gdb) while $tmp<$cpu_number
-+ >eval "tvariable $p_count%d",$tmp
-+ >set $tmp=$tmp+1
-+ >end
-+Example 1
-+This example define a tracepoint that count the times that call vfs_read
of each CPU.
-+
-+tvariable $p_count
-+set $tmp=0
-+while $tmp<$cpu_number
-+ eval "tvariable $p_count%d",$tmp
-+ set $tmp=$tmp+1
-+ end
-+trace vfs_read
-+actions
-+ teval $p_count=$p_count+1
-+ end
-+Then you can show how many vfs_read in each CPU after "tstart":
-+
-+(gdb) p $p_count0
-+$3 = 44802
-+(gdb) p $p_count1
-+$4 = 55272
-+(gdb) p $p_count2
-+$5 = 102085
-+(gdb) p $p_count3
-+Example 2
-+This example record stack dump of the function that close IRQ longest
time of each CPU.
-+
-+set pagination off
-+
-+tvariable $bt=1024
-+tvariable $p_count
-+tvariable $p_cc
-+set $tmp=0
-+while $tmp<$cpu_number
-+eval "tvariable $p_cc%d",$tmp
-+set $tmp=$tmp+1
-+end
-+
-+tvariable $ignore_error=1
-+
-+trace arch_local_irq_disable
-+ commands
-+ teval $p_count=$clock
-+ end
-+trace arch_local_irq_enable if ($p_count && $p_cc < $clock - $p_count)
-+ commands
-+ teval $p_cc = $clock - $p_count
-+ collect $bt
-+ collect $p_cc
-+ teval $p_count=0
-+ end
-+
-+enable
-+set pagination on
-+Special trace state variables $current_task, $current_task_pid,
$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format,
$printk_tmp ,$clock, $hardirq_count, $softirq_count and $irq_count
-+KGTP special trace state variables $current_task, $current_thread_info,
$cpu_id and $clock can very easy to access to some special value. You can
see them when GDB connects to the KGTP. You can use them in tracepoint
conditions or actions.
-+Access $current_task in tracepoint condition and action will get that
returns of get_current().
-+Access $current_task_pid in tracepoint condition and action will get that
returns of get_current()->pid.
-+Access $current_thread_info in tracepoint condition and action will get
that returns of current_thread_info().
-+Access $cpu_id in tracepoint condition and action will get that returns
of smp_processor_id().
-+Access $clock in tracepoint condition and action will get that returns of
local_clock() that return the timestamp in nanoseconds.
-+$rdtsc is only available on X86 and X86_64 architecture. Access it in
anytime will get current value of TSC with instruction RDTSC.
-+Access $hardirq_count in tracepoint condition and action will get that
returns of hardirq_count().
-+Access $softirq_count in tracepoint condition and action will get that
returns of softirq_count().
-+Access $irq_count in tracepoint condition and action will get that
returns of irq_count().
-+And KGTP has other special trace state variables $dump_stack,
$printk_level, $printk_format and $printk_tmp. All of them output their
values directly, as can be seen in
HOWTO#Howto_let_tracepoint_output_value_directly.
-+The following example counts in $c how many vfs_read calls that process
16663 does and collects the struct thread_info of current task:
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid ==
16663)
-+(gdb) tvariable $c
-+(gdb) actions
-+Enter actions for tracepoint 4, one per line.
-+End with a line saying just "end".
-+>teval $c=$c+1
-+>collect (*(struct thread_info *)$current_thread_info)
-+>end
-+(gdb) tstart
-+(gdb) info tvariables
-+Name Initial Current
-+$c 0 184
-+$current_task 0 <unknown>
-+$current_thread_info 0 <unknown>
-+$cpu_id 0 <unknown>
-+(gdb) tstop
-+(gdb) tfind
-+(gdb) p *(struct thread_info *)$current_thread_info
-+$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status =
0, cpu = 1, preempt_count = 2, addr_limit = {
-+ seg = 4294967295}, restart_block = {fn = 0xc0159fb0
<do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11,
-+ arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags
= 1, bitset = 78, time = 977063750,
-+ uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb,
expires = 335007449089}, poll = {
-+ ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78,
tv_nsec = 977063750}}},
-+ sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack =
0xef340044 "", uaccess_err = 0}
-+Another example shows how much sys_read() executes in each CPU.
-+
-+(gdb) tvariable $c0
-+(gdb) tvariable $c1
-+(gdb) trace sys_read
-+(gdb) condition $bpnum ($cpu_id == 0)
-+(gdb) actions
-+>teval $c0=$c0+1
-+>end
-+(gdb) trace sys_read
-+(gdb) condition $bpnum ($cpu_id == 1)
-+(gdb) actions
-+>teval $c1=$c1+1
-+>end
-+(gdb) info tvariables
-+Name Initial Current
-+$current_task 0 <unknown>
-+$cpu_id 0 <unknown>
-+$c0 0 3255
-+$c1 0 1904
-+sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. Please note
that this example just to howto use $cpu_id. Actially, this example use
per_cpu trace state variables is better.
-+
-+Special trace state variable $no_self_trace
-+$no_self_trace is different with the special trace state variables in the
previous section. It is used to control the behavior of tracepoint.
-+If the action of a tracepoint include a command access to the
$no_self_trace. The tracepoint will not trace anything if the current_task
is the a KGTP self process (GDB, netcat, getframe or some others process
that access to the interface of KGTP).
-+For example, if we want trace vfs_read or something that have process
context, and we don't want trace the operation of KGTP self process. Add
following command to the action:
-+
-+>collect $no_self_trace
-+Please note that the code that doesn't about process context (Irq
handler, softirq) doesn't need set this variable.
-+
-+Trace the function return with $kret
-+Sometime, set the tracepoint to the end of function is hard because the
Kernel is compiled with optimization. At this time, you can get help from
$kret.
-+$kret is a special trace state variable like $no_self_trace. When you set
value of it inside the action of tracepoint, this tracepoint be set with
kretprobe instead of kprobe. Then it can trace the end of this function.
-+Please note that this tracepoint must set in the first address of the
function in format "function_name".
-+
-+Following part is an example:
-+
-+#"*(function_name)" format can make certain that GDB send the first
address of function to KGTP.
-+(gdb) trace *vfs_read
-+(gdb) actions
-+>teval $kret=0
-+#Following part you can set commands that you want.
-+Use $ignore_error and $last_errno to ignore the error of tstart
-+If KGTP got any error of tstart, this command will get fail.
-+But sometime we need ignore this error and let KGTP keep work. For
example: If you set tracepoint on the inline function spin_lock. This
tracepoint will be set to a lot of addresses that some of them cannot be
set kprobe. It will make tstart get fail. You can use "$ignore_error"
ignore this error.
-+And the last error number will available in "$last_errno".
-+
-+(gdb) tvariable $ignore_error=1
-+This command will open ignore.
-+
-+(gdb) tvariable $ignore_error=0
-+This command will close ignore.
-+
-+Use $cooked_clock and $cooked_rdtsc the time without KGTP used
-+Access these two trace state variables can get the time without KGTP
used. Then we can get more close to really time that a part of code used
even if the actions of tracepoint is very complex. They will be introduce
in Cookbook (coming soon).
-+
-+Use $xtime_sec and $xtime_nsec get the timespec
-+Access these two trace state variables will return the time of day in a
timespec that use getnstimeofday.
-+$xtime_sec will access to the second part of a timespec.
-+$xtime_nsec will access to the nanosecond part of a timespec.
-+
-+Howto backtrace (stack dump)
-+Each time your program performs a function call, information about the
call is generated. That information includes the location of the call in
your program, the arguments of the call, and the local variables of the
function being called. The information is saved in a block of data called a
stack frame. The stack frames are allocated in a region of memory called
the call stack.
-+
-+Collect stack with $bt and use GDB command "backtrace"
-+Because this way is faster (just collect the stack when trace) and parse
out most of info inside the call stack (it can show all the stack info that
I introduce). So I suggest you use this way to do the stack dump.
-+First we need add the collect the stack command to the tracepoint action.
-+The general collect the stack command in GDB tracepoint is: In x86_32,
following command will collect 512 bytes of stack.
-+>collect *(unsigned char *)$esp@512
-+In x86_64, following command will collect 512 bytes of stack.
-+>collect *(unsigned char *)$rsp@512
-+In MIPS or ARM, following command will collect 512 bytes of stack.
-+>collect *(unsigned char *)$sp@512
-+These commands is so hard to remember, and the different arch need
different command.
-+KGTP have an special tracepoint trace state variable $bt. If tracepoint
action access it, KGTP will auto collect the $bt size (default value is
512) stack. For example, this command will collect 512 bytes stack memory:
-+
-+>collect $bt
-+If you want to change size of $bt, you can use following GDB command
before "tstart":
-+
-+(gdb) tvariable $bt=1024
-+Following part is an example about howto collect stack and howto use GDB
parse it:
-+
-+(gdb) target remote /sys/kernel/debug/gtp
-+(gdb) trace vfs_readdir
-+Tracepoint 1 at 0xffffffff8118c300: file
/home/teawater/kernel2/linux/fs/readdir.c, line 24.
-+(gdb) actions
-+Enter actions for tracepoint 1, one per line.
-+End with a line saying just "end".
-+>collect $bt
-+>end
-+(gdb) tstart
-+(gdb) shell ls
-+1 crypto fs include kernel mm
Module.symvers security System.map vmlinux
-+arch drivers hotcode.html init lib modules.builtin
net sound usr vmlinux.o
-+block firmware hotcode.html~ ipc Makefile modules.order
scripts source virt
-+(gdb) tstop
-+(gdb) tfind
-+Found trace frame 0, tracepoint 1
-+#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
-+ at /home/teawater/kernel2/linux/fs/readdir.c:24
-+24 {
-+(gdb) bt
-+#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
-+ at /home/teawater/kernel2/linux/fs/readdir.c:24
-+#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
-+#2 <signal handler called>
-+#3 0x00007f00253848a5 in ?? ()
-+#4 0x00003efd32cddfc9 in ?? ()
-+#5 0x00002c15b7d04101 in ?? ()
-+#6 0x000019c0c5704bf1 in ?? ()
-+#7 0x0000000900000000 in ?? ()
-+#8 0x000009988cc8d269 in ?? ()
-+#9 0x000009988cc9b8d1 in ?? ()
-+#10 0x0000000000000000 in ?? ()
-+(gdb) up
-+#1 0xffffffff8118c689 in sys_getdents (fd=<optimized out>,
dirent=0x1398c58, count=32768) at
/home/teawater/kernel2/linux/fs/readdir.c:214
-+214 error = vfs_readdir(file, filldir, &buf);
-+(gdb) p buf
-+$1 = {current_dir = 0x1398c58, previous = 0x0, count = 32768, error = 0}
-+(gdb) p error
-+$3 = -9
-+(gdb) frame 0
-+#0 vfs_readdir (file=0xffff8800c5556d00, filler=0xffffffff8118c4b0
<filldir>, buf=0xffff880108709f40)
-+ at /home/teawater/kernel2/linux/fs/readdir.c:24
-+24 {
-+From this example, we can see some GDB commands that parse the the call
stack:
-+bt is the alias of GDB commands backtrace that print a backtrace of the
entire stack: one line per frame for all frames in the stack.
-+up n is move n frames up the stack. For positive numbers n, this advances
toward the outermost frame, to higher frame numbers, to frames that have
existed longer. n defaults to one.
-+down n is move n frames down the stack. For positive numbers n, this
advances toward the innermost frame, to lower frame numbers, to frames that
were created more recently. n defaults to one. You may abbreviate down as
do.
-+frame n is select frame number n. Recall that frame zero is the innermost
(currently executing) frame, frame one is the frame that called the
innermost one, and so on. The highest-numbered frame is the one for main.
-+You can see that when you use up, down or frame to the different calll
stack frame, you can output the value of the arguments and local variables
of different call stack frame.
-+To get the more info about howto use GDB parse the call stack, please see
http://sourceware.org/gdb/current/onlinedocs/gdb/Stack.html
-+Collect stack of current function's caller with $_ret
-+If you just want to collect stack of current function's caller, please
use $_ret.
-+Please note that set the tracepoint that collect $_ret cannot in the
first address of function.
-+For example:
-+
-+(gdb) list vfs_read
-+360 }
-+361
-+362 EXPORT_SYMBOL(do_sync_read);
-+363
-+364 ssize_t vfs_read(struct file *file, char __user *buf, size_t
count, loff_t *pos)
-+365 {
-+366 ssize_t ret;
-+367
-+368 if (!(file->f_mode & FMODE_READ))
-+369 return -EBADF;
-+(gdb) trace 368
-+Tracepoint 2 at 0xffffffff8117a244: file
/home/teawater/kernel2/linux/fs/read_write.c, line 368.
-+(gdb) actions
-+Enter actions for tracepoint 2, one per line.
-+End with a line saying just "end".
-+>collect $_ret
-+>end
-+(gdb) tstart
-+(gdb) tstop
-+(gdb) tfind
-+Found trace frame 0, tracepoint 2
-+#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
-+ at /home/teawater/kernel2/linux/fs/read_write.c:368
-+368 if (!(file->f_mode & FMODE_READ))
-+(gdb) bt
-+#0 vfs_read (file=0xffff880141c46000, buf=0x359bda0 <Address 0x359bda0
out of bounds>, count=8192, pos=0xffff88012fa49f48)
-+ at /home/teawater/kernel2/linux/fs/read_write.c:368
-+#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
-+ at /home/teawater/kernel2/linux/fs/read_write.c:469
-+Backtrace stopped: not enough registers or memory available to unwind
further
-+(gdb) up
-+#1 0xffffffff8117a3ea in sys_read (fd=<optimized out>,
buf=<unavailable>, count=<unavailable>)
-+ at /home/teawater/kernel2/linux/fs/read_write.c:469
-+469 ret = vfs_read(file, buf, count, &pos);
-+(gdb) p ret
-+$2 = -9
-+You see that the caller of function vfs_read is sys_read. And the local
variable ret of sys_read is -9.
-+
-+Use $dump_stack to output stack dump through printk
-+Because this way need parse the stack when tracing and call printk
inside, so it will be slow, unsafe, unclear and cannot access a lot of info
of call stack. So I suggest you use the prev way to do stack dump.
-+KGTP has special trace state variable $dump_stack, "collect" it will let
Linux Kernel output stack dump through printk.
-+Following example lets Linux Kernel show the stack dump of vfs_readdir:
-+
-+target remote /sys/kernel/debug/gtp
-+trace vfs_readdir
-+ commands
-+ collect $dump_stack
-+ end
-+Then your kernel will printk like:
-+
-+[22779.208064] gtp 1:Pid: 441, comm: python Not tainted 2.6.39-rc3+ #46
-+[22779.208068] Call Trace:
-+[22779.208072] [<fe653cca>] gtp_get_var+0x4a/0xa0 [gtp]
-+[22779.208076] [<fe653d79>] gtp_collect_var+0x59/0xa0 [gtp]
-+[22779.208080] [<fe655974>] gtp_action_x+0x1bb4/0x1dc0 [gtp]
-+[22779.208084] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-+[22779.208088] [<c023f152>] ? __find_get_block_slow+0xd2/0x160
-+[22779.208091] [<c01a8c56>] ? delayacct_end+0x96/0xb0
-+[22779.208100] [<c023f404>] ? __find_get_block+0x84/0x1d0
-+[22779.208103] [<c05b6408>] ? _raw_spin_unlock+0x18/0x40
-+[22779.208106] [<c02e0838>] ? find_revoke_record+0xa8/0xc0
-+[22779.208109] [<c02e0c45>] ? jbd2_journal_cancel_revoke+0xd5/0xe0
-+[22779.208112] [<c02db51f>] ?
__jbd2_journal_temp_unlink_buffer+0x2f/0x110
-+[22779.208115] [<fe655c4c>] gtp_kp_pre_handler+0xcc/0x1c0 [gtp]
-+[22779.208118] [<c05b8a88>] kprobe_exceptions_notify+0x3d8/0x440
-+[22779.208121] [<c05b7d54>] ? hw_breakpoint_exceptions_notify+0x14/0x180
-+[22779.208124] [<c05b95eb>] ? sub_preempt_count+0x7b/0xb0
-+[22779.208126] [<c0227ac5>] ? vfs_readdir+0x15/0xb0
-+[22779.208128] [<c0227ac4>] ? vfs_readdir+0x14/0xb0
-+[22779.208131] [<c05b9743>] notifier_call_chain+0x43/0x60
-+[22779.208134] [<c05b9798>] __atomic_notifier_call_chain+0x38/0x50
-+[22779.208137] [<c05b97cf>] atomic_notifier_call_chain+0x1f/0x30
-+[22779.208140] [<c05b980d>] notify_die+0x2d/0x30
-+[22779.208142] [<c05b71c5>] do_int3+0x35/0xa0
-+How to use performance counters
-+Performance counters are special hardware registers available on most
modern CPUs. These registers count the number of certain types of hw
events: such as instructions executed, cachemisses suffered, or branches
mis-predicted - without slowing down the kernel or applications. These
registers can also trigger interrupts when a threshold number of events
have passed - and can thus be used to profile the code that runs on that
CPU.
-+The Linux Performance Counter subsystem called perf event can get the
value of performance counter. You can access it through KGTP perf event
trace state variables.
-+Please goto read the file tools/perf/design.txt in Linux Kernel to get
more info about perf event.
-+
-+Define a perf event trace state variable
-+Access an performance counter need define following trace state variable:
-+
-+"pe_cpu_"+tv_name Define the the CPU id of the performance counter.
-+"pe_type_"+tv_name Define the the type of the performance counter.
-+"pe_config_"+tv_name Define the the config of the performance counter.
-+"pe_en_"+tv_name This the switch to enable or disable the
performance counter.
-+ The performance counter is disable in default.
-+"pe_val_"+tv_name Access this variable can get the value of the
performance counter.
-+Define a per_cpu perf event trace state variable
-+Define a per_cpu perf event trace state variable is same with define
HOWTO#Per_cpu_trace_state_variables.
-+
-+"p_pe_"+perf_event type+string+CPU_id
***The diff for this file has been truncated for email.***
Other related posts:
- » [kgtp] r1394 committed - Add gtp_3.7_to_upstream.patch... - kgtp
