Revision: 939 Author: teawater Date: Fri Feb 24 19:10:54 2012 Log: Renmae gtp_quickstart.txt to quickstart.txt Rename gtp.txt to howto.txt http://code.google.com/p/kgtp/source/detail?r=939 Added: /trunk/howto.txt /trunk/quickstart.txt Deleted: /trunk/gtp.txt /trunk/gtp_quickstart.txt ======================================= --- /dev/null +++ /trunk/howto.txt Fri Feb 24 19:10:54 2012 @@ -0,0 +1,1519 @@ + Linux Kernel GDB tracepoint module (KGTP) + ========================================= + By Hui Zhu <teawater@xxxxxxxxx> + https://code.google.com/p/kgtp/wiki/HOWTO + 2012-02-21 + +Table of contents +----------------- + +What is KGTP +Report issues about KGTP +Get info about GDB tracepoint +Install GDB for KGTP +Get KGTP through http +Get KGTP through svn +Config KGTP +Compile KGTP +Install KGTP +Uninstall KGTP +Use KGTP with DKMS +How to get new version GDB +Howto use + Exec it + Make GDB connect to gtp + If GDB on current machine + If GDB on remote machine + Add module symbols to GDB + How to use getmod.py + How to use getmod + Access memory directly + Get register info from Kernel + Get the value of variable from Kernel + How to use use tracepoint condition + How to use trace state variables + Simple trace state variables + Per_cpu trace state variables + Special trace state variables $current_task, $current_task_pid, + $current_thread_info, $cpu_id, $dump_stack, $printk_level, + $printk_format, $printk_tmp, $clock, $rdtsc, $hardirq_count, + $softirq_count and $irq_count + Special trace state variable $no_self_trace + Trace the function return with $kret + Use $ignore_error and $last_errno to ignore the error of tstart + Use $cooked_clock and $cooked_rdtsc the time without KGTP used + Use $xtime_sec and $xtime_nsec get the timespec + How to use performance counters + Show all the traced data of current frame + Get backtrace info(stack dump) from Kernel + Howto let tracepoint output value directly + Output stack dump directly + Switch collect to output the value directly + Use printf command in actions + Get status of KGTP from Kernel + Set the trace buffer into a circular buffer + Do not stop tracepoint when the GDB disconnects + Howto show a variable whose value has been optimized away + Linux kernel "Compile with almost no optimization" patch + Update your GCC + How to get the function pointer point to + If the debug info of the function pointer is not optimized out + If the debug info of the function pointer is optimized out + How to use /sys/kernel/debug/gtpframe + Offline debug + How to use /sys/kernel/debug/gtpframe_pipe + Get the frame info with cat + Get the frame info with getframe + Get the frame info with GDB + + + + +What is KGTP +------------ + +KGTP is a realtime and lightweight Linux Kernel GDB debugger and tracer. ++It makes Linux Kernel supply a GDB remote debug interface. Then GDB in current +machine or remote machine (see "Make GDB connect to gtp") can debug and trace
+Linux through GDB tracepoint without stopping the Linux Kernel. +And even if the board doesn't have GDB on it and doesn't have interface for+remote debug. It can debug the Linux Kernel using offline debug (See "Offline
+debug"). +And it can work with Android +(See https://code.google.com/p/kgtp/wiki/HowToUseKGTPinAndroid). +It supports X86-32, X86-64, MIPS and ARM. + +For new user of KGTP, please go to see quickstart.txt. + + + + +Report issues about KGTP +------------------------+You can post it in https://code.google.com/p/kgtp/issues/list or write Email
+to teawater@xxxxxxxxxx + + + + +Get info about GDB tracepoint +-----------------------------+Please goto http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html
+ + + + +Install GDB for KGTP +------------------------+The GDB that older than 7.3 have some bugs of tracepoint. And some functions
+of GDB are not very well. +So please goto https://code.google.com/p/gdbt/ to install GDB for KGTP. + + + + +Get KGTP through http +---------------------+Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to download
+the package. + + + + +Get KGTP through svn +-------------------- +Some people have trouble with access to KGTP website. You can access kgtp +through svn: + +------------------------------------------------------------ +svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only +------------------------------------------------------------ + +kgtp-read-only/tags/ Present for each release of KGTP. +kgtp-read-only/trunk/ Present for the main trunk of KGTP. + + + + +Config KGTP +----------- + +Before compiling KGTP, you can choose which kernel you want build with and +which compiler you want by making changes to the Makefile in your KGTP +repository. +For example: + +------------------------------------------- +KERNELDIR := /lib/modules/`uname -r`/build +CROSS_COMPILE := +------------------------------------------- ++KERELDIR is set to the directory which holds the kernel you want to build for.
+By default, it is set to the kernel that you are running.+CROSS_COMPILE is set the compiler that you want to build the KGTP. Empty mean
+use current compiler. +ARCH is the architecture. + +------------------------------------------ +KERNELDIR := /home/teawater/kernel/bamd64 +CROSS_COMPILE :=x86_64-glibc_std- +ARCH := x86_64 +------------------------------------------ + +KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will +use x86_64-glibc_std-gcc. + + + + +Compile KGTP +------------ + +For normal use: + +--------- +cd kgtp/ +make +--------- + + + + +Compile KGTP with old Linux Kernel +---------------------------------- ++Most of time, KGTP can auto select right options to build with old Linux Kernel. +But if you want config special options with yourself, you can read following
+part. + +-------------------- +make AUTO=0 +-------------------- +With this option, KGTP will not auto select any build options. + +-------------------- +make AUTO=0 FRAME_SIMPLE=1 +-------------------- +With this option, KGTP will use simple frame instead of ring buffer.+The simple frame doesn't support gtpframe_pipe. Use it can make KGTP can build
+with old Kernel that doesn't support ring buffer. + +-------------------- +make AUTO=0 CLOCK_CYCLE=1 +-------------------- +With this option, $clock will return rdtsc value instead of local_clock. + +-------------------- +make AUTO=0 USE_PROC=1 +-------------------- +With this option, KGTP will use procfs instead of debugfs. + +The options can use together, for example: +----------------------------------- +make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1 +----------------------------------- +This build command make KGTP build success with Linux Kernel 2.6.18. + + + + +Install KGTP +------------ + +------------------ +cd kgtp/ +sudo make install +------------------ + + + + +Uninstall KGTP +-------------- + +-------------------- +cd kgtp/ +sudo make uninstall +-------------------- + + + + +Use KGTP with DKMS +------------------ + +-------------------- +cd kgtp/ +sudo make dkms +-------------------- +This commands will copy the files of KGTP to the directory that DKMS need. +Then you can use DKMS commands to control KGTP. +Please goto http://linux.dell.com/dkms/manpage.html to see how to use DKMS. + + + + +How to get new version GDB +-------------------------- ++The old version GDB such as GDB 7.2 have some bugs about tracepoint that KGTP
+need. So I suggest you update your version older than 7.2. +If you use UBUNTU, use can get the new version from +https://lkml.org/lkml/2011/6/4/65 + + + + +Howto use +--------- + +Exec it +------- + +If you have installed KGTP in your system, you can: + +------------------ +sudo modprobe gtp +------------------ + +Or you can use the kgtp module in the directory. + +------------------- +cd kgtp/ +sudo insmod gtp.ko +------------------- + + + +Make GDB connect to gtp +----------------------- + + +If GDB on current machine +------------------------- + +--------------------------------- +sudo gdb ./vmlinux +(gdb) target remote /sys/kernel/debug/gtp +Remote debugging using /sys/kernel/debug/gtp +0x0000000000000000 in ?? () +--------------------------------- +After that, you can begin to use GDB command trace the Linux Kernel. + + +If GDB on remote machine +------------------------ + +--------------------------------------------- +#Open the KGTP interface in current machine. +sudo su +nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp+(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old version
+netcat.) +#Let gdb connect to the port 1234 +gdb ./vmlinux +(gdb) target remote xxx.xxx.xxx.xxx:1234 +--------------------------------------------- +After that, you can begin to use GDB command trace the Linux Kernel. + + + +Add module symbols to GDB +------------------------- ++Sometimes you need to add a Linux kernel module's symbols to GDB to debug it. +Add symbols with hand is not very easy, so KGTP package include an GDB python
+script "getmod.py" and a program "getmod" can help you. + + +How to use getmod.py +-------------------- + +Connect to KGTP before use the getmod.py. +(gdb) source ~/kgtp/getmod.py +Then this script will auto load the Linux kernel module's symbols to GDB. + + +How to use getmod +----------------- +"getmod" is written by C so you can use it anywhere even if in an embedded +environment. +For example: + +--------------------------------------------------------------------------------+#Following command save Linux Kernel module info to the file ~/tmp/mi in GDB
+#command format. +sudo getmod >~/tmp/mi +#in gdb part: +(gdb) source ~/tmp/mi+add symbol table from file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
+ .text_addr = 0xf80de000 + .note.gnu.build-id_addr = 0xf80de088 + .exit.text_addr = 0xf80de074 + .init.text_addr = 0xf8118000 + .rodata.str1.1_addr = 0xf80de0ac + .rodata_addr = 0xf80de0c0 + __mcount_loc_addr = 0xf80de9c0 + .data_addr = 0xf80de9e0 + .gnu.linkonce.this_module_addr = 0xf80dea00+#After this GDB command, all the Linux Kernel module info is loaded into GDB.
+-------------------------------------------------------------------------------- + +If you use remote debug or offline debug, maybe you need change the base +directory. Following example is for it. + +--------------------------------------------------------------------------------+#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r /home/teawater/kernel/b26
+sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi +-------------------------------------------------------------------------------- + + + +Access memory directly +---------------------- + +After connect the KGTP, you can access most of memory directly. +For example, you can access to "jiffies_64" with following command: +--------------- +p jiffies_64 +--------------- + +Or you can access to the first entry of "static LIST_HEAD(modules)" with +following command: +--------------------------------------------------------------------------------+p *((struct module *)((char *)modules->next - ((size_t) &(((struct module *)0)->list))))
+-------------------------------------------------------------------------------- + + + +Get register info from Kernel +----------------------------- + +The following is an example that records the value of all registers +when "vfs_readdir" is called. + +-------------------------------------------------------------------------------- +(gdb) target remote /sys/kernel/debug/gtp +(gdb) trace vfs_readdir +Tracepoint 1 at 0xc01a1ac0: file +/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23. +(gdb) actions +Enter actions for tracepoint 1, one per line. +End with a line saying just "end". +>collect $reg +>end +(gdb) tstart +(gdb) shell ls +(gdb) tstop +(gdb) tfind +Found trace frame 0, tracepoint 1+#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
+ buf=0xc0d09f90) at readdir.c:23 +23 readdir.c: No such file or directory. + in readdir.c +(gdb) info reg +eax 0xc5528d00 -984445696 +ecx 0xc0d09f90 -1060069488 +edx 0xc01a1900 -1072031488 +ebx 0xfffffff7 -9 +esp 0xc0d09f8c 0xc0d09f8c +ebp 0x0 0x0 +esi 0x8061480 134616192 +edi 0xc5528d00 -984445696 +eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1> +eflags 0x286 [ PF SF IF ] +cs 0x60 96 +ss 0x8061480 134616192 +ds 0x7b 123 +es 0x7b 123 +fs 0x0 0 +gs 0x0 0 +(gdb) tfind +Found trace frame 1, tracepoint 1 +0xc01a1ac1 23 in readdir.c +(gdb) info reg +eax 0xc5528d00 -984445696 +ecx 0xc0d09f90 -1060069488 +edx 0xc01a1900 -1072031488 +ebx 0xfffffff7 -9 +esp 0xc0d09f8c 0xc0d09f8c +ebp 0x0 0x0 +esi 0x8061480 134616192 +edi 0xc5528d00 -984445696 +eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1> +eflags 0x286 [ PF SF IF ] +cs 0x60 96 +ss 0x8061480 134616192 +ds 0x7b 123 +es 0x7b 123 +fs 0x0 0 +gs 0x0 0 +-------------------------------------------------------------------------------- + + + +Get the value of variable from Kernel +------------------------------------- + +The following is an example that records the value of "jiffies_64" when the +function "vfs_readdir" is called: + +-------------------------------------------------------------------------------- +(gdb) target remote /sys/kernel/debug/gtp +(gdb) trace vfs_readdir+Tracepoint 1 at 0xc01ed740: file /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
+(gdb) actions +Enter actions for tracepoint 1, one per line. +End with a line saying just "end". +>collect jiffies_64 +>collect file->f_path.dentry->d_iname +>end +(gdb) tstart +(gdb) shell ls+arch drivers include kernel mm Module.symvers security System.map virt +block firmware init lib modules.builtin net sound t vmlinux +crypto fs ipc Makefile modules.order scripts source usr vmlinux.o
+(gdb) tstop +(gdb) tfind +Found trace frame 0, tracepoint 1+#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580 <filldir64>, buf=0xd6dfdf90)
+ at readdir.c:24 +24 { +(gdb) p jiffies_64 +$1 = 4297248706 +(gdb) p file->f_path.dentry->d_iname +$1 = "b26", '\000' <repeats 28 times> +-------------------------------------------------------------------------------- + + + +How to use use tracepoint condition +----------------------------------- + +http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html +Like breakpoints, we can set conditions on tracepoints. The speed of+tracepoints is faster than breakpoints because KGTP can do all the condition
+checks. +For example: + +------------------------------ +(gdb) trace handle_irq +(gdb) condition 1 (irq == 47) +------------------------------ + +This action of tracepoint 1 will work only when irq number is 47. + + + +How to use trace state variables +-------------------------------- + +http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html +Tracepoints have special variables. The variables can be traced directly, +or used in tracepoint conditions.+Note that just GDB 7.2.1 and later versions support use trace state variables +directly, the old version of GDB can show the value of trace state variables
+through command "info tvariables". + + +Simple trace state variables +---------------------------- + +Define a trace state variable $c. + +------------------- +(gdb) tvariable $c +------------------- + +Trace state variable $c is created with initial value 0. +The following action uses $c to count how many irqs happened in the kernel. + +----------------------------------------------------------------------- +(gdb) trace handle_irq +(gdb) actions +Enter actions for tracepoint 3, one per line. +End with a line saying just "end". +>collect $c #Save current value of $c to the trace frame buffer. +>teval $c=$c+1 #Increase the $c. +>end +----------------------------------------------------------------------- ++Also, you can set a value of variable to trace state variable, but don't forget
+covert variable to "uint64_t". + +----------------------------------------------------------------------- +>teval $c=(uint64_t)jiffies_64 +----------------------------------------------------------------------- + +You can get the current value of $c while the trace is running or stopped. + +---------------------------------- +(gdb) tstart +(gdb) info tvariables +$c 0 31554 +(gdb) p $c +$5 = 33652 +(gdb) tstop +(gdb) p $c +$9 = 105559 +---------------------------------- + +When using tfind, you can parse the trace frame buffer. If the value of a +trace state variable is collected, you can parse it out. + +------------------------------ +(gdb) tstop +(gdb) tfind +(gdb) info tvariables +$c 0 0 +(gdb) p $c +$6 = 0 +(gdb) tfind 100 +(gdb) p $c +$7 = 100 +------------------------------ ++If need, the tracepoint action that access the simple trace state variables will
+auto lock the spin lock for trace state variables. So it can handle race +condition about trace state variables.+The following example is OK even if it running a machine that have more than
+one CPU. +------------------------------------- +teval $c=$c+1 +------------------------------------- + + +Per_cpu trace state variables +----------------------------- + +Per_cpu trace state variables are special simple trace state variables.+When tracepoint action access to it, it will access to this CPU special trace
+state variables. + +It have 2 advantages:+1. The tracepoint actions that access to per_cpu trace state variables don't +have the race conditon issue. So it don't need lock the spin lock for trace +state variables. It is faster than simple trace state variables on multi-core
+machine.+2. Write the action that count some CPU special thing with it is easier than
+simple trace state variables. + +To define per_cpu trace state variables, you need named it in +format: +"per_cpu_"+string+CPU_id +or +"pc_"+string+CPU_id +Following example will define a series of per_cpu trace state variables +in a 4 COREs CPU machine with string "count": +------------------------------------- +tvariable $pc_count0 +tvariable $pc_count1 +tvariable $pc_count2 +tvariable $pc_count3 +------------------------------------- + +You can use compatibility better way to do it: +----------------------------------------- +set $tmp=0 +while $tmp<$cpu_number + eval "tvariable $pc_count%d",$tmp + set $tmp=$tmp+1 +end +----------------------------------------- + +Tracepoint action can access anyone of a series of per_cpu trace state +variables. KGTP will auto access the one of CPU that it running on. +For example: +---------------------------------------- +trace vfs_read +actions +teval $pc_count0=$pc_count0+1 +end +----------------------------------------+These GDB commands define a tracepoint that count the times that call vfs_read
+of each CPU. + + +Special trace state variables $current_task, $current_task_pid, +$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format, +$printk_tmp, $clock, $rdtsc, $hardirq_count, $softirq_count and $irq_count +--------------------------------------------------------------------------- + +KGTP special trace state variables $current_task, $current_thread_info,+$cpu_id and $clock can very easy to access to some special value. You can see +them when GDB connects to the KGTP. You can use them in tracepoint conditions
+or actions.+Access $current_task in tracepoint condition and action will get that returns
+of get_current(). +Access $current_task_pid in tracepoint condition and action will get that +returns of get_current()->pid.+Access $current_thread_info in tracepoint condition and action will get that
+returns of current_thread_info(). +Access $cpu_id in tracepoint condition and action will get that returns of +smp_processor_id(). +Access $clock in tracepoint condition and action will get that returns of +local_clock() that return the timestamp in nanoseconds.+$rdtsc is only available on X86 and X86_64 architecture. Access it in anytime
+will get current value of TSC with instruction RDTSC.+Access $hardirq_count in tracepoint condition and action will get that returns
+of hardirq_count().+Access $softirq_count in tracepoint condition and action will get that returns
+of softirq_count(). +Access $irq_count in tracepoint condition and action will get that returns +of irq_count(). ++And KGTP has other special trace state variables $dump_stack, $printk_level,
+$printk_format and $printk_tmp. All of them output their values directly, +as can be seen in "Howto let tracepoint output value directly". ++The following example counts in $c how many vfs_read calls that process 16663
+does and collects the struct thread_info of current task: + +-------------------------------------------------------------------------------- +(gdb) target remote /sys/kernel/debug/gtp+(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid == 16663)
+(gdb) tvariable $c +(gdb) actions +Enter actions for tracepoint 4, one per line. +End with a line saying just "end". +>teval $c=$c+1 +>collect (*(struct thread_info *)$current_thread_info) +>end +(gdb) tstart +(gdb) info tvariables +Name Initial Current +$c 0 184 +$current_task 0 <unknown> +$current_thread_info 0 <unknown> +$cpu_id 0 <unknown> +(gdb) tstop +(gdb) tfind +(gdb) p *(struct thread_info *)$current_thread_info+$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status = 0, cpu = 1, preempt_count = 2, addr_limit = { + seg = 4294967295}, restart_block = {fn = 0xc0159fb0 <do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11, + arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags = 1, bitset = 78, time = 977063750, + uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb, expires = 335007449089}, poll = { + ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78, tv_nsec = 977063750}}}, + sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack = 0xef340044 "", uaccess_err = 0}
+-------------------------------------------------------------------------------- + +Another example shows how much sys_read() executes in each CPU. + +-------------------------------------- +tvariable $c0 +tvariable $c1 +trace sys_read + condition $bpnum ($cpu_id == 0) + commands + teval $c0=$c0+1 + end +trace sys_read + condition $bpnum ($cpu_id == 1) + commands + teval $c1=$c1+1 + end +info tvariables +Name Initial Current +$current_task 0 <unknown> +$cpu_id 0 <unknown> +$c0 0 3255 +$c1 0 1904 +-------------------------------------- + +sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. + + +Special trace state variable $no_self_trace +-------------------------------------------- + +$no_self_trace is different with the special trace state variables in the +previous section. It is used to control the behavior of tracepoint.+If the action of a tracepoint include a command access to the $no_self_trace. +The tracepoint will not trace anything if the current_task is the a KGTP self
+process (GDB, netcat, getframe or some others process that access to the +interface of KGTP).+For example, if we want trace vfs_read or something that have process context,
+and we don't want trace the operation of KGTP self process. Add following +command to the action: + +-------------------------------------- +collect $no_self_trace +-------------------------------------- + +Please note that the code that doesn't have process context (Irq handler, +softirq) doesn't need set this variable. + + +Trace the function return with $kret +------------------------------------ ++Sometime, set the tracepoint to the end of function is hard because the Kernel
+is compiled with optimization. At this time, you can get help from $kret. + +$kret is a special trace state variable like $no_self_trace. When you set +value of it inside the action of tracepoint, this tracepoint be set with +kretprobe instead of kprobe. Then it can trace the end of this function. + +Following part is an example: + +------------------------------------------------------------------------------ +target remote /sys/kernel/debug/gtp +#"*(function_name)" format can make certain that GDB send the first address +#of function to KGTP. +trace *(vfs_read) +action +teval $kret=0 +#Following part you can set commands that you want. +------------------------------------------------------------------------------ + + +Use $ignore_error and $last_errno to ignore the error of tstart +--------------------------------------------------------------- + +If KGTP got any error of tstart, this command will get fail. +But sometime we need ignore this error and let KGTP keep work. +For example: If you set tracepoint on the inline function spin_lock. +This tracepoint will be set to a lot of addresses that some of them cannot +be set kprobe. It will make tstart get fail. You can use "$ignore_error" +ignore this error. +And the last error number will available in "$last_errno". + +--------------------------------- +tvariable $ignore_error=1 +--------------------------------- +This command will open ignore. + +--------------------------------- +tvariable $ignore_error=0 +--------------------------------- +This command will close ignore. + + +Use $cooked_clock and $cooked_rdtsc the time without KGTP used +-------------------------------------------------------------- + +Access these two trace state variables can get the time without KGTP used.+Then we can get more close to really time that a part of code used even if the
+actions of tracepoint is very complex. They will be introduce in +Cookbook (coming soon). + + +Use $xtime_sec and $xtime_nsec get the timespec +----------------------------------------------- + +Access these two trace state variables will return the time of day in +a timespec that use getnstimeofday. +$xtime_sec will access to the second part of a timespec. +$xtime_nsec will access to the nanosecond part of a timespec. + + + +How to use performance counters +------------------------------- ++Performance counters are special hardware registers available on most modern +CPUs. These registers count the number of certain types of hw events: such as +instructions executed, cachemisses suffered, or branches mis-predicted - without
+slowing down the kernel or applications. These registers can also trigger +interrupts when a threshold number of events have passed - and can thus be +used to profile the code that runs on that CPU. ++The Linux Performance Counter subsystem called perf event can get the value of
+performance counter. You can access it through KGTP perf event trace state +variables. + +Please goto read the file tools/perf/design.txt in Linux Kernel to get more +info about perf event. + + +Define a perf event trace state variable +---------------------------------------- + +Access an performance counter need define following trace state variable: +----------------------------------------------------------------------------- +"pe_cpu_"+tv_name Define the the CPU id of the performance counter. +"pe_type_"+tv_name Define the the type of the performance counter. +"pe_config_"+tv_name Define the the config of the performance counter. +"pe_en_"+tv_name This the switch to enable or disable the performance + counter. + The performance counter is disable in default. +"pe_val_"+tv_name Access this variable can get the value of the + performance counter. +----------------------------------------------------------------------------- + + +Define a per_cpu perf event trace state variable +------------------------------------------------ + +Define a per_cpu perf event trace state variable is same with define +"Per_cpu_trace_state_variables". + +------------------------------------------ +"pc_pe_"+perf_event type+string+CPU_id +------------------------------------------ ++Note that if you define a per_cpu perf event trace state variable, you will not
+need define the "pe_cpu_" because KGTP already get it from per_cpu id. + + +The perf event type and config +------------------------------ + +The type of perf event can be: +----------------------------------------------------------- +0 PERF_TYPE_HARDWARE +1 PERF_TYPE_SOFTWARE +2 PERF_TYPE_TRACEPOINT +3 PERF_TYPE_HW_CACHE +4 PERF_TYPE_RAW +5 PERF_TYPE_BREAKPOINT +----------------------------------------------------------- + +If the type is 0(PERF_TYPE_HARDWARE), the config can be: +----------------------------------------------------------- +0 PERF_COUNT_HW_CPU_CYCLES +1 PERF_COUNT_HW_INSTRUCTIONS +2 PERF_COUNT_HW_CACHE_REFERENCES +3 PERF_COUNT_HW_CACHE_MISSES +4 PERF_COUNT_HW_BRANCH_INSTRUCTIONS +5 PERF_COUNT_HW_BRANCH_MISSES +6 PERF_COUNT_HW_BUS_CYCLES +7 PERF_COUNT_HW_STALLED_CYCLES_FRONTEND +8 PERF_COUNT_HW_STALLED_CYCLES_BACKEND +----------------------------------------------------------- + +If the type is 3(PERF_TYPE_HW_CACHE), the config need to divide to 3 parts: +First one is cache id, it need be << 0 before set to config: +----------------------------------------------------------- +0 PERF_COUNT_HW_CACHE_L1D +1 PERF_COUNT_HW_CACHE_L1I +2 PERF_COUNT_HW_CACHE_LL +3 PERF_COUNT_HW_CACHE_DTLB +4 PERF_COUNT_HW_CACHE_ITLB +5 PERF_COUNT_HW_CACHE_BPU +----------------------------------------------------------- +Second one is cache op id, it need be << 8 before set to config: +----------------------------------------------------------- +0 PERF_COUNT_HW_CACHE_OP_READ +1 PERF_COUNT_HW_CACHE_OP_WRITE +2 PERF_COUNT_HW_CACHE_OP_PREFETCH +----------------------------------------------------------- +Last one is cache op result id, it need be << 16 before set to config: +----------------------------------------------------------- +0 PERF_COUNT_HW_CACHE_RESULT_ACCESS +1 PERF_COUNT_HW_CACHE_RESULT_MISS +----------------------------------------------------------- +If you want get the perf count of PERF_COUNT_HW_CACHE_L1I(1), +PERF_COUNT_HW_CACHE_OP_WRITE(1) and PERF_COUNT_HW_CACHE_RESULT_MISS(1), +you can use: +----------------------------------------------------------- +tvariable $pe_config_cache=1 | (1 << 8) | (1 << 16) +----------------------------------------------------------- + +tools/perf/design.txt in Linux Kernel have more info about type and config +of perf event. + + +Enable and disable all the perf event in a CPU with $pc_pe_en +------------------------------------------------------------- + +I think the best way that count a part of code with performance counters is+enable all the count in the begin of the code and disable all of them in the +end. You can do it with "pe_en_". But if you have a lot of perf event trace +state variables. That will make the tracepoint action very big. $pc_pe_en is
+for this issue. + +You can enable all the perf event trace state variables in current CPU with +following action: +-------------------- +teval $pc_pe_en=1 +-------------------- +Disable them with set $pc_pe_en to 0. +-------------------- +teval $pc_pe_en=0 +-------------------- + + +GDB scripts to help with set and get the perf event trace state variables +------------------------------------------------------------------------- ++Following is a GDB script define two commands dpe and spe to help define and
+show the perf event trace state variables.+You can put it to the ~/.gdbinit or your tracepoint script. Then you can use
+this two commands in GDB directly. +------------------------------------------------------------------------------- +define dpe + if ($argc < 2) + printf "Usage: dpe pe_type pe_config [enable]\n" + end + if ($argc >= 2) + set $tmp=0 + while $tmp<$cpu_number + eval "tvariable $pc_pe_type_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg0 + eval "tvariable $pc_pe_config_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg1 + eval "tvariable $pc_pe_val_%d%d_%d=0",$arg0, $arg1, $tmp + if ($argc >= 3) + eval "tvariable $pc_pe_en_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg2 + end + set $tmp=$tmp+1 + end + end +end + +document dpe +Usage: dpe pe_type pe_config [enable] +end + +define spe + if ($argc != 2 && $argc != 3) + printf "Usage: spe pe_type pe_config [cpu_id]\n" + end + if ($argc == 2) + set $tmp=0 + while $tmp<$cpu_number+ eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $tmp
+ set $tmp=$tmp+1 + end + end + if ($argc == 3)+ eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $arg2
+ end +end + +document spe +Usage: spe pe_type pe_config [cpu_id] +end +------------------------------------------------------------------------------- ++Following is an example to use it get the performance counters of function tcp_v4_rcv:
+------------------------------------------------------------------------------- +#Connect to kgtp +target remote /sys/kernel/debug/gtp+#Define 3 pe tvs for PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_CACHE_MISSES and PERF_COUNT_HW_BRANCH_MISSES.
+dpe 0 0 +dpe 0 3 +dpe 0 5 +#enable the performance counters of this CPU in the begin of this function. +trace tcp_v4_rcv ***The diff for this file has been truncated for email.*** ======================================= --- /dev/null +++ /trunk/quickstart.txt Fri Feb 24 19:10:54 2012 @@ -0,0 +1,250 @@ + Linux Kernel GDB tracepoint module (KGTP) quick start + ===================================================== + By Hui Zhu <teawater@xxxxxxxxx> + https://code.google.com/p/kgtp/wiki/Quickstart + 2011-09-12 + +Table of contents +----------------- +Ubuntu +Fedora + + + + +Ubuntu +------ + +Install GDB for KGTP +-------------------- + +This GDB's filename is different with the current GDB that you are using. +So please don't worry that it affect current GDB that your are using. + +For the Ubuntu 10.04 or later, running the following line at a terminal: +sudo add-apt-repository ppa:teawater/gdb-$(lsb_release -rs) +sudo apt-get update +sudo apt-get install gdb-release ++For the Ubuntu older than 10.04, please go to https://code.google.com/p/gdbt/
+get howto install GDB for KGTP from source. + + + +Install Linux kernel packages that KGTP need +-------------------------------------------- + +Please ignore this section if the Linux kernel of your system is built by +yourself. + +Install the Linux kernel debug image +------------------------------------ + +Add debug source to the sources list of Ubuntu +---------------------------------------------- ++Create an /etc/apt/sources.list.d/ddebs.list by running the following line at
+a terminal:+echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
+sudo tee -a /etc/apt/sources.list.d/ddebs.list ++Stable releases (not alphas and betas) require three more lines adding to the
+same file, which is done by the following terminal command:+echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse +deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse +deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
+sudo tee -a /etc/apt/sources.list.d/ddebs.list + +Import the debug symbol archive signing key: +sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 + +Then run: +sudo apt-get update + +Get Linux kernel debug image +---------------------------- +sudo apt-get install linux-image-$(uname -r)-dbgsym + + +Install the Linux kernel headers +-------------------------------- + +Please ignore this section if the Linux kernel of your system is built by +yourself. +sudo apt-get install linux-headers-generic + + +Install the Linux kernel source +------------------------------- + +Install the source package: +sudo apt-get install linux-source + +Uncompress the source package: +sudo mkdir -p /build/buildd/+sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2 -C /build/buildd/ +sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//') /build/buildd/linux-$(uname -r | sed 's/-.*//')
+ + + +Install GCC +----------- + +sudo apt-get install gcc + + + +Get and build KGTP +------------------ + +Install subversion: +sudo apt-get install subversion + +Get the source of KGTP with subversion and put it to directory "kgtp": +svn checkout https://kgtp.googlecode.com/svn/trunk kgtp + +Build KGTP: +cd kgtp +make + + + +Use KGTP +-------- + +Mount the sysfs and debugfs: +sudo mount -t sysfs none /sys/ +sudo mount -t debugfs none /sys/kernel/debug/ + +Insert the KGTP module to the current Linux Kernel: +cd kgtp +sudo insmod gtp.ko + +Use GDB connect to KGTP: +sudo gdb-release /usr/lib/debug/boot/vmlinux-$(uname -r) +(gdb) target remote /sys/kernel/debug/gtp + +Do a very simple trace: +(gdb) trace vfs_readdir+Tracepoint 1 at 0xc02289f0: file /build/buildd/linux-2.6.35/fs/readdir.c, line 23.
+(gdb) actions +Enter actions for tracepoint 1, one per line. +End with a line saying just "end". +>collect $reg +>end +(gdb) tstart +(gdb) shell ls +vmlinux-2.6.35-30-generic +(gdb) tstop +(gdb) tfind +Found trace frame 0, tracepoint 1 +#0 vfs_readdir (file=0x0, filler=0x163d8ae3, buf=0x18c0) at readdir.c:23 +23 { + + + +End +--- + +Now, you can begin to rock and roll your Linux kernel with KGTP and GDB. +Please go to see gtp.txt to get more message about howto use KGTP. + + + + +Fedora +------ + +Install GDB for KGTP +-------------------- + +Please go to https://code.google.com/p/gdbt/ get howto install GDB for KGTP +from source. + + + +Install Linux kernel packages that KGTP need +-------------------------------------------- + +Please ignore this section if the Linux kernel of your system is built +by yourself. + + +Install the Linux kernel debug image +------------------------------------ + +sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo + + +Install the Linux kernel devel package +-------------------------------------- + +sudo yum install kernel-devel-$(uname -r) + + + +Install GCC +----------- + +sudo yum install gcc + + + +Get and build KGTP +------------------ + +Install subversion: +sudo yum install subversion + +Get the source of KGTP with subversion and put it to directory "kgtp": +svn checkout https://kgtp.googlecode.com/svn/trunk kgtp + +Build KGTP: +cd kgtp +make + + + +Use KGTP +-------- + +Mount the sysfs and debug fs: +sudo mount -t sysfs none /sys/ +sudo mount -t debugfs none /sys/kernel/debug/ + +Insert the KGTP module to the current Linux Kernel: +cd kgtp +sudo insmod gtp.ko + +Use GDB connect to KGTP: +sudo gdb-release /usr/lib/debug/lib/modules/$(uname -r)/vmlinux +(gdb) target remote /sys/kernel/debug/gtp + +Do a very simple trace: +(gdb) trace vfs_readdir +Tracepoint 1 at 0xffffffff8110ec9b: file fs/readdir.c, line 23. +(gdb) actions +Enter actions for tracepoint 1, one per line. +End with a line saying just "end". +>collect $reg +>end +(gdb) tstart +(gdb) shell ls+co.patch getframe getmod.c gtp.mod.c gtp.txt perf_event.c +dkms.conf getframe.c getmod.py gtp.mod.o Makefile ring_buffer.c +dkms_others_install.sh getgtprsp.pl gtp.c gtp.o modules.order ring_buffer.h +dkms_others_uninstall.sh getmod gtp.ko gtp.patch Module.symvers
+(gdb) tstop +(gdb) tfind +Found trace frame 0, tracepoint 1+#0 vfs_readdir (file=0xffff880019d3df00, filler=0xffffffff8110eb16 <filldir>, buf=0xffff880003b39f38)
+ at fs/readdir.c:23 +23 { + + + +End +--- + +Now, you can begin to rock and roll your Linux kernel with KGTP and GDB. +Please go to HOWTO to get more message about howto use KGTP. ======================================= --- /trunk/gtp.txt Fri Feb 24 19:09:02 2012 +++ /dev/null @@ -1,1519 +0,0 @@ - Linux Kernel GDB tracepoint module (KGTP) - ========================================= - By Hui Zhu <teawater@xxxxxxxxx> - https://code.google.com/p/kgtp/wiki/HOWTO - 2012-02-21 - -Table of contents ------------------ - -What is KGTP -Report issues about KGTP -Get info about GDB tracepoint -Install GDB for KGTP -Get KGTP through http -Get KGTP through svn -Config KGTP -Compile KGTP -Install KGTP -Uninstall KGTP -Use KGTP with DKMS -How to get new version GDB -Howto use - Exec it - Make GDB connect to gtp - If GDB on current machine - If GDB on remote machine - Add module symbols to GDB - How to use getmod.py - How to use getmod - Access memory directly - Get register info from Kernel - Get the value of variable from Kernel - How to use use tracepoint condition - How to use trace state variables - Simple trace state variables - Per_cpu trace state variables - Special trace state variables $current_task, $current_task_pid, - $current_thread_info, $cpu_id, $dump_stack, $printk_level, - $printk_format, $printk_tmp, $clock, $rdtsc, $hardirq_count, - $softirq_count and $irq_count - Special trace state variable $no_self_trace - Trace the function return with $kret - Use $ignore_error and $last_errno to ignore the error of tstart - Use $cooked_clock and $cooked_rdtsc the time without KGTP used - Use $xtime_sec and $xtime_nsec get the timespec - How to use performance counters - Show all the traced data of current frame - Get backtrace info(stack dump) from Kernel - Howto let tracepoint output value directly - Output stack dump directly - Switch collect to output the value directly - Use printf command in actions - Get status of KGTP from Kernel - Set the trace buffer into a circular buffer - Do not stop tracepoint when the GDB disconnects - Howto show a variable whose value has been optimized away - Linux kernel "Compile with almost no optimization" patch - Update your GCC - How to get the function pointer point to - If the debug info of the function pointer is not optimized out - If the debug info of the function pointer is optimized out - How to use /sys/kernel/debug/gtpframe - Offline debug - How to use /sys/kernel/debug/gtpframe_pipe - Get the frame info with cat - Get the frame info with getframe - Get the frame info with GDB - - - - -What is KGTP ------------- - -KGTP is a realtime and lightweight Linux Kernel GDB debugger and tracer. --It makes Linux Kernel supply a GDB remote debug interface. Then GDB in current -machine or remote machine (see "Make GDB connect to gtp") can debug and trace
-Linux through GDB tracepoint without stopping the Linux Kernel. -And even if the board doesn't have GDB on it and doesn't have interface for-remote debug. It can debug the Linux Kernel using offline debug (See "Offline
-debug"). -And it can work with Android -(See https://code.google.com/p/kgtp/wiki/HowToUseKGTPinAndroid). -It supports X86-32, X86-64, MIPS and ARM. - -For new user of KGTP, please go to see quickstart.txt. - - - - -Report issues about KGTP --------------------------You can post it in https://code.google.com/p/kgtp/issues/list or write Email
-to teawater@xxxxxxxxxx - - - - -Get info about GDB tracepoint -------------------------------Please goto http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html
- - - - -Install GDB for KGTP --------------------------The GDB that older than 7.3 have some bugs of tracepoint. And some functions
-of GDB are not very well. -So please goto https://code.google.com/p/gdbt/ to install GDB for KGTP. - - - - -Get KGTP through http -----------------------Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to download
-the package. - - - - -Get KGTP through svn --------------------- -Some people have trouble with access to KGTP website. You can access kgtp -through svn: - ------------------------------------------------------------- -svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only ------------------------------------------------------------- - -kgtp-read-only/tags/ Present for each release of KGTP. -kgtp-read-only/trunk/ Present for the main trunk of KGTP. - - - - -Config KGTP ------------ - -Before compiling KGTP, you can choose which kernel you want build with and -which compiler you want by making changes to the Makefile in your KGTP -repository. -For example: - -------------------------------------------- -KERNELDIR := /lib/modules/`uname -r`/build -CROSS_COMPILE := -------------------------------------------- --KERELDIR is set to the directory which holds the kernel you want to build for.
-By default, it is set to the kernel that you are running.-CROSS_COMPILE is set the compiler that you want to build the KGTP. Empty mean
-use current compiler. -ARCH is the architecture. - ------------------------------------------- -KERNELDIR := /home/teawater/kernel/bamd64 -CROSS_COMPILE :=x86_64-glibc_std- -ARCH := x86_64 ------------------------------------------- - -KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will -use x86_64-glibc_std-gcc. - - - - -Compile KGTP ------------- - -For normal use: - ---------- -cd kgtp/ -make ---------- - - - - -Compile KGTP with old Linux Kernel ----------------------------------- --Most of time, KGTP can auto select right options to build with old Linux Kernel. -But if you want config special options with yourself, you can read following
-part. - --------------------- -make AUTO=0 --------------------- -With this option, KGTP will not auto select any build options. - --------------------- -make AUTO=0 FRAME_SIMPLE=1 --------------------- -With this option, KGTP will use simple frame instead of ring buffer.-The simple frame doesn't support gtpframe_pipe. Use it can make KGTP can build
-with old Kernel that doesn't support ring buffer. - --------------------- -make AUTO=0 CLOCK_CYCLE=1 --------------------- -With this option, $clock will return rdtsc value instead of local_clock. - --------------------- -make AUTO=0 USE_PROC=1 --------------------- -With this option, KGTP will use procfs instead of debugfs. - -The options can use together, for example: ------------------------------------ -make AUTO=0 FRAME_SIMPLE=1 CLOCK_CYCLE=1 ------------------------------------ -This build command make KGTP build success with Linux Kernel 2.6.18. - - - - -Install KGTP ------------- - ------------------- -cd kgtp/ -sudo make install ------------------- - - - - -Uninstall KGTP --------------- - --------------------- -cd kgtp/ -sudo make uninstall --------------------- - - - - -Use KGTP with DKMS ------------------- - --------------------- -cd kgtp/ -sudo make dkms --------------------- -This commands will copy the files of KGTP to the directory that DKMS need. -Then you can use DKMS commands to control KGTP. -Please goto http://linux.dell.com/dkms/manpage.html to see how to use DKMS. - - - - -How to get new version GDB --------------------------- --The old version GDB such as GDB 7.2 have some bugs about tracepoint that KGTP
-need. So I suggest you update your version older than 7.2. -If you use UBUNTU, use can get the new version from -https://lkml.org/lkml/2011/6/4/65 - - - - -Howto use ---------- - -Exec it -------- - -If you have installed KGTP in your system, you can: - ------------------- -sudo modprobe gtp ------------------- - -Or you can use the kgtp module in the directory. - -------------------- -cd kgtp/ -sudo insmod gtp.ko -------------------- - - - -Make GDB connect to gtp ------------------------ - - -If GDB on current machine -------------------------- - ---------------------------------- -sudo gdb ./vmlinux -(gdb) target remote /sys/kernel/debug/gtp -Remote debugging using /sys/kernel/debug/gtp -0x0000000000000000 in ?? () ---------------------------------- -After that, you can begin to use GDB command trace the Linux Kernel. - - -If GDB on remote machine ------------------------- - ---------------------------------------------- -#Open the KGTP interface in current machine. -sudo su -nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp-(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old version
-netcat.) -#Let gdb connect to the port 1234 -gdb ./vmlinux -(gdb) target remote xxx.xxx.xxx.xxx:1234 ---------------------------------------------- -After that, you can begin to use GDB command trace the Linux Kernel. - - - -Add module symbols to GDB -------------------------- --Sometimes you need to add a Linux kernel module's symbols to GDB to debug it. -Add symbols with hand is not very easy, so KGTP package include an GDB python
-script "getmod.py" and a program "getmod" can help you. - - -How to use getmod.py --------------------- - -Connect to KGTP before use the getmod.py. -(gdb) source ~/kgtp/getmod.py -Then this script will auto load the Linux kernel module's symbols to GDB. - - -How to use getmod ------------------ -"getmod" is written by C so you can use it anywhere even if in an embedded -environment. -For example: - ----------------------------------------------------------------------------------#Following command save Linux Kernel module info to the file ~/tmp/mi in GDB
-#command format. -sudo getmod >~/tmp/mi -#in gdb part: -(gdb) source ~/tmp/mi-add symbol table from file "/lib/modules/2.6.39-rc5+/kernel/fs/nls/nls_iso8859-1.ko" at
- .text_addr = 0xf80de000 - .note.gnu.build-id_addr = 0xf80de088 - .exit.text_addr = 0xf80de074 - .init.text_addr = 0xf8118000 - .rodata.str1.1_addr = 0xf80de0ac - .rodata_addr = 0xf80de0c0 - __mcount_loc_addr = 0xf80de9c0 - .data_addr = 0xf80de9e0 - .gnu.linkonce.this_module_addr = 0xf80dea00-#After this GDB command, all the Linux Kernel module info is loaded into GDB.
--------------------------------------------------------------------------------- - -If you use remote debug or offline debug, maybe you need change the base -directory. Following example is for it. - ----------------------------------------------------------------------------------#/lib/modules/2.6.39-rc5+/kernel is replaced to sudo ./getmod -r /home/teawater/kernel/b26
-sudo ./getmod -r /home/teawater/kernel/b26 >~/tmp/mi --------------------------------------------------------------------------------- - - - -Access memory directly ----------------------- - -After connect the KGTP, you can access most of memory directly. -For example, you can access to "jiffies_64" with following command: ---------------- -p jiffies_64 ---------------- - -Or you can access to the first entry of "static LIST_HEAD(modules)" with -following command: ----------------------------------------------------------------------------------p *((struct module *)((char *)modules->next - ((size_t) &(((struct module *)0)->list))))
--------------------------------------------------------------------------------- - - - -Get register info from Kernel ------------------------------ - -The following is an example that records the value of all registers -when "vfs_readdir" is called. - --------------------------------------------------------------------------------- -(gdb) target remote /sys/kernel/debug/gtp -(gdb) trace vfs_readdir -Tracepoint 1 at 0xc01a1ac0: file -/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23. -(gdb) actions -Enter actions for tracepoint 1, one per line. -End with a line saying just "end". ->collect $reg ->end -(gdb) tstart -(gdb) shell ls -(gdb) tstop -(gdb) tfind -Found trace frame 0, tracepoint 1-#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
- buf=0xc0d09f90) at readdir.c:23 -23 readdir.c: No such file or directory. - in readdir.c -(gdb) info reg -eax 0xc5528d00 -984445696 -ecx 0xc0d09f90 -1060069488 -edx 0xc01a1900 -1072031488 -ebx 0xfffffff7 -9 -esp 0xc0d09f8c 0xc0d09f8c -ebp 0x0 0x0 -esi 0x8061480 134616192 -edi 0xc5528d00 -984445696 -eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1> -eflags 0x286 [ PF SF IF ] -cs 0x60 96 -ss 0x8061480 134616192 -ds 0x7b 123 -es 0x7b 123 -fs 0x0 0 -gs 0x0 0 -(gdb) tfind -Found trace frame 1, tracepoint 1 -0xc01a1ac1 23 in readdir.c -(gdb) info reg -eax 0xc5528d00 -984445696 -ecx 0xc0d09f90 -1060069488 -edx 0xc01a1900 -1072031488 -ebx 0xfffffff7 -9 -esp 0xc0d09f8c 0xc0d09f8c -ebp 0x0 0x0 -esi 0x8061480 134616192 -edi 0xc5528d00 -984445696 -eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1> -eflags 0x286 [ PF SF IF ] -cs 0x60 96 -ss 0x8061480 134616192 -ds 0x7b 123 -es 0x7b 123 -fs 0x0 0 -gs 0x0 0 --------------------------------------------------------------------------------- - - - -Get the value of variable from Kernel -------------------------------------- - -The following is an example that records the value of "jiffies_64" when the -function "vfs_readdir" is called: - --------------------------------------------------------------------------------- -(gdb) target remote /sys/kernel/debug/gtp -(gdb) trace vfs_readdir-Tracepoint 1 at 0xc01ed740: file /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
-(gdb) actions -Enter actions for tracepoint 1, one per line. -End with a line saying just "end". ->collect jiffies_64 ->collect file->f_path.dentry->d_iname ->end -(gdb) tstart -(gdb) shell ls-arch drivers include kernel mm Module.symvers security System.map virt -block firmware init lib modules.builtin net sound t vmlinux -crypto fs ipc Makefile modules.order scripts source usr vmlinux.o
-(gdb) tstop -(gdb) tfind -Found trace frame 0, tracepoint 1-#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580 <filldir64>, buf=0xd6dfdf90)
- at readdir.c:24 -24 { -(gdb) p jiffies_64 -$1 = 4297248706 -(gdb) p file->f_path.dentry->d_iname -$1 = "b26", '\000' <repeats 28 times> --------------------------------------------------------------------------------- - - - -How to use use tracepoint condition ------------------------------------ - -http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html -Like breakpoints, we can set conditions on tracepoints. The speed of-tracepoints is faster than breakpoints because KGTP can do all the condition
-checks. -For example: - ------------------------------- -(gdb) trace handle_irq -(gdb) condition 1 (irq == 47) ------------------------------- - -This action of tracepoint 1 will work only when irq number is 47. - - - -How to use trace state variables --------------------------------- - -http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html -Tracepoints have special variables. The variables can be traced directly, -or used in tracepoint conditions.-Note that just GDB 7.2.1 and later versions support use trace state variables -directly, the old version of GDB can show the value of trace state variables
-through command "info tvariables". - - -Simple trace state variables ----------------------------- - -Define a trace state variable $c. - -------------------- -(gdb) tvariable $c -------------------- - -Trace state variable $c is created with initial value 0. -The following action uses $c to count how many irqs happened in the kernel. - ------------------------------------------------------------------------ -(gdb) trace handle_irq -(gdb) actions -Enter actions for tracepoint 3, one per line. -End with a line saying just "end". ->collect $c #Save current value of $c to the trace frame buffer. ->teval $c=$c+1 #Increase the $c. ->end ------------------------------------------------------------------------ --Also, you can set a value of variable to trace state variable, but don't forget
-covert variable to "uint64_t". - ------------------------------------------------------------------------ ->teval $c=(uint64_t)jiffies_64 ------------------------------------------------------------------------ - -You can get the current value of $c while the trace is running or stopped. - ----------------------------------- -(gdb) tstart -(gdb) info tvariables -$c 0 31554 -(gdb) p $c -$5 = 33652 -(gdb) tstop -(gdb) p $c -$9 = 105559 ----------------------------------- - -When using tfind, you can parse the trace frame buffer. If the value of a -trace state variable is collected, you can parse it out. - ------------------------------- -(gdb) tstop -(gdb) tfind -(gdb) info tvariables -$c 0 0 -(gdb) p $c -$6 = 0 -(gdb) tfind 100 -(gdb) p $c -$7 = 100 ------------------------------- --If need, the tracepoint action that access the simple trace state variables will
-auto lock the spin lock for trace state variables. So it can handle race -condition about trace state variables.-The following example is OK even if it running a machine that have more than
-one CPU. -------------------------------------- -teval $c=$c+1 -------------------------------------- - - -Per_cpu trace state variables ------------------------------ - -Per_cpu trace state variables are special simple trace state variables.-When tracepoint action access to it, it will access to this CPU special trace
-state variables. - -It have 2 advantages:-1. The tracepoint actions that access to per_cpu trace state variables don't -have the race conditon issue. So it don't need lock the spin lock for trace -state variables. It is faster than simple trace state variables on multi-core
-machine.-2. Write the action that count some CPU special thing with it is easier than
-simple trace state variables. - -To define per_cpu trace state variables, you need named it in -format: -"per_cpu_"+string+CPU_id -or -"pc_"+string+CPU_id -Following example will define a series of per_cpu trace state variables -in a 4 COREs CPU machine with string "count": -------------------------------------- -tvariable $pc_count0 -tvariable $pc_count1 -tvariable $pc_count2 -tvariable $pc_count3 -------------------------------------- - -You can use compatibility better way to do it: ------------------------------------------ -set $tmp=0 -while $tmp<$cpu_number - eval "tvariable $pc_count%d",$tmp - set $tmp=$tmp+1 -end ------------------------------------------ - -Tracepoint action can access anyone of a series of per_cpu trace state -variables. KGTP will auto access the one of CPU that it running on. -For example: ----------------------------------------- -trace vfs_read -actions -teval $pc_count0=$pc_count0+1 -end ------------------------------------------These GDB commands define a tracepoint that count the times that call vfs_read
-of each CPU. - - -Special trace state variables $current_task, $current_task_pid, -$current_thread_info, $cpu_id, $dump_stack, $printk_level, $printk_format, -$printk_tmp, $clock, $rdtsc, $hardirq_count, $softirq_count and $irq_count ---------------------------------------------------------------------------- - -KGTP special trace state variables $current_task, $current_thread_info,-$cpu_id and $clock can very easy to access to some special value. You can see -them when GDB connects to the KGTP. You can use them in tracepoint conditions
-or actions.-Access $current_task in tracepoint condition and action will get that returns
-of get_current(). -Access $current_task_pid in tracepoint condition and action will get that -returns of get_current()->pid.-Access $current_thread_info in tracepoint condition and action will get that
-returns of current_thread_info(). -Access $cpu_id in tracepoint condition and action will get that returns of -smp_processor_id(). -Access $clock in tracepoint condition and action will get that returns of -local_clock() that return the timestamp in nanoseconds.-$rdtsc is only available on X86 and X86_64 architecture. Access it in anytime
-will get current value of TSC with instruction RDTSC.-Access $hardirq_count in tracepoint condition and action will get that returns
-of hardirq_count().-Access $softirq_count in tracepoint condition and action will get that returns
-of softirq_count(). -Access $irq_count in tracepoint condition and action will get that returns -of irq_count(). --And KGTP has other special trace state variables $dump_stack, $printk_level,
-$printk_format and $printk_tmp. All of them output their values directly, -as can be seen in "Howto let tracepoint output value directly". --The following example counts in $c how many vfs_read calls that process 16663
-does and collects the struct thread_info of current task: - --------------------------------------------------------------------------------- -(gdb) target remote /sys/kernel/debug/gtp-(gdb) trace vfs_read if (((struct task_struct *)$current_task)->pid == 16663)
-(gdb) tvariable $c -(gdb) actions -Enter actions for tracepoint 4, one per line. -End with a line saying just "end". ->teval $c=$c+1 ->collect (*(struct thread_info *)$current_thread_info) ->end -(gdb) tstart -(gdb) info tvariables -Name Initial Current -$c 0 184 -$current_task 0 <unknown> -$current_thread_info 0 <unknown> -$cpu_id 0 <unknown> -(gdb) tstop -(gdb) tfind -(gdb) p *(struct thread_info *)$current_thread_info-$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status = 0, cpu = 1, preempt_count = 2, addr_limit = { - seg = 4294967295}, restart_block = {fn = 0xc0159fb0 <do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11, - arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11, flags = 1, bitset = 78, time = 977063750, - uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb, expires = 335007449089}, poll = { - ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78, tv_nsec = 977063750}}}, - sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack = 0xef340044 "", uaccess_err = 0}
--------------------------------------------------------------------------------- - -Another example shows how much sys_read() executes in each CPU. - --------------------------------------- -tvariable $c0 -tvariable $c1 -trace sys_read - condition $bpnum ($cpu_id == 0) - commands - teval $c0=$c0+1 - end -trace sys_read - condition $bpnum ($cpu_id == 1) - commands - teval $c1=$c1+1 - end -info tvariables -Name Initial Current -$current_task 0 <unknown> -$cpu_id 0 <unknown> -$c0 0 3255 -$c1 0 1904 --------------------------------------- - -sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. - - -Special trace state variable $no_self_trace --------------------------------------------- - -$no_self_trace is different with the special trace state variables in the -previous section. It is used to control the behavior of tracepoint.-If the action of a tracepoint include a command access to the $no_self_trace. -The tracepoint will not trace anything if the current_task is the a KGTP self
-process (GDB, netcat, getframe or some others process that access to the -interface of KGTP).-For example, if we want trace vfs_read or something that have process context,
-and we don't want trace the operation of KGTP self process. Add following -command to the action: - --------------------------------------- -collect $no_self_trace --------------------------------------- - -Please note that the code that doesn't have process context (Irq handler, -softirq) doesn't need set this variable. - - -Trace the function return with $kret ------------------------------------- --Sometime, set the tracepoint to the end of function is hard because the Kernel
-is compiled with optimization. At this time, you can get help from $kret. - -$kret is a special trace state variable like $no_self_trace. When you set -value of it inside the action of tracepoint, this tracepoint be set with -kretprobe instead of kprobe. Then it can trace the end of this function. - -Following part is an example: - ------------------------------------------------------------------------------- -target remote /sys/kernel/debug/gtp -#"*(function_name)" format can make certain that GDB send the first address -#of function to KGTP. -trace *(vfs_read) -action -teval $kret=0 -#Following part you can set commands that you want. ------------------------------------------------------------------------------- - - -Use $ignore_error and $last_errno to ignore the error of tstart ---------------------------------------------------------------- - -If KGTP got any error of tstart, this command will get fail. -But sometime we need ignore this error and let KGTP keep work. -For example: If you set tracepoint on the inline function spin_lock. -This tracepoint will be set to a lot of addresses that some of them cannot -be set kprobe. It will make tstart get fail. You can use "$ignore_error" -ignore this error. -And the last error number will available in "$last_errno". - ---------------------------------- -tvariable $ignore_error=1 ---------------------------------- -This command will open ignore. - ---------------------------------- -tvariable $ignore_error=0 ---------------------------------- -This command will close ignore. - - -Use $cooked_clock and $cooked_rdtsc the time without KGTP used --------------------------------------------------------------- - -Access these two trace state variables can get the time without KGTP used.-Then we can get more close to really time that a part of code used even if the
-actions of tracepoint is very complex. They will be introduce in -Cookbook (coming soon). - - -Use $xtime_sec and $xtime_nsec get the timespec ------------------------------------------------ - -Access these two trace state variables will return the time of day in -a timespec that use getnstimeofday. -$xtime_sec will access to the second part of a timespec. -$xtime_nsec will access to the nanosecond part of a timespec. - - - -How to use performance counters -------------------------------- --Performance counters are special hardware registers available on most modern -CPUs. These registers count the number of certain types of hw events: such as -instructions executed, cachemisses suffered, or branches mis-predicted - without
-slowing down the kernel or applications. These registers can also trigger -interrupts when a threshold number of events have passed - and can thus be -used to profile the code that runs on that CPU. --The Linux Performance Counter subsystem called perf event can get the value of
-performance counter. You can access it through KGTP perf event trace state -variables. - -Please goto read the file tools/perf/design.txt in Linux Kernel to get more -info about perf event. - - -Define a perf event trace state variable ----------------------------------------- - -Access an performance counter need define following trace state variable: ------------------------------------------------------------------------------ -"pe_cpu_"+tv_name Define the the CPU id of the performance counter. -"pe_type_"+tv_name Define the the type of the performance counter. -"pe_config_"+tv_name Define the the config of the performance counter. -"pe_en_"+tv_name This the switch to enable or disable the performance - counter. - The performance counter is disable in default. -"pe_val_"+tv_name Access this variable can get the value of the - performance counter. ------------------------------------------------------------------------------ - - -Define a per_cpu perf event trace state variable ------------------------------------------------- - -Define a per_cpu perf event trace state variable is same with define -"Per_cpu_trace_state_variables". - ------------------------------------------- -"pc_pe_"+perf_event type+string+CPU_id ------------------------------------------- --Note that if you define a per_cpu perf event trace state variable, you will not
-need define the "pe_cpu_" because KGTP already get it from per_cpu id. - - -The perf event type and config ------------------------------- - -The type of perf event can be: ------------------------------------------------------------ -0 PERF_TYPE_HARDWARE -1 PERF_TYPE_SOFTWARE -2 PERF_TYPE_TRACEPOINT -3 PERF_TYPE_HW_CACHE -4 PERF_TYPE_RAW -5 PERF_TYPE_BREAKPOINT ------------------------------------------------------------ - -If the type is 0(PERF_TYPE_HARDWARE), the config can be: ------------------------------------------------------------ -0 PERF_COUNT_HW_CPU_CYCLES -1 PERF_COUNT_HW_INSTRUCTIONS -2 PERF_COUNT_HW_CACHE_REFERENCES -3 PERF_COUNT_HW_CACHE_MISSES -4 PERF_COUNT_HW_BRANCH_INSTRUCTIONS -5 PERF_COUNT_HW_BRANCH_MISSES -6 PERF_COUNT_HW_BUS_CYCLES -7 PERF_COUNT_HW_STALLED_CYCLES_FRONTEND -8 PERF_COUNT_HW_STALLED_CYCLES_BACKEND ------------------------------------------------------------ - -If the type is 3(PERF_TYPE_HW_CACHE), the config need to divide to 3 parts: -First one is cache id, it need be << 0 before set to config: ------------------------------------------------------------ -0 PERF_COUNT_HW_CACHE_L1D -1 PERF_COUNT_HW_CACHE_L1I -2 PERF_COUNT_HW_CACHE_LL -3 PERF_COUNT_HW_CACHE_DTLB -4 PERF_COUNT_HW_CACHE_ITLB -5 PERF_COUNT_HW_CACHE_BPU ------------------------------------------------------------ -Second one is cache op id, it need be << 8 before set to config: ------------------------------------------------------------ -0 PERF_COUNT_HW_CACHE_OP_READ -1 PERF_COUNT_HW_CACHE_OP_WRITE -2 PERF_COUNT_HW_CACHE_OP_PREFETCH ------------------------------------------------------------ -Last one is cache op result id, it need be << 16 before set to config: ------------------------------------------------------------ -0 PERF_COUNT_HW_CACHE_RESULT_ACCESS -1 PERF_COUNT_HW_CACHE_RESULT_MISS ------------------------------------------------------------ -If you want get the perf count of PERF_COUNT_HW_CACHE_L1I(1), -PERF_COUNT_HW_CACHE_OP_WRITE(1) and PERF_COUNT_HW_CACHE_RESULT_MISS(1), -you can use: ------------------------------------------------------------ -tvariable $pe_config_cache=1 | (1 << 8) | (1 << 16) ------------------------------------------------------------ - -tools/perf/design.txt in Linux Kernel have more info about type and config -of perf event. - - -Enable and disable all the perf event in a CPU with $pc_pe_en -------------------------------------------------------------- - -I think the best way that count a part of code with performance counters is-enable all the count in the begin of the code and disable all of them in the -end. You can do it with "pe_en_". But if you have a lot of perf event trace -state variables. That will make the tracepoint action very big. $pc_pe_en is
-for this issue. - -You can enable all the perf event trace state variables in current CPU with -following action: --------------------- -teval $pc_pe_en=1 --------------------- -Disable them with set $pc_pe_en to 0. --------------------- -teval $pc_pe_en=0 --------------------- - - -GDB scripts to help with set and get the perf event trace state variables -------------------------------------------------------------------------- --Following is a GDB script define two commands dpe and spe to help define and
-show the perf event trace state variables.-You can put it to the ~/.gdbinit or your tracepoint script. Then you can use
-this two commands in GDB directly. -------------------------------------------------------------------------------- -define dpe - if ($argc < 2) - printf "Usage: dpe pe_type pe_config [enable]\n" - end - if ($argc >= 2) - set $tmp=0 - while $tmp<$cpu_number - eval "tvariable $pc_pe_type_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg0 - eval "tvariable $pc_pe_config_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg1 - eval "tvariable $pc_pe_val_%d%d_%d=0",$arg0, $arg1, $tmp - if ($argc >= 3) - eval "tvariable $pc_pe_en_%d%d_%d=%d",$arg0, $arg1, $tmp, $arg2 - end - set $tmp=$tmp+1 - end - end -end - -document dpe -Usage: dpe pe_type pe_config [enable] -end - -define spe - if ($argc != 2 && $argc != 3) - printf "Usage: spe pe_type pe_config [cpu_id]\n" - end - if ($argc == 2) - set $tmp=0 - while $tmp<$cpu_number- eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $tmp
- set $tmp=$tmp+1 - end - end - if ($argc == 3)- eval "printf \"$pc_pe_val_%%d%%d_%%d=%%ld\\n\",$arg0, $arg1, $tmp, $pc_pe_val_%d%d_%d", $arg0, $arg1, $arg2
- end -end - -document spe -Usage: spe pe_type pe_config [cpu_id] -end -------------------------------------------------------------------------------- --Following is an example to use it get the performance counters of function tcp_v4_rcv:
-------------------------------------------------------------------------------- -#Connect to kgtp -target remote /sys/kernel/debug/gtp-#Define 3 pe tvs for PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_CACHE_MISSES and PERF_COUNT_HW_BRANCH_MISSES.
-dpe 0 0 -dpe 0 3 -dpe 0 5 -#enable the performance counters of this CPU in the begin of this function. -trace tcp_v4_rcv ***The diff for this file has been truncated for email.*** ======================================= --- /trunk/gtp_quickstart.txt Thu Dec 29 04:31:48 2011 +++ /dev/null @@ -1,250 +0,0 @@ - Linux Kernel GDB tracepoint module (KGTP) quick start - ===================================================== - By Hui Zhu <teawater@xxxxxxxxx> - https://code.google.com/p/kgtp/wiki/Quickstart - 2011-09-12 - -Table of contents ------------------ -Ubuntu -Fedora - - - - -Ubuntu ------- - -Install GDB for KGTP --------------------- - -This GDB's filename is different with the current GDB that you are using. -So please don't worry that it affect current GDB that your are using. - -For the Ubuntu 10.04 or later, running the following line at a terminal: -sudo add-apt-repository ppa:teawater/gdb-$(lsb_release -rs) -sudo apt-get update -sudo apt-get install gdb-release --For the Ubuntu older than 10.04, please go to https://code.google.com/p/gdbt/
-get howto install GDB for KGTP from source. - - - -Install Linux kernel packages that KGTP need --------------------------------------------- - -Please ignore this section if the Linux kernel of your system is built by -yourself. - -Install the Linux kernel debug image ------------------------------------- - -Add debug source to the sources list of Ubuntu ----------------------------------------------- --Create an /etc/apt/sources.list.d/ddebs.list by running the following line at
-a terminal:-echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
-sudo tee -a /etc/apt/sources.list.d/ddebs.list --Stable releases (not alphas and betas) require three more lines adding to the
-same file, which is done by the following terminal command:-echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse -deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse -deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
-sudo tee -a /etc/apt/sources.list.d/ddebs.list - -Import the debug symbol archive signing key: -sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 - -Then run: -sudo apt-get update - -Get Linux kernel debug image ----------------------------- -sudo apt-get install linux-image-$(uname -r)-dbgsym - - -Install the Linux kernel headers --------------------------------- - -Please ignore this section if the Linux kernel of your system is built by -yourself. -sudo apt-get install linux-headers-generic - - -Install the Linux kernel source -------------------------------- - -Install the source package: -sudo apt-get install linux-source - -Uncompress the source package: -sudo mkdir -p /build/buildd/-sudo tar vxjf /usr/src/linux-source-$(uname -r | sed 's/-.*//').tar.bz2 -C /build/buildd/ -sudo mv /build/buildd/linux-source-$(uname -r | sed 's/-.*//') /build/buildd/linux-$(uname -r | sed 's/-.*//')
- - - -Install GCC ------------ - -sudo apt-get install gcc - - - -Get and build KGTP ------------------- - -Install subversion: -sudo apt-get install subversion - -Get the source of KGTP with subversion and put it to directory "kgtp": -svn checkout https://kgtp.googlecode.com/svn/trunk kgtp - -Build KGTP: -cd kgtp -make - - - -Use KGTP --------- - -Mount the sysfs and debugfs: -sudo mount -t sysfs none /sys/ -sudo mount -t debugfs none /sys/kernel/debug/ - -Insert the KGTP module to the current Linux Kernel: -cd kgtp -sudo insmod gtp.ko - -Use GDB connect to KGTP: -sudo gdb-release /usr/lib/debug/boot/vmlinux-$(uname -r) -(gdb) target remote /sys/kernel/debug/gtp - -Do a very simple trace: -(gdb) trace vfs_readdir-Tracepoint 1 at 0xc02289f0: file /build/buildd/linux-2.6.35/fs/readdir.c, line 23.
-(gdb) actions -Enter actions for tracepoint 1, one per line. -End with a line saying just "end". ->collect $reg ->end -(gdb) tstart -(gdb) shell ls -vmlinux-2.6.35-30-generic -(gdb) tstop -(gdb) tfind -Found trace frame 0, tracepoint 1 -#0 vfs_readdir (file=0x0, filler=0x163d8ae3, buf=0x18c0) at readdir.c:23 -23 { - - - -End ---- - -Now, you can begin to rock and roll your Linux kernel with KGTP and GDB. -Please go to see gtp.txt to get more message about howto use KGTP. - - - - -Fedora ------- - -Install GDB for KGTP --------------------- - -Please go to https://code.google.com/p/gdbt/ get howto install GDB for KGTP -from source. - - - -Install Linux kernel packages that KGTP need --------------------------------------------- - -Please ignore this section if the Linux kernel of your system is built -by yourself. - - -Install the Linux kernel debug image ------------------------------------- - -sudo yum --enablerepo=fedora-debuginfo install kernel-debuginfo - - -Install the Linux kernel devel package --------------------------------------- - -sudo yum install kernel-devel-$(uname -r) - - - -Install GCC ------------ - -sudo yum install gcc - - - -Get and build KGTP ------------------- - -Install subversion: -sudo yum install subversion - -Get the source of KGTP with subversion and put it to directory "kgtp": -svn checkout https://kgtp.googlecode.com/svn/trunk kgtp - -Build KGTP: -cd kgtp -make - - - -Use KGTP --------- - -Mount the sysfs and debug fs: -sudo mount -t sysfs none /sys/ -sudo mount -t debugfs none /sys/kernel/debug/ - -Insert the KGTP module to the current Linux Kernel: -cd kgtp -sudo insmod gtp.ko - -Use GDB connect to KGTP: -sudo gdb-release /usr/lib/debug/lib/modules/$(uname -r)/vmlinux -(gdb) target remote /sys/kernel/debug/gtp - -Do a very simple trace: -(gdb) trace vfs_readdir -Tracepoint 1 at 0xffffffff8110ec9b: file fs/readdir.c, line 23. -(gdb) actions -Enter actions for tracepoint 1, one per line. -End with a line saying just "end". ->collect $reg ->end -(gdb) tstart -(gdb) shell ls-co.patch getframe getmod.c gtp.mod.c gtp.txt perf_event.c -dkms.conf getframe.c getmod.py gtp.mod.o Makefile ring_buffer.c -dkms_others_install.sh getgtprsp.pl gtp.c gtp.o modules.order ring_buffer.h -dkms_others_uninstall.sh getmod gtp.ko gtp.patch Module.symvers
-(gdb) tstop -(gdb) tfind -Found trace frame 0, tracepoint 1-#0 vfs_readdir (file=0xffff880019d3df00, filler=0xffffffff8110eb16 <filldir>, buf=0xffff880003b39f38)
- at fs/readdir.c:23 -23 { - - - -End ---- - -Now, you can begin to rock and roll your Linux kernel with KGTP and GDB. -Please go to HOWTO to get more message about howto use KGTP.