funcslower - trace kernel functions slower than a threshold (microseconds). Uses Linux ftrace.

       Brendan Gregg

       This  uses  the  Linux ftrace function graph profiler to time kernel functions and filter them based on a
       latency threshold. Latency outliers can be studied this way, confirming  their  presence,  duration,  and
       rate. This tool is a proof of concept using Linux ftrace capabilities on older kernels.

       The  output  format  is  based  on the ftrace function graph trace format, described in the kernel source
       under Documentation/trace/ftrace.txt. Use the -H option to print column headings.  Note that  the  output
       may  be  shuffled  when  different  CPU  buffers  are  read; check the CPU column for changes, or include
       timestamps (-t) and post sort.

       WARNING: This uses dynamic tracing of kernel functions, and could cause kernel panics or  freezes.  Test,
       and know what you are doing, before use.

       Since this uses ftrace, only the root user can use this tool.

       Trace calls to vfs_read(), showing events slower than 10 ms:
              # funcslowervfs_read10000

       Same as above, but include column headers, event timestamps, and process names:
              # funcslower-HPtvfs_read10000

       Trace slow vfs_read()s for PID 198 only:
              # funcslower-p198vfs_read10000

       The  output format depends on the kernel version, and headings can be printed using -H. The format is the
       same   as   the   ftrace   function   trace   format,   described   in   the    kernel    source    under
       Documentation/trace/ftrace.txt.

       Typical fields are:

       TIME   (Shown with -t.) Time of event, in seconds.

       CPU    The CPU this event occurred on.

       TASK/PID
              (Shown with -P.) The process name (which could include dashes), a dash, and the process ID.

       DURATION
              Elapsed  time  during  the  function  call, inclusive of children. This is also inclusive of sleep
              time, unless -C is used.

       FUNCTION CALLS
              Kernel function returns.

       funcslower - trace kernel functions slower than a threshold (microseconds). Uses Linux ftrace.

       -a     All info. Same as -HPt.

       -C     Function durations measure on-CPU time only (exclude sleep time).

       -d seconds
              Set  the  duration  of  tracing, in seconds. Trace output will be buffered and printed at the end.
              This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

              The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If  you
              think events are missing, try increasing that size.

       -h     Print usage message.

       -H     Print column headers.

       -p PID Only trace kernel functions when this process ID is on-CPU.

       -L TID Only trace kernel functions when this thread ID is on-CPU.

       -P     Show process names and process IDs with every line of output.

       -t     Show timestamps on every line of output.

       funcstring
              A  function  name  to trace, which may include file glob style wildcards ("*") at the beginning or
              ending of a string only. Eg, "vfs*" means match "vfs" followed by anything. Since  the  output  is
              verbose, you probably only want to trace single functions, and not use wildcards.

       latency_us
              Minimum function duration to trace, in units of microseconds. This is filtered in-kernel.

       Linux

       OVERHEADS: Timing and filtering is performed  in-kernel  context,  costing  lower  overheads  than  post-
       processing  in user space. If you trace frequent events (eg, pick a common function and a low threshold),
       you might want to try the "-d secs" option, which buffers events in-kernel instead of printing them live.

       It's a good idea to start with a high threshold (eg, "100000" for 100 ms) then to  decrease  it.  If  you
       start low instead, you may start printing too many events.

       FTRACE function graph, which you may already have enabled and available on recent kernels. And awk.

funccount(8), functrace(8), funcgraph(8), kprobe(8)

USER COMMANDS                                      2014-07-30                                      funcslower(8)

       This is from the perf-tools collection:

              https://github.com/brendangregg/perf-tools

       Also  look  under the examples directory for a text file containing example usage, output, and commentary
       for this tool.

       Unstable - in development.

funcslower [-aChHPt] [-p PID] [-L TID] [-d secs] funcstring latency_us