criticalstat - A tracer to find and report long atomic critical sections in kernel

       Joel Fernandes

       criticalstat  traces  and  reports  occurrences  of  atomic  critical  sections in the kernel with useful
       stacktraces showing the origin of them. Such critical sections frequently occur due to use of  spinlocks,
       or  if  interrupts  or  preemption  were  explicitly disabled by a driver. IRQ routines in Linux are also
       executed with interrupts disabled. There are many reasons. Such critical sections are a  source  of  long
       latency/responsive issues for real-time systems.

       This  works  by probing the preempt/irq and cpuidle tracepoints in the kernel.  Since this uses BPF, only
       the root user can use this tool. Further, the kernel has to be built with certain CONFIG options enabled.
       See below.

       Run with default options: irq disabled for more than 100 uS
              # criticalstat

       Find sections with preemption disabled for more than 100 uS.
              # criticalstat-p

       Find sections with IRQ disabled for more than 500 uS.
              # criticalstat-d500

       Find sections with preemption disabled for more than 500 uS.
              # criticalstat-p-d500

       criticalstat - A tracer to find and report long atomic critical sections in kernel

       -h     Print usage message.

       -p     Find long sections where preemption was disabled on local CPU.

       -i     Find long sections where interrupt was disabled on local CPU.

       -d DURATION
              Only identify sections that are longer than DURATION in microseconds.

       Linux

       This tool can cause overhead if the application is spending a lot of time in kernel mode. The overhead is
       variable  but  can  be 2-4% of performance degradation. If overhead is seen to be too much, please pass a
       higher DURATION to the -d option to filter more aggressively.

       Enable CONFIG_PREEMPT_TRACER, CONFIG_PREEMPTIRQ_EVENTS (CONFIG_PREEMPTIRQ_TRACEPOINTS in kernel 4.19  and
       later) and CONFIG_DEBUG_PREEMPT. Additionally, the following options should be DISABLED on older kernels:
       CONFIG_PROVE_LOCKING, CONFIG_LOCKDEP.

       Linux kernel's preemptoff and irqoff tracers.

USER COMMANDS                                      2018-06-07                                    criticalstat(8)

       This is from bcc.

              https://github.com/iovisor/bcc

       Also look in the bcc distribution for a companion _examples.txt file containing  example  usage,  output,
       and commentary for this tool.

       Unstable - in development.

criticalstat[-h][-p][-i][-dDURATION]