CareerCup

I think it is not related to Kernel..

they may be asking at user space side only

do a kill -6 to get a core dump and debug the core

It may be a kernel question. There might be a way to get into the kernel and pause the process, attach a debugger to the paused process. It is very likely that you would have pause in the offending piece of code. In any case, you can then set breakpoints etc.

kill -6 is one solution as one user mentioned

Set conditional breakpoints by having counters at multiple places in the suspected process. Based on one counters hitting a certain count you can break and look into the possible buggy code area

You can have throttled printf mechanism based on the above counter logic

A normal process executing in user mode cannot consume 90% of CPU as the process will get switched out after it consumes its time share slot.

If the process is a Real Time process and has a bug in its code, inspite of getting preempted after its timeshare, it could get the cpu right back if it is the highest priority process. So, it may show 90% accumulated cpu usage over time, but cannot consume 90% in one continuous grab of cpu time.

The only way for the cpu to be 90% consumed by one process is some sort of kernel problem. Usually, there is some sort of an infinite loop OR a very long running process that is not preempting and giving up the CPU.

One way to debug such problems is to use kernel tracers. For example, refer to ktracer on HPUX. This tool traces kernel procedure calls, arguments and timing. So, if there is a loop and a particular stack trace is being repeatedly executed leading to a loop, we can see it in ktracer output.

A similar tool is Solaris Dtrace.

execute command 'top'
find the offending process "id"
then read this:
http://dirac.org/linux/gdb/06-Debugging_A_Running_Process.php

Use different machine and connect it to sluggish machine using firewire to debug kernel.

Use pstack to see the stack of all the threads.

I liked the core dump and pstack methods. gcov and gprof would also be of great help