CPU and scheduler visualization
Sometimes it might be necessary to visualize how threads are interacting in the program. Happy Solaris developers can use their Thread Scheduling Visualizer to look into guts of a Java program. Unfortunately, TSV is based on DTrace and thus is not available for Linux. But it is not a problem though. We can easily get the same functionality by combining SystemTap and MATLAB. Moreover, this approach is not restricted to any particular language.
The first part of the solution is a sched_switch SystemTap script. It collects data on context switches in the thread scheduler. Every time the context is being switched, the SystemTap probe is executed. It prints thread ID, current timestamp in nanoseconds, the number of microseconds since the last context switch and the type of event (thread scheduled for running or de-scheduled).
This is pretty cool, but going through the log manually is not fun. So to visualize interactions I have written a ReadSched MATLAB function. The ReadSched accepts three parameters: path to the file with the script output, start time and end time of the
period you want to visualize in milliseconds. Here is a call to the function to analyze a Log_10threads.txt log file and visualize all the activities occured between 2nd and 6th seconds:
ReadSched('/home/alexta/ParallelProgramAnalysis/SystemTap/scripts/Log_10threads.txt',2000,6000)
Here how visualization looks:
The X axis denotes time, while Y axis of the graph corresponds to different threads. For each thread a "high" level corresponds to the running state, low level - to the waiting state. Threads running on different CPUs are shown in different colors.
You can download the script and MATLAB function here. Hopefully, you will found this script useful for uncovering subtle events in your own applications. For example, it can be seen that besides threads that performs actual work, Java also runs multiple additional threads. Some of them (on the bottom) seem to be related to garbage collection.