Of course, Ellexus are developers of the best tracing tool in the world so we would recommend using Breeze, but if you ever find yourself debugging a system without access to Breeze you may have to resort to using ltrace or strace to debug your flow and discover its dependencies. Since we like you so very very much here is a tutorial on how to get started with ltrace.
Lets start by tracing ltrace on xterm, the default shell terminal in Linux. Ltrace and Breeze both trace library calls so we can use Breeze to see what ltrace is doing. You will need to use the -f option in order follow forks. This means that when xterm starts a bash shell as a child process you will be able to trace that too.
Breeze commandline: > trace ltrace -f xterm
The graph generated by Breeze below shows us how ltrace works. It is the root node shown in green. It runs xterm (top orange node) which in turn calls bash (bottom right in brown). All the files accessed are shown in pink. This information can be extracted from ltrace by looking for the ‘exec’ call. Programs are called in two stages – a new process is created with fork then the new process it turned into the target program with a call to exec(). Bash is a mult-process program because it forks many times without calling exec(). This is why it is shown in brown in the Breeze graph.
You can see the call to bash in the ltrace output below. It is probably worth noting that exec() is actually a family of library functions that allow you to call programs in a number of different ways.
execlp("/bin/bash", "bash", NULL, NULL, 0xfefefefefefefef
Ltrace uses the special files /proc//exe to lookup the location of each of the running processes. The temporal view in Breeze shows ltrace reading from /proc/2446/exe and /proc/2447/exe just before executing the processes with those process ids.
Once ltrace has found the program it attaches its hooks to it via a kernel program called ptrace and listens to all the library and system calls made to the Linux kernel via the GNU C library …and ladies and gentlemen, I really do mean “all” library calls. You will have to wade though a lot of output like the following. Each of the functions will have a man page. Calls like open() and close() you can probably guess, but others like fcntl() might need some research. At this point grep is your friend so learn to use it wisely.
A similar tool to ltrace is strace, which works in the same way, but shows system calls instead of library calls. At this point I should probably explain the difference: system calls are requests made to the Linux kernel. Programs use system calls to access system resources such as opening a file or connecting to the network. Library calls are calls made to the GNU C library which provides a more programmer-friendly interface to the system.
The advantage of strace over ltrace is that the information is often simpler – not all library calls map to system calls. The disadvantage is that not all library calls have the same name as their system call equivalent. For example, the fork() library call uses the clone() system call which can also be used for threading as well as making a new process.
Ltrace and strace only work on executables, but the good news is that you can still trace scripts by tracing the script interpreter:
> ltrace python myscript.py
The take home message is that you shouldn’t be afraid of ltrace or strace as a last resort, but that is exactly where they should stay. Ellexus Breeze can give you a lot of the same information in a more user-friendly format so why not give it a go next time instead of reading the library calls one at a time.
Ellexus are the developers of Breeze, a Linux dependency tracing tool that shows you what your programs are doing as they run. You can quickly search trace data to trouble shoot a problem build or installation.