Tracing technology: ltrace and strace

We are often asked if our I/O profiling tools use ltrace or strace under the hood. In fact, they use a different way to intercept file and network I/O called LD_PRELOAD.

LD_PRELOAD can be faster and more flexible, but is much harder to get right. However, if you just want to find out what your application is doing and you don’t have access to Breeze then you can implement your own solution with ltrace or strace.

What’s the difference between ltrace and strace?

Both ltrace and strace give you similar information, but with an important difference:

  • strace intercepts system calls make by the glibc and other libraries directly into the Linux Kernel. It uses a tool called ptrace to inspect the system calls of a process.
  • ltrace intercepts library calls and system calls made by your application to C libraries such as the glibc. It doesn’t use LD_PRELOAD, but the information that you get is very similar.

Library calls and system calls are often confused for the following reasons:

  • System calls don’t have a function name, but have to be given one to make it easy for developers to remember what they do. This name is used in the output of strace and looks like a function call or library call, but it isn’t.
  • Many system calls have the same name and functionality as library calls. For example, in glibc there is an open() library symbol that simply calls the open system call. For these calls, the information from ltrace and strace will match. This isn’t always the case though and there are many other library calls such as fopen() and open64() that use the open system call in a slightly different way, so just measuring the open library call won’t catch all the system calls.
  • In glibc there are two functions with confusing names:
    • The library symbol system() is actually a way of a process forking and executing another program. The system() library call has got nothing to do with Linux system calls.
    • The library symbol syscall() is a way of making Linux system calls directly. It is a function that simply makes the system call as specified.

To see how ltrace and strace work, we traced them with Breeze in an earlier blog post.

Tips for using strace and ltrace

Both tools will show you a lot of detail that you don’t need, but you’ll probably want to turn on a lot of the options to make sure that you get the process and timing information you need. Unless you know you don’t need the information, it’s a good idea to turn most options on.

For simple queries such as “Does program X access file Y?” then you can just trace your program and grep for the file. Make sure that you are looking at the trace for the right program. Both strace and ltrace will record PIDs, but only if you select the -f option to follow forks. Without this, the tools will only trace the top-level process.

The Breeze product suite shows you I/O operations for each program, but if you are using strace or ltrace you have to work out which program is which for yourself. To do this, grep for exec() calls to find the start of a program then use grep again to separate the actions for the correct PID. Finally, you can grep for the specific open() call or network operation that you are looking for.

If you have any further questions about strace, ltrace or our own tools, get in touch – our expert team will help you find the right tools for you.