Case study:

Debugging a slow software build

The problem:

A customer, Ahmed Ismail, software engineer at Mentor Graphics, found that a software build was taking too long and wanted to use Breeze to find out why.

Analysing shell commands

The solution:

Ahmed used Breeze to trace an incremental software build with a <30 seconds run. Once the trace file was generated, he started the UI and imported the trace. He quickly saw that hundreds of shell calls were being made within the incremental build.

He started to expand the shell commands in Breeze one by one to get more details. As Breeze shows the commandline arguments he could see they were all doing the same thing:

cd some_location; pwd

Ahmed then went back to the makefiles and found some shell calls like this:

MODELTECH_DIR = $(shell cd $(MTI_ROOT); pwd)/modeltech

After a bit of research he discovered that it is very bad practice to use recursive assignment with shell commands in makefiles (see article on makefile performance from Electric Cloud).

Overall Breeze gives good detailed information about a short trace that can be used to improve runtime when you look at specific parts of the run. It is also good for gaining more information about our programs and scripts. Breeze can easily be used to walkthrough a script by understanding the IO operations over time.

Ahmed Ismail, software engineer at Mentor Graphics

The result:

Ahmed made the below change in a few places and after rerunning observed about 87% improvement in incremental build (16 sec to 2 sec).

MODELTECH_DIR := $(shell cd $(MTI_ROOT); pwd)/modeltech