One of the tools we use the most here at cPanel is strace. This tool is great for finding the root cause of issues when logs aren't giving enough information as this utility will give us insight to the actual system calls. Hence, we can see why something is failing.
The following is a quick overview of the strace utility.
Strace is a very useful diagnostic, instructional and debugging tool in Linux which traces systems calls and signals used by a program.
It is most commonly used when a program fails to work as expected and the error logs offer few clues as to the issue. It is also great for instructional purposes.
How to use it?
To use strace you can simply invoke strace followed by the program you’d like to strace. EX:
This will output all the system calls and signals used by preforming an ls on a directory.
However, this will give you a lot more information than needed. This is why it’s very useful to understand strace flags as this will help you get to the bottom of your issue a lot quicker.
Stracing cpsrvd
Here's an example that runs strace, creates an output file at /root/strace.cpsrvd, and watches the system calls to the cpsrvd process:
-o : Suggested - will write the output to a file, in this example /root/strace.cpsrvd
-ff : Suggested - follow forks and generate one file per PID, i.e., /root/strace.cpsrvd.12345
-f : Optional: using only one -f option instead of two would log everything to one file instead of multiple if that is desired
-v : Suggested - print unabbreviated output for environment, stat, etc. calls
-tt : Suggested - adds microsecond timestamps to each call - useful for determining the source of a performance problem or can also be used as an identifier to reference a particular entry in the strace output
-s 128 : Suggested - increases the string size that is printed in syscall output. Default is 32 which can often cut off useful information but also keeps the output small. Increasing this value further is recommend if you find that potentially useful information is being cut off in the syscall output.
-e : Optional - tells strace to only output the matching system calls, i.e. open().
The following is a quick overview of the strace utility.
Strace is a very useful diagnostic, instructional and debugging tool in Linux which traces systems calls and signals used by a program.
It is most commonly used when a program fails to work as expected and the error logs offer few clues as to the issue. It is also great for instructional purposes.
How to use it?
To use strace you can simply invoke strace followed by the program you’d like to strace. EX:
Code:
strace lsHowever, this will give you a lot more information than needed. This is why it’s very useful to understand strace flags as this will help you get to the bottom of your issue a lot quicker.
Stracing cpsrvd
Here's an example that runs strace, creates an output file at /root/strace.cpsrvd, and watches the system calls to the cpsrvd process:
Code:
strace -o /root/strace.cpsrvd -ffvtt -s 128 -e open -p `cat /var/run/cpsrvd.pid`-ff : Suggested - follow forks and generate one file per PID, i.e., /root/strace.cpsrvd.12345
-f : Optional: using only one -f option instead of two would log everything to one file instead of multiple if that is desired
-v : Suggested - print unabbreviated output for environment, stat, etc. calls
-tt : Suggested - adds microsecond timestamps to each call - useful for determining the source of a performance problem or can also be used as an identifier to reference a particular entry in the strace output
-s 128 : Suggested - increases the string size that is printed in syscall output. Default is 32 which can often cut off useful information but also keeps the output small. Increasing this value further is recommend if you find that potentially useful information is being cut off in the syscall output.
-e : Optional - tells strace to only output the matching system calls, i.e. open().