apache nobody process and cpu high load

[benito]

Hello, since a week one of my servers have a couple of httpd processes owned by nobody that are taking a lot of cpu use. If I restart apache everything goes back to normal until start escalating again.

Anyone knows how to identify who is using that process?

Thanks

 

[cPRex]

Hey there! All Apache processes will be owned by the user "nobody" so seeing that showing up is normal.

There isn't an easy way to identify which website is causing that behavior outside of checking the logs at that time and seeing if that matches up with anything. If this is a shared hosting environment I would *strongly* recommend looking into CloudLinux as that would allow you to set CPU and RAM limits for each account on the machine, and will track accounts that reach their limits.

 

[benito]

Thanks @cPRex, you're right, the processes with an owner where the lsphp, not httpd.

I have already have cloudlinux but that now was helpfull to identificate the user using that apache process.

Finally I was able to identify the customer that owns these apache processes that runs for days using "lsof -p #PID"

 

[cPRex]

I'm glad to hear you found a good resolution!

 

[benito]

Hello!

Although I now know whose process it is, I don't know why this happens. Is there a way to make apache processes not last forever? In this case I have three apache processes from the same client active for several days.

 

[cPRex]

Has the "Timeout" value on your server been changed from the default of 300 seconds? You can check this value here: https://support.cpanel.net/hc/en-us...onfigure-the-Apache-Timeout-directive-via-WHM

 

[benito]

Has the "Timeout" value on your server been changed from the default of 300 seconds? You can check this value here: https://support.cpanel.net/hc/en-us...onfigure-the-Apache-Timeout-directive-via-WHM

Yes, that was the first thing I look. At least in WHM it says "300", just to be sure I changed the value to 100 to see if it makes any difference.

 

[cPRex]

If you run "ps aux | grep httpd" does it show the process name as "/usr/sbin/httpd -k start" ? If so, are you also running nginx?

 

[benito]

If you run "ps aux | grep httpd" does it show the process name as "/usr/sbin/httpd -k start" ? If so, are you also running nginx?

Hello, yes, it shows as httpd -k start and I'm not using nginx.

I will move the customer to another server to see if have the same behavior.

In this list the problem is the PID 3387516

[[email protected] ~]# ps aux | grep httpd
nobody 158119 0.6 0.3 4590624 125756 ? Sl 06:56 0:55 /usr/sbin/httpd -k start
nobody 160954 0.5 0.3 4590624 124900 ? Sl 07:00 0:47 /usr/sbin/httpd -k start
nobody 162132 0.5 0.3 4590624 125676 ? Sl 07:00 0:51 /usr/sbin/httpd -k start
nobody 162470 0.6 0.3 4590624 127572 ? Sl 07:00 0:54 /usr/sbin/httpd -k start
nobody 189656 0.6 0.3 4590624 124772 ? Sl 07:30 0:42 /usr/sbin/httpd -k start
nobody 190290 0.5 0.3 4590624 122876 ? Sl 07:30 0:40 /usr/sbin/httpd -k start
nobody 258331 0.7 0.3 4590624 120360 ? Sl 08:30 0:23 /usr/sbin/httpd -k start
root 329353 0.0 0.0 112812 968 pts/1 S+ 09:24 0:00 grep --color=auto httpd
root 3385237 0.0 0.1 375976 64192 ? Ss Feb06 0:09 /usr/sbin/httpd -k start
nobody 3387516 84.8 0.4 4590612 151056 ? Sl Feb06 860:26 /usr/sbin/httpd -k start
nobody 4085450 0.0 0.1 374720 48116 ? S 02:06 0:18 /usr/sbin/httpd -k start
nobody 4085455 0.0 0.1 375976 48148 ? S 02:06 0:01 /usr/sbin/httpd -k start
[[email protected] ~]#

 

[cPRex]

If the issue is isolated to just one user, this is an excellent use case for CloudLinux, as that is specifically designed to keep a single user from overloading an entire system.

 

[benito]

If the issue is isolated to just one user, this is an excellent use case for CloudLinux, as that is specifically designed to keep a single user from overloading an entire system.

I have cloudlinux on my servers. You think I have to open a ticket with them for this?

Thanks

 

[cPRex]

Do you have CageFS installed?

 

[benito]

Yes, I have it installed and working.

After the latest changes I made in "Golbal apache configuration" I do not have any httpd process running for hours.

Maybe its fixed? I put all values by default. I had the option "Symlink Protection" enabled. After reading the warning about that option I suspect that maybe it had something to do with the problem.

I'll check tomorrow to see if it happens again or close the issue.

 

[benito]

Sadly the problem is back.

 

[cPRex]

I don't think Apache processes being old is an issue. My personal server, with very little traffic, has several:

# ps aux | grep http
root      1314  0.0  0.4 225216  9612 ?        Ss   16:10   0:00 /usr/sbin/httpd -k start
nobody    2497  0.0  0.2 226436  5964 ?        S    16:18   0:00 /usr/sbin/httpd -k start
nobody    2498  0.0  0.2 226436  5964 ?        S    16:18   0:00 /usr/sbin/httpd -k start
nobody    2499  0.0  0.2 226436  5968 ?        S    16:18   0:00 /usr/sbin/httpd -k start
nobody    2500  0.0  0.2 226436  5964 ?        S    16:18   0:00 /usr/sbin/httpd -k start
nobody    2501  0.0  0.2 226436  5964 ?        S    16:18   0:00 /usr/sbin/httpd -k start

Those are all at least three hours old.

If your processes are causing load on the server that's something else, but not due to the age.

 

[benito]

In my case, the processes that are running long time also are using 100% cpu time. When I kill the process, the load goes back to normal again.

 

[cPRex]

It's time to open a ticket with our team then so we can take a look.

 

[benito]

It's time to open a ticket with our team then so we can take a look.

Done #94529836

 

[cPRex]

Thanks - I'm following along on my end now.

 

[cPRex]

Our team was able to confirm that the server isn't experiencing high load, as the load rarely gets above 4, yet you have an 8-core CPU. We also determined you are running the Apache worker mpm, which is why you are seeing those parent processes linger.