Looks like Apache keeps getting restarted?

[cwalke32477]

I have been looking at my processes, and I keep noticing several times
where it appears Apache is starting up over and over. However, I am not really sure what I am looking at.
Can anyone tell me if the following is normal behavior? This is a daily occurance, and happens all day, every day.
I haven't had any load issues, or downtimes, but it concerns me.

13404 (Trace) (Kill) nobody 0 0.4   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13031 (Trace) (Kill) nobody 0 0.3   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13445 (Trace) (Kill) root 0 0.3 0.1 tailwatchd 
10914 (Trace) (Kill) nobody 0 0.2   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11550 (Trace) (Kill) nobody 0 0.2   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
12992 (Trace) (Kill) nobody 0 0.2   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13273 (Trace) (Kill) nobody 0 0.2   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
21698 (Trace) (Kill) root 0 0.1   0.7 /usr/local/apache/bin/httpd -k start -DSSL 
8491 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
9946 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10076 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10527 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10888 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10912 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10913 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
10918 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11289 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11482 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11736 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
12473 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
12542 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13014 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13027 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13274 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13330 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13407 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13408 (Trace) (Kill) nobody 0 0.1   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13062 (Trace) (Kill) bearfaq 0 0.1 0.1 /usr/bin/php /home/bearfaq/public_html/lang.php 
13606 (Trace) (Kill) root 0 0.1 0.1 whostmgrd - serving 75.110.140.22 
9172 (Trace) (Kill) nobody 0 0.1 0.0 [httpd] <defunct> 
5035 (Trace) (Kill) root 0 0.0   1.7 /usr/sbin/clamd 
5111 (Trace) (Kill) root 0 0.0   0.6 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5 
9508 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11709 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11710 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
11733 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
12474 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13011 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13012 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13013 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13025 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13026 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13028 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13029 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13254 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13368 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13377 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13378 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13393 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13394 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13395 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13401 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13402 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13403 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13405 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
13406 (Trace) (Kill) nobody 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL 
31733 (Trace) (Kill) root 0 0.0   0.6 /usr/local/apache/bin/httpd -k start -DSSL

 

[cwalke32477]

I'm still seeing this occur.
Can anyone tell me what this is

 

[dalem]

Not enough information to tell

ran your server out of memory (just a guess)

 

[cPanelTristan]

Do you have a process killer on the machine like prm or LFD killing high CPU processes? Do you have something like sim that is set to monitor and restart Apache if it's down?

whereis prm
whereis lfd
whereis sim

If you have any of these, the above commands will show the path to the installation. If LFD is set to kill off high CPU processes, then it will show the kills in the log at /var/log/lfd.log location.

Next, you may wish to review the Apache error log at /usr/local/apache/logs/error_log location during the times you see the killed processes to see what it shows. Logs are your best friend. They are how to track down what's happening as something should be logging somewhere if Apache is either being purposely killed or restarted.

 

[cwalke32477]

Actually I do have LFD setup.
I have 8GB RAM, only use about 30% so that's not it.
I'll have to check LFD, I would bet that's the culprit.
Thanks for the info

 

[cwalke32477]

I have disabled LFD, and still get the same thing.
So I I checked the apache error logs, and cannot find anything that says it's restarting.
There are no major errors showing, using tail-f, just a bunch file not found errors giving 404's on people's sites.
I wouldn't think apache would restart each time a 404 comes up.

 

[cPanelTristan]

Hi Chris,

Are you possibly hitting MaxClients or ServerLimit settings on the machine? Check this in the /usr/local/apache/logs/error_log file:

grep -i maxclient /usr/local/apache/logs/error_log

If you are hitting that limit, additional connections will fail. This doesn't mean Apache is restarting, and in fact, what you pasted earlier were kills not restarts, which are different. You will not find restarts in the log, you should check the times you see the kils and see what it shows in the error log for Apache at the time on what is happening.

If you do find MaxClients or ServerLimit is the issue, then you can increase both in WHM > Apache Configuration > Global Configuration area. Please ensure to increase both values to the same number for those fields (MaxClients and ServerLimit).

Thanks.

 

[cwalke32477]

Looks like it has hit the limit twice in the past week.
However, that kill happens on a daily basis, and several times per day.
Hundreds even. And it always happens in blocks, around the clock.
Could it it be a mod_sec rule killing it?

On a further note, my server is setup to handle 1000 simultaneous connections.
I had to add custom entries in to config files, since the whm limit was something like 256 or 512, I can't remember.

[Tue Jan 25 15:42:09 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Jan 26 13:27:29 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting

 

[cPanelTristan]

Hi Chris,

In 11.28 WHM, the config files (include files) isn't required for adding ServerLimit so that MaxClients can be increased. You can simply do both in WHM > Apache Configuration > Global Configuration area.

As for mod_security, it's doubtful it would kill Apache processes en masse like this. Again, check the logs for when it's being killed to see what processes are showing.

Thanks.

 

[cwalke32477]

good to know about the change in 11.28's gui setting for apache.
On an interesting note, I am seeing that apache kill or restart on a new server I just deployed, with only 1 empty test account.
On it, it too, is restarting apache, as above. Again in a block several times. The block of restarts or kill (not sure what it is doing) are in blocks of 5 on the new server

 

[cPanelTristan]

Hi Chris,

Please open up a ticket then in WHM > Support Center > Contact cPanel or from the link in my signature for us to see what's going on. The only thing I know that does this type of activity would be a process killer, which you stated isn't being used.

Thanks.

 

[coderiser]

you may want to try updating cpanel and double check on the memory spikes. As memory spikes are normally what causes apache to crash.