SOLVED Extremely long "blocked" and "TLS setup" timings until I restart Apache?

[Benjamin D.]

Hi, I would like to know if you guys have any pointers or ideas for what to check as I'm facing a new issue that I've never experienced before and it's been quite constant now for almost 2 days straight.

I'm experiencing extremely long blocked and TLS setup timings until I restart Apache. Restarting Apache takes approximately 5 seconds and then everything is SO DAMN QUICK, like I can load a web page on any of this server's PHP website almost instantaneously. But after a random amount of time in the neighborhood of an hour or less, my server does the same again until I restart Apache. I'm getting tired of restarting Apache and I would like to know what I can check to resolve this?

NOTE: When this happens, WHM still remains quick on port 2087, it does not seem to be affected by this issue, only the server's hosted web pages and resources are.

What I have checked so far:

1) iftop, which indicates no issue, always less than 10% bandwidth used (e.g. 30-40mbps out of 1000mbps that I have) with a ton of q ready to serve.

2) top, which indicates no CPU and RAM related bottlenecks at all, server load is always in the range of 4 to 5 out of 8 (has been like this for +7 years) and RAM is always less than 10% used.

3) This is what Firefox shows when the issue is happening: (see attached)

 

[cPRex]

Hey there! The next time this happens, can you post the output from the scoreboard section of "apachectl status" so I can see that? I'm specifically looking for the area that looks like this, which is typically near the bottom of the output:

 _________________________.........................______________
 _______________________________________W_____________________...
 ......................_________________________

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current proces

 

[Benjamin D.]

It's still happening, I left it happen while I was waiting for somebody to answer here.

To answer your question, here's what I see AND thanks for replying to me

RRRKRRKRRRGKKWRRRRRRRRRWRRRRRRRKRGRRRKKKRRWRRKKWRRRRKRRRKRKRKRRK
RRRRGRKKKRRRRKWKKRKRGRRRRRRKRRRRRRKRRRRRRRRRRRKGKRWRRRRRKRRRKRKR
KRRRRRRKRRRRRGRRRRRRKK

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process


... Am I to understand from this that I need more Apache workers? I'm on Server Limit = 256 (default) and Max Request Workers = 150 (default).

... While I was waiting for a reply to the above, I went ahead for fun and rebooted Apache with Server Limit = 300 and Max Request Workers = 200 and it went great for about an hour and now it looks like this and the slowliness is the same (and I noticed that there's a 15% to 20% CPU usage increase at all time now which is still OK) :

LKKRRRRRRKRRLKRRRRRKKWRKRRRRKRRRRRRKKRKRKRRKRRKRRRRRRLRRKRRKKRWR
KRRRRKWRRRRRKRRRRKRKRRRRRKRRKWKKRKWKKRRRKKRRRRKRRRKRRRRKWRRRRRRR
RKRRRKLRWRKRWWRRWWRRRKKWRRRRRWKKRKWRRRRRKRKRRRRRRKRKKRWWWRRKRRRR
RRRKRKRR

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

While we're here, what is the reason why WHM says the default Apache 2 TimeOut setting is 300 seconds while the standard is 60? core - Apache HTTP Server Version 2.4

 

[cPRex]

It's much less likely that your server is suddenly not able to handle the traffic, and much more likely that you're experiencing a Denial of Service attack of some sort.

If you check "apachectl fullstatus" you can see what connections are being made to the server. It will be a LOT of output, but may give you more ideas on what site is being reached/attacked.

I'd check there, and if it does look like an attack you can use something like mod_evasive to help stop that. If that tool isn't enough, you may need a more advanced external solution to resolve the issue.

As far as the defaults, I'm not sure, but we tweak many options from the defaults, and it wouldn't be related to your issue.

 

[Benjamin D.]

I was considering being attacked as well as this kind of traffic has never been an issue in the past +7 years with this server. Though, I remember having a DDoS attack last year and the datacenter where my server is located has automatic mitigation solutions that automatically stop the DDoS traffic and then sends me an email to inform me about this. They have not sent me anything over the last 2 days during which this "attack" has been conducted.

 

[cPRex]

From the data presented so far, it looks like an attack, whether or not you've received an external notification from somewhere or not. There really isn't another explanation unless your one of your sites got far more popular than it used to (which would also show up in the traffic logs)

 

[Benjamin D.]

Since the issue still exists, I've executed: apachectl fullstatus

Is there something in that log that I should watch for?

For instance, I don't know if it's normal or not, but I'm seeing more than 100 slightly different IP addresses all beginning with the same 2 first parts of the network ID are exactly the same, but the last 2 parts of the IP differ. They all seem to come from the same country according to IP-TRACKER.ORG

 

[cPRex]

It's not a log that you can watch, it's a real-time snippet of the traffic Apache is handling.

You want to look through the output one line at a time, and see if the same IP address in the "Client" column is being repeated a large number of times, and if so, if that matches a particular URL from the "Request" column. If the former, you can block the IP and see if that lowers the load. If the latter, the attack could be directed at a specific site or page.

 

[Benjamin D.]

Yes, I'm aware that the "log" is live, sorry for the lack of proper technical terms. But yes, I understand that it's live.

Their technique is that there is no unique IP that repeats many time. They literally use a whole block of network dedicated to attacking the server. This is what I see AND NOTE that there are more than 100 IP's that begin with the same network ID 11.111 (redacted for privacy even though I don't think they actually deserve this) :

11.111.3.4
11.111.16.19
11.111.9.12
11.111.67.34
11.111.154.124
11.111.6.22
11.111.213.4
11.111.172.14
11.111.173.45
... more than 100 different IP that all begin with the same 2 parts.

 

[cPRex]

Definitely an attack. I'd block that whole 11.111 range (although it says it's based out of Ohio) and hopefully that's all you need to do!

 

[Benjamin D.]

Note that it's not really 11.111. I've redacted this for their privacy although they don't deserve any IMO. They're overseas, I checked on IP-TRACKER.ORG already.

I've blocked the whole 11.111.0.0 (redacted for privacy) range of IP with a single HTACCESS line and the server is now very fast and idle workers seem to be available at any given time now. Been testing for 15 minutes. Looks real good now.

THANKS! :D This is the kind of support I'm expecting from software I'm paying 40$/month for. I appreciate your help and thanks for helping me to figure this out!

 

[cPRex]

Ah, I gotcha. Time to block that range, then!

 

[Benjamin D.]

Please close this thread and thanks again for all your very informative help

 

[cPRex]

You're very welcome!