Very high CPU loads from brute force attempts - CSF/LFD

msklut · Apr 26, 2022

Over the last few days, we've experienced nearly 100% CPU load on a daily basis. Throughout the day there are hundreds of csf and lfd processes running, all of which are to block IP addresses from brute force attempts to our website's admin page. There are also hundreds of mysqld and httpd processes which are related to the login/database attempts. What can we do to bring down the CPU load? The number of tasks running has never been this high. It's now anywhere from 200 - 600 tasks. CSF and cPhulk are configured to block these attacks, but there are far too many that it's overloading the server and regular tasks (e.g. sending/receiving email from server) are nearly impossible.

LF_MODSEC Log

Code:

[Tue Apr 26 07:25:21.273168 2022] [:error] [pid 11162] [client 173.249.19.246:59052] [client 173.249.19.246] ModSecurity: Access denied with code 406 (phase 1).
Pattern match "Mozilla/5.0 \\\\(X11; Ubuntu; Linux x86_64; rv:62\\\\.0\\\\) Gecko\\\\/20100101 Firefox\\\\/62\\\\.0" at REQUEST_HEADERS:User-Agent.
[file "/etc/apache2/conf.d/modsec/modsec2.user.conf"] [line "1"] [id "91996789"] [msg "BAD UA BLOCK"] [hostname "mydomain.com"]
[uri "/admin/"] [unique_id "YmfWoRM2Fjtq6jMg5hEDNgAAAAw"]

top (there are hundreds of these processes/commands)

Code:

Tasks: 329 total,   2 running, 323 sleeping,   1 stopped,   3 zombie
%Cpu(s): 70.8 us, 18.4 sy,  0.0 ni,  0.0 id,  9.5 wa,  0.0 hi,  1.3 si,  0.0 st
KiB Mem :  3880140 total,   545588 free,  1364884 used,  1969668 buff/cache
KiB Swap:  4194300 total,  3958600 free,   235700 used.  2191960 avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND      
30930 root      20   0  179440  30608   2824 S  14.6  0.8   0:00.43 csf                                         
30926 root      20   0  179280  30420   2820 S  14.2  0.8   0:00.42 csf                                         
30931 root      20   0  179284  30412   2820 S  14.2  0.8   0:00.42 csf                                         
30932 root      20   0  179280  30484   2820 S  14.2  0.8   0:00.42 csf                                         
30925 root      20   0  179132  30388   2820 S  13.9  0.8   0:00.41 csf                                         
30928 root      20   0  179132  30408   2820 S  13.6  0.8   0:00.40 csf                                         
30929 root      20   0  179136  30420   2820 S  13.6  0.8   0:00.40 csf                                         
30934 root      20   0  179136  30392   2820 R  13.6  0.8   0:00.40 csf                                         
30935 root      20   0  179136  30420   2820 R  13.6  0.8   0:00.40 csf                                         
30927 root      20   0  179000  30052   2820 R  13.2  0.8   0:00.39 csf                                         
30936 root      20   0  178212  29304   2816 R  12.9  0.8   0:00.38 csf                                         
30937 root      20   0  178340  29528   2820 R  12.9  0.8   0:00.38 csf                                         
30933 root      20   0  173964  27292   2800 R  11.9  0.7   0:00.35 csf                 
22275 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 148.202.167.75      
23194 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 38.135.34.49        
23196 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 138.97.220.166      
23198 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 212.47.227.85       
23356 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 163.172.53.199      
23473 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 189.254.45.110      
23930 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 91.238.161.177      
24066 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 62.173.139.188      
24181 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 178.32.202.97       
24182 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 128.199.241.20      
24185 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 178.128.151.87      
24190 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 79.175.127.171      
24340 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 45.80.153.73        
24342 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 12.12.141.226       
24344 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 212.7.211.113       
24345 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 217.115.118.126     
24348 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 79.172.201.113      
24351 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 76.245.195.148      
24434 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 51.178.185.66       
24437 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 141.94.32.98        
24438 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 178.128.55.40       
24535 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 138.97.220.166      
24699 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 202.29.148.67       
24875 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 159.203.28.59       
24877 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 31.173.68.7         
24878 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 188.40.33.77        
24879 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 203.23.49.192       
24880 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 162.214.104.98      
25106 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 43.229.77.90        
25107 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 38.135.34.49        
25109 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 178.62.213.36       
25110 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 103.144.82.1        
25111 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 207.180.213.165     
25446 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 45.79.68.53         
25447 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 47.88.23.114        
25448 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 139.59.68.9         
25449 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 46.101.188.174      
25450 root      20   0  188452  35236    896 S   0.0  0.9   0:00.01 lfd - (child) blocking 207.180.236.152     
25451 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 203.210.87.64       
25837 root      20   0  188452  35236    896 S   0.0  0.9   0:00.02 lfd - (child) blocking 76.103.114.159      
25935 root      20   0  188452  35236    896 S   0.7  0.9   0:00.02 lfd - (child) blocking 180.242.130.79      
25939 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 103.41.204.29       
25940 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 138.201.142.73      
25941 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 182.70.248.147      
25944 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 178.128.155.255     
25946 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 51.79.248.189       
25948 root      20   0  188452  35236    896 S   0.3  0.9   0:00.01 lfd - (child) blocking 92.205.25.196       
21206 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 178.128.150.247     
23352 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 213.187.11.93       
23688 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 185.21.217.56       
23694 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 51.77.214.27        
23923 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 185.148.3.93        
24064 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 51.15.181.37        
24697 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 145.131.25.246      
24874 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 83.137.145.154      
25103 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 12.12.141.226       
25258 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 219.153.110.7       
25259 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 68.183.175.58       
25261 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 185.149.103.55      
25445 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 196.41.123.124      
25452 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 51.77.214.27        
25639 root      20   0  188452  35232    892 S   0.0  0.9   0:00.02 lfd - (child) blocking 185.116.215.125     
25727 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 190.105.205.100     
25728 root      20   0  188452  35232    892 S   0.0  0.9   0:00.01 lfd - (child) blocking 51.178.136.52
1351 nobody    20   0  547088  22652   3624 S   0.3  0.6   0:00.13 httpd                                               
3111 nobody    20   0  546820  21616   2900 S   0.3  0.6   0:00.09 httpd                                               
3157 nobody    20   0  546820  21616   2900 S   0.3  0.6   0:00.09 httpd

kodeslogic · Apr 26, 2022

From the output you shared, it seems an attack from multiple IPs, you should reach out to your data center or server provider, as they have specialized equipment to put in place to help mitigate the attack until it slows down or dies off.

Other options:
- Using CDN or external firewall services such as Cloudflare can help buffer traffic to the server.
OR
- Consult a cPanel certified System administrator to help you to analyze the situation and guide you to the next best step.

cPRex · Apr 26, 2022

I would also recommend an external tool. If you try and handle this locally, you'll still need to use your server's CPU processing power to handle the traffic. An external solution would completely remove your CPU from the equation, and your hosting provider likely has an external firewall or DoS prevention tools available.

msklut · Apr 26, 2022

We've had the VPS configured with CSF and cPhulk for several years now with no issues until now. Are there any Mod_Security rules or anything like that you would recommend looking at?

cPRex · Apr 26, 2022

mod_security wouldn't handle that type of traffic, as that watches uploads through the web and not brute force attacks.

msklut · Apr 26, 2022

cPRex said:
mod_security wouldn't handle that type of traffic, as that watches uploads through the web and not brute force attacks.

Look at the logs below. It shows Mod_Security rules are working to block the following...

Code:

Message: Access denied with code 406 (phase 1). Pattern match "Mozilla/5.0 \\(X11; Ubuntu; Linux x86_64; rv:62\\.0\\) Gecko\\/20100101 Firefox\\/62\\.0" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec/modsec2.user.conf"] [line "1"] [id "91996789"] [msg "BAD UA BLOCK"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 80.253.246.193] ModSecurity: Access denied with code 406 (phase 1). Pattern match "Mozilla/5.0 \\\\\\\\(X11; Ubuntu; Linux x86_64; rv:62\\\\\\\\.0\\\\\\\\) Gecko\\\\\\\\/20100101 Firefox\\\\\\\\/62\\\\\\\\.0" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec/modsec2.user.conf"] [line "1"] [id "91996789"] [msg "BAD UA BLOCK"] [hostname "mydomain.com"] [uri "/admin/"] [unique_id "YmhEsUkj-UhCPIMszGtGEQAAAA8"]
Action: Intercepted (phase 1)
Stopwatch: 1651000497800531 351 (- - -)
Stopwatch2: 1651000497800531 351; combined=47, p1=20, p2=0, p3=0, p4=0, p5=27, sr=0, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.0.2.
Server: Apache
Engine-Mode: "ENABLED"

cPRex · Apr 26, 2022

For that particular request, it seems they are trying to log in to domain.com/admin and their activity is triggering mod_security. However, if this is happening as frequently as you report, it would still be best to look to external firewall solutions since your server would have to process anything related to mod_security.

Ideally, if the traffic can be stopped before they are even able to reach the server, that would be the ideal solution.

msklut · Apr 26, 2022

cPRex said:
For that particular request, it seems they are trying to log in to domain.com/admin and their activity is triggering mod_security. However, if this is happening as frequently as you report, it would still be best to look to external firewall solutions since your server would have to process anything related to mod_security.

Ideally, if the traffic can be stopped before they are even able to reach the server, that would be the ideal solution.

OK. Can you think of anything else we should look at before going with an external option? We have not had these type of attacks before -- that's what's alarming and confusing.

cPRex · Apr 26, 2022

I personally don't have additional ideas, but that's mostly how everything works - I don't *need* to enable a slow query log, until I notice database slowness. I don't *need* to increase my RAM until my users grow and it starts running low. You didn't *need* to explore DoS protection options, until today.

If you'd like to submit a ticket to our team we could at least check the system for known security problems and ensure cPanel itself is working well.

msklut · Apr 26, 2022

cPRex said:
I personally don't have additional ideas, but that's mostly how everything works - I don't *need* to enable a slow query log, until I notice database slowness. I don't *need* to increase my RAM until my users grow and it starts running low. You didn't *need* to explore DoS protection options, until today.

If you'd like to submit a ticket to our team we could at least check the system for known security problems and ensure cPanel itself is working well.

OK. We've had CSF and cPhulk since the beginning. Just odd how it only started presenting an issue years later.

Mise · Apr 26, 2022

the overload could be because csf should execute the blocking and write the ips inside the csf.deny file. The attack seem to be string chains inside the User Agent header, and then returning a 406 error. Perhaps you could mitigate the situation moving the blocking into the .htacces of the atacked website, and then disabling the modsecurity rule. In that way the csf could be free of all that work

The ideal thing would be studying the logs, to see what those headers contains. And then build the precise rule inside the .htaccess with related strings. Something like this although with your strings:

RewriteCond %{HTTP_USER_AGENT} (libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]

... and so on

you can find many examples of HTTP_USER_AGENT for .htaccess everywhere

In that way you could disable the modsecurity rule, to convert that overload in simple common requests. And csf would be free of all that work.
In case of different attacked websites you could include these rules inside the Apache configuration and later disable that modsecurity rule

msklut · Apr 27, 2022

Mise said:
the overload could be because csf should execute the blocking and write the ips inside the csf.deny file. The attack seem to be string chains inside the User Agent header, and then returning a 406 error. Perhaps you could mitigate the situation moving the blocking into the .htacces of the atacked website, and then disabling the modsecurity rule. In that way the csf could be free of all that work

The ideal thing would be studying the logs, to see what those headers contains. And then build the precise rule inside the .htaccess with related strings. Something like this although with your strings:

RewriteCond %{HTTP_USER_AGENT} (libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]

... and so on

you can find many examples of HTTP_USER_AGENT for .htaccess everywhere

In that way you could disable the modsecurity rule, to convert that overload in simple common requests. And csf would be free of all that work.
In case of different attacked websites you could include these rules inside the Apache configuration and later disable that modsecurity rule

have you heard of this? Looks like it has all the htaccess rewrite rules that would be necessary:

7G Firewall | Perishable Press

The 7G Firewall is here! 7G is now out of beta and ready for production sites. So you can benefit from the powerful protection of the latest nG Firewall...

perishablepress.com

msklut · Apr 27, 2022

The .htaccess rules seem to be working pretty well. Thanks for the suggestion. Quick question... Apache logs are showing an error for this rewrite rule. Any ideas?

RewriteCond: cannot compile regular expression

Code:

<IfModule mod_rewrite.c>
    RewriteCond %{HTTP_USER_AGENT} (acapbot|acoonbot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|extract|eyenetie|feedfinder|flaming|flashget|flicky|foobot|g00g1e|getright|gigabot|go-ahead-got|gozilla|grabnet|grafula|harvest|heritrix|httrack|icarus6j|jetbot|jetcar|jikespider|kmccrew|leechftp|libweb|linkextractor|linkscan|linkwalker|loader|masscan|miner|mechanize|morfeus|moveoverbot|netmechanic|netspider|nicerspro|nikto|ninja|nutch|octopus|pagegrabber|petalbot|planetwork|postrank|proximic|purebot|pycurl|python|queryn|queryseeker|radian6|radiation|realdownload|scooter|seekerspider|semalt|siclab|sindice|sistrix|sitebot|siteexplorer|sitesnagger|skygrid|smartdownload|snoopy|sosospider|spankbot|spbot|sqlmap|stackrambler|stripper|sucker|surftbot|sux0r|suzukacz|suzuran|takeout|teleport|telesoft|true_robots|turingos|turnit|vampire|vikspider|voideye|webleacher|webreaper|webstripper|webvac|webviewer|webwhacker|winhttp|wwwoffle|woxbot|xaldon|xxxyy|yamanalab|yioopbot|youda|zeus|zmeu|zune|zyborg) [NC]

    RewriteRule .* - [F,L]

</IfModule>

msklut · May 6, 2022

Following up on this... We noticed immediate changes when we enabled the DoS rules in ModSecurity (which are disabled by default).

/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/crs-setup.conf (below shows the rule enabled without the '#')

Code:

#
# Optional DoS protection against clients making requests too quickly.
#
# When a client is making more than 100 requests (excluding static files) within
# 60 seconds, this is considered a 'burst'. After two bursts, the client is
# blocked for 600 seconds.
#
# Requests to static files are not counted towards DoS; they are listed in the
# 'tx.static_extensions' setting, which you can change in this file (see
# section "HTTP Policy Settings").
#
# For a detailed description, see rule file REQUEST-912-DOS-PROTECTION.conf.
#
# Uncomment this rule to use this feature:
#
SecAction \
"id:900700,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.dos_burst_time_slice=60',\
  setvar:'tx.dos_counter_threshold=100',\
  setvar:'tx.dos_block_timeout=600'"

Mise · May 15, 2022

I would try somehting like this depending of the attack strings:

RewriteCond %{HTTP_USER_AGENT} ^.*(string1|string2|string3.....).*$ [NC]
RewriteRule ^ 406 [L,R]

letters between [..] like [NC] are conditional flags when there is more than one rule, and also for other purposes. Errors can appears because this reason. Here one manual:

RewriteRule Flags - Apache HTTP Server Version 2.4

that modsec rule 900700 could cause memory impact depending the load; another issue to investigate as you do, with try and check.

Good luck

Thread starter	Similar threads	Forum	Replies	Date
A	Load very high	Security	3	Aug 18, 2023
M	New very dangerous security bug/feature in cpanel filter function	Security	1	Sep 20, 2022
K	Enabling DNSSEC resulting in DNS propagation/Email delivery issues	Security	1	Nov 12, 2021
J	imunify360 constant cron job every minute	Security	5	May 31, 2021
B	Extremely high server load - BDflush running very high- possible hack	Security	9	Jul 20, 2005

Search

Search

Very high CPU loads from brute force attempts - CSF/LFD

msklut

Well-Known Member

kodeslogic

Well-Known Member

cPRex

Jurassic Moderator

msklut

Well-Known Member

cPRex

Jurassic Moderator

msklut

Well-Known Member

cPRex

Jurassic Moderator

msklut

Well-Known Member

cPRex

Jurassic Moderator

msklut

Well-Known Member

Mise

Well-Known Member

msklut

Well-Known Member

7G Firewall | Perishable Press

msklut

Well-Known Member

msklut

Well-Known Member

Mise

Well-Known Member