In Progress CPANEL-43160 - cPHulk database issues leading to high cpu user for /usr/sbin/nft --json list ruleset

[James Row]

Almalinux 8.8.0 standard kvm
Cpanel 112.0.7

I see high cpu user for: /usr/sbin/nft --json list ruleset

cPHulk has lots of records for One-day Blocks

Narrowed it down to cPHulk database issues, hope someone can explain.

[root@buy ~]# /usr/local/cpanel/3rdparty/bin/sqlite3 /var/cpanel/hulkd/cphulk.sqlite
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
sqlite> select count(*) from ip_lists;
391987
sqlite> .quit

If you want to see how senseless those records are see this:

[root@buy ~]# /usr/local/cpanel/3rdparty/bin/sqlite3 /var/cpanel/hulkd/cphulk.sqlite
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
sqlite> select * from ip_lists limit 50;
||4|AD
||4|AD
||4|AD
||4|AD
||4|AD
etc

​

​ ​ [root@buy ~]# nft list chain inet filter cphulk table inet filter { chain cphulk { ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ... ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop } } In fact chain cphulk should have exactly 1 line looking like "ip saddr @cphulk-TempBan drop". so the entire chain should look like this: table inet filter { chain cphulk { ip saddr @cphulk-TempBan drop } } This reads to human like this: if IP source address is in table cphulk-TempBan drop the packet. So, this rule is not to block exact IP, this rule is to tell firewall to what table/list to look for IP to block. And the actual table is cphulk-TempBan, see [root@buy ~]# /usr/sbin/nft list set inet filter cphulk-TempBan table inet filter { set cphulk-TempBan { type ipv4_addr timeout 23h59m59s elements = { 46.148.40.13 expires 23h30m30s990ms, 46.148.40.49 expires 23h30m30s513ms, 46.148.40.60 expires 23h30m31s599ms, 46.148.40.61 expires 23h30m29s432ms, 46.148.40.62 expires 23h30m30s312ms, 46.148.40.63 expires 23h30m29s913ms, 46.148.40.64 expires 23h30m30s663ms, 46.148.40.65 expires 23h30m30s42ms, 46.148.40.66 expires 23h30m29s865ms, 46.148.40.67 expires 23h30m31s331ms, 46.148.40.68 expires 23h30m31s169ms, 46.148.40.69 expires 23h30m29s460ms, 46.148.40.70 expires 23h30m29s222ms, 46.148.40.72 expires 23h30m29s553ms, 46.148.40.73 expires 23h30m29s577ms, 46.148.40.77 expires 23h30m29s363ms, 46.148.40.94 expires 23h30m29s671ms, 46.148.40.107 expires 23h30m30s965ms, 46.148.40.110 expires 23h30m32s79ms, 46.148.40.111 expires 23h30m31s304ms, 46.148.40.112 expires 23h30m29s937ms, 46.148.40.113 expires 23h30m30s688ms, 46.148.40.114 expires 23h30m30s739ms, 46.148.40.115 expires 23h30m30s212ms, 46.148.40.116 expires 23h30m30s66ms, 46.148.40.117 expires 23h30m30s387ms, 46.148.40.118 expires 23h30m31s195ms, 46.148.40.119 expires 23h30m31s118ms, 46.148.40.120 expires 23h30m30s238ms, 46.148.40.121 expires 23h30m31s277ms, 46.148.40.122 expires 23h30m31s66ms, 46.148.40.135 expires 23h30m31s435ms, 46.148.40.140 expires 23h30m30s287ms, 46.148.40.141 expires 23h30m29s743ms, 46.148.40.142 expires 23h30m30s115ms, 46.148.40.143 expires 23h30m29s600ms, 46.148.40.144 expires 23h30m30s890ms, 46.148.40.145 expires 23h30m29s695ms, 46.148.40.146 expires 23h30m30s565ms, 46.148.40.147 expires 23h30m30s940ms, 46.148.40.148 expires 23h30m29s317ms, 46.148.40.149 expires 23h30m29s816ms, 46.148.40.151 expires 23h30m29s648ms, 46.148.40.152 expires 23h30m29s269ms, 46.148.40.153 expires 23h30m29s792ms, 46.148.40.183 expires 23h30m29s410ms, 46.148.40.185 expires 23h30m30s362ms, 46.148.40.186 expires 23h30m31s357ms, 46.148.40.189 expires 23h30m29s482ms, 46.148.40.190 expires 23h30m30s764ms, 46.148.40.191 expires 23h30m30s486ms, 46.148.40.192 expires 23h30m29s841ms, 46.148.40.193 expires 23h30m29s292ms, 46.148.40.195 expires 23h30m29s528ms, 46.148.40.197 expires 23h30m31s93ms, 46.148.40.198 expires 23h30m31s143ms, 79.110.48.24 expires 23h30m51s522ms, 79.110.48.30 expires 23h30m43s971ms, 79.110.48.113 expires 23h30m43s70ms, 89.107.129.123 expires 23h38m17s980ms, 103.56.61.132 expires 23h39m2s993ms, 141.98.10.150 expires 23h30m29s246ms, 141.98.10.220 expires 23h30m40s960ms } } } So, above is that actual list of IPs to ban. So those lines "ip saddr @cphulk-TempBan drop" are all bogus! Only 1 line is needed. Turns out cphulk has some bug that created enties that it should not. Hope you understand.

 

[James Row]

Response from Cpanel support
===

I have confirmed access to your server and have reviewed the state of the firewall in regards to the cphulk rules. There appears to be an issue with cPHulk adding duplicate lines to the nftables chains in the process of adding blocks based on its configured policies. At last count there were around 24 thousand duplicate entries in the nftable for cPHulk: chain cphulk { ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop ip saddr @cphulk-TempBan drop cPs# nft list chain inet filter cphulk | wc -l 24434 This is an issue we have escalated to our development team in case ID CPANEL-43160. When cPHulk detects an IP that is brute forcing and has triggered the automatic blocking procedure, a duplicate tempban reference is placed in the chain along with it. There is no workaround to prevent this behaviour yet, but I have attached the details to this ticket to the case to help them examine and resolve it as quickly as possible. For now, you should be able to use the following commands to clear the cPHulk task queue, flush the duplicate rules, and add a single rule back to ensure that the tempban rules are read: mv -v /var/cpanel/taskqueue{,.bak} nft flush chain inet filter cphulk nft add rule inet filter cphulk ip saddr @cphulk-TempBan drop /scripts/restartsrv_queueprocd --hard Clearing this queue and flushing the duplicate rules should allow the nftables management to operate faster when cPHulk detects intrusions to your server.

 

[cPRex]

Thanks for sharing that case number here! I'm watching this now and I'll be sure to post any updates I hear!