In Progress CPANEL-42825 - Ratelimit incoming mail to non-existent accounts

[SimpleTechGuy]

Hi, so I recently migrated to a new server and since the migration been getting thousands of emails to non existent accounts every day (coincidence? maybe. I don't know) I did keep the same ip, just a different box with the same host.

I think it is a dictionary attack, but not sure. Each attack is coming from a different ip address and each ip address is sending 100-200 emails at a time. They are coming from every country on the list including the united states and each one is a different domain name. Some are even using comcast and spectrum ips. All 100+ emails from each ip are happening in a 1 second time frame. Most of them are rejected by sender verify, but some make it to "no such user here" reject. As far as I can tell these attacks seem to be harmless and exim/cpanel/spamassassin are doing their jobs, but it's destroying the logs.

It seems like exim is ratelimiting them somehow, but I don't know for sure. Wondering if anyone knows of a way to do something like this:

If 10.20.30.40 sends more than 5 emails to a non-existent account, then block 10.20.3.40

thanks,

 

[SimpleTechGuy]

I'm now getting so much spam that the message below is popping up on my Mail Delivery Report for a 24 hour lookup: Right now I have over 5,000 failures on my server in just 12 hours. I really hope someone has a solution to this. thanks,

This query produced a result set that exceeds the reportable limit. If you cannot find what you are looking for by paging through these results, please restrict the query further by narrowing your search.

 

[SimpleSonic]

In your Exim Configuration Manager, just make sure the following are enabled:

• Dictionary attack protection
• Ratelimit suspicious SMTP servers
• Ratelimit incoming connections with only failed recipients

They should be enabled by default, if not you may want to enable them.

 

[cPRex]

What @ResellerWiz said above. Can you confirm those are enabled?

 

[SimpleTechGuy]

Hi @cPRex, all of the settings mentioned are currently set to "on".

I noticed that I was seeing this log several times at one point (generally after 100 connections from the same ip): message = "Number of failed recipients exceeded. Come back in a few hours." I was able to find that message in the Advanced config in the Dictionary Attack Protection, but I haven't been about to figure out how it works yet. I'm using the default config below:

  warn
    log_message = "Detected Dictionary Attack (Let $rcpt_fail_count bad recipients though before eng
aging)"
    condition = ${if > {${eval:$rcpt_fail_count}}{4}{yes}{no}}
    set acl_m7 = 1

  warn
    condition = ${if eq {${acl_m7}}{1}{1}{0}}
    ratelimit = 0 / 1h / strict / per_conn
    log_message = "Increment Connection Ratelimit - $sender_fullhost because of Dictionary Attack"

  drop
    condition = ${if eq {${acl_m7}}{1}{1}{0}}
    message = "Number of failed recipients exceeded.  Come back in a few hours."

--

I'm still trying to figure out how the ratelimit formula works: ratelimit = 0 / 1h / strict / per_conn

 

[cPRex]

Exim averages the ratelimits over time. However, the default values on a system are these:

ratelimit = 1.2 / 1h / strict / per_conn / noupdate

so it appears yours may have been altered. I'd put back the defaults, but then bump the "1h" to something more aggressive like '8h' so it keeps blocking the connection for at least 8 hours.

 

[SimpleTechGuy]

So for my default ratelimit in ratelimit (Ratelimit suspicious SMTP servers) it is already set to what you posted ratelimit = 1.2 / 1h / strict / per_conn / noupdate

But for the dictionary_attack (Dictionary attack protection) it is set to ratelimit = 0 / 1h / strict / per_conn

Can you confirm the default ratelimit in the dictionary_attack (Dictionary attack protection) section in Advanced Config?

Also, and either way, that would only change the time of the ratelimit right? And that's great, but they seem to have an infinate number of IP addresses, so I'm really hoping to find a way to limit each ip from sending more than 3 or 4 emails before the ratelimit kicks in and blocks them.

 

[SimpleTechGuy]

My current status:

 

[cPRex]

Ah yes, the specific section under the dictionary attack is different, and that is normal.

I'd recommend bumping the hour value to 8 under the dictionary section then as well and see if that helps to calm things down.

 

[SimpleTechGuy]

So I think the problem now is that the emails are not getting rate limited at all. I haven't seen any log messages that indicate ratelimiting. It's almost as if it's not working at all. The crazy thing is that all this started after the migration. I really think something happened during the migration that's causing the issue. I've got a ticket in now, will report back what I find.

 

[SimpleSonic]

So I think the problem now is that the emails are not getting rate limited at all. I haven't seen any log messages that indicate ratelimiting. It's almost as if it's not working at all. The crazy thing is that all this started after the migration. I really think something happened during the migration that's causing the issue. I've got a ticket in now, will report back what I find.

That is weird that it only started after migrating. During the migration did you choose to copy the Exim config from the source server? If so, maybe Exim has some inconsistencies?

Either way, I’m interested to see what the support team finds.

 

[SimpleTechGuy]

Yes sir, it was a full migration, I selected to migrate everything, but and although there were some error messages with cphulk and a few of the individual accounts, I don't remember seeing anything related to exim. I also ran the "Validate Exim configuration" from the api and it reported okay. I will report back as I learn more.

I really have a sneaking suspicion that this is related to my ACLs... I've noticed that the ACL visualizer is reporting incorrect information. Not sure if this is a bug or of it is only affecting me, but let's say I turn off sender verify, restart exim and run the visualizer, then sender verify acl should be removed, but it's not. Any chance someone could check and see if this is happening on your servers as well?

 

[cPRex]

It sounds like it might be time to make a ticket on this issue.

 

[SimpleTechGuy]

Made one yesterday, still waiting on someone to pick it up.

 

[cPRex]

Can you post the number here so I can follow along?

 

[SimpleTechGuy]

#95064510

 

[cPRex]

Thanks for that - I'm following along with that now.

 

[SimpleTechGuy]

Turns out this is a known issue (see below). I'm not sure, but the Internal Case ID looks pretty old....

After further review, I have found that your server is currently facing a known issue with the Dictionary Attack protection at this time with case ID CPANEL-42825.

When the "Default Address" is set to the option to "Fail" a message to a user that does not exist, it creates the following Alias to process this:
[-redacted- ~]cPs# cat /etc/valiases/-redacted-.com
*: :fail: No such person at this address.

At this time, the "Fail" Alias is being processed earlier in the mail process than the Dictionary Attack Protection implementation, Spam Filters, RBL checks, and other protections. Unfortunately, this is allowing Dictionary Attacks to be performed on the server as these protections will not trigger. Instead, the server simply responds with your "Fail" message and continues.

Eventually, the attackers are hitting a real email address; as such, the Alias is not processed and the Dictionary Attack Protection is triggered. This then sees the previous attempts, and is why you are seeing "delayed" counts

Wondering if anyone knows why valiases are still being used since the email alias feature has been removed since v106?

So apparently the only solution right now is:

At this time, the only known workaround for this is to set the "Default Address" configuration in cPanel to any option other than:
"Discard the email while your server processes it by SMTP time with an error message."

Which really isn't a solution because trusted clients would not be notified in the event of a misspelled email.

I was able to verify this prior to creating the ticket by setting the accounts Default Address to blackhole [Discard (Not Recommended)] and the spam was stopped by RBLs, but blackhole really isn't an option for me. Forwarding isn't an option either as a legitimate sender would never be notified of a fail delivery. So it's looking like the only other solution is to just deal with the spam and hope it quits or find some kind of 3rd party email filter.

This is honestly a bit frustrating. I've never experimented with the "pipe to a program" option. Maybe it's possible to pipe those messages back to exim, or maybe pipe to some type of 3rd party mail filter that would check for an active user and if not found then send the 550 error?

At the end of the day it seems like it should be an easy thing to move the processing of the valias to some point after the ratelimiting checks... But I don't know....

 

[SimpleTechGuy]

It turns out the internal case ID CPANEL-42825 is a fairly new case after all and is expected to be fixed, although not sure when. in the meantime I have an idea for a temporary solution.

In exim Advanced config, the option smtp_accept_max is set by default to 100. This allows maximum 100 connections to smtp and is effectively limiting the attacks to 100, however, I don't want to edit this as it would likely cause issues with legit mail, so I found this smtp_accept_max_per_host which as the following definition from exim:

This option restricts the number of simultaneous IP connections from a single host (strictly, from a single IP address) to the Exim daemon. The option is expanded, to enable different limits to be applied to different hosts by reference to $sender_host_address. Once the limit is reached, additional connection attempts from the same host are rejected with error code 421. This is entirely independent of smtp_accept_reserve. The option’s default value of zero imposes no limit. If this option is set greater than zero, it is required that smtp_accept_max be non-zero.


Warning: When setting this option you should not use any expansion constructions that take an appreciable amount of time. The expansion and test happen in the main daemon loop, in order to reject additional connections without forking additional processes (otherwise a denial-of-service attack could cause a vast number or processes to be created). While the daemon is doing this processing, it cannot accept any other incoming connections.

What concerns me is the "Warning". Does anyone know what expansion constructions are?

 

[SimpleTechGuy]

Found this definition of "String Expansions"; Guessing this is what that warning is talking about. Assuming the warning is directly related to String Expressions inside the smtp_accept_max_per_host variable. I'm going to try setting this on the server and see what happens. Wish me luck!