SOLVED SSL input filter read failed solutions

[jeffschips]

Hello and Happy Halloween to all!

I see a lot of "SSL input filter read failed" in my logs and the suggested solution is to modify the Apache Virtual Hosts with Include Files to ip-based hosts.

1) where is this file located and;
2) is this a good solution (I don't like modifying configuration files too much).

Thanks.

 

[jeffschips]

The general solution to remove these "SSL input filter read failed" errors according to a search of others with this issue, is to replace virtual host names with IP addresses of the affected domain name. However, my httpd.conf already is using ip addresses. Therefore it seems useless to follow the cpanel-provided documentation on how to make changes to virtual hostnames as provided here:

cPanel & WHM Developer Portal

With its world-class support and rich feature set, cPanel & WHM has been the industry-leading web hosting platform for over 20 years. Trusted worldwide by our technology partners WordPress, CloudLinux, LiteSpeed, and more.

documentation.cpanel.net

Anybody have a solution to this?

 

[andrew.n]

can you paste us the errors you see?

 

[jeffschips]

Thank you. Logs filling up with this kind of reports:


[Sun Nov 01 03:19:42.763600 2020] [ssl:info] [pid 16205:tid 47373256558336] [client xx.52.xx.40:63864] AH01964: Connection to child 83 established (server domain.com:443)
[Sun Nov 01 03:19:43.350089 2020] [ssl:info] [pid 16205:tid 47373252355840] (70014)End of file found: [client xx.52.xx.40:63864] AH01991: SSL input filter read failed.
[Sun Nov 01 03:19:43.754538 2020] [ssl:info] [pid 16235:tid 47373235545856] [client xx.52.xx.40:65196] AH01964: Connection to child 201 established (server domain.com:443)
[Sun Nov 01 03:19:44.335558 2020] [ssl:info] [pid 16235:tid 47373233444608] (70014)End of file found: [client xx.52.xx.40:65196] AH01991: SSL input filter read failed.
[Sun Nov 01 03:19:44.623799 2020] [:error] [pid 16236:tid 47373153212160] [client xx.52.xx.40:49940] [client xx.52.xx.40] ModSecurity: Access denied with code 403 (phase 1). Matched phrase "/.env" at REQUEST_URI. [file "/etc/apache2/conf.d/modsec_vendor_configs/comodo_apache/02_Global_Generic.conf"] [line "120"] [id "210492"] [rev "3"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "domain.com"] [uri "/.env"] [unique_id "X55voGi9nyWwB8Gsy-KITgAAAQI"]
[Sun Nov 01 03:19:44.727841 2020] [ssl:info] [pid 16210:tid 47373153212160] [client xx.52.xx.40:50026] AH01964: Connection to child 130 established (server domain.com:443)
[Sun Nov 01 03:19:45.705761 2020] [ssl:info] [pid 16210:tid 47373225039616] (70014)End of file found: [client xx.52.xx.40:50026] AH01991: SSL input filter read failed.

 

[cPRex]

I searched through some tickets for that error and there wasn't just one issue that caused the "SSL input filter read failed" message to show up in the logs. I also see this error happens with Apache across multiple systems, and isn't unique to cPanel.

Have you made any customizations to the Apache configuration or tempaltes on the machine? If so, reverting those would be a good place to start troubleshooting.

We also don't recommend trying to modify the configuration directly as cPanel will overwrite that during the nightly updates. If you did want to read a bit about customizing Apache on a cPanel server, I'd recommend the following documentation:

Advanced Apache Configuration | cPanel & WHM Documentation

EasyApache 4 provides several ways for you to customize your Apache configuration.

docs.cpanel.net

 

[jeffschips]

Thanks that's useful. I did read about customizing the apache config files and they mostly address this error by suggesting to change virtual hostnames to ip address which is how my virtual host already is. So it's not that.

I haven't made any customizations to the apache files other than the cpanel/WHM standard install process. So can't figure out why this is happening.

 

[cPRex]

Do you happen to know which site triggers these? If so, that would at least let you reproduce while you adjusted server settings.

 

[jeffschips]

The reports are showing up randomly across multiple virtual hosts. A tech note here:

52794 – Log file shows ssl port as 443

bz.apache.org

says "Adding "ServerName space-station:83" will solve your issue".

I'm not so certain and actually don't know how to or would venture to mess with a custom directive.

There are a lot of reports out there of this issue with hardly any suggesting a solution other than to change virtual hosts file from domainname.com to ip address.

Thoughts?

 

[cPRex]

*I* personally don't have any additional thoughts based on the data I have here. You're always welcome to open a ticket from WHM or using the link in my signature to have our team do some more in-depth checking on this.

 

[jeffschips]

thanks cPRex appreciate the assistance!

 

[cPRex]

If you do submit a ticket, let us know what you find out here! Of get me the ticket number and I can update this thread once we get a resolution.

 

[jeffschips]

Of course - will do!

 

[andrew.n]

Hmmm it could be related to EA-6020 which was a case a few years ago.

Apache has a race-condition when it kills and restarts its piped-logging processes on graceful restart before all of its children handling ongoing client connections have finished, resulting in a "Broken pipe" error when those children attempt to log to a pipe that no longer exists.

Because it is a race-condition, this will be more likely to happen on busier servers where httpd children are servicing a client request in the middle of a graceful restart, and unlikely to be seen on idle servers.

Can you try to disable Piped Logging via "WHM >> Apache Configuration >> Piped Log Configuration" to see if the error messages go away?

 

[jeffschips]

Hi. Piped logging was disabled a long time ago. The reason is because if it's not disabled then reactive web firewalls like csf and other, which respond to logged events, are delayed and thus protection is delayed - at least that's my understanding.

So, yes, piped logging was disabled already.

 

[jeffschips]

If you do submit a ticket, let us know what you find out here! Of get me the ticket number and I can update this thread once we get a resolution.

Issue submited to cpanel tech support. Thank you. Where do I post the ticket number?

 

[cPRex]

You can just post it here - the ticket number isn't top secret or sensitive :D

 

[jeffschips]

Thank you:
93877290

 

[cPRex]

Thanks - I'm watching the ticket on my end too so hopefully we'll hear something soon!

 

[cPRex]

So after some testing, our technicians found the LogLevel value in the Apache settings was changed from the default of "warn" to "info" which was causing these to be logged in Apache, even when these weren't necessarily related to an issue. I've added some emphasis from the earlier log that @jeffschips provided to make this more clear:

[Sun Nov 01 03:19:45.705761 2020] [ssl:info] [pid 16210:tid 47373225039616] (70014)End of file found: [client xx.52.xx.40:50026] AH01991: SSL input filter read failed.

Glad we were able to help track that down!

 

[jeffschips]

SOLVED: Indeed my tests show no more "SSL input filter read failed" when changing the log level to "warn" in the apache configuration settings in cpanel. Thanks for all the great assistance by all and also the very professional Cpanel techs!