The service “cpanel-dovecot-solr”/"clamd" appears to be down

[Olufemi Lawal]

I've been receiving notification emails saying that these services are down and then after about 4 minutes I receive emails sayings: The service “cpanel-dovecot-solr”/"clamd" is now operational.

Here's some more information:

For Clamd:
Service Name clamd
Service Status failed
Notification The service “clamd” appears to be down.
Service Check Method The system’s command to check or to restart this service failed.
Number of Restart Attempts 1
Service Check Raw Output (XID 34q7h5) The “clamd” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_clamd” reported error number 255 when it ended.
Startup Log No startup log
Memory Information
Used 2.52 GB
Available 8.48 GB
Installed 11 GB
Load Information 3.33 4.36 2.67
Uptime 91 days, 9 hours, 43 minutes, and 31 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 1.95 0.01 0.17 0.01 0.00 97.86 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

For Dovecot_solr:
Number of Restart Attempts 1
Service Check Raw Output (XID nu9p7p) The “cpanel_dovecot_solr” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_cpanel_dovecot_solr” reported error number 255 when it ended.
Startup Log No startup log
Memory Information
Used 2.08 GB
Available 8.92 GB
Installed 11 GB
Load Information 2.96 4.43 2.63
Uptime 91 days, 9 hours, 42 minutes, and 53 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 1.95 0.01 0.17 0.01 0.00 97.86 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

I'm not sure if this is associated but we've also been receiving spam emails that falsely appear to have been send from an email address on our server.

Any help on how best to proceed will be very much appreciated.

 

[GOT]

Hard to say with any certainty. These are both mail related services and if you are having a massive spam outbreak as you post implies they could be crashing under load.

How big is your mail queue?

Might look in /var/log/maillig and messages around the time of the notice to see if theres any messages about crashes or terminated processes

 

[Olufemi Lawal]

Thanks for the reply GOT.

I looked into the maillog and and then exim_paniclog and saw a lot of OOM crash message. The processes spamd, clamd and java were being killed due to a lack of memory.

I'm looking into ways to limit the amount of RAM that ClamAV takes. There doesn't seem to be an easy fix. Any other anti-virus suggestions or other solutions?

 

[GOT]

You post says you have 11 GB ram. That is an odd amount. And its unlikely that these services are actually what are taking up all your ram. I would start looking at your ram utilization in other areas. Prime suspects would be mysql, apahce and php typically.

 

[cPanelLauren]

Hi @Olufemi


Can you show me an example of the oom message? Are these present in just /var/log/maillog or in /var/log/messages as well?

 

[Olufemi Lawal]

Hi Lauren,

I've only found these OOM messages in the /var/log/messages. Here is an excerpt.

Oct 24 04:52:57  PAM-hulk[5445]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES

Oct 24 04:53:06 2 PAM-hulk[5445]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES

Oct 24 05:38:44 : [7824298.758448] Out of memory in UB 125883: OOM killed process 9235 (clamd) score 0 vm:949668kB, rss:20452kB, swap:524776kB

Oct 24 05:38:50 kernel: [7824304.760221] Out of memory in UB 125883: OOM killed process 9078 (java) score 0 vm:6212272kB, rss:61516kB, swap:286936kB

Oct 24 05:38:53  kernel: [7824308.202905] Out of memory in UB 125883: OOM killed process 5944 (spamd child) score 0 vm:241512kB, rss:2740kB, swap:108428kB

Oct 24 05:38:54 n kernel: [7824309.132341] Out of memory in UB 125883: OOM killed process 27727 (spamd) score 0 vm:223584kB, rss:4840kB, swap:93488kB

Oct 24 05:38:55  kernel: [7824310.329803] Out of memory in UB 125883: OOM killed process 26635 (mysqld) score 0 vm:6794980kB, rss:34864kB, swap:59984kB

Oct 24 05:38:57  kernel: [7824311.478852] Out of memory in UB 125883: OOM killed process 8123 (php) score 0 vm:246188kB, rss:45852kB, swap:28876kB

Oct 24 05:39:03  kernel: [7824318.147747] Out of memory in UB 125883: OOM killed process 8166 (php) score 0 vm:247616kB, rss:41084kB, swap:36680kB

Oct 24 05:39:05  kernel: [7824319.621564] Out of memory in UB 125883: OOM killed process 8190 (php) score 0 vm:246916kB, rss:48248kB, swap:29012kB

Oct 24 05:39:08  kernel: [7824323.041045] Out of memory in UB 125883: OOM killed process 8155 (php) score 0 vm:246916kB, rss:41908kB, swap:35284kB

Oct 24 05:39:12  kernel: [7824326.960922] Out of memory in UB 125883: OOM killed process 8185 (php) score 0 vm:247320kB, rss:53124kB, swap:24508kB

Oct 24 05:39:14  kernel: [7824328.533322] Out of memory in UB 125883: OOM killed process 8186 (php) score 0 vm:246912kB, rss:50936kB, swap:26396kB

Oct 24 05:39:15  kernel: [7824329.412481] Out of memory in UB 125883: OOM killed process 8184 (php) score 0 vm:244384kB, rss:45620kB, swap:29516kB

Oct 24 05:52:46 PAM-hulk[9779]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES

Oct 24 05:52:50 PAM-hulk[9787]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES

Oct 24 05:52:53  PAM-hulk[9787]: Brute force detection active: 550 LOGIN DENIED -- TOO MANY FAILURES

Here's an example of the /var/log/exim_paniclog entry:

2018-10-25 00:37:12 1gFaC9-0004zB-G7 spam acl condition: all spamd servers failed
2018-10-25 00:37:33 1gFaCU-0004ze-UI spam acl condition: all spamd servers failed
2018-10-25 00:37:52 1gFaCm-00051L-Gj spam acl condition: all spamd servers failed
2018-10-25 00:37:54 1gFaDK-0004ze-1D malware acl condition: clamd /var/clamd : unable to connect to UNIX socket (/var/clamd): Connection refused
2018-10-25 00:38:24 1gFaDK-0004ze-1D spam acl condition: all spamd servers failed
2018-10-25 00:40:04 1gFaFP-0005C1-SN malware acl condition: clamd /var/clamd : unable to connect to UNIX socket (/var/clamd): Connection refused
2018-10-25 00:40:34 1gFaFP-0005C1-SN spam acl condition: all spamd servers failed
d

Thanks for your help!

 

[cPanelLauren]

These do show you're running out of memory plain and simple:

Oct 24 05:39:12  kernel: [7824326.960922] Out of memory in UB 125883: OOM killed process 8185 (php) score 0 vm:247320kB, rss:53124kB, swap:24508kB

Oct 24 05:39:14  kernel: [7824328.533322] Out of memory in UB 125883: OOM killed process 8186 (php) score 0 vm:246912kB, rss:50936kB, swap:26396kB

Oct 24 05:39:15  kernel: [7824329.412481] Out of memory in UB 125883: OOM killed process 8184 (php) score 0 vm:244384kB, rss:45620kB, swap:29516kB

I was hoping to see that this turned out to process memory limits but it's not - this is the system memory and you might want to check with your provider to see if there's anything they can do to assist you with this issue further.

 

[behzad neissari]

I update WHM in my centos server to CENTOS 7.6 andwhm 78.0.13 and after that all my email stay in Delivery Queue and cpanel-dovecot-solr is down and not start.

i try to restart service and get following message:

[root@centos ~]# systemctl status cpanel-dovecot-solr
● cpanel-dovecot-solr.service - Solr for cPanel Dovecot
Loaded: loaded (/etc/systemd/system/cpanel-dovecot-solr.service; disabled; ve ndor preset: disabled)
Active: inactive (dead)
[root@centos ~]# systemctl start cpanel-dovecot-solr
[root@centos ~]# systemctl status cpanel-dovecot-solr
● cpanel-dovecot-solr.service - Solr for cPanel Dovecot
Loaded: loaded (/etc/systemd/system/cpanel-dovecot-solr.service; disabled; ve ndor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-02-28 06:03:09 UTC; 4s ago
Process: 20134 ExecStop=/home/cpanelsolr/bin/solr stop (code=exited, status=1/ FAILURE)
Process: 19666 ExecStart=/home/cpanelsolr/bin/solr start -noprompt -h ${SOLR_H OSTNAME} -p ${SOLR_PORT} -m ${SOLR_MEMORY} -a ${SOLR_JVM_OPTS} (code=exited, sta tus=0/SUCCESS)
Process: 19655 ExecStartPre=/usr/local/cpanel/3rdparty/scripts/cpanel_dovecot_ solr_firewall start (code=exited, status=0/SUCCESS)
Main PID: 19861 (code=exited, status=143)

Feb 28 06:02:39 centos.example.com systemd[1]: Starting Solr for cPanel Dov....
Feb 28 06:02:50 centos.example.com solr[19666]: [194B blob data]
Feb 28 06:02:50 centos.example.com solr[19666]: Started Solr server on port...!
Feb 28 06:02:50 centos.example.com systemd[1]: Started Solr for cPanel Dovecot.
Feb 28 06:03:09 centos.example.com systemd[1]: cpanel-dovecot-solr.service:...a
Feb 28 06:03:09 centos.example.com systemd[1]: cpanel-dovecot-solr.service:...1
Feb 28 06:03:09 centos.example.com systemd[1]: Unit cpanel-dovecot-solr.ser....
Feb 28 06:03:09 centos.example.com systemd[1]: cpanel-dovecot-solr.service ....
Hint: Some lines were ellipsized, use -l to show in full.

Please help me

 

[GOT]

I would first start with removing the Solr plug in. See if that resolves the issues and if so then you could try re-installing it.

 

[cPanelLauren]

Hi @behzad neissari

This doesn't look like there's an error it appears that Dovecot Solr is disabled:

Loaded: loaded (/etc/systemd/system/cpanel-dovecot-solr.service; disabled; ve ndor preset: disabled)

You can enable it by going to WHM>>Service Configuration>>Service Manager -> Check Enabled + Monitor next to cpanel-dovecot-solr

This is most likely not the cause of the mail queue issues, what's present in /var/log/exim_mainlog for the mail in the queue?


Thanks!